PROJECT / AI / KUBERNETES INCIDENT RESPONSE
PrometheusAI.
PrometheusAI is a Kubernetes incident-response demo that investigates alerts with observability data and read-only cluster tools, proposes an allowlisted remediation, and validates it with a server-side dry run. A separate policy service checks the exact action and human approval before executing it, then records the decision and verifies the outcome. It supports deterministic fake-model, Ollama, and Bedrock backends.
WHAT IT DOES
- Investigate alerts using Prometheus, Loki, runbooks, and read-only Kubernetes tools.
- Limit remediations to allowlisted restart, undo, and scale operations.
- Require a successful server-side dry run and human approval bound to the action hash.
- Separate agent read access from the policy service's Kubernetes write identity.
- Record decisions and verify execution outcomes with deterministic evaluation and tests.
SKILLS LEARNED
- Designing stateful AI agent workflows with LangGraph and pluggable model backends.
- Separating LLM-generated proposals from deterministic authorization and execution.
- Applying least-privilege Kubernetes RBAC and isolating services with NetworkPolicies.
- Building Python APIs and CLI workflows for incident investigation and approvals.
- Collecting observability evidence from Prometheus, Loki, and Kubernetes.
- Testing safety boundaries with pytest, policy tests, and deterministic scenario evaluations.
TECHNOLOGIES
PythonLangGraphFastAPIKubernetesPrometheusLokiPostgreSQLpgvectorpytest