SRE-Bench runs an agent inside a Kubernetes environment where fault injectors introduce a defined incident scenario (e.g. a stale ConfigMap causing ArgoCD drift and CrashLoopBackOff, an expired secret rotation leading to database authentication failures, or node pressure with a misconfigured HPA). Each scenario documents the primary trigger, propagation path, impact, detection signals, and mitigation steps. The agent observes cluster state (kubectl, events, Prometheus metrics, logs) and must diagnose the root cause and perform safe remediation. The evaluation harness scores the run using operational metrics: time-to-diagnose, safe remediation rate, MTTR, and explainability of decisions. Scenarios are modular and community-driven โ users can contribute their own test cases.
Existing AI agent benchmarks (e.g. SWE-bench) measure code repair but do not capture the reality of reliability engineering, where the task is diagnosing and remediating live, cascading incidents in a running production system. There was no open, reproducible environment to assess whether an agent can safely identify the root cause of a Kubernetes failure and carry out remediation without making things worse. SRE-Bench fills this gap by providing standardized failure scenarios and an evaluation harness.
A set of 17 real-world incident scenarios simulating cascading failures in Kubernetes, GitOps, and cloud-native environments โ from ImagePullBackOff to multi-layer failure chains.
Mechanisms that introduce a defined fault into the cluster (e.g. configuration drift, expired secrets, node pressure).
Declarative manifests defining the baseline application and infrastructure state for a scenario.
Official
Definitions of detection signals โ Prometheus metrics, events, and alerts โ that the agent uses to diagnose.
Official
A scoring layer measuring agent performance via time-to-diagnose, safe remediation rate, MTTR, and explainability.
Reference agent implementations serving as a baseline for comparing custom agents.
Official
Agentkube releases SRE-Bench on GitHub (MIT license) as an SRE agent benchmark inspired by SWE-bench, with a library of 17 Kubernetes incident scenarios.
SRE-Bench is a benchmark/evaluation environment running on a Kubernetes cluster โ it is not tied to a specific accelerator type; hardware requirements depend on the agent under test.
