Robots Atlas>ROBOTS ATLAS
Evaluation

SRE-Bench

2025ExperimentalPublished: 4 September 2026Updated: 4 September 2026Published
Key innovation
An open, reproducible benchmark that ports the SWE-bench idea from code repair to Site Reliability Engineering โ€” evaluating AI agents on realistic, cascading production incidents in Kubernetes clusters.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)Agent runtime
Use cases
Evaluating AI/LLM agents on Kubernetes incident responseBenchmarking autonomous SRE / DevOps agentsTesting root-cause analysis (RCA) in cloud-native environmentsMeasuring agent remediation safety and MTTRObservability triage (Prometheus, alerts) by agentsProof-of-concept environment for autonomous Kubernetes management

How it works

SRE-Bench runs an agent inside a Kubernetes environment where fault injectors introduce a defined incident scenario (e.g. a stale ConfigMap causing ArgoCD drift and CrashLoopBackOff, an expired secret rotation leading to database authentication failures, or node pressure with a misconfigured HPA). Each scenario documents the primary trigger, propagation path, impact, detection signals, and mitigation steps. The agent observes cluster state (kubectl, events, Prometheus metrics, logs) and must diagnose the root cause and perform safe remediation. The evaluation harness scores the run using operational metrics: time-to-diagnose, safe remediation rate, MTTR, and explainability of decisions. Scenarios are modular and community-driven โ€” users can contribute their own test cases.

Problem solved

Existing AI agent benchmarks (e.g. SWE-bench) measure code repair but do not capture the reality of reliability engineering, where the task is diagnosing and remediating live, cascading incidents in a running production system. There was no open, reproducible environment to assess whether an agent can safely identify the root cause of a Kubernetes failure and carry out remediation without making things worse. SRE-Bench fills this gap by providing standardized failure scenarios and an evaluation harness.

Components

Scenario libraryTest tasks / incident cases

A set of 17 real-world incident scenarios simulating cascading failures in Kubernetes, GitOps, and cloud-native environments โ€” from ImagePullBackOff to multi-layer failure chains.

Fault injectorsIncident trigger

Mechanisms that introduce a defined fault into the cluster (e.g. configuration drift, expired secrets, node pressure).

Kubernetes manifestsBaseline environment

Declarative manifests defining the baseline application and infrastructure state for a scenario.

Official

Observability specsDetection signals

Definitions of detection signals โ€” Prometheus metrics, events, and alerts โ€” that the agent uses to diagnose.

Official

Evaluation harnessScoring and evaluation

A scoring layer measuring agent performance via time-to-diagnose, safe remediation rate, MTTR, and explainability.

Baseline agentsReference baseline

Reference agent implementations serving as a baseline for comparing custom agents.

Official

Implementation

Evolution

2025
Open-source release of SRE-Bench by Agentkube
Inflection point

Agentkube releases SRE-Bench on GitHub (MIT license) as an SRE agent benchmark inspired by SWE-bench, with a library of 17 Kubernetes incident scenarios.

SRE-bench โ€” GitHub repository (paper)

Hardware requirements

Primary

SRE-Bench is a benchmark/evaluation environment running on a Kubernetes cluster โ€” it is not tied to a specific accelerator type; hardware requirements depend on the agent under test.