Robots Atlas>ROBOTS ATLAS
Evaluation

Embedded Evaluations

ActivePublished: 17 November 2026Updated: 17 November 2026Published
Key innovation
Moving evaluation from a pre-deployment phase on static datasets into the running system: model outputs are scored live on real traffic, closing the loop between production and quality control.
Category
Evaluation
Abstraction level
Pattern
Operation level
Evaluation (runtime)Application
Use cases
Live quality monitoring of LLM responses on production trafficGuardrails blocking unsafe or policy-violating responses in real timeDetecting anomalies, drift, and quality regressions in production, with alertingEvaluating AI agent actions (step correctness, tool use) at runtimeBuilding offline-evaluation datasets from real production traces

How it works

The AI system is instrumented so that each (or a sampled) model output is recorded as a trace together with its context. Evaluators are attached to these traces and run automatically: an LLM judge scoring quality/safety without a reference answer, smaller classifier models, or simple rules and regular expressions. In monitoring mode the evaluation results feed dashboards, anomaly detection, and alerts (online evaluation). In guardrail mode the check is built into the request–response path and executed synchronously: if a response fails the check it is blocked, corrected, or escalated to a human before reaching the user. Cost and latency are controlled via sampling, filters, and reserving the heaviest checks for critical risks. The results also feed a longer loop: building offline-evaluation datasets and tuning the system.

Problem solved

Offline evaluation on static datasets does not capture how a model behaves on real, shifting production traffic — new query types, drift, edge cases, and safety risks. Embedded evaluations solve this by scoring outputs at the point and time they are produced and enabling a reaction (alert or block) before a bad response reaches the user.