Embedded Evaluations
How it works
The AI system is instrumented so that each (or a sampled) model output is recorded as a trace together with its context. Evaluators are attached to these traces and run automatically: an LLM judge scoring quality/safety without a reference answer, smaller classifier models, or simple rules and regular expressions. In monitoring mode the evaluation results feed dashboards, anomaly detection, and alerts (online evaluation). In guardrail mode the check is built into the request–response path and executed synchronously: if a response fails the check it is blocked, corrected, or escalated to a human before reaching the user. Cost and latency are controlled via sampling, filters, and reserving the heaviest checks for critical risks. The results also feed a longer loop: building offline-evaluation datasets and tuning the system.
Problem solved
Offline evaluation on static datasets does not capture how a model behaves on real, shifting production traffic — new query types, drift, edge cases, and safety risks. Embedded evaluations solve this by scoring outputs at the point and time they are produced and enabling a reaction (alert or block) before a bad response reaches the user.