Robots Atlas>ROBOTS ATLAS
Safety

Safety Classifier

2022ActivePublished: 18 November 2026Updated: 18 November 2026Published
Key innovation
Moving content-safety judgment into a separate, specialized classifier that runs alongside the main model, instead of relying solely on the base model's own alignment training (RLHF).
Category
Safety
Abstraction level
Technika
Operation level
InferenceSystemSecurity operations
Use cases
Moderation of LLM-generated contentUser input filtering (prompt screening)Model output filteringDetecting jailbreak and prompt-injection attemptsGuardrails in AI applications and agentsEnforcing safety and compliance policies

How it works

1) A risk taxonomy (categories such as violence, sexual content, hate, self-harm, illegal activity, etc.) and policies are defined. 2) A classifier is trained — from lightweight embedding-based models to fine-tuned LLMs (e.g. Llama Guard on Llama 2, ShieldGemma on Gemma 2) — on data labelled against that taxonomy, often using synthetic data and active learning. 3) At runtime the classifier evaluates the user input (prompt classification) and/or the model output (response classification), returning a safe/unsafe label, categories and a score. 4) Based on a decision threshold, the orchestration layer blocks, allows, redacts or escalates the content. The classifier runs as a separate pipeline step, so it can be swapped and updated without changing the main model.

Problem solved

Tuning the LLM itself (e.g. with RLHF) does not guarantee that every harmful request is refused or that every generated answer is safe, and changing the model's behaviour is slow and expensive. A fast, updatable and auditable layer is needed to independently filter harmful inputs and outputs and to catch jailbreaks, without retraining the main model.

Components

Safety taxonomy & policiesDefines the risk categories and rules against which content is judged.

A structured set of harmful-content categories (e.g. violence, hate, self-harm) and decision rules.

Official

Classifier modelPredicts a safety label for the content being evaluated.

Ranges from a lightweight embedding-based classifier to a fine-tuned LLM guard (e.g. Llama Guard 7B, ShieldGemma 2B/9B).

Official

Input/output screening stagePipeline point: evaluates the input before the model and/or the output after it.

Invokes the classifier before and after the main model's generation.

Decision threshold & policy enforcementMaps the classifier score to an action: block / allow / redact / escalate.

Thresholding and enforcement logic based on the score and categories.

Official

Implementation

Implementation pitfalls
Added latency and costMedium

Every request needs an extra classifier pass (often two: input and output), increasing latency and cost.

Fix:Smaller/distilled classifier, batching, asynchronous output screening, streaming with interruption.
Over-refusal (false positives)High

An overly aggressive threshold or taxonomy blocks safe content and hurts usability.

Fix:Threshold calibration, evaluation on control sets, per-category thresholds, regression tests.
Taxonomy–policy mismatchMedium

A classifier trained on a different taxonomy may not match the product's safety policy.

Fix:Fine-tune to your own taxonomy or use models with a customizable taxonomy (e.g. Llama Guard).
Evasion of the classifier itselfHigh

Adversarial inputs (jailbreaks, obfuscation) can bypass the classifier just as they bypass the main model.

Fix:Adversarial training, red-teaming, synthetic data, constitutional-classifier-style approaches.
Language and domain coverage gapsMedium

Accuracy drops for languages and domains underrepresented in the training data.

Fix:Augment data with missing languages/domains, monitor accuracy in production.

Evolution

Original paper · 2022 · Todor Markov
A Holistic Approach to Undesired Content Detection in the Real World
Todor Markov, Chong Zhang, Sandhini Agarwal, Lilian Weng
2022
OpenAI — undesired-content detection classifier (basis of the Moderation API)
Inflection point

The 'A Holistic Approach to Undesired Content Detection' paper formalizes building a dedicated moderation classifier.

2023
Llama Guard — LLM-based input/output safety classifier
Inflection point

Meta releases Llama Guard (on Llama 2-7B) with a customizable risk taxonomy and separate prompt and response classification.

2024
ShieldGemma — family of moderation classifiers based on Gemma 2

Google publishes a set of classifiers (2B/9B/27B) scoring prompts and responses across four harm categories.

2025
Anthropic — constitutional classifiers defending against universal jailbreaks
Inflection point

Classifiers trained on synthetic data generated from a constitution; reported ~23.7% inference overhead and +0.38% refusals on production traffic.

Hyperparameters (configurable axes)

Safety taxonomyCritical

The set of risk categories the classifier recognizes; some models allow customizing it (zero/few-shot).

Decision thresholdHigh

The score cutoff above which content is blocked — a trade-off between false positives and false negatives.

Screening scope (input/output)High

Whether only the input, only the output, or both are screened.

Classifier size / latency budgetMedium

Trade-off between a lightweight classifier (low latency) and a large LLM guard (higher accuracy).

Computational complexity

Execution paradigm

Primary mode
Dense

A typical classifier runs a dense forward pass (all parameters active) for each piece of content it evaluates.

Activation pattern
All paths active

Parallelism

Parallelism level
Partially parallel

The classifier itself is a standard inference pass (parallel internally and across requests), but as a separate pipeline stage it adds sequential latency before/after the main model's generation.

Scope
Inference

Hardware requirements

Primary

Fine-tuned LLM-based classifiers (Llama Guard, ShieldGemma) are standard Transformer inference, accelerated by tensor cores.

Possible

Lightweight classifiers (e.g. embedding-based or small models) can run on CPU at acceptable latency.