Safety Classifier
How it works
1) A risk taxonomy (categories such as violence, sexual content, hate, self-harm, illegal activity, etc.) and policies are defined. 2) A classifier is trained — from lightweight embedding-based models to fine-tuned LLMs (e.g. Llama Guard on Llama 2, ShieldGemma on Gemma 2) — on data labelled against that taxonomy, often using synthetic data and active learning. 3) At runtime the classifier evaluates the user input (prompt classification) and/or the model output (response classification), returning a safe/unsafe label, categories and a score. 4) Based on a decision threshold, the orchestration layer blocks, allows, redacts or escalates the content. The classifier runs as a separate pipeline step, so it can be swapped and updated without changing the main model.
Problem solved
Tuning the LLM itself (e.g. with RLHF) does not guarantee that every harmful request is refused or that every generated answer is safe, and changing the model's behaviour is slow and expensive. A fast, updatable and auditable layer is needed to independently filter harmful inputs and outputs and to catch jailbreaks, without retraining the main model.
Components
A structured set of harmful-content categories (e.g. violence, hate, self-harm) and decision rules.
Official
Ranges from a lightweight embedding-based classifier to a fine-tuned LLM guard (e.g. Llama Guard 7B, ShieldGemma 2B/9B).
Official
Invokes the classifier before and after the main model's generation.
Thresholding and enforcement logic based on the score and categories.
Official
Implementation
Every request needs an extra classifier pass (often two: input and output), increasing latency and cost.
An overly aggressive threshold or taxonomy blocks safe content and hurts usability.
A classifier trained on a different taxonomy may not match the product's safety policy.
Adversarial inputs (jailbreaks, obfuscation) can bypass the classifier just as they bypass the main model.
Accuracy drops for languages and domains underrepresented in the training data.
Evolution
The 'A Holistic Approach to Undesired Content Detection' paper formalizes building a dedicated moderation classifier.
Meta releases Llama Guard (on Llama 2-7B) with a customizable risk taxonomy and separate prompt and response classification.
Google publishes a set of classifiers (2B/9B/27B) scoring prompts and responses across four harm categories.
Classifiers trained on synthetic data generated from a constitution; reported ~23.7% inference overhead and +0.38% refusals on production traffic.
Hyperparameters (configurable axes)
The set of risk categories the classifier recognizes; some models allow customizing it (zero/few-shot).
The score cutoff above which content is blocked — a trade-off between false positives and false negatives.
Whether only the input, only the output, or both are screened.
Trade-off between a lightweight classifier (low latency) and a large LLM guard (higher accuracy).
Computational complexity
Execution paradigm
A typical classifier runs a dense forward pass (all parameters active) for each piece of content it evaluates.
Parallelism
The classifier itself is a standard inference pass (parallel internally and across requests), but as a separate pipeline stage it adds sequential latency before/after the main model's generation.
Hardware requirements
Fine-tuned LLM-based classifiers (Llama Guard, ShieldGemma) are standard Transformer inference, accelerated by tensor cores.
Lightweight classifiers (e.g. embedding-based or small models) can run on CPU at acceptable latency.