Prompt filtering
How it works
An incoming prompt passes through a filtering layer placed in front of the model. The layer classifies the input: it matches patterns and terms against blocklists, runs a safety classifier that returns risk categories (e.g. violence, sexual content, hate, self-harm), or queries a separate judge model for policy compliance. Based on the result and decision thresholds, the prompt is passed unchanged, rewritten/sanitized, flagged for review, or rejected outright with a refusal message. Only a passing query reaches the generative model.
Problem solved
Generative models do not themselves enforce content policy or law, so an adversarial or careless user can steer them into producing harmful, illegal, or rights-infringing content. Prompt filtering addresses this at the earliest stage — it stops a risky input before the model performs any costly generation.
Components
A model that classifies the input content and returns risk categories with confidence scores.
Sets of terms, phrases, and patterns that are always blocked or always allowed.
Regular expressions and heuristics detecting known attack structures (e.g. jailbreak phrases).
A separate language model that judges a prompt’s compliance with content policy in ambiguous cases.
A defined set of rules and categories specifying what is disallowed and how the system responds.
Execution paradigm
The decision (pass / rewrite / reject) depends conditionally on the content of the specific prompt.
Parallelism
Each prompt is filtered independently, so filtering parallelizes across requests.