1) The application sends a request to the gateway using a single, unified API (usually OpenAI-compatible) instead of a specific provider's API. 2) The gateway authenticates the request (e.g. with a virtual key) and enforces policies: rate limits, budgets, input guardrails. 3) The routing layer selects the target model/provider based on rules (cost, latency, availability, load balancing, semantic matching). 4) A cache (exact or semantic) is optionally checked; a hit returns a response without calling the model. 5) The gateway translates the request into the selected provider's native format and forwards it (with streaming support). 6) On error, retries and failover to a backup model are triggered. 7) The response is normalized back to the unified format, passes output guardrails, is optionally cached, and metrics (tokens, cost, latency) are recorded.
Integrating an application directly with multiple LLM providers leads to fragmentation: each provider has a different API, authentication scheme and response format. This causes duplicated integration code, no central control over cost and limits, poor observability, exposure to single-provider outages and rate limits, and scattered key and security management. An LLM gateway centralizes these concerns in a single layer.
Exposes one consistent interface (usually OpenAI-compatible) and normalizes requests and responses regardless of provider.
Official
Selects the target model/provider based on rules: cost, latency, availability, load weights or semantic matching.
Official
Stores responses to reduce cost and latency; supports exact (hash) and semantic (vector-similarity) caching.
Official
Authenticates requests (virtual keys) and enforces rate limits, budgets and access control.
Official
Inspects input and output content: moderation, sensitive-data (PII) detection and redaction, blocking of disallowed requests.
Official
Records logs, metrics and traces: token usage, cost, latency, error rates and cache-hit rates.
Official
Translate normalized requests into each provider's native API and back, including streaming handling and error mapping.
Official
The gateway adds another hop between the application and the provider, increasing latency.
All LLM traffic passes through the gateway; its failure blocks every call.
Too low a similarity threshold returns irrelevant, stale responses.
A unified API can hide provider-specific features (e.g. tools, reasoning modes).
Accurate token and cost accounting across streaming and different tokenizers is difficult.
The gateway centralizes provider keys, becoming a high-value target.
API gateways (e.g. Kong, Netflix Zuul) become a standard pattern for centralizing traffic in microservices architectures — the foundation for later LLM gateways.
OpenAI's release of the GPT-3 API kicks off a proliferation of proprietary language-model APIs and the integration-fragmentation problem.
Tools such as LiteLLM and Portkey, plus MLflow AI Gateway, emerge and standardize access to multiple LLM providers.
Cloudflare AI Gateway and Kong AI Gateway bring LLM gateways into mainstream infrastructure; semantic routing, guardrails and semantic caching become common.
Time complexity: O(1) na żądanie (narzut proxy/routingu). Space complexity: O(P + C).
The gateway is I/O-bound; the main overhead is the extra network hop to the provider and — with semantic caching/routing — vector-similarity search.
Rule for selecting the provider/model for a request.
Ordered list of backup models used after an error or rate-limit hit.
Cache mode and its parameters (TTL, similarity threshold for semantic cache).
Rate limits (RPM/TPM) and cost budgets per key, user or team.
Retry count, backoff strategy and request timeouts.
Weights distributing traffic across equivalent deployments/keys.
Moderation, PII-redaction and content-blocking rules on input/output.
The execution path depends on request content and routing rules; a single backend is activated (with optional failover).
For each request the gateway conditionally selects a single provider/model from the configured routes.
A stateless gateway scales horizontally; requests are independent and processed in parallel.
The gateway is a stateless, I/O-bound network service; it runs on commodity CPUs and scales horizontally. The actual inference happens on the provider side.
GPUs bring no benefit to the gateway; the only exception is a local embedding model used for semantic routing or caching.
