Infrastructure

AI Gateway

2023ActivePublished: 27 July 2026Updated: 27 July 2026Published

Key innovation

Introduces a single, unified access point (proxy) between an application and multiple LLM providers, moving routing, failover, caching, cost control, security and observability out of application code into an infrastructure layer.

How it works

1) The application sends a request to the gateway using a single, unified API (usually OpenAI-compatible) instead of a specific provider's API. 2) The gateway authenticates the request (e.g. with a virtual key) and enforces policies: rate limits, budgets, input guardrails. 3) The routing layer selects the target model/provider based on rules (cost, latency, availability, load balancing, semantic matching). 4) A cache (exact or semantic) is optionally checked; a hit returns a response without calling the model. 5) The gateway translates the request into the selected provider's native format and forwards it (with streaming support). 6) On error, retries and failover to a backup model are triggered. 7) The response is normalized back to the unified format, passes output guardrails, is optionally cached, and metrics (tokens, cost, latency) are recorded.

Problem solved

Integrating an application directly with multiple LLM providers leads to fragmentation: each provider has a different API, authentication scheme and response format. This causes duplicated integration code, no central control over cost and limits, poor observability, exposure to single-provider outages and rate limits, and scattered key and security management. An LLM gateway centralizes these concerns in a single layer.

Components

Unified API LayerEntry point and format-translation layer.

Exposes one consistent interface (usually OpenAI-compatible) and normalizes requests and responses regardless of provider.

INUnified chat/completions/embeddings request.

OUTResponse in a unified format, with streaming support.

OpenAI-compatibleMost common variant — emulates the OpenAI API.

Native passthroughPasses requests through in the provider's native format.

Official

Router & Load BalancerTraffic-routing decision.

Selects the target model/provider based on rules: cost, latency, availability, load weights or semantic matching.

INRequest metadata and route configuration.

OUTSelected backend decision.

Latency-basedRoutes to the fastest available backend.

Cost-basedOptimizes request cost.

SemanticSelects a model based on the query's content/intent.

Official

Response CacheCost and latency reduction.

Stores responses to reduce cost and latency; supports exact (hash) and semantic (vector-similarity) caching.

INRequest hash or embedding.

OUTCached response or pass-through to the model.

Exact cacheHit only for identical requests.

Semantic cacheHit for semantically similar requests (similarity threshold).

Official

Policy & Governance EngineAccess, cost and limit control.

Authenticates requests (virtual keys) and enforces rate limits, budgets and access control.

INIdentity and request metadata.

OUTPolicy decision and budget/limit update.

Virtual keysVirtual keys with attached budget and permissions.

Token bucket rate limitingTraffic limiting via the token-bucket algorithm.

Official

Guardrails EngineContent safety and compliance.

Inspects input and output content: moderation, sensitive-data (PII) detection and redaction, blocking of disallowed requests.

INInput text or model output.

OUTGuardrail-check result.

PII redactionDetection and masking of personal data.

Content moderationBlocking policy-violating content.

Official

Observability & TelemetryMonitoring and auditing.

Records logs, metrics and traces: token usage, cost, latency, error rates and cache-hit rates.

INMetadata for each call.

OUTData exported to monitoring systems.

OpenTelemetry exportStandardized export of traces and metrics.

Official

Provider AdaptersIntegration with specific providers.

Translate normalized requests into each provider's native API and back, including streaming handling and error mapping.

INRequest in the unified format.

OUTRequest translated for the provider.

Cloud provider adapterAdapter for OpenAI/Anthropic/Google, etc.

Self-hosted adapterAdapter for self-hosted models (e.g. vLLM).

Official

Implementation

Reference implementations

Cloudflare AI Gateway

Hosted service · Cloudflare

Implementation pitfalls

Added latency and network hopMedium

The gateway adds another hop between the application and the provider, increasing latency.

Fix:Co-locate with the app/provider, use connection pooling, transparent streaming and a lightweight processing overhead.

Single point of failure (SPOF)High

All LLM traffic passes through the gateway; its failure blocks every call.

Fix:High-availability deployment (multiple instances, multiple regions), health checks and circuit breakers.

Incorrect semantic-cache hitsMedium

Too low a similarity threshold returns irrelevant, stale responses.

Fix:Tune the similarity threshold, use short TTLs and scope the cache key (e.g. per user/context).

Lowest-common-denominator APIMedium

A unified API can hide provider-specific features (e.g. tools, reasoning modes).

Fix:Pass through native parameters and offer a native mode alongside the OpenAI-compatible mode.

Token-accounting and streaming complexityMedium

Accurate token and cost accounting across streaming and different tokenizers is difficult.

Fix:Use provider-matched tokenizers, count on the response side and reconcile with provider billing.

Key sprawl and securityHigh

The gateway centralizes provider keys, becoming a high-value target.

Fix:Virtual keys for clients, secret storage in a vault, key rotation and least-privilege access.

Evolution

2015

API Gateway pattern popularized in microservices

API gateways (e.g. Kong, Netflix Zuul) become a standard pattern for centralizing traffic in microservices architectures — the foundation for later LLM gateways.

2020

Commercial LLM APIs (GPT-3)

OpenAI's release of the GPT-3 API kicks off a proliferation of proprietary language-model APIs and the integration-fragmentation problem.

2023

First dedicated LLM gateways

Inflection point

Tools such as LiteLLM and Portkey, plus MLflow AI Gateway, emerge and standardize access to multiple LLM providers.

2024

AI gateways from cloud and API-management vendors

Cloudflare AI Gateway and Kong AI Gateway bring LLM gateways into mainstream infrastructure; semantic routing, guardrails and semantic caching become common.

Hyperparameters (configurable axes)

Routing strategyCritical

Rule for selecting the provider/model for a request.

latency-basedFastest available backend.

cost-basedCheapest suitable model.

round-robin / weightedLoad balancing.

semanticSelection by request content.

Fallback chainHigh

Ordered list of backup models used after an error or rate-limit hit.

[gpt-4o, claude-3-5-sonnet, gemini-1.5-pro]Order of attempts.

CachingMedium

Cache mode and its parameters (TTL, similarity threshold for semantic cache).

exact, ttl=3600sExact cache.

semantic, threshold=0.95Semantic cache with a similarity threshold.

Rate limits & budgetsHigh

Rate limits (RPM/TPM) and cost budgets per key, user or team.

1000 RPM / 100k TPMRequests and tokens per minute.

$50/dzień per kluczDaily budget per key.

Retries & timeoutsHigh

Retry count, backoff strategy and request timeouts.

retries=3, backoff=exponentialExponential backoff.

timeout=30sPer-request timeout.

Load-balancing weightsMedium

Weights distributing traffic across equivalent deployments/keys.

azure:70 / openai:3070/30 traffic split.

Guardrail policiesMedium

Moderation, PII-redaction and content-blocking rules on input/output.

pii_redaction=onSensitive-data redaction.

moderation=blockBlocking disallowed content.

Computational complexity

Time complexity: O(1) na żądanie (narzut proxy/routingu). Space complexity: O(P + C).

Compute bottleneck

Network latency and the extra hop

The gateway is I/O-bound; the main overhead is the extra network hop to the provider and — with semantic caching/routing — vector-similarity search.

Depends on

Latencja dostawcyRozmiar indeksu semantycznegoWspółbieżność

Execution paradigm

Primary mode

Conditional

The execution path depends on request content and routing rules; a single backend is activated (with optional failover).

Activation pattern

Input dependent

Routing mechanism

For each request the gateway conditionally selects a single provider/model from the configured routes.

Parallelism

Parallelism level

Fully parallel

A stateless gateway scales horizontally; requests are independent and processed in parallel.

Scope

InferenceAcross devices

Constraints

!The cache and rate-limit/budget counters require shared storage (e.g. Redis) to stay consistent across gateway instances.

Hardware requirements

Primary

The gateway is a stateless, I/O-bound network service; it runs on commodity CPUs and scales horizontally. The actual inference happens on the provider side.

Limited

GPUs bring no benefit to the gateway; the only exception is a local embedding model used for semantic routing or caching.