Robots Atlas>ROBOTS ATLAS
Infrastructure

API key rate limiting

ActivePublished: 28 July 2026Updated: 28 July 2026Published
Key innovation
Ties rate limits and quotas to an individual API key, enabling fair resource sharing, backend protection and per-client cost control without changing the underlying model/services.
Category
Infrastructure
Abstraction level
Pattern
Operation level
ServingApplicationSystem
Use cases
Protecting LLM APIs from abuse and overloadFair resource sharing across clients/plansCost control (budgets, per-key quotas)Enforcing pricing tiersPreventing token reselling/abuseStability of API gateways and model backends

How it works

1) Each request is identified by its API key (and possibly user/plan). 2) The gateway checks the usage counters for that key against configured limits (e.g. RPM, TPM, quota) using an algorithm (token bucket / sliding window). 3) If the limit is not exceeded — the request passes and counters are updated (in shared storage, e.g. Redis). 4) If the limit is exceeded — the request is rejected (HTTP 429, often with a Retry-After header) or delayed/queued.

Problem solved

Without limits, a single client can overload the backend, exhaust provider quotas or run up uncontrolled costs. Per-API-key rate limiting protects the system, ensures fair access and keeps costs under control.

Components

Usage counterState / measurement

Stores current usage per key (requests/tokens in a window), usually in shared storage (e.g. Redis) for cross-instance consistency.

Official

Limiting algorithmDecision

Decides whether a request fits within the limit: token bucket, leaky bucket or sliding window.

Token bucketAllows bursts up to the bucket capacity.
Sliding windowMore accurate counting over a moving time window.

Official

Policy and enforcement pointEnforcement

Definition of limits per key/plan (RPM/TPM/quota) and the point where a request is allowed, rejected (429) or delayed.

Official

Implementation

Reference implementations
one-api (token/limit management)
songquanpeng
Kong / NGINX rate limiting
Kong Inc.
Implementation pitfalls
Accuracy vs performance (distributed)Medium

Strict global limits require a consistent counter, adding latency; local counters are faster but less accurate.

Fix:Shared store (Redis) with atomic ops, approximate counting, key sharding.
Thundering herd after window resetMedium

With a fixed window, all clients may hit simultaneously after reset, causing a load spike.

Fix:Sliding window, jitter, token bucket.
Unclear limit responsesLow

Missing clear headers (limit/remaining/Retry-After) makes it hard for clients to handle 429 correctly.

Fix:Standard rate-limit headers and Retry-After, documentation.

Hyperparameters (configurable axes)

Limit dimensionsCritical

What is limited: requests (RPM/RPD), tokens (TPM), concurrency, budget.

1000 RPM / 100k TPMRequests and tokens per minute.
$50/dzienCost budget.
AlgorithmHigh

Token bucket, leaky bucket, fixed/sliding window.

token bucketHandles bursts.
ScopeHigh

Per API key, per user, per plan/group, global.

Computational complexity

Time complexity: O(1) na zadanie. Space complexity: O(liczba aktywnych kluczy).

Compute bottleneck

Counter consistency in distributed mode

Maintaining accurate, consistent counters across many gateway instances at low latency is the main challenge (accuracy vs performance trade-off).

Depends on
Wspoldzielony magazyn (np. Redis)

Execution paradigm

Primary mode
Conditional

The outcome depends on the current usage of the given key.

Activation pattern
Input dependent
Routing mechanism

An allow/deny decision based on counters, without routing.

Parallelism

Parallelism level
Fully parallel

Checks are independent per request; the challenge is global counter consistency.

Scope
InferenceAcross devices
Constraints
!Accurate global limits require a shared, consistent counter across instances.

Hardware requirements

Primary

Limit checking is a lightweight, I/O-bound operation; it runs on any CPU. The key is a fast, shared counter store (e.g. Redis).