LiteLLM

LiteLLM is an open-source Python library and self-hosted proxy (MIT License) providing a unified OpenAI-compatible interface for calling 100+ LLM providers: OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Groq, Together, Cohere, Mistral, Ollama, xAI, DeepSeek, Alibaba Qwen, Perplexity, Hugging Face, and many more. Created by BerriAI Inc. (YC W23), first released in July 2023, currently ~19-20k GitHub stars and one of the most-used open-source Python AI projects. Install: uv add litellm or pip install litellm. Proxy Server: uv tool install 'litellm[proxy]' or Docker docker.litellm.ai/berriai/litellm:latest.

Two main usage modes: (1) Python SDK - drop-in replacement for the OpenAI client. Write code once, switch models by changing the string (model='openai/gpt-4o' → model='anthropic/claude-3-5-sonnet-20241022'). All responses in the OpenAI Chat Completions format. Exception handling mapped to OpenAI types (AuthenticationError, RateLimitError, APIError) - existing code works out of the box. (2) Proxy Server / LLM Gateway - self-hosted OpenAI-compatible gateway on port 4000. The client (OpenAI SDK, LangChain, any OpenAI-compatible SDK) connects to the proxy instead of directly to the provider. The proxy handles: virtual keys, cost tracking, guardrails, admin UI, load balancing.

Key enterprise features: Router - retry, fallback, and load balancing across multiple deployments of the same model (e.g. Azure GPT-4 East + Azure GPT-4 West + OpenAI direct); Virtual Keys - the proxy generates short-lived per-team/user keys with budget and rate limits; Cost Tracking - per-request cost calculation for 100+ models, aggregated by tags/keys/teams with alerts; Observability - single-line integrations with Langfuse, MLflow, Helicone, Lunary, OpenTelemetry, PostHog, Datadog via litellm.success_callback; Guardrails - PII masking, content filtering, prompt injection detection, custom callbacks. Admin UI - a dashboard for spend, users, keys, and teams monitoring.

Agent and MCP Gateway: LiteLLM is not just an LLM proxy - a single endpoint handles LLMs, A2A (agent-to-agent), and MCP tools. A2A Agents - an agent from any provider is invoked identically to an LLM (through the completion() interface). MCP Gateway - a central MCP endpoint with per-key access control, tool filtering, and audit logs. Eliminates the need for a separate agent gateway or MCP gateway - LiteLLM consolidates the layers. Support for a debug tool /utils/transform_request that shows exactly what LiteLLM sends to a provider (useful for debugging prompt formatting, headers, provider-specific params).

LiteLLM Enterprise: paid tier for production enterprise deployment. Includes: SSO/SAML (OneLogin, Okta, Auth0, Google Workspace, Azure AD/Microsoft Entra), advanced audit logs (immutable, per-request), hardened guardrails (fine-tuned PII masking, prompt injection detection), fine-grained RBAC and multi-team management, priority 24/7 support. Trust Center at trust.litellm.ai with SOC 2 Type II and GDPR compliance. Enterprise Docker registry: docker.litellm.ai/berriai/litellm-enterprise. Competition: OpenRouter (managed cloud + markup), Portkey, Vercel AI Gateway, Helicone AI Gateway. LiteLLM wins on a self-hosted-first policy and provider breadth. Enterprise customers: fintech, healthcare, government (data residency + sovereign cloud + audit compliance).