SGLang
High-performance, open-source (Apache 2.0) serving framework for LLMs and multimodal models; known for RadixAttention and a zero-overhead scheduler. Hosted by LMSYS.
Description
SGLang is a high-performance serving framework for large language models and multimodal models, designed for low-latency, high-throughput inference across hardware configurations — from a single GPU to large distributed clusters. Released under the Apache 2.0 license, it is hosted under LMSYS (a non-profit open-source organization), with contributions from NVIDIA, AMD, Intel and xAI, among others.
Key features include RadixAttention (prefix caching), a zero-overhead CPU scheduler, prefill-decode disaggregation and speculative decoding, continuous batching with paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, multiple quantization formats (FP4/FP8/INT4/AWQ/GPTQ) and multi-LoRA batching.
SGLang is deployed at scale across 400,000+ GPUs globally, trusted by enterprises such as Google Cloud, Microsoft Azure, AWS, Alibaba and Tencent, and by leading research institutions (Stanford, MIT, UC Berkeley).