vLLM
Fast, open-source (Apache 2.0) library and server for LLM inference and serving; known for PagedAttention and continuous batching.
Description
vLLM is a fast and easy-to-use open-source library for LLM inference and serving, released under the Apache 2.0 license. It was created at UC Berkeley's Sky Computing Lab and is now maintained by a community of more than 2,000 contributors across academia and industry (a PyTorch Foundation hosted project since 2025).
Its key innovation is PagedAttention — efficient memory management for attention key-value data. vLLM uses continuous batching with chunked prefill and prefix caching, provides an OpenAI-compatible API server (plus the Anthropic Messages API and gRPC), supports quantization (FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors) and distributed inference (tensor, pipeline, data, expert and context parallelism).
It supports a wide range of hardware: NVIDIA and AMD GPUs, x86/ARM/PowerPC CPUs, and specialized accelerators.