Inference Endpoints enable deployment of any model from Hugging Face Hub to dedicated, fully managed production infrastructure with autoscaling and scale-to-zero in a few clicks, eliminating the need to manually manage containers, scaling, and security.
An Inference Endpoint is built from a model hosted on the Hugging Face Hub. The user selects a model, task, cloud provider, region, instance type, and security settings, while Hugging Face manages the full lifecycle of the container running the model. The endpoint exposes an HTTP API and can be configured for autoscaling, scale-to-zero, private networking, analytics, and version updates. Integration with the huggingface_hub library also enables programmatic creation and management of endpoints.
Inference Endpoints address the complexity and cost of deploying AI models to production. Without such a service, teams must independently manage containers, environment configuration, scaling, security, monitoring, and infrastructure selection. Inference Endpoints simplify this process by providing a ready-made, managed inference layer for models from the Hub, enabling teams to deploy models faster and serve production traffic without building their own serving platform from scratch.
Stores versioned model weights and configuration files on Hugging Face Hub; downloaded at endpoint startup and loaded by the inference engine.
The trained model parameters and associated files stored and versioned on the Hugging Face Hub repository. Downloaded securely at endpoint startup and loaded by the inference engine. Users can optionally pin a specific commit revision.
Software that loads the model and handles inference requests. Can be TGI, vLLM, SGLang, TEI, or a custom Docker image.
The software runtime that loads the model and handles inference requests. Packaged as a Docker container. Hugging Face provides prebuilt containers for popular engines: Text Generation Inference (TGI), vLLM, SGLang, and Text Embeddings Inference (TEI). Users can also specify a custom Docker image.
Dynamically adjusts the number of endpoint replicas based on CPU/GPU utilization or pending request count; supports scale-to-zero during idle periods.
Monitors hardware utilization (CPU or GPU, default threshold 80% over 1 minute) or pending request count and adds or removes replicas accordingly. Scaling up occurs every minute, scaling down every 2 minutes with a 300-second stabilization window. Scale-to-zero reduces replicas to 0 after a configurable inactivity period (default 1 hour), with a cold-start delay on the next request.
Controls access to the endpoint via three security levels: public, authenticated (HF token), and private (VPC connection).
Enforces one of three access levels: Public (no authentication required), Protected (requires a valid Hugging Face access token), or Private (accessible only via a private VPC connection using AWS PrivateLink or Azure equivalent, not exposed to the public internet). Data in transit is encrypted with TLS/SSL.
Exposes the model as an HTTP URL through which clients send inference requests. The response format depends on the configured task and engine.
The URL exposed to clients for inference requests. Accepts HTTP requests with payloads in supported content types (application/json, image/*, audio/*, text/*). The API format follows the inference engine's conventions (e.g., OpenAI-compatible chat completions for TGI/vLLM LLM endpoints).
Fully parallel
Individual inference requests are independent HTTP calls handled by separate replicas. Multiple replicas run in parallel to handle concurrent traffic. Within each replica, the inference engine may use batching and parallel GPU execution.
Dense
All paths active
Each inference request is processed by a full model forward pass on the assigned replica. No conditional routing or sparse activation at the serving layer level.
Cloud provider
The cloud infrastructure provider on which the endpoint runs. Determines available instance types and supported regions.
Instance type / accelerator
The compute hardware used to run the endpoint. Options include CPU instances and GPU instances of various sizes (e.g., NVIDIA A10G, L4, A100). Determines throughput, latency, and cost.
Min / max replicas
Defines the lower and upper bounds for the autoscaler. Min replicas set to 0 enables scale-to-zero. Min replicas ≥ 2 recommended for high-availability production workloads.
Inference engine / container
The Docker container or inference engine used to serve the model. Auto-selected by Hugging Face based on model type; can be overridden.
Access level / endpoint type
Controls network access and authentication requirements for the endpoint.
Computational characteristics
When an endpoint is configured with scale-to-zero and receives a request after the inactivity period, it must restart from 0 replicas. During initialization, the proxy returns HTTP 503. Cold start time depends on model size and instance type and can range from tens of seconds to several minutes for large LLMs.
Set minimum replicas to at least 1 for latency-sensitive production workloads. Use the 'X-Scale-Up-Timeout' request header to control timeout behavior. For intermittent workloads, consider accepting the cold start trade-off for cost savings.
The autoscaler checks CPU/GPU utilization every minute and scales up by adding replicas, with a 300-second stabilization window after scale-down. Sudden traffic spikes may cause temporary request queuing or errors before new replicas become available.
Pre-warm by setting a higher minimum replica count before anticipated traffic spikes. Use pending-requests-based autoscaling (experimental) for faster response to load changes. Design client-side retry logic to handle transient 502/503 errors.
Deploying a large model on an instance with insufficient VRAM or RAM causes the container to fail to load the model. Conversely, over-provisioning an unnecessarily large GPU instance increases cost without performance benefit.
Check model VRAM requirements before selecting an instance type. Use the Hugging Face model catalog or documentation to find recommended hardware configurations. Enable quantization (e.g., GPTQ, AWQ) via TGI or vLLM to reduce memory requirements for large LLMs.
If the auto-detected task or container type is incorrect for the deployed model, the endpoint may start but return incorrect results or fail requests. Custom models or non-standard architectures may not be auto-recognized.
Explicitly specify the task type and container type in the endpoint configuration. Use a custom Docker container with a custom handler class for models not natively supported by HF inference containers.
Launch of Inference Endpoints (October 2022)
breakthroughHugging Face launched Inference Endpoints in October 2022 as a dedicated, managed inference serving product, replacing the paid tier of the Serverless Inference API. Initial support for AWS and Azure, with CPU and GPU instances.
Added support for Google Cloud TPU v5e
Hugging Face added Google Cloud TPU v5e support to Inference Endpoints in partnership with Google Cloud, enabling cost-effective inference for LLMs including Gemma, Llama, and Mistral via Optimum TPU and TGI.
Support for vLLM, SGLang, TEI as built-in inference engines
Inference Endpoints extended built-in support to multiple open-source inference engines (vLLM, SGLang, Text Embeddings Inference), in addition to TGI, allowing users to choose the engine best suited to their model and workload.
Most production LLM and diffusion model deployments on Inference Endpoints use NVIDIA GPU instances (A10G, L4, A100, H100). GPU inference is required for practical throughput on large transformer models.
Available GPU instances vary by cloud provider and region. Pricing starts from $0.5 per GPU/hr.
CPU instances are supported and suitable for smaller models (classification, embeddings, NLP tasks under ~1B parameters) where GPU cost is not justified. Pricing starts from $0.032 per CPU core/hr.
CPU-based endpoints use Intel Xeon or comparable processors with AVX instructions for accelerated matrix operations.
Google Cloud TPU v5e support was added in 2024 for LLM inference (Gemma, Llama, Mistral) via Optimum TPU. As of 2024, TPU availability on Inference Endpoints has been suspended pending further updates.
TPU support availability should be verified in current Hugging Face documentation.
| Title | Publisher | Type |
|---|---|---|
| Inference Endpoints Official product documentation for Inference Endpoints. | Hugging Face | documentation |
| About Inference Endpoints Description of managed container lifecycle, scaling, and monitoring. | Hugging Face | documentation |
| Inference Endpoints in huggingface_hub Programmatic endpoint management via the huggingface_hub library. | Hugging Face | documentation |
| Autoscaling Documentation for autoscaling endpoints. | Hugging Face | documentation |
| Getting Started with Hugging Face Inference Endpoints Post announcing the service from October 2022. | Hugging Face | blog |
| Analytics and Metrics Documentation for endpoint analytics and metrics. | Hugging Face | documentation |
Description of managed container lifecycle, scaling, and monitoring.
Programmatic endpoint management via the huggingface_hub library.
Post announcing the service from October 2022.
