An inference accelerator maps the model's computation graph onto hardware multiply-accumulate units. Linear layers and convolutions are lowered to matrix multiplications and executed on a systolic array or a grid of tensor cores, where data streams through a mesh of MAC units, minimizing data movement to and from memory. Model weights and activations are quantized to low precision (INT8/FP8), increasing operations per cycle and reducing memory-bandwidth demand. A compiler (e.g. TVM, AWS Neuron, XLA) lowers the model to an intermediate representation, chooses memory layout, operator fusion and scheduling, then emits instructions for the accelerator. At serving time, requests are frequently grouped into batches to maximize compute-unit utilization; for autoregressive (LLM) decoding, memory bandwidth rather than raw compute becomes the bottleneck.
Running inference at scale on CPUs and general-purpose GPUs is energy-hungry and memory-bandwidth bound, driving up operating cost and latency for production models. An inference accelerator addresses efficient, low-cost, low-latency serving of trained models — both in the data center and on power-constrained edge devices.
A grid of multiply-accumulate (MAC) units that performs matrix multiplication by streaming data rhythmically between neighboring cells, minimizing accesses to main memory.
A large, fast SRAM buffer holding weights and activations close to the compute units, reducing traffic to external memory (HBM/DRAM), the main source of latency and energy cost.
A datapath supporting INT8, FP8 and BF16 formats with scaling, enabling multiplied throughput over FP32 with a controlled accuracy loss.
Official
A software layer (e.g. XLA, AWS Neuron, Apache TVM) that lowers the model graph to an intermediate representation, selects operator fusion, memory layout and scheduling, and emits code for the accelerator.
Official
Aggressive quantization (INT8/FP8) can materially degrade model quality without calibration or quantization-aware training (QAT).
Batch-1 LLM decoding fails to saturate MAC units — memory bandwidth, not TOPS, governs performance.
Uncommon operators may be unsupported, forcing costly fallback to CPU/GPU and hurting latency.
Increasing batch size raises throughput but inflates per-request latency (p99).
The first broadly deployed dedicated data-center inference accelerator.
Formal TPU performance analysis and release of an open deep-learning accelerator architecture. The same year Apple introduces the Neural Engine (A11).
Popularized INT8 inference on serving-optimized GPUs.
A dedicated AWS inference chip unveiled at re:Invent, lowering serving cost.
FP8 enters LLM inference, alongside architectures optimized for ultra-low-latency decoding (Groq LPU).
At low arithmetic intensity — typical of batch-1 autoregressive LLM decoding — performance is bounded by memory bandwidth (weight movement) rather than raw compute (roofline model). Large batches and high arithmetic intensity shift the bottleneck toward the MAC units.
Inference arithmetic format (INT8, FP8, BF16, FP16).
Theoretical operations per second, usually quoted per precision.
Memory (HBM/DRAM) bandwidth — critical for LLM decoding.
Amount of fast local memory that limits external-memory traffic.
Trade-off between serving latency and throughput.
MAC-grid dimensions (e.g. 256x256) determining peak GEMM compute.
Power draw limit — critical at the edge.
Standard accelerators run dense GEMM (all paths active). Support for sparsity (e.g. 2:4) and MoE models is an optional extension.
Batched inference is highly parallel (data, channels, tokens). Autoregressive LLM decoding is sequential across tokens but parallel within each step.
The TPU is the flagship example of a systolic-array inference accelerator.
Tensor Core GPUs (e.g. T4/H100) are widely used for INT8/FP8 inference.
FPGAs enable configurable low-latency inference pipelines at the cost of lower compute density.
Neuromorphic chips offer ultra-low-power inference for spiking networks, but in niche applications.
Vector-extension CPUs can run inference, but with lower energy efficiency and throughput.
