Robots Atlas>ROBOTS ATLAS
Inference

INT8

2017ActivePublished: 29 August 2026Updated: 29 August 2026Published
Key innovation
Representing a neural network's floating-point weights and activations as 8-bit integers with a scale and zero-point, enabling integer-arithmetic inference — roughly 4x less memory and higher throughput with minimal accuracy loss.
Category
Inference
Abstraction level
Pattern
Operation level
Architecture blockModel
Use cases
Inference on edge and mobile devicesAccelerating models on GPUs with INT8 Tensor CoresDeployment on NPUs/TPUs and power-constrained acceleratorsReducing inference cost for LLMs and vision modelsModel compression for real-time applications

How it works

1) Choose a mapping scheme: affine (asymmetric, with scale and zero-point) or symmetric (zero-point = 0). 2) Determine the value range per tensor — per-tensor (one scale for the whole tensor) or per-channel (a separate scale per weight channel, more accurate for convolutional layers). 3) Calibration: run a representative dataset to estimate activation ranges (methods: min/max, percentile, KL-divergence/entropy minimization). 4) Quantize: q = round(x / scale) + zero_point, clamped to the INT8 range. 5) Integer inference: multiplications and accumulations run in INT8/INT32 with requantization between layers. The model can be obtained via PTQ (quantizing a trained model, fast, no retraining) or QAT (inserting fake-quant ops into the training graph so the network learns robustness to quantization — higher accuracy at the cost of extra training).

Problem solved

FP32/FP16 models are large and expensive at inference — they consume significant memory and bandwidth, making deployment on edge, mobile, and resource-constrained accelerators difficult. INT8 reduces model size ~4x along with memory-bandwidth and compute demands, enabling faster and cheaper inference at an acceptable accuracy cost.

Implementation

Reference implementations
PyTorch Quantization
Python · PyTorch
Official
TensorFlow Lite Post-Training Quantization
Python · Google
Official
NVIDIA TensorRT INT8
C++/Python · NVIDIA
Official
Implementation pitfalls
Poor activation calibrationHigh

A non-representative calibration set or naive min/max clips outliers and causes large accuracy drops.

Fix:Use representative data and percentile or KL-minimization methods; consider QAT.
Per-tensor where per-channel is neededMedium

Layers with high inter-channel range variance lose accuracy under a single shared scale.

Fix:Use per-channel quantization for convolutional and linear layer weights.
Activation outliers in LLMsHigh

Large activation outliers in transformers break naive INT8 activation quantization.

Fix:Apply techniques like SmoothQuant / mixed precision for outlier channels.

Evolution

Original paper · 2017 · CVPR 2018 · Benoit Jacob
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko
2017
Integer-arithmetic-only inference (Jacob et al.)
Inflection point

Google paper defining the INT8 quantization scheme with QAT, deployed in TensorFlow Lite / gemmlowp.

2020
Integer Quantization white paper (NVIDIA)

Systematization of PTQ and QAT for INT8 on GPUs (Wu et al.), including calibration and per-channel.

2022
LLM.int8() and SmoothQuant

INT8 methods adapted to large language models with outlier handling.

Hyperparameters (configurable axes)

Quantization methodCritical

PTQ (post-training) vs QAT (quantization-aware training). QAT usually yields higher accuracy at the cost of extra training.

PTQFast, no retraining
QATHigher accuracy, requires training
Scaling granularityHigh

Per-tensor (one scale per tensor) vs per-channel (scale per channel). Per-channel is more accurate for convolutional weights.

per-tensor
per-channel
Calibration methodHigh

How activation ranges are estimated during PTQ.

min/max
percentile
KL / entropyUsed e.g. in TensorRT
Mapping schemeMedium

Symmetric (zero-point = 0) vs affine/asymmetric (with a zero-point).

symmetric
asymmetric / affine

Execution paradigm

Primary mode
Dense

INT8 is a numeric-representation technique applied to inference; the compute flow stays dense, only the arithmetic precision changes.

Activation pattern
All paths active

Parallelism

Parallelism level
Fully parallel

INT8 operations are fully parallel on matrix units (Tensor Cores, NPUs).

Scope
Inference

Hardware requirements

Primary

Tensor Cores have dedicated high-TOPS INT8 datapaths.

Good fit

TPUs/NPUs natively support INT8 integer arithmetic.

Good fit

SIMD instructions (AVX-512 VNNI) accelerate INT8 multiply-accumulate on CPUs.

Good fit

Integer logic maps efficiently onto FPGA DSP resources.