Inference

INT8

2017ActivePublished: 29 August 2026Updated: 29 August 2026Published

Key innovation

Representing a neural network's floating-point weights and activations as 8-bit integers with a scale and zero-point, enabling integer-arithmetic inference — roughly 4x less memory and higher throughput with minimal accuracy loss.

Category

Inference

Abstraction level

Pattern

Operation level

Architecture blockModel

Use cases

Inference on edge and mobile devicesAccelerating models on GPUs with INT8 Tensor CoresDeployment on NPUs/TPUs and power-constrained acceleratorsReducing inference cost for LLMs and vision modelsModel compression for real-time applications

How it works

1) Choose a mapping scheme: affine (asymmetric, with scale and zero-point) or symmetric (zero-point = 0). 2) Determine the value range per tensor — per-tensor (one scale for the whole tensor) or per-channel (a separate scale per weight channel, more accurate for convolutional layers). 3) Calibration: run a representative dataset to estimate activation ranges (methods: min/max, percentile, KL-divergence/entropy minimization). 4) Quantize: q = round(x / scale) + zero_point, clamped to the INT8 range. 5) Integer inference: multiplications and accumulations run in INT8/INT32 with requantization between layers. The model can be obtained via PTQ (quantizing a trained model, fast, no retraining) or QAT (inserting fake-quant ops into the training graph so the network learns robustness to quantization — higher accuracy at the cost of extra training).

Problem solved

FP32/FP16 models are large and expensive at inference — they consume significant memory and bandwidth, making deployment on edge, mobile, and resource-constrained accelerators difficult. INT8 reduces model size ~4x along with memory-bandwidth and compute demands, enabling faster and cheaper inference at an acceptable accuracy cost.

Implementation

Reference implementations

PyTorch Quantization

Python · PyTorch

TensorFlow Lite Post-Training Quantization

Python · Google

NVIDIA TensorRT INT8

C++/Python · NVIDIA

Implementation pitfalls

Poor activation calibrationHigh

A non-representative calibration set or naive min/max clips outliers and causes large accuracy drops.

Fix:Use representative data and percentile or KL-minimization methods; consider QAT.

Per-tensor where per-channel is neededMedium

Layers with high inter-channel range variance lose accuracy under a single shared scale.

Fix:Use per-channel quantization for convolutional and linear layer weights.

Activation outliers in LLMsHigh

Large activation outliers in transformers break naive INT8 activation quantization.

Fix:Apply techniques like SmoothQuant / mixed precision for outlier channels.

Evolution

Original paper · 2017 · CVPR 2018 · Benoit Jacob

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko

2017

Integer-arithmetic-only inference (Jacob et al.)

Inflection point

Google paper defining the INT8 quantization scheme with QAT, deployed in TensorFlow Lite / gemmlowp.

2020

Integer Quantization white paper (NVIDIA)

Systematization of PTQ and QAT for INT8 on GPUs (Wu et al.), including calibration and per-channel.

2022

LLM.int8() and SmoothQuant

INT8 methods adapted to large language models with outlier handling.

Sources

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation

PyTorch Quantization

TensorFlow Lite Post-Training Quantization