Custom AI silicon
How it works
A team designs the chip for a specific AI compute profile: at its core are matrix-multiply units (e.g. systolic arrays), supported by low-precision arithmetic (FP8/INT8/bfloat16), large high-bandwidth memory (HBM) and fast inter-chip interconnects that let chips be linked into large clusters (pods). The design (RTL) is brought to tape-out and manufactured at a fab (e.g. TSMC), often with a silicon-implementation partner (e.g. Broadcom, Marvell). To be useful, the chip also needs a software stack (compilers, drivers, libraries), which is often the hardest part given the dominance of the CUDA ecosystem.
Problem solved
General-purpose GPUs are expensive, power-hungry and not always optimal for the narrow profile of AI compute; their supply and pricing are dominated by a single vendor. Dedicated ASICs cut cost and power draw and reduce dependence on external GPUs.
Components
The compute core optimized for matrix multiplication - the dominant AI operation.
FP8/INT8/bfloat16 formats increasing throughput and energy efficiency.
Official
Fast memory feeding data to the compute units.
Official
Fast links connecting many chips into large clusters (pods).
Official
Compilers and libraries enabling models to run on the custom chip.
Implementation
Lack of a mature software stack (compilers, libraries) limits adoption despite good hardware performance.
ASIC design and manufacturing carry large NRE costs and multi-month cycles; risk that the chip becomes outdated as models evolve.
An ASIC is optimized for a narrow compute profile; new model types may run poorly on it.
Evolution
The first commercial bipolar gate arrays appear (Ferranti, Fairchild Micromatrix) - the beginning of circuits designed for a specific application.
Logic-synthesis tools mature, compiling HDL (Verilog/VHDL) RTL descriptions into gate-level netlists, popularizing standard-cell-based ASIC design.
Google deployed TPU v1 for inference in its own data centers, kicking off the wave of dedicated AI accelerators.
Google publishes a detailed TPU performance analysis (ISCA 2017) and introduces TPU v2 with training support, demonstrating the performance-per-watt advantage of a dedicated ASIC over GPUs/CPUs.
AWS announced Inferentia (inference), then Trainium (training); more cloud providers began building their own accelerators.
Deployment of new in-house chip generations: Microsoft Maia 100, AWS Trainium2 and Google TPU v5 - custom silicon becomes standard among cloud providers.
Model providers enter custom silicon - OpenAI and Broadcom announced the Jalapeno inference ASIC, alongside Meta MTIA and Microsoft Maia.
Hyperparameters (configurable axes)
The fabrication node (e.g. 5 nm, 3 nm) determines transistor density, energy efficiency and cost.
Supported formats (INT8, FP8, bfloat16, FP4) affect throughput and accuracy.
The number and size of multiply-accumulate units determine peak compute (TOPS/FLOPS).
The amount and bandwidth of high-bandwidth memory bound the size of models served and throughput.
Bandwidth of inter-chip links determines how efficiently compute scales into pods.
Non-recurring engineering (NRE) cost and the length of the cycle to tape-out affect whether a custom chip pays off.
Compute bottleneck
AI accelerator performance is often limited by the bandwidth of feeding data (weights and activations) to the compute units rather than by raw compute; hence AI ASICs integrate HBM and large on-chip SRAM buffers.
Execution paradigm
A typical AI ASIC runs dense matrix multiplication - all MAC units are active every cycle; the hardware is optimized for a fixed, predictable compute profile.
Parallelism
Matrix multiplication is massively parallel; systolic arrays perform many MAC operations at once, and fast interconnects let compute scale across many chips (pods).