A team designs the chip for a specific AI compute profile: at its core are matrix-multiply units (e.g. systolic arrays), supported by low-precision arithmetic (FP8/INT8/bfloat16), large high-bandwidth memory (HBM) and fast inter-chip interconnects that let chips be linked into large clusters (pods). The design (RTL) is brought to tape-out and manufactured at a fab (e.g. TSMC), often with a silicon-implementation partner (e.g. Broadcom, Marvell). To be useful, the chip also needs a software stack (compilers, drivers, libraries), which is often the hardest part given the dominance of the CUDA ecosystem.
General-purpose GPUs are expensive, power-hungry and not always optimal for the narrow profile of AI compute; their supply and pricing are dominated by a single vendor. Dedicated ASICs cut cost and power draw and reduce dependence on external GPUs.
The compute core optimized for matrix multiplication - the dominant AI operation.
FP8/INT8/bfloat16 formats increasing throughput and energy efficiency.
Official
Fast memory feeding data to the compute units.
Official
Fast links connecting many chips into large clusters (pods).
Official
Compilers and libraries enabling models to run on the custom chip.
Lack of a mature software stack (compilers, libraries) limits adoption despite good hardware performance.
ASIC design and manufacturing carry large NRE costs and multi-month cycles; risk that the chip becomes outdated as models evolve.
An ASIC is optimized for a narrow compute profile; new model types may run poorly on it.
Google deployed TPU v1 for inference in its own data centers, kicking off the wave of dedicated AI accelerators.
AWS announced Inferentia (inference), then Trainium (training); more cloud providers began building their own accelerators.
Model providers enter custom silicon - OpenAI and Broadcom announced the Jalapeño inference ASIC, alongside Meta MTIA and Microsoft Maia.
