HBM
How it works
In HBM, 4 to 16 DRAM dies are stacked on top of each other and vertically connected by thousands of TSVs and microbumps, forming a very wide interface (1024-bit per stack, and 2048-bit from HBM4) split into many independent channels (e.g. 16 channels of 64 bits in HBM3). At the bottom of the stack sits a base/logic die with buffers and test logic that talks to the processor memory controller. The whole stack is placed next to the GPU/accelerator on a silicon interposer (2.5D packaging) that routes dense signal traces over a short distance. Because the bus is extremely wide, HBM reaches high aggregate bandwidth at a relatively low per-pin clock, lowering the energy per transferred bit compared with GDDR.
Problem solved
Memory bandwidth has become the main bottleneck of AI and HPC accelerators (the "memory wall"): traditional GDDR/DDR require wide on-PCB buses, consume significant power and cannot keep up with growing GPU compute. HBM shortens the memory-to-processor distance, multiplies the number of I/O lines and runs at lower per-pin clocks, delivering far higher bandwidth per watt and unlocking compute that would otherwise be starved by memory access.
Components
4 to 16 DRAM memory layers stacked vertically on top of each other, forming a single memory stack.
Vertical electrical connections passing through the silicon die that link the stacked DRAM layers and provide a wide, short interface.
A die at the bottom of the stack containing buffer circuitry and test logic; it mediates between the DRAM layers and the processor memory controller.
Fine solder connections joining adjacent layers of the stack and the stack to the base die.
A silicon substrate routing dense signal traces between the HBM stack and the GPU/CPU/accelerator over a very short distance (2.5D packaging).
Implementation
TSV stacks and interposer assembly (e.g. TSMC CoWoS) are expensive, and packaging throughput is often a supply bottleneck for AI accelerators.
HBM is integrated on the processor package; capacity is fixed at manufacturing, smaller than DDR modules and cannot be expanded.
Dense, tall DRAM stacks (12–16 layers) impede heat dissipation and may force clock throttling.
Evolution
JEDEC defines HBM: 1024-bit interface per stack, up to 4 DRAM layers, ~128 GB/s per stack.
The AMD Fiji-based GPU is the first product to use first-generation HBM.
Up to 8 layers, 8 GB per stack and up to 256 GB/s; used in NVIDIA Tesla P100 and V100 among others.
Higher clocks (up to ~3.6 Gbps/pin), ~460 GB/s per stack and up to 24 GB (12 layers).
16 channels of 64 bits, up to 6.4 Gbps/pin (~819 GB/s per stack); powers NVIDIA H100.
SK hynix and Micron: >9.2 Gbps/pin and >1.2 TB/s per stack, 24–36 GB; powers NVIDIA H200 and Blackwell.
Doubled interface to 2048 bits per stack, up to 64 GB (16 layers); foundation of the NVIDIA Rubin platform.
Hyperparameters (configurable axes)
Number of DRAM layers in the stack — affects capacity and partly bandwidth.
Width of a single stack bus in bits.
Maximum capacity of a single HBM stack.
Transfer speed per single I/O pin.
Division of the wide stack bus into independent channels.
Hardware requirements
Modern AI GPUs (NVIDIA H100/H200/Blackwell) are memory-bandwidth bound; HBM supplies the TB/s that tensor cores need.
Google TPU accelerators use HBM as main memory to feed large matrix-multiply units.