In HBM, 4 to 16 DRAM dies are stacked on top of each other and vertically connected by thousands of TSVs and microbumps, forming a very wide interface (1024-bit per stack, and 2048-bit from HBM4) split into many independent channels (e.g. 16 channels of 64 bits in HBM3). At the bottom of the stack sits a base/logic die with buffers and test logic that talks to the processor memory controller. The whole stack is placed next to the GPU/accelerator on a silicon interposer (2.5D packaging) that routes dense signal traces over a short distance. Because the bus is extremely wide, HBM reaches high aggregate bandwidth at a relatively low per-pin clock, lowering the energy per transferred bit compared with GDDR.
Memory bandwidth has become the main bottleneck of AI and HPC accelerators (the "memory wall"): traditional GDDR/DDR require wide on-PCB buses, consume significant power and cannot keep up with growing GPU compute. HBM shortens the memory-to-processor distance, multiplies the number of I/O lines and runs at lower per-pin clocks, delivering far higher bandwidth per watt and unlocking compute that would otherwise be starved by memory access.
4 to 16 DRAM memory layers stacked vertically on top of each other, forming a single memory stack.
Vertical electrical connections passing through the silicon die that link the stacked DRAM layers and provide a wide, short interface.
A die at the bottom of the stack containing buffer circuitry and test logic; it mediates between the DRAM layers and the processor memory controller.
Fine solder connections joining adjacent layers of the stack and the stack to the base die.
A silicon substrate routing dense signal traces between the HBM stack and the GPU/CPU/accelerator over a very short distance (2.5D packaging).
TSV stacks and interposer assembly (e.g. TSMC CoWoS) are expensive, and packaging throughput is often a supply bottleneck for AI accelerators.
HBM is integrated on the processor package; capacity is fixed at manufacturing, smaller than DDR modules and cannot be expanded.
Dense, tall DRAM stacks (12โ16 layers) impede heat dissipation and may force clock throttling.
JEDEC defines HBM: 1024-bit interface per stack, up to 4 DRAM layers, ~128 GB/s per stack.
The AMD Fiji-based GPU is the first product to use first-generation HBM.
Up to 8 layers, 8 GB per stack and up to 256 GB/s; used in NVIDIA Tesla P100 and V100 among others.
Higher clocks (up to ~3.6 Gbps/pin), ~460 GB/s per stack and up to 24 GB (12 layers).
16 channels of 64 bits, up to 6.4 Gbps/pin (~819 GB/s per stack); powers NVIDIA H100.
SK hynix and Micron: >9.2 Gbps/pin and >1.2 TB/s per stack, 24โ36 GB; powers NVIDIA H200 and Blackwell.
Doubled interface to 2048 bits per stack, up to 64 GB (16 layers); foundation of the NVIDIA Rubin platform.
Number of DRAM layers in the stack โ affects capacity and partly bandwidth.
Width of a single stack bus in bits.
Maximum capacity of a single HBM stack.
Transfer speed per single I/O pin.
Division of the wide stack bus into independent channels.
Modern AI GPUs (NVIDIA H100/H200/Blackwell) are memory-bandwidth bound; HBM supplies the TB/s that tensor cores need.
Google TPU accelerators use HBM as main memory to feed large matrix-multiply units.
