Model and hardware teams work together: the model architecture (e.g. the attention mechanism, memory-access patterns, numeric precision) is chosen for the hardware's capabilities, while at the same time the hardware (compute units, memory, interconnect, supported formats) is designed for the profile of the target models. An intermediate layer - compilers and kernels (e.g. XLA, specialized attention kernels) - maps the model to efficient execution. Iterative feedback (profiling, benchmarks) tunes both components to each other across successive generations.
A model designed without regard for hardware can be inefficient (e.g. memory-bandwidth bound), and an accelerator designed without knowledge of the target models can be mismatched. Optimizing the two layers separately leaves a lot of performance 'on the table'.
Compute and memory-access patterns (e.g. attention, MoE, precision) chosen for the hardware.
Compute units, memory, interconnect and formats designed for the target models.
Compilers (e.g. XLA) and specialized kernels mapping the model to efficient execution on the hardware.
Official
Profiling and benchmarks that iteratively tune the model and hardware to each other.
Official
A model too tightly optimized for one accelerator may run poorly on another.
Co-design requires close collaboration between model and hardware teams with different cycles and skills.
Google Brain introduced the bfloat16 format together with the TPU, showing the benefits of designing precision alongside the hardware.
FlashAttention was designed around the GPU memory hierarchy (SRAM/HBM), greatly speeding up attention - a flagship example of model-hardware co-design.
Hopper introduced hardware FP8 support and a Transformer Engine tailored to transformer models.
