Robots Atlas>ROBOTS ATLAS
Infrastructure

Hardware-model co-design

2017ActivePublished: 6 August 2026Updated: 6 August 2026Published
Key innovation
Designing the model architecture and the hardware (and software) together so they fit each other - instead of optimizing the model and the accelerator in isolation.
Category
Infrastructure
Abstraction level
Paradigm
Operation level
ModelArchitecture blockServing
Use cases
Optimizing training and inference of large modelsDesigning low-precision formats (FP8/bfloat16) together with the acceleratorHardware-aware algorithms (e.g. FlashAttention)Model providers' own silicon tailored to their architecturesQuantization and dataflow matched to hardware

How it works

Model and hardware teams work together: the model architecture (e.g. the attention mechanism, memory-access patterns, numeric precision) is chosen for the hardware's capabilities, while at the same time the hardware (compute units, memory, interconnect, supported formats) is designed for the profile of the target models. An intermediate layer - compilers and kernels (e.g. XLA, specialized attention kernels) - maps the model to efficient execution. Iterative feedback (profiling, benchmarks) tunes both components to each other across successive generations.

Problem solved

A model designed without regard for hardware can be inefficient (e.g. memory-bandwidth bound), and an accelerator designed without knowledge of the target models can be mismatched. Optimizing the two layers separately leaves a lot of performance 'on the table'.

Components

Model architectureThe model side of co-design.

Compute and memory-access patterns (e.g. attention, MoE, precision) chosen for the hardware.

Hardware acceleratorThe hardware side of co-design.

Compute units, memory, interconnect and formats designed for the target models.

Compiler / kernel layerThe model-hardware bridge.

Compilers (e.g. XLA) and specialized kernels mapping the model to efficient execution on the hardware.

Official

Feedback loopIterative optimization.

Profiling and benchmarks that iteratively tune the model and hardware to each other.

Official

Implementation

Implementation pitfalls
Over-fitting to one hardware targetMedium

A model too tightly optimized for one accelerator may run poorly on another.

Fix:Keeping an abstraction layer (compilers) and testing on multiple platforms.
Cost of team coordinationMedium

Co-design requires close collaboration between model and hardware teams with different cycles and skills.

Fix:Shared benchmarks, clear interfaces and common performance metrics.

Evolution

2017
bfloat16 and the TPU - a numeric format for the accelerator

Google Brain introduced the bfloat16 format together with the TPU, showing the benefits of designing precision alongside the hardware.

2022
FlashAttention - a hardware-aware algorithm
Inflection point

FlashAttention was designed around the GPU memory hierarchy (SRAM/HBM), greatly speeding up attention - a flagship example of model-hardware co-design.

2022
Transformer Engine and FP8 (NVIDIA Hopper)

Hopper introduced hardware FP8 support and a Transformer Engine tailored to transformer models.