AI Infrastructure
How it works
Accelerators (GPUs/TPUs/ASICs) are assembled into servers, and servers into clusters linked by low-latency, high-bandwidth interconnect (e.g. NVLink within a node, InfiniBand between nodes). A storage layer holds training data and model files, while memory close to the processors handles active computation. Large training jobs are distributed across many machines, cutting computation time at the cost of coordination and communication between workers. Everything sits in data centres that provide power, backup power and cooling. A software layer (drivers, compilers, ML frameworks) together with orchestration and job scheduling (e.g. Kubernetes, Slurm) allocates accelerators, schedules jobs and monitors the cluster, while model-serving systems handle inference. Resources can be self-owned or rented from cloud providers, including specialised GPU clouds (neoclouds).
Problem solved
Training and running large AI models requires compute, network bandwidth, memory and power far beyond a single machine. AI infrastructure solves the problem of supplying, interconnecting and efficiently using these resources at scale, so that computation can be distributed across many devices at acceptable cost, latency and energy consumption.
Components
Specialised processors performing massively parallel matrix operations at the core of training and inference.
Accelerators assembled into servers and clusters; usable capacity depends on memory, networking and software, not just chip counts.
High-bandwidth, low-latency links between processors, servers and clusters, critical for distributed training.
Memory close to the processors handles active computation, while storage retains training data, intermediate results and model files.
Buildings, racks, power, backup systems and networking โ the physical facility for operating computing hardware.
Power supply and distribution, backup systems and cooling; accelerator clusters have high power density and large cooling demand.
Drivers, compilers and ML frameworks plus systems for accelerator allocation, job scheduling and cluster monitoring.
Platforms providing accelerators and AI services for rent, including specialised GPU cloud providers.
Evolution
The success of deep networks trained on GPUs launched the era of accelerator-based infrastructure.
The arrival of a purpose-built AI accelerator (ASIC) designed for machine-learning workloads.
Training models with hundreds of billions of parameters forced large-scale distributed clusters with fast interconnect.
Surging demand for GPU compute drove specialised GPU clouds (neoclouds) and massive investment in data centres and power.
Hyperparameters (configurable axes)
Number of GPUs/TPUs in the cluster; determines achievable training scale.
Inter-node network bandwidth and latency; a bottleneck for distributed training.
Available electrical power (MW) constraining the size and siting of the infrastructure.
Power per rack driving the cooling requirements for accelerator clusters.
Hardware requirements
Tensor-core GPUs are the dominant accelerator for training and inference of large models.
Purpose-built ASICs (TPUs) are used for large-scale training and inference.
CPUs handle supporting tasks and inference of smaller or optimised models.
FPGAs are sometimes used for specialised, power-efficient inference.