AI Factory
How it works
An AI factory operates as an integrated, full hardware-software stack. Data pipelines turn raw, unstructured data into high-quality tokens for training. The hardware layer is accelerated computing: high-performance GPUs, CPUs, fast networking, storage and advanced cooling. The software layer is modular, scalable and API-driven, binding everything into one cohesive system. Inference is a continuous, iterative process: trained models generate predictions in real time, and the outputs feed back into the system, creating a data flywheel that improves the models. Software such as NVIDIA Dynamo orchestrates inference across thousands of GPUs, using disaggregated serving (separating the prefill and generation phases onto different GPUs) to maximize resource utilization and token revenue. The whole facility can be designed and simulated as a digital twin (e.g. an Omniverse Blueprint) before construction. Key operational metrics are token throughput, latency, and performance per watt and per megawatt.
Problem solved
General-purpose data centers are not optimized for the economics of AI: the scale, cost and energy use of generative AI and reasoning models (which emit tens of thousands of tokens per prompt) demand infrastructure designed holistically for maximum token throughput per watt and per megawatt, rather than for universal computing.
Components
Transform raw, unstructured data into high-quality, structured tokens that models can learn from.
High-performance GPUs, CPUs, storage and advanced cooling forming the factory's hardware layer.
Fast networking (e.g. NVLink, InfiniBand/Ethernet) tying thousands of accelerators into one scale-up/scale-out system.
Full-stack inference software (e.g. NVIDIA Dynamo) orchestrating token generation across thousands of GPUs with disaggregated serving.
Automate the AI lifecycle (from hyperparameter tuning to deployment), sustaining high throughput and reliability.
A virtual environment (e.g. Omniverse Blueprint) to design, simulate and optimize the factory before construction.
Implementation
Electrical power availability is often a hard limit on AI-factory scale; planning must include the grid and substation, not just server racks.
The high power density of accelerators requires advanced cooling (e.g. liquid); underestimating it causes throttling and performance loss.
Inefficient inference orchestration across a large GPU fleet raises token cost and wastes throughput.
Evolution
At the GTC keynote (March 18, 2024) Huang said that in the future data centers will be thought of as AI factories whose goal is to generate revenue in the form of intelligence.
At GTC (March 18, 2025) NVIDIA unveiled Dynamo, the successor to Triton Inference Server, to scale reasoning-model inference in AI factories and maximize token revenue via disaggregated serving.
Hyperparameters (configurable axes)
Total electrical power of the factory, usually the primary scaling constraint.
Tokens generated per unit time: the factory's primary output metric.
Energy efficiency; determines the unit cost of a token.
Number of accelerators tied into one system; determines maximum throughput.
Bandwidth of the GPU interconnect fabric; critical for distributed inference.
Compute bottleneck
The dominant constraint of an AI factory is available electrical power and energy efficiency; the goal is maximum token throughput within a given power budget. NVIDIA explicitly optimizes its platforms for the "lowest cost per token per megawatt" and performance per watt.
Parallelism
AI-factory workloads are distributed (scale-up and scale-out) across thousands of GPUs; inference of a single reasoning model remains partly sequential at the token-generation level.
Hardware requirements
An AI factory is built on accelerated computing; high-performance GPUs with tensor cores are its compute core.