Groq's compiler schedules every operation and data transfer in time ahead of execution (static, deterministic scheduling), so the chip needs no caches or branch prediction. Data streams through arrays of compute units (matrix multiplication, vector functions) in a fixed cadence, and weights and activations reside in fast on-chip SRAM. For large models, many LPUs are connected in a deterministically communicating network, allowing the model to scale while preserving predictable latency. The result is very fast, repeatable token generation.
LLM inference on GPUs is bounded by memory (HBM) bandwidth and suffers from variable latency due to reactive, non-deterministic execution. The LPU addresses this with deterministic dataflow and on-chip SRAM, delivering predictably low latency and high throughput.
The dataflow core executing operations in a deterministic, scheduled cadence.
Large, fast memory for weights and activations instead of HBM.
Schedules every operation and data transfer ahead of time.
Links many LPUs for large models with predictable communication.
Official
Large models require sharding across many LPUs.
Performance and correctness depend on the compiler's static schedule.
Jonathan Ross (co-creator of Google's TPU) founds Groq.
Groq describes the Tensor Streaming Processor โ a deterministic dataflow processor.
LLM inference on the LPU goes mainstream via GroqCloud.
Execution is dense and fully deterministic (static compiler scheduling), with no reactive hardware mechanisms.
Parallelism across compute arrays and multiple chips; autoregressive decoding remains sequential across tokens.
