Robots Atlas>ROBOTS ATLAS
Inference

Forward pass

1958ActivePublished: 17 November 2026Updated: 17 November 2026Published
Key innovation
Defines a neural network as a composition of functions evaluated layer by layer from input to output — the computation that produces a prediction while also storing the activations needed for the later backward pass.
Category
Inference
Abstraction level
Primitive
Operation level
ModelLayerInferenceTraining
Use cases
Inference: producing predictions from a trained modelThe first phase of a training step before the backward passContext prefill in autoregressive modelsProfiling a model's compute cost and latency

How it works

1) Load the input x (a batch of examples) and the model parameters. 2) For each layer l from first to last: compute a_l = f_l(a_{l-1}; W_l), where f_l is the layer operation (e.g. linear + activation). 3) Store intermediate activations if training is planned (needed for backprop). 4) At the end obtain the output (logits → softmax → probabilities, or regression values). 5) In inference return the prediction; in training compute the loss against the labels, which ends the forward phase and begins the backward phase.

Problem solved

The forward pass answers the basic question: how to compute a model's output from its input and parameters. It is also a prerequisite for learning — without computing activations and the loss in the forward direction, gradients cannot be computed in the backward direction.

Components

Layer operationsPerform the actual computation from input to output.

Constituent transforms: matrix multiplications, activation functions, normalizations, attention — applied in sequence.

Intermediate activationsConnect layers and enable later gradient computation.

The outputs of successive layers; in training they are cached for use in the backward pass.

Output layerTurns representations into the final prediction.

The final transform producing logits/probabilities or regression values.

Evolution

Original paper · 1958 · Psychological Review · Frank Rosenblatt
The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain
Frank Rosenblatt
1958
Perceptron
Inflection point

Rosenblatt formalizes computing a neuron's output from its inputs — the germ of the forward pass.

1986
Backpropagation establishes the forward/backward pair
Inflection point

Rumelhart, Hinton, and Williams describe backpropagation, which explicitly uses the activations stored in the forward pass.

2017
Transformer and parallel forward over tokens

The transformer architecture enables a parallel forward pass over all positions during training.

Hyperparameters (configurable axes)

Batch sizeHigh

Number of examples processed in parallel; affects throughput and memory use.

1Low-latency inference.
256+High-throughput training / batched inference.
Numerical precisionHigh

The number format (FP32/FP16/BF16/INT8) used in computation; affects speed, memory, and accuracy.

BF16Standard for training/inference on modern GPUs.
INT8Low-cost quantized inference.

Computational complexity

Time complexity: O(suma po warstwach kosztu warstwy); warstwa liniowa: O(B·d_in·d_out). Space complexity: Inferencja: O(maks. rozmiar aktywacji warstwy); Trening: O(suma aktywacji wszystkich warstw).

Execution paradigm

Primary mode
Dense

For dense models the forward pass activates all parameters; in sparse models (MoE) only a subset is activated.

Activation pattern
All paths active
Additional modes
Sparse
Routing mechanism

In a classic dense network all paths are active; routing appears only in sparse architectures such as MoE.

Parallelism

Parallelism level
Partially parallel

Operations within a layer and examples in a batch are fully parallel; layers are sequential.

Scope
InferenceTrainingAcross tokensAcross devices
Constraints
!Layer l depends on the output of layer l-1, so layers run sequentially (though operations within a layer are highly parallel).
!During token-by-token generation each position depends on the previous ones, limiting parallelism across tokens.

Hardware requirements

Primary

The forward pass is dominated by matrix multiplications, ideal for GPU tensor cores.

Good fit

TPU systolic arrays are optimized for the matrix operations of the forward pass.