Forward pass
How it works
1) Load the input x (a batch of examples) and the model parameters. 2) For each layer l from first to last: compute a_l = f_l(a_{l-1}; W_l), where f_l is the layer operation (e.g. linear + activation). 3) Store intermediate activations if training is planned (needed for backprop). 4) At the end obtain the output (logits → softmax → probabilities, or regression values). 5) In inference return the prediction; in training compute the loss against the labels, which ends the forward phase and begins the backward phase.
Problem solved
The forward pass answers the basic question: how to compute a model's output from its input and parameters. It is also a prerequisite for learning — without computing activations and the loss in the forward direction, gradients cannot be computed in the backward direction.
Components
Constituent transforms: matrix multiplications, activation functions, normalizations, attention — applied in sequence.
The outputs of successive layers; in training they are cached for use in the backward pass.
The final transform producing logits/probabilities or regression values.
Evolution
Rosenblatt formalizes computing a neuron's output from its inputs — the germ of the forward pass.
Rumelhart, Hinton, and Williams describe backpropagation, which explicitly uses the activations stored in the forward pass.
The transformer architecture enables a parallel forward pass over all positions during training.
Hyperparameters (configurable axes)
Number of examples processed in parallel; affects throughput and memory use.
The number format (FP32/FP16/BF16/INT8) used in computation; affects speed, memory, and accuracy.
Computational complexity
Time complexity: O(suma po warstwach kosztu warstwy); warstwa liniowa: O(B·d_in·d_out). Space complexity: Inferencja: O(maks. rozmiar aktywacji warstwy); Trening: O(suma aktywacji wszystkich warstw).
Execution paradigm
For dense models the forward pass activates all parameters; in sparse models (MoE) only a subset is activated.
In a classic dense network all paths are active; routing appears only in sparse architectures such as MoE.
Parallelism
Operations within a layer and examples in a batch are fully parallel; layers are sequential.
Hardware requirements
The forward pass is dominated by matrix multiplications, ideal for GPU tensor cores.
TPU systolic arrays are optimized for the matrix operations of the forward pass.