Architecture

MLP

1986ActivePublished: 21 October 2026Updated: 21 October 2026Published

Key innovation

Stacking fully connected layers with nonlinear activations, trained end-to-end by backpropagation — enabling learning of non-linearly-separable functions (e.g. XOR) and universal function approximation, overcoming the limitations of the single perceptron.

How it works

Forward pass: the input data flows sequentially through the layers; each layer computes a weighted sum (a matrix multiply of the weights with the input vector) plus a bias, then applies an activation function. Loss computation: the network output is compared with the target using a loss function. Backpropagation: gradients of the loss with respect to all weights are computed from the output layer back to the input layer using the chain rule. Weight update: an optimizer (e.g. SGD, Adam) adjusts the weights in the direction of the descending gradient. The cycle repeats over successive batches of data until convergence.

Problem solved

The single Rosenblatt perceptron could only separate linearly separable classes and failed on simple problems such as XOR. The MLP solves this by adding hidden layers with nonlinear activations, allowing it to model complex, nonlinear relationships. What was missing was an effective way to train such networks — backpropagation provided it, enabling efficient end-to-end training of multilayer networks.

Components

Input layerData entry point

Accepts the input feature vector; the number of neurons matches the dimensionality of the input data.

Hidden layer (fully connected)Linear transformation and intermediate representation

A dense layer computing a weighted sum of inputs plus a bias; parameterized by a weight matrix. An MLP has one or more such layers.

Activation functionIntroducing nonlinearity

A nonlinearity applied after the linear transformation (sigmoid, tanh, ReLU and variants); without it the network would collapse to a single linear transformation.

ReLURectified Linear Unit; the default choice in modern networks.

Sigmoid / tanhHistorical saturating functions, prone to vanishing gradients.

Official

Output layerProducing the output

The final layer producing the prediction; its activation depends on the task (softmax for multiclass classification, linear for regression).

Implementation

Reference implementations

scikit-learn MLPClassifier / MLPRegressor

Implementation pitfalls

Vanishing and exploding gradientsHigh

Deep MLPs with saturating activations (sigmoid, tanh) lose the gradient signal, which stalls training.

Fix:Use ReLU and its variants, proper weight initialization (He/Xavier), and normalization (batch/layer norm).

OverfittingMedium

The large parameter count of fully connected layers easily fits noise in the training data.

Fix:Dropout, L2 regularization (weight decay), early stopping and data augmentation.

Poor fit for structured dataMedium

The MLP ignores the spatial or temporal structure of the data (images, sequences) and requires a huge number of parameters.

Fix:Use convolutional networks for images, RNNs or Transformers for sequences; keep the MLP for tabular data and as a component.

Sensitivity to feature scalingLow

Unstandardized input features slow down or destabilize gradient-based training.

Fix:Normalize or standardize the inputs before training.

Evolution

Original paper · 1986 · Nature · David E. Rumelhart

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams

1958

Rosenblatt perceptron

Frank Rosenblatt introduces the perceptron — a single-layer linear classifier, the precursor of the MLP.

1969

Minsky and Papert critique

The book "Perceptrons" shows that a single perceptron cannot solve XOR — contributing to the first AI winter.

1986

Backpropagation for multilayer networks

Inflection point

Rumelhart, Hinton and Williams popularize backpropagation, making practical training of MLPs possible.

1989

Universal approximation theorem

Cybenko (1989), followed by Hornik (1991), prove that an MLP with a single hidden layer can approximate any continuous function.

2011

Adoption of ReLU

The ReLU activation (Glorot et al., 2011) mitigates the vanishing-gradient problem and eases training of deep networks.

2021

MLP-Mixer

An all-MLP architecture (Tolstikhin et al.) achieves competitive results in computer vision, renewing interest in fully connected layers.

Hyperparameters (configurable axes)

Depth (number of hidden layers)High

Number of hidden layers; greater depth increases representational capacity but makes training harder (vanishing gradients).

1Shallow MLP — sufficient as a universal approximator given enough width.

2-3Typical range for tabular data.

Width (neurons per layer)High

Number of neurons in a hidden layer; affects model capacity and, quadratically, the parameter count.

Activation functionCritical

The choice of nonlinearity (ReLU, GELU, tanh, sigmoid) affects training dynamics and gradient flow.

Computational complexity

Time complexity: O(Σₗ nₗ₋₁·nₗ). Space complexity: O(Σₗ nₗ₋₁·nₗ).

Compute bottleneck

Dense matrix multiplications (GEMM) between layers

The dominant cost is the dense weight-matrix multiplications; parameter count and compute grow quadratically with layer width.

Execution paradigm

Primary mode

Dense

All neurons and all connections are active for every input — there is no conditional routing or sparse activation.

Activation pattern

All paths active

Routing mechanism

Parallelism

Parallelism level

Partially parallel

Computation within a layer (matrix multiply) and across samples in a batch is fully parallel, but layers must be computed sequentially (the output of one is the input of the next).

Scope

TrainingInferenceAcross devices

Hardware requirements

Primary

Dense matrix multiplications map ideally onto GPU tensor cores.

Good fit

TPUs are optimized for the matrix operations typical of dense layers.

Possible

Smaller MLPs run efficiently on CPUs with AVX vector instructions.