ConvNeXt
How it works
ConvNeXt is built by a sequence of changes applied to ResNet: (1) adjusting the per-stage compute ratio to (3,3,9,3) in the Swin style; (2) a patchify stem, replacing the input 7x7 stride-2 conv plus max-pool with a non-overlapping 4x4 stride-4 convolution; (3) a ResNeXt-style block with depthwise convolution and wider channels; (4) an inverted bottleneck (roughly 4x channel expansion, like a Transformer FFN); (5) moving up and enlarging the depthwise kernel to 7x7; (6) replacing ReLU with GELU and using a single activation per block; (7) replacing BatchNorm with LayerNorm and using fewer normalization layers; (8) separate downsampling layers (2x2 stride-2 conv with LayerNorm) between stages. The model remains hierarchical and fully convolutional.
Problem solved
After the rise of Vision Transformers, a common claim was that the attention mechanism is essential for state-of-the-art computer vision, even though hierarchical variants (Swin) reintroduced the inductive biases of convolutions. ConvNeXt investigates how much of the transformer advantage stems from attention itself versus modern training and architectural choices, and shows that a well-designed pure ConvNet can match transformers.
Components
Replaces ResNet's input 7x7 stride-2 conv and max-pool with a non-overlapping 4x4 stride-4 convolution, analogous to ViT patch embedding.
A large 7x7 per-channel kernel acting as spatial information mixing, analogous to the Swin attention window.
A block expanding channels (~4x) and projecting them back, in the style of the Transformer FFN and MobileNetV2/ResNeXt.
Layer normalization replacing BatchNorm; the number of normalization layers per block is reduced.
GELU replaces ReLU, with a single activation per block instead of several.
Dedicated 2x2 stride-2 layers with LayerNorm between stages, instead of downsampling inside the residual block.
Implementation
Evolution
Residual networks, the baseline that ConvNeXt modernizes.
Applying a pure transformer to image classification.
A hierarchical ViT with shifted windows; the main reference point for ConvNeXt.
A modernized ResNet matching transformers on ImageNet, COCO and ADE20K.
Extension with self-supervised FCMAE pretraining and a Global Response Normalization (GRN) layer.
ConvNeXt used as a convolutional backbone in Meta's DINOv3 model family.
Hyperparameters (configurable axes)
Family variants differing in channel width, depth and compute cost.
Number of blocks per stage; ConvNeXt-T/S uses (3,3,9,3) while B/L/XL uses (3,3,27,3).
Number of channels across hierarchy stages.
Size of the large depthwise convolution kernel; 7x7 by default.
Computational complexity
Time complexity: O(H·W·C·K² + H·W·C²). Space complexity: O(H·W·C).
Execution paradigm
A pure ConvNet with no attention and no routing; all computational paths are active for every input.
Parallelism
Convolutions are parallel across spatial positions and channels; there is no sequential token dependency.
Hardware requirements
Dense convolutions map well to GPUs, but 7x7 depthwise convolutions are memory-bandwidth bound and underutilize tensor cores relative to their low FLOP count.
Standard convolution operations are broadly supported across accelerators.