Vision Tower
How it works
1) The image is split into patches and turned into a sequence of patch embeddings. 2) A stack of Transformer (ViT) layers processes this sequence, producing hidden features. 3) A specific hidden layer is selected (e.g. the penultimate one) along with a feature mode: "patch" (patch tokens without the CLS token) or "cls_patch". 4) The resulting features Z_v are projected by a projection matrix/module W into the LLM embedding dimension: H_v = W · Z_v. 5) The resulting visual tokens H_v are inserted into the text token sequence and fed to the language model. The tower itself is usually frozen; mainly the projector (and optionally the LLM) is trained.
Problem solved
Language models operate only on text tokens and cannot ingest an image as input. The Vision Tower solves this by encoding the image into a representation that can be projected into the LLM embedding space. Using a pretrained, frozen encoder avoids expensive visual pretraining from scratch and transfers rich visual knowledge (e.g. from CLIP) into the multimodal model.
Components
Splits the image into fixed-size patches (e.g. 14×14 px) and linearly embeds each patch as a vector, adding a positional embedding.
A stack of Transformer layers with multi-head self-attention processes the patch embeddings, building a representation that accounts for the global image context.
Official
Selects the hidden layer (e.g. the penultimate one) and the mode: "patch" (drop the CLS token) or "cls_patch" (keep both CLS and patches).
Official
Implementation
Evolution
Introduction of the ViT architecture — an image as a sequence of patches processed by a Transformer.
CLIP contrastively trains a separate image tower and text tower; the image tower becomes a reusable visual encoder.
LLaVA connects a frozen CLIP ViT-L/14 encoder as the vision_tower to an LLM via a linear projector, popularizing this pattern and name.
Hyperparameters (configurable axes)
Which pretrained image encoder serves as the tower (e.g. CLIP ViT-L/14, SigLIP).
Input image resolution; higher yields more patch tokens and higher cost.
Patch size in pixels; smaller patch = more tokens, finer granularity.
Which hidden layer features are taken from (e.g. the penultimate one).
Whether the encoder is frozen during training or trained.
Computational complexity
Time complexity: O(N² · d). Space complexity: O(N²).
Execution paradigm
A standard ViT performs dense computation — all patches pass through all layers.
Parallelism
Patches are processed in parallel within a layer; Transformer layers run sequentially.
Hardware requirements
A ViT-based tower is matmul-heavy, ideal for GPU tensor cores.
Transformer computation maps well onto TPUs.