1) The image is split into patches and turned into a sequence of patch embeddings. 2) A stack of Transformer (ViT) layers processes this sequence, producing hidden features. 3) A specific hidden layer is selected (e.g. the penultimate one) along with a feature mode: "patch" (patch tokens without the CLS token) or "cls_patch". 4) The resulting features Z_v are projected by a projection matrix/module W into the LLM embedding dimension: H_v = W · Z_v. 5) The resulting visual tokens H_v are inserted into the text token sequence and fed to the language model. The tower itself is usually frozen; mainly the projector (and optionally the LLM) is trained.
Language models operate only on text tokens and cannot ingest an image as input. The Vision Tower solves this by encoding the image into a representation that can be projected into the LLM embedding space. Using a pretrained, frozen encoder avoids expensive visual pretraining from scratch and transfers rich visual knowledge (e.g. from CLIP) into the multimodal model.
Splits the image into fixed-size patches (e.g. 14×14 px) and linearly embeds each patch as a vector, adding a positional embedding.
A stack of Transformer layers with multi-head self-attention processes the patch embeddings, building a representation that accounts for the global image context.
Official
Selects the hidden layer (e.g. the penultimate one) and the mode: "patch" (drop the CLS token) or "cls_patch" (keep both CLS and patches).
Official
Introduction of the ViT architecture — an image as a sequence of patches processed by a Transformer.
CLIP contrastively trains a separate image tower and text tower; the image tower becomes a reusable visual encoder.
LLaVA connects a frozen CLIP ViT-L/14 encoder as the vision_tower to an LLM via a linear projector, popularizing this pattern and name.
Time complexity: O(N² · d). Space complexity: O(N²).
Which pretrained image encoder serves as the tower (e.g. CLIP ViT-L/14, SigLIP).
Input image resolution; higher yields more patch tokens and higher cost.
Patch size in pixels; smaller patch = more tokens, finer granularity.
Which hidden layer features are taken from (e.g. the penultimate one).
Whether the encoder is frozen during training or trained.
A standard ViT performs dense computation — all patches pass through all layers.
Patches are processed in parallel within a layer; Transformer layers run sequentially.
A ViT-based tower is matmul-heavy, ideal for GPU tensor cores.
Transformer computation maps well onto TPUs.
