1) A visual encoder (ViT, CLIP, SigLIP, ConvNeXt) converts an image into a sequence of patch tokens (typically 14x14 or 28x28 patches). 2) A projection (MLP or Q-Former / Perceiver Resampler) maps visual tokens into the LLM's embedding space. 3) The LLM (Transformer decoder) receives the sequence: [SYSTEM] + [image patch tokens] + [USER text prompt]. 4) During decoding, the LLM autoregressively generates the answer โ optionally with an emitted chain-of-thought (thinking tokens) before the final answer. 5) Reasoning-optimized models generate an additional visual grounding chain (e.g., "Let me look at the red region... [region marker] ... I see 3 spheres..."). 6) Tool-use models can invoke external functions (OCR, detector, calculator, Python) in a loop โ the local tool's output enters the context and influences the next step. 7) Final answer โ text, JSON, code, bounding boxes.
Recognition models (classification, detection, segmentation) excel at mapping image -> label but cannot perform multi-step operations requiring the integration of perception with logic, planning, and world knowledge. The query "how many red spheres larger than the cube on the right" requires: (1) object detection, (2) colour and shape classification, (3) spatial reasoning, (4) size comparison, (5) counting. No single classifier can do this by definition. Visual reasoning is therefore a key competency for realistic applications: robotics, document analysis, medical imaging, multimodal assistants, and desktop-use agents.
Length of chain-of-thought before the answer. Shorter = faster and cheaper, longer = higher accuracy on hard tasks. GPT-5.6 Sol/Kimi K3 support reasoning_effort=max; Gemini 3 Thinking automatically scales the budget to task difficulty.
Default 224x224 or 336x336 tokens for ViT. High-res models (Qwen2-VL, InternVL) handle up to 4K via dynamic patching. Higher resolution = better OCR and detail but higher cost (quadratic in token count).
Enabling tools (Python, OCR, search) radically improves accuracy on tasks requiring symbolic precision (mathematics, precise counting). Costs latency and requires a secure sandbox.
Models supporting multiple images (interleaved images/text, video frames) will be key for temporal reasoning. Gemini 3 Pro up to 3h video, Qwen2-VL up to 20 min. Requires an adequate context window (1M+ for video).
Structured prompt ("Answer in JSON: {count: N, boxes: [...]}") gives high accuracy for downstream tasks. Free-form works better for open explanations and tutoring.
For visual reasoning, greedy (temperature=0) or very low temperature is most common. High temperature causes region and count hallucinations.
Most VLMs are dense transformer decoders + a dense ViT encoder. Modern MoE models (Kimi K3, DeepSeek VL2) use sparse activation for the LLM part, but the visual encoder remains dense. All layers are active per token.
The visual encoder is fully parallel (classic CNN/ViT forward pass). LLM decoding is sequential (autoregressive) โ each token depends on the previous ones. Multi-image batching works in parallel. Device-level parallelism via tensor/pipeline parallelism for large VLMs.
