Visual Reasoning

2015ActivePublished: 3 May 2015Updated: 17 July 2026Published

Key innovation

Transferring multi-step reasoning from the purely linguistic domain (chain-of-thought in LLMs) to the multimodal domain while preserving the strength of reasoning. Modern VLMs (from Flamingo/LLaVA/GPT-4V) learned in parallel: (1) to align visual and linguistic embedding spaces (via CLIP-like contrastive learning + instruction tuning), (2) to perform chain-of-thought with references to image regions, (3) to invoke tools (detectors, OCR, code) when precise symbolic reasoning is needed. The result: a single model can solve a problem requiring perception + logic + knowledge + planning, accessible via a simple API.

How it works

1) A visual encoder (ViT, CLIP, SigLIP, ConvNeXt) converts an image into a sequence of patch tokens (typically 14x14 or 28x28 patches). 2) A projection (MLP or Q-Former / Perceiver Resampler) maps visual tokens into the LLM's embedding space. 3) The LLM (Transformer decoder) receives the sequence: [SYSTEM] + [image patch tokens] + [USER text prompt]. 4) During decoding, the LLM autoregressively generates the answer — optionally with an emitted chain-of-thought (thinking tokens) before the final answer. 5) Reasoning-optimized models generate an additional visual grounding chain (e.g., "Let me look at the red region... [region marker] ... I see 3 spheres..."). 6) Tool-use models can invoke external functions (OCR, detector, calculator, Python) in a loop — the local tool's output enters the context and influences the next step. 7) Final answer — text, JSON, code, bounding boxes.

Problem solved

Recognition models (classification, detection, segmentation) excel at mapping image -> label but cannot perform multi-step operations requiring the integration of perception with logic, planning, and world knowledge. The query "how many red spheres larger than the cube on the right" requires: (1) object detection, (2) colour and shape classification, (3) spatial reasoning, (4) size comparison, (5) counting. No single classifier can do this by definition. Visual reasoning is therefore a key competency for realistic applications: robotics, document analysis, medical imaging, multimodal assistants, and desktop-use agents.

Components

Visual encoder (ViT / CLIP / SigLIP)Converts an image into a sequence of vectors (patch tokens). Commonly a Vision Transformer trained contrastively (CLIP) or with sigmoid loss (SigLIP) on billions of image-text pairs from the web.

Projector / adapter (MLP, Q-Former, Perceiver Resampler)Maps visual embeddings from the encoder dimension to the LLM dimension. Q-Former (BLIP-2) and Perceiver Resampler (Flamingo) additionally reduce token count (learnable query tokens).

LLM decoder (Transformer)Autoregressively generates the answer from the concatenated [image + text] sequence. Usually a pretrained LLM (Llama, Qwen, GPT) fine-tuned on visual instruction data (LLaVA-instruct, ShareGPT-4V).

Chain-of-thought / thinking budgetReasoning-time compute mechanism: the model generates a thinking sequence (thinking tokens) before the answer. Models with controllable reasoning effort (Gemini Thinking, Kimi K3, GPT-5.6 Sol with reasoning_effort=max) substantially improve visual reasoning at the cost of latency.

Tool use / function callingThe model can call specialised tools for precise symbolic reasoning: OCR (for text), object detector (for precise counting), Python (for calculations), search (for external knowledge). Critical for accuracy on tasks requiring symbolic precision.

Grounding annotations (bounding boxes, points, masks)Structured output binding textual conclusions to image regions. Standard formats: bounding boxes [x1,y1,x2,y2], points (x,y), Set-of-Mark (numbered regions), scene graph. Used in robotics, documents, UI agent workflows.

Implementation

Reference implementations

LLaVA (Large Language and Vision Assistant)

Qwen2.5-VL (Alibaba)

InternVL (Shanghai AI Lab)

VisProg (Compositional Visual Reasoning by Programming)

Implementation pitfalls

Critical

Fix:VLMs sometimes "see" things that are not in the image — especially when the prompt suggests their presence (leading questions). Mitigation: neutral prompts, verify via tool use (call detector), grounding chain with bounding boxes, testing on the POPE benchmark (Polling-based Object Probing Evaluation).

High

Fix:Many models "solve" hard visual reasoning tasks by pattern-matching to training data rather than actually reasoning. The VERIFY benchmark (2025) and MMMU-Pro are specifically designed to detect these shortcuts. Use contrast sets and held-out compositional test sets.

High

Fix:VLMs are weak at precise counting (>10 objects), geometric measurements, angles, and lengths. Delegate to external tools (SAM detector + counter, symbolic geometry via Python). Modern reasoning-optimized models (Kimi K3) substantially reduce this problem.

High

Fix:Changing wording drastically changes the answer. The model may correctly count 5 dogs when asked about "dogs" but not about "four-legged animals". Mitigation: prompt ensembling, self-consistency (multiple paraphrased queries), tool use.

Medium

Fix:The model may return bounding boxes in different formats (normalised 0-1, absolute pixels, [x,y,w,h] vs [x1,y1,x2,y2], YOLO format). Enforce a schema via structured output APIs (OpenAI/Anthropic response_format, JSON schema), validate with Pydantic.

Medium

Fix:Full 1080p 30fps video for 1 minute = ~50k image tokens for the encoder alone. Optimisations: frame subsampling, spatial downscaling to task-relevant regions, keyframe extraction, using a smaller VLM for initial region selection.

Evolution

Original paper · 2015

VQA: Visual Question Answering

2015

2016

2019

2021

2022

2023

2024

2025

2026

Hyperparameters (configurable axes)

Reasoning effort / thinking budgetCritical

Length of chain-of-thought before the answer. Shorter = faster and cheaper, longer = higher accuracy on hard tasks. GPT-5.6 Sol/Kimi K3 support reasoning_effort=max; Gemini 3 Thinking automatically scales the budget to task difficulty.

Input image resolutionHigh

Default 224x224 or 336x336 tokens for ViT. High-res models (Qwen2-VL, InternVL) handle up to 4K via dynamic patching. Higher resolution = better OCR and detail but higher cost (quadratic in token count).

Tool use enabledHigh

Enabling tools (Python, OCR, search) radically improves accuracy on tasks requiring symbolic precision (mathematics, precise counting). Costs latency and requires a secure sandbox.

Multi-image / video contextMedium

Models supporting multiple images (interleaved images/text, video frames) will be key for temporal reasoning. Gemini 3 Pro up to 3h video, Qwen2-VL up to 20 min. Requires an adequate context window (1M+ for video).

Prompt structure (structured vs free-form)Medium

Structured prompt ("Answer in JSON: {count: N, boxes: [...]}") gives high accuracy for downstream tasks. Free-form works better for open explanations and tutoring.

Temperature / samplingLow

For visual reasoning, greedy (temperature=0) or very low temperature is most common. High temperature causes region and count hallucinations.

Execution paradigm

Primary mode

Dense

Most VLMs are dense transformer decoders + a dense ViT encoder. Modern MoE models (Kimi K3, DeepSeek VL2) use sparse activation for the LLM part, but the visual encoder remains dense. All layers are active per token.

Activation pattern

All paths active

Parallelism

Parallelism level

Partially parallel

The visual encoder is fully parallel (classic CNN/ViT forward pass). LLM decoding is sequential (autoregressive) — each token depends on the previous ones. Multi-image batching works in parallel. Device-level parallelism via tensor/pipeline parallelism for large VLMs.

Scope

InferenceAcross devices