Multimodal
Vision-Language Model (VLM)
Category
Multimodal
Abstraction level
Paradigm
Operation level
ModelInference
Use cases
Image captioning and visual question answering (VQA)Multimodal AI assistantsDocument and chart analysisRobotics perception (basis for VLA models)
How it works
A vision encoder encodes the image into feature vectors that a projection module maps into the language model's space; the LLM processes the combined visual features and text to generate a response. Training uses large image-text pair datasets.
Problem solved
How to build a model that understands both image and language and can converse and reason about an image?
Strengths & limitations
Strengths
✓Combines visual perception with language reasoning
✓Versatility (many multimodal tasks)
✓Foundation for agents and robots acting in the physical world
Limitations
✗Hallucinations when describing images
✗Limits in precise object localization and counting
✗High cost of multimodal training