Robots Atlas>ROBOTS ATLAS
Multimodal

Multimodal AI

2021ActiveUpdated: 14 August 2026Published
Key innovation
Lets a single AI system process and/or generate multiple modalities (text, image, audio, video) in a shared representation space, instead of separate models per modality.
Category
Multimodal
Abstraction level
Paradigm
Operation level
ModelTrainingInference
Use cases
Answering questions about images and video (visual QA)Image captioning and descriptionGenerating images/video from textCross-modal retrieval (text<->image)Voice-and-vision assistants and real-time translationVision-Language-Action robotics (perception->action)Analyzing documents with text, tables and charts

How it works

Each modality is encoded into feature vectors by a dedicated encoder (e.g. ViT for images, a tokenizer for text, an audio encoder). The representations are projected into a shared space and combined (early, late or joint fusion) — e.g. via token concatenation, cross-attention or contrastive learning (as in CLIP). A shared backbone (usually a Transformer) processes the combined representations, and appropriate heads/decoders produce output in the target modality (text, image, audio). Models can be built by attaching encoders to an existing LLM or trained natively multimodal from scratch.

Problem solved

The world is inherently multimodal, and unimodal models (e.g. text-only) cannot relate information across image, audio and text. Multimodal AI addresses the problem of siloed, narrow models by enabling joint understanding and generation across modalities.

Key mechanisms

Dedicated modality encoders (e.g. a Vision Transformer)
A shared (joint) representation space
Modality fusion: early / late / joint (cross-attention, concatenation)
Contrastive learning that aligns modalities (e.g. CLIP)
A shared Transformer backbone and heads/decoders for output modalities

Strengths & limitations

Strengths
✓Joint understanding of content across modalities
✓Flexibility: one system instead of many narrow models
✓Knowledge transfer across modalities
✓Richer, more natural interactions (text+image+voice)
✓A foundation for agents and robots acting in the physical world
Limitations
✗Modality alignment is hard (misalignment, representation drift)
✗Costly training and inference (many encoders, large data)
✗Scarcity of well-paired multimodal data
✗Cross-modal hallucinations (e.g. describing things absent from an image)
✗Uneven quality across modalities; data biases and gaps

Components

Modality encodersInput / feature extraction

Separate encoders turning each modality (image, audio, text) into feature vectors.

Official

Projection & fusion moduleAlignment and combination

Projects modality representations into a shared space and combines them (concatenation, cross-attention).

Official

Shared backboneComputational core

A shared model (usually a Transformer) that processes the combined representations.

Output heads / decodersOutput

Produce output in the target modality (text, image, audio).

Official

Implementation

Reference implementations
CLIP
Python · OpenAI
Official
LLaVA
Python · LLaVA (Haotian Liu et al.)
Official
Hugging Face Transformers — modele multimodalne
Python · Hugging Face
Official
Implementation pitfalls
Modality misalignmentHigh

Representations of different modalities are poorly aligned, breaking cross-modal grounding.

Fix:Contrastive learning (e.g. CLIP), careful data pairing, retrieval validation.
Cross-modal hallucinationHigh

The model describes objects absent from an image or confuses modality content.

Fix:Grounding, better instruction data, faithfulness-focused evaluation.
Visual token blow-upMedium

High-res images/video produce huge numbers of tokens, raising cost and latency.

Fix:Token compression, adaptive resolution, pooling.
Scarcity of paired dataMedium

High-quality pairs (e.g. audio-text) are scarce for rarer modalities.

Fix:Synthetic data, self-supervised pretraining, augmentation.

Evolution

2015
Show and Tell — image captioning

Early vision-language coupling: generating image captions.

2021
CLIP — contrastive image-text alignment
Inflection point

A shared image-text embedding space learned contrastively; an inflection for multimodality.

2021
DALL·E — text-to-image generation

Generative multimodality: images from text descriptions.

2022
Flamingo — few-shot VLM

A vision-language model capable of in-context learning over interleaved images and text.

2023
GPT-4V — multimodal LLM goes mainstream

Broad availability of image understanding in a large LLM.

2024
GPT-4o / Gemini — native multimodality
Inflection point

Models trained natively on text, image, audio and video with low latency.

Hyperparameters (configurable axes)

Supported modalitiesCritical

Which modalities are supported on input and output.

text, imageA classic VLM.
text, image, audio, videoA natively multimodal model (e.g. GPT-4o, Gemini).
Fusion strategyHigh

When and how modalities are fused.

early / jointJoint encoding from the start (native multimodality).
lateSeparate encoders, combined at the end.
Training approachHigh

Natively multimodal training vs. bolting encoders onto an existing LLM.

nativeTrained from scratch on multiple modalities.
adapter/bolt-onVision encoder + projection into an existing LLM (e.g. LLaVA).

Computational complexity

Time complexity: O(Σ_m C_m + F). Space complexity: O(Σ_m T_m · d).

Execution paradigm

Primary mode
Dense

Most multimodal transformers are dense; some large models use MoE (mixture) for efficiency.

Activation pattern
All paths active
Additional modes
Mixture

Parallelism

Parallelism level
Partially parallel

Per-modality encoders can be computed in parallel; the shared Transformer backbone has the usual training/inference parallelism.

Scope
TrainingInferenceAcross devices

Hardware requirements

Primary

Training and inference of heavy vision/video encoders and large transformers.

Good fit

Large multimodal models trained/served on TPUs (e.g. Gemini).