Each modality is encoded into feature vectors by a dedicated encoder (e.g. ViT for images, a tokenizer for text, an audio encoder). The representations are projected into a shared space and combined (early, late or joint fusion) — e.g. via token concatenation, cross-attention or contrastive learning (as in CLIP). A shared backbone (usually a Transformer) processes the combined representations, and appropriate heads/decoders produce output in the target modality (text, image, audio). Models can be built by attaching encoders to an existing LLM or trained natively multimodal from scratch.
The world is inherently multimodal, and unimodal models (e.g. text-only) cannot relate information across image, audio and text. Multimodal AI addresses the problem of siloed, narrow models by enabling joint understanding and generation across modalities.
Separate encoders turning each modality (image, audio, text) into feature vectors.
Official
Projects modality representations into a shared space and combines them (concatenation, cross-attention).
Official
A shared model (usually a Transformer) that processes the combined representations.
Produce output in the target modality (text, image, audio).
Official
Representations of different modalities are poorly aligned, breaking cross-modal grounding.
The model describes objects absent from an image or confuses modality content.
High-res images/video produce huge numbers of tokens, raising cost and latency.
High-quality pairs (e.g. audio-text) are scarce for rarer modalities.
Early vision-language coupling: generating image captions.
A shared image-text embedding space learned contrastively; an inflection for multimodality.
Generative multimodality: images from text descriptions.
A vision-language model capable of in-context learning over interleaved images and text.
Broad availability of image understanding in a large LLM.
Models trained natively on text, image, audio and video with low latency.
Time complexity: O(Σ_m C_m + F). Space complexity: O(Σ_m T_m · d).
Which modalities are supported on input and output.
When and how modalities are fused.
Natively multimodal training vs. bolting encoders onto an existing LLM.
Most multimodal transformers are dense; some large models use MoE (mixture) for efficiency.
Per-modality encoders can be computed in parallel; the shared Transformer backbone has the usual training/inference parallelism.
Training and inference of heavy vision/video encoders and large transformers.
Large multimodal models trained/served on TPUs (e.g. Gemini).
