Native Multimodal Architecture

Multimodal Large Language Models (MLLMs) are architectures built on top of pretrained decoder-only large language models (LLMs) that incorporate additional modality-specific encoders and a modality interface to process non-textual inputs—most commonly images, but also audio, video, and 3D data. The canonical MLLM architecture consists of three core modules: (1) a modality encoder that converts raw non-text inputs (e.g., image patches processed by a Vision Transformer) into token embeddings; (2) a modality interface (connector) that bridges the encoder and LLM spaces via projection layers, Q-Former modules (as in BLIP-2), or cross-attention layers (as in Flamingo); and (3) the pretrained LLM backbone that performs reasoning and text generation over the resulting combined token sequence. Optionally, a modality generator can be appended to produce non-textual outputs. During training, MLLMs typically proceed through pretraining (cross-modal alignment, with the LLM and often the encoder frozen), instruction tuning, and alignment tuning stages. The primary computational bottleneck is the quadratic complexity O(n²) of self-attention with respect to the total token count, which becomes severe when visual inputs are converted into hundreds or thousands of tokens—particularly for high-resolution images or video. Three main connector paradigms are used: projection-based (linear layer or MLP), query-based (Q-Former with learnable queries), and fusion-based (cross-attention into frozen LLM layers). Representative models include Flamingo (DeepMind, 2022), BLIP-2 (Salesforce, 2023), LLaVA (2023), GPT-4V (OpenAI, 2023), Gemini (Google, 2023), and Claude (Anthropic).

Mixture of Experts (MoE) is an architecture in which a model is composed of multiple parallel sub-networks — the experts — along with a gating (routing) network that determines, for each input, which subset of experts to activate and how to combine their outputs. The gating network produces a weighting over experts; in the original soft formulation (Jacobs et al., 1991), all experts are weighted and summed. In the sparse formulation (Shazeer et al., 2017), only the top-k scoring experts are activated, and the remaining experts produce no output and incur no compute cost for that input. In the context of large language models, MoE is typically applied as a replacement for the feed-forward network (FFN) sub-layer within each Transformer block. Each token is routed to a small number of expert FFNs (commonly top-1 or top-2), with the router being a learned linear projection followed by a softmax. The outputs of the selected experts are weighted by the corresponding router scores and summed. A central challenge in sparse MoE is load balancing: without explicit regularization, the router tends to collapse onto a small set of preferred experts, leaving others undertrained. This is addressed via auxiliary load balancing losses added to the training objective, which encourage a roughly uniform distribution of tokens across experts. Expert parallelism is the standard distributed training and inference strategy for large MoE models: each expert is placed on a separate device, so that the total parameter count scales with the number of devices without increasing per-device memory or per-token FLOPs proportionally. The capacity factor controls the maximum number of tokens each expert can process per batch; tokens that overflow the capacity are either dropped or passed through a residual connection. Tuning the capacity factor is a critical practical consideration.