Multimodal AI refers to models that natively process or generate more than one modality — typically some combination of text, image, audio, video, code, and action (tool use, embodied control). The term "natively" distinguishes recent systems from early ones: early systems glued separate encoders together, while frontier systems increasingly use a single transformer with modality-agnostic tokenization.
Modalities
The modalities most commonly combined are:
- Text — the original and still-dominant modality.
- Image — input (vision-language models) and output (image generation).
- Audio — speech recognition, speech synthesis, and increasingly "audio-in, audio-out" realtime conversation.
- Video — input (video understanding) and output (video generation, as in Sora and Veo).
- Code — often treated as a sub-modality of text but with distinct capabilities.
- Action — tool calls, UI interactions, and robot control, represented as "action tokens" in recent vision-language-action (VLA) models.
- Embodied / sensor — 3D scenes, depth, tactile, and proprioception, used in frontier robotics.
Architectural approaches
Four approaches to combining modalities are commonly distinguished:
- Bolted-on encoders. A frozen or separately trained vision encoder (CLIP, SigLIP) feeds embeddings into an LLM. This approach is cheap and flexible and was dominant through 2023 (for example LLaVA and early GPT-4V pipelines as publicly reported).
- Joint training with modality adapters. Vision and language are trained jointly with adapter layers, giving better integration while remaining staged.
- Native multimodal. One model with one tokenizer space, end-to-end joint training, in which speech tokens, image tokens, and text tokens are treated uniformly. GPT-4o (May 2024) and Gemini 1.0/2.0 were the first widely deployed examples.
- Vision-Language-Action (VLA). Action tokens are added for robotics or agent control, as in Google RT-2, Physical Intelligence π0, and DeepMind Gemini Robotics.
History
Contrastive image-text training arrived with CLIP (OpenAI, 2021), which became foundational for the LLaVA/Flamingo era. Flamingo (DeepMind, 2022) was an early few-shot vision-language model. GPT-4V (OpenAI, Sep 2023) brought vision capability into a frontier chat model at scale.
GPT-4o (OpenAI, May 2024) was the first native multimodal frontier model, with reduced latency on audio because speech did not detour through text. The Gemini 1.0/1.5/2.0 line (Google DeepMind, 2023–2024) was marketed as native multimodal from launch and pushed long-context video understanding. Video generation advanced with Sora (OpenAI, Feb 2024) and Veo (Google, 2024).
Anthropic's Claude 3 / 3.5 / 4 / 4.5 / 4.6 models (2024–2025) included vision capability across all Claude 3 and later models, and Computer Use added action. Later frontier models — Gemini 2.5 / 3, GPT-4.5, and GPT-5.x (2025–2026) — became fully native across text, image, audio, and video.
Current state
All frontier Western models (GPT, Claude, Gemini) are multimodal at input, and most are multimodal at output for image and audio; video generation remains a separate model lineage in most cases. Chinese frontier models (Qwen, DeepSeek, Kimi, GLM) are also multimodal or are shipping multimodal variants. The "omni" framing — one model spanning all modalities — dominates frontier roadmaps. Robotics multimodality through VLA models is an active frontier, with Physical Intelligence, Skild, Figure, Tesla Optimus, and Google DeepMind all training large VLA systems.
Policy and safety relevance
Multimodality expands the attack surface for prompt injection: instructions can be embedded in images (typographic attacks, steganographic instructions) and audio (hidden commands in music or ambient noise). Image, audio, and video generation underpin deepfake, NCII, and election-interference concerns, and are a driver of provenance and watermarking regulation. Image and video generation also trigger stringent CSAM regulatory regimes globally.
Multimodal models can identify people, transcribe private conversations, and track behaviors, which intersects with EU AI Act (Regulation 2024/1689) biometric restrictions and broader surveillance policy. Action-capable multimodal models (Computer Use, Operator) add physical-world and digital-infrastructure risk. Evaluation lags behind: most benchmarks are text-based, and multimodal safety evaluation is less developed.
See also
Relationships
- depends-on: Scaling Laws — same scaling story, extended across modalities
- related: Agentic AI — action is a modality; agents are increasingly multimodal
- related: Prompt Injection — multimodal inputs expand the injection surface
- related: Data Provenance, C2PA, and Watermarking — multimodal outputs drive provenance regulation
- related: AI and Content Moderation, EU AI Act (Regulation 2024/1689) — synthetic media and biometric regulation
- instance-of: Gemini 3 / Gemini 3 Pro, Claude Opus 4.6, GPT-5.4 Thinking, Qwen3