AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

World Models

high confidence · updated 2026-08-13

Cross-cutting concept page for World Models: interactive predictive systems that simulate spatial-temporal environments in response to actions, P(s_{t+1} | s_t, a_t). Distinct from LLMs (model language) and video models (predict statistically plausible next frame). Promoted as the foundation-model class for embodied AI, robotics, and (per advocates) a path to AGI that LLMs alone cannot provide.

Contents

A World Model is an interactive predictive model that simulates spatial-temporal environments in response to actions. Where an LLM predicts $P(x_{t+1} | x_t)$ over tokens and a video model predicts $P(x_{t+1} | x_t)$ over pixels, a World Model predicts $P(s_{t+1} | s_t, a_t)$ — the next state conditioned on prior state and an action. Advocates promote the class as the foundation-model family for embodied AI and robotics, and, in the strongest framing, as a path to AGI that LLMs alone cannot provide.

According to McCormick & De Witte 2026, the action conditioning is what distinguishes a World Model from a generative video model: a video model is "a dream you stand in and watch", whereas a World Model is "a lucid dream you can shape".

The case for World Models

McCormick & De Witte frame the central argument as one of computational cost. Traditional simulation engines scale at $O(N)$ to $O(N^2)$ in the number of agents, objects, and interactions, so simulating thousands of stochastic human-like agents (for example, a soccer-stadium crowd) becomes computationally intractable. A World Model collapses the entire simulation into a fixed-cost forward pass through a neural network whose weights have absorbed real-world dynamics from training video. The authors present this as the proposed solution to a long-standing robotics bottleneck: machines must respond to real-world events in fixed time, but classical simulation costs grow with scene complexity (World Models: Computing the Uncomputable — Packy McCormick & Pim De Witte (Not Boring, March 2026)).

The advocates' further position is that language is "a lossy compression of reality" and code is "rule-bound and unable to handle the stochastic messiness of reality." On this view LLMs can describe clapping but not clap, and can talk about gravity but cannot know gravity the way a toddler does (World Models: Computing the Uncomputable — Packy McCormick & Pim De Witte (Not Boring, March 2026)). The bet is that physical embodiment requires a model that learned by interacting — observation-action-observation — rather than by reading text about physical behavior. McCormick & De Witte summarize the distinction as: "LLMs learn the structure of language. World Models learn the structure of causality."

The counter-position from the LLM-scaling camp is that a sufficiently large LLM with good multimodal grounding is itself an implicit World Model. Angjoo Kanazawa (UC Berkeley), quoted in the Béchard Scientific American piece, states: "In a way, I would say that the LLM already has a very good world model; it's just we don't really understand how it's doing it." Kanazawa adds that LLMs cannot update their training data in real time: "How do you develop an intelligent LLM vision system that can actually have streaming input and update its understanding of the world and act accordingly? That's a big open problem. I think AGI is not possible without actually solving this problem."

History

McCormick & De Witte 2026 organize the modern history into four waves:

  • Wave 0 (1990-1991) — Pre-deep-learning vision. Schmidhuber, *Making the World Differentiable* (1990) proposes an RNN that learns world dynamics with an agent inside the model. Sutton, Dyna (1991), unifies learning, planning, and reacting in one architecture.
  • Wave 1 (2018-2019) — "Can this work?" Ha & Schmidhuber's *World Models* paper demonstrates V (vision) + M (memory) + C (controller) on car-racing and FPS games. SimPLe / Atari 100k benchmark in 2019.
  • Wave 2 (2020-2022) — "Can it match human performance?" DreamerV2 (Hafner, Google DeepMind) hits human-level on a 55-game Atari benchmark on a single GPU. MuZero (DeepMind 2020) predicts in abstract latent space (no observable dream), beating Atari, Go, chess, and shogi. JEPA (LeCun 2022) — Joint Embedding Predictive Architecture; "predict meaning, not pixels". IRIS (Micheli, Alonso 2022) brings autoregressive transformer scaling to World Modeling.
  • Wave 3-4 (2023-2026) — Scale-up era: Genie / Genie 2 (DeepMind), Cosmos (NVIDIA), V-JEPA (Meta), Sora-style generative video (OpenAI), Marble (World Labs, late 2025). Ai2's MolmoBot (2025) demonstrates sim-to-real zero-shot transfer for robotic manipulation. General Intuition's action-conditioned-from-gaming-clips approach (2026, $133.7M Seed).

Schools of thought

By 2026 the field had roughly converged on three competing approaches:

  1. Generative pixel-space — World Labs, Google DeepMind, NVIDIA Cosmos, and Sora-derived work. These generate observable, watchable 3D worlds, useful for AR/VR, video, and design.
  2. Latent / JEPA (LeCun, AMI Labs) — Predict abstract representations rather than pixels, avoiding the "blurriness problem" of pixel prediction by sidestepping pixel space entirely.
  3. Action-conditioned (General Intuition) — Train on observation-action-observation data from gaming clips and similar sources, with the action serving as the compression. Optimized for embodied agents.

The camps differ on what the approach is ultimately for. McCormick & De Witte argue that good action-conditioned World Models are the path to AGI; LeCun argues the JEPA-style latent approach is essential; the generative camp sees the immediate market in entertainment and digital twins. Jim Fan, NVIDIA Director of Robotics, has said: "I bet with full confidence that 2026 will mark the first year that Large World Models lay real foundations for robotics, and for multimodal AI more broadly." Alexandre LeBrun, CEO of AMI Labs, has offered a skeptical reading of the term's commercial uptake: "My prediction is that 'World Models' will be the next buzzword. In six months, every company will call itself a World Model to raise funding." Demis Hassabis (DeepMind), in a late-April 2026 development-log item, projected AGI by 2030 while noting that reasoning and persistent memory are still missing, which he framed as an argument for World-Model-style additions to LLMs.

Active players

  • General Intuition — Pim De Witte, Kent Rollins. $133.7M Seed. Action-conditioned models from gaming clips.
  • World Labs (Fei-Fei Li) — $1B+ raised; launched Marble (3D worlds from text, images, video, and 3D layouts).
  • AMI Labs (Yann LeCun) — $1.03B raised; JEPA-descended philosophy.
  • Google DeepMind — Genie, V-JEPA, robotics policy work.
  • Runway — $860M raised ($5.3B February 2026 valuation); first world model launched December 2025, second planned for 2026; Q2 2026 +$40M ARR. A video-AI pivot toward world models, with biological world models and anti-aging described as a Germanidis long-horizon goal (Runway Started by Helping Filmmakers — Now It Wants to Beat Google at AI (Bellan, TechCrunch, May 2026)).
  • NVIDIA Cosmos — World Foundation Models positioned as compute infrastructure.
  • Niantic Spatial — Visual positioning model trained on 30B Pokémon Go / Ingress crowdsourced images; partnership with Coco Robotics for last-mile delivery (March 2026 MIT Technology Review).
  • OpenAI — A post-Sora shift: per April 2026 reporting, reallocating resources from the shuttered Sora video app to "longer-term world simulation research."
  • Figure AI — Helix-02 with System 0 (May 13, 2026): a whole-body controller trained on more than 1,000 hours of human motion data that replaced 109,000+ lines of hand-written C++, with robots running fully autonomous 8-hour shifts (Source: interestingengineering.com).

China's National People's Congress adopted the 15th Five-Year Plan in March 2026, which designates "embodied intelligence" as one of ten priority industry tracks and directs a national buildout of robot "training grounds" to generate proprietary physical-AI training data at scale. China is the largest single state actor explicitly committing to the world-models / embodied-AI paradigm (Source: sixdegreesofrobotics.substack.com).

Robotics deployment: Niantic Spatial and Coco Robotics

The Niantic Spatial partnership with Coco Robotics is a concrete instance of World Model deployment for robotics. Pokémon Go's 30B images of urban landmarks, tagged with accurate location markers, form the training corpus. The model can pinpoint location to within "a few centimeters" from a handful of snapshots, addressing the GPS-failure problem in dense urban canyons. Coco's 1,000+ delivery robots in LA, Chicago, Jersey City, Miami, and Helsinki use the model for last-mile navigation and accurate pickup-spot positioning.

Brian McClendon (Niantic Spatial CTO) describes the underlying problem: "The urban canyon is the worst place in the world for GPS… If you look at that blue dot on your phone, you'll often see it drift 50 meters." John Hanke (Niantic Spatial CEO) frames the connection between the game and the robots: "It turns out that getting Pikachu to realistically run around and getting Coco's robot to safely and accurately move through the world is actually the same problem." (Source: Heaven, *MIT Technology Review*, March 10 2026)

NVIDIA's vendor taxonomy and build pipeline

NVIDIA's corporate glossary entry "What Are World Models and How Are They Built?" (captured May 3, 2026) supplies a vendor-of-record definition and taxonomy that later coverage frequently maps back to. It defines a world model as a neural network that understands the dynamics of the real world — physics and spatial properties — from multimodal inputs (text, image, video, motion), and that generates videos simulating realistic physical environments. It introduces the term World Foundation Model (WFM) for a world model meeting the scale and generalizability bar of a foundation model — trained on massive unlabeled datasets, adaptable to a broad range of physical-AI tasks via post-training — and names NVIDIA Cosmos as its flagship WFM. The entry frames world models as a move from 2D digital outputs to, in its phrasing, "tangible, real-world experiences" (Source: nvidia.com).

The taxonomy has three types. Prediction models synthesize continuous motion from text prompts, input video, or by interpolating between two images, for video synthesis, animation, and robotic motion planning. Style-transfer models use ControlNet-style structured guidance (segmentation maps, lidar, depth maps, edge detection) to condition outputs while preserving layout and motion, for digital-twin simulation and environmental reconstruction. Reasoning models take multimodal inputs and apply chain-of-thought reasoning trained via reinforcement learning, for distinguishing real from synthetic data, selecting robot training data, predicting robotic actions, and logistics optimization (Source: nvidia.com).

The described build pipeline has four stages: data processing and curation over petabytes of multimodal data (filtering, annotation by vision-language models, classification, and semantic deduplication via video embeddings); tokenization of high-dimensional visual data into discrete tokens (integers) or continuous tokens (vectors); post-training, either unsupervised for generalization and domain adaptation or supervised for task-specific features and reasoning, using frameworks including NVIDIA NeMo; and reinforcement learning, presented as the substrate for reasoning models and the adapt-plan-decide loops associated with robotics and autonomous systems (Source: nvidia.com).

The glossary emphasizes three application areas: autonomous vehicles, via pre-labeled encoded video for training the AV stack and predictive simulation of new traffic patterns, weather, and lighting; robotics, via photorealistic synthetic data and predictive world states supporting simulation-based skill acquisition and reduced real-world testing risk; and video analytics, via natural-language question answering over video, automated summarization, and event localization, with industrial-safety and smart-city uses such as defect detection, crowd flows, and traffic (Source: nvidia.com). See Transportation — AI Deployment, Embodied AI vs AGI (China's Race Redirect).

As a vendor statement from a primary supplier of the underlying compute, the entry is best read alongside the independent framing in World Models: Computing the Uncomputable.

Relationships

Confidence is high for the technical definition and for the existence of the three schools, and high for the claim that major capital is flowing into World Model startups (World Labs $1B+, AMI $1.03B, General Intuition $133.7M). It is medium for capability claims about sim-to-real transfer, which remain promising but narrow demonstrations, and low (and contested) for the strongest position that World Models are necessary for AGI, which is the advocates' bet rather than the consensus.

Sources

Categories: Concepts & Frameworks | global | concept | world-models | embodied-ai | robotics | foundation-models | agi-paths | jepa | dreamer | action-conditioned
What links here (18)