Three Competing AI World Model Architectures — And How They Aim to Master Physical Reality

As enterprises push AI out of web browsers and into robots, vehicles, factories and hospitals, the limits of today’s large language models (LLMs) are becoming clearer. LLMs are powerful at pattern-matching text, but they lack grounded understanding of physics and causality. That gap is now driving intense interest — and multibillion‑dollar funding — into “world models,” architectures designed to let AI systems reason about and act within the physical world.

Recent mega-rounds for AMI Labs (a $1.03 billion seed round) and Fei‑Fei Li’s World Labs ($1 billion) underscore how strategic these models have become for robotics, autonomous driving, manufacturing, healthcare, and spatial computing. Yet “world model” is not a single technology. Three distinct architectural families are emerging, each making different tradeoffs around real-time control, spatial fidelity, and compute cost.

This article unpacks those three approaches — JEPA-style latent world models, Gaussian splat spatial generative models, and end-to-end generative simulators — and closes with how they may converge into hybrid stacks alongside LLMs.

Why LLMs Hit a Wall in the Physical World

LLMs succeed by predicting the next token in a sequence of text. That mechanism is effective for capturing abstract knowledge encoded in language, but it does not force the model to build an internal physics or causality model. As a result, LLMs and even vision-language models (VLMs) struggle when actions must have reliable real-world consequences — from moving a robot arm to navigating an intersection.

Turing Award winner Richard Sutton summarized this limitation in a conversation with podcaster Dwarkesh Patel, explaining that LLMs essentially mimic what people say rather than modeling the world itself. That means they have limited capacity to truly learn from embodied experience or respond robustly to changes that weren’t in their training data.

Google DeepMind CEO Demis Hassabis has described the result as “jagged intelligence”: today’s models can solve math olympiad problems yet fail at basic physical reasoning. They can talk about friction and momentum, but they don’t reliably predict what happens if a robot pushes a box on a wet floor.

These gaps become visible in production systems. VLM-based agents can be brittle, failing under minor lighting changes or small shifts in camera angle. For robotics, automotive, and safety‑critical workflows, that fragility is untenable.

World models aim to address this by giving AI systems an internal simulator they can query: a structured representation of how entities, geometry, and forces evolve over time. But there are competing visions of what this simulator should look like and how tightly it should be coupled to perception and control.

JEPA: built for real-time

The first major approach, used by AMI Labs, is built around the Joint Embedding Predictive Architecture (JEPA). Instead of predicting pixels, JEPA-style world models learn compact latent representations that encode the underlying dynamics of a scene.

The design mirrors how humans intuitively process the world. When watching a car drive down a street, we track speed, direction, and likely trajectory, not the exact pattern of highlights on each leaf in the background. We focus on the variables that matter for prediction and control.

JEPA models operationalize this idea. Rather than forcing the network to output a full next video frame, they learn to map observations into a latent space and predict how that latent state will evolve. Irrelevant details are discarded; the model concentrates on core interaction rules between objects and agents.

For practitioners, this yields several concrete properties:

• Robustness to nuisance variation: Because the model does not try to reconstruct all pixels, small visual or background changes are less likely to break behavior.
• Compute and memory efficiency: Operating in a low‑dimensional latent space reduces both training data needs and inference latency.
• Suitability for tight control loops: Low latency and compact state make JEPA architectures attractive for robotics, autonomous driving, and other domains where real‑time responsiveness is critical.

AMI is already exploring these properties in high‑pressure operational settings. In partnership with healthcare company Nabla, AMI is applying JEPA-style models to simulate operational complexity and reduce cognitive load in fast-paced clinical environments — a scenario where systems must react quickly and reliably to changing conditions.

Yann LeCun, a key architect of JEPA and a co‑founder of AMI, has emphasized in an interview with Newsweek that JEPA-based world models are designed to be controllable: they can be given goals and, by construction, are oriented toward accomplishing those goals. For engineering teams, that suggests a pathway to more predictable, goal-driven control policies sitting on top of a learned world representation.

The tradeoff is that JEPA prioritizes abstract dynamics over visual or spatial richness. If your primary need is fine-grained, photorealistic environment construction, other approaches may fit better — but if you care about control performance and latency, JEPA-like architectures are emerging as a compelling default.

Gaussian splats: built for space

The second architectural family focuses less on fast control and more on spatial fidelity. World Labs, founded by Fei‑Fei Li, epitomizes this track through generative models that construct full 3D environments using Gaussian splats.

A Gaussian splat is a way of representing a 3D scene as a large collection of tiny, parameterized particles. Each particle encodes position, shape, color, and how it interacts with light. When rendered together, these particles form detailed 3D scenes that can be viewed from arbitrary viewpoints.

World Labs’ systems take an initial prompt — an image or a textual description — and generate a 3D Gaussian splat representation. Unlike conventional video generation, the output is a navigable 3D scene that can be imported directly into physics and 3D engines such as Unreal Engine. Human users or AI agents can then move through the environment, interact with objects, and run simulations from any camera angle.

The core benefit is a drastic reduction in time and one-time generation cost for complex, interactive 3D environments. This directly targets what Fei‑Fei Li has called the “wordsmiths in the dark” problem: LLMs have rich linguistic capabilities but lack spatial experience and physical context. World Labs’ Marble model is designed to give AI agents that missing spatial awareness by embedding them in generated 3D worlds.

For technical teams, this suggests several use cases:

• Spatial computing and XR: Quickly generating immersive 3D environments for headsets or spatial interfaces.
• Interactive entertainment: Building game-like worlds or simulations that agents and players can explore.
• Industrial and architectural design: Creating rich digital twins or concept spaces far faster than manual modeling.
• Static training grounds for robots: Producing diverse virtual environments in which embodied agents can be trained offline.

This approach is less focused on split-second real-time control than on high-fidelity, navigable worlds. That orientation aligns with enterprise backers such as Autodesk, which has heavily supported World Labs to integrate these models into industrial design workflows. For design and simulation teams, this kind of “world factory” offers a way to rapidly iterate environments before physical prototyping.

End-to-end generation: built for scale

The third approach fuses world modeling, rendering, and physics into a single generative system. Instead of generating a static 3D asset to be passed into an external engine, these end-to-end models act as the engine themselves.

DeepMind’s Genie 3 and Nvidia’s Cosmos are two examples. Both ingest an initial prompt plus a continuous stream of user or agent actions. In response, the model generates subsequent frames of the environment in real time, while internally computing dynamics, lighting, and object interactions.

DeepMind has demonstrated Genie 3 maintaining object permanence and consistent physics at 24 frames per second without an explicit memory module. From the outside, this looks like a self-contained game engine driven purely by a neural model: you act; the world responds; the model keeps track of what exists where and how it should behave.

Nvidia’s Cosmos uses a similar paradigm to power large-scale synthetic data factories. For autonomous vehicle and robotics developers, this means being able to synthesize rare and dangerous edge cases — near collisions, unusual weather, complex multi-agent traffic scenarios — without orchestrating costly real-world tests. Waymo, another Alphabet company, has built its own world model on top of Genie 3 specifically for training self-driving cars, illustrating how these architectures can be adapted for domain-specific autonomy stacks.

The advantages of this end-to-end method include:

• Unified interface: A single model handles scene evolution, physics, and rendering.
• Massive synthetic data throughput: Ideal for generating large, diverse datasets for training perception and control systems.
• Flexible interactivity: Agents and humans can interact with the world while the model adapts in real time.

The downside is cost. Continuously rendering both physics and pixels at interactive frame rates demands significant compute. For organizations considering such systems, capacity planning and hardware strategy become first-order concerns.

Yet this expense may be unavoidable for the level of physical understanding Hassabis argues is required. To operate safely in unconstrained environments, AI systems need a deep internal model of causality and dynamics — exactly what these “end-to-end engines” aim to learn.

What this means for robotics and autonomy builders

For teams building robots, autonomous vehicles, or complex physical workflows, these three world modeling strategies map to different priorities.

If you are constrained by latency and tight control loops — for example, manipulating objects on a conveyor belt or coordinating robots in a warehouse — JEPA-style latent models offer a path to efficient prediction and control without overpaying for visual fidelity.

If your focus is on environment design, user experience, or spatial interfaces — such as AR/VR applications, industrial design, or training environments where photorealism and navigability matter — Gaussian splat models are attractive. They let you generate rich 3D spaces that plug into existing engines and tools.

If your main bottleneck is data — especially edge cases for safety-critical systems — then end-to-end generative simulators like Genie 3 and Cosmos align with those needs. They let you treat the model as a synthetic data factory and closed-loop simulator, albeit at a high compute cost.

Across all three, LLMs remain relevant but in a different role: as reasoning and communication layers. Rather than being the sole brain of an embodied system, LLMs sit alongside world models, translating human intent into goals, interpreting outcomes, and coordinating tasks.

Hybrid architectures and the road ahead

As these lines of work mature, hybrid architectures are already emerging. Instead of choosing one world model type, systems can combine them, using each where its strengths dominate.

One example is DeepTempo’s LogLM, which integrates ideas from LLMs and JEPA-style representations to detect anomalies and threats in security and network logs. While this is a cyber, not physical, domain, the pattern is instructive: a language-like interface fused with a learned world representation tailored to the data’s structure.

In physical domains, similar patterns are likely: JEPA-inspired latent models for fast control, Gaussian splats or related techniques for environment authoring and visualization, and end-to-end generative engines for high-volume simulation and edge-case synthesis — all orchestrated by LLM-based agents.

For AI practitioners and technical decision-makers, the key is to treat “world models” as a design space, not a single product. Clarify whether your priority is real-time control, spatial richness, or synthetic data scale, and then align with the architecture family — or combination — that best matches those constraints. The funding momentum behind AMI Labs, World Labs, and large efforts at DeepMind and Nvidia suggests that whichever mix wins out, world models will become core infrastructure for any AI system that must reliably touch the physical world.