PAN 2: A foundation world model for physical intelligence

Introduction

Today, we are thrilled to announce PAN 2, a foundation world model for physical intelligence.

Much of what makes an AI system intelligent with language comes from its ability to predict the next word. Physical intelligence, competence in acting in the physical world, rests in the same way on predicting the next world, or simulating how it will change in response to actions. Catching a ball requires anticipating contact and timing; pouring a cup of coffee, the physics of liquid flow; and stepping around a passer-by, the trajectory of another person across viewpoints. A system’s physical intelligence thus lies not in the abstract quality of its representations or the dexterity of its actions alone, but in how faithfully it can simulate the consequences of those actions. These simulations must produce observations that can be checked against the real world.

We call a model with this ability a world model, and it enables simulative reasoning, where an agent imagines the outcomes of its candidate actions before choosing among them. The more faithful the simulation, the closer those choices come to the right ones in the real world, and the more an agent can learn and plan without costly, slow or irreversible physical trials. To be useful, the simulated futures must be physically plausible sequences of consequences rather than renderings of whatever outcome that was requested. What it means to simulate the world well can be broken down into six dimensions of physical intelligence (6D-PI), from obeying the laws of physics to keeping the world moving on its own (see Evaluating World Models).

PAN 2 is designed to deliver this simulation at scale. It follows the Generative Latent Prediction (GLP) architecture, which learns latent world states through supervision on the next observations they reconstruct. A large language model (LLM), the conceptual reasoning backbone, predicts the next world state as a structured, time-grounded description of who acts, when, and how the camera moves. A latent diffusion model (LDM), the perceptual reasoning backbone, then predicts the perceptual detail that language leaves open, such as contact, deformation, and motion. An external agent can compare these simulated outcomes, revise plans, and choose what to execute, with new observations informing subsequent reasoning.

We build PAN 2 by coupling a pretrained LLM for conceptual reasoning with a perceptual reasoning backbone that we train from scratch on diverse real-world footage, and designing the text-based state representation that connects them. Its simulations must preserve action timing, physical interactions, and spatial coherence. This motivates PAN 2’s grounding in time, physics, and space.

Grounded in time, physics, and space

Explore all three themes: Time, Physics, and Space.Select a card to watch its video and browse individual examples.

01Time2:2802Physics2:1003Space2:22

Time-grounded actions. In the physical world, timing affects outcomes. The same press of the brake can stop a car short or end in a collision, depending on when it comes. The conceptual reasoning backbone (LLM) ties each world state and agent action to a precise timestamp, so the timing and order of actions are explicit for simulation. Each simulation builds on the outcome before it, so consequences carry forward: a car that swerves to avoid one obstacle faces the road from a new position, and every later decision starts from there.

Jump to a demo

Shirt foldingPaper planeCrossing trafficPaper wickingSteeping teaCooking eggsTimed jumpsJump timingCovered bowlCup shuffleHourglass

0:00

0:00

How PAN 2 compares

PAN 2 is built for AI agents to use, so we judge it mainly by automated checks of what its simulations show. We build our own test cases for the six dimensions of physical intelligence (6D-PI) and for multi-step plans (see Evaluating World Models), and complement them with public physics (Physics-IQ) and robotics (RBench) benchmarks. PAN 2 completes 75.1% of individual steps and 34.0% of full plans, achieving the highest full-plan completion among the models we studied while remaining competitive on single actions across the six dimensions. It would also rank first among open models and third overall on Physics-IQ Verified in the public leaderboard setting. We also assess simulation quality through blinded human comparisons based on action adherence, physical plausibility, and world consistency. PAN 2 ranks second among the four evaluated open models.

Architecture

How does PAN 2 turn an action instruction into a simulated future? The answer lies in how it splits reasoning from rendering. The PAN 2 architecture comprises an encoder, a backbone that reasons by predicting the next state, and a visual decoder that renders that state into the next observation. Formally, PAN 2 models the next observation, a video, given the current observation (an optional image with text) and an action (an instruction in text with the duration to simulate), through estimated latent states, as shown in the equation below:

From reasoning to simulation

The encoder represents the starting observation in two ways: the vision encoder of a pretrained vision-language model captures its conceptual content, while a VAE encoder preserves perceptual detail as video latents. The conceptual reasoning backbone, an LLM, uses the observation and the action to predict a text-based next state comprising global scene attributes and time-grounded events. The perceptual reasoning backbone, an LDM implemented as an MMDiT, uses this text-based state and optional starting-frame latents to predict video latents over a window of up to 20 seconds. The VAE decoder renders these latents into observation frames. The text-based state serves as the intermediate representation between the two backbones: the LLM reasons explicitly about events, actions, and camera motion, while the LDM predicts the perceptual details the description leaves open.

Both backbones operate on latent states, with the perceptual state decoded into observations. Supervision on these generated observations grounds latent-state learning in the observed world.

Evaluating World Models

A simulation can look right and still be wrong. A door may swing open before it is unlocked, or the camera may barely turn when asked to look behind. Metrics of video quality don’t catch this, so we built our own test cases, organized around the six dimensions of physical intelligence: physical consistency, causal coherence, entity persistence, constraint satisfaction, viewpoint invariance, and objective dynamism. Together, they ask whether simulations obey physical laws, follow actions in order, preserve entities and their states, respect action prerequisites, keep the world unchanged as the camera moves, and keep the world moving beyond the agent’s actions. We run them in two studies, single actions and multi-step plans; a VLM judge answers yes/no questions about each simulation, and VGGT-Ω verifies camera motion from estimated geometry.

Full plans compound on single steps

Across all six models, full-plan completion is lower than step completion because errors accumulate across a plan. The results follow a single compounding curve (R² = 0.986): as individual steps become more reliable, each improvement yields a larger gain in full-plan completion. The fitted curve suggests that raising step completion from 75% to 90% would roughly double full-plan completion, from one-third to two-thirds.

What we learned. All six models we studied fall on a single curve, where full-plan completion is a power-exponential function of step completion. Today’s leading world models sit where that curve turns steep. As individual steps become more reliable, we expect rapid gains in full-plan simulation that resemble capability emergence, bringing simulative reasoning within reach.

Technical Details

Building a world model for simulative reasoning requires answering several key technical questions: where language ends and the latent begins, what kind of data to learn dynamics from, what training on data alone misses, and how to train efficiently at scale. In this section, we examine each decision and its measured consequences.

Where does language end and the latent begin?

The text-based state defines the division of work between PAN 2’s two backbones. The LLM reasons about the conceptual content of the next state, and the LDM fills in the perceptual details left open by the text-based state. In PAN 2, we make subjects, the timing of events and actions, and camera motion explicit, while leaving geometry, appearance, and the fine details of motion to the LDM. This choice raised two questions.

Is the LLM doing more than tidying the prompt? Many video generators already put an LLM in front of a diffusion model to rewrite prompts. To see whether PAN 2’s conceptual backbone does more, we kept the LDM fixed and varied the level of conceptual reasoning by PAN 2’s LLM: none (the action instruction goes directly to the LDM), format-only (the LLM rewrites the input into the structured format without reasoning), or full (the LLM reasons about how the world will evolve and writes a complete text-based state).

Without conceptual reasoning, the model’s single-action score drops by nearly half and full-plan completion falls to zero, showing the LDM alone is insufficient for reasoning about the timing, events, and camera motion required for simulating action sequences. Format-only reasoning recovers most of the single-action gap but only part of the multi-step gap, confirming that the depth of conceptual reasoning, not just input formatting, is what drives plan simulation.

What we learned. Conceptual reasoning is integral to PAN 2’s simulation, helping actions, events, and camera movements unfold in the right order and at the right time. We expect advances in their LLM components to further improve the simulation capabilities of world models.

How should the states be written? We train the LDM on the same structured state descriptions it receives from the LLM at inference, aligning its training and test-time inputs. Each training clip is therefore paired with a structured JSON caption comprising two components. Global scene attributes capture the initial scene and subject identities, along with persistent context such as lighting, viewing perspective, and visual style. Time-grounded events describe subject actions, observed changes in object and scene states, and camera movements over explicit time intervals. This structure makes subjects, action timing, and camera motion explicit in the states the LDM learns to simulate.

We generate these annotations in three stages, which the example below steps through: a temporal scaffold, in which a vision–language model (VLM) marks when events and actions happen; camera evidence, in which estimated geometry gives the camera motion (VGGT-Ω); and caption synthesis, in which a VLM combines the two and adds detailed descriptions to produce the final structured caption.

▶ Play↻ Replay

Speed

1×

←→

First pass · Events and action intervals0.0 / 12 s

First passTemporally grounded

Event 10–3 sStack blocks

0.3–0.9 s0.9–2.4 s2.4–2.8 s

Event 23–8 sCat by the window

3.7–5.7 s5.7–6.5 s6.5–7.8 s7.8–8 s

Event 38–12 sTower unchanged

8–8.5 s8.5–9.3 s9.5–12 s

E1E2E3

Time

Events

Event 1Event 2Event 3

Camera

staticpan_rightstaticpan_leftstatic

Actions

S1S1S1S4S4S4S4S4S4S1

0 s3 s5 s8 s10 s12 s

View full caption ↗

ComponentsJSON↓ JSON

Subjects

S1Child's handsSmall light-skinned hands and forearms, with a pale blue short sleeve.S2Red wooden blockRed-painted wooden cube with slightly worn edges.S3Blue wooden blockDark blue-painted wooden cube, similar in size to the red cube, with worn corners.S4Ginger catSmall short-haired ginger tabby cat with darker stripes, a pale muzzle and chest, triangular ears and a long tail.

Initial frame

A worn wooden tabletop fills the lower half of the view. A red cube stands to the left of a blue cube near the middle of the table, with a gap between them. Two small hands rest at the near edge. Low wooden shelves with toys, baskets and books stand in the left background. A large white-framed window occupies the upper right, with patterned curtains beside it and a toy-filled basket below.

Clip context

Style

Photorealistic continuous shot

Lighting

Window daylight, warm tones, soft shadows

View

Low viewpoint just behind the child at table height. Hands and forearms occupy the foreground; a pale sleeve, shoulder and the side of the head enter the left foreground during stacking.

Event 1 · 0–3 sstatic

event_description

The hands reach around the red cube, lift it clear of the tabletop and position it over the blue cube. The red cube is lowered onto the blue cube and released. The hands withdraw, leaving an upright two-block tower, red above blue, near the middle of the table.

contained_actions

0.3–0.9 s

S1 reaches for and grips S2.

0.9–2.4 s

S1 lifts S2 and places it on top of S3.

2.4–2.8 s

S1 releases S2 and withdraws from the tower.

Event 2 · 3–8 span_right + static

3–5 span_right

5–8 sstatic

event_description

The table and tower leave the view, revealing carpet, the edge of a patterned play mat, a toy basket below the window and the wall to its right. A ginger cat is seated beside the white baseboard near an electrical outlet, visible from about 3.7 seconds. After sitting upright, it lowers its head toward its forepaws and the carpet, shifts its front paws and crouches close to the wall. Near the end of the interval its head starts to rise again.

contained_actions

3.7–5.7 s

S4 sits upright beside the baseboard.

5.7–6.5 s

S4 lowers its head toward its forepaws and shifts its front paws.

6.5–7.8 s

S4 crouches with its head close to the carpet beside the wall.

7.8–8 s

S4 starts to raise its head.

Event 3 · 8–12 span_left + static

8–9.75 span_left

9.75–12 sstatic

event_description

The cat raises its head and settles upright facing toward the table. It remains briefly visible by the wall as the table and two-block tower come back into view, then passes out of the right edge around 9.4 seconds. The tower remains upright with red on blue; the hands rest along the near table edge. The final view holds on the unchanged stack, with the window and toy shelves in the background.

contained_actions

8–8.5 s

S4 raises its head and settles upright facing toward the table.

8.5–9.3 s

S4 remains seated beside the baseboard.

9.5–12 s

S1 rests at the near table edge below the unchanged stack of S2 on S3.

EventsCamera motionVLM caption

Click any component in the timeline to see its details.

We compared this format with two plain-text alternatives, one emphasizing diverse content and one following a fixed structure. Since a caption describes the conceptual state that the perceptual reasoning backbone expands on, we evaluated each format by how well an LDM trained on it predicts held-out videos, using validation loss. The JSON captions contained the most detail and yielded the lowest loss.

What we learned. By capturing more of the world’s complexity in text-based states, we leave less for the LDM to infer when simulating perceptual details. This improves prediction without increasing the LDM’s capacity.

Learning dynamics from the real world

A world model learns physical dynamics from the observations it is trained on, so we select data for real-world relevance rather than visual appeal. We filter video clips for motion and quality, deduplicate them, and balance them across a taxonomy of topics (expand Data taxonomy and balancing below for details).

We tested whether excluding synthetic content improves simulation quality. After the earliest phase of training, we exclude gameplay, animation, computer-generated renders, and simulation videos, so the model learns dynamics from the real world rather than from stylized approximations of it. We trained two 9B models for 40,000 steps, one with synthetic content and one without. In blinded comparisons based on action adherence, physics, and world consistency, the model trained without synthetic content won 58% of votes.

What we learned. What we leave out of training data matters as much as what we put in. To learn faithful physical dynamics, a world model benefits from real-world observations rather than synthetic approximations.

Reward-based learning

After flow-matching training, the LDM still makes mistakes that its training objective doesn’t penalize directly (e.g., an object that vanishes mid-clip, or a hand that passes through the cup it’s grabbing). We thus refine it through reward-based learning on its own generated videos. Following RewardDance’s generative scoring principle, we use the frozen VisionReward model to answer 13 questions covering object structure and persistence, motion, composition, and perceptual detail, and use its answers to compute the reward. Over 600 steps, the total reward and each of its parts rise steadily.

What does the reward actually change? Comparing the model before and after reward-based learning shows what a reward adds, and what it can cost. Causal coherence rose by more than eight points and full-plan completion by nearly four, even though the reward never scores the order of events. Viewpoint invariance and physical consistency, on which the reward offers little signal, went down.

Because a learned reward covers only part of what faithful simulation requires, we build our final model as a weighted average of the checkpoints before and after reward-based learning to retain its gains while mitigating reward hacking (model soups, rewarded soups).

What we learned. Reward-based learning complements flow matching by improving capabilities important for simulative reasoning, even when they are not directly rewarded. However, designing rewards that capture all aspects of faithful simulation and preventing reward hacking remain challenges, as they do for LLMs.

Scaling Flow Matching with Per-Head Muon

Training a 29B diffusion transformer from scratch requires an efficient optimizer. PAN 2 uses a hybrid optimizer built around per-head Muon.

Per-head Muon reshapes a momentum update so that a few high-energy directions do not dominate it. For attention, we split the Q/K/V update into head-sized matrices, run five Newton–Schulz steps on each head independently, and then reassemble the results. This lets each head balance its own update spectrum instead of being normalized as part of one large fused matrix. It does not force different heads to be orthogonal; they still interact through attention and the training objective.

Does the advantage hold as models grow? We compared the two optimizers on models ranging from 185M to 1.52B parameters, tuning the batch size and learning rate at each model size. Per-head Muon achieved lower validation loss at all four sizes. The gap narrows as models grow but persists across the range we tested.

What we learned. Applying Muon selectively, with per-head updates where appropriate, improves training efficiency across all model sizes we tested, reaching comparable validation loss with substantially less compute.

Agent Learning in World Simulation

The point of a world model is to give agents a place to practice, where they can try actions, see their consequences, and adjust without the cost or risk of acting in the real world. Decisions can then draw on experience accumulated through interaction, with simulated consequences challenging the agent’s assumptions and guiding its next actions. This requires three capabilities from the world model. It must carry the state of the world from one action to the next. It must preserve information the agent cannot directly observe, and reveal them only through consequences, just as the real world does. And it must extend and maintain the world as the agent explores it. PAN 2 already does the first two.

Carrying the state across turns

PAN 2 can carry a simulated world forward one action at a time. In this example, a person folds a paper plane in simulation. Starting from an image of a sheet of paper on a table, we give PAN 2 one folding step at a time, and the last frame of each clip becomes the starting image for the next. Fold by fold, the sheet becomes a plane.

In this case, the last frame carries everything the next step needs, as nothing is hidden. The real world, however, is full of things an image doesn’t show.

Holding what the agent cannot see

PAN 2 can also hold facts that the agent doesn’t know, and let the agent discover them through acting. A ball sits in a tube closed by two caps: a blue one that comes off, and a red one that is glued shut. The glue is invisible in the starting image; it is stated only in the text given to PAN 2, never to the agent. An LLM agent is asked to retrieve the ball from the tube. In each attempt, it proposes a short plan, PAN 2 simulates it, and the agent judges from the 5-second clip whether it has succeeded; if not, it tries again with all its previous plans, clips and judgements in view, in a setup similar to Reflexion.

LLM agent ⇄ PAN 2· plan, 5-second simulation, review; every attempt starts from this image

Known only to the world model: the blue cap comes off; the red cap is glued shut.

1. 1Pull red capBall still inside
2. 2Twist red capReviewed as “cap off” (incorrect)
3. 3Invert; shakeBall still inside
4. 4Twist red; pinchBall still inside
5. 5Pull red; shakeBall still inside
6. 6Remove blue capBall exits

The agent first pulls the red cap, then twists it. Misreading the second simulation, it concludes that the cap has come off, and spends its next three attempts trying to force the ball out of that end, whether it be inverting and shaking the tube, pinching and twisting the red cap, or pulling it again. On its sixth attempt, it turns to the blue cap, and the ball comes out. The agent learned a property of the world it could not see, through simulated interaction alone. In this case, the text-based state carries what the image cannot.

Next steps for agent learning

Two steps remain before agents can train in world simulation at scale. The first is a world that grows and stays consistent as the agent explores it, which is the subject of an upcoming post. This requires coordinated conceptual and perceptual memory built into the model, along with increasingly faithful simulations of action consequences. The second is training agents on experience gathered across simulated episodes, so that what they learn carries across tasks and ultimately improves decisions in the physical world.

Try PAN 2

Predicting the next word gave us models that reason with language. Likewise, we believe predicting the next world is how models will learn to reason about the physical world. PAN 2 is our step in that direction: a world model that reasons about what happens, renders how it unfolds, and gives agents a simulator to practice in before they act.

PAN 2 is open, with weights, training code and inference code. Try it on your own scenes, build on it, and tell us what you simulate, and where it breaks.

Open weight→

Download the model weights from Hugging Face.

Training code→

Inspect, finetune, or train your own model from scratch.

Inference code

Run PAN 2 with vLLM, HF Diffusers or FastVideo.

@misc{pan2026pan2,
  title        = {{PAN 2}: A Foundation World Model for Physical Intelligence},
  author       = {{PAN Team}},
  howpublished = {Technical blog, Institute of Foundation Models, MBZUAI},
  year         = {2026},
  month        = nov
}

References

1.Eric Xing, Mingkai Deng, and Jinyu Hou. Critique of World Model. 2025. Paper.

2.Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. 2024. Paper.

3.Lipman et al. Flow Matching for Generative Modeling. ICLR, 2023. Paper.

4.Jianyuan Wang et al. VGGT-Ω. CVPR, 2026. Project and paper.

5.Jiazheng Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation (ReFL). NeurIPS, 2023. Paper.

6.Jie Wu et al. RewardDance: Reward Scaling in Visual Generation. 2025. Paper.

7.Jiazheng Xu et al. VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation. Paper · Code.

8.Keller Jordan. Muon: An Optimizer for Hidden Layers in Neural Networks. 2024. Article.

9.Jingyuan Liu et al. Muon Is Scalable for LLM Training. 2025. Paper.

10.Saman Motamed et al. Do Generative Video Models Understand Physical Principles? 2025. Paper · Benchmark and Verified leaderboard.

11.NVIDIA et al. Cosmos 3: Omnimodal World Models for Physical AI. 2026. Paper · Official model reference.

12.Lightricks. LTX-2.5. 2026. Official model card · Code.

13.MiniMax. MiniMax-H3. 2026. Official repository · Model card.

14.Black Forest Labs. FLUX 3. 2026. Official announcement.

15.ByteDance Seed. Seedance 2.5. 2026. Official project page.

16.Yufan Deng et al. Rethinking Video Generation Model for the Embodied World (RBench). 2026. Paper · Benchmark and code.

17.Juyi Lin et al. PhyGround: Benchmarking Physical Reasoning in Generative World Models. 2026. Paper · Project.

18.Mitchell Wortsman et al. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time. ICML, 2022. Paper.

19.Alexandre Ramé et al. Rewarded Soups: Towards Pareto-Optimal Alignment by Interpolating Weights Fine-Tuned on Diverse Rewards. NeurIPS, 2023. Paper.

20.Noah Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023. Paper.