Top View
01 · RIGID · AGIBOT G2
Pick Fruit
Rigid-Body Manipulation
WorldOracle
Benchmarking Action-Conditioned World Models against Real-World Executions
A real-world benchmark for testing whether predicted futures remain coherent with the actions that produce them.
Leaderboard
AgiBot G2 · 30s · 8 Models
| Rank | Model | Contributor | Single-View Score | Multi-View Score | PSC | KIAF | ISA | MVAC |
|---|---|---|---|---|---|---|---|---|
| Loading published results… | ||||||||
No published model results for this platform and duration yet.
AgiBot G2 · 30s · All models shown equally.
Overview
WorldOracle evaluates action-conditioned world models against paired real-world executions. Given the same initial observation and low-level action sequence, it asks whether a predicted rollout reproduces the real physical consequences and task-state transitions. The benchmark measures physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence across different interaction dynamics and embodiments. Its evaluation suite includes long, multi-segment episodes with both successful and failed executions, revealing distinct model strengths and failure modes across varied action outcomes and behavior distributions.
Framework
WorldOracle evaluates ACWMs against paired real-world executions from two data distributions: human-operated trajectories and policy rollouts collected on a real robot. For both distributions, each recorded action trajectory is replayed by the ACWM and compared with its paired real execution.
Data Samples
The examples shown are all drawn from the WorldOracle validation set.
Real-World Ground Truth
Four Interactions with Distinct Dynamics.
Every reference is a recorded physical rollout—not a simulator-rendered target. Paired executions preserve both successful and failed outcomes across rigid, deformable, fluid, and articulated-object interactions.
Top View
02 · DEFORMABLE · AGIBOT G2
Hang Towel
Deformable-Object Interaction
Top View
03 · FLUID · AGIBOT G2
Pour Water
Fluid Interaction
Top View
04 · ARTICULATED · AGIBOT G2
Pull Drawer
Articulated-Object Interaction
Embodiments
Three Embodiments. One Evaluation Framework.
WorldOracle evaluates heterogeneous agents and action spaces using paired real-world execution references. AgiBot G2 and YAM support synchronized Top, Left, and Right views for three-view evaluation.
Top View
Left View
Right View
01 · DUAL-ARM ROBOT
AgiBot G2
Top View · Left View · Right View
Top View
Left View
Right View
02 · BIMANUAL PLATFORM
YAM
Top View · Left View · Right View
Egocentric View
03 · EGOCENTRIC MANIPULATION · 15 FPS
Human Hand
Egocentric View
Success and Failure
Real Actions. Different Outcomes.
Real-world data includes both successful actions and failed attempts. Explore recorded GT excerpts from four tasks—capturing not only what works, but also missed grasps, dropped objects, and spills.
SuccessTop View
Fruit Placed in the Bowl
The yellow lemon is placed in the bowl and remains inside.
FailureTop View
Placement Miss
The placement attempt ends with both fruits still on the table.
Showing Pick Fruit success and failure excerpts.
Metrics
Evaluation Directions and Scoring Logic.
WorldOracle measures execution coherence along four complementary dimensions. The first three use the primary top-head view; multi-view action coherence uses all three synchronized views.
Physical State Correctness
At each interaction segment’s outcome keyframe, a VLM judges physical-state criteria annotated from the paired real-world keyframe. Criterion scores are averaged within segments and then across the episode.
Criteria → Segment Score → Episode Average
Interaction Spatial Alignment
At sampled frames, DINOv3 features inside annotated object and arm or hand masks are matched between real and predicted observations. Spatial displacement and correspondence confidence reward action-relevant entities near their real positions.
Masked Features → Spatial Matches → Temporal Average
Key Interaction Area Fidelity
Within masked regions of manipulated objects and robot arms or hands, PSNR, LPIPS, and V-JEPA measure pixel, perceptual, and semantic fidelity. Their normalized scores are combined geometrically, so one severe local error lowers the result.
Interaction Crops → PSNR / LPIPS / V-JEPA → Geometric Mean
Multi-View Action Coherence
Robot-arm alignment is measured independently against the paired real execution in synchronized top, left-hand, and right-hand views, then aggregated. A frame scores highly only when motion is aligned in every view.
Per-View Arm Alignment → Cross-View Average → Episode Average
Benchmark Coverage
One Benchmark. Multiple Worlds.
Embodiments
AgiBot G2
YAM
Human Hand
Data Distributions
Human-Operated
Policy-Rollout
Models Studied
IRASim · Ctrl-World
DreamDojo · Cosmos 3
GE-Sim 2.0