01 · Get started

Submission Overview

Each submission account represents one team and one fixed model. Email verification and organizer approval unlock submissions to WorldOracle-Core, WorldOracle-Policy, and WorldOracle-Long. Both open-weight and closed-source models may participate; the leaderboard reports certification and public-resource status separately.

WorldOracle evaluates ACWMs on human-operated trajectories and policy rollouts collected on real robots. The same framework covers AgiBot G2, YAM, and Human Hand. Select a Track and platform, then submit the required prediction videos; the Track selection determines the evaluation protocol.

02 · Eligibility & registration

One email, one registered model.

One registration email represents one model. A team can add models through Same Team, New Model. The original team credentials authorize each addition, while every model receives its own model name, organization, registration email, password, and quota under the shared team attribution. Organizer approval activates submission access.

Public resources are optional at registration. Organizer verification of any valid model resource assigns Certified status. Open Source status requires verified public code and verified public weights for the exact registered model or method. Resource links may use any public HTTP or HTTPS host.

Each approved account receives two evaluations per Track and six evaluations total in a rolling seven-day window. The first formal submission starts the shared window for all Tracks.

03 · Evaluation tracks

Choose a Track

Each submission enters one Track and one platform. Use separate submissions to enter multiple Tracks. WorldOracle-Core and WorldOracle-Long support AgiBot G2, YAM, and Human Hand; WorldOracle-Policy supports AgiBot G2 policy rollouts.

04 · Package specification

Prepare Your Submission

The portal uses the organizer-approved model bound to the account. Select the Track and platform, then add optional submission notes. Each formal queue entry receives the next submission number for that account and Track.

Track limits and required coverage

• WorldOracle-Core: one ZIP below 3 GiB. Submit the first 30 seconds of every required episode using recorded actions.
• WorldOracle-Policy: one ZIP below 3 GiB. Submit the complete policy-distribution rollout for AgiBot G2 and use action_source: policy in the manifest.
• WorldOracle-Long: one ZIP below 12 GiB. Submit every complete episode using recorded actions and preserve its full duration.

The size limit applies to both the uploaded ZIP and its declared expanded contents. Packages may contain up to 20,000 entries. Admission rejects ZIP symlinks, encrypted entries, duplicate or unsafe paths, and packages where manifest.json is not at the ZIP root.

Identify every video by its official task / episode_id / view values in manifest.json. Frame indices start at zero and remain contiguous; synchronized views share the same frame count, frame rate, and duration. Submit every required trajectory as provided. Task-specific success filtering and silent trajectory replacement are prohibited.

Use schema preview-2. Set benchmark_track to main, policy, or long_horizon to match the form. Set evaluation.horizon to 30s, policy, or full; use action_source: policy for WorldOracle-Policy and action_source: recorded for the other Tracks; and set view_mode to single or multi. The schema accepts only its documented top-level fields.

Single-view input directory format

submission.zip
├── manifest.json              # Required; metadata and prediction index
├── README.md                  # Required; reproduction and alignment details
├── predictions/               # One top-head MP4 per required episode
│   └── pickfruit/ep000000/top_head.mp4
├── config/                    # Recommended; inference settings and seeds
└── logs/                      # Recommended; generation and evaluation logs

Set evaluation.view_mode to single. Required view: top_head.mp4. Each episode contains exactly one prediction.

Three-view input directory format

submission.zip
├── manifest.json              # Required; metadata and prediction index
├── README.md                  # Required; reproduction and alignment details
├── predictions/               # Three synchronized MP4 files per episode
│   └── pickfruit/ep000000/
│       ├── top_head.mp4
│       ├── hand_left.mp4
│       └── hand_right.mp4
├── config/                    # Recommended; inference settings and seeds
└── logs/                      # Recommended; generation and evaluation logs

Set evaluation.view_mode to multi. Required views: top_head · hand_left · hand_right. All three predictions use identical frame counts, frame rates, and durations.

Each prediction entry must include task, episode_id, view, fps, path. The standard output is H.264 MP4, yuv420p, no audio, source resolution preserved, and the platform's native benchmark frame rate. In README.md, explain the action-to-frame mapping, whether the initial frame is included, the original resolution, and any unavoidable resampling.

Required compression profile

Encode each original JPEG or PNG sequence directly to its final MP4. Apply this profile: libx264, the slow preset, two-pass average bitrate 2830k, yuv420p, a 60-frame GOP, scene-cut disabled, and no audio. This direct path avoids an extra transcoding generation.

The five AgiBot G2 baseline packages—IRASim, DreamDojo, Ctrl-World, Cosmos 3, and GE-Sim 2.0—use the same 2830k target bitrate in both passes. The first ffmpeg command analyzes the sequence; the second produces the final video.

The commands below use 30 FPS for AgiBot G2 and YAM. For Human Hand, change both -framerate values to 15. The example input is one frame_00000.jpg sequence. For six-digit PNG input, use frame_%06d.png.

ffmpeg -nostdin -hide_banner -loglevel error -y \
  -framerate 30 -start_number 0 -i 'frame_%05d.jpg' \
  -map 0:v:0 -an -c:v libx264 -preset slow -b:v 2830k \
  -pix_fmt yuv420p -vsync 0 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pass 1 -passlogfile 'top_head-pass' -f null /dev/null

ffmpeg -nostdin -hide_banner -loglevel error -y \
  -framerate 30 -start_number 0 -i 'frame_%05d.jpg' \
  -map 0:v:0 -an -c:v libx264 -preset slow -b:v 2830k \
  -pix_fmt yuv420p -vsync 0 -g 60 -keyint_min 60 -sc_threshold 0 \
  -pass 2 -passlogfile 'top_head-pass' -movflags +faststart \
  top_head.mp4

2830k sets the target file size; preset slow improves compression efficiency at that rate. Average bitrate keeps package size more predictable than CRF. Estimate the total as bitrate × the sum of all video durations ÷ 8, plus MP4 and ZIP overhead. Apply one bitrate to every view and episode. If the ZIP exceeds the Track limit, lower that bitrate uniformly and record the value in README.md.

Package and verify

# Run from the directory containing manifest.json, README.md and predictions/
zip -0 -r submission.zip manifest.json README.md predictions/
unzip -t submission.zip
shasum -a 256 submission.zip

-0 stores the already-compressed MP4 files without wasting time recompressing them. Before uploading, confirm the ZIP is below the selected Track limit and that manifest.json is at the ZIP root. Every Track package must include the canonical comparison episode. A single-view submission publishes its top_head clip; a three-view submission publishes one synchronized composite with top_head above, hand_left at bottom left, and hand_right at bottom right. All other package files remain private.

Keep the package limited to evaluation artifacts. Exclude passwords, API keys, private tokens, personal data, raw GT frames, caches, checkpoints, and unrelated experiment outputs.

05 · Submission wizard

Validate and Submit

Run the public structural-validation template locally before uploading:

python validate_package.py submission.zip --track core --platform agibot-g2 --view-mode single

Set --track to core, policy, or long; set --platform to agibot-g2, yam, or human-hand; and set --view-mode to single or multi. These required arguments must match both manifest.json and the portal form. Human Hand supports Single-View only.

The public template checks ZIP safety, manifest fields, official task names, file references, view completeness, and the actual H.264 codec, frame rate, duration, dimensions, and first decoded frame. It requires ffprobe and ffmpeg. Authoritative test-set coverage, full-frame alignment, evaluation criteria, metric implementations, service paths, and credentials remain service-side checks. After upload, the evaluation service repeats the public checks before evaluation.

In the submission portal, follow four steps:

1. Model and Track: check your assigned model, choose a Track and platform, and optionally add submission notes.
2. Upload Materials: choose a ZIP. The browser uploads resumable parts and the background worker checks the assembled package and records its SHA-256 checksum.
3. Declarations: confirm authenticity, permissions, and the absence of secrets.
4. Confirm and Submit: check the summary and enter the automatic evaluation queue.

Saved steps remain in your account. After an interruption, reopen the same draft and reselect the identical file to resume. Replacing the package resets its declarations to prevent mismatched submissions. After final submission, the record is read-only unless the organizers request changes.

Track admission checks inspect video headers and the first decoded frame, enforce the platform frame rate (30 FPS for AgiBot G2 and YAM; 15 FPS for Human Hand), and validate view counts, synchronization, and WorldOracle-Core's 30-second limit. Full evaluation and scoring begin after admission; admission alone does not establish result validity.

06 · What happens next

Evaluation and Publication

My Submissions reports five public stages: Draft, Submitted, Evaluating, Review, and Accepted. Evaluating shows overall progress while keeping individual metric execution private.

1. Draft
2. Submitted
3. Evaluating
4. Review
5. Accepted

Leaderboard replacement compares both final aggregate scores for the same model, Track, and platform. A new accepted result replaces the complete public row when either its Single-View Score or Multi-View Score is strictly higher. A missing score is treated as 0; component metrics are not compared for replacement, and scores from different submissions are not merged.

• Single-View after Single-View: replaces the row only when the new Single-View Score is higher.
• Single-View after Multi-View: replaces the row only when the new Single-View Score is higher; the replacing row has no Multi-View Score.
• Multi-View after Single-View: replaces the row because the published missing Multi-View Score is treated as 0.
• Multi-View after Multi-View: replaces the row when either the new Single-View Score or Multi-View Score is higher.

07 · Fair and reproducible

Rules and FAQ