Evaluation
Evaluation Rules
These rules define formal evaluation, quota use, and leaderboard replacement.
Tracks and quota
Each formal submission enters one Track and one supported platform. An approved account receives two evaluations per Track and six evaluations total in a rolling seven-day window. The first formal submission starts the shared window.
Formal evaluation
Upload validation checks package structure and safety. Formal queue entry consumes quota and begins benchmark evaluation. Public progress follows Draft, Submitted, Evaluating, Review, and Accepted.
VLM evaluator version
The paper reports GPT-5.5 VLM-evaluator scores. The live Leaderboard uses GPT-5.6 Luna for both re-evaluated baselines and new submissions, keeping live entries comparable. Evaluator-version differences may produce different paper and live Leaderboard scores.
Leaderboard replacement
For the same model, Track, and platform, each accepted submission is compared with the complete published row. The new result replaces that row when either its final Single-View Score or final Multi-View Score is strictly higher. An unavailable score is treated as 0. Scores and component metrics from different submissions are never merged.
- Single-View after Single-View: replace only when the new Single-View Score is higher.
- Single-View after Multi-View: replace only when the new Single-View Score is higher; the replacement has no Multi-View Score.
- Multi-View after Single-View: replace because the current unavailable Multi-View Score is treated as 0.
- Multi-View after Multi-View: replace when either aggregate score is higher.
The evaluation service and the Submission Guide package specification govern admission and scoring.