Evaluation
Integrity & Review
These rules keep results comparable across research methods and system designs.
Comparable evaluation
Use the registered model and declared evaluation setup consistently across submitted tasks. Record material changes to checkpoints, inference configuration, action inputs, or model routing in the submission notes.
Fair participation
Generate every submitted output with the registered system under the declared setup. Hidden evaluation material, another participant's private submission, fabricated outputs, and task-specific manual selection violate this requirement.
Organizer review
Organizers may inspect packages, logs, public resources, and reproducibility details to resolve incomplete information or integrity questions. Results remain in Review until those questions are resolved. Confirmed link failures, model-material mismatches, and evaluation errors may lead to corrected labels or results.
Blacklisting
Check bypass, non-public benchmark-data scraping, service manipulation, and cheating may place the related account, model, or team on the blacklist. Organizers may remove associated submissions and block further access.
Clarifications and corrections
Participants may provide explanations and supporting evidence before review is finalized. Confirmed corrections preserve the underlying submission history and record the change.
Review protects fair participation and leaderboard integrity.