- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RoboWorM is a benchmark for evaluating structural consistency and inference efficiency in robotic video world models. It comprises 1,605 episode–camera-track conditions corresponding to 1,305 distinct trajectories from five source datasets, covering single-arm, bimanual, and humanoid settings. The paper introduces KIVA, which measures rigidity and jerk violations in trajectories estimated by a learned video-to-keypoint reader, and RIGIS, which combines predicted multi-view geometric consistency with camera-parameter stability. It evaluates three world-model families and nine acceleration methods across approximately 29,000 generated rollouts. Human studies report 73.5% segment-level classification agreement for calibrated KIVA and a mean per-morphology Spearman correlation of 0.749 for RIGIS. The acceleration experiments investigate whether conventional visual-quality retention can coexist with structural degradation.
Critical strengths
- A useful evaluation scope with substantial experimental coverage. Combining multiple robot morphologies, synchronized views, and inference acceleration addresses an important practical evaluation problem. The approximately 29,000 rollouts provide substantial material for studying structural failures that conventional video metrics may overlook. This breadth is a meaningful contribution independently of the stronger deployment claims.
- Promising empirical alignment with human judgments, particularly for RIGIS. RIGIS achieves per-morphology Spearman correlations of 0.706, 0.724, and 0.816, compared with 0.291, 0.342, and 0.344 for the reported Gemma + CoT baseline (Table D5). These results support further investigation of geometry-based evaluation. KIVA also shows useful aggregate agreement, although its advantage is less uniform across morphologies and depends partly on label-based calibration.
Critical weaknesses
- The validity of the core physical-consistency measurements is insufficiently established. KIVA computes violations on predicted keypoints, but the paper does not report held-out keypoint accuracy or controlled experiments separating actual structural abnormalities from reader failures under occlusion or appearance changes. Its explanation invokes increased cross-seed prediction variance, while its score uses the mean prediction; this connection is not experimentally validated. RIGIS assumes stable camera geometry, yet the benchmark includes wrist cameras, without specifying which views enter the reported scores or how legitimate camera motion is handled. These issues directly affect whether the benchmark measures the advertised properties. Human agreement provides supporting evidence, but does not resolve these ambiguities, particularly given initial annotator agreement of 53.6% for embodied physics and 56.0% for multi-view consistency.
- The reported results cannot be tied to a sufficiently consistent and reproducible protocol. Section 4.1 uses a fixed KIVA boundary of 1, Appendix B.1 describes human-fitted thresholds, and Appendix E.2 explicitly advises against human-label fitting. RIGIS uses exponential error mappings in the main text but different mappings in Eq. B.8. Training/evaluation episode separation is not documented for the fine-tuned checkpoints. For acceleration, Appendix C.2 states that most timings use small tuning samples and that the uniform timing pass is complete for only one configuration; the rule behind “winning all 12 configurations” is also unspecified. Together, these gaps prevent reliable interpretation of both the benchmark rankings and the headline efficiency conclusions.
Questions for authors
- Which exact implementation produced each reported result? Please reconcile KIVA’s fixed versus fitted thresholds, morphology-specific use of jerk, and 16-frame versus three-second segmentation, as well as the two RIGIS score mappings. Were VLM agreement scores computed on the same held-out videos as KIVA in each calibration draw?
- What evidence separates KIVA reader error from actual video abnormalities? Please provide any existing held-out keypoint accuracy, real-video false-alarm analysis, or controlled perturbation results. Is there evidence connecting cross-seed prediction variance to the violations computed from the mean trajectory?
- Which camera pairs were used for RIGIS and its human validation? Were moving wrist cameras included, and, if so, how was expected motion distinguished from erroneous geometric drift? Do the geometric and pose components independently align with human labels?
- Please specify the episode-level relationship between model training, reader training, reference calibration, and evaluation. For acceleration, which measured configurations and decision rule support the “12/12” claim, and which results remain supported under the uniform timing protocol?
4 comments
Response to Reviewer nw9S — Part 2: RIGIS & data separation
Which camera pairs were used for RIGIS and its human validation?
We thank the reviewer for this question, which helps clarify the camera setup and the scope of RIGIS validation.
Camera pairs. RIGIS and human assessment use exactly the same two fixed-camera views for each embodiment: cam_high and cam_low (bimanual), cam_high_rgb and cam_third_view (humanoid, Markovian tasks), cam_front_rgb and cam_head_rgb (humanoid, non-Markovian tasks), and exterior_1_left and exterior_2_left (single-arm). Wrist cameras and other dynamic-camera views are excluded. This setup is specified for RIGIS in Section 3.4.2 and for human evaluation in Box D.1.2, as quoted below. Camera stability therefore measures deviations from an expected static setup, not legitimate camera motion.
Component-level agreement. Both evaluations target a balanced, joint assessment of geometric consistency and camera stability. RIGIS weights the two components equally, and Box D.1.2 includes both criteria at every rating level without prioritizing either. To assess each component independently, we computed its agreement with the same human labels. Both components align positively with human ratings. Their combination achieves the highest mean agreement, indicating that the two components capture complementary aspects of human judgment.
| Metric | Bimanual Spearman | Bimanual RankAcc | Humanoid Spearman | Humanoid RankAcc | Single-arm Spearman | Single-arm RankAcc | Mean Spearman | Mean RankAcc |
|---|---|---|---|---|---|---|---|---|
| S_geo | 0.482 | 0.734 | 0.337 | 0.661 | 0.671 | 0.829 | 0.497 | 0.741 |
| S_pose | 0.727 | 0.856 | 0.657 | 0.823 | 0.771 | 0.887 | 0.718 | 0.856 |
| RIGIS = 0.5 (S_geo + S_pose) | 0.706 | 0.847 | 0.724 | 0.862 | 0.816 | 0.914 | 0.749 | 0.874 |
Supporting Text from the Manuscript Section 3.4.2, line 304: "Rather than blindly estimating spatial parameters, RIGIS leverages the dataset's known static camera configurations and workspace limits." Box D.1.2, line 1973–1975: "You will evaluate the synchronization and consistency between two views of generated videos depicting a robot performing actions. In addition to spatial consistency, camera stability is also a factor (ideally, both camera views are completely static)."
Please specify the episode-level relationship between model training, reader training, reference calibration, and evaluation.
We thank the reviewer for highlighting these concerns. In short, world-model training, reader training, and reference calibration all use the same real episodes, and evaluation rollouts are conditioned on largely overlapping episodes. We explain each relationship below.
(i) World-model training and evaluation. We acknowledge that most evaluation episodes overlap with the world models’ fine-tuning data, so strict episode-level separation is not guaranteed. Similar to imitation learning[1][2], fine-tuning uses multiple demonstrations of the same task, which may differ in execution, allowing the world models to learn a distribution of plausible task trajectories rather than memorize a single fixed sequence. During training, models learn to generate short video chunks sampled from these demonstrations. Evaluation, however, follows a different protocol: each model receives only the initial observations and model-specific conditioning signals, without access to ground-truth future frames. It then generates a full-length autoregressive rollout, with subsequent chunks conditioned on previously generated frames and, where applicable, refined task prompts. Rollouts extend to durations comparable to the original task demonstrations, allowing us to examine temporal drift and accumulated structural errors even on familiar episodes. Importantly, acceleration methods are compared against their dense baselines using identical checkpoints, conditioning, and evaluation episodes, enabling paired comparisons of acceleration-induced degradation. While memorization effects cannot be fully ruled out, our evaluation assesses the consistency of autoregressive generation rather than reconstruction of a fixed training trajectory.
(ii) KIVA reader. The reader is trained only on real videos with forward-kinematics supervision and never sees generated rollouts (train / test episodes: 271 / 29 bimanual, 234 / 26 humanoid, 108 / 12 single-arm). We use the same real episodes as world-model training by design, so the reader learns the normal kinematics that the world model should reproduce.
Response to Reviewer nw9S — Part 3: Calibration & acceleration
(iii) Reference calibration. θ_d is the 95th percentile of the reader's error on its real training episodes, i.e., the error level on real footage where no violation exists. It is then fixed, and a generated segment is flagged as a violation when s_S > 1 (Sec. 3.4.1). Since the reader's error is lowest on its training data, this θ_d is a strict reference. Human-fitted cuts appear only in App. D.3 for the three-level human comparison. RIGIS uses a pretrained geometry estimator with no training or threshold fitting on our data.
(iv) The "12/12" claim. It covers 12 model, embodiment, and view settings (Sec. 4.4, Fig. 5, Figs. C2 and C3). In each setting, we compare the speedup of every supported acceleration method against its dense baseline and record the family of the fastest method. Caching is fastest in all 12 settings, with an average speedup of 1.7× and an average RIGIS change of -0.004. "12/12" therefore refers to speedup leadership. Quality is evaluated separately with KIVA, RIGIS, PSNR, FVD, Visual Integrity, and Text Alignment, not assumed to be preserved. For example, WorldCache reaches 2.28× on DreamGen humanoid multi-view with RIGIS changing from 0.860 to 0.853.
(v) Timing protocol. For each method, we start from the settings recommended in its original paper and tune nearby values on a small subset, checking quality metrics and visual outputs to avoid severe degradation. The selected configuration is then fixed for all evaluation episodes. The tuning subset is used only for configuration selection. All reported speedups are measured over complete episode rollouts on the full evaluation set against matched dense baselines.
Supporting Text from the Revised manuscript We additionally verified the timing provenance of the speedups reported in Figures 5, C2, and C3 against the full-evaluation paired timing records and clarified their coverage in Appendix C.2. These reported speedups are distinct from the additional uniform timing pass, which remains incomplete and is not used for the main comparisons.
Appendix C.2, line 1659–1663: “Configuration selection. For each acceleration method, we initialize the hyperparameters from the settings recommended in its original paper and tune nearby values on a small subset. We examine both quantitative quality metrics and visual outputs to avoid configurations with severe degradation. Once selected, the hyperparameters are fixed for the full benchmark evaluation. The tuning subset is used only for configuration selection, not for reporting speedups.”
Appendix C.2, line 1686–1689: “Aggregation: After fixing the selected configurations, we measure generation times over complete evaluation rollouts, rather than the tuning subset. Within each setting, dense and accelerated models are evaluated on the same episodes. We compute speedup as the ratio of their mean generation times, using each backbone's native timing unit.”
Appendix C.2, line 1691–1695: “Timing provenance and coverage. Configuration tuning and speedup evaluation are conducted separately. The small tuning subset is used only to select hyperparameters. Final speedups are measured on complete evaluation rollouts with fixed configurations, using matched dense and accelerated runs on the same episodes. We aggregate the recorded generation times as described above, rather than using timing results from the tuning trials.”
Appendix C.2, line 1701–1705: “The uniform timing pass is intended as an additional verification of the reported speedups, rather than their primary measurement source. All reported speedups are derived from full-evaluation paired timing measurements with fixed configurations. At the time of writing, the uniform pass has been completed for DreamDojo humanoid single-view only. We therefore do not interpret the reported results across all 12 settings as independently verified under this additional protocol.”
References [1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, R. Martín-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Conference on Robot Learning (CoRL), PMLR 164:1678–1690, 2021.
We would like to thank Reviewer nw9S once again for the thoughtful review and constructive suggestions. We have carefully addressed each of the raised concerns through additional experiments and clarifications during the rebuttal, including the clarified KIVA scoring protocol, held-out keypoint accuracy, real-video false-alarm rates and controlled perturbation results for KIVA, the separation between RIGIS training data and evaluation episodes, the decision-rule sensitivity analysis, and the timing protocol for the acceleration audit. If you have an opportunity to revisit our responses before the discussion period ends, we would greatly appreciate any further feedback or remaining concerns that we could clarify. Thank Reviewer again for your time and consideration.
Response to Reviewer nw9S — Part 1: Scoring protocol and KIVA reliability
We thank reviewer nw9S for the careful review and for recognizing our benchmark’s experimental coverage and RIGIS’s promising agreement with human judgments. Below, we address the concerns about metric validity, protocol consistency, and reproducibility.
Which exact implementation produced each reported result?
We apologize for the inconsistent and overlapping descriptions, which we have corrected and reorganized for clarity in the revised manuscript. All reported results come from the implementation described below, and none of the corrections changes any reported value.
Fixed vs. fitted thresholds. We have reorganized the descriptions of KIVA reporting and human-score alignment (Sec. 3.4.1, l. 293–295, Sec. 4.1, l. 357–358, App. B.1, l. 1249–1250, App. D.3, l. 2309–2319). All reported KIVA results, including Table 2 and the acceleration results, use the same definition. Each segment's error is normalized by θ_d, the 95th percentile of the reader's error on real videos. A segment is flagged as a violation when s_S > 1, and KIVA is the resulting violation rate over the rollout:
KIVA=N100∑i=1N1![sSi>1]
sSi=maxd∈DeθdvSid
θd=Q0.95(vRd:R∈R)
The human-fitted thresholds, now described only in App. D.3, are used only for the human-score alignment, to map s_S to the raters' Good / Medium / Bad scale. They do not affect KIVA or any reported value.
Jerk. All morphologies use both rigidity and jerk, and the segment score is the larger of the two ratios, s_S = max(s_rig, s_jerk).
Segmentation. The "three-second" window in Appendix D.1.2 (line 1931) is corrected to match the main text. All results use 16-frame windows (1.0 to 1.6 s) for both KIVA and human raters.
RIGIS mapping. The term "exponential decay" in Sec. 3.4.2 (lines 314–315) is corrected to match Eq. B.7. All reported RIGIS results use the mappings in Eq. B.7, linear for RGB errors and quadratic for normalized depth errors.
VLM agreement. KIVA’s three-level cuts are fitted on 9 labeled videos per morphology, so KIVA cannot be scored on those 9 videos. We therefore repeat the evaluation 1000 times. Each time, we pick 9 videos at random for fitting, score KIVA on the remaining videos, and average the agreement over the 1000 repeats (73.5%). VLMs need no fitting, so we originally scored them once on all videos (58.4%). Scoring the VLM on exactly the same remaining videos in each repeat gives 58.5% (App. D.3, lines 2404–2405), so the comparison is unchanged.
What evidence separates KIVA reader error from actual video abnormalities?
Robustness to visual perturbations. We agree that keypoint prediction errors can arise from both physical violations and visual disturbances. KIVA should be sensitive to physical violations while remaining robust to appearance changes. To assess this, we compute KIVA scores on 8,093 generated segments, then apply occlusion, lighting changes, and motion blur to the same segments while preserving their underlying robot geometry and motion. We recompute KIVA scores and compare each perturbed segment with its original score. Small score changes would support robustness to these visual perturbations.
Perturbations change the violation rate by at most 2.6 pp, so KIVA is robust to these appearance changes.
Held-out keypoint accuracy. We additionally evaluate KIVA on held-out real videos, using these recordings as physically valid references to estimate its false-alarm rate. KIVA flags 9.6%, 4.6%, and 13.2% of segments for bimanual, humanoid, and single-arm robots, respectively. The corresponding predicted bone-length errors are 9.0, 7.4, and 2.3 mm. These results quantify the residual false alarms arising from keypoint estimation and the violation thresholds, with the highest rate occurring in the single-arm setting.
Is there evidence connecting cross-seed prediction variance to the violations computed from the mean trajectory? Yes. The reader is a diffusion model, so we sample it 5 times per segment and KIVA checks the average skeleton. When the samples disagree, averaging pulls the joints toward each other and shortens the bones, which the rigidity check flags. Across the 8,093 segments, the disagreement between samples correlates with the KIVA score (Spearman ρ = 0.49 / 0.56 / 0.56, Table D6).
Revised the manuscript: App. D.3, Table D6 (lines 2424–2476): held-out keypoint accuracy, real-video false-alarm rates, perturbation results on real clips, and the correlation between sample disagreement and KIVA.