acceptodds
Under review as a conference paper at ICLR 2027

RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models

Abstract

Video world models (VWMs) are rapidly emerging as scalable neural simulators for robotic manipulation. However, most benchmarks only evaluate them as open-world video generators, using Vision-Language Models (VLMs) that lack strict 3D spatial and kinematic reasoning. To bridge this gap, we introduce RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity. RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, spanning both Markovian and long-horizon non-Markovian tasks under single- and multi-view setups. To explicitly measure physical consistency, we propose two deterministic, VLM-free metrics: KIVA for intra-view kinematic stability, and RIGIS for multi-view geometric consistency. By exploiting intrinsic robotic invariants, KIVA improves hallucination detection alignment with human annotations by 25.8% relative to the strongest frontier VLMs, while RIGIS achieves a 130% higher Spearman correlation with human preferences. Furthermore, since real-world deployment requires low inference latency, we utilize RoboWorM to conduct the first large-scale audit of the quality-efficiency trade-off. Evaluating 9 state-of-the-art acceleration techniques across 3 leading world models (over 29,000 generated rollouts), we demonstrate how these acceleration methods can retain superficial visual quality while silently compromising deployment-critical physical constraints. Using our metrics, we achieve 2.3 maximum inference speed-up with minimal quality loss.

Primary area:
datasets and benchmarks
Keywords:
Video World Models, Robot Learning, Physical Consistency, Inference Acceleration, Kinematic Consistency, Multi-View Geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nw9SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboWorM is a benchmark for evaluating structural consistency and inference efficiency in robotic video world models. It comprises 1,605 episode–camera-track conditions corresponding to 1,305 distinct trajectories from five source datasets, covering single-arm, bimanual, and humanoid settings. The paper introduces KIVA, which measures rigidity and jerk violations in trajectories estimated by a learned video-to-keypoint reader, and RIGIS, which combines predicted multi-view geometric consistency with camera-parameter stability. It evaluates three world-model families and nine acceleration methods across approximately 29,000 generated rollouts. Human studies report 73.5% segment-level classification agreement for calibrated KIVA and a mean per-morphology Spearman correlation of 0.749 for RIGIS. The acceleration experiments investigate whether conventional visual-quality retention can coexist with structural degradation.

Critical strengths
  1. A useful evaluation scope with substantial experimental coverage. Combining multiple robot morphologies, synchronized views, and inference acceleration addresses an important practical evaluation problem. The approximately 29,000 rollouts provide substantial material for studying structural failures that conventional video metrics may overlook. This breadth is a meaningful contribution independently of the stronger deployment claims.
  2. Promising empirical alignment with human judgments, particularly for RIGIS. RIGIS achieves per-morphology Spearman correlations of 0.706, 0.724, and 0.816, compared with 0.291, 0.342, and 0.344 for the reported Gemma + CoT baseline (Table D5). These results support further investigation of geometry-based evaluation. KIVA also shows useful aggregate agreement, although its advantage is less uniform across morphologies and depends partly on label-based calibration.
Critical weaknesses
  1. The validity of the core physical-consistency measurements is insufficiently established. KIVA computes violations on predicted keypoints, but the paper does not report held-out keypoint accuracy or controlled experiments separating actual structural abnormalities from reader failures under occlusion or appearance changes. Its explanation invokes increased cross-seed prediction variance, while its score uses the mean prediction; this connection is not experimentally validated. RIGIS assumes stable camera geometry, yet the benchmark includes wrist cameras, without specifying which views enter the reported scores or how legitimate camera motion is handled. These issues directly affect whether the benchmark measures the advertised properties. Human agreement provides supporting evidence, but does not resolve these ambiguities, particularly given initial annotator agreement of 53.6% for embodied physics and 56.0% for multi-view consistency.
  2. The reported results cannot be tied to a sufficiently consistent and reproducible protocol. Section 4.1 uses a fixed KIVA boundary of 1, Appendix B.1 describes human-fitted thresholds, and Appendix E.2 explicitly advises against human-label fitting. RIGIS uses exponential error mappings in the main text but different mappings in Eq. B.8. Training/evaluation episode separation is not documented for the fine-tuned checkpoints. For acceleration, Appendix C.2 states that most timings use small tuning samples and that the uniform timing pass is complete for only one configuration; the rule behind “winning all 12 configurations” is also unspecified. Together, these gaps prevent reliable interpretation of both the benchmark rankings and the headline efficiency conclusions.
Questions for authors
  1. Which exact implementation produced each reported result? Please reconcile KIVA’s fixed versus fitted thresholds, morphology-specific use of jerk, and 16-frame versus three-second segmentation, as well as the two RIGIS score mappings. Were VLM agreement scores computed on the same held-out videos as KIVA in each calibration draw?
  2. What evidence separates KIVA reader error from actual video abnormalities? Please provide any existing held-out keypoint accuracy, real-video false-alarm analysis, or controlled perturbation results. Is there evidence connecting cross-seed prediction variance to the violations computed from the mean trajectory?
  3. Which camera pairs were used for RIGIS and its human validation? Were moving wrist cameras included, and, if so, how was expected motion distinguished from erroneous geometric drift? Do the geometric and pose components independently align with human labels?
  4. Please specify the episode-level relationship between model training, reader training, reference calibration, and evaluation. For acceleration, which measured configurations and decision rule support the “12/12” claim, and which results remain supported under the uniform timing protocol?
4 comments
AuthorsOpenReview ↗

Response to Reviewer nw9S — Part 1: Scoring protocol and KIVA reliability

We thank reviewer nw9S for the careful review and for recognizing our benchmark’s experimental coverage and RIGIS’s promising agreement with human judgments. Below, we address the concerns about metric validity, protocol consistency, and reproducibility.

Which exact implementation produced each reported result?

We apologize for the inconsistent and overlapping descriptions, which we have corrected and reorganized for clarity in the revised manuscript. All reported results come from the implementation described below, and none of the corrections changes any reported value.

Fixed vs. fitted thresholds. We have reorganized the descriptions of KIVA reporting and human-score alignment (Sec. 3.4.1, l. 293–295, Sec. 4.1, l. 357–358, App. B.1, l. 1249–1250, App. D.3, l. 2309–2319). All reported KIVA results, including Table 2 and the acceleration results, use the same definition. Each segment's error is normalized by θ_d, the 95th percentile of the reader's error on real videos. A segment is flagged as a violation when s_S > 1, and KIVA is the resulting violation rate over the rollout:

KIVA=100N∑i=1N1![sSi>1]\mathrm{KIVA}=\frac{100}{N}\sum_{i=1}^{N}\mathbb{1}\\!\left[s_{S_i}>1\right]

sSi=max⁡d∈DevSidθds_{S_i}=\max_{d\in\mathcal{D}_e}\frac{v^d_{S_i}}{\theta_d}

θd=Q0.95(vRd:R∈R)\theta_d=Q_{0.95}\big(\\{v^d_R : R\in\mathcal{R}\\}\big)

The human-fitted thresholds, now described only in App. D.3, are used only for the human-score alignment, to map s_S to the raters' Good / Medium / Bad scale. They do not affect KIVA or any reported value.

Jerk. All morphologies use both rigidity and jerk, and the segment score is the larger of the two ratios, s_S = max(s_rig, s_jerk).

Segmentation. The "three-second" window in Appendix D.1.2 (line 1931) is corrected to match the main text. All results use 16-frame windows (1.0 to 1.6 s) for both KIVA and human raters.

RIGIS mapping. The term "exponential decay" in Sec. 3.4.2 (lines 314–315) is corrected to match Eq. B.7. All reported RIGIS results use the mappings in Eq. B.7, linear for RGB errors and quadratic for normalized depth errors.

VLM agreement. KIVA’s three-level cuts are fitted on 9 labeled videos per morphology, so KIVA cannot be scored on those 9 videos. We therefore repeat the evaluation 1000 times. Each time, we pick 9 videos at random for fitting, score KIVA on the remaining videos, and average the agreement over the 1000 repeats (73.5%). VLMs need no fitting, so we originally scored them once on all videos (58.4%). Scoring the VLM on exactly the same remaining videos in each repeat gives 58.5% (App. D.3, lines 2404–2405), so the comparison is unchanged.

What evidence separates KIVA reader error from actual video abnormalities?

Robustness to visual perturbations. We agree that keypoint prediction errors can arise from both physical violations and visual disturbances. KIVA should be sensitive to physical violations while remaining robust to appearance changes. To assess this, we compute KIVA scores on 8,093 generated segments, then apply occlusion, lighting changes, and motion blur to the same segments while preserving their underlying robot geometry and motion. We recompute KIVA scores and compare each perturbed segment with its original score. Small score changes would support robustness to these visual perturbations.

Setting (violation rate %)BimanualHumanoidSingle-arm
Actual physical violations (no perturbation)83.346.485.9
+ Occlusion80.849.084.6
+ Lighting changes82.547.785.3
+ Motion blurs80.843.886.5

Perturbations change the violation rate by at most 2.6 pp, so KIVA is robust to these appearance changes.

Held-out keypoint accuracy. We additionally evaluate KIVA on held-out real videos, using these recordings as physically valid references to estimate its false-alarm rate. KIVA flags 9.6%, 4.6%, and 13.2% of segments for bimanual, humanoid, and single-arm robots, respectively. The corresponding predicted bone-length errors are 9.0, 7.4, and 2.3 mm. These results quantify the residual false alarms arising from keypoint estimation and the violation thresholds, with the highest rate occurring in the single-arm setting.

Is there evidence connecting cross-seed prediction variance to the violations computed from the mean trajectory? Yes. The reader is a diffusion model, so we sample it 5 times per segment and KIVA checks the average skeleton. When the samples disagree, averaging pulls the joints toward each other and shortens the bones, which the rigidity check flags. Across the 8,093 segments, the disagreement between samples correlates with the KIVA score (Spearman ρ = 0.49 / 0.56 / 0.56, Table D6).

Revised the manuscript: App. D.3, Table D6 (lines 2424–2476): held-out keypoint accuracy, real-video false-alarm rates, perturbation results on real clips, and the correlation between sample disagreement and KIVA.

AuthorsOpenReview ↗

Response to Reviewer nw9S — Part 2: RIGIS & data separation

Which camera pairs were used for RIGIS and its human validation?

We thank the reviewer for this question, which helps clarify the camera setup and the scope of RIGIS validation.

Camera pairs. RIGIS and human assessment use exactly the same two fixed-camera views for each embodiment: cam_high and cam_low (bimanual), cam_high_rgb and cam_third_view (humanoid, Markovian tasks), cam_front_rgb and cam_head_rgb (humanoid, non-Markovian tasks), and exterior_1_left and exterior_2_left (single-arm). Wrist cameras and other dynamic-camera views are excluded. This setup is specified for RIGIS in Section 3.4.2 and for human evaluation in Box D.1.2, as quoted below. Camera stability therefore measures deviations from an expected static setup, not legitimate camera motion.

Component-level agreement. Both evaluations target a balanced, joint assessment of geometric consistency and camera stability. RIGIS weights the two components equally, and Box D.1.2 includes both criteria at every rating level without prioritizing either. To assess each component independently, we computed its agreement with the same human labels. Both components align positively with human ratings. Their combination achieves the highest mean agreement, indicating that the two components capture complementary aspects of human judgment.

MetricBimanual SpearmanBimanual RankAccHumanoid SpearmanHumanoid RankAccSingle-arm SpearmanSingle-arm RankAccMean SpearmanMean RankAcc
S_geo0.4820.7340.3370.6610.6710.8290.4970.741
S_pose0.7270.8560.6570.8230.7710.8870.7180.856
RIGIS = 0.5 (S_geo + S_pose)0.7060.8470.7240.8620.8160.9140.7490.874

Supporting Text from the Manuscript Section 3.4.2, line 304: "Rather than blindly estimating spatial parameters, RIGIS leverages the dataset's known static camera configurations and workspace limits." Box D.1.2, line 1973–1975: "You will evaluate the synchronization and consistency between two views of generated videos depicting a robot performing actions. In addition to spatial consistency, camera stability is also a factor (ideally, both camera views are completely static)."

Please specify the episode-level relationship between model training, reader training, reference calibration, and evaluation.

We thank the reviewer for highlighting these concerns. In short, world-model training, reader training, and reference calibration all use the same real episodes, and evaluation rollouts are conditioned on largely overlapping episodes. We explain each relationship below.

(i) World-model training and evaluation. We acknowledge that most evaluation episodes overlap with the world models’ fine-tuning data, so strict episode-level separation is not guaranteed. Similar to imitation learning[1][2], fine-tuning uses multiple demonstrations of the same task, which may differ in execution, allowing the world models to learn a distribution of plausible task trajectories rather than memorize a single fixed sequence. During training, models learn to generate short video chunks sampled from these demonstrations. Evaluation, however, follows a different protocol: each model receives only the initial observations and model-specific conditioning signals, without access to ground-truth future frames. It then generates a full-length autoregressive rollout, with subsequent chunks conditioned on previously generated frames and, where applicable, refined task prompts. Rollouts extend to durations comparable to the original task demonstrations, allowing us to examine temporal drift and accumulated structural errors even on familiar episodes. Importantly, acceleration methods are compared against their dense baselines using identical checkpoints, conditioning, and evaluation episodes, enabling paired comparisons of acceleration-induced degradation. While memorization effects cannot be fully ruled out, our evaluation assesses the consistency of autoregressive generation rather than reconstruction of a fixed training trajectory.

(ii) KIVA reader. The reader is trained only on real videos with forward-kinematics supervision and never sees generated rollouts (train / test episodes: 271 / 29 bimanual, 234 / 26 humanoid, 108 / 12 single-arm). We use the same real episodes as world-model training by design, so the reader learns the normal kinematics that the world model should reproduce.

AuthorsOpenReview ↗

Response to Reviewer nw9S — Part 3: Calibration & acceleration

(iii) Reference calibration. θ_d is the 95th percentile of the reader's error on its real training episodes, i.e., the error level on real footage where no violation exists. It is then fixed, and a generated segment is flagged as a violation when s_S > 1 (Sec. 3.4.1). Since the reader's error is lowest on its training data, this θ_d is a strict reference. Human-fitted cuts appear only in App. D.3 for the three-level human comparison. RIGIS uses a pretrained geometry estimator with no training or threshold fitting on our data.

(iv) The "12/12" claim. It covers 12 model, embodiment, and view settings (Sec. 4.4, Fig. 5, Figs. C2 and C3). In each setting, we compare the speedup of every supported acceleration method against its dense baseline and record the family of the fastest method. Caching is fastest in all 12 settings, with an average speedup of 1.7× and an average RIGIS change of -0.004. "12/12" therefore refers to speedup leadership. Quality is evaluated separately with KIVA, RIGIS, PSNR, FVD, Visual Integrity, and Text Alignment, not assumed to be preserved. For example, WorldCache reaches 2.28× on DreamGen humanoid multi-view with RIGIS changing from 0.860 to 0.853.

(v) Timing protocol. For each method, we start from the settings recommended in its original paper and tune nearby values on a small subset, checking quality metrics and visual outputs to avoid severe degradation. The selected configuration is then fixed for all evaluation episodes. The tuning subset is used only for configuration selection. All reported speedups are measured over complete episode rollouts on the full evaluation set against matched dense baselines.

Supporting Text from the Revised manuscript We additionally verified the timing provenance of the speedups reported in Figures 5, C2, and C3 against the full-evaluation paired timing records and clarified their coverage in Appendix C.2. These reported speedups are distinct from the additional uniform timing pass, which remains incomplete and is not used for the main comparisons.

Appendix C.2, line 1659–1663: “Configuration selection. For each acceleration method, we initialize the hyperparameters from the settings recommended in its original paper and tune nearby values on a small subset. We examine both quantitative quality metrics and visual outputs to avoid configurations with severe degradation. Once selected, the hyperparameters are fixed for the full benchmark evaluation. The tuning subset is used only for configuration selection, not for reporting speedups.”

Appendix C.2, line 1686–1689: “Aggregation: After fixing the selected configurations, we measure generation times over complete evaluation rollouts, rather than the tuning subset. Within each setting, dense and accelerated models are evaluated on the same episodes. We compute speedup as the ratio of their mean generation times, using each backbone's native timing unit.”

Appendix C.2, line 1691–1695: “Timing provenance and coverage. Configuration tuning and speedup evaluation are conducted separately. The small tuning subset is used only to select hyperparameters. Final speedups are measured on complete evaluation rollouts with fixed configurations, using matched dense and accelerated runs on the same episodes. We aggregate the recorded generation times as described above, rather than using timing results from the tuning trials.”

Appendix C.2, line 1701–1705: “The uniform timing pass is intended as an additional verification of the reported speedups, rather than their primary measurement source. All reported speedups are derived from full-evaluation paired timing measurements with fixed configurations. At the time of writing, the uniform pass has been completed for DreamDojo humanoid single-view only. We therefore do not interpret the reported results across all 12 settings as independently verified under this additional protocol.”

References [1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, R. Martín-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Conference on Robot Learning (CoRL), PMLR 164:1678–1690, 2021.

We would like to thank Reviewer nw9S once again for the thoughtful review and constructive suggestions. We have carefully addressed each of the raised concerns through additional experiments and clarifications during the rebuttal, including the clarified KIVA scoring protocol, held-out keypoint accuracy, real-video false-alarm rates and controlled perturbation results for KIVA, the separation between RIGIS training data and evaluation episodes, the decision-rule sensitivity analysis, and the timing protocol for the acceleration audit. If you have an opportunity to revisit our responses before the discussion period ends, we would greatly appreciate any further feedback or remaining concerns that we could clarify. Thank Reviewer again for your time and consideration.

Reviewer_fciXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RoboWorM, a benchmark for evaluating the physical consistency and inference efficiency of robotic video world models. The authors argue that existing benchmarks rely heavily on visual quality metrics or VLM-based judgments, which often fail to capture geometric and kinematic inconsistencies in generated videos.

RoboWorM collects 1,605 evaluation samples from five public datasets, covering single-arm, bimanual, and humanoid robots, as well as single-view, multi-view, and different task horizons. The paper introduces two metrics: KIVA, which evaluates kinematic violations and motion irregularities, and RIGIS, which measures multi-view geometric consistency. The authors also evaluate nine inference acceleration methods across three world models to study the trade-off between generation speed and physical consistency. The results show that the proposed metrics achieve better agreement with human judgments than VLM-based evaluators on their respective evaluation tasks.

Critical strengths
  1. The motivation is clear and practically relevant. Existing video world model benchmarks tend to focus on visual realism, while robotic applications require stronger guarantees on physical consistency. Using robot kinematics and camera geometry as evaluation priors is a reasonable direction and provides a useful complement to existing metrics.

  2. The benchmark covers a reasonably diverse set of robotic configurations. It includes different robot embodiments, task types, camera setups, and rollout lengths. The dataset construction and evaluation protocols are also described in considerable detail, which should help with reproducibility and future extensions.

  3. The analysis of inference acceleration is a useful addition. Rather than focusing only on speed and visual quality, the authors examine how different acceleration strategies affect kinematic and geometric consistency. The experiments across nine acceleration methods and three world models provide some useful practical insights.

Critical weaknesses
  1. The validity of KIVA as a measure of physical consistency needs further justification. KIVA relies on a diffusion model trained on real videos to predict 3D robot keypoints, and then uses deviations in link lengths and motion trajectories to identify hallucinations. However, unstable keypoint predictions do not necessarily indicate physical errors. They could also result from visual distribution shifts, occlusions, or degraded image quality. Conversely, a generated video could maintain stable robot geometry while still showing physically incorrect behavior. It is therefore unclear to what extent KIVA measures actual physical violations rather than the robustness of the keypoint predictor. In addition, Table D4 shows that KIVA achieves a Spearman correlation of only 0.390 on single-arm tasks, compared with 0.631 for the best VLM baseline. This suggests that its effectiveness may vary considerably across embodiments.

  2. The benchmark does not sufficiently cover environmental hallucinations and object-level physical consistency. KIVA mainly focuses on the robot itself, while RIGIS evaluates geometric consistency across camera views. However, hallucinations involving objects in the environment are equally important for robotic world models. For example, objects may suddenly appear or disappear, move without contact, or behave inconsistently during manipulation. These errors may not be detected by either metric, especially when the same physically incorrect behavior appears consistently across all camera views. In fact, the human evaluation protocol explicitly excludes certain environmental interactions and task outcomes. While this is understandable for evaluating geometric consistency, it also limits how much the proposed metrics can tell us about the physical correctness of a generated world.

  3. The benchmark's robustness against overfitting and metric gaming remains unclear. All evaluation samples come from five publicly available datasets, with relatively fixed task and scene distributions. The paper does not sufficiently clarify how training and evaluation trajectories are separated, or how it prevents models from exploiting the known evaluation data. More importantly, since KIVA and RIGIS rely on relatively explicit constraints, a model could potentially be optimized specifically for link-length consistency, motion smoothness, and multi-view geometry without actually improving its ability to predict physical dynamics. Such a model might achieve high benchmark scores while having limited generalization. Some evaluation on unseen scenes or embodiments, or an experiment involving direct optimization toward the proposed metrics, would help establish the reliability of this benchmark.

  4. The paper focuses on the limitations of VLM-based evaluation but does not sufficiently explore its potential advantages. I agree that VLMs are not particularly reliable at precise 3D geometric reasoning or kinematic verification. However, they can still be useful for evaluating task semantics, object relationships, and high-level physical plausibility, which are difficult to capture with the proposed metrics. Existing benchmarks such as WorldArena already explore combining algorithmic metrics with VLM-based evaluation. In contrast, this paper mainly compares KIVA and RIGIS against standalone VLM judges, without investigating whether these approaches provide complementary information. The experiments demonstrate the advantages of the proposed metrics on specific evaluation dimensions, but do not establish whether a purely algorithmic approach is preferable to a hybrid evaluation framework.

Questions for authors
  1. KIVA reliability: How can the authors distinguish keypoint prediction errors caused by actual physical violations from those caused by occlusion, lighting changes, motion blur, or other visual distribution shifts? Also, why does KIVA perform substantially worse than the best VLM baseline on single-arm tasks? Does this indicate a limitation in its generalization across embodiments?

  2. Environmental hallucinations: How would KIVA and RIGIS handle hallucinations such as objects suddenly appearing or disappearing, moving without contact, or exhibiting physically incorrect state transitions? If a generated video is geometrically consistent across all views but physically incorrect, could it still receive a high score? It would be helpful to clarify the scope of physical consistency covered by the benchmark.

  3. Benchmark robustness: Since all evaluation data come from public datasets, how do the authors prevent data memorization or benchmark-specific optimization? Are training and evaluation trajectories strictly separated? Could the authors evaluate on unseen scenes or directly optimize a model toward KIVA and RIGIS to test whether score improvements actually reflect better physical modeling?

  4. Complementarity with VLMs: Have the authors considered combining KIVA and RIGIS with VLM-based evaluation? For example, algorithmic metrics could measure low-level geometric constraints, while VLMs could assess task semantics and high-level physical plausibility. Would such a hybrid approach improve agreement with human judgments or help detect errors missed by either method alone?

  5. Long-horizon evaluation: The distinction between Markovian and non-Markovian tasks appears to be strongly correlated with rollout length in some experiments. How can the authors separate performance degradation caused by longer rollouts from that caused by task complexity or historical dependencies? A controlled comparison with similar rollout lengths would help support the conclusions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fciX — Part 1: KIVA Reliability and Environmental Hallucinations

We thank reviewer fciX for their thoughtful and detailed review of our paper, and for recognizing RoboWorM’s practical relevance and broad experimental coverage. We address each concern below.

KIVA reliability and single-arm performance

Reviewer question: KIVA reliability: How can the authors distinguish keypoint prediction errors caused by actual physical violations from those caused by occlusion, lighting changes, motion blur, or other visual distribution shifts? Also, why does KIVA perform substantially worse than the best VLM baseline on single-arm tasks? Does this indicate a limitation in its generalization across embodiments?

Distinguish keypoint prediction errors caused by actual physical violations from those caused by occlusion, lighting changes, motion blur, or visual distribution shifts. Thanks for raising this point! We agree that keypoint prediction errors can come from both settings, and have clarified this limitation in our revision. While KIVA does not disentangle these two settings, our goal is to measure the actual physical violations, and therefore requires the keypoint prediction errors to be more sensitive to the real physical violation and more robust to other environmental changes such as occlusion. We verify this by first computing KIVA scores on 8,093 generated segments, and then apply visual perturbations (occlusion, lighting changes, motion blurs) and recompute the scores. The table reports KIVA's violation rate, i.e., the percentage of segments flagged as violations, on the generated segments with actual physical violations (first row) and after each perturbation.

Setting (violation rate %)BimanualHumanoidSingle-arm
Actual physical violations83.346.485.9
Occlusion80.849.084.6
Lighting changes82.547.785.3
Motion blurs80.843.886.5

Perturbations change the violation rate by at most 2.6 pp, so KIVA responds to motion, not appearance. Humanoid has a lower rate because its generated segments contain fewer kinematic violations, as also seen in Table 2. We clarified this limitation in Limitations (iii) (Sec. 5, lines 493–496) and added held-out false-alarm rates and perturbations of real clips to App. D.3 (lines 2424–2476, Table D6).

Low correlation with human judgement on single-arm tasks. Thank you for highlighting this! We visually inspected single-arm generated segments with the largest disagreements between KIVA and human annotators, and identified an interesting failure case specific to single-arm setup. In these generated segments, the camera viewpoint causes the hallucinated arm to visually blend in with the background (Fig. D4b, App. D.3). Human annotators can still recognize the physical violation in the arm and rate these segments as Bad. KIVA, on the other hand, assigns a Good score under this visual ambiguity. We hypothesize that, when a hallucinated robot arm blends into the background, the keypoint predictor falls back on plausible trajectories learned from real videos. The predicted keypoints therefore remain geometrically consistent despite the visual corruption, causing KIVA to miss the hallucination. This failure mode contributes to the lower single-arm correlation. We included these examples in App. D.3 (Fig. D4, lines 2376–2396, and paragraph “Single-arm disagreements”, lines 2359–2366) and clarified this limitation further in Limitations (iii) (Sec. 5, lines 493–496) in our revision.

Environmental hallucinations

Reviewer question: Environmental hallucinations: How would KIVA and RIGIS handle hallucinations such as objects suddenly appearing or disappearing, moving without contact, or exhibiting physically incorrect state transitions? If a generated video is geometrically consistent across all views but physically incorrect, could it still receive a high score? It would be helpful to clarify the scope of physical consistency covered by the benchmark.

We thank the reviewer for highlighting this important point, and we have refined the scope accordingly in Limitations (i) (Sec. 5, lines 489–491) of the revised manuscript. KIVA and RIGIS assess robot kinematics and cross-view geometric consistency, respectively, and do not cover object-level dynamics. KIVA evaluates only the robot's own kinematics from its keypoints, so it does not consider whether other objects in the scene are hallucinated. RIGIS detects object hallucinations that differ across views, but an error that appears identically in every view, such as an object moving without contact, keeps the scene consistent and can still score well on both metrics.

AuthorsOpenReview ↗

Response to Reviewer fciX — Part 2: Benchmark Robustness

Benchmark robustness:

Reviewer question: Benchmark robustness: Since all evaluation data come from public datasets, how do the authors prevent data memorization or benchmark-specific optimization? Are training and evaluation trajectories strictly separated? Could the authors evaluate on unseen scenes or directly optimize a model toward KIVA and RIGIS to test whether score improvements actually reflect better physical modeling?

Evaluation scope. By design, our metrics RIGIS and KIVA evaluate generated rollouts from the world models. We need to adapt these world models to the target robot and camera setup using the benchmark’s real episodes, because without this, rollouts collapse (App. C.1, Fig. C1) and the dense model is not a meaningful reference for acceleration (App. E.2). The evaluator stays independent of the generated rollouts it scores: KIVA’s reader and θ_d are fitted on real footage only, and RIGIS uses a pretrained VGGT-Ω with no training on our datasets.

Data memorization. Our evaluation follows a task-level setting analogous to imitation learning [1][2], where the same task is demonstrated through multiple trajectories with different execution styles. The video world model is fine-tuned on short video clips sampled from these demonstrations (e.g., 93 frames per chunk for DreamGen), learning a distribution of plausible task executions rather than a single fixed trajectory. At evaluation, generation starts from a real initial frame and proceeds autoregressively, with each subsequent chunk conditioned on the final frames of the previously generated chunk, rather than ground-truth future frames. Generation is also guided by refined task prompts (e.g., using the prompt refiner in DreamGen's Cosmos-Predict2 backbone), and continues until the rollout reaches a duration comparable to the original task demonstrations. Although the first chunk may benefit from overlap with the fine-tuning data, later chunks must evolve under the model's own generated context. Thus, the evaluation tests sustained task-conditioned generation rather than exact reconstruction of a single training demonstration.

Unseen scenes. Following the reviewer’s suggestion, we evaluate all frozen checkpoints (with no further fine-tuning) on 15 new scenes that none of the world models or the keypoint predictor was trained on: five from Stanford ALOHA-static, five GR-1 in-lab scenes, and five RoboCOIN tasks not used in the benchmark. Our goal is to verify two things: (i) whether KIVA and RIGIS still separate real, dense, and accelerated rollouts, and (ii) whether the model ranking in Table 2 still holds. We added this experiment into our revised manuscript at Table D7:For KIVA, we keep the predictor frozen, refit d on 25 real episodes per embodiment, and test on 25 other episodes. The pattern on unseen scenes matches the seen ones: real video is flagged at about 10% in both, and accelerated rollouts are flagged 3.0 to 5.8 times as often as real video in both, confirming KIVA still detects degraded rollouts on unseen scenes.

KIVA flag rate (%)Bimanual seenBimanual unseenHumanoid seenHumanoid unseen
Real video (held-out)9.510.38.89.7
Dense rollouts12.38.918.23.3
Accelerated rollouts29.130.642.456.6

For RIGIS, no changes are needed on the new scenes. Real video scores 0.874 to 0.884 across both scene sets, dense Ctrl-World rollouts stay at the real-video level (0.883 unseen vs. 0.877 seen), Ctrl-World ranks above DreamGen across all five new RoboCOIN tasks just as in Table 2, and accelerated rollouts score at or below dense ones in every column.

RIGIS (↑)Ctrl-World seenCtrl-World unseenDreamGen seenDreamGen unseen
Real video0.8800.8840.8740.882
Dense rollouts0.8770.8830.8560.823
Accelerated (WorldCache)0.8770.8420.8480.817

Optimizing toward the metrics. None of the evaluated world models was trained or tuned on KIVA or RIGIS, so the reported scores do not come from optimizing toward the metrics. We agree that a model could, in principle, learn to keep bone lengths fixed, motion smooth, and views consistent without better modeling physical dynamics. While we have not trained such a model, KIVA and RIGIS are not meant to be used alone: we report them together with PSNR (Table 2), which compares each rollout with the real episode frame by frame. A model that satisfies our constraints but does not reproduce the real motion therefore still scores poorly on PSNR.

AuthorsOpenReview ↗

Response to Reviewer fciX — Part 3: VLM Complementarity and Long-horizon Evaluation

Complementarity with VLMs

Reviewer question: Complementarity with VLMs: Have the authors considered combining KIVA and RIGIS with VLM-based evaluation? For example, algorithmic metrics could measure low-level geometric constraints, while VLMs could assess task semantics and high-level physical plausibility. Would such a hybrid approach improve agreement with human judgments or help detect errors missed by either method alone?

We thank the reviewer for this suggestion, and we have added it as future work in the revised manuscript (Sec. 5, line 493-495). We agree that KIVA and RIGIS are complementary to VLM-based evaluation. Our metrics measure low-level kinematic and geometric constraints, while VLMs can assess task semantics, task completion, and high-level physical plausibility. For example, a VLM could recognize the task phase so that KIVA tolerates abrupt motion at contact events, or identify task-relevant objects so that RIGIS focuses on the regions that matter most. We consider this hybrid direction necessary for an adequate evaluation of robotic world models.

Long-horizon evaluation:

Reviewer question: Long-horizon evaluation: The distinction between Markovian and non-Markovian tasks appears to be strongly correlated with rollout length in some experiments. How can the authors separate performance degradation caused by longer rollouts from that caused by task complexity or historical dependencies? A controlled comparison with similar rollout lengths would help support the conclusions.

Rollout duration (bimanual)DreamGen (markov/non-markov)DreamDojo (markov/non-markov)
~60s28.2% / 20.9%21.1% / 17.0%

Table A: Both task regimes have a matched rollout length of 60 seconds, where Markovian tasks show more violations than non-Markovian ones for both models.

Rollout duration (single-arm)DreamGen (markov/non-markov)DreamDojo (markov/non-markov)
~11s56.8% / –16.3% / –
~17s54.0% / –– / –
~22s– / 56.0%– / 23.1%
~40s– / 56.0%– / 39.5%

Table B: KIVA violation rates (%) across rollout durations within each task regime for single-arm robots. – indicate insufficient samples. Markovian tasks are generally shorter, providing sufficient samples at 11 and 17 seconds, whereas non-Markovian tasks provide sufficient samples at 22 and 40 seconds. Comparisons across durations are therefore made within each regime.

Thank you for highlighting the potential confounding between rollout duration and task complexity (section 4.3, RQ4). To investigate further, we add a comparison at matched rollout duration (in seconds) and a separate analysis of duration within each task regime.

We first examined the rollout-duration distributions across embodiments. Bimanual and humanoid tasks have overlapping rollout-duration distributions, whereas single-arm non-Markovian rollouts are consistently longer than Markovian ones. For matched-duration comparisons (table A), we report numbers for configuration with more than 50 rollouts in each task regime at the same duration. Only the bimanual configurations satisfy this criterion; the humanoid subset has insufficient samples at individual shared durations, and the single-arm subset lacks duration overlap. We therefore report matched-duration results for bimanual tasks (Table A) and comparisons across durations within each single-arm task regime (Table B).

For bimanual rollouts of approximately 60 seconds (Table A), non-Markovian tasks have lower KIVA violation rates than Markovian tasks for both DreamGen (20.9% vs. 28.2%) and DreamDojo (17.0% vs. 21.1%). Thus, non-Markovian tasks do not consistently exhibit greater kinematic degradation when duration is matched. For single-arm tasks (Table B), DreamDojo’s violation rate increases from 23.1% at approximately 22 seconds to 39.5% at approximately 40 seconds within the non-Markovian regime, whereas DreamGen remains at 56.0%. This indicates that the association between duration and degradation varies across models.

We will incorporate these comparisons to strengthen our analysis on why the observed degradation cannot be attributed to the task regime alone.

References

[1] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, R. Martín-Martín. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation. In Conference on Robot Learning (CoRL), PMLR 164:1678–1690, 2021.

[2] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, R. Martín-Martín. What Matters in Learning from Offline Human Demonstrations for

Reviewer_aBBdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RoboWorM introduces a benchmark for evaluating robotic video world models beyond perceptual quality, focusing on kinematic stability, multi-view consistency, and the effects of inference acceleration. It comprises 1,605 real-world episodes spanning single-arm, bimanual, and humanoid embodiments under different camera configurations and task horizons. The paper proposes KIVA, which assesses estimated robot keypoint trajectories, and RIGIS, which combines cross-view geometric consistency with camera stability. The metrics are validated against human judgments, and nine acceleration techniques are evaluated across three world-model families using over 29,000 generated rollouts. The central contribution is an audit of structural failures that conventional visual-quality metrics may overlook.

Critical strengths

The evaluation problem is well motivated. Generated robotic motion can appear visually convincing while exhibiting changing link lengths or inconsistent geometry across views. Incorporating embodiment and camera constraints provides a useful complement to perceptual and semantic evaluation.

The empirical coverage is substantial. The benchmark spans multiple robot morphologies, camera configurations, and task horizons, while the acceleration audit compares caching, sparse attention, and token reduction across different architectures. Examining structural fidelity alongside speed provides practical information often absent from conventional acceleration evaluations.

The metrics receive substantive human validation, using 291 single-view and 300 multi-view videos. The appendix excludes KIVA calibration videos from the reported agreement evaluation and examines variation across calibration draws. This provides a stronger empirical foundation than selected qualitative examples alone.

Critical weaknesses

KIVA’s performance is inconsistent across embodiments. Appendix D.3 reports a correlation of approximately 0.39 with human labels on single-arm videos. The aggregate agreement therefore does not establish equally reliable detection across the robot morphologies covered by the benchmark.

Physical violations are not sufficiently distinguished from evaluator errors. KIVA relies on a learned trajectory reader, whose errors under occlusion or difficult viewpoints can also produce apparent rigidity or jerk violations. The disagreement analysis does not clearly establish how often the metric detects failures of reconstruction rather than failures in the generated motion.

Transfer to new embodiments requires clearer characterization. Classification agreement improves substantially after calibration using embodiment-specific human labels. This is reasonable, but introduces an additional annotation requirement whose cost and effect on performance should be explicit.

The KIVA thresholding protocol is inconsistent across sections. Section 4.1 flags normalized scores above one, whereas Section 3.4.1 and Appendix B.1 describe embodiment-specific calibrated thresholds. The distinction directly affects reproduction of the reported violation rates.

The acceleration conclusions lack a clear sensitivity analysis. Converting continuous errors into violation rates can make comparisons sensitive to the decision boundary. It is unclear whether the principal acceleration rankings remain stable under continuous KIVA scores or alternative reasonable thresholds.

Questions for authors

Please identify the KIVA thresholds used for Table 2 and the acceleration results. Distinguish the real-video normalization level, the binary violation threshold, and the two cuts used for three-level human agreement. If Section 4.1 describes an outdated protocol, does correcting it affect any reported values? What explains the relatively weak single-arm correlation in Appendix D.3? Representative disagreement cases would help distinguish trajectory-reader errors from limitations of rigidity and jerk as proxies for the defects annotated by humans. Do the main acceleration conclusions hold under continuous KIVA scores or alternative reasonable thresholds? Any available sensitivity analysis would clarify whether the observed ordering is stable. For a new embodiment, what training data, human annotations, and calibration steps are required before KIVA can be used reliably? Please distinguish reusable components from embodiment-specific ones.

2 comments
AuthorsOpenReview ↗

Author response to reviewer aBBd - Part 1: KIVA Scoring and Reliability

We thank reviewer aBBd for their positive and constructive feedback on our manuscript. Below, we further address each point raised.

[Q1] KIVA thresholding protocol and reported results

Reviewer question: Please identify the KIVA thresholds used for Table 2 and the acceleration results. Distinguish the real-video normalization level, the binary violation threshold, and the two cuts used for three-level human agreement. If Section 4.1 describes an outdated protocol, does correcting it affect any reported values?

KIVA has a unified definition for all reported benchmark results, including Table 2 and the acceleration results. Each segment's rigidity and jerk errors are normalized by their respective real-video reference levels θd\theta_d, a segment is flagged as a violation when sS>1s_S>1, and KIVA is the resulting violation rate over the rollout:

KIVA=100N∑i=1N1[sSi>1]\mathrm{KIVA}=\frac{100}{N}\sum_{i=1}^{N}\mathbb{1}[s_{S_i}>1]

sSi=max⁡d∈DevSidθds_{S_i} = \max_{d\in\mathcal{D}_e} \frac{v^d_{S_i}}{\theta_d}

θd=Q0.95({vRd:R∈R})\theta_d=Q_{0.95}\left(\{v^d_R:R\in\mathcal{R}\}\right)

The two-cut calibrations are used only in the three-level human study, where they map continuous scores to discrete human ratings; they do not affect the KIVA violation rates in Table 2 or the acceleration results.

We apologize for the confusion and have restructured the revised manuscript accordingly:

  • Section 3.4.1, lines 293-295: KIVA is now explicitly defined as the violation rate, and the reference to human calibration is removed: "A segment S is flagged as a violation when sS>1, i.e., when its error exceeds the 95th percentile of real footage. KIVA is reported as the percentage of segments flagged as violations."
  • Section 4.1, lines 357-358: Restated to match Sec. 3.4.1, clarifying that KIVA results in Table 2 and Figure 5 are violation rates: "KIVA is reported as the violation rate, i.e., the percentage of segments flagged as violations (sS>1, Sec. 3.4.1), averaged over single-view rollouts."
  • Appendix B.1, lines 1249-1250: Reorganized into Violation threshold followed by Score aggregation, with human calibration removed.
  • Appendix D.3, line 2309-2319: The human-fitted cuts are now described here, as they are used only to compare KIVA with human ratings.

[Q2] KIVA reliability and single-arm disagreements

Reviewer question: What explains the relatively weak single-arm correlation in Appendix D.3? Representative disagreement cases would help distinguish trajectory-reader errors from limitations of rigidity and jerk as proxies for the defects annotated by humans.

Physical violations vs. keypoint predictor errors. KIVA should be sensitive to actual physical violations and robust to appearance changes. We test this in two ways. First, on held-out real video (Table D6), visual edits such as brightness, blur, or an occlusion patch preserve the motion and produce violation rates comparable to KIVA's false-alarm rates on unedited real footage. Splicing two clips so that the arm jumps between poses introduces motion discontinuities, and most spliced segments are flagged.

Second, on 8,093 generated segments, we apply occlusion, lighting changes, and motion blur, then recompute KIVA:

Setting (violation rate, %)BimanualHumanoidSingle-arm
Original generated segments83.346.485.9
+ Occlusion80.849.084.6
+ Lighting changes82.547.785.3
+ Motion blur80.843.886.5

Perturbations change the violation rate by at most 2.6 percentage points, supporting KIVA's robustness to these tested appearance changes.

Weak single-arm correlation. Among the 20 single-arm segments with the largest disagreements between KIVA and human labels, we identified a recurring failure case. The camera viewpoint causes a hallucinated arm to blend into the background (Fig. D4). Annotators still rate these segments as Bad, but KIVA does not flag them. We hypothesize that the reader falls back on plausible trajectories learned from real videos, so its predicted keypoints remain geometrically consistent despite the visual corruption. This failure mode contributes to the lower single-arm correlation, and we now explicitly acknowledge it as a limitation.

Manuscript revisions. We added held-out keypoint accuracy, real-video false-alarm rates, and controlled perturbations toApp. D.3 (lines 2424–2476, Table D6); generated-video perturbations to App. D.3 (line 2359-2366, Fig. D4); representative single-arm disagreement cases to App. D.3 (lines 2359-2366, Fig. D4); and the corresponding limitation to Sec. 5 (lines 493–496).

AuthorsOpenReview ↗

Response to Reviewer aBBd — Part 2: Acceleration Sensitivity and Transfer Requirements

[Q3] Sensitivity of acceleration conclusions

Reviewer question: Do the main acceleration conclusions hold under continuous KIVA scores or alternative reasonable thresholds? Any available sensitivity analysis would clarify whether the observed ordering is stable.

The main acceleration conclusions related to the single-view section in Table 2 and Figure 5 are based on the KIVA score, which measures the violation percentage with the protocol defined in Sections 3.4.1 and 4.1.

For a comprehensive sensitivity analysis of the decision methods, we re-ranked the accelerators in each of the 12 single-view evaluation settings (model × embodiment × task regime) in two new ways:

  • Continuous score: The first uses the continuous score sSs_S instead of the violation rate, so no threshold is involved.
  • Alternative threshold: The second uses a different threshold, γ1\gamma_1, the lower of the two cuts fitted on nine human-labeled videos per morphology for the human study.

In 11 of the 12 settings, all three scoring rules identify the same best family (caching in 7, sparse attention in 4). In the remaining one, DreamGen humanoid Markovian, only the cut γ1\gamma_1 swaps first place from sparse attention to caching, by a margin of 0.6 percentage points (70.9% vs. 71.5%).

We have added a Decision-rule sensitivity analysis to Appendix D.3 (lines 2415–2422, Table D5) of the revised manuscript.

[Q4] Requirements for a new embodiment

Reviewer question: For a new embodiment, what training data, human annotations, and calibration steps are required before KIVA can be used reliably? Please distinguish reusable components from embodiment-specific ones.

Applying KIVA to a new embodiment requires real robot data but no human labels, which are optional and used only for three-level grades.

Reusable: The reader architecture and training recipe, the rigidity and jerk definitions, and the KIVA scoring protocol.

Embodiment-specific, no human labels: (1) The robot's kinematic tree, which gives forward-kinematics keypoints, (2) 120–300 real episodes (0.4–2.8 h) with joint states, used to train the reader, and (3) θd\theta_d, the 95th percentile of the reader's error calculated on these real videos. With these three, KIVA reports violation rates, as used for all main results.

Optional human labels: Only if we want to convert KIVA into Good / Medium / Bad grades like a human rater. We use nine labeled videos per embodiment (three per level) to fit the two cuts (App. D.3, lines 2309–2319).

We have added this procedure to the revised manuscript in Appendix E.2, lines 2802–2809:

"KIVA requires real robot data but no human labels. The architecture, rigidity and jerk definitions, and training recipe are reusable. Three components are embodiment-specific: (1) the robot's kinematic tree, which gives forward-kinematics keypoints; (2) 120 to 300 real episodes (0.4 to 2.8 h) with joint states, used to train the morphology-specific reader; and (3) θ_d, the 95th percentile of the reader's error on these real videos. With these, KIVA reports violation rates (Sec. 3.4.1), as used for all main results. To convert KIVA into a human-level indicator (Good / Medium / Bad), a small set of human annotations is needed (e.g., we use 9 labeled videos per embodiment, 3 per level), on which the two cuts are then fitted following Appendix D.3."

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Strengths and How We Addressed the Main Concerns

Dear Area Chair and Reviewers,

We thank the reviewers for their careful and constructive reviews. We have answered each review point by point and uploaded a revised PDF, with all changes marked in blue. The Summary of Revisions table lists each change that addresses a reviewer comment, with its location in the manuscript and the reviewer who raised it. Below, we summarize the strengths the reviewers recognized and how we addressed their main concerns.

Contributions and strengths recognized by the reviewers

  • Contribution
    • Well-motivated, practically relevant problem. Generated robot motion can look convincing while violating kinematics or cross-view geometry, and embodiment and camera constraints complement perceptual and semantic metrics (aBBd, fciX).
    • Substantial empirical coverage. The benchmark spans multiple morphologies, camera setups, task types and horizons, with about 29,000 rollouts (aBBd, fciX, nw9S).
    • Acceleration audit. Evaluating structural fidelity alongside speed across caching, sparse attention and token reduction provides practical information absent from conventional acceleration evaluations (aBBd, fciX).
  • Soundness
    • Human validation of the metrics, with calibration videos excluded from the reported agreement (aBBd). RIGIS aligns strongly with human judgments across morphologies (nw9S).
  • Presentation: detailed dataset construction and protocols that support reproducibility and extension (fciX).

How we addressed the main concerns

The reviewers' concerns fall into four themes.

  1. KIVA reliability (aBBd W1, W2, fciX W1, nw9S W1). The reviewers asked whether KIVA flags physical violations or keypoint predictor errors. On real videos, KIVA flags at most 10.6 / 0.4 / 9.6% of segments after appearance-only perturbations (bimanual / humanoid / single-arm), against 87.2 / 100 / 97.9% for synthetic motion discontinuities (App. D.3, Table D6). We also added representative single-arm disagreement cases and the corresponding limitation (Sec. 5, App. D.3).
  2. Consistent scoring protocol (aBBd W4, W5, nw9S W2, Q4). All reported KIVA results now use one fixed rule, the violation rate against the 95th percentile of real-video errors, and human-fitted thresholds appear only in the human alignment (App. D.3). The leading acceleration family stays unchanged in 11 of 12 settings under the continuous score and an alternative threshold (App. D.3, Table D5). We also documented configuration selection and timing provenance of the speedups (Sec. 4.1, App. C.2) and corrected the RIGIS mapping (Sec. 3.4.2).
  3. Scope and generalization (aBBd W3, fciX W2–W4, nw9S Q4). We now state that the main comparisons are in-distribution evaluations (App. D.3) and that KIVA and RIGIS measure robot kinematics and cross-view geometry, not complete physical correctness (Sec. 5). On five new RoboCOIN tasks with frozen models, Ctrl-World keeps its RIGIS advantage over DreamGen (App. D.3, Table D8). App. E.2 describes how to apply KIVA to a new embodiment, and VLM-based semantic evaluation is added as future work (Sec. 5).
  4. Rollout length versus task regime (fciX Q5). We added a comparison at matched rollout duration and a separate analysis of duration within each task regime. At about 60 s on bimanual tasks, non-Markovian tasks have lower KIVA violation rates than Markovian tasks for both DreamGen and DreamDojo. Within the single-arm non-Markovian regime, the association between duration and degradation varies across models. The observed degradation therefore cannot be attributed to the task regime alone (Sec. 4.3, App. C.2, Tables C6 and C7).

The remaining minor questions and presentation suggestions are fully addressed in the individual replies.

We hope the additional evidence demonstrates that RoboWorM makes a meaningful contribution towards reliable evaluation of movement realism, consistency, and acceleration effects in embodied video world models. We would be grateful to know whether these replies address your concerns.

AuthorsOpenReview ↗

General Response: Summary of Revisions

All changes are marked in blue in the revised PDF. Line numbers refer to the revised manuscript.

ChangeLocationRaised by
KIVA defined as the percentage of segments flagged as violations (sS>1s_S > 1). Table 2 KIVA column labelled as a percentageSec. 3.4.1, l. 293–295; Sec. 4.1, l. 357–358; Table 2, l. 393aBBd W4, Q1; nw9S W2, Q1
RIGIS mapping corrected to match Eq. B.7 (linear for RGB, quadratic for normalized depth)Sec. 3.4.2, l. 314–315nw9S W2, Q1
Speedup defined as the ratio of mean dense to mean accelerated generation time over complete evaluation rollouts, with dense and accelerated runs on the same episodesSec. 4.1, l. 362–363nw9S W2, Q4
Rollout length and task regime, with a pointer to the matched-duration analysisSec. 4.3 (RQ4), l. 419, 423–427fciX Q5
Limitations (i): scope narrowed to robot kinematics and cross-view geometrySec. 5, l. 489–491fciX W2, Q2
Limitations (iii): keypoint predictor errors under occlusion or appearance changes, and hallucinated arms that blend into the backgroundSec. 5, l. 493–496aBBd W1, W2, Q2; fciX W1, Q1
VLM-based evaluation combined with RoboWorM added as future workSec. 5, l. 498–501fciX W4, Q4
Violation rule in the full KIVA protocol restated as sS>1s_S > 1, without human calibrationApp. B.1, l. 1249–1250aBBd W4, Q1; nw9S W2, Q1
Configuration selection, aggregation and timing provenance of the reported speedups. Scope of the uniform timing pass and of the 12/12 speedup claimApp. C.2, l. 1659–1663, 1686–1695, 1701–1711nw9S W2, Q4
Markovian vs. non-Markovian comparison at matched rollout duration, and duration within each single-arm regimeApp. C.2, l. 1749–1764; Tables C5, C6 and C7fciX Q5
Segmentation corrected to 16-frame windowsApp. D.1.2, l. 1931; Table D2, l. 2008; App. D.1.3, l. 2020; Fig. D1, l. 2099nw9S Q1
Human-fitted thresholds described only for the human-score alignmentApp. D.3, l. 2309–2319; Eq. D.1aBBd W4, Q1; nw9S W2, Q1
Single-arm disagreement cases, covering keypoint predictor errors and robots that blend into the backgroundApp. D.3, l. 2359–2366; Fig. D4aBBd W1, W2, Q2; fciX W1, Q1
VLM agreement recomputed on the same held-out videos as KIVAApp. D.3, l. 2404–2405nw9S Q1
Decision-rule sensitivity under the continuous score and an alternative thresholdApp. D.3, l. 2415–2422; Table D5aBBd W5, Q3
Held-out keypoint accuracy, false-alarm rates on real video, controlled perturbations of real clips, and the link between sample disagreement and KIVAApp. D.3, l. 2424–2429, 2458–2465; Table D6nw9S W1, Q2; aBBd W2; fciX W1, Q1
Episode-level separation of world-model training, keypoint predictor training and reference calibration. Table 2 rankings stated as in-distribution performanceApp. D.3, l. 2479–2489nw9S W2, Q4; fciX W3, Q3
Evaluation on 15 unseen scenes with frozen checkpointsApp. D.3, l. 2491–2496; Tables D7 and D8fciX W3, Q3
Procedure for applying KIVA to a new embodiment, separating reusable and embodiment-specific componentsApp. E.2, l. 2802–2809aBBd W3, Q4; nw9S W2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.