- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ActionTrace addresses action-consequence collapse in robot video world models. It combines camera-aligned kinematic conditioning, a recurrent state carried across video chunks, and branch supervision contrasting trajectories collected from the same simulator reset. Experiments evaluate prescribed action interventions, closed-loop policy evaluation on 50 RoboTwin tasks, and component ablations. The method obtains Pearson correlations of 0.9805 for four newly trained policies and 0.9920 for the five WorldArena policies. A modified transfer of its conditioning components to Ctrl-World also improves agreement with previously reported real-world policy outcomes.
Critical strengths
- The paper addresses a central requirement for embodied world models: predictions must distinguish the consequences of successful and unsuccessful controls, rather than merely produce plausible task videos. Projecting robot trajectories provides spatial grounding, while carrying state across chunks is a reasonable mechanism for retaining earlier interaction outcomes. Same-reset branch collection also provides a useful experimental basis for learning action dependence.
- The evaluation includes several complementary forms of evidence. Beyond image fidelity, the paper tests explicit interventions, evaluates newly trained policies excluded from fine-tuning data collection, and reports matched component ablations. The appendices disclose success-rate calibration, task-bootstrap uncertainty, and the limitations of the real-world transfer. These details make the findings more informative than a comparison based only on video metrics or aggregate correlation.
Critical weaknesses
- The recursive paired outcome supervision has an unresolved consistency problem. Appendix A.3 states that a paired pass can share A’s generated conditioning image and carried state, substitute B’s controls, and receive B’s recorded terminal label, even though this label need not describe executing B’s controls after A’s prefix. This associates a potentially incorrect outcome with the conditioned state–action sequence. It is particularly concerning for a method whose main objective is faithful action consequences. The branch-supervision ablation removes paired passes collectively and therefore does not isolate this source of inconsistent supervision. A controlled comparison that masks or recomputes invalid paired outcome targets is needed to establish the soundness of this training choice.
- The evidence for robust consequence prediction and practical policy evaluation remains weaker than the headline correlations suggest. The intervention study contains only 24 pairs from eight resets; at step 128, ActionTrace correctly predicts 21/48 pair–seed cases versus 18/48 for X-WAM (+ data), with the improvement confined to early release (Table 9). For the four new policies, all paired bootstrap intervals for metric differences include zero (Table 12). On WorldArena, the 0.9920 correlation accompanies an increase in MAE from 0.0426 for the data-only model to 0.0950, indicating a substantial ranking–calibration tradeoff. The hardware-related evaluation uses previously reported outcomes for seven tasks and three policies; the transferred system omits projected features at evaluation and does not test same-reset branch supervision on real interactions. It is useful supporting evidence, but does not establish real-world validity of the full method.
Questions for authors
- Can you isolate paired terminal supervision by masking labels that are invalid after A’s prefix, while holding branch ranking, factual supervision, architecture, and training schedule fixed? How often does the present recipe apply such labels?
- Do the improvements persist across a more diverse policy set and additional intervention resets? Please quantify uncertainty and report both ranking quality and absolute success-rate calibration.
- Is there controlled hardware evidence that the model distinguishes different physical outcomes from matched initial states? In the Ctrl-World transfer, what do the kinematic embedding and consequence state contribute separately?
Thank you for your careful review and for identifying the paired-supervision issue. We have completed a controlled retraining comparison and expanded the evaluation. The intervention and model-comparison additions are highlighted in blue in the manuscript.
W1 / Q1: “Masking labels that are invalid after A's prefix,” with the other training choices held fixed.
Thank you for pointing out this inconsistency. We acknowledge that some paired continuations in the original training receive B's terminal label while being conditioned on A's history, although B's recorded outcome need not apply after A's prefix. To isolate its effect, we retrained ActionTrace with these paired terminal targets masked in both training stages, keeping branch ranking, factual outcome and future-latent supervision, architecture, and schedule unchanged. We compare the two models on the five WorldArena policies.
The two runs yield nearly identical results: Pearson differs by 0.002, both recover the complete simulator ordering, and MAE/RMSE remain similar. This limited effect is consistent with the auxiliary role of the outcome objective, weighted at 0.05 during consequence-state training and 0.01 during joint training. The comparison indicates that the target-assignment issue has little effect on the reported policy-ranking result and does not alter our core empirical conclusion.
Q2: “Do the improvements persist across a more diverse policy set and additional intervention resets?”
We expanded our evaluation of paired-outcome accuracy to 30 independent resets across six tasks, with three interventions and two seeds per reset (180 pair–seed cases per model at each step).
ActionTrace improves in all three intervention types at both steps, with fewer clear errors and similar indeterminate proportions overall. Section 4.2 and Appendix B.1 now include these results.
In the four-policy evaluation, ActionTrace's Pearson gain over X-WAM (+ data) is +0.0621 (95% CI: [−0.0495, +0.1929]). Against OSCAR on the same policies, the Pearson gain is +0.1310 (95% CI: [−0.0372, +1.5303]), with MAE decreasing from 11.45 to 6.90 pp and RMSE from 13.61 to 7.60 pp (Tables 1 and 12).
For the five-policy results, we agree that ranking and absolute error should be considered separately. X-WAM (+ data) achieves lower MAE but reverses the two strongest policies, whereas ActionTrace recovers the complete simulator ordering. As discussed in Section 4.3 and Limitations, our principal gain is more reliable identification of relative policy strength.
W2 / Q3: “Controlled hardware evidence” and what the kinematic embedding and consequence state “contribute separately.”
We appreciate your emphasis on controlled physical evidence. We have not conducted matched-state hardware interventions; our controlled tests instead use simulator resets to reproduce the same initial state across action branches and isolate action effects. The Ctrl-World experiments complement this analysis by testing whether the transferred components improve policy evaluation against reported real-world outcomes. We added the two single-component variants below under matched DROID training and evaluation settings.
Both components independently improve correlation, and combining them yields the best Pearson, MAE, and RMSE. These ablations support complementary contributions to policy evaluation on Ctrl-World and are now included in Table 7 and Appendix B.6.
We appreciate your careful assessment and the opportunity to clarify these points. We hope the controlled retraining and expanded evaluation address your main concerns, and we welcome any further questions.