acceptodds
Under review as a conference paper at ICLR 2027

ActionTrace: Learning Action Consequences in Robot World Models

Abstract

Robot world models offer a way to evaluate policies through imagined rollouts, provided that their predictions reflect the consequences of the supplied actions. However, from the same initial scene, a model may generate similar, visually plausible futures for actions with different physical outcomes, obscuring differences between policies. To preserve these distinctions, we introduce ActionTrace, a robot world model that learns action consequences through spatial and temporal conditioning. It projects joint trajectories into the camera view to guide predicted robot motion and carries an action-updated consequence state across video segments, allowing earlier actions to inform later predictions. We reinforce this action dependence during training by favoring executed controls over alternatives from the same simulator reset for each observed future. Under paired action interventions, ActionTrace predicts object outcomes with higher overall accuracy across recursive rollouts. Across four newly trained policies on 50 manipulation tasks, ActionTrace recovers their ordering with a Pearson correlation of 0.9805 between predicted and simulator success rates. On the WorldArena benchmark, it also recovers the ordering of five released policies, reaching a Pearson correlation of 0.9920. The model achieves these policy-evaluation gains while retaining comparable long-horizon video fidelity. Adapting its components to Ctrl-World extends these policy-evaluation improvements to real-world outcomes across seven tasks and three policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Robot Manipulation, Policy Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w1mnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActionTrace addresses action-consequence collapse in robot video world models. It combines camera-aligned kinematic conditioning, a recurrent state carried across video chunks, and branch supervision contrasting trajectories collected from the same simulator reset. Experiments evaluate prescribed action interventions, closed-loop policy evaluation on 50 RoboTwin tasks, and component ablations. The method obtains Pearson correlations of 0.9805 for four newly trained policies and 0.9920 for the five WorldArena policies. A modified transfer of its conditioning components to Ctrl-World also improves agreement with previously reported real-world policy outcomes.

Critical strengths
  1. The paper addresses a central requirement for embodied world models: predictions must distinguish the consequences of successful and unsuccessful controls, rather than merely produce plausible task videos. Projecting robot trajectories provides spatial grounding, while carrying state across chunks is a reasonable mechanism for retaining earlier interaction outcomes. Same-reset branch collection also provides a useful experimental basis for learning action dependence.
  2. The evaluation includes several complementary forms of evidence. Beyond image fidelity, the paper tests explicit interventions, evaluates newly trained policies excluded from fine-tuning data collection, and reports matched component ablations. The appendices disclose success-rate calibration, task-bootstrap uncertainty, and the limitations of the real-world transfer. These details make the findings more informative than a comparison based only on video metrics or aggregate correlation.
Critical weaknesses
  1. The recursive paired outcome supervision has an unresolved consistency problem. Appendix A.3 states that a paired pass can share A’s generated conditioning image and carried state, substitute B’s controls, and receive B’s recorded terminal label, even though this label need not describe executing B’s controls after A’s prefix. This associates a potentially incorrect outcome with the conditioned state–action sequence. It is particularly concerning for a method whose main objective is faithful action consequences. The branch-supervision ablation removes paired passes collectively and therefore does not isolate this source of inconsistent supervision. A controlled comparison that masks or recomputes invalid paired outcome targets is needed to establish the soundness of this training choice.
  2. The evidence for robust consequence prediction and practical policy evaluation remains weaker than the headline correlations suggest. The intervention study contains only 24 pairs from eight resets; at step 128, ActionTrace correctly predicts 21/48 pair–seed cases versus 18/48 for X-WAM (+ data), with the improvement confined to early release (Table 9). For the four new policies, all paired bootstrap intervals for metric differences include zero (Table 12). On WorldArena, the 0.9920 correlation accompanies an increase in MAE from 0.0426 for the data-only model to 0.0950, indicating a substantial ranking–calibration tradeoff. The hardware-related evaluation uses previously reported outcomes for seven tasks and three policies; the transferred system omits projected features at evaluation and does not test same-reset branch supervision on real interactions. It is useful supporting evidence, but does not establish real-world validity of the full method.
Questions for authors
  1. Can you isolate paired terminal supervision by masking labels that are invalid after A’s prefix, while holding branch ranking, factual supervision, architecture, and training schedule fixed? How often does the present recipe apply such labels?
  2. Do the improvements persist across a more diverse policy set and additional intervention resets? Please quantify uncertainty and report both ranking quality and absolute success-rate calibration.
  3. Is there controlled hardware evidence that the model distinguishes different physical outcomes from matched initial states? In the Ctrl-World transfer, what do the kinematic embedding and consequence state contribute separately?
1 comment

Thank you for your careful review and for identifying the paired-supervision issue. We have completed a controlled retraining comparison and expanded the evaluation. The intervention and model-comparison additions are highlighted in blue in the manuscript.

W1 / Q1: “Masking labels that are invalid after A's prefix,” with the other training choices held fixed.

Thank you for pointing out this inconsistency. We acknowledge that some paired continuations in the original training receive B's terminal label while being conditioned on A's history, although B's recorded outcome need not apply after A's prefix. To isolate its effect, we retrained ActionTrace with these paired terminal targets masked in both training stages, keeping branch ranking, factual outcome and future-latent supervision, architecture, and schedule unchanged. We compare the two models on the five WorldArena policies.

ModelPearson ↑Spearman ↑MAE (pp) ↓RMSE (pp) ↓
Original ActionTrace0.99201.00009.5010.21
ActionTrace with paired-target masking0.99001.00009.3010.24

The two runs yield nearly identical results: Pearson differs by 0.002, both recover the complete simulator ordering, and MAE/RMSE remain similar. This limited effect is consistent with the auxiliary role of the outcome objective, weighted at 0.05 during consequence-state training and 0.01 during joint training. The comparison indicates that the target-assignment issue has little effect on the reported policy-ranking result and does not alter our core empirical conclusion.

Q2: “Do the improvements persist across a more diverse policy set and additional intervention resets?”

We expanded our evaluation of paired-outcome accuracy to 30 independent resets across six tasks, with three interventions and two seeds per reset (180 pair–seed cases per model at each step).

Control stepX-WAM (+ data)ActionTraceGain (pp), 95% CI
6420.0%30.0%+10.0 [5.00, 15.00]
12818.3%27.2%+8.9 [3.33, 15.00]

ActionTrace improves in all three intervention types at both steps, with fewer clear errors and similar indeterminate proportions overall. Section 4.2 and Appendix B.1 now include these results.

In the four-policy evaluation, ActionTrace's Pearson gain over X-WAM (+ data) is +0.0621 (95% CI: [−0.0495, +0.1929]). Against OSCAR on the same policies, the Pearson gain is +0.1310 (95% CI: [−0.0372, +1.5303]), with MAE decreasing from 11.45 to 6.90 pp and RMSE from 13.61 to 7.60 pp (Tables 1 and 12).

For the five-policy results, we agree that ranking and absolute error should be considered separately. X-WAM (+ data) achieves lower MAE but reverses the two strongest policies, whereas ActionTrace recovers the complete simulator ordering. As discussed in Section 4.3 and Limitations, our principal gain is more reliable identification of relative policy strength.

W2 / Q3: “Controlled hardware evidence” and what the kinematic embedding and consequence state “contribute separately.”

We appreciate your emphasis on controlled physical evidence. We have not conducted matched-state hardware interventions; our controlled tests instead use simulator resets to reproduce the same initial state across action branches and isolate action effects. The Ctrl-World experiments complement this analysis by testing whether the transferred components improve policy evaluation against reported real-world outcomes. We added the two single-component variants below under matched DROID training and evaluation settings.

Ctrl-World variantPearson ↑Spearman ↑MAE (pp) ↓RMSE (pp) ↓
Continued baseline0.60070.582821.9026.86
Kinematic conditioning only0.65960.639222.1430.76
Consequence conditioning only0.67080.653520.2428.81
Both components0.72830.652715.9521.24

Both components independently improve correlation, and combining them yields the best Pearson, MAE, and RMSE. These ablations support complementary contributions to policy evaluation on Ctrl-World and are now included in Table 7 and Appendix B.6.

We appreciate your careful assessment and the opportunity to clarify these points. We hope the controlled retraining and expanded evaluation address your main concerns, and we welcome any further questions.

Reviewer_xgHCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ActionTrace aims to make robot video world models better reflect the consequences of supplied actions. It combines projected robot trajectories, a recurrent consequence state, and supervision contrasting action sequences from the same initial state. Experiments evaluate action-dependent object outcomes, policy ranking in simulation, and agreement with real-world policy performance through adaptation to Ctrl-World.

Critical strengths

The paper provides thorough ablations of the conditioning components, branch supervision, and recurrent state. These comparisons help establish which design choices contribute to the reported policy-evaluation gains.

Critical weaknesses
  1. The evidence for reliable action-consequence prediction remains limited. Paired-outcome accuracy is below 50%, with a 6.25 percentage-point improvement over the data-matched baseline. This evaluation covers only 24 action pairs involving relatively simple grasping and release interactions. Although policy-ranking results are encouraging, these findings leave uncertainty about reliability on more complex interactions.

  2. The motivation and novelty need clearer explanation. The introduction introduces several mechanisms without sufficiently explaining why each is necessary or how they differ from existing action-conditioning and memory approaches. This makes the central contribution and its significance harder to assess.

Questions for authors
  1. Section 4.5 shows improved agreement with real-world success rates after transferring selected components to Ctrl-World. Does this adaptation use same-reset branch supervision? Which contributions are supported by this experiment?

  2. What accounts for the low paired-outcome accuracy-- incorrect physical outcomes, visually ambiguous generations, or both?

  3. How robust is the improvement in correlation with real-world success rates in Table 7?

1 comment

Thank you for your careful review and for recognizing the value of our ablations. We have expanded the intervention evaluation and added Ctrl-World component ablations. Manuscript revisions are highlighted in blue.

W1 / Q2: “Only 24 action pairs” and “incorrect physical outcomes, visually ambiguous generations, or both?”

We expanded the evaluation to 30 independent resets across six tasks and separately classified correct, incorrect, and visually indeterminate outcomes. A pair is correct if both branches match their simulator outcomes, incorrect if either clearly contradicts its expected outcome, and otherwise indeterminate.

Among each model's determinate pairs, correct / (correct + incorrect) increases from 33.3% for X-WAM (+ data) to 50.5% for ActionTrace at step 64 and from 32.0% to 49.5% at step 128. ActionTrace produces more correct outcomes and fewer clear errors, with similar indeterminate proportions. Three interventions and two seeds per reset give 180 cases per model at each step; entries are count (%).

Control stepModelCorrectIncorrectIndeterminate
64X-WAM (released)28 (15.6%)76 (42.2%)76 (42.2%)
64X-WAM (+ data)36 (20.0%)72 (40.0%)72 (40.0%)
64ActionTrace54 (30.0%)53 (29.4%)73 (40.6%)
128X-WAM (released)24 (13.3%)74 (41.1%)82 (45.6%)
128X-WAM (+ data)33 (18.3%)70 (38.9%)77 (42.8%)
128ActionTrace49 (27.2%)50 (27.8%)81 (45.0%)

The original eight resets and their scores are retained; all three models score lower on the 22 added resets, reducing the overall accuracy. Across the full expanded set, ActionTrace improves paired-outcome accuracy over X-WAM (+ data) by 10.0 pp at step 64 (95% CI: [5.00, 15.00]) and 8.9 pp at step 128 (95% CI: [3.33, 15.00]), with gains across all three intervention types. These results and uncertainty estimates are now in Section 4.2 and Appendix B.1.

W2: “The motivation and novelty need clearer explanation.”

Thank you for asking us to clarify the design rationale. Our goal is to generate futures that reflect the supplied actions. This requires a training objective that rewards using actions, together with representations that help the model predict their effects.

However, the scene and instruction can suggest a plausible continuation even when the actions would lead elsewhere. Same-reset Branch Supervision holds these cues fixed and favors the executed controls over alternatives for the same observed future. To turn these action differences into video predictions, Camera-Aligned Kinematic Grounding maps commanded motion into the camera view. Because the effects of that motion also depend on earlier interactions, Persistent Consequence Conditioning carries interaction information across video chunks.

Together, the supervision encourages the model to distinguish actions, while the conditioning helps it represent their consequences in space and over time. This connection is central to preserving policy differences in imagined rollouts.

Q1: “Does this adaptation use same-reset branch supervision?” Which contributions does it support?

Thank you for asking us to clarify this. The Ctrl-World adaptation does not use same-reset branch supervision. We train on recorded DROID trajectories and evaluate with the camera-independent kinematic embedding and recurrent consequence state. The improvement over matched continued training of Ctrl-World supports the transferability of our kinematic and consequence conditioning to another video backbone for policy evaluation.

Q3: “How robust is the improvement in correlation” in Table 7?

We trained two single-component variants with matched DROID data, initialization, and schedule. Each uses 420 evaluation rollouts across seven tasks and three policies; metrics compare the 21 task–policy success rates with reported real-world rates.

Ctrl-World variantPearson ↑Spearman ↑MAE (pp) ↓RMSE (pp) ↓
Continued baseline0.60070.582821.9026.86
Kinematic conditioning only0.65960.639222.1430.76
Consequence conditioning only0.67080.653520.2428.81
Both components0.72830.652715.9521.24

Each component improves correlation over the continued baseline; combining them gives the highest Pearson and lowest MAE and RMSE. These matched comparisons support contributions from both components beyond continued training alone. Table 7 and Appendix B.6 now include the ablations.

We hope this evidence addresses your concerns and would be grateful if you could revisit your rating. We welcome further questions.

Reviewer_rheWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World models are increasingly improving in generating realistic videos from initial scene. However, they still lack accurate action following and two different action sequences from the same initial scene can generate similar plausible video in current robot world models. The authors try to mitigate this by projecting joint trajectories into the camera view to guide action-conditioned motion, carrying a recurrent consequence state across video chunks, and using same-reset branch supervision to favor executed actions over alternatives from the same reset. In simulation the model showed high correlation with policy success rates while maintaining video fidelity comparable to its baseline over long horizon rollouts. Further, transferring its components to Ctrl-World also improved agreement with real-world policy outcomes.

Critical strengths
  1. ActionTrace achieves high Pearson correlation with simulator success rates and recovers the correct policy ordering. In the real-world transfer, its components raise Pearson correlation over the matched Ctrl-World continuation.
  2. The paper compares performance across different world models, and rollouts by different policies and on different environments - real as well as sim.
  3. Model ablations are informative (Table3-5) to study the impact of different components.
Critical weaknesses
  1. Since camera-aligned kinematic grounding is a contribution on this work, the authors should run baselines from the two cited papers in related works (Wang et al., 2025; Wu & Gao, 2026).
  2. The commanded actions might diverge from the actual states. The paper does not study how this divergence might affect performance and if the generated video follows the commanded action trajectory or deviates.
  3. More qualitative examples are needed to assess the correctness and quality of generation. Even though the ActionTrace trajectory in Figure 4b matches real world outcome, the generation diverges from real world (sim in this case). It would be interesting to see more success/failure cases and closed loop rollouts.
Questions for authors
  1. Why does the world model condition on language instructions which can bias the generation?
  2. Table 10 and 13 show the model is more pessimistic about the outcome while maintaining policy rankings. What remedies can be applied to mitigate this new issue.
1 comment

Thank you for your thoughtful review and constructive suggestions for strengthening the evaluation. We have added an OSCAR comparison and expanded the intervention evaluation. Manuscript revisions are highlighted in blue.

W1: “The authors should run baselines from the two cited papers.”

Given the availability of released models and the rebuttal timeline, we focused on OSCAR and evaluated its released model under our four-policy protocol.

World modelPearson ↑Spearman ↑MAE (pp) ↓RMSE (pp) ↓
OSCAR0.84940.800011.4513.61
X-WAM (+ data)0.91840.80009.5010.71
ActionTrace0.98051.00006.907.60

ActionTrace achieves higher correlation and lower success-rate errors than OSCAR, while recovering the complete simulator ordering. OSCAR reverses the two strongest policies. This comparison is now included in Table 1, with rollout details in Appendix B.2.

W2: “The commanded actions might diverge from the actual states.”

We agree that commanded motion should not be equated with the realized physical state. The kinematic raster is a conditioning signal, not a hard constraint on the output trajectory. Generation also depends on the current observation and the consequence state, allowing predictions to reflect preceding interactions rather than simply reproduce the commands.

W3: “More qualitative examples are needed.”

Your distinction between correct object outcomes and faithful video generation is helpful. We expanded the intervention evaluation to 30 independent resets across six tasks and separated correct, incorrect, and visually indeterminate outcomes. On this expanded set, ActionTrace improves paired-outcome accuracy over X-WAM (+ data), with relative gains of 50.0% at step 64 (30.0% versus 20.0%) and 48.5% at step 128 (27.2% versus 18.3%). It also reduces clear errors from 70 to 50 at step 128, with similar indeterminate proportions. Section 4.2 and Appendix B.1 now report these results and the outcome breakdown.

Q1: “Why does the world model condition on language instructions which can bias the generation?”

We share your concern that a task prompt can bias generation toward a plausible successful continuation. We retain language to preserve the conditioning interface of the pretrained backbone and because it supplies important task semantics. For example, the current image and motion commands alone may not specify the target object or intended goal. Our aim is to keep this guidance while making predictions more responsive to the actions actually supplied. ActionTrace therefore strengthens spatial and temporal action conditioning and uses same-reset comparisons in which the instruction stays fixed but the actions differ. This trains the model to distinguish action-dependent outcomes within the same task.

Q2: “What remedies can be applied” to pessimistic success-rate estimates?

Thank you for highlighting this point. It identifies an important distinction between selecting policies reliably and estimating their absolute success probabilities. Our paper focuses on the former: preserving action-dependent performance differences so that imagined rollouts recover the relative strengths of policies. A dedicated treatment of absolute calibration goes beyond this study's central objective, but your observation points to a valuable next step toward practical use. For instance, a strictly increasing calibration map fitted on independent held-out rollouts could adjust success-rate estimates while preserving policy ordering. Assessing the success judge's own bias on simulator videos could further inform this calibration.

We hope the additional comparison and clarifications strengthen your confidence in the paper. Thank you again for your constructive feedback; we would be happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.