- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when visual future prediction actually helps World Action Models (WAMs). The authors show that future conditioning has only a small effect on average performance, but can cause large drops on a few precision-sensitive tasks. They call this future-induced negative transfer. To address this, the paper proposes Utility-WAM: a lightweight residual adapter corrects harmful future-induced action changes, and a gate decides when future generation is worth the extra computation. Experiments on LIBERO, RoboTwin 2.0, and real-world bimanual tasks show improved success rates while reducing unnecessary video generation.
Critical strengths
-
Clear and useful problem formulation. The comparison between future-conditioned and future-free paths under the same checkpoint is well controlled, and the paper goes beyond benchmark averages to analyze task-level failures. This makes the negative-transfer observation quite convincing within the evaluated model.
-
The method is simple and practical. The adapter is lightweight, keeps the base WAM frozen, and the ablations clearly separate the effects of alignment, error regularization, and routing. The real-world experiments are also a nice addition.
Critical weaknesses
-
The main negative-transfer result is still based on one shared Stage-1 checkpoint. The strongest failures on RoboTwin are also heavily concentrated on two tasks. The FAST-WAM comparisons help, but those models are trained separately, so they are not a fully controlled replication. It would be useful to see whether the same failure pattern appears across multiple independently trained checkpoints.
-
The gate supervision could be better justified. The gate is trained from an action-MSE-based utility label, while the actual goal is closed-loop task success. I would like to see more evidence that this proxy reliably predicts when future generation is actually useful.
Questions for authors
-
How stable is the negative-transfer pattern across independently trained Stage-1 checkpoints? In particular, do Press Stapler and Place Can Basket remain the most strongly affected tasks?
-
How well does the gate's action-error utility correlate with closed-loop success? Reporting AUROC/AUPRC or some correlation analysis would help clarify this.
-
Could you compare against a gate trained on the original uncorrected future-conditioned branch? This would help isolate whether the adapter-derived supervision is actually important for routing.
2 comments
Thank you for your answer. My concerns have been resolved, and I will keep my original score.
Thank you for the helpful review. Revisions incorporated in the manuscript are marked in red. The new checkpoint and retraining results below are reported only in this response.
1. Weakness 1: Single-checkpoint evidence and task concentration.
We evaluated an additional independently trained Stage-1 checkpoint. Within each checkpoint, both branches share weights, with adapter and gate disabled. Changes below are future-conditioned minus future-free success, in percentage points.
Both checkpoints reproduce small aggregate effects and severe losses on Press Stapler and Place Can Basket, still ranked first and second. This controlled replication complements the FAST-WAM comparisons.
2. Weakness 2: Justification of action-MSE-based gate supervision.
Section 3.3 explains why we use paired action-error utility: a query's episode-level effect is difficult to observe offline because subsequent routing and execution also affect success. The adapter is the core component correcting future-induced Δa; the gate uses whether its corrected action reduces expert-action MSE relative to the future-free reference (Eq. 5).
Table 3 reports that gating raises LIBERO success from 98.6% to 99.0% and RoboTwin success from 92.7% to 93.7%, while reducing Neff from 10 to 8.70 and 2.45, respectively. We also retrained the adapter and gate with two additional seeds, keeping Stage-1 fixed. Across three training seeds (mean ± sample standard deviation, %):
Selective improves over Correction in every seed on both benchmarks. Separately, Fig. 2 shows better success–compute curves than random routing and static truncation at intermediate budgets.
On Place Can Basket, gating increases the deficit from 14 to 17 points (Table 10). Chunk-averaged MSE may not fully reflect precise final placement, potentially explaining this decrease. Appendix C.3 discusses the task-level proxy gap. Supervising the gate with episode-level rollout outcomes remains future work (Section 6).
3. Question 1: Negative-transfer stability across independent Stage-1 checkpoints.
Aggregate success rates of the two independently trained Stage-1 checkpoints differ by less than 1 percentage point; Press Stapler and Place Can Basket remain the two most harmed tasks in the same order (W1).
4. Question 2: Discrimination and closed-loop evidence.
AP exceeds prevalence on both benchmarks, supporting offline utility-label ranking.
Appendix B, Table 7 reports offline discrimination using existing gate scores on cached validation/model-selection inputs, without retraining. Each branch's cached action MSE is averaged over two paired inference draws before applying y=1[uact>0] (Eq. 5; Section 3.3):
This cached-label diagnostic is reported separately from threshold calibration and uses no closed-loop-return labels. In closed-loop deployment, gating improves success while reducing video computation (W2). A single Δa's effect depends on later routing and execution; random routing adds stochastic branch choices. Supervising the gate with episode-level rollout outcomes remains future work (W2).
5. Question 3: Comparison with a gate trained on the uncorrected branch.
Adapter-only achieves higher success than gate-only on both benchmarks. The full framework further improves success while reducing computation relative to adapter-only.
Gate-only is included in Table 3 and Fig. 2; Appendix C.8 (Tables 14–15) reports aggregate success/computation and the two severe-task results. We train the same gate architecture using utility labels from the uncorrected future-conditioned and future-free branches, keeping Stage-1 frozen and omitting the adapter during label generation, training, and inference.
This supports the combined design of correction and its corresponding gate supervision; the different branches and budgets prevent isolating label-source effects.
We hope these clarifications address your concerns. We welcome further discussion.