- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Direct Experience World-Model Optimization (DEWO), a method for improving world-action models using deployment experience. It revisits states along failed rollouts to collect successful and failed continuations. Successful examples supervise both video and action prediction, while failed examples supervise video prediction without action imitation. The method also introduces success-conditioned action guidance, activated by a value head when progress stalls. Experiments cover simulated dexterous manipulation and real-world grasp-and-place tasks.
Critical strengths
s1. The paper addresses a meaningful question: how to learn from failed robot executions without imitating their unsuccessful actions. Using their observed visual outcomes as training targets is a sensible approach.
s2. The experiments cover multiple WAM formulations, five simulated dexterous tasks, and two real-world hand platforms. This provides useful evidence about the method’s applicability across different models and hardware.
s3. The paper includes helpful ablations on successful and failed visual supervision, experience selection, and guidance timing. Its detailed spatial evaluation also makes the limits of the real-world improvements visible.
Critical weaknesses
w1. The practical cost of collecting the proposed data is unclear. Real-world collection requires human inspection of failures, restoration of comparable scene and robot configurations, and repeated attempts from these configurations. This may require substantial human effort, particularly for dexterous manipulation. The paper does not report enough information to assess whether this procedure is more efficient than collecting additional demonstrations or human corrections.
w2. The evidence that better visual prediction leads to better action generation is limited. The ablations show that adding failed visual futures improves both prediction loss and task success, which is useful evidence. However, these improvements alone do not establish that better visual representations explain the control gains. It also remains unclear whether collecting more successful experience under the same collection budget would provide similar benefits.
w3. The method combines many components, but its main source of improvement remains unclear. These include replay exploration, critical-fragment selection, outcome labels, visual supervision, a value head, and selective CFG. Although the paper provides component ablations, the connection between its main claim about world-model learning and its reliance on inference-time guidance needs stronger evidence. On the two guidance-ablation tasks, DEWO without CFG performs slightly worse than the initial model on one task and slightly better on the other. The added complexity therefore needs clearer justification against simpler alternatives.
Questions for authors
q1. How much human time and how many robot attempts are required to collect the retained successful and failed continuations? Under the same collection budget, how does DEWO compare with collecting additional successful demonstrations or human corrections?
q2. Could the authors isolate the contribution of failed-video training by keeping successful-action data and inference settings fixed, including a comparison with CFG disabled? How does adding failed videos compare with adding more successful videos under a matched budget?
q3. Which components are necessary to obtain the reported gains? In particular, how much improvement comes from training on the selected experience, and how much comes from selective CFG? How was the CFG scale of 0.2 selected, and how sensitive are the results to this choice?
4 comments
Response to Reviewer zF9Y (2/3)
W2&Q2. Isolating the contribution of failed visual supervision
The original ablation retains successful-action supervision while adding visual targets. Adding failed futures to successful visual supervision raises success from 88.7% to 90.7% on Water Plant and 74.0% to 77.3% on Fold Glasses. Held-out prediction loss also decreases. These results support improved predictive learning alongside improved control, but do not by themselves establish that a particular change in visual representations causes the control gains.
Matched successful-video and failed-video controls with CFG disabled. We additionally evaluated OpenWAM, providing a complementary backbone with additional large-scale pretraining. The two SFT controls share the same common training data, initialization, successful-action supervision, visual-supervision quantity, and training-update budget. They differ only in whether the additional visual targets come from successful or failed trajectories. These additional videos introduce no extra action targets, and both SFT controls are evaluated without CFG under the same inference settings. Full DEWO is included as a separate reference for the complete method.
Table 2. Success rates (%, ↑) with OpenWAM; 50 trials per task and condition.
| Training variant | Inference | Fold Glasses | Pick Bucket | Pinch Tongs | Water Plant | Four-task avg. |
|---|---|---|---|---|---|---|
| SFT + additional successful videos | No CFG | 76 | 94 | 84 | 88 | 85.5 |
| SFT + additional failed videos | No CFG | 80 | 94 | 94 | 74 | 85.5 |
| DEWO | Value-gated CFG | 84 | 96 | 94 | 92 | 91.5 |
With CFG disabled, additional failed videos improve Fold Glasses and Pinch Tongs relative to additional successful videos, match Pick Bucket, and reduce Water Plant. Both controls average 85.5%. Thus, failed visual experience is useful on some tasks, but does not uniformly outperform an equal amount of successful visual experience.
Full DEWO reaches 91.5%. This six-point advantage reflects the combined experience-selection, learning, and guidance approach, rather than failed visual supervision alone. The matched budget here refers to training-data volume and updates. We will make the CFG-disabled control setting explicit and clarify the scope of the mechanism claims in the revised manuscript.
Response to Reviewer zF9Y (3/3)
W3&Q3. Contributions of experience learning and selective guidance
The components follow one design logic: learning from ITP experience and selectively using that learning during execution. Successful and failed continuations supervise physical consequences; successful actions make feasible behavior accessible through a success condition. The same replay outcomes supervise progress estimation. The value gate determines when to intervene, while the learned success condition supplies the action direction.
Training and inference contributions. The original guidance ablation evaluates the same adapted FastWAM checkpoint under different inference settings.
Table 3. Success rates (%); three 50-scene sets per task and condition. Initial-policy results are included for reference.
| Task | Initial policy | DEWO without CFG | Continuous CFG | Value-gated CFG |
|---|---|---|---|---|
| Water Plant | 88.7 | 86.7 | 78.0 | 90.7 |
| Fold Glasses | 72.0 | 73.3 | 52.0 | 77.3 |
Unguided performance remains close to the initial policy. Selective guidance adds 4.0 percentage points on both tasks relative to unguided DEWO, while continuous guidance reduces performance. These results show that the control benefit depends on how the learned success condition is applied. The visual-supervision comparisons in W2&Q2 address the contribution of predictive learning under controlled action data and inference settings.
Experience selection and simpler alternatives. Figure 4c compares critical experience with full trajectories at matched data sizes and training budgets: success increases from 89.3% to 90.7% on Water Plant and 74.0% to 77.3% on Fold Glasses. In an additional OpenWAM comparison on Fold Glasses, DEWO with ITP anchors achieves 84.0%, versus 80.0% when anchors are placed two seconds before GPT-annotated visible failures, under the same replay-budget limit and training budget.
We also compare simpler learning and intervention choices on Fold Glasses. SFT on ITP experience and unguided DEWO both achieve 80.0%. With the same DEWO model and value gate, an equal-magnitude random perturbation achieves 80.0%, compared with 84.0% for success-conditioned CFG. This supports the learned direction beyond an arbitrary intervention.
Table 4. Guidance timing on Fold Glasses; success rates (%), 50 trials per condition, and a single-trigger budget for all guided variants.
| No guidance | Random gating | Fixed-schedule gating | Progress-based gating |
|---|---|---|---|
| 80.0 | 80.0 | 82.0 | 84.0 |
Together, these comparisons support experience selection, the learned success direction, and selective activation against simpler alternatives.
CFG-scale selection and sensitivity. We empirically selected w = 0.2 using collection scenes whose spatial configurations differ from the evaluation scenes. The three 50-scene evaluation sets were not used for hyperparameter selection. The same scale is used across all five FastWAM simulation tasks.
We evaluated additional CFG scales on Fold Glasses using OpenWAM, holding the trained checkpoint, value-gating rule, and evaluation protocol fixed.
Table 5. CFG-scale sensitivity on Fold Glasses using OpenWAM; 50 trials per condition.
| CFG scale w | Success rate (%) |
|---|---|
| 0 | 80.0 |
| 0.1 | 80.0 |
| 0.2 | 84.0 |
| 0.3 | 84.0 |
We will include more results and the parameter-selection protocol in the revised appendix.
Thanks for the additional experiments. The video comparisons and CFG ablations addressed my main concerns. If time permits, I’d appreciate a brief discussion of how DEWO’s recovery trials compare with additional demonstrations or human corrections under the same collection budget. I’m raising my score from 2 to 3.
Response to Reviewer zF9Y (1/3)
We appreciate the reviewer's concern regarding interaction cost and human effort. DEWO integrates collection with deployment-time capability assessment: the same recovery trials assess the policy's capabilities and provide successful and failed continuations for adaptation.
Simulation collection budget. Task-stage cues help localize candidate replay regions. Each window allows up to 20 anchors with ten continuations each, and querying stops at the first zero-success anchor. Across 116 trajectories with identifiable ITPs, the selected windows average 5.28 queried anchors (median: 3), corresponding to 52.8 continuation attempts per window, compared with the maximum of 200.
Human effort on physical robots. Our grasping sequences are short, with recognizable alignment, contact, and stabilization stages. Reviewing these stages to diagnose failures also identifies candidate replay anchors, allowing anchor selection to share the existing inspection workflow. Scene restoration is quick in this tabletop setup, and the policy generates four continuations per anchor without additional expert demonstrations.
We will report human inspection and restoration time, together with robot interaction costs, in the revised appendix.
Learning from the same deployment experience. We have also completed a complementary control comparing SFT and DEWO on the same newly collected experience, using Tennis Ball with the Sharpa hand. Within each backbone, both methods start from the same initial checkpoint. Evaluation uses a 3 × 3 grid with ten trials per cell. R0 is the initial policy; R1 and R2 follow one and two adaptation rounds.
Table 1. OpenWAM: grasping success, % (successes/90).
The original real-world backbone shows a similar pattern under its complete grasp-and-place metric: from 13.3% (12/90) initially, SFT reaches 15.6% (14/90) and 8.9% (8/90), while DEWO reaches 23.3% (21/90) after each adaptation round.
SFT improves after the first round but declines after the second. In the OpenWAM control, this decline accompanies less stable grasping. Our interpretation is that repeated SFT may overfit the limited deployment data and disturb the pretrained action distribution. These results support DEWO's use of the same collected experience; they do not measure the relative acquisition efficiency of recovery trials and additional demonstrations or human corrections.