- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes World-ISI, an iterative self-improvement framework for world-action models (WAMs) that jointly updates action generation, world prediction, and value estimation using self-collected interaction data. The authors introduce World-Expectation Optimization (WEO), which weights successful action trajectories based on the difference between observed returns and the model's predicted values, while using failed trajectories for world and value supervision. The framework is evaluated on 24 RoboCasa tasks and real-world WidowX and ALOHA robots. Results show that World-ISI improves the average RoboCasa success rate from 66.7% to 70.1% over four iterations, along with improvements in future prediction quality and real-world manipulation performance.
Critical strengths
-
Promising self-improvement framework. I appreciate the idea of jointly updating action, world, and value predictions using self-collected experience, especially the use of failed trajectories for prediction supervision.
-
Well-structured comparison of optimization strategies. The authors compare WEO with SFT using the same fixed dataset and training budget, achieving 69.0% vs. 67.5% success at 6k steps. This helps distinguish the effect of the proposed optimization from additional data collection.
-
Encouraging empirical results. World-ISI improves RoboCasa success from 66.7% to 70.1%, with additional improvements in world prediction metrics and real-world robot experiments.
Critical weaknesses
-
Limited evaluation of self-improvement under evolving task distributions. Although World-ISI demonstrates consistent improvements across iterations and includes some OOD evaluations, the iterative training remains focused on the same 24 RoboCasa tasks. It is unclear whether the framework can continually adapt to new tasks, changing visual conditions, or environment dynamics while retaining previously learned capabilities. A stronger continual learning evaluation would make the self-improvement claim more convincing.
-
Insufficient evidence for the contribution of world and value prediction. The auxiliary-loss ablations also change training sampling ratios (80/10/10 vs. 100/0/0), making it difficult to isolate the benefits of prediction supervision. Moreover, value prediction accuracy and calibration are not directly evaluated, despite their central role in WEO. The paper therefore provides limited evidence of how improved world/value modeling contributes to policy improvement.
-
Confounding between expectation weighting and action-loss scaling. Under binary rewards, WEO assigns weights ≥ 1 to all successful samples, increasing the overall action-loss contribution relative to world/value losses. Thus, its improvement over SFT may partly result from stronger action supervision rather than expectation-aware weighting. A mean-weight-matched SFT baseline is needed to distinguish these effects.
Questions for authors
-
Self-improvement and generalization. Can the authors provide stronger evidence of self-improvement under evolving task distributions, such as introducing new tasks, visual variations, or changing dynamics across iterations? How well does the model retain previously learned capabilities?
-
Effectiveness of world and value prediction. Figure 5 shows that adding world and value losses improves policy success, but it remains unclear how these prediction capabilities contribute to the improvement. Can the authors provide more direct evidence, especially the accuracy or calibration of value predictions across iterations? A controlled ablation with matched policy-sample exposure would also help clarify the contribution of auxiliary prediction supervision.
-
Effectiveness of WEO weighting. Can the authors compare WEO with a constant-weight SFT baseline matched to WEO's average action-loss weight? This would help clarify whether the improvement comes from expectation-aware weighting or simply stronger action supervision.
I am open to raising my rating if the authors can convincingly address these concerns, particularly regarding the contributions of world/value prediction and the effectiveness of WEO.
6 comments
Authors' Response to Reviewer GWji (2)
Q3:
We thank the reviewer for suggesting this control. WEO does not normalize its action weights to have unit mean. We trained a constant-weight SFT baseline using the same fixed dataset, sampling procedure, and a 3,000-step training budget as WEO. We assigned every successful action sample a weight of 3.7176, the estimated mean action weight under WEO. Results are shown in the table below.
| Method | Constant-weight SFT | WEO |
|---|---|---|
| Success rate | 66.1% | 67.4% |
WEO outperformed the constant-weight baseline by 1.3 percentage points, suggesting that expectation-aware weighting contributes to policy improvement even when the average action weight is matched. We have incorporated these results into Appendix A.5 of our manuscript.
Thanks for the clarification and additional experiments. I am satisfied with the responses to Q1 and Q3. For Q2, the matched policy-sample comparison also helps address my concern about sampling.
My remaining concern is about value prediction. The lower training MSE shows better fitting of the training targets, but it does not show whether the values used by WEO are accurate on new rollouts. Value training uses observed futures, while WEO uses predicted futures during collection. Could you evaluate these pre-execution value predictions against actual outcomes on separate test rollouts across iterations?
Authors' Response to Reviewer GWji
We thank the reviewer for their thoughtful evaluation and constructive feedback. We appreciate the recognition of our self-improvement framework, the effectiveness of WEO compared to SFT, and the encouraging results across simulation and real-world robotic tasks. We address the reviewer's questions as follows.
Q1:
Generalization under distribution shifts. In RoboCasa, we evaluate on held-out object instances across five kitchen scenes per task, including two with visual styles absent from training (Sec. 4.1). On WidowX, we collect demonstrations and iterative rollouts only in the standard training configurations and reserve the variations for testing. For the hotpot task, we test with different ingredients or rearrange familiar ones. For the brick task, we vary the bricks, the bowls, or the brick placements. After two iterations, performance improves in both hotpot settings and with different bricks and bowls, while performance on unfamiliar brick placements returns close to its initial level. On ALOHA, we test vegetable transfer by swapping the cucumber and eggplant positions or replacing the cucumber with a carrot. The cucumber–eggplant completion score rises from 39.6 to 66.2, while the carrot–eggplant score remains at 59.6 (Sec. 4.5). These experiments show where improvements transfer to held-out conditions. We acknowledge that continual adaptation to new tasks or changing dynamics during training remains an important direction for future research, and we are committed to further investigating these challenges in our future work.
Retention of previously learned capabilities. We additionally evaluated the base policy and all four World-ISI checkpoints on the same 240 in-domain RoboCasa initial conditions across the 24 tasks, keeping object instances and initial conditions fixed across checkpoints. Every iteration exceeded the base success rate by 2.5–6.3 percentage points. We have incorporated these results into Appendix A.2 (Table 5) of our manuscript.
Q2:
We respectfully clarify that when both the world-model and value losses are disabled, only policy samples have an active loss. Continuing to draw world-model or value samples would therefore use part of the batch for samples that contribute zero gradient. We use 100/0/0 sampling so that every sample contributes to the action objective. At the same 3,000-step training budget, this gives the policy-only baseline 25% more policy samples than WEO receives with 80/10/10 sampling (2.40M versus 1.92M). We have added this explanation and the loss breakdown to Appendix A.5 (Table 7).