- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates whether task improvements from world-model post-training depend on correct next-observation targets. It compares ground-truth targets, mismatched observations, and independent random rewards under GRPO on ALFWorld and ScienceWorld, with an additional self-distillation comparison on ALFWorld. Evaluation measures prediction accuracy, single-attempt success, and task coverage over 64 attempts. Mismatched targets substantially reduce prediction accuracy while retaining substantial task gains, and random-reward training also improves coverage. Behavioral analyses identify increased consideration of alternative actions and reduced looping after training. An extension to VisualWebArena demonstrates coverage improvements from random-reward training in a visual browsing environment.
Critical strengths
A well-designed audit of an important attribution problem.
The mismatched-target and random-reward controls directly test whether correct prediction supervision is necessary for task gains, while separate prediction evaluation checks that target corruption actually changes predictive performance. This provides a useful methodology for assessing what world-model post-training contributes to agent performance.
A consequential empirical finding supported across multiple settings.
The separation between prediction accuracy and task performance appears in both text environments and in the self-distillation experiment. The random-reward coverage improvement also extends to VisualWebArena. These results provide concrete evidence that task gains alone are insufficient to establish improved environment prediction as their source, an important consideration for interpreting and designing future agent-training methods.
Critical weaknesses
The broader conclusions about world-model learning are not sufficiently supported
The experiments show that substantial task gains can be retained without correct prediction targets, but it remains unclear whether this finding extends beyond the chosen training methods. Further comparisons across different ways of learning and using environment dynamics are needed to support broader conclusions about the contribution of prediction learning to agent performance.
Questions for authors
1. Does the same finding hold for directly supervised prediction training?
Why does the study not include a directly supervised next-state prediction baseline, such as Implicit World Modeling (IWM) [1]? With GT and MIS prediction targets followed by identical action fine-tuning, would MIS still retain most of the gains achieved by GT?
2. What motivates the implementation choices and training duration?
The original RWML collects ALFWorld trajectories using the model being trained, whereas this paper uses DeepSeek-V4-Flash. What motivates this change, and are the original easy-example subsampling and reward-text preprocessing procedures retained? The public RWML script uses entropy and KL coefficients of 0 and 0.001, respectively, whereas Table 6 uses 0.001 and 0.01. What motivates these adjustments?
COMAP combines ground-truth next-state supervision with distillation from an EMA-updated teacher, whereas this paper uses a fixed teacher and only the distillation loss. Why was this implementation chosen?
All conditions are trained for one epoch and evaluated at the final checkpoint. Why was this training duration selected? Do the relative performances of GT, MIS, and COIN persist when the training budget is extended equally across conditions?
3. How many independent training runs underlie Table 2?
The confidence intervals for the pass@k curves are obtained by resampling test tasks and do not capture training variability. How large are the GT–MIS differences relative to variation across training runs? Do the gains from COIN recur across different training seeds?
4. Could annotation errors affect the action-revision findings in Table 3?
Appendix C.2 reports that an independent review confirmed 40 of 60 steps labeled as action revision. Are the false positives concentrated in particular training conditions? Could these errors change the reported trend that post-training increases action revision?
Reference