- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper study how the choice of posterior gradient estimation function influences the learnt world model performance. Authors utilized DeamerV3 as backbone and studied different backward estimation functions like ST, ST-GS, and GST across different intrinsic reward methods. Downstream performance on Crafter environment validates that different backward estimators could lead to big different in overall performance and learned skills despite same backbone and data.
Critical strengths
- This paper studied a previously under explored problem of world model training: the choice of posterior gradient estimator during backward learning, which is interesting and worthy studying.
- The empirical results shows big differences in terms of performance and skills on subtasks across different backward estimators, highlighting the small choices can lead to significant downstream agent behaviors.
Critical weaknesses
- This paper conduct experiments use only one world model backbone Dreamer v3. More empirical experiments on extra world model architecture could further strengthen the generalization of this findings.
- The paper only presents empirical observations of this problem but provides little analysis on the mechanism of the problem. While the problem is interesting, a deeper investigation into how different estimator leads to different downstream performance may solidify and strengthen the whole study.
Questions for authors
- It would be super exciting to scale the model and the environment to larger and more complex scale to observe whether the findings still exist. This would provide powerful insights for the world model researcher in real world complex settings.
- A more detailed analysis of the learning dynamics under different posterior gradient estimators would help explain the observed differences. Whether different models's skills diverge at the earlier or latter stages of the training? Would training for more steps leads to more divergence or convergence?
4 comments
Response to Reviewer wpH3 (Part 2)
Following up on our previous response about additional world-model backbones, we report scores and skills for HD and RND in EMERALD, a world model with Transformer dynamics. Within each method, all three estimators use the same number of training steps.
| Drive | Training steps | Metric | ST | ST-GS | GST |
|---|---|---|---|---|---|
| HD | 8M | Crafter score | 2.612 | 2.228 | 5.138 |
| HD | 8M | Wood-pickaxe success rate (%) | 1.17 | 1.17 | 21.88 |
| HD | 8M | Stone-collection success rate (%) | 0.00 | 0.39 | 4.69 |
| RND | 5M | Crafter score | 3.955 | 10.288 | 3.316 |
| RND | 5M | Wood-pickaxe success rate (%) | 7.81 | 83.20 | 8.59 |
| RND | 5M | Stone-collection success rate (%) | 2.34 | 69.14 | 0.78 |
At these training stages, the Transformer world model also shows differences in tool-crafting and resource-gathering skills. Thank you for suggesting an additional backbone. This extends the original observations to another world model.
Thank you to the authors for the additional analyses and experiments with more backbones, which I found very interesting! I encourage the authors to explore these findings further in future work. Given these improvements, I will raise my score from 2 to 3.
Response to Reviewer wpH3
Thank you for your positive assessment and constructive feedback. Your suggestions have helped us strengthen the evidence and make the study more complete. We will incorporate the additional analyses and more complete experimental results into the next version of the paper, and continue investigating these findings in greater depth in future work. We sincerely appreciate your thoughtful review and valuable contributions to improving the quality of our work.
Response to Reviewer wpH3 (Part 1)
Thank you for recognizing the value of this question. To address mechanism and learning dynamics, we extended the fixed-experience control and added training-stage score and skill statistics. To address the single-backbone concern, we added early-stage evaluations in another world model.
1. When Skills Diverge and Whether They Converge
We compare Crafter scores and skills across training windows. Both tables use training-episode statistics and report differences (ST-GS minus ST); positive values indicate higher values under ST-GS. ± is the sample SD of paired differences across training runs. The first table reports score differences.
HD's gap grows, RND retains a positive mean gap in the later half, and P2E's gap first widens then narrows. The second table gives selected skill success-rate differences in percentage points (pp).
The timing and form of divergence depend on the drive. HD's mean wood-sword gap appears early, RND's stone-sword gap mainly grows mid-training, and P2E's tool skills change in opposite directions despite similar late-stage scores. Similar overall scores therefore do not imply similar skill profiles. The methods show no shared pattern of narrowing gaps within this budget.
2. Mechanism
We test model-learning effects without new exploration data. Online training data change with policies; the original Appendix C.2, Table C1 tests RND/P2E action rankings under fixed experience. New HD/RND controls fix experience and reward functions at early and late stages to test whether local prediction differences arise without new interaction data.
Branches share initial model and optimizer states, data, rewards, and random numbers. Only the posterior gradient estimator differs during 50 joint model, policy, and value updates. The table reports MAE of 15-step intrinsic-return predictions on held-out trajectories, following recorded actions (lower is better). Training estimator identifies how the starting checkpoint was trained.
After these updates, ST gives lower MAE for HD from 5M checkpoints, whereas ST-GS gives lower MAE from 50M checkpoints. For RND, ST-GS gives lower MAE at both stages. These differences arise without newly collected experience.
The original Figure 4 (§5.3) also tests sample–surrogate correspondence. Independent noise preserves both marginal distributions at fixed logits but changes their pairing, and online performance changes under this intervention. Together with the fixed-experience control, this identifies local prediction-learning differences without new interaction data and effects of sample–surrogate correspondence on online capabilities. The current results do not fully explain how these local learning differences lead to long-term skill differences. We will investigate this link in future work.
3. Other World-Model Backbones
To address the single-backbone concern, we compared posterior gradient estimators for HD and RND in EMERALD (Burchi and Timofte, 2025), a world model with Transformer dynamics. The table gives official Crafter scores at a common 1M budget, using 256 evaluation episodes per policy.
ST-GS scores highest for HD and ST for RND at this common early stage. We will report longer-budget evaluations in a follow-up to test whether these early differences persist.
4. Model and Environment Scale
Larger models and more complex environments remain untested.
References