acceptodds
Under review as a conference paper at ICLR 2027

Posterior Gradient Estimation Changes the Capabilities of Intrinsically Driven Agents

Abstract

Changing only posterior gradient estimation in world models with discrete latent states can lead intrinsically driven agents to acquire markedly different capabilities. We investigate this effect by changing how gradients are estimated through posterior samples across intrinsic-drive methods. Experiments across methods and tasks reveal substantial changes in performance, with both the magnitude and direction depending on the drive and task. Policies with similar aggregate scores can also acquire markedly different skill profiles. These findings show that posterior gradient estimation affects both performance and the skills acquired under intrinsic rewards.

Primary area:
reinforcement learning
Keywords:
Intrinsic Drives, World Models, Straight-through Gradient Estimation, Model-based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wpH3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper study how the choice of posterior gradient estimation function influences the learnt world model performance. Authors utilized DeamerV3 as backbone and studied different backward estimation functions like ST, ST-GS, and GST across different intrinsic reward methods. Downstream performance on Crafter environment validates that different backward estimators could lead to big different in overall performance and learned skills despite same backbone and data.

Critical strengths
  1. This paper studied a previously under explored problem of world model training: the choice of posterior gradient estimator during backward learning, which is interesting and worthy studying.
  2. The empirical results shows big differences in terms of performance and skills on subtasks across different backward estimators, highlighting the small choices can lead to significant downstream agent behaviors.
Critical weaknesses
  1. This paper conduct experiments use only one world model backbone Dreamer v3. More empirical experiments on extra world model architecture could further strengthen the generalization of this findings.
  2. The paper only presents empirical observations of this problem but provides little analysis on the mechanism of the problem. While the problem is interesting, a deeper investigation into how different estimator leads to different downstream performance may solidify and strengthen the whole study.
Questions for authors
  1. It would be super exciting to scale the model and the environment to larger and more complex scale to observe whether the findings still exist. This would provide powerful insights for the world model researcher in real world complex settings.
  2. A more detailed analysis of the learning dynamics under different posterior gradient estimators would help explain the observed differences. Whether different models's skills diverge at the earlier or latter stages of the training? Would training for more steps leads to more divergence or convergence?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wpH3 (Part 1)

Thank you for recognizing the value of this question. To address mechanism and learning dynamics, we extended the fixed-experience control and added training-stage score and skill statistics. To address the single-backbone concern, we added early-stage evaluations in another world model.

1. When Skills Diverge and Whether They Converge

We compare Crafter scores and skills across training windows. Both tables use training-episode statistics and report differences (ST-GS minus ST); positive values indicate higher values under ST-GS. ± is the sample SD of paired differences across training runs. The first table reports score differences.

Training windowHDRNDP2E
0–5M+0.04 ± 0.24+0.38 ± 1.00−0.27 ± 0.96
20–30M+16.77 ± 1.91+2.88 ± 2.06−3.99 ± 2.13
40–50M+23.37 ± 0.94+3.48 ± 2.28−0.46 ± 2.46

HD's gap grows, RND retains a positive mean gap in the later half, and P2E's gap first widens then narrows. The second table gives selected skill success-rate differences in percentage points (pp).

MethodSkillWindow (M)Difference ± SD (pp)
HDWood sword5–10+20.91 ± 23.50
RNDStone sword15–20+0.00 ± 0.99
RNDStone sword20–25+20.45 ± 10.01
RNDStone sword25–30+60.35 ± 5.93
P2EStone sword15–20−21.74 ± 33.01
P2EWood sword20–25+8.32 ± 32.56
P2EWood sword25–30+23.61 ± 30.18
P2EStone sword40–50−49.40 ± 47.85
P2EWood sword40–50+40.55 ± 39.61

The timing and form of divergence depend on the drive. HD's mean wood-sword gap appears early, RND's stone-sword gap mainly grows mid-training, and P2E's tool skills change in opposite directions despite similar late-stage scores. Similar overall scores therefore do not imply similar skill profiles. The methods show no shared pattern of narrowing gaps within this budget.

2. Mechanism

We test model-learning effects without new exploration data. Online training data change with policies; the original Appendix C.2, Table C1 tests RND/P2E action rankings under fixed experience. New HD/RND controls fix experience and reward functions at early and late stages to test whether local prediction differences arise without new interaction data.

Branches share initial model and optimizer states, data, rewards, and random numbers. Only the posterior gradient estimator differs during 50 joint model, policy, and value updates. The table reports MAE of 15-step intrinsic-return predictions on held-out trajectories, following recorded actions (lower is better). Training estimator identifies how the starting checkpoint was trained.

MethodCheckpoint stageTraining estimatorInitial MAEAfter ST updatesAfter ST-GS updates
HD5MST0.129650.091950.13392
HD5MST-GS0.784370.119690.12043
HD50MST0.302660.159680.15467
HD50MST-GS0.0838250.139810.068385
RND5MST0.018020.021210.01986
RND5MST-GS0.0121020.0179560.015451
RND50MST0.0017170.0023100.001562
RND50MST-GS0.0023460.0025580.002432

After these updates, ST gives lower MAE for HD from 5M checkpoints, whereas ST-GS gives lower MAE from 50M checkpoints. For RND, ST-GS gives lower MAE at both stages. These differences arise without newly collected experience.

The original Figure 4 (§5.3) also tests sample–surrogate correspondence. Independent noise preserves both marginal distributions at fixed logits but changes their pairing, and online performance changes under this intervention. Together with the fixed-experience control, this identifies local prediction-learning differences without new interaction data and effects of sample–surrogate correspondence on online capabilities. The current results do not fully explain how these local learning differences lead to long-term skill differences. We will investigate this link in future work.

3. Other World-Model Backbones

To address the single-backbone concern, we compared posterior gradient estimators for HD and RND in EMERALD (Burchi and Timofte, 2025), a world model with Transformer dynamics. The table gives official Crafter scores at a common 1M budget, using 256 evaluation episodes per policy.

DriveSTST-GSGST
HD1.8252.4331.400
RND1.7511.3861.313

ST-GS scores highest for HD and ST for RND at this common early stage. We will report longer-budget evaluations in a follow-up to test whether these early differences persist.

4. Model and Environment Scale

Larger models and more complex environments remain untested.

References

  • Burchi, M., and Timofte, R. Accurate and Efficient World Modeling with Masked Latent Transformers. ICML, 2025.
AuthorsOpenReview ↗

Response to Reviewer wpH3 (Part 2)

Following up on our previous response about additional world-model backbones, we report scores and skills for HD and RND in EMERALD, a world model with Transformer dynamics. Within each method, all three estimators use the same number of training steps.

DriveTraining stepsMetricSTST-GSGST
HD8MCrafter score2.6122.2285.138
HD8MWood-pickaxe success rate (%)1.171.1721.88
HD8MStone-collection success rate (%)0.000.394.69
RND5MCrafter score3.95510.2883.316
RND5MWood-pickaxe success rate (%)7.8183.208.59
RND5MStone-collection success rate (%)2.3469.140.78

At these training stages, the Transformer world model also shows differences in tool-crafting and resource-gathering skills. Thank you for suggesting an additional backbone. This extends the original observations to another world model.

Thank you to the authors for the additional analyses and experiments with more backbones, which I found very interesting! I encourage the authors to explore these findings further in future work. Given these improvements, I will raise my score from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer wpH3

Thank you for your positive assessment and constructive feedback. Your suggestions have helped us strengthen the evidence and make the study more complete. We will incorporate the additional analyses and more complete experimental results into the next version of the paper, and continue investigating these findings in greater depth in future work. We sincerely appreciate your thoughtful review and valuable contributions to improving the quality of our work.

Reviewer_afKcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper benchmarks different gradient estimators for categorical distributions in Dreamer's discrete recurrent state-space model, evaluating their effects on optimization behavior and final performance across different intrinsic-reward methods.

Critical strengths
  • The paper is clearly written and well-scoped.
  • The baselines are extensive, and homeostatic drive-based (HD) method is interesting for open-ended task like crafter.
  • The experiments do illustrate the spectrum of combinations of gradient estimators for categorical and intrinsic reward modules.
Critical weaknesses
  • Lack of novelty. While it is valuable to provide insights into how gradient estimators for categorical distributions affect the performance of intrinsically motivated agents, it is unclear why and how these estimators lead to such effects.
  • The straight-through gradient estimators, whether using probabilities from the vanilla version or soft samples from Gumbel-Softmax, are still biased because they do not account for the gradient of the sampling process. Given this, why not compare the proposed method with Gumbel-Softmax in both the forward and backward passes and investigate a range of temperature values? Dreamer's latent state is not required to be one-hot and can therefore consist of soft samples.
  • I think other factors may also interact with the latent state. For example, the decoder may be indiscriminate of the latent states as long as reconstruction loss is low. Moreover, the intrinsic modules are learned on top of non-stationary latent states. The extent to which these factors affect the effectiveness of intrinsic rewards remains unclear.
  • The independent backward noise experiment is counterintuitive: a more biased gradient estimator does not lead to evident decrease in performance, suggesting that the gradients of the latent states are less decisive than those of other components of the world model. This defeats the idea of seeking a better gradient estimator for categorical latent states, as the resulting performance gains appear limited, particularly when only task rewards are used. Further investigation is needed to identify the key factors underlying the observed performance differences.
  • Although the experiments appear to have been conducted rigorously (e.g., using multiple random seeds and horizontal comparisons), their coverage is limited. Intrinsic motivation is intended to encourage exploration and reduce reliance on task-specific rewards. However, the Atari and DMC tasks selected in this study primarily involve dense rewards. Both suites include sparse-reward tasks that require extensive exploration. Evaluating these tasks could help clarify how latent state learning affects exploratory behavior. Although Crafter can be considered a sparse-reward benchmark, the overall experimental coverage remains limited.
  • The experiments provide little insight into why different gradient estimators lead to different performance or offer practical guidance on when each estimator is preferable. Moreover, it remains unclear why the performance gap is nearly zero when task rewards are used. An examination of the training curves for Crafter shows that the performance of different gradient estimators is largely indistinguishable, except for HD. This raises a fundamental question: if the ST performs adequately in most cases, why should practitioners switch to alternative estimators that may incur additional computational costs?
  • Minor: gradient from loss function to latent state should not be transposed, which otherwise leads to incorrect matrix multiplication with the softmax Jacobian.
Questions for authors

When you say "Intrinsic-drive methods train only on their own intrinsic rewards, with a separate task-reward control in Crafter", what does "a separate task-reward control" mean here, do you fine tune the intrinsic-reward methods once being trained with task rewards like P2E?

9 comments
AuthorsOpenReview ↗

Response to Reviewer afKc (Part 1)

Thank you for your suggestions. We added soft-forward, temperature-range, sparse-task, and timing comparisons. We also clarify how reward modules depend on representations, what independent noise tests, and how the task-reward control is trained.

1. Contribution and Practical Relevance

Our contribution is evidence that posterior gradient estimation changes the relative performance of intrinsic-drive methods and the skills learned within each. RND and P2E change ranking across estimators in the paper's Table 1; the paper's Figures 2–3 show different tool skills despite similar scores. Reward-design comparisons therefore need to control posterior gradient estimation, while implementation choices require checking both scores and desired skills.

Prior work studies ST-GS variance reduction (Paulus et al., 2021), GST (Fan et al., 2022), and discrete-action estimators (Tilbury et al., 2023). We add evidence on their consequences for long-term intrinsic-drive capabilities and evaluation.

2. Do the Decoder and Changing Latents Explain Performance Differences?

Changing latents directly changes P2E's reward inputs. HD's fixed physiological reward and RND's image reward generator do not read world-model latents (Appendix D.2), so this direct pathway cannot explain the effects across all three drives. HD/RND still use model representations for reward prediction and policy learning, where changing latents may affect learning. The independent contribution of decoder supervision has not been tested.

In the HD/RND fixed-experience control, branches share initial model and optimizer states, data, rewards, and random numbers. Only the posterior gradient estimator differs during 50 joint model, policy, and value updates. The table reports MAE of 15-step intrinsic-return predictions on held-out trajectories, following recorded actions (lower is better). Training estimator identifies how the starting checkpoint was trained.

MethodCheckpoint stageTraining estimatorInitial MAEAfter ST updatesAfter ST-GS updates
HD5MST0.129650.091950.13392
HD5MST-GS0.784370.119690.12043
HD50MST0.302660.159680.15467
HD50MST-GS0.0838250.139810.068385
RND5MST0.018020.021210.01986
RND5MST-GS0.0121020.0179560.015451
RND50MST0.0017170.0023100.001562
RND50MST-GS0.0023460.0025580.002432

After the updates, the lower-MAE estimator for HD depends on the checkpoint stage; ST-GS gives lower MAE for RND at both stages. These differences arise without new exploration data.

3. Independent Noise and Bias

Using independent backward noise does not necessarily increase gradient bias. At fixed logits, it preserves the sample and surrogate marginal distributions but changes their pairing and potentially the mean gradient. Whether bias increases or decreases requires comparison with the true gradient (Appendix A.2).

In Figure 4 (§5.3) of the revised manuscript, independent backward noise lowers HD's and RND's mean scores and raises E3B's relative to standard ST-GS. These changes show that sample–surrogate pairing affects behavior learning, with the direction of the effect depending on the intrinsic drive.

4. Sparse Exploration and Coverage

Following your sparse-exploration suggestion, we added visual Cartpole Swingup Sparse under DreamerV3. RND trains only on intrinsic rewards; sparse task rewards are used only for evaluation. The table compares ST and GST after 5M environment steps (including resets), with 256 episodes per frozen policy. ± is SD across independent training runs. Reach rate is the fraction of episodes receiving positive task reward.

MetricSTGST
Task return23.02 ± 31.010.06 ± 0.09
Reward-region reach rate (%)50.98 ± 68.221.37 ± 1.93

At 5M, ST has higher mean return and reach rate, alongside greater training variation. This tests how posterior gradient estimation affects discovery and visitation of task-relevant regions. We will extend method coverage with RND/ST-GS and ICM ST/ST-GS/GST results in a follow-up under the same evaluation protocol.

References

  • Paulus, M. B., et al. Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator. ICLR, 2021.
  • Fan, T.-H., et al. Training Discrete Deep Generative Models via Gapped Straight-Through Estimator. ICML, 2022.
  • Tilbury, C. R., et al. Revisiting the Gumbel-Softmax in MADDPG. ALA Workshop at AAMAS, 2023.
AuthorsOpenReview ↗

Response to Reviewer afKc (Part 2)

5. Soft Forward Passes and Temperature

A Crafter control replaces observation-posterior and imagined-prior samples with Gumbel-Softmax soft states, retaining architecture, latent dimensions, and categorical KL. Entries compare frozen policies with hard-forward ST-GS at temperature 1 and 20M; ± is paired-difference SD.

MethodHard-forward scoreSoft-forward scoreScore difference (soft minus hard)
HD19.24210.261−8.982 ± 9.729
RND17.84925.437+7.587 ± 2.250
P2E13.46426.539+13.075 ± 1.369

Training with soft states raises mean scores for RND and P2E and lowers HD's. The next table compares P2E's soft-forward temperatures at a common 10M budget.

TemperatureScoreScore difference (this temperature minus 1)
0.56.199−12.047 ± 2.415
118.247—
211.675−6.571 ± 4.017

P2E's soft-state performance also depends on temperature, with 1 best among the tested values. To test backward-temperature effects with hard forward sampling unchanged, we vary only ST-GS backward temperature at 20M, using the first table's hard-forward baselines.

MethodScore at backward temperature 1Score at backward temperature 0.5Score difference (0.5 minus 1), mean ± SD
HD19.2425.775−13.468 ± 2.298
P2E13.46413.358−0.106 ± 5.429

Lowering backward temperature alone sharply reduces HD's mean score, while P2E's means remain close. Thus, backward settings can change performance even with hard states. Soft states also change model inputs and P2E's reward inputs; their gains do not establish more accurate gradients for the original discrete objective.

HD's matched longitudinal evaluations give the following paired score differences (mean ± SD).

HD contrastDifference at 20MDifference at 50M
Backward temperature 0.5 minus 1 (hard forward)−13.468 ± 2.298−2.827 ± 1.953
Soft minus hard forward (temperature 1)−8.982 ± 9.729−4.899 ± 2.242

At 50M, both modified settings have lower mean scores than the hard-forward, backward-temperature-1 baseline, but the gaps are smaller than at 20M. We will report longer-budget P2E comparisons and update its soft-temperature comparison at 20M in a follow-up.

6. Performance Differences and Computational Cost

On whether ST is adequate, non-HD methods also show practically meaningful score changes. These final scores from paper Table 1 include the task-reward control; entries are mean ± SD.

MethodSTST-GSGST
RND20.02 ± 1.6824.57 ± 2.1024.40 ± 1.20
P2E21.90 ± 2.5322.68 ± 2.0831.21 ± 3.45
E3B18.49 ± 0.3320.56 ± 3.8322.76 ± 3.24
ICM13.72 ± 5.1321.68 ± 0.1020.61 ± 1.63
Task reward70.63 ± 1.9473.24 ± 0.9569.98 ± 0.87

ST-GS improves RND/ICM and GST gives P2E larger gains, providing practical benefits beyond HD. Estimator choice also changes the RND/P2E ranking and hence the evaluation of reward designs. We have not yet identified why task-reward effects are smaller.

Estimators add no parameters. Warmed-up full-update timings use the same machine, HD model, and data; ± is the SD of timing-round means.

EstimatorFull update (ms)
ST688.89 ± 12.10
ST-GS689.63 ± 7.50
GST692.03 ± 2.91

Full-update times for ST-GS and GST are close to those for ST under the same timing conditions.

7. Which Matrix Is Transposed in the Gradient Formula?

We transpose the prediction-head Jacobian, using column gradients. For a head F:RK→RMF:\mathbb R^K\to\mathbb R^M, JFJ_F is M×KM\times K, and c=JF⊤∇y^Lc=J_F^\top\nabla_{\hat y}L is the loss gradient with respect to the latent state. At temperature 1, the posterior-logit gradient estimate is J(p)⊤cJ(p)^\top c, where J(p)J(p) is the softmax Jacobian at surrogate probabilities pp (paper §§3.1–3.2).

8. Is the Task-Reward Control Fine-Tuned from an Intrinsic-Drive Agent?

The paper's Crafter task-reward control is a separate agent trained from scratch using only achievement rewards. Intrinsic-drive agents are trained separately on their own intrinsic rewards and receive no subsequent task-reward fine-tuning (paper §4, Appendix D.2).

AuthorsOpenReview ↗

Response to Reviewer afKc (Part 3)

Following up on our previous responses on soft forward passes, temperature ranges, and sparse exploration, we report longer-training comparisons of forward relaxation and backward temperature, the 20M soft-forward temperature comparison, and the added ICM control.

1. Soft Forward Passes and Backward Temperature with Longer Training

The soft-forward experiment replaces both observation-posterior and imagination-prior samples with Gumbel-Softmax soft states. The table extends HD/P2E frozen-policy evaluations to a common 50M budget. The hard-forward baseline uses ST-GS with backward temperature 1, and the soft-forward temperature is also 1. Results summarize paired training runs common to these conditions. Their aggregation scope differs from that of Table 1 in the paper. Score-column ± values are sample SDs across training runs. Difference-column ± values are sample SDs of paired differences.

MethodHard-forward scoreSoft-forward scoreScore difference (soft minus hard)
HD28.671 ± 1.23923.771 ± 1.003−4.899 ± 2.242
P2E22.233 ± 1.63231.097 ± 1.020+8.864 ± 0.612

At the same training budget, soft-forward training raises P2E's score but lowers HD's. Soft states are therefore not a uniformly better alternative for the two intrinsic drives tested.

The next table compares score differences at 20M and 50M from the same P2E training runs, using the same frozen-policy evaluation protocol. The baseline is hard-forward ST-GS with backward temperature 1. Entries are mean ± sample SD of paired score differences.

P2E contrastScore difference at 20MScore difference at 50M
Soft minus hard forward (temperature 1)+13.075 ± 1.369+8.864 ± 0.612
Backward temperature 0.5 minus 1 (hard forward)−0.106 ± 5.429−2.113 ± 4.543

P2E's soft-forward gain shrinks with training but remains at 50M. Lowering only the backward temperature with hard forward sampling does not produce the same mean-score gain.

2. Soft-Forward Temperature Range at 20M

We extend P2E's three-temperature soft-forward comparison to a common 20M budget, using the same frozen-policy evaluation protocol.

Soft-forward temperatureScore, mean ± SDScore difference (this temperature minus 1), mean ± SD of paired differences
0.510.783 ± 2.041−15.756 ± 1.577
126.539 ± 0.464—
212.515 ± 3.733−14.024 ± 3.269

Among the tested temperatures, the intermediate setting gives P2E a higher mean score than either extreme. Performance does not improve monotonically with the degree of relaxation. Thank you for suggesting soft forward passes and a range of temperatures. These comparisons provide a more complete test of this continuous latent-state alternative.

3. Full Method Comparisons in Sparse Exploration

Following your sparse-task coverage suggestion, we report all three estimators for RND and ICM in Cartpole Swingup Sparse. Training uses only the corresponding intrinsic rewards, with sparse task rewards used only for evaluation. All conditions use the same 5M budget, with 256 evaluation episodes per frozen policy. ± is the sample SD across training runs. Reach rate is the fraction of episodes receiving positive task reward at least once.

MethodMetricSTST-GSGST
RNDTask return23.02 ± 31.019.37 ± 10.160.06 ± 0.09
RNDReward-region reach rate (%)50.98 ± 68.2254.88 ± 43.371.37 ± 1.93
ICMTask return14.87 ± 11.102.61 ± 0.0310.29 ± 10.86
ICMReward-region reach rate (%)99.61 ± 0.0086.91 ± 3.0488.09 ± 16.85

Under ST-GS, ICM still frequently reaches the reward region but accumulates less task reward, so the difference goes beyond whether the region is discovered. For RND, GST gives both lower reach rate and lower return.

Reviewer_4KtCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that changing only how gradients pass through discrete latent-state samples can largely change the capabilities. The authors focus on world modes using DreamerV3. They compare three existing gradient estimators, ST / ST-GS / GST, and keeping reward computation, architecture, and forward sampling rules fixed. Experiments show that performance changes depend on both the intrinsic reward method and the task, with no consistently good estimator. Alsp, there is an additional experiment, showing that the correspondence between forward sampling noise and the backward one affects learning. The main contribution is an empirical demonstration that a seemingly minor implementation choice can shape both overall performance and which skills agents acquire, though the mechanisms behind these effects seem to remain unresolved.

Critical strengths
  1. The authors keep the reward computation, model architecture, and forward sampling rule fixed within each method. They only change how gradients pass through the discrete posterior samples. That makes the comparison fairly convincing, the performance differences come from changing this part of training, even though the resulting models and experiences gradually become different.

  2. The differences are large enough to matter in practice.

  3. The skill analysis is informative. I find this finding interesting, that two agents can look equally capable according to the benchmark score while having quite different abilities.

Critical weaknesses
  1. The authors show that the effect exists, but do not really explain the full chain. Does the estimator change which information the world model represents? Does it improve reward prediction? Or does an early behavioral difference lead the agent to collect different data, which then changes everything else? The experiments do not separate these possibilities. Training on the same fixed dataset would help isolate model-learning effects from exploration feedback.

  2. Better task performance does not necessarily mean better optimization of the intrinsic objective. These agents train on intrinsic rewards, but much of the evaluation uses task scores. An agent could score higher because its behavior happens to match the benchmark better, without becoming better at maximizing its own training objective. The paper acknowledges this issue, but reporting intrinsic returns alongside task performance would make the interpretation stronger.

  3. The scope is still fairly narrow. All experiments use DreamerV3, and the detailed skill analysis comes from Crafter. The additional environments provide useful evidence, but mainly cover RND and P2E.

  4. The study uses five training seeds, but the skill plots and independent-noise figure do not show error bars.

Questions for authors

Please see the details above.

  1. Are there more explanations about mechanisms?

  2. How do you decide the estimator settings? Seems to use temperature 1 and GST gap 1.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4KtC (Part 1)

Thank you for recognizing the controlled comparisons and skill analysis. We extended the original fixed-data control, added intrinsic-return evaluations, temperature tests, and generalization tests, and added error bars to Figures 3–4.

1. Mechanism and Fixed-Data Control

To test model-learning effects without new exploration data, we extend the original RND/P2E action-ranking control (Appendix C.2, Table C1). The new HD/RND controls fix experience and reward functions at early and late stages to test how posterior gradient estimators affect return-prediction learning.

Branches share initial model and optimizer states, data, rewards, and random numbers. Only the posterior gradient estimator differs during 50 joint model, policy, and value updates. The table reports MAE of 15-step intrinsic-return predictions on held-out trajectories, following recorded actions (lower is better). Training estimator identifies how the starting checkpoint was trained.

MethodCheckpoint stageTraining estimatorInitial MAEAfter ST updatesAfter ST-GS updates
HD5MST0.129650.091950.13392
HD5MST-GS0.784370.119690.12043
HD50MST0.302660.159680.15467
HD50MST-GS0.0838250.139810.068385
RND5MST0.018020.021210.01986
RND5MST-GS0.0121020.0179560.015451
RND50MST0.0017170.0023100.001562
RND50MST-GS0.0023460.0025580.002432

After these updates, ST gives lower MAE for HD from 5M checkpoints, whereas ST-GS gives lower MAE from 50M checkpoints. For RND, ST-GS gives lower MAE at both stages. These differences arise without newly collected experience.

This control identifies short-horizon prediction-learning differences on fixed data. The current results do not fully explain how these local learning differences lead to long-term skill differences. We will investigate this link in future work.

2. Intrinsic Returns and Task Performance

To test whether Crafter-score gains accompany higher intrinsic returns, we evaluate frozen 50M policies. Each frozen ST- or ST-GS-trained RND/P2E reward module scores both policies' trajectories, using the same discount factor as in training. Entries are paired differences (ST-GS minus ST); positive values indicate higher values under ST-GS. ± is the sample SD of paired differences.

MethodCrafter-score differenceDiscounted intrinsic-return difference: ST rewardDiscounted intrinsic-return difference: ST-GS reward
HD+21.18 ± 3.48+0.19010 ± 0.00200 (fixed physiological reward)Same
RND+4.94 ± 5.25+0.02527 ± 0.02174+0.02112 ± 0.02356
P2E+0.34 ± 4.41+0.08267 ± 0.11979+0.02999 ± 0.09932

Note: Paired evaluation uses a different checkpoint set from paper Table 1.

HD and RND have higher mean Crafter scores and discounted intrinsic returns under ST-GS than ST. RND's mean return difference is positive under both frozen reward functions. P2E's score and discounted-return differences change sign across runs.

The next table distinguishes discounted gains from undiscounted reward totals on the same HD trajectories.

HD return statisticDifference (ST-GS minus ST), mean ± SD
Discounted+0.19010 ± 0.00200
Undiscounted−0.28561 ± 0.04832

HD's ST-GS endpoint policy has higher discounted but lower undiscounted return than ST. HD rewards are successive drive reductions, so their undiscounted sum reflects the initial-to-final drive difference; discounting retains temporal weights throughout the episode.

AuthorsOpenReview ↗

Response to Reviewer 4KtC (Part 2)

3. Backbone and Task Coverage

We compared posterior gradient estimators for HD and RND in EMERALD (Burchi and Timofte, 2025), a world model with Transformer dynamics. The table gives official Crafter scores at a common 1M budget, using 256 evaluation episodes per policy.

DriveSTST-GSGST
HD1.8252.4331.400
RND1.7511.3861.313

ST-GS scores highest for HD and ST for RND at this common early stage. We will report longer-budget evaluations in a follow-up to test whether these early differences persist.

To broaden task coverage, we added visual Cartpole Swingup Sparse under DreamerV3. RND trains only on intrinsic rewards; sparse task rewards are used only for evaluation. The table compares ST and GST after 5M environment steps (including resets), with 256 episodes per frozen policy. ± is SD across independent training runs. Reach rate is the fraction of episodes receiving positive task reward.

MetricSTGST
Task return23.02 ± 31.010.06 ± 0.09
Reward-region reach rate (%)50.98 ± 68.221.37 ± 1.93

At 5M, ST has higher mean return and reach rate, alongside greater training variation. This adds evidence on how posterior gradient estimation affects discovery and visitation of task-relevant regions. We will extend method coverage with RND/ST-GS and ICM ST/ST-GS/GST results in a follow-up under the same evaluation protocol.

4. Error Bars in the Skill and Independent-Noise Figures

We have revised the manuscript. Figures 3 (§5.2) and 4 (§5.3) now include standard-deviation error bars. With error bars, the differences in HD's wood-pickaxe and stone-sword success and RND's stone-sword success remain clearly visible in Figure 3. P2E has similar overall scores, yet higher mean wood-sword success and lower stone-sword success under ST-GS than ST, illustrating different skill profiles.

In Figure 4, independent backward noise lowers HD's and RND's mean scores and raises E3B's relative to standard ST-GS. The effect of the correspondence between sampling and the backward surrogate therefore depends on the intrinsic-drive method.

5. Temperature and GST Gap

We use temperature 1 and GST gap 1 as shared reference settings across methods, with the gap following Fan et al. (2022). This tests backward-rule effects without method-specific hyperparameter selection. The added Crafter control keeps hard forward sampling unchanged and changes only ST-GS backward temperature to 0.5. Entries are frozen-policy evaluations at 20M; ± is the sample SD of paired differences.

MethodScore at temperature 1Score at temperature 0.5Score difference (temperature 0.5 minus 1)
HD19.2425.775−13.468 ± 2.298
P2E13.46413.358−0.106 ± 5.429

At 20M, lowering backward temperature substantially reduces HD's mean score, while P2E's means are similar. Temperature sensitivity thus depends on the drive.

HD's longitudinal comparison also includes Gumbel-Softmax soft-forward states at temperature 1. Entries are paired score differences; ± is their sample SD.

HD contrastDifference at 20MDifference at 50M
Backward temperature 0.5 minus 1 (hard forward)−13.468 ± 2.298−2.827 ± 1.953
Soft minus hard forward (temperature 1)−8.982 ± 9.729−4.899 ± 2.242

At 50M, both modified settings have lower mean scores than the hard-forward, backward-temperature-1 baseline, but the gaps are smaller than at 20M. We will report the longer-budget P2E comparisons in a follow-up.

References

  • Burchi, M., and Timofte, R. Accurate and Efficient World Modeling with Masked Latent Transformers. ICML, 2025.
  • Fan, T.-H., et al. Training Discrete Deep Generative Models via Gapped Straight-Through Estimator. ICML, 2022.
AuthorsOpenReview ↗

Response to Reviewer 4KtC (Part 3)

Following up on our previous responses, we report 50M backward-temperature results, the full method comparisons in the sparse task, and longer-budget evaluations in EMERALD.

1. Backward Temperature at a Longer Training Budget

We extend the HD/P2E ST-GS backward-temperature comparison to frozen-policy evaluations at a common 50M budget, keeping hard forward sampling unchanged. The table summarizes paired training runs common to these conditions. Its aggregation scope differs from that of Table 1 in the paper. Scores are mean ± sample SD across training runs. Differences are reported as the mean ± sample SD of paired differences.

MethodScore at backward temperature 1Score at backward temperature 0.5Score difference (temperature 0.5 minus 1)
HD28.671 ± 1.23925.844 ± 0.714−2.827 ± 1.953
P2E22.233 ± 1.63220.119 ± 2.911−2.113 ± 4.543

At 50M, the lower backward temperature does not yield a higher mean final score for either method. HD's gap is smaller than at 20M, showing that the magnitude of the early temperature effect changes over training.

2. Method Coverage in a Sparse Task

We now report the full ST/ST-GS/GST comparisons for RND and ICM in Cartpole Swingup Sparse. Each method trains only on its own intrinsic rewards, with sparse task rewards used only for evaluation. The table follows the previous 5M budget and 256-episode evaluation protocol for each frozen policy. ± is the sample SD across training runs. Reach rate is the fraction of episodes receiving positive task reward at least once.

MethodMetricSTST-GSGST
RNDTask return23.02 ± 31.019.37 ± 10.160.06 ± 0.09
RNDReward-region reach rate (%)50.98 ± 68.2254.88 ± 43.371.37 ± 1.93
ICMTask return14.87 ± 11.102.61 ± 0.0310.29 ± 10.86
ICMReward-region reach rate (%)99.61 ± 0.0086.91 ± 3.0488.09 ± 16.85

In the added ICM comparison, ST-GS gives a lower mean task return, while GST gives both lower reach rate and lower return for RND. The two drives respond differently to estimator choice in the same sparse task. Thank you for highlighting the limited method coverage. The added ICM comparison provides evidence from another type of intrinsic drive in an additional task.

3. Scores and Skills in EMERALD

To address the backbone-coverage concern, we add results from EMERALD, a world model with Transformer dynamics. Within each method, all three estimators use the same number of training steps. The table reports official Crafter scores and success rates for two skills.

DriveTraining stepsMetricSTST-GSGST
HD8MCrafter score2.6122.2285.138
HD8MWood-pickaxe success rate (%)1.171.1721.88
HD8MStone-collection success rate (%)0.000.394.69
RND5MCrafter score3.95510.2883.316
RND5MWood-pickaxe success rate (%)7.8183.208.59
RND5MStone-collection success rate (%)2.3469.140.78

At these training stages, differences among EMERALD configurations appear in both scores and tool-crafting and resource-gathering skills, adding world-model evidence beyond DreamerV3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Discussion for Paper 31384

Dear Area Chair,

We would like to briefly highlight the paper’s contribution and the main developments during the discussion.

Our study shows that, with the intrinsic reward design fixed within each method, changing posterior gradient estimation can alter the relative performance of intrinsic-drive methods and produce different skill profiles even when aggregate scores are similar.

Reviewer 4KtC found the controlled comparisons fairly convincing, the effects practically meaningful, and the skill analysis informative, and maintained their positive assessment after the additional experiments. Reviewer wpH3 recognized the value of this underexplored question and raised their score following the additional analyses and backbone experiments.

Following the reviewers’ suggestions, we added fixed-experience controls, intrinsic-return evaluations, temperature and soft-forward comparisons, and experiments with a Transformer-based world model and a sparse-reward task. These additions broaden the evaluation and strengthen the empirical evidence.

Regarding Reviewer afKc’s questions about the contribution, our latest responses (Parts 4–7) further clarify that this is an empirical analysis using existing estimators as controlled interventions, not a proposal for a universally better replacement for ST. The different responses across intrinsic-drive methods and the distinct skill profiles at similar aggregate scores are central findings of the study.

These controlled findings provide an empirical basis for comparing intrinsic drives and the capabilities they produce. They also raise concrete questions for theoretical and mechanistic research into how local differences in posterior gradient estimation affect long-term capability development and why these effects vary across intrinsic drives. Thank you for your consideration.

Best regards,

Authors of Paper 31384

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.