- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
FLASH studies whether visual model-based reinforcement learning can benefit from previously collected experience. It selects anchors to improves downstream policy learning through episodic retrieval of past experiences. FLASH use these retrieved groups and randomly picked episodes to construct an imagination batch for actor-critic learning. Experiment on Atari 100k shows marginal improvement over two baselines (i.e., DreamerV3 and Drama).
Critical strengths
- The idea of explicitly retrieving of structurally historical contexts for downstream policy is well-motivated.
- The paper provides extensive experimental details. The authors include evaluation protocol, robustness analysis, and computational overhead in the supplementary.
Critical weaknesses
The proposed mechanism is not closely connected to the motivation.
Lines 185-186 observe that “selected task-relevant information remains decodable from lightweight latent representations under clean observations”. However, Section 3 does not demonstrate that the baseline policy or critic fails to exploit those variables. A method motivated by this finding should provide evidence that its intervention helps the policy or critic make better use of the identified task-relevant information.
FLASH is more naturally interpreted as a prioritization-based modification of imagination-context sampling. FLASH instead selects anchors using TD-error magnitude modulated by temporal value differences, retrieves hash-matched historical contexts, and incorporates these contexts into the imagination batch with group-normalized weights. Although it differs from conventional prioritized experience replay in where and how the selected experience enters training, it remains unclear why this particular design follows from the decodability findings. Changing the training-context distribution could plausibly improve downstream learning, but the paper does not adequately distinguish improved utilization of task-relevant latent information from a more general benefit of emphasizing informative training contexts.
The empirical evidence does not yet establish statistically reliable improvements.
Although the evaluation includes two world-model backbones, the aggregate improvements are modest. Table 2 does not report confidence intervals for the performance differences. Therefore, the reported results are insufficient to determine whether the improvements are reliable under training randomness.
The score-combination ablation raises an additional concern about sensitivity to the salience formulation and its configuration. In Table 21, the additive variant achieves a mean HNS of 0.984. This is numerically below the STORM baseline reported in Table 2. The paper notes that the ablations and the main evaluation use different seed sets, but the substantial drop warrants a clearer investigation of when the proposed sampling mechanism helps and when it can degrade performance.
Questions for authors
- How statistically reliable are the full-benchmark improvements? The improvements reported in Table 2 are generally modest. Could the authors provide confidence intervals or bootstrap intervals for the aggregate performance differences, including mean HNS, median HNS, IQM, and optimality gap? In addition, it would be helpful if all ablation studies used the same set of training seeds. Currently, the use of different seed subsets across ablations makes it difficult to directly compare the magnitude of different design choices.
- Could the authors clarify the discrepancy between the originally reported baseline results and the reproduced baselines used in this paper? For example, the original STORM paper reports a mean HNS of approximately 1.267, whereas the reproduced STORM baseline in Table 2 achieves 1.105. Similarly, the original DRAMA results are reported to be around 1.05 mean HNS, while the reproduced DRAMA baseline in this paper is 0.948. Could the authors explain the source of these discrepancies?
- What are the concrete advantages of FLASH over existing plug-and-play mechanisms [1-4]? In particular, it would be useful to discuss whether the advantage lies in sample efficiency, computational overhead, or compatibility with different world-model architectures. A direct empirical comparison with representative plug-and-play alternatives would further strengthen this claim.
- Could the authors provide a motivation that is more directly connected to the proposed mechanism? Would it be more appropriate to motivate FLASH by analyzing deficiencies in the imagination-context distribution—for example, whether uniform or frequency-based context sampling under-represents informative, rare, or outcome-critical situations? Such an analysis would establish a more direct logical connection between the identified problem and the proposed retrieval-based sampling mechanism.
- Could the authors clarify the notation used in Figure 4? From the methodology and appendix, it appears that denotes the replay-buffer position. Could the authors explicitly define this notation in the caption to avoid ambiguity?
References
[1] Morihira, N., Nahar, A., Bharadwaj, K., Kato, Y., Hayashi, A. and Harada, T., 2026. R2-Dreamer: Redundancy-reduced world models without decoders or augmentation. In International Conference on Learning Representations.
[2] Peng, Z.H., Li, S., Li, Z., Ruan, S., Liu, Y. and He, Y., 2026. From Observations to Events: Event-Aware World Model for Reinforcement Learning. In International Conference on Learning Representations.
[3] Zhang, W., Jelley, A., McInroe, T., Storkey, A. and Wang, G., 2026. Object-centric world models from few-shot annotations for sample-efficient reinforcement learning. In International Conference on Learning Representations.
[4] Ma, H., Wu, J., Feng, N., Xiao, C., Li, D., Hao, J., Wang, J. and Long, M., 2024. Harmonydream: Task harmonization inside world models. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:33983-34007.
8 comments
[2/5] W1 — Is the mechanism connected to the motivation? (W1, Q4)
W1:The proposed mechanism is not closely connected to the motivation. [Lines 185-186] observe that "selected task-relevant information remains decodable from lightweight latent representations under clean observations". However, [Section 3] does not demonstrate that the baseline policy or critic fails to exploit those variables. A method motivated by this finding should provide evidence that its intervention helps the policy or critic make better use of the identified task-relevant information. FLASH is more naturally interpreted as a prioritization-based modification of imagination-context sampling. FLASH instead selects anchors using TD-error magnitude modulated by temporal value differences, retrieves hash-matched historical contexts, and incorporates these contexts into the imagination batch with group-normalized weights. Although it differs from conventional prioritized experience replay in where and how the selected experience enters training, it remains unclear why this particular design follows from the decodability findings. Changing the training-context distribution could plausibly improve downstream learning, but the paper does not adequately distinguish improved utilization of task-relevant latent information from a more general benefit of emphasizing informative training contexts.
Q4:Could the authors provide a motivation that is more directly connected to the proposed mechanism? Would it be more appropriate to motivate FLASH by analyzing deficiencies in the imagination-context distribution—for example, whether uniform or frequency-based context sampling under-represents informative, rare, or outcome-critical situations? Such an analysis would establish a more direct logical connection between the identified problem and the proposed retrieval-based sampling mechanism.
We agree that decodability alone does not show that the baseline critic under-uses the information. To address this concern, we now provide that evidence. On the same variable our probes decode, FLASH's critic orders values by task progress more strongly than STORM's, and a control rules out the "emphasis only" explanation.
a) Does FLASH help the critic use the decoded variable? Igloo stage is decodable from STORM's latent (98.3% mean class accuracy on clean frames; [Sec. 3.3, Fig. 2(b)]). To test whether the critic uses it, we paste each of the 17 igloo stages into 17,421 fixed Frostbite contexts (all other pixels, actions, and latent-sampling seeds fixed) and measure the per-context rank correlation between stage and predicted value [Table R8, App. J.1]:
Table R8.
| Critic (one fixed checkpoint each) | Spearman ρ [95% CI] | Kendall τ_b |
|---|---|---|
| STORM | 0.076 [0.045, 0.110] | 0.040 |
| TD-error-only | 0.087 [0.045, 0.133] | 0.059 |
| Random-Neighbor | 0.108 [0.084, 0.132] | 0.052 |
| Full FLASH | 0.162 [0.127, 0.199] | 0.134 |
Although the latent encodes igloo stage, STORM's critic barely reflects it. FLASH doubles the alignment (+0.087, CI [0.051, 0.121]; [Table 14]), and removing either component lowers it (both paired-difference CIs exclude zero; [Table 16]). The STORM vs. FLASH comparison was already in [App. I] of the submission ([App. J] in the revised paper).
b) Utilization or general emphasis? The Random-Neighbor control separates the two explanations. It keeps the anchor-selection rule, group sizes, context allocation, and loss weights, so the emphasis on informative contexts is unchanged; only the hash-matched neighbors become random replay contexts. If the gain came from emphasis alone, this variant would match FLASH. It does not: mean HNS is 0.892 vs 1.163 [Table 30], and its critic alignment is 0.108 vs 0.162. The design follows from the finding: if the information is in the latent but the critic does not use it, the intervention should change what the critic trains on, not the encoder. FLASH therefore trains salient events together with similar past situations, so the critic sees how outcomes differ among similar states (e.g., scoring vs. non-scoring in Pong; [Fig. 4]).
c) Q4. Thank you for this suggestion; we agree that it gives a more direct motivation, and our Frostbite analysis provides one instance of it [App. E]. In 4 of 5 seed pairs, both STORM and STORM+FLASH collected successful igloo episodes before retrieval starts, from an identical shared warmup. With uniform sampling, STORM nevertheless ends lower in all four pairs: 2162.6 vs 2402.8 mean return, difference +240 [150, 330]. In the fifth pair, neither agent succeeded, and both agents achieve similarly low returns (254.5 vs 258.5). This pattern is what one expects if uniform sampling under-uses rare, outcome-critical experience that the agent already has. We have added this framing to the motivation in [Sec. 3].
[3/5] W2 — Statistical reliability and baseline reproduction (W2, Q1a, Q2)
W2:The empirical evidence does not yet establish statistically reliable improvements. Although the evaluation includes two world-model backbones, the aggregate improvements are modest. [Table 2] does not report confidence intervals for the performance differences. Therefore, the reported results are insufficient to determine whether the improvements are reliable under training randomness.
Q1a:How statistically reliable are the full-benchmark improvements? The improvements reported in [Table 2] are generally modest. Could the authors provide confidence intervals or bootstrap intervals for the aggregate performance differences, including mean HNS, median HNS, IQM, and optimality gap?
Q2:Could the authors clarify the discrepancy between the originally reported baseline results and the reproduced baselines used in this paper? For example, the original STORM paper reports a mean HNS of approximately 1.267, whereas the reproduced STORM baseline in [Table 2] achieves 1.105. Similarly, the original DRAMA results are reported to be around 1.05 mean HNS, while the reproduced DRAMA baseline in this paper is 0.948. Could the authors explain the source of these discrepancies?
We agree that the gains needed uncertainty estimates. The mean-HNS gain is statistically supported on all three agents. Some other metrics are not, and we report every interval. STORM and DRAMA now use 5 seeds per game and TWISTER uses 3, with per-seed scores [App. N]. We resample matched training-run pairs within each game (50,000 paired bootstrap replicates) and recompute each metric. Bold intervals exclude zero [Table R9]:
Table R9.
| FLASH − Baseline | STORM | DRAMA | TWISTER |
|---|---|---|---|
| Mean HNS ↑ | +0.0484 [+0.0020, +0.0941] | +0.0660 [+0.0063, +0.1335] | +0.0793 [+0.0106, +0.1569] |
| Median HNS ↑ | +0.0492 [+0.0021, +0.0935] | +0.0141 [−0.0532, +0.0806] | +0.1674 [−0.0187, +0.3638] |
| IQM ↑ | +0.0167 [−0.0047, +0.0375] | +0.0200 [−0.0169, +0.0576] | +0.0770 [+0.0351, +0.1286] |
| Optimality gap ↓ | −0.0071 [−0.0172, −0.0004] | −0.0091 [−0.0297, +0.0119] | −0.0170 [−0.0400, +0.0021] |
Q2 (baseline gap): Thank you for the question. Other studies also report lower reproduction scores for STORM and DRAMA, and every FLASH comparison uses a matched baseline.
- With 5 seeds, our baselines are now STORM 1.132 (previously 1.105) and DRAMA 0.975 (previously 0.948).
- Independent re-runs report similar numbers. For STORM, a team including STORM's first author obtained ≈1.14 with the released code [Ref. 5 - Sec. 4.1, Table 11], and GIT-STORM reports 0.947 [Ref. 6 - Table 5]. For DRAMA, EDELINE reports 0.989 [Ref. 7 - Table 1].
- Within each agent, baseline and FLASH share seeds, code, hyperparameters, budget, warmup, and evaluation, and they are bit-for-bit identical until retrieval starts [Sec. 5.1, App. C]. The FLASH effect is therefore measured against a same-condition baseline.
[1/5] Summary of our response
We thank Reviewer T3bp for the careful review, and for finding the idea of explicitly retrieving historical contexts for downstream policy learning well-motivated and the experimental details extensive.
The review raises three key concerns: the link between the decodability motivation and the mechanism (W1, Q4), statistical reliability and baseline reproduction (W2, Q1a, Q2), and the additive-variant result and ablation seeds (W3, Q1b). It also asks two questions, on related plug-and-play methods (Q3) and on notation (Q5). In response, we:
Revisions are marked in blue in the revised PDF.