acceptodds
Under review as a conference paper at ICLR 2027

FLASH: Improving Downstream Experience Utilization in Visual Model-Based Reinforcement Learning through Task-Relevant Episodic Retrieval

Abstract

Recent advances in visual model-based reinforcement learning have continually improved control performance through higher-fidelity generative world models and visual representations, often at additional computational cost. However, our probing experiments show that lightweight latent representations can retain substantial task-relevant information even when visual reconstruction is imperfect, suggesting that representation fidelity alone may not determine downstream control performance. These findings suggest that downstream utilization of task-relevant information already available in latent representations may itself be an important bottleneck. We therefore propose Fast Latent Anchoring via Salience and Hashing (FLASH), which uses salient learning events as retrieval anchors to identify structurally similar historical contexts through a hash-indexed episodic memory. FLASH jointly reuses these contexts in Actor-Critic learning, improving downstream experience utilization without replacing the underlying encoder or world-model architectures. Across Atari 100k, FLASH improves STORM's mean and median human-normalized scores by 4.71% and 8.37%, respectively, and DRAMA's by 8.33% and 5.35%. As a plug-and-play module that can be readily integrated into existing agents, FLASH achieves these performance gains with a 5.11–17.61% increase in wall-clock training time across the evaluated configurations.

Primary area:
reinforcement learning
Keywords:
Visual Reinforcement Learning, Model-Based Reinforcement Learning, Episodic Retrieval, Experience Utilization, Actor-Critic Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T3bpOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

FLASH studies whether visual model-based reinforcement learning can benefit from previously collected experience. It selects anchors to improves downstream policy learning through episodic retrieval of past experiences. FLASH use these retrieved groups and randomly picked episodes to construct an imagination batch for actor-critic learning. Experiment on Atari 100k shows marginal improvement over two baselines (i.e., DreamerV3 and Drama).

Critical strengths
  1. The idea of explicitly retrieving of structurally historical contexts for downstream policy is well-motivated.
  2. The paper provides extensive experimental details. The authors include evaluation protocol, robustness analysis, and computational overhead in the supplementary.
Critical weaknesses

The proposed mechanism is not closely connected to the motivation.

Lines 185-186 observe that “selected task-relevant information remains decodable from lightweight latent representations under clean observations”. However, Section 3 does not demonstrate that the baseline policy or critic fails to exploit those variables. A method motivated by this finding should provide evidence that its intervention helps the policy or critic make better use of the identified task-relevant information.

FLASH is more naturally interpreted as a prioritization-based modification of imagination-context sampling. FLASH instead selects anchors using TD-error magnitude modulated by temporal value differences, retrieves hash-matched historical contexts, and incorporates these contexts into the imagination batch with group-normalized weights. Although it differs from conventional prioritized experience replay in where and how the selected experience enters training, it remains unclear why this particular design follows from the decodability findings. Changing the training-context distribution could plausibly improve downstream learning, but the paper does not adequately distinguish improved utilization of task-relevant latent information from a more general benefit of emphasizing informative training contexts.

The empirical evidence does not yet establish statistically reliable improvements.

Although the evaluation includes two world-model backbones, the aggregate improvements are modest. Table 2 does not report confidence intervals for the performance differences. Therefore, the reported results are insufficient to determine whether the improvements are reliable under training randomness.

The score-combination ablation raises an additional concern about sensitivity to the salience formulation and its configuration. In Table 21, the additive variant achieves a mean HNS of 0.984. This is numerically below the STORM baseline reported in Table 2. The paper notes that the ablations and the main evaluation use different seed sets, but the substantial drop warrants a clearer investigation of when the proposed sampling mechanism helps and when it can degrade performance.

Questions for authors
  1. How statistically reliable are the full-benchmark improvements? The improvements reported in Table 2 are generally modest. Could the authors provide confidence intervals or bootstrap intervals for the aggregate performance differences, including mean HNS, median HNS, IQM, and optimality gap? In addition, it would be helpful if all ablation studies used the same set of training seeds. Currently, the use of different seed subsets across ablations makes it difficult to directly compare the magnitude of different design choices.
  2. Could the authors clarify the discrepancy between the originally reported baseline results and the reproduced baselines used in this paper? For example, the original STORM paper reports a mean HNS of approximately 1.267, whereas the reproduced STORM baseline in Table 2 achieves 1.105. Similarly, the original DRAMA results are reported to be around 1.05 mean HNS, while the reproduced DRAMA baseline in this paper is 0.948. Could the authors explain the source of these discrepancies?
  3. What are the concrete advantages of FLASH over existing plug-and-play mechanisms [1-4]? In particular, it would be useful to discuss whether the advantage lies in sample efficiency, computational overhead, or compatibility with different world-model architectures. A direct empirical comparison with representative plug-and-play alternatives would further strengthen this claim.
  4. Could the authors provide a motivation that is more directly connected to the proposed mechanism? Would it be more appropriate to motivate FLASH by analyzing deficiencies in the imagination-context distribution—for example, whether uniform or frequency-based context sampling under-represents informative, rare, or outcome-critical situations? Such an analysis would establish a more direct logical connection between the identified problem and the proposed retrieval-based sampling mechanism.
  5. Could the authors clarify the notation pp used in Figure 4? From the methodology and appendix, it appears that pp denotes the replay-buffer position. Could the authors explicitly define this notation in the caption to avoid ambiguity?

References

[1] Morihira, N., Nahar, A., Bharadwaj, K., Kato, Y., Hayashi, A. and Harada, T., 2026. R2-Dreamer: Redundancy-reduced world models without decoders or augmentation. In International Conference on Learning Representations.

[2] Peng, Z.H., Li, S., Li, Z., Ruan, S., Liu, Y. and He, Y., 2026. From Observations to Events: Event-Aware World Model for Reinforcement Learning. In International Conference on Learning Representations.

[3] Zhang, W., Jelley, A., McInroe, T., Storkey, A. and Wang, G., 2026. Object-centric world models from few-shot annotations for sample-efficient reinforcement learning. In International Conference on Learning Representations.

[4] Ma, H., Wu, J., Feng, N., Xiao, C., Li, D., Hao, J., Wang, J. and Long, M., 2024. Harmonydream: Task harmonization inside world models. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235:33983-34007.

8 comments
AuthorsOpenReview ↗

[1/5] Summary of our response

We thank Reviewer T3bp for the careful review, and for finding the idea of explicitly retrieving historical contexts for downstream policy learning well-motivated and the experimental details extensive.

The review raises three key concerns: the link between the decodability motivation and the mechanism (W1, Q4), statistical reliability and baseline reproduction (W2, Q1a, Q2), and the additive-variant result and ablation seeds (W3, Q1b). It also asks two questions, on related plug-and-play methods (Q3) and on notation (Q5). In response, we:

  1. Linked the motivation to the mechanism with a critic test on the same variable our probes decode (Frostbite igloo stage). FLASH's critic tracks igloo stage about twice as strongly as STORM's (Spearman 0.162 vs 0.076), and removing either FLASH component lowers it.
  2. Separated "using task-relevant information" from "emphasizing informative contexts" with a random-neighbor control that keeps the emphasis but removes relevance. Mean HNS drops from 1.163 to 0.892.
  3. Reported confidence intervals with 5 seeds for STORM and DRAMA plus a third agent (TWISTER). The mean-HNS CIs exclude zero on all three agents.
  4. Explained the baseline gap, the additive result (now frequency-matched), the seed subsets, and FLASH's relation to [Ref. 1–4], and defined pp in [Fig. 4].

Revisions are marked in blue in the revised PDF.

AuthorsOpenReview ↗

[2/5] W1 — Is the mechanism connected to the motivation? (W1, Q4)

W1: The proposed mechanism is not closely connected to the motivation. [Lines 185-186] observe that "selected task-relevant information remains decodable from lightweight latent representations under clean observations". However, [Section 3] does not demonstrate that the baseline policy or critic fails to exploit those variables. A method motivated by this finding should provide evidence that its intervention helps the policy or critic make better use of the identified task-relevant information. FLASH is more naturally interpreted as a prioritization-based modification of imagination-context sampling. FLASH instead selects anchors using TD-error magnitude modulated by temporal value differences, retrieves hash-matched historical contexts, and incorporates these contexts into the imagination batch with group-normalized weights. Although it differs from conventional prioritized experience replay in where and how the selected experience enters training, it remains unclear why this particular design follows from the decodability findings. Changing the training-context distribution could plausibly improve downstream learning, but the paper does not adequately distinguish improved utilization of task-relevant latent information from a more general benefit of emphasizing informative training contexts.

Q4: Could the authors provide a motivation that is more directly connected to the proposed mechanism? Would it be more appropriate to motivate FLASH by analyzing deficiencies in the imagination-context distribution—for example, whether uniform or frequency-based context sampling under-represents informative, rare, or outcome-critical situations? Such an analysis would establish a more direct logical connection between the identified problem and the proposed retrieval-based sampling mechanism.

We agree that decodability alone does not show that the baseline critic under-uses the information. To address this concern, we now provide that evidence. On the same variable our probes decode, FLASH's critic orders values by task progress more strongly than STORM's, and a control rules out the "emphasis only" explanation.

a) Does FLASH help the critic use the decoded variable? Igloo stage is decodable from STORM's latent (98.3% mean class accuracy on clean frames; [Sec. 3.3, Fig. 2(b)]). To test whether the critic uses it, we paste each of the 17 igloo stages into 17,421 fixed Frostbite contexts (all other pixels, actions, and latent-sampling seeds fixed) and measure the per-context rank correlation between stage and predicted value [Table R8, App. J.1]:

Table R8.

Critic (one fixed checkpoint each)Spearman ρ [95% CI]Kendall τ_b
STORM0.076 [0.045, 0.110]0.040
TD-error-only0.087 [0.045, 0.133]0.059
Random-Neighbor0.108 [0.084, 0.132]0.052
Full FLASH0.162 [0.127, 0.199]0.134

Although the latent encodes igloo stage, STORM's critic barely reflects it. FLASH doubles the alignment (+0.087, CI [0.051, 0.121]; [Table 14]), and removing either component lowers it (both paired-difference CIs exclude zero; [Table 16]). The STORM vs. FLASH comparison was already in [App. I] of the submission ([App. J] in the revised paper).

b) Utilization or general emphasis? The Random-Neighbor control separates the two explanations. It keeps the anchor-selection rule, group sizes, context allocation, and loss weights, so the emphasis on informative contexts is unchanged; only the hash-matched neighbors become random replay contexts. If the gain came from emphasis alone, this variant would match FLASH. It does not: mean HNS is 0.892 vs 1.163 [Table 30], and its critic alignment is 0.108 vs 0.162. The design follows from the finding: if the information is in the latent but the critic does not use it, the intervention should change what the critic trains on, not the encoder. FLASH therefore trains salient events together with similar past situations, so the critic sees how outcomes differ among similar states (e.g., scoring vs. non-scoring in Pong; [Fig. 4]).

c) Q4. Thank you for this suggestion; we agree that it gives a more direct motivation, and our Frostbite analysis provides one instance of it [App. E]. In 4 of 5 seed pairs, both STORM and STORM+FLASH collected successful igloo episodes before retrieval starts, from an identical shared warmup. With uniform sampling, STORM nevertheless ends lower in all four pairs: 2162.6 vs 2402.8 mean return, difference +240 [150, 330]. In the fifth pair, neither agent succeeded, and both agents achieve similarly low returns (254.5 vs 258.5). This pattern is what one expects if uniform sampling under-uses rare, outcome-critical experience that the agent already has. We have added this framing to the motivation in [Sec. 3].

AuthorsOpenReview ↗

[3/5] W2 — Statistical reliability and baseline reproduction (W2, Q1a, Q2)

W2: The empirical evidence does not yet establish statistically reliable improvements. Although the evaluation includes two world-model backbones, the aggregate improvements are modest. [Table 2] does not report confidence intervals for the performance differences. Therefore, the reported results are insufficient to determine whether the improvements are reliable under training randomness.

Q1a: How statistically reliable are the full-benchmark improvements? The improvements reported in [Table 2] are generally modest. Could the authors provide confidence intervals or bootstrap intervals for the aggregate performance differences, including mean HNS, median HNS, IQM, and optimality gap?

Q2: Could the authors clarify the discrepancy between the originally reported baseline results and the reproduced baselines used in this paper? For example, the original STORM paper reports a mean HNS of approximately 1.267, whereas the reproduced STORM baseline in [Table 2] achieves 1.105. Similarly, the original DRAMA results are reported to be around 1.05 mean HNS, while the reproduced DRAMA baseline in this paper is 0.948. Could the authors explain the source of these discrepancies?

We agree that the gains needed uncertainty estimates. The mean-HNS gain is statistically supported on all three agents. Some other metrics are not, and we report every interval. STORM and DRAMA now use 5 seeds per game and TWISTER uses 3, with per-seed scores [App. N]. We resample matched training-run pairs within each game (50,000 paired bootstrap replicates) and recompute each metric. Bold intervals exclude zero [Table R9]:

Table R9.

FLASH − BaselineSTORMDRAMATWISTER
Mean HNS ↑+0.0484 [+0.0020, +0.0941]+0.0660 [+0.0063, +0.1335]+0.0793 [+0.0106, +0.1569]
Median HNS ↑+0.0492 [+0.0021, +0.0935]+0.0141 [−0.0532, +0.0806]+0.1674 [−0.0187, +0.3638]
IQM ↑+0.0167 [−0.0047, +0.0375]+0.0200 [−0.0169, +0.0576]+0.0770 [+0.0351, +0.1286]
Optimality gap ↓−0.0071 [−0.0172, −0.0004]−0.0091 [−0.0297, +0.0119]−0.0170 [−0.0400, +0.0021]

Q2 (baseline gap): Thank you for the question. Other studies also report lower reproduction scores for STORM and DRAMA, and every FLASH comparison uses a matched baseline.

  • With 5 seeds, our baselines are now STORM 1.132 (previously 1.105) and DRAMA 0.975 (previously 0.948).
  • Independent re-runs report similar numbers. For STORM, a team including STORM's first author obtained ≈1.14 with the released code [Ref. 5 - Sec. 4.1, Table 11], and GIT-STORM reports 0.947 [Ref. 6 - Table 5]. For DRAMA, EDELINE reports 0.989 [Ref. 7 - Table 1].
  • Within each agent, baseline and FLASH share seeds, code, hyperparameters, budget, warmup, and evaluation, and they are bit-for-bit identical until retrieval starts [Sec. 5.1, App. C]. The FLASH effect is therefore measured against a same-condition baseline.
Reviewer_ccwGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates experience utilization in visual model-based reinforcement learning: even when lightweight latent representations preserve task-relevant information, downstream policy and value networks may fail to exploit it effectively. The authors propose FLASH, which combines normalized TD-error and temporal value differences to identify salient events as anchors. It retrieves historical contexts using PCA-based binary hashing and replaces a subset of randomly sampled contexts with retrieved contexts to initialize Actor-Critic imagination. The method preserves the underlying encoder, world-model, and Actor-Critic architectures, while using candidate re-verification and index rebuilding to address representation drift. Experiments cover 26 Atari 100k games and two base agents, STORM and DRAMA. The authors report relative improvements in mean human-normalized score of 4.71% and 8.33%, respectively, with approximately 5.11%–17.61% additional training time. Probing experiments, case studies, and component ablations provide supplementary validation.

Critical strengths
  1. The intervention is clearly defined and potentially useful in practice. FLASH directly changes the contexts used to initialize Actor-Critic imagination without requiring a redesign of the world-model architecture. It also addresses representation drift during training through candidate re-encoding and global index rebuilding. Positive aggregate results on two base agents, together with explicitly reported runtime overhead, make this approach worth further investigation. (Section 4; Appendices C–D)

  2. The appendices carefully disclose several limitations of the diagnostic evidence. The authors correctly distinguish training seeds from evaluation episodes, bootstrap matched training-seed pairs in the Frostbite analysis, and acknowledge that background corruption introduces distribution shift for the encoder. They also recognize that the value-ordering experiment does not directly establish more accurate value estimation. These choices improve the auditability of the evidence and avoid presenting case studies as definitive demonstrations of a general mechanism. (Appendices E, I, and J)

Critical weaknesses
  1. The central ablations do not sufficiently isolate the contribution of similar-context retrieval or establish that the proposed mechanism outperforms simpler sampling strategies. The neighbor ablation compares n=16n=16 with n=1n=1, but the group-weighting scheme in §4.3 simultaneously changes the number of random contexts and the loss weights. For example, when all 16 groups are populated:
  • With n=16n=16, there are 768 random contexts, and the loss-normalization denominator is 768+16=784768+16=784.
  • With n=1n=1, there are 1008 random contexts, and the normalization denominator is 1024. Thus, maintaining the same physical batch size does not control the training distribution or sample weighting. This comparison cannot adequately distinguish benefits from task-relevant neighbors from effects caused by changes in the proportion, correlation, or weighting of random samples. The most directly relevant missing control would fix the anchors, group size, context count, and weights, while replacing hash-selected neighbors with random neighbors. The paper also lacks empirical comparisons against simpler strategies such as TD-error-only prioritization. Moreover, the additive-versus-multiplicative ablation uses different thresholds, and Appendix L.3 explicitly acknowledges that trigger frequencies were not matched. These gaps directly affect the contribution assessment. The results suggest that the complete configuration may be effective, but they do not sufficiently establish that task-relevant retrieval and the proposed salience combination are responsible. Notably, the central Frostbite example scores 2054.9 with neighbors versus 2058.1 without neighbors, providing no evidence of a neighbor-retrieval benefit in that particular game. (Table 19)
  1. The statistical reliability of the main performance gains has not been established. The main experiments use four training seeds per game, but the PDF does not report individual-seed main results, confidence intervals for aggregate improvements, or probability-of-improvement estimates. On Atari 100k, where the paper itself emphasizes substantial seed variability, these point estimates are insufficient to assess whether the improvements reproduce reliably. Table 2 also indicates that the magnitude and breadth of the gains require careful interpretation:
  • STORM and DRAMA improve on 17/26 and 15/26 games, respectively.
  • Both agents achieve an IQM increase of only 0.016.
  • Calculating contributions from the reported scores shows that approximately 49% of DRAMA’s net aggregate mean improvement comes from Boxing. These observations do not demonstrate that the method is ineffective. They do mean that relative mean improvements alone cannot substantiate the main effectiveness claim. The matched-seed Frostbite analysis is useful supplementary evidence, but it cannot replace uncertainty analysis across the full benchmark. The authors should report confidence intervals for the differences between FLASH and the baselines using training-run resampling stratified by game, following established evaluation methodology.
Questions for authors
  1. Can you provide the per-game, per-training-seed scores underlying Table 2, together with confidence intervals for the differences in mean, median, and IQM? Were the baseline and FLASH runs conducted using fully matched seeds and identical evaluation protocols?

  2. With anchors, group size, loss weights, and context counts held fixed, do hash-selected neighbors outperform random neighbors? With trigger frequency and training budget matched, does the complete salience score outperform TD-error-only selection? These controls would directly establish whether the proposed additional machinery is necessary.

  3. In Algorithm 1, candidates are re-encoded using the current encoder and compared against the anchor’s registered key, but it is unclear whether the anchor itself is re-encoded. How are anchor keys and the queue refreshed after encoder updates or PCA rebuilding? If an old anchor key is retained, how does the procedure ensure that candidates match the anchor under the current representation?

5 comments
AuthorsOpenReview ↗

[1/4] Summary of our response

We thank Reviewer ccwG for the careful review, and for finding the intervention clearly defined and potentially useful in practice without redesigning the world-model architecture, and the aggregate results positive on two base agents.

The review asks whether retrieval and the salience score are really responsible for the gains (W1, Q2), whether the gains are statistically reliable (W2, Q1), and how anchor keys stay consistent with the current encoder (Q3). In response, we:

  1. Ran both controls specified in the review. (a) Hash vs. random neighbors with the anchor rule, group size, context counts, and loss weights fixed: mean HNS 1.163 vs 0.892. (b) Full salience vs. TD-error-only at matched trigger frequency: 1.166 vs 1.096.
  2. Checked the additive variant’s trigger frequency with a threshold sweep: at threshold 4.0, its trigger frequency approximately matches Full FLASH’s, and the full score wins all four aggregate metrics.
  3. Reported per-seed scores and game-stratified paired bootstrap CIs, with 5 seeds for STORM and DRAMA and a third agent (TWISTER). The mean-HNS CIs exclude zero for all three agents.
  4. Answered Q1b (yes, fully matched seeds and protocol) and Q3. Q3 identified a real gap: the query anchor is not re-encoded. We acknowledge it and describe the fix.

Revisions are marked in blue in the revised PDF.

AuthorsOpenReview ↗

[2/4] W1 — Do retrieval and the salience score cause the gains? (W1, Q2)

W1: The central ablations do not sufficiently isolate the contribution of similar-context retrieval or establish that the proposed mechanism outperforms simpler sampling strategies. The neighbor ablation compares (n=16) with (n=1), but the group-weighting scheme in [§4.3] simultaneously changes the number of random contexts and the loss weights. For example, when all 16 groups are populated: With (n=16), there are 768 random contexts, and the loss-normalization denominator is (768+16=784). With (n=1), there are 1008 random contexts, and the normalization denominator is 1024. Thus, maintaining the same physical batch size does not control the training distribution or sample weighting. This comparison cannot adequately distinguish benefits from task-relevant neighbors from effects caused by changes in the proportion, correlation, or weighting of random samples. The most directly relevant missing control would fix the anchors, group size, context count, and weights, while replacing hash-selected neighbors with random neighbors. [W1's points on TD-error-only selection and thresholds are quoted below.] The results suggest that the complete configuration may be effective, but they do not sufficiently establish that task-relevant retrieval and the proposed salience combination are responsible. Notably, the central Frostbite example scores 2054.9 with neighbors versus 2058.1 without neighbors, providing no evidence of a neighbor-retrieval benefit in that particular game. [Table 19]

Q2a: With anchors, group size, loss weights, and context counts held fixed, do hash-selected neighbors outperform random neighbors?

We agree that the original n=16n=16 vs. n=1n=1 ablation changes the random-context count and loss weights along with retrieval, so it cannot isolate neighbor relevance, and that it shows no benefit in Frostbite. Hash-selected neighbors outperform random neighbors in the new control, including in Frostbite.

To address Q2a, we compared Full FLASH with a Random-Neighbor variant using the same anchor-selection rule, group size (16), number of random contexts outside the groups, and loss weights. The variants differ only in how they fill the 15 non-anchor slots in each group: Full FLASH uses hash-matched replay contexts, while Random-Neighbor samples valid replay contexts uniformly [Table 30, App. M.4].

Table R5.

STORM, 26 gamesHash neighbors (Full FLASH)Random neighbors
Mean HNS ↑1.1630.892
Median HNS ↑0.5180.530
IQM ↑0.5740.510
Optimality gap ↓0.4430.466
Frostbite return2054.91824.6

Hash retrieval wins on mean, IQM, and optimality gap, while the median is slightly lower [Table R5]. In Frostbite, hash neighbors add 230 return over random neighbors. The flat n=16n=16 vs. n=1n=1 result is consistent with the confound the review identifies: n=1n=1 hands the freed slots back to random contexts with unit weight. The critic shows the same pattern. On identical Frostbite contexts, Full FLASH's values track igloo stage more closely than Random-Neighbor's (Spearman 0.162 vs 0.108; the paired CI of the difference excludes zero; [App. J.3]). One caveat: the rules are matched, but the realized anchors can differ once training diverges.

W1 (cont.): The paper also lacks empirical comparisons against simpler strategies such as TD-error-only prioritization. Moreover, the additive-versus-multiplicative ablation uses different thresholds, and [Appendix L.3] explicitly acknowledges that trigger frequencies were not matched. These gaps directly affect the contribution assessment.

Q2b.: With trigger frequency and training budget matched, does the complete salience score outperform TD-error-only selection? These controls would directly establish whether the proposed additional machinery is necessary.

Thank you for these suggestions. Yes. At matched trigger frequency, the full score beats both TD-error-only and additive selection on all four aggregate metrics [Table R6]. Threshold sweeps now set each variant to Full FLASH's ≈6 anchors per update (within 1%). Retrieval, context allocation, loss weights, and training budget are identical:

Table R6.

STORM, 26 gamesThreshold (sweep)Anchors / updateMean HNSMedianIQMOpt. gap ↓
Full vs TD-error-only [Tables 31–32]3.5 vs 3.8 (3.5–4.0)5.99 vs 5.961.166 vs 1.0960.547 vs 0.4950.621 vs 0.6020.434 vs 0.435
Full vs Additive [Tables 28–29]3.5 vs 4.0 (3.5–4.5)5.99 vs 6.031.163 vs 0.9840.536 vs 0.4580.599 vs 0.5650.446 vs 0.455

The margins are largest in mean and median HNS and small in optimality gap. Matching the frequency does not make the variants pick the same anchors, but it removes the "more or fewer anchors" explanation. The TD-error-only comparison uses 3 matched seeds per game.

AuthorsOpenReview ↗

[3/4] W2 — Statistical reliability (W2, Q1)

W2: The statistical reliability of the main performance gains has not been established. The main experiments use four training seeds per game, but the PDF does not report individual-seed main results, confidence intervals for aggregate improvements, or probability-of-improvement estimates. On Atari 100k, where the paper itself emphasizes substantial seed variability, these point estimates are insufficient to assess whether the improvements reproduce reliably. [Table 2] also indicates that the magnitude and breadth of the gains require careful interpretation: STORM and DRAMA improve on 17/26 and 15/26 games, respectively. Both agents achieve an IQM increase of only 0.016. Calculating contributions from the reported scores shows that approximately 49% of DRAMA's net aggregate mean improvement comes from Boxing. These observations do not demonstrate that the method is ineffective. They do mean that relative mean improvements alone cannot substantiate the main effectiveness claim. The matched-seed Frostbite analysis is useful supplementary evidence, but it cannot replace uncertainty analysis across the full benchmark. The authors should report confidence intervals for the differences between FLASH and the baselines using training-run resampling stratified by game, following established evaluation methodology.

Q1a: Can you provide the per-game, per-training-seed scores underlying [Table 2], together with confidence intervals for the differences in mean, median, and IQM?

Q1b: Were the baseline and FLASH runs conducted using fully matched seeds and identical evaluation protocols?

We agree that point estimates alone could not establish reliability. We now report per-seed scores and the requested intervals. The mean-HNS gain is significant on all three agents, while several other metrics are not, and we report every interval.

Q1a: To address your concern about training variability, we increased STORM and DRAMA from 4 to 5 seeds per game and added TWISTER with 3 seeds. We also added the full scores underlying Table 2 for every game and training seed, covering both baseline and FLASH, together with per-game 95% CIs for their mean score differences [Tables 33–35, App. N].

Table R7 reports 95% CIs for the aggregate differences using a paired bootstrap stratified by game. Bold intervals exclude zero:

Table R7.

FLASH − BaselineSTORMDRAMATWISTER
Mean HNS ↑+0.0484 [+0.0020, +0.0941]+0.0660 [+0.0063, +0.1335]+0.0793 [+0.0106, +0.1569]
Median HNS ↑+0.0492 [+0.0021, +0.0935]+0.0141 [−0.0532, +0.0806]+0.1674 [−0.0187, +0.3638]
IQM ↑+0.0167 [−0.0047, +0.0375]+0.0200 [−0.0169, +0.0576]+0.0770 [+0.0351, +0.1286]
Optimality gap ↓−0.0071 [−0.0172, −0.0004]−0.0091 [−0.0297, +0.0119]−0.0170 [−0.0400, +0.0021]

Breadth (W2). We agree that breadth matters as much as the mean. With the expanded runs, FLASH improves 16/26 (STORM), 15/26 (DRAMA), and 15/26 (TWISTER) games. Boxing now accounts for ≈40% of DRAMA's mean-HNS gain (previously 49%), and DRAMA's four aggregate metrics still favor FLASH when Boxing is excluded. Our effectiveness claim now rests on the paired intervals above rather than on relative mean improvement.

Q1b: Yes. Within each agent, baseline and FLASH use identical training seeds, architectures, hyperparameters, interaction budgets, warmup settings, and evaluation protocols [Sec. 5.1, App. C]. Up to retrieval activation at 50k steps, their episodes and updates are bit-for-bit identical.

Reviewer_W1DDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that visual model-based agents underuse task-relevant information already present in their latent states, and that the bottleneck lies in how past experience is reused for actor-critic learning. It proposes FLASH, a plug-in module for imagination-based agents. FLASH selects salient transitions as anchors with a score that combines normalized temporal-difference error and the change in the critic's value. It then retrieves contexts with matching codes from a PCA-based hash index over the replay buffer, and it places the anchors and their neighbours into the imagination batch for actor-critic training. Experiments on Atari 100k with STORM and DRAMA report gains in mean and median human-normalized scores at a wall-clock cost of about 5 to 18 per cent.

Critical strengths

S1. The paper addresses a practical question, namely how a world-model agent reuses the experience it has already collected, and the module leaves the encoder, world model and actor-critic architectures unchanged. The probing analysis in Sec. 3 supports the premise that task-relevant variables remain decodable from the latent state, and the overhead on STORM is modest at 5 to 7 per cent (Table 5). This makes the approach easy to adopt in existing agents, which is the main reason it merits consideration.

Critical weaknesses

W1. The method is an incremental and largely empirical adaptation of prioritized replay to world-model agents. Its anchor score ranks transitions by temporal-difference error, which is the priority signal of prioritized experience replay (Schaul et al., ICLR 2016). Prioritized replay has also been tailored to world-model agents before. Curious Replay (Kauvar et al., ICML 2023), which the paper cites, prioritizes experience for DreamerV3, DreamerV3-XP prioritizes trajectories by return, reconstruction loss and value error, and Simulus includes prioritized world-model replay. The design of FLASH rests on hand-set choices, such as the salience threshold of 3.5 and the 10-bit PCA hash, without principled justification. Retrieval only requires matching hash keys, and the similarity of retrieved neighbours is shown only through qualitative examples (Fig. 4, Appendix F), although RAM labels are available (Sec. 3). The authors should compare against these prioritization methods and justify why anchor retrieval improves on them.

W2. The experiments are too limited to support the plug-and-play claim. FLASH is evaluated only on Atari 100k and only with STORM and DRAMA, which are no longer strong baselines. STORM with FLASH reaches a mean human-normalized score of 1.157, below the 1.267 reported for STORM itself and well below DIAMOND (1.46, NeurIPS 2024) and TWISTER (1.62, ICLR 2025). Although Sec. 4 presents FLASH as applicable to Dreamer-style and diffusion world models, no Dreamer-family agent is tested. The gains are also small, with IQM rising by 0.016 on both agents over four seeds and without variance. Results on DreamerV3 and a stronger recent agent, with more seeds and at least one further benchmark, are needed.

Questions for authors

Q1 (clarification). How does FLASH differ, in mechanism and in effect, from prioritizing the same anchors with prioritized replay or Curious Replay? A comparison at matched compute would show whether neighbour retrieval adds anything beyond prioritization.

Q2 (requested experiments). Can the authors report FLASH on DreamerV3 and on a stronger recent agent such as DIAMOND or TWISTER, with more seeds and confidence intervals? Results on a second benchmark such as Crafter would test the plug-and-play claim.

Q3 (clarification). How similar are the retrieved neighbours to their anchors? The RAM labels used in Sec. 3 would allow a quantitative check of retrieval precision.

6 comments
AuthorsOpenReview ↗

[1/5] Summary of our response

We thank Reviewer W1DD for the detailed review, and for recognizing that task-relevant variables remain decodable from the latent state and that FLASH leaves the encoder, world-model, and actor–critic architectures unchanged, which makes it easy to adopt in existing agents.

The review raises two main concerns: whether FLASH adds anything beyond prioritized replay (W1, Q1, Q3), and whether the experiments are broad and reliable enough (W2, Q2). In response, we:

  1. Ran new controls that isolate what FLASH adds to prioritization: TD-error-only anchor selection (PER's signal), and hash-matched vs. random neighbors. Full FLASH wins both (mean HNS +0.070 and +0.271).
  2. Quantified retrieval precision with RAM labels, as suggested. Retrieved neighbors are closer to their anchors than random neighbors in all three games tested.
  3. Added TWISTER, the stronger agent named in Q2, and raised STORM and DRAMA to 5 seeds, with per-seed scores and confidence intervals. TWISTER+FLASH reaches 1.646 mean HNS, and the mean-HNS CIs exclude zero for all three agents.
  4. Explained the threshold and hash-width choices, why we did not add DreamerV3 or Crafter, and the gap to the published STORM score.

Revisions are marked in blue in the revised PDF.

AuthorsOpenReview ↗

[2/5] W1 (part 1) — Is FLASH more than prioritized replay? (W1, Q1)

W1: The method is an incremental and largely empirical adaptation of prioritized replay to world-model agents. Its anchor score ranks transitions by temporal-difference error, which is the priority signal of prioritized experience replay [Schaul et al., ICLR 2016]. Prioritized replay has also been tailored to world-model agents before. Curious Replay [Kauvar et al., ICML 2023], which the paper cites, prioritizes experience for DreamerV3, DreamerV3-XP prioritizes trajectories by return, reconstruction loss and value error, and Simulus includes prioritized world-model replay. [W1's points on hand-set choices and neighbour similarity are quoted in 3/5.] The authors should compare against these prioritization methods and justify why anchor retrieval improves on them.

Q1: How does FLASH differ, in mechanism and in effect, from prioritizing the same anchors with prioritized replay or Curious Replay? A comparison at matched compute would show whether neighbour retrieval adds anything beyond prioritization.

Thank you for raising this connection. We agree that FLASH builds on prioritization, since its score uses TD-error. However, we respectfully disagree that it is only an adaptation of prioritized replay. FLASH differs in two ways. First, its score adds a signed temporal value-difference term, intended to favor transitions involving task-relevant changes [Sec. 4.1]. Second, rather than setting replay frequency, the score selects anchors that retrieve similar past contexts for joint actor–critic training [Secs. 4.2–4.3], aiming to help the learner contrast similar situations and their outcomes [Sec. 5.2, Fig. 4]. The controls below support the contributions of both components.

Mechanism. The methods listed use a priority to decide how often an experience is replayed [Table R1]:

Table R1.

MethodPriority signalWhat the score decidesRetrieves related past contexts?
PERTD-errorReplay probability of that transitionNo
Curious ReplayVisit count + world-model lossReplay probability for world-model trainingNo
DreamerV3-XPReturn, reconstruction loss, value errorReplay probability of trajectoriesNo
SimulusPrioritized world-model replaySamples for world-model trainingNo
FLASHTD-error × signed value increase ΔVt=V(st)−V(st−1)\Delta V_t=V(s_t)-V(s_{t-1})Which transitions become anchorsYes: up to 15 hash-matched neighbors per anchor, trained together in the actor–critic imagination batch

Effect. We ran the controls that test what FLASH adds beyond prioritization, on the same agent (STORM, 26 games, matched seeds; [Table R2]):

Table R2.

ComparisonWhat is removedMean HNSIQMPaper
Full FLASH vs TD-error-only anchors (PER's signal; 5.99 vs 5.96 anchors/update)Value-increase term1.166 vs 1.0960.621 vs 0.602[Tables 31–32, App. M.5]
Full FLASH vs anchor-only (same anchor rule, no neighbors)Retrieval1.164 vs 1.0420.565 vs 0.538[Table 26, App. M.1]
Full FLASH vs random neighbors (same anchor rule, group size, weights)Relevance of neighbors1.163 vs 0.8920.574 vs 0.510[Table 30, App. M.4]

Each control removes one part of FLASH, and each lowers mean HNS. Anchor-only is the closest analogue of "prioritizing the same anchors": it adds the selected anchors to the batch without neighbors, and it trails Full FLASH by 0.122 [Table 26, App. M.1].

Matched compute (Q1). All controls use the same 100k environment steps, the same number of updates, and the same 1,024-context imagination batch. Retrieved contexts replace random ones rather than being added. Actor–critic compute is therefore identical, and hash vs. random neighbors differ only in which contexts fill the same slots. FLASH's training overhead is only 5.1–7.4% wall-clock on STORM [Table 6, App. C.1].

Changes: new TD-error-only control [Tables 31–32, App. M.5] and random-neighbor control [Table 30, App. M.4]; the latter is also summarized in [Table 4].

AuthorsOpenReview ↗

[3/5] W1 (part 2) — Hand-set choices and retrieval precision (W1, Q3)

W1 (cont.). The design of FLASH rests on hand-set choices, such as the salience threshold of 3.5 and the 10-bit PCA hash, without principled justification.

We agree that both values were set empirically rather than derived. Below is the reasoning behind each value and the evidence that the chosen setting works.

  • Threshold τz\tau_z=3.5. The score St=ReLU(ztTD)⋅softplus(ztΔV)S_t=\mathrm{ReLU}(z^{\mathrm{TD}}_t)\cdot\mathrm{softplus}(z^{\Delta V}_t) uses running z-scores, so the threshold sets the joint strength of the TD-error and value-change signals. For example, TD-error must be about 2.7 SD above its running mean when ΔVt\Delta V_t is 1 SD above its mean, and about 5.0 SD when ΔVt\Delta V_t is at its mean. New [App. C.2] gives a reference scale under an idealized independent-Gaussian model: τz=0\tau_z=0 removes threshold filtering, while τz≥5.69\tau_z\geq5.69 admits at most one candidate per batch with probability ≥95%. We set 3.5 between these ends. In training it gives about 6 anchors per update, so retrieved groups fill ≈96 of the 1,024 imagination contexts (≈9%) and the rest stay random. We chose this level so that retrieval stays targeted while most of the batch stays diverse. In existing development runs, lowering τz\tau_z to 3.4 left mean HNS unchanged (BattleZone, Frostbite, UpNDown; 7 paired runs; difference +0.003 [−0.016, +0.022]).
  • 10-bit hash. The bit width trades the number of neighbors available against their similarity. Each extra bit halves the average number of transitions per key. With 10 bits (1,024 keys) and a 50k–100k-transition replay buffer during retrieval, a key holds ≈50–100 transitions on average, against the 15 neighbors a group needs. In development runs, 11 bits often returned too few neighbors, and 9 bits often returned visually dissimilar frames. At 10 bits, retrieved neighbors are much more similar than random ones [Table R3] and give higher returns than random neighbors [Table 30].

W1 (cont.): Retrieval only requires matching hash keys, and the similarity of retrieved neighbours is shown only through qualitative examples [Fig. 4, Appendix F], although RAM labels are available [Sec. 3].

Q3. How similar are the retrieved neighbours to their anchors? The RAM labels used in [Sec. 3] would allow a quantitative check of retrieval precision.

Thank you for the suggestion; we ran exactly this check. Hash-retrieved neighbors are clearly more similar to their anchors than random replay contexts in all three games. For each anchor, we compare its retrieved neighbors with the same number of uniformly sampled replay contexts. Gower-style similarity is scored from RAM-decoded object positions and game-state variables on a [0, 1] scale. RAM is used only for this evaluation, never for retrieval. Results are averaged over three trained STORM+FLASH checkpoints per game:

Table R3.

GameHash-retrievedRandomGain
Pong0.8560.771+0.085
Seaquest0.7190.605+0.114
Frostbite0.8370.448+0.389

The gain is nearly identical across the three checkpoints. Full results of per game-checkpoint is in revised paper [Table 12].

Changes: new table and appendix [Sec. 5.2, Table 3, App. G].

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

We thank the AC and all reviewers for their careful and constructive reviews. We are encouraged that the reviewers found the idea of explicitly retrieving historical contexts for downstream policy learning well-motivated (T3bp), the intervention clearly defined and easy to adopt without redesigning the world model (W1DD, ccwG), the evidence that task-relevant variables remain decodable from the latent state informative (W1DD), the experimental details extensive (T3bp), and the aggregate results positive on two base agents (ccwG).

Our answers in brief (details in each reviewer thread; revisions are in blue in the revised PDF):

  1. Both components contribute beyond prioritization (W1DD, ccwG, T3bp). On STORM across 26 games, full salience beats TD-error-only selection (PER's priority signal) at matched trigger frequency, 1.166 vs 1.096 mean HNS. Hash-matched neighbors beat random neighbors with the same anchor-selection rule, group size, and loss weights, 1.163 vs 0.892 [App. M.4–M.5].
  2. The mean-HNS gains are statistically supported on all three agents (W1DD, ccwG, T3bp). STORM and DRAMA now use 5 seeds per game, and 3 for TWISTER, with per-seed scores [App. N]. Game-stratified paired bootstrap 95% CIs for the mean-HNS gain exclude zero: STORM +0.048 [+0.002, +0.094], DRAMA +0.066 [+0.006, +0.134], TWISTER +0.079 [+0.011, +0.157]. Some other metrics' intervals include zero, and we report them all.
  3. The gains hold on a stronger agent (W1DD). TWISTER+FLASH improves all four aggregate metrics over TWISTER across the 26-game benchmark (mean HNS 1.567 → 1.646; [Table 2]).
  4. Retrieved neighbors are measurably similar to their anchors (W1DD). By RAM-defined state similarity, hash-retrieved neighbors beat random ones by 0.085 (Pong), 0.114 (Seaquest), and 0.389 (Frostbite) on a [0, 1] scale [Table 3].
  5. FLASH's critic uses the decoded task variable more (T3bp). On fixed Frostbite contexts, the critic's alignment with igloo stage rises from 0.076 (STORM) to 0.162 (FLASH; Spearman). Removing either component lowers it [App. J.2–J.3].
  6. Ablation thresholds match Full FLASH’s trigger frequency (ccwG, T3bp). Threshold sweeps set the additive and TD-error-only variants to Full FLASH's ≈6 anchors per update [Tables 28, 32]. We also explain the choice of τz=3.5\tau_z=3.5 and the 10-bit hash (W1DD).
  7. Baseline reproduction (W1DD, T3bp). Our STORM baseline (1.132) is close to the ≈1.14 that a team including STORM's first author obtained with the released code. Every FLASH comparison is matched to its own baseline.
  8. One implementation gap, acknowledged (ccwG). The query anchor's hash key is not refreshed at query time. We describe the fix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.