acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

Abstract

World models predict future observations from current experience and actions, yet prediction can depend on observations seen far in the past. Episodic memory preserves past observations for later recall; however, as memory accumulates, it raises a fundamental question: which memories are useful for the current prediction, and which available retrieval cues should be trusted to find them? This is challenging because fixed criteria based on recency, pose overlap, or visual similarity can be unreliable across environments and queries. We propose Future-Aware Recall (FAR), a framework that learns episodic recall from future-aware predictive supervision and adaptive multi-cue scoring. During training, FAR measures predictive utility by the conditional log-likelihood of the realized future given recalled context, approximated by negative diffusion prediction loss, and uses it to train a retriever that remains future-blind at inference. The retriever learns cue-specific relevance and automatically determines which available retrieval cues, such as time, pose, vision, and audio, to trust for each query when selecting memories. Across three complementary settings, FAR outperforms hand-designed recall even with the same retrieval cues, automatically adapts which available cues to trust, and recalls the right history as the world changes. Together, these results establish FAR as a flexible, principled approach to episodic memory access in world models.

Primary area:
generative models
Keywords:
World Model, Memory, Recall

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_MhogOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a retrieval augmented method in the context of world models (adapting the EMDR2 framework, and adding an adaptive cue fusion). Historical observations (visual, sound, ... at each time stamp) are saved in an external memory. The main goal of the proposed method is to learn a retriever that extracts a top-K subset of observations maximizing the predictive utility (right term of Eq. 5) together with the current scoring preferences (left term of Eq. 5). The predictive utility is only known at training time (given the realized future), so at train time a (blind) scorer is learnt for approaching it (through Eqs. 5, 6 that adapts EMDR2). The importance of each cue is adapted for each query through a cue-fusion mechanism. The scorer and the world model are trained jointly.

The method is applied in 3 world-modeling settings: LoopNav (Minecraft visual + metadata), SoundSpaces (indoor navigation + sound), and AI2-Thor (interaction in environments), and is compared against most-recent observations (temporal recency), WorldMem (taking the top-K observations with high FOV overlap with the goal), and LongLive-RAG (top-K based on embedding similarity), showing improvement over those baselines on multiple metrics (e.g. correct prediction of future state, PSNR).

Critical strengths
  • The paper is relatively clear to read, regarding the motivation and the methodology; the companion website was also introducing the topic and the proposed method well
  • Compared to the proposed baselines, the capability of FAR is clear
Critical weaknesses
  • The baselines are weak w.r.t. the evaluated world-modeling settings: the temporal recency is the simplest possible baseline (this method is not appearing in Table 2) and the other two: WorldMem (high FOV overlap with the goal) and LongLive-RAG (based on embedding similarity) are not considering the audio cue. One reason might be that the authors decided to compare against external retriever methods only.
Questions for authors

Experimental setting (K and chunking)

Baselines (already stated in the weaknesses above)

  • In SoundSpaces the sound cue is critical, and is not present for the other compared methods, giving an unfair advantage to the proposed methodology. For AI2-THOR, this is also apparent in Fig. 9, since the constant guess is better compared to both Temporal and WorldMem baselines. Table 2 doesn't show any method managing the audio cue. I suggest adding other baselines, including multi-cue ones, even if there are outside the external retriever setting.

Selection of K

  • The value of K is not detailed. It might be corresponding to what is stated in l.1205, but I'm not sure. The typical value of K and the total number of observations should appear in the main document.
  • What is typical number of observations after chunking (is it appearing on Table 5?)?
  • The following question is: What is the performance using the proposed method with all observations (so following the whole method, without taking the top-K at the end).

Training computation

  • One stated limitation of the proposed method is the "additional training-time" l.482, but there is no quantification of this time in the paper.
  • The predictive utility is learnt for each setting, that might be very costly.

Relation with EMDR2

  • What is the main difference with EMDR2, apart from the cue fusion? I didn't put this into critical weaknesses, since I think there is substantial adaptation done, but the contribution could be better highlighted.

Minor comments

  • In l.240: the relevance is judged based on each cue alone before fusion. There are settings in which the visual/sound interaction is critical. This would be interesting to consider as a future work, but it is not affecting my assessment.
  • In Tab. 1, why lambda=0.5 in numerical? Shouldn't lambda be a vector combining the different cues? What is the learnt value for SoundSpace in particular?
  • Regarding the predictive utility, ideally a single foundational trained model could be able to have a sense of it.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer Mhog (Part 1 of 3)

We thank the reviewer for investing time in assessing our approach and for the helpful suggestions on baselines, computational cost, and multimodal retrieval. Revisions in the paper are highlighted in blue.


[C1] The baseline set is limited to recency-, FOV-, and embedding-similarity-based retrieval, and Temporal is absent from Table 2. Include stronger memory baselines beyond external retrievers.

We refer the reviewer to Section 4.4 (Additional baseline comparisons; lines 500–509) and Table 1 of the revised paper, where we add comparisons with MemLearner and MASt3R-SLAM on LoopNav alongside Temporal and the existing retrieval baselines. We have also added Temporal (Navigation World Model) to the conceptual comparison in Table 6 (Appendix B), explicitly identifying its time-based retrieval rule. FAR-Multi-Cue outperforms both additional methods across all three metrics; relative to MemLearner, it improves PSNR by 2.674 and reduces LPIPS and DreamSim by 29.9% and 36.9%, respectively.


[C2] Additional audio or agent-state cues may give FAR an unfair advantage over the baselines, particularly in SoundSpaces and AI2-THOR-dyn. Include stronger multi-cue baselines, including methods outside the external-retriever setting.

We thank the reviewer for this suggestion. We agree that the benefit of additional retrieval cues should be distinguished from that of learning predictive relevance. Our experiments explicitly examine both aspects, including comparisons where the available retrieval cues are matched.

First, FAR-Meta outperforms WorldMem using only the same metadata cue types (time and pose), without relying on audio or historical agent-state retrieval cues. On SoundSpaces, even with matched 1,000-frame retrieval pools, FAR-Meta achieves 17.295 versus 14.339 PSNR on SoundSpaces (endpoint scans) and 18.659 versus 13.979 on SoundSpaces (periodic scans) (Section A.1, Table 3). Similarly, on AI2-THOR-dyn (Section 4.3, Figure 9), FAR-Meta achieves 66.7% accuracy, compared with 32.5% for WorldMem and 41.8% for Temporal, also exceeding the 50% constant-guess reference. These comparisons show that FAR's improvements are not simply attributable to access to additional modalities.

Second, FAR-Multi-Cue addresses the complementary question of how to effectively exploit multiple cues available in the interaction history. Rather than imposing handcrafted rules for combining modalities or prescribing which cue should be preferred, FAR learns both cue-specific relevance and query-dependent cue weights through future-aware predictive supervision (Section 3.2). Thus, the multi-cue experiments demonstrate how FAR can learn to exploit complementary historical information, while the same-cue comparisons establish its advantage without relying on additional retrieval modalities.

Regarding additional multi-cue baselines, existing approaches are designed around specific retrieval criteria, such as temporal recency, FOV overlap, or visual similarity. Incorporating audio is therefore not a direct extension of these methods, as it requires defining new relevance functions and mechanisms for combining heterogeneous cues that are not specified by the original approaches. Learning how to combine such cues, rather than manually prescribing their relative importance, is precisely one of FAR's central motivations. We therefore prioritize controlled same-cue comparisons while separately evaluating the benefits of learned multi-cue retrieval. The revised paper describes learned cue fusion in Section 3.2, presents matched-pool comparisons in Section A.1 (Table 3), and clarifies the shared AI2-THOR-dyn components and the distinction between retrieval cues and generator conditioning in Sections 4.3 and D.3–D.4. We have revised the quantitative-results paragraph of Section 4.2 (lines 366–368) to explicitly clarify that FAR-Meta and WorldMem are compared with available retrieval cues restricted to metadata (time and pose), while FAR-Multi-Cue separately evaluates the benefit of making audio available for retrieval.


AuthorsOpenReview ↗

Reply to Reviewer Mhog (Part 2 of 3)


[C3] State the typical value of KK, the total number of historical observations, and the number of candidates after chunking explicitly in the main text.

We thank the reviewer for pointing out the lack of clarity. The value of KK and the total number of historical observations are detailed in Table 10, Section D.4 of the revised paper. Specifically, KK is determined by the context size as K=context size−1K = \text{context size} - 1, since one slot is reserved for the current observation. We have clarified this relationship in Section 3.3 (Temporally structured recall; lines 284–289) of the revised paper: the four rollout benchmarks use K=11K=11, selecting from up to 100 representatives obtained by dividing a 1,000-frame pool into chunks of 10 frames. Table 10 in Section D.4 provides the dataset-specific configurations.


[C4] Report performance when using all observations retained by the chunking procedure, without the final Top-K selection.

We refer the reviewer to Section A.2 (Table 4) of the revised paper, where we compare selecting the top 11 contexts from 100 temporal chunks against using all 100 chunk representatives. Using all representatives yields PSNR/LPIPS/DreamSim of 17.184/0.342/0.078, compared with 17.267/0.337/0.082 for Top-K selection. The small differences indicate comparable prediction quality while Top-K selection reduces the number of recalled contexts from 100 to 11.


[C5] Quantify the additional training cost of setting-specific predictive-utility learning.

We refer the reviewer to Section A.3 (Table 5) of the revised paper, where we quantify end-to-end training cost on LoopNav using four H100-equivalent GPUs. For 700K steps, Temporal requires approximately 130 hours, while FAR-Meta, FAR-Visual, and FAR-Multi-Cue require 141, 141, and 142 hours, respectively. This corresponds to approximately 8–9% additional training time, including predictive-utility evaluation and retriever optimization; WorldMem requires approximately 175 hours in the same setting, reflecting the computational overhead of its iterative greedy FOV-based memory selection. Thus, despite its additional predictive-utility evaluation and retriever optimization, FAR trains faster than WorldMem in this comparison.


[C6] Clarify FAR's theoretical and practical differences from EMDR2 beyond cue fusion.

Theoretical distinction. We thank the reviewer for this suggestion. As shown in Appendix C.2, both EMDR2 and FAR arise from the same latent-variable maximum-likelihood principle, where downstream predictive likelihood provides supervision for discrete retrieval. However, FAR extends the retriever parameterization to incorporate multiple cue-specific relevance functions and query-dependent fusion weights. As derived in Appendix C.3, although the underlying posterior-matching objective retains the EMDR2-style gradient, FAR's gradient decomposes into two learning pathways: one optimizing relevance scores within each cue and another optimizing cue weights λ\lambda based on their contribution to predictive utility. Thus, future-aware supervision learns not only which memories to retrieve, but also which cues to trust for each query.

Practical adaptations. FAR further adapts this framework to video diffusion world models through several design choices: (1) approximating predictive utility using the negative diffusion prediction loss of the observed future, rather than reader likelihood for question answering; (2) contrastive pretraining of retrieval encoders with JEPA-style representations, combined with frozen, cacheable memory embeddings and lightweight trainable query adapters; and (3) temporally chunked Top-K retrieval to reduce redundant selection from long observation histories. Together with adaptive multi-cue fusion, these adaptations enable predictive-utility-based retrieval from evolving multimodal episodic memories while preserving the world model's conditioning interface. The revised paper details the latent-variable formulation in Section C.2 and the two learning pathways in Section C.3. Sections 3.2–3.3 and D.1 describe adaptive cue fusion, the diffusion-loss utility surrogate, pretrained cacheable representations, and temporally chunked Top-K retrieval.


AuthorsOpenReview ↗

Reply to Reviewer Mhog (Part 3 of 3)


[C7] Fusion weights should form a vector with one scalar per cue, so clarify the scalar notation λ=0.5\lambda=0.5 in Table 1. Report the learned SoundSpaces weights.

We thank the reviewer for pointing out the ambiguous notation. The fusion weight λ\boldsymbol{\lambda} is a vector with one scalar weight per retrieval cue, as defined in Eq. (8). In Table 1, the uniform-fusion baseline uses λ=(λ1,λ2)=(0.5,0.5)\boldsymbol{\lambda}=(\lambda_1,\lambda_2)=(0.5,0.5), assigning equal weight to metadata and visual relevance scores without a prior preference for either cue. We have revised the uniform-fusion notation in Table 1 and explained the equal-weight vector in Section 4.4 of the revised paper (lines 515–518).

For SoundSpaces, the learned weights after convergence average approximately 0.95 for metadata (pose and time) and 0.05 for sound. These are averages of query-dependent weights, rather than a single fixed mixture shared across all queries. The strong metadata preference suggests that pose and time provide the primary retrieval signal, while sound serves as a complementary tie-breaking cue when metadata assigns similar relevance to multiple context frames. We have added these average weights and their interpretation in Section 4.2 of the revised paper (lines 373–375).


[C8] Could a general-purpose pretrained foundation model estimate predictive utility without setting-specific training?

Thank you for the suggestion. We agree that a pretrained foundation model could potentially estimate predictive utility across environments, but our current design has two motivations.

First, semantic relevance is distinct from downstream predictive utility: a memory that appears relevant to a foundation model may not reduce the particular world model's prediction error. For example, a visually similar view of a closed refrigerator may be less useful for predicting an opening action than an older observation revealing its contents. FAR uses the downstream model's prediction loss to learn which memories actually help that model under its conditioning interface.

Second, we seek a controlled comparison of retrieval strategies: introducing an additional foundation-model utility estimator would also introduce external pretrained knowledge and additional computation, making it harder to attribute gains to the retrieval mechanism itself. Our current approach therefore learns predictive relevance from the shared downstream world model's feedback. A transferable foundation-model estimator remains a promising extension, provided its alignment with downstream predictive utility and its additional computational cost are evaluated explicitly.


[C9] Independent cue scoring may miss jointly informative visual–audio interactions. Discuss joint cue modeling as a future direction.

We agree that independently scoring each cue before fusion may not fully capture interactions between modalities, such as cases where visual and audio observations are jointly informative despite being individually ambiguous. Modeling such interactions through joint cue-specific scorers is an interesting extension of FAR. However, evaluating their benefits also requires benchmarks where cross-modal interactions are essential for long-horizon prediction. While audio-visual navigation datasets exist, benchmarks combining multimodal observations, loop closure, and history-dependent future prediction remain relatively limited. Our SoundSpaces experiments provide a controlled evaluation of complementary retrieval cues, but do not explicitly isolate nonlinear cross-modal interactions. We acknowledge this limitation and leave joint cue modeling and its evaluation on richer multimodal memory benchmarks to future work.


Reviewer_FBjQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author proposes Future-Aware Recall (FAR), a learned episodic-memory retriever for video diffusion world models. During training, it uses the observed future’s diffusion prediction loss to supervise memory relevance, while retrieval remains future-blind at inference. Cue-specific scorers and query-dependent weights combine available signals such as time, pose, vision, and audio, with temporal chunking used to reduce redundant retrieval. Experiments on LoopNav, SoundSpaces, and AI2-THOR report improvements in rollout quality and history-dependent state prediction over fixed retrieval strategies.

Critical strengths
  • The predictive-relevance formulation is well motivated: posterior matching provides a coherent way to supervise discrete retrieval using downstream prediction quality while learning which available cues to trust, rather than relying on fixed similarity rules.

  • The component ablation provides useful evidence beyond the overall performance gains: Table 1 shows improvements from query adaptation and learned cue fusion, with learned fusion reducing DreamSim from 0.088 to 0.082 compared with equal weighting.

Critical weaknesses
  • Unequal memory pools weaken a central baseline comparison. Appendix C.4 allows FAR to search up to 1,000 memory frames but caps WorldMem at 200 on LoopNav and SoundSpaces. Access to a larger history may itself improve recall, so the comparisons in Sections 4.1–4.2 do not cleanly isolate the benefit of learned relevance even when retrieval cue types are matched; results using identical candidate pools are needed.

  • The off-scene dynamics evidence needs tighter qualification. Appendix C.3–C.4 introduces ground-truth historical agent annotations, a supervised agent-state predictor, an auxiliary state term in the retrieval reward, and state-conditioned generation, while evaluating a one-step probe rather than an autoregressive rollout. These are legitimate design choices, but it is unclear which additions are shared across baselines, making Figure 9’s gains difficult to attribute specifically to FAR’s recall mechanism rather than differences in supervision or conditioning.

Questions for authors
  • Why is WorldMem restricted to 200 frames when FAR accesses up to 1,000? Do you have existing comparisons using identical candidate-memory pools, and do they preserve the reported same-cue advantage?

  • In AI2-THOR-dyn, do Temporal, WorldMem, FAR-Meta, and FAR-Multi-Cue share the same agent-state predictor, auxiliary supervision, and generator conditioning? Please distinguish additional information used only for retrieval from information supplied to the prediction pipeline.

  • Figure 7 labels its intervention “Post-Interaction Observation Removed,” while the caption describes histories with and without the object-placement event. Is the underlying interaction actually changed, or is only its resulting observation removed from memory? This distinction matters for interpreting the example as prediction of different world states versus sensitivity to missing evidence.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer FBjQ (Part 1 of 2)

We are grateful for the reviewer's time and detailed assessment, which helped us clarify the evaluation protocol and the interpretation of our experiments. Revisions in the paper are highlighted in blue.


[C1] FAR's larger candidate pool may explain its advantage over WorldMem, even with matched cue types. Provide identical-pool comparisons and establish whether the same-cue advantage persists.

We thank the reviewer for raising this concern. The 200-frame cap was adopted to mitigate temporally redundant retrieval in WorldMem, whose greedy FOV-based selection does not explicitly enforce temporal diversity. To address the fairness concern, we additionally evaluate WorldMem with the same 1,000-frame retrieval pool available to FAR.

Increasing WorldMem's retrieval pool from 200 to 1,000 frames produces negligible performance changes across all three benchmarks. For example, PSNR changes from 15.372 to 15.340 on LoopNav, 14.298 to 14.339 on SoundSpaces (endpoint scans), and 13.917 to 13.979 on SoundSpaces (periodic scans), while LPIPS and DreamSim remain nearly unchanged. This limited sensitivity to pool size is consistent with our observation of temporally redundant retrieval in WorldMem, suggesting that access to more historical frames does not necessarily yield more informative recalled contexts.

Importantly, FAR-Meta continues to substantially outperform WorldMem when both methods have access to the same 1,000-frame retrieval pool and use the same metadata cues (time and pose). Compared with WorldMem at 1,000 frames, FAR-Meta improves PSNR by 1.809, 2.956, and 4.680 dB on LoopNav, SoundSpaces (endpoint scans), and SoundSpaces (periodic scans), respectively, while reducing DreamSim by 39.4%, 46.8%, and 63.1%. LPIPS also improves consistently across all three benchmarks.

These controlled comparisons demonstrate that FAR's same-cue advantage persists when retrieval-pool size is matched and cannot be explained simply by access to a larger history. We report these results in Section A.1 (Table 3) and clarify the original WorldMem configuration in Section D.4 (Evaluation operating point; lines 1339–1347) of the revised paper.


[C2] Clarify whether all AI2-THOR-dyn baselines share the agent-state predictor, auxiliary supervision, generator conditioning, and evaluation protocol. Distinguish retrieval-only information from prediction-pipeline inputs so the gains can be attributed to retrieval.

We thank the reviewer for raising this concern. We clarify that all four methods (Temporal, WorldMem, FAR-Meta, and FAR-Multi-Cue) use the same agent-state prediction architecture, auxiliary state supervision, state-conditioned generator architecture, and evaluation protocol. The differences lie in how historical memories are retrieved, rather than in the supervision or conditioning provided to the downstream prediction pipeline.

Importantly, we distinguish information used for memory retrieval from information used for future prediction. All methods use historical agent-state annotations to support the auxiliary state predictor. During training, the generator receives ground-truth target agent states through teacher forcing; at inference, all methods rely on their predicted agent states, without access to ground-truth future information.

For retrieval, the key distinction is that FAR learns which historical memories and available cues are most useful for future prediction, rather than relying on fixed, hand-designed relevance rules. In particular, FAR-Multi-Cue learns to adaptively weight historical agent-state information alongside metadata according to their predictive utility for the current query. This information comes from the same recorded history, rather than privileged future observations. The auxiliary state term in FAR's retrieval reward provides supervision for learning memory relevance, not additional conditioning for its generator.

Thus, Figure 9 compares retrieval strategies under matched downstream architectures and supervision, without granting FAR privileged future information. Notably, FAR-Meta already outperforms the heuristic baselines using the same metadata cue types, while FAR-Multi-Cue further demonstrates the benefit of adaptively exploiting available historical information. We have clarified the shared components in Section 4.3 (Off-scene dynamics prediction; lines 459–468) and Section D.3 (Second-agent supervision in AI2-THOR-dyn; lines 1309–1317), and the conditioning and inference protocol in Section D.4 (Generation conditioning, lines 1329–1338; AI2-THOR-dyn inference, lines 1349–1354) of the revised paper.


AuthorsOpenReview ↗

Reply to Reviewer FBjQ (Part 2 of 2)


[C3] Does Figure 7 change the underlying object-placement interaction or only remove its resulting observation from memory? Clarify whether the example demonstrates different world states or sensitivity to missing historical evidence.

We thank the reviewer for pointing out this ambiguity. We clarify that the underlying interaction and world state remain unchanged; only the resulting post-interaction observation is removed from episodic memory. Specifically, the agent places the tomato inside the refrigerator in both conditions, and the subsequent current observation and opening action are identical. The intervention changes only whether the post-placement observation is available for memory retrieval.

Thus, Figure 7 examines the effect of removing historical evidence, rather than comparing different underlying world states. The ground-truth future remains the same in both conditions; what changes is the information available to the model for predicting it. We agree that the original caption and discussion could incorrectly suggest that the underlying object-placement event differs. We have revised the Figure 7 caption and its accompanying discussion in Section 4.3 (Sensitivity to historical evidence; lines 450–458) of the revised paper to explicitly describe the memory-removal intervention and clarify the intended interpretation.


Reviewer_FBjQOpenReview ↗

Response to Author Rebuttal

I apprecaite the authors for the response and efforts, which addressed my previous two concerns on the apple-to-apple comparison and the AI2-THOR-dyn method. Therefore, I raised my score to 3.

We thank the reviewer for their thoughtful follow-up and for reconsidering the score! We appreciate their time and constructive feedback, which helped us strengthen the controlled comparisons and clarify the AI2-THOR-dyn methodology.

Reviewer_HG1TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FAR, a world‑model episodic memory retriever trained via predictive‑utility supervision, with inference strictly future‑blind. Its modular design fuses heterogeneous retrieval cues and supports cached memory without modifying the generator interface. Experiments and ablations show improvements over a strong pre‑trained visual retriever across multiple environments. However, comparisons are confounded by mismatched memory‑frame budgets, insufficient baselines for cue fusion, and limited experimental setup that restricts general claims on off‑scene dynamics.

Critical strengths
  1. The predictive-utility formulation provides a well-motivated learning signal for memory retrieval while keeping inference independent of future observations.
  2. The modular architecture combines heterogeneous retrieval cues and supports cached memory representations without changing the generator’s conditioning interface.
  3. Experiments cover complementary retrieval challenges, and ablations demonstrate improvements beyond a strong pretrained visual retriever.
Critical weaknesses
  1. FAR accesses up to 1,000 memory frames while WorldMem is capped at 200 on LoopNav and SoundSpaces, confounding the comparison of retrieval methods (Appendix C.4).
  2. Comparing adaptive fusion only against equal-weight fusion does not establish its advantage over a learned query-independent mixture (Table 1).
  3. AI2-THOR-dyn relies on structured agent-state supervision and a one-step evaluation in previously seen rooms, limiting support for broader off-scene dynamics claims (Appendices C.3–C.4).
Questions for authors
  1. Could you clarify how the differing maximum memory‑frame budgets (1000 for FAR versus 200 for WorldMem) affect retrieval performance, and present fair comparisons under identical memory‑frame limits on LoopNav and SoundSpaces?
  2. Why is adaptive cue fusion only benchmarked against equal‑weight fusion? Can you add comparisons against learned query‑independent mixture baselines, and discuss whether the AI2‑THOR‑dyn setup constrains claims for general off‑scene dynamics?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer HG1T (Part 1 of 2)

We appreciate the reviewer's careful reading and thoughtful feedback on comparison fairness, cue fusion, and the scope of our dynamics results. Thank you for your time; revisions in the paper are highlighted in blue.


[C1] The unequal candidate-pool limits for FAR and WorldMem may confound the LoopNav and SoundSpaces comparisons. Explain their effect and report results under identical limits.

We thank the reviewer for raising this concern. The 200-frame cap was adopted to mitigate temporally redundant retrieval in WorldMem, whose greedy FOV-based selection does not explicitly enforce temporal diversity. To address the fairness concern, we additionally evaluate WorldMem with the same 1,000-frame retrieval pool available to FAR. Cached-key FAR takes 210.6 ms/step (9.4 ms retrieval), versus 399.3 ms/step for WorldMem with a 1,000-frame pool.

Increasing WorldMem's retrieval pool from 200 to 1,000 frames produces negligible performance changes across all three benchmarks. For example, PSNR changes from 15.372 to 15.340 on LoopNav, 14.298 to 14.339 on SoundSpaces (endpoint scans), and 13.917 to 13.979 on SoundSpaces (periodic scans), while LPIPS and DreamSim remain nearly unchanged. This limited sensitivity to pool size is consistent with our observation of temporally redundant retrieval in WorldMem, suggesting that more history need not improve recalled contexts, even with greater retrieval computation than FAR.

Importantly, FAR-Meta continues to substantially outperform WorldMem when both methods have access to the same 1,000-frame retrieval pool and use the same metadata cues (time and pose). Compared with WorldMem at 1,000 frames, FAR-Meta improves PSNR by 1.809, 2.956, and 4.680 dB on LoopNav, SoundSpaces (endpoint scans), and SoundSpaces (periodic scans), respectively, while reducing DreamSim by 39.4%, 46.8%, and 63.1%. LPIPS also improves consistently across all three benchmarks.

These controlled comparisons demonstrate that FAR's same-cue advantage persists when retrieval-pool size is matched and cannot be explained simply by access to a larger history. We report these results in Section A.1 (Table 3) and clarify the original WorldMem configuration in Section D.4 (Evaluation operating point; lines 1339–1347) of the revised paper.


[C2] Comparison with uniform fusion does not establish the benefit of query-dependent fusion. Add a learned query-independent mixture baseline.

We thank the reviewer for suggesting this stronger control. We have added a learned query-independent mixture to Table 1 in Section 4.4 of the revised paper; the interpretation and learned weights are discussed in lines 515–522. On LoopNav, constant and per-query fusion achieve close performance: PSNR/LPIPS/DreamSim are 17.313/0.331/0.079 and 17.267/0.337/0.082, respectively, with slightly better mean scores for constant fusion. The learned constant mixture assigns 0.91 to metadata and 0.09 to vision. This strong metadata preference suggests that a global weighting is effective for this navigation setting, with visual information providing a complementary tie-breaking signal.

AI2-THOR provides direct evidence that query adaptivity responds to different predictive information needs. Unlike navigation queries dominated by geometric relevance, an opening action can require recalling a container's contents, while staying in place or turning around calls for different historical views. We therefore conduct a paired analysis holding the query timepoint, initial pose, and candidate pool fixed while changing the requested action (Section 4.4, lines 477–483; Table 2). The mean visual weight rises to 0.405 for open, compared with 0.316 for stay and 0.287 for turn, consistent with greater reliance on visual evidence for interaction-dependent prediction.

This adaptation is reflected in the memories selected for each action. In Figure 10, the same current observation yields a view of the refrigerator interior for open, the closed refrigerator for stay, and the surrounding room for turn. The example-specific visual weights are 0.44, 0.34, and 0.29, respectively. Thus, query adaptivity allows FAR to change both cue preferences and retrieved evidence according to the requested prediction, even when the available history is unchanged.

Together, the results show that learned constant fusion is competitive on LoopNav, while AI2-THOR demonstrates action-dependent cue weighting and retrieval consistent with the query's information needs.


AuthorsOpenReview ↗

Reply to Reviewer HG1T (Part 2 of 2)


[C3] Structured agent-state supervision and one-step evaluation in seen rooms limit the evidence for general off-scene dynamics prediction. Clarify the scope and limitations of the AI2-THOR-dyn results.

We thank the reviewer for highlighting these limitations. We would like to clarify that we do not claim to solve general off-scene dynamics prediction. Rather, as discussed in Section 4.3, AI2-THOR-dyn serves as a controlled evaluation to verify whether FAR can retrieve motion-informative historical observations that improve future prediction.

We acknowledge that this setting makes several simplifying assumptions, including structured agent-state supervision, deterministic motion along a predefined patrol route, and one-step evaluation in previously seen environments. These assumptions allow us to isolate the contribution of memory retrieval but limit generalization to more complex scenarios.

For instance, when an independent agent follows a stochastic or history-dependent policy, its future state may not be reliably inferred from a short historical context; similarly, long-horizon autoregressive prediction may accumulate errors that cannot be addressed by retrieval alone. We have explicitly clarified these limitations and the intended scope of our claims in Section 4.3 (Off-scene dynamics prediction; lines 459–468) and Section 5 (lines 534–538) of the revised paper. Sections D.2–D.4 provide the scene-split, supervision, and one-step evaluation details.


AuthorsOpenReview ↗

A Kind Reminder for Comments

We appreciate Reviewer HG1T's thoughtful suggestions on experimental controls and the scope of our conclusions. We have strengthened the revised paper in the following ways:

  • Fair retrieval-pool comparison: Added WorldMem evaluations with a 1,000-frame pool matching FAR (Appendix A.1, Table 3). WorldMem's performance changes negligibly, and FAR-Meta continues to outperform it using the same metadata cue types.
  • Analysis of the benefit of query-dependent fusion: Added a learned constant-mixture baseline showing similar performance to per-query fusion on LoopNav, where metadata dominates and vision may serve as a tie-breaking cue (Section 4.4, Table 1). In AI2-THOR, changing only the requested action changes cue weights and retrieved evidence, demonstrating a non-trivial role for query-dependent fusion when prediction needs vary (Table 2, Figure 10).
  • Scope of dynamics claims: Explicitly describe AI2-THOR-dyn as a controlled, one-step evaluation with structured state supervision, predefined motion, and seen rooms, and acknowledge the limits on broader generalization (Sections 4.3 and 5; Appendices D.2–D.4).

Thank you again for helping us make both the evidence and its limitations clearer. We would be happy to address any remaining questions; revisions in the paper are highlighted in blue.

Reviewer_szGBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Future-Aware Recall, a learning-based episodic memory retrieval framework for world models. The method uses the actual future observation during training to estimate how useful each candidate memory is for future prediction. Predictive utility is defined and estimated to supervise the learning of the memory retriever. The framework additionally introduces adaptive multi-cue fusion to dynamically combine different retrieval signals. Experiments demonstrate improvements over conventional memory retrieval strategies in long-horizon visual prediction, multimodal retrieval, and changing-world scenarios.

Critical strengths
  1. The paper addresses an important and well-motivated problem. The authors identify a meaningful limitation of existing episodic memory retrieval methods, i.e., the most similar historical observation is not necessarily the most useful one for future prediction.

  2. In FAR, future observations are used only during training to supervise the retriever. During inference, the retriever selects memories without access to future observations. This makes the approach practical for prediction tasks.

  3. The experimental evaluation covers several different scenarios to comprehensively verify the proposed method. The ablations and visualizations provide useful evidence of effectiveness.

Critical weaknesses
  1. The appendix states that evaluation uses retrieval pools of up to 1,000 historical frames. However, WorldMem retains a 200-frame retrieval pool cap on LoopNav and SoundSpaces, whereas FAR can access the larger pool. While the authors explain that the WorldMem restriction matches its training configuration, I think evaluating FAR and WorldMem with identical retrieval pool sizes is achievable.

  2. The paper's central contribution is learning retrieval relevance from future predictive utility. However, FAR also benefits from several other components, including contrastively pretrained retrieval encoders, learned query adapters, and adaptive cue fusion. Table 1 investigates several of these components, but it does not provide a completely controlled comparison of the same trainable retrieval architecture with and without future-aware predictive supervision.

  3. One major claim is that FAR can automatically determine which retrieval cues to trust for different queries. The results show that combining multiple cues generally improves prediction. However, these improvements alone do not fully establish that the model is dynamically adjusting its cue preferences according to the current situation.

Questions for authors
  1. Appendix C.4 indicates that WorldMem uses a 200-frame retrieval pool on LoopNav and SoundSpaces, while FAR can use up to 1,000 frames. Could the authors clarify the reason for this discrepancy and provide comparisons using the same retrieval pool size for both approaches?

  2. Could the authors provide an ablation that removes future-aware predictive supervision while keeping the retrieval encoder, query adapter, and cue-fusion architecture unchanged? This would help quantify the specific contribution of the proposed training objective.

  3. Since the predictive utility of each memory is evaluated individually during training, while multiple memories are jointly used during inference, I wonder how FAR handles cases in which several individually uninformative memories become useful when combined?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer szGB (Part 1 of 2)

We thank the reviewer for the time and care devoted to evaluating our work and for the constructive suggestions on experimental controls and retrieval supervision. Revisions in the paper are highlighted in blue.


[C1] WorldMem is capped at 200 candidate frames on LoopNav and SoundSpaces, whereas FAR can access 1,000, potentially confounding the comparison. Explain the discrepancy and compare the methods under identical candidate-pool sizes.

We thank the reviewer for raising this concern. The 200-frame cap was adopted to mitigate temporally redundant retrieval in WorldMem, whose greedy FOV-based selection does not explicitly enforce temporal diversity. To address the fairness concern, we additionally evaluate WorldMem with the same 1,000-frame retrieval pool available to FAR. Cached-key FAR takes 210.6 ms/step (9.4 ms retrieval), versus 399.3 ms/step for WorldMem with a 1,000-frame pool.

Increasing WorldMem's retrieval pool from 200 to 1,000 frames produces negligible performance changes across all three benchmarks. For example, PSNR changes from 15.372 to 15.340 on LoopNav, 14.298 to 14.339 on SoundSpaces (endpoint scans), and 13.917 to 13.979 on SoundSpaces (periodic scans), while LPIPS and DreamSim remain nearly unchanged. This limited sensitivity to pool size is consistent with our observation of temporally redundant retrieval in WorldMem, suggesting that more history need not improve recalled contexts, even with greater retrieval computation than FAR.

Importantly, FAR-Meta continues to substantially outperform WorldMem when both methods have access to the same 1,000-frame retrieval pool and use the same metadata cues (time and pose). Compared with WorldMem at 1,000 frames, FAR-Meta improves PSNR by 1.809, 2.956, and 4.680 dB on LoopNav, SoundSpaces (endpoint scans), and SoundSpaces (periodic scans), respectively, while reducing DreamSim by 39.4%, 46.8%, and 63.1%. LPIPS also improves consistently across all three benchmarks.

These controlled comparisons demonstrate that FAR's same-cue advantage persists when retrieval-pool size is matched and cannot be explained simply by access to a larger history. We report these results in Section A.1 (Table 3) and clarify the original WorldMem configuration in Section D.4 (Evaluation operating point; lines 1339–1347) of the revised paper.


[C2] Isolate the contribution of future-aware predictive supervision by removing it while keeping the retrieval encoder, query adapter, and cue-fusion architecture unchanged.

We thank the reviewer for emphasizing the importance of isolating future-aware predictive supervision from architectural improvements. We would like to clarify that Table 1 in Section 4.4 already provides a relevant control through the comparison between “Enc. Pretrained” and “+ MLP Adapter,” and explain why this comparison is informative.

Our query adapter uses a residual parameterization, q=h+MLPϕ(h)q=h+\mathrm{MLP}_{\phi}(h), with its final layer initialized to zero. Consequently, adding the adapter initially leaves the pretrained retrieval function exactly unchanged. More importantly, under our retriever objective (Eq. 16 in Section C.3), removing future-aware predictive utility by setting uθ,i=0u_{\theta,i}=0 makes the future-aware posterior identical to the future-blind retrieval distribution (Eqs. 13 and 15):

=\operatorname{softmax}_i(s_{\phi,i}+u_{\theta,i}) =\operatorname{softmax}_i(s_{\phi,i}) =r_\phi(i).$$ The resulting posterior-matching objective therefore provides zero gradient for the retriever. **Without an alternative training signal, the adapter remains at its initialization, recovering the original pretrained retrieval function.** Thus, “Enc. Pretrained” serves as a functional reference for the configuration without future-aware supervision. **Table 1 shows that future-aware adaptation improves retrieval beyond the pretrained similarity function.** Training the residual adapter with FAR increases PSNR from 16.248 to 16.480, reduces LPIPS from 0.390 to 0.374, and reduces DreamSim from 0.100 to 0.091. Both configurations use the same pretrained visual encoder and retrieval cues, so these improvements demonstrate the benefit of learning predictive relevance without changing the underlying encoder or introducing additional retrieval information. While this comparison does not isolate future-aware supervision against every possible alternative training objective, it provides a direct no-supervision reference for the vision-only setting. The subsequent ablations separately examine the benefits of additional cues and adaptive fusion. We have clarified the residual initialization in Section D.1 (Cue-specific retrieval scores and cacheable representations; lines 1070–1074) and this interpretation of Table 1 in Section 4.4 (Component ablations; lines 511–522) of the revised paper; Section C.3 gives the posterior-matching formulation in Eqs. 13–16 (lines 965–993). ---
AuthorsOpenReview ↗

Reply to Reviewer szGB (Part 2 of 2)


[C3] Improved performance from combining cues does not establish query-dependent adaptation. Provide direct evidence that FAR changes its cue preferences with the prediction query.

We thank the reviewer for this suggestion. To provide direct evidence of query-dependent cue adaptation, we conduct a controlled, paired analysis in AI2-THOR. We hold the query timepoint, initial agent pose, and retrieval memory pool fixed, while varying the requested action: open (interaction without camera movement), stay (no interaction or movement), and turn (180° rotation). We examine the learned visual cue weight λvisual\lambda_{\mathrm{visual}}, with the remaining weight assigned to the metadata cue (time and pose).

The learned cue weights vary systematically with the prediction query, even when the available memories are identical. The mean visual weight is 0.405 for open, compared with 0.316 for stay and 0.287 for turn. The corresponding 10th–90th percentile ranges are [0.368, 0.431], [0.221, 0.388], and [0.244, 0.339], respectively. In particular, open receives a visual weight 0.089 higher than stay and 0.118 higher than turn on average. This pattern is consistent with greater reliance on visual information for interaction-dependent predictions, where historical observations can help distinguish different object states. For example, camera pose and time alone do not indicate whether a container is open or closed, whereas visual observations can reveal its state and, when open, its contents.

These controlled results provide direct evidence that FAR adjusts its cue preferences according to the current prediction query, rather than relying on fixed cue weights. This complements our existing prediction-quality comparisons by directly examining the learned fusion mechanism. We have included this analysis in Section 4.4 (Query-dependent cue adaptation; lines 477–483) and Table 2 of the revised paper, with a supporting qualitative retrieval example in Figure 10.


[C4] How does FAR handle memories that are individually uninformative but jointly useful, given that training evaluates candidate utility independently while inference uses multiple memories?

We thank the reviewer for raising this important point. We agree that memories can be individually uninformative yet become useful when combined. It is important, however, to distinguish FAR's general formulation from the approximation used for efficient retriever training.

In our formulation (Eqs. 2, 4, and 5), predictive utility is defined over the complete recalled context Ct\mathcal{C}_t, rather than individual memories. The world model is also trained and evaluated using the jointly recalled context, allowing it to combine information across multiple memories. The singleton utility and resulting update in Eqs. 14–16 (Section C.3) are a computational approximation for assigning retrieval credit, not an independence assumption imposed on the world model. Nevertheless, this approximation does not explicitly capture interactions between memories that are informative only when selected together, and such combinations may be missed during retrieval.

The challenge is computational: evaluating joint utility over arbitrary sets of KK memories requires considering up to (NK)\binom{N}{K} combinations, and even evaluating all pairs involves O(N2)O(N^2) candidates. One practical approach is to incorporate known temporal structure into individual retrieval candidates. For example, in AI2-THOR-dyn, each recalled context contains information from two timepoints separated by 1.5 seconds, allowing it to capture local motion rather than relying on a single observation. This reduces the number of possible temporally paired candidates to O(N)O(N), avoiding unrestricted pairwise search.

This structured construction captures certain forms of complementary information efficiently, although it does not resolve arbitrary higher-order interactions. Extending the practical retriever to assign predictive credit jointly over multiple selected memories, potentially through structured candidate groups or restricted set-level evaluation, is a natural direction for future work. Section C.3 of the revised paper clarifies the distinction between set-level modeling and candidate-level retrieval credit, including the limitation concerning higher-order interactions (lines 1009–1014); Section 5 acknowledges this limitation (lines 531–534), while Sections 4.3 (lines 463–465) and D.3 (Second-agent supervision in AI2-THOR-dyn; lines 1295–1312) describe the temporally paired context construction.


AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer szGB for the constructive feedback on comparison fairness, predictive supervision, and the retrieval mechanism. Our responses and revised paper address these concerns through the following evidence and clarifications:

  • Matched retrieval pools: Added WorldMem results with the same 1,000-frame pool as FAR (Appendix A.1, Table 3). Increasing WorldMem's pool has negligible effects, while FAR-Meta retains its advantage with matched pool sizes and metadata cue types.
  • Future-aware supervision: Clarified the zero-initialized residual adapter and the pretrained-retrieval reference in Table 1 (Sections 4.4 and D.1). In the vision-only comparison, future-aware adaptation improves PSNR from 16.248 to 16.480 without changing the pretrained encoder or retrieval cues.
  • Query-dependent cue adaptation: Added a paired AI2-THOR analysis with the timepoint, initial pose, and memory pool fixed (Section 4.4, Table 2, Figure 10). Mean visual weights differ across open, stay, and turn queries (0.405, 0.316, and 0.287), accompanied by action-dependent retrieval.
  • Complementary memories and limitations: Clarified that the formulation evaluates complete recalled contexts, whereas efficient training approximates retrieval credit at the candidate level (Appendix C.3). We explicitly acknowledge that this approximation can miss memories useful only in combination (Section 5).

We appreciate the reviewer's time and would welcome any remaining questions about these revisions. Changes in the revised paper are highlighted in blue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Kind Reminder for Comments

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and careful evaluation of our work. We have provided detailed point-by-point responses and updated the manuscript with additional experiments and clarifications addressing the reviewers’ concerns. The key changes are summarized in our individual reviewer responses, and all revisions in the manuscript are highlighted in blue.

We would greatly appreciate the Area Chair’s help in inviting reviewers to engage in further discussion and consider our responses. We welcome any remaining questions and would be happy to provide further clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.