- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a retrieval augmented method in the context of world models (adapting the EMDR2 framework, and adding an adaptive cue fusion). Historical observations (visual, sound, ... at each time stamp) are saved in an external memory. The main goal of the proposed method is to learn a retriever that extracts a top-K subset of observations maximizing the predictive utility (right term of Eq. 5) together with the current scoring preferences (left term of Eq. 5). The predictive utility is only known at training time (given the realized future), so at train time a (blind) scorer is learnt for approaching it (through Eqs. 5, 6 that adapts EMDR2). The importance of each cue is adapted for each query through a cue-fusion mechanism. The scorer and the world model are trained jointly.
The method is applied in 3 world-modeling settings: LoopNav (Minecraft visual + metadata), SoundSpaces (indoor navigation + sound), and AI2-Thor (interaction in environments), and is compared against most-recent observations (temporal recency), WorldMem (taking the top-K observations with high FOV overlap with the goal), and LongLive-RAG (top-K based on embedding similarity), showing improvement over those baselines on multiple metrics (e.g. correct prediction of future state, PSNR).
Critical strengths
- The paper is relatively clear to read, regarding the motivation and the methodology; the companion website was also introducing the topic and the proposed method well
- Compared to the proposed baselines, the capability of FAR is clear
Critical weaknesses
- The baselines are weak w.r.t. the evaluated world-modeling settings: the temporal recency is the simplest possible baseline (this method is not appearing in Table 2) and the other two: WorldMem (high FOV overlap with the goal) and LongLive-RAG (based on embedding similarity) are not considering the audio cue. One reason might be that the authors decided to compare against external retriever methods only.
Questions for authors
Experimental setting (K and chunking)
Baselines (already stated in the weaknesses above)
- In SoundSpaces the sound cue is critical, and is not present for the other compared methods, giving an unfair advantage to the proposed methodology. For AI2-THOR, this is also apparent in Fig. 9, since the constant guess is better compared to both Temporal and WorldMem baselines. Table 2 doesn't show any method managing the audio cue. I suggest adding other baselines, including multi-cue ones, even if there are outside the external retriever setting.
Selection of K
- The value of K is not detailed. It might be corresponding to what is stated in l.1205, but I'm not sure. The typical value of K and the total number of observations should appear in the main document.
- What is typical number of observations after chunking (is it appearing on Table 5?)?
- The following question is: What is the performance using the proposed method with all observations (so following the whole method, without taking the top-K at the end).
Training computation
- One stated limitation of the proposed method is the "additional training-time" l.482, but there is no quantification of this time in the paper.
- The predictive utility is learnt for each setting, that might be very costly.
Relation with EMDR2
- What is the main difference with EMDR2, apart from the cue fusion? I didn't put this into critical weaknesses, since I think there is substantial adaptation done, but the contribution could be better highlighted.
Minor comments
- In l.240: the relevance is judged based on each cue alone before fusion. There are settings in which the visual/sound interaction is critical. This would be interesting to consider as a future work, but it is not affecting my assessment.
- In Tab. 1, why lambda=0.5 in numerical? Shouldn't lambda be a vector combining the different cues? What is the learnt value for SoundSpace in particular?
- Regarding the predictive utility, ideally a single foundational trained model could be able to have a sense of it.
6 comments
Reply to Reviewer Mhog (Part 2 of 3)
[C3] State the typical value of , the total number of historical observations, and the number of candidates after chunking explicitly in the main text.
We thank the reviewer for pointing out the lack of clarity. The value of and the total number of historical observations are detailed in Table 10, Section D.4 of the revised paper. Specifically, is determined by the context size as , since one slot is reserved for the current observation. We have clarified this relationship in Section 3.3 (Temporally structured recall; lines 284–289) of the revised paper: the four rollout benchmarks use , selecting from up to 100 representatives obtained by dividing a 1,000-frame pool into chunks of 10 frames. Table 10 in Section D.4 provides the dataset-specific configurations.
[C4] Report performance when using all observations retained by the chunking procedure, without the final Top-K selection.
We refer the reviewer to Section A.2 (Table 4) of the revised paper, where we compare selecting the top 11 contexts from 100 temporal chunks against using all 100 chunk representatives. Using all representatives yields PSNR/LPIPS/DreamSim of 17.184/0.342/0.078, compared with 17.267/0.337/0.082 for Top-K selection. The small differences indicate comparable prediction quality while Top-K selection reduces the number of recalled contexts from 100 to 11.
[C5] Quantify the additional training cost of setting-specific predictive-utility learning.
We refer the reviewer to Section A.3 (Table 5) of the revised paper, where we quantify end-to-end training cost on LoopNav using four H100-equivalent GPUs. For 700K steps, Temporal requires approximately 130 hours, while FAR-Meta, FAR-Visual, and FAR-Multi-Cue require 141, 141, and 142 hours, respectively. This corresponds to approximately 8–9% additional training time, including predictive-utility evaluation and retriever optimization; WorldMem requires approximately 175 hours in the same setting, reflecting the computational overhead of its iterative greedy FOV-based memory selection. Thus, despite its additional predictive-utility evaluation and retriever optimization, FAR trains faster than WorldMem in this comparison.
[C6] Clarify FAR's theoretical and practical differences from EMDR2 beyond cue fusion.
Theoretical distinction. We thank the reviewer for this suggestion. As shown in Appendix C.2, both EMDR2 and FAR arise from the same latent-variable maximum-likelihood principle, where downstream predictive likelihood provides supervision for discrete retrieval. However, FAR extends the retriever parameterization to incorporate multiple cue-specific relevance functions and query-dependent fusion weights. As derived in Appendix C.3, although the underlying posterior-matching objective retains the EMDR2-style gradient, FAR's gradient decomposes into two learning pathways: one optimizing relevance scores within each cue and another optimizing cue weights based on their contribution to predictive utility. Thus, future-aware supervision learns not only which memories to retrieve, but also which cues to trust for each query.
Practical adaptations. FAR further adapts this framework to video diffusion world models through several design choices: (1) approximating predictive utility using the negative diffusion prediction loss of the observed future, rather than reader likelihood for question answering; (2) contrastive pretraining of retrieval encoders with JEPA-style representations, combined with frozen, cacheable memory embeddings and lightweight trainable query adapters; and (3) temporally chunked Top-K retrieval to reduce redundant selection from long observation histories. Together with adaptive multi-cue fusion, these adaptations enable predictive-utility-based retrieval from evolving multimodal episodic memories while preserving the world model's conditioning interface. The revised paper details the latent-variable formulation in Section C.2 and the two learning pathways in Section C.3. Sections 3.2–3.3 and D.1 describe adaptive cue fusion, the diffusion-loss utility surrogate, pretrained cacheable representations, and temporally chunked Top-K retrieval.
Reply to Reviewer Mhog (Part 3 of 3)
[C7] Fusion weights should form a vector with one scalar per cue, so clarify the scalar notation in Table 1. Report the learned SoundSpaces weights.
We thank the reviewer for pointing out the ambiguous notation. The fusion weight is a vector with one scalar weight per retrieval cue, as defined in Eq. (8). In Table 1, the uniform-fusion baseline uses , assigning equal weight to metadata and visual relevance scores without a prior preference for either cue. We have revised the uniform-fusion notation in Table 1 and explained the equal-weight vector in Section 4.4 of the revised paper (lines 515–518).
For SoundSpaces, the learned weights after convergence average approximately 0.95 for metadata (pose and time) and 0.05 for sound. These are averages of query-dependent weights, rather than a single fixed mixture shared across all queries. The strong metadata preference suggests that pose and time provide the primary retrieval signal, while sound serves as a complementary tie-breaking cue when metadata assigns similar relevance to multiple context frames. We have added these average weights and their interpretation in Section 4.2 of the revised paper (lines 373–375).
[C8] Could a general-purpose pretrained foundation model estimate predictive utility without setting-specific training?
Thank you for the suggestion. We agree that a pretrained foundation model could potentially estimate predictive utility across environments, but our current design has two motivations.
First, semantic relevance is distinct from downstream predictive utility: a memory that appears relevant to a foundation model may not reduce the particular world model's prediction error. For example, a visually similar view of a closed refrigerator may be less useful for predicting an opening action than an older observation revealing its contents. FAR uses the downstream model's prediction loss to learn which memories actually help that model under its conditioning interface.
Second, we seek a controlled comparison of retrieval strategies: introducing an additional foundation-model utility estimator would also introduce external pretrained knowledge and additional computation, making it harder to attribute gains to the retrieval mechanism itself. Our current approach therefore learns predictive relevance from the shared downstream world model's feedback. A transferable foundation-model estimator remains a promising extension, provided its alignment with downstream predictive utility and its additional computational cost are evaluated explicitly.
[C9] Independent cue scoring may miss jointly informative visual–audio interactions. Discuss joint cue modeling as a future direction.
We agree that independently scoring each cue before fusion may not fully capture interactions between modalities, such as cases where visual and audio observations are jointly informative despite being individually ambiguous. Modeling such interactions through joint cue-specific scorers is an interesting extension of FAR. However, evaluating their benefits also requires benchmarks where cross-modal interactions are essential for long-horizon prediction. While audio-visual navigation datasets exist, benchmarks combining multimodal observations, loop closure, and history-dependent future prediction remain relatively limited. Our SoundSpaces experiments provide a controlled evaluation of complementary retrieval cues, but do not explicitly isolate nonlinear cross-modal interactions. We acknowledge this limitation and leave joint cue modeling and its evaluation on richer multimodal memory benchmarks to future work.
Reply to Reviewer Mhog (Part 1 of 3)
We thank the reviewer for investing time in assessing our approach and for the helpful suggestions on baselines, computational cost, and multimodal retrieval. Revisions in the paper are highlighted in blue.
[C1] The baseline set is limited to recency-, FOV-, and embedding-similarity-based retrieval, and Temporal is absent from Table 2. Include stronger memory baselines beyond external retrievers.
We refer the reviewer to Section 4.4 (Additional baseline comparisons; lines 500–509) and Table 1 of the revised paper, where we add comparisons with MemLearner and MASt3R-SLAM on LoopNav alongside Temporal and the existing retrieval baselines. We have also added Temporal (Navigation World Model) to the conceptual comparison in Table 6 (Appendix B), explicitly identifying its time-based retrieval rule. FAR-Multi-Cue outperforms both additional methods across all three metrics; relative to MemLearner, it improves PSNR by 2.674 and reduces LPIPS and DreamSim by 29.9% and 36.9%, respectively.
[C2] Additional audio or agent-state cues may give FAR an unfair advantage over the baselines, particularly in SoundSpaces and AI2-THOR-dyn. Include stronger multi-cue baselines, including methods outside the external-retriever setting.
We thank the reviewer for this suggestion. We agree that the benefit of additional retrieval cues should be distinguished from that of learning predictive relevance. Our experiments explicitly examine both aspects, including comparisons where the available retrieval cues are matched.
First, FAR-Meta outperforms WorldMem using only the same metadata cue types (time and pose), without relying on audio or historical agent-state retrieval cues. On SoundSpaces, even with matched 1,000-frame retrieval pools, FAR-Meta achieves 17.295 versus 14.339 PSNR on SoundSpaces (endpoint scans) and 18.659 versus 13.979 on SoundSpaces (periodic scans) (Section A.1, Table 3). Similarly, on AI2-THOR-dyn (Section 4.3, Figure 9), FAR-Meta achieves 66.7% accuracy, compared with 32.5% for WorldMem and 41.8% for Temporal, also exceeding the 50% constant-guess reference. These comparisons show that FAR's improvements are not simply attributable to access to additional modalities.
Second, FAR-Multi-Cue addresses the complementary question of how to effectively exploit multiple cues available in the interaction history. Rather than imposing handcrafted rules for combining modalities or prescribing which cue should be preferred, FAR learns both cue-specific relevance and query-dependent cue weights through future-aware predictive supervision (Section 3.2). Thus, the multi-cue experiments demonstrate how FAR can learn to exploit complementary historical information, while the same-cue comparisons establish its advantage without relying on additional retrieval modalities.
Regarding additional multi-cue baselines, existing approaches are designed around specific retrieval criteria, such as temporal recency, FOV overlap, or visual similarity. Incorporating audio is therefore not a direct extension of these methods, as it requires defining new relevance functions and mechanisms for combining heterogeneous cues that are not specified by the original approaches. Learning how to combine such cues, rather than manually prescribing their relative importance, is precisely one of FAR's central motivations. We therefore prioritize controlled same-cue comparisons while separately evaluating the benefits of learned multi-cue retrieval. The revised paper describes learned cue fusion in Section 3.2, presents matched-pool comparisons in Section A.1 (Table 3), and clarifies the shared AI2-THOR-dyn components and the distinction between retrieval cues and generator conditioning in Sections 4.3 and D.3–D.4. We have revised the quantitative-results paragraph of Section 4.2 (lines 366–368) to explicitly clarify that FAR-Meta and WorldMem are compared with available retrieval cues restricted to metadata (time and pose), while FAR-Multi-Cue separately evaluates the benefit of making audio available for retrieval.