acceptodds
Under review as a conference paper at ICLR 2027

Visual Memory Fields: A Brain-Inspired Spatial Scaffold for Embodied Memory

Abstract

Embodied agents need memory that can integrate growing streams of local, viewpoint-dependent observations into a persistent representation under finite capacity. Existing systems often trade off bounded storage against long-horizon retention and flexible reuse. Inspired by hippocampal-entorhinal systems, we propose **Visual Memory Fields**, which bind episode-specific sensory content to a reusable spatial scaffold represented as a continuously addressable field over position and heading. We instantiate this scaffold with a sparse Fourier basis shared across different maze layouts, and update episode-specific content online through compact sufficient statistics rather than storing observation history. A fixed Fourier support keeps the sufficient statistics—and therefore persistent memory—independent of experience length, while budget-constrained search allocates the available capacity across pose frequencies. This bounded representation can then be queried and reused after accumulation, rather than being tied to the objective under which experience was collected. We test this property in Memory Maze, first through view-centric recall over long histories and randomly withheld views, and then through scene-centric object localization using a separately supervised reader on the frozen field. We further show that the resulting scene belief can be re-expressed in candidate action frames for downstream control. Together, these results demonstrate **Visual Memory Fields** as a persistent, bounded, and reusable form of scaffolded embodied memory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spatial Scaffolding, Episodic Visual Memory, Embodied Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pX21OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

I used LLM assistance primarily to polish the language and improve the clarity of my review. The critical assessments, arguments, and overall recommendation originated from my independent reading and judgment.

Critical strengths
  1. The paper addresses the challenge of maintaining bounded, persistent visual memory that can support multiple downstream tasks. The separation of shared spatial structure from episode-specific content provides a reasonable and interpretable memory design.

  2. The Fourier-based sufficient-statistic representation provides measurable storage savings while preserving long-term visual information. The experiments demonstrate improvements in visual recall and object localization, with additional evidence of benefits for downstream navigation.

Critical weaknesses

Limited generalization and incomplete efficiency evaluation. All experiments are conducted in Memory Maze, a simplified environment with known agent poses and largely static scenes. It remains unclear whether VMF scales to more complex, dynamic, or realistic embodied environments. Furthermore, while the paper demonstrates reduced persistent storage, it lacks a comprehensive memory–accuracy–compute trade-off analysis, particularly regarding retrieval latency and computational scalability.

Questions for authors

Can the authors demonstrate that VMF's advantages persist in more complex and realistic environments, particularly under matched memory–accuracy–compute budgets?

Reviewer_yHHrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Visual Memory Field (VMF) that binds episodic-specific sensory content to a reusable spatial scaffold. The model is evaluated in Memory Maze (Pasukonis et al., 2023).

Critical strengths

The idea of having a model that separates episodic observation from a reusable relational structure is interesting and a relevant question (Whittington et al., 2020; Chandra et al., 2025). Unfortunately, the contributions of this paper cannot be properly assessed because of the weaknesses listed below.

Critical weaknesses
  1. The work is not situated within the existing literature

The introduction mostly summarize the paper rather than positioning it. After reading it, a reader could not determine which prior approaches the work builds on, how it differs from them, or which gap this paper addresses.

The motivation presents VMF as a solution to a dichotomy between a memory that grows and a bounded recurrent states shaped by its training readout. No specific work is given as example for either category, and, more importantly, this framing omits a substantial body of work between these two extremes. Without engaging with previous work, the motivation for the proposed model is unclear.

The related work section does not resolve this. It briefly lists prior models, but does not state their limitations or explain how VMF is an improvement. For example, the sentence "VMF turns this organization into a supplied-pose Fourier kernel with episode-wise regression" describe VMW, but does not explain what problem it solves. The extended related work section in Appendix B mostly restates section 2, adding little clarification.

  1. The methods are difficult to follow and insufficiently supported

Figure 1 (the main figure going over the model) has a brief caption that does not explain the model or the notations. Moreover, the main text never discusses, explains, or refers to any of the individual panels. In its current form, the figure does not help the reader understand the method.

Several claims are stated without explanation or support. For example, "Orthonormal column pairs in V make Q(θ)Q(θ′)=Q(θ+θ′)Q(\theta)Q(\theta ') = Q(\theta + \theta ') and Q(θ)T=Q(−θ)Q(\theta)^{T} = Q(- \theta). Thus alignment handles predictable latent rotation, while mixed address modes retain viewpoint-dependent content.”.

Several components are used without being defined. For example,

a) "A frozen codec maps image o_t to normalized latent z_t = E(o_t)". Function E is never defined, described only by its output dimensionality (256) and by the fact it is pretrained (Appendix A.1). It is unknown what the function is and what the normalization mentioned is.

b) The function D in equation (4) is not defined in the main text. The only information, found in Appendix A.1, is that it is a decoder going from "latents back to images".

c) In section 3.5, a function R appears without definition or justification, and is never mentioned in the main text again.

The appendix provides limited support. For example, Appendix C states multiple propositions with brief or missing arguments:

  • The full proof of Proposition 1 reads: "Proof. Apply cos(a)cos(b)+sin(a)sin(b)=cos(a−b)cos(a) cos(b) + sin(a) sin(b) = cos(a − b), integer periodicity, and the angle-addition identities."
  • Proposition 3 reads: "The Sherman–Morrison identity applied to A+xxTA + xx^T yields Eq. equation ??."
Questions for authors

In its current form, the paper is not in a state where its contribution can be evaluated. The methods rely on undefined components and on claims that are not supported. The introduction and related work do not situate the work or identify the gap the paper addresses.

Reviewer_vvgxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Visual Memory Fields (VMF), a visual memory architecture for embodied agents, aiming to address long-term visual experience retention and reuse under limited storage capacity. The method employs a shared Fourier-based spatial scaffold to encode position and heading, stores visual information through online-updated sufficient statistics, and optimizes the frequency set under a storage budget to achieve a fixed-capacity, continuously queryable memory representation. Experiments in the Memory Maze environment cover long-horizon visual recall, reconstruction of unobserved viewpoints, object localization, and downstream navigation control, demonstrating the potential of the proposed architecture for visual information retention and task reuse.

Critical strengths
  1. The research problem is meaningful and well motivated. This paper addresses how embodied agents can retain long-term visual experiences under limited storage resources while supporting queries from different downstream tasks. The motivation is clear, and the separation between the spatial scaffold and task-specific readers provides an interesting design perspective.

  2. The method offers certain technical contributions. The authors exploit the algebraic structure of Fourier features to compress regression statistics and optimize the spatial frequency set under a storage budget. The fixed-capacity property is mathematically justified, and the structural ablation studies support the effectiveness of frequency selection and heading alignment.

  3. The experimental evaluation is relatively comprehensive. Beyond long-horizon visual recall and unobserved-view reconstruction, the paper evaluates downstream applications through object localization and navigation experiments. In particular, comparisons involving full-history memory, recent-window memory, and geometric information provide useful evidence of the importance of long-term visual information for object localization.

Critical weaknesses
  1. The source of the downstream control improvements remains unclear. In the navigation experiments, Global not only employs an object-location reader trained with additional supervision but also modifies the motion prediction inputs for candidate actions. Therefore, its substantial performance improvements over Local and PPO may result from more direct access to target-location information and a different action prediction mechanism, rather than solely from the VMF memory architecture. A direct comparison that replaces different memory methods within an otherwise identical control framework is currently missing. Additionally, some mathematical expressions in Figure 1 appear to contain garbled characters.

  2. The experimental validation of storage efficiency remains insufficient. Although the paper theoretically establishes that memory capacity does not grow with history length and reports storage savings compared with a conventional Gram matrix, the experiments mainly focus on specific storage budgets. Comparisons of reconstruction accuracy versus memory consumption across different capacity constraints are lacking. Furthermore, the computational costs associated with online memory updates, memory queries, and downstream readers are not sufficiently analyzed, making it difficult to comprehensively assess the resource efficiency of the proposed method.

  3. The advantages over existing memory methods are not entirely consistent. Although VMF achieves competitive results on the main visual reconstruction metrics, Matched-Q Fourier still outperforms VMF on certain object localization metrics in the Paired-64 setting. Moreover, the standard SMT configuration in the appendix performs better than its matched-Q counterpart. These observations require further explanation to distinguish the benefits of the proposed architecture from the effects of training configurations.

  4. Generalization and applicability remain somewhat limited. All experiments are conducted in the Memory Maze environment and assume access to accurate agent poses. Experiments involving larger mazes also require refitting the spatial scaffold. Therefore, the current results primarily establish the effectiveness of the method in a specific simulated environment, while its robustness to pose estimation errors, applicability to dynamic environments, and transferability across different scene scales remain insufficiently investigated.

Questions for authors
  1. Could the authors replace VMF with Fourier memory or other existing memory methods while keeping the object-location reader, control policy architecture, and candidate action prediction process consistent? Such a comparison would help isolate the contribution of VMF itself to navigation performance.

  2. Could the authors provide comparisons of visual reconstruction accuracy, object localization performance, and computational costs under different storage budgets? In particular, does VMF maintain consistent advantages from frequency optimization and compressed storage across different memory capacity constraints?

  3. How do the authors explain the superior performance of Matched-Q Fourier on certain object localization metrics in Paired-64, as well as the substantial performance degradation of SMT under the matched-Q setting? Could these results indicate sensitivity to baseline configurations or training hyperparameters?

  4. How sensitive is the proposed method to pose estimation errors? Since spatial addressing directly depends on the input poses, localization errors during real-world operation may lead to inconsistencies between memory writing and retrieval. Could the authors discuss their potential impact and the corresponding limitations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.