acceptodds
Under review as a conference paper at ICLR 2027

Long-horizon Video Generation with Slot Memory Addressed By Region Tokens

11 views

Abstract

Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on its generative prior elsewhere. On long trajectories in Memory Maze and Minecraft, SMART maintains high-fidelity recall far beyond its training horizon while using constant memory and per-step cost.

Primary area:
generative models
Keywords:
video diffusion models, state space models, long-horizon video generation, spatial memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iSoqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes a method called SMART for long-horizon video generation, which first compresses each frame into multiple region tokens and writes them into a fixed number of slots based on camera parameters. When writing the memory into slots, only the selected slots are updated, while the other slots remain unchanged. When generating new frames, the system reads the memory using the target camera parameters and produces a coarse latent draft for the target view. The generated results are then written back into memory. Experiments on Memory Maze and Minecraft show that the method maintains good reconstruction quality and scene consistency over long videos.

Critical strengths
  1. Slot Memory system greatly reduces the space needed to store memory while still providing good reconstruction quality.

  2. Paper is well written and easy to understand. Besides, the experiments and ablations also validate the effectiveness of the proposed modules and the efficiency of the overall system.

Critical weaknesses
  1. Concepts proposed in this paper are similar to those in existing work in the community, the core idea is to maintain a memory and use camera parameters to retrieve the required historical information. Besides, the paper does not directly compare proposed memory retrieval module with methods such as FOV IoU, the current experiments change several components at the same time, it is unclear how much of the improvement comes from the routing design itself.

  2. All the experiments and visualizations in the paper use Memory Maze and Minecraft scenes ,which are relatively simple, and I am concerned that the method may perform differently in real-world scenes. Can the proposed slot memory recover fine details, such as text and patterns, in real-world scenes or more complex environments? I am also concerned about whether these low-level details are preserved in the slots and can be accurately retrieved.

Questions for authors
  1. See Weaknesses 2.

  2. Because the slot memory has limited capacity, will earlier memories fade after the model has seen many different scenes?

5 comments

We thank the reviewer for the feedback and for noting the memory efficiency and clarity of the paper. We address the concerns below.


Q1. The core idea, maintaining a memory and using camera parameters to retrieve historical information, resembles existing work. What is conceptually new in SMART?

Of course we do not claim that using camera parameters to retrieve past content is new (Section 2). Our contribution is a learned, implicit memory for video, in which the camera determines how the memory is organized. Besides the conceptual novelty, this also brings a clear practical advance over explicit, retrieval-based memories.

In language modeling, implicit memory is well established. State-space models (Gu et al., 2021; Gu & Dao, 2023), gated linear attention (Yang et al., 2023) and slot memories (Zhang et al., 2024; Afzal et al., 2026) compress an unbounded history into a constant-size learned state, where decay by recency and routing by content are natural choices. Video breaks both assumptions, because what must be recalled depends mainly on locality. Current video adaptations of these models (Po et al., 2025; StateSpaceDiffuser) are bounded, but lack selective routing - they treat the camera only as an input, while updating the full state at every step. Yet, naively introducing content-based routing confuses visually similar places (please see our answer to Q2, where content-only keys lose up to 10.1 dB). Our contribution is to introduce selective, pose-based routing for a bounded implicit memory.

Explicit, retrieval-based memories as in WorldMem (Xiao et al., 2026), MosaicMem (Yu et al., 2026), and Lyra 2.0 (Bahmani et al., 2026) store an ever-growing history and use the camera only to select what is read. SMART keeps the bounded, learned state of implicit memory but uses the camera as its address: pose alone decides which slot each region is written to (other slots stay unchanged, so forgetting depends on where, not when); values are re-posed by the target camera; and every read returns a fused draft. Storage is therefore constant, and a memory read fuses multiple observations, rather than copying a best-matching ones. Crucially, despite maintaining strictly constant storage, SMART still outperforms these explicit retrieval methods, because memory read can fuse multiple observations, not just copy a single one.


Q2. How does SMART compare directly with camera-based retrieval such as FOV IoU, and how much of the gain comes from the routing design itself?

Direct comparison with FOV-based retrieval. The comparison to GeoCache (Table 1) implements that comparison in a strong form: following Lyra 2.0, frames are even retrieved such that they cover as many target pixels as possible after reprojection with ground-truth depth, which is a stronger criterion than frustum overlap. At 3,223 frames, SMART exceeds it with 4× less compute, and constant storage. We will better highlight this comparison in the manuscript.

Isolating the routing design. The memory-only ablations change one component at a time: dense vs. routed slots (Table 3), with vs. without SE(3) value transport (Table 3), heads and slots (Tables 3 and 5), and - new in this rebuttal - pose-only vs. joint vs. content-only addressing in the same sparse memory. Starting from our Memory Maze SMART-SM (16 slots, 98,304-float state), we changed only the write key (and the router input that follows it), keeping the architecture, the data and the 100k-step training recipe fixed: (i) pose-only keys (paper), (ii) joint keys with both pose and content based keys, (iii) content-only keys. We evaluate on 250 episodes per length:

FramesWrite keyPSNR ↑SSIM ↑LPIPS ↓
401Pose only (SMART)32.710.9400.024
401Joint32.230.9340.026
401Content only27.730.8800.077
1601Pose only (SMART)32.380.9360.024
1601Joint28.960.8920.059
1601Content only24.160.8250.148
3201Pose only (SMART)31.940.9310.026
3201Joint23.240.8010.172
3201Content only21.850.7760.205

At 3,201 frames horizon, with everything else fixed, content-only keys and joint keys degrade severely relative to pose-only keys. This isolates pose-based routing as the component behind long-horizon retention.

Q3. Can the memory preserve fine details (text, patterns) in real-world scenes?

In line with prior art (Pasukonis et al., 2022; Po et al., 2025; Chandratreya et al., 2026; Xiao et al., 2026), we chose Memory Maze and Minecraft for our experiments, because they provide exact ground-truth poses, enforce long revisits, and come with unambiguous ground truth. They contain some details, but not as many as real-world clips (which are short and include few returns, offering little signal). Minecraft is open-terrain environment of size 256×256, with significant texture. In contrast to the baselines, SMART is able to consistently reconstruct details such as the sun position (Fig. 11). Regarding fine details, SMART is a coarse-to-fine system by design: the memory provides a coarse draft, which the diffusion transformer then refines based on its prior. The fidelity of the draft scales with capacity (Table 5).

We fully agree that real-world scenes are an important next step. For this first study, we had to stick with established benchmarks and concentrate on whether pose-addressed memory makes long-context video generation feasible. Scaling requires substantial compute resources to accommodate for both the memory capacity and the diffusion backbone. We simply did not have the computing resources to do this. We note that nothing in the addressing mechanism is specific to synthetic environments, and hope that future work building on ours will scale pose-addressed memory to real-world scenes.


Q4. Do earlier memories fade after the model has seen many scenes?

Within a scene, forgetting is tied to overwrites rather than to elapsed time: unselected slots receive neither decay nor writes, so a stored region fades when a new observation is routed to the same slot. Table 3 shows this: SMART loses only 0.8 dB from 401 to 3,201 frames, whereas a dense state of the same size loses 11.4 dB.


We thank the reviewer again and hope the new comparisons address the concerns. We will clarify our comparison with a camera-based retrieval method (GeoCache) and the position of SMART as an implicit, pose-addressed memory relative to explicit retrieval-based memories. We will also add the write-key ablation.

Thank you for your response and the additional experiments.

I have also considered the other reviewers' concerns and your responses, and I think some of the concerns have been addressed. However, the ability to faithfully preserve and recall fine details, especially in real-world scenes, remains unclear. It would be grate to include experiments on more complex scenes in the revised version.

For these reasons, I will maintain my original score of 2.

Reviewer_q6gdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SMART, a fixed-capacity recurrent memory for long-horizon video generation. Camera-derived region tokens determine memory addressing and routing, while visual content determines stored values and overwrite strength. Sparse updates preserve unselected slots, and camera-dependent transformations transport stored values between viewpoints. Given a target camera, the memory produces a coarse latent draft and confidence map to condition a chunk-causal diffusion transformer. Experiments on Memory Maze and Minecraft demonstrate improved reconstruction fidelity over long trajectories with bounded recurrent storage.

Critical strengths
  1. The paper addresses a meaningful problem in long-horizon video generation. Using geometry to organize a fixed-size memory is intuitive and well motivated.

  2. SMART achieves strong long-horizon reconstruction with lower reported FLOPs. The results cover two environments with different backbones and resolutions.

  3. The memory-only ablations support the benefits of sparse updates and geometric transport for long-term retention.

Critical weaknesses
  1. The contribution beyond existing sparse memory methods is not clearly established. The ablations do not isolate pose-based addressing, and some baselines lack explicit pose inputs. This makes the source of the reported gains difficult to assess. 
  2. The capacity–retention tradeoff is insufficiently evaluated. Longer trajectories do not establish reliable recall as spatial coverage and slot interference increase, leaving the operating limits of the fixed-size memory unclear.
Questions for authors
  1. A comparison of pose-only, content-based, and joint addressing within the same sparse memory would help clarify the contribution beyond Raven.
  2. The spatial coverage at each evaluation horizon needs clarification. Recall measurements as the explored area grows would help establish the limits of the fixed-size memory.
  3. The pose information available to each baseline needs clarification. Any results with matched pose inputs would help separate the benefit of the memory design from access to explicit poses.
2 comments

We thank the reviewer for the constructive questions, which helped us to elaborate further on our pose-based router design. We address each of the questions below.


Q1. How do pose-only, content-based, and joint addressing compare in the same sparse memory?

We agree that the addressing comparison is a valuable addition to our experiments. As requested, we ran the ablation on router input choice. Starting from our Memory Maze SMART-SM (16 slots, 98,304-float state), we changed only the write key (and the router input that follows it), keeping the architecture, the data and the 100k-step training recipe fixed: (i) pose-only keys (paper), (ii) joint keys with both pose and content, (iii) content-only keys. We evaluate and report SMART-SM on 250 episodes per sequence length:

FramesWrite keyPSNR ↑SSIM ↑LPIPS ↓
401Pose only32.710.9400.024
401Joint32.230.9340.026
401Content only27.730.8800.077
1601Pose only32.380.9360.024
1601Joint28.960.8920.059
1601Content only24.160.8250.148
3201Pose only31.940.9310.026
3201Joint23.240.8010.172
3201Content only21.850.7760.205

Content in the key costs little at the training length but degrades sharply with increasing horizon: joint and content-only keys lose 8.7 dB and 10.1 dB relative to pose-only keys at 3,201 frames. In revisit-heavy environments visually similar views recur at different places, so content keys route different locations into the same slots and overwrite them, whereas pose keys partition the memory by location and keep recall nearly flat. Together with Table 3 (dense vs. sparse at equal size; with vs. without SE(3) value transport), this isolates the components: sparsity alone is not sufficient, pose-based addressing is what makes the sparse memory scale with horizon, and value transport improves quality at every horizon. Beyond addressing, SMART differs from Raven in what the memory is asked to do (App. A.1): random-access reads at arbitrary target cameras (past or future) instead of next-token prediction, read-only queries that don’t modify the state, and a coarse draft as interface to a subsequent generator.


Q2. What is the spatial coverage at each horizon, and how does recall change as the explored area grows?

We agree that this should be shown and add three analyses for horizon and coverage of evaluation episodes.

  • Coverage per horizon. The forward leg is a stochastic breadth-first coverage sweep that heads for the nearest unvisited cell (App. A.2).
Forward leg (frames)40116013201
Visited cells of the 15×15 maze (%)58.285.789.0
Horizons1 - 6464 - 256256 - 768768 - 1792
Visited range of Minecraft scene (%)30.240.364.3100
Frames in the already-visited ground (%)64.451.645.147.0
  • Recall vs. explored area. We finetune the same 16-slot SMART-SM memory (Table 3) for 50k steps on a 31×31 Memory Maze (approximately 4× more cells) and also finetune for 32 memory slots:
Frames15×15 (cells / PSNR)31×31 (cells / PSNR) with 16 memory slots31×31 (PSNR) with 32 memory slots
40184 / 32.7191 / 30.3430.68
1601124 / 32.38328 / 25.9027.72
3201129 / 31.94492 / 24.8927.07

On 15×15 the forward leg saturates the maze by 1,601 frames and recall stays within 0.8 dB. On 31×31 with 32 slots, recall falls by 2.96 dB as coverage grows from 91 to 328 cells, then by only 0.65 dB from 328 to 492 cells, so the store degrades gracefully beyond its capacity instead of collapsing. The operating limit moves with capacity: doubling the slots from 16 to 32 adds +0.3 dB at 401 frames but +2.2 dB at 3,201 frames, consistent with Table 5 in the paper on 15×15 (+0.3 dB vs. +1.3 dB).

Q3. Which pose inputs does each baseline receive, and how do the results change with matched pose inputs?

We clarify the inputs below and will add this table to the paper:

DatasetMethodCamera pose inputOther geometry
Memory MazeCausal (100F / Full)pose, PRoPE (same as SMART)–
Memory MazeDFoTpose, PRoPE (same as SMART)–
Memory MazeGeoCache (Lyra 2.0)pose (same as Lyra2.0)GT simulator depth
Memory MazeSMARTpose–

On Memory Maze, all baselines in Table 1 receive the same GT poses as SMART, and GeoCache additionally receives GT depth, so the gains there are not explained by pose access. Within our own model, the dense-memory ablation (Table 3) and the content-key ablation (see answer for Q1) have identical pose inputs and still collapse beyond the training horizon, showing that it matters how the pose is used (as the address), not merely whether it is available. On Minecraft, we aimed to match the evaluation protocol to MilliVid, from the training dataset to the DiT architecture. For that reason, none of the DiT models in our Minecraft evaluations include camera conditioning.


We thank the reviewer again and hope these results clarify the contribution. We are grateful for their point on clarification of input pose information to the models and we will make this clarification in the paper as well.

Reviewer_rCFLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SMART, a slot memory addressed by region tokens for long-horizon video generation. The memory is a fixed-size gated slot recurrence, where the write keys, read queries and routing are computed only from the camera pose, and the values are computed from the frame content and transformed into a shared frame with the writing camera. Each write only updates the slot that matches its pose, so old content does not decay with time. When queried with a target camera, the memory returns a coarse draft latent and a confidence map, which are concatenated to the input of a chunk-causal diffusion transformer. Experiments on memory maze and minecraft show better long-horizon recall than dfot, a lyra 2 style geometry cache and millivid, with constant memory and lower compute.

Critical strengths
  • Decoupling the memory addressing (pose) from the stored content (frame) is a clean and interesting idea, and makes the forgetting depend on where rather than when.

  • The ablations and analysis are convincing, e.g., dense memory with the same size collapses beyond the training horizon while the slot memory holds, and Fig. 7 shows the slots are really organized by position.

  • Long-horizon recall far beyond the training length, with a constant memory size and much lower compute than the baselines.

Critical weaknesses
  • It would be nice to also compare with long-context state-space video world models (po et al.), which is cited and uses a very similar memory maze / minecraft setting and model size. Currently the recurrent memory baseline is a statespacediffuser reimplementation in Tab. 6, which is action-driven and only on short trajectories. If the authors could reproduce po et al. under the same setting, it would better show the benefit of pose addressed slots over temporal recurrence. Also the geocache baseline with GT depth looks a bit weak for an explicit memory in a static maze, more details on how it is trained and run would help.

  • The memory has a fixed capacity and the slots are addressed by absolute pose, so in larger or open-ended scenes different regions have to share slots and overwrite each other, which is also mentioned in the limitations. The current scenes are quite small with GT poses from the simulator. Some analysis on how the recall changes with scene size, and how robust it is to noisy poses, would make the claim of long-horizon generation more convincing

Questions for authors

Seek weakness. Overall I'm positive

4 comments

We thank the reviewer for the positive assessment and for highlighting the decoupling of addressing and content, the dense-vs-slot ablation, and the slot organization in Fig. 7. We address the remaining concerns below. We thank the reviewer for suggesting the scene-size analysis, which we have added, see Q3.


Q1. How does SMART compare with Po et al. (2025)?

We agree that Po et al. (2025) is the closest implicit-memory approach to ours, and we did position SMART relative to it in the paper. We tried to include it as a baseline, and to keep settings close to it, e.g., our Memory Maze generator already follows its model size (250M, App. A.3). However, to our knowledge no code or checkpoints are publicly available. We contacted the authors for further details before the submission deadline but could not obtain code, weights or data from them. We attempted a reimplementation, but were not able to achieve comparable performance, so we prefer not to report numbers that might misrepresent the method.

Instead, we isolate the mechanism that distinguishes the two designs under controlled conditions: a recurrent state that compresses along time and receives the camera as conditioning input, vs. a state that is addressed by the camera. In Table 3, SMART-Dense is SMART with the routed slots replaced by a dense state of the same size, and with identical pose inputs. It matches SMART at the training length (32.9 vs. 32.7 dB) but drops to 21.5 dB at 3,201 frames, while SMART stays at 31.9 dB.

When compressing recurrences through time with SMART-Dense and StateSpaceDiffusers, every step updates the full state and the camera enters as conditioning, so retention depends on elapsed time. In SMART the pose is the address, and unselected slots pass through a step unchanged (Eq. 3). We will make this discussion explicit in the related work.

Q2. GeoCache looks weak. How is it trained and run?

Thank you for your attention and indeed this was something we were puzzled by as well. GeoCache strictly followed the official Lyra 2.0 repository’s model implementation and training (App. A.5). We made further efforts and configuration changes to improve the implementation by increasing the spatial slots from 5 retrieved frames to 9 frames while also seeding the sparse 3D cache with the full forward leg and avoiding the default spatial downsampling of FramePack on the low-resolution data. We further notice that training in mixed precision leads to inferior results when the public training code run as released. As a result of further training the model at higher precision and gathering all frames to the sparse 3D cache instead of subsampling them, we observe that GeoCache achieves better results, yet still inferior to our method.

MethodPFLOPs↓PSNR↑SSIM↑LPIPS↓FID↓FVD↓PFLOPs↓PSNR↑SSIM↑LPIPS↓FID↓FVD↓
(201+21)+201 = 423 frames(401+21)+401 = 823 frames
Causal (100F)0.09625.00.8090.1316.394.060.19321.80.7390.1988.857.19
Causal (Full)0.11730.00.8910.034‾\underline{0.034}5.36‾\underline{5.36}2.13‾\underline{2.13}0.29924.30.7890.1007.964.61
GeoCache (GT-D)0.45426.30.8430.0756.643.580.93626.30.8430.072‾\underline{0.072}6.292.43
DFoT0.56632.00.9190.0305.741.891.13028.4‾\underline{28.4}0.865‾\underline{0.865}0.0795.54‾\underline{5.54}1.89‾\underline{1.89}
Ours0.108‾\underline{0.108}31.1‾\underline{31.1}0.9190.0405.222.250.213‾\underline{0.213}29.90.9000.0465.081.83
MethodPFLOPs↓PSNR↑SSIM↑LPIPS↓FID↓FVD↓PFLOPs↓PSNR↑SSIM↑LPIPS↓FID↓FVD↓
(801+21)+801 = 1623 frames(1601+21)+1601 = 3223 frames
Causal (100F)0.39420.40.7090.22710.38.880.82419.10.6770.26611.611.1
Causal (Full)0.85020.80.7180.21047.432.62.72718.30.6690.351158.5199.6
GeoCache (GT-D)1.87227.1‾\underline{27.1}0.853‾\underline{0.853}0.060‾\underline{0.060}5.981.873.77227.4‾\underline{27.4}0.857‾\underline{0.857}0.0555.881.73
DFoT2.25926.60.8380.1035.48‾\underline{5.48}1.604.51824.00.7950.1475.72‾\underline{5.72}1.80
Ours0.430‾\underline{0.430}29.30.8900.0515.031.66‾\underline{1.66}0.900‾\underline{0.900}27.60.8630.070‾\underline{0.070}5.001.76‾\underline{1.76}

GeoCache behaves as expected from an explicit memory: it does not degrade with horizon (PSNR 26.3 → 27.4 from 423 to 3,223 frames). With camera-based retrieval and conditioning, the DiT has to learn the association of the frames which is a challenging task which explains its inferior quality. We will open-source our training code for the baselines as well as the checkpoints and we hope that the updated results clarify our GeoCache baseline.

Q3. With fixed capacity, how does recall change with larger scene?

We agree that this is indeed the central trade-off, which we acknowledge in the Limitations. We therefore test the memory on a 4× larger maze and vary the number of slots. As expected, larger memory can handle larger scene sizes (recall and capacity interact as the explored area grows).

We finetune the SMART-SM memory (memory-only retrieval as in Table 3) for 50k steps on a 31×31 Memory Maze, with 16 and with 32 slots, and evaluate on 250 episodes per horizon. Visited cells denotes the mean number of distinct cells the forward leg visits.

MazeModelFramesVisited cellsPSNR ↑SSIM ↑LPIPS ↓
15×15SMART-SM (16 slots)4018432.710.9400.024
15×15SMART-SM (16 slots)160112432.380.9360.024
15×15SMART-SM (16 slots)320112931.940.9310.026
31×31SMART-SM (16 slots)4019130.340.9160.044
31×31SMART-SM (32 slots)4019130.680.9200.041
31×31SMART-SM (16 slots)160132825.900.8430.113
31×31SMART-SM (32 slots)160132827.720.8750.081
31×31SMART-SM (16 slots)320149224.890.8230.136
31×31SMART-SM (32 slots)320149227.070.8620.096

SMART works well in the 4× larger maze. On 15×15, the forward leg recall stays within 0.8 dB across horizons. On 31×31 with 32 slots, SMART retains 27.1 dB after 3,201 frames over 492 visited cells, 3.4× the open area of the entire 15×15 maze. From 1,601 to 3,201 frames, the explored area grows by 50%, yet recall drops by only 0.65 dB. Even with the original 16 slots, recall degrades gracefully and does not collapse. Capacity sets the operating limit: doubling the slots gains +0.3 dB at 401 frames, +1.8 dB at 1,601 and +2.2 dB at 3,201, consistent with the 15×15 trend in Table 5 (+0.3 vs. +1.3 dB). Capacity is a free parameter of SMART: the paper fixes the state to 98,304 floats only to match the MilliVid/FramePack context budget, but the slot budget can be adjusted to the expected scene size. We will add this study and the measured operating range to the paper.


Q4. How robust is SMART to noisy poses?

Thank you for pointing this out. We have tested the memory module (memory-only retrieval as in Table 3), where the pose acts as the address. The poses of all written frames are perturbed with i.i.d. Gaussian noise on camera position and yaw, with the per-episode RMS set to the target translation error and rotation error. We compare the impact of noise on Memory Maze’s 401-frame retrieval evaluation dataset where exact-pose baseline achieves 32.63 dB. The agent moves ≈0.25 cells per frame and turns are 18.7° on average.

Position error (% distance of one frame step)4%8%12%16%20%
PSNR loss (dB)0.030.130.280.460.69
Rotation error (% of one frame's turn angle)1.3%2.7%4%5.3%8%
PSNR loss (dB)0.020.100.230.400.77

Recall degrades smoothly and about linearly, without sudden collapse: within 0.5 dB up to 16% of a frame step or 5.3% of a camera turn. This is without any pose-noise augmentation during training, which should improve robustness further. Some sensitivity is also expected by design as the pose specifies the output view and the memory should not be invariant to it. We will add this study to the appendix.


We thank the reviewer again and are happy to discuss further.

Thanks for the response and additional experiments. My concerns are further addressed. It would also be good to also discuss pose drift over time in the limitations, as the noise study only uses small i.i.d. noise. I remain positive and will keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their feedback and the detailed evaluation of our work. Reviewers found the decoupling of pose-derived addressing from content-derived values "clean and interesting" (rCFL) and "intuitive and well motivated" (q6gd), the ablations "convincing" (rCFL) and supportive of each module (q6gd, iSoq), and noted strong long-horizon recall with constant memory and lower compute (rCFL, q6gd, iSoq).


During the rebuttal, we added new experiments and several clarifications, summarized below.

1. Source of the gains and isolating pose-based addressing (q6gd, iSoq). In a new ablation inside the same memory architecture, we show that joint and content-only keys degrade severely compared to pose-only keys at 3,201 frames, while pose-only keys lose only 0.8 dB. Together with Table 3 (dense vs. routed slots at equal size and with vs. without SE(3) transport), this isolates our design choices for the pose-addressed memory.

2. Analysis on scene size and capacity (rCFL, q6gd, iSoq). We report spatial coverage per horizon on both datasets and evaluate the memory on a 4× larger 31×31 maze. With 32 slots, SMART retains 27.1 dB after 3,201 frames over 3.4× the open area of the entire 15×15 maze. Doubling the memory slots adds up to 2.2 dB, so the operating range moves with capacity.

3. Analysis on robustness to pose noise (rCFL). Without any noise augmentation in training, recall stays within 0.5 dB under position errors up to 16% of a frame step or rotation errors up to 5.3% of a camera turn. Some sensitivity is also expected by design as the pose specifies the output view and the memory should not be invariant to it.

4. Inquiries on baselines (rCFL, q6gd, iSoq). We clarify the pose inputs of all baselines, explain why Po et al. (2025) could not be reproduced, and detail our strengthened GeoCache implementation, with all baseline details in the revised appendix.

5. Real-world scenes (iSoq). Preserving fine detail is important, but it is a complementary problem. In line with recent practice (Po et al., 2025; Chandratreya et al., 2026; Xiao et al., 2026), we evaluate on Memory Maze and Minecraft, the standard datasets for long-horizon memory. Transferring to large-scale real data would require over 5,000 H100 GPU-hours per training run which is a separate effort beyond the scope of this principled paper.


Revisions. The revised paper is already uploaded with all changes in blue color. In the main paper, we explicitly state our contributions in more detail, update our experiments and ablations according to (1) and (4). We include further details on (2), (3), and (4) in our appendix.

We thank the reviewers again for their insightful feedback and the constructive discussion. We hope that our answers and the revised paper address their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.