- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Learning-to-Retrieve, a method that replaces selected softmax attention layers in a pretrained video diffusion Transformer with gated linear attention layers (GLA or GDN), using their recurrent states as internal memory. Building on this architecture, the authors introduce two key components: (1) a retrieval gate modulated by camera Plucker rays, which selectively retains historical information in the recurrent states on a per-channel basis; and (2) a binary retrieval trigger, which determines when to activate retrieval mode. During training, it is supervised using 3D revisitability labels computed offline through depth estimation, back-projection, and reprojection. As a result, inference requires neither an external memory bank nor 3D conditioning, while the computational cost per denoising step remains independent of video length. Experiments are conducted on three backbone models(Wan2.1, Wan2.2, and SANA-WM) and evaluated on Echo-Memory and Revisit-Bench benchmarks.
Critical strengths
-
I like how the paper reframes "external memory retrieval" as "conditional gated access to linear attention states." It seems to draw on the idea of hybrid attention in llm, where full attention captures dense pixel-level information while linear attention compresses semantic information, allowing the model to track past information through its internal states. I think this is a pretty interesting direction, and I suspect it was partly inspired by kimi linear and ARL2.
-
No need for 3D information or external memory at inference time. The 3D information is only used to generate supervision labels during training, so the model doesn't rely on it at all during inference. This avoids error propagation from 3D reconstruction and removes the engineering overhead of maintaining an external retrieval module.
-
Clear efficiency advantages. The denoising latency stays constant at around 0.5 seconds per step.
-
Comprehensive evaluation. The experiments cover three backbone models, two state update mechanisms, and two benchmarks.
Critical weaknesses
1.Some concerns about the fairness of the comparisons. In Table 1, about half of the baseline results are taken directly from the Echo-Memory paper, which seems to contradict the authors' claim that all methods are evaluated under exactly the same settings. Also, Context-as-Memory is not marked as retrained in Table 1, but Appendix C says it was retrained. This inconsistency makes the comparison a bit questionable.
2.Lack of analysis of the underlying mechanisms. The paper doesn't report the precision or recall of the retrieval trigger, provide any visualization of gamma_t, or investigate what information is actually stored in the recurrent states. As a result, there isn't much direct evidence to support the claim that the model has learned when to retrieve information and what information to retrieve.
3.Insufficient evaluation of long-term memory capabilities. Most of the quantitative evaluations are conducted on relatively short sequences of around 81 frames. More fundamentally, I think the fixed capacity of the recurrent states is a major limitation. I'm actually not convinced that DeltaNet-style approaches are a good solution to memory in visual models. The core idea originally comes from LLMs, where the main motivation is more about improving context efficiency rather than solving memory itself. For text, compressing information into abstract semantic representations is often acceptable. However, visual tasks, especially revisiting previously seen scenes, often require preserving dense, fine-grained token-level information that may be lost during compression.
Questions for authors
See above weakness. especially at the third point. I'm not sure this is the right direction for visual memory. I even suspect that a sparse attention-based approach might work better, since it could preserve access to detailed visual information instead of forcing everything into a compressed recurrent state. The supplementary video demos also seem to support this concern, as the actual performance on memory revisiting doesn't look particularly convincing.
6 comments
Weakness2. Lack of analysis of the underlying mechanisms. The paper doesn't report the precision or recall of the retrieval trigger, provide any visualization of gamma_t, or investigate what information is actually stored in the recurrent states. As a result, there isn't much direct evidence to support the claim that the model has learned when to retrieve information and what information to retrieve.
Answer. Thank you for your suggestion. We have supplemented the quantitative recognition results of the trigger, spatiotemporal activation visualizations, and an internal behavior analysis of the camera-conditioned gate.
(1) Quantitative recognition of the retrieval trigger.
Using 3D re-visibility labels as ground truth, we evaluate the trigger token-by-token on the Echo-Memory test set.
- Test videos contain 243 frames.
- A token grid is used.
- Only valid tokens with Pi3X depth confidence higher than 0.3 are counted.
- Taking the trigger as the predicted positive class and re-visibility as the ground-truth positive class, we compute Precision, Recall, F1, and Accuracy.
| Triggering Method | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| Always | 0.21 | 1.00 | 0.32 | 0.21 |
| Random | 0.21 | 0.21 | 0.21 | 0.72 |
| L2R trigger | 0.76 | 0.74 | 0.74 | 0.94 |
L2R trigger achieves an F1 of 0.74, higher than Always (0.32) and Random (0.21). Its Precision is 0.76 and Recall is 0.74, indicating that the model achieves a good balance between triggering accuracy and coverage of revisited regions.
Always has a Recall of 1.00 because it predicts all valid tokens as positive; its Precision is 0.21, reflecting the proportion of true re-visibility positive samples in the test set. In contrast, L2R selectively activates the trigger according to spatial revisitation needs.
(2) Visualization of the spatiotemporal behavior of the trigger.
Figure 6 in the revised paper (page 20) illustrates the trigger activation probabilities across different spatial locations and their temporal variations. The figure selects the four test windows with the highest proportion of re-visible regions in the return segment:
| Visual Element | Meaning |
|---|---|
| Orange contour | Target region corresponding to the geometric re-visibility label. |
| Shades of blue | Trigger activation probability at different spatial locations. |
| Right-side curves | Temporal variation of the proportion of re-visibility targets and trigger-activated tokens relative to valid tokens. |
The visualizations show that:
- When the camera turns away from historical regions, the trigger remains largely inactive.
- When the camera turns back toward historical regions, activations concentrate within the orange target region.
- The proportion of trigger-activated tokens varies with the size of the re-visible region: it increases as the region expands and decreases as it shrinks.
Therefore, both the spatial activation locations and temporal variation trends of the trigger correspond to geometric revisitation needs.
(3) Internal behavior of the camera-conditioned gate.
The trigger determines when to activate historical state modulation, while the camera-conditioned gate determines how to modulate the historical state once activated.
Figure 7 in the revised paper (page 21) compares two types of tokens in the re-visible region at frame 238:
- Previously visible tokens: Already visible 24 frames earlier at frame 214.
- Newly re-entering tokens: Re-entering the current field of view at frame 238.
The gate distributions exhibit distinct patterns:
- For previously visible tokens, is heavily concentrated near 1, indicating that the corresponding historical states are retained.
- For newly re-entering tokens, exhibits a broader distribution spanning low and high values, with one portion near 1 and the other near 0, corresponding to retention and suppression of the historical state, respectively.
This indicates that the camera-conditioned gate selectively modulates the historical states corresponding to different tokens based on the current spatial condition.
In summary, the recurrent state update formulation explains how historical information is stored; the Precision, Recall, and F1 of the trigger demonstrate when the model activates historical state modulation; and the visualization of illustrates how the model selectively modulates historical information. These three pieces of evidence jointly support L2R's internalized historical information access mechanism.
Revision: We have added the above internal behavior analysis, quantitative results, and corresponding visualizations in Section 4.6 and Appendix E in the revised PDF.
Weakness3 & Question. Insufficient evaluation of long-term memory capabilities. Most of the quantitative evaluations are conducted on relatively short sequences of around 81 frames. More fundamentally, I think the fixed capacity of the recurrent states is a major limitation. I'm actually not convinced that DeltaNet-style approaches are a good solution to memory in visual models. The core idea originally comes from LLMs, where the main motivation is more about improving context efficiency rather than solving memory itself. For text, compressing information into abstract semantic representations is often acceptable. However, visual tasks, especially revisiting previously seen scenes, often require preserving dense, fine-grained token-level information that may be lost during compression. I'm not sure this is the right direction for visual memory. I even suspect that a sparse attention-based approach might work better, since it could preserve access to detailed visual information instead of forcing everything into a compressed recurrent state. The supplementary video demos also seem to support this concern, as the actual performance on memory revisiting doesn't look particularly convincing.
Answer. First, we apologize for having uploaded incorrect visualizations in the previous supplementary materials (which were derived from checkpoints at 1k/5k iteration steps). We have now replaced them with the corresponding videos generated from the 5k-iteration-step checkpoint.
Now, we are focusing on addressing the concerns you have raised.
We have supplemented 1000-frame long-sequence autoregressive generation results and further constructed multi-location revisitation trajectories to quantitatively evaluate generation consistency when repeatedly visiting the same camera pose.
It is necessary to distinguish the experimental purposes corresponding to three sequence lengths:
| Sequence Length | Experimental Purpose |
|---|---|
| 81 frames | Quantitative evaluation on Revisit-Bench. |
| 243 frames | Trigger behavior analysis in Echo-Memory test samples. |
| 1000 frames | Qualitative demonstration of long-sequence autoregressive generation. |
Regarding the relationship between fixed-capacity recurrent states and visual memory, we respond from three aspects: task positioning, architectural design, and experimental results.
(1) Task positioning: scene revisitation rather than unbounded scene exploration.
This paper focuses on scene revisitation consistency rather than unbounded scene exploration. The two have different requirements for historical information:
- Scene exploration: Continuously enters new spatial regions and needs to constantly expand the coverage of historical information.
- Scene revisitation: The camera re-enters previously observed spatial regions and needs to recover and maintain the consistency of historical scene content.
L2R is oriented toward historical information access tasks within finite physical scenes, accumulating historical information into the recurrent state and selectively modulating it based on the current camera condition. For such revisitation tasks in finite scenes, unboundedly expanding all historical context may not necessarily be reasonable.
(2) Hybrid architecture: balancing historical state capacity and fine-grained contextual interaction.
We do not replace all attention layers with L2R layers; instead, we retain softmax-attention layers, forming a hybrid architecture together with recurrent memory.
| Module | Role |
|---|---|
| L2R layers | Continuously accumulate historical tokens in the state. |
| Softmax-attention layers | Preserve token-level interactions within the attention context, providing fine-grained context modeling capability. |
This design does not treat the recurrent state as a lossless storage for all historical visual information, but rather leverages the complementarity of the two mechanisms to strike a balance among historical information accumulation, fine-grained contextual interaction, and memory efficiency.
(3) Layer-ratio ablation: experimental basis for fixed capacity and the hybrid architecture.
We evaluate the impact of the ratio between the two layer types on revisitation performance through different memory-to-softmax layer ratios.
| Architecture | Memory/Softmax ratio | Replay PSNR | Loop PSNR |
|---|---|---|---|
| L2R-GLA | 1:1 | 14.26 | 13.55 |
| L2R-GLA | 3:1 | 14.74 | 13.14 |
| L2R-GLA | 7:1 | 10.52 | 10.37 |
The results show that the 3:1 layer ratio achieves the highest Replay PSNR (14.74 dB), and 1:1 achieves the highest Loop PSNR (13.55 dB); under 7:1, the two metrics drop to 10.52 dB and 10.37 dB, respectively.
These results indicate that increasing the proportion of L2R layers does not continuously improve revisitation performance. We ultimately adopt the 3:1 configuration, primarily considering replay fidelity while maintaining competitive loop-closure performance.
Answer. Thank you for pointing out the annotation issue regarding the sources of the experimental results. In Table 1, results not marked with
*are cited from Echo-Memory, while methods marked with*were retrained by us. Furthermore, we conducted an independent reproduction following the experimental environment and memory-capacity benchmark protocol of Echo-Memory, and report the reproduced results in this response.(1) Sources of results in Table 1.
Table 1 follows the memory-capacity benchmark evaluation protocol of Echo-Memory:
*: Adopt the results reported in the Echo-Memory paper.*: Adopt the results obtained from our own retraining.Therefore, citing values from Echo-Memory does not mean we describe them as results retrained by ourselves.
(2) Complete reproduction under a unified environment.
To provide directly comparable experimental evidence, we retrained and evaluated the relevant methods under the same software/hardware environment and memory-capacity benchmark evaluation protocol as Echo-Memory.
The reproduced results show that:
These results demonstrate the performance of different methods across each evaluation dimension: L2R-GLA excels in historical scene replay and open-domain revisitation, while VMem achieves better results on Loop PSNR and Loop LPIPS.
(3) Annotation of Context as Memory.
We sincerely apologize for the errors in our previous submission. To ensure a fair comparison and consistency throughout the paper, we have removed Appendix C and uploaded additional, accurate videos generated by L2R to the Supplementary Material.
We hope these supplementary materials address your concerns.