acceptodds
Withdrawn from ICLR 2027

Learning to Retrieve: Internalizing Memory Retrieval for Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Video world models aim to generate explorable, 3D-consistent scene videos conditioned on camera trajectories. Existing approaches often rely on external memory systems that explicitly retrieve previously observed content to mitigate scene drift during long-horizon generation. However, these auxiliary memory pathways operate outside the model's internal generative dynamics, preventing the model from intrinsically learning when and what historical information should be retrieved. We propose to internalize memory retrieval into the generation process, allowing retrieval to emerge as an intrinsic behavior of the video world model rather than relying on an external memory system. Based on this principle, we introduce Learning-to-Retrieve (L2R), which repurposes the model's persistent internal state as a memory for historical context. A camera-conditioned retrieval gate selectively accesses relevant historical information from this state, determining what to retrieve, while a retrieval trigger determines when to retrieve. We further supervise the trigger with a 3D re-visibility signal, activating retrieval when previously observed content re-enters the current view while otherwise preserving the existing context. Together, these components enable the model to intrinsically acquire memory retrieval behavior and incorporate relevant historical observations into generation without a separate retrieval pathway. Across multiple base models and camera-revisit benchmarks, L2R improves long-term scene consistency while eliminating the need for an external memory bank or 3D conditions.

Primary area:
generative models
Keywords:
Video world model, Memory retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_gBeKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learning-to-Retrieve, a method that replaces selected softmax attention layers in a pretrained video diffusion Transformer with gated linear attention layers (GLA or GDN), using their recurrent states as internal memory. Building on this architecture, the authors introduce two key components: (1) a retrieval gate modulated by camera Plucker rays, which selectively retains historical information in the recurrent states on a per-channel basis; and (2) a binary retrieval trigger, which determines when to activate retrieval mode. During training, it is supervised using 3D revisitability labels computed offline through depth estimation, back-projection, and reprojection. As a result, inference requires neither an external memory bank nor 3D conditioning, while the computational cost per denoising step remains independent of video length. Experiments are conducted on three backbone models(Wan2.1, Wan2.2, and SANA-WM) and evaluated on Echo-Memory and Revisit-Bench benchmarks.

Critical strengths
  1. I like how the paper reframes "external memory retrieval" as "conditional gated access to linear attention states." It seems to draw on the idea of hybrid attention in llm, where full attention captures dense pixel-level information while linear attention compresses semantic information, allowing the model to track past information through its internal states. I think this is a pretty interesting direction, and I suspect it was partly inspired by kimi linear and ARL2.

  2. No need for 3D information or external memory at inference time. The 3D information is only used to generate supervision labels during training, so the model doesn't rely on it at all during inference. This avoids error propagation from 3D reconstruction and removes the engineering overhead of maintaining an external retrieval module.

  3. Clear efficiency advantages. The denoising latency stays constant at around 0.5 seconds per step.

  4. Comprehensive evaluation. The experiments cover three backbone models, two state update mechanisms, and two benchmarks.

Critical weaknesses

1.Some concerns about the fairness of the comparisons. In Table 1, about half of the baseline results are taken directly from the Echo-Memory paper, which seems to contradict the authors' claim that all methods are evaluated under exactly the same settings. Also, Context-as-Memory is not marked as retrained in Table 1, but Appendix C says it was retrained. This inconsistency makes the comparison a bit questionable.

2.Lack of analysis of the underlying mechanisms. The paper doesn't report the precision or recall of the retrieval trigger, provide any visualization of gamma_t, or investigate what information is actually stored in the recurrent states. As a result, there isn't much direct evidence to support the claim that the model has learned when to retrieve information and what information to retrieve.

3.Insufficient evaluation of long-term memory capabilities. Most of the quantitative evaluations are conducted on relatively short sequences of around 81 frames. More fundamentally, I think the fixed capacity of the recurrent states is a major limitation. I'm actually not convinced that DeltaNet-style approaches are a good solution to memory in visual models. The core idea originally comes from LLMs, where the main motivation is more about improving context efficiency rather than solving memory itself. For text, compressing information into abstract semantic representations is often acceptable. However, visual tasks, especially revisiting previously seen scenes, often require preserving dense, fine-grained token-level information that may be lost during compression.

Questions for authors

See above weakness. especially at the third point. I'm not sure this is the right direction for visual memory. I even suspect that a sparse attention-based approach might work better, since it could preserve access to detailed visual information instead of forcing everything into a compressed recurrent state. The supplementary video demos also seem to support this concern, as the actual performance on memory revisiting doesn't look particularly convincing.

6 comments

Weakness1. Some concerns about the fairness of the comparisons. In Table 1, about half of the baseline results are taken directly from the Echo-Memory paper, which seems to contradict the authors' claim that all methods are evaluated under exactly the same settings. Also, Context-as-Memory is not marked as retrained in Table 1, but Appendix C says it was retrained. This inconsistency makes the comparison a bit questionable.

Answer. Thank you for pointing out the annotation issue regarding the sources of the experimental results. In Table 1, results not marked with * are cited from Echo-Memory, while methods marked with * were retrained by us. Furthermore, we conducted an independent reproduction following the experimental environment and memory-capacity benchmark protocol of Echo-Memory, and report the reproduced results in this response.

(1) Sources of results in Table 1.

Table 1 follows the memory-capacity benchmark evaluation protocol of Echo-Memory:

  • Methods without *: Adopt the results reported in the Echo-Memory paper.
  • Methods with *: Adopt the results obtained from our own retraining.

Therefore, citing values from Echo-Memory does not mean we describe them as results retrained by ourselves.


(2) Complete reproduction under a unified environment.

To provide directly comparable experimental evidence, we retrained and evaluated the relevant methods under the same software/hardware environment and memory-capacity benchmark evaluation protocol as Echo-Memory.

MethodReplay PSNR↑Replay SSIM↑Replay LPIPS↓Loop PSNR↑Loop SSIM↑Loop LPIPS↓O-V↑
I2V baseline (Wan2.1)9.730.4090.5279.280.3050.61711.57
Spatial Memory13.370.4140.54510.230.2800.6237.01
VideoSSM12.140.3460.56912.170.3020.60835.16
ARL²12.730.3260.59412.130.3090.63343.06
Context as Memory, K=512.230.4160.49111.290.3190.60454.89
Context as Memory, K=2012.050.4320.50311.470.3740.58758.63
WorldMem12.340.4710.53611.590.3480.59459.74
VMem13.710.5020.50713.170.3520.58164.73
L2R-GLA14.740.5130.43013.140.3750.60372.57
L2R-GDN13.860.4090.55812.810.3740.61469.37

The reproduced results show that:

  • L2R-GLA achieves the best results on Replay PSNR, Replay SSIM, Replay LPIPS, Loop SSIM, and O-V.
  • VMem achieves the best results on Loop PSNR and Loop LPIPS.
  • L2R-GDN also achieves a higher O-V than all compared methods except L2R-GLA.

These results demonstrate the performance of different methods across each evaluation dimension: L2R-GLA excels in historical scene replay and open-domain revisitation, while VMem achieves better results on Loop PSNR and Loop LPIPS.


(3) Annotation of Context as Memory.

We sincerely apologize for the errors in our previous submission. To ensure a fair comparison and consistency throughout the paper, we have removed Appendix C and uploaded additional, accurate videos generated by L2R to the Supplementary Material.

We hope these supplementary materials address your concerns.

Weakness2. Lack of analysis of the underlying mechanisms. The paper doesn't report the precision or recall of the retrieval trigger, provide any visualization of gamma_t, or investigate what information is actually stored in the recurrent states. As a result, there isn't much direct evidence to support the claim that the model has learned when to retrieve information and what information to retrieve.

Answer. Thank you for your suggestion. We have supplemented the quantitative recognition results of the trigger, spatiotemporal activation visualizations, and an internal behavior analysis of the camera-conditioned gate.

(1) Quantitative recognition of the retrieval trigger.

Using 3D re-visibility labels as ground truth, we evaluate the trigger token-by-token on the Echo-Memory test set.

  • Test videos contain 243 frames.
  • A 22×4022 \times 40 token grid is used.
  • Only valid tokens with Pi3X depth confidence higher than 0.3 are counted.
  • Taking the trigger as the predicted positive class and re-visibility as the ground-truth positive class, we compute Precision, Recall, F1, and Accuracy.
Triggering MethodPrecisionRecallF1Accuracy
Always0.211.000.320.21
Random0.210.210.210.72
L2R trigger0.760.740.740.94

L2R trigger achieves an F1 of 0.74, higher than Always (0.32) and Random (0.21). Its Precision is 0.76 and Recall is 0.74, indicating that the model achieves a good balance between triggering accuracy and coverage of revisited regions.

Always has a Recall of 1.00 because it predicts all valid tokens as positive; its Precision is 0.21, reflecting the proportion of true re-visibility positive samples in the test set. In contrast, L2R selectively activates the trigger according to spatial revisitation needs.


(2) Visualization of the spatiotemporal behavior of the trigger.

Figure 6 in the revised paper (page 20) illustrates the trigger activation probabilities across different spatial locations and their temporal variations. The figure selects the four test windows with the highest proportion of re-visible regions in the return segment:

Visual ElementMeaning
Orange contourTarget region corresponding to the geometric re-visibility label.
Shades of blueTrigger activation probability at different spatial locations.
Right-side curvesTemporal variation of the proportion of re-visibility targets and trigger-activated tokens relative to valid tokens.

The visualizations show that:

  • When the camera turns away from historical regions, the trigger remains largely inactive.
  • When the camera turns back toward historical regions, activations concentrate within the orange target region.
  • The proportion of trigger-activated tokens varies with the size of the re-visible region: it increases as the region expands and decreases as it shrinks.

Therefore, both the spatial activation locations and temporal variation trends of the trigger correspond to geometric revisitation needs.


(3) Internal behavior of the camera-conditioned gate.

The trigger determines when to activate historical state modulation, while the camera-conditioned gate γt\gamma_t determines how to modulate the historical state once activated.

Figure 7 in the revised paper (page 21) compares two types of tokens in the re-visible region at frame 238:

  • Previously visible tokens: Already visible 24 frames earlier at frame 214.
  • Newly re-entering tokens: Re-entering the current field of view at frame 238.

The gate distributions exhibit distinct patterns:

  • For previously visible tokens, γt\gamma_t is heavily concentrated near 1, indicating that the corresponding historical states are retained.
  • For newly re-entering tokens, γt\gamma_t exhibits a broader distribution spanning low and high values, with one portion near 1 and the other near 0, corresponding to retention and suppression of the historical state, respectively.

This indicates that the camera-conditioned gate selectively modulates the historical states corresponding to different tokens based on the current spatial condition.

In summary, the recurrent state update formulation explains how historical information is stored; the Precision, Recall, and F1 of the trigger demonstrate when the model activates historical state modulation; and the visualization of γt\gamma_t illustrates how the model selectively modulates historical information. These three pieces of evidence jointly support L2R's internalized historical information access mechanism.


Revision: We have added the above internal behavior analysis, quantitative results, and corresponding visualizations in Section 4.6 and Appendix E in the revised PDF.

Weakness3 & Question. Insufficient evaluation of long-term memory capabilities. Most of the quantitative evaluations are conducted on relatively short sequences of around 81 frames. More fundamentally, I think the fixed capacity of the recurrent states is a major limitation. I'm actually not convinced that DeltaNet-style approaches are a good solution to memory in visual models. The core idea originally comes from LLMs, where the main motivation is more about improving context efficiency rather than solving memory itself. For text, compressing information into abstract semantic representations is often acceptable. However, visual tasks, especially revisiting previously seen scenes, often require preserving dense, fine-grained token-level information that may be lost during compression. I'm not sure this is the right direction for visual memory. I even suspect that a sparse attention-based approach might work better, since it could preserve access to detailed visual information instead of forcing everything into a compressed recurrent state. The supplementary video demos also seem to support this concern, as the actual performance on memory revisiting doesn't look particularly convincing.

Answer. First, we apologize for having uploaded incorrect visualizations in the previous supplementary materials (which were derived from checkpoints at 1k/5k iteration steps). We have now replaced them with the corresponding videos generated from the 5k-iteration-step checkpoint.

Now, we are focusing on addressing the concerns you have raised.


We have supplemented 1000-frame long-sequence autoregressive generation results and further constructed multi-location revisitation trajectories to quantitatively evaluate generation consistency when repeatedly visiting the same camera pose.

It is necessary to distinguish the experimental purposes corresponding to three sequence lengths:

Sequence LengthExperimental Purpose
81 framesQuantitative evaluation on Revisit-Bench.
243 framesTrigger behavior analysis in Echo-Memory test samples.
1000 framesQualitative demonstration of long-sequence autoregressive generation.

Regarding the relationship between fixed-capacity recurrent states and visual memory, we respond from three aspects: task positioning, architectural design, and experimental results.

(1) Task positioning: scene revisitation rather than unbounded scene exploration.

This paper focuses on scene revisitation consistency rather than unbounded scene exploration. The two have different requirements for historical information:

  • Scene exploration: Continuously enters new spatial regions and needs to constantly expand the coverage of historical information.
  • Scene revisitation: The camera re-enters previously observed spatial regions and needs to recover and maintain the consistency of historical scene content.

L2R is oriented toward historical information access tasks within finite physical scenes, accumulating historical information into the recurrent state and selectively modulating it based on the current camera condition. For such revisitation tasks in finite scenes, unboundedly expanding all historical context may not necessarily be reasonable.


(2) Hybrid architecture: balancing historical state capacity and fine-grained contextual interaction.

We do not replace all attention layers with L2R layers; instead, we retain softmax-attention layers, forming a hybrid architecture together with recurrent memory.

ModuleRole
L2R layersContinuously accumulate historical tokens in the state.
Softmax-attention layersPreserve token-level interactions within the attention context, providing fine-grained context modeling capability.

This design does not treat the recurrent state as a lossless storage for all historical visual information, but rather leverages the complementarity of the two mechanisms to strike a balance among historical information accumulation, fine-grained contextual interaction, and memory efficiency.


(3) Layer-ratio ablation: experimental basis for fixed capacity and the hybrid architecture.

We evaluate the impact of the ratio between the two layer types on revisitation performance through different memory-to-softmax layer ratios.

ArchitectureMemory/Softmax ratioReplay PSNRLoop PSNR
L2R-GLA1:114.2613.55
L2R-GLA3:114.7413.14
L2R-GLA7:110.5210.37

The results show that the 3:1 layer ratio achieves the highest Replay PSNR (14.74 dB), and 1:1 achieves the highest Loop PSNR (13.55 dB); under 7:1, the two metrics drop to 10.52 dB and 10.37 dB, respectively.

These results indicate that increasing the proportion of L2R layers does not continuously improve revisitation performance. We ultimately adopt the 3:1 configuration, primarily considering replay fidelity while maintaining competitive loop-closure performance.

Reviewer_TtsdOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Learning-to-Retrieve (L2R) to improve long-term scene consistency in camera-controlled video world models. L2R accumulates past information in a recurrent attention state and selectively modifies that state using a camera-conditioned gate and a retrieval trigger. The trigger is supervised with re-visibility labels derived from depth and camera poses. The authors apply the design to Gated Linear Attention and Gated DeltaNet, and evaluate replay, loop closure, video quality, and camera control on several video backbones. The paper reports higher consistency than prior methods on some evaluations and shows that denoising-step latency stays nearly constant over long rollouts.

Critical strengths
  1. A practical direction that addresses long-term scene consistency and memory efficiency together. Long video generation needs to retain past information without paying for attention over the full history or for managing an external 3D memory, so this goal is important. Combining camera conditioning and re-visibility supervision with a recurrent state is a reasonable approach to the problem. The method also needs no explicit 3D correspondence or external memory retrieval at inference, which can be a practical advantage.
  2. Evaluation across several backbones and state-update rules, with failure cases disclosed. The paper reports results on Wan2.1, Wan2.2, and SANA-WM, each with GLA and GDN, which is welcome. It also provides ablations on the components and the layer ratio. It discloses the performance drop and failure cases at high memory-layer ratios as well. These help readers understand the method’s scope and the limits of a fixed-capacity state.
Critical weaknesses
  1. The inputs and computation of the retrieval trigger, a core component, are not sufficiently defined. The paper does not explain at the architectural level how past information is compressed, or what current information is added for the trigger to operate. It also does not report how accurately the trigger works or how much it affects generation. Explicit equations for this module, its information flow, and its performance are needed.
  2. The need to tell the model when to retrieve previously generated information is questionable. Would it not be enough to always retrieve, and simply provide no past information when nothing relevant to the target view exists? Retrieving past information is not costly, so I doubt that deciding whether to retrieve is a significant contribution.
Questions for authors
  1. What are the exact inputs to the retrieval trigger, and how is it computed? How is past information compressed into the state, and what current information is combined with it to decide when to retrieve? How accurately does the trigger predict re-visibility, and how much does its accuracy affect generation quality?
  2. Line 165 says the video is compressed into a single 1D token. How is this compression done, and how much information is lost? Is this why the first and last frames are often geometrically inconsistent, and why the generated results sometimes do not follow the reference image?
  3. Why do the baseline PSNR values differ so much from those in the original papers? What is the exact evaluation protocol, including the data, resolution, trajectory length, and which frames are compared? Can you show the results separately for RealEstate10K and DL3DV?
  4. What happens if the model always retrieves, without the trigger? If retrieving past information is not costly, why is it necessary to decide when to retrieve?
12 comments

Thank you for your insightful and comprehensive feedback. We have carefully studied your comments and argue that your concerns can be addressed.

Weakness1 & Question1. The inputs and computation of the retrieval trigger, a core component, are not sufficiently defined. The paper does not explain at the architectural level how past information is compressed, or what current information is added for the trigger to operate. It also does not report how accurately the trigger works or how much it affects generation. Explicit equations for this module, its information flow, and its performance are needed. What are the exact inputs to the retrieval trigger, and how is it computed? How is past information compressed into the state, and what current information is combined with it to decide when to retrieve? How accurately does the trigger predict re-visibility, and how much does its accuracy affect generation quality?

Answer. We are pleased to provide a further explanation of the trigger module and the information flow of the recurrent state. L2R compresses historical context into a recurrent state and achieves internalized access to historical information through the retrieval trigger and camera-conditioned gate. Below, we describe the state construction, state modulation, trigger computation, and experimental results, respectively.

(1) How is historical information compressed into the recurrent state?

In L2R, key-value associations of historical tokens are accumulated into the recurrent state via outer products. Let ktk_t and vtv_t denote the key and value of the current token, respectively, and let StS_t denote the recurrent state. Omitting additional gating:

St=St−1+vtkt⊤.S_t = S_{t-1} + v_t k_t^\top.

Unrolling gives:

St=∑i=1tviki⊤.S_t = \sum_{i=1}^{t} v_i k_i^\top.

Therefore, StS_t is a fixed context representation formed by accumulating historical tokens, rather than a single token or a simple concatenation of historical token sequences. This design avoids explicitly retaining all historical tokens and enables continuous propagation of historical context through the state.


(2) How does L2R achieve historical information access?

Building upon the above recurrent representation, L2R introduces the retrieval trigger rtr_t and the camera-conditioned gate γt\gamma_t. The actual state update is:

St={Diag⁡(γt)St−1+vtkt⊤,rt=1,St−1+vtkt⊤,rt=0.S_t = \begin{cases} \operatorname{Diag}(\gamma_t)S_{t-1} + v_t k_t^\top, & r_t = 1, \\ S_{t-1} + v_t k_t^\top, & r_t = 0. \end{cases}

The responsibilities of the two components are as follows:

  • rtr_t decides when to activate additional state modulation.
  • γt\gamma_t decides how to modulate the historical state according to the current camera condition.

When rt=0r_t = 0, the historical state is still retained and jointly updated with the current token; when rt=1r_t = 1, the model additionally performs camera-conditioned modulation of the historical state.

Weakness1 & Question1.

(3) Inputs and computation process of the trigger.

The retrieval trigger predicts whether to activate historical information access based on the current token feature and camera pose. Specifically, it takes two inputs:

  • Token feature xtx_t: The input feature of the memory layer at the current token, i.e., the hidden state of the DiT block modulated by AdaLN.
  • Camera encoding ϕt\phi_t: The camera-condition encoding obtained by projecting the camera pose ctc_t of the current frame, where ctc_t denotes the translation and rotation matrices relative to the reference frame, and the projection matrix WcW_c is shared with the camera-conditioned gate ϕt=Wcct.\phi_t = W_c c_t.

The computation of the trigger consists of two steps:

Step 1. Predict the trigger logit.

The token feature xtx_t and camera encoding ϕt\phi_t each pass through an independent linear prediction head, and the two prediction results are summed to obtain the scalar trigger logit:

r^t=wx⊤xt+wϕ⊤ϕt+b.\hat{r}_t = w_x^\top x_t + w_\phi^\top \phi_t + b.

Step 2. Binarize to obtain the trigger.*

A sigmoid function is applied to the trigger logit, followed by binarization with a threshold of 0.5 to obtain the final triggering result:

rt=1[σ(r^t)>0.5]=1[r^t>0].r_t = \mathbb{1}\left[\sigma(\hat{r}_t) > 0.5\right] = \mathbb{1}\left[\hat{r}_t > 0\right].

Gradients are stopped through the binarization operation. Ultimately, rt∈0,1r_t \in \\{0, 1\\} controls whether to activate additional historical information access and state modulation.

The trigger controls historical information access as follows:

TriggerHistorical Information AccessState Modulation
rt=0r_t = 0Uses recurrent state as context directlyDoes not activate the camera-conditioned gate; existing historical state receives no additional modulation
rt=1r_t = 1Reads and modulates historical state componentsActivates the camera-conditioned gate, selectively modulates the historical state via γt\gamma_t, and adds the read-out historical information component to the current computation

In summary, the retrieval trigger decides when to access historical information based on the current token feature and camera pose, while the camera-conditioned gate decides how to modulate the historical state. Together, they trigger decisions and modulate the state, jointly enabling L2R's conditioned access to historical information.


(4) Re-visibility supervision and generation quality.

The geometric supervision of the trigger is constructed from 3D re-visibility labels through the following procedure:

  1. Estimate the depth of each frame.
  2. Unproject surfaces corresponding to historical tokens into 3D points.
  3. Project these points onto the current frame according to the camera pose.
  4. Label surfaces that were previously observed, subsequently left the field of view or became occluded, and eventually reappeared as positive samples.

The trigger is supervised using weighted binary cross-entropy, with a positive sample weight of w+=3.w_+ = 3.

The overall training objective is:

L=Ldenoise+λrLr,λr=0.05.\mathcal{L} = \mathcal{L}_{\mathrm{denoise}} + \lambda_r \mathcal{L}_r, \qquad \lambda_r = 0.05.

The token-by-token recognition performance of the trigger is shown in the following table. The result is the average of the metrics across all test cases.

Triggering MethodPrecisionRecallF1Accuracy
Always0.211.000.320.21
Random0.210.210.210.72
L2R trigger (ours)0.760.740.740.94

In terms of generation quality, Table 3 shows:

  • Camera-guided retrieval: Replay PSNR is 13.88 dB.
  • Adding an independent trigger: Replay PSNR increases to 14.02 dB.
  • Adding re-visibility supervision: Replay PSNR further increases to 14.74 dB.

Thus, the effect of the trigger is reflected in both geometric label recognition and generation quality improvement.


(5) Parameter overhead.

According to Appendix A.1, L2R layers reuse the query, key, value, and output projections of the replaced self-attention layers. Additional parameters mainly come from the projection layers of the retrieval gate and trigger.

Taking Wan2.1 as an example:

Layer Type or Replacement SettingParameter Count
Standard softmax self-attention layer9.45M
L2R-GLA layer11.20M
Replacing the first three attention layers in every four blocks+5.25M per group (+1.75M per replaced layer)

This design introduces an additional mechanism for historical information access while preserving the main projection structure of the backbone.


Revision: We have added the mathematical definition of the recurrent state, the actual inputs and computation process of the trigger, re-visibility supervision, and the training objective in Section 3, and added the quantitative evaluation and internal behavior analysis of the trigger in Section 4.6. The revised paper has been submitted.

Weakness2 & Question4. The need to tell the model when to retrieve previously generated information is questionable. Would it not be enough to always retrieve, and simply provide no past information when nothing relevant to the target view exists? Retrieving past information is not costly, so I doubt that deciding whether to retrieve is a significant contribution. What happens if the model always retrieves, without the trigger? If retrieving past information is not costly, why is it necessary to decide when to retrieve?

Answer. The purpose of introducing the retrieval trigger is not to reduce the computational cost of historical information retrieval, but to avoid indiscriminately activating retrieval at all time steps.

(1) Design motivation for selective triggering.

During generation, a video world model continuously receives changing camera viewpoints. Two types of viewpoints correspond to different state modulation needs:

  • No historical scene revisitation occurs: Additional modulation of the historical state is unnecessary.
  • Previously observed scenes re-enter the field of view: The model needs to access relevant historical information based on the current spatial condition.

If retrieval is always activated, the model will continuously modulate the historical state even when there is no clear need for historical scene revisitation in the current viewpoint, potentially introducing unnecessary state changes. The role of the retrieval trigger is precisely to match the additional modulation to actual spatial revisitation needs.


(2) Controlled experiment with continuous activation.

The Camera-guided retrieval configuration in Table 3 has no independent retrieval trigger and continuously activates camera-conditioned state modulation. A comparison with the result after adding an independent trigger is as follows:

MetricContinuously Activated Camera-guided Retrieval+ Retrieval Trigger
Replay PSNR13.8814.02
Loop PSNR12.7613.11
O-V68.7471.42

After adding the independent trigger:

  • Replay PSNR improves by 0.14 dB.
  • Loop PSNR improves by 0.35 dB.
  • O-V improves by 2.68.

This indicates that continuously activating camera-conditioned state modulation differs in effect from selectively activating it according to geometric revisitation needs. The independent trigger makes the additional state modulation correspond more accurately to spatial revisitation events, thereby improving generation results.


(3) Geometric recognition capability of the trigger.

In the token-by-token evaluation, Always achieves an F1 of 0.32 and Random achieves 0.21, whereas L2R trigger reaches 0.74.

Therefore, the contribution of the retrieval trigger lies in learning when to activate additional historical state modulation, rather than reducing the computational overhead of the retrieval operation itself.

Reviewer_yKWpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper internalizes memory retrieval in video world models using recurrent attention states, camera-conditioned gating, and a learned re-visibility trigger. It is designed to improve scene consistency without an expanding external memory bank. Experiments evaluate memory capacity, camera revisits, transfer across backbones, and efficiency as the generated history grows.

Critical strengths
  1. Maintaining scene consistency over a growing video history is an important problem. Combining camera geometry with internal memory makes sense, and the GLA/GDN and Wan/SANA results extend the study beyond one implementation.

  2. The long-context measurements show the efficiency benefit of bounded state. The memory-ratio ablation also reveals the quality loss from replacing too much ordinary attention.

Critical weaknesses
  1. The retrieval rule is hard to follow. Equation 2 uses a local window for for 𝑟𝑡𝑟_{𝑡}=0, while Appendix B retains 𝑆t−1𝑆_{t−1}​ in that branch; it is unclear whether the state is read or only carried forward. The trigger ablation raises another question. Line 215 stops gradients through the hard trigger, yet Table 3 adds the trigger before its supervision. The learning signal for the trigger-only variant is unspecified, which makes this central ablation difficult to interpret.

  2. The presentation is poor. Appendix A.2.2 lists fields in RealCam-Vid train.npz at length, while the training/test split for Revisit-Bench gets too little explanation. The benchmark is sampled from those records, using the same source datasets as training. Appendix A.2.3 removes duplicate source videos within the benchmark, but gives no rule for excluding training videos. Whether the reported revisit results hold on unseen source scenes is therefore uncertain.

Questions for authors
  1. How is the local-window branch implemented?
  2. Are all benchmark source videos excluded from training? What split or exclusion rule was used?
  3. How accurately does the trigger identify reappearing surfaces compared with fixed or unguided activation?
5 comments

Weakness1. The retrieval rule is hard to follow. Equation 2 uses a local window for rt=0r_t=0, while Appendix B retains St−1S_{t-1} in that branch; it is unclear whether the state is read or only carried forward. The trigger ablation raises another question. Line 215 stops gradients through the hard trigger, yet Table 3 adds the trigger before its supervision. The learning signal for the trigger-only variant is unspecified, which makes this central ablation difficult to interpret.

Answer. After careful analysis of your comments, we believe this concern arises from conflating the conceptual retrieval formulation with the actual implementation in L2R, although we explained lines 174–188 of the original paper.

We would still like to clarify this for you: it is important to distinguish between the conceptual context-selection rule in Eq. (2) and the actual recurrent-state implementation in Eq. (3). The two describe different levels of abstraction, and there is no conflict in the state access rules.

(1) Eq. (2): Conceptual context-selection rule.

Eq. (2) describes two context-selection modes of a general retrieval policy:

  • When rt=1r_t = 1, context is retrieved from historical information.
  • When rt=0r_t = 0, local-window context is used.

This equation is used to explain the basic concept of retrieval behavior and does not mean that L2R actually implements an independent local-window retrieval branch.


(2) Eq. (3): Implementation in L2R.

L2R internalizes historical information access into the persistent recurrent state st−1s_{t-1}:

ct=γtst−1,If rt=1,c_t = \gamma_t s_{t-1}, \text{If } r_t = 1, ct=st−1,If rt=0,c_t = s_{t-1}, \text{If } r_t = 0,

where:

  • ctc_t: the historical context currently used.
  • st−1s_{t-1}: the recurrent state from the previous step.
  • γt\gamma_t: the camera-conditioned gate.

Therefore, the state usage under both cases is clear:

  • When rt=0r_t = 0, the model still uses the full historical recurrent state as context, without switching to a local window or discarding the historical state.
  • When rt=1r_t = 1, the camera-conditioned gate is additionally activated to selectively modulate the historical state.

This is consistent with the state update formulation in Appendix B.


(3) Evaluation of triggering strategies.

To examine the role of the trigger, we compare three triggering strategies while keeping all other model settings unchanged:

  1. Always: Setting rt≡1r_t \equiv 1 for all tokens.
  2. Random: Within each test case, randomly selecting tokens to trigger according to the true positive ratio of re-visibility in that window.
  3. L2R trigger: Using the triggering results learned by the model.

The evaluation is conducted on the Echo-Memory. We use 3D re-visibility labels as ground truth.

Triggering MethodRevisit F1↑Replay PSNR↑
Always0.3213.88
Random0.2112.41
L2R trigger (ours)0.7414.74

Revisit F1 measures how effectively the trigger identifies geometric re-visibility positive samples. L2R trigger outperforms both Always and Random.

In addition, the ablation in Table 3 shows:

SettingReplay PSNR
Camera-guided retrieval13.88
+ Retrieval trigger14.02
+ Re-visibility supervision14.74

These results verify the role of our triggering from the trigger's geometric recognition capability and generation quality, respectively.

Please note that triggering retrieval with an "Always" policy causes the L2R behavior to collapse into a GLA + camera-guided retrieval gate configuration. For the specific reason, please refer to our response to "Reviewer 5tQg Question 2."


(4) Training signal for the trigger-only setting.

The full L2R uses a hard binary trigger and stops gradients through the binarization operation. Therefore, the trigger is trained via the re-visibility supervision in Eq. (5).

The third row of Table 3 does not use re-visibility supervision. To allow this variant to learn the trigger from the video generation loss, we replace the hard binary decision with soft gating, directly using the sigmoid probability σ(logit)\sigma(\mathrm{logit}) as a continuous rt∈(0,1)r_t \in (0, 1).

Thus, the difference between the two settings is:

  • Third row of Table 3: Learning the trigger solely from the generation objective.
  • Fourth row of Table 3: Explicitly supervising the trigger using geometric re-visibility.

After adding geometric supervision, Replay PSNR increases from 14.02 dB to 14.74 dB, and O-V increases from 71.42 to 72.57. This indicates that the generation loss alone is insufficient to provide accurate supervision for triggering timing, whereas geometric re-visibility provides a direct supervision signal for "when retrieval is needed".


Revision: We have clarified the distinction in abstraction levels between Eq. (2) and Eq. (3) in Section 3.2, and supplemented the training signal for the trigger and the corresponding ablation settings.

Weakness2 & Question2. The presentation is poor. Appendix A.2.2 lists fields in RealCam-Vid train.npz at length, while the training/test split for Revisit-Bench gets too little explanation. The benchmark is sampled from those records, using the same source datasets as training. Appendix A.2.3 removes duplicate source videos within the benchmark, but gives no rule for excluding training videos. Whether the reported revisit results hold on unseen source scenes is therefore uncertain. Are all benchmark source videos excluded from training? What split or exclusion rule was used?

Answer. All samples in Revisit-Bench come from the test split of RealCam-Vid rather than the train split. Therefore, the source videos in the benchmark do not participate in model training based on the RealCam-Vid train split.

(1) Training/test split.

RealCam-Vid is composed of data sources such as RealEstate10K, DL3DV, and MiraData, and provides an explicit train/test split. We follow its original split:

  • Model training: Uses the train split of RealCam-Vid.
  • Revisit-Bench: All evaluation samples are selected from the held-out test split of RealCam-Vid.

The camera trajectories for training and testing are also clearly distinct:

  • Training trajectory A→B→C: The camera moves continuously and observes new viewpoints, primarily corresponding to continuous exploration.
  • Test trajectory A→B→A: The camera leaves the initially observed region and then returns, requiring the model to recover previously observed scene information.

Thus, the evaluation simultaneously examines generalization capability on unseen source scenes and the model's ability to recover historical scene information under revisitation trajectories.


(2) Intra-benchmark deduplication.

RealCam-Vid contains a large number of videos longer than 81 frames, some of which inherently include looping trajectories around the scene. Splitting the same video into multiple 81-frame sequences would produce evaluation clips with highly similar scene textures and camera motions.

Therefore, we perform intra-benchmark deduplication during the construction of Revisit-Bench to avoid repeated appearance of similar evaluation samples.

Two independent steps need to be distinguished:

  • Train/test split: The original RealCam-Vid partition guarantees isolation between training and testing source videos.
  • Intra-benchmark deduplication: Controls the duplication of similar evaluation samples within Revisit-Bench.

(3) Revision.

We have updated Appendix A.2.2, and explicitly state that model training uses the train partition while Revisit-Bench exclusively uses the held-out test partition of RealCam-Vid. We also add the structural differences between training and testing trajectories and the specific role of intra-benchmark deduplication in Appendix A.2.3 to eliminate ambiguity regarding data partitioning.

Question1. How is the local-window branch implemented?

Answer. Upon careful review, our paper does not mention a "local-window branch," nor does L2R implement an independent "local-window retrieval" branch. We kindly ask that your questions focus on the content presented in the paper.

However, after careful analysis of your comment, we think this question may stem from a misunderstanding that interprets the conceptual formulation in Eq. (2) as a concrete implementation detail. We clarify that the local window in Eq. (2) is only used to describe the conceptual context-selection rule, and is not an independent branch in the actual architecture.

The actual implementation of L2R is carried out by the persistent recurrent state in Eq. (3):

  • When rt=0r_t = 0, the model uses the state st−1s_{t-1} from the previous time step as historical context.
  • When rt=1r_t = 1, the model selectively modulates this state via the camera-conditioned gate γt\gamma_t.

Therefore, the trigger determines when to activate additional historical state modulation, and the gate determines how to modulate the historical state. Both branches use the persistent recurrent state as the carrier of historical context, without involving an explicit switch between a local window and the recurrent state.

Reviewer_5tQgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learning-to-Retrieve (L2R) to improve long-term scene consistency in video world models. L2R integrates memory retrieval into recurrent attention through a camera-conditioned gate and a retrieval trigger supervised by 3D re-visibility signals. Experiments across multiple backbones demonstrate improved revisit consistency and inference efficiency without external memory modules.

Critical strengths
  • Long-term scene consistency is fundamental to video world models, and reducing reliance on external memory is practically valuable.

  • Reformulating memory retrieval as conditional access to a persistent recurrent state is an intuitive and meaningful architectural contribution.

  • Experiments and ablations demonstrate meaningful improvements in revisit consistency across multiple architectures.

Critical weaknesses
  • The retrieval gate directly modifies the recurrent state, making it unclear whether the method performs genuine retrieval or primarily improves conditional forgetting and state management.

  • L2R uses geometric re-visibility to supervise the retrieval trigger. However, reappearing content may still be available in recent context or may have already been forgotten from the recurrent state. Conversely, historical information may be useful even without re-visibility. The paper could better justify this supervision assumption and discuss its limitations.

  • L2R compresses historical information into a fixed-size recurrent state, which may cause interference between memories of different spatial locations. For example, retrieving information from one location may affect the ability to recall other previously visited locations. The paper does not sufficiently discuss how L2R handles such interference as the number of observed locations increases.

Questions for authors
  1. How do you distinguish the proposed retrieval mechanism from camera-conditioned forgetting or state modulation, given that the gate directly modifies the recurrent state?

  2. What is the main advantage of separating the retrieval trigger and camera-conditioned gate instead of using a unified gating mechanism?

  3. How does L2R prevent memory interference when repeatedly revisiting multiple locations, such as A → B → C → A → B → C?

  4. Why is geometric re-visibility an appropriate proxy for retrieval necessity, especially when historical information may already be corrupted or outdated?

6 comments

Thanks for your insightful comments. We list your advice and questions followed by detailed answers, then argue that your concerns can be addressed.

Weakness1&Question1. The retrieval gate directly modifies the recurrent state, making it unclear whether the method performs genuine retrieval or primarily improves conditional forgetting and state management. How do you distinguish the proposed retrieval mechanism from camera-conditioned forgetting or state modulation, given that the gate directly modifies the recurrent state?

Answer. Thank you for raising this question. We argue that retrieval and forgetting are two complementary perspectives of historical information selection:

  • Retrieval: Retaining and utilizing historical information relevant to the current generation.
  • Forgetting: Suppressing historical information irrelevant to the current generation.

L2R achieves this selective access to historical information through camera-conditioned modulation of the recurrent state. Therefore, state modulation is the concrete implementation mechanism of retrieval, rather than its opposite.

(1/4) Implementation form of retrieval.

Specifically, in L2R, historical context is not explicitly stored as independent historical frames, but is instead accumulated through the recurrent state of GLA (Gated Linear Attention) or GDN (Gated DeltaNet).

Let ktk_t and vtv_t denote the key and value vectors of the current token, respectively, and let StS_t denote the recurrent state accumulating historical tokens. Omitting additional gating, its update form is:

St=St−1+vtkt⊤=∑i=1tviki⊤.S_t = S_{t-1} + v_t k_t^\top = \sum_{i=1}^{t} v_i k_i^\top.

Thus, StS_t is an accumulated representation of historical tokens, rather than an explicit storage of all historical tokens or frames.

Building upon this, L2R introduces two vectors with distinct responsibilities:

ComponentSymbolRole
Retrieval triggerrt∈{0,1}r_t \in \{0, 1\}Decides when to activate additional camera-conditioned state modulation.
Camera-conditioned gateγt∈(0,1)\gamma_t \in (0, 1)Selectively modulates the historical state according to the current camera condition.

Specifically, its state update form is:

St=γtSt−1+vtkt⊤,If rt=1,S_t = \gamma_t S_{t-1} + v_t k_t^\top, \text{If } r_t = 1,

St=St−1+vtkt⊤,If rt=0.S_t = S_{t-1} + v_t k_t^\top, \text{If } r_t = 0.

Notably, depending on the magnitude of the gate, γt\gamma_t in L2R serves different functions:

  • A larger γt\gamma_t retains more of the historical state;
  • A smaller γt\gamma_t imposes stronger suppression on the historical state.

Hence, the essence of γt\gamma_t in L2R is to adaptively modulate the historical state according to the current camera condition.


(2/4) Experimental verification.

Table 3 in our paper directly verifies the roles of camera-conditioned retrieval, the retrieval trigger, and 3D re-visibility supervision.

SettingReplay PSNRLoop PSNRO-V
GLA10.5512.7466.13
+ Camera-guided retrieval13.8812.7668.74
+ Retrieval trigger14.0213.1171.42
+ Re-visibility supervision14.7413.1472.57

The results exhibit a clear, progressive improvement:

  • Camera-conditioned retrieval brings the primary performance gain, raising Replay PSNR from 10.55 dB to 13.88 dB.
  • Adding an independent trigger increases Replay PSNR to 14.02 dB, while also improving Loop PSNR and O-V to 13.11 and 71.42, respectively.
  • Incorporating 3D re-visibility supervision further improves Replay PSNR and O-V to 14.74 dB and 72.57.

These results validate the effectiveness of L2R's internalized historical information access from the perspective of generation quality.

Weakness1&Question1.

(3/4) Internal behavior analysis.

Beyond generation quality, we further analyze the model's internal historical information access process from two aspects: trigger activation behavior and the camera-conditioned gate.

Spatiotemporal behavior of the trigger. Figure 6 in the revised paper (page 20) illustrates the trigger activation probabilities across different spatial locations and their temporal variations:

  • When the camera turns away from historical regions, the trigger remains largely inactive.
  • When the camera turns back toward previously observed regions, activations concentrate on the re-visible areas.
  • The proportion of tokens with rt=1r_t = 1 varies with the size of the revisited region: it increases as the region expands and decreases as it shrinks.

Selective modulation of the gate. Figure 7 in the revised paper (page 21) compares two types of tokens within the re-visible region at frame 238:

  • Previously visible tokens: Already visible 24 frames earlier at frame 214.
  • Newly re-entering tokens: Re-entering the current field of view at frame 238.

The gate distributions exhibit distinct patterns:

  • For previously visible tokens, γt\gamma_t is heavily concentrated near 1, indicating that the corresponding historical states are retained.
  • For newly re-entering tokens, γt\gamma_t displays a bimodal distribution: one portion approaches 1, retaining the corresponding historical states; the other approaches 0, suppressing the corresponding historical state information.

Together, these two analyses demonstrate that the trigger learns when to activate state modulation, while the camera-conditioned gate determines how to modulate the historical state once activated.


(4/4) Broader research significance.

We do not claim that the state modulation adopted by L2R is the only way to achieve historical information access. This paper explores a broader research direction: enabling generative models to autonomously access and utilize internal historical information based on their current spatial state. L2R provides a concise baseline for this direction, and we also hope to inspire the community to explore more effective implicit historical information access mechanisms.

In summary, L2R does not perform traditional retrieval over historical frames via explicit addressing; instead, it internalizes retrieval into the video generation model through camera-conditioned recurrent state access. Table 3 in the paper and the internal behavior analyses above verify this mechanism from the perspectives of generation quality and model behavior, respectively.


Revision: We have added an explanation of the recurrent state as the carrier of historical context in Section 3.1, and included quantitative and visual analyses of the trigger and camera-conditioned gate in Appendix E. The revised paper has been submitted.

Weakness2 & Question 4. L2R uses geometric re-visibility to supervise the retrieval trigger. However, reappearing content may still be available in recent context or may have already been forgotten from the recurrent state. Conversely, historical information may be useful even without re-visibility. The paper could better justify this supervision assumption and discuss its limitations. Why is geometric re-visibility an appropriate proxy for retrieval necessity, especially when historical information may already be corrupted or outdated?

Answer. We do not claim that re-visibility is a complete criterion for the actual retrieval necessity of every token; rather, we adopt it as a spatial revisitation supervision signal with clear geometric meaning. We use 3D re-visibility as the geometric supervision signal for the retrieval trigger. It identifies 3D surfaces that were previously observed, subsequently left the field of view or became occluded, and eventually re-entered the field of view, directly corresponding to the spatial revisitation task of video world models.

(1) Why adopt geometric re-visibility?

Geometric re-visibility and visual similarity between adjacent frames describe different relationships:

SignalRelationship DescribedLimitation or Role
Visual similarity between adjacent framesLocal temporal continuity, reflecting appearance changes between adjacent frames.Primarily describes local temporal relationships.
3D re-visibility (ours)Spatial correspondence across time, identifying the same previously observed 3D surface re-entering the current field of view.Capable of establishing correspondence via geometric projection even when multiple different viewpoints separate the two observations.

Therefore, 3D re-visibility provides cross-view spatial revisitation supervision, rather than relying solely on local appearance similarity to judge the need for historical information access.


(2) Distinguishing geometric revisitation events from information integrity in the recurrent state.

Here, two questions need to be distinguished: whether a geometric revisitation occurs, and whether the relevant historical information is still preserved in the recurrent state. 3D re-visibility identifies the revisitation event, whereas the actual modulation of the historical state is performed by different modules in L2R.

ComponentQuestion AnsweredSpecific Role
3D re-visibilityDoes spatial revisitation occur?Identifies 3D surfaces that were previously observed, left the field of view or became occluded, and are now re-entering the field of view.
Retrieval trigger rtr_tWhen to activate state modulation?Decides whether to activate additional camera-conditioned state modulation based on geometric revisitation supervision.
Camera-conditioned gate γt\gamma_tHow to modulate the historical state?Selectively retains or suppresses different historical states according to the current camera condition.
Softmax attentionHow to provide fine-grained contextual interaction?Preserves token-level interactions within the attention context, complementing L2R.

Geometric revisitation does not imply that all relevant historical information needs to be modified to the same extent. Figure 7 in the revised paper (page 21) compares frame 214 and frame 238, and summarizes the gate distributions of two types of tokens in the re-visible region at frame 238:

  • Previously visible tokens: Already visible at frame 214 and still visible at frame 238. Their γt\gamma_t values are heavily concentrated near 1, indicating that the corresponding historical states are retained.
  • Newly re-entering tokens: Not visible at frame 214, re-entering the field of view at frame 238. Their γt\gamma_t values exhibit a bimodal distribution: one portion approaches 1, retaining the corresponding historical states; the other approaches 0, suppressing the corresponding historical state information.

These results indicate that when geometric revisitation occurs, the model does not apply a uniform modification to all relevant historical states, but instead selectively retains or suppresses states according to the current camera condition.

The camera-conditioned gate modulates information already existing in the recurrent state, rather than reconstructing information that has already been lost from the state. Meanwhile, L2R retains softmax-attention layers to maintain token-level contextual interactions within the attention context.

Therefore, 3D re-visibility serves as a geometric supervision signal tailored to the spatial revisitation task, and does not imply that all useful historical information must be triggered by re-visibility. Recurrent state modulation and softmax attention jointly support historical context modeling.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewers and Area Chair,

We sincerely thank the reviewers and the Area Chair for their thoughtful comments and constructive suggestions. In response, we have revised the manuscript to clarify L2R's motivation, formulation, and implementation, and to add evaluations and analyses. All changes are highlighted in blue. The main revisions are summarized below.


1. Abstract and Introduction (Pages 1–2).

We clarify L2R's two components: the retrieval trigger determines when camera-conditioned retrieval is activated, while the camera-conditioned gate determines how historical information is modulated. We also clarify the role of 3D re-visibility supervision and that explicit 3D correspondence is not required at inference.


2. Preliminaries and Motivation (Section 3.1, Pages 3–4).

We explain how historical tokens are accumulated by the recurrent state. We explain why retrieval activation and state modulation are separated: a scene revisit may make previous observations useful, but does not necessarily require additional modulation of the historical state. This clarifies the limitation of applying a camera-conditioned gate at every recurrent update and motivates L2R's trigger-and-gate design. We also connect frame-level external retrieval to token-level context selection.


3. L2R Formulation and Implementation (Section 3.2, Pages 4–6).

We clarify L2R's information flow and implementation:

  • Recurrent-state access: We specify the roles of rtr_t and γt\gamma_t. When rt=0r_t=0, additional camera-conditioned modulation is inactive, while the regular state transition and current-token update continue.
  • Camera-conditioned gate: We define the shared camera-condition feature ϕt\phi_t and explain how the gate controls retention or suppression of historical state components.
  • Retrieval trigger: We add its explicit computation, including token and camera-condition inputs, learnable weights and bias, sigmoid probability, and hard binary activation with stop-gradient.
  • Training and inference: We clarify the use of offline 3D re-visibility supervision during training and trigger operation at inference without explicit depth-based correspondence or an external geometric retrieval module.

4. Architecture and Evaluation Setup (Sections 3.3 and 4.1, Page 6).

We clarify the complementary roles of recurrent memory and retained softmax-attention layers, and summarize the evaluation settings for memory capacity, video quality, camera controllability, and revisit consistency.


5. Ablation Study (Section 4.4, Page 9).

We clarify the different trigger-learning signals in the trigger-only ablation and fully supervised L2R. The former uses continuous sigmoid gating optimized through the generation objective without explicit re-visibility supervision; the latter uses a hard binary trigger and supervises its pre-binarization probability with 3D re-visibility labels.


6. Quantitative Retrieval-Trigger Evaluation (New Section 4.6, Page 10).

We add an evaluation against offline 3D re-visibility labels on Echo-Memory revisitation cases. We compare L2R with always trigger and random trigger using precision, recall, F1, accuracy, and revisit PSNR. L2R outperforms them.


7. Revisit-Bench Data and Sampling (Appendix A.2, Pages 14–15).

  • We clarify that all 300 source scenes are selected from the held-out test partition of RealCam-Vid, while its training partition is reserved for model training.
  • We detail video-level deduplication, camera-pose validity checks, and motion filtering, and distinguish benchmark-internal deduplication from the original train/test split.

8. Retrieval-Behavior Visualizations (New Appendix E, Pages 19–21).

We add two analyses.

  • Figure 6 visualizes trigger probabilities and 3D re-visibility targets across four revisitation sequences.
  • Figure 7 compares gate distributions for tokens visible 24 frames earlier and tokens newly re-entering the field of view.

9. Conclusion (Page 10).

We revise the Conclusion to summarize the decoupled retrieval design, 3D re-visibility supervision, and evaluations.


We hope these revisions help enhance quality and reproducibility of L2R.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.