acceptodds
Withdrawn from ICLR 2027

GLIMPSE: GLOBAL–LOCAL INDEXING WITH MULTI- PRECISION SPECTRAL ENCODING FOR EXPLORABLE VIDEO WORLD MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A camera-controlled video world model must keep a scene consistent when the camera returns to it. Memories that render stored features into the target frame resolve estimated depth and pose into pixel positions, where geometric error becomes ghosting or duplicated structure. Positional transport avoids that resampling by relocating a historical patch’s address rather than its content, yet still encodes the projected coordinate at full precision in every rotary band, whatever the quality of the geometry behind it. An erroneous address retains full positional amplitude and can compete with correctly aligned evidence. We present GLIMPSE, a geometry-addressed memory interface that adapts positional precision and spectral attenuation to geometric reliability. It organizes historical evidence into a geometry-addressed local memory for view-specific appearance reuse and a recurrent global memory for scene-level context, with footprint-guided retrieval under a fixed local read budget. Under matched retrieval and read budgets, GLIMPSE improves geometric correspondence and photometric consistency over deterministic positional transport.

Primary area:
generative models
Keywords:
Video Generation, Camera Control, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qTDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies scene memory in camera-controlled video generation, focusing on recovering consistent structure and appearance when revisiting previously observed regions. The proposed method, GLIMPSE, combines a frozen recurrent global memory with a local patch memory retrieved via geometry. To make memory retrieval less sensitive to projection errors, its STaPE module extends PRoPE by using different coordinate precisions across frequency bands and attenuating spatial frequencies based on geometric reliability and temporal lag. Built on Wan2.2-TI2V-5B, the approach is evaluated on camera-trajectory replay, 100 Y-trajectory revisit cases, and component ablations, supplemented by longer-rollout examples and partial inference-time profiling.

Critical strengths

S1. The Y-trajectory setup is well-motivated. By observing both sides of a scene before pulling back to a wider view, the camera forces the model to recover content from different timestamps. This effectively tests whether the memory design can integrate multiple past observations.

S2. The ablation study is solid, covering coordinate precision, spectral attenuation, the dual memory branches, and positional-error sensitivity. Under fixed retrieval and read budgets (Table 5), full STaPE improves Inliers@5 (5.690K to 6.480K) and reduces RGB MAE@5 (0.135 to 0.109) over continuous PE warping, justifying the proposed modifications within the current system.

Critical weaknesses

W1. The advantages over existing methods in detail recovery and efficiency are not sufficiently established. In the supplementary video, GLIMPSE does not show a consistent edge over GEN3C during revisits; in fact, GEN3C appears to better preserve local details in several examples. This aligns with the trade-off seen in Table 4: while GLIMPSE yields more matches and geometric inliers, GEN3C achieves lower RGB MAE@1/@5 (0.079/0.086 vs. 0.093/0.091). Because these errors are computed only over each method’s own accepted correspondences, they do not establish which method better recovers the overall appearance. Moreover, several margins in the main comparison tables are narrow and lack confidence intervals or multi-run variance to confirm their statistical reliability. Regarding efficiency, while Tables 12 and 14 report partial component timings, critical metrics—such as end-to-end latency, peak GPU memory, history-storage growth, profiling hardware, and baseline timings under comparable settings—are missing. This makes it difficult to judge whether the visual gains justify the added memory and geometric overhead.

W2. Missing comparisons with recent world models and limited evaluation horizons restrict the assessment of the memory contribution. The paper omits comparisons with WorldPlay [R1], Matrix-Game 3.0 [R2], LingBot-World [R3], Lyra 2.0 [R4], and MagicWorld [R5], all of which tackle long-horizon generation or scene retention. Notably, WorldPlay and Lyra 2.0 are already cited and have publicly available inference code and weights. Lyra 2.0 is particularly relevant, as it also uses historical geometry to retrieve observations and establish target-view correspondences to mitigate spatial forgetting. Comparing against these methods is necessary to contextualize the benefits in revisit detail, long-term consistency, and inference cost (or, if differing control interfaces, model scales, or checkpoints preclude a fair comparison, these constraints should be explicitly discussed). Finally, the main quantitative revisit evaluation is capped at 241 frames. The 721-frame examples and timings lack corresponding baseline quality comparisons at that horizon, leaving it unclear whether the memory benefits hold over longer exploration. Taken together with W1, these gaps lead me to lean toward weak rejection.

References:

  • [R1] Sun et al. WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
  • [R2] Wang et al. Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory.
  • [R3] Robbyant Team. Advancing Open-source World Models.
  • [R4] Shen et al. Lyra 2.0: Explorable Generative 3D Worlds.
  • [R5] Li et al. MagicWorld: Towards Long-Horizon Stability for Interactive Video World Exploration.
Questions for authors

Q1. Revisit quality. Where does GLIMPSE improve detail recovery over GEN3C? Could you compare corresponding scene regions to clarify the tradeoff between matching coverage and RGB error in Table 4, and report how the main results vary across scenes or random seeds? Q2. Inference cost. What hardware was used for Tables 12 and 14, and which pipeline stages are included? Could you report end-to-end latency, peak GPU memory, and storage use at different history lengths, with baseline measurements at the same hardware, resolution, and output length? Q3. Long-horizon comparisons. Why were WorldPlay and Lyra 2.0, as well as Matrix-Game 3.0, LingBot-World, and MagicWorld, excluded? Which can be evaluated on the same revisit trajectories, and which are limited by their interfaces or available checkpoints? Do you have memory-consistency comparisons at 721 frames or longer?

4 comments

Thank you for recognizing the value of the Y-trajectory setup and the controlled evidence for STaPE under fixed retrieval and read budgets. We appreciate your specific questions about revisit detail, practical inference cost, and longer-horizon comparisons. We have expanded the evaluation in these directions and present the results below.

Q1. Where does GLIMPSE improve over GEN3C in recovering details? GLIMPSE's clearest advantage is broader geometrically consistent correspondence coverage. In Table 4, GLIMPSE and GEN3C obtain 5.067K and 3.791K Inliers@5, with inlier rates of 79.353% and 77.133%, respectively. GEN3C has a lower RGB MAE (0.086 versus 0.091), but each method is evaluated at its own accepted matches. The support regions therefore differ. This is a trade-off between correspondence coverage and color error on matched regions, rather than a single ranking of overall historical-content recovery. Under shared real reference frames and evaluation regions in replay, GLIMPSE and GEN3C achieve PSNR values of 22.94 and 22.78 dB, and LPIPS values of 0.2683 and 0.2700. GLIMPSE is slightly better on both, complementing the broader geometric correspondence coverage. The paired intervals reported above address variability across cases.

Q2. What is the end-to-end inference cost? We add a comparison with official Lyra 2.0 on a single A800, using the same source image, requested trajectory, and 352×640 resolution. Each method generates 721 frames continuously, with costs recorded at 81, 241, and 721 frames. GLIMPSE uses Wan2.2-TI2V-5B, 50 sampling steps, and CFG 5; Lyra 2.0 uses its official 14B model, 35 steps, and CFG 5. The entries are prefixes of one run per method. Timing starts at the first denoising step and includes subsequent geometry updates, retrieval, and decoding. GLIMPSE includes per-window video saving; Lyra saves after generation. At 721 frames, generation takes 2096.4 and 4334.2 seconds, respectively. Including final saving and cleanup, these become 2120.0 and 4341.5 seconds; including initialization, they are 2389.5 and 5112.9 seconds. The fixed local read budget avoids attending to all historical tokens, while the measured pipeline cost includes growing geometry and cache overhead.

FramesGLIMPSE time (s)Lyra 2.0 time (s)GLIMPSE peak allocated (GiB)Lyra 2.0 cumulative peak allocated (GiB)
81191.1488.644.066.1
241601.51453.544.066.1
7212096.44334.245.168.2

At 721 frames, GLIMPSE's PyTorch allocated and reserved peaks are 45.1 and 78.6 GiB. Lyra's memory figures come from a separate measurement under the same configuration: cumulative allocated peaks at 81 and 241 frames are both 66.11 GiB, with reserved peaks of 66.65 GiB. Its principal history tensors and caches at 81, 241, and 721 frames occupy 2.11, 2.72, and 4.55 GiB on the CPU, and approximately 8.4, 8.5, and 8.5 MiB on the GPU.

Q3. How does GLIMPSE compare with recent methods and over longer horizons? We add comparisons with recent methods at 241 frames and longer-horizon evaluation at 721 frames. We use official models and native sampling settings, including the official 14B checkpoint for Lyra 2.0. Methods receive the same source observations, with the requested trajectories adapted to their respective control interfaces. Recent-method comparison on Y-traj. Within the rebuttal period, we evaluate the five methods on a shared subset of 50 Y-traj. cases, each containing 241 frames. Match and inlier counts are in thousands; the seven quality and consistency scores are normalized scores multiplied by 100. Compared with recent methods that use larger generators or large dedicated training datasets, GLIMPSE does not have an advantage in model or training-data scale, yet remains competitive, particularly in revisit correspondence coverage and appearance consistency.

MetricGLIMPSELyra 2.0WorldPlayMatrix-Game 3.0MagicWorld
Matches@5 ↑ (K)7.4221.9823.9787.0861.198
Inliers@5 ↑ (K)6.5101.7523.3206.5541.043
CLIP-IQA+ ↑20.5310.4928.8422.711.87
Aesthetic score ↑27.6015.7624.7730.067.15
Gram style consistency ↑87.7377.5535.9986.6177.52
DINO consistency ↑83.4782.0385.0182.3269.52
CLIP consistency ↑93.494492.556493.493093.407190.1764
Optical Flow score ↑62.1559.2950.5159.7552.49
Normalized Flow AEPE score ↑65.5354.6980.0155.0057.14
Rotation error ↓ (°)12.98814.62316.24821.94628.905
Translation error ↓1.8612.1531.9972.4322.430

721-frame quality comparison The 721-frame quality comparison uses 20 scenes, with outputs resized to 352×640. CLIP-T measures text alignment across all frames; CLIP-F measures semantic consistency over all 720 adjacent-frame pairs. Scores are averaged equally across cases and multiplied by 100.

MethodFramesCLIP-T ↑CLIP-F ↑
GLIMPSE72133.09298.625
Lyra 2.072132.56797.951
WorldPlay72127.97698.214
Matrix-Game 3.072130.06898.406
MagicWorld72127.22497.872

Long-interval correspondence GLIMPSE does not use post-training specifically designed to suppress visual drift, and accumulated drift becomes a more important bottleneck over sufficiently long rollouts. We will clarify this in the limitations section. To examine long-interval consistency, we evaluate 20 cases at 721 frames. Using only the requested camera trajectories, we select the closest-pose pair between frames 0–240 and 480–720, and use the same pair for each method. Image preprocessing, GIM-DKM matching, and geometric-inlier thresholds follow the original sparse-matching evaluation. All methods in this long-interval evaluation are evaluated on the same 20 cases.

MethodMatches ↑ (K)Geometric inliers ↑ (K)RGB MAE ↓
GLIMPSE8.9028.5010.0641
Lyra 2.08.7107.6510.0829
Matrix-Game 3.09.6028.4470.0823
MagicWorld0.6530.5920.0736
GEN3C8.8818.4400.0702

GEN3C achieves 8.881K matches and 8.440K geometric inliers, with an RGB MAE of 0.0702 on the 20-case evaluation, indicating strong long-interval consistency. GLIMPSE likewise shows competitive geometric correspondence and local photometric consistency. Accumulated drift remains relevant, but these results show that useful historical consistency can persist over longer rollouts.On the same 20-case evaluation, WorldPlay and ReWorld obtain 0.865K and 2.523K matches, 0.825K and 1.798K geometric inliers, and RGB MAE values of 0.0790 and 0.1388, respectively. RGB MAE is computed over each method's valid matching support.

Reviewer_qTDPOpenReview ↗

Reply to the rebuttal

Thanks to the authors for the response and the extra experiments added during the rebuttal.

The new inference cost breakdown (compared with Lyra 2.0) and the 721-frame comparisons against recent world models (like WorldPlay, Matrix-Game 3.0, etc.) are solid and answer my questions on practical overhead and long-term rollouts.

However, for Q1, I am still not convinced about the actual visual improvement of GLIMPSE over GEN3C. I get your point about the trade-off between match coverage (Inliers@5) and RGB MAE, and I see the new PSNR/LPIPS numbers (22.94 vs 22.78). But the reality is, the metrics are extremely close, and I couldn't really see a clear visual advantage over GEN3C in the supplementary videos when recovering details from memory. Because of this, I still feel the actual contribution and novelty of the paper are somewhat limited in this aspect.

So while I appreciate the effort in adding the new baselines, my main concern remains, and I will keep my current score.

Thank you for clarifying your concern. We would like to point to a concrete visual difference in the historic-building example (“Mountain village,” 00:58–01:02 in the supplementary video, particularly at 01:00). During the rightward view, GLIMPSE produces a more coherent roofline and tiled roof, whereas GEN3C exhibits visibly irregular roof tiles and distorted architectural details. These differences are easier to inspect by pausing at the indicated timestamp than during full-video playback. They concern architectural structure and appearance, rather than simply a larger number of detected matches.

We do not base our claim on the 0.16 dB PSNR margin alone. The aggregate scores indicate comparable overall reconstruction quality, but they do not capture every localized structural difference illustrated by this example. Our claim is not that GLIMPSE outperforms GEN3C in every scene or on every metric, but that it provides competitive visual quality while improving geometric correspondence under revisits.

Finally, the novelty of STaPE is distinct from the performance gap between two complete systems. Its contribution is to address uncertain projected locations within memory attention, without resampling historical appearance. Under the same backbone, training recipe, retrieval, and read budget, Table 5 shows that replacing continuous positional transport with STaPE increases Inliers@5 from 5.690K to 6.480K (+13.9%) and reduces RGB MAE@5 from 0.135 to 0.109 (−19.3%). The controlled perturbation experiments further test this mechanism under identical address errors. Thus, the contribution is supported by an isolated, non-trivial improvement in memory addressing, alongside concrete visual benefits—not by a narrow PSNR advantage over GEN3C.

Reviewer_3mTJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLIMPSE, a global-local memory framework for camera-controlled explorable video world models that combines a global recurrent memory built from a frozen CUT3R state with a local pointer memory in which historical patches are indexed by geometry. By introducing STaPE, which assigns band-specific coordinate precision to the rotary frequency bands and attenuates high spatial frequencies according to depth smoothness and temporal lag when addressing retrieved patches in the target view, the authors report improved camera control and cross-view consistency under revisits.

Critical strengths
  • The problem addressed is clearly defined. Existing positional transport assigns full positional precision to historical patches even when the estimated geometry is inaccurate. This paper identifies this as the problem and proposes a solution that adapts only the precision and frequency content of each address to its geometric reliability, without resampling the historical appearance. Experiments with controlled perturbations of the projected coordinates confirm that the degradation is smaller than that of PE warp.
  • The validation is thorough. In addition to the system-level comparison with external methods, the authors conduct ablations that separately isolate the two components of STaPE, the global and local memory branches, whether keys or values are modulated, the reliability signals used to set the attenuation width, and the training augmentations. Furthermore, they propose a new evaluation protocol, Y-traj., that combines regions observed at different times into a single view, which is valuable because it directly tests consistency under revisits.
Critical weaknesses
  • Many of the evaluations do not directly measure whether the model recovers the previously observed content. Although the camera-replay task has a ground-truth source video, the paper reports only distribution-level visual-quality metrics such as FID and FVD and does not include results comparing generated frames against the ground truth using PSNR or LPIPS. The Y-traj. evaluation also relies on the number of matched pixels and geometric inliers between generated frames, but these counts are influenced by image sharpness and texture; the less sharpness-sensitive quantities in Table 4, the inlier rate and the RGB MAE, show much smaller differences, and GEN3C achieves the lowest RGB MAE. Furthermore, although the method uses DA3 geometry for both training and inference, the trajectory errors in Table 1 are also computed from DA3-derived poses, so systems whose outputs are consistent with DA3 may have an advantage on these metrics. Several of the margins over the external methods are also small, and since neither the number of seeds nor confidence intervals are reported for the comparison tables, it is impossible to determine whether these differences fall within the margin of error. The authors should provide evaluations against ground-truth videos, trajectory errors in Table 1 computed with a pose estimator independent of DA3, and statistical support for the reported differences.
  • While the idea of reducing the precision of positional addresses according to their geometric reliability is reasonable, much of the specific design of STaPE lacks justification. In the formula determining the attenuation width σ, the coefficients for depth smoothness and temporal lag are hard-coded, and the authors provide no clear justification for the assumption that older observations yield less reliable transported addresses. In fact, according to the appendix, the correlation between σ and the measured address error is only ρ = 0.33. Regarding multi-precision coordinates, the authors give no reason for dividing the rotary frequency pairs into three equal bands. Furthermore, rounding causes the spatial phase to change discontinuously when a perturbation crosses a bin boundary, so this design does not necessarily make the addressing more robust to projection errors, and its effect on its own is small. The rationale for each design choice, or a comparison with other conceivable alternatives, should be provided.
Questions for authors
  • I understand that in this paper the training augmentations, particularly future-as-memory, which supplies real post-target observations as training-only LPM donors, have a substantial effect. According to Table 7, removing it worsens the rotation and translation errors. In contrast, the external methods being compared presumably do not use this augmentation, and their training data and backbones also differ. I understand that the external comparisons are intended to be system-level and that the internal ablations isolate STaPE; still, how much of the improvement over the external methods is attributable to STaPE, and how much to the rest of the system, including the training recipe? For example, could you provide results comparing the PE warp control in Table 5, which is trained with the same augmentations, against the external methods? Since the evaluation set of Table 5 differs from those of Tables 1–4, this would require running the PE warp control on the latter sets or the external methods on the ablation set. If the PE warp control also outperforms the external methods, I believe that much of the improvement over them is due to these shared components rather than STaPE.
3 comments

Thank you for recognizing the motivation behind reliability-aware addressing, the robustness demonstrated by the perturbation tests, and the breadth of our ablations. We also appreciate your assessment of Y-traj. as a useful test of combining observations across time. Your questions help sharpen the distinction between content recovery, geometric evaluation, and the contribution of STaPE; we address each with the existing controls and additional results below.

Q1. Do the metrics measure content recovery rather than sharpness or self-consistency?

We have added camera-replay PSNR and LPIPS using shared evaluation frames and regions. GLIMPSE achieves the highest PSNR (22.94 dB) and lowest LPIPS (0.2683) among the three methods, providing direct evidence against real target frames.

MethodPSNR ↑ (dB)LPIPS ↓
GLIMPSE22.940.2683
GEN3C22.780.2700
UCM22.070.2850

Pixel-aligned metrics require corresponding real target frames, which are generally unavailable during free exploration. Replaying an observed trajectory also makes it easier to reuse historical 2D frames or patches directly. We therefore introduced Y-traj. to test whether a model can combine complementary regions observed at different times. Table 4 reports not only match counts, but also geometric inliers, inlier rates, and RGB errors, distinguishing correspondence coverage, geometric validity, and local photometric consistency. Replay and Y-traj. thus test complementary aspects: recovery against a real reference and memory composition at a new viewpoint.

Q2. Does using DA3 for both generation and evaluation bias the results?

We address this concern with two additional controls: independent camera evaluation using ViPE on the full Table 1 evaluation set, and inference-time geometry replacement using VGGT on the full Table 5 evaluation set without retraining. In addition, the correspondence and photometric results in Table 5 and the controlled perturbation experiment in Table 10 use GIM-DKM, providing evidence through a separate evaluation pipeline.

Independent camera evaluation

ViPE processes the same generated videos, with evaluation frames, resolution, scale handling, and the original requested trajectories held fixed.

MethodDA3 rotation ↓ (°)DA3 translation NRMSE ↓ViPE rotation ↓ (°)ViPE translation NRMSE ↓
GLIMPSE0.770.1020.790.110
UCM0.880.0590.930.067
GEN3C1.290.3421.310.351

ViPE preserves the rotation and translation rankings obtained with DA3. GLIMPSE achieves the lowest rotation error among the evaluated methods: 0.79°, compared with 0.93° for UCM and 1.31° for GEN3C.

Replacing inference-time geometry

We hold model weights, seeds, inputs, requested trajectories, read budgets, and sampling parameters fixed, and replace DA3 with VGGT for retrieval, projection, and historical geometry updates, without retraining. The DA3 baseline is the full STaPE configuration from Table 5, and both are evaluated with the same matching pipeline.

Inference geometryInliers@5 ↑ (K)RGB MAE@5 ↓
DA3 (Table 5, full STaPE)6.4800.109
VGGT6.5090.113

Both metrics remain close after replacement without retraining, supporting the ability of the GLIMPSE memory interface to work with another geometry estimator.

Q3. Are the reported differences statistically reliable?

Each method uses seed 42 for every case in the paper's comparisons. We additionally compute case-level paired bootstrap confidence intervals for the differences between methods. The paired confidence intervals exclude zero for the reported comparisons, supporting the correspondence improvements and the controlled STaPE gains.

ComparisonMetricFirst meanSecond meanDifference95% paired CI
GLIMPSE − UCM (Table 4)Inliers@5 ↑ (K)5.0674.373+0.694[+0.309, +1.021]
GLIMPSE − GEN3C (Table 4)Inliers@5 ↑ (K)5.0673.791+1.276[+0.752, +1.550]
GLIMPSE − GEN3C (Table 4)Inlier Rate@5 ↑ (%)79.35377.133+2.220 pp[+0.783, +2.944]
STaPE − PE warp (Table 5)Inliers@5 ↑ (K)6.4805.690+0.790[+0.408, +1.187]
STaPE − PE warp (Table 5)RGB MAE@5 ↓0.1090.135−0.026[−0.038, −0.014]

Q4. What motivates the reliability signals, band partition, and quantization?

Depth smoothness and temporal lag provide complementary cues. The former indicates projection risk near surface boundaries; the latter captures the possibility of accumulated geometric drift during exploration. The fixed weights of 0.75 and 0.25 prioritize local depth information while using temporal information as a supplementary cue. Together, these cues define a token-specific reliability proxy. The appendix reports a positive correlation between attenuation width and measured address error (Spearman ρ = 0.329). Table 9 offers a more direct functional test: permuting widths across tokens while preserving their distribution degrades performance, showing that where attenuation is applied matters. The precision groups are fixed; the attenuation width varies with token reliability. Frequency-dependent precision is motivated by the different phase responses to the same coordinate error: a displacement δ produces a larger phase shift at a higher frequency. ReRoPE's analysis of low-frequency redundancy likewise motivates treating rotary bands differently. Our focus is unreliable spatial addresses: integer coordinates at high frequencies, half-cell coordinates at intermediate frequencies, and continuous coordinates at low frequencies. Equal thirds provide a simple fixed allocation across these three representations. The boundary change identified by the reviewer concerns the quantized branch. Full STaPE also retains continuous low-frequency coordinates and attenuates unreliable frequency components. Tables 5 and 10 evaluate the resulting combination through component ablations and matched perturbations. Multi-precision alone reduces RGB MAE from 0.135 to 0.122; when added to attenuation, it increases inliers from 6.097K to 6.480K and reduces RGB MAE from 0.120 to 0.109. These results support the complementarity of the two components.

Q5. How much of the external improvement comes from STaPE rather than the backbone or training recipe?

The PE-warp control in Table 5 retains GLIMPSE's backbone, training augmentations, retrieval, local read budget, and global memory, changing only the addressing mechanism. Following the reviewer's suggestion, we evaluate UCM on that same evaluation set using its official checkpoint and repository-default configuration.

MethodInliers@5 ↑ (K)RGB MAE@5 ↓
UCM5.4080.153
GLIMPSE + PE warp5.6900.135
GLIMPSE + STaPE6.4800.109

GLIMPSE + PE warp already improves over UCM by 5.2% in inliers and 11.8% in RGB MAE, reflecting the combined differences in the shared system configuration. With those settings unchanged, STaPE provides a further 13.9% increase in inliers and a 19.3% reduction in RGB MAE. The total improvements over UCM are 19.8% and 28.8%, respectively. The shared system and training recipe therefore contribute, but do not account for STaPE's additional gain under matched conditions.

We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.

Reviewer_9g4xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present GLIMPSE, a memory module for autoregressive video generation. A common problem with autoregressive generation is long horizon consistency. A simple test like revisiting the same area after walking around for a couple of seconds will show that the scene has completely changed. This is usually caused by limited context length during inference. GLIMPSE overcomes this by paying attention to global context through two branches: the global branch, which pays attention to a state created through CUT3R, and a local branch that retrieves relevant video patches from global context history. GLIMPSE then applies a new positional embedding scheme on these retrieved patches, using a method they call STaPE, which they show through quantitative and qualitative evaluations, performs better than conventional approaches.

Critical strengths
  • Evaluations are thorough and show substantial enough gains in their method compared to relevant baselines. Especially the comparisons on WorldScore is convincing.

  • One of my initial concerns was the disentanglement in efficacy between the retrieval method and the positional embedding. But the authors did a great job in ablating these. Showing that STaPE does at least lead to non-trivial gains compared to conventional RoPE.

  • The STaPE method is simple, but proven to be quite effective. Methods that make minimal changes but realistic gains should be rewarded given that these methods often work even at larger scales.

  • The Y-traj protocol is a nice touch for a principled way of evaluating the revisiting case.

Critical weaknesses
  • I'm not entirely sure if STaPE is effective only for their patch retrieval heuristics or somthing that is generally beneficial when applied to existing patch retrieval methods, or even whole frame retrieval methods for memory.

  • I would only really consider the positional embedding part of this paper a truly novel addition, but even then, it can be difficult to justify that it is truly novel too. As one aspect of it is quantisation, and the other is frequency dampening which has been explored before in prior works.

Questions for authors
  • Can you show that the gains from RoPE to STaPE is independent of the retrieval scheme? Right now STaPE has only been tested on the current patch retrieval heuristic, but it is important to see if STaPE is applicable in the case where other retrieval (whole frame or even no retrieval) strategies are used.

  • The current retrieval scheme is not occlusion aware. Are there cases you can sample where this actually matters? For example, trying to walk through a house with dense corridors and rooms would be a good example. Here, the occlusion matters a lot.

2 comments

Thank you for highlighting the simplicity and effectiveness of STaPE, the controlled separation of retrieval and positional encoding, and the strength of the WorldScore evaluation. We also appreciate your recognition of Y-traj. as a principled revisit test. Below, we address the transfer of STaPE across retrieval strategies, its contribution, and the retrieval design for occluded scenes.

Q1. Does STaPE depend on the proposed patch retrieval strategy?

The gains are not limited to footprint retrieval. With causal z-buffer retrieval and the token budget held fixed, STaPE improves Inliers@5 from 5.949K to 6.852K (+15.2%). This is consistent with the footprint-based comparison in Table 5. STaPE operates on the addresses of already retrieved tokens and does not require changing the retriever. These two controlled comparisons demonstrate its effectiveness under different retrieval strategies.

RetrievalAddress encodingInliers@5 ↑ (K)
Causal z-bufferPE warp5.949
Causal z-bufferSTaPE6.852

Q2. What is novel beyond quantization and frequency attenuation?

STaPE addresses a specific failure mode of video memory: historical content can remain intact while its transported address is unreliable. Preserving original features avoids resampling artifacts, but does not prevent erroneous projections from interfering with attention through positional encoding. STaPE assigns coordinate precision by frequency band and modulates the spatial spectrum of memory keys and values using token-specific reliability. Unreliable addresses therefore no longer participate at full high-frequency amplitude. The contribution is a reliability-aware memory-addressing operator that couples band-wise coordinate precision with token-specific spectral modulation. The experiments test the relevant alternatives directly. Table 5 isolates the two components under fixed retrieval and read budgets. Table 8 compares frequency-selective attenuation with scalar damping. Table 9 preserves the width distribution while permuting its assignment to tokens. Table 10 holds retrieved content fixed and applies identical address perturbations. Together, these results support the roles of frequency selection and token-specific reliability, beyond simply reducing overall memory strength.

Q3. How does retrieval handle occlusion in corridors or rooms?

We use depth-based, coverage-first retrieval: nearby views are considered first, complementary views are selected by their additional coverage, and historical patches are read under a fixed token budget. This design uses geometry to locate relevant content while retaining complementary evidence across views for target-conditioned attention. Under footprint retrieval, Table 5 shows that STaPE improves Inliers@5 from 5.690K to 6.480K and reduces RGB MAE from 0.135 to 0.109. In the additional causal z-buffer control, STaPE likewise improves Inliers@5 from 5.949K to 6.852K, demonstrating its benefits under both retrieval strategies.

We sincerely thank the reviewer for the constructive feedback. We hope our response has addressed your concerns. As the discussion deadline is approaching its end, we wanted to send a gentle final reminder. We fully respect your judgment and simply want to ensure no unresolved concern remains. Please let us know if anything further would be helpful.

Comments on the paper

1 comment

We thank the reviewers for recognizing the clear motivation of GLIMPSE, the simplicity and effectiveness of STaPE, the strength of the controlled ablations, and the value of the Y-trajectory protocol. The reviews highlight that STaPE delivers meaningful gains in geometric correspondence and photometric consistency under matched retrieval and read budgets, and that the ablations carefully distinguish the effects of positional encoding, retrieval, and the global–local memory design. We also appreciate the positive assessment of the broad evaluation, including WorldScore, and of Y-traj. as a principled test of recovering complementary observations across time. These strengths underscore the practical value of reliable memory addressing for camera-controlled video world models. We will upload an updated manuscript PDF incorporating the additional results and clarifications shortly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.