- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies scene memory in camera-controlled video generation, focusing on recovering consistent structure and appearance when revisiting previously observed regions. The proposed method, GLIMPSE, combines a frozen recurrent global memory with a local patch memory retrieved via geometry. To make memory retrieval less sensitive to projection errors, its STaPE module extends PRoPE by using different coordinate precisions across frequency bands and attenuating spatial frequencies based on geometric reliability and temporal lag. Built on Wan2.2-TI2V-5B, the approach is evaluated on camera-trajectory replay, 100 Y-trajectory revisit cases, and component ablations, supplemented by longer-rollout examples and partial inference-time profiling.
Critical strengths
S1. The Y-trajectory setup is well-motivated. By observing both sides of a scene before pulling back to a wider view, the camera forces the model to recover content from different timestamps. This effectively tests whether the memory design can integrate multiple past observations.
S2. The ablation study is solid, covering coordinate precision, spectral attenuation, the dual memory branches, and positional-error sensitivity. Under fixed retrieval and read budgets (Table 5), full STaPE improves Inliers@5 (5.690K to 6.480K) and reduces RGB MAE@5 (0.135 to 0.109) over continuous PE warping, justifying the proposed modifications within the current system.
Critical weaknesses
W1. The advantages over existing methods in detail recovery and efficiency are not sufficiently established. In the supplementary video, GLIMPSE does not show a consistent edge over GEN3C during revisits; in fact, GEN3C appears to better preserve local details in several examples. This aligns with the trade-off seen in Table 4: while GLIMPSE yields more matches and geometric inliers, GEN3C achieves lower RGB MAE@1/@5 (0.079/0.086 vs. 0.093/0.091). Because these errors are computed only over each method’s own accepted correspondences, they do not establish which method better recovers the overall appearance. Moreover, several margins in the main comparison tables are narrow and lack confidence intervals or multi-run variance to confirm their statistical reliability. Regarding efficiency, while Tables 12 and 14 report partial component timings, critical metrics—such as end-to-end latency, peak GPU memory, history-storage growth, profiling hardware, and baseline timings under comparable settings—are missing. This makes it difficult to judge whether the visual gains justify the added memory and geometric overhead.
W2. Missing comparisons with recent world models and limited evaluation horizons restrict the assessment of the memory contribution. The paper omits comparisons with WorldPlay [R1], Matrix-Game 3.0 [R2], LingBot-World [R3], Lyra 2.0 [R4], and MagicWorld [R5], all of which tackle long-horizon generation or scene retention. Notably, WorldPlay and Lyra 2.0 are already cited and have publicly available inference code and weights. Lyra 2.0 is particularly relevant, as it also uses historical geometry to retrieve observations and establish target-view correspondences to mitigate spatial forgetting. Comparing against these methods is necessary to contextualize the benefits in revisit detail, long-term consistency, and inference cost (or, if differing control interfaces, model scales, or checkpoints preclude a fair comparison, these constraints should be explicitly discussed). Finally, the main quantitative revisit evaluation is capped at 241 frames. The 721-frame examples and timings lack corresponding baseline quality comparisons at that horizon, leaving it unclear whether the memory benefits hold over longer exploration. Taken together with W1, these gaps lead me to lean toward weak rejection.
References:
- [R1] Sun et al. WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling.
- [R2] Wang et al. Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory.
- [R3] Robbyant Team. Advancing Open-source World Models.
- [R4] Shen et al. Lyra 2.0: Explorable Generative 3D Worlds.
- [R5] Li et al. MagicWorld: Towards Long-Horizon Stability for Interactive Video World Exploration.
Questions for authors
Q1. Revisit quality. Where does GLIMPSE improve detail recovery over GEN3C? Could you compare corresponding scene regions to clarify the tradeoff between matching coverage and RGB error in Table 4, and report how the main results vary across scenes or random seeds? Q2. Inference cost. What hardware was used for Tables 12 and 14, and which pipeline stages are included? Could you report end-to-end latency, peak GPU memory, and storage use at different history lengths, with baseline measurements at the same hardware, resolution, and output length? Q3. Long-horizon comparisons. Why were WorldPlay and Lyra 2.0, as well as Matrix-Game 3.0, LingBot-World, and MagicWorld, excluded? Which can be evaluated on the same revisit trajectories, and which are limited by their interfaces or available checkpoints? Do you have memory-consistency comparisons at 721 frames or longer?
4 comments
Q3. How does GLIMPSE compare with recent methods and over longer horizons? We add comparisons with recent methods at 241 frames and longer-horizon evaluation at 721 frames. We use official models and native sampling settings, including the official 14B checkpoint for Lyra 2.0. Methods receive the same source observations, with the requested trajectories adapted to their respective control interfaces. Recent-method comparison on Y-traj. Within the rebuttal period, we evaluate the five methods on a shared subset of 50 Y-traj. cases, each containing 241 frames. Match and inlier counts are in thousands; the seven quality and consistency scores are normalized scores multiplied by 100. Compared with recent methods that use larger generators or large dedicated training datasets, GLIMPSE does not have an advantage in model or training-data scale, yet remains competitive, particularly in revisit correspondence coverage and appearance consistency.
| Metric | GLIMPSE | Lyra 2.0 | WorldPlay | Matrix-Game 3.0 | MagicWorld |
|---|---|---|---|---|---|
| Matches@5 ↑ (K) | 7.422 | 1.982 | 3.978 | 7.086 | 1.198 |
| Inliers@5 ↑ (K) | 6.510 | 1.752 | 3.320 | 6.554 | 1.043 |
| CLIP-IQA+ ↑ | 20.53 | 10.49 | 28.84 | 22.71 | 1.87 |
| Aesthetic score ↑ | 27.60 | 15.76 | 24.77 | 30.06 | 7.15 |
| Gram style consistency ↑ | 87.73 | 77.55 | 35.99 | 86.61 | 77.52 |
| DINO consistency ↑ | 83.47 | 82.03 | 85.01 | 82.32 | 69.52 |
| CLIP consistency ↑ | 93.4944 | 92.5564 | 93.4930 | 93.4071 | 90.1764 |
| Optical Flow score ↑ | 62.15 | 59.29 | 50.51 | 59.75 | 52.49 |
| Normalized Flow AEPE score ↑ | 65.53 | 54.69 | 80.01 | 55.00 | 57.14 |
| Rotation error ↓ (°) | 12.988 | 14.623 | 16.248 | 21.946 | 28.905 |
| Translation error ↓ | 1.861 | 2.153 | 1.997 | 2.432 | 2.430 |
721-frame quality comparison The 721-frame quality comparison uses 20 scenes, with outputs resized to 352×640. CLIP-T measures text alignment across all frames; CLIP-F measures semantic consistency over all 720 adjacent-frame pairs. Scores are averaged equally across cases and multiplied by 100.
| Method | Frames | CLIP-T ↑ | CLIP-F ↑ |
|---|---|---|---|
| GLIMPSE | 721 | 33.092 | 98.625 |
| Lyra 2.0 | 721 | 32.567 | 97.951 |
| WorldPlay | 721 | 27.976 | 98.214 |
| Matrix-Game 3.0 | 721 | 30.068 | 98.406 |
| MagicWorld | 721 | 27.224 | 97.872 |
Long-interval correspondence GLIMPSE does not use post-training specifically designed to suppress visual drift, and accumulated drift becomes a more important bottleneck over sufficiently long rollouts. We will clarify this in the limitations section. To examine long-interval consistency, we evaluate 20 cases at 721 frames. Using only the requested camera trajectories, we select the closest-pose pair between frames 0–240 and 480–720, and use the same pair for each method. Image preprocessing, GIM-DKM matching, and geometric-inlier thresholds follow the original sparse-matching evaluation. All methods in this long-interval evaluation are evaluated on the same 20 cases.
| Method | Matches ↑ (K) | Geometric inliers ↑ (K) | RGB MAE ↓ |
|---|---|---|---|
| GLIMPSE | 8.902 | 8.501 | 0.0641 |
| Lyra 2.0 | 8.710 | 7.651 | 0.0829 |
| Matrix-Game 3.0 | 9.602 | 8.447 | 0.0823 |
| MagicWorld | 0.653 | 0.592 | 0.0736 |
| GEN3C | 8.881 | 8.440 | 0.0702 |
GEN3C achieves 8.881K matches and 8.440K geometric inliers, with an RGB MAE of 0.0702 on the 20-case evaluation, indicating strong long-interval consistency. GLIMPSE likewise shows competitive geometric correspondence and local photometric consistency. Accumulated drift remains relevant, but these results show that useful historical consistency can persist over longer rollouts.On the same 20-case evaluation, WorldPlay and ReWorld obtain 0.865K and 2.523K matches, 0.825K and 1.798K geometric inliers, and RGB MAE values of 0.0790 and 0.1388, respectively. RGB MAE is computed over each method's valid matching support.
Reply to the rebuttal
Thanks to the authors for the response and the extra experiments added during the rebuttal.
The new inference cost breakdown (compared with Lyra 2.0) and the 721-frame comparisons against recent world models (like WorldPlay, Matrix-Game 3.0, etc.) are solid and answer my questions on practical overhead and long-term rollouts.
However, for Q1, I am still not convinced about the actual visual improvement of GLIMPSE over GEN3C. I get your point about the trade-off between match coverage (Inliers@5) and RGB MAE, and I see the new PSNR/LPIPS numbers (22.94 vs 22.78). But the reality is, the metrics are extremely close, and I couldn't really see a clear visual advantage over GEN3C in the supplementary videos when recovering details from memory. Because of this, I still feel the actual contribution and novelty of the paper are somewhat limited in this aspect.
So while I appreciate the effort in adding the new baselines, my main concern remains, and I will keep my current score.
Thank you for clarifying your concern. We would like to point to a concrete visual difference in the historic-building example (“Mountain village,” 00:58–01:02 in the supplementary video, particularly at 01:00). During the rightward view, GLIMPSE produces a more coherent roofline and tiled roof, whereas GEN3C exhibits visibly irregular roof tiles and distorted architectural details. These differences are easier to inspect by pausing at the indicated timestamp than during full-video playback. They concern architectural structure and appearance, rather than simply a larger number of detected matches.
We do not base our claim on the 0.16 dB PSNR margin alone. The aggregate scores indicate comparable overall reconstruction quality, but they do not capture every localized structural difference illustrated by this example. Our claim is not that GLIMPSE outperforms GEN3C in every scene or on every metric, but that it provides competitive visual quality while improving geometric correspondence under revisits.
Finally, the novelty of STaPE is distinct from the performance gap between two complete systems. Its contribution is to address uncertain projected locations within memory attention, without resampling historical appearance. Under the same backbone, training recipe, retrieval, and read budget, Table 5 shows that replacing continuous positional transport with STaPE increases Inliers@5 from 5.690K to 6.480K (+13.9%) and reduces RGB MAE@5 from 0.135 to 0.109 (−19.3%). The controlled perturbation experiments further test this mechanism under identical address errors. Thus, the contribution is supported by an isolated, non-trivial improvement in memory addressing, alongside concrete visual benefits—not by a narrow PSNR advantage over GEN3C.
Thank you for recognizing the value of the Y-trajectory setup and the controlled evidence for STaPE under fixed retrieval and read budgets. We appreciate your specific questions about revisit detail, practical inference cost, and longer-horizon comparisons. We have expanded the evaluation in these directions and present the results below.
Q1. Where does GLIMPSE improve over GEN3C in recovering details? GLIMPSE's clearest advantage is broader geometrically consistent correspondence coverage. In Table 4, GLIMPSE and GEN3C obtain 5.067K and 3.791K Inliers@5, with inlier rates of 79.353% and 77.133%, respectively. GEN3C has a lower RGB MAE (0.086 versus 0.091), but each method is evaluated at its own accepted matches. The support regions therefore differ. This is a trade-off between correspondence coverage and color error on matched regions, rather than a single ranking of overall historical-content recovery. Under shared real reference frames and evaluation regions in replay, GLIMPSE and GEN3C achieve PSNR values of 22.94 and 22.78 dB, and LPIPS values of 0.2683 and 0.2700. GLIMPSE is slightly better on both, complementing the broader geometric correspondence coverage. The paired intervals reported above address variability across cases.
Q2. What is the end-to-end inference cost? We add a comparison with official Lyra 2.0 on a single A800, using the same source image, requested trajectory, and 352×640 resolution. Each method generates 721 frames continuously, with costs recorded at 81, 241, and 721 frames. GLIMPSE uses Wan2.2-TI2V-5B, 50 sampling steps, and CFG 5; Lyra 2.0 uses its official 14B model, 35 steps, and CFG 5. The entries are prefixes of one run per method. Timing starts at the first denoising step and includes subsequent geometry updates, retrieval, and decoding. GLIMPSE includes per-window video saving; Lyra saves after generation. At 721 frames, generation takes 2096.4 and 4334.2 seconds, respectively. Including final saving and cleanup, these become 2120.0 and 4341.5 seconds; including initialization, they are 2389.5 and 5112.9 seconds. The fixed local read budget avoids attending to all historical tokens, while the measured pipeline cost includes growing geometry and cache overhead.
At 721 frames, GLIMPSE's PyTorch allocated and reserved peaks are 45.1 and 78.6 GiB. Lyra's memory figures come from a separate measurement under the same configuration: cumulative allocated peaks at 81 and 241 frames are both 66.11 GiB, with reserved peaks of 66.65 GiB. Its principal history tensors and caches at 81, 241, and 721 frames occupy 2.11, 2.72, and 4.55 GiB on the CPU, and approximately 8.4, 8.5, and 8.5 MiB on the GPU.