- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents SeaMem, a seen-aware memory for long-horizon camera-controlled world generation. The main idea of the paper is to dientangle the reconstruction and generation process. Specifically, it first conducts reconstruction by rendering the scenes reconstructed implicitly from the input tokens into explicit images. Two lightweight predictors then estimate unseen and detail-deficient region masks, which are then used to guide a frozen NVS backbone to go through the generation process. Experiments on RealEstate10K and ACID using SEVA and ViewCrafter demonstrate the proposed metho improves short-range generation quality and long-horizon revisit consistency, with additional component and predictor-feature ablations.
Critical strengths
- The motivation of disentangling generation and reconstruction is good and insteresting.
- Experiments demonstrate the proposed method improves short range memory and generation quality for novel view synthesis.
- The paper is well-written and easy to understand.
Critical weaknesses
-
Although SeaMem stores historical observations as implicit tokens, it still relies on recovered target-view images to condition the downstream NVS backbone, which is explicit, and very similar to those point-cloud-memory based methods. As point cloud could also provide visible regions and invisbile regions, just like the proposed mask prediction module, I would recommend that the authors to provide a comparison with point-cloud-based memory under the same backbone and demonstrate the effectiveness of the proposed method.
-
The other concern of me is that the method appears incremental, mainly combining the original one-stage LVSM network with an extra initialization stage to improve its robustness. However, this two-stage design also involves some possible drawbacks:
a) The additional recovery and mask-prediction stages introduce computational overhead.
b) The recovered content in the first stage may have artifacts and errors which could be preserved and propagated to the downstreaming networks to provide some wrong guidance.
I would recommend that the authors report end-to-end inference time and peak GPU memory usage for different history lengths, evaluate robustness to recovery errors, and clarify how incorrect historical content can be corrected to better justify the proposed two-stage design.
Questions for authors
N.A.
3 comments
Dear Reviewer,
As the discussion phase is approaching its end, we would like to kindly follow up on our previous responses. We have carefully considered your concerns regarding the value of learned generation and the reproducibility of our repair method. If possible, we would sincerely appreciate your feedback on whether our clarifications and additional experiments have adequately addressed your main concerns. Should any questions remain, we would be happy to provide further clarification during the remaining discussion period.
Thank you again for your time and valuable feedback on our work.
Dear Reviewer,
Thank you again for your time and constructive feedback. As the discussion period is nearing its end, we would greatly appreciate your consideration of our latest response above. We hope the additional clarifications and evidence address your concerns, and would be grateful if you could reconsider your assessment in light of them. Please let us know if any concerns remain unresolved; we remain available to clarify them. Thank you for your consideration.
Clarifications on Implicit Memory, Novelty, Computational Cost, and Recovery Reliability
Thank you for your constructive feedback. We address memory representations, contributions beyond LVSM, computational cost, and recovery-error propagation, with additional experimental results.
1. Implicit Memory versus Explicit Point-Cloud Caching (W1)
We agree that SeaMem and point-cloud memory can both provide target-view content and coverage information. However, SeaMem is an implicit memory framework, fundamentally different from explicit point-cloud memory in how historical information is stored, queried, and utilized. Point-cloud memory stores explicit 3D samples and retrieves content through geometric projection. In contrast, SeaMem stores compressed latent tokens and recovers content through camera-conditioned attention, without constructing or rendering an explicit 3D representation. Historical support is estimated from recovery attention statistics rather than projected point occupancy. Importantly, the image-based interface does not change the implicit nature of SeaMem's stored memory. Instead of serving as an ordinary conditioning image, the recovered content and predicted masks guide the NVS backend to selectively generate or refine content in appropriate regions.
Following your suggestion, we have added a comparison with a Gen3C-style point-cloud cache with the same SEVA backend in Appendix D.2, Table 8 (lines 976–985). SeaMem + SEVA achieves better performance across all three settings. These results demonstrate the effectiveness of SeaMem compared with point-cloud cache.
2. Novelty and Contributions Beyond Existing Memory Paradigms (W2)
To our knowledge, SeaMem is the first world-generation memory framework to predict explicit historical-support masks from implicit memory for spatially selective anchoring. Unlike implicit context memory, this guidance reduces unnecessary resynthesis and hallucination-induced inconsistencies. Unlike explicit 3D memory, it avoids reconstruction and fusion that may accumulate geometric errors from inconsistent observations.
Specifically, SeaMem compresses observed views into latent tokens and queries them with the target camera to recover historical content. Recovery attention statistics provide visibility and detail-support cues for lightweight mask predictors. The resulting unseen and detail-deficient masks guide a frozen NVS backend to preserve supported content, refine insufficient details, and synthesize unseen regions.
3. End-to-End Computational Cost (W2)
We have added complete-pipeline measurements at two rollout lengths in Appendix D.1, Table 7 (lines 953–965). We evaluate the 63-view short-range and 243-view long-horizon revisit trajectories, containing 60 and 240 generated targets, respectively. Total generation time increases from 196.1 to 197.5 seconds (+0.7%) for short-range generation and from 797.3 to 824.2 seconds (+3.4%) for long-horizon revisit. Peak GPU memory increases from 11.87 to 12.79 GiB and from 11.87 to 15.10 GiB, respectively. Overall, SeaMem introduces minimal end-to-end runtime overhead and only a modest increase in peak GPU memory usage.
4. Historical Preservation and Error Correction (W2)
Preserving historically supported content is a deliberate design choice in SeaMem. Allowing the generator to freely rewrite previously observed regions may introduce hallucinated changes and cause scene drift during long-horizon generation. SeaMem therefore applies stronge anchoring to well-supported seen regions, weaker anchoring to detail-deficient regions, and no direct anchoring to unseen regions.
To evaluate the reliability of recovered content as a memory anchor, we additionally compare its recovery quality against ground truth in seen, non-detail-deficient regions on both datasets:
These results support the quality of recovered content as a memory anchor. We appreciate your suggestion and will explore explicit error-detection and correction mechanisms in future work.