acceptodds
Withdrawn from ICLR 2027

Memory Matters: A Diagnostic Memory Benchmark for Online 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed-forward online 3D reconstruction is a challenging long-context reasoning problem, where large geometry models rely on memory mechanisms to retain historical information and preserve reconstruction accuracy. However, existing benchmarks mainly measure final reconstruction quality, offering limited insight into what memory retains, when it updates, how it handles unreliable inputs, and how efficiently it compresses scene history. We introduce Memo3R, a diagnostic benchmark that evaluates memory in online 3D reconstruction along four dimensions: persistence, plasticity, robustness, and compactness. For each dimension, we design controlled causal protocols and diagnostic metrics that isolate memory behavior. Benchmarking representative implicit- and explicit-memory methods across indoor and outdoor scenes reveals memory-specific trade-offs beyond conventional reconstruction metrics, providing practical guidance for future long-context 3D reconstruction models.

Primary area:
datasets and benchmarks
Keywords:
Memory, Feed-forward Online 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5kpROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Memo3R uses existing datasets and reconstruction metrics to diagnose memory in online 3D reconstruction across four dimensions: persistence, plasticity, robustness, and compactness.

Critical strengths

The problem studied in this paper, how to comprehensively evaluate the memory used in online 3D reconstruction, is highly valuable to research in this field.

Critical weaknesses
  1. As a benchmark study, this paper still relies entirely on existing datasets and commonly used quality metrics. The four proposed capabilities are evaluated mainly through different input settings, which remains considerably limited.
  2. Regarding the compactness, I believe the proposed evaluation primarily measures how well different memory mechanisms tolerate repetitive frames and high-density observations. However, it does not introduce appropriate metrics for measuring changes in memory usage, information compression ratios, or the actual amount of information retained under these input settings. Therefore, I do not believe that the current evaluation truly measures “compactness”.
Questions for authors
  1. Please check the weaknesses.
  2. I would like the authors to provide a more convincing way to evaluate “compactness”.
Reviewer_K5vLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Memo3R, a diagnostic benchmark for evaluating memory mechanisms in online 3D reconstruction. The authors propose four evaluation dimensions along with corresponding evaluation. Experiments on ten methods across datasets shows different memory-related behaviors and trade-offs in long-term information retention, adaptation to scene changes, robustness to corrupted observations, and handling of redundant information.

Critical strengths
  1. A good perspective on memory evaluation. The paper introduces four complementary evaluation dimensions, providing a new framework for analyzing memory behavior in online 3D reconstruction.
  2. Comprehensive experimental evaluation. The paper evaluates diverse memory architectures, providing a broad comparison of existing methods.
  3. Clear presentation. The paper is well-organized, with clearly presented motivations, experimental analyses and illustrations which further facilitate understanding.
Critical weaknesses
  1. Unclear practical value of the proposed memory diagnostics. The proposed metrics characterize different memory behaviors, but their relationship with actual reconstruction quality remains unclear. Since better memory scores do not necessarily imply better reconstruction performance, the paper does not sufficiently demonstrate how these diagnostics provide additional insights or guide future model design.

  2. Limited experimental coverage. Although the experiments cover diverse scenes, each category is evaluated on only one dataset, which makes it difficult to determine whether the observed trends are generalizable or specific to the selected datasets. Additional evaluations on datasets with similar scene would strengthen the conclusions.

  3. The evaluated methods differ substantially and their performance trends are not always consistent across datasets, which may indicate model-specific properties or scene preferences, weakening the conclusions about implicit and explicit memory.

Questions for authors
  1. Generalizability across datasets. Since each scene category is evaluated on only one dataset, could the authors evaluate an additional dataset with similar scene characteristics (e.g., TUM RGB-D dynamic sequences alongside Bonn) to verify whether the observed memory-related trends remain consistent?

  2. Fairness of Cross-Method Comparisons. The comparisons may be affected by differences in backbones, training configurations, and memory strategies. For example, in Sec. 4.1, InfiniteVGGT and OVGGT are used to support the persistence advantage of explicit memory, but both introduce additional memory management strategies over StreamVGGT. Could the authors include StreamVGGT as a baseline to better isolate the effects of KV caching itself?

  3. Practical Implications of Memory Diagnostics. The authors suggest that reconstruction quality alone offers limited insight into what memory retains. Since memory ultimately serves reconstruction, I wonder what we can learn when methods achieve similar reconstruction quality but differ in memory scores, or when better memory scores do not translate into better reconstruction quality. Could the authors provide a concrete example of how such findings could offer practical guidance for future long-context 3D reconstruction models?

Reviewer_J67LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Memo3R, a benchmark for memory in online 3D reconstruction. It evaluates ten models on 7-Scenes, Bonn, and Virtual KITTI 2. The four tests measure reconstruction of an earlier image after intervening frames, adaptation to edited scenes, retention after corrupted inputs, and sensitivity to sampling density. The paper also reports memory-budget comparisons, object-removal tests, and qualitative reconstructions.

Critical strengths

The benchmark asks questions that standard reconstruction scores do not answer and tries to compare different kinds of memory designs in online 3D reconstruction. The motivation is good, and the number of experiments across different methods, datasets, and critical design choices is large.

Critical weaknesses
  1. Training strategies affect the conclusions about the memory design. Section 3 acknowledges differences in training data, sequence length, and optimization. However, reporting relative degradation does not remove these differences. A model trained on noisy inputs may simply be better at handling noise. Dividing by its clean-input error does not remove that advantage. Training length and sampling patterns can likewise affect the retention curves. The results tell us how these checkpoints behave, but do not by themselves show that explicit or implicit memory caused the differences. So I think a table of known training conditions would help, and it is important to isolate the effects of the memory design and the training strategies, as it's the main contribution of this paper.
  2. The test construction introduces another ambiguity. Changing stride changes camera motion and the number of updates, as well as redundancy. In the plasticity test, the comparisons also change history length and repetition. Thus, it makes it hard to tell whether a drop in reconstruction quality comes from difficulty compressing information or adapting to a changed scene.
  3. The common online protocol is not specified consistently. Section 3 says that all methods run frame by frame without chunking, while Appendix D says that LingBot-Map buffers eight images and violates this protocol. What was actually done in the evaluation? The warm-up, scoring, and frozen-memory query rules need to be stated. Other small things, like CD and aggregation details: how are point clouds aligned and scaled?
Questions for authors

My weak-reject rating is mainly due to two concerns: the tests do not yet separate memory design from training and input-history effects, and the online/query conditions are unclear. The following points would affect my assessment:

  1. Training strategies: I want to see a fair experimental design that separates the training strategies (e.g. noise simulation, training sequence length in one batch, ...) and the true effect of the memory design choice, which is the core contribution this paper wants to state.
  2. Density test: What supports interpreting Figure 7 as compactness, rather than sensitivity to motion and the number of updates? A control separating repeated observations from additional scene information would help; alternatively, the claim could be narrowed to sampling-density sensitivity.
  3. Online and query rules: Which frames are scored during Lingbot-Map's initialization, and what observations are available then? For the frozen-memory queries, which state updates are disabled in each model?

I would like to see these concerns addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.