acceptodds
Withdrawn from ICLR 2027

Matisse: Evidence-Space Reasoning for Active 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How can a 3D reconstruction system acquire and retain useful information to understand the geometry of a scene from partial views under a limited computation budget? Existing active view acquisition methods typically estimate uncertainty over observed or instantiated geometry, limiting their ability to reason about unseen structure, while long-horizon reconstruction methods often retain redundant observations. We introduce Matisse, a training-free framework that unifies active reconstruction and keyframe selection by leveraging evidence provided by a pretrained generative 3D model. Matisse estimates Evidential Uncertainty from cross-attention evidence associated with 3D latent tokens and derives an Evidential Information Gain to guide both view acquisition and keyframe selection based on the expected reduction in posterior entropy. Matisse supports multi-object scenes through occlusion-aware, object-balanced aggregation and propagates uncertainty through intermediate latents to avoid full reconstruction during planning. Matisse reduces Chamfer distance by 12.7%, 3.8%, and 9.2% on GSO30, YCB-V, and Replica, respectively, relative to the best baseline on each dataset, and achieves a end-to-end speedup over the best active reconstruction baseline on GSO30 with the same reconstruction backend. In the GSO30 keyframe selection experiment for long-horizon reconstruction, Matisse achieves comparable Chamfer distance using 14% of the input views compared with Stream3D.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Active Reconstruction, Multi-view 3D Generation, Keyframe Selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AyqWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Matisse reuses a frozen SAM3D generator and Stream3D's canonical evidential memory to guide active reconstruction and keyframe selection. It interprets attention-derived weights as Gaussian observation precisions, obtains token uncertainty alpha/(alpha + accumulated evidence), and approximates information gain by summing uncertainty over visible predicted occupied tokens. Object-balanced aggregation and viewpoint-diversity weighting extend the criterion to multi-object acquisition, while sparse-structure decoding reduces planning cost. The main paper reports GSO30 and YCB-V reconstruction results and a GSO30 keyframe experiment; additional validation is provided in the appendices.

Critical strengths
  1. Using one spatially shared evidence representation for both acquisition and retention is an interesting conceptual direction. It connects two decisions that are often treated separately and gives a simple diminishing-return rationale for directing observations toward weakly supported regions. The Gaussian derivations are internally consistent under their stated assumptions.
  2. The implementation makes candidate evaluation inexpensive without uncertainty-specific training. Planning uses sparse-structure decoding rather than full structured-latent generation, and candidate scores are batched on the GPU. This offers practical value for active reconstruction with a costly frozen generative backbone.
Critical weaknesses
  1. The comparisons do not establish the full acquisition-and-retention claim. Figure 3 compares selections from a spiral trajectory with Stream3D's chronological processing, without equal-budget random, uniform, or pose-diversity selections from the same candidate pool. The additional YCB-V keyframe study appears only in Appendix C.6 and repeats the selected-frames-versus-prefix comparison. Thus, the 14% view-count result does not isolate the scoring contribution. The abstract also highlights a 9.2% Replica CD reduction, whereas Section 6.3 states the gain and places the complete quantitative comparison in Appendix C.1. Part of the headline cross-dataset evidence is consequently not directly assessable from the main-paper comparisons. The GSO30 and YCB-V results deserve credit, but they cannot substitute for evidence supporting the broader claims. The appendix also does not report reconstruction uncertainty on method differences, and MAGICIAN's beam search is disabled on Replica. These limitations concern evidence sufficiency and comparability; supplementary experiments are not inherently invalid or prohibited.
  2. The main paper does not supply sufficient direct validation of the central uncertainty interpretation. Theorems 1 and 3 assume independent Gaussian observations and define their precision to equal the attention-derived weight. They do not establish that actual attention scores measure geometric reliability or that correlated views provide independent evidence. The practical score further substitutes binary visibility for future evidence and uses a first-order entropy approximation. Section 6.4 announces visibility, symmetry, and model-confidence analyses but places all corresponding evidence in Appendix D; numerical pseudo-evidence sensitivity is also deferred to Appendix C.4. These analyses concern the central formulation, rather than peripheral detail. Because ICLR explicitly states that reviewers are not required to read appendices, the essential validation should be sufficiently presented in the nine-page main paper. I examined the appendix, and its examples still do not test posterior calibration or whether predicted gain ranks actual reconstruction improvements. Moreover, covariance over tokens in a decoded completion does not establish uncertainty that is global in scene-hypothesis space. The issue therefore extends beyond presentation to the validity and scope of the interpretation.
Questions for authors
  1. What direct evidence validates attention-derived precision and predicted information gain against geometric reliability and actual improvement from acquiring a view? Please identify the support in the main paper and explain how repeated or nearly identical views affect the independence and uncertainty assumptions.
  2. Does the keyframe advantage persist against random, uniformly spaced, and pose-diversity selections at the same view budget and from the same full candidate pool? Please clarify whether selection has access to the entire sequence's poses or must operate causally.
  3. What paired uncertainty estimates support the headline 9.2% Replica gain, and how sensitive is the comparison to the disabled MAGICIAN beam search? The complete comparison is currently confined to Appendix C.1; clarifying the evidence and baseline configuration could materially change the assessment of cross-dataset generality.
Reviewer_CdQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Matisse, a training-free framework that unifies active 3D reconstruction and keyframe selection using evidence from a pretrained generative 3D model. The method formulates the evidence memory in Stream3D as a Gaussian estimation problem, derives an Evidential Uncertainty (EU) measure from the posterior covariance, and uses an approximate Evidential Information Gain (EIG) score to select informative views. The framework also incorporates an exploration factor and supports multi-object scenes through occlusion-aware, object-balanced aggregation.

Experiments on GSO30, YCB-V, and Replica demonstrate promising reconstruction performance. The paper reports relative Chamfer Distance reductions of 12.7%, 3.8%, and 9.2%, respectively, compared with the best-performing baseline on each dataset. It also demonstrates the potential of the proposed criterion for reducing the number of input views in keyframe selection. While the problem is relevant and the framework is practically appealing, the empirical justification of the uncertainty formulation and the approximation used for information gain requires further investigation.

Critical strengths

S1. A unified and efficient framework for observation selection.

Matisse uses a shared evidence-space formulation for active viewpoint selection and keyframe selection. Reusing a pretrained generative reconstruction pipeline without training an additional selection policy is attractive from a practical perspective. The use of intermediate representations to evaluate candidate views also offers a plausible way to reduce the computational overhead of planning.

S2. Promising results across multiple reconstruction settings.

The evaluation covers single-object reconstruction, multi-object scenes, and indoor environments. The reported improvements on GSO30 and Replica are encouraging, while the YCB-V results show a smaller but positive improvement in Chamfer Distance. The additional keyframe-selection experiments also suggest that the proposed criterion can reduce the number of retained observations while maintaining comparable reconstruction quality in the evaluated settings.

Critical weaknesses

W1. The empirical validation of Evidential Uncertainty remains incomplete.

The authors interpret Stream3D's cross-attention evidence weights as contributions to observation precision and derive EU from a Gaussian posterior covariance. Appendix D provides useful sanity checks, showing that uncertainty responds plausibly to visibility, approximately respects object symmetry, and generally increases as image details are removed by Gaussian blur.

However, these analyses do not directly establish whether EU predicts actual geometric reconstruction errors. The paper does not report a quantitative correlation or ranking analysis between token-level EU and ground-truth geometric errors, nor does it establish whether EU provides an advantage over simpler alternatives such as raw evidence magnitude.

This distinction matters because EU directly determines the subsequent information-gain score. If EU does not reliably identify regions with greater reconstruction errors, the theoretical motivation for using it to guide view selection becomes less convincing. A quantitative evaluation relating predicted uncertainty to actual reconstruction errors would strengthen the central claim.

W2. The practical information-gain approximation lacks direct empirical validation.

The paper derives an information-gain expression under a Gaussian model and explicitly states the assumptions behind the first-order approximation. The practical score then replaces unknown future association weights with binary geometric visibility. This approximation is computationally attractive, but visibility alone does not necessarily determine how much useful evidence a future observation will provide.

Although Appendix A.2 provides the mathematical justification, the paper does not directly evaluate whether the approximate score in Eq. (7) preserves the candidate-view ranking induced by Eq. (6), or whether it reliably predicts the actual reconstruction benefit of selected views. The small-increment assumption may also be questionable for tokens with little accumulated evidence under the binary-visibility approximation.

These concerns do not imply that the approximation is incorrect. Rather, its practical reliability remains insufficiently demonstrated. A quantitative comparison of candidate-view rankings and approximation errors would help establish whether the proposed score is a reliable surrogate for information gain.

Questions for authors

Q1. Validation of EU. Can the authors report a quantitative relationship between token-level EU and ground-truth geometric reconstruction errors? How does EU compare with raw evidence magnitude or visibility-based measures in identifying regions with large reconstruction errors?

Q2. Reliability of the EIG approximation. How closely does the approximate score in Eq. (7) preserve the candidate-view ranking of Eq. (6)? Can the authors evaluate ranking consistency or compare the predicted information gain with the actual reconstruction improvement obtained after executing selected views?

Q3. Contribution of the information-gain formulation. Table 3 shows that removing information gain degrades the reported metrics, which supports its practical utility. However, what exact selection rule is used in the “w/o Information Gain” variant? Can the authors compare the proposed method with uniform visible-token coverage to clarify the contribution of uncertainty weighting?

Reviewer_TKsMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on active 3D reconstruction that proactively seeks the next best views and keeps informative views from redundant view sampling. It reformulates the Adaptive Evidential Memory (AEM) as a measure for reducing the posterior entropy of the latent 3D features from 2D view observations. Experiments show that the method achieves better results than previous methods.

Critical strengths
  • The idea of reinterpreting and reformulating the existing AEM into a probabilistic model is interesting
  • The model might be among the first few to explore active reconstruction for latent diffusion 3D models
  • The shown results are better than it's competitors
Critical weaknesses
  • Insufficient dataset samples

While I appreciate the better performance across three datasets, the sampled objects are much smaller than in the original dataset. For instance, GSO contains 1030 objects, but the paper samples only 30. Why did you choose those 30 objects, and what's the performance on the full dataset? This leaves the results a bit less convincing.

  • Limited Improvements over Random Sampling

It's a bit surprising to find in Tables 1 and 2 that the default "Random" setup generally ranks 2nd in geometry with a limited margin compared to the proposed method. Moreover, the other methods show no advantage over the "Random" setup. That drives me to question the real usefulness of the proposed approach.

  • Limited applications scope

While the methodology is interesting, it builds on a precomputed AEM score under Steam3D; it's not yet verified whether other related approaches using the same latent grids (TRELLIS, SAM3D) can work similarly well under this approach. The applicability is therefore limited. Considering that the improvement is relatively limited compared to "Random", it seems to be a critical issue.

Questions for authors
  • Evaluation under a larger/full subset of GSO
  • Explain the relative marginal improvement over "Random"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.