acceptodds
Withdrawn from ICLR 2027

Lightweight Test-Time Geometry Unification for Multi-View Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feed‑forward 3D reconstruction models can predict several offset sheets for one physical surface. Reconciling these predictions requires both cross‑view agreement and preservation of local surface structure: independent voxel shifts can narrow layers while introducing cracks between neighboring pixels. We introduce a self‑supervised test‑time refiner that addresses these coupled requirements without target‑scene depth or 3D reference supervision. Closed‑form per‑view similarity alignment first corrects rigid‑plus‑scale drift. A 0.9 M‑parameter residual network then corrects the remaining non‑rigid component of multi‑view geometric inconsistency, using multi‑view image features and geometric self‑supervision with displacement‑field regularization to preserve local structure. Wall‑corner and curved‑edge comparisons show that this second stage brings residual sheets closer together while retaining surface continuity. It reduces cross‑view centroid gaps in all 53 evaluated NRGBD/DTU scene–model cases. On 6 658 patches from six NRGBD scenes, its adjacent‑pixel GT edge‑vector error is lower than full‑strength direct () and smoothed voxel shifts in 94.7% and 87.5% of cases, respectively. The same refiner applies to Pi3 and VGGT‑ without per‑base retraining, improving Accuracy and Normal Consistency across the evaluated object‑level, indoor, and outdoor benchmarks. The complete pipeline takes about 3.7 minutes for a representative 15‑view scene on one A800 GPU. A fixed‑count analysis shows that redundant layers can lower Completeness error while raising Accuracy error, motivating evaluation of surface structure alongside point distances.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-view reconstruction, feed-forward 3D reconstruction, test-time refinement, geometric consistency, self-supervised learning, point maps

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xQkXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a lightweight test-time geometry unification framework for reducing cross-view layering artifacts in feed-forward multi-view reconstruction. The method consists of a closed-form per-view Sim(3) alignment stage followed by a small image-conditioned residual network optimized per scene using cross-view geometric consistency, soft collapse, and displacement-field regularization. The framework is evaluated with Pi3 and VGGT-Ω\Omega on DTU, ETH3D, ScanNet++, NRGBD, and 7-Scenes, together with detailed controls on cross-view gaps, corresponding-pixel errors, local surface structure, and Accuracy/Completeness trade-offs.

Critical strengths
  1. The paper carefully studies a meaningful failure mode of feed-forward multi-view reconstruction. Rather than relying only on standard nearest-neighbor reconstruction metrics, the paper explicitly evaluates cross-view sheet separation, corresponding-pixel position error, local edge structure, and the Accuracy/Completeness trade-off. This provides a more informative analysis of geometry unification than standard point-distance evaluation alone.

  2. The experimental analysis is unusually transparent and comprehensive. The method is evaluated with two different reconstruction backbones and across object-level, indoor, and outdoor datasets. The supplementary material also reports negative results and metric regressions, including cases where Completeness or symmetric distance worsens after refinement, which improves confidence in the reported empirical observations.

Critical weaknesses
  1. Stage 2 operates on geometric associations that are constructed before Stage 1, even though Stage 1 explicitly changes the geometry. The paper states that voxel assignments, camera associations, and multi-view reprojection results are cached before Stage 1 and reused during Stage 2. However, Stage 1 subsequently applies an independent Sim(3) transformation to every view. For example, after a view is translated, rotated, or rescaled, a point may belong to a different voxel, project to a different image patch, or change its visibility in another view, while Stage 2 still uses the associations computed from the pre-alignment geometry. The supplementary material further confirms that tokenization and reprojection are not rebuilt after alignment and that the original camera centers are retained. This creates an inconsistency between the geometry being refined and the spatial/image associations conditioning the refinement network.

  2. The Stage-2 objective leaves non-trivial geometric deformations unconstrained. Both the consensus and collapse terms constrain disagreement primarily along estimated surface normals, while the smoothness term penalizes second differences of the displacement field. As a concrete example, for a planar surface with normal n=(0,0,1)n=(0,0,1), a constant tangential displacement d=(a,0,0)d=(a,0,0) produces zero normal displacement for the consensus/collapse terms and zero second difference for the smoothness term. Thus, the geometry can undergo a systematic tangential shift without being penalized by any of the three losses. The supplementary material itself notes that the regularizer does not bound tangential translation or affine distortion, leaving a non-trivial null space in the core test-time objective.

Questions for authors

Questions for Authors

  1. Why does Eq. (2) fit every point in a voxel to the same cross-view centroid rather than aligning per-view voxel centroids to the consensus? For points in one voxel written as pi=μm,v+ϵip_i=\mu_{m,v}+\epsilon_i with ∑iϵi=0\sum_i \epsilon_i=0, the corresponding objective can be decomposed as ∑i∥sRpi+t−cm∥2=nm∥sRμm,v+t−cm∥2+s2∑i∥ϵi∥2\sum_i \lVert sRp_i+t-c_m\rVert^2 = n_m\lVert sR\mu_{m,v}+t-c_m\rVert^2 + s^2\sum_i\lVert\epsilon_i\rVert^2. The second term penalizes the within-voxel spread proportionally to s2s^2 and therefore appears to introduce a bias toward shrinking the fitted scale, even though the stated purpose of Stage 1 is to correct inter-view Sim(3) drift. Is Eq. (2) exactly the objective used in the implementation? If so, how is this scale-shrink bias avoided beyond rejecting fitted scales outside [0.5,2][0.5,2]?

  2. How are the coordinate frames kept consistent between Stage 1 and Stage 2? Stage 1 applies an independent Sim(3) transformation pi′=svRvpi+tvp_i' = s_vR_vp_i+t_v to each view, while the supplement states that tokenization and reprojection are performed before alignment and are not rebuilt afterward, and that the initial camera centers are retained for Stage-2 viewing weights. After the point geometry has changed, why are the pre-alignment voxel assignments, visibility relations, image-patch associations, and camera centers still geometrically valid for Stage 2?

  3. What prevents Stage 2 from drifting along tangential or locally affine directions that are not constrained by the stated losses? Both LconsL_{\mathrm{cons}} and LcolL_{\mathrm{col}} constrain primarily along-normal displacement, while LsmoothL_{\mathrm{smooth}} penalizes displacement curvature. For a planar surface with normal n=(0,0,1)n=(0,0,1), a constant tangential displacement d=(a,0,0)d=(a,0,0) has d⊤n=0d^\top n=0 and zero second difference, so it can remain unpenalized by all three terms. The supplement also notes that the regularizer does not bound tangential translation or affine distortion. Is there an additional constraint or architectural property that removes this null space in practice?

  4. How should the contribution of the learned Stage-2 refiner be interpreted relative to filtering and Stage 1? On the nine-scene NRGBD control, CD changes from 13.643 mm for the raw base output to 8.636 mm after filtering and 8.180 mm after Stage 1, while Stage 2 further changes it only to 8.127 mm. On DTU, Stage 2 improves Accuracy but worsens Completeness and symmetric distance for both Pi3 and VGGT-Ω\Omega. Should the principal claim of Stage 2 therefore be understood specifically as improving cross-view consolidation and point-position accuracy rather than consistently improving overall reconstruction quality?

  5. How sensitive is the method to the arbitrary scale of the predicted coordinate frame? The voxel sizes, Huber widths, displacement scale, and several activation thresholds are specified in predicted units rather than metric units, and the supplement explicitly notes that rescaling the input geometry can change voxel groups and active thresholds. Is any normalization applied before refinement to ensure that the same hyperparameters have comparable meaning across scenes and different base reconstruction models?

Reviewer_JvjYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses “layering” in feed-forward multi-view reconstruction, where a single surface is reconstructed as multiple offset sheets across views. The proposed approach aims to reconcile these predictions at test time, without ground truth, by enforcing consistency between overlapping views in the predicted coordinate frame. It combines a per-view similarity alignment with a learned non-rigid refinement optimized for each scene. The main novelty is the image-conditioned, self-supervised test-time refinement applied directly to the decoded point maps of a frozen backbone, without rendering, explicit feature matching, or backbone adaptation. The approach is evaluated on DTU, ETH3D, ScanNet++, NRGBD, and 7-Scenes using two backbones. The results show improvements in accuracy and normal consistency over the base models and competing feed-forward and test-time methods, supported by analyses of cross-view gaps, local edges, and paired positions.

Critical strengths

The paper adresses a relevant problem. Layering is a real and visible limitation of current feed-forward reconstruction models, and refining their outputs without modifying the backbone is a practical setting.

The proposed pipeline is backbone-agnostic. The same refiner applies to e.g. Pi3 and VGGT-Ω without retraining, and improves accuracy and normal consistency on several benchmarks.

The supplement reports informative controls (filtering, alignment, PCA, direct shifts, free offsets) as well as regressions, which allows a fair assessment of each stage’s contribution.

Critical weaknesses

My main criticism concerns the overall positioning of the paper. The method essentially refines feed-forward predictions to improve cross-view consistency, a problem traditionally addressed by bundle adjustment and depth fusion. The uncalibrated setting is inherited from the backbone, whose cameras the method keeps essentially fixed, and although it avoids explicit feature matching, it relies on implicit geometric and appearance correspondences. Yet established refinements initialized from the same predictions (bundle adjustment, depth fusion, Gaussian-splatting methods such as InstantSplat, 2DGS or PGSR) are not compared, and rendering-based methods are dismissed based on outdated NeRF/NeuS runtimes.

The core contribution has marginal impact. According to the supplement’s own controls, most of the reported gain comes from input filtering and per-view alignment. Stage 2, presented as the main contribution, only marginally reduces the paired position error relative to alignment alone (Table 2), and it consistently worsens Completeness on DTU. Its local edge error is also slightly higher than with alignment alone, so the claim of preserved surface structure rests mainly on comparisons with deliberately strong direct shifts and on selected examples.

The fixed-count analysis of layering, listed among the contributions, shows that offset sheets lower completeness error while raising accuracy error. This is a direct instance of a well-known property of one-sided distance metrics: redundant off-surface points favor completeness at the expense of accuracy, which is why MVS benchmarks such as Middlebury (Seitz et al., 2006) and Tanks and Temples (Knapitsch et al., 2017) report both directions or an F-score. Isolating the effect of layering at a fixed point count is a clean illustration and helps interpret the completeness regressions of the method, but it is not a contribution in itself.

Questions for authors

Comparison with standard refinements. How does the method compare, in accuracy, completeness and runtime, with bundle adjustment followed by depth fusion, and with Gaussian-splatting refinements (e.g. InstantSplat, 2DGS), all initialized from the same Pi3/VGGT predictions? Are there regimes (few views, low texture) where these fail and your method succeeds?

Effect of Stage 2. The gain in paired position error over alignment is small. Can you show cases or metrics where Stage 2 has a clear, practically relevant benefit, for example on downstream meshing or normal estimation?

Completeness regressions. Is the Completeness degradation on DTU only due to the removal of duplicated sheets, or also to a misplaced consensus surface? A precision/recall or F-score at fixed thresholds would help separate the two effects.

Reviewer_mkqYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the layering artifact in feed-forward multi-view reconstruction models such as Pi3 and VGGT-Ω, where a physical surface is predicted as several offset sheets by different views, and proposes a lightweight test-time refinement.

The framework contains 2 stages: Stage 1 fits a closed-form per-view Sim(3) transform toward the cross-view voxel-centroid consensus, which removes per-view rigid-plus-scale drift. Stage 2 optimizes a 0.9M-parameter residual network per scene for 300 steps with three self-supervised losses, namely cross-view centroid consensus, soft per-view collapse, and a displacement-field Laplacian. The goal of this stage is to reduce the remaining non-rigid offsets while largely preserving local surface structure.

Experiments on DTU, ETH3D, ScanNet++, NRGBD, and 7-Scenes report improved Accuracy and Normal Consistency for both backbones.

Critical strengths
  1. The problem is real and clearly defined. Figure 1 shows the layering artifacts of per-view decoded feed-forward models in along-normal cross-sections, and correcting them is worthwhile.

  2. The method is lightweight, with a 0.9M-parameter refiner applied unchanged to both Pi3 and VGGT-Ω.

  3. The evaluation is broad, covering several datasets, and the experiments are extensive, including a thorough set of controls and per-scene results in the supplement.

Critical weaknesses
  1. The gain from Stage 2 is not established, and Ours does not clearly outperform the parameter-free Direct control. On DTU, Supplementary Table 3 shows that almost all of the Accuracy reduction comes from input filtering and Stage 1 (1.217 to 1.063 to 0.816 mm), while Stage 2 adds only 0.01 mm and worsens Completeness and CD on all 22 scenes. On NRGBD the CD gain from Stage 2 is 0.05 mm, and no breakdown is given for ETH3D or ScanNet++. Meanwhile, Direct with α = 0.25, a scalar interpolation with no network and no optimization, achieves lower CD on DTU, lower Edge-8 error in all three cohorts, and a comparable gap on NRGBD (Supplementary Tables 5 and 6), with Ours leading only on paired error by 0.07 to 0.6 mm. Given that Stage 2 accounts for 99 percent of the runtime, the evidence does not justify its complexity.

  2. The evidence for Stage 2 in Section 4.3 is not convincing. The only metric Ours consistently wins is the paired error, by 0.05 to 1.4 percent with no variance reported. Gap reduction is what the losses optimize, so it is not independent evidence. Every comparison is against Direct with α = 1, the worst setting in the authors’ own sweep, so the 94.7 and 87.5 percent win rates in the abstract are over the weakest control. The Figure 4 examples were picked for beating Direct, and the patch audit covers six of nine NRGBD scenes with no stated reason. The authors disclose all of this in the supplement, but it leaves the main text’s claims about Stage 2 without much support.

  3. I could not find evidence that any of the three losses is actually needed. Dropping the collapse loss gives better CD on NRGBD (7.843 versus 7.967, Supplementary Table 8) and better Accuracy and NC on DTU (Supplementary Table 9), and the authors admit that no loss is claimed necessary for every metric. L_cons and L_col also seem to work against each other, since one lets the surface float and the other pins it to Stage 1. And the target of L_col is just the Direct α = 1 shift the paper argues against, so in the end the network is learning something between Align and Direct, which a single α already gives you.

  4. The paper is hard to read, and a good part of that is avoidable. The introduction already points to Figure 4 (page 8) to explain what Stage 2 does, before Stage 1, Stage 2, Align, or Direct have even been introduced, and the pipeline overview in Figure 2 is never cited anywhere in the main text. The abstract talks about “Direct (α = 1)”, but I had to go to Supplementary Section 2.1 to find out what Direct and α actually are. The same metric shows up as paired error, fixed-pixel error, and corresponding-pixel error depending on the section. The gates, weights, and normals for the three losses are all “in the supplement”, so Equations 5 to 7 cannot really be followed from the main paper. And the conclusion is four sentences with no limitations, even though the supplement is quite open about them.

  5. Several details needed to reproduce the method are missing. The confidence and depth-discontinuity filtering is never specified, even though it accounts for most of the gain (Supplementary Tables 1 and 3). The network is described in a few sentences, with no information on the image encoder, how viewing directions enter the attention, or how voxel tokens are gathered for the offset MLP, and the only architectural detail given is a config name (“3d static2d”). The ScanNet pretraining the method warm-starts from is not described at all, yet random initialization needs five times more steps to catch up.

Questions for authors

Overall, I think the problem this paper goes after is a meaningful one. But the contribution of Stage 2 is not separated from filtering and Stage 1, the evidence in Section 4.3 does not hold up against the authors’ own Direct sweep, and too many details needed to reproduce the method are missing. That is what drives my current score. My specific questions are laid out in the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.