acceptodds
Under review as a conference paper at ICLR 2027

NativeRefine: Nuisance-Aware Refinement of Feed-Forward 3D Geometry

28 views

Abstract

We introduce NativeRefine, a portable, training-free refinement layer that turns frozen feed-forward 3D predictions into scene-adapted geometry using RGB correspondences. Its key principle is to separate fitting freedom from output complexity. Bounded, centered source-ray variables provide temporary structural flexibility for camera fitting; a second stage fixes the fitted cameras and refits a compact, native-relative depth field. The emitted reconstruction retains native detail and preserves the window-wide geometric mean of depth exactly. Across four frozen backbones and 600 windows from three benchmarks, 2,400 model–window evaluations demonstrate broad applicability without retraining or architecture-specific tuning. With known intrinsics, point and shared-scale translation errors improve together in 10 of 12 cohort–backbone cells against each of native prediction and bundle adjustment with the same dense refit. Two additional recordings tested after method lock retain translation gains of 27.6% and 26.9% over native MapAnything and Pi3X. RGB-only transfer to VGGT and DA3 improves point error in 9 of 12 cells without external calibration. The complete 1,200-evaluation extension runs on a 16 GB consumer GPU, bringing compact, native-preserving optimization to existing reconstruction models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Feed-forward 3D reconstruction, Geometric refinement, Correspondence matching, Test-time optimization, Multi-view geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e9xjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

A training-free, test-time refinement layer for frozen feed-forward multi-view 3D predictions, driven only by RGB correspondences. Stage 1 jointly updates anchored cameras and a smooth depth field, with one bounded nuisance variable per unique matched source ray. Stage 2 fixes the fitted cameras, discards the nuisance, and refits the persistent depth field to the original correspondences. Evaluated across multiple frozen backbones and datasets against the unrefined prediction are done.

Critical strengths

Portable and architecture-agnostic. One fixed configuration is applied unchanged across several distinct backbones, requires no training.

Critical weaknesses
  1. Comparison to prior work is too narrow to support the novelty of the method. Related work surveys a substantial line of directly relevant methods, but only one is actually run, and only on part of the data and metrics — the rest are cited, but not compared against. Without testing against the closest existing methods, it’s hard to tell whether the gains come from a genuinely new mechanism.

  2. The relationship to that prior work is not clearly stated, which makes the paper’s own contribution hard to pin down. Several cited methods already combine a frozen/learned prediction with test-time, correspondence-driven correction — conceptually close to this paper’s core idea. The paper doesn’t explain what specifically differs from them, so a reader can’t tell which parts of the design are new versus inherited.

Questions for authors

1.With the temporary per-ray variables removed, the two optimization stages appear to share the same objective over the persistent depth field once the camera is fixed. How much of the two-stage design’s benefit holds independently of the per-ray mechanism, versus being attributable to that mechanism alone?

  1. The external baseline is evaluated only on camera accuracy and underperforms the unrefined prediction in most of those cases. Could the authors clarify why. Is this a genuine limitation of that method, or a difference in configuration between the two methods?
Reviewer_MeMrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents the NativeRefine framework for refining dense geometry and cameras using RGB correspondences. The framework leverages a two-stage design that separates the fitting freedom from the final output. The first stage optimizes cameras through temporary source-ray adjustments and the second stage discards the temporary variables and refits the depth field by fixing the cameras. To demonstrate the effectiveness of the proposed framework, the experiments span four frozen backbones and more than 2000 model-window samples. The results show improvements in both dense points and camera pose estimation.

Critical strengths
  • The idea of decoupling the camera fitting process and the final dense representation is interesting and novel.
  • The framework does not require backbone tuning.
  • The framework is tested across four frozen backbones to show the generalizability of the framework.
Critical weaknesses
  • The improvement of using a decoupled two-stage design is not consistent, and the discussion in Section 4.5 emphasizes improvements only in favorable settings and metrics. However, Table 19 shows that the full 2-stage method improves both endpoints in only one of four settings (MapAnything / TartanAir).
Questions for authors
  • The effect of introducing nuance needs to be clarified.
Reviewer_7nJqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NativeRefine is a training-free framework for refining depth and camera poses predicted by frozen feed-forward 3D reconstruction models using fixed RGB correspondences. Its central design separates the representation used during optimization from the representation retained in the output. Stage 1 jointly optimizes cameras, a smooth native-relative depth correction, and temporary per-source-ray depth variables. Stage 2 discards the temporary variables, fixes the fitted cameras, and refits the smooth depth correction. The final depth preserves the native prediction’s window-wide geometric mean.

Critical strengths
  • The method addresses a concrete optimization problem. A compact depth representation can restrict camera fitting, while unconstrained local corrections can compromise the final reconstruction. Separating temporary fitting variables from persistent output variables is a reasonable approach to this tension.
  • The framework operates without retraining the backbone. Its depth-and-camera interface allows application to multiple reconstruction models without architecture-specific refinement networks.
  • The calibrated experiments show useful improvements. NativeRefine improves both primary endpoints over native predictions in 10 of 12 cohort–backbone combinations. Some translation gains are substantial, particularly for MapAnything on TartanAir.
  • The paper includes relevant mechanism controls. No-nuisance fitting, direct nuisance emission, warm versus cold refitting, and camera-only substitutions provide evidence beyond comparisons against native predictions. These controls expose meaningful tradeoffs, although the main narrative does not fully reflect them.
  • The evaluation discloses several important limitations. The manuscript distinguishes known from predicted intrinsics, reports window-level regressions, identifies independent physical groups, and acknowledges mixed camera performance in RGB-only transfer.
Critical weaknesses
  • The central mechanism is not consistently supported. The strongest concern is that the proposed design does not reliably outperform its simpler alternatives on the designated primary endpoints.
  • In Table 19, Pi3X/TartanAir performs better without nuisance variables on both point error and shared-scale translation.
  • On both nuScenes backbones, the no-nuisance control has lower shared-scale translation error than NativeRefine.
  • For MapAnything/TartanAir, directly emitting nuisance-derived corrections outperforms the proposed constrained output on both primary endpoints.
  • The importance of warm-start depth transfer is unclear. Tables 8-13 show nearly identical results between the proposed warm refit and a cold refit using the same fitted cameras. This suggests that transferring Stage 1 depth coefficients contributes little to the final result. The authors should report full-precision differences and convergence curves. If warm initialization offers no meaningful accuracy or efficiency benefit, it should not be presented as a substantive contribution.
  • The evaluation does not cleanly establish improved camera-geometry coupling. Shared-scale translation depends on a scale estimated from predicted depth. Consequently, Stage 2 can improve the reported translation score despite leaving camera poses unchanged. This is a legitimate reconstruction metric, but it cannot independently establish improved camera estimation. Additionally, nuScenes point targets cover only the reference view. Its point score therefore does not depend on the fitted non-reference cameras and cannot demonstrate improved multiview geometry. The evaluation needs a decomposition of camera versus depth-scale effects, together with cross-view consistency or fused-reconstruction measurements.
  • The BA comparison contains several confounds. The BA baseline and NativeRefine differ in observation sharing, confidence weighting, robust loss, camera constraints, and optimization procedure. BA uses independent two-observation landmarks, whereas NativeRefine shares source-ray corrections across target observations. Therefore, the observed advantage cannot be attributed specifically to the proposed representation.
Questions for authors

See Weakness section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.