acceptodds
Withdrawn from ICLR 2027

Rectify Pose and Capture Ambiguity: 3D Spatial-Aware GS in the Wild

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

104 views

Abstract

Recent 3DGS and NeRF variants have significantly advanced for real-world scenarios. However, they still struggle to capture ambiguous distractors that are visually or semantically similar to static regions and remain susceptible to inaccurate camera poses. We found that these two challenges are inherently coupled, making them difficult to address independently and leading to artifact-prone 3D reconstruction. Motivated by these findings, we introduce PAGauss, a novel framework that jointly addresses these challenges. Specifically, unlike previous works that focus on 2D representations, PAGauss leverages 3D spatial relationships across viewpoints. PAGauss consists of two novel components: a spatial-aware adaptive mask and a spatial-aware camera pose alignment. The spatial-aware adaptive mask leverages 3D multi-view consistency to robustly capture ambiguous distractors regardless of their colors or semantic similarity. The spatial-aware camera pose alignment rectifies imprecise camera poses using a source refinement strategy with distractor-free images. To stabilize camera pose alignment, we introduce a multi-stage training strategy, improving 3D Gaussian alignment. We also create and release the WildArena dataset that includes challenging scenarios with corresponding mask annotations. To our knowledge, WildArena is the first dataset to provide such mask annotations, enabling robustness analysis over baselines. Through extensive experiments, we demonstrate that PAGauss achieves superior results and robustness across in-the-wild datasets. Our source code and dataset will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Novel View Synthesis, Unconstrained scenarios

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_hGqTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses 3D Gaussian Splatting from unconstrained photo collections, arguing that transient distractor removal and camera pose errors are coupled problems that should be solved jointly. The proposed method, PAGauss, builds on RobustSplat and adds two modules. First, a spatial aware adaptive mask uses Depth Anything 3 (DA3) per pixel confidence residuals aggregated over SAM instances with a learnable threshold to identify distractors, complemented by a context aware expansion based on DA3 feature similarity. Second, a spatial aware pose alignment regresses per image SE(3) residuals toward poses estimated by DA3 from rendered reference views. Training follows a three stage schedule. The authors also contribute WildArena, a dataset of 82 phone captured scenes with distractor mask annotations. Results are reported on five benchmarks with rendering, pose, and segmentation metrics.

Critical strengths
  1. The masking module introduces a simple and effective signal based on DA3 multi view confidence residuals. Plug in experiments (Table 5b, Table 12) show that this signal improves several existing methods, suggesting it captures genuinely useful information beyond what photometric or semantic residuals provide. This is the most transferable contribution of the paper.

  2. WildArena provides 82 scenes with distractor mask annotations and a unified evaluation protocol. The field currently lacks standardized mask level evaluation for in the wild reconstruction, and this dataset could serve as a valuable community resource if fully documented and released.

Critical weaknesses
  1. The pose refinement mechanism is underspecified and appears to have a circularity problem. Renderings produced at the current pose estimate are used to obtain DA3 poses, which are then used as supervision targets for that same pose. The paper does not explain how DA3 can recover information about the true pose that is not already encoded in the rendering, nor does it describe how poses from DA3's arbitrary similarity frame are aligned to the SfM coordinate frame before the loss in Eq. 10 is computed. Without resolving this, the theoretical basis for the pose component is unclear.

  2. The reported improvements over the strongest baselines are modest (within 0.25 dB on most established benchmarks without pose optimization), and the reliability of the results is undermined by numerous table inconsistencies. Memory values appear permuted between Tables 23 and 24, timing values are duplicated across rows in Tables 20 and 21, per scene results are duplicated in Tables 18 and 19, and several averages in Table 4b do not match the per scene values. These issues need to be resolved before the numerical claims can be properly evaluated.

Questions for authors

Clarifications that could materially change the assessment:

  1. In the source refinement step, exactly which images are input to DA3: rendered images only, or the real target image together with rendered references? Please explain why DA3 applied to renderings produced at pose estimate T_hat_i provides information about the true pose of I_i rather than simply recovering T_hat_i.

  2. How are the DA3 estimated poses, which live in an arbitrary similarity frame for each forward pass, aligned to the SfM coordinate frame before computing the loss in Eq. 10? Is a Sim(3) or Umeyama alignment solved, and if so over which point correspondences?

  3. How were the ground truth poses for Table 4b obtained? If WildArena includes clean captures (as Table 9 indicates), could you report pose errors against ground truth computed independently from those clean images, rather than from SfM on prior methods' renderings?

  4. Could you clarify the duplicated values across tables (for example, the shared memory numbers in Tables 23 and 24, copied timing rows in Tables 20 and 21, and the averages in Table 4b that do not match the per scene values)?

Suggestions for strengthening the paper in a revision:

  1. Adding RefineSplat to the WildArena comparison (Table 2) and citing NeRF HuGS, which also combines multi view consistency heuristics with SAM instance masks, would help clarify the novelty of the masking component.

  2. A controlled quantitative analysis of DA3 confidence residuals as a distractor detection signal (for example, AUROC against ground truth masks compared with photometric and DINO residuals) would strengthen the motivation beyond the qualitative evidence in Fig. 4.

Reviewer_rc5bOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a framework for in the wild, distractor-free gaussian reconstruction called PAGuass which builds upon RobustSplat. The authors introduce spacial aware masks by using the uncertainty maps predicted by DA3 together with SAM3 segmentation masks to filter out dynamic objects. Since SfM poses might contain noise in in-the-wild scenes, the authors also propose to align the camera poses together with the gaussians during training with a bias towards DA3 predicted poses. In order to evaluate the method, the authors introduce a new dataset WildArena with 8 captured scenes and manually annotated distractor masks. The method is extensively evaluated on multiple datasets and the evaluation includes segmentation metrics, pose error and ablation results.

Critical strengths

Spatial aware distractor masks: The idea of using 3D foundation models like DA3 on rendered distractor free scenes as a transient signal to filter distractors is an interesting idea and might be worth to explore further. Though I have major concerns about the evaluation soundness since its value as shown in tbl12 is only shown for a single scene. The paper also never shows qualitative results for the uncertainty masks (before or after segmentation mask filtering).

Critical weaknesses

Unclear evaluation settings: It is often very unclear or impossible to know what is measured. Only 6 of the claimed 82 WildArena scenes are evaluated without clarification. Ablation settings and rows are underspecified. They vary in scene selection, mostly cover only a single scene and under specify the row settings. The pose GT is ambiguous as the authors seemingly rerun SfM on distractor free images (5c) while App. C mentions that SfM is run on renderings of prior methods. Neither of which is validated to be correct, or more accurate then plain SfM (which should handle distractors in outlier filtering like RANSAC already). Generally results are never discussed nor explained.

Core claim not supported: The papers core claim that previous methods 'fail to distinguish ambiguous distractors from static elements with similar color or semantics' (L 041) is never tested and their core contribution Mspatial\mathcal{M}_{spatial} contains a significant amount of semantic information as well (SAM Masks, DA3 feature similarity) which is the main thing driving their performance (Table 5c).

Questions for authors
  1. Which of the 82 wildArena scene are evaluated and why these six? How many scenes are annotated and what was the selection criteria for the ablation scenes?

  2. Table 5c: what is the performance of Mcontext\mathcal{M}_{context} alone? Why is Mgeo\mathcal{M}_{geo} worse than the baseline?

  3. How exactly are GT poses obtained? Since masks are available, the clean way would be to mask out matches inside the distractor masks and run on real images. Are these poses also used for table3?

  4. Can you show UU, U^\hat{U} and β\beta for example images?

Reviewer_FNZVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PAGauss targets in-the-wild 3DGS with two coupled problems: ambiguous distractors and inaccurate SfM poses. Built on RobustSplat, it (i) detects distractors from the change in DA3 confidence when a ground-truth image is replaced by its rendering, aggregated over SAM instances, and (ii) refines per-image poses by regressing them toward DA3 poses estimated on distractor-free renderings, with a three-stage training schedule. The paper also introduces WildArena, a dataset of 82 scenes with distractor mask annotations.

Critical strengths
  • The paper treats distractors and pose errors as a joint problem, which is a relevant angle for in-the-wild reconstruction.
  • WildArena provides distractor mask annotations, which would be useful to the community if released.
  • The evaluation covers five datasets and many baselines.
Critical weaknesses
  1. Limited novelty. The method mainly combines existing components:

    • a 3D foundation-model confidence signal;
    • SAM instance-level aggregation with a mean/variance threshold, which AsymGS, DualSplat, and DroneSplat already use (as acknowledged in App. C);
    • standard per-image pose residuals (Eq. 9) regularized toward foundation-model poses.

    The paper does not clearly explain what is new beyond this combination.

  2. Data inconsistencies in the reported results.

    • In Tab. 23 (Ambiguous Wild) and Tab. 24 (NeRF On-the-go), four methods (3DGS, Mip-Splatting, DroneSplat, RobustSplat) report the same four memory values on different datasets, only placed under different scenes. For example, 3DGS has 413.04 / 447.08 / 247.98 / 396.69 for Humanoid / Porter / Lounge / Jockey in Tab. 23, and 447.08 / 413.04 / 396.69 / 247.98 for Mountain / Fountain / Corner / Patio in Tab. 24. The memory unit is also not given.
    • In Fig. 8-(b), LongSplat+RobustSplat reports an IoU of 0.936, which exceeds its accuracy of 0.830. This is not possible under any standard IoU definition, and IoU is not defined in the paper.

    These errors reduce confidence in the reported numbers.

Questions for authors
  1. How are IoU, precision, and recall defined and averaged in Fig. 8-(b) and Tab. 10?
  2. Can the authors check Tab. 23/24 and the other appendix tables, correct them, and state the memory unit?
  3. What is the technical contribution beyond combining DA3 confidence, SAM instance aggregation, and pose regression?
Reviewer_ScmBOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAGauss, a 3DGS framework that jointly addresses ambiguous distractors and inaccurate camera poses using 3D multi-view consistency and distractor-free pose refinement. It also introduces the WildArena dataset with annotated distractor masks. Experiments demonstrate improvements in reconstruction quality, segmentation, and pose accuracy across in-the-wild datasets.

Critical strengths

1.The paper addresses the interplay between distractor removal and camera pose refinement in unconstrained 3D reconstruction. The proposed framework is intuitive and practically relevant. 2.The method achieves competitive results across five in-the-wild datasets, supported by ablation studies examining the contributions of individual components. 3.WildArena provides 82 challenging scenes with annotated distractor masks, offering a useful benchmark for evaluating robustness in real-world reconstruction.

Critical weaknesses

1.Some reported IoU values exceed the corresponding precision or recall, which is inconsistent under a common evaluation protocol. The reported Average values also do not consistently match the arithmetic means of the listed metrics. 2.Eq. (6) introduces a learnable weighting parameter, but its optimization procedure is unclear, particularly given the hard thresholding operation in Eq. (7). 3.The masking pipeline closely resembles RefineSplat in its use of instance-level aggregation, adaptive thresholding, and complementary masks. The main advances appear to lie in DA3-based confidence residuals and render-based pose refinement. While useful, the methodological contribution appears incremental, and the distinction from closely related masking approaches should be more clearly established

Questions for authors

1.Could the authors clarify the definitions and aggregation protocols for the segmentation metrics and explain the numerical inconsistencies in Figure 8(b) and Table 10?

2.How is the learnable weighting parameter in Eq. (6) optimized, and how is gradient propagation handled through the hard thresholding operation in Eq. (7)?

3.How does the proposed mask construction improve upon RefineSplat beyond changing the uncertainty signal? A controlled comparison using the same DA3-derived signal and identical pose-refinement settings would help isolate this contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.