acceptodds
Withdrawn from ICLR 2027

AnchorSplat: Feed-Forward 3D Referring Segmentation in Gaussian Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grounding a referring expression in sparse, uncalibrated views requires both semantic discrimination and 3D consistency. Feed-forward reconstruction models supply cameras, depth, and renderable Gaussians, yet substituting them for the backbone of a multiview referring model consistently lowers accuracy. We trace this failure to two mechanisms. Reconstruction pretraining progressively attenuates the appearance cues that referring expressions rely on, and pixel-aligned Gaussian rendering allows each primitive to copy the label of the pixel that created it. We present AnchorSplat, which predicts view masks and a referred Gaussian scene in a single forward pass. A frozen semantic encoder anchors the referring state, and zero-initialized gates admit reconstruction features while preserving its initial referring function. A frame-invariant correspondence graph transports language-conditioned states across views, and an opacity-gated readout labels each Gaussian under owner-excluded rendering, which evaluates a primitive only in views that did not create it. On the 9,508 test samples of MVRefer, AnchorSplat improves global mIoU from 39.9 to 41.0 and [email protected] from 41.5 to 44.6 over MVGGT, whereas direct substitutions with VGGT-Ω, DA3, and AMB3R reach at most 32.8 global mIoU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Referring Segmentation, 3D Gaussian Splatting, Feed-Forward Reconstruction, Language Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DyGuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

AnchorSplat predicts referring masks and a renderable Gaussian scene from sparse, uncalibrated images and a text query. It combines DINOv2 semantics with DA3 geometry, cross-view feature transport, and owner-excluded rendering. On MVRefer, it improves global mIoU from 39.9 to 41.0 over MVGGT. Additional experiments evaluate novel-view rendering.

Critical strengths
  1. Predicting a renderable, language-selected Gaussian scene from sparse uncalibrated views is a useful extension beyond view masks and lifted points. It allows the selected object to be inspected from other viewpoints.
  2. Owner-excluded rendering addresses a concrete supervision shortcut: a Gaussian could otherwise copy its source pixel’s label without learning agreement across views. This is a reasonable design worth investigating.
Critical weaknesses
  1. The main components are not sufficiently isolated in the ablations. Table 5 mainly compares backbone substitutions with the full model. It does not show a DINOv2-only baseline or separately remove the geometry ladder and cross-view transport. I therefore cannot tell whether the gains come from the proposed geometry mechanisms or the added semantic features.

  2. The benefit of the Gaussian readout is not directly established. The main segmentation metrics evaluate view masks and lifted points, while Table 4 measures rendering quality. Object-region PSNR does not tell us whether the selected Gaussians exclude background or preserve the entire target. A comparison against direct mask assignment and owner-inclusive rendering, using the same geometry and held-out-view mask IoU, is needed to support this contribution.

  3. Poor figure quality makes the qualitative results difficult to assess. The pipeline in Figure 3 is overcrowded, with tiny labels and visibly compressed text. The result panels in Figures 6–9 are also too small, and many details remain unclear even when enlarged. The comparisons make it difficult to inspect object boundaries, background leakage, and fine-grained differences between methods. This is not just a cosmetic issue: the presentation limits my ability to verify the claimed improvements. Larger panels, readable labels, and clear close-ups of corresponding regions are needed.

Questions for authors
  1. Can you provide matched ablations for the DINOv2-only baseline, geometry ladder, and cross-view transport? These would help establish which components account for the improvement.

  2. Do any reported segmentation scores directly evaluate the final Gaussian readout? On the same DA3 Gaussians, how does owner-excluded rendering compare with direct mask assignment and owner-inclusive rendering in held-out-view mask IoU? Please include clear visual comparisons with ground truth.

3 comments
AuthorsOpenReview ↗

Comment 1 of 3. Matched ablations for DINOv2, the ladder, and transport (W1, Q1)

We thank the reviewer for the detailed assessment and the concrete experimental requests. We trained every requested control on all 9,508 MVRefer test queries with matched budgets and added them to Table 5a (Section 5.5) and Table S1 (Section S2.1). Throughout our responses, pairs report global mIoU / [email protected] (%), and brackets give paired 95% scene-bootstrap intervals.

Matched protocol. All referring controls use identical data, views, evaluator, four-head decision, and 20,000 referring updates. Transport variants are calibrated from one shared ladder for 100 updates (Table S7). Table 5a now separates these component controls from backbone controls.

DINOv2-only baseline. With every geometry gate and transport parameter fixed at zero, the decoder reads DINOv2 tokens alone and reaches 39.53 / 39.87. This is below the released MVGGT (39.9 / 41.5) and below MVGGT continued for the same 20,000 updates (40.14 / 43.74). The added semantic features therefore do not explain our improvement.

Removing the geometry ladder. Adding gated DA3 features to the same anchor raises the result to 40.88 / 44.28, a gain of 1.35 [0.78, 1.91] / 4.41 that holds on Hard, Easy, Unique, and Multiple. Geometry contributes cross-view evidence, raising Neg mIoU from 73.72 to 80.21 and View mIoU from 66.00 to 69.75. A standard aligned DA3 interface trained with the same budget reaches only 34.29 / 35.77, 6.59 [5.64, 7.55] points below the ladder, so the gain depends on how geometry enters the decoder.

Removing cross-view transport. From the same ladder, full transport reaches 41.01 / 44.64, against 40.88 / 44.35 with transport disabled and 40.62 / 43.73 with uniform aggregation over the same four neighbors. Full transport improves [email protected] over both alternatives on all four splits. Uniform aggregation lowers Neg mIoU from 80.19 to 78.46 because it spreads referent activation into views without a matching surface, whereas our confidence and distance weights keep Neg mIoU at 80.17. The transport graph sends 72.91% of its weight to target patches against a 3.80% prevalence baseline (Section S2.3).

Attribution. Under identical budgets, each mechanism improves on the stage that precedes it (39.53, 40.88, and 41.01 global mIoU, with 39.87, 44.28, and 44.64 [email protected]). On the same frozen DA3 features, the complete design exceeds direct substitution by 10.08 / 15.40, and its margin over MVGGT grows with views, reaching 71.56 against 65.63 at 32 views (Table 3a). A decoder-free probe explains why the semantic anchor is needed. Same-object ranking in DA3 falls by 7.93 points [3.50, 12.69] from layer 4 to layer 39, close to chance, while DINOv2 stays 18.97 points above deep DA3 (Section 5.3, Table S2b).

We address W2, Q2, and W3 in the following comments.

AuthorsOpenReview ↗

Comment 2 of 3. Direct evaluation of the Gaussian readout (W2, Q2)

A distinct output. The readout produces what view masks and lifted points cannot provide, a referred Gaussian scene that can be rendered from any viewpoint. Its gradients reach only the readout head, so it never alters the reported view masks (Section 4.6). We therefore added a dedicated Gaussian-level evaluation in exactly the requested setting, with the same DA3 geometry and held-out-view mask IoU (Section 5.4, Table S6b).

Protocol. All variants share identical DA3 Gaussians. Gaussian mIoU scores primitives matched within 2 cm to held-out GT sensor-depth surfaces over 21 queries. Any selected background primitive enters the union and any missed referent primitive lowers the intersection, so the metric measures background exclusion and target completeness directly, which object-region PSNR cannot. Referred masks rendered with opacity ojsjo_js_j are scored by IoU on 28 target-visible held-out frames that are never input views.

Results on identical Gaussians (Gaussian mIoU and held-out mask mIoU).

  1. Direct mask assignment from the MVGGT+DA3 state, 17.82 and 13.58.
  2. Direct mask assignment from the AnchorSplat state, 22.95 and 15.05.
  3. Owner-inclusive readout head, 23.02 and 15.21.
  4. Owner-excluded readout head (ours), 23.02 and 15.21. The anchored referring state accounts for the largest gain, +5.13 Gaussian mIoU and +1.47 mask mIoU over MVGGT+DA3, and the readout adds 0.07 and 0.16 over direct assignment at a cost of 0.015 s per pass (Table 3b). Absolute values are lower than view-mask mIoU because every primitive is judged against GT surfaces in unseen frames.

Owner-excluded versus owner-inclusive rendering. The two heads were trained for the same 400 updates with identical seed and sample order, and both refine the same anchored owner logit, which explains their equal held-out accuracy. They differ in the signal they learn from. Under owner-inclusive rendering, source-owner primitives carry 44.41% of rendered opacity and 31.46% [26.74, 36.56] of primitive-logit rendering gradients, and this share can be lowered by copying each pixel's own label. Exclusion sets the share to zero, so every gradient enforces agreement with other views, which is the condition for Proposition 3 (Section S2.4). Negating one view's owner logits changes the owner-inclusive render by 54.89 percentage points and the owner-excluded render by exactly zero in all 50 cases.

Visual comparison with ground truth. Figure 7 now places GT beside the full-scene, selection, and referred renders, with matched 2×2\times insets on object boundaries. Figure S13 shows two expressions selecting different Gaussians from one shared reconstruction, illustrating that only opacity depends on the query.

AuthorsOpenReview ↗

Comment 3 of 3. Figure quality (W3) and summary of contributions

Figure 3. We re-laid out the pipeline figure with a minimum font size of 7 pt and removed the vertical compression that distorted its text. All figures are now included at their native aspect ratio and full resolution.

Figures 6 to 9. We added local magnifications to Figures 6 to 9 to enhance fine details. Each 2×2\times inset enlarges an identical region across methods, so object boundaries, background leakage, and fine differences between methods can be inspected directly against ground truth.

Supplementary views. The two enlarged single-case comparisons (Figures S9 and S10) and the nine-scene gallery (Figure S11) provide full-page views with point IoU on a shared canvas.

Summary of contributions and evidence.

  1. Diagnosis. We identify why reconstruction backbones fail as referring backbones. A decoder-free probe shows object identity in DA3 falling to near chance with depth, and the source-owner shortcut is measured directly, carrying 31.46% of rendering gradients and shifting owner-inclusive renders by 54.89 percentage points.
  2. Output. AnchorSplat returns view masks and a referred Gaussian scene from sparse uncalibrated views in one forward pass, now evaluated at the Gaussian level and in held-out views.
  3. Mechanisms. Under matched budgets, the semantic anchor, geometry ladder, and transport each improve on the preceding stage, while owner exclusion makes primitive supervision fully cross-view at no cost in held-out accuracy. \item Results. AnchorSplat improves 17 of 20 MVRefer entries and all nine ScanRefer entries over MVGGT, whereas every direct reconstruction substitution loses 7.1 to 9.0 points.

We hope these results resolve the reviewer's concerns. Should any question remain, we would welcome further comments and will promptly clarify or provide additional experiments.

Reviewer_eYXuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents AnchorSplat, a feed-forward framework for 3D referring segmentation from sparse, uncalibrated images. It combines DINOv2 semantic features with DA3 geometry through a semantic-anchored geometry ladder and cross-view feature transport. An owner-excluded rendering strategy is further introduced to produce language-conditioned Gaussian scenes.

Critical strengths

1.The paper addresses an interesting problem by extending multiview referring segmentation to renderable Gaussian scenes without per-scene optimization.

2.The experimental results are promising. AnchorSplat improves over MVGGT on MVRefer and significantly outperforms direct backbone substitutions. The authors also provide comparisons across different settings and additional analyses in the supplementary material.

Critical weaknesses

1.The authors attribute the performance drop from backbone replacement to semantic erosion. However, I am not fully convinced that the analysis in Figure 2 establishes this explanation. Could the degradation instead come from feature distribution mismatch between the reconstruction backbone and the referring decoder?

2.The semantic-anchored geometry ladder in Section 4.2 seems closely related to standard feature projection and gated residual fusion. What makes this design particularly different from simpler alternatives? Also, Proposition 1 mainly establishes that the original decoder is recovered when the new modules are initialized to zero. While this appears reasonable, its practical significance is unclear, as it does not guarantee semantic preservation during training.

3.It is difficult to isolate the contributions of semantic anchoring, geometric transport, and owner-excluded rendering. More importantly, Section S5.1 describes additional feature alignment (1,500 steps) and behavior distillation (512 steps). These steps may play an important role in bridging the feature spaces, but their individual contributions are not evaluated. This makes it difficult to tell where the performance gains actually come from.

4.Most quantitative referring results are based on view masks or lifted points, while the rendering metrics mainly evaluate full-scene reconstruction. The quality of the actual language-conditioned Gaussian selection is less directly evaluated.

5.The writing is sometimes difficult to follow, with several uncommon terms and abrupt transitions between concepts. Some important implementation details are scattered across the main paper and supplement. Several figures, especially the qualitative comparisons, are also too small to inspect clearly.

Questions for authors

Please refer to the Critical Weaknesses section

4 comments
AuthorsOpenReview ↗

Comment 1 of 4. Semantic erosion versus feature distribution mismatch (W1)

We thank the reviewer for the careful reading and for raising this alternative explanation, which we tested directly. Throughout our responses, pairs report global mIoU / [email protected] (%) on all 9,508 MVRefer test queries, and brackets give paired 95% scene-bootstrap intervals.

Two hypotheses, two predictions. A distribution mismatch lies between DA3 features and the referring decoder, so it should vanish when no decoder is involved and shrink when the features are aligned to the decoder basis. Erosion lies in the features themselves, so it should persist in both settings. We ran one experiment for each prediction.

Experiment 1, decoder-free probe (Section 5.3, Table S2b). We rank frozen tokens directly, with the same cached inputs and patch normalization for every representation. For each query, a patch of the GT object in another view must outrank a background patch whose 3D distance is matched within 5%, so geometry alone cannot separate them, and chance is 50%. Over 47,387 triplets in 46 scenes, same-object ranking in DA3 falls from 58.46% at layer 4 to 51.84% at layer 20 and 50.53% at layer 39, a drop of 7.93 points [3.50, 12.69] to near chance. With RGB-distinct alternatives (281,334 triplets), appearance ranking falls from 78.07% to 51.29%. DINOv2 reaches 69.50% on identity, 18.97 points [14.67, 23.20] above deep DA3. Shallow DA3 retains color but not object identity, whereas DINOv2 retains both, which is why the anchor uses DINOv2. Because the probe involves no decoder, this decline cannot stem from a decoder mismatch.

Experiment 2, explicit alignment (Section 5.5, Table 5a). We fitted an interface that maps DA3 tokens to the DINOv2 basis of the decoder, using 1,500 feature-fitting and 512 behavior-distillation updates, inserted it into the direct DA3 substitution, and trained for the same 20,000 referring updates. Alignment raises the substitution from 30.93 / 29.24 to 34.29 / 35.77, so mismatch accounts for 3.36 of the 8.60 points that separate direct substitution from the DINOv2 anchor alone (39.53). The aligned model still trails the anchor by 5.24 points.

Consistency with training dynamics. A pure mismatch should also shrink as the decoder adapts. After 20 epochs on DA3 features, the direct substitution remains at 30.93 (Table 5b), consistent with information loss rather than a calibration gap.

Conclusion. Mismatch exists but explains the smaller part of the loss, while the larger part follows the erosion measured without any decoder. Accordingly, the anchored decoder keeps the strongest within-view discrimination against color-matched alternatives, with a stage-12 contrast of 3.94 against 3.58, 3.23, and 3.13 for the VGGT-Ω\Omega, DA3, and AMB3R substitutions (Section S2.2).

We address W2 to W5 in the following comments.

AuthorsOpenReview ↗

Comment 2 of 4. Distinctiveness of the ladder and role of Proposition 1 (W2)

Difference from projection and gated residual fusion. The design choice lies in which stream carries the referring state. In standard fusion, reconstruction tokens form the residual stream and other features are projected into it. Our ladder reverses these roles. DINOv2 tokens form the residual stream, and each DA3 block half enters as a gated increment at its assigned stage, alternating local and global halves in native block order. Three controls isolate these choices.

  1. Semantic stream. The aligned DA3 control realizes the standard design, projecting the same DA3 features into the decoder basis under the same budget. Our ladder exceeds it by 6.59 [5.64, 7.55] / 8.51 (40.88 / 44.28 against 34.29 / 35.77).
  2. Native block order. Permuting the order lowers 43.04 / 48.67 to 42.47 / 46.50 (Table 5b).
  3. Stage-wise geometric increments. Relative to the anchor alone, the ladder raises Neg mIoU from 73.72 to 80.21 and View mIoU from 66.00 to 69.75 (Table S1), so geometry adds cross-view evidence without displacing appearance. This agrees with the MVGGT finding that late fusion outperforms early and middle fusion (Table 6).

Practical role of Proposition 1. The proposition serves two concrete purposes. First, it defines an exact semantic reference. At θg=0\theta_g=0 every view logit equals that of the decoder driven by DINOv2 alone, which is precisely our anchor-only control. Every trained model can therefore be measured against its own reference, and Table 5a reports this measurement. The trained ladder exceeds the reference on every split (Hard 24.86 against 23.56, Easy 51.53 against 50.15, Unique 65.69 against 64.70, Multiple 34.90 against 33.47).

Second, its bound ties any drift from the reference to the norms of the gated injections and transport increments, weighted by downstream Lipschitz constants. Geometry can modify the referring function only through these terms, so the DINOv2 state is never replaced by a new feature basis, which is the failure mode of direct substitution. Empirically, discrimination survives training. On the 50-scene diagnostic cohort, AnchorSplat attains the highest IoU at all four supervised stages (45.17, 44.63, 45.84, and 45.15, against at most 43.69 for any substitution, Section S2.5) and the highest color-matched contrast (Section S2.2). Further geometry training of the calibrated model for 200 updates changes global mIoU by only 0.13 (Table 5b), indicating that the learned increments are stable.

AuthorsOpenReview ↗

Comment 3 of 4. Component contributions and the alignment updates (W3)

Component isolation. Table 5a now contains matched full-test controls in which each row adds one mechanism to the previous one with identical data and budgets.

  1. Anchor only, 39.53 / 39.87.
  2. With the geometry ladder, 40.88 / 44.28, a gain of 1.35 [0.78, 1.91] / 4.41.
  3. With calibrated transport, 41.01 / 44.64, a gain of 0.13 / 0.29 over disabled transport and 0.39 / 0.91 over uniform neighbor aggregation, with [email protected] improving on all four splits. Owner-excluded rendering acts on the Gaussian output and is evaluated with Gaussian-level metrics, which we detail under W4.

Role of the 1,500 alignment and 512 distillation updates. These updates fit an entry interface from DA3 tokens to the DINOv2 basis of the referring decoder, combining normalized squared error and cosine distance for feature fitting with distillation through the frozen decoder (Section S5.1). The anchor of AnchorSplat already consists of DINOv2 tokens, so its referring state does not depend on such a mapping. To quantify the interface itself, we transplanted it into the direct DA3 substitution and trained for the same 20,000 referring updates. It adds 3.36 / 6.53 (from 30.93 / 29.24 to 34.29 / 35.77), yet the aligned model remains 6.59 [5.64, 7.55] points below our ladder. The alignment therefore cannot account for our improvement.

Role of the training budget. MVGGT continued for the same 20,000 updates reaches 40.14 / 43.74, below our ladder before calibration. Continued geometry training, balanced BCE, and a boundary refiner keep the calibrated model between 40.31 and 41.01 (Table 5b). The improvement thus comes from the anchored architecture rather than from additional optimization.

Where the gains come from. Read together, the controls give a consistent picture. The semantic anchor recovers 8.60 of the points lost by direct DA3 substitution. The geometry ladder adds cross-view evidence, raising Neg mIoU by 6.49 and [email protected] by 4.41. Calibrated transport adds [email protected] on every split while preserving Neg mIoU. Table S1 (Section S2.1) gives full metrics for every control, and Table S7 lists every budget.

AuthorsOpenReview ↗

Comment 4 of 4. Gaussian selection (W4) and presentation (W5)

W4, evaluation protocol. We added Gaussian-level evaluation of the language-conditioned selection (Section 5.4, Table S6b). All variants share identical DA3 Gaussians. Primitives matched within 2 cm to held-out GT sensor-depth surfaces are scored by Gaussian mIoU over 21 queries, so selected background primitives enter the union and missed referent primitives lower the intersection. Referred masks rendered with opacity ojsjo_js_j are scored by mask IoU on 28 target-visible held-out frames that are never input views. Both metrics assess the referred scene itself rather than full-scene reconstruction.

W4, results. Labels from the MVGGT+DA3 state reach 17.82 Gaussian mIoU and 13.58 mask mIoU. AnchorSplat raises them to 22.95 and 15.05 through its anchored referring state, and its readout head reaches 23.02 and 15.21. Absolute values are lower than view-mask mIoU because every primitive is judged against GT surfaces in unseen frames. Owner-inclusive and owner-excluded heads trained with the same budget reach identical scores, while exclusion removes the 31.46% source-owner share of rendering gradients and makes all primitive supervision cross-view (Section S2.4). Figure 7 now places GT beside the full-scene, selection, and referred renders, and Figure S13 shows two expressions selecting different Gaussians from one shared reconstruction.

W5, terminology. Section 4 now defines the semantic anchor, geometry ladder, and owner at first use and states the three transitions handled in Sections 4.2 to 4.4. Section 4.2 explains how the ladder differs from residual fusion.

W5, organization. Section 4.6 gathers the training stages and gradient routing, Section 5.5 gathers all matched controls, and Table S7 lists the recipe together with every control budget, so implementation details no longer need to be collected across the paper and supplement.

W5, figures. We re-laid out Figure 3 and added local magnifications to Figures 6 to 9 to enhance fine details. All figures are now included at their native aspect ratio and full resolution, which removes the vertical compression that reduced legibility, and the insets enlarge identical regions across methods.

We hope these results resolve the reviewer's concerns. Should any question remain, we would welcome further comments and will promptly clarify or provide additional experiments.

Reviewer_XgFDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 3D referring segmentation under sparse and uncalibrated views. It proposes a feed-forward framework, AnchorSplat, which combines frozen DINOv2 semantic features with DA3 reconstruction features through a semantic-anchored geometry ladder. It also introduces geometric semantic transport for cross-view feature propagation and owner-excluded Gaussian rendering to encourage cross-view consistency. Experiments on MVRefer and ScanRefer show that the proposed method achieves improvements over MVGGT.

Critical strengths

The paper is clearly motivated. The identified semantic erosion and source-view label-copy shortcut provide plausible explanation for the proposed designs. In particular, owner-excluded rendering is an interesting approach to enforcing cross-view label consistency.

Critical weaknesses

The contributions of the key technical components are insufficiently validated. The paper lacks controlled ablations to isolate the effectiveness of semantic anchoring, geometric semantic transport, and owner-excluded rendering. The technical advantages over simpler alternatives remain unclear, while inference time and memory consumption increase substantially.

Questions for authors
  1. Could authors provide controlled ablation results under the same training budget: (i) the semantic anchor alone with all gates and transport parameters fixed at zero, (ii) gated geometry injection without transport, and (iii) the same anchored design with Pi3 as the geometry backbone?
  2. How does the proposed geometric semantic transport compare with simpler alternatives, such as nearest-neighbor feature aggregation or geometry-based cross-view feature averaging?
  3. Since the readout inputs are detached and rendering gradients only train the readout head, owner-excluded rendering does not affect the reported view-mask or lifted-point metrics. Could the authors compare it with owner-inclusive rendering using Gaussian-level metrics?
  4. Given the relatively modest improvements over MVGGT and the substantially higher inference latency and memory usage, how do the authors justify the additional complexity? Could an MVGGT variant with Pi3, trained for the same additional budget, achieve comparable performance?
4 comments
AuthorsOpenReview ↗

Comment 1 of 4. Controlled ablations (Q1)

We sincerely thank the reviewer for the precise questions, which helped us isolate each component. We trained every requested control on all 9,508 MVRefer test queries under matched budgets and integrated the results into Table 5a (Section 5.5) and Table S1 (Section S2.1). Throughout our responses, pairs report global mIoU / [email protected] (%), and brackets give paired 95% scene-bootstrap intervals.

Matched protocol. All referring controls share the same data, eight views, evaluator, four-head decision, and 20,000 referring updates, which matches the retained epoch of our first training stage. Transport variants are calibrated from one shared ladder for the same 100 updates (Table S7).

(i) Semantic anchor alone. With every ladder gate, descriptor map, and transport parameter fixed at zero, the decoder reads DINOv2 tokens alone and reaches 39.53 / 39.87.

(ii) Gated geometry without transport. Adding the DA3 ladder to the same anchor raises the result to 40.88 / 44.28, a gain of 1.35 [0.78, 1.91] / 4.41. Global mIoU improves on all four splits (Hard 23.56 to 24.86, Easy 50.15 to 51.53, Unique 64.70 to 65.69, Multiple 33.47 to 34.90). The gain arises where 3D evidence is needed. Neg mIoU rises from 73.72 to 80.21 and View mIoU from 66.00 to 69.75, showing that geometry tells the decoder which views do not contain the referent. Calibrating transport then yields the full model, 41.01 / 44.64.

(iii) Pi3 as the geometry backbone. Pi3 predicts point maps and cameras but no Gaussian primitives, so a Pi3 ladder cannot return the referred Gaussian scene that defines our task. We understand the underlying question as whether our gain merely reflects a stronger backbone, and our backbone controls answer it directly. DA3 is the weaker referring backbone. Direct DA3 substitution reaches 30.93 / 29.24, and even with a fitted DA3-to-DINOv2 interface it reaches 34.29 / 35.77, both well below Pi3-based MVGGT at 39.9 / 41.5. Starting from this weaker backbone, AnchorSplat exceeds MVGGT and surpasses the same-backbone substitution by 10.08 / 15.40. The improvement therefore comes from the anchored design rather than from the choice of backbone.

Summary. Under identical budgets, each stage improves on the previous one (39.53, 40.88, and 41.01 global mIoU, with 39.87, 44.28, and 44.64 [email protected]). The anchor alone does not explain the gain, since it remains below MVGGT continued for the same budget (40.14 / 43.74).

We address Q2 to Q4 in the following comments.

AuthorsOpenReview ↗

Comment 2 of 4. Comparison with simpler alternatives (Q2)

We compared three cross-view updates calibrated from the same ladder with identical data and budgets (Table 5a, Table S1).

  1. Nearest-neighbor feature aggregation, with uniform weights over the same four nearest other-view patches, reaches 40.62 / 43.73.
  2. Disabling the update reaches 40.88 / 44.35.
  3. Our geometric semantic transport reaches 41.01 / 44.64, adding 0.39 / 0.91 over uniform aggregation and 0.13 / 0.29 over no transport. Full transport improves [email protected] over both alternatives on every split (Hard 25.04 against 24.75 and 23.88, Easy 57.67 against 57.39 and 56.94, Unique 76.26 against 75.72 and 75.88, Multiple 37.02 against 36.80 and 35.99).

Why naive aggregation falls short. Uniform aggregation lowers Neg mIoU from 80.19 to 78.46 and View mIoU from 69.74 to 68.65, because it pushes referent activation into views whose patches have no matching surface. Our weights differ in two respects. The confidence-weighted kernel πab\pi_{ab} favors close and reliable correspondences, and the support mass ρa\rho_a attenuates transfer when the nearest match is far. Full transport therefore keeps Neg mIoU at 80.17 while raising View mIoU to 69.83. The bracket in Eq.~(6) is a neighbor-minus-self residual, so a patch state moves only by its disagreement with reliable neighbors.

Geometry-based averaging. Confidence-weighted geometric averaging is the special case WQ=IW_Q=I, bQ=0\mathbf b_Q=0 of Eq.~(6), for which Proposition 2 guarantees a convex combination of a patch and its neighbors. Our learned WQW_Q contains this case and additionally selects which channels move.

The graph carries semantic identity. Independently of the learned update, 72.91% of the outgoing weight of target patches reaches target patches in other views, against a 3.80% prevalence baseline drawn from the same candidate pool. This is a 19.19-fold enrichment with a paired difference of 69.11 points [61.77, 75.85] (Section S2.3). Under scaling, translation, and rotation of the reconstruction frame, over 99.99% of edges are preserved and the outgoing mass changes by less than 0.0023, confirming the frame invariance stated in Proposition 2.

Scaling with views. The complete model converts added views into accuracy. From 4 to 32 views AnchorSplat gains 10.66 global mIoU against 6.95 for MVGGT and reaches 71.56 / 82.00 at 32 views against 65.63 / 74.00, whereas direct substitutions stagnate or degrade (Table 3a).

AuthorsOpenReview ↗

Comment 3 of 4. Owner exclusion evaluated at the Gaussian level (Q3)

Rendering gradients indeed do not reach the view masks or lifted points, and this routing is intentional. Rendering and owner losses update the readout head HH through detached inputs, so primitive supervision cannot alter the reported view-mask results (Section 4.6). Owner exclusion therefore contributes to the second output of AnchorSplat, the referred Gaussian scene, and is properly evaluated on Gaussians. We added this evaluation (Section 5.4, Table S6b in Section S4.1).

Protocol. All variants share identical DA3 Gaussians. Primitives matched within 2 cm to held-out GT sensor-depth surfaces are scored by Gaussian mIoU over 21 queries, so selected background primitives enter the union and missed referent primitives reduce the intersection. Referred masks are rendered into 28 target-visible held-out frames that are never input views.

Results. Direct owner assignment from the MVGGT+DA3 state gives 17.82 Gaussian mIoU and 13.58 mask mIoU. Direct assignment from the AnchorSplat state gives 22.95 and 15.05. Owner-inclusive and owner-excluded heads, trained for the same 400 updates with identical seed and sample order, both reach 23.02 and 15.21.

What exclusion changes. Both heads refine the same detached anchored owner logit, which explains their equal held-out accuracy. Their training signals differ. Under owner-inclusive rendering, source-owner primitives carry 44.41% of rendered opacity and 31.46% [26.74, 36.56] of primitive-logit rendering gradients across 64 queries in 32 training scenes (Section S2.4). This share can be reduced by copying each pixel's own label, which conveys no information about other views. Exclusion sets the owner share to zero and raises the cross-view share from 68.54% to 100%. Under this condition, Proposition 3 guarantees that a low rendering loss implies label agreement among the primitives of other views at every covered pixel, a guarantee that owner-inclusive rendering cannot provide. The intervention in Section S2.4 confirms the mechanism. Negating one view's owner logits changes the owner-inclusive render by 54.89 percentage points and the owner-excluded render by exactly zero in all 50 cases.

Summary. Owner exclusion turns the entire primitive supervision into a cross-view consistency signal at no cost in held-out accuracy, while the anchored referring state supplies the main Gaussian-level gain, +5.13 Gaussian mIoU and +1.47 mask mIoU over MVGGT+DA3 on identical primitives.

AuthorsOpenReview ↗

Comment 4 of 4. Matched budget, improvement, and cost (Q4)

Matched MVGGT training. We continued MVGGT with Pi3 for the same 20,000 referring updates used by our first stage. It reaches 40.14 / 43.74. Our ladder already exceeds it before transport calibration (40.88 / 44.28), and the full model reaches 41.01 / 44.64. AnchorSplat leads on all four splits (Easy 51.89 against 50.46, Unique 66.21 against 65.12, Multiple 34.95 against 34.12, Hard 24.67 against 24.61) and in Pos mIoU (44.13 against 42.13). Longer MVGGT training saturates, as 20,000 additional updates add 0.24 global mIoU to the released 39.9.

Additional optimization of our model. Extra optimization does not drive our result either. Continued geometry training of the calibrated model for 100 or 200 updates gives 40.93 and 40.88, and balanced BCE or a boundary refiner keeps it between 40.31 and 41.01 (Table 5b).

Size of the improvement. The gain over MVGGT (+1.1 global mIoU, +3.1 [email protected], 17 of 20 MVRefer entries, and all nine ScanRefer entries) is obtained on a backbone that loses 9.0 points under direct substitution. Relative to the same DA3 backbone, our design adds 10.08 / 15.40. The margin also grows with views, reaching 5.93 global mIoU and 8.0 [email protected] at 32 views (Table 3a).

Cost. Eight-view inference takes 0.594 s and 11.46 GB on one RTX 4090 D (Table 3b, Section S5.1). DA3 geometry accounts for 0.433 s and returns cameras, depths, and renderable Gaussians, which MVGGT does not provide. All referring and readout modules together take 0.163 s, below the 0.217 s of the corresponding MVGGT modules, and the Gaussian readout itself takes 0.015 s. For several expressions on the same views, geometry and DINOv2 tokens are computed once, so each further query adds at most 0.163 s. Among the reconstruction backbones, AnchorSplat is faster than MVGGT+AMB3R (0.826 s) and costs 0.137 s more than MVGGT+DA3 while scoring 10.08 points higher.

Justification. The additional cost buys an output that MVGGT cannot produce, a referred Gaussian scene that can be rendered and verified from novel views, together with higher global mIoU at every view count from 4 to 32.

We hope these results resolve the reviewer's concerns. Should any question remain, we would welcome further comments and will promptly clarify or provide additional experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for Submission 47745

We thank all the reviewers and the Area Chair for their efforts in the review process, which helped us improve the quality of our paper.

AnchorSplat grounds a free-form expression in both view masks and a renderable Gaussian scene from sparse, uncalibrated RGB views in one forward pass. Its central design keeps semantic tokens as the referring decoder's residual stream, introduces reconstruction features through an ordered, gated geometry ladder, and transports language-conditioned states along geometric correspondences. An opacity-gated Gaussian readout uses owner-excluded rendering for cross-view supervision and supports verification of the selected scene from novel viewpoints.

The revised text makes each mechanism's role explicit. Section 4 defines the anchor, ladder, and owner at first use. Sections 4.2--4.3 distinguish semantic-state updates from feature fusion and explain the correspondence weights. Sections 4.4 and 4.6 specify detached inputs and which losses update each output. Sections 5.3--5.5 connect semantic diagnostics, Gaussian-level evaluation, and matched ablations to the corresponding method claims.

For reviewer DyGu, matched controls evaluated on all 9,508 queries separate the contribution of DINOv2 from those of the ladder and transport: global mIoU increases from 39.53 with the semantic anchor alone to 40.88 with the ladder and 41.01 with full transport. The ladder gain is 1.35 points, with a paired 95% interval of [0.78, 1.91] (Section 5.5, Table 5a). Direct Gaussian evaluation further establishes the value of the referred-scene output: on identical geometry and a shared GT-matched primitive set, anchored labels raise Gaussian mIoU from 17.82 to 22.95; rendered selection is also evaluated on 28 held-out target-visible frames (Section 5.4, Table S6b).

For reviewer eYXu, decoder-free and interface-aligned controls test semantic erosion, feature mismatch, and the role of interface initialization. Over 47,387 geometry-matched GT triplets, DA3's same-object ranking falls from 58.46% to 50.53% across depth, without involving a referring decoder (Section 5.3, Table S2b). Explicit interface alignment improves global mIoU to 34.29, while the anchored ladder reaches 40.88 (Section 5.5, Table 5a). Section 4.2 also explains Proposition 1's exact semantic reference and its bound on deviations from that reference.

For reviewer XgFD, shared-initialization calibration compares full transport with disabled and uniform-neighbor updates: 41.01 versus 40.88 and 40.62 global mIoU. The ladder also exceeds matched-referring-budget MVGGT, 40.88 versus 40.14 (Section 5.5, Table 5a). Gaussian-level controls clarify the readout's supervision: owner exclusion removes the 31.46% source-owner share of rendering gradients, while paired heads retain the same 23.02 Gaussian mIoU and 15.21 held-out mask mIoU (Section 5.4, Sections S2.4 and S4.1).

We hope that these revisions and responses address the concerns of the reviewers and the Area Chair. Should any concerns remain, please feel free to raise them; we will respond promptly and conduct additional experiments as needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.