acceptodds
Under review as a conference paper at ICLR 2027

Semantic and Spatial Disambiguation for Multi-view 3D Referring Expression Segmentation

Abstract

Multi-view 3D referring expression segmentation (MV-3DRES) aims to reconstruct a 3D scene from sparse multi-view RGB images and segment the object specified by a natural language description. However, sparse observations result in incomplete and fragmented object representations, making it difficult for existing methods to capture category-specific semantics and distinguish object instances based on their spatial locations. To address these challenges, we propose MV-S²D, a framework that jointly mitigates semantic and spatial ambiguities in MV-3DRES. Specifically, we first introduce a Semantic Category Enhancement (SCE) strategy that strengthens category-specific representations of the target object through text-guided semantic alignment and enhancement, thereby improving semantic discrimination. To mitigate spatial instance ambiguity, we then develop a Spatial Position Guidance (SPG) mechanism that estimates text-conditioned confidence scores over scene locations to identify target-relevant regions. Finally, we design a multi-view aggregation strategy to improve the cross-view consistency of segmentation predictions. Experiments on MVRefer demonstrate that MV-S²D achieves state-of-the-art performance, outperforming MVGGT by 7.3 points in [email protected] on the challenging Multiple subset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D referring expression segmentation, multi-view learning, sparse-view perception, vision-language learning, referent disambiguation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GmTzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work tackles the task of Multi-View 3D Referring Expression Segmentation, the task proposed by MVGGT, given multiple sparse images of a scene and a detailed description, the segmentation should be found. The authors propose to improve MVGGT in a) semantic category ambiguity, b) spatial instance ambiguity, c) lack of cross-view consistency. As solution, they propose a) Semantic Category Enhancement, b) Spatial Position Guidance, c) Cross-view Query Aggregation. They add new losses for a) deep loss and SCE loss, and for c) CQA loss. The components are well ablated, however each component could be better ablated with its individual losses. Overall there is an improvement of 2.9 percentage points in IoU for fair 224x224 training resolution comparison. In the main paper they always evaluate their method with the unfair 448x448 training resolution. While the improvement is significant, the method changes are more task-and-method tailored. SPG could be further applied but is very close to Mask2Former without comparison. CQA could be used for other multi-view tasks, however the parameter count also increases which is not shown in comparison with adding an additional layer to MVGGT, language tokens using.

Critical strengths

Method specific problems are well motivated, a) semantic category ambiguity, b) spatial instance ambiguity, c) lack of cross-view consistency.

All three components SCE, SPG, and CQA are well ablated and each shows improvement.

Overall there is an improvement of 3.3 percentage points in IoU.

SPG does not introduce new objectives, and its underlying idea is to use thresholded previous masks to guide later layers to mask out a single instance rather than multiple.

Critical weaknesses

Overall the task addressed is a end-user task and the motivational task example is not well-posed for me. The description of “The round table is to the right of the couch” does not describe the as-marked table well. Of course this is a single example, but it was deliberately chosen by the authors, and it is still not convincing.

The method is hard to understand, a lot of architecture changes at different stages, and additional losses, without explaining the losses in the main paper.

The general idea of SPG, guiding later layers with masked attention, was already introduced in Mask2Former.

The deep loss at multiple levels is not ablated and is very similar to the ideas of UNet++, Mask2Former.

The individual CQA losses are not ablated.

It is unclear if a validation set with early-stopping is used.

All ablations should be done in 224x224 training resolution to compare fairly with other works and to compare with MVGGT in the main paper the training resolution of 224x224 should be used.

Questions for authors

Given that the proposed changes are very much task and method tailored, how could they be applied in more general scenarios?

Are you using early stopping with a validation set?

11 comments
AuthorsOpenReview ↗

Response (Part 1)

We sincerely thank the reviewer for the positive evaluation of our work and the constructive suggestions. We appreciate your recognition of our problem motivation and module designs. Below, we address each of your concerns through additional ablation studies and comparative experiments.

1. Regarding the Motivation Example

We thank the reviewer for pointing out the lack of precision in the referring expression shown in Figure 1. The original annotation is "The small table. The small round table is on the right by the couch." For a more concise illustration, we shortened the original sentence when preparing the motivation figure. We apologize for the confusion this may have caused. We have restored the complete original expression in Figure 1 of the revised manuscript.

2. Clarification of the Method Architecture and Training Objectives

We thank the reviewer for pointing out that the method description is not sufficiently clear. We have revised the Method section to improve its clarity and readability. Our method does not simply stack modules at different stages of the network. Instead, each module is specifically designed to address a distinct ambiguity issue in sparse multi-view 3D referring expression segmentation:

· Semantic category ambiguity. We introduce SCE, which enhances language features through learnable category anchors and a Token–Channel gating mechanism, thereby reducing category confusion.

· Spatial instance ambiguity. We introduce SPG, which combines reverse text–visual attention with historical mask confidence to select candidate locations and construct soft spatial prompts, enabling the model to distinguish between candidate instances belonging to the same category.

· Cross-view prediction inconsistency. We further introduce CQA, which aggregates complementary information from different views through shared queries to improve cross-view segmentation consistency.

Furthermore, we have clarified the training objective by distinguishing the losses inherited from the baseline from those newly introduced in this work. The final training objective is formulated as follows: Ltotal=Lseg+0.5Ldeep+0.04LSCE+0.3LCQA-aux+λbg(e)Lbg+0.02Lp2p\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{seg}} + 0.5\mathcal{L}_{\mathrm{deep}} + 0.04\mathcal{L}_{\mathrm{SCE}} + 0.3\mathcal{L}_{\mathrm{CQA\text{-}aux}} + \lambda_{\mathrm{bg}}(e)\mathcal{L}_{\mathrm{bg}} + 0.02\mathcal{L}_{\mathrm{p2p}} The origin and role of each loss term are summarized in Table R1.

Table R1: Composition, Origin, and Function of the Loss Terms

Loss FunctionOriginFunction
Lseg\mathcal{L}_{\mathrm{seg}}MVGGT baselineBCE and Dice supervision for the final mask
Ldeep\mathcal{L}_{\mathrm{deep}}MVGGT baselineDeep supervision of predictions from intermediate multimodal layers
LSCE\mathcal{L}_{\mathrm{SCE}}Newly introducedSupervises semantic anchors through category cross-entropy
LCQA−aux\mathcal{L}_{\mathrm{CQA-aux}}Newly introducedBCE and Dice supervision for the CQA-aggregated mask
Lbg\mathcal{L}_{\mathrm{bg}}Newly introducedSuppresses foreground responses outside the GT target region
Lp2p\mathcal{L}_{\mathrm{p2p}}Newly introducedImproves target-background feature discriminability through patch-level contrastive learning

We have added a consolidated explanation of the training objective and a more detailed description of the loss components to the main text of the revised manuscript.

AuthorsOpenReview ↗

Response (Part 2)

3. Relationship Between SPG and Mask2Former

While both our SPG and Mask2Former [1] leverage intermediate mask predictions to provide spatial cues, they adopt fundamentally different strategies for deriving and applying spatial guidance.

· Different attention mechanisms. Mask2Former employs target queries to perform cross-attention with visual features. SPG additionally introduces reverse text–visual attention, using text tokens as Queries and visual tokens as Keys to compute the relevance between textual descriptions and visual locations.

· Different spatial guidance strategies. Mask2Former directly uses historical binary masks to constrain the attention region. In contrast, SPG combines historical mask confidence to select candidate locations, constructs soft spatial prompts based on text relevance, and injects them into visual features through a gated residual connection.

To compare the effectiveness of these two designs, we adapted the Mask-guided Cross-Attention (MCA) mechanism from Mask2Former to MVGGT to replace SPG.

Table R2: Performance Comparison Between SPG and Mask-guided Cross-Attention (MCA)

MethodGlobal mIoU[email protected][email protected]
MV-S²D (MCA)42.258.445.1
MV-S²D (SPG)42.859.146.3

As shown in Table R2, the results show that MV-S²D (SPG) outperforms MV-S²D (MCA) on all evaluation metrics. Compared with simply using historical masks to constrain the attention region, the text-relevance modeling and soft spatial prompts introduced by SPG can more effectively localize the referred target. These results demonstrate the performance advantage of SPG over Mask-guided Cross-Attention.

We also conducted additional ablation experiments (Table R3) on the internal components of SPG by separately removing reverse text–visual attention and historical mask confidence filtering. Removing either component leads to a performance degradation, further validating the effectiveness of the SPG design. The results are presented below.

Table R3: Ablation Study of the Internal Components of SPG

SettingGlobal mIoU[email protected][email protected]
w/o Reverse Attention41.757.744.7
w/o Confidence Filtering41.557.744.2
Full SPG42.859.146.3

4. Ablation Study of the Multi-Layer Deep Loss

The multi-layer Deep Loss follows the deep-supervision strategy adopted in the MVGGT baseline. Deep supervision is a commonly used training strategy in segmentation tasks and is not a newly introduced component of our method. To further investigate its impact on model performance, we conducted an additional ablation study (Table R4) by removing the Deep Loss.

Table R4: Ablation Study of the Multi-Layer Deep Supervision Loss

SettingGlobal mIoU[email protected][email protected]
w/o Deep Loss41.357.544.3
Full42.859.146.3

5. Individual Ablation Studies of the CQA Loss Components

CQA incorporates an auxiliary segmentation loss, a background suppression loss, and a patch-level contrastive loss. We conducted additional ablation studies (Table R5) to evaluate their individual contributions.

Table R5: Individual Ablation Studies of the CQA Auxiliary Loss Components

SettingGlobal mIoU[email protected][email protected]
w/o CQA Aux Seg41.958.144.7
w/o Background Suppression42.158.245.2
w/o Patch Contrastive41.857.845.1
Full42.859.146.3

The results show that removing any of the auxiliary losses leads to a performance degradation, validating the effectiveness of all three loss components.

6. Whether Early Stopping Was Used

We did not use validation-based Early Stopping. Instead, we trained the model for a fixed number of epochs and selected the best checkpoint according to validation performance. This strategy is consistent with MVGGT [2], IPDN [3], Position3D [4], and 3D-STMN [5], and is a commonly adopted model-selection protocol in this field, ensuring a fair comparison.

As shown in Table R6, the model achieves its best performance at Epoch 22 (42.82%), while its performance at Epoch 30 is 42.74%, only 0.08 points lower than the best performance. This indicates that the model has largely converged by approximately Epoch 22.

Table R6: Validation Performance at Different Training Epochs

MetricEpoch 1Epoch 5Epoch 10Epoch 15Epoch 20Epoch 22Epoch 25Epoch 28Epoch 29Epoch 30
Global mIoU26.5033.4738.8240.6741.4442.8242.2942.7542.7542.74
[email protected]40.7951.1454.3857.3357.0559.1358.6758.9258.9358.95
[email protected]21.0930.5440.9543.0144.2946.3045.9046.1546.2046.19
AuthorsOpenReview ↗

Response (Part 3)

7. Fairness of Training Resolution

We would like to clarify that all module ablation studies in our paper were conducted using a training resolution of 224×224, consistent with MVGGT. The appendix section entitled "More Experimental Results" in the original manuscript already includes performance comparisons under the same training resolution, as shown in Table R7 below.

Table R7: Performance Comparison Between MVGGT and MV-S²D at Different Training Resolutions

MethodTraining ResolutionGlobal mIoU[email protected][email protected]Multiple mIoUMultiple [email protected]
MVGGT224×22439.955.941.533.833.5
MV-S²D224×22442.859.146.336.938.8
MV-S²D448×44843.257.947.837.440.8

Under the same training resolution, our method also achieves a gain of 2.9 points in Global mIoU over MVGGT, demonstrating the effectiveness of the proposed method. We have incorporated the results under the 224×224 training resolution into the main text of the revised manuscript to provide a more comprehensive evaluation of our method.

8. Applicability to More General Scenarios

To address your concern, we have conducted additional experiments in the revised manuscript to further evaluate the generalizability of our proposed framework. The results demonstrate that our disambiguation mechanisms are not limited to the specific MVGGT architecture or sparse multi-view inputs, but can also be transferred across different datasets, model architectures, and tasks.

Specifically, we conducted cross-dataset experiments on ReferIt3D (Nr3D and Sr3D) [6] under zero-shot cross-dataset (Table R8) and supervised settings (Tables R9, R10). Moreover, we transferred the core disambiguation mechanisms (SCE and SPG) to MDIN [7] and conducted experiments on 3D-GRES (Table R11) and 3D-RES (Table R12) using the Multi3DRes and ScanRefer datasets, respectively.

Table R8: Zero-shot Cross-Dataset Evaluation Results on Nr3D and Sr3D

DatasetMethodGlobal mIoU[email protected][email protected]
Nr3DMVGGT23.534.922.7
Nr3DMV-S²D25.237.025.7
Sr3DMVGGT21.632.321.2
Sr3DMV-S²D22.232.622.2

Table R9: Supervised Training Results on Nr3D

MethodGlobal mIoU[email protected][email protected]
MVGGT26.539.126.8
MV-S²D28.341.529.2

Table R10: Supervised Training Results on Sr3D

MethodGlobal mIoU[email protected][email protected]
MVGGT25.440.223.4
MV-S²D26.841.525.6

Table R11: Transfer Results on the 3D-GRES Task Using MDIN

MethodmIoU[email protected][email protected]
MDIN40.258.934.9
MDIN + SCE + SPG42.060.937.0

Table R12: Transfer Results on the 3D-RES Task Using MDIN

MethodmIoU[email protected][email protected]
MDIN45.254.949.6
MDIN + SCE + SPG47.557.252.3

These results demonstrate the general applicability of our proposed method across different datasets, model architectures, and tasks. We have incorporated the above experimental results and corresponding discussions into the revised manuscript.

References

[1] Bowen Cheng et al. "Masked-attention mask transformer for universal image segmentation." CVPR, 2022.

[2] Changli Wu et al. "MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation." CVPR, 2026.

[3] Qi Chen et al. "Ipdn: Image-enhanced prompt decoding network for 3d referring expression segmentation." AAAI, 2025.

[4] Yabing Wang et al. "Spatial Matters: Position-Guided 3D Referring Expression Segmentation." CVPR, 2026.

[5] Changli Wu et al. "3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation." AAAI, 2024.

[6] Panos Achlioptas et al. "Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes." ECCV, 2020.

[7] Changli Wu et al. "3d-gres: Generalized 3d referring expression segmentation." ACM MM, 2024.

Reviewer_rnBUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MV-S2D, a framework for multi-view 3D referring expression segmentation under sparse RGB observations. It introduces Semantic Category Enhancement (SCE) to strengthen category-specific language representations, Spatial Position Guidance (SPG) to emphasize target-relevant visual regions, and Cross-view Query Aggregation (CQA) to enhance cross-view consistency. Experiments on MVRefer demonstrate promising performance, particularly on the challenging Multiple subset.

Critical strengths
  1. Clear presentation and comprehensive experiments. The paper is well-written and easy to follow, with clearly presented and well-formulated equations. The figures and tables are clear, visually appealing, and well-organized. The experimental evaluation is comprehensive and reflects substantial effort.
  2. Promising empirical results. The proposed method achieves competitive performance on MVRefer, with particularly notable improvements on the challenging Multiple subset involving multiple candidate instances.
Critical weaknesses
  1. Limited convergence analysis. The paper does not provide training or validation curves to characterize the optimization process. Including such curves would provide additional insights into convergence behavior and training stability.
Questions for authors
  1. Could the authors provide training and validation curves to demonstrate convergence behavior and training stability?
  2. What motivates the choice of DINOv2 and RoBERTa as the visual and language encoders, respectively? Could more recent and advanced models further improve performance?
2 comments

We sincerely thank the reviewer for the positive evaluation of our work and the constructive suggestions. We appreciate your recognition of our comprehensive experiments and the performance improvements achieved by our method. Below, we address each of your concerns through additional ablation studies and comparative experiments.

1. Analysis of Model Convergence and Training Stability

We have organized the training logs, which show that the training process remained stable throughout. The corresponding results are presented in Table R1. We have added the training and validation curves to the supplementary material of the revised manuscript.

Table R1: Training Loss, Validation Loss, and Global mIoU at Different Training Stages

MetricEpoch 1Epoch 5Epoch 10Epoch 15Epoch 20Epoch 25Epoch 30
Training Loss4.45423.52832.45112.19881.94041.80141.7546
Validation Loss2.58562.30842.24362.12652.09132.11192.1137
Global mIoU26.5033.4738.8240.6741.4442.2942.74

2. Selection and Replacement of Visual and Language Encoders

2.1. Selection of the DINOv2 Visual Encoder

We retain the pretrained visual-geometry reconstruction branch based on π³ [1] used in MVGGT to ensure a fair comparison. Since π³ adopts DINOv2 as its underlying visual encoder, replacing DINOv2 with DINOv3 alone would affect the compatibility of the original reconstruction framework.

We appreciate your constructive suggestion. Due to the limited revision period and computational resources, we were unable to reliably integrate and thoroughly evaluate this alternative framework. Nevertheless, we plan to explore stronger and more recent backbones, such as Depth Anything 3 [2] and VGGT-omega [3], in our future work to further improve the proposed framework.

2.2. Selection of the RoBERTa-base Language Encoder

To investigate whether newer text encoders can improve performance, we additionally evaluated MPNet-base [4], DeBERTa-base [5], and ModernBERT-base [6], and compared them with the original RoBERTa-base.

Table R2: Performance Comparison of Different Text Encoders on MVRefer

Text EncoderGlobal mIoU[email protected][email protected]
RoBERTa-base42.859.146.3
MPNet-base41.857.445.2
DeBERTa-base41.457.144.3
ModernBERT-base39.254.342.0

As shown in Table R2, replacing the original text encoder with newer alternatives does not necessarily lead to performance improvements in the current multi-view 3D referring expression segmentation task. We attribute this observation to several possible factors:

· Text feature distributions. Different encoders employ different network architectures and pretraining objectives, resulting in differences in semantic representations, feature distributions, and contextual modeling. These differences may affect the representation of fine-grained object information.

· Pretraining corpora and tokenization strategies. Different models are trained on different pretraining corpora and adopt different tokenization strategies, which may influence how object names, attribute descriptions, and spatial relations are learned and encoded.

· Compatibility with downstream visual interaction. Text features produced by different encoders may differ in their representation spaces and semantic granularity, potentially affecting the alignment between textual and visual features as well as the effectiveness of cross-modal interactions.

References

[1] Yifan Wang et al. "π³: Permutation-Equivariant Visual Geometry Learning." ICLR, 2026.

[2] Haotong Lin et al. "Depth Anything 3: Recovering the Visual Space from Any Views." ICLR, 2026.

[3] Jianyuan Wang et al. "VGGT-Ω." CVPR, 2026.

[4] Kaitao Song et al. "Mpnet: Masked and permuted pre-training for language understanding." NeurIPS, 2020.

[5] Pengcheng He et al. "Deberta: Decoding-enhanced bert with disentangled attention." ICLR, 2021.

[6] Benjamin Warner et al. "Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference." ACL, 2025.

The authors have addressed my concerns. I will maintain my original positive score.

Reviewer_x5SLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript proposed a framework that jointly mitigates semantic and spatial ambiguities in Multi-view 3D Referring Expression Segmentation task. Specifically, it introduces a Semantic Category Enhancement (SCE) strategy that strengthens category-specific representations of the target object through text-guided semantic alignment and enhancement, a Spatial Position Guidance (SPG) mechanism that estimates text-conditioned confidence scores over scene locations to identify target-relevant regions, and a multi-view aggregation strategy that improves the cross-view consistency of segmentation predictions. Experiments on the MVRefer benchmark shows that the proposed method outperforming MVGGT.

Critical strengths
  1. Well-structured problem formulation. This manuscript models the failure modes under sparse views as two separable forms of referential ambiguity, semantic category ambiguity and spatial instance ambiguity, and maps each module to one ambiguity type. This decomposition makes the design rationale and the ablation structure transparent.
  2. Systematic ablation and hyperparameter analysis. Tables 3–5 examine each component in isolation and in combination, and sweep the two most design-sensitive hyperparameters (the SPG confidence threshold and the SPG starting fusion layer). The analysis covers both aggregate and subset-level metrics (Hard, Multiple).
  3. Useful negative result. Table 6 adapts three representative disambiguation strategies from dense 3D referring segmentation (IPDN, Position3D, EG-3DVG) to the same MV-3DRES baseline and shows that none of them improves over the baseline, whereas the proposed SCE and SPG do. This directly supports the paper's argument that sparse-view disambiguation requires designs different from those developed for dense 3D inputs.
Critical weaknesses
  1. Novelty relative to MVGGT is incremental. The task definition, benchmark, and evaluation protocol are entirely inherited from MVGGT (CVPR26); this manuscript introduces no new task, setting, or benchmark. At the mechanism level, SCE is a learnable category-anchor representation with token-channel gating, which resembles existing category-prototype and prompt-enhancement designs; SPG combines a reversed text-to-visual attention with confidence-based position filtering, and reversed attention has precedents in grounded-visual and referring-understanding works; CQA uses shared queries for cross-view aggregation, a standard mechanism in multi-view transformers. Each module is an adapted combination of established techniques. The contribution is at the component level rather than at the level of a new mechanism or paradigm, which is a limited novelty basis for an ICLR submission.

  2. Open-vocabulary behavior of the category anchors is not addressed. SCE relies on a predefined set of K=19 categories plus an unknown class, and its category prediction is supervised by the target category. How the anchors behave for object categories outside this taxonomy, and whether the method degrades for open-vocabulary expressions, is not discussed.

Questions for authors
  1. How do the category anchors behave for referring expressions whose target categories fall outside the predefined set, and is the category-supervision signal always available for the benchmark annotations?
2 comments
AuthorsOpenReview ↗

Response (Part 1)

We sincerely thank the reviewer for the positive evaluation of our work and the constructive suggestions. We appreciate your recognition of our problem formulation and comparative experiments. Below, we address each of your concerns through additional ablation studies and comparative experiments.

1. Novelty of the Proposed Method

We acknowledge that MVGGT serves as our baseline, and we follow its MVRefer benchmark, data split, and evaluation protocol to ensure a fair comparison. However, MV-S²D differs fundamentally from MVGGT in terms of motivation, methodological design, and experimental analysis.

First, Motivation. MVGGT primarily addresses efficient end-to-end 3D referring segmentation from sparse multi-view RGB inputs and the associated foreground gradient dilution problem. Our MV-S²D focuses on two underexplored challenges caused by incomplete and fragmented observations: semantic category ambiguity and spatial instance ambiguity. Our goal is to jointly resolve these two types of ambiguity for more reliable target identification.

Second, Method. Rather than simply modifying the existing architecture or optimization strategy, we introduce a task-specific semantic and spatial disambiguation framework with three novel mechanisms: SCE enhances category-specific language representations, SPG provides text-guided spatial localization, and CQA aggregates complementary target information across views. These mechanisms explicitly address referential ambiguity from complementary perspectives.

Third, Experimental Validation. Beyond overall performance comparisons, we conduct targeted ablation studies and semantic–spatial disambiguation analyses to validate the effectiveness and complementary roles of the proposed components. These results further demonstrate that MV-S²D introduces a dedicated semantic–spatial disambiguation framework for sparse-view 3D referring expression segmentation.

Moreover, our individual modules also differ from existing approaches in several important aspects.

Existing methods use category prototypes (COSeg [1]) for visual feature matching or learnable prompt vectors (Prompt-RIS [2], CoDe [3]) to enhance feature representations. In contrast, SCE focuses on leveraging category semantics to resolve semantic category ambiguity.

Conventional reverse-attention methods (Mono3DVG-EnSD [4], TransRefer3D [5]) exchange the roles of Query and Key/Value for feature interaction. SPG, however, does not perform Value aggregation. Instead, it combines text relevance with historical mask confidence to identify target-relevant visual locations.

Unlike multi-view Transformers (ViewRefer [6], PanSt3R [7]) that employ shared queries for cross-view aggregation, CQA further incorporates language-guided weighting and mask fusion to integrate cross-view predictions.

Furthermore, we conduct additional experiments to validate the generalizability and transferability of the proposed semantic and spatial disambiguation mechanisms in MV-S²D. Specifically, we evaluate our method on Nr3D and Sr3D [8] under both zero-shot cross-dataset (Table R1) and supervised settings (Table R2, R3). Moreover, we transfer the core mechanisms (SCE and SPG) to MDIN [9] and evaluate them on 3D-GRES (Table R4) and 3D-RES (Table R5) under dense-input settings.

The results consistently show that our proposed mechanisms bring performance improvements across different datasets, training settings, model architectures, and tasks. These findings demonstrate that our contributions go beyond incremental improvements over MVGGT, providing generalizable semantic and spatial disambiguation mechanisms that can be transferred to diverse datasets, model architectures, and 3D referring tasks.

Table R1: Zero-shot Cross-Dataset Evaluation Results on Nr3D and Sr3D

DatasetMethodGlobal mIoU[email protected][email protected]
Nr3DMVGGT23.534.922.7
Nr3DMV-S²D25.237.025.7
Sr3DMVGGT21.632.321.2
Sr3DMV-S²D22.232.622.2

Table R2: Supervised Training Results on Nr3D

MethodGlobal mIoU[email protected][email protected]
MVGGT26.539.126.8
MV-S²D28.341.529.2

Table R3: Supervised Training Results on Sr3D

MethodGlobal mIoU[email protected][email protected]
MVGGT25.440.223.4
MV-S²D26.841.525.6

Table R4: Transfer Results on the 3D-GRES Task Using MDIN

MethodmIoU[email protected][email protected]
MDIN40.258.934.9
MDIN + SCE + SPG42.060.937.0

Table R5: Transfer Results on the 3D-RES Task Using MDIN

MethodmIoU[email protected][email protected]
MDIN45.254.949.6
MDIN + SCE + SPG47.557.252.3
AuthorsOpenReview ↗

Response (Part 2)

2. Behavior of Category Anchors for Categories Outside the Predefined Set

For target objects outside the predefined category set, SCE handles them through the Unknown category. Specifically, the category anchors consist of 18 predefined categories and one Unknown category. During training, objects belonging to categories outside this predefined set are uniformly mapped to the Unknown category for supervision.During inference, category semantics are aggregated through soft category selection, while the residual connection preserves information from the original text, allowing the model to localize targets belonging to unseen categories.

3. Performance on Open-Vocabulary Referring Expressions

We selected target categories that fall outside the predefined category set but are present in the test set, obtaining 143 referring expressions across 23 distinct target categories. We evaluated the models on this subset (Table R6).

Table R6: Performance Comparison on Categories Outside the Predefined Category Set

MethodGlobal mIoU
MVGGT26.76
MV-S²D w/o SCE30.33
MV-S²D Full30.61

On this subset of categories outside the predefined category set, Full MV-S²D achieves a gain of 3.85 points over MVGGT and a further gain of 0.28 points over MV-S²D without SCE. These results indicate that SCE also benefits open-vocabulary generalization.

4. Availability of Category Supervision Signals

The annotations in the benchmark consistently provide category supervision signals. In our training set, every target object has an "object_name" field, resulting in 100% category supervision coverage. We directly use the existing category annotations to construct auxiliary supervision, with all objects outside the 18 predefined categories being mapped to the Unknown category.

References

[1] Zhaochong An et al. "Rethinking few-shot 3d point cloud semantic segmentation." CVPR, 2024.

[2] Chao Shang et al. "Prompt-driven referring image segmentation with instance contrasting." CVPR, 2024.

[3] Ji-Jia Wu et al. "Image-text co-decomposition for text-supervised semantic segmentation." CVPR, 2024.

[4] Yuzhen Li et al. "Mono3DVG-EnSD: Enhanced Spatial-aware and Dimension-decoupled Text Encoding for Monocular 3D Visual Grounding." AAAI, 2026.

[5] Dailan He et al. "TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding." ACM MM, 2021.

[6] Zoey Guo et al. "Viewrefer: Grasp the multi-view knowledge for 3d visual grounding." ICCV, 2023.

[7] Lojze Žust et al. "Panst3r: Multi-view consistent panoptic segmentation." ICCV, 2025.

[8] Panos Achlioptas et al. "Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes." ECCV, 2020.

[9] Changli Wu et al. "3d-gres: Generalized 3d referring expression segmentation." ACM MM, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewer Comments

We sincerely thank all reviewers for taking the time to review our manuscript and provide constructive suggestions. We also appreciate the reviewers' positive evaluations of our problem motivation, module designs, ablation studies, and performance improvements.

In this paper, we propose MV-S²D (Semantic and Spatial Disambiguation) to address three key challenges in sparse multi-view 3D referring expression segmentation: (1) Semantic category ambiguity. We introduce SCE, which enhances language features through learnable category anchors and a Token–Channel gating mechanism, thereby reducing category confusion. (2) Spatial instance ambiguity. We introduce SPG, which combines reverse text–visual attention with historical mask confidence to select candidate locations and construct soft spatial prompts, enabling the model to distinguish between candidate instances belonging to the same category. (3) Cross-view prediction inconsistency. We further introduce CQA, which aggregates complementary information from different views through shared queries to improve cross-view segmentation consistency.

On MVRefer, compared with MVGGT, our method achieves a gain of 3.3 points in Global mIoU and a larger improvement of 7.3 points in [email protected] on the challenging Multiple subset.

In response to the reviewers' comments, we have conducted the following additional experiments and analyses, and incorporated the corresponding results directly into the revised manuscript and the responses to the respective reviewers for further discussion:

  • Clarified the differences between MV-S²D and MVGGT in terms of research motivation and methodological design, and further explained how SCE, SPG, and CQA differ in design from existing methods based on category prototypes, prompt-based feature enhancement, reverse attention, and multi-view Transformers.

  • Conducted zero-shot cross-dataset experiments on Nr3D and Sr3D (Appendix Table 22) and supervised cross-dataset experiments (Main Text Table 10). We also transferred SCE and SPG to MDIN and evaluated them on the 3D-GRES and 3D-RES tasks (Main Text Table 11).

  • Analyzed the category configuration and category supervision mechanism in SCE (Section 3.2 of the main text) and conducted an open-vocabulary generalization evaluation (Main Text Table 9).

  • Added training and validation loss curves (Appendix Figure 7), as well as training loss, validation loss, and Global mIoU at different training epochs (Appendix Table 19).

  • Added comparative experiments involving RoBERTa-base, MPNet-base, DeBERTa-base, and ModernBERT-base as text encoders (Appendix Table 18), together with an explanation of the selection of the visual-geometry backbone.

  • Improved the motivation and detailed mechanisms of SCE, SPG, and CQA, added the complete training objective and explanations of the individual loss components, and clarified the distinction between losses inherited from MVGGT and those newly introduced in this work (Sections 3.2–3.5 and Equation 11 of the main text).

  • Added ablation studies on the internal components of SPG (Appendix Table 20) and an independent ablation study removing the multi-layer Deep Loss (Appendix Table 21). We also conducted individual ablation studies by removing the auxiliary segmentation loss, background suppression loss, and patch-level contrastive loss in CQA (Appendix Table 15).

  • Added fair comparisons under different training resolutions (Main Text Table 1) and revised Figure 1 to restore the complete original referring expression from the dataset.

These additional experiments and analyses further demonstrate the methodological novelty of MV-S²D and validate its effectiveness, generalizability, and transferability across different datasets, model architectures, and tasks. We have also improved the method descriptions, experimental results, and relevant discussions in the revised manuscript according to the reviewers' suggestions.

Once again, we sincerely thank all reviewers for their valuable comments. In the following, we address the concerns raised by each reviewer individually, together with the corresponding experimental results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.