- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work tackles the task of Multi-View 3D Referring Expression Segmentation, the task proposed by MVGGT, given multiple sparse images of a scene and a detailed description, the segmentation should be found. The authors propose to improve MVGGT in a) semantic category ambiguity, b) spatial instance ambiguity, c) lack of cross-view consistency. As solution, they propose a) Semantic Category Enhancement, b) Spatial Position Guidance, c) Cross-view Query Aggregation. They add new losses for a) deep loss and SCE loss, and for c) CQA loss. The components are well ablated, however each component could be better ablated with its individual losses. Overall there is an improvement of 2.9 percentage points in IoU for fair 224x224 training resolution comparison. In the main paper they always evaluate their method with the unfair 448x448 training resolution. While the improvement is significant, the method changes are more task-and-method tailored. SPG could be further applied but is very close to Mask2Former without comparison. CQA could be used for other multi-view tasks, however the parameter count also increases which is not shown in comparison with adding an additional layer to MVGGT, language tokens using.
Critical strengths
Method specific problems are well motivated, a) semantic category ambiguity, b) spatial instance ambiguity, c) lack of cross-view consistency.
All three components SCE, SPG, and CQA are well ablated and each shows improvement.
Overall there is an improvement of 3.3 percentage points in IoU.
SPG does not introduce new objectives, and its underlying idea is to use thresholded previous masks to guide later layers to mask out a single instance rather than multiple.
Critical weaknesses
Overall the task addressed is a end-user task and the motivational task example is not well-posed for me. The description of “The round table is to the right of the couch” does not describe the as-marked table well. Of course this is a single example, but it was deliberately chosen by the authors, and it is still not convincing.
The method is hard to understand, a lot of architecture changes at different stages, and additional losses, without explaining the losses in the main paper.
The general idea of SPG, guiding later layers with masked attention, was already introduced in Mask2Former.
The deep loss at multiple levels is not ablated and is very similar to the ideas of UNet++, Mask2Former.
The individual CQA losses are not ablated.
It is unclear if a validation set with early-stopping is used.
All ablations should be done in 224x224 training resolution to compare fairly with other works and to compare with MVGGT in the main paper the training resolution of 224x224 should be used.
Questions for authors
Given that the proposed changes are very much task and method tailored, how could they be applied in more general scenarios?
Are you using early stopping with a validation set?
11 comments
Response (Part 2)
3. Relationship Between SPG and Mask2Former
While both our SPG and Mask2Former [1] leverage intermediate mask predictions to provide spatial cues, they adopt fundamentally different strategies for deriving and applying spatial guidance.
· Different attention mechanisms. Mask2Former employs target queries to perform cross-attention with visual features. SPG additionally introduces reverse text–visual attention, using text tokens as Queries and visual tokens as Keys to compute the relevance between textual descriptions and visual locations.
· Different spatial guidance strategies. Mask2Former directly uses historical binary masks to constrain the attention region. In contrast, SPG combines historical mask confidence to select candidate locations, constructs soft spatial prompts based on text relevance, and injects them into visual features through a gated residual connection.
To compare the effectiveness of these two designs, we adapted the Mask-guided Cross-Attention (MCA) mechanism from Mask2Former to MVGGT to replace SPG.
Table R2: Performance Comparison Between SPG and Mask-guided Cross-Attention (MCA)
| Method | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| MV-S²D (MCA) | 42.2 | 58.4 | 45.1 |
| MV-S²D (SPG) | 42.8 | 59.1 | 46.3 |
As shown in Table R2, the results show that MV-S²D (SPG) outperforms MV-S²D (MCA) on all evaluation metrics. Compared with simply using historical masks to constrain the attention region, the text-relevance modeling and soft spatial prompts introduced by SPG can more effectively localize the referred target. These results demonstrate the performance advantage of SPG over Mask-guided Cross-Attention.
We also conducted additional ablation experiments (Table R3) on the internal components of SPG by separately removing reverse text–visual attention and historical mask confidence filtering. Removing either component leads to a performance degradation, further validating the effectiveness of the SPG design. The results are presented below.
Table R3: Ablation Study of the Internal Components of SPG
| Setting | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| w/o Reverse Attention | 41.7 | 57.7 | 44.7 |
| w/o Confidence Filtering | 41.5 | 57.7 | 44.2 |
| Full SPG | 42.8 | 59.1 | 46.3 |
4. Ablation Study of the Multi-Layer Deep Loss
The multi-layer Deep Loss follows the deep-supervision strategy adopted in the MVGGT baseline. Deep supervision is a commonly used training strategy in segmentation tasks and is not a newly introduced component of our method. To further investigate its impact on model performance, we conducted an additional ablation study (Table R4) by removing the Deep Loss.
Table R4: Ablation Study of the Multi-Layer Deep Supervision Loss
| Setting | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| w/o Deep Loss | 41.3 | 57.5 | 44.3 |
| Full | 42.8 | 59.1 | 46.3 |
5. Individual Ablation Studies of the CQA Loss Components
CQA incorporates an auxiliary segmentation loss, a background suppression loss, and a patch-level contrastive loss. We conducted additional ablation studies (Table R5) to evaluate their individual contributions.
Table R5: Individual Ablation Studies of the CQA Auxiliary Loss Components
| Setting | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| w/o CQA Aux Seg | 41.9 | 58.1 | 44.7 |
| w/o Background Suppression | 42.1 | 58.2 | 45.2 |
| w/o Patch Contrastive | 41.8 | 57.8 | 45.1 |
| Full | 42.8 | 59.1 | 46.3 |
The results show that removing any of the auxiliary losses leads to a performance degradation, validating the effectiveness of all three loss components.
6. Whether Early Stopping Was Used
We did not use validation-based Early Stopping. Instead, we trained the model for a fixed number of epochs and selected the best checkpoint according to validation performance. This strategy is consistent with MVGGT [2], IPDN [3], Position3D [4], and 3D-STMN [5], and is a commonly adopted model-selection protocol in this field, ensuring a fair comparison.
As shown in Table R6, the model achieves its best performance at Epoch 22 (42.82%), while its performance at Epoch 30 is 42.74%, only 0.08 points lower than the best performance. This indicates that the model has largely converged by approximately Epoch 22.
Table R6: Validation Performance at Different Training Epochs
| Metric | Epoch 1 | Epoch 5 | Epoch 10 | Epoch 15 | Epoch 20 | Epoch 22 | Epoch 25 | Epoch 28 | Epoch 29 | Epoch 30 |
|---|---|---|---|---|---|---|---|---|---|---|
| Global mIoU | 26.50 | 33.47 | 38.82 | 40.67 | 41.44 | 42.82 | 42.29 | 42.75 | 42.75 | 42.74 |
| [email protected] | 40.79 | 51.14 | 54.38 | 57.33 | 57.05 | 59.13 | 58.67 | 58.92 | 58.93 | 58.95 |
| [email protected] | 21.09 | 30.54 | 40.95 | 43.01 | 44.29 | 46.30 | 45.90 | 46.15 | 46.20 | 46.19 |
Response (Part 3)
7. Fairness of Training Resolution
We would like to clarify that all module ablation studies in our paper were conducted using a training resolution of 224×224, consistent with MVGGT. The appendix section entitled "More Experimental Results" in the original manuscript already includes performance comparisons under the same training resolution, as shown in Table R7 below.
Table R7: Performance Comparison Between MVGGT and MV-S²D at Different Training Resolutions
| Method | Training Resolution | Global mIoU | [email protected] | [email protected] | Multiple mIoU | Multiple [email protected] |
|---|---|---|---|---|---|---|
| MVGGT | 224×224 | 39.9 | 55.9 | 41.5 | 33.8 | 33.5 |
| MV-S²D | 224×224 | 42.8 | 59.1 | 46.3 | 36.9 | 38.8 |
| MV-S²D | 448×448 | 43.2 | 57.9 | 47.8 | 37.4 | 40.8 |
Under the same training resolution, our method also achieves a gain of 2.9 points in Global mIoU over MVGGT, demonstrating the effectiveness of the proposed method. We have incorporated the results under the 224×224 training resolution into the main text of the revised manuscript to provide a more comprehensive evaluation of our method.
8. Applicability to More General Scenarios
To address your concern, we have conducted additional experiments in the revised manuscript to further evaluate the generalizability of our proposed framework. The results demonstrate that our disambiguation mechanisms are not limited to the specific MVGGT architecture or sparse multi-view inputs, but can also be transferred across different datasets, model architectures, and tasks.
Specifically, we conducted cross-dataset experiments on ReferIt3D (Nr3D and Sr3D) [6] under zero-shot cross-dataset (Table R8) and supervised settings (Tables R9, R10). Moreover, we transferred the core disambiguation mechanisms (SCE and SPG) to MDIN [7] and conducted experiments on 3D-GRES (Table R11) and 3D-RES (Table R12) using the Multi3DRes and ScanRefer datasets, respectively.
Table R8: Zero-shot Cross-Dataset Evaluation Results on Nr3D and Sr3D
| Dataset | Method | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|---|
| Nr3D | MVGGT | 23.5 | 34.9 | 22.7 |
| Nr3D | MV-S²D | 25.2 | 37.0 | 25.7 |
| Sr3D | MVGGT | 21.6 | 32.3 | 21.2 |
| Sr3D | MV-S²D | 22.2 | 32.6 | 22.2 |
Table R9: Supervised Training Results on Nr3D
| Method | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| MVGGT | 26.5 | 39.1 | 26.8 |
| MV-S²D | 28.3 | 41.5 | 29.2 |
Table R10: Supervised Training Results on Sr3D
| Method | Global mIoU | [email protected] | [email protected] |
|---|---|---|---|
| MVGGT | 25.4 | 40.2 | 23.4 |
| MV-S²D | 26.8 | 41.5 | 25.6 |
Table R11: Transfer Results on the 3D-GRES Task Using MDIN
| Method | mIoU | [email protected] | [email protected] |
|---|---|---|---|
| MDIN | 40.2 | 58.9 | 34.9 |
| MDIN + SCE + SPG | 42.0 | 60.9 | 37.0 |
Table R12: Transfer Results on the 3D-RES Task Using MDIN
| Method | mIoU | [email protected] | [email protected] |
|---|---|---|---|
| MDIN | 45.2 | 54.9 | 49.6 |
| MDIN + SCE + SPG | 47.5 | 57.2 | 52.3 |
These results demonstrate the general applicability of our proposed method across different datasets, model architectures, and tasks. We have incorporated the above experimental results and corresponding discussions into the revised manuscript.
References
[1] Bowen Cheng et al. "Masked-attention mask transformer for universal image segmentation." CVPR, 2022.
[2] Changli Wu et al. "MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation." CVPR, 2026.
[3] Qi Chen et al. "Ipdn: Image-enhanced prompt decoding network for 3d referring expression segmentation." AAAI, 2025.
[4] Yabing Wang et al. "Spatial Matters: Position-Guided 3D Referring Expression Segmentation." CVPR, 2026.
[5] Changli Wu et al. "3d-stmn: Dependency-driven superpoint-text matching network for end-to-end 3d referring expression segmentation." AAAI, 2024.
[6] Panos Achlioptas et al. "Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes." ECCV, 2020.
[7] Changli Wu et al. "3d-gres: Generalized 3d referring expression segmentation." ACM MM, 2024.
Response (Part 1)
We sincerely thank the reviewer for the positive evaluation of our work and the constructive suggestions. We appreciate your recognition of our problem motivation and module designs. Below, we address each of your concerns through additional ablation studies and comparative experiments.
1. Regarding the Motivation Example
We thank the reviewer for pointing out the lack of precision in the referring expression shown in Figure 1. The original annotation is "The small table. The small round table is on the right by the couch." For a more concise illustration, we shortened the original sentence when preparing the motivation figure. We apologize for the confusion this may have caused. We have restored the complete original expression in Figure 1 of the revised manuscript.
2. Clarification of the Method Architecture and Training Objectives
We thank the reviewer for pointing out that the method description is not sufficiently clear. We have revised the Method section to improve its clarity and readability. Our method does not simply stack modules at different stages of the network. Instead, each module is specifically designed to address a distinct ambiguity issue in sparse multi-view 3D referring expression segmentation:
Furthermore, we have clarified the training objective by distinguishing the losses inherited from the baseline from those newly introduced in this work. The final training objective is formulated as follows: Ltotal=Lseg+0.5Ldeep+0.04LSCE+0.3LCQA-aux+λbg(e)Lbg+0.02Lp2p The origin and role of each loss term are summarized in Table R1.
Table R1: Composition, Origin, and Function of the Loss Terms
We have added a consolidated explanation of the training objective and a more detailed description of the loss components to the main text of the revised manuscript.