- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission presents AnyGranGS, which is a framework for language-guided 3D Gaussian Splatting segmentation across objects, parts, part sets, and conditioned parts. The method firstly obtains an initial mask using an MLLM and SAM, and then associates it with a coarse target proxy in a semantic Gaussian field. It constructs target-centered virtual views, and select complementary observations using marginal visibility gain. For spatially conditioned queries, it estimates object-centric semantic axes, and filters candidate parts according to their 3D positions. Finally, it aggregates visibility-weighted foreground and background evidence to refine Gaussian membership. This work also introduce Gran-LERF and Gran-3D-OVS, and reports substantial improvements on these two benchmarks and Ref-LERF.
Critical strengths
1. The submission addresses two concrete limitations with well-motivated designs.
Intuitively, fine-grained parts often require closer and less occluded observations, while object-relative directions cannot reliably be interpreted as image-relative directions. The distinction between these problems are clear.
2. The view-selection objective accounts for complementary observations.
Rather than selecting views solely by their individual visibility scores, VGS measures the additional coverage each view provides, which helps avoid observing the same regions repeatedly.
3. Using explicit geometry to constrain language-based spatial reasoning is a useful way.
SAC does not entirely rely on the MLLM to identify the correct side independently in each image. Instead, it check candidate locations against estimated object-centric directions.
4. The reported gains are substantial, and the evaluation protocol tests a fixed 3D prediction across views.
The method improves average performance on all three benchmarks. Inferring a Gaussian mask once per query, and rendering that frozen prediction into all evaluation views is also a meaningful way to assess 3D segmentation.
Critical weaknesses
- The novelty needs to be more clearly distinguished from the integration of existing components.
TAO uses bounding-sphere-based camera placement, VGS uses a maximum-coverage-style objective, SAC combines a pretrained orientation estimator with coordinate-sign filtering, and the refinement stage build on prior work. This combination may be effective, but the submission needs stronger evidence to claim which new design choices are essential beyond a straightforward integration. The “agentic” framing is also not fully supported by the described fixed pipeline, which does not show feedback-driven re-observation or adaptive decision-making. The present method is a directly integrated pipeline from my viewpoint.
- SAC does not fully establish the cross-view consistency claimed in the paper.
Sec. 4.2 estimates a separate orientation for each view, and transforms it into world coordinates. Expressing predictions in the same coordinate system does not make them consistent: different views may still produce conflicting orientations or axis flips. No shared orientation estimation, cross-view fusion, or conflict-resolution mechanism is described. The method therefore applies object-centric constraints per view, but it does not explicitly resolve disagreement between the estimated frames.
- There is an unexplained discrepancy in the main ablation tables.
The default setting uses . Tab. 4 reports 80.45 mIoU for the full method (in the first line), whereas Tab. 6 reports 77.45 for VGS at (in the sixth line). Meanwhile, the Random and MAX results match exactly across the two tables. If the VGS configurations differ, those differences need to be stated clearly. Otherwise, this appears to be an error in the key experimental result tables.
- The comparisons do not sufficiently separate methodological gains from backbone and observation-budget advantages.
The submission specifies its own MLLM, SAM model, and view count, but does not clearly establish whether the strongest baselines use comparable models, image inputs, or inference budgets. Runtime, model-call counts, and cost are also missing. SAGO is discussed as a related method that acquires virtual views, but no comparison or explanation for its omission is provided.
Questions for authors
- What's the reason of the discrepancy between Tab. 4 and 6?
Do 80.45 and 77.45 correspond to different configurations or query subsets? And Which setting is used in Fig. 5?
- How are conflicting semantic orientations across views handled?
Have the authors measured agreement between the predicted matrices in world coordinates? How does per-view orientation estimation compare with estimating one shared frame, or fusing predictions across views?
- Are the strongest baselines evaluated with matched backbones and observation budgets?
Please clarify their MLLMs, segmentation models, numbers of input views, and model-call counts. How much of the improvement do the method have when these factors are controlled?
6 comments
Response to Reviewer ALfa (Part 2/4)
W3: There is an unexplained discrepancy in the main ablation tables. The default setting uses (). Tab. 4 reports 80.45 mIoU for the full method (in the first line), whereas Tab. 6 reports 77.45 for VGS at () (in the sixth line). Meanwhile, the Random and MAX results match exactly across the two tables. If the VGS configurations differ, those differences need to be stated clearly. Otherwise, this appears to be an error in the key experimental result tables.
A3: Thank you for pointing out this inconsistency. After rechecking the experimental records, we identified a reporting error involving the SAC configuration. We apologize for this oversight and clarify the discrepancy below.
(1) Different configurations in Tables 4 and 6. The VGS result of 77.45 at in Table 6 was mistakenly taken from an experiment with SAC disabled, whereas the Full result of 80.45 in Tables 1 and 4 was obtained with SAC enabled. Thus, the discrepancy comes from different SAC configurations, not different query subsets.
(2) Why Random and MAX match across Tables 4 and 6. The w/o VGS (Random) and w/o VGS (MAX) results of 74.88 and 76.67 in Table 4 were also taken from the SAC-disabled experiments reported in Table 6. Consequently, the original Table 4 compared Random and MAX without SAC against Full with SAC, rather than using the same configuration.
We have corrected Tables 4 and 6 using results obtained with SAC enabled. The corrected results are presented below.
Revised Table 4. Ablation of TAO.
| Setting | mIoU ↑ | mBIoU ↑ |
|---|---|---|
| Full | 80.45 | 79.83 |
| Fixed views + Random | 71.01 | 70.37 |
| Fixed views + VGS | 75.70 | 74.80 |
| w/o VGS (Random) | 77.28 | 76.26 |
| w/o VGS (MAX) | 78.81 | 78.15 |
Revised Table 6. Detailed results of candidate-view selection on Gran-LERF Figurines (mIoU, %).
| Random | MAX | VGS | |
|---|---|---|---|
| 2 | 66.87 | 70.15 | 77.33 |
| 4 | 73.20 | 74.44 | 79.80 |
| 6 | 75.55 | 79.21 | 80.22 |
| 8 | 76.83 | 78.84 | 81.24 |
| 10 | 79.14 | 80.11 | 80.43 |
| 12 | 77.28 | 78.81 | 80.45 |
| 16 | 80.03 | 80.08 | 81.72 |
With the corrected SAC-enabled configuration, VGS outperforms Random and MAX by 3.17 and 1.64 mIoU points at , respectively, and consistently achieves the highest mIoU across all tested values of . We have updated Tables 4 and 6 accordingly in the revised manuscript.
W4.1: The comparisons do not sufficiently separate methodological gains from backbone and observation-budget advantages. The submission specifies its own MLLM, SAM model, and view count, but does not clearly establish whether the strongest baselines use comparable models, image inputs, or inference budgets. Runtime, model-call counts, and cost are also missing.
A4.1: Thank you for raising concerns about the comparability of model backbones, image inputs, and inference budgets. We clarify the setting used by the the strongest baseline REALM and AnyGranGS below:
- MLLM: Both REALM and AnyGranGS use the same Gemini-3.6-Flash.
- SAM model: Both methods use SAM 2.1 Hiera-L.
- Image inputs: AnyGranGS uses 13 images per query (1 reference + 12 selected views), while REALM uses an average of 26 images on Gran-LERF and Ref-LERF and 23 images on Gran-3D-OVS.
- Inference budgets: We report the per-query runtime, model-call counts, and cost below.
Per-query runtime, MLLM calls, and API cost (Gemini-3.6-Flash):
| Method | Runtime (s) | MLLM calls | Input tokens | Output tokens | Est. cost (USD) |
|---|---|---|---|---|---|
| REALM | 314.08 | 26 | 37,347 | 5,583 | 0.04895 |
| AnyGranGS | 123.73 | 2 | 18,232 | 10,388 | 0.05263 |
Runtime are reported as mean. MLLM calls and token counts are per-query averages rounded to the nearest integer; estimated API costs are mean values per query.
For candidate-mask generation, AnyGranGS feeds all selected images to the MLLM in a single call, whereas REALM's released implementation processes images individually through separate calls. Consequently, AnyGranGS reduces MLLM calls from 26 to 2 and achieves a 2.54× speedup in average runtime, with a slightly higher API cost.
Under the same MLLM and SAM backbones, AnyGranGS outperforms REALM by 15.37, 26.20, and 12.58 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively, as reported in Tables 1–3 of the manuscript.
Response to Reviewer ALfa (Part 3/4)
W4.2: SAGO is discussed as a related method that acquires virtual views, but no comparison or explanation for its omission is provided.
A4.2: SAGO's code was first released on September 24, 2026, just one day before our submission deadline, leaving insufficient time to complete the experiments before submission. Therefore, SAGO was not included in our initial submission. We have now conducted comparisons using its released code on all three benchmarks. The results are as follows:
Results on Gran-LERF:
| Method | Figurines (mIoU/mBIoU) | Ramen (mIoU/mBIoU) | Teatime (mIoU/mBIoU) | Avg. (mIoU/mBIoU) |
|---|---|---|---|---|
| SAGO | 30.94 / 30.12 | 31.37 / 22.87 | 34.89 / 31.53 | 32.40 / 28.17 |
| AnyGranGS | 80.45 / 79.83 | 77.18 / 70.06 | 72.25 / 68.59 | 76.63 / 72.83 |
Results on Gran-3D-OVS:
| Method | Bed (mIoU) | Bench (mIoU) | Lawn (mIoU) | Room (mIoU) | Sofa (mIoU) | Avg. (mIoU) |
|---|---|---|---|---|---|---|
| SAGO | 66.66 | 59.27 | 56.98 | 53.58 | 59.05 | 59.11 |
| AnyGranGS | 82.15 | 84.96 | 81.76 | 85.20 | 73.40 | 81.49 |
Results on Ref-LERF:
| Method | Figurines (mIoU) | Ramen (mIoU) | Teatime (mIoU) | Avg. (mIoU) |
|---|---|---|---|---|
| SAGO | 9.79 | 13.05 | 26.68 | 16.51 |
| AnyGranGS | 58.10 | 78.91 | 85.05 | 74.02 |
Our method outperforms SAGO by 44.23, 22.38, and 57.51 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively. We have incorporated these results into Tables 1–3 of the revised manuscript.
Response to Questions for Authors:
Q1: What's the reason of the discrepancy between Tab. 4 and 6? Do 80.45 and 77.45 correspond to different configurations or query subsets? And Which setting is used in Fig. 5?
A1: Please refer to our response to W3. Figure 5 originally used the SAC-disabled setting in Table 6. We have corrected Tables 4 and 6 and Figure 5 using results obtained with SAC enabled.
Q2: How are conflicting semantic orientations across views handled? Have the authors measured agreement between the predicted () matrices in world coordinates? How does per-view orientation estimation compare with estimating one shared frame, or fusing predictions across views?
A2: Thank you for these questions. We address them below.
(1) Handling conflicting semantic orientations across views. Please refer to our response to W2.
(2) Agreement between predicted orientations in world coordinates. We measure the agreement between the predicted matrices after transforming them into world coordinates, using 12 candidate views per query. The angular dispersion statistics are reported below:
Agreement between predicted orientations in world coordinates:
| Scene | Mean | |||
|---|---|---|---|---|
| Figurines | 39.9° | 0.0% (0/396) | 1.0% (4/396) | 1.0% (4/396) |
| Teatime | 54.4° | 17.6% (93/528) | 0.6% (3/528) | 17.8% (94/528) |
denotes the relative rotation angle between world-frame predictions. The last three columns report the percentages of view pairs with semantic X-axis (right), Y-axis (down), and Z-axis (front) disagreements exceeding .
The results show that predicted orientations can remain inconsistent across candidate views even after transformation into world coordinates**.
(3) Comparison of per-view orientation estimation, one single shared frame, and cross-view fusion. We compare the following settings on spatially conditioned part queries in Gran-LERF using the same candidate masks:
- w/o SAC: Removes Semantic Axis Constraints.
- Per-view estimation: Uses independently predicted orientations for each view.
- Single shared frame: Estimates one orientation from the reference image and applies it across candidate views.
- Cross-view fusion (Ours): Filters inconsistent orientation estimates and averages the remaining estimates to obtain a shared orientation, as described in our response to W2.
Comparison of per-view orientation estimation, a single shared frame, and cross-view fusion on spatially conditioned part queries in Gran-LERF (mIoU) :
| w/o SAC | Per-view estimation | Single shared frame | Cross-view fusion (Ours) |
|---|---|---|---|
| 62.6 | 62.5 | 61.3 | 70.8 |
Our cross-view fusion achieves the highest overall mIoU of 70.8%, outperforming per-view orientation estimation (62.5%) and a single shared frame (61.3%) by 8.3 and 9.5 points, respectively. These results demonstrate the effectiveness of fusing orientation predictions across views for spatially conditioned part segmentation.
Response to Reviewer ALfa (Part 1/4)
Response to 'Critical Weaknesses':
A1.1: Thank you for raising concerns about the novelty. Our contribution lies in the specific design of target-centric observation and semantic axis constraints, rather than the individual techniques. We clarify our key design choices and provide ablation evidence below.
(1) Key design. Specifically, to address inflexible view acquisition, we propose TAO to construct target-centric virtual views and VGS to select complementary views based on marginal Gaussian visibility gain. To address cross-view spatial ambiguity, we propose SAC to apply object-centric semantic axis constraints for spatially conditioned part segmentation. These designs are all essential to addressing these two challenges.
(2) Ablation evidence. The ablation results demonstrate the contributions of each module: TAO (80.45 vs. 75.70 mIoU with fixed views), VGS (80.45 vs. 77.28 without VGS), and SAC (75.9 vs. 71.0 without SAC). These results support the importance of our proposed design choices beyond a straightforward integration of existing techniques.
A1.2: Thank you for your insightful comments.
Following REALM [1], which claims to be MLLM-Agent framework, we name our method as an agentic framework.
Our framework actually contains adaptive decision-making. The MLLM interprets the query granularity and spatial relations to decide whether to ground the parent object or the queried target. It then uses target-centric observations from TAO/VGS to generate localization prompts for segmentation. These query-dependent decisions go beyond a purely fixed pipeline.
Prior research characterizes multimodal agentic frameworks through perception, reasoning, planning, memory, and action [2]. From this perspective, our method could be framed as "Agentic".
We acknowledge that feedback-driven re-observation is not included in the current framework and consider it a promising direction for future work.
References
[1] Shi, Changyue, et al. "REALM: an MLLM-agent framework for open world 3D reasoning segmentation and editing on gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
[2] Mokaria, Neel, et al. "A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications." Transactions on Machine Learning Research (TMLR), 2026.
A2: Thank you for pointing out this issue. We actually already account for cross-view orientation inconsistencies and handle through fusion of orientation estimates across views, but the corresponding mechanism was not clearly described in the original manuscript. We clarify it below.
Handling conflicting semantic orientations across views. To resolve disagreements among per-view axes, we first transform the orientation estimates into world coordinates, identify the rotation medoid, exclude estimates with large angular disagreements, and compute the chordal mean of the remaining estimates to obtain a shared orientation across candidate views. We then use this shared orientation for spatial semantic filtering.
This mechanism reduces disagreements between the estimated frames and establishes cross-view consistency, rather than merely expressing per-view predictions in the same coordinate system. We have supplemented the relevant details in Sec. 4.2 and the Appendix of the revised manuscript.