acceptodds
Under review as a conference paper at ICLR 2027

AnyGranGS: Agentic 3DGS Segmentation with Versatile Granularity

Abstract

3D Gaussian Splatting (3DGS) enables language-guided 3D scene understanding, while practical queries may target varying semantic granularities. Recent agentic methods leverage MLLMs for target reasoning and refinement, yet versatile-granularity segmentation remains challenged by (i) inflexible view acquisition, where predefined views may insufficiently observe fine-grained targets, and (ii) cross-view spatial ambiguity, where object-centric spatial relations can be inconsistently interpreted across viewpoints. We propose **AnyGranGS**, an agentic framework addressing both challenges. First, **Target-centric Adaptive Observation (TAO)** locates a coarse target proxy and constructs target-specific virtual views, while **Visibility Gain-based Selection (VGS)** selects complementary candidate views based on Gaussian visibility gain. Second, **Semantic Axis Constraints (SAC)** establish object-centric semantic axes for consistent part localization across views. Multi-view masks are then aggregated through visibility-weighted Gaussian evidence for target refinement. We further construct **Gran-LERF** and **Gran-3D-OVS** for versatile-granularity evaluation. Experiments on Gran-LERF, Gran-3D-OVS, and Ref-LERF demonstrate the effectiveness of AnyGranGS across diverse query granularities, improving the average mIoU over the strongest baselines by **15.37, 21.61, and 12.58 points**, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic Segmentation, Open-vocabulary Segmentation, Referring Segmentation, Agentic, 3DGS

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ALfaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission presents AnyGranGS, which is a framework for language-guided 3D Gaussian Splatting segmentation across objects, parts, part sets, and conditioned parts. The method firstly obtains an initial mask using an MLLM and SAM, and then associates it with a coarse target proxy in a semantic Gaussian field. It constructs target-centered virtual views, and select complementary observations using marginal visibility gain. For spatially conditioned queries, it estimates object-centric semantic axes, and filters candidate parts according to their 3D positions. Finally, it aggregates visibility-weighted foreground and background evidence to refine Gaussian membership. This work also introduce Gran-LERF and Gran-3D-OVS, and reports substantial improvements on these two benchmarks and Ref-LERF.

Critical strengths

1. The submission addresses two concrete limitations with well-motivated designs.

Intuitively, fine-grained parts often require closer and less occluded observations, while object-relative directions cannot reliably be interpreted as image-relative directions. The distinction between these problems are clear.

2. The view-selection objective accounts for complementary observations.

Rather than selecting views solely by their individual visibility scores, VGS measures the additional coverage each view provides, which helps avoid observing the same regions repeatedly.

3. Using explicit geometry to constrain language-based spatial reasoning is a useful way.

SAC does not entirely rely on the MLLM to identify the correct side independently in each image. Instead, it check candidate locations against estimated object-centric directions.

4. The reported gains are substantial, and the evaluation protocol tests a fixed 3D prediction across views.

The method improves average performance on all three benchmarks. Inferring a Gaussian mask once per query, and rendering that frozen prediction into all evaluation views is also a meaningful way to assess 3D segmentation.

Critical weaknesses
  1. The novelty needs to be more clearly distinguished from the integration of existing components.

TAO uses bounding-sphere-based camera placement, VGS uses a maximum-coverage-style objective, SAC combines a pretrained orientation estimator with coordinate-sign filtering, and the refinement stage build on prior work. This combination may be effective, but the submission needs stronger evidence to claim which new design choices are essential beyond a straightforward integration. The “agentic” framing is also not fully supported by the described fixed pipeline, which does not show feedback-driven re-observation or adaptive decision-making. The present method is a directly integrated pipeline from my viewpoint.

  1. SAC does not fully establish the cross-view consistency claimed in the paper.

Sec. 4.2 estimates a separate orientation �0��0� for each view, and transforms it into world coordinates. Expressing predictions in the same coordinate system does not make them consistent: different views may still produce conflicting orientations or axis flips. No shared orientation estimation, cross-view fusion, or conflict-resolution mechanism is described. The method therefore applies object-centric constraints per view, but it does not explicitly resolve disagreement between the estimated frames.

  1. There is an unexplained discrepancy in the main ablation tables.

The default setting uses �1��1�. Tab. 4 reports 80.45 mIoU for the full method (in the first line), whereas Tab. 6 reports 77.45 for VGS at �2��2� (in the sixth line). Meanwhile, the Random and MAX results match exactly across the two tables. If the VGS configurations differ, those differences need to be stated clearly. Otherwise, this appears to be an error in the key experimental result tables.

  1. The comparisons do not sufficiently separate methodological gains from backbone and observation-budget advantages.

The submission specifies its own MLLM, SAM model, and view count, but does not clearly establish whether the strongest baselines use comparable models, image inputs, or inference budgets. Runtime, model-call counts, and cost are also missing. SAGO is discussed as a related method that acquires virtual views, but no comparison or explanation for its omission is provided.

Questions for authors
  1. What's the reason of the discrepancy between Tab. 4 and 6?

Do 80.45 and 77.45 correspond to different configurations or query subsets? And Which setting is used in Fig. 5?

  1. How are conflicting semantic orientations across views handled?

Have the authors measured agreement between the predicted �0��0� matrices in world coordinates? How does per-view orientation estimation compare with estimating one shared frame, or fusing predictions across views?

  1. Are the strongest baselines evaluated with matched backbones and observation budgets?

Please clarify their MLLMs, segmentation models, numbers of input views, and model-call counts. How much of the improvement do the method have when these factors are controlled?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ALfa (Part 1/4)

Response to 'Critical Weaknesses':

W1.1: The novelty needs to be more clearly distinguished from the integration of existing components. TAO uses bounding-sphere-based camera placement, VGS uses a maximum-coverage-style objective, SAC combines a pretrained orientation estimator with coordinate-sign filtering, and the refinement stage build on prior work. This combination may be effective, but the submission needs stronger evidence to claim which new design choices are essential beyond a straightforward integration.

A1.1: Thank you for raising concerns about the novelty. Our contribution lies in the specific design of target-centric observation and semantic axis constraints, rather than the individual techniques. We clarify our key design choices and provide ablation evidence below.

(1) Key design. Specifically, to address inflexible view acquisition, we propose TAO to construct target-centric virtual views and VGS to select complementary views based on marginal Gaussian visibility gain. To address cross-view spatial ambiguity, we propose SAC to apply object-centric semantic axis constraints for spatially conditioned part segmentation. These designs are all essential to addressing these two challenges.

(2) Ablation evidence. The ablation results demonstrate the contributions of each module: TAO (80.45 vs. 75.70 mIoU with fixed views), VGS (80.45 vs. 77.28 without VGS), and SAC (75.9 vs. 71.0 without SAC). These results support the importance of our proposed design choices beyond a straightforward integration of existing techniques.

 

W1.2: The “agentic” framing is also not fully supported by the described fixed pipeline, which does not show feedback-driven re-observation or adaptive decision-making. The present method is a directly integrated pipeline from my viewpoint.

A1.2: Thank you for your insightful comments.

  • Following REALM [1], which claims to be MLLM-Agent framework, we name our method as an agentic framework.

  • Our framework actually contains adaptive decision-making. The MLLM interprets the query granularity and spatial relations to decide whether to ground the parent object or the queried target. It then uses target-centric observations from TAO/VGS to generate localization prompts for segmentation. These query-dependent decisions go beyond a purely fixed pipeline.

  • Prior research characterizes multimodal agentic frameworks through perception, reasoning, planning, memory, and action [2]. From this perspective, our method could be framed as "Agentic".

We acknowledge that feedback-driven re-observation is not included in the current framework and consider it a promising direction for future work.

References

[1] Shi, Changyue, et al. "REALM: an MLLM-agent framework for open world 3D reasoning segmentation and editing on gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Mokaria, Neel, et al. "A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications." Transactions on Machine Learning Research (TMLR), 2026.

 

W2: SAC does not fully establish the cross-view consistency claimed in the paper. Sec. 4.2 estimates a separate orientation (RvR_v) for each view, and transforms it into world coordinates. Expressing predictions in the same coordinate system does not make them consistent: different views may still produce conflicting orientations or axis flips. No shared orientation estimation, cross-view fusion, or conflict-resolution mechanism is described. The method therefore applies object-centric constraints per view, but it does not explicitly resolve disagreement between the estimated frames.

A2: Thank you for pointing out this issue. We actually already account for cross-view orientation inconsistencies and handle through fusion of orientation estimates across views, but the corresponding mechanism was not clearly described in the original manuscript. We clarify it below.

Handling conflicting semantic orientations across views. To resolve disagreements among per-view axes, we first transform the orientation estimates into world coordinates, identify the rotation medoid, exclude estimates with large angular disagreements, and compute the chordal mean of the remaining estimates to obtain a shared orientation across candidate views. We then use this shared orientation for spatial semantic filtering.

This mechanism reduces disagreements between the estimated frames and establishes cross-view consistency, rather than merely expressing per-view predictions in the same coordinate system. We have supplemented the relevant details in Sec. 4.2 and the Appendix of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ALfa (Part 2/4)

W3: There is an unexplained discrepancy in the main ablation tables. The default setting uses (Kc=12K_c = 12). Tab. 4 reports 80.45 mIoU for the full method (in the first line), whereas Tab. 6 reports 77.45 for VGS at (Kc=12K_c = 12) (in the sixth line). Meanwhile, the Random and MAX results match exactly across the two tables. If the VGS configurations differ, those differences need to be stated clearly. Otherwise, this appears to be an error in the key experimental result tables.

A3: Thank you for pointing out this inconsistency. After rechecking the experimental records, we identified a reporting error involving the SAC configuration. We apologize for this oversight and clarify the discrepancy below.

(1) Different configurations in Tables 4 and 6. The VGS result of 77.45 at Kc=12K_c=12 in Table 6 was mistakenly taken from an experiment with SAC disabled, whereas the Full result of 80.45 in Tables 1 and 4 was obtained with SAC enabled. Thus, the discrepancy comes from different SAC configurations, not different query subsets.

(2) Why Random and MAX match across Tables 4 and 6. The w/o VGS (Random) and w/o VGS (MAX) results of 74.88 and 76.67 in Table 4 were also taken from the SAC-disabled experiments reported in Table 6. Consequently, the original Table 4 compared Random and MAX without SAC against Full with SAC, rather than using the same configuration.

We have corrected Tables 4 and 6 using results obtained with SAC enabled. The corrected results are presented below.

Revised Table 4. Ablation of TAO.

SettingmIoU ↑mBIoU ↑
Full80.4579.83
Fixed views + Random71.0170.37
Fixed views + VGS75.7074.80
w/o VGS (Random)77.2876.26
w/o VGS (MAX)78.8178.15

Revised Table 6. Detailed results of candidate-view selection on Gran-LERF Figurines (mIoU, %).

KcK_cRandomMAXVGS
266.8770.1577.33
473.2074.4479.80
675.5579.2180.22
876.8378.8481.24
1079.1480.1180.43
1277.2878.8180.45
1680.0380.0881.72

With the corrected SAC-enabled configuration, VGS outperforms Random and MAX by 3.17 and 1.64 mIoU points at Kc=12K_c=12, respectively, and consistently achieves the highest mIoU across all tested values of KcK_c. We have updated Tables 4 and 6 accordingly in the revised manuscript.

W4.1: The comparisons do not sufficiently separate methodological gains from backbone and observation-budget advantages. The submission specifies its own MLLM, SAM model, and view count, but does not clearly establish whether the strongest baselines use comparable models, image inputs, or inference budgets. Runtime, model-call counts, and cost are also missing.

 

A4.1: Thank you for raising concerns about the comparability of model backbones, image inputs, and inference budgets. We clarify the setting used by the the strongest baseline REALM and AnyGranGS below:

  • MLLM: Both REALM and AnyGranGS use the same Gemini-3.6-Flash.
  • SAM model: Both methods use SAM 2.1 Hiera-L.
  • Image inputs: AnyGranGS uses 13 images per query (1 reference + 12 selected views), while REALM uses an average of 26 images on Gran-LERF and Ref-LERF and 23 images on Gran-3D-OVS.
  • Inference budgets: We report the per-query runtime, model-call counts, and cost below.

Per-query runtime, MLLM calls, and API cost (Gemini-3.6-Flash):

MethodRuntime (s)MLLM callsInput tokensOutput tokensEst. cost (USD)
REALM314.082637,3475,5830.04895
AnyGranGS123.73218,23210,3880.05263

Runtime are reported as mean. MLLM calls and token counts are per-query averages rounded to the nearest integer; estimated API costs are mean values per query.

For candidate-mask generation, AnyGranGS feeds all selected images to the MLLM in a single call, whereas REALM's released implementation processes images individually through separate calls. Consequently, AnyGranGS reduces MLLM calls from 26 to 2 and achieves a 2.54× speedup in average runtime, with a slightly higher API cost.

Under the same MLLM and SAM backbones, AnyGranGS outperforms REALM by 15.37, 26.20, and 12.58 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively, as reported in Tables 1–3 of the manuscript.

AuthorsOpenReview ↗

Response to Reviewer ALfa (Part 3/4)

W4.2: SAGO is discussed as a related method that acquires virtual views, but no comparison or explanation for its omission is provided.

A4.2: SAGO's code was first released on September 24, 2026, just one day before our submission deadline, leaving insufficient time to complete the experiments before submission. Therefore, SAGO was not included in our initial submission. We have now conducted comparisons using its released code on all three benchmarks. The results are as follows:

Results on Gran-LERF:

MethodFigurines (mIoU/mBIoU)Ramen (mIoU/mBIoU)Teatime (mIoU/mBIoU)Avg. (mIoU/mBIoU)
SAGO30.94 / 30.1231.37 / 22.8734.89 / 31.5332.40 / 28.17
AnyGranGS80.45 / 79.8377.18 / 70.0672.25 / 68.5976.63 / 72.83

Results on Gran-3D-OVS:

MethodBed (mIoU)Bench (mIoU)Lawn (mIoU)Room (mIoU)Sofa (mIoU)Avg. (mIoU)
SAGO66.6659.2756.9853.5859.0559.11
AnyGranGS82.1584.9681.7685.2073.4081.49

Results on Ref-LERF:

MethodFigurines (mIoU)Ramen (mIoU)Teatime (mIoU)Avg. (mIoU)
SAGO9.7913.0526.6816.51
AnyGranGS58.1078.9185.0574.02

Our method outperforms SAGO by 44.23, 22.38, and 57.51 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively. We have incorporated these results into Tables 1–3 of the revised manuscript.

 

Response to Questions for Authors:

Q1: What's the reason of the discrepancy between Tab. 4 and 6? Do 80.45 and 77.45 correspond to different configurations or query subsets? And Which setting is used in Fig. 5?

A1: Please refer to our response to W3. Figure 5 originally used the SAC-disabled setting in Table 6. We have corrected Tables 4 and 6 and Figure 5 using results obtained with SAC enabled.

 

Q2: How are conflicting semantic orientations across views handled? Have the authors measured agreement between the predicted (RvR_v) matrices in world coordinates? How does per-view orientation estimation compare with estimating one shared frame, or fusing predictions across views?

A2: Thank you for these questions. We address them below.

(1) Handling conflicting semantic orientations across views. Please refer to our response to W2.

(2) Agreement between predicted orientations in world coordinates. We measure the agreement between the predicted RvR_v matrices after transforming them into world coordinates, using 12 candidate views per query. The angular dispersion statistics are reported below:

Agreement between predicted orientations in world coordinates:

SceneMean Δθrot\Delta\theta_{\mathrm{rot}}ΔθX>90∘\Delta\theta_X>90^\circΔθY>90∘\Delta\theta_Y>90^\circΔθZ>90∘\Delta\theta_Z>90^\circ
Figurines39.9°0.0% (0/396)1.0% (4/396)1.0% (4/396)
Teatime54.4°17.6% (93/528)0.6% (3/528)17.8% (94/528)

Δθrot\Delta\theta_{\mathrm{rot}} denotes the relative rotation angle between world-frame predictions. The last three columns report the percentages of view pairs with semantic X-axis (right), Y-axis (down), and Z-axis (front) disagreements exceeding 90∘90^\circ.

The results show that predicted orientations can remain inconsistent across candidate views even after transformation into world coordinates**.

(3) Comparison of per-view orientation estimation, one single shared frame, and cross-view fusion. We compare the following settings on spatially conditioned part queries in Gran-LERF using the same candidate masks:

  • w/o SAC: Removes Semantic Axis Constraints.
  • Per-view estimation: Uses independently predicted orientations for each view.
  • Single shared frame: Estimates one orientation from the reference image and applies it across candidate views.
  • Cross-view fusion (Ours): Filters inconsistent orientation estimates and averages the remaining estimates to obtain a shared orientation, as described in our response to W2.

Comparison of per-view orientation estimation, a single shared frame, and cross-view fusion on spatially conditioned part queries in Gran-LERF (mIoU) :

w/o SACPer-view estimationSingle shared frameCross-view fusion (Ours)
62.662.561.370.8

Our cross-view fusion achieves the highest overall mIoU of 70.8%, outperforming per-view orientation estimation (62.5%) and a single shared frame (61.3%) by 8.3 and 9.5 points, respectively. These results demonstrate the effectiveness of fusing orientation predictions across views for spatially conditioned part segmentation.

Reviewer_vAW3OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies text-driven 3DGS segmentation across varying semantic granularities and proposes AnyGranGS. It improves fine-grained observation selection via object-centric virtual views and visibility gain, handles cross-view spatial relationships with semantic-axis constraints, and refines targets using multi-view Gaussian evidence. The authors also build Gran-LERF and Gran-3D-OVS, reporting notable mIoU gains on three benchmarks.

Critical strengths
  1. The object-adaptive observation design brings substantial value. Instead of being limited to original training cameras, TAO builds object-centric views based on coarse proxy scales, while VGS selects complementary views using marginal visibility coverage in Eq. (6)–(7). The consistent gains of VGS over Random/MAX in Table 4 and Table 6 confirm this design is not merely adding more views.

  2. The overall performance is strong and each component has standalone empirical support. AnyGranGS achieves mIoU improvements of 15.37, 21.61 and 12.58 points on Gran-LERF, Gran-3D-OVS and Ref-LERF respectively. SAC consistently boosts performance across four MLLMs, and removing multi-view refinement causes obvious accuracy drops. Therefore, the paper’s strength does not rely on one single module.

Critical weaknesses
  1. The MLLM setup for the agentic baseline lacks controlled comparison, leading to ambiguous attribution for the main performance gains. (Missing information) Section 5.1 states AnyGranGS uses Gemini-3.6-Flash, yet Table 3 only labels REALM as "our reproduced" without specifying the MLLM, prompt or decoding settings, or whether they match this work. The original REALM uses Qwen2.5-VL and selects local views from clustered cameras. More importantly, Table 5 shows switching only the MLLM changes mIoU after SAC from 41.0 to 75.9, meaning the backbone strongly affects this agentic pipeline. Thus the advantages in Tables 1–3 cannot be fully attributed to TAO/VGS/SAC. The authors should fully document the MLLM version, prompt, temperature/decoding, SAM and view budget used for reproduced REALM. If they differ from AnyGranGS, run a REALM control with the same Gemini-3.6-Flash, segmentation backbone and comparable view budget on at least one representative benchmark, reporting mIoU/mBIoU to separate framework gains from MLLM gains.

  2. The core claim of "versatile granularity" lacks direct granularity-wise breakdown evidence. (Insufficient evidence) Section 3 defines four granularity levels: object, part, part set and spatially conditioned part. But Tables 1–2 only report scene-averaged results without sample counts and performance for each granularity. It remains unclear whether overall improvements hold across all four query types or mainly come from easier categories. In particular, the SAC ablation only covers 6 spatial queries and 19 masks in Figurines, a scope far narrower than the paper’s claim of cross-scene, cross-granularity effectiveness. The authors should report sample counts, mIoU and mBIoU grouped by object / part / part-set / spatially-conditioned part using existing predictions. Also report aggregated results with and without SAC for the spatial subset. This analysis requires no retraining and directly validates the central claim of arbitrary granularity.

Questions for authors

1.Which MLLM version, prompt/decoding settings, SAM version and view count were used for the reproduced REALM‡? Do you have REALM and AnyGranGS results under the same Gemini-3.6-Flash backbone?

2.Can you report the sample count, mIoU and mBIoU for each of the four query granularities from existing predictions? Does this breakdown still show AnyGranGS outperforms the strongest baseline on every granularity?

3.For SAC, how consistent are the semantic axes from Orient Anything across different candidate views after transforming them into the world frame via camera poses? Do you have angular dispersion statistics of axis directions, failure cases, or paired per-spatial-query results with and without SAC? This information shapes my assessment of SAC’s cross-view consistency claim.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vAW3 (Part 1/2)

W1: The MLLM setup for the agentic baseline lacks controlled comparison, leading to ambiguous attribution for the main performance gains.

The authors should fully document the MLLM version, prompt, temperature/decoding, SAM and view budget used for reproduced REALM. If they differ from AnyGranGS, run a REALM control with the same Gemini-3.6-Flash, segmentation backbone and comparable view budget on at least one representative benchmark, reporting mIoU/mBIoU to separate framework gains from MLLM gains.

A1: Thank you for pointing out the missing information regarding the controlled comparison with REALM. We clarify the MLLM version, prompt, temperature/decoding settings, SAM model, and view budget used by REALM and AnyGranGS below:

  • MLLM version: Both REALM and AnyGranGS use the same Gemini-3.6-Flash.
  • Prompt: REALM uses its officially released prompts, while AnyGranGS uses our task-specific prompts.
  • Temperature/decoding: Both methods use a decoding temperature of 0.
  • SAM: Both methods use SAM 2.1 Hiera-L.
  • View budget: AnyGranGS uses 13 input images per query (1 reference + 12 selected views), while REALM uses an average of 26 images on Gran-LERF and Ref-LERF, and 23 images on Gran-3D-OVS.

Under the same MLLM and SAM backbones, AnyGranGS outperforms REALM by 15.37, 26.20, and 12.58 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively, as reported in Tables 1–3 of the manuscript.

 

W2.1: The core claim of "versatile granularity" lacks direct granularity-wise breakdown evidence. (Insufficient evidence) Section 3 defines four granularity levels: object, part, part set and spatially conditioned part. But Tables 1–2 only report scene-averaged results without sample counts and performance for each granularity. It remains unclear whether overall improvements hold across all four query types or mainly come from easier categories.

A2.1: Thank you for pointing out the lack of granularity-wise breakdown evidence. To directly verify our versatile-granularity claim, we report the sample counts and performance for each granularity. The results are shown below:

  • Per-granularity Sample Counts.

    Sample Counts of Gran-LERF and Gran-3D-OVS across different granularities:

    BenchmarkObjectPartPart SetSpatially Conditioned PartOverall
    Gran-LERF303581487
    Gran-3D-OVS23157651
  • Per-granularity Performance.

    Per-granularity Performance on Gran-LERF:

    MethodObject (mIoU/mBIoU)Part (mIoU/mBIoU)Part Set (mIoU/mBIoU)Spatially Conditioned Part (mIoU/mBIoU)
    SAGA59.85 / 55.3822.85 / 21.1412.85 / 11.128.83 / 7.43
    LaGa63.39 / 60.2831.68 / 29.1323.96 / 21.486.56 / 7.20
    Dr. Splat52.63 / 48.7123.36 / 20.7818.13 / 15.298.25 / 8.27
    SAGO48.14 / 44.3024.86 / 21.9320.71 / 16.3323.74 / 23.74
    ReLaGS65.33 / 62.2334.09 / 31.0625.52 / 21.7914.74 / 15.83
    REALM78.23 / 75.8959.71 / 57.0760.70 / 54.5828.67 / 28.48
    AnyGranGS85.81 / 82.4971.36 / 68.6381.65 / 78.1170.76 / 70.06

    Per-granularity Performance on Gran-3D-OVS:

    MethodObject (mIoU/mBIoU)Part (mIoU/mBIoU)Part Set (mIoU/mBIoU)Spatially Conditioned Part (mIoU/mBIoU)
    SAGA60.64 / 50.2436.65 / 32.1934.88 / 28.9819.72 / 12.62
    LaGa82.28 / 66.5542.08 / 36.3127.69 / 23.5725.20 / 20.51
    Dr. Splat54.84 / 47.3830.06 / 26.9429.87 / 24.7222.09 / 19.31
    SAGO79.42 / 69.6646.46 / 42.1141.47 / 35.7727.85 / 24.67
    ReLaGS79.18 / 74.0150.13 / 48.4237.95 / 32.9025.67 / 23.12
    REALM88.03 / 80.7629.98 / 27.4825.11 / 23.8816.42 / 15.55
    AnyGranGS86.87 / 77.3282.81 / 77.7773.38 / 65.3967.18 / 64.02

Overall, AnyGranGS achieves the best mIoU and mBIoU across all four granularities on Gran-LERF. On Gran-3D-OVS, AnyGranGS is slightly below REALM only at the object level (by 1.16 mIoU points), ranking second, while strongly outperforming all baselines on the other three granularities. These results show that our overall improvements hold across all four query types.

AuthorsOpenReview ↗

Response to Reviewer vAW3 (Part 2/2)

W2.2: In particular, the SAC ablation only covers 6 spatial queries and 19 masks in Figurines, a scope far narrower than the paper’s claim of cross-scene, cross-granularity effectiveness. The authors should report sample counts, mIoU and mBIoU grouped by object / part / part-set / spatially-conditioned part using existing predictions. Also report aggregated results with and without SAC for the spatial subset.

A2.2: Thank you for pointing out the narrow scope of the SAC ablation. We extended the SAC ablation to all 14 spatially conditioned part queries in Figurines (6 queries) and Teatime (8 queries). Note that Ramen is excluded because it primarily consists of round objects, making it unsuitable for spatial query annotation. The aggregated results with and without SAC for the spatial subset are presented below:

SAC ablation on spatially conditioned part queries in Gran-LERF:

SettingFigurines (mIoU)Teatime (mIoU)Overall (mIoU)
w/o SAC71.056.462.6
w/ SAC75.966.970.8

 

Response to 'Questions for Authors':

Q1: Which MLLM version, prompt/decoding settings, SAM version and view count were used for the reproduced REALM‡? Do you have REALM and AnyGranGS results under the same Gemini-3.6-Flash backbone?

A1: Please refer to our response to W1.

 

Q2: Can you report the sample count, mIoU and mBIoU for each of the four query granularities from existing predictions? Does this breakdown still show AnyGranGS outperforms the strongest baseline on every granularity?

A2: Please refer to our response to W2.1.

 

Q3: For SAC, how consistent are the semantic axes from Orient Anything across different candidate views after transforming them into the world frame via camera poses? Do you have angular dispersion statistics of axis directions, failure cases, or paired per-spatial-query results with and without SAC?

A3: Thank you for raising this question. To address your concerns about how consistent the semantic axes are from Orient Anything across different candidate views, we will provide angular dispersion statistics and report the paired per-spatial-query results with and without SAC.

(1) Angular dispersion statistics of axis directions. Following your suggestions, We report the angular dispersion of semantic axes from Orient Anything across 12 candidate views per query after transforming them into the world frame, the results are as follows:

Angular dispersion statistics of axis directions:

SceneMean Δθrot\Delta\theta_{\mathrm{rot}}ΔθX>90∘\Delta\theta_X>90^\circΔθY>90∘\Delta\theta_Y>90^\circΔθZ>90∘\Delta\theta_Z>90^\circ
Figurines39.9°0.0% (0/396)1.0% (4/396)1.0% (4/396)
Teatime54.4°17.6% (93/528)0.6% (3/528)17.8% (94/528)

Δθrot\Delta\theta_{\mathrm{rot}} denotes the relative rotation angle between world-frame predictions. The last three columns report the percentages of view pairs with semantic X-axis (right), Y-axis (down), and Z-axis (front) disagreements exceeding 90∘90^\circ.

The results indicate that inconsistencies exist among the semantic axes from Orient Anything across different candidate views.

(2) Paired per-spatial-query results with and without SAC. The aggregated results on all 14 spatially conditioned queries are presented in our response to W2.2. We have additionally added paired per-query quantitative results and visualizations comparing the results with and without SAC in the Appendix of the revised manuscript.

Dear Authors, thank you for the detailed responses and newly added experiments, including the per-granularity performance breakdown, semantic axis angular dispersion statistics, and expanded SAC ablation on spatially conditioned queries. These results substantially enrich the empirical evaluation of AnyGranGS.

Nevertheless, two important concerns remain unresolved. First, while the angular dispersion statistics demonstrate the inconsistency of raw semantic axes across views, the paper does not quantify how much SAC actively mitigates such axis disagreement. The existing aggregate mIoU comparison cannot fully validate the core claim that SAC enforces stable cross-view semantic axes. Second, AnyGranGS and REALM share identical MLLM and SAM backbones but operate under different view budgets. Even with the current results, we cannot isolate whether the large performance advantage stems from the proposed framework itself or simply the difference in the number of input views.

Given these remaining ambiguities on the core mechanism validation and fair comparison with REALM, I lower my score to 2. I also suggest that for future submissions, preparing and submitting the rebuttal earlier in the discussion window would allow adequate time for reviewer examination and follow-up clarifications.

AuthorsOpenReview ↗

Response to Reviewer vAW3

SAC Mechanism. We would like to respectifully clarify that SAC fuses semantic axes from different views into a shared object-centric coordinate frame, eliminating cross-view axis disagreement by construction (see our response A2 to Reviewer ALfa). Furthermore, Table 5 shows substantial performance improvements over the setting without SAC, validating its effectiveness.

Input Views. First, we would like to reiterate that AnyGranGS uses fewer input views than REALM under our evaluated settings. Thus, our improvements are not obtained simply by increasing the number of observations.

Second, the two methods employ different view-acquisition procedures. AnyGranGS constructs target-centric virtual views and selects a fixed number through VGS. In contrast, REALM follows a global-to-local strategy, where local views are retrieved from clustered training cameras based on the target instance. Consequently, REALM's input view count varies across targets, making it difficult to enforce an identical view budget without modifying REALM's original selection procedure.

We fully agree with you that an equal-view comparison would provide a more direct assessment of view-selection effectiveness. Nevertheless, the current comparison demonstrates that AnyGranGS achieves its improvements without requiring a larger number of input views.

About Late Rebuttal. Thank you for this suggestion. We sincerely apologize for submitting our rebuttal so late in the discussion period. The additional experiments and careful verification of the results took longer than anticipated, and we wanted to provide clear and reliable responses to the reviewers' concerns. We recognize that this left limited time for further discussion and will make every effort to submit our responses earlier in future review cycles.

Reviewer_AewfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies language-guided segmentation in a pretrained 3DGS scene where the query can target an object, a part, a set of parts, or a spatially conditioned part (e.g., "the front-left leg of Jake the Dog"). The method is a training-free pipeline. An MLLM and SAM 2 give a coarse mask in a reference view, which is lifted to a Gaussian cluster from a SAGA affinity field. Target-centric Adaptive Observation (TAO) places 48 virtual cameras on a Fibonacci sphere around this proxy. Visibility Gain-based Selection (VGS) then greedily picks 12 of them by marginal per-Gaussian visibility coverage. For spatial part queries, Semantic Axis Constraints (SAC) use Orient Anything in each view to get object axes and keep only part masks whose 3D centers lie on the required side. Multi-view masks are finally aggregated into per-Gaussian foreground/background evidence to refine the selection. The authors build two benchmarks (Gran-LERF: 87 queries / 252 masks; Gran-3D-OVS: 51 categories / 254 masks) and report average mIoU gains of 15.4, 21.6 and 12.6 points over the strongest baselines on Gran-LERF, Gran-3D-OVS and Ref-LERF.

Critical strengths
  1. The problem is relevant and well posed. Spatially conditioned part queries are a realistic case that current 3DGS language-field methods handle poorly. Fig. 1 identifies two concrete failure modes: fixed views under-observe small parts, and view-dependent spatial words are inconsistent across views. The pipeline addresses both without per-scene referring annotations or retraining.
  2. The reported gains are large and consistent across three benchmarks. The TAO ablation (Table 4, Fig. 5) separates the effect of the virtual-view pool from that of the selection rule. If the numbers hold up, this is a practical improvement over REALM-style agentic segmentation.
Critical weaknesses
  1. The evidence behind the ablations is thin and partly inconsistent. Tables 4 and 6 share the setting Kc = 12 on Figurines. Random (74.88) and MAX (76.67) match across the two tables, but VGS at Kc = 12 is 77.45 in Table 6 while "Full" is 80.45 in Table 4 and Table 1. The 3-point gap is unexplained and is about the size of the VGS benefit being claimed. All ablations use a single scene, and the SAC ablation has only 6 queries / 19 masks. No run-to-run variance is reported, although the pipeline depends on a sampled closed-source MLLM. As a result, I cannot tell how much of the improvement comes from TAO/VGS/SAC rather than from noise or from the stronger MLLM.
  2. The baselines and benchmarks do not yet support the main claim. It is unclear which MLLM the REALM reproduction used. If it was not Gemini-3.6-Flash, part of the gain may come from the model. The reproduced ReLaGS on Ref-LERF (5.72 / 1.79 mIoU on Ramen / Teatime) looks like a failed run. SAGO, the closest prior work on target-centric virtual views, is cited but not compared. The two new benchmarks are presented as a contribution, but the paper gives only their sizes: no annotation protocol, no distribution over object / part / part-set / spatial queries, no per-granularity results, and no statement on release. Without a per-granularity breakdown, the central "versatile granularity" claim is not directly tested. On the technical side, most components are existing tools (SAGA + HDBSCAN, Fibonacci sampling, greedy max-coverage selection, Orient Anything, FlashSplat-style voting). The novelty therefore rests on the empirical case, and that case needs to be solid.
Questions for authors

Clarifications that could change my assessment:

  1. Please reconcile Table 6 (VGS, Kc = 12: 77.45) with Tables 1/4 (Full: 80.45). Are these different runs or different configurations?
  2. Which MLLM was used for the REALM reproduction? Can you report REALM and AnyGranGS with the same MLLM?
  3. Can you give per-granularity results (object / part / part set / spatial part) on Gran-LERF and Gran-3D-OVS, and describe how the benchmarks were annotated and whether they will be released?
  4. What is the variance of the full pipeline over repeated runs, and what is the per-query runtime and API cost compared with REALM?
  5. SAC estimates the object frame independently in each view. How often are the per-view axes inconsistent (e.g., front/back flips on symmetric objects), and what happens to the accepted masks in that case?

Suggestions (not required for my decision): run the ablations on all scenes, compare against SAGO, and fix the typos ("Constrants", "candidate sasks", "sptial", "VGSselects"; "R3DGS" is undefined in Fig. 4).

5 comments
AuthorsOpenReview ↗

Response to Reviewer Aewf (Part 1/5)

Response to 'Critical Weaknesses':

W1.1: The evidence behind the ablations is thin and partly inconsistent. Tables 4 and 6 share the setting Kc = 12 on Figurines. Random (74.88) and MAX (76.67) match across the two tables, but VGS at Kc = 12 is 77.45 in Table 6 while "Full" is 80.45 in Table 4 and Table 1. The 3-point gap is unexplained and is about the size of the VGS benefit being claimed.

A1.1: Thank you for pointing out this inconsistency. After rechecking the experimental records, we identified a reporting error involving the SAC configuration. We apologize for this oversight and clarify the discrepancy below.

(1) Reason for the 3-point discrepancy. The VGS result of 77.45 at Kc=12K_c=12 in Table 6 was mistakenly taken from an experiment with SAC disabled, whereas the Full result of 80.45 in Tables 1 and 4 was obtained with SAC enabled. This difference in SAC configurations explains the 3-point discrepancy identified by the reviewer.

(2) Reason why Random and MAX match across Tables 4 and 6. The w/o VGS (Random) and w/o VGS (MAX) results of 74.88 and 76.67 in Table 4 were also taken from the SAC-disabled experiments reported in Table 6. Consequently, the original Table 4 compared Random and MAX without SAC against Full with SAC, rather than evaluating the three view-selection strategies under the same configuration.

We have corrected Tables 4 and 6 and Figure 5 using results obtained with SAC enabled. The corrected results are presented below.

Revised Table 4. Ablation of TAO.

SettingmIoU ↑mBIoU ↑
Full80.4579.83
Fixed views + Random71.0170.37
Fixed views + VGS75.7074.80
w/o VGS (Random)77.2876.26
w/o VGS (MAX)78.8178.15

Revised Table 6. Detailed results of candidate-view selection on Gran-LERF Figurines (mIoU, %).

KcK_cRandomMAXVGS
266.8770.1577.33
473.2074.4479.80
675.5579.2180.22
876.8378.8481.24
1079.1480.1180.43
1277.2878.8180.45
1680.0380.0881.72

With the corrected SAC-enabled configuration, VGS outperforms Random and MAX by 3.17 and 1.64 mIoU points at Kc=12K_c=12, respectively. Moreover, VGS consistently outperforms both alternatives across all evaluated values of KcK_c, further supporting its effectiveness.

We thank the reviewer again for identifying this issue. We have updated Tables 4 and 6 and Figure 5 in the revised manuscript accordingly to ensure consistent experimental configurations and accurate reporting.

AuthorsOpenReview ↗

Response to Reviewer Aewf (Part 2/5)

W1.2: All ablations use a single scene, and the SAC ablation has only 6 queries / 19 masks. No run-to-run variance is reported, although the pipeline depends on a sampled closed-source MLLM. As a result, I cannot tell how much of the improvement comes from TAO/VGS/SAC rather than from noise or from the stronger MLLM.

A1.2: Thank you for highlighting the limited ablation evidence. We have extended our ablations to all Gran-LERF scenes. We conducted three runs per setting and report the mean ± standard deviation (SD) to assess run-to-run variability.

(1) Extended TAO Ablation. We extended the TAO ablation to all three Gran-LERF scenes. The results are as follows:

SettingFigurines (mIoU/mBIoU)Ramen (mIoU/mBIoU)Teatime (mIoU/mBIoU)Overall (mIoU/mBIoU)
Full80.58 ± 0.58 / 79.95 ± 0.6378.21 ± 0.89 / 70.27 ± 0.8372.29 ± 1.45 / 68.52 ± 0.6877.03 ± 0.46 / 72.91 ± 0.30
Fixed views + Random74.97 ± 0.59 / 74.17 ± 0.6777.09 ± 4.69 / 70.01 ± 3.8760.93 ± 1.00 / 56.34 ± 1.0270.99 ± 1.74 / 66.84 ± 1.48
Fixed views + VGS78.67 ± 0.45 / 77.99 ± 0.4580.25 ± 0.42 / 72.99 ± 0.4761.81 ± 2.33 / 55.68 ± 2.1573.58 ± 0.65 / 68.89 ± 0.72
w/o VGS (Random)79.01 ± 1.88 / 78.20 ± 2.0571.24 ± 4.12 / 63.59 ± 4.2269.17 ± 4.67 / 65.03 ± 4.2773.14 ± 1.80 / 68.94 ± 1.79
w/o VGS (MAX)79.59 ± 0.84 / 78.92 ± 0.8474.92 ± 4.69 / 67.98 ± 4.6067.50 ± 3.84 / 63.45 ± 3.5774.00 ± 2.31 / 70.12 ± 2.09

The full method achieves the highest overall mIoU and mBIoU, improving mIoU by 6.04 and 3.45 points over Fixed views + Random and Fixed views + VGS, respectively. On Ramen, Fixed views + VGS performs better than Full (80.25% vs. 78.21% mIoU), possibly because its relatively simple targets and favorable original camera coverage already provide sufficient observations, reducing the benefit of virtual observation. We will further improve performance in such cases. Nevertheless, our method achieves the best overall performance across all three scenes, supporting the effectiveness of TAO across diverse scenes.

(2) Extended VGS Ablation. We further evaluated VGS across all three scenes under different candidate-view budgets (Kc∈2,4,6,8,10,12,16K_c \in {2,4,6,8,10,12,16}). The results are as follows:

KcK_cFigurines Random (mIoU)Figurines MAX (mIoU)Figurines VGS (mIoU)Ramen Random (mIoU)Ramen MAX (mIoU)Ramen VGS (mIoU)Teatime Random (mIoU)Teatime MAX (mIoU)Teatime VGS (mIoU)
265.52 ± 1.7772.56 ± 2.5377.40 ± 0.5269.91 ± 3.7273.00 ± 5.5478.48 ± 1.5448.37 ± 0.8761.26 ± 3.5565.39 ± 1.04
472.36 ± 1.9576.49 ± 1.9779.91 ± 0.6167.31 ± 5.0974.12 ± 5.0478.59 ± 1.0654.52 ± 2.0466.28 ± 3.3568.55 ± 1.18
677.39 ± 1.6879.39 ± 0.7080.27 ± 0.4770.48 ± 3.0974.59 ± 4.7878.33 ± 1.0559.99 ± 2.2867.05 ± 3.9271.29 ± 0.88
877.53 ± 0.6379.50 ± 0.8380.71 ± 0.7272.08 ± 3.5774.76 ± 4.6778.41 ± 1.0965.72 ± 3.7867.24 ± 3.8970.97 ± 1.51
1079.31 ± 0.9680.03 ± 0.4780.53 ± 0.5969.25 ± 6.7174.83 ± 4.5778.37 ± 0.9664.08 ± 3.1267.41 ± 3.8771.46 ± 0.95
1279.01 ± 1.8879.59 ± 0.8480.58 ± 0.5871.24 ± 4.1274.92 ± 4.6978.21 ± 0.8969.17 ± 4.6767.50 ± 3.8472.29 ± 1.45
1679.46 ± 0.6879.98 ± 0.3781.01 ± 0.8474.63 ± 2.2174.88 ± 4.7778.37 ± 1.0871.69 ± 4.4870.05 ± 4.4471.78 ± 1.59

VGS consistently achieves the highest mIoU compared with Random and MAX across different candidate-view budgets.

(3) Extended SAC Ablation. We extended the SAC ablation to all 14 spatially conditioned part queries in Figurines (6 queries) and Teatime (8 queries). Note that Ramen is excluded because it primarily consists of round objects, making it unsuitable for spatial query annotation. The results are presented below:

SettingFigurines(mIoU)Teatime(mIoU)
w/o SAC68.58 ± 6.1150.91 ± 4.81
w/ SAC77.04 ± 1.0869.11 ± 2.28

SAC improves mIoU by 8.46 and 18.20 points on Figurines and Teatime, respectively. These results further support the effectiveness of SAC across different scenes.

Furthermore, the relatively small overall run-to-run variance further supports the stability of our method. We will incorporate these extended ablations and results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Aewf (Part 3/5)

W2.1: The baselines and benchmarks do not yet support the main claim. It is unclear which MLLM the REALM reproduction used. If it was not Gemini-3.6-Flash, part of the gain may come from the model.

A2.1: Thank you for raising this concern. We clarify that both REALM and AnyGranGS were evaluated using the same MLLM (Gemini-3.6-Flash) and segmentation model (SAM 2.1 Hiera-L) in the original Tables 1–3. Therefore, the reported improvements are not due to differences in these model backbones.

 

W2.2: The reproduced ReLaGS on Ref-LERF (5.72 / 1.79 mIoU on Ramen / Teatime) looks like a failed run.

A2.2: Thank you for pointing out this issue. After carefully checking the code, we found that our previous reproduction in Table 3 mistakenly excluded the relation-graph reasoning stage. We have corrected the results. The revised results are shown below:

MethodFig. (mIoU)Ram. (mIoU)Tea.(mIoU)Avg. (mIoU)
ReLaGS34.5121.1621.9125.86
AnyGranGS58.1078.9185.0574.02

Even after correcting the ReLaGS reproduction, AnyGranGS still substantially outperforms ReLaGS across all three scenes, with an average improvement of 48.16 mIoU points. We have updated Table 3 in the manuscript accordingly.

 

W2.3: SAGO, the closest prior work on target-centric virtual views, is cited but not compared.

A2.3: SAGO's code was first released on September 24, 2026, just one day before our submission deadline, leaving insufficient time to complete the experiments before submission. Therefore, SAGO was not included in our initial submission. We have now conducted comparisons using its released code on all three benchmarks. The results are as follows:

Results on Gran-LERF:

MethodFigurines (mIoU/mBIoU)Ramen (mIoU/mBIoU)Teatime (mIoU/mBIoU)Avg. (mIoU/mBIoU)
SAGO30.94 / 30.1231.37 / 22.8734.89 / 31.5332.40 / 28.17
AnyGranGS80.45 / 79.8377.18 / 70.0672.25 / 68.5976.63 / 72.83

Results on Gran-3D-OVS:

MethodBed (mIoU)Bench (mIoU)Lawn (mIoU)Room (mIoU)Sofa (mIoU)Avg. (mIoU)
SAGO66.6659.2756.9853.5859.0559.11
AnyGranGS82.1584.9681.7685.2073.4081.49

Results on Ref-LERF:

MethodFigurines (mIoU)Ramen (mIoU)Teatime (mIoU)Avg. (mIoU)
SAGO9.7913.0526.6816.51
AnyGranGS58.1078.9185.0574.02

Our method outperforms SAGO by 44.23, 22.38, and 57.51 mIoU points on Gran-LERF, Gran-3D-OVS, and Ref-LERF, respectively. We have incorporated these results into Tables 1–3 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.