acceptodds
Under review as a conference paper at ICLR 2027

Spatially Structured Codebook Learning for 3D Instance-Level Perception of Aerial-Ground Scenes

Abstract

Recent advances in unified aerial–ground reconstruction have enabled photorealistic modeling of large-scale environments, yet fine-grained scene understanding remains underexplored. Achieving consistent instance-level semantics is particularly challenging due to sparse per-view observations over a large identity space and severe cross-view variations in projected scale and visible detail. These factors complicate scene-wide optimization and lead to over- or under-segmented 2D instance masks with inconsistent supervision. To address these challenges, we propose Spatially Structured Codebook Learning (SSCL), a scalable framework for instance-level aerial–ground scene understanding. SSCL first organizes the scene into spatial root regions, each with a compact codebook of leaves representing local instance identities, and jointly learns cross-root connectivity to fuse leaves into globally consistent instances. A granularity-aware codebook learning method is further introduced to heuristically mitigate conflicting instance supervision. We further introduce Aerial–Ground Segmentation (AG-Seg), a new benchmark dataset for evaluating instance segmentation and open-vocabulary retrieval in aerial–ground scenes. Extensive experiments on AG-Seg demonstrate substantial improvements over state-of-the-art methods in both tasks, enabling fine-grained interaction with large aerial–ground environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, 3D Scene Understanding, Aerial–Ground Scene Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ny42OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SSCL for instance-level 3D understanding in aerial–ground scenes. It combines spatially organized local codebooks with learned cross-region associations and granularity-aware supervision to handle inconsistent 2D masks. The authors also introduce AG-Seg, a seven-scene benchmark, and demonstrate strong improvements in instance selection and open-vocabulary retrieval.

Critical strengths
  1. Clear motivation and coherent design. Local codebooks and cross-region association directly address limited view coverage and inconsistent object segmentation.
  2. Strong empirical results. SSCL outperforms the evaluated baselines on all seven scenes in both tasks, improving average Click/Text mIoU by 19.4/10.9 percentage points.
  3. Useful benchmark and ablations. AG-Seg provides a valuable evaluation resource, while the ablations support the main design choices.
Critical weaknesses
  1. Limited evaluation of identity consistency. Per-image segmentation metrics do not directly measure whether identities remain consistent across aerial and ground views.
  2. Incomplete isolation of the main contribution. Matched-capacity codebook comparisons and simpler cross-region fusion baselines would better establish the benefit of the proposed structure.
  3. Scalability needs further evidence. Rendering efficiency is reported, but training memory and the cost of learning and fusing cross-root connections remain unclear.
Questions for authors
  1. Can you provide direct cross-view identity-consistency metrics and single-click results?
  2. How does SSCL compare with a global codebook of comparable capacity and a simpler fusion strategy?
  3. What are the training time and peak memory on the largest scene, and how sensitive are results to the number of roots and leaves?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer Ny42 (Part 1)

Thank you for your valuable feedback. We have addressed each comment below and appreciate your thorough review.

Can you provide direct cross-view identity-consistency metrics?

Thank you for suggesting a cross-view ID consistency evaluation. To address this concern, we introduce a new evaluation benchmark based on Horizon-GS synthetic scenes to quantify instance identity preservation both across viewpoints (aerial--street) and within individual domains (aerial--aerial and street--street). Annotation groups are constructed by associating physical-object masks using raw depth and camera geometry, followed by manual verification of uncertain correspondences. The resulting benchmark contributes 5,826 additional directed source--target validation pairs, including 1,354 in Colosseum, 1,188 in Elvenruin, and 3,284 in CitySample.

To assess cross-view identity consistency with the new annotations, we design a dedicated protocol for all three settings. Within each group, every eligible view serves once as the source, and all methods use the same fixed click inside the annotated mask to select their instance IDs. Each method then holds its selected ID fixed and renders the complete 2D mask in all other eligible views. Results are reported below:

SceneMethodOverallAerial–aerialAerial–streetStreet–street
ColosseumOurs0.7230.6970.6590.853
ColosseumAG$^2$aussian0.6350.5690.5630.781
ColosseumUnified-Lift0.4090.5000.2280.660
ColosseumInstanceGaussian0.5220.4010.4600.648
ElvenruinOurs0.8200.8430.7330.860
ElvenruinAG$^2$aussian0.5300.5910.5010.552
ElvenruinUnified-Lift0.5150.5140.2910.557
ElvenruinInstanceGaussian0.4970.5380.5040.520
CitySampleOurs0.8500.9310.8290.822
CitySampleAG$^2$aussian0.2680.1080.0920.374
CitySampleUnified-Lift0.4480.7970.0680.342
CitySampleInstanceGaussian0.2860.4060.5270.245

These results demonstrate that our approach still achieves substantial advantages in cross-view identity consistency. We will incorporate this key finding into the revised manuscript.

Can you provide direct single-click results?

We have completed a controlled single-click evaluation on Colosseum, CitySample, Elvenruin, Road, and Park. For every annotated instance, one point is chosen deterministically at the deepest interior location of its ground-truth mask, independently of every method's prediction. The table below compares that fixed single click with the original best-of-three selection on exactly the same frozen samples.

MethodSingle-click mIoUBest-of-three mIoU
Ours0.7670.778
Unified-Lift0.5970.618
AG$^2$aussian0.4850.501
InstanceGaussian0.3670.398
OpenGaussian0.2310.285

Under the single-click evaluation protocol, all evaluated methods exhibit a slight performance drop. However, the relative performance differences remain largely unchanged, and our overall performance advantage is consistently maintained.

AuthorsOpenReview ↗

Author Response to Reviewer Ny42 (Part 2)

How does SSCL compare with a global codebook of comparable capacity?

Regarding the reviewer's concern about capacity alignment, the capacity of a global codebook and that of our local framework have different meanings and are therefore not directly comparable. In a global codebook, each entry represents a complete scene-level instance identity. In contrast, each local leaf in our framework represents an instance or a spatial fragment of an instance within a specific root. Consequently, an object spanning multiple roots may occupy multiple local leaves, which are subsequently associated with a shared identity through fusion. Thus, matching the global codebook size to R×LR \times L aligns the nominal slot budgets, but does not imply an equivalent capacity for representing complete scene-level instances.

To further address the reviewer's request, we attempted to match Unified-Lift's codebook size to SSCL's numerical base-slot budget and evaluated additional smaller configurations where the matched setting exceeded the available GPU memory. All experiments were conducted on an RTX 4090.

SceneMethodBase slotsClick mIoUPeak training allocated (GiB)
ColosseumUnified-Lift1024 instances0.71411.4
ColosseumUnified-Lift1536 instances0.71114.1
ColosseumOurs64 × 24 = 1536 parts0.8008.1
CitySampleUnified-Lift3072 instances--OOM
CitySampleUnified-Lift2048 instances--OOM
CitySampleUnified-Lift1536 instances0.58419.6
CitySampleUnified-Lift1024 instances0.58216.4
CitySampleOurs128 × 24 = 3072 parts0.86810.2

The experimental results show that performance nearly saturates with a global codebook size of 1,024, and further increasing its capacity yields negligible performance gains. Meanwhile, larger global codebooks incur substantially higher computational and memory overhead, eventually leading to out-of-memory (OOM) errors. Given that CitySample is the largest scene in our evaluation, we consider a global codebook capacity of 1,024 reasonable for our current setting. These results demonstrate the practical scalability and memory-efficiency advantages of our local framework under the evaluated resource constraints.

How does SSCL compare with a simpler fusion strategy?

To compare SSCL with a simpler fusion strategy, we disable both physical gating and group-level connectivity aggregation in our “Connectivity-guided leaf fusion” (lines 339–350). This baseline directly takes the transitive closure of positive pairwise links and corresponds to the “w/o group-level connectivity & w/o physical gate” column below. We additionally evaluate a variant that removes only group-level connectivity aggregation while retaining physical gating, allowing us to assess the contribution of each mechanism. The Click mIoU results are reported below.

SceneFullw/o group-level connectivityw/o group-level connectivity & w/o physical gate
Colosseum0.8000.7880.766
Elvenruin0.8570.8020.785
CitySample0.8680.8530.759
Road0.6850.6700.624
Park0.6830.6740.664

Simple fusion strategies are prone to over-merging distinct objects, which can substantially degrade segmentation quality. The results show that both of our mechanisms effectively mitigate this issue.

AuthorsOpenReview ↗

Author Response to Reviewer Ny42 (Part 3)

How does SSCL compare with a simpler fusion strategy?

To compare SSCL with a simpler fusion strategy, we disable both physical gating and group-level connectivity aggregation in our “Connectivity-guided leaf fusion” (lines 339–350). This baseline directly takes the transitive closure of positive pairwise links and corresponds to the “w/o group-level connectivity & w/o physical gate” column below. We additionally evaluate a variant that removes only group-level connectivity aggregation while retaining physical gating, allowing us to assess the contribution of each mechanism. The Click mIoU results are reported below.

SceneFullw/o group-level connectivityw/o group-level connectivity<br>& w/o physical gate
Colosseum0.8000.7880.766
Elvenruin0.8570.8020.785
CitySample0.8680.8530.759
Road0.6850.6700.624
Park0.6830.6740.664

Simple fusion strategies are prone to over-merging distinct objects, which can substantially degrade segmentation quality. The results show that both of our mechanisms effectively mitigate this issue.

What are the training time and peak memory on the largest scene

CitySample is the largest of the five ablation scenes by registered anchor count. We compare the recorded training time and peak GPU memory of our method with two representative instance-level baselines: Unified-Lift, which uses a global codebook, and AG$^2$aussian, which learns instance features and recovers instances through clustering.

MethodRecorded timePeak GPU memory (GiB)
Ours2 h 20 min10.2
Unified-Lift4 h 25 min16.4
AG$^2$aussian1 h 54 min18.9

SSCL achieves the lowest peak GPU memory among the three methods. Compared with Unified-Lift, it reduces recorded training time by 47.2% and peak memory by 37.8%. Compared with AG$^2$aussian, it requires 26 additional minutes of training while reducing peak memory by 46.0%.

These efficiency differences can be partly explained by each method’s semantic representation and optimization strategy. SSCL restricts each anchor’s semantic assignment to the LL leaves within its root, reducing the matching and intermediate-tensor costs associated with Unified-Lift’s KK global entries. Cross-root connections receive sparse updates, further limiting their computational overhead. AG$^2$aussian learns low-dimensional anchor features and subsequently clusters them, which helps explain its shorter training time. Its anchor graph supports feature propagation among neighboring anchors, while graph storage and propagation contribute to memory usage.

Overall, SSCL has a moderately longer runtime than AG$^2$aussian but requires substantially less GPU memory, while outperforming Unified-Lift in both runtime and memory consumption.

How sensitive are results to the number of roots and leaves?

We assessed sensitivity to R and L using four additional configurations per scene, varying the overall model capacity by approximately ±30% around the default setting. The results are reported below:

SceneRLClick mIoUDelta vs. default
Colosseum64240.800default
Colosseum64160.791-0.009
Colosseum64320.804+0.004
Colosseum48240.772-0.028
Colosseum80240.811+0.011
CitySample128240.868default
CitySample128160.836-0.032
CitySample128320.871+0.003
CitySample96240.811-0.057
CitySample160240.854-0.014

Reducing either RR or LL consistently degrades performance on both scenes, indicating the importance of sufficient local capacity for capturing instance-level information. The effect of increasing capacity, however, is not always positive. On CitySample, increasing RR slightly reduces performance, potentially because excessive partitioning complicates cross-block fusion and increases the risk of incorrect associations. Despite these variations, the lowest-performing configuration on each scene still surpasses the strongest baseline (0.772 vs. 0.759 on Colosseum and 0.811 vs. 0.582 on CitySample). These results support a robust performance advantage across the tested configurations.

Thank the authors for their rebuttal. The rebuttal has addressed my concerns, and I would like to keep my score.

Reviewer_yUZLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies instance-level aerial–ground scene understanding and proposes SSCL. To address sparse supervision in large-scale aerial–ground scenes, SSCL divides the Gaussian scene into spatial regions, learns local instance codebooks within these regions, and fuses instances through cross-region connections. SSCL also introduces granularity-aware matching, fragment attachment, and purity-weighted supervision to mitigate inconsistencies in the segmentation granularity of 2D masks across views. The authors add annotations to seven existing scenes to construct the AG-Seg benchmark. On the two tasks of click selection and text retrieval, SSCL shows clear performance improvements over the comparison methods.

Critical strengths
  1. The problem addressed by the paper has practical significance, and the method’s design is also fairly well targeted. In large-scale aerial–ground scenes, a single view can cover only a small part of the scene, leading to sparse supervision for the optimization of global instance representations. Meanwhile, differences in scale and visible detail between aerial and ground views can easily lead to inconsistent mask granularity. The authors use local codebooks to reduce the identity space involved in local optimization to alleviate the optimization difficulties caused by sparse supervision. They then connect local instances through cross-region associations and address mask fragmentation and incorrect merging. These design choices correspond fairly clearly to the difficulties of the task and provide a useful reference for 3D instance segmentation in large scenes.

  2. The performance improvements brought by the method are fairly evident, and the authors provide a relatively extensive set of comparative experiments, ablation analyses, and application demonstrations, along with sufficient visualizations to give readers an intuitive understanding of the method’s results. Overall, the work is fairly solid and also provides convincing evidence of the method’s effectiveness on the evaluated tasks.

Critical weaknesses
  1. I still have questions about the validation of 3D instance completeness and cross-view consistency. The paper mainly evaluates performance using IoU after 2D projection, which can reflect segmentation quality in visible regions but is insufficient to show whether the entire object corresponds to the same 3D instance. In Click, selection is performed independently in each image, so high scores may still be obtained even if the parts visible in different views are split into different instances. I would like to see an instance selected in one view and then evaluated in other views while keeping its identity fixed, to verify global consistency more directly.

  2. I am still not quite sure how much the local codebooks and cross-region association each contribute. In Table 2, these two components are introduced together when switching from the global codebook to the authors’ framework. The subsequent ablations in Table 2 show the role of the granularity-aware components but do not separate these two changes. I would like the authors to conduct separate comparisons under comparable capacity and training settings. In addition, I would also like to see how the results differ between establishing associations directly from multi-view matching results and learning the connections.

Questions for authors
  1. Could the authors add a cross-view evaluation with a fixed instance identity, namely, selecting a 3D instance in one view and evaluating the segmentation results of the same instance in other views? In particular, how does the method perform in terms of instance completeness and consistency between aerial and ground views, and does it still show clear improvements over the baselines?

  2. In Table 2, how are capacity and training settings controlled for the global codebook and the local framework? Could the authors provide results from separate comparisons for local codebooks and cross-region fusion to show how much each contributes?

  3. Have the authors compared against an approach that directly aggregates multi-view matching results to establish cross-region associations? When the same matching evidence, confidence weights, and fusion rules are used, how does this approach compare with learning the connections in terms of performance?

  4. How does the paper handle cases of cross-view granularity overlap? For example, suppose there are three instances, A, B, and C: view 1 groups A and B together, while view 2 groups B and C together. Could the authors analyze when such cases would lead to incorrect merging of the final instances and how this would affect the model’s results?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer yUZL (Part 1)

Thanks for your insightful comments, we respond to the comments below:

Could the authors add a cross-view evaluation with a fixed instance identity, namely, selecting a 3D instance in one view and evaluating the segmentation results of the same instance in other views? In particular, how does the method perform in terms of instance completeness and consistency between aerial and ground views, and does it still show clear improvements over the baselines?

Thank you for suggesting an evaluation that follows the same selected instance across views.

To directly evaluate cross-view identity consistency, we construct a new evaluation set using the synthetic scenes from Horizon-GS to assess cross-view instance ID consistency between aerial and street views(aerial–street), complemented by aerial–aerial and street–street settings to assess consistency within each view type. Specifically, we form cross-view annotation groups by associating masks of the same physical object using raw depth maps and camera geometry, with human review of uncertain correspondences. In total, these verified associations provide 5,826 directed source--target validation pairs, comprising 1,354 in Colosseum, 1,188 in Elvenruin, and 3,284 in CitySample.

For evaluation, we introduce a unified fixed-ID retrieval protocol and report cross-view mask mIoU across all three settings. Within each annotation group, every eligible view serves as the source in turn. For each source view, all methods receive the same fixed click within the annotated object mask to select their respective instance IDs. Each selected ID is then kept unchanged to render the complete instance mask in every other eligible view.

The results are reported below.

SceneMethodOverallAerial–aerialAerial–streetStreet–street
ColosseumOurs0.7230.6970.6590.853
ColosseumAG$^2$aussian0.6350.5690.5630.781
ColosseumUnified-Lift0.4090.5000.2280.660
ColosseumInstanceGaussian0.5220.4010.4600.648
ElvenruinOurs0.8200.8430.7330.860
ElvenruinAG$^2$aussian0.5300.5910.5010.552
ElvenruinUnified-Lift0.5150.5140.2910.557
ElvenruinInstanceGaussian0.4970.5380.5040.520
CitySampleOurs0.8500.9310.8290.822
CitySampleAG$^2$aussian0.2680.1080.0920.374
CitySampleUnified-Lift0.4480.7970.0680.342
CitySampleInstanceGaussian0.2860.4060.5270.245

The updated experimental results further demonstrate the effectiveness of our method. Ours obtains object-balanced mIoU of 0.723/0.820/0.850, compared with the strongest overall baseline in each scene at 0.635/0.530/0.448. We will include these findings in the revised manuscript and investigate how our method performs in real-world scenes in future work.

AuthorsOpenReview ↗

Author Response to Reviewer yUZL (Part 2)

In Table 2, how are capacity and training settings controlled for the global codebook and the local framework?

To control the training conditions, we use the same frozen Urban-GS geometry and an 80k-iteration semantic training budget for both the global-codebook baseline and our local framework. Both maintain semantic features at the anchor level, with each decoded Gaussian inheriting its parent anchor’s semantic feature.

For the global codebook, we adopt Unified-Lift as the baseline and increase its capacity to 1,024 entries based on the expected number of scene-level instances. Each entry represents a global instance identity. For our local framework, we allocate 24 leaves per root, using 64 roots by default and 128 for CitySample, Road, and Park. Each leaf represents an instance or an instance fragment within its root. An object spanning multiple roots may therefore occupy several local leaves, so the nominal capacity of R×LR \times L describes the available local slots before fusion.

To further assess the effect of codebook capacity, we increase Unified-Lift’s global codebook to match our framework’s nominal slot budget. We evaluate 1,536 entries on Colosseum and attempt 3,072 entries on CitySample, with additional smaller configurations evaluated when GPU memory limits prevent the matched setting from running. All experiments are conducted on an RTX 4090, and the results are reported below:

SceneMethodBase slotsClick mIoUPeak training allocated (GiB)
ColosseumUnified-Lift1024 instances0.71411.4
ColosseumUnified-Lift1536 instances0.71114.1
ColosseumOurs64 × 24 = 1536 parts0.8008.1
CitySampleUnified-Lift3072 instances--OOM
CitySampleUnified-Lift2048 instances--OOM
CitySampleUnified-Lift1536 instances0.58419.6
CitySampleUnified-Lift1024 instances0.58216.4
CitySampleOurs128 × 24 = 3072 parts0.86810.2

Increasing Unified-Lift's codebook capacity from 1,024 to 1,536 entries did not yield a substantial performance improvement, while further increases were constrained by GPU memory on CitySample. Given that CitySample is the largest scene in our evaluation, we consider a global codebook capacity of 1,024 reasonable for our current setting.

Further implementation details are provided in Appendix A, "Detailed Implementation and Hyperparameters".

Could the authors provide results from separate comparisons for local codebooks and cross-region fusion to show how much each contributes?

We further separate the 'Our baseline framework' configuration in Table 2 into local codebooks and cross-region fusion to clarify their respective contributions. The Click mIoU results are reported below:

ConfigurationMeanColosseumElvenruinCitySampleRoadPark
Global codebook baseline0.6180.7140.6720.5820.5580.562
───────────────────────────────────────────────────────
Local codebook only0.6290.6780.7130.6400.5250.586
+ Connectivity-guided leaf fusion0.7310.7770.8190.7930.6040.662

Many instances span multiple spatial roots because root partitions do not align with object boundaries. Without fusion, a click retrieves only a root-local fragment and misses the remaining parts of the object, limiting the performance of “Local codebook only.” Nevertheless, this variant remains competitive with the global codebook baseline, suggesting that localizing optimization is still effective in large-scale scenes. Cross-region fusion reconnects these fragments, improving instance completeness and helping explain the observed performance gain.

AuthorsOpenReview ↗

Author Response to Reviewer yUZL (Part 3)

Have the authors compared against an approach that directly aggregates multi-view matching results to establish cross-region associations? When the same matching evidence, confidence weights, and fusion rules are used, how does this approach compare with learning the connections in terms of performance?

Thank you for suggesting this comparison. To address your concern, we implement a direct weighted aggregation baseline using the same definitions of matching targets and reliability weights, and compare it with learned connectivity under the same downstream fusion rules.

For this baseline, we freeze the final leaf assignments and replay the training views to collect multi-view matching evidence. Using the association targets ca,bvc_{a,b}^{v} and reliability weights wa,bvw_{a,b}^{v} defined in Eqs. (11)–(12), we compute the score for each cross-root leaf pair (a,b)(a,b) as Ca,bagg=∑vwa,bvca,bv∑vwa,bvC_{a,b}^{\mathrm{agg}} = \frac{\sum_v w_{a,b}^{v}c_{a,b}^{v}}{\sum_v w_{a,b}^{v}}, where both sums include only views providing valid co-observed matching evidence. Pairs with zero total weight are excluded from fusion. We substitute this aggregated score for the learned score Ca,b=sigmoid⁡(ℓa,b)C_{a,b}=\operatorname{sigmoid}(\ell_{a,b}) and report the comparison below.

SceneFullDirect weighted aggregation
Colosseum0.8000.787
CitySample0.8680.857
Elvenruin0.8570.844
Road0.6850.678
Park0.6830.667
Mean0.7780.767

Direct weighted aggregation is effective, with a mean Click mIoU of 0.767 versus 0.778 for Full, a modest gap of 1.1 percentage points. Our learned connectivity is jointly optimized with evolving leaf assignments, continuously adapting to multi-view matching evidence. Moreover, the iterative optimization itself is inexpensive. Connection parameters occupy approximately 9–36 MiB, and each supervised update takes 0.50–0.54 ms on average on an RTX 4090. The parameter storage, mean update time, and estimated cumulative update time are listed below.

SceneConnection parameters (MiB)Mean update (ms)Active iteration rangeEstimated total (s)
Colosseum8.990.53730k-80k26.8
CitySample35.990.50330k-80k25.1
Elvenruin8.990.50630k-80k25.3
Road35.990.52230k-80k26.1
Park35.990.52030k-80k25.8

The schedule-based estimate is approximately 25–27 seconds per scene for numerical updates on prepared pairs, excluding rendering, matching, pair construction, and final fusion. At this modest update cost, we consider retaining iterative optimization a reasonable trade-off for the accuracy gain.

Reviewer_33G9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper addresses 3D instance and open-vocabulary segmentation on large-scale aerial-ground 3D Gaussian Splatting (3DGS) scenes. Built upon a frozen Urban-GS backbone, the method lifts 2D SAM masks into 3D instance IDs and attaches CLIP features for click-based and text-based querying. The framework consists of three technical components: (1) a root-local representation that partitions anchors into spatial roots via K-means, assigning each root a small set of local instance labels with dominant-root rendering; (2) a granularity-aware matching module that pairs masks with local leaves using the harmonic mean of purity and coverage, while attaching small fragments and down-weighting under-segmented masks; and (3) a learned cross-root affinity matrix supervised by 2D mask co-occurrence, followed by greedy average-linkage merging at inference. The authors also introduce the AG-Seg benchmark across seven scenes (five synthetic and two real) with 5,231 click queries and 1,463 text queries, showing strong quantitative gains over existing baselines.

Critical strengths
  • Efficient spatial representation: Partitioning the scene into spatial roots enables fast rendering and low memory overhead on city-scale scenes. Supervising cross-root merging via 2D mask co-occurrence avoids fragile clustering heuristics.

  • Practical aerial-ground benchmark: The AG-Seg benchmark provides valuable dense click and text annotations across both aerial and ground views. The click evaluation protocol is rigorous and easy to follow.

Critical weaknesses
  • Unmeasured core claim (Figure 1, Section 4): The central claim of cross-view global consistency is not directly evaluated. As detailed in Section 4, both click and text protocols compute 2D IoU on individual images. A model can assign conflicting IDs to the same building across aerial and ground views and still achieve high scores. The paper lacks direct 3D instance metrics or cross-view ID consistency scores on the synthetic scenes. Results are also not reported separately for aerial and ground views.

  • Unfair capacity and missing baseline controls (Section 3.1, Table 2): The comparison with global baselines has a capacity confound. In Table 2, the baseline codebook is fixed to 1024, while the proposed root-local model uses a substantially larger label capacity. It is unclear how much gain comes from spatial decomposition rather than expanded capacity. In addition, the paper lacks a global baseline that restricts Hungarian matching to anchors visible in the current view.

  • Incomplete ablations and baseline fairness (Section 3.2, Table 3, Table 6): Key parameters R and L are chosen without ablation studies. Table 3 does not include a baseline without cross-root fusion. Per-scene results in Table 6 show that the granularity modules from Section 3.2 cause performance drops on certain scenes. Finally, the paper does not confirm whether all baselines in Table 1 use the exact same frozen Urban-GS geometry.

Questions for authors
  • Can you report direct 3D instance metrics or cross-view ID consistency on the synthetic scenes, with results broken down by aerial and ground views?

  • Are all baselines in Table 1 evaluated on the exact same frozen Urban-GS geometry, and how does the global codebook perform when its label capacity is matched to your model?

  • What are the quantitative results when cross-root fusion in Table 3 is completely removed, and how sensitive is performance to different values of R and L?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 33G9 (Part 1)

Thanks for your careful review. We respond to the comments below:

Can you report direct 3D instance metrics or cross-view ID consistency on the synthetic scenes, with results broken down by aerial and ground views?

Thank you for raising this question. Our original evaluation follows the practice adopted in prior work such as LangSplatV2 and AG$^2$aussian, which assesses segmentation quality in individual views.

We agree that directly evaluating cross-view instance ID consistency is meaningful and important. To address this concern, we construct a new evaluation set using the synthetic scenes from Horizon-GS to assess cross-view instance ID consistency between aerial and street-level views (aerial–street). We additionally include aerial–aerial and street–street evaluation settings to examine consistency within each view type. Specifically, we form cross-view annotation groups by associating masks of the same physical object using raw depth maps and camera geometry, with human review of uncertain correspondences. The accepted associations yield 5,826 additional directed source–target validation pairs: 1,354 in Colosseum, 1,188 in Elvenruin, and 3,284 in CitySample.

For evaluation, we specifically designed a metric to assess performance consistently across all three settings. Within each annotation group, every eligible view serves once as the source view. For each source view, all methods use the same fixed click inside the annotated object mask to select their respective instance IDs. Each method then keeps its selected ID fixed and renders the complete 2D mask corresponding to that ID in every other eligible view within the group.

The results are reported below.

SceneMethodOverallAerial–aerialAerial–streetStreet–street
ColosseumOurs0.7230.6970.6590.853
ColosseumAG$^2$aussian0.6350.5690.5630.781
ColosseumUnified-Lift0.4090.5000.2280.660
ColosseumInstanceGaussian0.5220.4010.4600.648
ElvenruinOurs0.8200.8430.7330.860
ElvenruinAG$^2$aussian0.5300.5910.5010.552
ElvenruinUnified-Lift0.5150.5140.2910.557
ElvenruinInstanceGaussian0.4970.5380.5040.520
CitySampleOurs0.8500.9310.8290.822
CitySampleAG$^2$aussian0.2680.1080.0920.374
CitySampleUnified-Lift0.4480.7970.0680.342
CitySampleInstanceGaussian0.2860.4060.5270.245

The updated experimental results further demonstrate the effectiveness of our method. We will include these findings in the revised manuscript and investigate how our method performs in real-world scenes in future work.

Are all baselines in Table 1 evaluated on the exact same frozen Urban-GS geometry

Yes. All methods in our experiments use the same frozen Urban-GS geometry.

AuthorsOpenReview ↗

Author Response to Reviewer 33G9 (Part 2)

how does the global codebook perform when its label capacity is matched to your model?

We set the global codebook capacity to 1,024 based on the expected max number of scene-level instances and computational cost. However, in our method, a local leaf represents part of an instance within a spatial root. Since an instance may span multiple roots and occupy several local leaves, the R×LR\times L local leaves generally correspond to fewer distinct scene-level instances. Thus, matching the global codebook size to R×LR\times L aligns the numerical slot budgets, but does not imply equivalent instance capacity.

To further address the reviewer's request, We attempted to match Unified-Lift's codebook size to SSCL's numerical base-slot budget and evaluated additional smaller configurations where the matched setting exceeded the available GPU memory. All experiments were conducted on an RTX 4090.

SceneMethodBase slotsClick mIoUPeak training allocated (GiB)
ColosseumUnified-Lift1024 instances0.71411.4
ColosseumUnified-Lift1536 instances0.71114.1
ColosseumOurs64 × 24 = 1536 parts0.8008.1
CitySampleUnified-Lift3072 instances--OOM
CitySampleUnified-Lift2048 instances--OOM
CitySampleUnified-Lift1536 instances0.58419.6
CitySampleUnified-Lift1024 instances0.58216.4
CitySampleOurs128 × 24 = 3072 parts0.86810.2

The experimental results show that performance nearly saturates with a global codebook size of 1,024, and further increasing its capacity yields negligible performance gains. Meanwhile, larger global codebooks incur substantially higher computational and memory overhead, eventually leading to out-of-memory (OOM) errors. Given that CitySample is the largest scene in our evaluation, we consider a global codebook capacity of 1,024 reasonable for our current setting. These results demonstrate the practical scalability and memory-efficiency advantages of our local framework under the evaluated resource constraints.

To avoid ambiguity, we will clarify in the revised manuscript how the physical interpretation of the codebook differs.

What are the quantitative results when cross-root fusion in Table 3 is completely removed?

We have added a five-scene ablation comparing Full with w/o Connectivity-guided leaf fusion to quantify the contribution of the complete post-fusion pipeline. The added ablation directly compares the results before and after Connectivity-guided leaf fusion:

SceneFullw/o Connectivity-guided leaf fusion
Colosseum0.8000.701
CitySample0.8680.667
Elvenruin0.8570.730
Road0.6850.586
Park0.6830.610
Mean0.7780.659

Ablating the proposed fusion scheme drops the mean score from 0.778 to 0.659, with performance degrading across all scenes. This loss stems primarily from spatial root boundaries misaligning with object boundaries, which splits single instances across multiple root regions. Without fusion, queries on local fragments fail to retrieve an instance's remaining parts, compromising completeness and mIoU. Fusion resolves this by linking cross-root fragments under a unified identity.

AuthorsOpenReview ↗

Author Response to Reviewer 33G9 (Part 3)

How sensitive is performance to different values of R and L?

We evaluated four additional R/L configurations on Colosseum and CitySample, varying the overall model capacity by approximately ±30% around the default setting to assess sensitivity. The results are reported below:

SceneRLClick mIoUDelta vs. default
Colosseum64240.800default
Colosseum64160.791-0.009
Colosseum64320.804+0.004
Colosseum48240.772-0.028
Colosseum80240.811+0.011
CitySample128240.868default
CitySample128160.836-0.032
CitySample128320.871+0.003
CitySample96240.811-0.057
CitySample160240.854-0.014

Both scenes exhibit performance degradation when the model capacity is reduced, suggesting that a larger capacity better captures instance-level information in complex scenes. On CitySample, increasing RR leads to a slight performance drop, indicating that excessive partitioning may increase the complexity of cross-block fusion and potentially introduce incorrect merging decisions.

Overall, we acknowledge that model capacity exerts only a minor impact on instance segmentation performance, and we therefore empirically set these two parameters. What's more, even the worst-performing configuration within the ±30% variation range consistently outperforms the strongest baseline (0.772 vs. 0.759 on Colosseum and 0.811 vs. 0.582 on CitySample). These results support a robust performance advantage across the tested configurations.

In addition, the paper lacks a global baseline that restricts Hungarian matching to anchors visible in the current view.

Our current global-codebook baseline already satisfies this visibility restriction, as only anchors visible in the current view contribute to the Hungarian matching scores.

The main bottleneck is that matching still considers the full codebook of KK global IDs, even though each view observes only a small subset of scene instances. This large candidate identity space makes optimization challenging under sparse per-view supervision. SSCL addresses this bottleneck through its root-local structure, which restricts each mask component’s matching candidates to the LL leaves of its corresponding spatial root while retaining R×LR \times L entries across the scene.

We will clarify this distinction and the baseline’s visibility restriction in the revised manuscript to avoid confusion.

The granularity modules from Section 3.2 cause performance drops on certain scenes.

Thank you for carefully examining the per-scene ablations. On Elvenruin, the observed decrease may be associated with fragmented or incomplete 2D supervision, where a mask captures only part of the corresponding leaf’s visible region. Such masks can receive low coverage scores under our purity–coverage criterion, making mask-to-leaf assignment less stable. This may explain the slight Click mIoU decrease from 0.819 to 0.813 when granularity-aware assignment is introduced. Across the other four scenes—Colosseum, CitySample, Road, and Park—the module improves Click mIoU by 2.0–5.4 percentage points, showing an overall benefit alongside this small scene-specific decrease.

Thank authoss for the detailed response and experiments, which have addressed most of my concerns, so I will keep my original score (soundness updated).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.