- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SSCL for instance-level 3D understanding in aerial–ground scenes. It combines spatially organized local codebooks with learned cross-region associations and granularity-aware supervision to handle inconsistent 2D masks. The authors also introduce AG-Seg, a seven-scene benchmark, and demonstrate strong improvements in instance selection and open-vocabulary retrieval.
Critical strengths
- Clear motivation and coherent design. Local codebooks and cross-region association directly address limited view coverage and inconsistent object segmentation.
- Strong empirical results. SSCL outperforms the evaluated baselines on all seven scenes in both tasks, improving average Click/Text mIoU by 19.4/10.9 percentage points.
- Useful benchmark and ablations. AG-Seg provides a valuable evaluation resource, while the ablations support the main design choices.
Critical weaknesses
- Limited evaluation of identity consistency. Per-image segmentation metrics do not directly measure whether identities remain consistent across aerial and ground views.
- Incomplete isolation of the main contribution. Matched-capacity codebook comparisons and simpler cross-region fusion baselines would better establish the benefit of the proposed structure.
- Scalability needs further evidence. Rendering efficiency is reported, but training memory and the cost of learning and fusing cross-root connections remain unclear.
Questions for authors
- Can you provide direct cross-view identity-consistency metrics and single-click results?
- How does SSCL compare with a global codebook of comparable capacity and a simpler fusion strategy?
- What are the training time and peak memory on the largest scene, and how sensitive are results to the number of roots and leaves?
4 comments
Author Response to Reviewer Ny42 (Part 2)
How does SSCL compare with a global codebook of comparable capacity?
Regarding the reviewer's concern about capacity alignment, the capacity of a global codebook and that of our local framework have different meanings and are therefore not directly comparable. In a global codebook, each entry represents a complete scene-level instance identity. In contrast, each local leaf in our framework represents an instance or a spatial fragment of an instance within a specific root. Consequently, an object spanning multiple roots may occupy multiple local leaves, which are subsequently associated with a shared identity through fusion. Thus, matching the global codebook size to aligns the nominal slot budgets, but does not imply an equivalent capacity for representing complete scene-level instances.
To further address the reviewer's request, we attempted to match Unified-Lift's codebook size to SSCL's numerical base-slot budget and evaluated additional smaller configurations where the matched setting exceeded the available GPU memory. All experiments were conducted on an RTX 4090.
| Scene | Method | Base slots | Click mIoU | Peak training allocated (GiB) |
|---|---|---|---|---|
| Colosseum | Unified-Lift | 1024 instances | 0.714 | 11.4 |
| Colosseum | Unified-Lift | 1536 instances | 0.711 | 14.1 |
| Colosseum | Ours | 64 × 24 = 1536 parts | 0.800 | 8.1 |
| CitySample | Unified-Lift | 3072 instances | -- | OOM |
| CitySample | Unified-Lift | 2048 instances | -- | OOM |
| CitySample | Unified-Lift | 1536 instances | 0.584 | 19.6 |
| CitySample | Unified-Lift | 1024 instances | 0.582 | 16.4 |
| CitySample | Ours | 128 × 24 = 3072 parts | 0.868 | 10.2 |
The experimental results show that performance nearly saturates with a global codebook size of 1,024, and further increasing its capacity yields negligible performance gains. Meanwhile, larger global codebooks incur substantially higher computational and memory overhead, eventually leading to out-of-memory (OOM) errors. Given that CitySample is the largest scene in our evaluation, we consider a global codebook capacity of 1,024 reasonable for our current setting. These results demonstrate the practical scalability and memory-efficiency advantages of our local framework under the evaluated resource constraints.
How does SSCL compare with a simpler fusion strategy?
To compare SSCL with a simpler fusion strategy, we disable both physical gating and group-level connectivity aggregation in our “Connectivity-guided leaf fusion” (lines 339–350). This baseline directly takes the transitive closure of positive pairwise links and corresponds to the “w/o group-level connectivity & w/o physical gate” column below. We additionally evaluate a variant that removes only group-level connectivity aggregation while retaining physical gating, allowing us to assess the contribution of each mechanism. The Click mIoU results are reported below.
| Scene | Full | w/o group-level connectivity | w/o group-level connectivity & w/o physical gate |
|---|---|---|---|
| Colosseum | 0.800 | 0.788 | 0.766 |
| Elvenruin | 0.857 | 0.802 | 0.785 |
| CitySample | 0.868 | 0.853 | 0.759 |
| Road | 0.685 | 0.670 | 0.624 |
| Park | 0.683 | 0.674 | 0.664 |
Simple fusion strategies are prone to over-merging distinct objects, which can substantially degrade segmentation quality. The results show that both of our mechanisms effectively mitigate this issue.
Author Response to Reviewer Ny42 (Part 3)
How does SSCL compare with a simpler fusion strategy?
To compare SSCL with a simpler fusion strategy, we disable both physical gating and group-level connectivity aggregation in our “Connectivity-guided leaf fusion” (lines 339–350). This baseline directly takes the transitive closure of positive pairwise links and corresponds to the “w/o group-level connectivity & w/o physical gate” column below. We additionally evaluate a variant that removes only group-level connectivity aggregation while retaining physical gating, allowing us to assess the contribution of each mechanism. The Click mIoU results are reported below.
| Scene | Full | w/o group-level connectivity | w/o group-level connectivity<br>& w/o physical gate |
|---|---|---|---|
| Colosseum | 0.800 | 0.788 | 0.766 |
| Elvenruin | 0.857 | 0.802 | 0.785 |
| CitySample | 0.868 | 0.853 | 0.759 |
| Road | 0.685 | 0.670 | 0.624 |
| Park | 0.683 | 0.674 | 0.664 |
Simple fusion strategies are prone to over-merging distinct objects, which can substantially degrade segmentation quality. The results show that both of our mechanisms effectively mitigate this issue.
What are the training time and peak memory on the largest scene
CitySample is the largest of the five ablation scenes by registered anchor count. We compare the recorded training time and peak GPU memory of our method with two representative instance-level baselines: Unified-Lift, which uses a global codebook, and AG$^2$aussian, which learns instance features and recovers instances through clustering.
| Method | Recorded time | Peak GPU memory (GiB) |
|---|---|---|
| Ours | 2 h 20 min | 10.2 |
| Unified-Lift | 4 h 25 min | 16.4 |
| AG$^2$aussian | 1 h 54 min | 18.9 |
SSCL achieves the lowest peak GPU memory among the three methods. Compared with Unified-Lift, it reduces recorded training time by 47.2% and peak memory by 37.8%. Compared with AG$^2$aussian, it requires 26 additional minutes of training while reducing peak memory by 46.0%.
These efficiency differences can be partly explained by each method’s semantic representation and optimization strategy. SSCL restricts each anchor’s semantic assignment to the leaves within its root, reducing the matching and intermediate-tensor costs associated with Unified-Lift’s global entries. Cross-root connections receive sparse updates, further limiting their computational overhead. AG$^2$aussian learns low-dimensional anchor features and subsequently clusters them, which helps explain its shorter training time. Its anchor graph supports feature propagation among neighboring anchors, while graph storage and propagation contribute to memory usage.
Overall, SSCL has a moderately longer runtime than AG$^2$aussian but requires substantially less GPU memory, while outperforming Unified-Lift in both runtime and memory consumption.
How sensitive are results to the number of roots and leaves?
We assessed sensitivity to R and L using four additional configurations per scene, varying the overall model capacity by approximately ±30% around the default setting. The results are reported below:
| Scene | R | L | Click mIoU | Delta vs. default |
|---|---|---|---|---|
| Colosseum | 64 | 24 | 0.800 | default |
| Colosseum | 64 | 16 | 0.791 | -0.009 |
| Colosseum | 64 | 32 | 0.804 | +0.004 |
| Colosseum | 48 | 24 | 0.772 | -0.028 |
| Colosseum | 80 | 24 | 0.811 | +0.011 |
| CitySample | 128 | 24 | 0.868 | default |
| CitySample | 128 | 16 | 0.836 | -0.032 |
| CitySample | 128 | 32 | 0.871 | +0.003 |
| CitySample | 96 | 24 | 0.811 | -0.057 |
| CitySample | 160 | 24 | 0.854 | -0.014 |
Reducing either or consistently degrades performance on both scenes, indicating the importance of sufficient local capacity for capturing instance-level information. The effect of increasing capacity, however, is not always positive. On CitySample, increasing slightly reduces performance, potentially because excessive partitioning complicates cross-block fusion and increases the risk of incorrect associations. Despite these variations, the lowest-performing configuration on each scene still surpasses the strongest baseline (0.772 vs. 0.759 on Colosseum and 0.811 vs. 0.582 on CitySample). These results support a robust performance advantage across the tested configurations.
Thank the authors for their rebuttal. The rebuttal has addressed my concerns, and I would like to keep my score.
Author Response to Reviewer Ny42 (Part 1)
Thank you for your valuable feedback. We have addressed each comment below and appreciate your thorough review.
Thank you for suggesting a cross-view ID consistency evaluation. To address this concern, we introduce a new evaluation benchmark based on Horizon-GS synthetic scenes to quantify instance identity preservation both across viewpoints (aerial--street) and within individual domains (aerial--aerial and street--street). Annotation groups are constructed by associating physical-object masks using raw depth and camera geometry, followed by manual verification of uncertain correspondences. The resulting benchmark contributes 5,826 additional directed source--target validation pairs, including 1,354 in Colosseum, 1,188 in Elvenruin, and 3,284 in CitySample.
To assess cross-view identity consistency with the new annotations, we design a dedicated protocol for all three settings. Within each group, every eligible view serves once as the source, and all methods use the same fixed click inside the annotated mask to select their instance IDs. Each method then holds its selected ID fixed and renders the complete 2D mask in all other eligible views. Results are reported below:
These results demonstrate that our approach still achieves substantial advantages in cross-view identity consistency. We will incorporate this key finding into the revised manuscript.
We have completed a controlled single-click evaluation on Colosseum, CitySample, Elvenruin, Road, and Park. For every annotated instance, one point is chosen deterministically at the deepest interior location of its ground-truth mask, independently of every method's prediction. The table below compares that fixed single click with the original best-of-three selection on exactly the same frozen samples.
Under the single-click evaluation protocol, all evaluated methods exhibit a slight performance drop. However, the relative performance differences remain largely unchanged, and our overall performance advantage is consistently maintained.