acceptodds
Under review as a conference paper at ICLR 2027

SPOR: Semantics-Preserving Ordinal Refinement for Open-Vocabulary Panoptic Segmentation

Abstract

Open-vocabulary panoptic segmentation requires recognizing categories unseen during training while accurately separating object instances and delineating coherent stuff regions. However, masks generated from appearance and semantic cues alone may erroneously merge semantically similar regions at different depth layers. We introduce SPOR (Semantics-Preserving Ordinal Refinement), a framework that uses within-image depth ordering to constrain spatial grouping in open-vocabulary panoptic segmentation. SPOR converts monocular depth estimates into normalized ordinal ranks, yielding a representation invariant to strictly increasing transformations of depth values. Rather than equating depth continuity with segment identity, it applies ordinal geometry selectively throughout mask formation. Ordinal affinities regulate local and cross-scale feature aggregation, while query-conditioned boundary routing uses reliable ordinal intervals to regulate cross-attention at uncertain boundaries while protecting confident mask cores. A training-only partition consistency objective combines ground-truth segment relations with ordinal gaps and cross-scale agreement to encourage within-segment coherence and cross-segment separation. Direct geometry-dependent modifications are confined to the mask-formation pathway, leaving pretrained semantic embeddings and the visual feature used for semantic pooling unchanged. Built on ODISE and fine-tuned on COCO Panoptic, SPOR achieves 55.5 PQ on COCO and 22.8 PQ in zero-shot transfer to ADE20K, while retaining broadly comparable performance across five open-vocabulary semantic segmentation benchmarks. These results suggest that ordinal scene structure can complement semantic recognition by constraining how pixels are grouped.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Open-Vocabulary Panoptic Segmentation, Ordinal Geometric Reasoning, Diffusion Models, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8bywOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPOR, a semantics-preserving ordinal refinement framework for open-vocabulary panoptic segmentation. It leverages ordinal depth information derived from monocular pseudo-depth estimates to guide feature aggregation, attention routing, and region partitioning. Experiments show improvements in segmentation performance while maintaining open-vocabulary generalization.

Critical strengths
  1. Well-motivated approach. Incorporating ordinal depth provides a reasonable way to improve spatial understanding in open-vocabulary panoptic segmentation.
  2. Clear presentation. The paper is well organized and clearly written, making the motivation, methodology, and experimental results easy to follow.
Critical weaknesses
  1. Limited performance improvements. Although the proposed method achieves improvements over the baseline, the gains are relatively marginal considering the additional complexity introduced by multiple modules.
  2. Insufficient qualitative analysis. More qualitative comparisons are needed, particularly for semantically similar instances at different depths and challenging occlusion scenarios, to better demonstrate the benefits of ordinal depth refinement.
Questions for authors
  1. Stability of performance improvements. The performance gains over baseline are relatively small. Results across multiple random seeds would help verify the stability of these improvements.
  2. Qualitative analysis. More visual comparisons in challenging scenarios, particularly involving semantically similar objects at different depths or complex occlusions, would better demonstrate the advantages of the proposed method.
  3. Robustness analysis. The paper lacks a systematic analysis of depth estimation errors. Evaluating the impact of noisy or inaccurate depth predictions on segmentation performance would help assess the robustness of the method.
  4. Effectiveness of ordinal depth. A comparison with simpler depth representations, such as normalized depth values or direct depth feature fusion, would help clarify the specific benefits of the proposed ordinal depth representation.
5 comments

Thank you for your detailed feedback.

To Q1:

We added two random seeds and evaluated all three seeds under the same predicted-depth inference setting. Tables A–B report the complete results. COCO PQ is 55.500 ± 0.048, with all three seeds outperforming ODISE, supporting the consistency of this modest gain across the evaluated runs. ADE20K PQ is 22.623 ± 0.216, indicating mean performance comparable to ODISE (22.6). These statistics are mean ± sample standard deviation. We will include the complete per-seed results and summary statistics in the revised manuscript.

While conducting the additional runs and checking the evaluation pipeline, we found that all original evaluations except COCO used RGB-only inference, bypassing ODFA and QOBR. This was inconsistent with the predicted-depth inference setting described in the manuscript. We will correct the main result tables and related analyses to consistently report predicted-depth inference results, and provide paired results from the same checkpoint under both inference settings in the appendix. The paired evaluation shows that predicted-depth inference improves mean PQ and AP on COCO and ADE20K while maintaining broadly comparable semantic segmentation performance. This supports ordinal geometry as a complementary cue for spatial grouping and instance prediction at test time. Accordingly, the revised main results will consistently use predicted-depth inference. The configuration without inference-time depth also retains comparable mean performance, offering flexibility in inference configuration. We thank the reviewer for prompting us to verify and correct this issue.

Table A: Multi-seed panoptic segmentation results of SPOR with predicted depth at inference on COCO and ADE20K, alongside the ODISE reference. Summary statistics are mean ± sample standard deviation across three seeds.

Method / SeedCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
ODISE55.446.065.222.614.429.9
SPOR (seed=22)55.44646.20565.28722.50414.17529.760
SPOR (seed=42)55.53446.22465.33022.87314.40529.471
SPOR (seed=62)55.52146.04265.45322.49414.24729.479
SPOR (Mean ± SD)55.500 ± 0.04846.157 ± 0.10065.357 ± 0.08622.623 ± 0.21614.276 ± 0.11729.570 ± 0.165

Table B: Multi-seed open-vocabulary semantic segmentation results of SPOR with predicted depth at inference, alongside the ODISE reference. Scores are mIoU (%); summary statistics are mean ± sample standard deviation across three seeds.

Method / SeedA-847PC-459A-150PC-59PAS-21
ODISE11.114.529.957.384.6
SPOR (seed=22)10.93714.61929.76057.10285.081
SPOR (seed=42)11.37115.22229.47156.90684.789
SPOR (seed=62)10.99814.48829.47956.97784.795
SPOR (Mean ± SD)11.102 ± 0.23514.776 ± 0.39229.570 ± 0.16556.995 ± 0.09984.888 ± 0.167

To Q2:

We added qualitative comparisons in the appendix (Figures 4–6), covering challenging scenes with same-class objects at different depths, partial occlusion, and overlapping contours. Compared with the baseline, SPOR delineates object boundaries more accurately (Figure 4) and separates same-class instances more clearly (Figure 5) in these examples. Figure 6 further presents representative results for instance assignment near occlusion boundaries and dense small-object segmentation.

We welcome further discussion if any concerns remain.

Thank you for your detailed feedback.

To Q3:

SPOR uses predicted depth from Depth Anything V2 as auxiliary information. To quantify the effect of depth noise and inaccuracies, we conducted depth perturbation experiments with the trained SPOR model, keeping RGB inputs and model weights fixed. All evaluations use Depth Anything V2 predictions: the unperturbed reference uses the original predictions, while the perturbed settings apply Gaussian noise or Gaussian blur to assess segmentation stability under degraded depth inputs.

We apply perturbations after the standard inference resizing and before ordinal-depth processing: (1) pixel-wise, zero-mean Gaussian noise with standard deviation σₙ = 0.10 or 0.20 is added to depth maps normalized to [0,1], followed by clipping to [0,1]. Both noise levels share the same per-image noise pattern, differing only in magnitude. (2) A 5 × 5 Gaussian kernel with spatial standard deviation τ = 1 pixel simulates degradation of depth boundaries and local details, without additional random noise. We completed 18 full evaluations across six evaluation tasks.

Under the tested noise and blur conditions, SPOR performs similarly to the unperturbed reference. The maximum absolute PQ changes are only 0.074 and 0.045 percentage points on COCO and ADE20K, respectively; the maximum absolute mIoU change across the five semantic segmentation benchmarks is only 0.151 percentage points. These small changes provide quantitative evidence of SPOR’s robustness to the tested depth noise and local smoothing.

Table C: Effect of depth perturbations on SPOR’s segmentation performance on COCO and ADE20K.

SettingCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
SPOR (σₙ = 0)55.53446.22465.33022.87214.40529.471
Gaussian Noise (σₙ = 0.10)55.50346.28365.36622.83414.39429.612
Gaussian Noise (σₙ = 0.20)55.46046.22265.31922.82714.43629.568
Gaussian Blur (5 × 5, τ = 1 px)55.51746.22665.32322.85514.40729.482

Table D: Effect of depth perturbations on SPOR’s cross-dataset semantic segmentation performance.

SettingA-847PC-459A-150PC-59PAS-21
SPOR (σₙ = 0)11.37115.22229.47156.90684.789
Gaussian Noise (σₙ = 0.10)11.22015.21029.61256.92884.783
Gaussian Noise (σₙ = 0.20)11.28715.22929.56856.92184.808
Gaussian Blur (5 × 5, τ = 1 px)11.31515.22929.48256.91284.793

We welcome further discussion if any concerns remain.

Thank you for your detailed feedback.

To Q4:

We added a comparison with normalized depth to assess the effectiveness of SPOR’s ordinal-depth representation. Normalization standardizes the value range but retains numerical differences between predicted depth values. SPOR instead converts depth into normalized within-image ranks, making geometric modeling depend on relative ordering rather than specific depth magnitudes.

As shown in Tables E–F, using normalized depth reduces performance on nine of the ten metrics. Ordinal depth therefore yields better overall segmentation performance in these experiments, supporting SPOR’s use of relative ranks to organize geometric information and assist segmentation.

Table E: Segmentation performance comparison between ordinal and normalized depth on COCO and ADE20K.

SettingCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
Ordinal depth55.53446.22465.33022.87314.40529.471
Normalized depth55.39045.91065.20022.52114.22329.758

Table F: Performance comparison between ordinal and normalized depth on five semantic segmentation benchmarks (mIoU).

SettingA-847PC-459A-150PC-59PAS-21
Ordinal depth11.37115.22229.47156.90684.789
Normalized depth11.00914.51929.75856.85084.496

We welcome further discussion if any concerns remain.

Reviewer_3JUPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors tackle open-vocabulary panoptic segmentation, aiming to improve the separation of objects and regions using estimated depth from Depth Anything v2. The proposed method uses relative depth ordering to guide mask prediction, adding several refinement modules and an auxiliary training loss to ODISE. Evaluation across seven datasets results in marginal to no improvement over the baseline.

Critical strengths

Open-vocabulary panoptic segmentation is an important research problem, and using depth guidance to distinguish individual instances is a reasonable direction.

Critical weaknesses

W1: Extremely limited contribution. The method adds substantial engineering to ODISE without demonstrating a convincing benefit. In the five-dataset semantic comparison, SPOR is worse than ODISE on A-150 (29.6 vs. 29.9 mIoU) and PC-59 (56.9 vs. 57.3), and unchanged on A-847. Improvements occur only on PC-459 and PAS-21. Together with the approximately 0.1/0.2-point COCO/ADE20K PQ gains and mixed component ablations, these results provide insufficient justification for the additional modules, loss, and depth-estimation requirement.

W2: Poor presentation. The paper introduces numerous modules, thresholds, and weighting rules without building a clear argument for why this particular combination is necessary. Important definitions (e.g. the query reliability score and routing schedule) remain unspecified. The method needs a substantially clearer explanation and stronger analysis connecting its design choices to the claimed grouping improvements. In general the paper is quite unstructured and hard to follow.

Questions for authors

Q1: Where does SPOR help, and where does it hurt? Please show representative cases where it improves, leaves unchanged, or degrades ODISE’s prediction, alongside ground truth and estimated depth. Do the improvements actually address the motivating problem of merging semantically similar regions at different depths?

Q2: What causes the failures? In unsuccessful cases, is the depth estimate incorrect, does the ordinal representation provide insufficient separation, or does the geometric constraint conflict with a valid object mask.

7 comments

Thank you for the detailed feedback.

To Q1:

We added cases where SPOR improves, largely preserves, or degrades ODISE's predictions, plus instance separation analysis by ordinal-depth gap on the full COCO/ADE20K validation sets. These show improved separation of adjacent same-class instances at different depths, addressing our motivation, but also incorrect splits or local mask omissions under some complex occlusions.

Qualitative analysis. Appendix Figures 4–5 show improved boundaries and instance separation, respectively; Figure 6 shows improvements in boundaries, region assignment under occlusion, and dense small-object recognition. Figures 7–8 show largely unchanged and degraded predictions. In Figure 5, estimated depth captures the front–back ordering of two sheep; SPOR separates them where ODISE merges them, better matching GT while retaining both "sheep" labels. In Figure 6, SPOR separates a foreground arm from a background person. Both address same-class merging across depths. Figure 7 largely preserves the main donut and person instance partitions. Figure 8 shows extra plant fragments and missing person-mask regions around overlapping legs, indicating possible incorrect geometric adjustments under complex occlusion or insufficient geometric cues.

Quantitative analysis. To test whether gains extend beyond individual examples, we construct adjacent, same-class, non-crowd thing pairs from the full COCO/ADE20K validation GT. Each instance has at least 100 pixels and 64 depth samples; minimum mask distance is at most 2% of the image diagonal, yielding 14,886/9,041 pairs. We convert estimated depth to continuous within-image ordinal values and define each pair's gap as the absolute difference between its GT instance medians. Within each dataset, pairs are sorted by gap and split into approximately equal-sized Small, Medium, and Large groups.

Table A shows higher overall correct separation and fewer merge errors in every gap group on both datasets; both metrics improve in the Large group. This supports using depth differences to distinguish semantically similar instances and mitigate same-class merging across depths.

Fewer merge errors do not guarantee higher correct separation in every group: ADE20K Small-group separation falls from 24.851% to 24.618%. Overall same-class split errors rise from 2.917% to 3.097% on COCO but fall from 1.704% to 1.622% on ADE20K. Together with degraded examples, this shows that local separation gains can accompany incorrect splits or mask omissions.

Table A: Instance separation by ordinal-depth gap on COCO and ADE20K. CS: correct separation; ME: merge error; SS: same-class split error.

GapMethodCOCO CS ↑COCO ME ↓COCO SS ↓ADE20K CS ↑ADE20K ME ↓ADE20K SS ↓
SmallODISE55.28010.2183.66524.85125.0171.754
SPOR56.44910.0363.80924.61824.4861.674
MediumODISE60.7826.8123.48123.59825.5232.094
SPOR61.4876.2273.64423.79723.8962.042
LargeODISE71.5444.4342.88733.11222.4621.590
SPOR72.4913.9503.02933.61021.9311.443
OverallODISE62.5357.1542.91727.18724.3341.704
SPOR63.4766.7383.09727.34223.4381.622

CS and ME rates are computed over instance pairs; SS rates over deduplicated GT instances within each group.

Table B shows nearly unchanged COCO thing PQ and a 0.114-point stuff PQ gain; ADE20K thing/stuff PQ improve by 0.188/0.148 percentage points. Modest overall PQ gains are consistent with the coexistence of local improvements, unchanged predictions, and degradations. Together with the separation statistics and same-class examples, these results support partial mitigation of the merging problem addressed in this work.

Table B: Thing/stuff panoptic quality on COCO and ADE20K (%).

MethodCOCO PQ_th ↑COCO PQ_st ↑ADE20K PQ_th ↑ADE20K PQ_st ↑
ODISE61.34446.47221.24525.670
SPOR61.34946.58621.43325.818

We welcome further discussion if any concerns remain.

Thank you for the detailed feedback.

To Q2:

Thank you for the suggestion. We distinguish two failure types: persistent merges that remain uncorrected, and new merges or splits relative to ODISE. To investigate their mechanisms, we conducted inference-pipeline diagnostics and ordinal interventions on a fixed failure cohort from COCO val2017, examining candidate separability, ordinal gaps, and within-instance geometric variation. The cohort contains 18 newly merged pairs, 861 persistently merged pairs, and 67 newly split instances, spanning 570 unique images.

Candidate separability is an important bottleneck for persistent merges. Among the 861 persistently merged pairs, 601 (69.8%) lack masks that separately match the two GT instances at the raw query stage: no two distinct candidates each achieve IoU > 0.5 with their respective GT instances. The 18 newly merged pairs show similar stage-specific differences: 11 lack separable candidates; 6 have correctly classified candidates that pass filtering but fail to separate during final competition, overlap handling, or assignment; and 1 involves class or background decisions. Thus, final merge errors involve both candidate representation and subsequent assignment and cannot be attributed to insufficient depth gaps from the outputs alone.

To directly test the role of ordinal gaps, we shift the two targets’ ordinal values by 0.05 in opposite directions, increasing their median gap by 0.10 while preserving each target’s internal ordinal variation. After excluding infeasible cases, such as out-of-range values or ordering ties, and targets whose baseline failures were not reproduced, 543 persistently merged and 15 newly merged pairs remain, totaling 558 pairs. None recover correct separation (0/558). Correct separation requires each GT instance to match a final prediction with the correct class, IoU > 0.5, and a distinct instance ID. Under the tested conditions, increasing ordinal contrast alone is therefore insufficient to restore separation; candidate separability and subsequent assignment are also limiting factors.

Some new split errors are sensitive to within-instance ordinal variation. A single object can span a substantial depth range, so internal geometric variation need not indicate an instance boundary. We preserve the target instance’s ordinal median and all inputs outside it, while reducing its internal ordinal deviations to 50%. This makes 3/67 instances meet the criterion of “correct class, IoU > 0.5, and no longer triggering a same-class split error.” This intervention provides evidence of a conflict between within-instance geometric variation and mask coherence in a few cases.

Regarding depth estimation, SPOR’s continuous ordinal representation reduces dependence on specific depth values and scale but retains the input’s near–far ordering and cannot itself correct ordering errors. We did not introduce independent ground-truth depth or manually annotated near–far relations in this experiment, so ordering errors remain a possible factor.

Together, the stage-level diagnostics and interventions indicate that persistent merges are constrained by candidate separability and subsequent assignment, while a few new split errors are sensitive to within-instance ordinal variation. These findings clarify SPOR’s limitations and show that geometric cues must work together with effective candidates and instance integrity.

These analyses diagnose failure mechanisms in a predefined, fixed failure cohort; overall performance is evaluated on the full validation sets. As shown in Q1, SPOR improves overall correct separation by 0.941/0.155 percentage points and reduces structural merge error rates by 0.416/0.896 percentage points on COCO/ADE20K, respectively, with modest thing/stuff PQ gains. Thus, despite local failures and over-segmentation risks, SPOR still provides overall separation gains on the full validation sets.

We welcome further discussion if any concerns remain.

Thank you for the detailed feedback.

To W1:

We added two seeds and evaluated all three with predicted-depth inference (Tables A–B). COCO PQ is 55.500 ± 0.048; all seeds exceed ODISE, supporting a stable, modest gain. Full per-seed statistics will be included in the revision.

We found that original non-COCO evaluations used RGB-only inference, bypassing ODFA/QOBR, contrary to the manuscript. We will correct the main tables/analyses and report same-checkpoint paired results with/without depth in the appendix. Predicted depth improves mean COCO/ADE20K PQ/AP with broadly comparable semantic performance, supporting ordinal geometry for test-time grouping and instance prediction. Without depth, performance remains comparable, offering flexibility. We thank the reviewer for prompting this correction.

Table A: Predicted-depth panoptic results; SPOR mean ± sample SD over three seeds.

MethodCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
ODISE55.446.065.222.614.429.9
SPOR (Mean ± SD)55.500±0.04846.157±0.10065.357±0.08622.623±0.21614.276±0.11729.570±0.165

Table B: Predicted-depth semantic results, mIoU (%); SPOR mean ± sample SD over three seeds.

MethodA-847PC-459A-150PC-59PAS-21
ODISE11.114.529.957.384.6
SPOR (Mean ± SD)11.102±0.23514.776±0.39229.570±0.16556.995±0.09984.888±0.167

Beyond cumulative ablation, removing ODFA/QOBR individually reduces COCO/ADE20K PQ by 0.09/0.11 and 0.19/0.23, respectively (Table C; same seed and predicted-depth protocol). ROPC's partition-consistency supervision complements feature aggregation and query routing: with ODFA/QOBR enabled, it adds 0.12/0.04 COCO PQ/AP, 0.09 ADE20K AP, and 0.37/0.19 A-847/PC-459 mIoU. Despite lower ADE20K PQ (22.95→22.87), these gains without inference overhead support its distinct contribution and retention.

Table C: SPOR component ablation. ✓/–: enabled/disabled. Semantic scores are mIoU (%); A-150 also reports ADE20K mIoU.

ODFAQOBRROPCCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APA-847PC-459A-150PC-59PAS-21
–––55.1545.9265.3022.2614.0011.1414.1029.7456.7784.75
✓––55.3246.0365.3622.8614.0611.0914.5329.6356.8284.64
✓✓–55.4146.1865.3122.9514.3111.0015.0329.6356.9484.79
–✓✓55.4446.0465.1322.7614.2210.8514.6829.6656.9884.69
✓–✓55.3446.2065.2322.6414.1910.9414.6629.4956.7784.96
✓✓✓55.5346.2265.3322.8714.4011.3715.2229.4756.9184.79

Normalized depth unifies the range but retains numerical intervals; SPOR uses normalized within-image ranks, avoiding dependence on depth magnitudes. Normalized depth lowers 9/10 metrics (Table D), supporting relative-rank geometry for segmentation.

Table D: Ordinal vs. normalized depth. Semantic scores are mIoU (%); A-150 also reports ADE20K mIoU.

DepthCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APA-847PC-459A-150PC-59PAS-21
Ordinal55.53446.22465.33022.87314.40511.37115.22229.47156.90684.789
Normalized55.39045.91065.20022.52114.22311.00914.51929.75856.85084.496

We welcome further discussion if any concerns remain.

Reviewer_WnzNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SPOR (Semantics-Preserving Ordinal Refinement), a framework that incorporates relative depth information into open-vocabulary panoptic segmentation to improve spatial grouping and instance separation. The method addresses the limitation of appearance- and semantics-driven segmentation, where semantically similar objects at different depth layers may be incorrectly merged. Instead of relying on metric depth values, SPOR converts monocular depth predictions into normalized ordinal ranks, providing a representation invariant to strictly increasing transformations of depth.

Built upon ODISE, SPOR introduces three components to integrate ordinal geometry into mask formation. Ordinal Depth Feature Aggregation (ODFA) uses depth-based affinities to regulate local and cross-scale feature aggregation. Query-Conditioned Ordinal Boundary Routing (QOBR) adjusts cross-attention using query-specific depth intervals while preserving confident mask regions. Reliable Ordinal Partition Consistency (ROPC) introduces a training-only objective that encourages consistent pixel grouping based on ground-truth segment relationships and ordinal depth differences. The framework restricts direct geometry-dependent modifications to the mask prediction pathway, leaving pretrained semantic representations unchanged.

The authors evaluate SPOR on COCO Panoptic and zero-shot ADE20K panoptic segmentation, together with five open-vocabulary semantic segmentation benchmarks. Results show modest improvements over ODISE, achieving 55.5 PQ on COCO and 22.8 PQ on ADE20K, while maintaining broadly comparable semantic segmentation performance across datasets. Component-wise ablations further demonstrate incremental improvements from the proposed modules.

Critical strengths

(1) The paper addresses an important limitation of open-vocabulary panoptic segmentation by incorporating geometric cues to distinguish semantically similar but spatially separated regions. The use of ordinal depth provides an intuitive way to introduce structural information without depending on metric depth estimation. (2) The three proposed modules incorporate ordinal geometry at complementary stages of mask formation, including feature aggregation, query attention, and partition supervision. The separation between geometric refinement and semantic feature extraction is also a reasonable design choice for preserving open-vocabulary recognition capabilities. (3) The paper evaluates both in-domain panoptic segmentation and cross-dataset open-vocabulary transfer, covering multiple semantic label spaces. The ablation studies and qualitative comparisons provide preliminary evidence that ordinal geometry can improve mask prediction while maintaining competitive semantic recognition performance.

Critical weaknesses

(1) Despite introducing three additional components, the improvements over the original ODISE baseline are relatively small. As shown in Table 1, SPOR improves PQ by only 0.1 and 0.2 percentage points on COCO and ADE20K, respectively, while ADE20K mIoU decreases slightly. Although Table 3 reports larger improvements over the matched component-free variant, the absolute gains remain modest. The paper does not report results across multiple random seeds or provide statistical variance, making it difficult to determine whether these improvements are robust and statistically meaningful. (2) Although Table 3 progressively introduces ODFA, QOBR, and ROPC, the ablation study only evaluates cumulative configurations rather than independently removing each component from the complete framework. Therefore, it is difficult to determine the individual contribution of each module and whether their improvements depend on interactions with other components. In particular, ROPC produces only a small additional improvement on COCO PQ and no visible improvement on ADE20K PQ. More controlled ablations would help establish whether all three components are necessary and complementary. (3) The main motivation of SPOR is to prevent incorrect merging of semantically similar regions across different depth layers. However, the evaluation primarily relies on aggregate PQ, AP, and mIoU, without directly analyzing instance separation, boundary accuracy, or performance on objects with significant depth discontinuities. As a result, the experiments do not clearly demonstrate whether the improvements originate from the specific structural failures that motivate the method. Furthermore, although the semantic feature pathway remains unchanged, several open-vocabulary semantic segmentation results decrease slightly, suggesting that preserving semantic features does not necessarily guarantee unchanged semantic predictions.

Questions for authors

(1) Given the relatively small PQ improvements over ODISE, could the authors report results across multiple random seeds or provide statistical variance for the main experiments? In particular, are the observed improvements on COCO and ADE20K consistently larger than the variation introduced by training randomness? (2) Could the authors provide independent ablations of ODFA, QOBR, and ROPC by removing each component from the complete framework? In particular, given the negligible additional PQ improvement from ROPC on ADE20K, what evidence supports its necessity? (3) Could the authors provide more direct evidence that SPOR improves instance separation and boundary prediction, particularly for semantically similar objects at different depth layers? For example, a breakdown of thing/stuff segmentation quality or boundary-specific performance would help substantiate the proposed motivation. Additionally, could the authors clarify how semantic preservation is evaluated beyond maintaining the original semantic feature pathway, given the slight performance decreases on several open-vocabulary benchmarks?

3 comments

Thank you for your detailed feedback.

To Q1:

We added two random seeds and evaluated all three seeds under the same predicted-depth inference setting. Tables A–B report the complete results. COCO PQ is 55.500 ± 0.048, with all three seeds outperforming ODISE, supporting the consistency of this modest gain across the evaluated runs. ADE20K PQ is 22.623 ± 0.216, indicating mean performance comparable to ODISE (22.6). These statistics are mean ± sample standard deviation. We will include the complete per-seed results and summary statistics in the revised manuscript.

While conducting the additional runs and checking the evaluation pipeline, we found that all original evaluations except COCO used RGB-only inference, bypassing ODFA and QOBR. This was inconsistent with the predicted-depth inference setting described in the manuscript. We will correct the main result tables and related analyses to consistently report predicted-depth inference results, and provide paired results from the same checkpoint under both inference settings in the appendix. The paired evaluation shows that predicted-depth inference improves mean PQ and AP on COCO and ADE20K while maintaining broadly comparable semantic segmentation performance. This supports ordinal geometry as a complementary cue for spatial grouping and instance prediction at test time. Accordingly, the revised main results will consistently use predicted-depth inference. The configuration without inference-time depth also retains comparable mean performance, offering flexibility in inference configuration. We thank the reviewer for prompting us to verify and correct this issue.

Table A: Multi-seed panoptic segmentation results of SPOR with predicted depth at inference on COCO and ADE20K, alongside the ODISE reference. Summary statistics are mean ± sample standard deviation across three seeds.

Method / SeedCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
ODISE55.446.065.222.614.429.9
SPOR (seed=22)55.44646.20565.28722.50414.17529.760
SPOR (seed=42)55.53446.22465.33022.87314.40529.471
SPOR (seed=62)55.52146.04265.45322.49414.24729.479
SPOR (Mean ± SD)55.500 ± 0.04846.157 ± 0.10065.357 ± 0.08622.623 ± 0.21614.276 ± 0.11729.570 ± 0.165

Table B: Multi-seed open-vocabulary semantic segmentation results of SPOR with predicted depth at inference, alongside the ODISE reference. Scores are mIoU (%); summary statistics are mean ± sample standard deviation across three seeds.

Method / SeedA-847PC-459A-150PC-59PAS-21
ODISE11.114.529.957.384.6
SPOR (seed=22)10.93714.61929.76057.10285.081
SPOR (seed=42)11.37115.22229.47156.90684.789
SPOR (seed=62)10.99814.48829.47956.97784.795
SPOR (Mean ± SD)11.102 ± 0.23514.776 ± 0.39229.570 ± 0.16556.995 ± 0.09984.888 ± 0.167

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

Thank you for your detailed feedback.

To Q2:

In addition to the original cumulative ablation, we added experiments that individually remove ODFA and QOBR from the full SPOR model, as shown in Tables C–D. For a fair comparison, all configurations use the same random seed and predicted-depth inference protocol. Removing ODFA reduces COCO/ADE20K PQ by 0.09/0.11, respectively; removing QOBR reduces them by 0.19/0.23. These comparisons support the contributions of both modules within the full framework.

ROPC complements ODFA’s feature aggregation and QOBR’s query routing through direct partition-consistency supervision. With ODFA and QOBR already enabled, adding ROPC further improves COCO PQ/AP by 0.12/0.04, ADE20K AP by 0.09, and A-847/PC-459 mIoU by 0.37/0.19. This indicates that the other two modules do not fully replace its contribution. Although ADE20K PQ does not improve (22.95 → 22.87), these additional gains, without extra inference overhead, provide empirical support for retaining ROPC in the full framework.

Table C: Component ablation of SPOR on COCO and ADE20K.

ConfigurationODFAQOBRROPCCOCO PQCOCO APCOCO mIoUADE20K PQADE20K APADE20K mIoU
SPOR w/o ODFA, QOBR, ROPC–––55.1545.9265.3022.2614.0029.74
SPOR w/o QOBR, ROPC✓––55.3246.0365.3622.8614.0629.63
SPOR w/o ROPC✓✓–55.4146.1865.3122.9514.3129.63
SPOR w/o ODFA–✓✓55.4446.0465.1322.7614.2229.66
SPOR w/o QOBR✓–✓55.3446.2065.2322.6414.1929.49
SPOR✓✓✓55.5346.2265.3322.8714.4029.47

“w/o” denotes “without.”

Table D: Component ablation of SPOR on five open-vocabulary semantic segmentation benchmarks. All scores are mIoU (%).

ConfigurationODFAQOBRROPCA-847PC-459A-150PC-59PAS-21
SPOR w/o ODFA, QOBR, ROPC–––11.1414.1029.7456.7784.75
SPOR w/o QOBR, ROPC✓––11.0914.5329.6356.8284.64
SPOR w/o ROPC✓✓–11.0015.0329.6356.9484.79
SPOR w/o ODFA–✓✓10.8514.6829.6656.9884.69
SPOR w/o QOBR✓–✓10.9414.6629.4956.7784.96
SPOR✓✓✓11.3715.2229.4756.9184.79

“w/o” denotes “without.”

Please let us know if any concerns remain unaddressed; we are happy to discuss them.

Thank you for your detailed feedback.

To Q3:

We added instance separation analysis by ordinal-depth gap, a thing/stuff breakdown, and boundary evaluation, and clarified semantic preservation evaluation.

(1) Instance separation and boundary prediction.

From the full COCO/ADE20K validation GT, we select adjacent pairs of same-class, non-crowd thing instances, each with at least 100 pixels and 64 depth samples. The minimum mask distance is at most 2% of the image diagonal, yielding 14,886/9,041 pairs. Estimated depth is converted to continuous within-image ordinal values; each pair's depth gap is the absolute difference between its GT instance medians. Within each dataset, pairs are sorted by gap and split into approximately equal-sized Small, Medium, and Large groups.

Table E shows higher overall correct separation and fewer merge errors in every gap group on both datasets; both metrics improve in the Large group. This aligns with using depth differences to distinguish semantically similar instances and supports mitigating same-class merging across depths. Revised appendix Figures 4–5 illustrate improved contours and boundaries between occluded instances, including bananas, sinks, trains, and sheep.

Table E: Instance separation by ordinal-depth gap on COCO and ADE20K. CS: correct separation; ME: merge error; SS: same-class split error.

GapMethodCOCO CS ↑COCO ME ↓COCO SS ↓ADE20K CS ↑ADE20K ME ↓ADE20K SS ↓
SmallODISE55.28010.2183.66524.85125.0171.754
SPOR56.44910.0363.80924.61824.4861.674
MediumODISE60.7826.8123.48123.59825.5232dent.094
SPOR61.4876.2273.64423.79723.8962.042
LargeODISE71.5444.4342.88733.11222.4621.590
SPOR72.4913.9503.02933.61021.9311.443
OverallODISE62.5357.1542.91727.18724.3341.704
SPOR63.4766.7383.09727.34223.4381.622

Table F shows nearly unchanged COCO thing PQ and a 0.114-point gain in stuff PQ. ADE20K thing/stuff PQ increase by 0.188/0.148 percentage points. These results further support partial alleviation of the instance separation problem addressed here.

Table F: Thing/stuff panoptic quality on COCO and ADE20K (%).

MethodCOCO PQ_th ↑COCO PQ_st ↑ADE20K PQ_th ↑ADE20K PQ_st ↑
ODISE61.34446.47221.24525.670
SPOR61.34946.58621.43325.818

(2) Semantic preservation.

We evaluate cross-dataset zero-shot semantic segmentation on five open-vocabulary benchmarks. SPOR is fine-tuned only on COCO. At test time, we switch to the target class vocabulary, retaining ODISE's text prompts and classification fusion rules, without training or fine-tuning on target annotations. Semantic transfer is measured by final pixel-prediction mIoU.

Under the same predicted-depth inference setting, three-seed means are comparable to ODISE on A-847, higher on PC-459/PAS-21, and slightly lower on A-150/PC-59, indicating preserved overall open-vocabulary transfer capability. “Semantic preservation” means retaining pretrained semantic representations and the original semantic feature pathway. Final classification still depends on predicted masks and decoder queries, so predictions or metrics need not remain unchanged on every benchmark. We welcome further discussion if any concerns remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.