acceptodds
Withdrawn from ICLR 2027

SEER: Semantic Evidence for Executable Region Propagation in Open-Vocabulary Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free open-vocabulary semantic segmentation (OVSS) has made strong progress by adapting CLIP-like representations to dense prediction, but its remaining errors are often local, structured, and tied to the active label space. A successful correction must be visually plausible, legal under the target vocabulary, and executable in the segmentation score field. We propose SEER, a training-free test-time repair layer for Semantic Evidence and Executable Region propagation. SEER casts post-prediction OVSS repair as evidence-conditioned label propagation. The OVSS score space proposes compact candidate labels, a frozen VLM verifies closed-set regional evidence, and an execution gate certifies whether the evidence is legally and semantically actionable. The accepted label is then propagated by an OVSS-side policy: coherent high-confidence regions allow broad propagation, while object- or boundary-risk cases use conservative score-grounded propagation. Across eight standard OVSS benchmarks, SEER reaches 58.0 average mIoU without updating OVSS weights. Matched-input controls isolate the repair mechanism, while resource measurements report the cost of the complete frozen system.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Open-Vocabulary Semantic Segmentation, Multimodal Large Model, Training-free, Region Propagation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_om1POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates OVSS post-prediction repair as executable region propagation by separating regional semantic evidence, label-space validity, and pixel support. It proposes SEER, a training-free framework that combines frozen VLM verification with OVSS score-guided pixel propagation. Extensive diagnostics evaluate compact candidates, executable gates, multi-view evidence, evidence-conditioned propagation, and complete-system cost.

Critical strengths
  1. The presented figures and tables are clear and effectively illustrate the workflow of the proposed method.

  2. This paper involves a substantial amount of work and demonstrates the effectiveness of the proposed method through both qualitative and quantitative evaluations.

Critical weaknesses
  1. Although the motivation is clear, it lacks a thorough analysis of the limitations of existing methods. For example, the authors should clarify why existing OVSS predictions and region propagation methods are insufficient and establish a stronger connection between these limitations and the proposed framework.

  2. The novelty is questionable. One of the core components of SEER is Executable Region Propagation, which shares substantial similarities with the methods proposed in Lposs [1] and CorrMatch [2]. However, neither of these works is cited or discussed in the paper, which further weakens the credibility of the claimed novelty.

    [1] Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation. CVPR 2025

    [2] Corrmatch: Label propagation via correlation matching for semi-supervised semantic segmentation. CVPR 2024

  3. Most of the methods included in the comparative experiments were published in 2025, with insufficient comparisons against methods published in 2026 to demonstrate the state-of-the-art performance of the proposed method.

Questions for authors

Please refer to 'Critical Weaknesses'

Reviewer_8My8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SEER, a training-free repair framework for existing open-vocabulary semantic segmentation predictions. It selects SAM2 proposals using prediction-side signals, constructs compact candidate labels from dense OVSS scores, and uses a frozen VLM to verify regional semantics. Accepted decisions are applied through adaptive updates that either relabel the full proposal or restrict changes to a score-supported subset. The method reports improvements across eight benchmarks, achieving an average mIoU of 58.0 with a ViT-L/14 base.

Critical strengths
  1. The separation of regional semantic verification from pixel-level updates is sensible: identifying a region’s category does not necessarily justify relabeling every pixel within it.
  2. The reported improvements are substantial and consistent across benchmarks, with particularly notable gains on ADE20K and Cityscapes.
  3. SEER can be applied without updating the underlying OVSS model, and the cross-base results suggest practical value as a reusable repair layer.
Critical weaknesses
  1. The writing of this manuscript is POOR, and the equations and notation insufficiently explain how the quantities are obtained or connected, particularly Eqs. (1), (3), and (14). Moreover, Eqs. 7 and 12) use only abstract representations instead of complete calculations, which increases the difficulty of understanding. The roles of uncertainuncertain, confidencelevelsconfidence levels, and region-form categories in Eqs. (9)–(11) also need clearer definitions, especially the VLM-reported judgments. The work may offer a useful engineering contribution, but the current writing makes it difficult to assess its impact.
  2. The reliability of the VLM-reported judgments is insufficiently justified. The verifier is asked to select a label, while reporting conf={high, medium, low} and form={coherent, boundary-like, mixed, unclear}. But the criteria for these judgments are not clearly defined, and the authors do not provide analysis results to verify whether the VLM's conclusions are correct, nor do they analyze erroneous failure cases.
  3. In Sec. 3.3 and Fig. 2, the VLM sees multiple views of the same proposal. But why? The authors did not discuss this. Is it an inference strategy or a specific setting for VLM? How are these multi-view evidences generated, and what are the advantages? Although the appendix provides crop settings and Table 5 reports performance gains, the main text does not clearly explain how the views are constructed and jointly presented to the VLM.
  4. The reliance on a large VLM verifier raises concerns about deployment cost. Table 6 shows relatively limited gains with the 0.8B VLM, while substantially larger improvements require a VLM of 4B or more. Since VLM verification is central to the repair process, this overhead is difficult to avoid for samples receiving repairs.
Questions for authors
  1. SEER depends on existing OVSS predictions. Is it influenced by post-processing techniques like PAMR?
  2. Table 22 reports fewer than one call per image on several benchmarks, implying that some test images receive no verification. The authors should explain how the audit budget is allocated across images if a VLM is not used.
Reviewer_JT3SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SEER is a training-free post-prediction repair layer placed on a frozen OVSS prediction Y0Y^0 and its dense score tensor SS. A fixed SAM2 proposal bank yields candidates, a budgeted priority function selects a top-K subset, a frozen VLM (Qwen3.5-9B) returns only a regional label/confidence/form triple over a closed candidate set, and an executable gate decides whether that evidence may rewrite the label — after which the pixel support is handed back to the OVSS score field. The conceptual contribution is a "label direction × pixel projector" formalization (Props./Lemma in Appendix B) plus an unusually complete diagnostic suite. Table 1 reports 56.2 mean mIoU (ViT-B/16, CorrCLIP 51.0) and 58.0 (ViT-L/14, CorrCLIP 53.6).

Critical strengths
  • Sharp framing. "The VLM supplies label direction, the segmenter decides pixel support" is written as an explicit executable action, and the failure mode label correct, support wrong → net harm is formalized. Reading the closed candidate set as a projection rather than a shortened prompt is cleaner than most VLM-repair work.
  • Well-controlled experiments. Table 4 compares score rule, CLIP crop classifier, VLM+score support, Dense-CRF and SEER on the same frozen state, proposals and audit budget. Cross-base plug-in is positive on 4 bases × 4 benchmarks with bootstrap CIs; auditor and proposal-source transfer are both covered; candidate-order replay moves the worst benchmark by only 0.0778 mIoU.
  • Unusually candid positioning. The Table 1 footnote states plainly that this is an accuracy–cost point, not an equal-resource comparison, and the paper declines to claim domain-blind development.
Critical weaknesses
  1. The mechanism evidence and the main result do not share a prediction state. Table 2's +7.52 rests on a self-built "independently frozen diagnostic" state, which the paper itself says is "not a reproduction of the published CorrCLIP row in Tab. 1" (its initial VOC21 is 72.88 vs the published 74.8). The mechanism conclusions of Tables 2–6, 21, 28, 29 therefore cannot be transferred to the Table 1 system as it stands. In parallel, the mechanism study covers four benchmarks (stated at lines 636 and 1737) while the conclusion reads "Across eight benchmarks" (line 776).

  2. No control isolates the VLM's net contribution — the most important gap. The free-form interface gives only +0.70 (exactly +0.00 on PC60/City/ADE), and the margin-score rule alone gives +2.64, while "VLM + score support" gives only +1.70. Without a control that keeps the same gate and propagation but replaces the VLM with (a) the score argmax over AiA_i or (b) a random label from Ai∖{bi}A_i\setminus\{b_i\}, it is impossible to tell how much of the gain comes from the closed-set form constraint and how much from regional semantic evidence.

  3. Constants were selected on the evaluation domains. The paper states that constants "were selected once from aggregate diagnostics on the evaluated natural-image validation domains, then frozen". It is therefore unclear whether the selection set and the reporting set overlap; no held-out selection protocol is reported.

  4. The gain covers only half the available headroom. Table 29 gives SEER +7.52 against a full-pool oracle of +16.05 (SEER/full = 0.50), suggesting that audit budget and candidate set, not the pixel-execution stage the paper emphasizes, are the binding constraints. Relatedly, the precision/recall asymmetry is never reported: raw accuracy and executed precision have different denominators (Table 15 caption), so the reader cannot learn how many correct repairs the gate discarded.

Questions for authors
  1. Please provide component ablations on the same CorrCLIP predictions used in Table 1 (mean gain over the eight benchmarks with paired CIs), and reconcile the four-benchmark mechanism scope with the eight-benchmark conclusion sentence.
  2. Please add the control that fixes the gate and propagation strategies while replacing the VLM decision with (a) arg⁡max⁡csˉi,c\arg\max_c \bar s_{i,c} over AiA_i and (b) a random draw from Ai∖{bi}A_i\setminus\{b_i\}; and reconcile margin-score +2.64 (Table 21) with "VLM+score support" +1.70 (Table 4).
  3. Do the hyperparameter selection set and the reporting set overlap? If you ran a held-out protocol, please report it.
  4. Please report the miss rate (correct repairs rejected by the gate), the false-repair rate, and the mIoU impact of each; and give base+kk coverage plus false-repair rates for a 100+ class vocabulary such as COCO-Object or COCO-Stuff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.