acceptodds
Under review as a conference paper at ICLR 2027

SCARP: JOINT RISK-AWARE CORRECTION OF HIGHRISK SEMANTIC SEGMENTATION

Abstract

Semantic segmentation errors with comparable mean Intersection over Union (mIoU) can induce markedly different downstream consequences. We study a failure mode that exposes this mismatch: a ghost, defined as a connected prediction of a high-risk class with no ground-truth support for that class. Because ghost identity is unobservable at test time, correction is inherently joint: the value of rerouting one high-risk component depends on what the scene retains elsewhere. We characterize exactly when such context induces a Keep–Reroute preference reversal and introduce SCARP (Scene-level Consequence-Aware Rerouting with Preservation), a training-free post-hoc decision layer that jointly optimizes high-risk component interventions under prediction-only constraints on removed high-risk evidence, without test-time ground truth. Against a matched additive program differing only in the scene term, coupling changes decisions on 7–21% of test images across four paired benchmarks and improves supported-component retention across all five domains. Strict preference reversals occur in 17 of 96 eligible cases and vanish without coupling. At matched retention on ACDC, SCARP further lowers Ghost-FDR, demonstrating an improved preservation–risk frontier.

Primary area:
optimization
Keywords:
Semantic segmentation, post-hoc correction, risk-aware decision making, structured prediction, safety-critical perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_m7QvOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies reliable semantic segmentation and introduces SCARP, a training-free, post-hoc decision layer that jointly optimizes interventions on high-risk components under prediction-only constraints on the removal of high-risk evidence, without requiring test-time ground truth. The experimental results provide some evidence of the method’s effectiveness.

Critical strengths

The methodological formulation and theoretical derivations are presented in considerable detail.

Critical weaknesses

The paper lacks both qualitative visualizations and quantitative analyses showing where ghost components typically occur and whether they exhibit any consistent spatial or semantic patterns. Such evidence would help readers better understand the phenomenon and evaluate the practical effectiveness of the proposed method.

Questions for authors

I am not sufficiently familiar with this specific area of semantic segmentation and therefore may not be able to provide a fully reliable assessment. Please feel free to correct me if I have misunderstood any aspect of the paper.

Reviewer_3AeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies high-risk errors in semantic segmentation and proposes SCARP, a training-free post-hoc correction framework. It formulates component-level Keep–Reroute decisions as a joint optimization problem, combining expected intervention costs, scene-level coupling, and preservation constraints. Experiments across five domains demonstrate context-dependent decision changes and improved retention of supported components compared with an additive formulation.

Critical strengths

Clear problem formulation and theoretical motivation. The paper connects component-level correction with structured decision-making. The analysis of context-dependent preference reversal provides a sound explanation for joint optimization.

Careful mechanism validation. The matched Joint–Additive comparison effectively isolates scene coupling. The controlled experiments across different settings and backbones provide useful evidence for the proposed mechanism.

Critical weaknesses

Narrow problem scope and limited practical significance. The formulation focuses on unsupported predictions of an aggregated high-risk class, covering only a restricted subset of segmentation errors. The proposed Ghost-FDR metric does not necessarily reflect meaningful downstream consequences, limiting the broader significance of the problem.

Limited generalizability of the risk formulation. The method relies on several application-specific choices, including spatial contexts, a confidence-based scene potential, and manually specified consequence matrices and preservation constraints. Their relationship to actual risk and their applicability to other scenarios are not sufficiently justified.

Insufficient evidence of practical correction benefits. Although joint optimization improves retention over the additive baseline, improvements over the original predictions are less convincing. Rerouting frequently produces incorrect semantic labels, and segmentation performance can degrade substantially. Matched-retention comparisons are also limited to ACDC, leaving the broader preservation–risk benefits unclear.

Questions for authors

Problem significance. How well does the Ghost-FDR formulation capture meaningful downstream risk? What makes this specific failure mode sufficiently important beyond the evaluated settings?

Method generality. How well do the proposed scene potential and preservation constraints generalize to different risk definitions, newer segmentation backbones and application scenarios?

Practical utility. Given the unreliable rerouting labels and degradation in segmentation accuracy, what evidence supports the practical value of SCARP? Could the authors provide downstream risk evaluation or expert assessment, alongside matched-retention comparisons on additional datasets?

Presentation. Could the authors include a method overview illustrating the complete decision pipeline and the role of scene-level coupling?

Reviewer_xfDJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the troublesome "ghost" false positives that plague high-risk semantic segmentation. It presents SCARP, a training-free post-hoc joint correction framework. Leveraging scene coupling and 0-1 mixed-integer linear programming (MILP), it attains an improved retention-risk tradeoff over diverse domains and model architectures.

Critical strengths
  • Formulates a practical, under-explored problem: aligning semantic segmentation with asymmetric downstream risks.
  • Strong theoretical grounding (formal proof of context-induced preference reversals, Prop 3.1).
  • Clean experimental ablation (Joint vs. Additive) isolating the exact contribution of the scene term.
Critical weaknesses
  • Rerouted destinations are semantically unreliable (only 3.25% match true classes; often degrades local pixel accuracy). The reroute acts as a cost-minimizing placeholder rather than a true semantic correction.
  • Performance relies heavily on tuning hyperparameters (αmin\alpha_{min}, budgets, ρ\rho), raising deployment friction.
Questions for authors
  1. Regarding the handling of unknown classes: Given that the redirected labels are semantically unreliable and hurt pixel-level mIoU, it would be interesting to know why the authors do not adopt an explicit unknown or abstention class.

  2. On real-time capability and scalability: As the number of scene components scales up significantly, does the worst-case MILP solving time pose a bottleneck for real-time vehicular or aerial deployment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.