SCARP: JOINT RISK-AWARE CORRECTION OF HIGHRISK SEMANTIC SEGMENTATION

Semantic segmentation errors with comparable mean Intersection over Union (mIoU) can induce markedly different downstream consequences. We study a failure mode that exposes this mismatch: a ghost, defined as a connected prediction of a high-risk class with no ground-truth support for that class. Because ghost identity is unobservable at test time, correction is inherently joint: the value of rerouting one high-risk component depends on what the scene retains elsewhere. We characterize exactly when such context induces a Keep–Reroute preference reversal and introduce SCARP (Scene-level Consequence-Aware Rerouting with Preservation), a training-free post-hoc decision layer that jointly optimizes high-risk component interventions under prediction-only constraints on removed high-risk evidence, without test-time ground truth. Against a matched additive program differing only in the scene term, coupling changes decisions on 7–21% of test images across four paired benchmarks and improves supported-component retention across all five domains. Strict preference reversals occur in 17 of 96 eligible cases and vanish without coupling. At matched retention on ACDC, SCARP further lowers Ghost-FDR, demonstrating an improved preservation–risk frontier.