acceptodds
Withdrawn from ICLR 2027

Same IoU, Different Decisions: Synthetic Mask Corruptions Mislead Classifiers Guided by Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Segmentation masks guide downstream classifiers through masking, cropping, auxiliary channels, and attention. Synthetic corruptions of ground-truth masks are a convenient substitute for repeatedly training segmenters, but their use assumes that matching intersection over union (IoU) also matches downstream effects. We test this assumption with a source-crossed audit that trains and evaluates classifiers on both synthetic masks and held-out segmenter predictions, while matching the two sources to the same IoU target for each image. On frozen PlantSeg test data, the two training sources respond differently to the same change in evaluation mask, yielding a +6.28 pp difference-in-differences (95% CI over six seeds [5.31, 7.26]). A preregistered BUSI replication gives +16.50 pp ([13.93, 19.07]), whereas a CUB extension using six seeds gives a smaller +2.07 pp interaction ([1.53, 2.62]) in the opposite practical direction: synthetic evaluation fails to reveal a +1.99 pp benefit from ROI cropping with predicted masks. On PlantSeg, synthetic validation consequently selects a masking method that incurs +1.91 pp deployment regret ([0.48, 3.34]). Matching fragmentation 350× more closely does not remove the discrepancy, and training masks obtained by cross fitting only partially mitigate it. Thus matching IoU for each image does not make synthetic and predicted masks interchangeable downstream. Their fidelity must be tested with a source-crossed downstream audit rather than inferred from mask similarity

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
segmentation-guided classification, synthetic mask corruption, mask provenance, IoU matching, downstream evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VbR6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work examines whether synthetic masks matched to segmenter predictions by per-image IoU provide reliable evaluations of segmentation-guided classifiers. It proposes a 2×22\times 2 source-crossed audit, reporting a 6.286.28 percentage-point interaction on PlantSeg and 1.911.91 percentage points of model-selection regret in a separate selection experiment.

Critical strengths
  • The source-crossed design, OOF predictions, and controlled comparisons provide convincing evidence of training-mask provenance effects on PlantSeg.
  • The model-selection experiment connects mask-source mismatch to a practical decision.
Critical weaknesses
  • Limited novelty and insight. IoU does not capture error location, topology, or retained background. Different downstream outcomes under equal IoU are therefore expected. The 2×22\times 2 crossing and difference-in-differences analysis use established methodology. The paper offers limited insight into when synthetic masks can serve as reliable proxies.
  • The synthetic masks used in the experiments differ substantially from real segmentation predictions. For example, on CUB, 99.55% of synthetic masks produce crops covering almost the entire image, compared with only 0.93% of predicted masks. Such a large difference makes the observed classification performance gap unsurprising. It remains unclear whether the findings hold when more realistic synthetic masks are used.
  • The abstract is overloaded with numerical results, confidence intervals, and technical details, making the main contribution difficult to grasp. The overall presentation could be more concise and focused.
Questions for authors
  • What is the insight beyond the expected consequences?
  • What is the practical benefit of the proposed audit, given that it requires real segmenter predictions and additional classifier training? And if real segmenter predictions are already available, why is it still necessary to use synthetic masks for model selection?
Reviewer_YbcDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether synthetic mask corruptions can replace the predictions of a real segmenter when masks are used as inputs to a downstream classifier (e.g., background zeroing, ROI cropping, an extra mask channel). For each image, the authors corrupt the GT mask so that its IoU matches the IoU of the corresponding segmenter prediction. They then train one classifier on synthetic masks (SS) and another on predicted masks (RR, obtained with 5-fold OOF predictions), and evaluate both classifiers on both mask sources. The main quantity is the difference-in-differences $\text{DiD} = (A_{SS} - A_{SR}) - (A_{RS} - A_{RR}), $ where the first index is the training source and the second is the evaluation source. On the PlantSeg test set, DiD is +6.28+6.28 pp. BUSI (+16.50+16.50 pp) and CUB (+2.07+2.07 pp) also show positive interactions, and RIM-ONE is inconclusive. The authors further show that synthetic validation changes model selection on PlantSeg (+1.91+1.91 pp regret), that matching fragmentation more closely does not remove the gap, and that OOF cross-fitting only partially mitigates it. They conclude that per-image IoU matching does not make synthetic and predicted masks interchangeable, and that a source-crossed audit should be used instead.

Critical strengths
  1. The question is relevant. Perturbing GT masks to simulate segmentation errors is a common shortcut, and checking whether this shortcut actually reflects the behavior of real segmenter outputs is a reasonable thing to do.

  2. The experiments are very thorough. The paper covers four domains, several segmenters (including two zero-shot ones), several fusion modes, and three backbones. Uncertainty is reported at the seed level and hierarchically, and there are additional checks for near-duplicate images, class imbalance, and residual IoU mismatch.

  3. The reporting is honest. Preregistered analyses are separated from post hoc ones, and failed predictions (Severstal, BUS-BRA) and inconclusive results (RIM-ONE) are kept in the paper. The limitations section is also quite clear about what is and is not claimed.

  4. The crossed design is more informative than just comparing ASSA_{SS} with ARRA_{RR}, since it separates the effect of changing the evaluation mask from the classifier's adaptation to its training masks.

Critical weaknesses
  1. The main finding seems somewhat expected. The DiD can be rewritten as (ASS+ARR)−(ARS+ASR)(A_{SS} + A_{RR}) - (A_{RS} + A_{SR}), i.e., source-matched cells minus mismatched cells. A positive DiD therefore mostly says that a classifier performs better when the test masks come from the same distribution as the training masks. The source probe in Figure 5A also shows that all synthetic sources are almost perfectly separable from deployment masks (AUC ≈1.0\approx 1.0). Given that the two mask distributions are clearly different, it is not very surprising that classifiers adapt to them differently. I am also not fully convinced that the community actually assumes per-image IoU matching is sufficient. As far as I understand, the cited perturbation works (e.g., Zwanenburg et al., 2019; Bruhns et al., 2025) mainly use perturbations to test the robustness of extracted features or downstream measurements, not to train classifiers that consume the masks.

  2. There is little insight beyond the observation itself. The paper shows that a discrepancy exists, but it does not identify which error properties cause it. The fragmentation ladder is the only attempt in this direction, and as the authors note, it changes several statistics at once: centroid distance improves from 0.489 to 0.113 while boundary distance worsens from 0.373 to 0.562. It also uses only three seeds, and the trend interval includes zero. As a result, the reader does not learn what should be matched, in addition to IoU, to make a synthetic proxy reliable.

  3. There is no actionable solution. The proposed audit requires predicted masks from the deployed segmenter. But if those masks are available, there is much less reason to use a synthetic proxy in the first place. So the audit is hard to apply in exactly the setting where a proxy is needed. Cross-fitting is a known technique (stacked generalization), its benefit here is partial (+1.10+1.10 pp pooled), and it depends on the architecture (only +0.10+0.10 pp with mixed signs for ViT-Base). Overall, the paper does not offer a better proxy or a practical recipe.

  4. The synthetic generators look unrealistic, which may limit the generality of the claims. Probabilistic mixing produces very scattered errors (median 586 connected components at σ=0\sigma = 0 in Table 8), while real segmenter errors are usually spatially coherent (boundary shifts, missed regions, contiguous false positives). Erosion and dilation miss the IoU target for more than half of the images (Table 23). The CUB result, which is presented as the "opposite direction", seems largely explained by this artifact: scattered false positives expand the bounding box, so 99.55% of synthetic boxes cover more than 99% of the image. I think this is closer to a property of this particular generator than to a general phenomenon that changes direction across domains.

  5. The practical significance is limited. On PlantSeg, the hard-zero classifier trained and evaluated on deployment masks reaches 75.73%, compared with 75.33% for the image-only baseline (Figure 5B), so mask guidance barely helps in the deployment setting. In the selection experiment, deployment validation also picks the image-only model. In addition, only 2 of 13 screened datasets pass the GT mask benefit gate (Table 9), which suggests that the setting where this issue matters is quite narrow. The large BUSI effect is also hard to interpret, because the same 98 images are used for both checkpoint selection and evaluation.

Questions for authors
  1. Does the interaction persist when the synthetic masks have more realistic, spatially coherent errors (see Weakness 4)? If the DiD becomes much smaller in that case, I think the main issue would be the realism of the generator rather than IoU matching itself.

  2. In what practical scenario would one run the proposed audit? If predicted masks are needed for the audit, could a small number of predicted masks, or masks from a different segmenter, be enough to validate a proxy for the target segmenter?

Suggestion 1. One way to address Weakness 4 would be to add a generator that produces more realistic, spatially coherent errors, for example adversarial perturbations of the input or of a segmenter's embedding [A, B], where the attack strength can be searched to match the per-image IoU target in the same way as the blend strength in probabilistic mixing.

[A] Janouskova, Klara, et al. "Single image test-time adaptation for segmentation." Transactions on Machine Learning Research 2024 (2024).

[B] Kim, Beomyoung, and Sung Ju Hwang. "Learning from Adversity: Semantic-Aware Mask Refinement through Adversarial Perturbation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Reviewer_fCRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an audit of the use of segmentation masks to guide downstream classification and their impact on classification performance. The authors demonstrate that synthetic masks generated by perturbing ground-truth annotations can achieve similar Intersection over Union (IoU) scores to masks predicted by segmentation models while producing substantially different downstream classification results. To investigate this discrepancy, the authors introduce a crossed evaluation protocol based on a Difference-in-Differences (DiD) metric. The proposed audit is evaluated across four datasets: PlantSeg, BUSI, CUB-200-2011, and RIM-ONE DL . The experiments reveal significant discrepancies between synthetic and predicted masks. The authors further demonstrate that relying on synthetic masks for model selection can lead to suboptimal decisions when deploying classifiers with real segmentation predictions.

Critical strengths

This paper presents a timely contribution that advances the evaluation of segmentation-guided classification. The empirical evidence is convincing, with strong results on PlantSeg, supporting findings on BUSI, and complementary results on CUB demonstrating that the impact of synthetic mask discrepancies can vary across datasets. I believe this work provides valuable insights for the research community and could inspire further investigations beyond classification, particularly in other downstream tasks that rely on intermediate segmentation predictions.

Critical weaknesses

While I appreciate the careful experimental methodology and the relevance of the findings, I feel that the work remains somewhat limited in terms of novelty and scope.

I believe this work could serve as the foundation for a broader and more impactful study. In particular, it would be interesting to investigate more systematically which characteristics of synthetic mask perturbations are responsible for the observed DiD discrepancies. Although the authors provide some initial investigations, a more detailed analysis could offer deeper insights into the underlying mechanisms.

Furthermore, exploring more sophisticated synthetic mask generation strategies, potentially through learned generative models that better reproduce the error distributions of real segmenters, could extend the contribution beyond identifying the limitations of existing perturbations toward developing more reliable alternatives.

Overall, I remain somewhat hesitant primarily because of the level of novelty. While I find the empirical evidence convincing and the findings relevant, I am currently leaning toward a borderline evaluation. I would be interested in hearing the authors' perspective on my concerns regarding the broader scope and potential impact of this work, as well as their responses to the questions raised below. I would be open to upgrading my score if the authors can further clarify and make a stronger case for the novelty and significance of their contribution.

Questions for authors
  1. Which characteristics of synthetic mask perturbations contribute most to the observed DiD? For instance, error localization, boundary deformation, fragmentation, or false-positive distribution.

  2. Since synthetic masks are generated by perturbing ground-truth annotations, could they remain too structurally similar to the original masks, limiting their ability to reproduce more complex or semantic errors made by real segmenters?

  3. The paper demonstrates that simple synthetic corruptions do not faithfully reproduce downstream behavior. However, an interesting next question is whether one could learn a corruption distribution that better approximates real segmentation errors, rather than relying on predefined transformations. For example, one could train a generative model to reproduce spatial error patterns observed in OOF predictions. The objective could be to minimize the discrepancy between synthetic and predicted masks in downstream classification, potentially measured through DiD.

  4. Although OOF and deployment predictions have similar IoU distributions, the deployment segmenter is trained on the full training set, whereas each OOF model uses only four folds. Could this difference affect the structure of segmentation errors and introduce a bias in the downstream evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.