- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work examines whether synthetic masks matched to segmenter predictions by per-image IoU provide reliable evaluations of segmentation-guided classifiers. It proposes a source-crossed audit, reporting a percentage-point interaction on PlantSeg and percentage points of model-selection regret in a separate selection experiment.
Critical strengths
- The source-crossed design, OOF predictions, and controlled comparisons provide convincing evidence of training-mask provenance effects on PlantSeg.
- The model-selection experiment connects mask-source mismatch to a practical decision.
Critical weaknesses
- Limited novelty and insight. IoU does not capture error location, topology, or retained background. Different downstream outcomes under equal IoU are therefore expected. The crossing and difference-in-differences analysis use established methodology. The paper offers limited insight into when synthetic masks can serve as reliable proxies.
- The synthetic masks used in the experiments differ substantially from real segmentation predictions. For example, on CUB, 99.55% of synthetic masks produce crops covering almost the entire image, compared with only 0.93% of predicted masks. Such a large difference makes the observed classification performance gap unsurprising. It remains unclear whether the findings hold when more realistic synthetic masks are used.
- The abstract is overloaded with numerical results, confidence intervals, and technical details, making the main contribution difficult to grasp. The overall presentation could be more concise and focused.
Questions for authors
- What is the insight beyond the expected consequences?
- What is the practical benefit of the proposed audit, given that it requires real segmenter predictions and additional classifier training? And if real segmenter predictions are already available, why is it still necessary to use synthetic masks for model selection?