- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CAVE, a framework for selecting medical images for local mask annotation under a limited budget. It uses counterfactual perturbations of predicted evidence regions and reflection consistency to assess spatial evidence before ground-truth masks become available. The authors evaluate CAVE on three medical imaging datasets. The original five-seed experiments show small improvements under dataset-specific configurations, while the independent ten-seed and matched-backbone experiments do not demonstrate improvements over the matched controls.
Critical strengths
-
Selecting valuable local annotations is an important challenge in medical image analysis. The idea of examining whether a model's predicted evidence supports its classification before requesting a mask is worth exploring.
-
The authors compare CAVE with uncertainty- and diversity-based selection methods and evaluate it under different annotation budgets. The matched comparisons help isolate the effect of the proposed acquisition signals.
Critical weaknesses
-
Under the unified ten-seed protocol, CAVE performs worse than the matched control on all three datasets. The original five-seed improvements are small and uncertain, and the matched-backbone experiments also show negative effects. The current evidence does not establish the proposed audit as an effective acquisition strategy.
-
The method assumes that changes in the classifier's response to predicted regions can help identify useful masks. However, the reported score–repair correlations are weak, and the selected samples do not consistently improve downstream localization.
-
The acquisition score combines perturbation responses, reflection consistency, uncertainty, and diversity, all of which are related to existing selection or interpretation techniques. Without a stronger explanation of the observed failures, it is difficult to identify a sufficiently clear contribution beyond this combination.
Questions for authors
-
What is the main insight that CAVE provides beyond the negative acquisition results? Can the authors identify conditions under which counterfactual auditing is expected to be useful?
-
Why does the proposed audit score show such limited association with post-query improvement? Can the authors investigate whether it primarily measures sensitivity to perturbations rather than actual localization errors?
-
Can the authors further analyze the individual spatial auditing signals under the unified protocol?