acceptodds
Under review as a conference paper at ICLR 2027

CAVE: Counterfactual Auditing of Visual Evidence for Budgeted Local Supervision

Abstract

Medical images often have diagnosis labels but few local masks to supervise predicted evidence. With a limited annotation budget, we must select masks before their target regions are visible. Uncertainty and diversity identify ambiguous or underrepresented images but do not test whether predicted regions contribute to the known-class response. Expert-ROI criteria require the mask that the query will acquire. We introduce Counterfactual Auditing of Visual Evidence (CAVE), which compares known-class responses when a predicted region is retained or removed and measures reflection stability before ranking mask queries. Acquired masks supervise the shared learner. Under the original validation-selected five-seed profiles, CAVE has higher mean Dice budget-AUC than its matched control on BUSI, ISIC2018, and RSNA and the highest RSNA mean among tested selectors, although paired intervals span zero. Under a separate ten-seed protocol counting every queried mask, its mean Dice budget-AUC is lower than the matched control on all three datasets. The audit measures spatial evidence before annotation, but its current score does not reliably identify valuable masks.

Primary area:
interpretability and explainable AI
Keywords:
medical image analysis, active learning, budgeted local supervision, visual evidence auditing, counterfactual perturbations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_4eN2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CAVE, a framework for selecting medical images for local mask annotation under a limited budget. It uses counterfactual perturbations of predicted evidence regions and reflection consistency to assess spatial evidence before ground-truth masks become available. The authors evaluate CAVE on three medical imaging datasets. The original five-seed experiments show small improvements under dataset-specific configurations, while the independent ten-seed and matched-backbone experiments do not demonstrate improvements over the matched controls.

Critical strengths
  1. Selecting valuable local annotations is an important challenge in medical image analysis. The idea of examining whether a model's predicted evidence supports its classification before requesting a mask is worth exploring.

  2. The authors compare CAVE with uncertainty- and diversity-based selection methods and evaluate it under different annotation budgets. The matched comparisons help isolate the effect of the proposed acquisition signals.

Critical weaknesses
  1. Under the unified ten-seed protocol, CAVE performs worse than the matched control on all three datasets. The original five-seed improvements are small and uncertain, and the matched-backbone experiments also show negative effects. The current evidence does not establish the proposed audit as an effective acquisition strategy.

  2. The method assumes that changes in the classifier's response to predicted regions can help identify useful masks. However, the reported score–repair correlations are weak, and the selected samples do not consistently improve downstream localization.

  3. The acquisition score combines perturbation responses, reflection consistency, uncertainty, and diversity, all of which are related to existing selection or interpretation techniques. Without a stronger explanation of the observed failures, it is difficult to identify a sufficiently clear contribution beyond this combination.

Questions for authors
  1. What is the main insight that CAVE provides beyond the negative acquisition results? Can the authors identify conditions under which counterfactual auditing is expected to be useful?

  2. Why does the proposed audit score show such limited association with post-query improvement? Can the authors investigate whether it primarily measures sensitivity to perturbations rather than actual localization errors?

  3. Can the authors further analyze the individual spatial auditing signals under the unified protocol?

Reviewer_KhUZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors consider a setting where image-level labels are known and there is a limited annotation budget for masks. The propose a method to select masks more efficiently. It doesn't work.

Critical strengths

None

Critical weaknesses
  • the proposed method does not actually work, all results are either negative or show a difference in the 3rd digit
  • the paper is quite hard to read and often uses terms in an unusual way.
  • the term "active learning" is not mentioned in the Intro or Related Work, while the paper is clearly about active learning (choosing regions for an oracle to label).
  • as much as I could understand from the unusual terms, the method uses a combination of multiple known heuristics, so I couldn't appreciate the algorithmic novelty
Questions for authors

None

Reviewer_Q1ZXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select images for local mask annotation under a fixed budget when every training image has a diagnosis label but few have masks. It proposes CAVE, a pre-query audit that compares the known-class response under retained and removed views of the predicted region, measures reflection stability, and combines these signals with uncertainty and feature distance to rank mask queries for a shared learner. On BUSI, ISIC2018 and RSNA, CAVE shows small gains with paired intervals spanning zero under the original five-seed profiles and lower Dice budget-AUC than its matched control under the ten-seed all-counted protocol. And the authors conclude that the current audit score does not reliably identify valuable masks.

Critical strengths
  • A well-controlled negative result, reported in full. Pre-specified decision rule, seed-paired runs, Holm correction, and the ten-seed and ResNet-18 results that contradict the authors' own five-seed numbers are all reported.
  • A relevant question, cleanly formulated. Testing whether the model's spatial evidence supports the known class before paying for a mask is a natural idea, Section 3.1 states the setting clearly, and the baseline set (12 selectors) is broad.
Critical weaknesses
  • The negative result is narrow and unexplained. One instantiation is tested and the paper never asks why it fails. Two likely causes follow from the design: at 1% to 5% budgets the evidence logits are barely trained, so the predicted region behind the retained and removed views is close to noise exactly when selection matters most (its quality on the pool is never reported); and mean filling creates out-of-distribution inputs, so the response gap may measure artifacts rather than evidence use (the argument of Hooker et al. 2019, cited but not tested). Without a diagnosis or simple variants (audit only after validation evidence Dice reaches a minimum; filter instead of rank; hard masks), the conclusion is about one score, not the idea. Also, in Table 14 Random is within about one SD of the best selector on all three datasets (BUSI 0.435 vs 0.454; ISIC2018 0.736 vs 0.744; RSNA 0.420 vs 0.431). If no selector separates from Random, the setup cannot detect an acquisition effect, which weakens both the positive and the negative claims.

  • Key details are missing, and the space was spent elsewhere. The learner architecture is never stated (ResNet-18 appears only as a robustness variant), nor input size, optimizer, schedule, lambda, how T is retrained each round, the shortlist size, or how the Table 16 profile weights were obtained (they look tuned on validation masks, an annotation resource that is not budget-counted). No code link. The five-seed matched control, the baseline for the reported +0.0062 / +0.0049 / +0.0010 differences, is described only as "a matched rule that omits the spatial audit" and its absolute scores never appear (the text cites Table 6, which lists the best external selector instead). Dice is also far below what standard segmentation networks reach on these datasets with similar or fewer masks: a ResNet34 U-Net trained on 1/8 of the BUSI masks reaches Dice 0.68 [1], and supervised-only networks with 10% of the ISIC2018 masks reach 0.86 [2], against Dice budget-AUC of about 0.45 and 0.74 here. Splits and learners differ, but a gap of 0.1 to 0.2 Dice says the shared learner is weak, and the audit depends on its evidence maps. Also, this is not a page-limit issue: about two pages go to two near-identical schematic figures (note these figures look so AI generated style) and eleven tables of 3 to 6 rows (note these tables' designs are so loose), four of which duplicate/overlap very much with appendix tables (3/12, 6/14, 8/13, 7/9), while there is no result figure at all.

Overall, I do not think this paper is ready for ICLR. The paper presentation, experiments, and figures do not meet the bar for the venue.

[1] Striving for Simplicity: Simple Yet Effective Prior-Aware Pseudo-Labeling for Semi-Supervised Ultrasound Image Segmentation

[2] SamDSK: Combining Segment Anything Model with Domain-Specific Knowledge for Semi-Supervised Learning in Medical Image Segmentation

Questions for authors
  • What is the learner (backbone, pretraining, input size, optimizer, epochs, lambda), and how is T retrained each round?
  • At each budget, what is the Dice of the soft evidence map on the pool? Does the audit score-repair correlation (Table 5) improve at larger budgets?
  • What exactly is the five-seed matched control (which components of Eq. 6 it keeps) and what are its absolute Dice budget-AUC values?
  • Is any selector reliably better than Random under either protocol? If not, how should the negative result be read?
  • In Table 14, PAAL, SBC-AL, KALM, EG-AL and VALIANT are 0.01 to 0.08 Dice below Random on ISIC2018, although they beat random selection in their own papers. Were the criterion-only adapters validated by reproducing the original papers' gains in their original settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.