- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes PCGA, which averages teacher attribution maps across label-preserving perturbations and uses their agreement to weight student attribution alignment. Experiments on Chest CT, RADGEN, and CIFAR-10 compare stability weighting with uniform and prediction-derived weights, supplemented by gradient controls and spatial interventions. The results attribute the main Chest CT gain chiefly to auxiliary-supervision strength, while the incremental benefit of image-specific weighting remains inconsistent.
Critical strengths
The gradient analysis and scale-only control distinguish changes in auxiliary-supervision strength from the effects of image-specific weighting. These controls, together with transparent reporting of negative results, provide useful evidence about the limitations of attribution stability as a supervision criterion (§§3.3, 5.2).
Critical weaknesses
- Insufficient support for the weighting criterion and its distinct methodological benefit. The experiments do not adequately establish that more stable explanations provide more useful student supervision. In the strongest Chest CT cosine comparison, PCGA’s advantage disappears after matching auxiliary-gradient strength: PCGA achieves 99.19% AUC versus 99.25% for scale-only uniform alignment (Table 1; §5.2). The CIFAR-10 advantage occurs in only one of three paired seeds, while the favorable norm-matched L1 follow-up is limited to validation results (Appendices F, I). These findings do not establish a reproducible incremental benefit from assigning stability-based weights to individual images.
- Insufficient validation of when stability is useful. The reported variation across tasks, losses, and target configurations does not establish conditions that predict when stability weighting improves learning. The gradient decomposition characterizes changes in update scale and direction, but does not establish when those changes are beneficial. Although the authors acknowledge the negative findings, the analysis does not sufficiently justify why this stability score should determine supervision strength or when assessing it separately for each image adds value.
Questions for authors
- What evidence establishes that the proposed stability score predicts an explanation’s downstream training utility and therefore justifies its use in determining supervision strength?
- Which result demonstrates a reproducible benefit from image-specific stability weighting after controlling auxiliary-gradient strength?
- What measurable condition predicts when stability weighting will improve learning, and where is that prediction independently validated?