- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method which is theoretically motivated to 1- identify the acquisition-policy risk, 2- construct a cross-fitted trining criterion, 3-establish a uniform excess-risk guarantee relative to the best predictor.
Critical strengths
1- Well-defined and meaningful learning problem.
2- Principled theoretical framework with practical optimization.
Note: I really liked your abstract, it clearly answered three main questions I always seek in an abstract, (WHY, WHAT, and HOW), well done!
Critical weaknesses
First of all, I would like to thank the author for their contribution to this paper. I found this paper interesting; however, answering these two concerns would lean my score towards clear acceptance and I would be happy to read authors responses regarding the notes bellow:
1- I believe adding a more realistic multi-environment imaging experiment (if possible with actual acquisition differences) and end-to-end model would make the validation under realistic acquisition shift sufficient. The Camelyon17 experiment only uses color synthetic color cues and labels, and frozen representation and four held-out slides. also no end-to-end image training is evaluated.
2-I believe identifying the conditions that correction consistently will outperforms policy weighting and also demonstrating a more substantial benefit under realistic errors (like nuisance-estimation) would resolve the limited demonstration of proposed correctoin advantage.
Questions for authors
See the Weaknesses.
3 comments
Thank you for taking the time to write a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.
I will maintain my current score.
We thank you for your constructive feedback and encouraging assessment, and are glad our revisions addressed your concerns. We believe your comments will help us further refine and strengthen the paper.
Thank you for your constructive comments.
Weakness 1. We have added end-to-end CNN experiments using simulated labels and the original Camelyon17 tumor labels. ERM, policy weighting, clipped weighting, and correction fine-tune both the encoder and classification head of an ImageNet-pretrained ResNet-18. Within each experiment, the methods use the same training data, initialization, and training budget; the weighted methods share the estimated acquisition probabilities. In both CNN experiments, we vary the amount of data used to estimate acquisition probabilities.
In the experiment with simulated labels, policy weighting and correction both had lower mean target Brier risk than ERM (Appendix A.1.1, Table 8).
With the original tumor labels, we use two fixed hematoxylin and eosin (H&E) stain profiles estimated from training images. Correction had lower mean target Brier risk than ERM in all three conditions, with relative reductions ranging from 4.5% to 6.7%, while policy weighting did not consistently improve on ERM. The paired confidence intervals for these risk differences included zero (Section 4.2, lines 513–517; Appendix A.1.2, Tables 10 and 12).
We simulate which stain version is observed for each training image (see Section 5 (lines 537–539) for the scope of this evaluation). All evaluations use images from the same four held-out slides.
Weakness 2. Correction reduces the sensitivity of the risk criterion to propensity estimation error when the conditional-risk regression is accurate enough. Sampling and optimization errors still matter, so this does not guarantee lower finite-sample risk in every setting (Section 3.2, lines 284–292 and 316–323).
We have added a matched comparison in which weighting and correction use the same propensity estimates and training records. In the simulation with estimated acquisition probabilities, correction reduces mean excess risk by 46.37% relative to matched policy weighting. The reduction relative to standard out-of-fold weighting is 37.6%; the latter uses four, rather than three, folds for each propensity fit. Here, excess risk is the gap from the minimum risk within the chosen predictor class. Absolute excess risks are reported in Appendix A, Table 1, and matched paired differences with simultaneous 95% confidence intervals in Table 2.
In this simulation, clipping weights at 2 gives lower mean risk than correction when acquisition probabilities are estimated, but higher mean risk when they are known (Appendix A, Table 1). With known acquisition probabilities, the benefit of correction over unclipped matched policy weighting remains uncertain (Appendix A, Table 2).
In the end-to-end CNN experiment with simulated labels, correction reduced mean excess risk above the raw-image Bayes risk by 14.83% and 20.26% relative to unclipped policy weighting in two conditions. The corresponding paired risk reductions were supported by simultaneous 95% confidence intervals; the remaining comparison was uncertain. A gain over clipped weighting was supported in one condition (Appendix A.1.1, Table 8).
With the original Camelyon17 tumor labels, correction had lower mean target Brier risk than policy weighting in all three conditions, with relative reductions ranging from 3.5% to 14.9%. Relative to clipped weighting, mean risk was 3.2–6.2% higher in two conditions and 11.1% lower in the other. The paired confidence intervals for risk differences against both weighting baselines included zero, so the additional benefit of correction remains uncertain (Appendix A.1.2, Table 11).