acceptodds
Under review as a conference paper at ICLR 2027

Learning under counterfactual shifts in image acquisition

Abstract

A change in image acquisition can alter the optimal image-only predictor even when class proportions and the overall image distribution remain unchanged, because the class probabilities associated with a given image can change under the deployed policy. We study classification under a prespecified change in the acquisition policy using historical data alone. The intervention preserves historical assignment support, avoiding an additional assignment-positivity requirement. Under conditional exchangeability and missing-at-random labels, we identify the target Brier risk and its Bayes predictor and derive an efficient-influence-function-corrected training criterion. With linear conditional-risk regression on metadata, the criterion reduces exactly to a Brier objective with precomputed record weights, enabling gradient-based training without repeated regression fitting. We establish a uniform excess-risk bound with product-form nuisance errors and a squared propensity- error term. Simulations and a semi-synthetic Camelyon17 study show that targeting the planned policy provides most of the improvement, while the additional benefit of correction varies with the predictor and sample size.

Primary area:
causal reasoning
Keywords:
causal inference, distribution shift, image classification, incremental interventions, orthogonal learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uM7EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method which is theoretically motivated to 1- identify the acquisition-policy risk, 2- construct a cross-fitted trining criterion, 3-establish a uniform excess-risk guarantee relative to the best predictor.

Critical strengths

1- Well-defined and meaningful learning problem.
2- Principled theoretical framework with practical optimization.

Note: I really liked your abstract, it clearly answered three main questions I always seek in an abstract, (WHY, WHAT, and HOW), well done!

Critical weaknesses

First of all, I would like to thank the author for their contribution to this paper. I found this paper interesting; however, answering these two concerns would lean my score towards clear acceptance and I would be happy to read authors responses regarding the notes bellow:

1- I believe adding a more realistic multi-environment imaging experiment (if possible with actual acquisition differences) and end-to-end model would make the validation under realistic acquisition shift sufficient. The Camelyon17 experiment only uses color synthetic color cues and labels, and frozen representation and four held-out slides. also no end-to-end image training is evaluated.

2-I believe identifying the conditions that correction consistently will outperforms policy weighting and also demonstrating a more substantial benefit under realistic errors (like nuisance-estimation) would resolve the limited demonstration of proposed correctoin advantage.

Questions for authors

See the Weaknesses.

3 comments

Thank you for your constructive comments.

Weakness 1. We have added end-to-end CNN experiments using simulated labels and the original Camelyon17 tumor labels. ERM, policy weighting, clipped weighting, and correction fine-tune both the encoder and classification head of an ImageNet-pretrained ResNet-18. Within each experiment, the methods use the same training data, initialization, and training budget; the weighted methods share the estimated acquisition probabilities. In both CNN experiments, we vary the amount of data used to estimate acquisition probabilities.

In the experiment with simulated labels, policy weighting and correction both had lower mean target Brier risk than ERM (Appendix A.1.1, Table 8).

With the original tumor labels, we use two fixed hematoxylin and eosin (H&E) stain profiles estimated from training images. Correction had lower mean target Brier risk than ERM in all three conditions, with relative reductions ranging from 4.5% to 6.7%, while policy weighting did not consistently improve on ERM. The paired confidence intervals for these risk differences included zero (Section 4.2, lines 513–517; Appendix A.1.2, Tables 10 and 12).

We simulate which stain version is observed for each training image (see Section 5 (lines 537–539) for the scope of this evaluation). All evaluations use images from the same four held-out slides.

Weakness 2. Correction reduces the sensitivity of the risk criterion to propensity estimation error when the conditional-risk regression is accurate enough. Sampling and optimization errors still matter, so this does not guarantee lower finite-sample risk in every setting (Section 3.2, lines 284–292 and 316–323).

We have added a matched comparison in which weighting and correction use the same propensity estimates and training records. In the simulation with estimated acquisition probabilities, correction reduces mean excess risk by 46.37% relative to matched policy weighting. The reduction relative to standard out-of-fold weighting is 37.6%; the latter uses four, rather than three, folds for each propensity fit. Here, excess risk is the gap from the minimum risk within the chosen predictor class. Absolute excess risks are reported in Appendix A, Table 1, and matched paired differences with simultaneous 95% confidence intervals in Table 2.

In this simulation, clipping weights at 2 gives lower mean risk than correction when acquisition probabilities are estimated, but higher mean risk when they are known (Appendix A, Table 1). With known acquisition probabilities, the benefit of correction over unclipped matched policy weighting remains uncertain (Appendix A, Table 2).

In the end-to-end CNN experiment with simulated labels, correction reduced mean excess risk above the raw-image Bayes risk by 14.83% and 20.26% relative to unclipped policy weighting in two conditions. The corresponding paired risk reductions were supported by simultaneous 95% confidence intervals; the remaining comparison was uncertain. A gain over clipped weighting was supported in one condition (Appendix A.1.1, Table 8).

With the original Camelyon17 tumor labels, correction had lower mean target Brier risk than policy weighting in all three conditions, with relative reductions ranging from 3.5% to 14.9%. Relative to clipped weighting, mean risk was 3.2–6.2% higher in two conditions and 11.1% lower in the other. The paired confidence intervals for risk differences against both weighting baselines included zero, so the additional benefit of correction remains uncertain (Appendix A.1.2, Table 11).

Thank you for taking the time to write a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.

I will maintain my current score.

We thank you for your constructive feedback and encouraging assessment, and are glad our revisions addressed your concerns. We believe your comments will help us further refine and strengthen the paper.

Reviewer_Ci7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Summary: Perturbations or changes in image acquisition can alter the image distribution, and deep learning models are sensitive to such changes. This paper studies classification under a prespecified change in the acquisition policy. The authors identify the Brier risk and its Bayes predictor, and they establish a uniform excess-risk bound with product-form nuisance errors and a squared propensity-error term. Their contributions are: (1) identifying the acquisition-policy risk, and (2) constructing a cross-fitted training criterion with a uniform excess-risk guarantee relative to the best predictor in the chosen class.

Critical strengths

Strengths:

  • Useful theory: the bound separates sampling error from product-type nuisance bias and states that the assignment correction is not doubly robust.

  • The training is practical, since precomputed record weights allow gradient-based training without refitting at every step.

Critical weaknesses

Weaknesses:

  • The abstract gets overly technical, and I wasn't able to understand the contributions clearly. I suggest keeping it high-level. It was hard to understand whats the significance of Brier risk in their context?

  • Why did the authors demonstrate their method on only one dataset? This limits the extent of the claims they can make.

  • For image construction in pathology, why do the authors use stains that do not match real-world shifts?

  • I do not think the evaluation setup for Camelyon17 is in line with the original work. It is standard for papers to train on three hospitals and evaluate on the remaining two (with one held out for validation).

Questions for authors
  • The baselines are weak. The main experiments use a four-parameter head or a small MLP on PCA-transformed ResNet-18 embeddings.

  • Can the authors evaluate on more real data with a natural acquisition change, such as multiple scanners or stain protocols?

  • What happens with end-to-end CNN training and negative record weights?

3 comments

Thank you for your constructive comments.

Weakness 1. We have reduced the technical detail in the Abstract. Brier risk measures the accuracy of predicted class probabilities under the planned acquisition policy. Its excess over Bayes risk is the mean squared distance between the predicted and true conditional class probabilities. Section 3 (lines 151–153 and 176–187) now states this interpretation alongside the identity.

Weakness 2 and 3, and Question 2. We have added an experiment using the tumor labels provided with Camelyon17. Using two stain profiles estimated from training images, we create two versions of each image with different hematoxylin and eosin (H&E) staining colors (Section 4.2, lines 513–517; Appendix A.1.2, Tables 10–12).

The experiment with simulated labels and color cues allows calculation of the risk under the planned policy and the corresponding Bayes risk (Appendix A, lines 882–883 and 918–923).

The imaging experiments use only Camelyon17, with simulated assignment to the staining conditions. Performance under naturally occurring changes in scanners or staining protocols remains untested (Section 5, lines 537–539).

Weakness 4. The standard hospital split evaluates generalization to unseen hospitals. Our experiment instead changes the acquisition assignment probabilities within historical support, while keeping the source population fixed. Training and evaluation use disjoint slides, and all methods are evaluated under the same planned policy. All replications use the same four held-out slides (Section 4.2, lines 467–473; Appendix A.1.2).

Question 1. We now compare ERM, policy weighting, clipped weighting, and correction using the same ImageNet-pretrained ResNet-18, training both the encoder and head. The methods share the training data, initialization, and training budget; the weighted methods also share propensity estimates within each condition. In both CNN experiments, we vary the amount of data used to estimate acquisition probabilities.

In the end-to-end CNN experiment with simulated labels, policy weighting and correction both had lower mean target Brier risk than ERM. Correction gains were supported over unclipped weighting in two conditions and over clipped weighting in one condition; the remaining comparisons were uncertain (Appendix A.1.1, Table 8).

In the experiment using the original Camelyon17 tumor labels, correction had lower mean target Brier risk than ERM and policy weighting in all three conditions, with relative reductions of 4.5–6.7% and 3.5–14.9%, respectively. However, the paired confidence intervals for the risk differences included zero, and clipped weighting had lower mean risk in some conditions. Section 4.2 (lines 513–517) and Appendix A.1.2, Tables 10–12, report Brier risk, classification accuracy, and the full comparisons.

Question 3. We used the original correction without clipping. In the end-to-end CNN experiment with simulated labels, the final weights applied to individual training losses were all positive (Appendix A.1.1, Table 9).

A correction term can be negative even when the final weight applied to the corresponding training loss is positive. If the final weight is negative, that record's contribution to the gradient reverses sign. Because Brier loss is bounded, the objective remains bounded below for any finite set of fixed weights, although boundedness alone does not guarantee stable optimization (Section 5, lines 537–539; Appendix J, Equation (56)).

Thank you for your responses, I have one quick follow up question, where are the results for this stated line? " We now compare ERM, policy weighting, clipped weighting, and correction"

Thank you for your follow-up question. In the revised manuscript, Table 8 (Appendix A.1.1, p. 21) reports the end-to-end ResNet-18 comparison with simulated labels, and Table 10 (Appendix A.1.2, p. 23) reports the corresponding comparison with the original Camelyon17 tumor labels. Both tables report target Brier risk and classification accuracy for all four methods.

The table below summarizes the mean target Brier risks from Tables 8 and 10. (Within each experiment, all three rows share the same ERM reference because historical ERM does not use the estimated acquisition probabilities.)

LabelsPropensity sample sizeHistorical ERMPolicy weightingClipped weightingCorrection
Simulated2,0480.60400.43930.43640.4353
Simulated4,0960.60400.43880.43620.4333
Simulated8,1920.60400.43240.43700.4320
Original tumor2,0480.10100.11340.09340.0965
Original tumor4,0960.10100.09760.08870.0942
Original tumor8,1920.10100.10720.10760.0956

Table 8 also reports paired Brier-risk differences and their 95% confidence intervals for the simulated labels. For the original tumor labels, these comparisons are reported in Tables 11 and 12 (p. 23).

Reviewer_MZVeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies image classification under shifts in image acquisition policies at deployment. It focuses on training a classifier for a future acquisition policy using only historical data. The authors propose a corrected risk objective that accounts for estimated propensities and missing labels, together with an efficient optimization strategy. The paper also provides theoretical analysis and evaluates the proposed approach through simulations and semi-synthetic experiments on Camelyon17.

Critical strengths
  1. The problem setting is meaningful for applications such as medical imaging, where acquisition policies may change and collecting new labeled data can be costly.
  2. The paper provides a clear theoretical formulation of the problem. In particular, the use of causal inference and counterfactual reasoning fits naturally with the problem setting, and the proposed solution is supported by careful theoretical analysis.
Critical weaknesses
  1. The experiments rely on simulated and semi-synthetic acquisition shifts. While these controlled settings are useful for evaluating the proposed method, it remains unclear whether the observed improvements would carry over to naturally occurring acquisition shifts.
  2. The Camelyon17 experiments mainly use fixed pretrained image features followed by PCA and relatively small prediction heads. While these experiments demonstrate the effectiveness of the method in simplified settings, it remains unclear how well it performs when training or fine-tuning modern image classifiers end to end.
Questions for authors
  1. The experiments mainly use fixed image representations, while the theoretical analysis considers a predefined predictor class. Does the theoretical guarantee still hold when the image feature extractor is trained jointly with the classifier?
  2. How does increasing classifier capacity affect the accuracy of conditional risk estimation and the benefit of the proposed correction? Does the method retain its computational efficiency and empirical advantages on larger models and more complex datasets?
  3. How robust is the proposed method to violations of its identification assumptions, particularly conditional exchangeability? What are the implications of unobserved confounding in acquisition assignment for the estimated target risk and the resulting classifier?
1 comment

Thank you for your constructive comments.

Weakness 1 and 2. We have added end-to-end CNN experiments using simulated labels and the original Camelyon17 tumor labels. We fine-tune both the encoder and classification head of an ImageNet-pretrained ResNet-18.

Within each experiment, ERM, policy weighting, clipped weighting, and correction use the same training data, initialization, and training budget. The weighted methods also share the estimated acquisition probabilities within each condition. In both CNN experiments, we vary the amount of data used to estimate acquisition probabilities.

In the end-to-end CNN experiment with simulated labels, policy weighting and correction both had lower mean target Brier risk than ERM. Correction gains were supported over unclipped weighting in two conditions and over clipped weighting in one condition; the remaining comparisons were uncertain (Appendix A.1.1, Table 8).

With the original tumor labels, we use two fixed hematoxylin and eosin (H&E) stain profiles estimated from training images. Correction had lower mean target Brier risk than ERM and policy weighting in all three conditions, with relative reductions of 4.5–6.7% and 3.5–14.9%, respectively. However, the paired confidence intervals for the risk differences included zero, and clipped weighting had lower mean risk in some conditions (Section 4.2, lines 513–517; Appendix A.1.2, Tables 10–12).

Assignment to the two staining conditions is simulated; performance under naturally occurring changes in scanners or staining protocols remains untested (Section 5, lines 537–539).

Question 1. Yes, subject to the conditions in Theorem 1. A predefined predictor class can include an encoder and head whose parameters are jointly learned. The uniform empirical-process term, conditional-risk estimation error, and optimization error must still be controlled; fixing an architecture alone does not ensure this (Section 3.2, lines 308–323).

Question 2. The record weights can be computed once before CNN training, without refitting the conditional-risk regression at every update (Section 3.1, lines 236–273). This reduction is exact because, with fixed metadata features and regularization, the fitted conditional risks are linear in the regression responses.

A larger predictor class requires accurate conditional-risk estimates over a larger set of predictors; increasing network size alone does not ensure a larger correction gain. The empirical comparison is limited to ResNet-18 on Camelyon17 (Section 4.2, lines 504–508; Appendix A.1.2, Table 10).

Question 3. Correction does not remove unobserved confounding. If conditional exchangeability fails, the risk under the planned policy may not be identifiable from the historical data. The resulting criterion may therefore select a classifier that is suboptimal under the planned policy (Section 2, lines 102–110; Section 5, lines 531–535).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Comment to All Reviewers

We thank all reviewers for their time and effort in reviewing our manuscript. The main revisions are marked in red in the revised manuscript. Section, appendix, table, and line references in our responses refer to this revised version.

  • We have added end-to-end experiments on Camelyon17 using both simulated labels and the original tumor labels provided with the dataset.
  • The matched simulation provides evidence of correction gains over unclipped policy weighting when acquisition probabilities are estimated (Appendix A, Table 2). The CNN experiment with simulated labels also showed correction gains over unclipped policy weighting in two of the three conditions, supported by simultaneous 95% confidence intervals (Appendix A.1.1, Table 8).
  • For the original labels, we simulate differences in hematoxylin and eosin (H&E) staining colors. Correction had lower mean target Brier risk than ERM and policy weighting, although the paired confidence intervals included zero (Section 4.2, lines 513–517; Appendix A.1.2, Tables 10–12). If accepted, we plan to add independent training runs under the same protocol for the camera-ready version to estimate the differences in target Brier risk more precisely.
  • We have clarified the interpretation of Brier risk and the conditions for jointly trained encoders (Section 3, lines 151–153 and 183–187; Section 3.2, lines 321–323).
  • We have also revised wording and notation throughout the manuscript and clarified the implementation and evaluation details. The appendix expands the asymptotic proofs for a fixed predictor and specifies the rate conditions (Appendices E and J.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.