acceptodds
Under review as a conference paper at ICLR 2027

Training Scores for Conditional Audits: Overlap Weighting and Power under Confounding

10 views

Abstract

Conditional audits often test a score trained for marginal prediction, even though that objective rewards the very confounders the audit conditions on. We study whether train-only overlap weighting can recover conditional signal lost during score construction. The procedure estimates discrete-stratum label proportions in training data, fits a weighted classifier, and freezes its scores before an independent exact-stratum permutation test. We establish its training-balance target and explain why independent testing preserves the randomization guarantee. A Gaussian power ceiling separates score-learning loss from limited overlap. Across 25 synthetic configurations, we evaluate effect size, overlap, sample size, noise, and confounder measurement error. In a fresh 500-seed alternative block, overlap weighting raises forest detection from 61.6% to 79.4%, a paired gain of 17.8 percentage points [13.4, 22.4], while its null rejection rate is 4.0%. Global class balancing yields 60.8% detection. Additional continuous-confound and repeated-clip experiments delineate the test's assumptions. The contribution is a controlled score-training study and reproducible audit procedure built from established weighting and randomization tools.

Primary area:
causal reasoning
Keywords:
Conditional audits, overlap weighting, confounding, permutation test, statistical power, covariate balance, model auditing, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TVLrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether overlap-weighted classifier training improves the power of an independent conditional audit. Training weights balance labels within discrete confounder strata; the resulting scores are frozen before within-stratum label permutation. The paper establishes the weighting target, explains conditional-test validity, derives a Gaussian power ceiling, and evaluates synthetic settings involving overlap, noise, confounder measurement error, and repeated observations. In 500 fresh alternative trials, weighted forests increase detection from 61.6% to 79.4%, with 4.0% rejection under the null.

Critical strengths

A useful intervention supported by a controlled comparison. The fresh-seed experiment holds datasets, learner settings, and permutation draws fixed. The weighted forest improves detection by 17.8 percentage points, with reported paired interval [13.4, 22.4], while global class balancing provides no comparable gain. This supports the specific value of within-stratum weighting in the studied setting.

Careful separation of validity and power. Independent training preserves the exact-stratum randomisation argument, while the overlap diagnostic and Gaussian ceiling distinguish limited test information from score-learning loss. The paper appropriately avoids extending these guarantees to arbitrary continuous-confounder binning or clinical inference.

Critical weaknesses

The demonstrated benefit is restricted to a narrow synthetic family. The central generator places the added label signal in a dedicated feature, and fresh seeds validate the same family. The continuous-confounder and repeated-clip experiments test randomization assumptions using fixed scores rather than the proposed training intervention. Thus, evidence for weighting’s effectiveness when signal and confounding interact within features remains limited.

The incremental contribution over other conditional-score constructions is not established. Comparisons include ordinary training, global class balancing, and an oracle coordinate, but not a learned score explicitly adjusting for strata or optimizing within-stratum discrimination. Given the established ingredients, such a comparison would help determine whether overlap weighting offers a distinctive practical advantage.

Questions for authors

Does the weighting benefit persist when conditional signal is distributed across features, interacts with confounders, or reverses direction across strata?

How does it compare with a stratum-adjusted or within-stratum discrimination objective using the same learners and permutation test?

Can the separate supplement provide the complete per-seed records and unchanged implementation for both the 7.8% weighted-logistic null result and the subsequent 5.02% result? The latter is reassuring, but cannot be independently checked from this PDF.

Reviewer_KDgzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a classifier trained for marginal prediction can lean on confounders and lose the information needed to detect within-stratum label associations, i.e., so a good predictor can be a poor score for a conditional independence test.

Critical strengths
  • The paper shows why good classification performance doesn't necessarily mean a model is useful for detecting signals beyond confounders.
  • The experimental setup is well designed in my perspective, with independent runs and several useful baselines, including class balancing, an oracle, and a metadata-only model.
  • The authors showed test cases where the method fails, particularly when confounders are not fully observed, rather than only reporting favorable results.
Critical weaknesses
  • Although the experiments are well designed, they are limited to synthetic data generated under assumptions that fit the proposed method. - -- Moreover, there is no evaluation using real learned representations or realistic confounders, where the problem would be more challenging.
  • The method also seems quite sensitive to whether the confounders are correctly identified. The authors show that incomplete conditioning can lead to a 90% rejection rate even when there is no true signal. Since real datasets often have missing or imperfect confounder information, I think this limitation deserves more attention/explanation. Also, several parts of the approach build on existing methods, so stronger practical evidence would help establish its contribution.
  • The calibration being the strongest result --> The weighted logistic model achieves 95.2%, but its null rejection rate is 7.8%, above the expected 5%. The authors address this with additional runs showing a rate of 5.02%, which is reassuring. However, the power/calibration results come from different sets of experiments, making the main result a little harder to interpret.
Questions for authors
  1. Any results on real datasets with demographic or acquisition-related confounders? How would the method handle incomplete or imperfect confounder information?
  2. How does the approach compare with methods that explicitly account for confounders during training, such as residualization or conditional-score learning?
  3. For the weighted logistic model, could you report both power and null rejection using the same 5,000 runs sets?
  4. Am I correct that the continuous-confounder experiment evaluates the permutation test using a fixed score, rather than the proposed training method? Could you clarify what this experiment tells us about your approach?
  5. Have you considered using estimated confounder probabilities together with a trained classifier, rather than assuming the true probabilities are known? Would the conclusions still hold?
Reviewer_dAJaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigate the shortcut learning problem in video-based clinical applications, trying to answer whether the one-dimensional score retains sufficient information related to the cohort of one label, or instead, just shortcut, biased information for prediction of the label. The effect of class unbalance is investigated through class-balanced forests, with all positive and negative data points multiplied by two constant values, or called factors. A procedure of overlap-trained conditional audit is proposed. Synthetic data is created to verify the proposal.

Critical strengths

Synthetic data is generated for verifying the proposed procedure. The results are therefore well reproducible.

Critical weaknesses

The introduction doesn’t cite any papers to give the audience a clear background.

Many notations in the equations are not defined, e.g. h() in eq. 1, l() in eq. 6.

There is a lack of problem statement and clear introduction of the motivating examples. During the theoretical discussion, the terms in the application, e.g. video embeddings, dementia mechanism, are suddenly used without any context.

The proposal is only tested on synthetic data that follows simple distributions. How are the distribution parameters determined?

Questions for authors

If the distribution hyperparameters are changed, will the conclusions still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.