- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether overlap-weighted classifier training improves the power of an independent conditional audit. Training weights balance labels within discrete confounder strata; the resulting scores are frozen before within-stratum label permutation. The paper establishes the weighting target, explains conditional-test validity, derives a Gaussian power ceiling, and evaluates synthetic settings involving overlap, noise, confounder measurement error, and repeated observations. In 500 fresh alternative trials, weighted forests increase detection from 61.6% to 79.4%, with 4.0% rejection under the null.
Critical strengths
A useful intervention supported by a controlled comparison. The fresh-seed experiment holds datasets, learner settings, and permutation draws fixed. The weighted forest improves detection by 17.8 percentage points, with reported paired interval [13.4, 22.4], while global class balancing provides no comparable gain. This supports the specific value of within-stratum weighting in the studied setting.
Careful separation of validity and power. Independent training preserves the exact-stratum randomisation argument, while the overlap diagnostic and Gaussian ceiling distinguish limited test information from score-learning loss. The paper appropriately avoids extending these guarantees to arbitrary continuous-confounder binning or clinical inference.
Critical weaknesses
The demonstrated benefit is restricted to a narrow synthetic family. The central generator places the added label signal in a dedicated feature, and fresh seeds validate the same family. The continuous-confounder and repeated-clip experiments test randomization assumptions using fixed scores rather than the proposed training intervention. Thus, evidence for weighting’s effectiveness when signal and confounding interact within features remains limited.
The incremental contribution over other conditional-score constructions is not established. Comparisons include ordinary training, global class balancing, and an oracle coordinate, but not a learned score explicitly adjusting for strata or optimizing within-stratum discrimination. Given the established ingredients, such a comparison would help determine whether overlap weighting offers a distinctive practical advantage.
Questions for authors
Does the weighting benefit persist when conditional signal is distributed across features, interacts with confounders, or reverses direction across strata?
How does it compare with a stratum-adjusted or within-stratum discrimination objective using the same learners and permutation test?
Can the separate supplement provide the complete per-seed records and unchanged implementation for both the 7.8% weighted-logistic null result and the subsequent 5.02% result? The latter is reassuring, but cannot be independently checked from this PDF.