- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper provides an information-theoretic perspective on separating covariate shift from label-rule (mechanism) change. The authors inspect (the mutual information between the source indicator and the (input, target) pair) and decompose it into a sum of and , where the first term diagnoses covariate shift and the second diagnoses the rule change. All three quantities can be expressed in terms of the Jensen-Shannon divergence under the assumption. The authors suggest using classifier-based estimators for the and terms to estimate the magnitude of covariate shift and mechanism change. They validate their approach on various setups (e.g., synthetic data, Electricity records). The results indicate that the proposed measures are partially informative for diagnosing the shifts and changes in question.
Critical strengths
The paper is easy to read. The proposed quantities are intuitive and rooted in information theory. The connection to the JS divergence is also neat. The analysis of why the two numbers must be read together is clean and useful for practical applications.
Critical weaknesses
- The contribution is modest, to say the least. While it is, to the best of my knowledge, solid, the provided analysis is neither sufficiently insightful nor immediately useful to justify acceptance at an A* conference.
- The proposed estimation method is biased, since the quantities are estimated via the difference in held-out classification log-losses. The work does identify this bias as the difference of two excess risks and gives a formal bound (Eq. 6), but that bound depends on unknown quantities and is not estimable from the data, so the practical size of the bias remains uncontrolled.
Questions for authors
Have you tried a variational lower bound on the JS divergence to at least pin the sign of the bias?
Response to Reviewer wb4G
We thank the reviewer for the accurate summary and the constructive question. The revised PDF marks changes in blue.
W2 (bias). We agree: the bound in Eq. 6 depends on unknown excess risks and cannot be evaluated from data. The revision complements it with an empirical calibration (Section 3): resample all labels from one pooled model of p(Y|X), rerun the same classifier pair, and compare the observed score with the resulting null distribution. On the submission's Table 1 settings, unchanged pairs are called resolved in 2 of 128 cases, a 0.30-bit change is resolved in 32 of 32 cases with the MLP and 25 of 32 with HGB, and the estimates that worried all three reviewers (HGB 0.001 for a true 0.30; MLP -0.011 for a true 0.03, rotated d = 128) are reported as not resolved. The calibration controls the decision, not the magnitude: attenuation remains and the scope paragraph says so.
Question (variational lower bound to pin the sign). Each held-out log loss already gives a variational lower bound on its own term: 1 - L_X bounds I(Z;X) from below, and 1 - L_XY bounds I(Z;X,Y) from below (bits, balanced Z). The conditional component is their difference, and a difference of two lower bounds has no fixed sign. Following your suggestion we tried to pin it with a nested construction: the joint classifier receives the input classifier's held-out log-odds as a feature, so in principle it can reproduce the input classifier. It did not work in finite samples, and we report that (Appendix F): negative estimates remain in 36 of 128 HGB cells and 61 of 128 MLP cells (plain estimator: 34 and 57), and for the overfitted d = 128 MLP the nested estimate is worse (mean -0.155 under the null, +0.29 under pure input shift). We therefore keep the signed estimate and use the null calibration to say when its sign and size are not informative.
W1 (size of the contribution). We understand the concern. The paper's claim is a way to read two numbers together and to know when the conditional number cannot be trusted; the calibration makes the second part operational and the randomized localization study (Section 4.2) tests the first on records with imposed changes. We have kept the claims at that size.