acceptodds
Under review as a conference paper at ICLR 2027

Separating Covariate Shift from Mechanism Change with Two Discriminators

Abstract

When labeled datasets differ, deciding what to inspect next requires separating a change in input frequencies from a change in the labeling rule. We develop a two-component diagnostic based on a balanced dataset source indicator : input information and conditional information , called conditional Jensen–Shannon discrepancy (CJSD). The components add to the joint Jensen–Shannon divergence. CJSD isolates label-rule differences on shared support, while the input component reveals how much conditional information remains observable. Two source classifiers estimate both components through held-out log losses, whose record-level differences connect a global score to regions worth inspecting. Controlled dataset pairs illustrate the separation and localize an imposed change in electricity records. Unaltered gas-sensor measurements demonstrate why a small conditional score can coexist with poor transfer when the inputs strongly separate. An equal-observation comparison with established conditional-information estimators assesses magnitude recovery, null behavior, and rotation sensitivity. Together, these results provide a practical way to read dataset differences and the limits of the evidence they supply.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
distribution shift, covariate shift, concept drift, conditional mutual information, discriminator-based estimation, Jensen-Shannon divergence, dataset diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wb4GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides an information-theoretic perspective on separating covariate shift from label-rule (mechanism) change. The authors inspect I(Z;X,Y)I(Z;X,Y) (the mutual information between the source indicator and the (input, target) pair) and decompose it into a sum of I(Z;X)I(Z;X) and I(Z;Y∣X)I(Z;Y \mid X), where the first term diagnoses covariate shift and the second diagnoses the rule change. All three quantities can be expressed in terms of the Jensen-Shannon divergence under the Z∼Bern(1/2)Z \sim \mathrm{Bern}(1/2) assumption. The authors suggest using classifier-based estimators for the I(Z;X)I(Z;X) and I(Z;Y∣X)I(Z;Y \mid X) terms to estimate the magnitude of covariate shift and mechanism change. They validate their approach on various setups (e.g., synthetic data, Electricity records). The results indicate that the proposed measures are partially informative for diagnosing the shifts and changes in question.

Critical strengths

The paper is easy to read. The proposed quantities are intuitive and rooted in information theory. The connection to the JS divergence is also neat. The analysis of why the two numbers must be read together is clean and useful for practical applications.

Critical weaknesses
  1. The contribution is modest, to say the least. While it is, to the best of my knowledge, solid, the provided analysis is neither sufficiently insightful nor immediately useful to justify acceptance at an A* conference.
  2. The proposed estimation method is biased, since the quantities are estimated via the difference in held-out classification log-losses. The work does identify this bias as the difference of two excess risks and gives a formal bound (Eq. 6), but that bound depends on unknown quantities and is not estimable from the data, so the practical size of the bias remains uncontrolled.
Questions for authors

Have you tried a variational lower bound on the JS divergence to at least pin the sign of the bias?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wb4G

We thank the reviewer for the accurate summary and the constructive question. The revised PDF marks changes in blue.

W2 (bias). We agree: the bound in Eq. 6 depends on unknown excess risks and cannot be evaluated from data. The revision complements it with an empirical calibration (Section 3): resample all labels from one pooled model of p(Y|X), rerun the same classifier pair, and compare the observed score with the resulting null distribution. On the submission's Table 1 settings, unchanged pairs are called resolved in 2 of 128 cases, a 0.30-bit change is resolved in 32 of 32 cases with the MLP and 25 of 32 with HGB, and the estimates that worried all three reviewers (HGB 0.001 for a true 0.30; MLP -0.011 for a true 0.03, rotated d = 128) are reported as not resolved. The calibration controls the decision, not the magnitude: attenuation remains and the scope paragraph says so.

Question (variational lower bound to pin the sign). Each held-out log loss already gives a variational lower bound on its own term: 1 - L_X bounds I(Z;X) from below, and 1 - L_XY bounds I(Z;X,Y) from below (bits, balanced Z). The conditional component is their difference, and a difference of two lower bounds has no fixed sign. Following your suggestion we tried to pin it with a nested construction: the joint classifier receives the input classifier's held-out log-odds as a feature, so in principle it can reproduce the input classifier. It did not work in finite samples, and we report that (Appendix F): negative estimates remain in 36 of 128 HGB cells and 61 of 128 MLP cells (plain estimator: 34 and 57), and for the overfitted d = 128 MLP the nested estimate is worse (mean -0.155 under the null, +0.29 under pure input shift). We therefore keep the signed estimate and use the null calibration to say when its sign and size are not informative.

W1 (size of the contribution). We understand the concern. The paper's claim is a way to read two numbers together and to know when the conditional number cannot be trusted; the calibration makes the second part operational and the randomized localization study (Section 4.2) tests the first on records with imposed changes. We have kept the claims at that size.

Reviewer_uU6GOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper mainly build on existing ingredients, e.g, MI chain rule, and classifier-based CMI estimation.

Detecting covariate shift versus conditional shift boils down to marginal and conditional independence testing. For scoring, other measures of dependence could be considered instead of MI, given the challenges of estimating it. What is less clear to me is the practical value of estimating the magnitude of these shifts. The authors need to empirically establish its practical relevance.

Also, the estimated loss difference includes unequal errors from the two classifiers. The strong discrimination AUC does not establish accurate CMI estimation.

Overall, I find the formulation useful, but I am not convinced that the method or its demonstrated practical benefits offer a substantial advance over existing approaches

Critical strengths

I think the problem is relevant, and the proposed procedure is simple.

Critical weaknesses

The paper mainly builds on existing ingredients, e.g., the MI chain rule and classifier-based CMI estimation.

Detecting covariate shift versus conditional shift boils down to marginal and conditional independence testing. For scoring, other measures of dependence could be considered instead of MI, given the challenges of estimating it. What is less clear to me is the practical value of estimating the magnitude of these shifts. The authors need to empirically establish its practical relevance.

Also, the estimated loss difference includes unequal errors from the two classifiers. Strong discrimination AUC does not establish accurate CMI estimation.

Overall, I find the formulation useful, but I am not convinced that the method or its demonstrated practical benefits offer a substantial advance over existing approaches.

Questions for authors

What are the main advantages of the proposed estimator over existing classifier-based CMI estimators, and can these be demonstrated under comparable fitting budgets?

In which experiments does using the magnitude of the score improve a downstream decision beyond what an existing conditional two-sample test provides?

How accurately does the estimator recover ground-truth CMI across challenging settings?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uU6G

We thank the reviewer for the comments and the three concrete questions. The revised PDF marks changes in blue.

On novelty. We agree that the chain rule and classifier-based estimation are known, and we do not claim a new estimator class (the introduction says the contribution is "this diagnostic formulation, implementation, and validation, rather than a new chain rule or a new information quantity"). What the paper adds is the reading: with a balanced source indicator the two components add to the joint Jensen-Shannon divergence, the input component bounds how much conditional information is observable, record-level loss differences connect the global score to regions to inspect, and, new in the revision, a calibration that says when the conditional score cannot be distinguished from classifier error.

On other dependence measures and on testing versus magnitude. A conditional independence test with any dependence measure answers whether a difference exists. We use log loss because it gives three things other measures do not give together: the input and conditional components add exactly to one joint divergence in common units, the input component caps the conditional one (Eq. 7), and the score decomposes over records. The revision adds the test-like reading you ask about as a complement, not a replacement (below).

Q1 (advantage over existing classifier-based CMI estimators, comparable budgets). Section 5 and Appendix E compare the paired-loss route with CCMI and the Mesner-Shalizi nearest-neighbor estimator on the same 12,000 observations; the two neural routes share architecture, update count and examples per update, with no tuning. RMSE over all 128 cells is 0.039 (paired MLP), 0.069 (CCMI), 0.107 (paired HGB), 0.113 (MS). The advantage is one objective for both components and record-level contributions; we do not claim uniform superiority, and HGB loses to CCMI on rotated inputs. A comparison with kernel- and classifier-based conditional independence tests (KCIT, CCIT) on the same settings is not in this revision; we are running it and will post the result in this thread.

Q2 (downstream decisions). The submission had no experiment where the score drives a decision. The revision adds one: choosing which records to inspect, with 50 randomly placed label changes in the electricity records (Section 4.2). Precision@k, mean ± SD over placements:

Record scoreChanged records, mass .10mass .30Band, mass .10mass .30
Paired loss difference (ours)0.49 ± 0.160.69 ± 0.040.46 ± 0.170.69 ± 0.05
Label loss, model fitted on A0.53 ± 0.100.72 ± 0.020.34 ± 0.050.53 ± 0.01
Label loss, pooled model0.23 ± 0.070.61 ± 0.020.16 ± 0.050.58 ± 0.02
Predictive entropy, pooled model0.30 ± 0.080.73 ± 0.030.30 ± 0.070.73 ± 0.03
Random0.10 ± 0.010.30 ± 0.010.10 ± 0.010.30 ± 0.01

The record contributions find changed records at two to five times the chance rate and are best for locating a small band; a label model fitted on a designated reference source is equally good or slightly better on individual records. We report this as is.

Q3 (ground-truth recovery in hard settings). Table 2 of the revision (Table 1 of the submission) reports recovery including the failures. The revision adds a conditional-null calibration that turns each estimate into "resolved" or "not resolved" (procedure in our reply to Reviewer SLQ2). Over all settings, unchanged pairs are called resolved in 2 of 128 cases; a 0.30-bit change is resolved in 32 of 32 cases (MLP) and 25 of 32 (HGB); a 0.03-bit change in 15 of 32 and 12 of 32. At rotated d = 128 neither learner resolves 0.03, and HGB resolves 0.30 in only 2 of 8 seeds (Appendix F, Table 11).

On AUC and unequal classifier errors. We agree that discrimination AUC does not establish estimation accuracy, and that the loss difference contains unequal errors. The calibration above measures exactly this error for the classifier pair in use and reports when it dominates.

AuthorsOpenReview ↗

Follow-up for Reviewer uU6G: comparison with CCIT and KCIT

As promised in our response, here is the comparison with conditional independence tests on the same laws (the submission's Table 1 settings: d = 8 and 128, axis-aligned and rotated inputs, four conditions, the same eight seeds, 6,000 records per source). The hypothesis tested is that the source is independent of the label given the inputs. Entries are the number of seeds, out of 8, in which each method rejects at the 5% level; the first two columns should be near zero and the last two near eight.

Method (rows used)dGeometryNullInput shiftRule (.03)Rule (.30)
Calibrated paired MLP (12,000)8axis0/80/87/88/8
Calibrated paired HGB (12,000)8axis0/80/88/88/8
CCIT (12,000)8axis0/80/80/87/8
KCIT, default widths (2,000)8axis5/80/85/83/8
KCIT, median widths (2,000)8axis0/80/80/83/8
Calibrated paired MLP (12,000)8rot0/81/88/88/8
Calibrated paired HGB (12,000)8rot0/80/82/88/8
CCIT (12,000)8rot0/80/80/86/8
KCIT, default widths (2,000)8rot2/83/82/82/8
KCIT, median widths (2,000)8rot1/81/80/82/8
Calibrated paired MLP (12,000)128axis0/80/80/88/8
Calibrated paired HGB (12,000)128axis0/80/82/87/8
CCIT (12,000)128axis0/80/80/81/8
KCIT, default widths (2,000)128axis8/88/88/88/8
KCIT, median widths (2,000)128axis4/83/83/82/8
Calibrated paired MLP (12,000)128rot0/80/80/88/8
Calibrated paired HGB (12,000)128rot0/81/80/82/8
CCIT (12,000)128rot0/80/80/80/8
KCIT, default widths (2,000)128rot8/88/88/88/8
KCIT, median widths (2,000)128rot3/83/84/81/8

How the tests were run. CCIT uses the authors' package with its default settings on all 12,000 records. KCIT uses the causal-learn implementation with its default (empirical) kernel widths and, as one alternative fixed in advance, median widths. KCIT has cubic cost, so it runs on a random source-balanced subsample of 2,000 records; a single test on all 12,000 did not finish in more than ten minutes on 46 cores. Nothing was tuned on these results. The calibrated rows are the paired-loss estimates with the conditional-null calibration from the revision (Appendix F).

What we read from it.

  1. CCIT controls false positives (0 of 64 unchanged pairs) and detects the 0.30-bit change at d = 8 (13 of 16) but rarely at d = 128 (1 of 16). It never detects the 0.03-bit change. The calibrated paired MLP detects 0.30 in 32 of 32 cases and 0.03 in 15 of 32, with 1 false positive in 64.

  2. KCIT is not calibrated on these laws in either configuration: with default widths it rejects every pair at d = 128, changed or not, and with median widths it still rejects 15 of 64 unchanged pairs. We do not take this as a general statement about KCIT. The labels here are a deterministic function of the inputs and the conditioning set has up to 128 dimensions, which is far from the setting the test was designed for, and a tuned kernel may do better.

  3. On your second question, this is the direct comparison with an existing conditional two-sample test under equal data. For the decision "is there a conditional difference", the calibrated score is at least as sensitive as CCIT in every setting, at the same false-positive level. In addition it returns a magnitude in bits and per-record contributions, which a test does not.

Limits. These are constructed laws with one sample size, and CCIT and KCIT were run with default settings, as were our learners. The rotated d = 128 setting remains hard for everyone: no calibrated method detects the 0.03 change there, and only the paired MLP detects the 0.30 change reliably.

Thank you for adding the conditional independence tests and the matched-budget estimator comparison. The localization experiment gives a concrete use for the per-record scores. My remaining question is whether knowing the magnitude of the overall shift in bits improves a downstream decision. Detecting a shift and ranking records for inspection can both be useful without requiring an accurate estimate of that magnitude.

Reviewer_SLQ2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper separates dataset differences into input shift and conditional label change using an information decomposition based on dataset-source indicators. Two classifiers estimate the components through held-out log losses, while individual loss differences provide descriptive regional contributions. Experiments examine controlled shifts, natural sensor data, and comparisons with existing conditional-information estimators.

Critical strengths
  1. The formulation clearly explains the role of shared support and why a small conditional score does not necessarily imply unchanged labeling rules.
  2. The controlled comparisons show improved overall estimation accuracy for paired MLPs, with transparent reporting of null controls and failure cases.
Critical weaknesses

W1. Insufficient reliability of the fitted diagnostic. The fitted score combines conditional discrepancy with an uncontrolled difference in classifier errors (Eq. 6). On rotated 128-dimensional inputs, HGB estimates a true 0.30-bit discrepancy as 0.001, while MLP estimates 0.03 as −0.017 (Table 1). No validated procedure distinguishes genuinely small changes from classifier failure, limiting the reliability of the proposed diagnosis. W2. Limited validation of the practical contribution. Evidence for successful diagnosis mainly comes from synthetic or artificially modified datasets. The localization example uses a preselected remapping region, while the natural gas-sensor study has unknown conditional discrepancy and strongly separated inputs. These experiments do not establish reliable identification or localization of genuine conditional changes, leaving the central practical contribution insufficiently demonstrated.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer SLQ2 (1/2): W1, reliability of the fitted diagnostic

We thank the reviewer for the careful reading, in particular of Eq. 6 and Table 1. Both weaknesses asked for evidence we did not have, so we ran the experiments; the revised PDF contains them (changes in blue).

W1 (reliability of the fitted diagnostic). We agree with the core point: the fitted score is the conditional discrepancy plus a difference of two excess risks, and the submission gave no validated way to tell a small true change from a classifier failure. The revision adds one (Section 3, "Calibrate the score against a conditional null").

Procedure. Fit one pooled model of p(Y|X) that ignores the source, resample every label from it, and rerun the whole estimator (same learners and budget, a freshly drawn fold split, a fresh training seed). Inputs are untouched and the label-rule difference is removed by construction, so 100 such replicates give the distribution of the fitted score for this classifier pair on these inputs when there is no conditional difference. The observed score is reported with the null 95% interval and a one-sided p-value; a score with p > .05 is reported as "not resolved", not as "no change".

Result on the rows you cite (rotated d = 128, the submission's Table 1 laws, folds and learners; eight seeds; bits):

LearnerdGeometryCondition (target)Estimate (mean ± SD)Null 95% intervalResolved (p ≤ .05)Rotation SD
HGB128rotNull-0.001 ± 0.003[-0.005, +0.005]0/80.002
HGB128rotInput shift-0.000 ± 0.002[-0.003, +0.003]1/80.001
HGB128rotRule (.03)+0.000 ± 0.002[-0.005, +0.005]0/80.002
HGB128rotRule (.30)+0.001 ± 0.004[-0.005, +0.005]2/80.004
MLP128rotNull-0.016 ± 0.034[-0.066, +0.064]0/80.031
MLP128rotInput shift-0.000 ± 0.024[-0.042, +0.048]0/80.021
MLP128rotRule (.03)-0.011 ± 0.021[-0.067, +0.063]0/80.033
MLP128rotRule (.30)+0.245 ± 0.030[-0.067, +0.061]8/80.038

The two rows you point to are now labeled for what they are. HGB estimates the 0.30 change as 0.001 and the procedure resolves it in only 2 of 8 seeds: it reports a learner failure where the raw score suggested a small change. The MLP estimate of -0.011 for the 0.03 change lies inside a null interval 0.13 bits wide and is reported as not resolved in 8 of 8 seeds, while the 0.30 change is resolved in 8 of 8.

Over all Table 1 settings (d = 8 and 128, axis-aligned and rotated; Appendix F, Table 11), pairs with an unchanged rule are called resolved in 2 of 128 cases (nominal 5%). The 0.30 change is resolved in 32 of 32 cases with the MLP and 25 of 32 with HGB; the 0.03 change in 15 of 32 and 12 of 32.

What the calibration does not do. It decides presence, not magnitude: attenuation remains (0.245 for a true 0.30), and we state this in the scope paragraph. We also report two things that did not work. Reusing the observed fold split in every replicate collapses the null of a deterministic learner and gives 6 of 64 false "resolved" calls for HGB, which is why the split is redrawn. A binary "unstable under rotation" flag marked correct detections as unstable, so we report the rotation spread as a number instead.

AuthorsOpenReview ↗

Response to Reviewer SLQ2 (2/2): W2, practical validation

(Continued: reply to W2.)

W2 (practical validation). The localization region in the electricity study was indeed fixed in advance. We repeated it with 50 randomly placed bands (random feature, random position, band mass 0.10 or 0.30; no method is told where the band is). Precision@k, k = number of positives, mean ± SD over placements (Section 4.2, Table 1 of the revision):

Record scoreChanged records, mass .10mass .30Band, mass .10mass .30
Paired loss difference (ours)0.49 ± 0.160.69 ± 0.040.46 ± 0.170.69 ± 0.05
Label loss, model fitted on A0.53 ± 0.100.72 ± 0.020.34 ± 0.050.53 ± 0.01
Label loss, pooled model0.23 ± 0.070.61 ± 0.020.16 ± 0.050.58 ± 0.02
Predictive entropy, pooled model0.30 ± 0.080.73 ± 0.030.30 ± 0.070.73 ± 0.03
Random0.10 ± 0.010.30 ± 0.010.10 ± 0.010.30 ± 0.01

Ranking records by their loss-difference contribution finds changed records at two to five times the chance rate and is the best of the compared scores for locating a small band. It is not uniformly best, and the revision says so: a label model fitted on the reference source alone ranks individual changed records equally well or slightly better, and for the large band the entropy of a pooled label model does as well. The contribution score needs no choice of reference source and also ranks source-A records inside the band.

For the gas-sensor data our claim is limited to the overlap effect: a small conditional score is not evidence of an unchanged rule when the inputs separate. Ground-truth conditional change is not observable in natural data, so imposed changes on real inputs remain our main evidence; the scope paragraph states this.

Comments on the paper

1 comment

Dear Reviewers,

The rebuttal period ends in less than two days. I kindly ask you to acknowledge the authors’ responses, and indicate whether your main concerns have been adequately addressed or further clarification is needed.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.