acceptodds
Under review as a conference paper at ICLR 2027

HellingerIEP: Stable Domain Adaptation under Contaminated Covariate Shift

Abstract

Importance weighting is a standard approach to domain adaptation under covariate shift, but direct density-ratio estimation can be unstable when the unlabeled target sample is contaminated. Existing robust methods often rely on user-specified trimming or reweighting mechanisms. We study how the divergence geometry of a density-ratio objective affects contamination sensitivity. We introduce HellingerIEP, a nonnegative kernel estimator derived from a variational Hellinger objective. Parameterizing the square root of the density ratio yields a convex optimization problem, and, for a fixed kernel dictionary and regularization level, we establish consistency for the corresponding regularized population solution and asymptotic normality under an interiority condition. Under a common square-root ratio parameterization, the Hellinger, KL, and Pearson target-score contributions contain factors of orders , , and , respectively. Thus, Hellinger attenuates added-point score contributions more strongly in high-ratio regions but has greater score magnitude in low-ratio regions, so no uniform sensitivity ordering follows. This nonuniformity suggests a stability–adaptation trade-off, since a large fitted ratio may represent either contamination or genuine target shift. Experiments on synthetic and benchmark domain adaptation tasks show that HellingerIEP is competitive in clean settings and more stable under the contamination regimes considered.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Domain adaptation, covariate shift, density-ratio estimation, importance weighting, Hellinger divergence, f-divergence, target contamination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3HfTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HellingerIEP, a kernel-based density-ratio estimator for domain adaptation under contaminated covariate shift. The method uses a variational Hellinger objective and a square-root parameterization to obtain a convex optimization problem. The authors establish consistency and asymptotic normality under fixed-dimensional assumptions and analyze local sensitivity to target contamination. Experiments show improved stability over several density-ratio estimation methods. I currently lean toward rejection, but I would be willing to reconsider my rating if the authors provide convincing clarification and evidence addressing these concerns.

Critical strengths
  1. The paper addresses a relevant problem, as density-ratio estimation can be unstable when the unlabeled target sample is contaminated.

  2. The square-root parameterization gives a convenient convex formulation with nonnegative density-ratio estimates.

  3. The experiments cover both density-ratio estimation and downstream prediction, with comparisons to several existing methods.

Critical weaknesses
  1. The methodological novelty is somewhat limited. Hellinger divergence and variational density-ratio estimation have been studied previously, so the main contribution appears to be their particular kernel implementation and contamination analysis.

  2. The theoretical analysis focuses on local score sensitivity, whereas the empirical claim concerns stability of the complete estimator under substantial contamination. The connection between these two levels of analysis is not fully established.

  3. The empirical improvements are encouraging, but the contamination mechanisms are highly controlled. It remains unclear how well the method handles contamination that resembles genuine target-domain shift.

Questions for authors

1. In Proposition 3.1 (Page 5), the comparison with KLIEP and uLSIF is based on target-score terms under a common square-root parameterization. Since the methods have different curvature matrices, how much of the reported stability difference is explained by the score terms alone? Could the authors provide a numerical comparison of the full contamination influence?

2. The kernel centers are selected from the target sample, which may itself be contaminated. How sensitive is HellingerIEP to this choice? For example, would using centers selected from the source sample or the clean portion of the target sample lead to substantially different results?

3. In Section 4, the Gaussian density-ratio experiment replaces 30% of target observations with the clean target mean. This creates a concentrated spike rather than a conventional outlier distribution. How would the method perform when the contamination occurs in a region with low source density, where the fitted density ratio may behave differently?

4. Theorem 3.1 establishes convergence to the regularized population minimizer wλ∗w_\lambda^*, rather than the true density ratio rr. Could the authors quantify how the regularization and kernel approximation errors affect downstream importance-weighted prediction?

5. In the benchmark experiments, contaminated observations are replaced by the mean feature vector, while prediction is evaluated on the clean target distribution. How does HellingerIEP perform when the target distribution contains a genuine high-density-ratio subpopulation that should receive large importance weights rather than be treated as contamination?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3HfT (1/2)

We want to thank the reviewer for their careful comments and for expressing openness to reconsider their rating. Below we address each of the stated criticisms separately, but we want to first stress how the reviewer's comments helped us to make our arguments and presentation stronger. The nonuniform sensitivity pattern suggested at the score level persists after accounting for curvature and the constraints, and HellingerIEP adapts to a genuine high-ratio subpopulation while remaining stable under central contamination. uLSIF shows exactly the opposite behavior, with high sensitivity to central contamination and practically no sensitivity to contamination outside the effective source domain. KLIEP's sensitivity is mostly uniform in the location of the contamination, and NEDIEP interpolates between the benefits of HellingerIEP and uLSIF. We added a discussion about these limitations and differences in sensitivity to the main manuscript. Below we give detailed responses to the reviewer's questions.

Q1 / W2. Score terms versus complete fitted-ratio sensitivity.

We agree that the score factors alone do not determine the sensitivity of the fitted estimator. We therefore added Appendix J, which computes the empirical local derivative of the normalized fitted ratio, conditional on the selected dictionary and hyperparameters. The calculation includes the centered contamination score, the method-specific inverse curvature, the normalization, and the derivative of the fitted-ratio map, and we compare the derivative with an exact one-point refit. For w~=w/Pn,Sw\tilde w=w/P_{n,S}w, we report

I(z)=∥dw~dε∣ε=0∥L2(Pn,S)\mathcal{I}(z)=\left\Vert \left.\frac{d\tilde{w}}{d\varepsilon}\right|_{\varepsilon=0} \right\Vert_{L_2(P_{n,S})} over point-mass contamination at z∈[−10,10]z\in[-10,10].

As stated above, considering curvature and constraints, HellingerIEP has the smallest fitted-ratio influence near the target center, where the ratio is high, and the largest in low-ratio regions. The empirical results at the level of the fitted ratio reinforce the ordering suggested by Proposition 3.1 from the score factors alone.

Q2. Dependence on contaminated kernel centers.

In Appendix I we replicate the downstream experiment of Appendix F.3 with target-selected, source-selected, and pooled dictionaries, keeping the number of candidate centers and the tuning grid the same for all methods.

The results indicate that the central-contamination advantage is not created by contaminated target-selected centers. Under 30%30\% contamination at the target center, HellingerIEP remains close to oracle weighting in d=1,5,10d=1,5,10, and its RMSE changes very little across the three dictionaries. In particular, its performance persists with source-selected centers. In the new Table 2 we report the average coefficient of variation (CV) across the different settings, and it shows that HellingerIEP is most stable across the different dictionaries. In the clean case, for example, HellingerIEP has an average CV of 0.07%0.07\% whereas KLIEP is a distant second with an average CV of 0.23%0.23\%. In the contaminated case, under both z=0z=0 and z=6z=6 contamination, HellingerIEP has the lowest CV.

Q3. Contamination in low-source-density regions.

We added an experiment in Appendix K.2 where we move a 30%30\% spike over z∈{0,…,10}z\in\{0,\ldots,10\}, from the central high-ratio region into regions where both the clean density ratio and the source density are small; the clean ratio decreases from r(0)≈1.82r(0)\approx1.82 to r(4)≈0.25r(4)\approx0.25 and r(6)≈0.02r(6)\approx0.02.

Similarly, in Appendix K.3 we consider a heavy-tailed Cauchy contamination. As suggested by the local sensitivity analysis above, the results are nonuniform and HellingerIEP is most advantageous with contamination in the high-ratio regions. For z≤2z\le2, HellingerIEP has the lowest ratio NMSE and the best or close-to-best prediction RMSE, and its ratio NMSE stays below KLIEP's and uLSIF's at every location. For z≳4z\gtrsim4, HellingerIEP still outperforms KLIEP and uLSIF in prediction RMSE, but NEDIEP and RuLSIF achieve even lower prediction RMSE. Under Cauchy contamination the behavior is dimension-dependent: uLSIF, RuLSIF, and NEDIEP retain their adaptation benefit in univariate problems, but as the dimension and contamination proportion increase no method clearly improves on source-only fitting. This is in line with the local sensitivity results, because the contamination then places more and more points in regions of negligible source density.

AuthorsOpenReview ↗

Response to Reviewer 3HfT (2/2)

Q4. Convergence to the regularized solution and downstream risk.

Reviewer QUEC raises a similar issue (W5) and we added the following target-risk decomposition, which is summarized at the end of Section 3 and stated in full in the new Appendix C.4. Under the covariate-shift assumption for the clean source and target distributions, for a fixed predictor hh and a target fitting sample contaminated at level ε∈[0,1)\varepsilon \in [0, 1), we have ∣EPS[w^nℓh]−RT(h)∣≤∥ℓh∥L2(PS)(∥w^n−wλ,ε⋆∥+∥wλ,ε⋆−wλ⋆∥+∥wλ⋆−r∥),\left|\mathbb E_{P_S}[\widehat w_n\ell_h]-R_T(h)\right|\le\Vert\ell_h\Vert_{L_2(P_S)}\Big(\Vert\widehat w_n-w_{\lambda,\varepsilon}^\star\Vert+\Vert w_{\lambda,\varepsilon}^\star-w_\lambda^\star\Vert+\Vert w_\lambda^\star-r\Vert\Big), where the three norms inside the parentheses are in L2(PSX)L_2(P_S^X), wλ,ε⋆w_{\lambda,\varepsilon}^\star is the population solution under contamination, and rr is the clean target-to-source density ratio. In the clean case, the middle term vanishes. Our theory provides an estimation rate and identifies a local contamination-sensitivity mechanism. Theorem 3.1 gives an Op(n−1/2)O_p(n^{-1/2}) estimation rate under its assumptions and Proposition 3.1 identifies the ratio-dependent target-score factors governing local contamination sensitivity. As detailed above, the new Appendix J evaluates the resulting normalized fitted-ratio influence after accounting for curvature and constraints. The reviewer is correct that our theory does not give a rate for the combined kernel-approximation and regularization bias. Appendix C.4 also shows how this term enters. Because wλ⋆w_\lambda^\star minimizes the penalized population criterion,

J(w_\lambda^\star)-J(r)\le\inf_{\beta\in\mathcal C}\left\\{\mathbb E_{P_S}\left[\frac{\\{g_\beta(X)-\sqrt{r(X)}\\}^{2}}{g_\beta(X)}\right]+\frac{\lambda}{2}\Vert\beta\Vert_2^2\right\\}.

The right-hand side is the best approximation of r\sqrt r achievable within the kernel class, plus the regularization penalty. When gλ⋆g_\lambda^\star and rr are bounded, this also bounds ∥wλ⋆−r∥L2(PSX)2\Vert w_\lambda^\star-r\Vert_{L_2(P_S^X)}^2 up to the constant sup⁡gλ⋆,sup⁡(gλ⋆+r)2\sup g_\lambda^\star\\,\sup(g_\lambda^\star+\sqrt r)^2. Appendix C.4 further extends the bound from a fixed predictor to one learned by weighted empirical risk minimization. Explicit rates would need approximation and complexity assumptions, which we leave to future work. Empirically, in the misspecified regression experiment of Appendix F.3, HellingerIEP recovers approximately 9393\\%, 9191\\%, and 8585\\% of the RMSE improvement of oracle weighting over source-only at 5050\\% contamination in d=1,5,10d=1,5,10.

Q5 / W3. Genuine high-ratio target structure.

We added Appendix L, where we consider a genuine two-component shift scenario. There, we increase the mass of a N(3,1)\mathcal N(3,1) component from 1010\\% to 3030\\%, or from 55\\% to 3535\\%, with no contamination. The density ratio therefore approaches 33 or 77, respectively, in the region dominated by this component.

The empirical results indicate that HellingerIEP adapts to the genuine high-ratio structure. It improves on source-only estimation in every configuration, reduces subpopulation RMSE by 0.040.04–0.170.17 (5959\\%–9696\\% of the oracle reduction), and is comparable to oracle weighting at d=1d=1. In d=5d=5 and d=10d=10, KLIEP adapts more fully, but overall target RMSE differs by at most 0.0060.006 (0.50.5\\%). These results align with the stability–adaptation trade-off: the local score concerns one added point, while a persistent subpopulation changes the criterion over a whole region, so attenuating a point's score contribution is not clipping its weight.

W1. Novelty and contribution.

The methodological contribution of this paper lies in developing and directly analyzing a restricted dual Hellinger estimator for contaminated covariate-shift adaptation, rather than in the Hellinger divergence or the general variational representation considered in isolation. Restriction to the kernel class changes the primal–dual projection geometry, so the classical robustness properties of minimum Hellinger estimation do not automatically transfer to HellingerIEP. We therefore analyze the implemented estimator itself, deriving its ratio-dependent target-contamination sensitivity and the resulting fitted-ratio perturbation, including centering, curvature, and constraints. The paper then examines the implications of this mechanism for downstream prediction under central contamination, low-ratio and heavy-tailed contamination, and a genuine high-ratio target subpopulation.

The paper’s contribution is thus the combination of the convex nonnegative square-root formulation, estimator-specific contamination analysis, and downstream stability–adaptation study. We have sharpened the introduction and discussion to distinguish these contributions from the previously known Hellinger divergence and variational density-ratio framework.

We hope the new evidence supports reconsideration of the rating. Thank you for your consideration.

Thanks for the response. I maintain my original rating.

Reviewer_JweVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper proposes HellingerIEP, a kernel-based density-ratio estimation method designed to improve the stability of covariate-shift adaptation when unlabelled target-domain samples are contaminated.

  2. The method introduces a variational objective derived from the Hellinger divergence and employs a square-root parameterisation of the density ratio, resulting in a convex optimisation problem over nonnegative kernel coefficients.

  3. The authors establish consistency and asymptotic normality for a fixed finite-dimensional kernel dictionary and regularisation parameter. They further analyse the local sensitivity of different density-ratio estimation objectives to target contamination, showing that the Hellinger objective exhibits stronger target-score attenuation in high-ratio regions than KL- and Pearson-based objectives.

  4. Experiments on synthetic Gaussian distributions and four tabular benchmark datasets demonstrate competitive performance under clean covariate shift and improved stability under the investigated contamination settings.

While the proposed formulation is mathematically appealing and the experimental results are encouraging, I have several concerns regarding the generalisability of the empirical findings, the limited scope of the theoretical guarantees, and the attribution of the observed improvements to the proposed divergence objective.

Critical strengths

The proposed square-root parameterisation leads to a convex optimisation problem while ensuring nonnegative density-ratio estimates. This formulation is relatively straightforward and avoids introducing an explicit contamination-specific trimming or reweighing parameter.

The comparison of target-score contributions under a common parameterisation provides useful insights into how divergence geometry affects sensitivity to contamination.

In particular, the observation that Hellinger-based estimation attenuates high-ratio target-score contributions more strongly than KL- and Pearson-based alternatives is interesting.

I also appreciate that the authors explicitly acknowledge the nonuniformity of this sensitivity comparison rather than claiming universal robustness.

HellingerIEP demonstrates consistently competitive performance in clean settings and substantial improvements under the considered contamination mechanisms.

The downstream prediction results are particularly encouraging, as the benefits of stable density-ratio estimation translate into improved classification accuracy and regression performance.

The supplementary experiments include comparisons with robust alternatives such as Trimmed-KLIEP, WeightedDRE, and γ\gamma-DRE, as well as experiments involving model misspecification and varying contamination locations.

These additional evaluations are useful and strengthen the empirical investigation.

Critical weaknesses

W1. Limited empirical generalizability due to small datasets and artificially constructed domain shifts

My primary concern is that the experimental evaluation is insufficient to establish the general applicability of the proposed method.

Although the paper evaluates HellingerIEP on four real-world tabular datasets (Diabetes, Abalone, Breast Cancer, and Wine), these datasets are relatively small, with the largest containing only 4,177 observations.

More importantly, the domain shifts and contamination mechanisms are artificially constructed rather than naturally occurring.

Specifically, in the benchmark experiments, covariate shift is generated through feature-dependent sample selection, while target contamination is introduced by replacing a fraction of target observations with the feature-wise mean of the clean target sample.

Consequently, these experiments primarily evaluate the proposed estimator under controlled synthetic distribution shifts, even when the underlying observations originate from real datasets.

While such experiments are useful for isolating specific statistical properties, they provide limited evidence regarding the method's effectiveness in realistic domain adaptation scenarios.

This limitation is particularly important because robustness under contaminated covariate shift is the central motivation of the paper.

Additionally, the synthetic experiments mainly consider low-dimensional settings, with downstream prediction experiments restricted to dimensions d∈{1,5,10}d\in\{1,5,10\} and density-ratio estimation experiments extending only to d=19d=19.

The behavior of the proposed estimator in substantially higher-dimensional settings remains unclear.

I believe the experimental evaluation would benefit significantly from:

  • Larger and more diverse datasets with naturally occurring domain shifts.
  • Realistic cross-domain adaptation benchmarks involving different acquisition environments, devices, or temporal distributions.
  • Experiments with substantially larger sample sizes and higher-dimensional feature representations.
  • Comparisons under multiple contamination mechanisms beyond concentrated central contamination.

For datasets where the covariate-shift assumption cannot be guaranteed, the authors should explicitly discuss this limitation and distinguish violations of covariate shift from contamination effects.

Without stronger empirical evidence, I find the broader applicability of the proposed method insufficiently supported.

Questions for authors
  1. Can the authors add a source-only (unweighted) baseline to all tabular benchmark results and report its gap to HellingerIEP? This would directly establish whether the adaptation benefit is real.

  2. Can the authors provide an ablation with a fixed or source-derived kernel dictionary, compared with the current target-derived centers? How strongly does contamination of the target sample affect the kernel dictionary itself?

  3. Can the authors compare Hellinger, KL, and Pearson objectives under matched direct, square-root, and log-linear parameterizations? How much of the improvement is attributable specifically to the Hellinger divergence rather than to parameterization or optimization differences?

  4. How does HellingerIEP perform under diffuse, heavy-tailed, multimodal, and out-of-support contamination, as well as contamination in low-ratio regions where the analysis predicts greater sensitivity? Is the robustness-adaptation trade-off observed empirically?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JweV (1/2)

We would like to thank the reviewer for their detailed feedback and constructive suggestions. The revised manuscript addresses each of the four questions and we added a real dataset with natural drift.

W1. Natural shifts, scale, and empirical generalizability.

In this revision we add an application to the Gas Sensor Array Drift dataset in Appendix M. The dataset contains 13,910 measurements with 128 original sensor features, recorded in 10 chronological batches over 36 months, with naturally occurring sensor drift and no injected contamination. We evaluate all nine adjacent-batch transfers.

The results indicate that HellingerIEP's worst loss relative to source-only on any transfer is below 22 percentage points of balanced accuracy. On the other hand, KLIEP, uLSIF, NEDIEP, and RuLSIF each lose more than 1010 percentage points on at least one transfer, while Direct-HellingerIEP is also comparatively stable. HellingerIEP also has the largest average gain, at 3.63.6 percentage points of balanced accuracy across all nine transfers, although this average is driven mainly by the transfer from batches 8 to 9. We therefore read the experiment as evidence of stability on a substantial natural-shift benchmark.

Because the original observations have 128 features, we apply label-free preprocessing identically for all methods before estimating the density ratio: an outlier screen that removes fewer than 0.30.3\\% of training observations (never applied to the evaluation sample), followed by standardization and PCA fitted on the source batch and applied to the target batch, retaining 9999\\% of the source variance (between 8 and 19 components across the nine transfers and 10 seeds). The new Remark M.1 records what HellingerIEP targets in this case. For a measurable representation Z=Π(X)Z=\Pi(X), the reduced-space ratio rΠ=dPTΠ/dPSΠr_\Pi=dP_T^\Pi/dP_S^\Pi satisfies rΠ(Π(X))=EPS[r(X)∣Π(X)]r_\Pi(\Pi(X))=\mathbb E_{P_S}[r(X)\mid\Pi(X)] almost surely, so HellingerIEP can be applied directly on the reduced space with population target rΠr_\Pi. If r(X)r(X) is measurable with respect to Π(X)\Pi(X), no density-ratio information is lost; if P(Y∣X)=P(Y∣Π(X))P(Y\mid X)=P(Y\mid\Pi(X)) and prediction is based on Π(X)\Pi(X), the importance-weighted target-risk identity remains exact on the reduced space; otherwise, dimension reduction introduces an additional representation error. For a fixed or independently constructed representation, our guarantees apply on the reduced space.

As the reviewer points out, exact covariate shift need not hold here: sensor drift can alter the sensor response to a given gas, potentially changing P(Y∣X)P(Y\mid X). We therefore present this as a natural-shift stress test, and Appendix M discusses this limitation explicitly.

Q1. Source-only baseline on the tabular benchmarks.

Table 1 now reports source-only for all four datasets. The paired comparison with HellingerIEP on clean data, over 100 replicates, is:

DatasetSource-onlyHellingerIEPΔ\Delta (SE)95% CI
Breast Cancer (acc., %)94.93 ± 2.4295.32 ± 2.18+0.39 (0.16)[+0.09, +0.70]
Wine (acc., %)95.67 ± 3.4793.81 ± 5.07−1.86 (0.47)[−2.78, −0.94]
Abalone (RMSE)2.433 ± 0.1862.405 ± 0.171−0.028 (0.005)[−0.038, −0.017]
Diabetes (RMSE)58.06 ± 4.3058.24 ± 4.16+0.19 (0.11)[−0.04, +0.41]

Performance is reported as mean ± standard deviation. Δ\Delta is the mean paired difference, HellingerIEP minus source-only; SE and 95% CI refer to this paired difference. Accuracy differences are in percentage points. Higher accuracy and lower RMSE are better.

HellingerIEP improves on source-only on the Breast Cancer and Abalone datasets, is statistically indistinguishable on the Diabetes dataset under the paired comparison, and performs significantly worse on the Wine dataset. Nevertheless, among the four importance-weighting methods in Table 1, HellingerIEP has the best mean performance for every dataset and contamination level.

This result shows that importance weighting does not always improve downstream prediction. When a predictor already approximates the target-optimal predictor well, importance weighting may offer little reduction in bias while increasing variance because of the lower effective sample size and additional density-ratio estimation step. This is likely why the adaptation methods do not work well on the Wine dataset. In the misspecified-regression experiment in Appendix F.3, however, the adaptation benefit is substantial: with half of the target sample contaminated, HellingerIEP recovers 9393\\%, 9191\\%, and 8585\\% of the improvement of oracle weighting over source-only in d=1,5,10d=1,5,10.

AuthorsOpenReview ↗

Response to Reviewer JweV (2/2)

Q2. Kernel dictionary.

In the new Appendix I we replicate the downstream experiment of Appendix F.3 with target-selected, source-selected, and pooled dictionaries, keeping the number of candidate centers and the tuning grid the same for all methods.

The results indicate that the advantage under central contamination is not created by selecting kernel centers from contaminated target observations. Under 3030\\% contamination at the target center, HellingerIEP remains close to oracle weighting in d=1,5,10d=1,5,10, and its RMSE changes very little across target-selected, source-selected, and pooled dictionaries. In particular, its performance persists with source-selected centers, through which target contamination cannot enter the dictionary. Under central contamination, its average coefficient of variation across dictionary constructions is CV‾HellingerIEP=0.18\overline{\text{CV}}_\text{HellingerIEP}=0.18\\% (Table 2). The competing methods are more sensitive to the choice of dictionary, with CV‾KLIEP=1.77\overline{\text{CV}}_\text{KLIEP}=1.77\\%, CV‾RuLSIF=3.98\overline{\text{CV}}_\text{RuLSIF}=3.98\\%, and CV‾uLSIF=26.47\overline{\text{CV}}_\text{uLSIF}=26.47\\%.

Q3. Objective versus parameterization.

The divergence objective and the ratio representation are intrinsically coupled in these estimators. The direct, square-root, and log-linear formulations induce different function classes, positivity and normalization constraints, regularization geometries, and, in some cases, different optimization problems. Consequently, comparing divergence objectives across ratio representations would not by itself isolate the effect of the objective from that of the representation.

We therefore provide complementary controlled comparisons. Proposition 3.1 compares the Hellinger, KL, and Pearson target-score contributions under a common square-root ratio class. Direct-HellingerIEP and LL-HellingerIEP retain the Hellinger criterion while changing the ratio representation. Appendix H varies the power-divergence objective under a common direct kernel class. Taken together, these comparisons support the paper’s central claim that divergence geometry is a substantive determinant of contamination sensitivity. The ratio representation, tuning, and optimization shape how this geometric effect is expressed in the finite-sample estimator.

Q4. Other contamination mechanisms and the trade-off.

This relates to questions Q3 and Q5/W3 of Reviewer 3HfT, and we added Appendix L to test a genuine high-ratio subpopulation with no contamination, Appendix K.2 where we move a 3030\\% spike from the target center into low-ratio regions, and Appendix K.3 to check the effects of heavy-tailed Cauchy contamination.

These new results suggest that the adaptation cost is small in the genuine-subpopulation experiment, and that HellingerIEP's benefits are greatest when contamination is within the effective range of the source distribution. They also demonstrate that as contamination affects low or negligible density regions, HellingerIEP is more affected than some other methods, as predicted by our theory in Proposition 3.1 and the newly added local sensitivity analysis in Appendix J.

When adding a genuine subpopulation of size growing from 1010\\% to 3030\\% of the data, or from 55\\% to 3535\\%, HellingerIEP improves on source-only in every configuration and lowers subpopulation RMSE by 0.040.04–0.170.17 (5959\\%–9696\\% of the oracle reduction). KLIEP adapts somewhat more fully in d=5d=5 and d=10d=10, but overall target RMSE differs by at most 0.0060.006 (0.50.5\\%).

The additional contamination experiments demonstrate that for a spike at z≤2z\le2 (i.e., close to the target covariate distribution), HellingerIEP has the lowest ratio NMSE and the lowest or near-lowest prediction RMSE. For z≳4z\gtrsim4 (i.e., in low or negligible density regions), HellingerIEP still outperforms KLIEP and uLSIF in prediction RMSE, but NEDIEP and RuLSIF achieve lower prediction RMSE. Under Cauchy contamination, where diffuse contamination is likely placed far away from the true target population, uLSIF, RuLSIF, and NEDIEP retain their adaptation benefit better, but no method clearly improves on source-only beyond 2020–3030\\% contamination in d=5d=5 and d=10d=10.

Taken together, the additional experiments show both sides of the predicted trade-off. HellingerIEP is most stable under central, high-ratio contamination and retains substantial adaptation benefit for a genuine high-ratio target subpopulation. Its advantage diminishes or reverses in low-ratio and heavy-tailed regimes, where NEDIEP and RuLSIF may adapt more fully.

We hope these results and the revisions according to the reviewer's suggestions support reconsideration of the rating. We thank the reviewer for their consideration.

Reviewer_QUECOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses unsupervised domain adaptation under covariate shift, focusing on the instability of density-ratio estimation when the unlabeled target data is corrupted. The authors propose Hellinger Importance Estimation Procedure, a kernel estimator derived from a variational formulation of the Hellinger divergence.

Critical strengths
  • Hellinger based parameter-free alternative to sample trimming or heuristic sample-weighting schemes.
  • thorough theoretical analysis regarding target score factors, explaining why HellingerIEP handles high-ratio target spikes/outliers better.
Critical weaknesses
  • under heavy target contamination, I doubt whether an unsupervised CV objective can be trusted.
  • if target contamination adds noise in regions with zero source support, the w−1w^{-1} term can inflate.
  • theorem 3.1 is derived based on assumption that the kernel center dictionary and λ\lambda are fixed and independent of nn; however, in practice, cm,λc_m, \lambda are chosen from target data, such that the theoretical guarantees do not account for data-dependent dictionary selection or hyperparameter tuning.
  • the experiments are mainly based on low-dimensional toy data; in high-dimensional spaces, most of the volume in space lies far away from any target sample since RBF kernels decay exponentially as distance from the centers grows.
  • would be better to provide the final target risk bound based on the proposed theoretical framework.
Questions for authors

see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QUEC (1/3)

We thank the reviewer for sharing their concerns and feedback. The revised manuscript addresses each of the five points through a new analysis or result.

W1. Cross-validation under heavy contamination.

Contamination can affect the cross-validation (CV) procedure, but our empirical results suggest that HellingerIEP's stability also extends to CV selection. For example, in Appendix F.3 we tune all methods by CV on the contaminated sample (there is no oracle selection). The results indicate that CV is stable for HellingerIEP with centrally located contamination. In d=5d=5, the same bandwidth configuration is chosen in at least 94 of 100 replicates at each contamination level from 00 to 5050\\%. In the other dimensions, the most frequent choice stays within one grid step as contamination increases. For KLIEP and uLSIF, the selected value changes much more, by up to a factor of 100100, as contamination increases.

Although stable selection does not necessarily mean that the selected configuration is good, the downstream performance of the HellingerIEP-tuned estimator also remains stable: in the misspecified regression experiment at 5050\\% contamination, HellingerIEP achieves 9393\\%, 9191\\%, and 8585\\% of the oracle RMSE improvement in d=1,5,10d=1,5,10.

Our explanation of this behavior is the following. Under central contamination, the contaminated observations lie in a region of substantial source overlap. Increasing the fitted ratio there reduces the reciprocal target term of the held-out Hellinger criterion but also increases its source term, which provides a counterbalance, so the selected configuration is stable in this regime. Under heavy-tailed contamination in regions of negligible source density, this counterbalance is much weaker; cross-validation then selects overly wide kernels and the adaptation benefit disappears. This is an empirical explanation of the observed behavior, not a theoretical guarantee of tuning consistency under contamination.

The exception is heavy-tailed Cauchy contamination (shown in the new Appendix K.3), where HellingerIEP is expected to be less stable (as suggested by Proposition 3.1 and the new local sensitivity results in Appendix J). There, as described above, cross-validation selects very wide kernels and HellingerIEP collapses to source-only performance.

W2. Contamination outside the source support.

The reviewer is absolutely correct, and the revised manuscript addresses this concern in multiple locations. First, Appendix J shows the local sensitivity of the methods to contamination in different regions. As stated also in the response to Reviewer 3HfT, considering curvature and constraints, HellingerIEP has the smallest fitted-ratio influence near the target center (high ratio), but the largest influence in low-ratio regions. These empirical results at the level of the fitted ratio reinforce the ordering suggested by Proposition 3.1 from the score factors alone.

Moreover, in Appendix K.2 we show performance as a 3030\\% spike is moved from the target center (z=0z=0) out to z=10z=10, where the source density is practically zero. Appendix K.3 adds Cauchy contamination, which, especially in higher dimensions, also places a lot of mass on regions with practically zero source density. From both of these experiments, we see that HellingerIEP is most beneficial in the presence of contamination close to the target center. RuLSIF and NEDIEP, on the other hand, perform better in low-ratio regions. From Figure 14 we can see that for z≤2z\le2, HellingerIEP has the lowest ratio NMSE and the lowest or near-lowest prediction RMSE. As z≳4z\gtrsim4, RuLSIF and NEDIEP can achieve lower prediction RMSE. But even in these regions, HellingerIEP performs better than KLIEP and uLSIF in prediction RMSE, and its ratio NMSE stays below both at every location.

AuthorsOpenReview ↗

Response to Reviewer QUEC (2/3)

W3. Fixed dictionary and λ\lambda in Theorem 3.1.

We agree with the reviewer that a fixed dictionary is not practical. We therefore added a corollary for data-dependent tuning and Appendix I to check the dependence on the dictionary.

The new Corollary C.1 shows that, when using sample splitting, the estimation guarantee extends to cross-validated tuning. Let the centers be chosen from an auxiliary sample that is independent of the fitting sample, and let the bandwidth and λ\lambda be selected from a fixed finite grid G\mathcal G of configurations satisfying the assumptions of Theorem 3.1 with λk>0\lambda_k>0 for every k∈Gk\in\mathcal G. Conditionally on the independent sample used to construct the dictionary, for any data-dependent choice k^∈G\hat k\in\mathcal G, including the one made by cross-validation,

∥w^n,k^−wk^⋆∥∞≤max⁡k∈G∥w^n,k−wk⋆∥∞=Op(n−1/2).\Vert\widehat w_{n,\hat k}-w^\star_{\hat k}\Vert_\infty\le\max_{k\in\mathcal G}\Vert\widehat w_{n,k}-w^\star_k\Vert_\infty=O_p(n^{-1/2}).

This corollary covers estimation error for the selected configuration, but optimal selection and post-selection normality are not covered, and it applies only to the sample-split variant. In contrast, our implementation, like KLIEP and uLSIF, draws the centers from the same observations used for fitting, and that reuse is not covered by the theory. We therefore test the dependence on the dictionary empirically in Appendix I. We compare target-selected, source-selected, and pooled centers. Under 3030\\% contamination at the target center, HellingerIEP's downstream performance changes very little across the three dictionaries, and its advantage persists with source-selected centers, which contain no contaminated observations. Across all considered settings, HellingerIEP has the most stable downstream results (Table 2). This empirically supports the stability of the implemented same-sample procedure.

W4. Low-dimensional data and RBF kernels in high dimensions.

In this revision we add an application to the Gas Sensor Array Drift dataset in Appendix M. The dataset contains 13,910 measurements with 128 original sensor features, recorded in 10 chronological batches over 36 months, with natural drift and no injected contamination. We evaluate all nine adjacent-batch transfers. All methods share the same label-free preprocessing: a screen for extreme sensor readings that removes fewer than 0.30.3\\% of training observations and is never applied to the evaluation sample, followed by standardization and PCA (details below and in Appendix M).

The results indicate that HellingerIEP’s worst loss relative to source-only on any transfer is below 22 percentage points of balanced accuracy. On the other hand, KLIEP, uLSIF, NEDIEP, and RuLSIF each lose more than 1010 percentage points on at least one transfer, while Direct-HellingerIEP is also comparatively stable. HellingerIEP also has the largest average gain, at 3.63.6 percentage points of balanced accuracy across all nine transfers, although this average is driven mainly by one transfer.

We agree with the reviewer that RBF-based density-ratio estimation faces difficulties as dimension grows since a finite set of centers may cover the relevant space poorly, and this limitation similarly affects KLIEP, uLSIF, and RuLSIF. For the gas-sensor application we therefore apply dimension reduction before estimating the ratio: standardization and PCA are fitted on the source batch and applied to the target batch, retaining 9999\\% of the source variance (between 8 and 19 components across the nine transfers and 10 seeds), and target labels are used only for evaluation. The new Remark M.1 identifies what HellingerIEP targets in this case. For Z=Π(X)Z=\Pi(X), HellingerIEP targets the pushforward density ratio rΠ=dPTΠ/dPSΠr_\Pi=dP_T^\Pi/dP_S^\Pi, where rΠ(Π(X))=EPS[r(X)∣Π(X)]r_\Pi(\Pi(X))=\mathbb E_{P_S}[r(X)\mid\Pi(X)]. If the representation is ratio-sufficient, the full density ratio is preserved exactly; if it is outcome-sufficient, the importance-weighted target-risk identity for predictors based on Π(X)\Pi(X) is preserved exactly; otherwise, dimension reduction introduces a representation-approximation term. For a fixed or independently constructed representation, our guarantees apply on the reduced space. The gas-sensor experiment thus evaluates a practically relevant high-dimensional pipeline, while the theory identifies the corresponding population target and the additional approximation term. Moreover, the Hellinger criterion itself is not tied to the RBF expansion: the kernel features could be replaced by a representation suited to high-dimensional inputs, which we leave for future work.

AuthorsOpenReview ↗

Response to Reviewer QUEC (3/3)

W5. Target risk bound.

Reviewer 3HfT raises a similar issue (Q4) and we added the following target-risk decomposition, which is summarized at the end of Section 3 and stated in full in the new Appendix C.4. Under the covariate-shift assumption for the clean source and target distributions, for a fixed predictor hh and a target fitting sample contaminated at level ε∈[0,1)\varepsilon \in [0, 1), we have ∣EPS[w^nℓh]−RT(h)∣≤∥ℓh∥L2(PS)(∥w^n−wλ,ε⋆∥+∥wλ,ε⋆−wλ⋆∥+∥wλ⋆−r∥),\left|\mathbb E_{P_S}[\widehat w_n\ell_h]-R_T(h)\right|\le\Vert\ell_h\Vert_{L_2(P_S)}\Big(\Vert\widehat w_n-w_{\lambda,\varepsilon}^\star\Vert+\Vert w_{\lambda,\varepsilon}^\star-w_\lambda^\star\Vert+\Vert w_\lambda^\star-r\Vert\Big), where the three norms inside the parentheses are in L2(PSX)L_2(P_S^X), wλ,ε⋆w_{\lambda,\varepsilon}^\star is the population solution under contamination, and rr is the clean target-to-source density ratio. In the clean case, the middle term vanishes.

We can see that our theory provides an estimation rate and identifies a local contamination-sensitivity mechanism. Theorem 3.1 gives an Op(n−1/2)O_p(n^{-1/2}) estimation rate under its assumptions and Proposition 3.1 identifies the ratio-dependent target-score factors governing local contamination sensitivity. As detailed above, the new Appendix J evaluates the resulting normalized fitted-ratio influence after accounting for curvature and constraints. The reviewer is correct that our theory does not give a rate for the combined kernel-approximation and regularization bias. Appendix C.4 also shows how this term enters. Since wλ⋆=wβλ⋆w_\lambda^\star=w_{\beta_\lambda^\star} minimizes the penalized population criterion, the exact excess-criterion identity J(wβ)−J(r)=EPS[gβ(X)−r(X)2/gβ(X)]J(w_\beta)-J(r)=\mathbb E_{P_S}\big[\\{g_\beta(X)-\sqrt{r(X)}\\}^2/g_\beta(X)\big] gives J(w_\lambda^\star)-J(r)\le\inf_{\beta\in\mathcal C}\left\\{\mathbb E_{P_S}\left[\frac{\\{g_\beta(X)-\sqrt{r(X)}\\}^{2}}{g_\beta(X)}\right]+\frac{\lambda}{2}\Vert\beta\Vert_2^2\right\\}, so the excess criterion of the regularized solution is bounded by a best-in-class approximation term plus a regularization contribution. This transfers to the L2L_2 term in the decomposition because (w−r)2=(g−r)2/g,g,(g+r)2(w-r)^2=\\{(g-\sqrt r)^2/g\\}\\,g\\,(g+\sqrt r)^2: when gλ⋆g_\lambda^\star and rr are bounded (the former holds under Assumption 3.1; the latter is an additional condition on the clean ratio), ∥wλ⋆−r∥L2(PSX)2≤sup⁡gλ⋆,sup⁡(gλ⋆+r)2,J(wλ⋆)−J(r)\Vert w_\lambda^\star-r\Vert_{L_2(P_S^X)}^2\le\sup g_\lambda^\star\\,\sup(g_\lambda^\star+\sqrt r)^2\\,\\{J(w_\lambda^\star)-J(r)\\}.

Appendix C.4 also extends the fixed-predictor bound to a predictor h^\widehat h fitted by weighted empirical risk minimization over a class H\mathcal H, with hH⋆h_{\mathcal H}^\star the target-risk minimizer in H\mathcal H: RT(h^)−RT(hH⋆)≤2sup⁡h∈H∣Pn,Sw^nℓh−RT(h)∣≤2sup⁡h∈H∣(Pn,S−PS)w^nℓh∣+2sup⁡h∈H∣PS(w^n−r)ℓh∣.R_T(\widehat h)-R_T(h_{\mathcal H}^\star)\le 2\sup_{h\in\mathcal H}\left|P_{n,S}\\{\widehat w_n\ell_h\\}-R_T(h)\right|\le 2\sup_{h\in\mathcal H}\left|(P_{n,S}-P_S)\\{\widehat w_n\ell_h\\}\right|+2\sup_{h\in\mathcal H}\left|P_S\\{(\widehat w_n-r)\ell_h\\}\right|. If sup⁡h∈H∥ℓh∥L2(PS)≤LH\sup_{h\in\mathcal H}\Vert\ell_h\Vert_{L_2(P_S)}\le L_{\mathcal H}, the second term is bounded by 2LH∥w^n−r∥L2(PSX)2L_{\mathcal H}\Vert\widehat w_n-r\Vert_{L_2(P_S^X)}, which decomposes into the estimation, contamination, and approximation–regularization components above. Explicit rates in MM and λ\lambda, and for the empirical-process term, would need approximation and complexity assumptions together with sample splitting or cross-fitting to handle the dependence between the fitted weights and the source empirical process, which we leave to future work. Empirically, in the misspecified regression setting of Appendix F.3, HellingerIEP recovers approximately 9393\\%, 9191\\%, and 8585\\% of the source-to-oracle RMSE improvement at 5050\\% contamination in d=1,5,10d=1,5,10.

We hope the new evidence supports reconsideration of the rating and we thank the reviewer for their consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions for the Area Chair

We would like to thank the area chair for handling our submission and provide an overview of our changes in response to the reviewers' suggestions. Most reviewers' concerns overlap, and we summarize below how we addressed them in the revision. Additional details can be found in the individual responses.

  1. Kernel centers taken from a contaminated target sample (3HfT Q2, JweV Q2, QUEC W3). The revised manuscript adds the downstream experiment with source-selected and pooled centers alongside target-selected ones (Appendix I). Under 3030\\% central contamination, HellingerIEP remains close to oracle weighting and changes very little across target-selected, source-selected, and pooled dictionaries, with the lowest variation across the three (Table 2). Its performance therefore persists with source-selected centers, through which target contamination cannot enter the dictionary.

  2. Other types of contamination (3HfT Q3, JweV Q4, QUEC W2). We added an experiment that moves the 30% spike away from the target center into low-ratio regions with negligible source density (Appendix K.2). We also added a simulation with heavy-tailed Cauchy contamination (Appendix K.3).

  3. Contamination vs. genuine target shift (3HfT W3/Q5, JweV Q4). We added a setting with a genuine high-ratio subpopulation and no contamination (Appendix L). There, HellingerIEP improves on source-only in every configuration and recovers 59%–96% of the oracle reduction in subpopulation RMSE.

  4. Link between the score analysis and the full estimator (3HfT W2/Q1/Q4, QUEC W1, W3, W5). Appendix J follows the score analysis through to the fitted weights. It computes the local derivative of the normalized fitted ratio, with curvature and constraints included, and checks it against one-point refits. On the theory side, Section 3 now ends with a summary of a three-term target-risk decomposition, Appendix C.3 adds a finite-grid tuning corollary, and Appendix C.4 states the decomposition in full, bounds the approximation–regularization term, and extends the risk bound to a predictor learned by weighted empirical risk minimization. Appendix F.3 also shows that cross-validation on the contaminated sample selects a stable configuration for HellingerIEP.

  5. Real data with natural shift (JweV W1, QUEC W4). Appendix M adds an application to the Gas Sensor Array Drift dataset, which has 13,910 measurements, 128 features, and nine adjacent-batch transfers with natural drift, together with a remark on what HellingerIEP targets after dimension reduction. Table 1 now also includes a source-only baseline for the four tabular datasets (JweV Q1).

  6. Divergence vs. parameterization (JweV Q3). The paper addresses the attribution question through complementary controlled comparisons. Proposition 3.1 holds the square-root ratio class fixed and compares the Hellinger, KL, and Pearson target-score geometries. Appendix H holds a common direct kernel class fixed and varies the power-divergence objective. Appendices F, G, and K.3 retain the Hellinger criterion while examining square-root, direct, and log-linear ratio representations. Together, these comparisons support the paper’s central claim that divergence geometry is a substantive determinant of contamination sensitivity. The ratio representation, tuning, and optimization shape how this geometric mechanism is expressed in finite-sample performance.

Overall, the additional results support the main claims of the paper. The fitted-ratio experiments highlight the nonuniform sensitivity pattern suggested by the score analysis. The downstream experiments underscore these results. With 50% central contamination, HellingerIEP recovers 85%–93% of the oracle improvement over source-only. On the gas-sensor data, its worst loss relative to source-only is below 2 percentage points of balanced accuracy, while KLIEP, uLSIF, RuLSIF, and NEDIEP each lose more than 10 percentage points on at least one transfer; HellingerIEP also has the largest average gain and Direct-HellingerIEP is also comparatively stable.

The new local analysis identifies greater Hellinger score sensitivity in low-ratio regions as suggested by the reviewers, and Appendix J makes the corresponding fitted-ratio sensitivity pattern explicit. Consistent with this limitation, NEDIEP and RuLSIF achieve lower prediction RMSE for spikes far from the target center in Appendix K.2. The revision discusses these limitations in more depth alongside the central-contamination results.

In the manuscript, we added Appendices I–M, extended Appendix F.3, added a summary of the target-risk decomposition to Section 3 with the full statement in Appendix C.4, added the tuning corollary to Appendix C.3, added source-only to Table 1, and clarified the contribution and its scope in the introduction and discussion.

We are grateful to the reviewers for their comments, which helped us improve the paper considerably. We thank them and the AC for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.