- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HellingerIEP, a kernel-based density-ratio estimator for domain adaptation under contaminated covariate shift. The method uses a variational Hellinger objective and a square-root parameterization to obtain a convex optimization problem. The authors establish consistency and asymptotic normality under fixed-dimensional assumptions and analyze local sensitivity to target contamination. Experiments show improved stability over several density-ratio estimation methods. I currently lean toward rejection, but I would be willing to reconsider my rating if the authors provide convincing clarification and evidence addressing these concerns.
Critical strengths
-
The paper addresses a relevant problem, as density-ratio estimation can be unstable when the unlabeled target sample is contaminated.
-
The square-root parameterization gives a convenient convex formulation with nonnegative density-ratio estimates.
-
The experiments cover both density-ratio estimation and downstream prediction, with comparisons to several existing methods.
Critical weaknesses
-
The methodological novelty is somewhat limited. Hellinger divergence and variational density-ratio estimation have been studied previously, so the main contribution appears to be their particular kernel implementation and contamination analysis.
-
The theoretical analysis focuses on local score sensitivity, whereas the empirical claim concerns stability of the complete estimator under substantial contamination. The connection between these two levels of analysis is not fully established.
-
The empirical improvements are encouraging, but the contamination mechanisms are highly controlled. It remains unclear how well the method handles contamination that resembles genuine target-domain shift.
Questions for authors
1. In Proposition 3.1 (Page 5), the comparison with KLIEP and uLSIF is based on target-score terms under a common square-root parameterization. Since the methods have different curvature matrices, how much of the reported stability difference is explained by the score terms alone? Could the authors provide a numerical comparison of the full contamination influence?
2. The kernel centers are selected from the target sample, which may itself be contaminated. How sensitive is HellingerIEP to this choice? For example, would using centers selected from the source sample or the clean portion of the target sample lead to substantially different results?
3. In Section 4, the Gaussian density-ratio experiment replaces 30% of target observations with the clean target mean. This creates a concentrated spike rather than a conventional outlier distribution. How would the method perform when the contamination occurs in a region with low source density, where the fitted density ratio may behave differently?
4. Theorem 3.1 establishes convergence to the regularized population minimizer , rather than the true density ratio . Could the authors quantify how the regularization and kernel approximation errors affect downstream importance-weighted prediction?
5. In the benchmark experiments, contaminated observations are replaced by the mean feature vector, while prediction is evaluated on the clean target distribution. How does HellingerIEP perform when the target distribution contains a genuine high-density-ratio subpopulation that should receive large importance weights rather than be treated as contamination?
3 comments
Response to Reviewer 3HfT (2/2)
Q4. Convergence to the regularized solution and downstream risk.
Reviewer QUEC raises a similar issue (W5) and we added the following target-risk decomposition, which is summarized at the end of Section 3 and stated in full in the new Appendix C.4. Under the covariate-shift assumption for the clean source and target distributions, for a fixed predictor and a target fitting sample contaminated at level , we have where the three norms inside the parentheses are in , is the population solution under contamination, and is the clean target-to-source density ratio. In the clean case, the middle term vanishes. Our theory provides an estimation rate and identifies a local contamination-sensitivity mechanism. Theorem 3.1 gives an estimation rate under its assumptions and Proposition 3.1 identifies the ratio-dependent target-score factors governing local contamination sensitivity. As detailed above, the new Appendix J evaluates the resulting normalized fitted-ratio influence after accounting for curvature and constraints. The reviewer is correct that our theory does not give a rate for the combined kernel-approximation and regularization bias. Appendix C.4 also shows how this term enters. Because minimizes the penalized population criterion,
J(w_\lambda^\star)-J(r)\le\inf_{\beta\in\mathcal C}\left\\{\mathbb E_{P_S}\left[\frac{\\{g_\beta(X)-\sqrt{r(X)}\\}^{2}}{g_\beta(X)}\right]+\frac{\lambda}{2}\Vert\beta\Vert_2^2\right\\}.
The right-hand side is the best approximation of achievable within the kernel class, plus the regularization penalty. When and are bounded, this also bounds up to the constant . Appendix C.4 further extends the bound from a fixed predictor to one learned by weighted empirical risk minimization. Explicit rates would need approximation and complexity assumptions, which we leave to future work. Empirically, in the misspecified regression experiment of Appendix F.3, HellingerIEP recovers approximately , , and of the RMSE improvement of oracle weighting over source-only at contamination in .
Q5 / W3. Genuine high-ratio target structure.
We added Appendix L, where we consider a genuine two-component shift scenario. There, we increase the mass of a component from to , or from to , with no contamination. The density ratio therefore approaches or , respectively, in the region dominated by this component.
The empirical results indicate that HellingerIEP adapts to the genuine high-ratio structure. It improves on source-only estimation in every configuration, reduces subpopulation RMSE by – (– of the oracle reduction), and is comparable to oracle weighting at . In and , KLIEP adapts more fully, but overall target RMSE differs by at most (). These results align with the stability–adaptation trade-off: the local score concerns one added point, while a persistent subpopulation changes the criterion over a whole region, so attenuating a point's score contribution is not clipping its weight.
W1. Novelty and contribution.
The methodological contribution of this paper lies in developing and directly analyzing a restricted dual Hellinger estimator for contaminated covariate-shift adaptation, rather than in the Hellinger divergence or the general variational representation considered in isolation. Restriction to the kernel class changes the primal–dual projection geometry, so the classical robustness properties of minimum Hellinger estimation do not automatically transfer to HellingerIEP. We therefore analyze the implemented estimator itself, deriving its ratio-dependent target-contamination sensitivity and the resulting fitted-ratio perturbation, including centering, curvature, and constraints. The paper then examines the implications of this mechanism for downstream prediction under central contamination, low-ratio and heavy-tailed contamination, and a genuine high-ratio target subpopulation.
The paper’s contribution is thus the combination of the convex nonnegative square-root formulation, estimator-specific contamination analysis, and downstream stability–adaptation study. We have sharpened the introduction and discussion to distinguish these contributions from the previously known Hellinger divergence and variational density-ratio framework.
We hope the new evidence supports reconsideration of the rating. Thank you for your consideration.
Thanks for the response. I maintain my original rating.
Response to Reviewer 3HfT (1/2)
We want to thank the reviewer for their careful comments and for expressing openness to reconsider their rating. Below we address each of the stated criticisms separately, but we want to first stress how the reviewer's comments helped us to make our arguments and presentation stronger. The nonuniform sensitivity pattern suggested at the score level persists after accounting for curvature and the constraints, and HellingerIEP adapts to a genuine high-ratio subpopulation while remaining stable under central contamination. uLSIF shows exactly the opposite behavior, with high sensitivity to central contamination and practically no sensitivity to contamination outside the effective source domain. KLIEP's sensitivity is mostly uniform in the location of the contamination, and NEDIEP interpolates between the benefits of HellingerIEP and uLSIF. We added a discussion about these limitations and differences in sensitivity to the main manuscript. Below we give detailed responses to the reviewer's questions.
Q1 / W2. Score terms versus complete fitted-ratio sensitivity.
We agree that the score factors alone do not determine the sensitivity of the fitted estimator. We therefore added Appendix J, which computes the empirical local derivative of the normalized fitted ratio, conditional on the selected dictionary and hyperparameters. The calculation includes the centered contamination score, the method-specific inverse curvature, the normalization, and the derivative of the fitted-ratio map, and we compare the derivative with an exact one-point refit. For w~=w/Pn,Sw, we report
I(z)=dεdw~ε=0L2(Pn,S) over point-mass contamination at z∈[−10,10].
As stated above, considering curvature and constraints, HellingerIEP has the smallest fitted-ratio influence near the target center, where the ratio is high, and the largest in low-ratio regions. The empirical results at the level of the fitted ratio reinforce the ordering suggested by Proposition 3.1 from the score factors alone.
Q2. Dependence on contaminated kernel centers.
In Appendix I we replicate the downstream experiment of Appendix F.3 with target-selected, source-selected, and pooled dictionaries, keeping the number of candidate centers and the tuning grid the same for all methods.
The results indicate that the central-contamination advantage is not created by contaminated target-selected centers. Under 30% contamination at the target center, HellingerIEP remains close to oracle weighting in d=1,5,10, and its RMSE changes very little across the three dictionaries. In particular, its performance persists with source-selected centers. In the new Table 2 we report the average coefficient of variation (CV) across the different settings, and it shows that HellingerIEP is most stable across the different dictionaries. In the clean case, for example, HellingerIEP has an average CV of 0.07% whereas KLIEP is a distant second with an average CV of 0.23%. In the contaminated case, under both z=0 and z=6 contamination, HellingerIEP has the lowest CV.
Q3. Contamination in low-source-density regions.
We added an experiment in Appendix K.2 where we move a 30% spike over z∈{0,…,10}, from the central high-ratio region into regions where both the clean density ratio and the source density are small; the clean ratio decreases from r(0)≈1.82 to r(4)≈0.25 and r(6)≈0.02.
Similarly, in Appendix K.3 we consider a heavy-tailed Cauchy contamination. As suggested by the local sensitivity analysis above, the results are nonuniform and HellingerIEP is most advantageous with contamination in the high-ratio regions. For z≤2, HellingerIEP has the lowest ratio NMSE and the best or close-to-best prediction RMSE, and its ratio NMSE stays below KLIEP's and uLSIF's at every location. For z≳4, HellingerIEP still outperforms KLIEP and uLSIF in prediction RMSE, but NEDIEP and RuLSIF achieve even lower prediction RMSE. Under Cauchy contamination the behavior is dimension-dependent: uLSIF, RuLSIF, and NEDIEP retain their adaptation benefit in univariate problems, but as the dimension and contamination proportion increase no method clearly improves on source-only fitting. This is in line with the local sensitivity results, because the contamination then places more and more points in regions of negligible source density.