acceptodds
Under review as a conference paper at ICLR 2027

Robust -Aggregation under Covariate Shift with Pairwise Overlap

Abstract

We study squared-loss aggregation of a fixed dictionary of predictors under covariate shift: a labeled sample of size comes from a source population with input law , while the target risk is under another input law . With known density ratios, the difficulty of a comparison depends on the weights where its two predictors disagree. We capture this dependence through an original-pair overlap coefficient and construct a robust -aggregate from median-of-means (MOM) estimates of risk differences and squared distances. Coordinatewise robust estimates can produce comparison cycles and non-Euclidean distance matrices. Our method repairs the distances by a root-distance projection onto the Euclidean distance matrices, then combines the predictors by solving a monotone variational inequality on the simplex. With exact population quantities, this rule is the classical uniform-prior -aggregate with mixing parameter . With noisy pairwise quantities, the variational form also accepts nonzero comparison cycles. The inputs are the density-ratio weights, a failure probability , and a bound on predictions such that, for a source input-response pair , the source conditional second moment satisfies wherever candidate predictors disagree. Without a supplied overlap bound, the resulting convex mixture achieves a leading-one oracle inequality of order , including settings in which the global second moment of the density ratio is infinite. Matching lower bounds show the worst-case dependence on overlap, dictionary size, and confidence.

Primary area:
learning theory
Keywords:
statistical learning theory, covariate shift, Q-aggregation, robust estimation, importance weighting, median-of-means

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6b8XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies robust aggregation of a fixed dictionary of regression predictors under covariate shift with squared loss. The authors introduce a dictionary-dependent pairwise overlap coefficient and build a robust Q-aggregation procedure using median-of-means estimates of pairwise risk differences and distances, Euclidean distance-matrix repair, and a variational-inequality aggregation step.

Critical strengths

S1. The use of pairwise loss differences avoids unnecessarily strong moment requirements.

S2. The paper addresses two genuine consistency problems created by coordinatewise robust estimation.

S3. The connection to classical Q-aggregation is conceptually useful.

S4. The theory includes matching lower bounds for the main rate dependence.

Critical weaknesses

W1. The method assumes the density ratio is known exactly, which removes one of the main practical difficulties of covariate shift.

W2. Some assumptions (e.g. uniformly boundedness) are strong.

W3. The finite-sample oracle bound has extremely large constants and is often practically vacuous.

W4. The empirical results do not show a compelling predictive advantage for the proposed method.

W5. The algorithm is quite complicated and its individual ingredients are largely established tools.

W6. It seems that the real-data experiments do not involve naturally occurring covariate shift.

W7. The paper does not study robustness to violations of the exact covariate-shift assumption.

Questions for authors

See weaknesses.

Reviewer_gbhKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper takes Equation (10) as its objective function and derives several theoretical results, but provides no clear motivation or justification for this choice.

Critical strengths

None

Critical weaknesses

Is Equation (10) the objective function? Its motivation is unclear. The authors should explain why this objective is appropriate and how it relates to the problem under study.

The introduction contains several technical terms that are not adequately explained, such as “a sharp oracle inequality governed by this dictionary-dependent quantity.” These terms should be clarified to make the introduction more accessible.

The introduction includes excessive mathematical detail. For example, the paragraph beginning “Two features of squared-loss aggregation guide the construction. Comparing losses cancels the common term wY2w Y^2 (where YY denotes the response), which would otherwise require stronger response moments to control an individual weighted risk. The remaining pairwise difference has variance governed by (1). Robust estimation creates a separate difficulty: writing TijT_{i j} for the robust estimate of the target-risk difference between fif_i and fjf_j, medians taken pair by pair can yield Tij+Tjk+Tki≠0T_{i j}+T_{j k}+T_{k i} \neq 0 (i.e., the three estimated comparisons around a cycle need not sum to zero). Such a cycle prevents us from representing all comparisons as differences of one empirical-risk vector, as ordinary QQ-aggregation would require. ” The introduction should instead explain the main challenges and the proposed approach at a conceptual level.

A central claim is that “a large weight is harmless for comparing two models at an input where they agree.” However, the authors do not provide a clear explanation or a concrete example to support this claim. They should explain the underlying reasoning and illustrate its significance.

The definition of TijT_{ij} is unclear. The expression to which the median-of-means (MOM) estimator is applied appears to be a scalar. The authors should specify the sample-level quantities, how the observations are partitioned into blocks, and how the block means and their median are computed.

Questions for authors

The authors should explain why this objective Equation (10) is appropriate and how it relates to the problem under study.

Reviewer_rhurOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies a setting similar to that of Q-aggregation, regression with a finite family of predictors, with the wrinkle of also being interested in covariate shift. The main observation is that we don't need uniformly bounded importance weights, rather we only need these where predictors disagree. This leads to a more general transfer learning theorem (which requires a more complex aggregation procedure) with a bound in terms of a new "transfer coefficient" that captures the ratio between "variance" of the weighted source disagreement and the target disagreement, between pairs.

I think there is a decent amount of potential in the result of this paper. The topic is interesting and relevant, the proposed method and result seem kind of cool, etc. However, the presentation is quite poor; I am pretty sure the entire manuscript (text of the paper) is AI generated. This comes with the now-well-known presentation issues for theoretical works, in particular, (a) there is almost no motivation for this particular problem and (b) there is almost no intuition for the new definitions, the algorithm design, or anything else. Put another way, I do not think the presentation makes the paper accessible to even near-experts in the field, and this, to me, is sufficient grounds for rejection.

Critical strengths

As noted above, the topic is interesting and relevant, the proposed method and result seem kind of cool, etc. The insight that one does not need bounded importance weights outside of the disagreement region is simple but insightful, and the fact that one can leverage this to improve guarantees is nice.

Critical weaknesses

As noted above, the presentation is quite poor; I am pretty sure the entire manuscript (text of the paper) is AI generated. This comes with the now-well-known presentation issues for theoretical works, in particular, (a) there is almost no motivation for this particular problem and (b) there is almost no intuition for the new definitions, the algorithm design, or anything else. Put another way, I do not think the presentation makes the paper accessible to even near-experts in the field, and this, to me, is sufficient grounds for rejection.

Questions for authors

I don't really have any questions, I think the paper cannot be accepted without a substantial re-write which would be too significant for the rebuttal/revision process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.