- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
The paper provide a general decomposition of the excess risk into a learning term and an error term associated with distribution drift, and prove oracle inequalities for the learning error under mixing conditions.
-
They analyze the drift error in the square loss setting and provide explicit bounds for structural-break and smoothly drifting processes.
Critical strengths
S1. Clear and meaningful learning–drift decomposition.
The paper decomposes the out-of-sample excess risk of nonstationary weighted ERM into a learning error and a drift error, making the key trade-off in weight selection particularly clear. Spreading the weights increases the effective sample size and reduces estimation error, but may increase the drift error due to the mismatch between historical and current distributions. This decomposition also provides a natural structure for the subsequent theoretical analysis.
S2. Strong uniformity of Theorem 3.2.
The main result is not restricted to a prespecified fixed weight vector, but holds simultaneously over , thereby accommodating data-dependent weight selection. Moreover, the hypothesis class is allowed to depend on the weights. This is more general than a fixed-weight analysis and better reflects practical settings involving adaptive weighting.
S3. Relatively sharp fast rates for dependent data.
Through weight-dependent localization, the paper uses -mixing for coupling and -mixing for localized variance control. Under an appropriate local -to- control condition, the analysis achieves an iid-order leading rate even under polynomial mixing.
Critical weaknesses
W1. The concrete drift analysis in Section 4 has a more limited scope than the general framework may suggest.
The detailed analysis of the drift term in Section 4 primarily focuses on the squared loss and further requires structural assumptions on the evolution of the regression function, density ratios, and related quantities. Therefore, while the title and the general framework suggest broad applicability, the complete learning-plus-drift excess-risk characterization applies to a somewhat narrower setting. The authors also acknowledge in the discussion that more explicit drift analyses require additional assumptions.
W2. The theory allows data-dependent weights, but no concrete data-driven weight-selection procedure is provided.
Although the uniform result accommodates weights selected based on the observed data, the paper does not develop a practical procedure for selecting such weights from data.
W3. The empirical validation is relatively limited.
The experiments mainly consider a scalar location model and serve primarily to illustrate the theoretical phenomena. They therefore provide limited evidence that the proposed framework or its implications lead to practical improvements on realistic nonstationary prediction tasks.
Questions for authors
Q1. Data-driven weight selection.
The uniformity over allows the selected weight vector to be data-dependent. However, the optimal learning–drift trade-off involves a drift term that is generally unknown. Is it possible to construct a fully data-driven weight-selection rule, based only on the observed sequence, that provably achieves or approximates the oracle trade-off developed in the paper?
Q2. Applicability of the local -to- condition.
The iid-order leading rate under polynomial mixing appears to rely critically on the local -to- control in Assumption A3. Could the authors clarify which commonly used hypothesis classes satisfy the favorable condition? Conversely, what rate should one expect when such local control is unavailable?
Q3. Practical relevance of the examples for Assumption A3.
The two examples in Example 3.1 used to illustrate Assumption A3 appear somewhat restrictive relative to hypothesis classes commonly used in practice. In particular: (1) the scalar class seems mainly illustrative and is rarely used as a practical prediction class; (2) for the linear feature class, how restrictive is the required eigenvalue condition in realistic settings, and is it generally straightforward to verify? For example, does the condition continue to hold when is the feature map associated with a Gaussian kernel? (3) More broadly, can an analogous local -to- condition be established for neural-network hypothesis classes, and if so, under what assumptions?
2 comments
Thank you for your helpful response. I will maintain my positive rating.
We thank the reviewer for the positive evaluation and helpful suggestions. We clarified the scope of the drift analysis in the contributions (W1) and agree that our contribution is theoretical (W3). We expanded the explanation of Cw,∞ in the applications, added Gaussian features to Example 3.1, and extended the discussion on weight selection (W2). Changes in the manuscript are marked with orange font.
Q1. Data-driven weight selection.
Future-risk guarantees require assumptions relating past and future distributions. Our theorem already controls learning for any admissible data-dependent choice. However, constructing a selector that provably approximates the optimal learning–drift trade-off is a separate challenge, since the drift contribution is generally unobservable.
This question motivated dedicated research even under more restrictive settings. Mazzetto and Upfal (2023) establish adaptive window selection for independent observations, with an algorithm that involves a generally non-computable discrepancy quantity and knowledge of unknowable ground-truth parameters. Huang and Wang (2025) use stability-based comparisons of empirical losses, relying on independent repetitions at every time point and a quasi-stationarity condition. So even in these works solely dedicated to weight selection under independence, the oracle bounds require substantial additional assumptions. Developing such selection guarantees in our dependent setting would similarly require a substantial analysis in a restricted setting, which is beyond the general scope of our paper.
Q2. The role of Cw,∞ in the applications.
The favorable case Cw,∞>0 in (A3), combined with finite Kρ, enables the sharper polynomial-mixing rates (Section 3.3). Theorem 3.2 also covers Cw,∞=0, although its bound can then be slower.
Section 5 and Appendix C.1–C.3 give concrete examples:
Linear models: Bounded features and a uniformly positive lower Gram eigenvalue give Cw,∞>0. For a fixed feature dimension, this recovers the iid-order parametric rate under the stated polynomial-mixing conditions.
Basis expansions: Histogram bases on q equal bins for uniformly distributed inputs satisfy Cw,∞=q−1/2>0. As the basis grows, Cw,∞ decreases, so recovering the iid-order rate n−2/3 up to logarithms requires faster polynomial decay than for fixed-dimensional linear models.
ReLU networks: The condition Cw,∞>0 does not generally hold for these classes. Our application therefore uses Cw,∞=0 and recovers the usual smooth nonparametric rate only under exponential mixing, up to logarithms. For networks with fixed, bounded hidden features and a trainable linear output layer, uniformly positive definite feature Gram matrices give Cw,∞>0.
Q3. Feature geometry and Gaussian kernels.
The Gram condition can be checked from assumptions on the input distribution. It is a standard well-conditioning assumption covering many common fixed-dimensional regression models, although it can fail when features become nearly collinear.
For finite Gaussian expansions, take m fixed distinct centers and Φj(z)=m−1/2e−∥z−cj∥22/(2σ2). The constant is Cw,∞=infP∈Pλmin(EP[Φ(Z)Φ(Z)⊤]) and of order m−1/2 for typical nondegenerate distributions. For the full infinite-dimensional Gaussian RKHS, however, we must take Cw,∞=0.