acceptodds
Under review as a conference paper at ICLR 2027

Robust Statistical Learning from Streaming Data: An Adaptive Transfer Perspective

Abstract

Statistical online learning is challenging when streaming batches may follow different models. This paper studies a contamination setting in which each batch follows a central model with high probability, and a batch-specific model otherwise. To address this online heterogeneity, we approach the problem from a transfer learning perspective. Starting with a point estimation example, we first develop a robust one-pass algorithm that simultaneously updates an estimate of the central parameter, and estimates the parameter of each current batch. We then extend this online method to a linear regression problem and establish time-uniform error bounds: When the current batch follows the central model, the updated central estimator becomes increasingly accurate until reaching a contamination-dependent error floor. When the current batch follows a heterogeneous model, the proposed method adaptively transfers information from previous estimates whenever beneficial, yielding a batch-specific estimator that is potentially more accurate than purely local estimation. The method also provides an online heterogeneity detection procedure. Numerical experiments demonstrate the benefits of our robust online method for learning both central and batch-specific models.

Primary area:
learning theory
Keywords:
Online learning, Transfer learning, Contamination model, Streaming data, Robust statistics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QYG2OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies one-pass online learning when each incoming batch follows a central model with probability 1 − ε and a batch-specific model with probability ε (Bernoulli, i.i.d. across batches), and proposes a bidirectional transfer procedure: the running central estimate is used as source information to estimate the current batch parameter via a firm-thresholding operator, while the current batch feeds back into the central estimate through a clipped-score (Huber-type) update. Two settings are analysed — a Gaussian location model and linear regression — with time-uniform finite-sample bounds showing that the central estimator improves as 1/(nb) up to a contamination floor of order ε²/n, that the batch-specific estimator is never worse than local estimation and is strictly better when the shift is small, and that the same machinery yields an online heterogeneity test with zero false discoveries. Experiments are simulations only (p ∈ {10, 30}, n = 100, B = 50 batches, 50 Monte Carlo replications).

Critical strengths
  • The bidirectional-transfer principle is well posed: a mildly heterogeneous batch both benefits from and contributes to the central estimate, with no need for ε or a separation gap.
  • The method is genuinely one-pass with O(1)/O(p) memory and time-uniform bounds over all batches, including an adaptive min that interpolates between local estimation and transfer.
  • The heterogeneity test with zero false discoveries and the data-driven scale estimator ν̂ make the procedure usable without knowing σ or κ₀.
Critical weaknesses
  1. Every guarantee is conditional on a single "trusted" first batch that fixes the projection interval for the whole stream, and a contaminated batch 1 is neither analysed nor recoverable.
  2. The central rate is not order-optimal — its contamination term carries an extra factor p against the σ²(p/(nb) + ε²/n) benchmark — yet it is framed as nearly minimax-optimal.
  3. Algorithm 2 cannot be instantiated as written, because its thresholds and detection rule depend on unspecified constants C₁–C₆ and the practical ν̂ relies on the unanalysed λ_min(XᵀX).
  4. The Bernoulli, sub-Gaussian and ε ≤ 1/4 assumptions are far weaker than the adversarial federated motivation, while the experiments instantiate only a very special alternating shift.
  5. The empirical study is simulations only (one linear model, one ε, no runtime or detection metrics), and the offline ARMUL beats it on the central error in every setting.
Questions for authors
  1. Every guarantee is conditional on a single "trusted" first batch that fixes the projection interval for the whole stream, and a contaminated batch 1 is neither analysed nor recoverable.
  2. The central rate is not order-optimal — its contamination term carries an extra factor p against the σ²(p/(nb) + ε²/n) benchmark — yet it is framed as nearly minimax-optimal.
  3. Algorithm 2 cannot be instantiated as written, because its thresholds and detection rule depend on unspecified constants C₁–C₆ and the practical ν̂ relies on the unanalysed λ_min(XᵀX).
  4. The Bernoulli, sub-Gaussian and ε ≤ 1/4 assumptions are far weaker than the adversarial federated motivation, while the experiments instantiate only a very special alternating shift.
  5. The empirical study is simulations only (one linear model, one ε, no runtime or detection metrics), and the offline ARMUL beats it on the central error in every setting.
Reviewer_3tPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates online statistical learning from streaming data batches in the presence of batch-level contamination, where each incoming batch follows a central model with probability 1−ϵ1-\epsilon and a batch-specific heterogeneous model with probability ϵ\epsilon. The authors propose a one-pass, O(p)O(p)-memory bidirectional online transfer framework that simultaneously estimates the current-batch parameter via a radial firm-thresholding operator and updates the central parameter via a coordinate-wise clipped-score (Huberized) recursion projected onto a reference interval. The manuscript establishes finite-sample, simultaneous error bounds for both the Gaussian location model and linear regression without requiring prior knowledge of ϵ\epsilon or a minimum signal separation distance, and introduces a companion online heterogeneity detection procedure, validating the approach through numerical simulations.

Critical strengths
  1. The theoretical analysis establishes non-asymptotic, simultaneous error bounds for both the central model and heterogeneous batch models without requiring prior knowledge of the contamination rate ϵ\epsilon or a minimum signal separation distance, improving upon existing streaming approaches that rely on asymptotic guarantees and hard batch rejection.

  2. By leveraging continuous partial transfer, the proposed estimator provably and empirically outperforms naive local OLS on mildly shifted heterogeneous batches while preventing negative transfer under large shifts and achieving finite-sample family-wise error control for online anomaly detection.

Critical weaknesses
  1. The theoretical guarantees and projection operators rely critically on the assumption that the very first batch (b=1b=1) is an uncontaminated reference batch β(1)=β∗\boldsymbol{\beta}^{(1)} = \boldsymbol{\beta}^*, to construct fixed projection intervals Ij\mathcal{I}_j; if the first batch is contaminated or experiences a large finite-sample deviation, the central estimator is permanently locked away from β∗\boldsymbol{\beta}^\ast for all subsequent batches b≥2b\ge 2, which could be mitigated by discussing a robust initialization.

  2. Because the central parameter is updated coordinate-wise rather than via low-dimensional spectral projections, the contamination error floor in linear regression scales suboptimally with dimension as O~(σ2pϵ2/n)\tilde{O}(\sigma^2 p \epsilon^2 / n) rather than the minimax optimal O~(σ2ϵ2/n)\tilde{O}(\sigma^2 \epsilon^2 / n) established in prior robust learning literature (e.g., Byzantine-robust distributed learning: Towards optimal statistical rates, Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett, 2018; Streaming algorithms for high-dimensional robust statistics, Ilias Diakonikolas, Daniel M. Kane, Ankit Pensia, and Thanasis Pittas, 2022).

  3. The empirical evaluation of the heterogeneous-model error HbH_b compares against online baselines that only estimate a single central parameter (forcing them to use m^(b)\hat{\mathbf{m}}^{(b)} as a proxy for β^(b)\hat{\boldsymbol{\beta}}^{(b)}), and the authors might consider comparing against online streaming methods that explicitly estimate batch-specific parameters or equipping test-based baselines with a local OLS fallback upon rejection (e.g., Renewable quantile regression with heterogeneous streaming datasets, Xuerong Chen and Senlin Yuan, 2024; Real-time regression analysis of streaming clustered data with possible abnormal data batches, Lan Luo, Ling Zhou, and Peter X.-K. Song, 2023).

Questions for authors

See the weaknesses. I also have the following points to make:

  1. The Abstract states that the paper extends the online method to a linear regression problem and establishes "time-uniform error bounds." While the Gaussian location model in Section 2 operates over an unbounded horizon b=2,3,…b = 2, 3, \dots (Theorem 2), the linear regression formulation in Section 3 fixes a prespecified batch budget BB (page 5, line 255) and requires the per-batch sample size to satisfy n≥C6[p+log⁡(B/δ)]n \ge C_6[p + \log(B/\delta)] (Theorem 3, page 7, line 335; used to control the sample covariance union bound P((EX(B))c)≤2Be−cn≤δ/4\mathbb{P}((\mathcal{E}_X^{(B)})^c) \le 2Be^{-cn} \le \delta/4 in Equation 35).

  2. On lines 324–325, the paper highlights that the central and current-batch estimates in Algorithm 2 require only O(p)O(p) persistent storage between batches (which compares favorably to the O(p2)O(p^2) summary matrices stored by renewable estimation methods). However, within each batch bb, computing the batchwise OLS estimator β~(b)=(X(b)⊤X(b))†X(b)⊤y(b)\widetilde{\boldsymbol{\beta}}^{(b)} = (\mathbf{X}^{(b)\top}\mathbf{X}^{(b)})^\dagger \mathbf{X}^{(b)\top}\mathbf{y}^{(b)} (and λmin⁡(X(b)⊤X(b))\lambda_{\min}(\mathbf{X}^{(b)\top}\mathbf{X}^{(b)}) in Remark 2) requires O(np+p2)O(np + p^2) transient working memory during batch processing and O(np2+p3)O(np^2 + p^3) time per batch, in contrast to first-order online baselines (such as LAD SGD and online quantile regression) that operate in O(np)O(np) time and O(p)O(p) total memory per batch without matrix inversions. Explicitly stating the per-batch computational and working-memory complexity alongside the O(p)O(p) persistent storage claim would provide a more complete operational comparison.

Reviewer_pF2gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses robust online learning where each batch follows a central model with high probability, and a batch-specific model otherwise. The authors propose a one-pass online learning algorithm based on transfer learning framework for estimating the central parameter and batch-specific parameters. The proposed method is developed first for point estimation and then linear regression. The authors show a time-uniform error bounds for both estimates. The experiments show the effectiveness of the proposed method in improving estimation accuracy for central and batch-specific models.

Critical strengths
  1. A unified framework for bidirectional online transfer when batch-level contamination exists: I think that the combination of robust central updates and adaptation to batch-specific differences is well motivated and the proposed framework offers an effective way to combine them.
  2. Joint finite-sample guarantees for central and batch-specific estimation: The theoretical analysis includes both central parameter and parameter for current batch. This theoretical result shows how accumulated information across batches improves central estimation, and when information from the central model can improve estimation for current batch.
Critical weaknesses
  1. Limited algorithmic novelty: The proposed algorithm is based on existing clipped-score updates and firm thresholding. It seems that the algorithmic novelty of this study lies in the combination of these components in an online setting. However, I am not convinced whether this study establishes how well the joint update works, in comparison with employing existing robust online estimation and adaptive transfer methods sequentially. More detailed discussion is necessary compared to, for example, Pesme & Flammarion, 2020 and Li et al., 2022.

  2. Limitation on theoretical optimality: My concern is the dimension dependence in the contamination term in Theorem3. In Remark1, the authors mention that the bound contains an additional factors that are dimension-dependent. This study does not clarify whether it is intrinsic to the problem addressed in this paper or simply result of the theoretical analysis.

  3. Selection of baseline methods: Most of the online methods estimate only the central parameter, whereas the proposed method outputs a batch-specific estimate as well. This makes it difficult to evaluate and interpret the comparison of batch-specific estimation error. For example, I recommend that the authors conduct experiments with a simpler two-stage baseline that first updates a robust central estimator and then shrinks the current batch estimator toward it, with the amount of shrinkage determined from the data.

  4. Limited experimental design: This study focuses on linear model. I think that experiments on broader model classes would strengthen the claim of this study.

Questions for authors
  1. Dependence on dimension in Theorem3: Could you clarify whether the additional dimension-dependent factor in the contamination term is due to limitation of the proof, or theoretical reasons under the one-pass and memory constraints? Furthermore, if possible, a sharper bound or a lower-bound is preferable.

  2. Comparison with simpler transfer baseline: How well does the proposed algorithm work in comparison with two-stage procedure that combines an existing robust online central estimator with adaptive shrinkage of the current batch's OLS estimator toward a central estimate?

  3. Effectiveness of scale calibration: Remark2 suggests a calibration of the scale parameter. How sensitive is the method to this choice?

  4. Online detection performance: Corollary3 shows theoretical analysis on online detection performance. Could you show empirical Type-I error and detection power of the proposed heterogeneity test?

  5. Experimental design: Could you show effectiveness of the proposed method for a broader model classes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.