acceptodds
Under review as a conference paper at ICLR 2027

Score-Preserving Continual Lasso under Task Shift

Abstract

Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We study continual learning for high-dimensional sparse linear regression, where regularization creates a difficulty absent from continual ridge regression: that recursively reusing an -penalized estimator can propagate earlier shrinkage and accumulate regularization bias across tasks. We propose score-preserving continual Lasso (SP-CL), which separates sparse estimation from state transmission by preserving the historical squared-loss score. Under squared loss, its cumulative estimator exactly recovers a full-history Lasso minimizer. When task coefficients coincide, it inherits the full-history sparse rate. For related task-specific coefficients, SP-CL uses a regularized current-task correction that leaves the transmitted state unchanged and attains the offline source-assisted rate under an oracle related-task condition. We also derive Overall/BWT/FWT decompositions that distinguish historical information loss, task conflict, and current-task specialization, and validate the resulting regimes in simulation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, high-dimensional statistics, sparse regression, Lasso, statistical learning theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_gXmGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an implementation method for pooled Lasso in continual learning scenarios without replay. First, it points out that under L1 regularization, the previous Lasso solution cannot be directly used as the center of the quadratic historical penalty because its shrinkage bias accumulates with each task, causing the cumulative estimate to deviate from the joint training objective. The paper proposes the SP-CL algorithm, which maintains a cumulative curvature HtH_t, utu_t satisfying Ht×ut=GtH_t \times u_t=G_t (i.e., a first-order score without penalty contribution), and a cumulative sample size NtN_t at each task boundary. In each step, it solves a quadratic penalized Lasso centered at ut−1u_{t-1}, and proves that this yields a minimum point of the pooled Lasso. On the evaluation side, it decomposes the evaluation metrics into task conflict, information loss, and specialization for analysis. Experiments on both synthetic and real datasets validate the proposed mechanism.

Critical strengths

The paper proposes an accurate solution for pooled Lasso in continual learning scenarios without replay, and uses experiments to verify that the obtained cumulative solution is indeed consistent with the offline pooled Lasso trained on all historical data, while the method of directly passing the center is significantly degraded.

Critical weaknesses

Replay-based methods were excluded from the paper's protocol, but they remain a typical and strong family of methods in continual learning. The paper did not compare against any replay baselines (such as ER, Chaudhry et al., 2019) with equivalent memory budgets as a reference.

Questions for authors

Q1 [Suggestion]. Could a replay-based method be added as a reference under the same memory budget, and could a storage efficiency analysis of SP-CL be added?

4 comments
AuthorsOpenReview ↗

Response to Reviewer gXmG (1/2)

Response to Reviewer gXmG, part 1 of 2.

Thank you for suggesting a replay comparison under the same memory budget.

Our primary focus is regularization-based continual learning under a protocol that retains approved historical summaries. DataSHIELD provides a related access setting: individual records remain at their source, and the analyst receives disclosure-controlled aggregate outputs. We additionally evaluate experience replay under the same retained-memory budget, allowing the learner to retain historical samples.

1. Replay implementation and memory budget

Our primary baseline is experience replay (ER; Chaudhry et al., 2019) with reservoir sampling (Vitter, 1985), using Lasso for fitting. A uniform reservoir Mt−1\mathcal M_{t-1} contains mtm_t past labeled rows before task tt. Each current and cached sample receives weight one:

\left\{\frac{\|Y_t-X_tb\|_2^2+\|Y_{\mathcal M_{t-1}}-X_{\mathcal M_{t-1}}b\|_2^2}{2(n_t+m_t)} +\lambda_t^{\mathrm{ER}}\|b\|_1\right\}.$$ This is equal weighting per sample, not equal total weight per group. The reservoir is updated after fitting. We separately tune replay's penalty using the actual training size $n_t+m_t$; current observations are not counted twice. We retain historical-weighted replay as a supplementary control: it multiplies the historical residual sum by $N_{t-1}/m_t$ and normalizes by $2N_t$. The versions use identical buffer capacities, seeds, and splits, with separate validation-only penalty selection and the same Lasso KKT tolerance. In the long-stream experiment, arrays, both prediction heads, schedules, bookkeeping, and metadata give SP-CL a budget of 468,723 bytes. Both replay versions use the largest feasible buffer: 240 rows taking 467,177 bytes; one more row exceeds the budget. Current inputs and temporary solver workspace are accounted for separately. Penalties are selected on four independent pilot streams before evaluation, without formal-test outcomes or true coefficients. SP-CL's $O(p^2)$ state preserves historical squared loss independently of stream length; the comparison tests its benefit once a fixed reservoir cannot retain all history. **2. Results after the reservoir fills** Appendix I.1 uses $p=240$, $s=8$, $n_t=40$ and $T=30$, giving $N_T=1,200$. Replay retains all history when fitting task 7 and first subsamples it at task 8. Final population excess risks exclude noise and report mean $\pm$ SE over 20 paired streams: | Setting and evaluation target | SP-CL generalist | Replay (equal) | Replay (weighted) | | --- | ---: | ---: | ---: | | Shared coefficients | $0.033\pm0.002$ | $0.173\pm0.014$ | $0.239\pm0.020$ | | Task shift: current task | $0.695\pm0.016$ | $0.878\pm0.055$ | $1.030\pm0.061$ | | Task shift: previous-task average | $0.226\pm0.002$ | $0.384\pm0.016$ | $0.423\pm0.022$ | The shift path keeps support fixed with $\|\beta_T^\star-\beta_1^\star\|_1=3$. Equal replay improves on the weighted control, while generalist retains lower current and historical risks in these settings. Figure 8 shows both replay trajectories across the capacity threshold. The separate chronological forecasts yield the following prospective response MSEs. Penalties use earlier validation data; replay losses average five electricity or ten SPX reservoir seeds. | Stream | Generalist | Specialist | Replay (equal) | Replay (weighted) | | --- | ---: | ---: | ---: | ---: | | Nord Pool | 16.691254 | 17.529177 | 17.225040 | 17.861476 | | PJM | 24.593975 | 25.150766 | 26.051670 | 28.984537 | | Oxford-Man SPX | 0.424512 | 0.381271 | 0.405277 | 0.431155 | | Financial text | 0.150242 | 0.143020 | 0.150242 | 0.150242 | Equal replay improves on weighted replay in all three market streams. On SPX it also outperforms generalist, while specialist remains better than both replay variants.
AuthorsOpenReview ↗

Response to Reviewer gXmG (2/2)

Response to Reviewer gXmG, part 2 of 2.

Permanent holdouts assess historical retention under a distinct protocol (Tables 7–9). Both replay versions retain 157 daily rows for NP/PJM and 200 rows for SPX. Generalist has lower final Overall and historical MSE than either replay under both validation criteria on all three streams. SPX historical MSEs for generalist/equal/weighted replay are .43947/.44640/.45841 under current-task validation and .42387/.45396/.44953 under history-aware validation. Both criteria use development data only, with frozen rules and ten seed-specific losses per replay version. Appendix I.4.1 also reports current-task metrics and BWT/FWT. BWT uses each method's own task-time baseline and is interpreted alongside absolute historical MSE; FWT compares with current-only Lasso after current-task training (Section 2; Proposition 2).

3. Agreement with full-history replay

When all history fits, mt=Nt−1m_t=N_{t-1}, so both replay weightings give the same pooled-Lasso objective as SP-CL at a matched penalty. A separate full-buffer check gives an objective gap below 2.23×10−162.23\times10^{-16}. Financial text's sparse CSR history fits the matched budget; replay and generalist agree at the reported precision. This equality checks the historical objective, whereas the finite-buffer comparisons evaluate a different information constraint.

The permanent-company-holdout text evaluation also retains all 21,775 eligible historical reports. Replay/generalist predictions agree within 3.20×10−123.20\times10^{-12} across all evaluation updates (Appendix I.5.1, Table 12). Equal and weighted replay share this full-history row.

Dear Reviewer gXmG,

As the discussion period is approaching its end, we would like to kindly follow up on our response. We sincerely appreciate your thoughtful feedback and hope that our rebuttal and revisions have addressed your concerns.

If any concerns remain, we would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards, Authors of Submission 46828

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts in addressing my concerns.

The replay comparison under a strictly matched memory budget is thorough and convincing. SP-CL's consistent advantage on the synthetic streams and most real-data benchmarks, together with the full-history equivalence verification, satisfactorily resolves my concern.

Overall, I have decided to raise my score from 2 to 3.

Reviewer_mBnEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continual learning for high-dimensional sparse linear regression. A key challenge in the sparse setting is the propagation of earlier shrinkage and accumulation of regularization bias across tasks. To address these challenges, the paper proposes Score Preserving Continual LASSO (SP-CL), a variant of LASSO which maintains information (Gram matrix and cross-moments) from previous tasks. The paper provides identifiability, statistical recovery guarantees, along with some experiments.

Critical strengths
  1. The paper is well-written, I had no issue parsing through the results.
  2. The authors tackle an important problem in continual learning of catastrophic forgetting particularly in the setting where sparsity is important. Their proposed algorithm SP-CL is backed by identifiability and statistical recovery results in high-dimensional regimes.
Critical weaknesses
  1. The authors fail to acknowledge the literature on online LASSO raising concerns regarding novelty of this work. For instance, the practice of preserving covariance matrices is common in online LASSO; below are few papers that have similar idea:

    • Garrigues, Pierre, and Laurent Ghaoui. "An homotopy algorithm for the lasso with online observations." Advances in neural information processing systems 21 (2008).
    • D. Angelosante, J. A. Bazerque and G. B. Giannakis, "Online Adaptive Estimation of Sparse Signals: Where RLS Meets the ℓ1\ell_1-Norm," in IEEE Transactions on Signal Processing, vol. 58, no. 7, pp. 3436-3447, July 2010.
    • Babadi, Behtash, Nicholas Kalouptsidis, and Vahid Tarokh. "SPARLS: The sparse RLS algorithm." IEEE Transactions on Signal Processing 58, no. 8 (2010): 4013-4025.
  2. Following from earlier point, even in CL literature there are plethora of algorithms such as GEM, GPM, OWM, EWC that store statistics as memory.

  3. The main-text claims are stronger than the proofs. Specifically, the proof for the support-recovery claim is unclear (line 886 - 887). See Question 2 and 3.

  4. The paper considers a noisy sparse linear model, and LASSO type optimization problem. Despite the tractability of this setting the authors have not stated the precise constants (despite their promise (line 367) of availability in Appendix C ) in their Corollary 1, and Theorem 2. In the literature, such explicit-constants for LASSO do exist (e.g., Bickel et al. 2009). Please state the constants, or show empirically that the observed rates in section 6 match the predicted scaling.

Questions for authors
  1. Could the authors comment on the differences of their work with respect to vast literature on online LASSO, and memory based CL techniques?

  2. In Appendix D.1, the proof of Corollary 1 assumes that Nt≳s2log⁡pN_t \gtrsim s^2\log p (line 881) which is stronger compared to its statement requiring just Nt≳slog⁡pN_t \gtrsim s\log p (line 291). Could the authors clarify what are the exact conditions for which their final bounds apply ?

  3. Could the authors provide precise arguments for the signed-support claim in Corollary 1?

  4. Could the authors clarify the experiments design for Figure 1? Specifically in line 204: what does it mean by Gaussian AR(1) in the context?

  5. Tightness of bounds in Corollary 1, and Theorem 2 compared to the empirical observations in section 6 are missing. Can the authors comment whether the bounds are valid, and observe the same rates?

  6. In Figure, how does effective sparsity of "Current-only Lasso" compare with "SP-CL cumulative"?

  7. Minor corrections: In the line 022: the acronyms "BWT", "FWT" not defined until later making it hard to parse the sentence for the readers.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mBnE (1/4)

Response to Reviewer mBnE, part 1 of 4.

We thank the reviewer for the careful assessment. We have revised the positioning, separated estimation from selection conditions, expanded the proofs, and added controlled rate diagnostics. Page numbers below refer to the revised manuscript.

Q1a and Weakness 1: differences from online sparse regression.

We thank the reviewer for these references. We now cite and discuss all three in the Introduction and have added two representative baselines—cumulative-score Lasso and exponentially weighted Lasso—to evaluate cumulative estimation and adaptive tracking (Section 6; Appendix I.1). We also agree that retaining the Gram matrix and cross-moments is standard in online Lasso, and we do not claim it as a contribution. With the same data and penalty, cumulative-score Lasso and the SP-CL generalist minimize the same objective; in our experiment, their maximum coefficient difference is 4.24×10−134.24\times10^{-13}.

We study continual learning in high-dimensional sparse linear regression, with an explicit focus on retaining performance on earlier tasks while learning subsequent tasks. This emphasis differs from the main questions addressed by the cited online sparse-regression literature. Garrigues and El Ghaoui (2008) study how to update a Lasso solution as observations arrive. Angelosante et al. (2010) and Babadi et al. (2010) develop recursive sparse estimation, including mechanisms for tracking a changing signal, where older observations are downweighted because they are less representative of the current target. We study continual learning, where earlier tasks remain evaluation targets after the task distribution changes, so the object of analysis is forgetting and transfer. The two lines share mathematical tools but treat historical information differently.

Within continual learning, we examine the sparse extension of parameter-centered quadratic regularization, a mechanism represented by EWC and the framework of Zhao et al. (2024). Sparsity also has a concrete role in modern continual learning: LG-CBM and CI-CBM use sparsity-promoting regularization on linear concept-to-class prediction heads to support interpretability. Motivated by these sparse heads, we consider how such heads could be learned through parameter-centered regularization, with historical information represented by a quadratic penalty.

What is new.

  1. A failure of directly transmitting ℓ1\ell_1-penalized centers. Our fixed-coordinate, squared-loss model isolates a compatibility issue in this formulation: directly reusing the previous ℓ1\ell_1-penalized head as the quadratic center carries earlier penalty contributions into subsequent historical objectives. Proposition 1 and Example 1 show that this mechanism can erase previously learned signals even without task conflict, motivating the separation between the sparse predictor and the score-preserving historical state. Figure 1 examines the corresponding signal attenuation under noisy, correlated designs. This identifies a source of forgetting arising from the transmission mechanism itself.

  2. The correction. SP-CL separates the sparse predictor from a historical score carrier satisfying Htut=GtH_tu_t=G_t. Its equivalence to pooled Lasso establishes that the cumulative historical squared loss is preserved without the distortion introduced by transmitting penalized centers. The role of this equivalence is to certify faithful historical information transmission within the CL procedure. This use of an offline reference is consistent with the broader approach of analyzing continual regularization against offline estimation benchmarks, as in Zhao et al. (2024).

  3. Adaptation under task shift. The specialist adds a sparse current-task correction without overwriting the historical state, with a rate guarantee under the stated related-task conditions (Theorem 2(c)). In Section 6, at h=12h=12 it attains post-update FWT (Section 2) of 8.238.23, against 6.456.45 for the generalist, the matched cumulative-Lasso estimator. Proposition 2 ties this difference to the current-task prediction gain and characterizes the cost on historical tasks.

AuthorsOpenReview ↗

Response to Reviewer mBnE (2/4)

Response to Reviewer mBnE, part 2 of 4.

Empirical comparison. Permanent real-data holdouts now provide complementary evidence (Appendices I.4.1 and I.5.1): on financial text, specialist has lower mean task-time MSE and higher FWT under both validation criteria, while its final predictor and final historical risk coincide with generalist. Oxford-Man also exhibits an adaptation–retention trade-off under current-task validation; history-aware validation selects no correction.

Q1b and Weakness 2: differences from memory-based continual learning.

We agree that retaining historical information is an established component of continual learning. The taxonomy of Wang et al. (2024) distinguishes regularization-, replay-, optimization-, representation-, and architecture-based approaches, with connections across these categories. Our work focuses on quadratic regularization-based continual learning, where a parameter-centered quadratic term represents historical information.

The cited methods use retained information in different ways. GEM stores historical examples to construct gradient constraints; GPM stores activation-derived subspace bases for gradient projection; and OWM recursively maintains a projection matrix to restrict updates along previously observed input directions. These mechanisms address interference through constraints on parameter updates. Our analysis addresses the fidelity of the historical-loss representation used in a regularized objective.

EWC is particularly relevant because it uses parameter anchors and Fisher-based weights to construct quadratic penalties. For a sparse linear analogue of this approach, our question is whether the historical center remains valid after an ℓ1\ell_1 penalty is introduced. The exact-curvature failure established above shows that even a single consolidated quadratic can transmit earlier penalty contributions when its center is passed directly from the penalized predictor. SP-CL addresses this issue by separating the historical score carrier from the sparse predictor. This identifies the specific aspect of memory design studied here: the validity of the historical quadratic and its consequences for sparse signal retention.

Q2: distinct pooled-Lasso conditions for estimation and signed-support recovery.

Corollary 1 concerns the cumulative estimator under shared coefficients. Its estimation bounds require Condition 1, Nt≳slog⁡(2p/ξ)N_t\gtrsim s\log(2p/\xi), and λw,t≍log⁡(2p/ξ)/Nt\lambda_{w,t}\asymp\sqrt{\log(2p/\xi)/N_t} with a sufficiently large constant; they do not require irrepresentability or beta-min. Signed-support recovery uses additional selection conditions. The original statement referred this conclusion to the additional conditions in Appendix D.1; we now display its stronger sample-size requirement directly in the corollary (p. 6).

By Theorem 1, the SP-CL cumulative estimator is a pooled-Lasso minimizer for the same data and penalty. At a fixed task tt, it therefore inherits the corresponding pooled-Lasso estimation and selection guarantees without additional statistical requirements from sequential processing. The s2log⁡ps^2\log p sample-size condition belongs to our explicit pooled-Lasso support-recovery proof; it is not a statistical cost of SP-CL.

For precision, Appendix D.1, Eqs. (31)–(32) (p. 20), gives the following sufficient conditions. Let S=supp⁡(β⋆)S=\operatorname{supp}(\beta^\star), s=∣S∣≥1s=|S|\ge1, βmin⁡=min⁡k∈S∣βk⋆∣\beta_{\min}=\min_{k\in S}|\beta_k^\star|, λ=λw,t>0\lambda=\lambda_{w,t}>0, and u=log⁡(8p2/ξ)u=\log(8p^2/\xi). In addition to Condition 1, assume

\|\Sigma_{S^cS}\Sigma_{SS}^{-1}\|_\infty\le1-\gamma, \quad 0<\gamma\le1,$$ and $$N_t\ge C_0\max\{1,M_S^2s^2/\gamma^2\}u,\quad \lambda\ge(C_1\sigma_\varepsilon/\gamma)\sqrt{u/N_t},\quad \beta_{\min}\ge3M_S\lambda.$$ Here the matrix $\infty$-norm is the maximum absolute row sum, and $C_0,C_1$ depend on design tails and the covariance upper bound. For fixed $M_S,\gamma$, the $s^2\log(2p/\xi)$ scaling comes from the entrywise covariance control used in this proof; we do not claim it is a sharp or necessary threshold. The finite-confidence conclusion is at least $1-\xi$; probability tending to one requires a sequence $\xi\to0$ satisfying the stated conditions.
AuthorsOpenReview ↗

Response to Reviewer mBnE (3/4)

Response to Reviewer mBnE, part 3 of 4.

Q3 and Weakness 3: explicit support-recovery proof and transfer to SP-CL.

Appendix D.1 now gives an explicit primal–dual witness proof for the pooled Lasso (pp. 20–21; Lemma 3 and Eq. (33), p. 20). Let Σ^=X⊤X/Nt\widehat\Sigma=X^\top X/N_t, a=X⊤ε/Nta=X^\top\varepsilon/N_t, and q=sgn⁡(βS⋆)q=\operatorname{sgn}(\beta_S^\star). Construct

\widetilde w_S=\beta_S^\star+\widehat\Sigma_{SS}^{-1}(a_S-\lambda q).$$ On $\|\widehat\Sigma-\Sigma\|_{\max}\le\gamma/(8M_Ss)$ and $\|a\|_\infty\le\gamma\lambda/8$, the proof establishes $\|\widehat\Sigma_{SS}^{-1}\|_\infty\le2M_S$ and hence the active-coordinate error bound $$\|\widetilde w_S-\beta_S^\star\|_\infty \le9M_S\lambda/4<\beta_{\min}.$$ Thus all active signs are correct. Writing $\widehat Q=\widehat\Sigma_{S^cS}\widehat\Sigma_{SS}^{-1}$, the inactive dual vector obeys $$\|\widehat Qq+(a_{S^c}-\widehat Qa_S)/\lambda\|_\infty\le1-\gamma/4<1.$$ These verify the KKT conditions for the full pooled problem. Strict inactive feasibility forces every minimizer to vanish on $S^c$; positive definiteness of $\widehat\Sigma_{SS}$ then gives uniqueness, even when the full Gram matrix is singular. Bernstein's inequality and a union bound establish the required events jointly with probability at least $1-\xi$ under Eq. (32). By Theorem 1, SP-CL must equal this unique pooled-Lasso solution on that event and therefore has the same signed support under the same conditions. **Q4: Gaussian AR(1).** AR(1) denotes correlation across feature coordinates: independently sampled rows $x_{ti}\sim N(0,\Sigma)$ with $\Sigma_{k\ell}=0.5^{|k-\ell|}$. Observations and task designs are independent. Figure 1 uses $p=200,s=10,n_t=40,T=30$, noise standard deviation 1, and 50 streams with shared coefficients. This is now explicit near Figure 1 (p. 5) and in Section 6 (p. 9). **Q5 and Weakness 4: classical guarantees, explicit constants, and empirical rates.** By Theorem 1, Corollary 1 inherits the corresponding pooled-Lasso estimation guarantee; Theorem 2(a) applies the same Lasso argument to the current batch, under our stated conditions (Bickel et al., 2009). For Theorem 2(b), we control the extra heterogeneity score with the added bound $M_t\le B$ and adjusted penalty before applying the Lasso inequality (Appendix C, Eqs. (22)–(24), p. 18). For the specialist in Theorem 2(c), we follow the squared-loss two-stage analysis of Tian and Feng (2023). Appendix C (pp. 18–20) handles the shared current data through joint events. Its strengthened sufficient conditions remain fixed $h\le h_0$, $h\sqrt{L/n_t}\le1$, and $n_t,N_{t-1}\gtrsim sL$. Lemma 2, Eq. (25) (p. 19), gives explicit Lasso constants. For error $e$ around an $s$-sparse target, let $a$ be the full empirical loss score. If $\lambda\ge2\|a\|_\infty$ and the cone-RE lower bound is $\kappa_l/2$, then $$\|e\|_2\le6\sqrt{s}\lambda/\kappa_l,\quad \|e\|_1\le24s\lambda/\kappa_l,\quad \|e\|_\Sigma^2\le36\kappa_us\lambda^2/\kappa_l^2.$$ Appendix C gives parameter dependencies for the remaining concentration and specialist constants; they are not fully numerically calibrated. Appendix I.2 (pp. 31–32; Figure 9, p. 32) reports 30 repetitions per setting with prespecified penalty schedules. For shared coefficients, mean squared coefficient-estimation error divided by $s\log(2p/.05)/N$ ranges from 4.188 to 4.469; mean $\ell_1$ error divided by $s\sqrt{\log(2p/.05)/N}$ ranges from 2.016 to 2.073. The log–log sample-size slope is −1.007 (pointwise 95% interval [−1.051, −0.964]), and the sparsity slope is 1.021 [0.980, 1.062], consistent with the predicted grid scaling. Appendix I.2 also examines the heterogeneity score. Specialist mean errors divided by $sL/N+\min\{h\sqrt{L/n_t},h^2\}$ range from 2.096 to 16.400, a wider range than under shared coefficients. These diagnostics assess finite-grid scaling, not sharpness or tightness; they do not numerically calibrate the constants in the high-probability guarantees. **Q6: effective sparsity.** Section 6/Figure 2 (p. 9) defines effective support by $|\widehat\beta_k|>10^{-3}$. Appendix B.4 (pp. 16–17) reports the full selection diagnostics. At the final task ($p=1000,s=10,n_t=40,T=20$; 50 streams), Current-only selects $12.98\pm0.48$ coordinates versus SP-CL's $10.06\pm0.03$ (mean ± SE). Mean true-positive rates are 0.252 versus 1.000; mean false-discovery proportions are 0.805 versus 0.0055. SP-CL achieves the correct signed support in 47/50 streams at the final task; each of the other three contains one false positive. Both Lasso methods use the calibration in [Wainwright (2009, Section VII)](https://arindam.cs.illinois.edu/courses/s14cs598/paper/spt1.pdf),
Reviewer_ygNHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a continual sparse linear regression problem where tasks arrive sequentially. Each task has its specific coefficient vector and historical data can never be revisited. The authors address the problem by showing that passing the penalized lasso estimates through tasks will accumulate the shrinkage. They propose SP-CL, which instead passes on a state which keeps the historical (squared) loss score. The paper show that the estimator equals the offline pooled lasso estimator and a sparse correction fitted on arriving data adapts it to the current task without altering the state. The paper also provides error bounds under tasks shift and decompositions of overall, BWT and FWT. Simulations further support the results.

Critical strengths
  1. The paper clearly explain the reason why ℓ1\ell_1 penalty leads to a problem in continual linear regression and identifies the difficulties with insights.
  2. The proposed method is simple and effective, as shown in Theorem 1 that the cumulative estimator equals the offline pooled Lasso estimator at every step.
  3. The metric decomposition separates forgetting from task conflict, which provides a good interpretation.
Critical weaknesses
  1. Practical setting. Given the ℓ1\ell_1 penalty and the simulation settings, the paper targets high-dimensional linear regression. Then the transmitted state is a p×pp \times p matrix, which may require even more storage than the raw data. On the other hand, when Nt>pN_t > p, least square or ridge estimator can be computed from the same state, and the need for sparsity in the cumulative estimator is not that clear to me. The author should clarify which constraint motivates the setting of maintaining a matrix instead of storing raw data (privacy, data access, etc.). They should also specify the regime in which SP-CL is preferable to those simpler alternatives.

  2. Task heterogeneity in Theorem 2 (b). Under task shift, the pooled score contains the term Nt−1∑j≤t(Xj⊤Xj−njΣ)(βj∗−wt∗)N_t^{-1} \sum_{j \leq t} (X_j^\top X_j -n_j \Sigma ) ( \beta_j^* - w_t^* ), which is with mean zero, but its magnitude increases with the dispersion of the task coefficient, e.g. $ TC_t $. The theorem didn't ask for any assumption on controlling this term and the penalty does not consider it, either. The constants, discussed in Appendix C, depend only on κ\kappa's so they do not absorb this one. Since following theoretical results depend on this theorem, a detailed discussion on this theorem about extra conditions will be necessary.

  3. The empirical evaluation relies mainly on simulations. More real data analysis would strengthen the numerical section.

  4. Related online methods for high dimensional regression are not fully discussed in the experiment part. More comparison with baseline methods will make the paper stronger.

Questions for authors

Please address the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ygNH (1/3)

Response to Reviewer ygNH, part 1 of 3.

We thank the reviewer for recognizing the shrinkage mechanism, exact pooled-Lasso equivalence, and metric interpretation. We address the four weaknesses below.

Weakness 1. Historical-data access and the regimes where SP-CL is useful.

(a) Why retain a matrix when Nt<pN_t<p? The motivating constraint is access to individual records. In DataSHIELD, records remain at contributing sites while approved GLM analyses return scores and information matrices. This provides a concrete precedent for access to approved summaries without access to individual-level history.

Under our historical-access protocol, SP-CL retains curvature and an unpenalized score to reconstruct cumulative squared loss and recover pooled Lasso. The matrix serves this information-access constraint; Nt<pN_t<p itself does not imply storage savings. When all raw history can instead be retained and reused, full-history Lasso solves the same objective. Our replay comparisons evaluate that alternative access setting.

(b) Why Lasso when least squares or ridge use the same summaries? A sparse estimator is useful when the cumulative target is sparse, the design and task dispersion satisfy the stated conditions, and variable selection matters. The sample requirement scales with sw,tLs_{w,t}L, making sw,tL≪Nts_{w,t}L\ll N_t with non-negligible p/Ntp/N_t particularly informative. Even when Nt>pN_t>p, Lasso exploits a small active set and supports selection under additional conditions, whereas ridge generally stays dense.

(c) When does bounded storage help? Appendix I.1 uses p=240,nt=40,T=30p=240,n_t=40,T=30, hence NT=1200N_T=1200. SP-CL uses 468,723 bytes versus 467,177 for the largest feasible 240-row reservoir. Over 20 paired shared-coefficient streams with eight signals, final excess risks are .033±.002.033\pm.002 for generalist, .516±.013.516\pm.013 for continual ridge, and .173±.014.173\pm.014 for equal replay (mean ±\pm SE), using independently selected pilot penalties. Task shift and s=32,120s=32,120 are also evaluated.

Under the commuting empirical-Hessian structure in Zhao et al. (2024), an exact O(p)O(p) evolving state is possible in an available common basis; total storage is O(p)O(p) when that basis is implicit or compact (Appendices A.4/G). Appendix J separately gives an O(p)O(p) scalar-curvature variant without a shared basis. Its KKT-based center matches the current smooth surrogate's gradient at the sparse solution, preserving that minimizer at a matched penalty while approximating curvature. The appendix reports this local property and empirical comparisons, and explains its generalized implicit-FTRL formulation.

Weakness 2. The missing heterogeneity term in Theorem 2(b).

Thank you for pointing out this missing term. The reviewer is correct: common covariance makes the additional score mean zero but does not control its size. We have added an explicit task-dispersion bound to Theorem 2(b), Mt=max⁡j≤t∥βj⋆−wt⋆∥Σ≤B,M_t=\max_{j\le t}\|\beta_j^\star-w_t^\star\|_\Sigma\le B, where BB is fixed independently of dimension, sample sizes, sparsity, and time. Under the stated design and sparsity assumptions, with Nt≳sw,tLN_t\gtrsim s_{w,t}L, a sufficiently large multiplier in λw,t≍(σε+B)L/Nt\lambda_{w,t}\asymp(\sigma_\varepsilon+B)\sqrt{L/N_t} yields

\lesssim(\sigma_\varepsilon+B)^2s_{w,t}L/N_t.$$ Here $L=\log(C_{\mathrm u}pt(t+1)/\delta)$, and the guarantee holds on a joint event of probability at least $1-\delta$ at all times satisfying the theorem's conditions. Thus the original sample-size rate is retained within this explicitly bounded-dispersion class. Original Condition 1 and sparsity alone do not imply the new bound. For the general dispersion-dependent calculation, let $d_{j,t}=\beta_j^\star-w_t^\star$ and $\nu_t^2=\sum_{j\le t}(n_j/N_t)\|d_{j,t}\|_\Sigma^2$. Under the stated design assumptions, the additional score satisfies, on the joint high-probability event, $$R_t=N_t^{-1}\sum_{j\le t}(X_j^\top X_j-n_j\Sigma)d_{j,t}, \qquad \|R_t\|_\infty\le C\{\nu_t\sqrt{L/N_t}+M_tL/N_t\}.$$ For $N_t\ge CL$, a sufficient penalty controlling both observation noise and heterogeneity is $$\lambda_{w,t}\ge C\{(\sigma_\varepsilon+\nu_t)\sqrt{L/N_t}+M_tL/N_t\}.$$ These population dispersion quantities calibrate the theoretical sufficient penalty. Appendix C, Eqs. (22)–(24), p. 18, give the derivation.
AuthorsOpenReview ↗

Response to Reviewer ygNH (2/3)

Response to Reviewer ygNH, part 2 of 3.

The corrected first-stage bound also enters the second-stage score through Xt⊤Xt(w^t−wt⋆)/ntX_t^\top X_t(\widehat w_t-w_t^\star)/n_t. Appendix D.2 uses

\qquad s_{w,t}\lambda_{w,t}\le c_\Delta\lambda_{\Delta,t}$$ to control this perturbation, with the explicit $c_\Delta$ given there, and states the additional sample-size, support-compatibility, irrepresentability, and beta-min conditions for signed-support recovery (p. 22, Eqs. (35)–(38)). The self-contained proof of Theorem 2(c) now requires $n_t\gtrsim sL$, strengthening the original $n_t\gtrsim L$ condition, together with $N_{t-1}\gtrsim sL$, fixed $h\le h_0$, and $h\sqrt{L/n_t}\le1$ in the stated related-task class. Its specialist rate form is retained under these revised sufficient conditions. Exact pooled equivalence and the deterministic metric identities remain valid independently of these statistical restrictions. See Theorem 2, pp. 7–8, and Appendices C–D, pp. 18–23. **Weakness 3. Real-data evaluation.** Following your suggestions, we have added four chronological streams with fixed feature dictionaries: the **Nord Pool and PJM** experiments predict 24 hourly electricity prices using 247 lag/forecast/weekday coordinates; **Oxford-Man** predicts SPX log-realized variance using 396 lagged volatility coordinates; **Kogan financial text** predicts twelve-month log volatility using 10,000 binary word indicators and past volatility. Market updates are monthly, and text updates are annual. [Electricity source](https://doi.org/10.1016/j.apenergy.2021.116983), [Oxford-Man replication](https://doi.org/10.1093/ectj/utad003), [original financial corpus](https://www.cs.cmu.edu/~ark/10K/data/) Methods share inputs and forecast targets within each stream. Preprocessing uses earlier reference data; tuning uses designated development data, then rules are frozen. Predictions precede label-based updates, with twelve-month label maturity enforced for text. Continual ridge uses only two selected scalars: $\lambda_1=\lambda_{\mathrm{init}}$ and $\lambda_t=\lambda_0N_{t-1}/n_t$ for $t\ge2$. Equal replay weights each current/cached row by one and selects its penalty separately; historical-weighted replay remains supplementary. Initialization, splits and grids are specified in Appendices B.4 and I.4–I.5. The following **prospective response MSEs** use original-price units for electricity. Replay averages five electricity or ten SPX seed-specific losses. | Stream | Generalist | Specialist | Continual ridge | Current-only | Replay (equal) | | --- | ---: | ---: | ---: | ---: | ---: | | Nord Pool | 16.691254 | 17.529177 | 18.281273 | 30.880356 | 17.225040 | | PJM | 24.593975 | 25.150766 | 28.305193 | 55.806534 | 26.051670 | | Oxford-Man SPX | 0.424512 | 0.381271 | 0.468779 | 0.702657 | 0.405277 | | Financial text | 0.150242 | 0.143020 | 0.195967 | 0.176448 | 0.150242 | Generalist beats specialist on electricity; specialization improves SPX and text. Both SP-CL heads beat the continual ridge in these four streams. SPX equal replay (.405277) beats generalist (.424512), while specialist (.381271) remains better than both replay variants. Full financial history fits the replay budget, so its agreement with generalist verifies pooled equivalence, not a storage saving. Text's final effective-coordinate counts are 322/325 for generalist/specialist versus 5,485 for ridge at threshold $10^{-3}$; true support is unknown. **Historical retention and transfer.** Separate permanent holdouts evaluate the primary methods and weighted-replay control after each update (Tables 7–9 and 12). Both current-task and history-aware validation criteria are reported, with frozen selected rules. On all three market streams, generalist has lower final Overall and historical MSE than continual ridge, current-only, and both replay variants under both criteria. Specialist does not improve electricity Overall or mean task-time MSE. On Oxford-Man, current-task validation improves specialist's mean task-time MSE (.40672 versus .46366) but worsens final historical MSE (.52082 versus .43947); history-aware validation selects no correction. BWT must be read alongside absolute historical MSE because its baseline differs by method (Section 2; Proposition 2). These holdout splits differ from the prospective forecasts above. These four additional streams have cumulative $N_t>p$ during testing; the main simulation separately studies $N_T=800<p=1000$. Appendix I gives detailed results and the ridge-intercept sensitivity check.
AuthorsOpenReview ↗

Response to Reviewer ygNH (3/3)

Response to Reviewer ygNH, part 3 of 3.

The climate study in Appendix K examines score transmission in the cumulative high-dimensional regime (NT=658<p=1073N_T=658<p=1073). Its task-balanced, shared-penalty protocol provides a retrospective diagnostic, reported separately from the four forecasting streams above.

Weakness 4. Related sequential sparse-regression methods and experimental baselines.

We thank the reviewer for this helpful comment. Following your suggestions, we have added two baselines from the online sparse-regression literature and expanded the related-work discussion (Introduction, Section 6, Appendix I.1).

Relation to online sparse regression. The sequential sparse-regression literature covers incremental Lasso optimization and adaptive sparse estimation, with cumulative and time-weighted objectives (Garrigues and El Ghaoui, 2008; Angelosante et al., 2010; Babadi et al., 2010). Our setting differs in its evaluation target: in continual learning, earlier tasks remain evaluation targets as later tasks arrive. Downweighting older observations can improve tracking of a changing current target, and our evaluation also measures what that costs on earlier tasks. We have expanded the discussion of these three works accordingly.

Within this CL setting, we identify the historical-objective distortion caused by using the previous ℓ1\ell_1-penalized predictor as the next historical quadratic center, and correct it through score-preserving transmission (Proposition 1; Theorem 1).

Added baselines.

  • Cumulative-score Lasso retains the Gram matrix and cross-moments and solves the matched cumulative objective. By Theorem 1, both methods minimize the same cumulative objective; in this experiment, their maximum coefficient difference is 4.24×10−134.24\times10^{-13}. This baseline is therefore a check that the score-preserving update loses nothing relative to the cumulative online objective. The SP-CL specialist additionally addresses current-task adaptation beyond this matched cumulative objective through a sparse correction that leaves the historical state unchanged. In the Section 6 evaluation at h=12h=12, the specialist attains post-update FWT (Section 2) of 8.238.23, against 6.456.45 for the generalist (equivalently, for cumulative Lasso). Proposition 2 ties this difference to the average current-task gain and characterizes the accompanying cost on historical tasks.

  • Adaptive EW-Lasso discounts historical moments to track changing tasks. Its decay and penalty choices are selected on independent pilot streams, and the selected schedule is frozen for evaluation. In the task-shift experiment in Appendix I.1, its final current-task population excess risk is lower than the SP-CL generalist's (0.2480.248 versus 0.6950.695), while its average excess risk on preceding tasks is higher (0.4200.420 versus 0.2260.226). Evaluating both quantities makes the adaptation–retention trade-off explicit.

These baselines implement cumulative and exponentially weighted sparse-regression objectives; they are not reproductions of the specific homotopy or SPARLS solvers. Together with direct-center transmission, parameter-only continual ridge, current-only Lasso, and memory-matched experience replay with reservoir sampling and Lasso fitting, they evaluate historical information transmission, current-task adaptation, and sample retention. The Introduction, Section 6, and Appendix I.1 provide the related-work discussion, objectives, tuning protocols, and results.

The real-data holdout evaluation (Appendices I.4.1 and I.5.1) complements these simulations, reporting absolute historical risk, current-task performance, and BWT/FWT for both the generalist and the specialist.

Reviewer_fntwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continual sparse linear regression and proposes SP-CL. It stores the regularized estimate separately from the historical state, preserving the squared-loss score so that the cumulative estimator is equivalent to Lasso fitted to the pooled historical data. The paper uses Overall, BWT, and FWT to analyze the trade-off between generalist and specialist estimation, and evaluates the method through simulations and a chronological climate-prediction experiment.

Critical strengths
  1. Separating historical data information from the shrinkage introduced by regularization is a clear and useful idea. The equivalence to pooled Lasso gives the mechanism a precise mathematical interpretation, while the score and objective diagnostics directly test whether the historical state is preserved.

  2. The generalist-specialist analysis helps distinguish historical information loss from task conflict. The simulations illustrate this trade-off, and the climate experiment tests the cumulative estimator on a real chronological data stream.

Critical weaknesses
  1. Theorem 2(b) appears to omit an additional source of random variation caused by task heterogeneity. At the population pooled target, the empirical score contains the original noise term as well as 1Nt∑j≤t(Xj⊤Xj−njΣ)(βj⋆−wt⋆).\frac{1}{N_t}\sum_{j\le t}(X_j^\top X_j-n_j\Sigma)(\beta_j^\star-w_t^\star). A common covariance makes this term mean zero, but its scale still depends on the differences between task coefficients. For example, consider two equal-sized scalar Gaussian-design tasks with coefficients b+Ab+A and b−Ab-A. Their population pooled target is bb, but the score still has standard deviation A/nA/\sqrt{n} even with no observation noise.

Theorem 2(b) does not state a corresponding restriction on coefficient contrasts or adjust the regularization strength or convergence rate to account for them. Appendix C, p. 16, applies the standard Lasso bound directly. The authors therefore need to explain how this additional term is controlled and whether it changes the first-stage bound used in Appendix D.

Questions for authors
  1. How does Theorem 2(b) control the additional score term above? Does it require a restriction on the differences between task coefficients, or a regularization strength that depends on the degree of heterogeneity? If so, how would this affect the first-stage bound in Appendix D?

  2. How is the historical-loss budget εt\varepsilon_t in Appendix F chosen using only the available data? As task shift increases, how does the resulting rule compare with the generalist, specialist, and current-only estimators?

4 comments
AuthorsOpenReview ↗

Response to Reviewer fntw (1/2)

Response to Reviewer fntw, part 1 of 2.

We thank the reviewer for identifying the heterogeneity-score omission and for requesting an operational historical-budget rule. We corrected the theorem and its downstream arguments and added the budget rule and comparative experiments. Page and equation references below refer to the revised manuscript.

Weakness 1 and Q1. Heterogeneity in Theorem 2(b) and its consequences for Appendix D.

We agree that the original proof omitted control of the additional score caused by task heterogeneity. We have added an explicit bound on task dispersion to Theorem 2(b). Under this added condition, the original sample-size rate holds with constants depending on that bound. Appendix D now uses the corrected first-stage error and states the additional conditions needed for support recovery.

The omitted term. At the pooled population target wt⋆=∑j≤t(nj/Nt)βj⋆w_t^\star=\sum_{j\le t}(n_j/N_t)\beta_j^\star, the full score is

\qquad R_t=\frac1{N_t}\sum_{j\le t}(X_j^\top X_j-n_j\Sigma)(\beta_j^\star-w_t^\star).$$ The term $R_t$ combines empirical Gram fluctuations with differences between task coefficients. Common covariance makes its expectation zero, while its magnitude depends on those differences. Original Condition 1 alone therefore did not supply the required control. **Correction to Theorem 2(b).** We now assume $$\max_{j\le t}\|\beta_j^\star-w_t^\star\|_\Sigma\le B,$$ where $B$ is fixed independently of dimension, sample size, sparsity, and time. Under the stated design and noise conditions, Bernstein concentration gives, for $N_t\gtrsim L$, $$\|R_t\|_\infty\le CB\sqrt{L/N_t}, \qquad \|\mathcal S_t\|_\infty\le C(\sigma_\varepsilon+B)\sqrt{L/N_t}$$ on the joint high-probability event. Here $L=\log(C_{\mathrm u}pt(t+1)/\delta)$ for the simultaneous-in-time guarantee, and $C$ depends on the design-tail and covariance bounds. A sufficiently large multiplier in $\lambda_{w,t}\asymp(\sigma_\varepsilon+B)\sqrt{L/N_t}$ therefore controls the full score. With the theorem's sparsity and sample-size conditions, this yields squared estimation error of order $(\sigma_\varepsilon+B)^2s_{w,t}L/N_t$. Thus, the sample-size rate is preserved within the explicitly bounded-dispersion class. For growing dispersion, Appendix C also gives a sufficient penalty in terms of the weighted root-mean-square and maximum task contrasts. These dispersion measures are defined through population coefficients and enter the theoretical penalty calibration. See Theorem 2(b), pp. 7–8, and Appendix C, Eqs. (22)–(24), p. 18. Figure 9, p. 32, illustrates the additional score increasing with task dispersion. **Consequences for Appendix D.** The corrected first-stage bound is $$\|\widehat w_t-w_t^\star\|_1\le\frac{24s_{w,t}\lambda_{w,t}}{\kappa_l}.$$ A larger penalty needed to control heterogeneity therefore enlarges this error bound. Because the second stage starts from $\widehat w_t$, its score contains the perturbation $X_t^\top X_t(\widehat w_t-w_t^\star)/n_t$. Appendix D.2 controls this perturbation by imposing $s_{w,t}\lambda_{w,t}\le c_\Delta\lambda_{\Delta,t}$ and explicitly states the sample-size, support-compatibility, irrepresentability, and beta-min conditions for signed-support recovery. The argument uses deterministic inequalities on the joint concentration event, allowing dependence between $\widehat w_t$ and the current batch. See Appendix D.2, pp. 22–23, with Eqs. (35)–(38) on p. 22. **Related correction to Theorem 2(c).** We also provide a self-contained specialist-error proof in Appendix C, pp. 18–20. Its sufficient conditions explicitly include fixed $h\le h_0$, $h\sqrt{L/n_t}\le1$, and $n_t,N_{t-1}\gtrsim sL$. In particular, the current-batch requirement has been strengthened from $n_t\gtrsim L$ to $n_t\gtrsim sL$. The specialist rate is asserted under these revised conditions. Theorem 1's exact pooled equivalence and the metric decompositions follow from deterministic identities and hold independently of these statistical restrictions. **Q2. Choosing the historical-loss budget and comparing the resulting predictor.** **Who specifies the budget, and what the algorithm computes.** The user specifies $\varepsilon_t\ge0$ as the tolerated increase in average historical training loss. Given this tolerance, SP-CL uses its retained state and the current batch to compute the largest admissible correction fraction $\widehat\eta_t$. Thus $\varepsilon_t$ expresses the user's retention preference, while $\widehat\eta_t$ determines how much current-task adaptation is deployed.
AuthorsOpenReview ↗

Response to Reviewer fntw (2/2)

Response to Reviewer fntw, part 2 of 2.

The reference is the current generalist w^t\widehat w_t, which already incorporates the current batch. The budget limits the additional historical cost of specialization. We use half-squared loss, so tolerating an increase of cc in historical training MSE corresponds to εt=c/2\varepsilon_t=c/2. The tolerance or its schedule is specified before deployment.

Computing the adaptation from available information. For t≥2t\ge2, define β^t(η)=w^t+ηΔ^t,0≤η≤1.\widehat\beta_t(\eta)=\widehat w_t+\eta\widehat\Delta_t,\qquad 0\le\eta\le1. The endpoints are generalist and full specialist. Let L^<t\widehat L_{<t} denote the sample-weighted average historical half-squared training loss. Appendix F gives

=A_t\eta^2+B_t\eta,$$ $$A_t=\frac{\widehat\Delta_t^\top H_{t-1}\widehat\Delta_t}{2N_{t-1}},\qquad B_t=\frac{\widehat\Delta_t^\top H_{t-1}(\widehat w_t-u_{t-1})}{N_{t-1}}.$$ The retained summaries provide these coefficients exactly because the historical response-square constant cancels. The correction-step KKT conditions show that current training loss is nonincreasing along this path. Consequently, the constrained training-loss minimum is attained at $$\widehat\eta_t=\max\{\eta\in[0,1]:Q_t(\eta)\le\varepsilon_t\}.$$ Feasibility follows from $Q_t(0)=0$. A zero budget can allow positive adaptation when the correction reduces historical training loss. Appendix F, Eqs. (41)–(45), pp. 23–24, give the closed form and degenerate cases. If current validation loss is the selection criterion, minimize it over the feasible interval instead. The exact guarantee concerns aggregate historical training loss relative to the current generalist. Per-task population risks and BWT require separate evaluation. **Behavior as task shift increases.** In the common-covariance population model, at fixed current-to-total sample ratio and fixed absolute budget, the largest feasible population correction fraction is nonincreasing in the distance between the current coefficient and the historical population target (Appendix F, Eqs. (47)–(48), p. 24). Empirical fractions also reflect estimation noise. We compare generalist, specialist, current-only, and the budgeted predictor using 50 paired streams, five history lengths, eight shift levels, fixed Lasso penalties, and prespecified budgets $\{0,.01,.05,.10,.25,.50,1\}$. Across 14,000 budgeted outputs, the maximum positive empirical violation is $6.7\times10^{-16}$. At $m=16$ and $\varepsilon=.5$, mean $\widehat\eta$ decreases from $1.000$ to $.942$, $.612$ and $.411$ at $h=0,8,12,16$, respectively. At $m=16,h=16$, mean population excess risks are | Predictor | Current | Historical | | --- | ---: | ---: | | Generalist | 14.953 | 4.627 | | Specialist | 3.551 | 10.442 | | Budgeted, $\varepsilon=.5$ | 9.002 | 5.534 | | Current-only | 11.226 | 14.247 | The budgeted predictor improves current risk relative to the generalist while retaining more historical performance than the full specialist. At zero shift, the generalist has lower risk than the specialist ($.0976$ versus $.1156$). These risks use full squared-error units, whereas the budget bounds an empirical half-loss increment. The large-shift grid is a sensitivity study whose cells need not satisfy every sufficient condition of Theorem 2(c). See Appendix I.3, pp. 32–33, Figure 10.

The authors’ response has addressed my concern, and I will increase my score.

Thank you for your follow-up. We appreciate your careful reading and constructive feedback, which helped us strengthen the paper.

Reviewer_tEfDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies continual learning for high-dimensional sparse linear regression across sequential tasks. The authors show that centering a quadratic continual regularizer at the previous ℓ1\ell_1-penalized estimator accumulates historical penalty subgradients and can cause support collapse. They propose Score-Preserving Continual Lasso, which removes the current ℓ1\ell_1 subgradient via a pseudoinverse correction before transmitting the state, thereby recovering the exact offline pooled Lasso minimizer. Under task shift, a second-stage ℓ1\ell_1-penalized offset on the current batch adapts the cumulative estimator without modifying the transmitted state, achieving offline high-dimensional transfer learning rates. The authors also derive decompositions of Overall, BWT, and FWT metrics and evaluate the method on synthetic streams and a climate reanalysis dataset.

Critical strengths
  • Formulation of Recursive Shrinkage Bias: Proposition 1 and Examples 1–2 demonstrate how naively transmitting an ℓ1\ell_1-penalized parameter center accumulates historical subgradients and attenuates active features.

  • Rigorous Theoretical Guarantees and Metric Decompositions: The exact equivalence to offline pooled Lasso in Theorem 1, the anytime transfer rates in Theorem 2, and the Overall/BWT/FWT decompositions isolating task conflict in Proposition 2 are mathematically sound and clearly presented.

Critical weaknesses
  • Limited Practical Applicability and Weak Connection to Continual Learning Scenarios: The primary concern with this work is its application scope. Linear Lasso over fixed, pre-specified raw features is rarely used in modern continual learning, where catastrophic forgetting primarily arises in representation learning and deep neural networks with evolving feature spaces. While the authors motivate sparse continual learning by citing sparse computation in neural networks, high-dimensional linear feature selection on fixed covariates is fundamentally an online or streaming summary-statistic regression problem rather than a typical continual learning application. Beyond the low-task-count climate forecasting stream in Appendix G—where the second-stage task-shift specialist is not evaluated—it remains unclear what concrete continual learning applications require sequential linear Lasso.

  • Practical Memory Trade-offs in the High-Dimensional Regime: The motivation for Lasso is the high-dimensional regime where the feature dimension pp exceeds the sample size. However, transmitting the exact curvature matrix Ht∈Rp×pH_t \in \mathbb{R}^{p \times p} requires O(p2)O(p^2) storage. Whenever p>Ntp > N_t—such as in Figure 2 where p=1000p = 1000 and NT=800N_T = 800—maintaining the p×pp \times p matrix HtH_t requires more memory than simply storing all raw historical samples (X1:t,Y1:t)(X_{1:t}, Y_{1:t}) in O(Ntp)O(N_t p) memory, which limits the practical deployment of the algorithm when pp is large.

  • Positioning in Table 1 Against Theoretical Frameworks: Table 1 contrasts SP-CL directly against prior works such as Li et al. (2023), Zhao et al. (2024, 2025), and Karpel et al. (2026) via capability checkmarks. This comparison is not entirely fair: those prior works are theoretical analyses investigating the fundamental statistical behavior and forgetting trade-offs of standard ℓ2\ell_2 continual learning, whereas this paper proposes a new two-stage algorithm that transmits full O(p2)O(p^2) sufficient statistics. Presenting them side-by-side in a feature table conflates analytical frameworks with algorithmic design.

Questions for authors

Concrete Continual Learning Applications: Could the authors elaborate on realistic continual learning applications where high-dimensional linear Lasso on a fixed feature space is deployed, and how SP-CL could connect to continual representation learning or deep neural networks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tEfD (1/3)

Response to Reviewer tEfD, part 1 of 3.

Thank you for asking us to clarify the applications, historical state, and relationship to previous theory. We have expanded the discussion and added sequential evaluations with observed responses.

Weakness 1 and Q1. Concrete applications and the connection to deep continual learning

We study continual learning in a sparse linear regression setting, where tasks arrive sequentially, and regression coefficients may vary across tasks. The learner uses current-task data and retained historical information, without requiring access to the full historical dataset. Shared feature coordinates can represent measured covariates, a predetermined dictionary, or a frozen learned representation. Sparsity enables variable selection and makes the predictors' contributing variables explicit.

Why this is continual learning rather than only streaming regression. Under squared loss, the transmitted state is a summary statistic, and it is what makes the cumulative fit equal to the full-history Lasso (Theorem 1). What separates the problem from streaming Lasso with a common coefficient is (i) task-specific coefficients, which create a stability–adaptation trade-off; (ii) the propagation of earlier penalty subgradients when an ℓ1\ell_1-penalized predictor is directly reused as the next historical quadratic center (Proposition 1); and (iii) evaluation by Overall, BWT and FWT, with decompositions that distinguish task conflict, estimation error, and the historical cost of current-task specialization (Proposition 2). This extends the existing theory of continual linear regression (Li et al., 2023; Zhao et al., 2024; Karpel et al., 2026) from ℓ2\ell_2 to sparse estimation.

Concrete applications. Appendix H discusses the application settings, and Appendices I.4–I.5 report four real-world forecasting streams:

  • Nord Pool and PJM electricity prices: 24 hourly prediction heads use a common 247-coordinate dictionary of lagged prices, exogenous forecasts, and weekday indicators, with monthly updates. The public benchmark supplies observed prices and forecasts. Data source
  • Oxford-Man realized variance: daily SPX log-realized variance is predicted from 396 coordinates: 22 lagged volatility measurements for each of 18 markets, with monthly updates. Replication source
  • Financial-report risk: annual Kogan MD&A cohorts use a frozen 10,000-word binary dictionary plus past volatility (p=10,001p=10{,}001) to predict subsequent twelve-month log volatility. Reports enter training only after their response windows mature. Original corpus

Within each stream, methods share the inputs and forecast targets. Preprocessing is fixed on earlier reference data, parameters use designated development data, and predictions precede label-based updates. Appendices I.4–I.5 specify initialization, dates and candidate rules. The following are sample-weighted prospective response MSEs:

StreamGeneralistSpecialistContinual ridgeCurrent-only LassoReplay (equal)
Nord Pool16.69125417.52917718.28127330.88035617.225040
PJM24.59397525.15076628.30519355.80653426.051670
Oxford-Man SPX0.4245120.3812710.4687790.7026570.405277
Financial text0.1502420.1430200.1959670.1764480.150242

Replay gives every current and cached sample equal weight; independently tuned historical-weighted replay remains a supplementary control. Generalist has lower MSE than continual ridge in all four streams. Specialist improves on generalist for SPX and financial text; on SPX, its correction adjusts only the intercept. Equal replay (.405277) beats SPX generalist (.424512), while specialist (.381271) is best among these comparators. Financial replay matches generalist because all eligible sparse historical rows fit its budget. True supports are unknown; simulation separately examines NT<pN_T<p.

AuthorsOpenReview ↗

Response to Reviewer tEfD (2/3)

Response to Reviewer tEfD, part 2 of 3.

Historical retention and transfer. Separate permanent holdouts never enter supervised fitting. Appendices I.4.1 and I.5.1 report Overall, historical/current MSE, mean task-time MSE, and BWT/FWT under both current-task and history-aware validation. Across all three market streams, generalist has lower final Overall and historical MSE than continual ridge, current-only, and both replay variants under both criteria. These results use different splits from the prospective table. BWT is interpreted with absolute historical risk because its baseline differs across methods; FWT compares with current-only Lasso after current-task training (Section 2; Proposition 2).

The climate study in Appendix K examines score transmission in the cumulative high-dimensional regime (NT=658<p=1073N_T=658<p=1073). Its task-balanced, shared-penalty protocol provides a retrospective diagnostic, reported separately from the four forecasting streams above.

Connection to deep continual learning. LG-CBM and CI-CBM use sparsity-promoting regularization on linear concept prediction heads, motivating sparse interpretable heads in CL. SP-CL directly covers a squared-loss regression readout y=ϕ(z)⊤βt⋆+εy=\phi(z)^\top\beta_t^\star+\varepsilon with the complete feature map fixed, including both backbone and concept map when applicable. Evolving concept maps and classification losses require further analysis (Introduction; Appendix H).

Weakness 2. Historical state, storage, and the high-dimensional regime

SP-CL's O(p2)O(p^2) state preserves the entire historical empirical squared loss up to a parameter-independent constant, independently of stream length. It supports sparse estimation when Nt<pN_t<p, without implying that the matrix is smaller than those raw samples. Appendix G separately counts historical state, deployed heads, metadata, and temporary workspace.

The long-stream comparison uses p=240p=240, nt=40n_t=40, T=30T=30, hence NT=1200N_T=1200. SP-CL, including both heads, uses 468,723 bytes; the largest feasible 240-row replay buffer uses 467,177 bytes and first subsamples history when fitting task 8. Over 20 paired shared-coefficient streams, final excess risk is 0.033±0.0020.033\pm0.002 for generalist versus 0.173±0.0140.173\pm0.014 for equal replay (mean ±\pm SE; Appendix I.1). Market buffers hold 157 daily rows shared by 24 heads for electricity and 200 rows for SPX. All financial-text history fits; matched penalties then give the same pooled-Lasso objective.

Exact structured compression. Under the empirical commutativity assumption in Zhao et al. (2024), Assumption 4.2, Appendix A.4 gives an exact O(p)O(p) evolving state conditional on an available fixed common eigenbasis UU. It retains diagonal curvature and carrier coordinates, preserving pooled-Lasso equivalence with the ℓ1\ell_1 penalty in the original coordinates. Total storage is O(p)O(p) when UU is implicit or compact; a dense stored UU costs O(p2)O(p^2).

Supplementary local-score variant. Appendix J constructs an O(p)O(p)-state scalar-curvature variant without a shared basis. Its KKT-based center matches the current smooth surrogate's gradient at the sparse solution, preserving that minimizer at the same penalty while approximating historical curvature. The appendix separately reports this local property, prediction/sparsity comparisons, and the relationship to generalized implicit FTRL. Its prospective MSE is below the same-grid continual-ridge reference on all four real streams.

Weakness 3. Theoretical contribution and comparability with previous work

Following your suggestion, we have replaced the checkmarks with a compact comparison of task coefficients, historical-state size, current-task adaptation, and main theoretical result. The state-size and adaptation columns describe each update mechanism, so SP-CL's O(p2)O(p^2) state is visible next to the others, and the final column states the theoretical contribution. Table notes specify that the Zhao et al. (2024) row refers to their practical matrix-weighted version and that SP-CL's adaptation guarantee requires controlled task differences; Appendix G.1 gives the underlying assumptions and update rules.

AuthorsOpenReview ↗

Response to Reviewer tEfD (3/3)

Response to Reviewer tEfD, part 3 of 3.

We keep these works in the table because each of them, like ours, analyzes a specific update rule for continual linear regression and asks how regularization and retained historical information determine statistical performance. Our paper is also primarily theoretical in this setting: it characterizes shrinkage accumulation and its consequences for signal retention, proves pooled-Lasso equivalence, and gives sparse recovery guarantees under explicit conditions. The revised table distinguishes their analyses of regularization and prediction/transfer risks from our analysis of penalty-induced transmission bias and sparse recovery.

Zhao et al. (2024) provide a direct comparison at the level of historical-loss representation. Their practical generalized quadratic update (Section 4.3, Eq. (7)) uses the cumulative historical Gram matrix, scaled by the current sample size, as its regularization weight. A general dense implementation retains a p×pp\times p matrix together with a parameter estimate; SP-CL's O(p2)O(p^2) historical state is of the same order as that of this practical matrix-weighted implementation. The empirical commutativity assumption in their analysis also motivates our exact shared-eigenbasis representation in Appendix A.4, and Appendix G accounts explicitly for the availability and storage cost of that basis.

What SP-CL adds. A quadratic historical penalty reproduces the historical data loss when its center satisfies Kv=gKv=g, where KK and gg are the historical Gram and cross-moment statistics. An ℓ1\ell_1-penalized predictor generally violates this identity through its penalty subgradient. Consequently, even retaining the exact historical curvature does not suffice when that predictor is directly reused as the next center. We characterize the resulting accumulation, establish its consequences for sparse signals, and construct a separate score-preserving carrier that restores the pooled-Lasso objective while allowing current-task specialization.

Zhao et al. (2025) provide a complementary high-dimensional theoretical comparison. They analyze a scalar continual-ridge update that transmits only the previous parameter vector, and derive asymptotic prediction and transfer risks. SP-CL studies sparse estimation and variable selection with a richer historical state. The revised table lists these differences in retained state and main theoretical results as explicit columns. Together, the two Zhao papers connect to both the historical-state mechanism and the high-dimensional statistical questions studied here, which is why they sit alongside SP-CL in Table 1.

Li et al. (2023) and Karpel et al. (2026) are kept on the same basis: both analyze ℓ2\ell_2-regularized continual linear regression, and their rows now state their update and result in the same terms.

Dear Reviewer tEfD,

As the discussion period is approaching its end, we would like to kindly follow up on our response. We sincerely appreciate your thoughtful feedback and hope that our rebuttal and revisions have addressed your concerns.

If any concerns remain, we would be happy to provide further clarification.

Thank you again for your time and consideration.

Best regards, Authors of Submission 46828

Comments on the paper

3 comments
~Itay_Evron1OpenReview ↗

A possible connection

Dear authors,
Thank you for the interesting manuscript.

We noticed a connection between Eq. (6) and Theorem 1 in your manuscript and Proposition 5.5 of our ICML 23 paper (see proof on Page 43). Specifically, in linear regression, a parameter-space continual regularizer, weighted by the Gram matrix of previous tasks 1,…,t−11,\dots,t-1, is equivalent to minimizing tasks 1,…,t1,\dots,t jointly. Of course, your manuscript additionally includes an ℓ1\ell_1 regularization term, which our paper did not consider.

We would appreciate it if you could discuss this connection in the next version.

Good luck with your submission!

AuthorsOpenReview ↗

Overview of revisions

Overview of revisions

We thank the reviewers for their detailed feedback. The revision retains SP-CL's central construction: separate the sparse predictor from the historical score carrier, recover the pooled-Lasso objective, and fit a current-task correction without overwriting that historical state. The changes below clarify its statistical conditions, scope, and empirical evidence. Detailed storage accounting and support diagnostics are in Appendices G and B.4. Page references use the clean manuscript.

  1. Statistical statements and proofs. We explicitly separate estimation conditions from signed-support conditions, supply the primal–dual witness argument and constants, and account for task-dispersion noise in Theorem 2(b). The specialist analysis states the conditions needed for its two-stage bound. The historical-loss budget is defined relative to the current generalist and used to choose a feasible correction. Locations: Corollary 1 and Theorem 2; Appendices C–F, pp. 18–24.

  2. Positioning and storage. Table 1 now compares assumptions, retained state, current-task adaptation, and main results; Table 2 compares statistical rates without an implementation-dependent raw-history column. The discussion distinguishes SP-CL's transmission mechanism from incremental sparse-regression solvers and other CL memory mechanisms. We also discuss the Gram-weighted connection in Evron et al. (2023), and explicitly cite the classical Lasso argument and the sources of reservoir replay. Storage accounting separates persistent state from workspace; exact compression in a shared eigenbasis counts the basis explicitly. Locations: Introduction/Table 1, pp. 1–3; Appendix A.4, pp. 13–14; G–H, pp. 24–27.

  3. Comparisons and practical applications. We add memory-matched replay, cumulative-score and adaptive sparse-regression references, and four observed-response streams: Nord Pool, PJM, Oxford-Man SPX, and financial-report text. Sample-equal replay is the primary replay comparator; independently tuned historical-weighted replay is supplementary. Separate permanent holdouts evaluate Overall, historical/current error, and BWT/FWT. Prospective forecasts and permanent-holdout evaluations are reported separately. Common metric and selection definitions are collected at the start of Appendix I; numerical checks are consolidated in I.6. Section 2 clarifies that FWT is measured after current-task training relative to current-only Lasso. The parameter-only continual-ridge comparison uses a fixed, validation-selected two-parameter rule. Appendix K, pp. 45–49, reports the climate experiment with a clarified retrospective tuning protocol. Locations: Appendix B.4, pp. 16–17; I.1, pp. 28–31; I.4–I.6, pp. 33–43.

  4. Calibration and interpretation. Figure 2 uses Wainwright's support-recovery calibration with known generating sparsity and noise, without post-selection OLS; the full selection diagnostics are in Appendix B.4. Real-data penalties instead use observed training scores and earlier development data. Rate and budget studies clarify what each experiment establishes. Locations: Section 6/Figure 2, p. 9; Appendix B.4, pp. 16–17; I.2–I.3, pp. 31–33.

  5. Optional compact-state extension. Appendix J, pp. 44–45, separately presents a scalar-curvature construction and its local KKT matching property, with exploratory comparisons and its relationship to implicit FTRL. This extension is distinct from the exact historical-loss construction used by the main theorems.

Reading the highlighted copy. Blue text identifies substantive main-text revisions addressing the reviews. Updated main-text tables and figures are marked by blue captions, with their contents in normal colors. In the appendices, blue headings identify new or substantively revised sections; their contents remain black for readability. Black text within a marked appendix section does not imply unchanged content. Minor editorial changes are not individually marked; deleted passages are omitted.

The anonymous supplementary package contains code, configurations, selected rules, source information, and saved results, with portable verification commands.

The AI use statement has been updated to describe assistance with literature discovery, mathematical formulation and proofs, experimental design and code, analysis, and drafting/editing. The authors retain responsibility for the work.

AuthorsOpenReview ↗

Connection to Proposition 5.5

Thank you for pointing out this connection. Proposition 5.5 of your ICML 2023 paper establishes the connection between a Gram-weighted parameter-space regularizer and joint fitting in jointly realizable linear regression. We have added this discussion alongside Zhao et al. (2024) in Section 3.1 and Appendix G.1. Our sparse extension addresses the additional penalty contributions introduced when an ℓ1\ell_1-penalized estimate is reused as the historical center.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.