acceptodds
Withdrawn from ICLR 2027

Width or Interaction Fidelity? Allocating a Fixed State Budget in Analytic Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Analytic continual learners fit a closed-form ridge readout on frozen features and carry only its sufficient statistics, the continuation state, across tasks. Expanding features into many random coordinates improves accuracy, but the Gram matrix in this state grows quadratically with width. We ask how a fixed continuation-state budget should be split between representation width and the fidelity of stored feature interactions. Because the feature-label statistic and the Gram diagonal grow only linearly with width, we keep them exact and compress only the interactions with **Diagonal-Preserving Spectral Ridge (DPSR)**, which merges per-task spectra into a rank-bounded factor and solves a full-width readout in closed form. On three class-incremental benchmarks with ViT-B/16 features, DPSR stays within points of a full-Gram readout of the same expansion in stage-average accuracy while storing less continuation state. At equal budgets of MiB with penalties chosen by generalized cross-validation (GCV), trading interaction rank for width beats exact ridge on a narrower expansion in of configurations by points (three seeds, paired intervals above zero) and loses points where too little rank remains. With validation-and-refit selection, gains are smaller, clearest on CIFAR-100 and shared by other corrected sketches, and a penalty chosen before truncation can collapse accuracy after refitting. Under GCV, width thus outweighs interaction fidelity down to a dataset-dependent rank; the savings concern persistent state, not training time or peak memory.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class-incremental learning, analytic continual learning, memory-efficient learning, representation expansion

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ve9oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how a fixed continuation state budget (memory) in class incremental learning should be split between representation width and interaction fidelity. The proposed method DPSR keeps inexpensive part (feature strengths and feature-class association) accurately and stores an approximation of feature-feature relations. On three benchmarks, DPSR achieves almost the same accuracy as full gram readout with 88-89% less continuation state.

Critical strengths
  1. The problem is significant and clearly stated The paper pose a real problem in the domain of analytic continual learning i.e the tradeoff between representation quality and storage accuracy. The research question is specific and experimentally testable.

  2. Gains from using the DPSR method are significant DPSR gives accuracy of full Gram model while reducing memory by 88-89%.

  3. There is sufficient experimental rigor to support the hypotheses -- Comparison done against multiple baselines LR, FD, RFD, FD + Diag etc. Evaluates on multiple datasets, encoders, memory budgets, ranks, task sequences etc. Reports results on multiple seeds. Rigorous analysis on failure conditions.

  4. The paper provides theoretical backing of its design choices and bounds on prediction errors.

Critical weaknesses
  1. The method's novelty is limited
  2. A budget-matched span-restricted learner using a LoRanPAC could have been a baseline++ and such results could really show the significance of DPSR.
  3. I found the readability slightly jarring; especially the way some paragraphs are starting (Fix a continuation budget β, Three observations motivate the allocation etc)
Questions for authors
  1. Do you have information regarding how much the individual components contribute to the DPSR accuracy (especially the diagonal correction)?

  2. Appendix E.4 mentions narrow exact widths (packed storage of upper triangle). Do you have the results for it? (Table 1 and Table 16)

  3. The central allocation claim is tested mainly on three datasets. Could you extend the matched-budget experiments to ImageNet-R/A and at least one omitted benchmark from Table 1 of LoRanPAC (https://arxiv.org/pdf/2410.00645), particularly StanfordCars?

Reviewer_YrgWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary
  • This paper investigates the trade-off between representation width and interaction fidelity in analytic continual learning. The authors presents Diagonal-Preserving Spectral Ridge (DPSR), a method that retains exact per-coordinate statistics (the feature-label statistic) while compressing off-diagonal feature interactions into a rank-bounded factor
Critical strengths
  • The problem formulation is compelling. Framing the continual learning memory constraint as a direct trade-off between representation width and interaction fidelity is interesting
Critical weaknesses
  • The presentation requires significant rework. The methodology is difficult to follow and is fill with heavy notation without sufficient high-level, intuitive explanations. As a reader, it is very difficult to grasp how the proposed DPSR method actually works step-by-step
  • Because the narrative and methodology are opaque, the ultimate goal and the practical takeaways of the experiments are lost. It is difficult to evaluate the validity of the results when the core mechanics of the proposed solution cannot be easily understood
Questions for authors
  • Could you provide a high-level, intuitive explanation (maybe a simplified step-by-step walkthrough) of how the DPSR method processes a single task, completely stripped of the heavy mathematical notation?
Reviewer_NaDtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate a fixed continuation-state budget between representation width and interaction fidelity in analytic continual learning. The authors propose DPSR, which keeps the feature-label statistic and Gram diagonal exact, compresses off-diagonal interactions into a rank-bounded factor via two-stage SVD truncation, and solves a full-width ridge readout via the Woodbury identity. Experiments on CIFAR-100, CUB-200, and VTAB-50 with frozen ViT-B/16 features evaluate DPSR against narrow exact ridge and streaming matrix sketches under GCV and validation-and-refit protocols.

Critical strengths
  1. The trade-off between expansion width and Gram approximation under a fixed memory budget is clearly formulated, and the authors candidly report negative results under validation-and-refit as well as the first-task collapse on CUB-200.
Critical weaknesses
  1. The main conclusion that width outweighs interaction fidelity is an artifact of penalty selection and an unfair baseline, and DPSR shows no real advantage over simpler sketches.

    • Protocol sensitivity: The headline "17 of 18" win rate relies on task-local GCV, under which the narrow exact baseline degrades by 0.98-1.92 points on CUB-200 compared to validation-and-refit. When evaluated under validation-and-refit (Table 1), DPSR actually underperforms narrow exact ridge on CUB-200 (-0.579 at 20 MiB and -2.933 at 43 MiB), while on CIFAR-100 its modest gains (+0.3 to +0.6) are matched by existing sketches (RFD and FD+diag).

    • Equivalence to a penalty shift: Proposition 1 shows that isotropic completion is algebraically equivalent to low-rank ridge (LR) with a shifted penalty λ+δˉ\lambda + \bar{\delta}. Consistently, when using a shared 40-value penalty grid (Section 4.2), all 12 paired confidence intervals between DPSR and LR contain zero. Thus, exact diagonal completion mainly acts as an implicit regularization shift rather than recovering lost interaction geometry.

    • Unfair memory accounting: The "matched-budget" comparison stores the symmetric exact Gram as a dense m×mm \times m matrix rather than in packed upper-triangular form, which would allow the narrow exact baseline to be 1.4×1.4\times wider at the same budget. Moreover, at the nominal 43 MiB budget, wide methods actually consume ~76 MiB of total state versus ~54 MiB for narrow exact ridge (Table 27).

  2. The full-width Woodbury readout is numerically unstable after spectral truncation, and diagonal completion fails to regularize the orthogonal subspace.

    • Amplification outside the retained span: Unlike span-restricted readouts (e.g., PCR or LoRanPAC), Eq. (4) applies the Woodbury inverse to the full label statistic SS, including its component S⊥S_{\perp} orthogonal to the retained rank-rr factor. As shown in Lemma 2, truncation removes the spectral curvature outside the retained span, causing S⊥S_{\perp} to be amplified by a factor of (σ2+λ)/λ(\sigma^2 + \lambda)/\lambda when λ\lambda is small.

    • Collapse under validation-and-refit: Because the validation split on the first CUB-200 task has fewer samples than the rank capacity (nfit<rn_{\text{fit}} < r), the validation state is exact (S⊥=0S_{\perp} = 0) and selects a small penalty (λ=102\lambda = 10^2). After refitting on the full task (nrefit>rn_{\text{refit}} > r), truncation introduces a tiny residual S⊥S_{\perp} (norm fraction <0.034%< 0.034\%) that dominates the classifier and causes accuracy to collapse from ~98% to 28.9-55.0% (Table 4). Notably, RFD and FD+diag do not collapse at the same λ\lambda, demonstrating that DPSR's diagonal completion fails to provide stable curvature outside the retained span.

    • Contradiction with known conditioning properties: Prior work [1], which this paper overlooks, has shown that naively scaling up random projection width induces ill-conditioned feature matrices that destabilize analytic ridge updates, motivating compact, data-guided projections instead. Advocating wider random expansions while truncating the Gram spectrum exacerbates this ill-conditioning without providing a penalty selection rule that remains valid after truncation.

[1] Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

Questions for authors
  1. Since a shared 40-value penalty grid yields no statistically significant difference between DPSR and plain low-rank ridge (all 12 paired intervals contain zero), what is the concrete algorithmic benefit of exact diagonal completion beyond shifting the effective regularization penalty?

  2. In Table 4, RFD and FD+diag remain stable (96.7-98.7%) at the exact same penalty (λ=102\lambda = 10^2) where DPSR collapses to 28.9-55.0%. Why does DPSR's diagonal completion fail to regularize S⊥S_{\perp} when RFD and FD+diag succeed?

  3. How does the width-versus-rank comparison change if (i) the narrow exact baseline stores only the upper triangle of the symmetric Gram matrix (1.4×1.4\times wider at the same budget), and (ii) budgets are matched by total state rather than continuation state alone?

Reviewer_TLfTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between representation width and interaction fidelity under a fixed continuation-state budget in analytic continual learning. The authors propose Diagonal-Preserving Spectral Ridge (DPSR), which retains exact coordinate energies and feature-label statistics while compressing the Gram matrix through sequential spectral truncation. A diagonal correction and Woodbury solve produce a full-width classifier. Experiments show substantial continuation-state savings relative to the full Gram, while the advantage over narrower exact readouts depends on the budget and penalty selection.

Critical strengths
  • The studied resource-allocation problem is meaningful. Separating representation width from the cost of storing feature interactions provides a useful perspective on analytic continual learning.

  • The experimental analysis is extensive and transparent. The paper includes several sketching baselines, additional encoders and task streams, and detailed resource measurements. The investigation of the validation-to-refit collapse is informative, and the authors clearly acknowledge negative results.

Critical weaknesses
  • The main matched-budget comparison is incomplete. Storing only the Gram matrix's upper triangle would increase the exact baseline's width from 3,307 to 4,647 on CIFAR-100 at 43 MiB (Appendix E.4). This stronger baseline should be evaluated to establish the claimed width advantage.

  • The allocation benefit is sensitive to penalty selection. Under validation and refit, DPSR has lower mean accuracy than narrow exact ridge at all three CUB-200 budgets (Table 1). The authors should repeat the width-rank study with penalty selection that accounts for the deployed compressed state.

  • The methodological novelty is limited. DPSR combines established spectral updates, diagonal correction and the Woodbury identity. With finer penalty tuning, all twelve paired intervals against LR or isotropic completion include zero (Table 19), leaving the additional benefit of exact diagonal bookkeeping unclear.

  • The practical benefit is narrow. At 43 MiB on CIFAR-100, DPSR requires more total state (76.10 vs. 53.93 MiB) and fitting time (12.8 vs. 0.9 seconds per task) than narrow exact ridge (Tables 25 and 27). A total-state-matched comparison is needed to establish practical value.

  • The failure analysis remains incomplete. DPSR's first-task accuracy drops below 56% in two CUB-200 seeds, while RFD and FD+diag remain above 96% at the same penalty (Appendix F.7). The authors should explain this difference and evaluate a remedy for the refit collapse.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.