acceptodds
Under review as a conference paper at ICLR 2027

CARGO: Carrying Curvature Across Tasks for Continual Pruning of Large Language Models

Abstract

Continual pruning has recently emerged as a promising direction that progressively updates pruned LLMs as new tasks arrive, while retaining knowledge from previously encountered tasks. Existing methods focus primarily on which weights to prune, leaving unexplored whether the remaining weights can be actively adapted to compensate for the information lost through continual pruning. We find that combining accumulated-importance-based weight pruning with existing single-task weight reconstruction strategies can improve final performance but leads to larger forgetting, calling for a dedicated design for continual pruning. To this end, we first formulate a progressive weight reconstruction problem for continual pruning to jointly minimize the layer-wise pruning error of the pruned model over all tasks seen so far. Under fixed layer inputs and an invertible sum of task Hessians, we show that optimal compensation for a single-weight deletion depends on historical tasks only through their accumulated Hessian, enabling efficient bluereconstruction without retaining past calibration data. Building on this insight, we next propose CARGO (**C**urvature **A**ccumulation and **R**egime-**G**ated C**o**nsolidation), a training-free continual pruning framework that carries curvature information across tasks to guide both weight selection and reconstruction. To robustly consolidate curvature over time, CARGO further incorporates spectral cleaning to suppress noisy low-energy curvature components and a task-correlation-aware directional gate to selectively retain historical curvature. Experiments across three model families and two task streams show that CARGO substantially improves continual pruning performance over the strongest baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Continual Pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TKHLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CARGO starts from a cross-task layerwise reconstruction objective and shows that, under fixed historical layer inputs and single-weight deletion, historical data can be represented by an accumulated second-moment matrix. In practice, it spectrally truncates and normalizes current-task curvature, measures current support along historical eigenvectors, gates or decays historical eigenvalues, and then performs SparseGPT-style sequential pruning and reconstruction using a damped surrogate curvature matrix. The experiments report high continual-pruning ACC in most settings and generally improved BWT.

Critical strengths

The problem setting and methodological direction are valuable. Compared with accumulating only per-weight scores, using cross-task input second moments to guide compensation is a sensible advance for continual pruning. The single-weight OBS result is also concise and complete.

Critical weaknesses
  1. The theory does not cover the implemented algorithm. The proposition concerns a true accumulated matrix and one-coordinate deletion, while CARGO uses a processed surrogate matrix and sequentially deletes many weights. The paper neither proves that this surrogate corresponds to a well-defined cross-task objective nor shows that the result under the final fixed mask is close to the joint optimum.
  2. Basis stability and the experimental protocol have serious flaws. Thresholding individual eigenvectors is not evidently invariant to basis rotations within a repeated-eigenvalue subspace. Moreover, multiple tables report different results for identical settings, and each model, task stream, and sparsity level uses different hyperparameters without explaining whether they were selected on an independent validation set.
Questions for authors
  1. Are Eqs. (7)–(10) invariant to orthogonal changes of basis within repeated-eigenvalue subspaces? If not, please use subspace-level gating and report stability under nearly repeated eigenvalues.
  2. Were the parameters in Table 7 selected using an independent validation set? Does CARGO retain its advantage with one global hyperparameter setting, multiple random task orders, and a baseline that jointly solves for the weights under the fixed final mask?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TKHL Part1

Dear reviewer TKHL,

We sincerely thank you for the encouraging assessment and constructive suggestions. We conducted extensive experiments and address your concerns point by point below.

Weakness 1: The proposition does not cover the processed curvature or multiple-weight deletion, and the relationship between the implemented solution and the joint optimum is unclear.

Thank you for suggesting a clearer connection between the theory and the implemented algorithm. CARGO uses SparseGPT’s column-wise pruning and weight reconstruction procedure; our design provides the cross-task curvature used by that procedure.

Let Ck=ZkZk⊤C_k=Z_kZ_k^\top denote the task-kk curvature contribution, incorporating the prescribed cleaning and normalization. Let BjB_j scale each historical eigendirection by the square root of its gate value, so that Mt=BtMt−1Bt⊤+CtM_t=B_tM_{t-1}B_t^\top+C_t. Defining At,k=BtBt−1⋯Bk+1A_{t,k}=B_tB_{t-1}\cdots B_{k+1} and At,t=IA_{t,t}=I, expanding the recursion gives:

Mt=∑k=1tAt,kCkAt,k⊤.M_t=\sum_{k=1}^{t}A_{t,k}C_kA_{t,k}^{\top}.

Lt(δ)=12∑k=1t∥Zk⊤At,k⊤δ∥22=12δ⊤Mtδ.\mathcal{L}_t(\delta)=\frac{1}{2}\sum_{k=1}^{t} \lVert Z_k^{\top}A_{t,k}^{\top}\delta\rVert_2^2 =\frac{1}{2}\delta^{\top}M_t\delta.

Thus, with the collected layer inputs and curvature states fixed, MtM_t is exactly the Hessian of this cross-task reconstruction objective. The single-deletion OBS score and reconstruction update follow from this objective; sequential multi-weight reconstruction uses SparseGPT’s scalable approximation.

We further fixed the same final CARGO mask on Llama-2-7B / TRACE at 50% sparsity and jointly reconstructed all retained weights in each row using the same CARGO curvature, with damping. The original column-wise and joint reconstructions achieve final ACC of 37.24 and 36.53, respectively. This shows that our column-wise reconstruction achieves the same level of performance as Joint reconstruction does. These results show that our column-wise reconstruction achieves performance comparable to joint reconstruction.

Reconstruction methodACC
CARGO column-wise reconstruction37.24
Joint reconstruction using CARGO curvature36.53
AuthorsOpenReview ↗

Response to Reviewer TKHL Part2

Q1&Weakness Part1: Are the equations invariant to orthogonal changes of basis within repeated-eigenvalue subspaces? How are nearly repeated eigenvalues handled?

Thank you so much for identifying the basis ambiguity under repeated eigenvalues and suggesting subspace-level gating. We have implemented this correction and added rotation-invariance checks, perturbation tests for nearly repeated eigenvalues, and full task-stream experiments.

Subspace-level definition. The original eigenvector-level gate is not generally invariant to orthogonal basis changes within repeated positive-eigenvalue eigenspaces. We replace it with one shared gate per exact eigenspace. For t≥2t\geq2, write Mt−1=∑kλkPkM_{t-1}=\sum_k\lambda_kP_k, where λk\lambda_k are the distinct positive eigenvalues, Pk=UkUk⊤P_k=U_kU_k^\top are their orthogonal spectral projectors, and rk=rank⁡(Pk)r_k=\operatorname{rank}(P_k). With CtC_t denoting the cleaned and normalized current-task curvature, define:

m_k=\frac{c_k}{c_k+\lambda_k/(t-1)+\epsilon},$$ $$g_k=\begin{cases}1,&m_k\geq\alpha,\\\beta,&m_k<\alpha,\end{cases} \qquad M_t=\sum_k g_k\lambda_kP_k+C_t.$$ Replacing any subspace basis $U_k$ with $U_kQ_k$, for orthogonal $Q_k$, leaves $P_k$ unchanged and therefore leaves $c_k$, $g_k$, and $M_t$ unchanged. When all positive eigenvalues are distinct, $r_k=1$ and this definition reduces to the original direction-wise update. Numerically, we group only positive eigenvalues that are identical in their computed floating-point values; strictly distinct, nearly repeated eigenvalues remain separate. The correction retains the original $\alpha$, $\beta$, numerical stabilizer, cleaning procedure, and reconstruction solver, introducing no new hyperparameter. **Rotation and update checks.** On small matrices containing one or multiple repeated-eigenvalue blocks, we tested 20 orthogonal basis changes per block, including sign flips and column permutations. The maximum relative Frobenius difference between subspace updates was below $4.1\times10^{-16}$ in float64 and $1.0\times10^{-7}$ in float32, with no gate changes. The two rules were bit-identical in the simple-spectrum tests. In a full subspace-gate run on Llama-2-7B / TRACE at 50% sparsity with $(\beta,\alpha)=(0.9,0.3)$, we also measured the difference between the two updates on identical inputs and the same eigendecomposition: $\Delta=\lVert F_{\mathrm{sub}}-F_{\mathrm{orig}}\rVert_F/\lVert F_{\mathrm{orig}}\rVert_F$ . The rules differed at 17 of 1,568 module-update steps, with mean $\Delta=1.4\times10^{-8}$ and maximum $\Delta=2.1\times10^{-6}$. These measurements quantify local update differences on the same inputs. **Perturbation stability for nearly repeated eigenvalues.** We selected 18 saved $4096\times4096$ module states from a real subspace-gate run, covering different tasks, attention/MLP modules, and states with relatively high mixed-cluster curvature mass. We added PSD perturbations $E=\delta\|X\|_Fvv^\top$ to either the historical curvature $M_{t-1}$ or current curvature $C_t$, testing 10 random unit directions per magnitude. The table reports the relative Frobenius change in the subspace update when perturbing historical curvature in float32: | Perturbation magnitude | Median output change | Maximum output change | |---|---:|---:| | $10^{-8}$ | $2.1\times10^{-5}$ | $6.1\times10^{-5}$ | | $10^{-7}$ | $2.3\times10^{-5}$ | $6.5\times10^{-5}$ | | $10^{-6}$ | $3.0\times10^{-5}$ | $7.6\times10^{-5}$ | | $10^{-5}$ | $4.1\times10^{-5}$ | $1.2\times10^{-4}$ The two rules had the same perturbation response on these saved states, and perturbing the current-task curvature Ct also produced small output changes. The exact-subspace correction removes basis ambiguity for exact repeats. Strictly distinct, nearly repeated eigenvalues retain direction-wise decisions, and both rules can undergo gate switches in synthetic tests where support scores cross the threshold. The table reports measured behavior for the tested real states and perturbation magnitudes. **Full task-stream performance.** We ran both rules on Llama-2-7B / TRACE at 50% sparsity with the same gate parameters, carrying the subspace-rule curvature into subsequent pruning and reconstruction steps. The individual-run results are: |Gate rule | ACC | BWT | |---|---:|---:| | Original direction-wise gate | 38.59 | +0.25 | | Subspace-level gate | 37.78 | −0.99 | The final ACC values differ by 0.81 percentage points. We report ACC and BWT for both rules to show full-stream performance when the subspace rule is actually applied. We appreciate your suggestion, which helped remove the mathematical ambiguity for exact repeats and motivated stability checks tied to the implemented update.
AuthorsOpenReview ↗

Response to Reviewer TKHL Part3

Question 2 weakness 2 part 2: Was an independent validation set used? Does the advantage persist with shared parameters, multiple task orders, and a joint solve under the fixed final mask?

Thank you for these concrete validation suggestions. We provide the parameter-selection procedure, shared-parameter analysis, task-order experiments, and fixed-mask joint-solve results below.

Independent validation set. Thank you for asking us to clarify the parameter-selection procedure. For each setting, we select hyperparameters using a fixed validation split comprising 15% of the training data. Both the sensitivity analysis in Figure 4 and the shared-parameter analysis below report results on the test sets.

Shared parameters. Thank you for suggesting an evaluation with shared hyperparameters. We examined a single gate configuration, (β,α\beta, \alpha) = (0.7, 0.5), across all 16 model–task-stream–sparsity settings. In these individual-run results, CARGO achieves higher ACC than the strongest of COPAL, Accum. Wanda, Full reconstruction, and Protected reconstruction in every setting, with an average improvement of 3.18 percentage points. For example, on Ministral-3-8B / Long-seq at 50% sparsity, it achieves 77.67 ACC, compared with 76.13 for the strongest baseline. These results support using one shared configuration across the tested models, task streams, and sparsity levels without adjusting the gate parameters for each setting.

Task order. To evaluate the performance under different task orders, we evaluated two additional, predetermined random task orders while retaining the original parameters, β\beta=0.9 and α\alpha=0.3. The results below are for Llama-2-7B / TRACE at 50% sparsity; entries report ACC (BWT).

MethodNew order 1New order 2
Copal33.12(+11.53)34.97(+2.88)
Accum. Wanda35.96(+0.60)35.92(-0.99)
Full reconstruction36.04(−0.45)37.94(+0.09)
Protected reconstruction36.33(−0.77)37.61(−0.94)
ACC.Hessian37.18(−0.48)36.34(−1.40)
CARGO40.02(+3.47)38.81(+1.54)

Under the first new order, CARGO improves ACC over ACC. Hessian by 2.84 points. Under the second, its ACC improves by 0.87 points on full reconstruction, with higher BWT. CARGO maintains positive BWT under both orders. These results show that its effectiveness extends beyond the original task order. Our conclusion applies to the tested orders and does not imply insensitivity to every possible task permutation.

Joint solve under a fixed mask. On Llama-2-7B / TRACE at 50% sparsity, we fixed CARGO's final mask and used damped joint reconstruction. The original column-wise solution and the joint solution using CARGO curvature achieve ACC of 37.24 and 36.53, respectively. These results show that our column-wise reconstruction achieves performance comparable to joint reconstruction.

Revision We have added the task-order experiments to the appendix. Time and computational constraints prevent us from completing the remaining experiments within the rebuttal period. Once these experiments are complete, we will add an ablation study of the eigenspace-level gate.

Reviewer_vukgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CARGO which is a training-free continual pruning method. By accumulating Hessian matrices across tasks, it simultaneously guides weight pruning and the reconstruction of remaining weights to solve continual pruning problem. Its core innovation lies in the use of spectral denoising and directional forgetting gating, which preserve important parameter directions shared across tasks while attenuating outdated information. Experimental results demonstrate that CARGO outperforms existing continual pruning methods.

Critical strengths
  1. The paper provides a solid theoretical foundation for using accumulated Hessians to preserve historical knowledge without storing past data.
  2. The spectral cleaning and directional gating mechanisms offer an interesting way to balance knowledge retention and adaptation.
  3. The method shows consistent improvements across different LLMs, tasks, and sparsity levels without requiring retraining.
Critical weaknesses

1.The strong accumulated-Hessian baseline is missing from the 70% and 2:4 sparsity experiments, making it difficult to assess the actual benefits of cleaning and gating. 2.The paper lacks a comparison with offline pruning using combined task data, making it difficult to assess the effectiveness of its continual pruning strategy.

Questions for authors

same as above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vukg

Dear Reviewer vukg,

We sincerely thank you for carefully reading our paper and providing thoughtful feedback. We are encouraged by the positive comments.

Weakness 1: Without ACC. Hessian at 70% and 2:4 sparsity, the additional effects of cleaning and gating cannot be assessed.

First, we want to clarify that ACC. Hessian is not an independent baseline but rather an ablated variant of our proposed CARGO method, in which the cleaning and gating components are removed. Second, to answer the reviewer's question, we further added ACC.Hessian comparisons at 70% and 2:4 sparsity. The results for Llama-2-7B are shown below, with CARGO retaining the parameter configuration for each setting:

SettingACC.Hessian ACCCARGO ACCCARGO − ACC.Hessian
TRACE, 70%30.8934.29+3.40
Long-seq, 70%46.3149.12+2.81
TRACE, 2:434.4935.43+0.94
Long-seq, 2:457.1258.73+1.61

At 70% sparsity, CARGO improves ACC by 3.40 and 2.81 percentage points on TRACE and Long-seq, respectively. Under 2:4 pruning, it improves TRACE and Long-seq ACC by 0.94 and 1.61 points. These results show higher ACC for CARGO in these tested settings, with the additional benefits of cleaning and gating particularly evident at 70% sparsity.

Weakness 2: A comparison with offline pruning using data from all tasks is missing.

Thank you for pointing out the important comparison with offline pruning. We added offline joint pruning, pooling 128 calibration sequences from each of the eight TRACE tasks and pruning once from the same dense model. Final ACC at 50% sparsity is shown below:

ModelOffline joint pruningCARGO
Llama-2-7B36.2336.27
Qwen3-8B51.0750.99
Ministral-3-8B47.8048.37

All methods in the table report mean ACC over three runs. CARGO differs from offline joint pruning by +0.04, −0.08, and +0.57 points across the three models. Carrying curvature across tasks therefore achieves final performance close to pooled access to all task data while accessing only the current task data at each step.

Revision

Due to time and computational constraints, we cannot finish the experiments under all settings in the paper. We will add the 30% and 70% sparsity, 2:4 semi-structure, and offline experiment results in the camera-ready version after we finish all the experiments.

Thank you for your response, it addresses my concerns. I will maintain my original score.

Dear reviewer,

We sincerely appreciate your constructive feedback. Your valuable suggestions significantly improved the quality of our paper.

Again, thank you so much for your time and effort in reviewing our work.

Best, Authors

Reviewer_Rmt5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free continual pruning method that leverages cross-task curvature for weight selection and reconstruction. Theoretically, it shows that information from all historical tasks can be compressed into a single accumulated Hessian. The proposed spectral cleaning removes curvature directions corrupted by estimation noise, while task-aware gating avoids domination by earlier tasks. Experiments on multiple LLM architectures demonstrate the effectiveness of the method.

Critical strengths

1.The paper addresses continual pruning with clear motivation and experimental support. 2.The paper provides theoretical proof that historical task information can be compressed into an accumulated Hessian matrix without retaining any calibration data. It supports long task sequences and provides valuable insights to this field. 3.Experiments across multiple model architectures and sparsity levels demonstrate the effectiveness of the proposed method.

Critical weaknesses

1.Although weight reconstruction can mitigate information loss during pruning, the model's generalization performance on downstream tasks may degrade due to the absence of foundational task data in the calibration set. 2.Do Eqs. 8 and 9 imply that when there is a large disparity between new and old tasks, the new task cannot preserve many feature directions from old tasks, thereby weakening all these directions? Based on this, does it suggest that the design in Section 4.3 does not account for the cross-task transferability of specific feature directions, but instead exhibits sensitivity to the task execution order? 3.The reported baseline methods are relatively limited. It would be beneficial to add more comparative baselines. For example, adopting a diagonal Fisher information matrix in EWC to the curvature approximation and applied to continuous pruning. This would help clarify the differences between existing techniques and the proposed method.

Questions for authors

1.The threshold α\alpha and the decay factor β\beta appear to be highly dependent on the model architecture and the dataset. Their optimal values can differ substantially under different settings, which requires manually tuning and make it difficult to apply the method to new models or tasks. Is there any guideline or underlying principle for choosing α\alpha and β\beta? 2.Although the proposed method is training-free, it remains unclear how its performance compares to that of trainable methods. 3.For a 7B-scale model, the curvature state alone already requires approximately 28 GB of storage. Please provide comparisons with other baseline methods about storage overhead and computational efficiency. 4.What is the performance upper bound achievable by retaining previous task calibration samples and employing the exact joint Hessian matrix?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Rmt5 Part 1

Dear Reviewer Rmt5,

We sincerely appreciate your recognition of the motivation and significance of our work. We are encouraged by the positive comments.

Weakness1. Could calibration data that do not cover foundational capabilities harm performance on other downstream tasks?

Thank you for raising the question of foundational capability retention. Prior work supports your concern about calibration coverage.[1] identifies single-calibration-set activation statistics as a source of calibration bias that harms downstream cross-task generalization. [2] also find that calibration-data choice substantially affects pruned-model performance; using target-task data does not necessarily outperform using pretraining data, even on that target task.

Our goal is to continually adapt to new tasks without fine-tuning while retaining the ability to perform previously encountered tasks. To this end, CARGO combines historical curvature with current-task information for weight selection and weight reconstruction, so that reconstruction accounts for pruning errors on both new and previous tasks. We use ACC and BWT to assess final performance across the task stream and retention on previous tasks, respectively. Preserving general capabilities beyond the task stream is also important, but it is a separate research question from the continual adaptation and retention addressed in this work.

[1] Xiaoyun Liu, Divya Saxena, Jiannong Cao, Yuqing Zhao, Yiying Dong, and Penghui Ruan. GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models. arXiv preprint arXiv:2603.13418, 2026.

[2] Abhinav Bandari, Lu Yin, Cheng-Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning. In Proceedings of EMNLP, 2024.

Weakness 2. When tasks differ substantially, does the gate weaken historical directions, overlook transferability, and depend on task order?

Thank you for highlighting how task differences and arrival order may affect historical retention. This is an important aspect of understanding the gate’s behavior. The gate adjusts the contribution of historical curvature to the reconstruction objective. When a historical direction does not meet the current task's support threshold, its contribution is multiplied by a positive factor β\beta and continues to participate in subsequent pruning and reconstruction. The current task's curvature contribution is added in full. This update controls the relative contributions of historical and current information.

We use the current task’s support for historical directions as a signal for information reuse. The order dependence you raise merits examination, since an online update can vary with task arrival order. To investigate this, we evaluated two additional, predetermined random task orders while retaining the original parameters, β\beta=0.9 and alphaalpha =0.3. The results below are for Llama-2-7B / TRACE at 50% sparsity; entries report ACC (BWT).

MethodNew order 1New order 2
Copal33.12 (+11.53)34.97 (+2.88)
Accum. Wanda35.96 (+0.60)35.92 (-0.99)
Full reconstruction36.04 (-0.45)37.94 (+0.09)
Protected reconstruction36.33 (-0.77)37.61 (-0.94)
ACC.Hessian37.18 (-0.48)36.34 (-1.40)
CARGO40.02 (+3.47)38.81 (+1.54)

Under the first new order, CARGO improves ACC over ACC. Hessian by 2.84 points. Under the second, its ACC improves by 0.87 points on full reconstruction, with higher BWT. CARGO maintains positive BWT under both orders. These results show that its effectiveness extends beyond the original task order. Our conclusion applies to the tested orders and does not imply insensitivity to every possible task permutation.

AuthorsOpenReview ↗

Response to Reviewer Rmt5 Part 2

Weakness 3: Add a continual pruning baseline based on EWC or diagonal Fisher information.

We thank the reviewer for this valuable suggestion. We would like to emphasize that continual LLM pruning is an emerging research direction, with only one existing work directly investigating this problem. Consequently, the number of directly applicable baselines is inherently limited. We have already gone beyond existing continual pruning methods by adapting representative approaches from LLM pruning to our setting.

Regarding the suggested EWC-based approach, we added an EWC-inspired baseline that uses historical Fisher importance to constrain weight reconstruction for the current task. At the dense weights, we estimate the empirical Fisher from squared gradients on answer tokens in the calibration sequences and accumulate it across tasks. For the current task, we construct $ K_t=H_t+\gamma\operatorname{diag}(f_{\mathrm{past}}). $ Both pruning scores and weight reconstruction use KtK_t. The current Hessian supplies the correlations between weights needed for reconstruction, while the historical Fisher imposes stronger constraints on changes to historically important coordinates. A purely diagonal curvature matrix has no off-diagonal correlations through which deleting one weight can induce adjustments to others. We therefore retain the current task's Hessian for reconstruction. In the implementation, the empirical Fisher is averaged over output rows so that each layer shares one reconstruction matrix. Before accumulation, each task's Fisher vector is rescaled to match the mean diagonal magnitude of that task's Hessian. The available results for Llama-2-7B on TRACE at 50% sparsity are:

MethodACC
Fisher, γ\gamma=0.137.03
Fisher, γ\gamma=0.535.20
Fisher, γ\gamma=1.035.55
CARGO39.27

We report all three values of γ\gamma. CARGO exceeds the highest ACC among these Fisher configurations by 2.24 points.

Question1: Is there any guideline or underlying principle for choosing α\alpha and β\beta

We can explain the parameters’ roles and provide guidance using the existing grid results. Section 5.4 evaluates the full 4×4 grid, β∈[0.7,0.8,0.9,0.95]\beta\in [0.7, 0.8, 0.9, 0.95 ] and α∈[0.3,0.4,0.5,0.7]\alpha \in [0.3, 0.4, 0.5, 0.7], for Qwen3-8B at 50% sparsity. Across this range, ACC varies by approximately 2.4 points on TRACE and 1.1 points on Long-seq. Performance in these settings therefore isn't concentrated at a single narrow parameter choice.

We also examined the shared configuration (β\beta,α\alpha)=(0.7,0.5) across all 16 model–task-stream–sparsity settings. In these individual-run results, its ACC exceeds the highest value among COPAL, Accum. Wanda, Full, and Protected reconstruction in every setting, with a mean difference of +3.18 percentage points. For example, it achieves 77.67 ACC on Ministral-3-8B / Long-seq / 50%, exceeding the highest baseline result of 76.13 by 1.54 points. This shows that one configuration can be used across several settings.

Question2: How does CARGO compare with methods that allow training?

There is currently no baseline that directly combines continual pruning and fine-tuning under our setting. To address your suggestion, we combined pruning with masked LoRA training to construct three trainable comparators based on Full reconstruction, COPAL, and CARGO, each with historical-data replay. All three trainable comparators below are our adaptations to the continual pruning setting. After each pruning step, we train the adapter for two epochs on the calibration sequences retained from all tasks seen so far: 128 sequences per task, totaling 1,024 at the final step. The adapters target q_proj and v_proj, with rank 8 and scaling parameter 16, and contain approximately 4.19M parameters, or 0.06% of the model. The low-rank update is masked by the current pruning mask so that pruned positions remain zero. The adapter is carried across tasks, detached during pruning, and attached for training and evaluation.

MethodGradient trainingHistorical data retainedACCBWT
CARGONoNo39.27−0.28
Full + masked LoRA + replayYesYes39.63−2.97
COPAL + masked LoRA + replayYesYes40.03+18.47
CARGO + masked LoRA + replayYesYes42.77−0.88

Training-free CARGO achieves 39.27 ACC, within 0.36 and 0.76 points of Full and COPAL with training and replay, respectively. Under the same training setting, CARGO achieves 42.77 ACC, exceeding Full and COPAL by 3.14 and 2.74 points and improving on training-free CARGO by 3.50 points. These results also show that adding training and historical-data replay is applicable to CARGO: under this configuration, masked LoRA plus replay increases its ACC from 39.27 to 42.77. CARGO provides competitive performance without training and can also be combined with subsequent fine-tuning to further improve continual adaptation.

AuthorsOpenReview ↗

Response to Reviewer Rmt5 Part 3

Question 3: What are the storage and computational costs of the approximately 28 GB curvature state?

Thank you for pointing out the need for clearer storage and computational comparisons. We provide the historical state sizes and complete experimental runtimes separately. We compare the core statistical tensors retained across tasks over the same modules in FP32. Llama-2-7B has 32 Transformer blocks, each with four attention projections and three MLP projections, a hidden dimension of 4,096, and an MLP intermediate dimension of 11,008. Accumulated per-weight scores require 25.90 GB, while accumulated curvature requires 28.40 GB. CARGO requires approximately 2.49 GB (9.6%) more historical state than score-accumulation methods and achieves higher ACC in this setting.

MethodStorage (GB)ACC
COPAL25.9033.85
Accum. Wanda25.9034.99
Full reconstruction25.9037.62
Protected reconstruction25.9037.55
CARGO28.4039.27

Computationally, cleaning and gating add two eigendecompositions per layer per task. The following table reports total experimental runtime, including pruning and evaluation, for Qwen3-8B / TRACE at 50% sparsity, together with individual-run ACC for this setting:

MethodTIme (hours)ACC
COPAL15.145.69
Accum. Wanda14.744.02
Full reconstruction15.646.71
Protected reconstruction19.945.79
CARGO16.051.45

Relative to Full reconstruction, CARGO adds approximately 0.4 hours (2.6%) to total experimental runtime while improving ACC by 4.74 points. Relative to COPAL, runtime increases by about 6.0% and ACC improves by 5.76 points. Compared with Protected reconstruction, CARGO reduces runtime by approximately 19.6% while improving ACC by 5.66 points. These totals include approximately 14.7 hours of shared evaluation, so the runtime percentages refer to the complete experiment. For continual adaptation when historical data cannot be retained, we believe these performance gains justify the additional storage and computational cost.

Question 4: What performance is achievable when historical calibration samples are retained, and the exact joint Hessian is used?

Thank you for suggesting a joint-reconstruction reference with retained historical data. For fixed layer inputs, computing a joint Hessian from all retained task inputs is mathematically equivalent to accumulating task Hessians sequentially. Let Xk∈Rd×nkX_k\in\mathbb R^{d\times n_k} be the fixed layer inputs of task k. For one output unit, represent the dense and sparse weights as column vectors w,w^∈Rdw,\widehat w\in\mathbb R^d, and write δ=w^−w\delta=\widehat w-w. Concatenate the task inputs: Xjoint=[X1,…,Xt].X_{\mathrm{joint}}=[X_1,\ldots,X_t].

\mathcal{L}(\delta) &= \sum_{k=1}^{t} \lVert X_k^{\top}\delta \rVert_2^2 = \lVert X_{\mathrm{joint}}^{\top}\delta \rVert_2^2 = \delta^{\top}H_{\mathrm{joint}}\delta, \\\\ H_{\mathrm{joint}} &= X_{\mathrm{joint}}X_{\mathrm{joint}}^{\top} = \sum_{k=1}^{t} X_kX_k^{\top} = H_{\mathrm{acc}}. \end{aligned}$$ We added offline joint pruning, pooling 128 calibration sequences from each of the eight TRACE tasks and pruning once from the same dense model. Final ACC at 50% sparsity is shown below: |Model | Offline joint pruning| CARGO | |---|---:|---:| | Llama-2-7B | 36.23 | 36.27 | | Qwen3-8B | 51.07 | 50.99 | | Ministral-3-8B | 47.80 | 48.37 | All methods in the table report mean ACC over three runs. CARGO differs from offline joint pruning by +0.04, −0.08, and +0.57 points across the three models. Carrying curvature across tasks therefore achieves final performance close to pooled access to all task data while accessing only the current task data at each step. **Revision** We add the storage comparison table and the new order experiments to the paper and move the time cost section from the appendix to the main paper. For the Fisher, training, and offline baselines, since we cannot finish experiments under all settings within the rebuttal period, we will add these in the camera-ready version after we finish all the experiments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision and Response to Reviewers

Dear Area Chair and Reviewers,

We thank the Area Chair for the engagement so far in the review process and the reviewers for their constructive feedback. Below, we summarize our responses to the concerns and the changes made to the revision.

1. Reconstruction objective — Reviewer TKHL

We provided a quadratic cross-task objective for CARGO’s processed curvature, with inputs and transformations fixed. CARGO follows SparseGPT’s column-wise reconstruction. Under the same final mask and curvature, column-wise and joint reconstruction achieve comparable ACC: 37.24 and 36.53.

2. Fisher and trainable baselines — Reviewer Rmt5

We provided an EWC-inspired Fisher baseline and trainable extensions using masked LoRA and replay. On Llama-2-7B / TRACE at 50% sparsity, CARGO with training and replay achieves 42.77 ACC, versus 39.63 for Full reconstruction and 40.03 for COPAL under the same training configuration.

3. Accumulated-Hessian comparisons — Reviewer vukg

We compared ACC. Hessian at 70% and 2:4 sparsity on TRACE and Long-seq to examine cleaning and gating beyond accumulation.

4. Historical calibration data — Reviewers Rmt5 and vukg

Rmt5 requested a joint-Hessian reference with historical data; vukg requested offline pooled pruning. CARGO achieves comparable performance to offline pooled pruning across three model families. For Rmt5, we also showed that pooled inputs and accumulated Hessians yield the same quadratic objective for fixed layer inputs.

5. Gate-parameter guidance — Reviewer Rmt5

We explained that α\alpha controls the retention threshold and betabeta controls decay. Lower α\alpha or higher betabeta preserves more historical curvature. The sensitivity analysis is now in Section 5.4, Figure 4.

6. Validation and shared parameters — Reviewer TKHL

Hyperparameters are selected using a fixed validation split comprising 15% of training data; the sensitivity and shared-parameter analyses report test results. With (beta,αbeta, \alpha) = (0.7, 0.5) across 16 settings, CARGO exceeds the strongest of the four original baselines in every individual-run comparison, averaging a 3.18-percentage-point ACC improvement.

7. Historical retention — Reviewer Rmt5

We clarified that historical curvature along directions below the current-task support threshold is scaled by betabeta, rather than removed, and still contributes to pruning and reconstruction.

8. Task order — Reviewers Rmt5 and TKHL

With unchanged gate parameters, CARGO maintains strong ACC and positive BWT under two new orders on Llama-2-7B / TRACE at 50% sparsity. Appendix B.2, Table 15 reports the sequences and results.

9. Basis invariance and stability — Reviewer TKHL

We provided an eigenspace-level gate and verified its basis invariance through rotation tests. Across 1,568 module-update steps, the two rules differed in 17 same-input updates, with mean and maximum relative curvature-update differences of 1.4 × 10⁻⁸ and 2.1 × 10⁻⁶.

For nearly repeated eigenvalues, perturbations of 10⁻⁸–10⁻⁵ across 18 saved module states produced a maximum relative update change of 1.2 × 10⁻⁴. In the full Llama-2-7B / TRACE experiment at 50% sparsity, the original and eigenspace-level gates achieved 38.59 and 37.78 ACC. Thus, final performance is similar in this setting, while the eigenspace formulation resolves the basis ambiguity.

10. Storage and runtime — Reviewer Rmt5

Relative to Full reconstruction, CARGO uses 9.6% more historical-state storage and gains 1.65 ACC points on Llama-2-7B / TRACE. On Qwen3-8B / TRACE, total experimental runtime increases by 2.6%, with a gain of 4.74 ACC points. These gains justify the additional storage and computation. Historical-state size is fixed across tasks.

11. Scope — Reviewer Rmt5

We clarified CARGO’s goal of adapting to new tasks while retaining earlier-task performance, and discussed literature on calibration data and general capabilities.

12. Reporting consistency — Reviewer TKHL

We checked corresponding results and clarified individual-run versus averaged results. Appendix B.1 reports means and standard deviations for larger models, ablations, sparsity comparisons, and 2:4 pruning.

Summary of Manuscript Revisions

  • Parameter sensitivity (Rmt5): Moved the existing analysis from the appendix to Section 5.4, Figure 4.
  • Runtime and storage (Rmt5): Moved the existing runtime analysis to Section 5.5 and added storage analysis. Tables 5 and 6 present both costs alongside ACC.
  • Task order (Rmt5, TKHL): Added task sequences, results, and analysis in Appendix B.2, Table 15.
  • Error bars and reporting (TKHL): Expanded Appendix B.1 with means and standard deviations and clarified individual-run versus averaged results.

Revisions are highlighted in blue. Thank you again for your time and suggestions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.