- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARGO starts from a cross-task layerwise reconstruction objective and shows that, under fixed historical layer inputs and single-weight deletion, historical data can be represented by an accumulated second-moment matrix. In practice, it spectrally truncates and normalizes current-task curvature, measures current support along historical eigenvectors, gates or decays historical eigenvalues, and then performs SparseGPT-style sequential pruning and reconstruction using a damped surrogate curvature matrix. The experiments report high continual-pruning ACC in most settings and generally improved BWT.
Critical strengths
The problem setting and methodological direction are valuable. Compared with accumulating only per-weight scores, using cross-task input second moments to guide compensation is a sensible advance for continual pruning. The single-weight OBS result is also concise and complete.
Critical weaknesses
- The theory does not cover the implemented algorithm. The proposition concerns a true accumulated matrix and one-coordinate deletion, while CARGO uses a processed surrogate matrix and sequentially deletes many weights. The paper neither proves that this surrogate corresponds to a well-defined cross-task objective nor shows that the result under the final fixed mask is close to the joint optimum.
- Basis stability and the experimental protocol have serious flaws. Thresholding individual eigenvectors is not evidently invariant to basis rotations within a repeated-eigenvalue subspace. Moreover, multiple tables report different results for identical settings, and each model, task stream, and sparsity level uses different hyperparameters without explaining whether they were selected on an independent validation set.
Questions for authors
- Are Eqs. (7)–(10) invariant to orthogonal changes of basis within repeated-eigenvalue subspaces? If not, please use subspace-level gating and report stability under nearly repeated eigenvalues.
- Were the parameters in Table 7 selected using an independent validation set? Does CARGO retain its advantage with one global hyperparameter setting, multiple random task orders, and a baseline that jointly solves for the weights under the fixed final mask?
3 comments
Response to Reviewer TKHL Part2
Q1&Weakness Part1: Are the equations invariant to orthogonal changes of basis within repeated-eigenvalue subspaces? How are nearly repeated eigenvalues handled?
Thank you so much for identifying the basis ambiguity under repeated eigenvalues and suggesting subspace-level gating. We have implemented this correction and added rotation-invariance checks, perturbation tests for nearly repeated eigenvalues, and full task-stream experiments.
Subspace-level definition. The original eigenvector-level gate is not generally invariant to orthogonal basis changes within repeated positive-eigenvalue eigenspaces. We replace it with one shared gate per exact eigenspace. For , write , where are the distinct positive eigenvalues, are their orthogonal spectral projectors, and . With denoting the cleaned and normalized current-task curvature, define:
m_k=\frac{c_k}{c_k+\lambda_k/(t-1)+\epsilon},$$ $$g_k=\begin{cases}1,&m_k\geq\alpha,\\\beta,&m_k<\alpha,\end{cases} \qquad M_t=\sum_k g_k\lambda_kP_k+C_t.$$ Replacing any subspace basis $U_k$ with $U_kQ_k$, for orthogonal $Q_k$, leaves $P_k$ unchanged and therefore leaves $c_k$, $g_k$, and $M_t$ unchanged. When all positive eigenvalues are distinct, $r_k=1$ and this definition reduces to the original direction-wise update. Numerically, we group only positive eigenvalues that are identical in their computed floating-point values; strictly distinct, nearly repeated eigenvalues remain separate. The correction retains the original $\alpha$, $\beta$, numerical stabilizer, cleaning procedure, and reconstruction solver, introducing no new hyperparameter. **Rotation and update checks.** On small matrices containing one or multiple repeated-eigenvalue blocks, we tested 20 orthogonal basis changes per block, including sign flips and column permutations. The maximum relative Frobenius difference between subspace updates was below $4.1\times10^{-16}$ in float64 and $1.0\times10^{-7}$ in float32, with no gate changes. The two rules were bit-identical in the simple-spectrum tests. In a full subspace-gate run on Llama-2-7B / TRACE at 50% sparsity with $(\beta,\alpha)=(0.9,0.3)$, we also measured the difference between the two updates on identical inputs and the same eigendecomposition: $\Delta=\lVert F_{\mathrm{sub}}-F_{\mathrm{orig}}\rVert_F/\lVert F_{\mathrm{orig}}\rVert_F$ . The rules differed at 17 of 1,568 module-update steps, with mean $\Delta=1.4\times10^{-8}$ and maximum $\Delta=2.1\times10^{-6}$. These measurements quantify local update differences on the same inputs. **Perturbation stability for nearly repeated eigenvalues.** We selected 18 saved $4096\times4096$ module states from a real subspace-gate run, covering different tasks, attention/MLP modules, and states with relatively high mixed-cluster curvature mass. We added PSD perturbations $E=\delta\|X\|_Fvv^\top$ to either the historical curvature $M_{t-1}$ or current curvature $C_t$, testing 10 random unit directions per magnitude. The table reports the relative Frobenius change in the subspace update when perturbing historical curvature in float32: | Perturbation magnitude | Median output change | Maximum output change | |---|---:|---:| | $10^{-8}$ | $2.1\times10^{-5}$ | $6.1\times10^{-5}$ | | $10^{-7}$ | $2.3\times10^{-5}$ | $6.5\times10^{-5}$ | | $10^{-6}$ | $3.0\times10^{-5}$ | $7.6\times10^{-5}$ | | $10^{-5}$ | $4.1\times10^{-5}$ | $1.2\times10^{-4}$ The two rules had the same perturbation response on these saved states, and perturbing the current-task curvature Ct also produced small output changes. The exact-subspace correction removes basis ambiguity for exact repeats. Strictly distinct, nearly repeated eigenvalues retain direction-wise decisions, and both rules can undergo gate switches in synthetic tests where support scores cross the threshold. The table reports measured behavior for the tested real states and perturbation magnitudes. **Full task-stream performance.** We ran both rules on Llama-2-7B / TRACE at 50% sparsity with the same gate parameters, carrying the subspace-rule curvature into subsequent pruning and reconstruction steps. The individual-run results are: |Gate rule | ACC | BWT | |---|---:|---:| | Original direction-wise gate | 38.59 | +0.25 | | Subspace-level gate | 37.78 | −0.99 | The final ACC values differ by 0.81 percentage points. We report ACC and BWT for both rules to show full-stream performance when the subspace rule is actually applied. We appreciate your suggestion, which helped remove the mathematical ambiguity for exact repeats and motivated stability checks tied to the implemented update.Response to Reviewer TKHL Part3
Question 2 weakness 2 part 2: Was an independent validation set used? Does the advantage persist with shared parameters, multiple task orders, and a joint solve under the fixed final mask?
Thank you for these concrete validation suggestions. We provide the parameter-selection procedure, shared-parameter analysis, task-order experiments, and fixed-mask joint-solve results below.
Independent validation set. Thank you for asking us to clarify the parameter-selection procedure. For each setting, we select hyperparameters using a fixed validation split comprising 15% of the training data. Both the sensitivity analysis in Figure 4 and the shared-parameter analysis below report results on the test sets.
Shared parameters. Thank you for suggesting an evaluation with shared hyperparameters. We examined a single gate configuration, () = (0.7, 0.5), across all 16 model–task-stream–sparsity settings. In these individual-run results, CARGO achieves higher ACC than the strongest of COPAL, Accum. Wanda, Full reconstruction, and Protected reconstruction in every setting, with an average improvement of 3.18 percentage points. For example, on Ministral-3-8B / Long-seq at 50% sparsity, it achieves 77.67 ACC, compared with 76.13 for the strongest baseline. These results support using one shared configuration across the tested models, task streams, and sparsity levels without adjusting the gate parameters for each setting.
Task order. To evaluate the performance under different task orders, we evaluated two additional, predetermined random task orders while retaining the original parameters, =0.9 and =0.3. The results below are for Llama-2-7B / TRACE at 50% sparsity; entries report ACC (BWT).
| Method | New order 1 | New order 2 |
|---|---|---|
| Copal | 33.12(+11.53) | 34.97(+2.88) |
| Accum. Wanda | 35.96(+0.60) | 35.92(-0.99) |
| Full reconstruction | 36.04(−0.45) | 37.94(+0.09) |
| Protected reconstruction | 36.33(−0.77) | 37.61(−0.94) |
| ACC.Hessian | 37.18(−0.48) | 36.34(−1.40) |
| CARGO | 40.02(+3.47) | 38.81(+1.54) |
Under the first new order, CARGO improves ACC over ACC. Hessian by 2.84 points. Under the second, its ACC improves by 0.87 points on full reconstruction, with higher BWT. CARGO maintains positive BWT under both orders. These results show that its effectiveness extends beyond the original task order. Our conclusion applies to the tested orders and does not imply insensitivity to every possible task permutation.
Joint solve under a fixed mask. On Llama-2-7B / TRACE at 50% sparsity, we fixed CARGO's final mask and used damped joint reconstruction. The original column-wise solution and the joint solution using CARGO curvature achieve ACC of 37.24 and 36.53, respectively. These results show that our column-wise reconstruction achieves performance comparable to joint reconstruction.
Revision We have added the task-order experiments to the appendix. Time and computational constraints prevent us from completing the remaining experiments within the rebuttal period. Once these experiments are complete, we will add an ablation study of the eigenspace-level gate.
Response to Reviewer TKHL Part1
Dear reviewer TKHL,
We sincerely thank you for the encouraging assessment and constructive suggestions. We conducted extensive experiments and address your concerns point by point below.
Weakness 1: The proposition does not cover the processed curvature or multiple-weight deletion, and the relationship between the implemented solution and the joint optimum is unclear.
Thank you for suggesting a clearer connection between the theory and the implemented algorithm. CARGO uses SparseGPT’s column-wise pruning and weight reconstruction procedure; our design provides the cross-task curvature used by that procedure.
Let Ck=ZkZk⊤ denote the task-k curvature contribution, incorporating the prescribed cleaning and normalization. Let Bj scale each historical eigendirection by the square root of its gate value, so that Mt=BtMt−1Bt⊤+Ct. Defining At,k=BtBt−1⋯Bk+1 and At,t=I, expanding the recursion gives:
Mt=∑k=1tAt,kCkAt,k⊤.
Lt(δ)=21∑k=1t∥Zk⊤At,k⊤δ∥22=21δ⊤Mtδ.
Thus, with the collected layer inputs and curvature states fixed, Mt is exactly the Hessian of this cross-task reconstruction objective. The single-deletion OBS score and reconstruction update follow from this objective; sequential multi-weight reconstruction uses SparseGPT’s scalable approximation.
We further fixed the same final CARGO mask on Llama-2-7B / TRACE at 50% sparsity and jointly reconstructed all retained weights in each row using the same CARGO curvature, with damping. The original column-wise and joint reconstructions achieve final ACC of 37.24 and 36.53, respectively. This shows that our column-wise reconstruction achieves the same level of performance as Joint reconstruction does. These results show that our column-wise reconstruction achieves performance comparable to joint reconstruction.