- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Gradient Skill Decomposition (GSD), which connects human-named skills with parameter-update directions in large language models. GSD extracts per-example LoRA-B gradients at a fixed original checkpoint, learns compact layer-wise representations, aligns them with skill annotations through a multilabel predictor, and maps the recovered directions back to parameter space. During subsequent supervised fine-tuning, it removes components of parameter displacement within selected skill subspaces to mitigate forgetting. Experiments include skill prediction on a synthetic algebra dataset, independent fine-tuning of three models on six continuation tasks, and sequential training under two task orders. The reported results show improved protected-task performance with broadly comparable continuation-task metrics, while a restricted skill-subset ablation provides preliminary evidence of selective protection.
Critical strengths
S1: Connecting skill interpretation with training intervention: The paper connects human-interpretable skill descriptions with the model’s parameter-update space and uses this correspondence for practical training interventions. The recovered skill directions are used not only to characterize the skills associated with individual examples, but also to define preservation constraints during subsequent fine-tuning. The method therefore goes beyond skill prediction or representation visualization by making skill interpretations actionable during optimization, providing a concrete and testable framework that connects training interpretability with capability preservation.
S2: Supporting skill-selective preservation and complementing replay: Compared with the replay baseline used in the paper, which preserves prior capabilities by rehearsing old-task examples, GSD provides a parameter-space mechanism for selecting preservation targets by skill name. It thus supports finer-grained capability preservation while mitigating catastrophic forgetting, with the skill-subset ablation providing preliminary empirical evidence of selectivity. Moreover, GSD can be combined with replay, and the combined method improves six-task-average retention over replay alone across all three evaluated models. This is consequential because the method offers more controllable preservation while adding value to existing replay-based training pipelines, rather than requiring replay to be replaced.
Critical weaknesses
Although GSD improves capability retention, the main experiments do not sufficiently isolate the additional contribution of skill-label alignment. Since Mean-Grad and GSD differ in protected subspace dimensionality, it remains unclear whether the gains arise from semantically guided direction selection or from protecting richer gradient structure. Comparisons with matched-dimensional gradient-subspace methods that do not use skill labels would help clarify this distinction.
Questions for authors
Both skill-basis construction and preservation evaluation currently rely on different splits of Math Gen. Do the authors have evidence that skill directions extracted from one dataset can preserve the same skill on an independently sourced dataset?
Response to weakness and question
Response summary
Thank you for recognizing the connection between skill interpretation and training intervention, the ability to select preservation targets by skill name, and the complementarity with replay. These strengths capture the paper's central contribution: a shared representation that describes the skills associated with examples and turns those descriptions into explicit constraints on later model updates. The original experiments support its practical value through improved retention in all 18 independent settings without replay and higher average retention from adding GSD to replay across all three models. Your questions help distinguish this operational contribution from the additional evidence needed to isolate the effect of skill-label alignment and establish protection across datasets.
Alignment contribution (W1). We add PCA and Random comparisons: GSD exceeds PCA on five of six tasks and Random on all six, with the highest average protected accuracy (Section 5.2, Table 4; Appendix I). Selected-skill studies separately test controllability (Section 5.1, Figure 3; Appendix H.5, Figure 8). Isolating label effects at matched per-layer rank remains open.
Cross-dataset protection (Q1). We add original-MATH evaluation using the unchanged Math Gen-derived basis and continual-task training. GSD improves accuracy over Vanilla by 6.5 and 11.0 percentage points after FinQA and SQuAD, respectively (Appendix K, Table 16), demonstrating protection beyond the generated evaluation corpus.
Critical Weaknesses
W1
Response location: Respond here; this weakness is not covered by a separate explicit question.
Response: GSD outperforms PCA on five of six tasks and Random on all six, achieving the highest average protected accuracy under the shared data and no-replay adaptation protocol (Section 5.2, Table 4).
PCA retains ten global components, so this comparison evaluates complete basis constructions rather than label effects at matched per-layer rank; Appendix I provides the basis definitions and continual task results. Alongside stronger retention, GSD offers explicit control over what to protect: users can select and combine targets by skill name through its learned skill-to-direction mapping (Section 5.1, Figure 3).
Paper references: Section 5.2, Table 4; Section 5.1, Figure 3; Appendix I, Table 15.
Questions for Authors
Q1
Response: Yes. Appendix K, Table 16 evaluates original-MATH algebra retention on MiniCPM5-1B after FinQA and SQuAD adaptation using the unchanged Math Gen-derived basis and continual-task training. Only the protected evaluation dataset changes; no basis is refitted on original MATH.
GSD improves accuracy over Vanilla by 6.5 percentage points after FinQA and 11.0 after SQuAD. A GSD variant achieves the highest original-MATH accuracy in both settings: Replay+Ours after FinQA and projection alone after SQuAD. The same learned directions therefore preserve algebra capability on independently sourced problems, extending protection beyond the generated Math Gen evaluation corpus.
Paper references: Appendix K, Table 16.