- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors observe that small specialist LLMs trained with RL retain much of their performance on non-target tasks. They propose Low-Rank Ascent (LRA) to induce focal forgetting, and ANTIGEN, which uses the subspaces identified by ascent to host high-plasticity parameters that continue improving the target past GRPO's plateau. Results are reported on Countdown, DeepScaleR, and OpenBookQA.
Critical strengths
- The phenomenon is interesting. RL specialists retaining non-target capabilities runs against the common catastrophic-forgetting narrative.
- The analysis is broad: MLP activation maps, Fisher ratios, gradient cosines, parameter Jaccard overlap, and CKA.
Critical weaknesses
-
The central mechanism is not supported by the authors' own ablations. The claim is that ANTIGEN gains come from recruiting non-target subspaces identified by ascent. But plain LoRA (60.8 ± 2.8) and random addresses (59.5 ± 2.2) are close to ANTIGEN (62.4 ± 1.1), with heavily overlapping error bars. Appendix C.3 shows random addresses and plain LoRA also compress late-layer donor Fisher ratios (by 27% and 18%, versus 16% for ANTIGEN). The authors themselves write that ascent-derived addresses are not required for this effect. This contradicts the subspace-recruitment narrative.
-
Peak-based reporting on the test split. Figure 3 and Table 6 report the maximum over checkpoints on the 500-item evaluation split. Selecting the best checkpoint on the test set inflates methods with noisy trajectories. Checkpoint selection should use a separate validation split, and final or window-mean results should be reported. Comparison windows also vary across arms (1620–1800, 2000, 2400), and arms are shown only where they cover the full window, which adds another selection layer.
Questions for authors
- Is the gap between ANTIGEN and plain LoRA or random addresses statistically significant with more seeds or a paired test?
- Can checkpoints be selected on a held-out validation split, and what are the final-checkpoint results?
2 comments
I have read authors' response and kept my score.
Response to Reviewer dvNq
We thank the reviewer for their careful review of our work. Their review is succinct but identifies our strengths and gaps with great attention to detail. We address these gaps point by point, with new results collected at https://anonymous.4open.science/r/antigen-rebuttal-31EB/, and we hope the reviewer will consider recommending acceptance.
Improvement over our ablations
Q1: Significance with more seeds
Q2: Peaks, windows, and validation