- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper defines learning as the performance change on the target task and forgetting as the equally weighted mean performance change across five non-target tasks. For each algorithm, checkpoints from different learning rates and training stages form a point set that is compared through its nondominated frontier and hypervolume relative to a fixed nadir point. Experiments cover Qwen2.5-1.5B-Instruct and LLaMA3.1-8B-Instruct, with Countdown and MMLU as target tasks. Additional studies examine KL regularization, LoRA rank, TrSFT clipping, degree of off-policy training, data scale, ten-epoch training, and recovery from forgetting.
Critical strengths
- The Pareto perspective addresses the right evaluation problem. Relative performance changes with learning rate and stopping time, so a single checkpoint can yield a misleading ranking. The four model–task settings demonstrate this clearly.
- The diagnostic experiments are thorough. The methods use the same filtered query sets, and the paper examines teacher traces, rollout reuse, data scale, extended training, earlier checkpoints, and benchmark-level sources of forgetting. These analyses make early-stage forgetting a credible and important empirical finding.
Critical weaknesses
- The frontiers and rankings lack statistical support. Several hypervolume gaps are only at the third or second decimal place, yet the paper reports no repeated runs, bootstrapped frontiers, hypervolume confidence intervals, or rank-stability probabilities. The adaptive stopping rule over learning rates also affects the estimated frontiers.
- The cost definition is not aligned with algorithmic budgets. Averaging five benchmarks equally can obscure domination by a single benchmark. RL, OPD, SFT, and LoRA also differ in rollout generation, batch size, teacher inference, and trainable parameter count. Without frontiers under matched FLOPs, tokens, or wall-clock time, the study cannot establish which method is preferable at equal cost.
Questions for authors
- If each main setting were run with 3–5 independent seeds and jointly bootstrapped over evaluation examples and training seeds, would the Pareto dominance relations, hypervolume rankings, and the claim that 83.35% of forgetting occurs before the first checkpoint remain stable?
- How do the frontiers for SFT, GRPO, REINFORCE, SDFT, and LoRA change under matched training FLOPs, generated tokens, or wall-clock time? Please include teacher and rollout costs, and report leave-one-benchmark-out or weighting-sensitivity analyses.
4 comments
Author Resposne 2/3
The cost definition is not aligned with algorithmic budgets. RL, OPD, SFT, and LoRA also differ in rollout generation, batch size, teacher inference, and trainable parameter count. Without frontiers under matched FLOPs, tokens, or wall-clock time, the study cannot establish which method is preferable at equal cost.
The cost studied in the paper is degradation of non-target capabilities. For this study, we do not need a claim about equal-compute optimality to establish the observed learning–forgetting trade-offs.
Matching compute would require accounting for SFT demonstration generation, RL rollouts, distillation teacher/student inference, and full-tuning versus LoRA update costs. It also requires choosing how to amortize demonstrations and whether to charge hyperparameter search. Therefore, equal compute implies that training data has to differ across algorithms. As such, with compute cost introduced as an additional aspect, is it not possible to have a fair comparison in every sense.
Author Response 3/3
Averaging five benchmarks equally can obscure domination by a single benchmark.
Appendix I, which decomposes forgetting by benchmark both along selected Pareto-front points and along training trajectories, directly addresses aggregation. The composition of forgetting therefore depends on the target task, with a particularly substantial safety contribution when MMLU is the target. Table 26 attributes 36.53% of Countdown-target forgetting to MMLU and 20.05% to MATH; for MMLU targets, WildJailbreak contributes 65.91% and IFEval 18.77%. Tables 27–28 give per-setting breakdowns. We will highlight these findings.
We do not establish that method rankings are invariant to changing the benchmark weights, but neither is such invariance required for the reported macro-average comparison. Equal weights define the evaluated objective, without implying equal realized contributions or preservation of every capability.
Please include teacher and rollout costs, and report leave-one-benchmark-out or weighting-sensitivity analyses.
For benchmark sensitivity, we report the each non-target benchmark's contributions to forgetting in Appendix I. We additionally evaluate benchmark omission on its 12 Countdown trajectories (two models × three algorithms × two learning rates). After each omission, we recompute the four-benchmark mean forgetting magnitude, d(t) = max(0, −ΔF(t)), at every saved checkpoint, calculate each run's d(first) / max_t d(t), and average across runs. Runs with zero maximum forgetting have undefined ratios and are excluded.
| Target | Omitted non-target benchmark | Mean first-checkpoint share of maximum within-run forgetting | Runs included |
|---|---|---|---|
| Countdown | MMLU | 78.6% | 12 |
| Countdown | IFEval | 79.8% | 12 |
| Countdown | MATH | 83.7% | 11 |
| Countdown | WildGuardTest | 81.1% | 12 |
| Countdown | WildJailbreak | 83.2% | 12 |
The early forgetting conclusion holds even under leave-one-benchmark-out settings. The first checkpoint accounts for 78.6%–83.7% of maximum within-run forgetting on average across these Countdown runs. Only MATH omission excludes a zero-forgetting run.
Dear Reviewer,
With the author response window closing soon, we would deeply appreciate it if you could go through our responses and reconsider your rating if they address your concerns.
Thank you again for your time and effort in reviewing our paper.
Best regards,
The Paper Authors.
Author Response 1/3
Thank you for your review. We address each point below.
The reported rankings describe the evaluated checkpoint sets; the paper does not establish that every small HV difference is statistically significant. Our main evidence is the different frontier shapes and attainable learning ranges in Figures 1–3, supported by trajectories in Figures 5 and 16–23.
While repeated runs are prohibitively expensive, we do provide results that address sensitivity to the tested data sizes and training durations. Approximately 6 learning rates × 10 algorithms × 2 models × 2 datasets require 240 training runs; 5 checkpoints evaluated on 6 benchmarks yield roughly 7,200 evaluations. Replication exceeds our compute budget. Existing checks vary SFT demonstrations (Appendix F.1.1), increase Countdown queries from 4,161 to 10,000 and MMLU queries from 9,645 to 12,000 for GRPO and REINFORCE (F.2.5), and extend Qwen/Countdown training from two to ten epochs for SFT, GRPO, and SDFT, including constant-learning-rate variants (F.2.6). More data has mixed effects without overturning the LLaMA comparison; longer training adds little forgetting, and two-epoch peak learning is close to the ten-epoch peak.
Repeating training and evaluation runs will be prohibitively expensive due to the number of algorithms, hyperparameters, models, datasets, and intermediate checkpoints, and evaluation benchmarks. However, we did have additional evidence for the early forgetting phenomenon.
Appendix F.3 reports 83.75%, 78.07%, 83.72%, and 87.87% first-checkpoint forgetting shares, averaging 83.35%. Figures 24–25 show most forgetting already occurs by approximately 10 steps for SFT/SDFT and 16 for GRPO on Qwen/Countdown. Figures 14–15 extend training to ten epochs with little additional forgetting. Appendix F.3 identifies exceptions to the timing observation.
The learning-rate stopping rule is validated by our pilot study on SFT. Across both Countdown backbones, we also tested
1e-7,5e-7,1e-3, and5e-3. The latter two exceed the stopping rate of5e-4; their checkpoints learn less and forget more than those at5e-4and are dominated by existing frontier points.Nevertheless, it is not possible to fully recover the true Pareto front, and our empirical fronts are approximations.