- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the trade-off between "capability acquisition" and "existing capability retention" in offline agent fine-tuning. Through controlled experiments, the authors find that standard full SFT leads to non-target capability degradation, while global Base-KL regularization or update-limiting strategies (such as PSFT) fail to properly resolve this issue. To address this, the authors propose PG-SFT (Privilege-Guided SFT), which leverages the "next action" as privileged information (hint) during training to compute turn-level information gain and adaptively adjust the supervision intensity. Experiments demonstrate that this method maintains or improves target task performance (e.g., on SWE-bench) while substantially preserving performance across other general benchmarks.
Critical strengths
-
Addresses a Crucial Practical Pain Point: Precisely focuses on the "capability tax" or catastrophic forgetting problem faced by large language models during the agent post-training phase, with comprehensive evaluation dimensions covering reasoning, code, tool use, and other benchmarks.
-
Clear and Natural Writing with Clean Visuals: The paper features a rigorous logical structure and fluent, natural expressions.
-
Rigorous Evaluation Protocol: Beyond simple downstream accuracy, the paper incorporates distribution-level diagnostic metrics such as NLL and Base-KL , mutually corroborating the adaptation and forgetting phenomena from both macro and micro perspectives.
Critical weaknesses
-
Insufficient In-Depth Comparison and Theoretical Depth Regarding Cutting-Edge Fundamental Works (Including Ziniu Li's and Tianshuo Liu's Works): Compared to existing literature that dissects generalization and forgetting mechanisms from a more fundamental level (e.g., Ziniu Li et al [1]. studying GEM from the perspective of output diversity collapse, and TS Liu et al [2]. analyzing micro-dynamic perspectives like clipping low-probability tokens to control gradient explosion and provide a generalizable initialization), PG-SFT remains an empirical heuristic design heavily dependent on agent-specific trajectory hints. Compared to these already published foundational works (e.g., Li et al. and Liu et al.), this paper lacks sufficient novelty, and its core motivation is indistinguishable from theirs.
-
Limited Methodological Generalizability: PG-SFT heavily relies on explicit "next action" hints within agent trajectories to compute knowledge gains. This restricts the method's ability to be directly transferred and generalized to general instruction tuning (SFT) or multimodal scenarios that lack explicit action decision trajectories.
-
Limitations in Experiment Scale and Model Backbones: The primary experiments are limited to 4B-scale models (Qwen3.5-4B and Qwen3-4B-Thinking) and a single software engineering agent dataset (Nemotron-SFT-SWE-v3). It remains unclear whether these findings scale to larger parameter models (e.g., 14B) or more diverse agent domains (e.g., web browsing, embodied AI).
-
Potential Biases Introduced by Evaluation Set Selection and Modification: For the target benchmark (SWE-bench), the authors utilize a self-curated/difficulty-screened subset of 89 instances and apply a conditional pass@3 scheduling protocol on the screened subset (Appendices D). This prevents direct comparison with standard leaderboard results and risks non-random impacts on different models due to the screening logic.
[1]. Li Z, Chen C, Xu T, et al. Preserving diversity in supervised fine-tuning of large language models[C]//International Conference on Learning Representations. 2025, 2025: 66127-66154.
[2]. Liu T S, Jia C, Liu H, et al. Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL[C]//Forty-third International Conference on Machine Learning.
Questions for authors
Please see weaknesses.
5 comments
2. W2: Dependence on explicit next-action hints
We thank the reviewer and appreciate this comment. PG-SFT requires an explicit next action in the demonstration and is therefore agent-specific. We want to emphasize that this is a deliberate choice rather than an unexpected limitation. The dependence is structural: in a multi-turn trajectory, an action changes the environment state and the decisions available later; an error at an early turn can make the remaining turns unrecoverable. A response-level or example-level signal cannot identify which turn was resolved by a decision, which is the quantity that sets . In single-turn SFT the turn-level coefficient collapses to an example-level weight, and in multimodal data there is no action sequence to attribute.
3. W3: Experiment scale, backbones, and agent domains
Our experiments currently cover two 4B-scale Qwen backbones and one software-engineering trajectory source (Nemotron-SFT-SWE-v3 with the OpenHands scaffold). We have not run larger models yet, mainly because of compute and time limits, and we will post further results in the discussion if they become available.
Within this regime, we added three control experiments to support our claims: a turn-NLL coefficient calibrated to match PG-SFT's total probability push, a within-trajectory shuffle of the PG-SFT coefficients (seed-42), and a fixed- sweep on Qwen3.5. They test whether a simpler allocation rule or a uniform intensity can reproduce the acquisition–retention profile.
| Backbone | Allocation | SWE-89 | GPQA /198 | BFCL /200 | LCB /120 |
|---|---|---|---|---|---|
| Qwen3.5-4B | PG-SFT | 59 | 150 | 111 | 61 |
| Qwen3.5-4B | Turn NLL | 58 | 138 | 110 | 60 |
| Qwen3.5-4B | Shuffled | 57 | 148 | 111 | 52 |
| Qwen3-4B-Thinking | PG-SFT | 27 | 131 | 110 | 85 |
| Qwen3-4B-Thinking | Turn NLL | 24 | 126 | 106 | 88 |
| Qwen3-4B-Thinking | Shuffled | 24 | 132 | 110 | 87 |
Across both backbones, no simpler rule reproduces this joint profile. NLL allocation is competitive on retention but consistently weaker on GPQA. Shuffled coefficients lose on the target task and, on Qwen3.5, on LCB. The fixed sweep trades LCB for SWE-89 with no single value matching both. PG-SFT therefore gives the most favorable acquisition–retention balance among the compared rules, although not on every individual retention benchmark.
4. W4: The 89-instance subset and the conditional pass@3 protocol
We accept both parts of the concern. The 89-instance scores are not comparable with SWE-bench Lite leaderboard mainly due to computation resource limits. Specifically, the random component is 50 of the 300 Lite instances (seed 42). The screened component is 50 instances chosen from public per-instance leaderboard records and a repeated-sampling study [3,4]. Selection uses external references, not any outcome of the compared objectives. Both components overlap on 11 SWE-bench instances, giving 89 tasks in the end. Screening raises solvability so that degradation on solvable tasks is visible. Currently, full Lite is still running and will be posted if it completes.
[1] Li Z, Chen C, Xu T, et al. Preserving diversity in supervised fine-tuning of large language models. ICLR 2025.
[2] Liu T S, Jia C, Liu H, et al. Clipping low-probability tokens in SFT yields a generalizable initialization for RL. ICML 2026.
[3] SWE-bench experiments
[4] SWE-bench Lite sampling study
Summary of changes
We will revise the paper to (i) add Li et al. and Liu et al. to related work section and clearly distinguish our novelty; (ii) describe PG-SFT as a parameter-free turn-level allocation rule with a probability-mass interpretation, not a theory of forgetting; (iii) add the turn-NLL, shuffled-coefficient, and fixed- ablations and report the trade-off performance; (iv) scope the method to agent trajectories with training-only action information; and (v) label the 89-instance set as diagnostic, report pass@1, and correct the Appendix D sentence on screening. We thank the reviewer for pushing us to make these distinctions explicit.
We sincerely hope our extended experiments and responses have addressed your concerns.
Have you examined high-weight turns to see how much of the probability gain comes from the reasoning itself, and how much comes from tokens that repeat the revealed action? Have you tried other hint recipes, such as revealing only the tool name, to see how they affect these predictions? I am pleased with the additional results and will raise my score accordingly.
We thank the reviewer for the comparison with Li et al. and Liu et al. and for the scope limits. We agree that retention during SFT is not a new motivation, and that the submitted version did not position PG-SFT against these works. Below we explain the difference in signal, granularity, and target, report the controls we added, and state the limits we accept.
1. W1: Relation to Li et al. and Liu et al., and the scope of our claim
Li et al. and Liu et al. explain why SFT degrades: reverse-KL fitting collapses output diversity [1], and low-probability expert tokens produce large per-token updates that accumulate into forgetting [2]. We share the symptom but study a different object: which turns inside a multi-turn agent trajectory should move away from the Base distribution, and by how much. The two lines are complementary: the cited works explain why SFT moves the model, while we study where to spend that movement in agent data.
Li et al. change the objective and PG-SFT changes the target. GEM is derived from a game-theoretic view of reverse-KL minimization with entropy regularization, so the model keeps multiple plausible outputs. PG-SFT has no entropy term and no diversity signal. Its target is a per-token interpolation between the frozen Base distribution and the demonstrated token, and the coefficient comes from a training-only action hint. Diversity preservation may follow from staying near Base, but it is not what we optimize or measure.
Liu et al. change the per-token update while PG-SFT changes the turn coefficient. Liu et al. bound per-token updates by clipping tokens with low Base probability, and evaluate out-of-distribution forgetting and downstream RL performance. PG-SFT neither clips nor discards tokens by Base probability and does not study RL initialization. Our method aggregates the action-conditioned gain over a turn and applies one coefficient to every token in that turn, reasoning and action alike.
Heuristic designs and theoretical contributions. The action hint is not an ad hoc choice. The demonstration's next action is training-time privileged information, and self-distillation methods such as OPSD show that such self-provided signals can supply a learning signal. We use the hint as a measurement instrument rather than a distillation target: it estimates how much of a turn is determined by its decision, and we set the coefficient so that the turn's total probability shift equals that gain. This is why no external annotation or rollout is needed, and why we do not distill the hint-conditioned distribution.
We agree that the paper does not provide a theory of forgetting, and we will say so explicitly. Li et al. and Liu et al. attribute SFT degradation to mode-seeking reverse-KL fitting and low-probability-token updates, respectively, but neither prescribes where supervision should be allocated in a multi-turn trajectory.
Our theoretical contribution is narrower: PG-SFT places the target in a trust region around the Base distribution. Relative to the Base model, the target moves αk(1−bt) probability mass to each demonstrated token, and the turn's total increase in demonstrated-token probability is Gk. The coefficient is a probability-mass budget. The fixed-intensity control matches the aggregate budget, so the downstream difference isolates allocation from amount. In our setting, a control that closely matches aggregate drift and NLL does not reproduce the same retention profile, so a theory of retention must be allocation-aware. We will add this characterization as a short proposition and state the scope of the claim. We will also add both references and the comparison table to related work, rewrite the introduction so that retention alone is not our novelty.