acceptodds
Under review as a conference paper at ICLR 2027

Guided by the HINT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning

Abstract

Supervised fine-tuning (SFT) on offline agent trajectories is the standard approach for training specialized tool-using agents, but forcing models to imitate reasoning and actions token by token may harm other capabilities (e.g., general reasoning, tool calling, code generation) of the base model. In this work, we focus on studying how SFT objectives balance the trade-off between acquiring new capabilities while preserving existing ones better on agent data? By comparing several baselines, standard SFT improves the target benchmark while lowering several non-target benchmark scores; meanwhile, simply constraining distributional drift using KL penalty or limiting the update magnitude did not avoid this regression trend. Motivated by recent token-wise adaptive learning objectives, this work proposed Privilege-Guided SFT (PG-SFT) to leverage turn-level information gain of agent trajectories as indicators to adjust supervision strength. PG-SFT yields a more favorable observed trade-off on the evaluated benchmarks, substantially reducing distributional drift and broad capability degradation at the cost of slight degradation on target-task performance. Our findings suggest that balancing the acquisition–retention trade-off depends not only on whether the model is anchored to its base behavior, but also on where and how strongly supervision should depart from that behavior.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
SFT, Privilege-conditioned guidance, Agent traces

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_re9HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between "capability acquisition" and "existing capability retention" in offline agent fine-tuning. Through controlled experiments, the authors find that standard full SFT leads to non-target capability degradation, while global Base-KL regularization or update-limiting strategies (such as PSFT) fail to properly resolve this issue. To address this, the authors propose PG-SFT (Privilege-Guided SFT), which leverages the "next action" as privileged information (hint) during training to compute turn-level information gain and adaptively adjust the supervision intensity. Experiments demonstrate that this method maintains or improves target task performance (e.g., on SWE-bench) while substantially preserving performance across other general benchmarks.

Critical strengths
  1. Addresses a Crucial Practical Pain Point: Precisely focuses on the "capability tax" or catastrophic forgetting problem faced by large language models during the agent post-training phase, with comprehensive evaluation dimensions covering reasoning, code, tool use, and other benchmarks.

  2. Clear and Natural Writing with Clean Visuals: The paper features a rigorous logical structure and fluent, natural expressions.

  3. Rigorous Evaluation Protocol: Beyond simple downstream accuracy, the paper incorporates distribution-level diagnostic metrics such as NLL and Base-KL , mutually corroborating the adaptation and forgetting phenomena from both macro and micro perspectives.

Critical weaknesses
  1. Insufficient In-Depth Comparison and Theoretical Depth Regarding Cutting-Edge Fundamental Works (Including Ziniu Li's and Tianshuo Liu's Works): Compared to existing literature that dissects generalization and forgetting mechanisms from a more fundamental level (e.g., Ziniu Li et al [1]. studying GEM from the perspective of output diversity collapse, and TS Liu et al [2]. analyzing micro-dynamic perspectives like clipping low-probability tokens to control gradient explosion and provide a generalizable initialization), PG-SFT remains an empirical heuristic design heavily dependent on agent-specific trajectory hints. Compared to these already published foundational works (e.g., Li et al. and Liu et al.), this paper lacks sufficient novelty, and its core motivation is indistinguishable from theirs.

  2. Limited Methodological Generalizability: PG-SFT heavily relies on explicit "next action" hints within agent trajectories to compute knowledge gains. This restricts the method's ability to be directly transferred and generalized to general instruction tuning (SFT) or multimodal scenarios that lack explicit action decision trajectories.

  3. Limitations in Experiment Scale and Model Backbones: The primary experiments are limited to 4B-scale models (Qwen3.5-4B and Qwen3-4B-Thinking) and a single software engineering agent dataset (Nemotron-SFT-SWE-v3). It remains unclear whether these findings scale to larger parameter models (e.g., 14B) or more diverse agent domains (e.g., web browsing, embodied AI).

  4. Potential Biases Introduced by Evaluation Set Selection and Modification: For the target benchmark (SWE-bench), the authors utilize a self-curated/difficulty-screened subset of 89 instances and apply a conditional pass@3 scheduling protocol on the screened subset (Appendices D). This prevents direct comparison with standard leaderboard results and risks non-random impacts on different models due to the screening logic.

[1]. Li Z, Chen C, Xu T, et al. Preserving diversity in supervised fine-tuning of large language models[C]//International Conference on Learning Representations. 2025, 2025: 66127-66154.

[2]. Liu T S, Jia C, Liu H, et al. Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RL[C]//Forty-third International Conference on Machine Learning.

Questions for authors

Please see weaknesses.

5 comments

We thank the reviewer for the comparison with Li et al. and Liu et al. and for the scope limits. We agree that retention during SFT is not a new motivation, and that the submitted version did not position PG-SFT against these works. Below we explain the difference in signal, granularity, and target, report the controls we added, and state the limits we accept.

1. W1: Relation to Li et al. and Liu et al., and the scope of our claim

Li et al. and Liu et al. explain why SFT degrades: reverse-KL fitting collapses output diversity [1], and low-probability expert tokens produce large per-token updates that accumulate into forgetting [2]. We share the symptom but study a different object: which turns inside a multi-turn agent trajectory should move away from the Base distribution, and by how much. The two lines are complementary: the cited works explain why SFT moves the model, while we study where to spend that movement in agent data.

WorkSignalGranularityInterventionPrimary outcome
Li et al. [1]entropy of the output distributionoutput distributionentropy-regularized objective (GEM)diversity, forgetting
Liu et al. [2]Base probability of each tokentokenclip low-probability tokensOOD forgetting, RL initialization
PG-SFTaction-conditioned gain GkG_kassistant turncoefficient αk\alpha_k on the targetacquisition–retention in agent SFT

Li et al. change the objective and PG-SFT changes the target. GEM is derived from a game-theoretic view of reverse-KL minimization with entropy regularization, so the model keeps multiple plausible outputs. PG-SFT has no entropy term and no diversity signal. Its target is a per-token interpolation between the frozen Base distribution and the demonstrated token, and the coefficient comes from a training-only action hint. Diversity preservation may follow from staying near Base, but it is not what we optimize or measure.

Liu et al. change the per-token update while PG-SFT changes the turn coefficient. Liu et al. bound per-token updates by clipping tokens with low Base probability, and evaluate out-of-distribution forgetting and downstream RL performance. PG-SFT neither clips nor discards tokens by Base probability and does not study RL initialization. Our method aggregates the action-conditioned gain over a turn and applies one coefficient to every token in that turn, reasoning and action alike.

Heuristic designs and theoretical contributions. The action hint is not an ad hoc choice. The demonstration's next action is training-time privileged information, and self-distillation methods such as OPSD show that such self-provided signals can supply a learning signal. We use the hint as a measurement instrument rather than a distillation target: it estimates how much of a turn is determined by its decision, and we set the coefficient so that the turn's total probability shift equals that gain. This is why no external annotation or rollout is needed, and why we do not distill the hint-conditioned distribution.

We agree that the paper does not provide a theory of forgetting, and we will say so explicitly. Li et al. and Liu et al. attribute SFT degradation to mode-seeking reverse-KL fitting and low-probability-token updates, respectively, but neither prescribes where supervision should be allocated in a multi-turn trajectory.

Our theoretical contribution is narrower: PG-SFT places the target in a trust region around the Base distribution. Relative to the Base model, the target moves αk(1−bt)\alpha_k(1-b_t) probability mass to each demonstrated token, and the turn's total increase in demonstrated-token probability is GkG_k. The coefficient is a probability-mass budget. The fixed-intensity control matches the aggregate budget, so the downstream difference isolates allocation from amount. In our setting, a control that closely matches aggregate drift and NLL does not reproduce the same retention profile, so a theory of retention must be allocation-aware. We will add this characterization as a short proposition and state the scope of the claim. We will also add both references and the comparison table to related work, rewrite the introduction so that retention alone is not our novelty.

2. W2: Dependence on explicit next-action hints

We thank the reviewer and appreciate this comment. PG-SFT requires an explicit next action in the demonstration and is therefore agent-specific. We want to emphasize that this is a deliberate choice rather than an unexpected limitation. The dependence is structural: in a multi-turn trajectory, an action changes the environment state and the decisions available later; an error at an early turn can make the remaining turns unrecoverable. A response-level or example-level signal cannot identify which turn was resolved by a decision, which is the quantity that sets αk\alpha_k. In single-turn SFT the turn-level coefficient collapses to an example-level weight, and in multimodal data there is no action sequence to attribute.

3. W3: Experiment scale, backbones, and agent domains

Our experiments currently cover two 4B-scale Qwen backbones and one software-engineering trajectory source (Nemotron-SFT-SWE-v3 with the OpenHands scaffold). We have not run larger models yet, mainly because of compute and time limits, and we will post further results in the discussion if they become available.

Within this regime, we added three control experiments to support our claims: a turn-NLL coefficient αkNLL=min⁡!(1,λ(1−e−NLLk))\alpha_k^{\mathrm{NLL}}=\min\\!\left(1,\lambda\left(1-e^{-\mathrm{NLL}_k}\right)\right) calibrated to match PG-SFT's total probability push, a within-trajectory shuffle of the PG-SFT coefficients (seed-42), and a fixed-α\alpha sweep on Qwen3.5. They test whether a simpler allocation rule or a uniform intensity can reproduce the acquisition–retention profile.

BackboneAllocationSWE-89GPQA /198BFCL /200LCB /120
Qwen3.5-4BPG-SFT5915011161
Qwen3.5-4BTurn NLL5813811060
Qwen3.5-4BShuffled α\alpha5714811152
Qwen3-4B-ThinkingPG-SFT2713111085
Qwen3-4B-ThinkingTurn NLL2412610688
Qwen3-4B-ThinkingShuffled α\alpha2413211087

Across both backbones, no simpler rule reproduces this joint profile. NLL allocation is competitive on retention but consistently weaker on GPQA. Shuffled coefficients lose on the target task and, on Qwen3.5, on LCB. The fixed sweep trades LCB for SWE-89 with no single value matching both. PG-SFT therefore gives the most favorable acquisition–retention balance among the compared rules, although not on every individual retention benchmark.

4. W4: The 89-instance subset and the conditional pass@3 protocol

We accept both parts of the concern. The 89-instance scores are not comparable with SWE-bench Lite leaderboard mainly due to computation resource limits. Specifically, the random component is 50 of the 300 Lite instances (seed 42). The screened component is 50 instances chosen from public per-instance leaderboard records and a repeated-sampling study [3,4]. Selection uses external references, not any outcome of the compared objectives. Both components overlap on 11 SWE-bench instances, giving 89 tasks in the end. Screening raises solvability so that degradation on solvable tasks is visible. Currently, full Lite is still running and will be posted if it completes.

[1] Li Z, Chen C, Xu T, et al. Preserving diversity in supervised fine-tuning of large language models. ICLR 2025.
[2] Liu T S, Jia C, Liu H, et al. Clipping low-probability tokens in SFT yields a generalizable initialization for RL. ICML 2026.
[3] SWE-bench experiments
[4] SWE-bench Lite sampling study

Summary of changes

We will revise the paper to (i) add Li et al. and Liu et al. to related work section and clearly distinguish our novelty; (ii) describe PG-SFT as a parameter-free turn-level allocation rule with a probability-mass interpretation, not a theory of forgetting; (iii) add the turn-NLL, shuffled-coefficient, and fixed-α\alpha ablations and report the trade-off performance; (iv) scope the method to agent trajectories with training-only action information; and (v) label the 89-instance set as diagnostic, report pass@1, and correct the Appendix D sentence on screening. We thank the reviewer for pushing us to make these distinctions explicit.

We sincerely hope our extended experiments and responses have addressed your concerns.

Have you examined high-weight turns to see how much of the probability gain comes from the reasoning itself, and how much comes from tokens that repeat the revealed action? Have you tried other hint recipes, such as revealing only the tool name, to see how they affect these predictions? I am pleased with the additional results and will raise my score accordingly.

Reviewer_dpo6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the trade-off between acquiring task-specific capabilities and preserving general capabilities during supervised fine-tuning of language model agents. In this paper, the authors argue that conventional SFT uniformly supervises all demonstrated tokens, potentially introducing unnecessary changes to the model's original behavior. To address this, the paper introduces Privilege-Guided Supervised Fine-Tuning (PG-SFT), which uses information from the demonstrated next action as a training-time hint to determine the supervision intensity for each agent turn. The method measures how much the hint increases the base model's probability of demonstrated tokens and uses this signal to interpolate between the original model distribution and the demonstrated targets. Experiments on Qwen3.5-4B and Qwen3-4B-Thinking evaluate acquisition on a SWE-bench Lite subset and retention on GPQA, BFCL, and LiveCodeBench. The results suggest that PG-SFT achieves a more favorable acquisition-retention trade-off than the tested baselines, with additional analyses examining distributional drift and the importance of adaptive supervision.

Critical strengths
  1. It is an important and practically relevant problem. Studying the acquisition-retention trade-off specifically for tool-using agents is valuable, since agent trajectories contain reasoning, tool calls, and implementation details that may not contribute equally to learning.
  2. Interesting use of privileged information for adaptive supervision. The proposed use of a future action as privileged information to estimate turn-level learning intensity is intuitive and potentially useful. Rather than directly distilling from the hint-conditioned model, PG-SFT uses the resulting probability gain to modulate supervision. This provides an interesting alternative to conventional token-level weighting and uniform KL regularization.
  3. Empirical improvements look good.
  4. The comparison of learning objectives and distributional behavior is also quite useful.
Critical weaknesses
  1. I think the baseline comparisons are not sufficiently controlled. A significant concern is that several baselines use substantially different optimization budgets and hyperparameters. I looked at the parameters in Appendix B and it seems like they all have different settings so it raises my concern about whether the comparisons are well controlled or not.
  2. There are also insufficient ablations to establish the importance of the privileged action hint. The paper attributes PG-SFT's improvements to action-informed supervision allocation, but it does not sufficiently demonstrate that the proposed coefficient captures decision-relevant information rather than a more general measure of token predictability. For example, the method could be compared against randomly permuted turn coefficients, coefficients based on ordinary turn-level perplexity, alternative confidence-based signals, or coefficients computed from shuffled action hints. An ablation reversing the coefficient allocation would also help determine whether the proposed direction of weighting is genuinely important.
  3. The fixed-intensity control does not conclusively establish a better acquisition-retention frontier.
  4. There is limited evaluation and nonstandard benchmark construction. The experimental validation is restricted to two closely related Qwen backbones, both at the 4B scale, and a single software-engineering agent domain. This limits conclusions about whether the proposed supervision principle generalizes across model families, sizes, or other agent environments.
  5. Lastly, the experiments appear to use a single training seed, without uncertainty estimates across independent runs. This is particularly concerning because several reported performance differences are relatively small.
Questions for authors
  1. Can the authors provide budget-matched comparisons with SFT+Base-KL and PSFT, including meaningful sweeps over KL coefficients, learning rates, and optimization steps?
  2. How does PG-SFT compare against randomly shuffled turn coefficients, perplexity-based coefficients, or coefficients computed using incorrect or shuffled action hints? Can the authors demonstrate that the proposed action-conditioned gain is more informative than simpler turn-level uncertainty measures?
  3. Could the authors report the full acquisition-retention curve obtained by sweeping the fixed coefficient rather than evaluating only α=0.35\alpha=0.35?
  4. Can the authors evaluate PG-SFT on the full SWE-bench Lite benchmark using a uniform sampling budget, and report paired confidence intervals or multiple training seeds?
4 comments

We thank the reviewer for identifying these threats to validity. We agree that the submitted paper did not sufficiently distinguish budget-matched comparisons from reference baselines, and that a single fixed-intensity point cannot establish a better acquisition–retention frontier. Therefore, below we separate the evidence into three levels: controlled comparisons within the PG-SFT target family, cross-objective comparisons with corrected budgets, and scope and uncertainty. We also correct several factual errors in the appendix and in the definition of the fixed-intensity control.

1. W1/Q1: Budget-matched comparisons and hyperparameters

We first correct the training-budget table. The Qwen3 SFT+Base-KL entry of 32 steps in Appendix B is a transcription error: the run used one epoch (156 steps), the same as SFT and PG-SFT. The table below lists the actual budgets.

BackboneObjectiveTrajectoriesEpochsStepsLR
Qwen3.5-4BSFT1,0001325×10−55\times10^{-5}
Qwen3.5-4BSFT+Base-KL1,0001325×10−55\times10^{-5}
Qwen3.5-4BPG-SFT1,0001325×10−55\times10^{-5}
Qwen3.5-4BFixed intensity1,0001325×10−55\times10^{-5}
Qwen3.5-4BPSFT1,00039610−610^{-6}
Qwen3-4B-ThinkingSFT4,99211565×10−55\times10^{-5}
Qwen3-4B-ThinkingSFT+Base-KL4,99211565×10−55\times10^{-5}
Qwen3-4B-ThinkingPG-SFT4,99211565×10−55\times10^{-5}
Qwen3-4B-ThinkingPSFT4,992346810−610^{-6}

First, all non-PSFT arms are budget-matched within each backbone. They use the same trajectories, global batch, learning rate, schedule, LoRA configuration, and number of optimizer steps; only the supervision objective and its objective-specific coefficient differ.

We keep PSFT at the low-learning-rate, multi-epoch recipe of the original method (LR 10−610^{-6}: 3 epochs, 96 steps on Qwen3.5 and 468 on Qwen3). A 32/156-step variant at 5×10−55\times10^{-5} would change the trust-region behavior and would no longer represent PSFT. We therefore label PSFT as a referencing baseline.

2. W2/Q2: What the coefficient captures

We really appreciate this comment, as comparisons between shuffling coefficients and other more intuitive baselines would support our statement that "action-conditioned coefficient captures decision-relevant information."

Ordinary turn-level uncertainty. We replace the action-conditioned gain with

αkNLL=min⁡!(1,λ(1−e−NLLk)),\alpha_k^{\mathrm{NLL}}=\min\\!\left(1,\lambda\left(1-e^{-\mathrm{NLL}_k}\right)\right),

where λ\lambda is calibrated to match PG-SFT's total training budget. This is the same as the perplexity-based control (PPLk=eNLLk\mathrm{PPL}_k=e^{\mathrm{NLL}_k}).

BackboneCoefficientSWE /89GPQA /198BFCL /200LCB /120
Qwen3.5-4BPG-SFT5915011161
Qwen3.5-4BNLL-based5813811060
Qwen3-4B-ThinkingPG-SFT2713111085
Qwen3-4B-ThinkingNLL-based2412610688

On Qwen3.5, ordinary turn uncertainty closely matches PG-SFT on SWE, BFCL, and LCB but is lower on GPQA. On Qwen3, the pattern is mixed: NLL is lower on SWE and BFCL, and slightly higher on LCB.

We also measure the alignment between the two coefficient. Spearman correlations between αNLL\alpha^{\mathrm{NLL}} and αPG\alpha^{\mathrm{PG}} are −0.207-0.207 for Qwen3.5 and −0.057-0.057 for Qwen3.

Turn-weights shuffle control. We permute the PG-SFT coefficients within each trajectory using a fixed seed-42 permutation. This preserves each trajectory's coefficients and changes only which assistant turn receives each coefficient.

BackboneCoefficientSWE /89GPQA /198BFCL /200LCB /120
Qwen3.5-4BPG-SFT5915011161
Qwen3.5-4BShuffled-α\alpha5714811152
Qwen3-4B-ThinkingPG-SFT2713111085
Qwen3-4B-ThinkingShuffled-α\alpha2413211087

Shuffling weights designation to turns slightly lower the performance on target benchmarks but retain on the rest benchmarks.

3. W3/Q3: The sweeping fixed-intensity

We appreciate the suggestions on sweeping across fixed-coefficients to support our statement. Accordingly, we completed a sweeping coefficients α∈0.2,0.5,0.7\alpha\in\\{0.2,0.5,0.7\\} experiment, and the acquisition-retention results of Qwen-3.5 are:

Fixed coefficientSWE /89GPQA /198BFCL /200LCB /120
0.20.25115011172
0.50.55514711159
0.70.75314810652

The completed fixed sweep does not produce a coefficient that dominates PG-SFT.

4. W4: Scope across models and agent environments

We agree that two Qwen backbones at the 4B scale and one software-engineering agent environment do not establish generalization across model families, model sizes, or agent domains. With limited time during the rebuttal period, we will first revise the abstract, experiments, and conclusion to scope our empirical claims. Follow up experiments will be conducted later.

5. W5/Q4: Uncertainty, full Swebench Lite, and independent seeds

We thank the reviewer for pointing this out. We did not run the full 300-instance SWE-bench Lite evaluation for every model, mainly because of the time cost of the agentic harness. We are continuing this evaluation and will post the results if they complete before the deadline. The 89-instance set is therefore a diagnostic subset, not a full SWE-bench Lite estimate.

The subset was constructed without using any outcome from the models compared in this paper. The random component is a uniform sample of 50 Lite instances drawn with seed 42. For the easy component, we first used public per-instance evaluation records from the SWE-bench experiments repository [1] and the SWE-bench Lite sampling study [2] to estimate instance solvability. From that pool we sampled 50 instances. The two components overlap on 11 instances, giving 89 unique instances. Because SWE-bench is difficult for 4B models, the easy component is included to increase discriminability, in particular to detect whether a method degrades on instances that are solvable in principle. It is not selected to favor any particular objective. The resulting set does show separation among the compared objectives, which is why we use it as a diagnostic comparison rather than a leaderboard-style estimate.

[1] SWE-bench experiments
[2] SWE-bench Lite sampling study

Summary of changes

We will revise the paper to (i) correct the budget table and the coefficient definition; (ii) narrow the coefficient claim to non-redundancy and alignment sensitivity; (iii) report the fixed sweep as a trade-off rather than a better frontier; (iv) scope the empirical claims to the evaluated backbones and domain; and (v) label the 89-instance set as a diagnostic and report uncertainty honestly. We thank the reviewer for pushing us to make these distinctions explicit.

We sincerely hope our extended experiments and responses have addressed your concerns.

Thank you for the clarifications and additional ablations. I will maintain my score, as the missing uncertainty estimates and incomplete baseline hyperparameter sweeps leave it unclear whether the reported improvements are reliable, particularly given the small and mixed gains over the new controls.

Thank you for your response. We are still doing some experiments at the moment, and hopefully will finish them before the rebuttal deadline. Please stay tuned for follow-up updates in the coming days. We’ll post them here as soon as they become available.

Reviewer_rH4HOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how supervised fine-tuning (SFT) affects the trade-off between acquiring software-agent capabilities and retaining other capabilities during offline fine-tuning. The main contribution of the paper is a method called PG-SFT that adjusts the strength with which a model learns from each recorded assistant turn. A frozen copy of the original model scores the demonstrated reasoning and action twice: once with the normal conversation history and once with an added hint that reveals the demonstrated action. If the hint makes the recorded text much more predictable, the turn receives a higher imitation weight. If the hint helps little, training puts more weight on preserving the original model’s predictions and, as a result, deteriorates less base model capabilities. This weight applies to every token in the turn. The student trains on the normal conversation history; the hint only helps calculate the weight and is unnecessary when the trained model solves new tasks.

The authors compare PG-SFT with standard fine-tuning and regularized alternatives on two 4B models, measuring software-agent performance and the retention of reasoning, tool use, and coding abilities. PG-SFT preserves more capabilities than standard fine-tuning while retaining most target-task gains. However, the evidence comes from one training seed and limited benchmark subsets. Some baselines use different training schedules, and the controls do not establish whether action-derived weights outperform simpler weighting strategies.

Critical strengths

The comparison against ordinary SFT provides promising evidence. Those arms share the training configuration within each backbone. On Qwen3.5, PG-SFT matches SFT’s 59/89 target successes while improving GPQA from 133 to 150, BFCL from 100 to 111, and LiveCodeBench from 48 to 61. This suggests a potentially useful intervention for preserving capabilities during agent adaptation.

The method makes a concrete distinction between privileged scoring and privileged distillation. The action hint selects supervision strength; the student does not directly imitate a hint-conditioned teacher. Combining downstream evaluation with held-out likelihood and KL diagnostics is also useful for examining the acquisition–retention trade-off.

Critical weaknesses

The proposed mechanism is insufficiently isolated. Equation 10 and Section 4.4 describe different averaging rules for the fixed-weight control. No shuffled-weight or simpler weighting control establishes the specific benefit of the action hint, and the fixed control outperforms PG-SFT on two retention benchmarks.

The comparisons leave robustness unclear. Several baselines use different training schedules—for example, 32 versus 156 steps—and receive limited hyperparameter exploration. One training seed, a selected 89-task target benchmark, and ambiguities in training and evaluation protocols further limit confidence in the conclusions.

Questions for authors

Was the fixed coefficient calculated using Equation 10’s probability-room-weighted mean or Section 4.4’s token-count-weighted mean? What are both values, and would correcting the matching criterion change the comparison?

Does PG-SFT retain an advantage against fixed-coefficient and learning-rate sweeps under matched training schedules? Does the privileged signal outperform shuffled turn coefficients or a simpler confidence-based allocation?

What training data exposure and optimizer-step count were actually used for Qwen3 PSFT? How should the “three epochs / 96 steps” statement in Table 6 be reconciled?

What exact attempt budget applies to each SWE-bench subset and method, including the overlapping instances? Do the conclusions hold separately on the random subset, and what uncertainty is observed across repeated runs?

3 comments

We thank the reviewer for a careful and constructive reading. Two of the points identify definitional errors and we've corrected both. The remaining points ask us to separate a controlled comparison from a reference baseline and to describe the evaluation set honestly. The following are our point-to-point responses.

1. W1/Q1: Corrected averaging rule to Eq. 10's probability-weighted mean

We thank the reviewer for catching this. The two averaging rules should never have been used interchangeably, and we accept such an inconsistency.

The submitted control used the token-count-weighted average, whereas Eq. 10 defines the probability-room-weighted mean that matches PG-SFT's total probability push. Matching ∑tα(1−bt)=∑kαkHk\sum_t \alpha(1-b_t)=\sum_k\alpha_kH_k gives

=\frac{527{,}387.442}{1{,}770{,}005.912} =0.297958\approx\mathbf{0.298},$$ so the coefficient that is matched to PG-SFT in the Eq. 10 sense is $0.298$, not $0.35$. We will report $0.298$ as the Eq. 10-matched control. We re-ran the control at the corrected intensity with the same 1,000 trajectories, LoRA configuration, and 32-step schedule. The result shows a slightly weaker fit to the training data, and a correspondingly smaller drift from Base. In general, the corrected coefficient does not overturn the comparison in Section 4.4. | Objective | SWE-89 | GPQA /198 | BFCL /200 | LCB /120 | |---|---:|---:|---:|---:| | PG-SFT | 59 | 150 | 111 | 61 | | Fixed $\alpha=0.35$ (token-weighted) | 50 | 140 | 114 | 72 | | Fixed $\alpha=0.30$ (Eq. 10: $\bar\alpha_H=0.298$) | 49 | 130 | 106 | 69 | ### 2. W1/Q2: Added fixed, shuffled, and confidence-based controls #### Fixed-coefficient sweep We thank the reviewer for pressing on this. We added a sweeping-$\alpha$ control experiments using the same Qwen3.5 1K data, LoRA configuration, and 32-step schedule as PG-SFT, and $\alpha$ is the weight on the demonstration term as in Eq. 10. | Objective | SWE-89 | GPQA /198 | BFCL /200 | LCB /120 | |---|---:|---:|---:|---:| | PG-SFT | **59** | **150** | **111** | 61 | | Fixed $\alpha=0.2$ | 51 | **150** | **111** | **72** | | Fixed $\alpha=0.5$ | 55 | 147 | **111** | 59 | | Fixed $\alpha=0.7$ | 53 | 148 | 106 | 52 | Lowering the coefficient to $0.2$ matches PG-SFT on GPQA and BFCL and buys 11 LCB instances, but resolves eight fewer SWE-89 instances. Raising it to $0.5$ or $0.7$ is weaker than PG-SFT on both sides. A fixed coefficient can therefore be tuned toward retention or toward acquisition, but no single value reproduces PG-SFT's combination. We will present phenomenon as an acquisition–retention trade-off and state in the revision that one fixed-intensity control cannot settle the question. #### Shuffled turn coefficients We thank the reviewer for this meaningful suggestion. We permute the PG-SFT coefficients within each trajectory using a fixed seed-42 permutation, which preserves changes only which assistant turn receives each coefficient. | Backbone | Objective | SWE-89 | GPQA /198 | BFCL /200 | LCB /120 | |---|---|---:|---:|---:|---:| | Qwen3.5-4B | PG-SFT | 59 | 150 | 111 | 61 | | Qwen3.5-4B | Shuffled $\alpha$ | 57 | 148 | 111 | 52 | | Qwen3-4B-Thinking | PG-SFT | 27 | 131 | 110 | 85 | | Qwen3-4B-Thinking | Shuffled $\alpha$ | 24 | 132 | 110 | 87 | On Qwen3.5 the shuffled arm loses two SWE-89 and nine LCB instances, with GPQA within two instances and BFCL unchanged. on Qwen3, target SWE-bench drops by 3 samples. Therefore, the coefficients are not interchangeable across turns. Yet, since the "learning process" still applies regardless the coefficients assigned, shuffled-$\alpha$ may still have a certain level of improvement on the results. #### A simpler confidence-based coefficient We also tested the simpler turn-level signal the reviewer has in mind. We replace the action-conditioned gain with $$\alpha_k^{\mathrm{NLL}}=\min\\!\left(1,\lambda\left(1-e^{-\mathrm{NLL}_k}\right)\right),$$ where $\lambda$ is calibrated before training to match PG-SFT's aggregate weight assignment. Because $\mathrm{PPL}=e^{\mathrm{NLL}}$, this control is a monotonic transformation of turn perplexity. NLL and PPL are not two independent alternatives. | Backbone | Coefficient | SWE-89 | GPQA /198 | BFCL /200 | LCB /120 | |---|---|---:|---:|---:|---:| | Qwen3.5-4B | PG-SFT | 59 | 150 | 111 | 61 | | Qwen3.5-4B | NLL-based | 58 | 138 | 110 | 60 | | Qwen3-4B-Thinking | PG-SFT | 27 | 131 | 110 | 85 | | Qwen3-4B-Thinking | NLL-based | 24 | 126 | 106 | 88 | The Spearman correlation between the two coefficients is $-0.207$ on Qwen3.5 and $-0.057$ on Qwen3. Therefore, hint-guidance is a different signal than the PPL-guidance. On Qwen3.5 the NLL arm almost matches PG-SFT . On Qwen3, it is lower on three benchmarks and slightly higher on LCB. We interpret the trade-off gap as evidence that the privileged signal carries information a model-internal confidence score does not.

3. W2/Q3: Corrections of training configurations and settings

We thank the reviewer for checking Table 6. The entry was wrong and the reviewer's reading of it was correct.

"3 epochs / 96 steps" was a transcription error. 96 steps belongs to the Qwen3.5 setting: 1,000 trajectories at global batch 32 give 32 steps per epoch, so three epochs is 96 steps. Qwen3 uses 4,992 trajectories, so one epoch is 156 steps and three epochs is 468. The table below lists the actual contracts. Table 6 in the paper will carry these corrected numbers.

BackboneObjectiveTrajectoriesEpochsOptimizer stepsLearning rate
Qwen3.5-4BSFT / SFT+Base-KL / PG-SFT1,0001325×10−55\times10^{-5}
Qwen3.5-4BPSFT1,00039610−610^{-6}
Qwen3-4B-ThinkingSFT / SFT+Base-KL / PG-SFT4,99211565×10−55\times10^{-5}
Qwen3-4B-ThinkingPSFT4,992346810−610^{-6}

Qwen3 PSFT therefore sees all 4,992 trajectories three times, and there is no inconsistency in the actual data exposure.

On the related concern about budgets: the non-PSFT arms are budget-matched within each backbone. PSFT is deliberately different. It follows the original method's conservative recipe (LR 10−610^{-6} and 10 epochs in Zhu et al., versus 2×10−52\times10^{-5} for their SFT baseline), and we use the same learning rate with three epochs. A 32- or 156-step version at 5×10−55\times10^{-5} would change the trust-region behavior and would no longer represent PSFT. We therefore keep PSFT as a reference baseline rather than presenting it as a budget controlled comparison.

4. W2/Q4: Justification of the 89-instance subset evaluation

We thank the reviewer for pressing on this, and we will describe the construction and status of the evaluation set in the Appendix.

We did not run the full 300-instance SWE-bench Lite evaluation for every model, mainly because of the time cost of the agentic harness. That evaluation is continuing, and we will post the results if they complete before the discussion period closes.

The subset was constructed without using any outcome from the models compared in this paper. The random component is a uniform sample of 50 Lite instances drawn with seed 42. For the easy component, we first used public per-instance evaluation records from the SWE-bench experiments repository [1] and the SWE-bench Lite sampling study [2] to estimate instance solvability. From that 300 sample pool we sampled 50 instances with seed 42. The two components overlap on 11 instances, so the deduplicated set has 50+50−11=8950+50-11=89 unique tasks. Because SWE-bench is difficult for 4B-scale models, the easy component increases distinguishability. In particular it reveals whether a method degrades on instances that are already solvable, and it was not selected to favor any particular objective.

The metric in our main comparison is pass@3: each instance receives up to three attempts and counts as resolved if at least one succeeds, so these numbers measure coverage of solvable tasks rather than first-attempt accuracy (which is usually the target of reinforcement learning stage). We will state this in the table caption and label the set as diagnostic.

[1] SWE-bench experiments
[2] SWE-bench Lite sampling study

Summary of changes

We will revise the paper to (i) report Eq. 10's probability-weighted coefficient (0.2980.298) as the matched fixed-intensity control and the token-weighted 0.350.35 as a sensitivity; (ii) add the fixed-coefficient sweep, the within-trajectory shuffle, and the NLL/PPL coefficient; (iii) present the sweep as an acquisition–retention trade-off rather than a better frontier, and narrow the mechanism claim to non-redundancy and alignment sensitivity; (iv) correct the step counts in Table 6 and label PSFT as a reference, non-budget-matched baseline; and (v) describe the 89-instance set as a diagnostic subset, state the pass@3 metric explicitly, and treat one- or two-instance differences as noise. We thank the reviewer for pushing us to make these distinctions explicit.

We hope our extended experiments and responses above have addressed your concerns.

Hi reviewer rH4H: Since the rebuttal period is approaching the end, we'd like to kindly ask whether you've gone through our new results and whether they've addressed your previous concerns. We understand that you might be busy with other reviewing tasks and your own submissions, but we sincerely appreciate your latest comments on our rebuttal contents and your willingness of updating the score. Thank you so much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.