- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Gradient-guided Fine-Tuning (GFT), which learns from worked solutions through an activation objective within the student model. The student replays both a reference solution and its own generated reasoning with the reference answer substituted in. At each decoder layer, GFT penalizes the positive projection of their activation difference onto a detached, filtered gradient of the reference-answer negative log-likelihood. The method also studies cross-entropy on answer-extraction failures and KL regularization to the best-validation checkpoint on student-generated contexts. Experiments with three Qwen3 sizes on GSM8K and two MATH subjects show improvements over the reported SFT baselines, while student-context KL substantially reduces post-peak generation collapse.
Critical strengths
-
An interesting way to use reference solutions without an additional teacher network. The same-student replay construction avoids cross-model representation matching, while the one-sided projection gives a precise alternative to matching complete activation vectors. The paper carefully specifies prediction positions, stop-gradient operations, normalization, and update ordering, and explains the local behavior of the objective in Eq. (8). Inference uses ordinary generation without an additional model or activation-matching procedure.
-
Substantial empirical gains supported by a useful stability analysis. Every reported GFT configuration on the two MATH subjects exceeds its corresponding SFT baseline. A single fixed configuration, uniform weighting with activation share 1 and no KL, also beats SFT in all nine task/model combinations in Table 1, so the overall improvement is not confined to the best configuration for each task. The study reports validation trajectories and post-peak behavior: the number of runs with a validation drop exceeding 20 percentage points falls from 47/54 without KL to 1/54 with student KL. Both the accuracy results and the stabilization strategy warrant attention.
Critical weaknesses
-
The experiments do not isolate the benefit of the central reference-guided activation objective. Eq. (8) shows that an active term follows the filtered answer-NLL gradient, with the reference determining the gate and update magnitude. Sweeping layer weights, extraction-failure CE, and KL does not establish whether this construction improves over directly optimizing answer NLL on the same student-prefix replays. A matched answer-only CE baseline using the same rollouts and KL regularizer, together with an ablation of the one-sided projection, would distinguish the proposed contribution from the benefits of on-policy answer supervision and regularization. This is the main issue behind my recommendation.
-
The baseline behavior makes the largest gains difficult to attribute. Section 5.1 reports that 44% of the Qwen3-1.7B SFT outputs on Precalculus reach the generation cap, and Appendix Figure 4 shows substantial SFT deterioration during training. The reported advantage on completed outputs is useful, but does not replace a stabilized SFT comparison or establish improvement over the initial student. Initial-model accuracy and SFT with the same student-context KL would help determine how much of the gain reflects better adaptation versus avoiding baseline degeneration. The headline gains also use the best test result among multiple configurations; the configuration-selection procedure needs clarification independently of the clearly specified validation-based checkpoint selection.
Questions for authors
-
How does GFT compare with directly minimizing reference-answer NLL on identical student-prefix replays, using the same valid examples, LoRA setup, KL reference, and validation-selection procedure? Does removing the one-sided gate materially change performance?
-
What are the initial students' accuracies, and how does SFT perform with the same student-context KL? In the Precalculus 1.7B analysis of completed outputs, is the reported accuracy gap computed on the same questions for both methods?
-
Were the three transferred weighting rules and the final objective settings selected using validation accuracy alone? Appendix E describes Table 4, which reports test accuracy, as the basis for selecting the three rules, while Figure 1 reports the best of 12 GFT runs. Please clarify the selection protocol and give the test performance of a configuration selected exclusively using validation results.
4 comments
Response to Reviewer WMDN (continued)
Adaptation versus avoided degeneration.
We agree that most of the margin over paper SFT on MATH reflects SFT degrading the starting model, and we will state this in the paper. Three results show that GFT nevertheless adapts, rather than only avoiding degeneration.
(i) The degradation is not an artifact of a weak baseline. SFT with KL remains below the starting model in all three seeds. Answer-CE and GFT both exceed the starting model in some configurations, but GFT achieves the higher three-seed mean.
(ii) SFT degrades within its first epoch, GFT keeps learning (Table R7). SFT peaks at its first check at every Precalculus size, with and without KL, already below the start, while 16 of 18 GFT + KL runs reach the starting validation accuracy or more, at later checkpoints (up to step 720).
(iii) GFT retains and often improves strong starting-model performance. Relative to the sample-aligned untrained figures supplied for Table R5, validation-selected GFT improves by +3.03 points on GSM8K 0.6B, +3.09 on GSM8K 1.7B, +1.69 on GSM8K 4B, +1.51 on Intermediate Algebra 0.6B, +4.13 on Intermediate Algebra 1.7B, and +4.21 on Precalculus 1.7B. These untrained values now agree with Tables R1–R2 for the same settings.
GFT thus learns from the same worked solutions that damage a strong student under SFT.
Table R7. Best-validation step, Precalculus (seed 1333; 84 steps = 1 epoch). GFT: mean over six runs (latest in brackets). Val. %: start, and SFT (no KL) best / last. Last column: GFT + KL runs ≥ start.
| Model | SFT | SFT + KL | GFT | GFT + KL | Start | SFT best / last | GFT + KL ≥ start |
|---|---|---|---|---|---|---|---|
| 0.6B | 60 | 60 | 100 (180) | 80 (120) | 44.0 | 16.0 / 12.0 | 4/6 |
| 1.7B | 60 | 60 | 80 (180) | 240 (720) | 62.7 | 38.7 / 18.7 | 6/6 |
| 4B | 60 | 60 | 140 (540) | 130 (180) | 80.0 | 60.0 / 25.3 | 6/6 |
Q3 and Weakness 2 (configuration selection). Selection protocol and a configuration selected on validation alone.
Checkpoints are always selected on validation alone. The three weightings were chosen from the GSM8K study of Appendix E, which reports test accuracy, as the reviewer notes, but validation does not separate the nine weightings (all within 0.8 points), and MATH and Qwen3-4B played no part in the choice. The final objective settings behind Figure 1 and the headline numbers (weighting, α and KL per setting) are the best test result of the 12 configurations, so they were not selected on validation alone. The configuration selected on validation alone beats SFT in all nine settings, by +0.7 to +2.5 points on GSM8K and +3.7 to +32.8 on MATH (Table R1), and in all six settings with three seeds (Table R5). As the reviewer notes, one fixed configuration (uniform, α=1, no KL) also beats SFT in all nine settings, so the improvement does not depend on per-task selection. The revision will report validation-selected results in the abstract, Figure 1 and Table 1, with the best of 12 as a secondary column.
We hope these additional results address your concerns regarding the baseline, robustness, scope, and mechanism. In light of this new evidence, we would be grateful if you would consider raising your score.
Thank you for the substantial additional experiments. The matched Answer-CE baseline, gate/filter ablations, and three-seed results address my main concerns and provide stronger evidence for the contribution of the proposed activation objective. The initial-model comparisons and validation-selected results also strengthen the empirical evaluation. I will raise my rating to 3 (Weak acceptance). I would appreciate clarification on two remaining points: (1) In Table R3, the no-gate, no-filter, and full-matching ablations have identical per-seed test accuracies for α=1 and α=0.9, despite differing validation trajectories in Table R4. Please confirm these entries. (2) The failure of SFT + KL alone does not establish that the observed degradation is unrelated to baseline configuration, particularly since SFT has already degraded at the first evaluation. Please clarify whether earlier stopping or a lower learning rate was evaluated; otherwise, please qualify the claim that the degradation is not an artifact of a weak baseline.
Response to Reviewer WMDN
Dear Reviewer,
Thank you for your positive reassessment and for raising your rating. We appreciate your constructive feedback and address the two remaining points below.
1. Identical ablation accuracies across α settings (Tables R3–R4).
Thank you for highlighting this observation. We understand the concern regarding the identical test accuracies across α settings despite differences in their validation trajectories. Table R3 reports test accuracy at the validation-selected checkpoint, whereas Table R4 tracks validation performance throughout training. For the ablation runs, checkpoints were selected at step 60, before the subsequent training trajectories diverged. Therefore, differences observed at later training steps do not affect the reported test accuracies at the selected checkpoints. Additionally, the relatively small evaluation sets (546 test and 75 validation problems) limit the resolution of accuracy measurements, making small performance differences less distinguishable. Together, these factors explain how identical test accuracies across α settings can coexist with different training dynamics. We have also added a note below Table R4 in our General Response to clarify this distinction.
2. SFT degradation and baseline configuration.
We thank the reviewer for this important observation. We agree that SFT degradation can depend on the baseline configuration, and we appreciate the opportunity to clarify our experimental protocol. To investigate learning-rate sensitivity, we evaluated three learning rates ({1e-5, 2e-5, 5e-5}) and employed validation-based checkpoint selection with early-stopping patience of 10. Under the reported protocol, SFT achieved its highest observed validation accuracy at the first evaluation (step 60), followed by further degradation. Moreover, adding validation-selected student KL did not improve SFT performance (28.88 ± 3.23% versus 28.88 ± 1.78% for standard SFT), whereas GFT achieved 59.34 ± 0.48%.
We acknowledge that even smaller learning rates or checkpoints before step 60 might improve SFT performance. However, very early checkpoints involve substantially fewer training updates and may primarily preserve the initial model's capabilities rather than demonstrate meaningful adaptation. Furthermore, identifying an optimal SFT configuration may require additional hyperparameter exploration. In contrast, GFT demonstrates consistent improvements and greater robustness across the evaluated learning rates, highlighting its reduced sensitivity to training configurations. Therefore, our findings support GFT's effectiveness under the evaluated experimental settings without ruling out further improvements through additional SFT tuning.
We appreciate your reconsideration and hope these clarifications address your remaining concerns. If you have any further questions or concerns, we would be happy to provide additional clarification.
Best regards,
The Authors
Response to Reviewer WMDN
We thank the reviewer for identifying the comparison that decides the paper's main claim, and for noting the same-student replay, the precise specification and the stability analysis. We ran exactly the requested matched baseline with three seeds on Qwen3-1.7B Precalculus. Its best three-seed mean improves on the starting model by 1.83 points, whereas GFT improves by 4.21 points and exceeds the best answer-CE mean by 2.38 points.
Weakness 1 and Q1. Answer NLL on identical student-prefix replays; removing the one-sided gate.
Our answer-CE baseline (G2, Table R2) minimizes the reference-answer NLL on the identical student-prefix replays, with the same rollouts, valid examples, LoRA setup, KL reference and validation selection, and its KL weight is selected on validation from {0, 0.01, 0.05, 0.1}. It reaches its best three-seed mean of 56.96 ± 0.92 with selected KL (α=0.9, β=0.1), while the no-KL variant reaches 56.17 ± 0.59, against 59.34 ± 0.48 for validation-selected GFT; GFT has the higher mean, although one answer-CE seed ties the lowest GFT seed at 58.79%.
This separates GFT from on-policy answer supervision and regularization: answer-CE uses matched rollouts and regularization, yet its best three-seed mean (56.96) remains 2.38 points below GFT (59.34), although it exceeds the starting model (55.13) by 1.83 points. In the α=1 comparisons, full GFT reaches 59.83 ± 0.21, versus 56.84 ± 0.46 for answer-CE and 55.07–56.90 for the activation ablations (Table R3).
Removing the gate or filter now reduces accuracy. At α=1, full GFT scores 59.83 ± 0.21, versus 56.90 ± 0.92 without the gate and 55.86 ± 0.73 without the filter (Table R3); the same ordering holds at α=0.9. Full matching without the answer gradient drops further to 55.07 ± 0.94 at α=1, supporting the role of answer-gradient-aware alignment. Table R4 reports both α settings: by step 660, GFT retains approximately 61–65% validation accuracy for α=1 and 63–67% for α=0.9, whereas full matching falls to approximately 0–4% and 11–25%, respectively. We will present the gate and filter as empirically supported design choices that constrain each active update to a local descent direction on answer NLL.
Weakness 2 and Q2. Initial students, SFT with the same student-context KL, and the completed-output gap.
Initial students. Table R1 gives all nine under the paper's protocol.
SFT with the same KL. With the same student-context KL to the best-validation checkpoint, weight selected on validation, SFT reaches 28.88 ± 3.23, the same mean as plain SFT (Table R2). The KL that stabilizes GFT does not stabilize SFT, so GFT's stability does not come from the regularizer alone.
Completed outputs. The reviewer is right that the paper's comparison used each model's own finished outputs, so the two accuracies referred to different questions. On the same questions (304 and 298 shared), GFT leads SFT by 20.7 and 22.2 points (74.34/74.83% vs. 53.62/52.68%; G4). This shared-subset analysis confirms that GFT's advantage over SFT persists even when both models are evaluated on identical completed questions.