acceptodds
Under review as a conference paper at ICLR 2027

Gradient-guided Fine-tuning

Abstract

Supervised fine-tuning (SFT) trains language models to reproduce reference responses token by token, although different reasoning trajectories can support the same correct answer. We introduce Gradient-guided Fine-Tuning (GFT), a method for learning from reference solutions through task-relevant activation alignment. GFT generates a student response and replays both reference and student reasoning through the same model. Gradients of the reference-answer negative log-likelihood identify answer-sensitive directions, while a directional activation-alignment objective guides parameter updates without directly optimizing full-response cross-entropy. We combine these activation-alignment losses across all decoder layers using normalized weights, without requiring model-specific layer selection. To mitigate late-training generation collapse, we incorporate KL regularization on student-generated contexts, preserving useful generation behavior while permitting representation-level adaptation. We evaluate GFT on three mathematical reasoning benchmarks, GSM8K, MATH Intermediate Algebra, and MATH Precalculus, across three model sizes: Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B. The best GFT configuration exceeds SFT by 0.8–3.6 percentage points on GSM8K and by 7–33 points on the two MATH subjects. These findings support gradient-guided activation alignment as a promising alternative to direct token imitation for adapting reasoning models. The code is available at https://anonymous.4open.science/r/GFT/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, mathematical reasoning, gradient-guided fine-tuning, activation alignment, training stability, KL regularization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WMDNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Gradient-guided Fine-Tuning (GFT), which learns from worked solutions through an activation objective within the student model. The student replays both a reference solution and its own generated reasoning with the reference answer substituted in. At each decoder layer, GFT penalizes the positive projection of their activation difference onto a detached, filtered gradient of the reference-answer negative log-likelihood. The method also studies cross-entropy on answer-extraction failures and KL regularization to the best-validation checkpoint on student-generated contexts. Experiments with three Qwen3 sizes on GSM8K and two MATH subjects show improvements over the reported SFT baselines, while student-context KL substantially reduces post-peak generation collapse.

Critical strengths
  1. An interesting way to use reference solutions without an additional teacher network. The same-student replay construction avoids cross-model representation matching, while the one-sided projection gives a precise alternative to matching complete activation vectors. The paper carefully specifies prediction positions, stop-gradient operations, normalization, and update ordering, and explains the local behavior of the objective in Eq. (8). Inference uses ordinary generation without an additional model or activation-matching procedure.

  2. Substantial empirical gains supported by a useful stability analysis. Every reported GFT configuration on the two MATH subjects exceeds its corresponding SFT baseline. A single fixed configuration, uniform weighting with activation share 1 and no KL, also beats SFT in all nine task/model combinations in Table 1, so the overall improvement is not confined to the best configuration for each task. The study reports validation trajectories and post-peak behavior: the number of runs with a validation drop exceeding 20 percentage points falls from 47/54 without KL to 1/54 with student KL. Both the accuracy results and the stabilization strategy warrant attention.

Critical weaknesses
  1. The experiments do not isolate the benefit of the central reference-guided activation objective. Eq. (8) shows that an active term follows the filtered answer-NLL gradient, with the reference determining the gate and update magnitude. Sweeping layer weights, extraction-failure CE, and KL does not establish whether this construction improves over directly optimizing answer NLL on the same student-prefix replays. A matched answer-only CE baseline using the same rollouts and KL regularizer, together with an ablation of the one-sided projection, would distinguish the proposed contribution from the benefits of on-policy answer supervision and regularization. This is the main issue behind my recommendation.

  2. The baseline behavior makes the largest gains difficult to attribute. Section 5.1 reports that 44% of the Qwen3-1.7B SFT outputs on Precalculus reach the generation cap, and Appendix Figure 4 shows substantial SFT deterioration during training. The reported advantage on completed outputs is useful, but does not replace a stabilized SFT comparison or establish improvement over the initial student. Initial-model accuracy and SFT with the same student-context KL would help determine how much of the gain reflects better adaptation versus avoiding baseline degeneration. The headline gains also use the best test result among multiple configurations; the configuration-selection procedure needs clarification independently of the clearly specified validation-based checkpoint selection.

Questions for authors
  1. How does GFT compare with directly minimizing reference-answer NLL on identical student-prefix replays, using the same valid examples, LoRA setup, KL reference, and validation-selection procedure? Does removing the one-sided gate materially change performance?

  2. What are the initial students' accuracies, and how does SFT perform with the same student-context KL? In the Precalculus 1.7B analysis of completed outputs, is the reported accuracy gap computed on the same questions for both methods?

  3. Were the three transferred weighting rules and the final objective settings selected using validation accuracy alone? Appendix E describes Table 4, which reports test accuracy, as the basis for selecting the three rules, while Figure 1 reports the best of 12 GFT runs. Please clarify the selection protocol and give the test performance of a configuration selected exclusively using validation results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WMDN

We thank the reviewer for identifying the comparison that decides the paper's main claim, and for noting the same-student replay, the precise specification and the stability analysis. We ran exactly the requested matched baseline with three seeds on Qwen3-1.7B Precalculus. Its best three-seed mean improves on the starting model by 1.83 points, whereas GFT improves by 4.21 points and exceeds the best answer-CE mean by 2.38 points.

Weakness 1 and Q1. Answer NLL on identical student-prefix replays; removing the one-sided gate.

Our answer-CE baseline (G2, Table R2) minimizes the reference-answer NLL on the identical student-prefix replays, with the same rollouts, valid examples, LoRA setup, KL reference and validation selection, and its KL weight is selected on validation from {0, 0.01, 0.05, 0.1}. It reaches its best three-seed mean of 56.96 ± 0.92 with selected KL (α=0.9, β=0.1), while the no-KL variant reaches 56.17 ± 0.59, against 59.34 ± 0.48 for validation-selected GFT; GFT has the higher mean, although one answer-CE seed ties the lowest GFT seed at 58.79%.

This separates GFT from on-policy answer supervision and regularization: answer-CE uses matched rollouts and regularization, yet its best three-seed mean (56.96) remains 2.38 points below GFT (59.34), although it exceeds the starting model (55.13) by 1.83 points. In the α=1 comparisons, full GFT reaches 59.83 ± 0.21, versus 56.84 ± 0.46 for answer-CE and 55.07–56.90 for the activation ablations (Table R3).

Removing the gate or filter now reduces accuracy. At α=1, full GFT scores 59.83 ± 0.21, versus 56.90 ± 0.92 without the gate and 55.86 ± 0.73 without the filter (Table R3); the same ordering holds at α=0.9. Full matching without the answer gradient drops further to 55.07 ± 0.94 at α=1, supporting the role of answer-gradient-aware alignment. Table R4 reports both α settings: by step 660, GFT retains approximately 61–65% validation accuracy for α=1 and 63–67% for α=0.9, whereas full matching falls to approximately 0–4% and 11–25%, respectively. We will present the gate and filter as empirically supported design choices that constrain each active update to a local descent direction on answer NLL.

Weakness 2 and Q2. Initial students, SFT with the same student-context KL, and the completed-output gap.

Initial students. Table R1 gives all nine under the paper's protocol.

SFT with the same KL. With the same student-context KL to the best-validation checkpoint, weight selected on validation, SFT reaches 28.88 ± 3.23, the same mean as plain SFT (Table R2). The KL that stabilizes GFT does not stabilize SFT, so GFT's stability does not come from the regularizer alone.

Completed outputs. The reviewer is right that the paper's comparison used each model's own finished outputs, so the two accuracies referred to different questions. On the same questions (304 and 298 shared), GFT leads SFT by 20.7 and 22.2 points (74.34/74.83% vs. 53.62/52.68%; G4). This shared-subset analysis confirms that GFT's advantage over SFT persists even when both models are evaluated on identical completed questions.

AuthorsOpenReview ↗

Response to Reviewer WMDN (continued)

Adaptation versus avoided degeneration.

We agree that most of the margin over paper SFT on MATH reflects SFT degrading the starting model, and we will state this in the paper. Three results show that GFT nevertheless adapts, rather than only avoiding degeneration.

(i) The degradation is not an artifact of a weak baseline. SFT with KL remains below the starting model in all three seeds. Answer-CE and GFT both exceed the starting model in some configurations, but GFT achieves the higher three-seed mean.

(ii) SFT degrades within its first epoch, GFT keeps learning (Table R7). SFT peaks at its first check at every Precalculus size, with and without KL, already below the start, while 16 of 18 GFT + KL runs reach the starting validation accuracy or more, at later checkpoints (up to step 720).

(iii) GFT retains and often improves strong starting-model performance. Relative to the sample-aligned untrained figures supplied for Table R5, validation-selected GFT improves by +3.03 points on GSM8K 0.6B, +3.09 on GSM8K 1.7B, +1.69 on GSM8K 4B, +1.51 on Intermediate Algebra 0.6B, +4.13 on Intermediate Algebra 1.7B, and +4.21 on Precalculus 1.7B. These untrained values now agree with Tables R1–R2 for the same settings.

GFT thus learns from the same worked solutions that damage a strong student under SFT.

Table R7. Best-validation step, Precalculus (seed 1333; 84 steps = 1 epoch). GFT: mean over six runs (latest in brackets). Val. %: start, and SFT (no KL) best / last. Last column: GFT + KL runs ≥ start.

ModelSFTSFT + KLGFTGFT + KLStartSFT best / lastGFT + KL ≥ start
0.6B6060100 (180)80 (120)44.016.0 / 12.04/6
1.7B606080 (180)240 (720)62.738.7 / 18.76/6
4B6060140 (540)130 (180)80.060.0 / 25.36/6

Q3 and Weakness 2 (configuration selection). Selection protocol and a configuration selected on validation alone.

Checkpoints are always selected on validation alone. The three weightings were chosen from the GSM8K study of Appendix E, which reports test accuracy, as the reviewer notes, but validation does not separate the nine weightings (all within 0.8 points), and MATH and Qwen3-4B played no part in the choice. The final objective settings behind Figure 1 and the headline numbers (weighting, α and KL per setting) are the best test result of the 12 configurations, so they were not selected on validation alone. The configuration selected on validation alone beats SFT in all nine settings, by +0.7 to +2.5 points on GSM8K and +3.7 to +32.8 on MATH (Table R1), and in all six settings with three seeds (Table R5). As the reviewer notes, one fixed configuration (uniform, α=1, no KL) also beats SFT in all nine settings, so the improvement does not depend on per-task selection. The revision will report validation-selected results in the abstract, Figure 1 and Table 1, with the best of 12 as a secondary column.

We hope these additional results address your concerns regarding the baseline, robustness, scope, and mechanism. In light of this new evidence, we would be grateful if you would consider raising your score.

Thank you for the substantial additional experiments. The matched Answer-CE baseline, gate/filter ablations, and three-seed results address my main concerns and provide stronger evidence for the contribution of the proposed activation objective. The initial-model comparisons and validation-selected results also strengthen the empirical evaluation. I will raise my rating to 3 (Weak acceptance). I would appreciate clarification on two remaining points: (1) In Table R3, the no-gate, no-filter, and full-matching ablations have identical per-seed test accuracies for α=1 and α=0.9, despite differing validation trajectories in Table R4. Please confirm these entries. (2) The failure of SFT + KL alone does not establish that the observed degradation is unrelated to baseline configuration, particularly since SFT has already degraded at the first evaluation. Please clarify whether earlier stopping or a lower learning rate was evaluated; otherwise, please qualify the claim that the degradation is not an artifact of a weak baseline.

AuthorsOpenReview ↗

Response to Reviewer WMDN

Dear Reviewer,

Thank you for your positive reassessment and for raising your rating. We appreciate your constructive feedback and address the two remaining points below.

1. Identical ablation accuracies across α settings (Tables R3–R4).

Thank you for highlighting this observation. We understand the concern regarding the identical test accuracies across α settings despite differences in their validation trajectories. Table R3 reports test accuracy at the validation-selected checkpoint, whereas Table R4 tracks validation performance throughout training. For the ablation runs, checkpoints were selected at step 60, before the subsequent training trajectories diverged. Therefore, differences observed at later training steps do not affect the reported test accuracies at the selected checkpoints. Additionally, the relatively small evaluation sets (546 test and 75 validation problems) limit the resolution of accuracy measurements, making small performance differences less distinguishable. Together, these factors explain how identical test accuracies across α settings can coexist with different training dynamics. We have also added a note below Table R4 in our General Response to clarify this distinction.

2. SFT degradation and baseline configuration.

We thank the reviewer for this important observation. We agree that SFT degradation can depend on the baseline configuration, and we appreciate the opportunity to clarify our experimental protocol. To investigate learning-rate sensitivity, we evaluated three learning rates ({1e-5, 2e-5, 5e-5}) and employed validation-based checkpoint selection with early-stopping patience of 10. Under the reported protocol, SFT achieved its highest observed validation accuracy at the first evaluation (step 60), followed by further degradation. Moreover, adding validation-selected student KL did not improve SFT performance (28.88 ± 3.23% versus 28.88 ± 1.78% for standard SFT), whereas GFT achieved 59.34 ± 0.48%.

We acknowledge that even smaller learning rates or checkpoints before step 60 might improve SFT performance. However, very early checkpoints involve substantially fewer training updates and may primarily preserve the initial model's capabilities rather than demonstrate meaningful adaptation. Furthermore, identifying an optimal SFT configuration may require additional hyperparameter exploration. In contrast, GFT demonstrates consistent improvements and greater robustness across the evaluated learning rates, highlighting its reduced sensitivity to training configurations. Therefore, our findings support GFT's effectiveness under the evaluated experimental settings without ruling out further improvements through additional SFT tuning.

We appreciate your reconsideration and hope these clarifications address your remaining concerns. If you have any further questions or concerns, we would be happy to provide additional clarification.

Best regards,
The Authors

Reviewer_HxfzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gradient-guided Fine-Tuning (GFT). The model first generates its own inference, then replays the reference solution and the student's inference, which is then replaced with the correct final answer. The negative log-likelihood gradient of the answer is used to selectively penalize the student-reference activation gap, which is related to the answer loss. The method can also incorporate auxiliary cross-entropy and KL regularization. Experiments cover three Qwen3 model sizes, GSM8K, and two MATH subsets; the best results for each setting improve the performance relative to SFT by 0.76–3.57 percentage points on GSM8K and 7.30–33.15 percentage points on the MATH subset.

Critical strengths

The supervision objective is clear and has a verifiable mechanism. Both replay branches are generated by the same student model, requiring no external teacher network or cross-model layer mapping; one-sided projective loss allows for the preservation of representation differences independent of the current answer gradient. The paper explicitly states the gradient cutoff and update direction, and acknowledges that the answer loss still provides cross-entropy information, making the method's mechanism easier to understand and verify. (Sections 3.2–3.3)

The benefits cover multiple configurations, and stability issues are directly compared. The two smaller models outperform SFT in 67 out of 72 configurations on GSM8K, and all main experimental configurations in the two MATH subsets also outperform the corresponding SFT. After adding KL, the number of runs where validation accuracy dropped by more than 20 points after the peak decreased from 47/54 to 1/54. This directly supports the role of the stabilizer and shows that the paper does not only present a single best result.

Critical weaknesses

The necessity of the core representation objective has not been verified separately. Equation (8) shows that the update proceeds along the direction of the filtered answer loss gradient, and the reference activation gap mainly determines the update intensity. Therefore, "directly optimizing the cross-entropy of the correct answer on the same student inference prefix" is a key comparison, but this is not provided in the paper. Existing ablation methods mainly change layer weights, auxiliary cross-entropy, and KL, and do not directly test the effects of removing one-sided gating, common direction filtering, or using full activation matching. The activation of approximately 46% of the gating terms only proves that the mechanism is working, not that it brings a final accuracy gain.

SFT baseline degradation leaves substantial uncertainty regarding the interpretation of the significant improvement. 44% of the outputs in the Precalculus SFT of Qwen3-1.7B reach the length limit, and the training curves also show early degradation in multiple SFT settings. The authors report that GFT still leads by 13–20 points in generating output, which is important counter-evidence; however, conditional comparisons of samples generated by different methods cannot replace initializing the model, fully tuning SFT, and comparing SFT with the same KL stabilizer. Furthermore, the experiment only had one training seed and matched the number of training steps rather than the computational cost, making it difficult to determine the reproducible net gain of the new objective itself.

Questions for authors
  1. Under the same student generation, answer replacement, sample selection, and KL conditions, what are the results of directly optimizing the cross-entropy of the correct answer? Does the accuracy steadily decrease after removing one-sided gating or using full activation matching?

  2. What are the accuracies of the three initialization models under the same evaluation protocol? Does step 0 participate in the selection of the optimal checkpoint? After tuning SFT or adding the same KL stabilizer, how much of the significant gap remains on Precalculus?

  3. Are there multiple sub-results with fixed representative configurations and comparisons of total GPU time? In particular, it is necessary to confirm whether the optimal gain of 0.76 on GSM8K for the 4B model is stable and can be retained within a reasonable training budget.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Hxfz

We thank the reviewer for the clear analysis of what Eq. (8) does and does not establish, and for noting that the gains hold across configurations and that the stability comparison is direct. The reviewer's two concerns are whether the representation objective matters beyond answer CE, and whether the gain over SFT survives against the initial models and a stabilized SFT. We ran both comparisons with three seeds on Qwen3-1.7B Precalculus.

Weakness 1 and Q1. Answer CE on the same student prefix; removing the gate; full activation matching.

We agree with the reviewer's reading of Eq. (8): an active term moves along the filtered answer-NLL gradient, and the reference sets the gate and the update magnitude. Answer CE on the same student prefix is therefore the key comparison. Under the same student generations, answer replacement, sample selection and KL (weight selected on validation), answer-CE reaches 53.97–56.96 across its variants, peaking at 56.96 ± 0.92 with selected KL, against 59.34 ± 0.48 for GFT (G2, Table R2). Three results show that the reference contributes more than an update magnitude.

(i) Answer-CE improves upon the starting model, but trails GFT: its best variant averages 56.96 ± 0.92 against 55.13 initially (+1.83 points), versus 59.34 ± 0.48 for GFT (+4.21 points) (Table R2). In the matched α=1 ablation (Table R3), full GFT achieves 59.83 ± 0.21 versus 56.84 ± 0.46 for answer-CE, demonstrating the importance of the proposed activation objective.

(ii) Full matching, which ignores the answer gradient and pulls every pooled coordinate toward the reference, reaches only 55.07 ± 0.94 at α=1, versus 59.83 ± 0.21 for full GFT (Table R3). This comparison supports the need for answer-gradient-aware alignment, not unrestricted activation matching.

(iii) GFT and answer-CE lead to different models, not only different accuracies: GFT has fewer length-capped outputs (7.7% vs. 12.8–16.6%) and lower seed variance (sd 0.48 vs. 0.59–1.80).

Does accuracy decrease without the gate or with full matching? Yes. At α=1 (Table R3), no gate reaches 56.90 ± 0.92, no filter 55.86 ± 0.73, and full matching 55.07 ± 0.94, all below full GFT (59.83 ± 0.21). At α=0.9 the corresponding means are again below full GFT (59.34 ± 0.48). Table R4 reports both α settings: at step 660, GFT retains approximately 61–65% validation accuracy for α=1 and 63–67% for α=0.9, while full matching falls to approximately 0–4% and 11–25%, respectively; α=1 full matching has 71–100% capped outputs. Together, these results favor selective, answer-gradient-aware alignment over unrestricted activation matching.

The 46% active share establishes that the gate operates; the ablations also show higher test accuracy with the gate and common-direction filter. We will explain their local-descent role (Eq. 8) and report the α=1 and α=0.9 training trajectories alongside the R3 comparisons.

AuthorsOpenReview ↗

Response to Reviewer Hxfz (continued)

Weakness 2 and Q2. SFT degradation, initial models, step 0, and the gap after tuning SFT or adding the same KL.

Initial models. Table R1 reports all three under the paper's protocol (same prompts, token cap, greedy decoding, scoring and full test set). The untrained student beats paper SFT in 6 of 9 settings, so SFT degradation does inflate the MATH margins, and we will add the untrained students to Table 1.

Step 0. Step 0 does not take part in checkpoint selection; the first validation check is at step 60. Including it would not change the GFT result on Precalculus 1.7B: the selected GFT checkpoints reach 69.3, 64.0 and 65.3% validation accuracy, above the starting model's 62.7%, so the result comes from training.

Gap after tuning SFT or adding the same KL. As the reviewer notes, SFT degrades: 42.9% of its outputs hit the length cap. SFT with the same student KL, weight selected on validation, reaches 28.88 ± 3.23 (43.2% capped), the same mean as plain SFT, against 59.34 ± 0.48 for GFT: the gap stays at 30.5 points, and all GFT seeds exceed every SFT seed with and without KL (Table R2). SFT peaks at its first check (step 60) at every Precalculus size, with and without KL, and only degrades afterwards (G4), so neither the KL nor longer training prevents the degradation. SFT damages a strong student (55.13 → 28.88), while GFT learns from the same worked solutions without this damage and ends above the starting model in all seeds.

Completed outputs. We agree that conditional accuracies over different samples cannot replace these comparisons. On the questions both models finish (304 and 298), GFT still leads by 20.7 and 22.2 points (G4); we will report this only as a diagnostic.

Q3 and Weakness 2 (single seed, step- vs. compute-matching). Repeated runs, GPU time, and the +0.76 gain of Qwen3-4B on GSM8K.

Repeated runs. Across five settings, 56 fixed GFT configurations were evaluated with three seeds (12 each on four settings, 8 on GSM8K 4B); 53 outperform SFT on mean accuracy (median sd 0.71). Validation-selected GFT also outperforms SFT in all six settings (Table R5). On Precalculus 1.7B, five fixed GFT variants have three seeds (Tables R2 and R3).

GPU time. Table R6 reports total time to early stopping. GFT trains for 4.4 H200 hours (720 steps), about as much as answer-CE (3.8–4.6) and SFT + KL (4.0–5.2), and about 10× plain SFT per step, because methods trained on student generations pay for generation. As SFT peaks at step 60 and then degrades, more SFT compute does not close the gap.

The +0.76 gain. The paper's seed-1333 test-selected GSM8K 4B gain is +0.76 points (Table R1). In the updated three-seed validation-selected comparison (Table R5), GFT reaches 92.14 ± 0.68 versus an SFT mean of 90.11%, a 2.03-point advantage. The separate linear, α=0.9, no-KL configuration reaches 92.87 ± 0.33 over three seeds. We will clearly report both single-seed and multi-seed improvements, highlighting GFT's consistent gains despite the strong initial performance of Qwen3-4B.

Presentation.

We will add pseudo-code for one GFT update and a notation table to Section 3, and move the selection protocol of this response into Section 4.

We hope these additional results address your concerns regarding the baseline, robustness, scope, and mechanism. In light of this new evidence, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our submission and providing constructive feedback. We have carefully prepared our rebuttal to address your concerns, supported by extensive additional experiments, analyses, and clarifications that further strengthen our findings.

We hope these additional results adequately address your concerns. If so, we would be grateful if you would consider raising your score. We would also be happy to address any further questions or concerns you may have.

Thank you again for your valuable feedback and consideration.

Best regards,
The Authors

Reviewer_h9yfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Gradient-guided Fine-Tuning (GFT), which uses reference worked solutions to train a model through its activations. The current student replays two sequences: the reference worked solution and its own reasoning with the answer replaced by the reference answer. At each decoder layer, GFT averages activations at the positions that predict answer tokens. It subtracts the reference activations from the student activations, then scales and filters this difference. The gradient comes from the negative log-likelihood (NLL) of the reference-answer tokens in the student-prefix replay. This gradient is filtered and normalized. The method projects the scaled, filtered activation difference onto this direction and penalizes only positive projections, using a squared loss. During the update, the reference activations and gradient direction are treated as constants.

The optional cross-entropy (CE) term is called “extraction-failure CE.” It trains on the full reference response only for examples where no answer can be extracted from the student's generation. The optional KL term is evaluated on unmodified student generations. It uses the checkpoint with the highest validation accuracy so far as the reference. Experiments compare GFT with supervised fine-tuning (SFT) on GSM8K and two MATH subjects, using three Qwen3 model sizes. The paper reports higher test accuracy at selected checkpoints and fewer sharp declines in validation accuracy later in training when KL is used.

Critical strengths
  1. The training procedure is carefully specified. The method uses reference text without requiring logits or hidden states from a separate teacher. Both replays use the same reference answer, so the activation comparison is not driven simply by different answer tokens. The paper explicitly states which quantities are detached from the gradient computation and explains that the gradient direction gives only a local prediction of how the answer loss will change. These details help readers understand and reproduce the method.

  2. The stability experiments examine what happens after a run reaches its best validation accuracy. Table 5 reports subsequent drops exceeding 20 percentage points in 47 of 54 runs without KL, compared with 1 of 54 runs with KL, across the evaluated configurations. The paper also acknowledges that KL can reduce peak accuracy on GSM8K. Reporting both effects makes the tradeoff between accuracy and training stability clear.

Critical weaknesses

Weakness 1

The comparison does not yet establish how much GFT improves over a comparably tuned and regularized SFT baseline. For Qwen3-1.7B on Precalculus, SFT achieves 26.92% accuracy, and 44% of its test outputs reach the generation length limit (Section 5.1). The authors acknowledge this and report that GFT also performs better among outputs that finish. However, if each model is evaluated only on the problems where its own output finishes, the two accuracies can refer to different subsets of problems.

The paper compares GFT with and without the optional extraction-failure CE and KL terms. Configurations without either optional term already outperform SFT on several tasks. These comparisons provide evidence about the effects of CE and KL within GFT. However, the paper does not report a comparable search over SFT training settings or SFT with an analogous KL regularizer. It remains unclear how much of GFT's advantage would persist with comparable baseline tuning and regularization. Matching optimizer steps also leaves the tradeoff between accuracy and computational cost unresolved; this does not invalidate the reported accuracy differences.

The choice of configuration and variation across training runs are particularly relevant when interpreting small gains. For Qwen3-4B on GSM8K, the best of twelve GFT configurations improves accuracy by 0.76 percentage points, or approximately ten more test problems answered correctly. All runs use the same training seed. This warrants caution about that comparison, while the larger gains on MATH remain meaningful evidence.

Weakness 2

The experiments do not establish whether the proposed activation objective improves on direct CE training of reference-answer tokens after the student's reasoning. Section 3.3 explains why GFT is not mathematically equivalent to answer CE with a fixed coefficient. The remaining question is whether this difference improves performance. A direct CE baseline should use the same rules for constructing student prefixes, replacing answers, and selecting valid pairs. Here, valid pairs are those whose answer spans are retained during replay. A comparable KL regularizer, with its weight selected on validation data for each objective, would make this comparison more informative.

Within the activation objective, the roles of the one-sided condition and the common-direction filter are not directly tested. The former penalizes only positive projections; the latter removes the dominant direction estimated from calibration activations. Figure 5 shows how often the one-sided condition is satisfied. It does not show whether excluding the remaining terms improves accuracy. Targeted ablations would help explain the contribution of these design choices.

Questions for authors

The highest priorities are clarifying how configurations were selected and providing the two baseline comparisons below. One representative setting, such as Qwen3-1.7B on Precalculus, would be sufficient for these comparisons.

  1. Were the three layer weighting rules transferred to other settings selected using validation accuracy alone? Was validation accuracy also the sole criterion for selecting the configurations behind the main “best GFT” results? Appendix E describes Table 4, which reports test accuracy, as the basis for selecting the weighting rules. Please distinguish selecting a checkpoint within a run from selecting a configuration across runs, and report the test result for a configuration selected using validation data alone.

  2. How does GFT compare with applying CE directly to the reference-answer tokens after student-generated reasoning? Please use the same rules for replacing answers and selecting valid pairs, along with the same training data, generation protocol, and evaluation procedure. Use the same definition of KL and a comparable rule for updating the reference checkpoint. Select the regularization weight on validation data for each objective. Identical numerical weights need not give comparable regularization when the training losses have different scales.

  3. On the same setting, how does GFT compare with the starting model and an SFT baseline whose hyperparameters are selected using validation data? For SFT, a comparable KL regularizer could use student-generated contexts and the checkpoint with the highest validation accuracy so far. Its weight should also be selected on validation data. Please report accuracy on the full test set as the main result, along with the fraction of outputs that reach the generation length limit. As a supplementary diagnostic, report accuracy on the shared subset where both models finish, together with its size. Because this subset depends on which outputs finish, it is only a diagnostic and cannot replace the comparison on the full test set.

Repeating a key comparison with additional training seeds and separately testing the one-sided condition and common-direction filter would strengthen the evidence. Reporting training time or GPU hours for a representative run would also clarify computational cost. These additions are secondary to the selection clarification and baseline comparisons above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer h9yf

We thank the reviewer for the precise and prioritized questions, and for noting that the procedure is carefully specified and that the stability analysis reports both sides of the KL trade-off. We ran every comparison the reviewer prioritized on the suggested setting, Qwen3-1.7B on Precalculus, with three seeds. In short, (1) the configuration selected on validation alone beats SFT in all nine settings; (2) answer CE with matched rules improves over the starting model (best mean +1.83 points), while GFT achieves the higher mean (+4.21 points) and improves in every seed; and (3) SFT with the same validation-selected KL stays 30.5 points below GFT, and the gap remains on the shared subset.

Q1. Were the weightings and the "best GFT" configurations selected on validation alone?

We separate the three levels of selection, as the reviewer asks (G1).

(i) Checkpoint within a run. Always validation accuracy alone; test accuracy is computed once, at the selected checkpoint.

(ii) Weighting rules. They were chosen from the GSM8K study of Appendix E, which reports test accuracy, as the reviewer notes. Validation would not have given a clearer choice: averaged over the four objective settings and both models, all nine weightings lie within 0.8 validation points (79.7–80.4%), with linear second, quadratic fourth and uniform fifth. MATH and Qwen3-4B were never used in this choice, so six of the nine settings test the transfer of a fixed choice.

(iii) Configuration across runs. The paper's "best GFT" takes the best test result of 12 configurations. Table R1 reports the configuration with the highest validation accuracy instead: it beats SFT in all nine settings, by +0.7 to +2.5 points on GSM8K and +3.7 to +32.8 on MATH. With three seeds and selection by mean validation accuracy (Table R5), it beats SFT in all six settings, e.g. 59.34 ± 0.48 vs. 28.88 ± 1.78 on Precalculus 1.7B. The camera-ready will use validation-only selection in the abstract, Figure 1 and Table 1, and will state the Appendix E protocol explicitly.

Q2 and Weakness 2. CE on the reference-answer tokens after the student's reasoning; the one-sided condition and the common-direction filter.

Our answer-CE baseline (G2, Table R2) follows each condition the reviewer lists: the same answer replacement and valid-pair rule (answer spans retained in the replay), training data, generation protocol and evaluation, and the same KL on student generations with the same best-validation reference update. Because identical KL weights need not regularize both losses equally, its KL weight is selected on validation from {0, 0.01, 0.05, 0.1}, for α=1 and α=0.9, whereas GFT keeps β=0.05 without tuning, which, if anything, favors the baseline.

Answer-CE reaches its best three-seed mean of 56.96 ± 0.92 with selected KL (α=0.9, β=0.1); the no-KL variant reaches 56.17 ± 0.59 (Table R2), against 59.34 ± 0.48 for validation-selected GFT. The best answer-CE mean improves on the starting model (55.13) by 1.83 points, whereas GFT improves by 4.21 points and leads answer-CE by 2.38 points; on-policy answer supervision and regularization alone do not account for the full gain. Answer-CE also produces about twice as many length-capped outputs (12.8–16.6% vs. 7.7%) and varies more across seeds (sd 0.59–1.80 vs. 0.48).

On the second part of Weakness 2 (gate and filter), Table R3 now shows a clear difference at the best checkpoint: at α=1, full GFT reaches 59.83 ± 0.21, compared with 56.90 ± 0.92 without the one-sided gate and 55.86 ± 0.73 without the common-direction filter. The corresponding α=0.9 results show the same ordering (59.34 ± 0.48 vs. 56.90 ± 0.92 and 55.86 ± 0.73). Thus, the ablations support both design choices beyond merely showing that the gate activates; we will explain their local-descent role (Eq. 8) and report the full comparisons. Table R4 now reports both α settings: at step 660, GFT stays at approximately 61–65% validation accuracy for α=1 and 63–67% for α=0.9, while full matching falls to approximately 0–4% and 11–25%, respectively.

AuthorsOpenReview ↗

Response to Reviewer h9yf (continued)

Q3 and Weakness 1 (different subsets). Starting model and SFT with validation-selected KL, on the full test set and the shared subset.

Full test set (main result, Table R2). The starting model reaches 55.13 (5.5% capped). SFT with the KL the reviewer describes, on student-generated contexts towards the best-validation checkpoint, with its weight selected on validation from {0, 0.01, 0.05, 0.1}, reaches 28.88 ± 3.23 (43.2% capped), the same mean as plain SFT (28.88 ± 1.78, 42.9%). GFT reaches 59.34 ± 0.48 (7.7% capped). All GFT seeds exceed the starting model and every SFT seed, with or without KL.

Shared subset (diagnostic only). SFT finishes 312/311 of the 546 test questions and GFT 507/501 (seeds 1334/1335), so 304 and 298 questions are finished by both. On them, GFT reaches 74.34/74.83% vs. 53.62/52.68% for SFT, a gap of 20.7/22.2 points, so SFT's deficit is not only truncation. As the reviewer notes, the subset depends on which outputs finish, so we will report it only as a diagnostic, next to the full-test results, and replace the paper's sentence on finished outputs.

Weakness 1. Comparable SFT tuning and regularization, step- vs. compute-matching, and the single-seed GSM8K 4B gain.

Tuning and regularization. The same KL does not help SFT (Q3), and SFT's best checkpoint is its first validation check (step 60, 0.7 epochs) at all three Precalculus sizes, with and without KL, after which it only degrades (G4). Earlier stopping is therefore already covered by the selection, and longer training or stronger regularization does not recover the starting model. GFT, trained on the same worked solutions, keeps improving (best steps up to 720) and ends above the starting model.

Compute. Matching optimizer steps gives GFT more compute, about 10× SFT per step (21.8 vs. 2.1 s on one H200; Table R6), about as much as answer-CE and SFT + KL, because all methods trained on student generations pay for generation. Giving SFT more compute does not help, since its validation accuracy only falls after step 60.

GSM8K 4B. The paper's +0.76-point result uses the seed-1333 test-selected configuration (Table R1). Across three seeds, validation-selected GFT scores 92.14 ± 0.68 versus an SFT mean of 90.11% (+2.03 points; Table R5); another GFT configuration (linear, α=0.9, no KL) reaches 92.87 ± 0.33. We will clearly distinguish the selection protocols and report the GSM8K 4B improvement alongside the stronger results on MATH benchmarks.

Secondary requests.

More seeds: three seeds for 56 configurations across five settings (12 each in four settings; 8 on GSM8K 4B), 53 above SFT on mean accuracy, median sd 0.71 (Table R5). Gate and filter: Tables R3 and R4. GPU hours: Table R6; the recommended GFT configuration trains for 4.4 H200 hours (720 steps) on Precalculus 1.7B.

We hope these additional results address your concerns regarding the baseline, robustness, scope, and mechanism. In light of this new evidence, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer,

We sincerely appreciate the time and effort you have devoted to reviewing our submission and providing constructive feedback. We have carefully prepared our rebuttal to address your concerns, supported by extensive additional experiments, analyses, and clarifications that further strengthen our findings.

We hope these additional results adequately address your concerns. If so, we would be grateful if you would consider raising your score. We would also be happy to address any further questions or concerns you may have.

Thank you again for your valuable feedback and consideration.

Best regards,
The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their careful reviews. They agree that GFT is carefully specified, learns from worked solutions without a teacher network or cross-model layer mapping, and that student KL removes post-peak collapse (47/54 runs without KL vs. 1/54 with KL). Shared questions are answered once below (G1–G6); each reviewer reply answers that reviewer's points in turn. All new results will be added to the paper.

Setup. As Reviewer h9yf suggested, controlled comparisons use Qwen3-1.7B on MATH Precalculus with three seeds (1333 from the paper, 1334, 1335). We also evaluated all nine untrained students and repeated the GFT configurations and SFT over three seeds on GSM8K (0.6B/1.7B/4B) and Intermediate Algebra (0.6B/1.7B); GSM8K 4B has eight three-seed GFT configurations, the other four settings twelve each (Table R5). The protocol is unchanged: validation every 60 steps, patience 10, greedy decoding, full test set at the best-validation checkpoint. Baseline KL weights are selected on validation; GFT keeps the paper's learning rate and β=0.05 without tuning.

G1. Configuration selection. Checkpoints are always selected on validation, and test accuracy is computed once, at the selected checkpoint. The three weightings came from the GSM8K study of Appendix E, which reports test accuracy, but validation does not separate them (all nine within 0.8 points), and MATH and Qwen3-4B played no part in the choice. The headline "best GFT" is the best of 12 on test. Selected on validation alone (Table R1), GFT beats SFT in all nine settings (+0.7 to +2.5 on GSM8K, +3.7 to +32.8 on MATH), and also with three seeds (Table R5).

Table R1. Test accuracy (%), seed 1333. Capped: untrained outputs at the 4,096-token limit. ∗Mean of two configurations tied on validation.

SettingUntrainedCapped %SFTGFT (val.)GFT (best of 12)Val. − SFT
GSM8K 0.6B63.080.563.1564.9066.72+1.75
GSM8K 1.7B80.740.580.8283.3284.00+2.50
GSM8K 4B90.450.291.8992.5792.65+0.68
Int. Alg. 0.6B31.238.017.3931.06∗34.44+13.67
Int. Alg. 1.7B52.166.951.5055.1558.80+3.65
Int. Alg. 4B68.669.662.2468.6671.21+6.42
Precalc. 0.6B34.805.715.5734.4337.73+18.86
Precalc. 1.7B55.135.526.9259.7160.07+32.79
Precalc. 4B70.157.152.0173.53∗74.73+21.52

G2. Answer cross-entropy on identical replays. Answer-CE replaces the activation loss by the reference-answer NLL (Eq. 2) on the same student-prefix replays and keeps everything else identical: rollouts, answer replacement, valid pairs, extraction-failure CE, LoRA, KL definition and reference update, and selection. Its KL weight is selected on validation, for α=1 and α=0.9. GFT reaches 59.34 ± 0.48, outperforming all answer-CE variants; the strongest answer-CE three-seed mean is 56.96 ± 0.92 (α=0.9, β=0.1; Table R2). The no-KL answer-CE variant (α=0.9, β=0) reaches 56.17 ± 0.59. Relative to the untrained student (55.13), the strongest answer-CE variant improves by 1.83 percentage points, whereas GFT improves by 4.21 points, a 2.38-point advantage over the best answer-CE mean.

Table R2. Precalculus 1.7B, test accuracy (%), 546 problems. β selected on validation from {0, 0.01, 0.05, 0.1}. †Tied on validation. ‡Validation-best of the paper's 12 configurations.

Methodβs1333s1334s1335Mean ± sdCapped %
Untrained student–55.1355.1355.1355.135.5
SFT (paper)–26.9230.4029.3028.88 ± 1.7842.9
SFT + student KL0.125.2729.8531.5028.88 ± 3.2343.2
Answer-CE, α=0.9056.4156.5955.4956.17 ± 0.5913.4
Answer-CE, α=0.90.157.8856.9656.0456.96 ± 0.9212.8
Answer-CE, α=10.01†56.2358.7955.3156.78 ± 1.8012.8
Answer-CE, α=10.1†52.5654.9554.4053.97 ± 1.2516.6
GFT, α=0.9, linear0.05‡59.7159.5258.7959.34 ± 0.487.7
AuthorsOpenReview ↗

General Response to All Reviewers (continued)

G3. Gate, filter and full matching. Table R3 ablates the activation objective at α=1 and α=0.9 (three seeds each). At α=1, full GFT reaches 59.83 ± 0.21%, above no one-sided gate (56.90 ± 0.92%), no common-direction filter (55.86 ± 0.73%), full activation matching (55.07 ± 0.94%), and answer-CE (56.84 ± 0.46%). At α=0.9, full GFT achieves 59.34 ± 0.48%, again exceeding every ablation (55.07–56.90%) and answer-CE (55.68 ± 1.28%). These comparisons support the contribution of the gate, common-direction filter and answer-gradient-aware matching. At step 660, the sample-aligned validation ranges for GFT are 61.33–65.33% at α=1 and 62.67–66.67% at α=0.9, versus 45.33–52.00%/52.00–57.33% without the gate, 50.67–56.00%/56.00–60.00% without the filter, and 0.00–4.00%/10.67–25.33% for full matching (Table R4).

Table R3. Ablations, Precalculus 1.7B, test accuracy (%; 546 test problems). All rows use linear weighting and β=0.05; mean ± sd is calculated across three seeds for each α. Values are rounded from integer correct-prediction counts.

Objectiveα=1 s1333α=1 s1334α=1 s1335α=1 mean ± sdα=0.9 s1333α=0.9 s1334α=0.9 s1335α=0.9 mean ± sd
GFT (full)60.0759.7159.7159.83 ± 0.2159.7159.5258.7959.34 ± 0.48
No one-sided gate57.8856.7856.0456.90 ± 0.9257.8856.7856.0456.90 ± 0.92
No common-direction filter55.1356.5955.8655.86 ± 0.7355.1356.5955.8655.86 ± 0.73
Full activation matching55.3154.0355.8655.07 ± 0.9455.3154.0355.8655.07 ± 0.94
Answer-CE (no activations)57.3356.7856.4156.84 ± 0.4654.4055.6856.9655.68 ± 1.28

Table R4. Validation trajectories at α=1 and α=0.9. Precalculus 1.7B, validation accuracy (%) over 75 problems (initial validation accuracy 62.67%); Capped % summarizes checks from steps 120–660. The GFT s1333 (paper) curve was not retained.

Objectiveα=1 Step 60α=1 Step 120α=1 Step 660α=1 Capped %α=0.9 Step 60α=0.9 Step 120α=0.9 Step 660α=0.9 Capped %
GFT (available seeds)58.67–62.6761.33–64.0061.33–65.331–1060.00–62.6761.33–64.0062.67–66.671–8
No one-sided gate60.00–65.3352.00–58.6745.33–52.0010–2560.00–64.0057.33–61.3352.00–57.338–20
No common-direction filter58.67–61.3353.33–58.6750.67–56.005–1558.67–61.3358.67–61.3356.00–60.003–12
Full activation matching54.67–61.3312.00–21.330.00–4.0071–10054.67–61.3329.33–45.3310.67–25.3340–70

Note. Table R3 reports test accuracy at the validation-selected checkpoint, whereas Table R4 presents validation trajectories throughout training. The ablation runs selected checkpoints at step 60, before their later validation trajectories diverged. Furthermore, the relatively small evaluation sets (546 test and 75 validation problems) limit the resolution of accuracy measurements, making minor performance differences less distinguishable. Consequently, identical test accuracies across α settings can coexist with different late-training behavior.

G4. Starting model, SFT with the same KL, and shared subset. Paper SFT falls below the untrained student in 6 of 9 settings, by up to 28.21 points (Table R1). SFT with the same student KL, weight selected on validation, reaches 28.88 ± 3.23, no better than plain SFT, while GFT exceeds the starting model in all three seeds and achieves a higher mean than any answer-CE variant (Table R2). On the questions both finish (304/298 for seeds 1334/1335), GFT reaches 74.34/74.83% vs. 53.62/52.68% for SFT. SFT peaks at its first check (step 60, 0.7 epochs) for all three Precalculus sizes, with or without KL, already below the starting model, and only degrades afterwards (validation 16.0/38.7/60.0% at best vs. 44.0/62.7/80.0% at start, 12.0–25.3% at the end).

AuthorsOpenReview ↗

General Response to All Reviewers (continued)

G5. Seeds, GPU time and the GSM8K 4B gain. Across five settings, 53 of 56 three-seed GFT configurations outperform the corresponding SFT mean (median sd 0.71; Table R5). On GSM8K 4B, validation-selected GFT achieves 92.14 ± 0.68 versus an SFT mean of 90.11% (Table R5), a 2.03-point advantage. This three-seed comparison is distinct from the paper's seed-1333 test-selected +0.76-point result (Table R1). Per step, GFT costs about as much as answer-CE and SFT + KL, roughly 10× plain SFT (Table R6).

Table R5. Three-seed test accuracy (%; mean ± sd where reported); GFT selected by mean validation accuracy. Initial accuracies are reported under the same evaluation protocol as Table R1. Last column: number of fixed GFT configurations outperforming mean SFT accuracy (53/56 across the first five settings; GSM8K 4B has eight three-seed configurations). Precalculus 1.7B is reported separately in Tables R2–R3.

SettingUntrainedSFTGFTGFT > SFT
GSM8K 0.6B63.0862.98 ± 1.2666.11 ± 0.5711/12
GSM8K 1.7B80.7479.6983.83 ± 0.4912/12
GSM8K 4B90.4590.1192.14 ± 0.686/8
Int. Alg. 0.6B31.2318.01 ± 0.6132.74 ± 1.6112/12
Int. Alg. 1.7B52.1651.16 ± 1.0956.29 ± 1.2212/12
Precalc. 1.7B55.1328.88 ± 1.7859.34 ± 0.4812/12

Table R6. Training cost to early stopping, Precalculus 1.7B, one H200. Upper bounds: jobs restart every 30 min and reload the model.

MethodStepsGPU hs / step
SFT6600.382.1
SFT + student KL6604.0–5.221.8–28.4
Answer-CE660–7203.8–4.620.8–24.1
GFT ablations6603.8–5.120.7–27.9
GFT (α=0.9)7204.421.8

G6. Recommended default: α=0.9 with KL. It is the validation-selected configuration on Precalculus 1.7B and has the lowest seed variance in Table R2. Extraction-failure CE (Section 3.5, one tenth of the gradient) keeps a signal on problems where the student produces no answer.

Changes to the paper. (1) Untrained students in Table 1; validation-selected headline results, best of 12 as secondary. (2) A new section with answer-CE, SFT + KL, the ablations with training curves, and GPU time. (3) Three seeds and an updated Limitations section. (4) The shared-subset diagnostic in Section 5.1. (5) Gate and filter as design choices; the GSM8K 4B gain described as small. (6) The weighting selection stated in Section 4 and Appendix E. (7) α=0.9 with KL as the default.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.