acceptodds
Under review as a conference paper at ICLR 2027

Speculative Fine-Tuning: Accelerated Supervised Fine-Tuning with Speculative Heads

Abstract

Supervised fine-tuning is a standard method for adapting large language models to downstream tasks, but it can still require substantial compute, especially when repeated across models, datasets, and deployment settings. In this work, we show that speculative decoding, originally introduced for inference acceleration, can also be used to reduce the number of optimization steps required for supervised fine-tuning. Our key observation is that gradients from speculative heads are strongly correlated with gradients from the main language-model head. We use this correlation to construct control-variate gradient estimators that reduce stochastic gradient variance while preserving the corresponding training objective. We further show that the standard multi-token prediction (MTP) objective admits a natural control-variate interpretation, connecting MTP training to variance reduction. Moreover, we find that the correlation varies across the layers of the model, allowing us to develop a layer-aware variant of the method. Across supervised fine-tuning experiments, our proposed optimizer requires up to **85%** fewer training samples, corresponding to a **6.7×** improvement in sample efficiency, and implies an estimated **2.65×** wall-clock speedup using measured per-step latency, while reducing the final main loss by up to **12.2%**. To our knowledge, this is the first work to use speculative decoding for optimizer updates, separating the optimization role of speculative heads from their conventional use as auxiliary prediction heads. Across five evaluation seeds, MTP+CV improves GSM8K accuracy by 3.08 and 2.98 percentage points on Qwen3.5-4B and Gemma-4 E2B, respectively.

Primary area:
learning theory
Keywords:
Speculative decoding, fine-tuning, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ntHMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses speculative-head gradients as layerwise control variates for supervised fine-tuning, estimating their mean with an EMA. It develops SGD and Adam variants and reports improved sample efficiency and downstream performance on GSM8K and XSum.

Critical strengths
  1. Using correlated auxiliary gradients for variance reduction is an interesting idea, supported by an exact-mean analysis and an analysis of EMA tracking error.
  2. Experiments cover full fine-tuning and LoRA, with comparisons against both standard SFT and MTP.
Critical weaknesses
  1. The main SGD experiments combine gradient correction with method-specific layerwise update scaling. This also changes effective learning rates, making it difficult to determine how much of the improvement comes from the control variate. Independently tuned baselines and a scaling-only control would strengthen the evidence.
  2. The headline 2.65× speedup is estimated from sample reduction and separately measured step latency. End-to-end time-to-quality measurements, including calibration and variation across training seeds, are needed to establish the practical benefit.
Questions for authors
  1. How much improvement remains against independently tuned Base/MTP baselines and a scaling-only control?
  2. Can you report end-to-end time to a fixed held-out quality target, including calibration, across independent training seeds?
1 comment

We are grateful to the reviewer for highlighting the practical comparison.

We agree that the practical comparison is elapsed time to the same held-out GSM8K accuracy, including the cost of calibration. Our 2.65× estimate concerns a main-head loss threshold. Table 4 measures final accuracy. Neither measurement by itself establishes how quickly a tuned method reaches a given accuracy.

Gradient correction at matched layerwise rates. Appendix F.1 multiplies both the mean update and its noise by a layer-specific factor. The SGD results therefore measure the performance of gradient correction together with calibrated scaling. The Adam experiments give us one direct answer about the role of that scaling. On Qwen3.5-0.8B, with the SGD multipliers omitted, Base requires about 2.6 times as many examples as MTP+CV in full tuning and 2.5 times as many in LoRA to reach Base’s final main-head loss (Figures 13–14). The combined method can therefore improve sample efficiency without that multiplier.

For SGD, your scaling-only baseline gives the ordinary gradient exactly the layerwise rates available to the corrected gradient. Comparing those two runs at the same global rate and data order isolates the effect of the estimator. Comparing the two ordinary objectives at independently selected learning rates also gives MTP credit for any improvement it achieves through auxiliary supervision alone.

On 0.8B GSM8K with four depths, the example counts below are normalized to tuned Base = 1.00. The last column is the fraction of examples saved by correcting the gradient when the ordinary and corrected updates both use the CV multipliers.

ObjectiveOrdinary, unit scaleOrdinary, CV scaleCV, CV scaleFraction saved at matched scale
Base1.000.860.650.24
MTP0.950.880.300.66

Elapsed time to held-out accuracy. Table 4 reports endpoint GSM8K exact-match accuracy, averaged over five decoding seeds. The entries below are fractions of answers scored correct.

ModelBaseBase+CVMTPMTP+CV
Qwen3.5-4B0.69110.69230.70140.7219
Gemma-4 E2B0.77390.77090.78260.8037

MTP+CV exceeds MTP by 0.0205 in accuracy on Qwen and by 0.0211 on Gemma. These endpoint differences motivate timing a shared accuracy target directly. That comparison varies the training seed while holding the evaluation questions and decoding rule fixed.

For the four-depth Qwen3.5-0.8B comparison below, the target is an absolute increase of 0.05 in held-out exact-match accuracy from the shared pretrained checkpoint, using greedy decoding. Time includes setup, training and evaluation through the first scheduled check reaching the target, with calibration charged to MTP+CV. Each row pairs the data order across methods within a training seed.

Training seedTuned Base (min)Tuned MTP (min)MTP+CV (min)Speedup over MTP
12825211.19×
23126241.08×
32624221.09×

The median paired speedup is 1.29× over Base and 1.09× over MTP. The latter gives the relevant margin for a practitioner who already uses auxiliary supervision.

We will qualify the wall-clock claim as a projection from the measured step costs and the observed sample reduction. We would appreciate your taking the Adam results into account when reassessing the sample-efficiency claim, since those runs directly address your concern about the SGD multiplier.

Reviewer_3GWJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using speculative-head gradients as control variates to accelerate supervised fine-tuning. It introduces layerwise coefficients calibrated before training and an exponential moving average to estimate the auxiliary-gradient mean, yielding SpecSGD and SpecAdam. The analysis covers exact-mean variance reduction, its combination with an MTP objective, and EMA tracking error. Experiments on GSM8K and XSum evaluate loss reduction and downstream quality across multiple models and tuning settings. The paper reports improved sample efficiency and estimates wall-clock gains using measured per-step latency.

Critical strengths
  1. The paper identifies a useful optimization role for pretrained speculative heads. Exploiting their gradient correlation with the main task provides a concrete approach to reducing stochastic-gradient noise, with layerwise calibration accounting for differences across the network.

  2. The empirical investigation includes four training regimes, gradient-variance diagnostics, and experiments across model scales, full fine-tuning, and LoRA. The analysis also explicitly acknowledges and bounds EMA-induced tracking error, making the distinction between the ideal estimator and practical implementation clear.

Critical weaknesses
  1. The SGD comparison does not sufficiently isolate the benefit of the proposed estimator. Appendix F.1 applies different layerwise variance-matching factors to different methods. These factors also rescale the mean update and therefore change the effective learning rates. Without independently tuned Base/MTP baselines and controls isolating this rescaling, it remains unclear how much of the reported sample-efficiency improvement requires the proposed control variate.

  2. The headline acceleration results need stronger validation. The 2.65× speedup is explicitly estimated from sample reduction and separately measured step latency, rather than measured through a complete time-to-target run including calibration. Moreover, the 6.7× setting lacks Base+CV and MTP results in Table 6. The five seeds used for downstream evaluation are decoding seeds, which do not establish robustness across independent training runs. These gaps limit confidence in the magnitude and reproducibility of the practical gains.

Questions for authors
  1. Comparison protocol: Do the gains persist with independently tuned learning rates for Base and MTP? Could you isolate the effects of the control variate and the layerwise rescaling in Appendix F.1?

  2. Headline results: For the setting producing the 6.7× sample-efficiency gain, could you provide the missing Base+CV and MTP results, together with measured time-to-target including calibration and variability across training seeds?

  3. Interpretation of MTP: Proposition 1 rewrites the corrected MTP+CV estimator. In what sense does it explain variance reduction in ordinary MTP? Under the assumed positive covariance and a positive auxiliary weight, the uncorrected MTP gradient has greater raw variance than the main gradient.

3 comments

We thank the reviewer for their comments on Proposition 1 and for highlighting the distinction between the observed gain in sample efficiency and the estimated wall-clock speedup.

We agree with your distinction. In the four-depth Qwen3.5-0.8B GSM8K run, MTP+CV reached Base’s final main-head loss with 0.15 times the examples used by Base. This gives the 6.7× sample-efficiency result. The 2.65× figure combines that observation with a separate step-latency benchmark. We will specify the configuration alongside the sample count in the abstract.

Independent tuning and layerwise scaling. Appendix F.1 fixes the global rate and batch order. Its layer multiplier scales the mean update, so the SGD curves cannot isolate variance reduction. Figure 8 provides a direct test of the estimator at fixed parameters. On XSum, the median observed MTP+CV/MTP gradient-variance ratio is 0.171 at initialization, close to the prediction of 0.169 from calibration. With the SGD multipliers omitted and a common Adam schedule, Base requires about 2.6 times as many examples as MTP+CV to reach Base’s final loss in 0.8B full tuning.

We use a separately tuned rate for each ordinary baseline in the proposed comparison. Within each objective, holding that rate fixed while transferring the CV multipliers to the ordinary gradient isolates the effect of changing the estimator. The ordinary MTP arm uses the current minibatch auxiliary gradient. Setting the base CV coefficient to zero in Eq. (4) would still replace that gradient by its EMA and would test a different estimator.

Examples to the original four-depth loss threshold, normalized to tuned Base = 1.0. Within each pair, both runs use the rate selected for the ordinary gradient.

ObjectiveOrdinary, unit scaleCV, unit scaleOrdinary, CV scaleCV, CV scale
Base1.00.810.860.65
MTP0.950.490.880.30

Reading across each row separates the gain from gradient correction at unit scale from the gain when both methods have access to the CV multipliers.

The two missing entries in Table 6. The submitted 0.8B row establishes the 6.7× comparison with Base. On 4B GSM8K, where all four methods were evaluated, Base+CV achieves 1.5×, ordinary MTP exhausts its budget before reaching the target, and MTP+CV achieves 4.3×. The two configurations differ in both model size and prediction depth. For the original 0.8B comparison below, the denominator is the count for the submitted Base run, before independent tuning.

0.8B, original Table 6 protocolBaseBase+CVMTPMTP+CV
Fraction of Base examples1.000.800.900.15
Sample-efficiency gain1.00×1.25×1.11×6.7×

Taking the reciprocal gives the sample-efficiency gain in Table 6.

End-to-end time and independent training seeds. Appendix I.1 reports 65.922 ms per Base step and 165.436 ms per MTP+CV step. At equal batch size, the corrected method must use fewer than 0.40 times the number of Base steps to recover its additional per-step cost, even before calibration. The five seeds in Tables 4–5 vary decoding on trained checkpoints.

The target is the original 0.8B loss threshold. The clock includes setup, training, calibration where required, and evaluation through the first scheduled threshold crossing. Each pair shares the same data order, with independent training seeds across rows.

Training seedBase (min)MTP+CV (min)Speedup
16.23.81.63×
26.03.91.54×
36.44.11.56×

The median of the paired time ratios is 1.56×. This comparison uses the same loss target as the sample-efficiency result and charges calibration to MTP+CV.

Interpretation of ordinary MTP. We agree with your variance calculation. For layerwise minibatch gradients mm and ss, the raw variance of m+λsm + λs exceeds that of mm by 2λC+λ2Vs2λ C + λ^2 V_s where CC is the inner-product covariance of the two gradients and VsV_s is the speculative gradient variance. Both terms are positive under the stated assumptions and λ>0λ > 0. Proposition 1 concerns the corrected estimator. For a fixed aggregate speculative loss with exact gradient mean μμ, the optimal coefficient shifts by λλ, giving m−α∗(0)(s−μ)+λμm - α^{*}(0)(s-μ) + λμ. Changing the outer weight changes the mean direction and leaves the random part unchanged. We will revise Section 3.3 and the abstract to state this result explicitly.

We would appreciate your reconsideration in light of this narrower claim and the direct evidence for variance reduction in Figure 8.

Thank you for adding the matched-scaling controls and the end-to-end timing results. These address my questions about the comparison protocol and the cost of calibration.

For practical use, I give the most weight to elapsed time to the same downstream accuracy target against tuned MTP. On Qwen3.5-0.8B, MTP+CV reached the five-percentage-point GSM8K improvement in less time in all three runs, but the median speedup was 1.09×, or about 8% less training time. I find the control-variate idea interesting, but this measured gain is modest, and it remains unclear how well it carries over to larger models.

Thank you for your answer. Regarding scaling to larger models, on Qwen3.5-0.8B, MTP+CV reached the fixed GSM8K accuracy target sooner than tuned MTP in all three paired runs. For the larger models, our experiments measure sample efficiency and final task accuracy. We report both below.

Sample efficiency under a fixed training budget. Sample efficiency matters in two practical settings. If training stops at a target loss, a more sample-efficient method can stop after processing fewer examples. If the example budget is fixed, the benefit can instead appear as a lower final loss. Table 1 reports both comparisons on GSM8K. The sample-efficiency gain is the number of examples Base needs to reach its final main-head loss divided by the number MTP+CV needs to reach that same loss.

Qwen3.5 modelTuningSample-efficiency gain over BaseRelative reduction in final loss at equal example budget
0.8BFull6.7×0.122
4BFull4.3×0.102
4BLoRA3.0×0.048
9BLoRA3.5×0.038

Table 6 also provides a larger-model comparison against ordinary MTP. On XSum with Qwen3.5-4B full fine-tuning, MTP and MTP+CV achieved sample-efficiency gains of 1.3× and 3.5× over Base, respectively. Thus, under the submitted SGD protocol, MTP+CV used approximately 0.37 times as many examples as ordinary MTP to reach the same main-head loss.

Final downstream accuracy. We also evaluated whether the optimization gains carry through to answer correctness. Table 4 reports final GSM8K exact-match accuracy averaged over five decoding seeds.

ModelMTPMTP+CVAbsolute gain
Qwen3.5-4B0.70140.7219+0.0205
Gemma-4 E2B0.78260.8037+0.0211

The reduction in examples to a fixed loss extends to the 4B and 9B Qwen settings, and MTP+CV also improves final GSM8K accuracy over MTP on both models evaluated for that metric. Our direct timing comparison against tuned MTP is limited to 0.8B. Establishing the elapsed-time gain at larger scales requires timing those models to the same downstream accuracy target.

Moreover, the reported wall-clock speedup includes the cost of a control-variate backward pass that we have not optimized at the kernel level. Base benefits from PyTorch’s optimized backward-pass kernel. Applying comparable optimization to MTP+CV could lower its cost per step and translate the observed reduction in training steps into a larger wall-clock gain. Our primary contribution is to establish the theoretical basis for variance reduction with speculative gradients and to show empirically that this approach can reduce the number of steps needed to reach a target loss.

Reviewer_ZefaOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Speculative Fine-Tuning, which uses gradients from speculative prediction heads as layerwise control variates to reduce stochastic-gradient variance during supervised fine-tuning. The method estimates the mean speculative gradient using an exponential moving average and integrates the corrected gradients into SGD and Adam. It also extends the formulation to multi-token prediction (MTP). Experiments report fewer training examples needed to reach the baseline’s final main-head loss, together with improvements in downstream performance.

Critical strengths
  1. Reusing pretrained speculative heads as sources of correlated gradient information is an interesting idea. The paper identifies a concrete way to exploit these heads during fine-tuning, and the observed variation in gradient correlation across layers provides a reasonable motivation for layerwise coefficients.
  2. The control-variate construction and its main derivations are sound under the stated assumptions. The relationship between the optimal coefficients for the main objective and the MTP objective is particularly clear. The paper also explicitly acknowledges the bias introduced by EMA mean estimation and analyzes its tracking error in Appendix E.
  3. Experiments compare predicted and observed gradient-variance ratios using a separate development set. The close agreement at initialization provides useful support for the conditional variance analysis.
Critical weaknesses
  1. Using lower-variance gradients to support larger steps is reasonable. However, the current experiments do not establish how much of the acceleration comes from the control variate versus the calibrated scaling, or whether the gains persist against well-tuned standard optimizers. Four comparisons would be particularly informative: independently tuned Base and MTP, CV updates without variance-matching scaling, base with the same layerwise scaling as CV, stronger momentum SGD and AdamW comparisons.
  2. The theoretical contribution should be stated more precisely. Proposition 1 characterizes the CV-corrected MTP update; it does not establish that ordinary MTP benefits from variance reduction. Ordinary MTP may still improve optimization through its mean direction or auxiliary supervision, but that mechanism is not established by the proposed decomposition. A more precise contribution statement could be re-claimed.
Questions for authors

NA

2 comments

We appreciate the reviewer's thoughtful feedback.

As the review points out, the layerwise multiplier in Appendix F.1 scales both the mean SGD update and its noise. We agree. The sample-efficiency gain in Table 1 belongs to the complete training procedure. The most direct evidence for the control variate itself comes from Figure 8. At initialization on XSum, the predicted median layerwise variance ratio for MTP+CV relative to MTP is 0.169. On disjoint minibatches, the observed ratio is 0.171. Both gradients are evaluated at the same parameters with the same EMA held fixed. Since this measurement precedes the optimizer step, changing a layer’s effective learning rate cannot explain it.

Contribution of the layerwise scaling. Two training comparisons help narrow the attribution. First, Base+CV and MTP+CV use the same layerwise multiplier in Appendix F.1. In the 4B GSM8K full-tuning row of Table 6, they reach Base’s final main-head loss with sample-efficiency gains of 1.5× and 4.3×, respectively. The additional gain from introducing the auxiliary objective in this comparison occurs at fixed scaling. Second, the Adam protocol in Appendix F.2 omits the SGD multipliers entirely. On Qwen3.5-0.8B, Base needs about 2.6 times as many examples as MTP+CV in full tuning and 2.5 times as many in LoRA to reach Base’s final GSM8K loss (Figures 13–14). These results establish that the combined method can yield a training gain without the SGD multiplier. They leave the gain over an independently tuned ordinary gradient to be quantified.

The proposed scaling-only control isolates this comparison. Within each objective, the ordinary and corrected gradients receive the same global learning rate, schedule and minibatch order. Transferring the frozen CV multipliers to the ordinary gradient makes the layerwise rates identical as well. Selecting the global rate independently for ordinary Base and ordinary MTP gives each baseline its own operating point.

The table gives the fraction of examples needed to reach the original Base loss threshold on Qwen3.5-0.8B GSM8K with four prediction depths and full tuning. The denominator for every entry is the count for tuned Base. Within each row, both gradients use the rate selected for the ordinary gradient.

ObjectiveOrdinary, unit scaleCV, unit scaleOrdinary, CV scaleCV, CV scale
Base1.000.810.860.65
MTP0.950.490.880.30

At unit scale, correcting the gradient reduces the Base example count by a fraction of 0.20. At matched CV scale, the reduction is 0.24, relative to the scaling-only baseline in the same row. For MTP the corresponding reductions are 0.47 and 0.66. These comparisons hold the auxiliary objective fixed, so the difference within each pair measures the contribution of the gradient correction.

Momentum SGD and AdamW. Figures 13–14 use shared Adam hyperparameters. They establish that the effect survives adaptive normalization in those runs. An independently tuned optimizer comparison addresses a stricter claim. For momentum SGD the relevant search covers learning rate and momentum. For AdamW it covers learning rate and weight decay. The comparison below gives ordinary MTP and MTP+CV equal search budgets and evaluates each at its own selected settings on the same held-out target.

OptimizerTuned MTP, examplesTuned MTP+CV, examplesMTP+CV / MTP
Momentum SGD160014400.90
AdamW120011400.95

Here the denominator is the independently tuned MTP baseline for each optimizer. The resulting ratios give the sample cost of using the corrected gradient after the optimizer has already been tuned.

Scope of Proposition 1. We agree with your reading and will narrow the contribution statement to the following result. For a fixed aggregate speculative loss, the exact-mean control variate preserves the population MTP gradient. At the optimal coefficient, the variance of the corrected estimator is independent of the outer MTP weight. Ordinary MTP retains the minibatch noise in its auxiliary gradient. Its benefits may come from changing the mean training direction or from auxiliary supervision. Proposition 1 establishes that we can retain that mean direction while removing the additional stochastic term introduced by the auxiliary loss. Appendix E quantifies the tracking error incurred when the auxiliary mean is estimated with an EMA.

Thank you for the response, I will maintain the positive score, however, after read other reviewers' feedback, I will adjust my confidence score a bit lower.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.