- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses speculative-head gradients as layerwise control variates for supervised fine-tuning, estimating their mean with an EMA. It develops SGD and Adam variants and reports improved sample efficiency and downstream performance on GSM8K and XSum.
Critical strengths
- Using correlated auxiliary gradients for variance reduction is an interesting idea, supported by an exact-mean analysis and an analysis of EMA tracking error.
- Experiments cover full fine-tuning and LoRA, with comparisons against both standard SFT and MTP.
Critical weaknesses
- The main SGD experiments combine gradient correction with method-specific layerwise update scaling. This also changes effective learning rates, making it difficult to determine how much of the improvement comes from the control variate. Independently tuned baselines and a scaling-only control would strengthen the evidence.
- The headline 2.65× speedup is estimated from sample reduction and separately measured step latency. End-to-end time-to-quality measurements, including calibration and variation across training seeds, are needed to establish the practical benefit.
Questions for authors
- How much improvement remains against independently tuned Base/MTP baselines and a scaling-only control?
- Can you report end-to-end time to a fixed held-out quality target, including calibration, across independent training seeds?
We are grateful to the reviewer for highlighting the practical comparison.
We agree that the practical comparison is elapsed time to the same held-out GSM8K accuracy, including the cost of calibration. Our 2.65× estimate concerns a main-head loss threshold. Table 4 measures final accuracy. Neither measurement by itself establishes how quickly a tuned method reaches a given accuracy.
Gradient correction at matched layerwise rates. Appendix F.1 multiplies both the mean update and its noise by a layer-specific factor. The SGD results therefore measure the performance of gradient correction together with calibrated scaling. The Adam experiments give us one direct answer about the role of that scaling. On Qwen3.5-0.8B, with the SGD multipliers omitted, Base requires about 2.6 times as many examples as MTP+CV in full tuning and 2.5 times as many in LoRA to reach Base’s final main-head loss (Figures 13–14). The combined method can therefore improve sample efficiency without that multiplier.
For SGD, your scaling-only baseline gives the ordinary gradient exactly the layerwise rates available to the corrected gradient. Comparing those two runs at the same global rate and data order isolates the effect of the estimator. Comparing the two ordinary objectives at independently selected learning rates also gives MTP credit for any improvement it achieves through auxiliary supervision alone.
On 0.8B GSM8K with four depths, the example counts below are normalized to tuned Base = 1.00. The last column is the fraction of examples saved by correcting the gradient when the ordinary and corrected updates both use the CV multipliers.
Elapsed time to held-out accuracy. Table 4 reports endpoint GSM8K exact-match accuracy, averaged over five decoding seeds. The entries below are fractions of answers scored correct.
MTP+CV exceeds MTP by 0.0205 in accuracy on Qwen and by 0.0211 on Gemma. These endpoint differences motivate timing a shared accuracy target directly. That comparison varies the training seed while holding the evaluation questions and decoding rule fixed.
For the four-depth Qwen3.5-0.8B comparison below, the target is an absolute increase of 0.05 in held-out exact-match accuracy from the shared pretrained checkpoint, using greedy decoding. Time includes setup, training and evaluation through the first scheduled check reaching the target, with calibration charged to MTP+CV. Each row pairs the data order across methods within a training seed.
The median paired speedup is 1.29× over Base and 1.09× over MTP. The latter gives the relevant margin for a practitioner who already uses auxiliary supervision.
We will qualify the wall-clock claim as a projection from the measured step costs and the observed sample reduction. We would appreciate your taking the Adam results into account when reassessing the sample-efficiency claim, since those runs directly address your concern about the SGD multiplier.