- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TipSweep, which tunes the learning rate using a small number of trainable suffix blocks and transfers it to runs with more trainable blocks. The optimal rate approximately scales as (k^{-1}) with FP32 master-weight updates and (k^{-1/2}) with direct BF16 updates. A first-order loss model incorporating update rounding provides a conditional explanation for this difference. Experiments cover supervised finetuning and direct preference optimization, while frozen-prefix caching enables proxy searches for a 6.7B model on a consumer GPU.
Critical strengths
-
Varying the number of trainable blocks provides a practical transfer axis within an existing pretrained checkpoint. This avoids needing a separately pretrained model with a different width or depth.
-
The empirical study covers several model families, SFT and DPO, and trainable depths ranging from one block to the full transformer. Held-out loss results, block-placement experiments, and attention-only and MLP-only ablations provide additional support for the observed trends.
-
The cached implementation makes the proposed search procedure concrete. The cost analysis includes cache construction and the final full-depth run, while the runtime evaluation distinguishes consumer-GPU and H200 usage and discloses the cache-storage requirement.
Critical weaknesses
-
The precision comparison changes Adam-moment and gradient precision together with weight-update accumulation. The rounding probes provide useful evidence, but they use states and update directions from FP32 trajectories. The experiments therefore do not fully isolate weight-update rounding as the cause of the different scaling exponents.
-
The transferred learning rate does not consistently match the best downstream performance shown in the sweeps. For the coding experiment, the transferred BF16 rate obtains 57.3% on HumanEval+, while the full-depth sweep peaks at roughly 64% at another rate. This single-run difference needs uncertainty estimates, but it makes a matched comparison against the best full-depth rate important.
-
The analysis assumes that optimal learning rates maintain comparable per-step loss reduction, then derives the scaling exponents under additional assumptions about block contributions and rounding. This is a useful consistency argument, but it does not establish why the same criterion identifies the rate minimizing final loss after a finite training run.
-
The main sweeps use one training seed, although the paper acknowledges that flat optima can make rate selection sensitive to noise. The reliability of selecting a proxy rate and transferring it to one expensive full-depth run is therefore not quantified. Repeated runs near the candidate rates would be more informative than point estimates of the fitted exponents alone.
Questions for authors
-
Does the exponent difference persist when Adam moments and gradient accumulation remain in FP32, while only weight-update accumulation precision changes?
-
Please report the loss and downstream-performance gaps between the transferred rate and the best full-depth rate under the same training recipe. How stable are these gaps across repeated seeds?
-
Could you clarify the connection between the assumed per-step loss-reduction criterion and the learning rate that minimizes final training loss, and distinguish the theoretical conclusions from the empirically supported transfer rule?
5 comments
Rebuttal Two
W2. The transferred learning rate does not consistently match the best downstream performance shown in the sweeps. For the coding experiment, the transferred BF16 rate obtains 57.3% on HumanEval+, while the full-depth sweep peaks at roughly 64% at another rate. This single-run difference needs uncertainty estimates, but it makes a matched comparison against the best full-depth rate important.
Q2. Please report the loss and downstream-performance gaps between the transferred rate and the best full-depth rate under the same training recipe. How stable are these gaps across repeated seeds?
Thank you for raising this point. We want the case study to mimic how TipSweep is used in practice, so Table 1 reports pass@1 at the transferred rate instead of the best pass@1 in the full-depth sweep. We agree that uncertainty estimates across seeds are important, and we add a multi-seed experiment (Section 5.3 and Appendix E.4 of the revision) to further evaluate the quality of the transferred rate.
Setup. The submission uses seed 42. On two more seeds, 11 and 13, we run a proxy sweep, scale the selected rate to with our scaling rules, and do a full-depth run at the transferred rate. The model is DeepSeek-Coder-6.7B-Base and the training recipe is unchanged. Under BF16, the transferred rate is on all three seeds. Under FP32, it is on seeds 13 and 42 and on seed 11.
For comparison, we also do full-depth runs on these two seeds at the best rates in the full-depth sweep of the submission. The rate with the highest HumanEval+ pass@1 is under BF16 and under FP32. The rate with the lowest final training loss is under FP32, and under BF16 it is the transferred rate itself. Due to limited time, we do not run a full-depth sweep on the new seeds.
Downstream performance. The table below reports HumanEval+ pass@1 (%) of the full-depth runs, with the mean and standard deviation over the three seeds.
| Precision | Learning rate | Seed 11 | Seed 13 | Seed 42 | Mean ± std |
|---|---|---|---|---|---|
| BF16 | Transferred | 60.4 | 57.9 | 57.3 | 58.5 ± 1.6 |
| 57.3 | 56.7 | 64.0 | 59.3 ± 4.1 | ||
| FP32 | Transferred | 59.8 | 60.4 | 57.9 | 59.3 ± 1.3 |
| 57.3 | 54.3 | 60.4 | 57.3 ± 3.0 | ||
| 60.4 | 61.0 | 58.5 | 60.0 ± 1.3 |
On seed 42, and score higher than the transferred rate, but this does not persist across seeds. On seeds 11 and 13, the transferred rate scores higher than both. Over the three seeds, its mean is 0.8 points below under BF16 and 2.0 points above under FP32. The transferred rate is also more consistent across seeds than and . Under FP32, scores 0.6 points higher than the transferred rate on every seed, which corresponds to one problem.
Training loss. The table below reports the final EMA-smoothed training loss of the same runs.
| Precision | Learning rate | Seed 11 | Seed 13 | Seed 42 |
|---|---|---|---|---|
| BF16 | Transferred | 0.2082 | 0.2094 | 0.2109 |
| 0.2138 | 0.2148 | 0.2165 | ||
| FP32 | Transferred | 0.2038 | 0.2034 | 0.2086 |
| 0.2047 | 0.2054 | 0.2077 | ||
| 0.2029 | 0.2022 | 0.2060 |
Under BF16, the transferred rate gives a lower loss than on every seed. Under FP32, gives the lowest loss on every seed, and the transferred rate is within 0.0026 of it.
Rebuttal Three
W3. The analysis assumes that optimal learning rates maintain comparable per-step loss reduction, then derives the scaling exponents under additional assumptions about block contributions and rounding. This is a useful consistency argument, but it does not establish why the same criterion identifies the rate minimizing final loss after a finite training run.
Q3. Could you clarify the connection between the assumed per-step loss-reduction criterion and the learning rate that minimizes final training loss, and distinguish the theoretical conclusions from the empirically supported transfer rule?
Thank you for raising this point. We agree the paper should make this connection clearer. The per-step loss-reduction criterion (Assumption 1) is a hypothesis about how the optimal learning rate scales with the number of trainable blocks . It states that this rate keeps the per-step loss reduction at as varies, so that learning progress neither vanishes nor explodes. This assumption is the loss-level counterpart of the criterion for learning rate transfer across model size, which chooses the rate so that each update changes the features by [1,2,3]. Since hides constants independent of , the assumption constrains the scaling of the optimal rate but does not identify its value at a given . TipSweep instead obtains the optimal rate at from the proxy sweep and then transfers it to other with the scaling rule. We have revised Section 4 to state this distinction explicitly.
The role of our analysis is to explain the precision-dependent scaling rules that we observe empirically in our learning rate sweeps, where we select the rate with the lowest EMA-smoothed final training loss. Its assumptions are supported by our profiling experiments. For Assumption 1, Figures 3a and 8 show that the per-step loss reduction at the empirically tuned learning rate stays on a comparable scale across .
[1] Yang & Hu (2021) Tensor programs iv: Feature learning in infinite-width neural networks.
[2] Hayou et al. (2024) The impact of initialization on lora finetuning dynamics.
[3] Hayou et al. (2024) LoRA+: Efficient low rank adaptation of large models.
W4. The main sweeps use one training seed, although the paper acknowledges that flat optima can make rate selection sensitive to noise. The reliability of selecting a proxy rate and transferring it to one expensive full-depth run is therefore not quantified. Repeated runs near the candidate rates would be more informative than point estimates of the fitted exponents alone.
Thank you for this suggestion. The main sweeps use one training seed because repeating them is expensive. Each additional seed adds a learning rate sweep for every , precision, and setting, and we chose to spend this compute on covering more models, datasets, and post-training regimes.
The multi-seed experiment in our response to W2 and Q2 quantifies the reliability of selecting a proxy rate and transferring it. For DeepSeek-Coder-6.7B-Base under both precisions, it includes a proxy sweep and a full-depth run at the transferred rate on each of three seeds. Under BF16, the proxy sweep selects the same rate on all three seeds. Under FP32, where the curves are flatter near their minima, the selected rate differs on one seed by a factor of , the smallest step of our grid. At the transferred rate, HumanEval+ pass@1 varies by at most five of the 164 problems across seeds.
We agree that repeated runs near the candidate rates are informative. Such runs are affordable only within a narrow range of learning rates, and TipSweep can help locate this range cheaply through the proxy sweep.
Rebuttal One
Thank you for your time and constructive feedback. For questions raised by multiple reviewers, we will provide the same response to each reviewer to ensure consistency. We have revised our manuscript based on your feedback, with all changes highlighted in blue.
Thank you for this suggestion. We have added the requested comparison in Appendix C.2 of the revised manuscript. We repeat the SFT runs of Llama-3.2-1B and DeepSeek-Coder-6.7B-Base and the DPO runs of Llama-3.2-1B-Instruct with direct BF16 updates, but keep the gradients, gradient accumulation, and Adam moments in FP32. All other settings are unchanged. For DeepSeek-Coder-6.7B-Base, the new runs use DeepSpeed ZeRO-3 on two GPUs as in its FP32 runs, only without the FP32 master copy. In all three settings, the new runs thus differ from their FP32 counterparts only in update precision. Results are shown in Figure 19 of the revision, and the table below summarizes the log2 of the optimal learning rate for each k and the fitted log-log slope β. For comparison, we also include the original results from Section 5.1, where the gradients and optimizer states are in BF16. Across all three settings, the optimal learning rate still scales approximately as k−1/2. The change in the scaling rule therefore comes from weight-update rounding, not from the precision of gradients or optimizer states.
We take the states and update directions for the rounding probes from FP32 runs to separate the effect of update rounding from other potential effects of BF16 training. The probes then vary only the learning rate, so they measure how ϕk(η) changes with η (Assumption 3) without comparing separate runs whose trajectories differ.