acceptodds
Under review as a conference paper at ICLR 2027

Precision-Aware Learning Rate Transfer Across Block Count in LLM Post-Training

Abstract

The post-training of pretrained large language models, including supervised finetuning (SFT) and direct preference optimization (DPO), is very sensitive to the learning rate, and tuning it through repeated runs is costly in both computation and memory. In this work, we propose TipSweep, a method to transfer the learning rate across the number of trainable transformer blocks. To do so efficiently, we search for the best learning rate by training only the last blocks, and show how to scale the selected rate for a subsequent set of blocks. Across SFT and DPO experiments with models ranging from 1 to 7 billion parameters, we observe that the optimal learning rate follows precision-dependent scaling rules: the optimal rate scales approximately as with float32 (FP32) master weights, but as when updates are applied directly to bfloat16 (BF16) weights without a master copy. We explain both rules using an analytical model that accounts for update rounding, under assumptions supported by profiling experiments. Finally, we develop a prototype that demonstrates the feasibility of learning rate search for a 6.7-billion-parameter model on a single consumer GPU with 16 GiB of memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Learning-Rate Transfer, Large Language Models, LLM Post-Training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_gTbaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TipSweep, which tunes the learning rate using a small number of trainable suffix blocks and transfers it to runs with more trainable blocks. The optimal rate approximately scales as (k^{-1}) with FP32 master-weight updates and (k^{-1/2}) with direct BF16 updates. A first-order loss model incorporating update rounding provides a conditional explanation for this difference. Experiments cover supervised finetuning and direct preference optimization, while frozen-prefix caching enables proxy searches for a 6.7B model on a consumer GPU.

Critical strengths
  • Varying the number of trainable blocks provides a practical transfer axis within an existing pretrained checkpoint. This avoids needing a separately pretrained model with a different width or depth.

  • The empirical study covers several model families, SFT and DPO, and trainable depths ranging from one block to the full transformer. Held-out loss results, block-placement experiments, and attention-only and MLP-only ablations provide additional support for the observed trends.

  • The cached implementation makes the proposed search procedure concrete. The cost analysis includes cache construction and the final full-depth run, while the runtime evaluation distinguishes consumer-GPU and H200 usage and discloses the cache-storage requirement.

Critical weaknesses
  • The precision comparison changes Adam-moment and gradient precision together with weight-update accumulation. The rounding probes provide useful evidence, but they use states and update directions from FP32 trajectories. The experiments therefore do not fully isolate weight-update rounding as the cause of the different scaling exponents.

  • The transferred learning rate does not consistently match the best downstream performance shown in the sweeps. For the coding experiment, the transferred BF16 rate obtains 57.3% on HumanEval+, while the full-depth sweep peaks at roughly 64% at another rate. This single-run difference needs uncertainty estimates, but it makes a matched comparison against the best full-depth rate important.

  • The analysis assumes that optimal learning rates maintain comparable per-step loss reduction, then derives the scaling exponents under additional assumptions about block contributions and rounding. This is a useful consistency argument, but it does not establish why the same criterion identifies the rate minimizing final loss after a finite training run.

  • The main sweeps use one training seed, although the paper acknowledges that flat optima can make rate selection sensitive to noise. The reliability of selecting a proxy rate and transferring it to one expensive full-depth run is therefore not quantified. Repeated runs near the candidate rates would be more informative than point estimates of the fitted exponents alone.

Questions for authors
  • Does the exponent difference persist when Adam moments and gradient accumulation remain in FP32, while only weight-update accumulation precision changes?

  • Please report the loss and downstream-performance gaps between the transferred rate and the best full-depth rate under the same training recipe. How stable are these gaps across repeated seeds?

  • Could you clarify the connection between the assumed per-step loss-reduction criterion and the learning rate that minimizes final training loss, and distinguish the theoretical conclusions from the empirically supported transfer rule?

5 comments
AuthorsOpenReview ↗

Rebuttal One

Thank you for your time and constructive feedback. For questions raised by multiple reviewers, we will provide the same response to each reviewer to ensure consistency. We have revised our manuscript based on your feedback, with all changes highlighted in blue.

W1. The precision comparison changes Adam-moment and gradient precision together with weight-update accumulation. The rounding probes provide useful evidence, but they use states and update directions from FP32 trajectories. The experiments therefore do not fully isolate weight-update rounding as the cause of the different scaling exponents.

Q1. Does the exponent difference persist when Adam moments and gradient accumulation remain in FP32, while only weight-update accumulation precision changes?

Thank you for this suggestion. We have added the requested comparison in Appendix C.2 of the revised manuscript. We repeat the SFT runs of Llama-3.2-1B and DeepSeek-Coder-6.7B-Base and the DPO runs of Llama-3.2-1B-Instruct with direct BF16 updates, but keep the gradients, gradient accumulation, and Adam moments in FP32. All other settings are unchanged. For DeepSeek-Coder-6.7B-Base, the new runs use DeepSpeed ZeRO-3 on two GPUs as in its FP32 runs, only without the FP32 master copy. In all three settings, the new runs thus differ from their FP32 counterparts only in update precision. Results are shown in Figure 19 of the revision, and the table below summarizes the log⁡2\log_2 of the optimal learning rate for each kk and the fitted log-log slope β\beta. For comparison, we also include the original results from Section 5.1, where the gradients and optimizer states are in BF16. Across all three settings, the optimal learning rate still scales approximately as k−1/2k^{-1/2}. The change in the scaling rule therefore comes from weight-update rounding, not from the precision of gradients or optimizer states.

SettingGradient & Optimizer precisionk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16k=32k=32Fitted β\beta
Llama-3.2-1B (SFT)FP32-13.5-14.0-14.5-15.0-15.5–-0.50
BF16-13.0-13.5-14.0-14.5-15.0–-0.50
DeepSeek-Coder-6.7B-Base (SFT)FP32-11.5-12.0-12.5-12.5-13.0-13.5-0.37
BF16-11.5-12.0-12.5-13.0-13.5-14.0-0.50
Llama-3.2-1B-Instruct (DPO)FP32-16.0-16.0-17.0-17.5-18.0–-0.55
BF16-15.5-16.0-16.5-17.0-17.5–-0.50

We take the states and update directions for the rounding probes from FP32 runs to separate the effect of update rounding from other potential effects of BF16 training. The probes then vary only the learning rate, so they measure how ϕk(η)\phi_k(\eta) changes with η\eta (Assumption 3) without comparing separate runs whose trajectories differ.

AuthorsOpenReview ↗

Rebuttal Two

W2. The transferred learning rate does not consistently match the best downstream performance shown in the sweeps. For the coding experiment, the transferred BF16 rate obtains 57.3% on HumanEval+, while the full-depth sweep peaks at roughly 64% at another rate. This single-run difference needs uncertainty estimates, but it makes a matched comparison against the best full-depth rate important.

Q2. Please report the loss and downstream-performance gaps between the transferred rate and the best full-depth rate under the same training recipe. How stable are these gaps across repeated seeds?

Thank you for raising this point. We want the case study to mimic how TipSweep is used in practice, so Table 1 reports pass@1 at the transferred rate instead of the best pass@1 in the full-depth sweep. We agree that uncertainty estimates across seeds are important, and we add a multi-seed experiment (Section 5.3 and Appendix E.4 of the revision) to further evaluate the quality of the transferred rate.

Setup. The submission uses seed 42. On two more seeds, 11 and 13, we run a k0=1k_0 = 1 proxy sweep, scale the selected rate to k=32k = 32 with our scaling rules, and do a full-depth run at the transferred rate. The model is DeepSeek-Coder-6.7B-Base and the training recipe is unchanged. Under BF16, the transferred rate is 2−142^{-14} on all three seeds. Under FP32, it is 2−172^{-17} on seeds 13 and 42 and 2−16.52^{-16.5} on seed 11.

For comparison, we also do full-depth runs on these two seeds at the best rates in the full-depth sweep of the submission. The rate with the highest HumanEval+ pass@1 is 2−132^{-13} under BF16 and 2−152^{-15} under FP32. The rate with the lowest final training loss is 2−162^{-16} under FP32, and under BF16 it is the transferred rate itself. Due to limited time, we do not run a full-depth sweep on the new seeds.

Downstream performance. The table below reports HumanEval+ pass@1 (%) of the full-depth runs, with the mean and standard deviation over the three seeds.

PrecisionLearning rateSeed 11Seed 13Seed 42Mean ± std
BF16Transferred60.457.957.358.5 ± 1.6
2−132^{-13}57.356.764.059.3 ± 4.1
FP32Transferred59.860.457.959.3 ± 1.3
2−152^{-15}57.354.360.457.3 ± 3.0
2−162^{-16}60.461.058.560.0 ± 1.3

On seed 42, 2−132^{-13} and 2−152^{-15} score higher than the transferred rate, but this does not persist across seeds. On seeds 11 and 13, the transferred rate scores higher than both. Over the three seeds, its mean is 0.8 points below 2−132^{-13} under BF16 and 2.0 points above 2−152^{-15} under FP32. The transferred rate is also more consistent across seeds than 2−132^{-13} and 2−152^{-15}. Under FP32, 2−162^{-16} scores 0.6 points higher than the transferred rate on every seed, which corresponds to one problem.

Training loss. The table below reports the final EMA-smoothed training loss of the same runs.

PrecisionLearning rateSeed 11Seed 13Seed 42
BF16Transferred0.20820.20940.2109
2−132^{-13}0.21380.21480.2165
FP32Transferred0.20380.20340.2086
2−152^{-15}0.20470.20540.2077
2−162^{-16}0.20290.20220.2060

Under BF16, the transferred rate gives a lower loss than 2−132^{-13} on every seed. Under FP32, 2−162^{-16} gives the lowest loss on every seed, and the transferred rate is within 0.0026 of it.

AuthorsOpenReview ↗

Rebuttal Three

W3. The analysis assumes that optimal learning rates maintain comparable per-step loss reduction, then derives the scaling exponents under additional assumptions about block contributions and rounding. This is a useful consistency argument, but it does not establish why the same criterion identifies the rate minimizing final loss after a finite training run.

Q3. Could you clarify the connection between the assumed per-step loss-reduction criterion and the learning rate that minimizes final training loss, and distinguish the theoretical conclusions from the empirically supported transfer rule?

Thank you for raising this point. We agree the paper should make this connection clearer. The per-step loss-reduction criterion (Assumption 1) is a hypothesis about how the optimal learning rate scales with the number of trainable blocks kk. It states that this rate keeps the per-step loss reduction at Θ(1)\Theta(1) as kk varies, so that learning progress neither vanishes nor explodes. This assumption is the loss-level counterpart of the criterion for learning rate transfer across model size, which chooses the rate so that each update changes the features by Θ(1)\Theta(1) [1,2,3]. Since Θ(1)\Theta(1) hides constants independent of kk, the assumption constrains the scaling of the optimal rate but does not identify its value at a given kk. TipSweep instead obtains the optimal rate at k0k_0 from the proxy sweep and then transfers it to other kk with the scaling rule. We have revised Section 4 to state this distinction explicitly.

The role of our analysis is to explain the precision-dependent scaling rules that we observe empirically in our learning rate sweeps, where we select the rate with the lowest EMA-smoothed final training loss. Its assumptions are supported by our profiling experiments. For Assumption 1, Figures 3a and 8 show that the per-step loss reduction at the empirically tuned learning rate stays on a comparable scale across kk.

[1] Yang & Hu (2021) Tensor programs iv: Feature learning in infinite-width neural networks.

[2] Hayou et al. (2024) The impact of initialization on lora finetuning dynamics.

[3] Hayou et al. (2024) LoRA+: Efficient low rank adaptation of large models.

W4. The main sweeps use one training seed, although the paper acknowledges that flat optima can make rate selection sensitive to noise. The reliability of selecting a proxy rate and transferring it to one expensive full-depth run is therefore not quantified. Repeated runs near the candidate rates would be more informative than point estimates of the fitted exponents alone.

Thank you for this suggestion. The main sweeps use one training seed because repeating them is expensive. Each additional seed adds a learning rate sweep for every kk, precision, and setting, and we chose to spend this compute on covering more models, datasets, and post-training regimes.

The multi-seed experiment in our response to W2 and Q2 quantifies the reliability of selecting a proxy rate and transferring it. For DeepSeek-Coder-6.7B-Base under both precisions, it includes a k0=1k_0 = 1 proxy sweep and a full-depth run at the transferred rate on each of three seeds. Under BF16, the proxy sweep selects the same rate on all three seeds. Under FP32, where the curves are flatter near their minima, the selected rate differs on one seed by a factor of 2\sqrt{2}, the smallest step of our grid. At the transferred rate, HumanEval+ pass@1 varies by at most five of the 164 problems across seeds.

We agree that repeated runs near the candidate rates are informative. Such runs are affordable only within a narrow range of learning rates, and TipSweep can help locate this range cheaply through the proxy sweep.

Reviewer_9dfVOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes using the local information already available in an N-gram embedding branch to guide subsequent Transformer computation. A learned certainty signal coordinates attention writes, reuse of the initial embedding, attention span, and preference for nearby context. Attention statistics update this signal across layers.

Critical strengths

The method addresses a real cost in post-training with a simple, useful idea. Learning rate tuning can consume several times the resources of the final training run. Using the number of trainable blocks as a transfer axis makes sense for pretrained checkpoints, whose architecture is already fixed. The caching implementation gives the idea practical substance: repeated trials can run on hardware that could not accommodate full-depth training. This could be particularly valuable for groups with limited access to large GPUs, since it lets them reserve those resources for the final run.

Critical weaknesses

I think the cost advantage needs a more realistic search baseline. The systems results establish that TipSweep can move tuning onto cheaper hardware. Its advantage over other economical tuning strategies is less clear. The comparison assumes that every candidate in the full-depth sweep completes an entire training run; shortened trials or early stopping are not evaluated.

The fixed costs also matter: the case study requires approximately 337 GiB of cached features and nearly four hours to construct them on the consumer GPU. At five candidates, total GPU hours are roughly comparable to the full-depth sweep, with the benefit coming from using less expensive hardware. These are reasonable tradeoffs, but they make the method’s attractiveness depend on the search budget and available hardware. A comparison at matched final quality would better establish when the additional caching workflow is worthwhile.

Questions for authors

The FP32 and BF16 configurations also differ in optimizer-state precision. Do you have a training comparison that changes weight-update rounding while holding optimizer-state precision fixed? The rounding probes are useful, but this comparison would strengthen the attribution of the different scaling rules to update rounding.

2 comments
AuthorsOpenReview ↗

Rebuttal One

Thank you for your time and constructive feedback. For questions raised by multiple reviewers, we will provide the same response to each reviewer to ensure consistency. We have revised our manuscript based on your feedback, with all changes highlighted in blue.

Q1. The FP32 and BF16 configurations also differ in optimizer-state precision. Do you have a training comparison that changes weight-update rounding while holding optimizer-state precision fixed? The rounding probes are useful, but this comparison would strengthen the attribution of the different scaling rules to update rounding.

Thank you for this suggestion. We have added the requested comparison in Appendix C.2 of the revised manuscript. We repeat the SFT runs of Llama-3.2-1B and DeepSeek-Coder-6.7B-Base and the DPO runs of Llama-3.2-1B-Instruct with direct BF16 updates, but keep the gradients, gradient accumulation, and Adam moments in FP32. All other settings are unchanged. For DeepSeek-Coder-6.7B-Base, the new runs use DeepSpeed ZeRO-3 on two GPUs as in its FP32 runs, only without the FP32 master copy. In all three settings, the new runs thus differ from their FP32 counterparts only in update precision. Results are shown in Figure 19 of the revision, and the table below summarizes the log⁡2\log_2 of the optimal learning rate for each kk and the fitted log-log slope β\beta. For comparison, we also include the original results from Section 5.1, where the gradients and optimizer states are in BF16. Across all three settings, the optimal learning rate still scales approximately as k−1/2k^{-1/2}. The change in the scaling rule therefore comes from weight-update rounding, not from the precision of gradients or optimizer states.

SettingGradient & Optimizer precisionk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16k=32k=32Fitted β\beta
Llama-3.2-1B (SFT)FP32-13.5-14.0-14.5-15.0-15.5–-0.50
BF16-13.0-13.5-14.0-14.5-15.0–-0.50
DeepSeek-Coder-6.7B-Base (SFT)FP32-11.5-12.0-12.5-12.5-13.0-13.5-0.37
BF16-11.5-12.0-12.5-13.0-13.5-14.0-0.50
Llama-3.2-1B-Instruct (DPO)FP32-16.0-16.0-17.0-17.5-18.0–-0.55
BF16-15.5-16.0-16.5-17.0-17.5–-0.50

W1. I think the cost advantage needs a more realistic search baseline. The systems results establish that TipSweep can move tuning onto cheaper hardware. Its advantage over other economical tuning strategies is less clear. The comparison assumes that every candidate in the full-depth sweep completes an entire training run; shortened trials or early stopping are not evaluated.

Thank you for raising this point. We agree that shortened trials and early stopping are both effective tuning strategies. However, these strategies are orthogonal to TipSweep. They reduce the number of training steps per trial, while TipSweep reduces the memory and computation per step. For example, full-depth training in our setup does not fit on the 16 GiB consumer GPU even at a batch size of 1 (Section 5.3). Shortening a trial does not lower this barrier, whereas TipSweep can run the whole search on this GPU. At the same time, these strategies are complementary to our method. Just as they are applied to full-depth trials, they can be applied to our cached proxy trials to further reduce the tuning cost. We leave this combination for future work and have noted it in Section 6 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Two

W2. The fixed costs also matter: the case study requires approximately 337 GiB of cached features and nearly four hours to construct them on the consumer GPU. At five candidates, total GPU hours are roughly comparable to the full-depth sweep, with the benefit coming from using less expensive hardware. These are reasonable tradeoffs, but they make the method’s attractiveness depend on the search budget and available hardware. A comparison at matched final quality would better establish when the additional caching workflow is worthwhile.

Thank you for this feedback. The main advantage of the two-stage prototype is that the whole learning rate search can run on a widely available consumer GPU, so the high-end GPU is reserved only for the final full-depth run, regardless of RR. In total GPU hours, it roughly matches the full-depth sweep at R=5R = 5, but about 80% of these hours are spent on the consumer GPU. We therefore expect the two-stage prototype to suit settings where consumer GPUs and disk storage are more accessible than high-end GPUs.

To remove the feature cache of the two-stage prototype, we introduce a cache-free prototype that leverages high-end GPUs for the search. Its details and results are in Appendix F of the revised manuscript. The core idea is to hold RR copies of the last k0k_0 blocks in GPU memory and train each copy at a different candidate learning rate. The frozen prefix processes each batch once and feeds its output to all copies, so a single pass over the dataset completes all RR proxy trials and no features are stored on disk. The selected rate is then scaled for the full-depth run following the same rules as in the two-stage prototype.

The table below reports the peak allocated GPU memory of this prototype in GiB, for DeepSeek-Coder-6.7B-Base with k0=1k_0 = 1 on a single 140 GiB H200. BB is the per-GPU batch size, and every sequence contains 1024 tokens. In our experiments, each additional candidate adds 1.51 GiB under BF16 and 3.02 GiB under FP32. At B=64B = 64, the H200 holds up to 35 candidates under FP32 (135.35 GiB), while 35 candidates under BF16 take only 81.07 GiB.

RRBF16, B=1B = 1BF16, B=16B = 16BF16, B=64B = 64FP32, B=1B = 1FP32, B=16B = 16FP32, B=64B = 64
519.8623.5135.8427.4131.4344.88
1027.4031.0543.3842.4946.5059.95
2042.4846.1358.4572.6576.6690.11
3565.0968.7581.07117.88121.90135.35
4072.6376.2988.61132.96136.98OOM

The table below compares its H200 GPU hours with those of the full-depth sweep. The search has a fixed cost of about half an hour for the forward pass through the frozen prefix and takes about 3 minutes per additional candidate. In the time of one full-depth run (1.48 h), the search covers 20 trials under BF16. Including the final full-depth run, the speed-up over the full-depth sweep is 3.3×3.3\times at R=5R = 5 and 10.0×10.0\times at R=20R = 20 under BF16, and 3.4×3.4\times and 10.5×10.5\times under FP32. This speed-up should keep growing with RR as long as the RR copies fit in GPU memory.

PrecisionRRSearchSearch + final runFull-depth sweepSpeed-up
BF1650.732.217.383.3×3.3\times
100.992.4714.766.0×6.0\times
151.242.7222.148.1×8.1\times
201.482.9629.5210.0×10.0\times
FP3250.762.699.143.4×3.4\times
101.032.9618.296.2×6.2\times
151.303.2327.438.5×8.5\times
201.573.5036.5810.5×10.5\times

We also compare the final quality under the same training recipe across three seeds in Section 5.3, Table 2, and Appendix E.4. These results compare the transferred rate with the full-depth sweep rates selected by HumanEval+ and training loss. The runtime comparison does not measure the cost of reaching a fixed quality target.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Rebuttal One

Thank you for your time and constructive feedback. For questions raised by multiple reviewers, we will provide the same response to each reviewer to ensure consistency. We have revised our manuscript based on your feedback, with all changes highlighted in blue.

W1-a. The theory isolates precision through strong assumptions, and their validation is incomplete and partly circular.

  • Assumption 1 is essentially the conclusion. The first-order model has no optimum, since ηDkϕk(η)\eta D_k\phi_k(\eta) grows monotonically in η\eta, so the optimum comes from Assumption 1 itself. Given Dk∝kD_k\propto k, Assumption 1 is equivalent to the observed ηk∗∝1/k\eta_k^*\propto 1/k, so it cannot be tested independently; Fig. 8, measured at rule-prescribed rates, only confirms Dk∝kD_k\propto k. The analysis should be presented as a consistency argument, not a derivation.

Thank you for raising this point. The first-order model expresses the per-step loss reduction as ηDkϕk(η)\eta D_k\phi_k(\eta), where DkD_k sums the nominal contributions of the trainable blocks and ϕk\phi_k is the update factor that accounts for update rounding. It is not meant to locate the optimum. Assumption 1 is a hypothesis about how the optimal learning rate scales with the number of trainable blocks kk. It states that this rate keeps the per-step loss reduction at Θ(1)\Theta(1) as kk varies, so that learning progress neither vanishes nor explodes. This is the loss-level counterpart of the criterion for learning rate transfer across model size, which chooses the rate so that each update changes the features by Θ(1)\Theta(1) [1,2,3]. Since Θ(1)\Theta(1) hides constants independent of kk, the assumption constrains the scaling of the optimal rate but does not identify its value. TipSweep obtains this value at k0k_0 from the proxy sweep and then transfers it to other kk with the scaling rule. We have revised Section 4 to state this distinction explicitly.

Assumption 1 is also not the conclusion. It gives an exponent only when combined with Dk=Θ(k)D_k=\Theta(k) from Assumption 2 and ϕk(η)=Θ(ηα)\phi_k(\eta)=\Theta(\eta^\alpha) from Assumption 3. The same criterion gives k−1k^{-1} under FP32, where ϕk=1\phi_k=1, and k−1/2k^{-1/2} under BF16, where ϕk(η)=Θ(η)\phi_k(\eta)=\Theta(\eta). Assumption 1 alone therefore cannot give these precision-dependent scaling rules.

We support Assumption 1 with the profiling in Figures 3a and 8, where the per-step loss reduction at the empirically tuned learning rate stays on a comparable scale across kk under both FP32 and BF16. In the BF16 panels of Figure 8 (b, d), a comparable loss reduction at η∝1/k\eta\propto 1/\sqrt{k} requires the realized sum D~k(η)=Dkϕk(η)\tilde{D}_k(\eta)=D_k\phi_k(\eta) to grow as k\sqrt{k}, which does not follow from Dk∝kD_k\propto k alone.

Overall, the role of our analysis is to explain the precision-dependent scaling rules that we observe empirically in our learning rate sweeps, under assumptions supported by our profiling experiments. In this sense, we agree that it can be viewed as a consistency argument.

[1] Yang & Hu (2021) Tensor programs iv: Feature learning in infinite-width neural networks.

[2] Hayou et al. (2024) The impact of initialization on lora finetuning dynamics.

[3] Hayou et al. (2024) LoRA+: Efficient low rank adaptation of large models.

W1-b. The theory isolates precision through strong assumptions, and their validation is incomplete and partly circular.

  • Other hyperparameters are fixed and absorbed into the Θ(⋅)\Theta(\cdot) constants, although some plausibly interact with kk: batch size sets gradient noise, and weight decay (fixed at 0) and Adam ϵ\epsilon change the weight and update magnitudes that govern the BF16 regime. The rules should be checked under at least one other batch size and a nonzero weight decay, or stated as conditional on these settings.

Thank you for this suggestion.

Batch size. Our experiments cover two batch sizes, 32 examples for SFT and 8 preference pairs for DPO. The optimal learning rate follows the same scaling rules in both.

Weight decay. We have added an experiment with a nonzero weight decay in Appendix C.1 (Figure 18) of the revised manuscript. We repeat the SFT sweeps of Llama-3.2-1B on Tülu 3 under direct BF16 updates with a weight decay of 0.01. All other settings are unchanged. The table below lists log⁡2\log_2 of the optimal learning rate for each kk, selected by the final training loss or by the validation NLL. The last column is the slope β\beta of a least-squares fit of log⁡2ηk\log_2 \eta_k against log⁡2k\log_2 k. The optimal learning rate scales as k−1/2k^{-1/2}, as the case without weight decay.

Criterionk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16β\beta
Final training loss-13-13.5-14-14.5-15-0.50
Validation NLL-13-13.5-14-14.5-15-0.50
AuthorsOpenReview ↗

Rebuttal Two

W1-c. The theory isolates precision through strong assumptions, and their validation is incomplete and partly circular.

  • Some observations do not fit the assumptions. The SFT first-order error is only 10−210^{-2}-10−110^{-1} (Fig. 7a, b), so training operates far from the curvature limit and the optimum is set by factors outside the model; DPO-FP32 at k=16k=16 reaches ≈0.5\approx 0.5-11 (Fig. 7c), close to the ϵ<1\epsilon<1 boundary of Proposition 4.1. These should be discussed rather than cited as support.

Thank you for raising this point. As in our reply to W1-a, our analysis is not meant to locate the optimum but to describe how it scales with the number of trainable blocks kk. It therefore does not assume that the optimum sits at a curvature limit. The first-order approximation in Proposition 4.1 requires the higher-order term of the loss change to stay below a fraction ϵ<1\epsilon<1 of the first-order term. The small error in SFT (10−210^{-2} to 10−110^{-1}) fits this condition well. For DPO under FP32, the error is larger at large kk and closer to the boundary of this condition, but it stays below 1 for every kk in Figure 7c and decreases steadily during training. Following your suggestion, we have revised Appendix A.2 to discuss this explicitly.

W2. Variables in the FP32/BF16 comparison are not well controlled.

  • For the 7B models, FP32 runs use DeepSpeed ZeRO-3 whereas BF16 runs use native PyTorch, so the implementation differs between the two arms. Both arms should be run under the same framework.
  • "Direct BF16" keeps weights, Adam moments, and gradient accumulation (e.g., 32 microbatches in Stage 2) all in BF16, while the theory models only weight rounding; ϕ\phi is also profiled with update directions saved from FP32 runs rather than from BF16 runs. An ablation that keeps BF16 weights but uses FP32 moments and accumulation would isolate weight rounding, and stochastic rounding of weight updates, which the theory predicts should restore k−1k^{-1}, would test it causally.

Thank you for this suggestion. We have added the requested comparison in Appendix C.2 of the revised manuscript. We repeat the SFT runs of Llama-3.2-1B and DeepSeek-Coder-6.7B-Base and the DPO runs of Llama-3.2-1B-Instruct with direct BF16 updates, but keep the gradients, gradient accumulation, and Adam moments in FP32. All other settings are unchanged. For DeepSeek-Coder-6.7B-Base, the new runs use DeepSpeed ZeRO-3 on two GPUs as in its FP32 runs, only without the FP32 master copy. In all three settings, the new runs thus differ from their FP32 counterparts only in update precision. Results are shown in Figure 19 of the revision, and the table below summarizes the log⁡2\log_2 of the optimal learning rate for each kk and the fitted log-log slope β\beta. For comparison, we also include the original results from Section 5.1, where the gradients and optimizer states are in BF16. Across all three settings, the optimal learning rate still scales approximately as k−1/2k^{-1/2}. The change in the scaling rule therefore comes from weight-update rounding, not from the precision of gradients or optimizer states. Since both DeepSeek-Coder-6.7B-Base arms now run under DeepSpeed ZeRO-3 on two GPUs, the implementation difference does not explain the gap between the two scaling rules here.

SettingGradient & Optimizer precisionk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16k=32k=32Fitted β\beta
Llama-3.2-1B (SFT)FP32-13.5-14.0-14.5-15.0-15.5–-0.50
BF16-13.0-13.5-14.0-14.5-15.0–-0.50
DeepSeek-Coder-6.7B-Base (SFT)FP32-11.5-12.0-12.5-12.5-13.0-13.5-0.37
BF16-11.5-12.0-12.5-13.0-13.5-14.0-0.50
Llama-3.2-1B-Instruct (DPO)FP32-16.0-16.0-17.0-17.5-18.0–-0.55
BF16-15.5-16.0-16.5-17.0-17.5–-0.50

We take the states and update directions for the rounding probes from FP32 runs to separate the effect of update rounding from other potential effects of BF16 training. The probes then vary only the learning rate, so they measure how ϕk(η)\phi_k(\eta) changes with η\eta (Assumption 3) without comparing separate runs whose trajectories differ. We agree that stochastic rounding of weight updates would be a more direct causal test. We leave it for future work and have noted this in Section 6 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Three

W3-a There is a gap between the training-loss optimum and downstream performance.

  • The optimal learning rate is defined by final training loss. For DPO, lower training loss does not imply a better policy, yet no downstream evaluation is reported for the DPO models.

Thank you for raising this point. We evaluate the trained DPO models on held-out preference pairs and report the validation DPO loss in Appendix C.1 (Figures 15 c & d, 16 c & d). For your convenience, the table below lists log⁡2\log_2 of the optimal learning rate under this validation loss for each kk. The last column is the slope β\beta of a least-squares fit of log⁡2ηk\log_2 \eta_k against log⁡2k\log_2 k. The optimal learning rate scales roughly as k−1k^{-1} under FP32 and k−1/2k^{-1/2} under BF16.

ModelPrecisionk=1k=1k=2k=2k=4k=4k=8k=8k=16k=16k=32k=32β\beta
Llama-3.2-1B-InstructFP32-19-19.5-20.5-21.5-22.5-0.90
BF16-15.5-16-16.5-17-17.5-0.50
OLMo-2-1124-7B-SFTFP32-16-17-18.5-19.5-20-21-1.00
BF16-14.5-15-15.5-16-16.5-17-0.50

We use the validation DPO loss as a proxy for the downstream performance of the DPO models and leave a direct downstream evaluation to future work. Both are noted in Section 6 of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.