acceptodds
Under review as a conference paper at ICLR 2027

One Learning Rate, Many Positional Steps: Position-Aware RoPE Preconditioning

Abstract

Rotary positional embeddings (RoPE) assign different frequencies to two-dimensional query/key pairs, yet standard optimizers ignore this structure. We identify a potential RoPE-induced learning-rate mismatch: the same parameter-space turn can induce different positional steps across pairs, with sensitive pairs potentially limiting the stable global learning rate. We derive an analytic sensitivity for each pair and show that the same frequency-dependent structure appears in a RoPE-specific component of the local generalized Gauss–Newton geometry. Based on this, we propose Position-Aware Preconditioning (PA), which preserves the scale-changing part of a base-optimizer update and recalibrates only its turning part using RoPE information. PA requires no extra forward/backward pass or per-parameter optimizer state and retains first-order convergence guarantees for its gradient specialization. On C4 pretraining with 130M- and 350M-parameter LLaMA-style models, PA improves AdamW and Muon, reaching baseline training-loss levels with 13–27% fewer optimizer updates.

Primary area:
optimization
Keywords:
Position-aware Optimization, RoPE, Pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_D5pdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The optimizers ignore RoPE's different frequency assignment to keys/queries. The paper identifies the sensitivity of each key/query pair and proposes a position-aware preconditioning, which keeps the length-changing part and rescales the turning part. Their preconditioning-aware mechanism can be cheaply added to the optimizers.

Critical strengths
  1. The paper proposes a cheap and lightweight wrapper with no extra cost to the optimizer states.
  2. They build a bridge between RoPE and preconditioning for optimizers.
  3. The paper has good ablations.
Critical weaknesses

Minor issues:

  1. The second and third paragraphs of the introduction need more citations.
  2. What is R in R(h\theta_i)?
  3. Mention what this sign, "⪰", means.
  4. It may be good to add the training/validation loss to Table 1 as well.
  5. What is your GPU setup?

Serious issues: 6. Unfortunately, the paper is not clear and is not very well-written. 6. The baselines are limited, and the models aren't big. 6. The paper uses some heavy maths and formalisms, which makes the paper hard to understand. 7. Some numbers don't match, e.g., Table 7’s AdamW number 3.420 is not the same as AdamW in Table 3: 3.441. The setup behind Table 7 needs to be explained.

Questions for authors
  1. I wonder how you handle the muon's orthogonality? You didn't address it!
  2. I also didn't understand why the sequence length is small. (sequence length of 256)
6 comments

Response to Reviewer D5pd

We thank the reviewer for the comments and suggestions. We provide a detailed clarification to address your concerns below.

1. Presentation and Clarification of Our Motivation and Method (regarding issues 6(1), 6(3), and Q1)

Thank you for pointing out the presentation issues. We have revised Sections 2–4 by bringing key formulas and lemma statements from the Appendix into the main text and making the connections between our positional analysis, the attention response, and the PA update more explicit. Below, we provide a brief overview of our motivation and method, with references to the revised manuscript.

Motivation Overview. We begin with the two-dimensional parameter cells associated with each RoPE frequency and separate an update’s local scale and turn components. After normalizing out the cell magnitude, we find that the same local turn can produce different positional responses across rotary pairs, which we quantify through the sensitivity SiS_i (Eq. (3); Proposition 2.1).

We then trace these turns into actual attention scores and find that the same rotary factor appears in the RoPE-specific response and its loss-weighted local GGN bound. This connects our positional measurement to the training objective while keeping this isolated component distinct from the full loss curvature (Lemma 3.1; Proposition 3.3).

Based on this connection, we preserve the base optimizer’s local scale component and adjust its turn using RoPE information, giving the PA update in Eq. (22). We then evaluate this correction in pretraining with both AdamW and Muon, where our results show fewer updates to reach matched training loss with modest per-step overhead.

Regarding Muon. PA acts on the query/key cells after Muon has completed its usual momentum, Newton–Schulz orthogonalization, and shape scaling, so the final corrected update generally does not preserve matrix orthogonality. PA preserves the base proposal’s local scale component and adjusts its turn. We now state this ordering and its effect on orthogonality explicitly in Section 4.1. We derive this correction from RoPE’s positional structure and use AdamW and Muon as two different base optimizers that supply the initial update. Our goal is to incorporate this additional model-specific information into the update and evaluate whether the correction improves pretraining efficiency. The paired Muon comparisons in Figures 1–2 show that PA-Muon reaches the baseline training loss in fewer updates, with modest per-step overhead reported in Table 1.

2. Experimental Scale and Sequence Length (regarding issue 6(2), 7, and Q2)

We evaluate PA through paired comparisons with AdamW and Muon, which construct updates in substantially different ways. Both optimizers are tested on the 130M and 350M LLaMA-style models, and we additionally evaluate the Muon v.s. PA-Muon pair on Qwen2.5-0.5B. Figures 1–2 show fewer updates to reach the corresponding baseline training loss across these comparisons, while Table 1 reports modest per-step overhead. We select learning rates separately for each method following the protocol in Appendix E.3. These comparisons test whether the RoPE-aware correction provides additional benefit across different base optimizers and model settings. Our model sizes reflect the resources available for comparisons with adequate training budgets, and validation at substantially larger scales remains valuable future work.

To examine a longer sequence length, we additionally train the 130M model at sequence length 1024, where both PA variants achieve lower final validation loss than their corresponding base optimizers. These runs use 5,000 updates and a global batch of 256 sequences, giving approximately 1.31B tokens and matching the token budget of our original 130M experiments. All four runs use 500 warmup updates, and training/evaluation seed 42. Both PA variants use κ=35\kappa=35. The method-specific learning rates and results are:

MethodMain learning rateFallback AdamW learning rateFinal validation loss
AdamW1.0×10−31.0\times10^{-3}—3.237305
PA-AdamW1.5×10−31.5\times10^{-3}—3.198975
Muon1.2×10−21.2\times10^{-2}2.4×10−32.4\times10^{-3}3.101807
PA-Muon2.8×10−22.8\times10^{-2}5.6×10−35.6\times10^{-3}3.063965

These results show that the validation-loss improvements also appear at sequence length 1024 in the tested 130M setting.

For the main experiments, we choose sequence length 256 to balance training cost with maintaining long training trajectories and completing tuning and ablations within our available compute budget. With a global batch of 256 sequences, the 130M, 350M, and Qwen2.5-0.5B runs use 20K, 60K, and 80K updates, corresponding to approximately 1.31B, 3.93B, and 5.24B tokens, respectively (Appendix E.1). Within each baseline–PA comparison, we match the sequence length, global batch size, and training budget. This sequence-length choice also has been commonly used in optimizer research. For example, GaLore [1] and LDAdam [4] use a maximum sequence length of 256 for its LLaMA/C4 pretraining experiments.

Regarding the AdamW values in Tables 3 and 7. These values correspond to different experimental configurations and are therefore not conflicting results for the same experiment. Table 3 reports the 130M AdamW configuration selected from the learning-rate sweep, with learning rate 1.0×10−31.0\times10^{-3}, evaluation seed 42, and validation loss 3.4413 (Appendix E.3). Table 7 reports the fixed-rate ablation configuration, with learning rate 1.5×10−31.5\times10^{-3}, evaluation seed 12,345, and validation loss 3.42035 (Appendix E.5). Both use training seed 42 and 10,000 updates. Appendix E.5 explicitly notes that the ablation protocol differs from the sweep protocol in both learning rate and evaluation seed.

3. Clarifications on Notation, Experimental Details, and References (regarding the minor issues)

Regarding notation (Minors 2 and 3), the original submission defines R(α)\mathbf R(\alpha) inline in Sec. 2 (Lines 110–113) as the 2×22\times2 planar rotation matrix with angle α\alpha. Thus, R(hθi)\mathbf R(h\theta_i) represents rotation by hθih\theta_i, where hh is the relative token displacement and θi\theta_i is the RoPE frequency of pair ii. To make this definition easier to locate, we now display it separately as Eq. (1). We use ⪰\succeq in its standard matrix-order sense: A⪰B\mathbf A\succeq\mathbf B means that A−B\mathbf A-\mathbf B is positive semidefinite, equivalently v⊤(A−B)v≥0\mathbf v^\top(\mathbf A-\mathbf B)\mathbf v\geq0 for every v\mathbf v. For clarity, we now state this convention explicitly in Sec. 3.3; in particular, Ba⪰0\mathbf B_a\succeq0 means that Ba\mathbf B_a is positive semidefinite.

For experimental reporting (Minors 4 and 5), we have added the plotted endpoint training and validation losses directly to Sec. 5.1. The (training, validation) loss pairs change from baseline to standard PA as follows: at 130M, (3.2981,3.3037)→(3.2930,3.2943)(3.2981,3.3037)\to(3.2930,3.2943) for AdamW and (3.2567,3.2606)→(3.2296,3.2334)(3.2567,3.2606)\to(3.2296,3.2334) for Muon; at 350M, (3.0076,3.0249)→(2.9933,3.0099)(3.0076,3.0249)\to(2.9933,3.0099) for AdamW and (2.9801,2.9994)→(2.9732,2.9912)(2.9801,2.9994)\to(2.9732,2.9912) for Muon. For Qwen2.5-0.5B with Muon, the corresponding pair changes from (3.2028,3.2264)(3.2028,3.2264) to (3.1986,3.2192)(3.1986,3.2192). We have also specified the GPU models at the beginning of Sec. 5. For the main experiments, the 130M and 350M LLaMA-style runs use 2 and 4 NVIDIA L40S GPUs, respectively, with 48 GB memory per GPU; Qwen uses 4 NVIDIA A800 GPUs with 80 GB memory per GPU. The GPU counts are reported in Appendix E.1.

Regarding references (Minor 1), the second and third paragraphs of our introduction primarily develop our own research motivation and the reasoning leading to PA, so not every step corresponds to a result drawn from prior work. To make the relevant background statements more explicit, we have added a citation to Martens [2] in the second paragraph, supporting the distinction between parameter changes and changes in model behavior, and to Bottou et al. [3] in the third paragraph, supporting the connection between local geometry and admissible step sizes. These additions provide references for the background underlying our discussion while retaining the presentation of our own RoPE-specific reasoning.

We thank the reviewer for the constructive feedback and would appreciate a reassessment in light of the revisions and additional experimental results. We would be very grateful if you could consider raising your score.

References

  1. Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. Proceedings of the 41st International Conference on Machine Learning, 235, 61121–61143.
  2. Martens, J. (2020). New Insights and Perspectives on the Natural Gradient Method. Journal of Machine Learning Research, 21(146), 1–76.
  3. Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization Methods for Large-Scale Machine Learning. SIAM Review, 60(2), 223–311.
  4. Robert, T., Safaryan, M., Modoranu, I. V., & Alistarh, D. (2025, May). Ldadam: Adaptive optimization from low-dimensional gradient statistics. In International Conference on Learning Representations (Vol. 2025, pp. 4778-4814).
Reviewer_ufFBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Position-Aware Preconditioning (PA), a lightweight wrapper that decomposes updates to RoPE-aligned query/key parameter cells into radial and angular components and attenuates the angular component using a fixed frequency-dependent profile. It derives an exact normalized positional-response identity, connects the rotary factor to an isolated generalized Gauss–Newton response, and proves stationarity guarantees for an ideal gradient specialization. C4 pretraining experiments with LLaMA-style and Qwen models evaluate PA with AdamW and Muon, together with profile and alignment ablations.

Critical strengths
  1. The method is simple and inexpensive, and the empirical benefit is reasonably consistent across two optimizer families and several model settings. Figures 1–2 and Table 1 report fewer updates to matched training loss with modest per-step overhead. The three-seed PA-Log comparisons in Table 6 improve validation loss in all paired runs, and Tables 10–11 examine downstream performance.

  2. The appendix provides useful controls beyond the main training curves: polar-only updates and exact sensitivity profiles in Table 7, strength/range sweeps in Table 8, and reversed, shuffled, constant, and random-pairing controls in Table 9. The theoretical claims are also carefully qualified, particularly the distinctions between the isolated response and full curvature, and between gradient-specialization guarantees and conditional adaptive-optimizer results.

Critical weaknesses
  1. The evidence for a useful optimizer is stronger than the evidence for the proposed curvature mechanism. Proposition 3.2 controls an isolated RoPE-specific GGN response; the full response includes frequency-dependent remaining geometry and potentially cancelling cross terms. The step-size improvement additionally assumes A.4. The authors acknowledge these distinctions, but the experiments do not directly establish that this component is the limiting curvature or stability factor in the actual training runs. This leaves the explanatory contribution narrower than the overall motivation suggests.

  2. The profile comparisons do not completely separate RoPE-aware geometry from attenuation strength and update-size effects. Table 7 uses the same numerical kappa for exact and theta-squared profiles, although Appendix C.1 explains the strength conversion needed for the low-angle approximation. Table 9's constant control matches the arithmetic mean multiplier, but not realized angular update norms, as the authors acknowledge. These are useful controls, yet a tuned constant-angular control or a strength-matched comparison would clarify the additional value of the proposed profile.

Questions for authors
  1. Can a checkpoint-level diagnostic of total loss sensitivity or local stability establish whether the sensitive RoPE bands actually limit the training step size, and how this relates to Proposition 3.2?

  2. How do the profile conclusions in Tables 7 and 9 change after matching or tuning attenuation strength, including the conversion in Appendix C.1, and comparing against a tuned constant-angular control?

  3. What sequence lengths are used in the main C4 runs? Since the analytic sensitivity depends on token displacement while PA-Log depends only on frequency rank at fixed head dimension, how should the profiles be selected when context length or RoPE base changes?

2 comments

Response to Reviewer ufFB

We thank the reviewer for the comments and suggestions. We provide a detailed clarification to address your concerns below.

1. Connecting Our Geometric Analysis to Actual Training Behavior (regarding W1 and Q1)

Thank you for this suggestion. We provide two complementary training-level comparisons to examine this connection. We find that PA-AdamW achieves lower validation loss at larger tested learning rates, and that amplifying angular updates in high-SiS_i bands produces larger validation-loss increases than the same amplification in low-SiS_i bands.

For clarity, Proposition 3.2 in the original submission is now Proposition 3.3 in the revised manuscript. This proposition bounds an isolated RoPE-specific GGN response to angular perturbations, with a frequency-dependent factor SiS_i. It motivates examining whether high-SiS_i bands are less tolerant of increased angular updates, without identifying this isolated response as the component limiting the training step size. The sufficient descent-step guarantee additionally relies on Assumption A.4.

First, PA-AdamW achieves lower validation loss at larger learning rates in the tested sweep.

We compare AdamW and standard PA-AdamW on the 130M model over 10,000 updates per configuration, using κ=35\kappa=35 for PA-AdamW. The final validation losses are:

Peak learning rateAdamW validation lossPA-AdamW validation loss (κ=35\kappa=35)
1.0×10−31.0\times10^{-3}3.44133.4376
1.5×10−31.5\times10^{-3}3.46043.4332
1.8×10−31.8\times10^{-3}3.51503.4571

Among these learning rates, AdamW achieves its best validation loss at 1.0×10−31.0\times10^{-3} and deteriorates as the learning rate increases. PA-AdamW instead improves at 1.5×10−31.5\times10^{-3} and exhibits substantially less deterioration at 1.8×10−31.8\times10^{-3}. These results support a broader useful learning-rate range for PA-AdamW in the tested sweep.

Second, amplifying angular updates in high-SiS_i bands produces larger validation-loss increases.

To examine the frequency dependence more directly, we compute the exact SiS_i using causal token-pair displacement weights at sequence length 256, rank the rotary pairs by SiS_i, and divide them into equally sized high-SiS_i and low-SiS_i groups. We decompose each AdamW increment into radial and tangential components, multiply only the tangential component of the selected Q/K cells by 1.25 or 1.5, and add the resulting increment directly to the parameters. We preserve the radial component, the weight-decay rule, and the ordinary AdamW update for all other cells.

These 130M runs use the same base learning rate of 1.5×10−31.5\times10^{-3} and are trained for 10,000 updates. The unmodified AdamW control has a final validation loss of 3.4604.

Angular multiplierHigh-SiS_i validation lossIncrease over controlLow-SiS_i validation lossIncrease over control
1.25×1.25\times3.4779+0.01753.4632+0.0028
1.5×1.5\times3.5040+0.04363.4765+0.0161

At both multipliers, amplifying angular updates in the high-SiS_i group leads to a larger increase in validation loss than applying the same multiplier to the equally sized low-SiS_i group. The difference is already apparent at 1.25×1.25\times and persists at 1.5×1.5\times, showing that the choice of frequency bands matters for this intervention over the training trajectory.

Together, these comparisons provide complementary training-level evidence connecting our analysis to actual pretraining behavior. We observe both better performance at larger learning rates with PA-AdamW and a frequency-dependent response to angular-update amplification, supporting our motivation for using RoPE sensitivity to calibrate angular updates.

2. Comparing the RoPE Profile with a Tuned Constant-Angular Control

Thank you for this suggestion. Tuning improves the constant-angular control, but Standard PA still achieves a validation loss approximately 0.00469 lower than the best tested constant setting.

As discussed in Appendix C.1, equal numerical values of κ\kappa do not imply matched attenuation for the exact and low-frequency profiles. To examine the underlying concern about frequency-dependent allocation versus overall attenuation strength, we follow your recommendation and evaluate a tuned constant-angular control under the original 130M ablation setting, with all runs trained for 10,000 updates.

We retain the same PA update construction but apply a shared angular multiplier to every RoPE pair. Specifically, we extend the mean-matched constant control in Table 9 by varying κ\kappa:

pi=c(κ)=1N∑j=1N(1+κθj2)−1,p_i=c(\kappa)=\frac{1}{N}\sum_{j=1}^{N}(1+\kappa\theta_j^2)^{-1},

where NN is the number of rotary pairs per attention head. At κ=35\kappa=35, this reproduces the original mean-matched constant control; varying κ\kappa allows us to tune the uniform attenuation strength. We compare these controls with Standard PA from Table 7 below.

Methodκ\kappaAngular multiplier pip_iValidation loss
Constant-angular150.8347493.40760
Constant-angular350.7902313.39698
Constant-angular550.7661043.39326
Constant-angular750.7494603.39387
Constant-angular1000.7339753.39436
Standard PA35(1+κθi2)−1(1+\kappa\theta_i^2)^{-1}3.38857

Tuning reduces the constant control’s validation loss from 3.39698 at κ=35\kappa=35 to 3.39326 at κ=55\kappa=55, narrowing the gap with Standard PA. Nevertheless, Standard PA retains an advantage over all tested constant settings. This supports the added value of allocating angular attenuation across RoPE frequencies in this setting, even after tuning the uniform control’s overall attenuation strength.

3. Sequence Length and Profile Selection (regarding Q3)

Our main C4 runs use a sequence length of 256 tokens, as stated in Appendix E.1. We choose this setting to balance training cost with maintaining sufficiently long training trajectories and completing the optimizer comparisons and ablations within our available compute budget. This setting also has been commonly used in optimizer research. GaLore [1] and LDAdam [2] use a maximum sequence length of 256 for LLaMA pretraining on C4.

When the context length or RoPE base changes, we would adapt the profiles as follows. For standard PA, we would update the θi2\theta_i^2 profile when the RoPE base changes and revalidate or adjust κ\kappa for the new setting. For PA-Log, we would retain the rank-based parameterization and retune λmax⁡\lambda_{\max} using validation loss, while keeping λmin⁡=1\lambda_{\min}=1. Our reported PA-Log configuration follows this empirical selection approach: Table 8(b) compares λmax⁡∈{2,4,8}\lambda_{\max}\in\{2,4,8\}, with λmax⁡=4\lambda_{\max}=4 giving the lowest validation loss.

We thank the reviewer again for the positive comment and thoughtful suggestions, which help us clarify the analysis and strengthen the experimental evaluation.

References

  1. Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., & Tian, Y. (2024). GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. Proceedings of the 41st International Conference on Machine Learning, 235, 61121–61143.

  2. Robert, T., Safaryan, M., Modoranu, I. V., & Alistarh, D. (2025, May). Ldadam: Adaptive optimization from low-dimensional gradient statistics. In International Conference on Learning Representations (Vol. 2025, pp. 4778-4814).

Reviewer_yh3SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies a potential issue in using RoPE for the global learning rate. To remedy this, the authors propose an optimizer preconditioning method that adjusts the effect of RoPE on updates in accordance with their provided derivation. Small pretraining results are provided as experimental evidence.

Critical strengths

This paper addresses a problem in a way that could be potentially important. Since RoPE is widely used, it would highly beneficial if we can obtain a significant improvement through a simple optimizer adjustment.

Critical weaknesses

The paper appears to filled with LLM-generated text in many places, which makes it quite difficult to read. In particular, I do not understand what most of Sections 3 and 4 are trying to say. The exact details of Algorithm 1 are also unclear, with Lines 2, 3, 8, and 9 having critical details deferred to the appendix or missing altogether.

Moreover, the experimental results do not convincingly support the effectiveness of the proposed method. In particular, the scale of the experiments is far too small, consisting only of pretraining experiments on models with less than 1B parameters. I would like to see experiments for settings besides pretraining and for larger models, typically at least 7B parameters. Otherwise, it is difficult to argue that the method offers a significant contribution.

Questions for authors

Given that the method is called Position-Aware Preconditioning, should the acronym be PAP instead of PA?

3 comments

Response to Reviewer yh3S

We thank the reviewer for the comments and suggestions. We provide detailed clarifications below to address your concerns.

1. Presentation and Clarification of Our Motivation and Method (regarding Weakness 1)

Thank you for pointing out the presentation issues. To meet the nine-page limit, we place some formulas and lemma statements in the appendix in the original submission. We recognize that this may have made parts of the argument harder to follow. With the additional page allowed for the rebuttal and camera-ready versions, we have revised Sections 2–4 by bringing key formulas and lemma statements into the main text and adding more intuitive explanations. We have also made the connections between these results more explicit.

To clarify our motivation and method, we summarize the reasoning behind PA in three steps. The references below follow the revised manuscript.

  1. The same turn can produce different positional responses. We first compare equal local turns across RoPE pairs, independently of cell magnitude, and quantify their unequal positional effects through the sensitivity SiS_i (Proposition 2.1). This motivates using the known RoPE structure to calibrate parameter updates.

  2. This frequency dependence also appears in actual attention. We trace the same rotary structure into the RoPE-specific attention response and its local loss-weighted GGN bound (Lemma 3.1; Proposition 3.3). This connects our positional analysis to the training objective while keeping the isolated response distinct from the full training curvature.

  3. PA uses this structure to adjust the update. We preserve the base optimizer’s local scale proposal and adjust only its turn using a RoPE-dependent multiplier (Eq. (22)). We then evaluate this correction in pretraining, where PA improves both AdamW and Muon with modest additional per-step cost.

We hope the above part can help you understand our overall claim.

Algorithm. We have revised the presentation of Algorithm 1 in the new version, including defining the parameter-group notation in Require and specifically clarifying Lines 2, 3, 8, and 9 of Algorithm 1. We provide an explanation as below:

  • Line 2: We form the base increment using the optimizer’s direction and learning rate, before applying decoupled weight decay.
  • Line 3: We explicitly enumerate the RoPE-aligned cells defined in Eq. (3) across all attention heads, Q/K projections, frequency pairs, and input columns.
  • Line 8: We retain the base updates for all remaining parameters and combine their updated values with the PA-corrected Q/K cells to assemble the candidate parameters W^g\widehat{\mathbf W}_g for each group.
  • Line 9: We commit Wt+1,g=W^g−ηt,gωgWt,g\mathbf W_{t+1,g}=\widehat{\mathbf W}_g-\eta_{t,g}\omega_g\mathbf W_{t,g}, computing the decay term from the retained pre-update parameters Wt\mathbf W_t.

2. Experimental Scope and Model Scale (regarding Weakness 2)

Our current empirical evidence concerns pretraining efficiency on the tested models. We focus on pretraining because training trajectories, validation losses, and the cost of reaching a matched loss directly measure the optimization benefit we study. This focus is also consistent with the pretraining evaluations in optimizer studies such as SOAP [1] and SWAN [2]. We additionally report zero-shot downstream evaluations of the resulting checkpoints in Tables 10–11 to assess the quality of the pretrained models. These evaluations show improved average downstream scores, with gains varying across individual tasks.

Across the four LLaMA-style model–optimizer settings, Figure 1 shows approximately 13%–27% fewer updates to reach the baseline training loss. Table 1 reports approximately 0.8%–6.1% per-step overhead and estimated wall-clock savings of 7.7%–23.8%. We agree that validation at 7B would provide valuable additional evidence. As a resource reference from a comparable optimizer study, GaLore [3] trains a 7B LLaMA model on C4 for 19.7B tokens using 64 A100 GPUs, and restricts the comparison to a single trial without hyperparameter tuning because of computational constraints. We do not have access to training resources at this scale. We have therefore prioritized controlled comparisons at the reported model scales, together with learning-rate sweeps and ablations, and leave validation at 7B and beyond to future work.

Besides, we use “PA” to abbreviate “Position-Aware” in Position-Aware Preconditioning, and use the same short name throughout the paper.

We deeply appreciate your constructive feedback. We hope our detailed clarifications have resolved your concerns. We would be very grateful if you could consider raising your score.

References

  1. SOAP: Improving and Stabilizing Shampoo using Adam.
  2. SWAN: Preprocessing SGD Enables Adam-Level Performance on LLM Training with Significant Memory Reduction.
  3. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.
AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer yh3S

As the deadline of discussion period is approaching, we would kindly like to inquire if you would get a chance to review our response and if there are any remaining questions we can address.

Your insights, both the constructive suggestions and areas of contention, have been crucial for us. We have done our best to address each point and clarify any misunderstandings. We are truly keen to have a constructive dialogue with you to refine our work further.

Best regards,

Authors of Submission 25799

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer yh3S,

As the discussion period is ending soon, we would appreciate it if you could review our responses and additional results. We would be happy to clarify any remaining concerns. Thank you for your time and consideration.

Best regards,

Authors of Submission 25799

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.