acceptodds
Under review as a conference paper at ICLR 2027

Learnable Adaptive Spectral Reparameterization for Improving LLM Pretraining

Abstract

Recent advances suggest that flatter and better-conditioned weight spectra are closely associated with improved LLM pretraining and generalization, motivating increasing interest in shaping the spectral geometry of model parameters. However, existing approaches typically rely on prescribed spectral transformations, whose predefined mappings impose strong priors on spectral evolution and may restrict the flexibility required during training. Moreover, different layers and parameter types can favor distinct spectral geometries, making a uniform transformation potentially suboptimal across the model. To address this limitation, we propose Learnable Adaptive Spectral Reparameterization (LASR), which enables individual weight matrices to learn their preferred spectral transformations from the training objective. Specifically, LASR constructs a learnable family of spectral mappings through a truncated K-th-order expansion of the reciprocal square-root function, with trainable coefficients controlling the contributions of different polynomial bases. This parameterization spans diverse feasible spectral amplification behaviors while ensuring bounded spectral reshaping. By jointly optimizing these coefficients with the model, each weight matrix adaptively learns its spectral transformation according to its training dynamics. Consequently, LASR turns spectral reshaping from a prescribed, model-wide operation into a weight-specific learnable mechanism. Extensive experiments across pretraining, downstream evaluations, and spectral analyses consistently demonstrate the effectiveness of LASR.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Pretraining, large language model, generalization, spectrum

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k1Q7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Polynomial weight preconditioning methods, such as [1], reshapes the spectra of weight matrices with a fixed polynomial transformation. This paper proposes learning a spectral transformation, one for each weight matrix, trained together with the model via the language-modeling objective. It uses a taylor expansion with bounded learnable coefficients.

The experiments test whether learning these transformations improves pretraining and downstream performance. LASR reaches the same training loss in fewer steps and improves average zero-shot accuracy over the standard and spectral baselines with both AdamW and Muon. An ablation finds that separate transformations for individual matrices outperform shared coefficients.

Critical strengths
  • (S1) End-to-end spectral shaping. The main appeal is learning spectral shaping end to end under the LM objective. This offers a way to reduce manual choices about target spectra and shaping strengths for individual matrices. The parameterization used by LASR gives a simple, bounded way to do this with few learned parameters per matrix.

  • (S2) Benefits of layer-specific adaptation. The sharing ablation provides evidence that adaption across different layers is useful. The gains over PC also support the usefulness of the approach in the studied studied in the paper.

Critical weaknesses
  • (W1) Unclear hyperparameter tuning. My main concern is the fairness of hyperparameter selection. All methods share a learning-rate schedule, but the paper doesn't explain how the learning rates were selected or whether they were tuned for each method. Additionaly, the authors perform a polynomial-degree sweep for LASR, while the baselines' hyperparameter tuning protocol is not reported. E.g., PC [1] offers different spectra shaping strengths. A common setting can suit different methods differently, so it's hard to tell whether the advantage survives with tuned baselines. See next point.

  • (W2) Modest gains and unclear uncertainty. LASR's average downstream gains over PC are less than one percentage point, and some tasks get worse. Additionally, given the sensitivity of downstream evaluations, I'd want to see that the gains survive variation across repeated evaluations of the same trained models.

Questions for authors
  • (Q1) LM objective and spectral shaping. Why do you expect minimizing the LM loss to steer the spectra toward better conditioning and generalization? The end to end concept is appealing, but I'd appreciate further motivation on this.
  • (Q2) Hyperparameter selection. How were the learning rates and hyperparameters selected for LASR and each baseline? Please clarify the search budgets and data used for tuning, including the degree sweep for LASR. I'd want to see whether the advantage holds with comparably tuned baselines.
  • (Q3) Evaluation repeatability. Do the gains presist after repeated evaluations with different evaluation seeds (given the noisiness of these downstream evals)?
  • (Q4) Wall-clock acceleration. After accounting for the extra time per iteration, does LASR reach the same loss faster than the base method and PC?.

Given convincing answers to the above questions I'd consider raising the score.


[1] Wang et al. PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training. arXiv:2606.06470, 2026.

6 comments
AuthorsOpenReview ↗

Response to Reviewer k1Q7 (1)

We sincerely thank you for reviewing our manuscript and providing valuable suggestions. We will address each of your concerns in detail in our rebuttal.

W1: Unclear hyperparameter tuning. My main concern is the fairness of hyperparameter selection. All methods share a learning-rate schedule, but the paper doesn't explain how the learning rates were selected or whether they were tuned for each method. Additionaly, the authors perform a polynomial-degree sweep for LASR, while the baselines' hyperparameter tuning protocol is not reported. E.g., PC [1] offers different spectra shaping strengths. A common setting can suit different methods differently, so it's hard to tell whether the advantage survives with tuned baselines. See next point.

We sincerely thank the reviewer for raising this important concern. Our framework is built upon the standard Megatron-LM pretraining pipeline, with general hyperparameters, including learning rates and optimizer settings, following established practices from baseline studies and technical reports of leading industry labs. We prioritize fairness and consistency by adopting the same training hyperparameters across all methods. Method-specific tuning based on limited training stages may overfit to intermediate performance, as improvements observed early in pretraining do not necessarily persist until convergence. Therefore, we adopt standardized pretraining configurations to ensure fair comparisons and avoid potential biases introduced by individually tuned learning rates.

For method-specific hyperparameters, we follow the configurations reported in the original papers wherever applicable, including polynomial coefficients, approximation degrees, and iteration counts, with necessary adaptations to our unified framework.

MethodPolynomial / Spectral MappingMethod-specific Hyperparameters
ONINewton iteration: Bt=1.5Bt−1−0.5Bt−13SB_t = 1.5B_{t-1} - 0.5B_{t-1}^3SIterations = 4; coefficients = (1.5, -0.5); epsilon = 1e-5
ENFp1(x)=x−αx3p_1(x) = x - \alpha x^3, p2(x)=x+αx3p_2(x) = x + \alpha x^3α = 0.1; spectral scale = 1.0
CANSp(x)=a1x+a3x3+a5x5p(x) = a_1x + a_3x^3 + a_5x^5Order = 5; iterations = 4; coefficients listed below
IFNSOP(R)=I+∑l=1LαlR2l−1P(R) = I + \sum_{l=1}^{L} \alpha_l R^{2^{l-1}}L = 14; fixed offline-fitted coefficients are [0.487649, 0.481625, 0.719537, 1.246460, 1.156044, 3.124816, 0.998430, 9.335488, -5.110362, 34.735029, -44.114691, 130.881855, -161.856947, 237.302668]
PC2g(x)=2.083x−1.643x3+0.560x5g(x) = 2.083x - 1.643x^3 + 0.560x^5Polynomial level = 2; power iterations = 10; gamma init = 1.0
PC4g(x)=3.625x−9.261x3+14.097x5−10.351x7+2.890x9g(x) = 3.625x - 9.261x^3 + 14.097x^5 - 10.351x^7 + 2.890x^9Polynomial level = 4; power iterations = 10; gamma init = 1.0

Coefficients of four iterations in the CANS method:

Stagea1a3a5
18.420294-24.91049118.472094
24.101229-3.0518560.574124
33.680982-2.7539650.540190
42.728092-2.0315490.458664

Most baseline methods employ fixed spectral control functions. Regarding the different spectral shaping configurations of PC mentioned by the reviewer, we strictly follow the settings recommended in the original paper, which adopts PC4 for AdamW and PC2 for Muon based on their experimental ablations.

For general hyperparameters, the learning rate is one of the most critical factors affecting pretraining performance. We therefore summarize the learning-rate configurations reported in the official technical reports of LLaMA3 and OLMoE, as well as the PC Layer paper, in the following table.

ReferenceModelOptimizerPeak LR
LLaMA 3 (Scaling)40M–16BAdamW2e-4–4e-4
LLaMA 38BAdamW3.0e-4
OLMoE7B-A1BAdamW4.0e-4
PC271MAdamW8.486e-4
PC1BAdamW3.0e-4
PC271MMuon4.243e-3
PC1BMuon1.5e-3

Based on these established pretraining configurations, we selected a balanced set of learning rates and general training hyperparameters that aligns with existing practices across different model scales and optimizers, ensuring a consistent and fair comparison among methods. Importantly, we deliberately avoided tuning these hyperparameters specifically for the performance at 10,000 iterations, as such short-horizon optimization could lead to overfitting to a limited training budget and may not generalize to longer pretraining runs. Instead, our choices follow established large-scale pretraining practices, aiming to ensure that the observed improvements reflect the effectiveness of the proposed method rather than hyperparameter choices tailored to a particular training horizon.

AuthorsOpenReview ↗

Response to Reviewer k1Q7 (2)

W2: Modest gains and unclear uncertainty. LASR's average downstream gains over PC are less than one percentage point, and some tasks get worse. Additionally, given the sensitivity of downstream evaluations, I'd want to see that the gains survive variation across repeated evaluations of the same trained models.

We sincerely thank the reviewer for raising this important concern. In fact, pretrained models naturally exhibit varying performance across different downstream tasks, and their overall capabilities are better reflected by evaluations across a broad range of benchmarks. Our original evaluation included several commonly used benchmark tasks adopted in prior studies. To address the reviewer's concern, we have further evaluated the 1B model on a more comprehensive benchmark suite (7 original + 12 additional), demonstrating the benefits of spectral control in model pretraining and further validating the effectiveness of our proposed adaptive spectral control method.

BASEONIENFCANSIFNSOPCLASR
MMLU23.0723.0423.0523.1123.1023.0623.01
ARC-E50.5150.7351.2252.4552.8253.0554.44
BOOLQ53.3353.4653.8552.6752.4553.1056.97
HELLAS.36.2036.4436.3937.2237.1937.3337.68
PIQA68.8568.5369.3469.8869.9370.1269.59
SCIQ75.1075.3075.7076.7077.5077.3078.20
WINOG.53.8853.7254.2553.4353.6654.1254.34
WMDP.15.5615.5015.5515.3015.4515.6615.82
TMLU21.8021.7421.9521.3721.2421.6722.25
PROST.20.3920.0518.4619.1320.2020.3620.88
AEXAMS16.6517.2017.1618.2817.8417.9419.87
ACLUE.17.5517.1417.5017.4317.6517.8418.48
ANLI.26.4526.5627.1327.3527.2227.9727.66
MATHQA19.0219.4719.2520.6619.3520.3721.20
HAERAE14.2014.3514.1214.2214.4714.2914.65
COMQA.17.4518.3317.6917.8818.2118.4618.82
LOGIQA18.1018.2118.2218.3718.2918.6519.06
MULTIRC40.4843.2442.4543.3944.1044.1445.77
RTE.46.2346.2546.7347.4048.2147.9448.95
AVG33.4133.6533.6834.0134.1534.3935.14

The expanded evaluation across 19 benchmarks provides stronger evidence of the generalization benefits of LASR. Specifically, LASR outperforms BASE and PC by 1.73 and 0.75 percentage points on average, achieving improvements on 18/19 and 16/19 tasks, respectively. Importantly, even after excluding BoolQ, where LASR exhibits a relatively large gain, the average improvements remain 1.62 and 0.58 percentage points over BASE and PC. These results suggest that the improvements are not driven by a single benchmark but are broadly observed across diverse downstream tasks. While individual task results may exhibit statistical uncertainty, the expanded evaluation provides complementary evidence of the effectiveness of adaptive spectral control.

AuthorsOpenReview ↗

Response to Reviewer k1Q7 (3)

Q1: LM objective and spectral shaping. Why do you expect minimizing the LM loss to steer the spectra toward better conditioning and generalization? The end to end concept is appealing, but I'd appreciate further motivation on this.

We sincerely thank the reviewer for this very insightful question. Our motivation is based on two considerations.

First, from an optimization perspective, standard LM pretraining minimizes f(W)f(W), whereas fixed spectral control methods optimize f(W+h(W))f(W+h(W)), where hh denotes a predefined spectral transformation. Since the optimal weight geometry is unknown, imposing a fixed transformation may introduce an optimization bias that is not necessarily aligned with the original objective. To address this limitation, LASR introduces learnable spectral control coefficients, which can be conceptually expressed as minimizing f(W+αh(W))f(W+\alpha h(W)). As learnable α\alpha approaches 0, the original parameterization is recovered, preserving access to the original solution space.

Second, our motivation is further supported by an empirical observation in the PC paper. PC selectively applies spectral preconditioning to the attention output projection and FFN weights, while its ablation studies reveal that extending spectral control to other modules does not consistently improve performance. This suggests that different network components may have heterogeneous spectral requirements, making predefined spectral transformations potentially suboptimal, particularly as model architectures evolve. Motivated by this observation, LASR allows the LM objective to directly guide layer-specific spectral transformations rather than relying on manually prescribed spectral structures. Our results in Appendix C.7 further support this motivation: some layers exhibit negligible spectral changes, whereas others undergo substantial spectral reshaping, indicating that the network learns distinct spectral preferences across layers. Thus, LASR leverages task-driven optimization to adapt spectral control to the actual requirements of different network components.

Compared with the first challenge, we consider the second more fundamental. Our experimental results suggest that the LM objective can effectively guide the search for favorable spectral configurations within a predefined spectral control space in the pretraining, rather than relying on fixed spectral transformations.

Q2: Hyperparameter selection. How were the learning rates and hyperparameters selected for LASR and each baseline? Please clarify the search budgets and data used for tuning, including the degree sweep for LASR. I'd want to see whether the advantage holds with comparably tuned baselines.

We sincerely thank the reviewer for this important question. As discussed in W1, our experimental protocol prioritizes fairness and consistency. Specifically, the general training hyperparameters, including the learning rate and optimizer settings, are adopted from established LLM pretraining practices reported in leading industry technical reports and the baseline papers, rather than being optimized for LASR.

Regarding the polynomial degree KK, we follow the experimental protocol of the PC paper, which evaluates different polynomial configurations from K=1K=1 to 44. We extend this investigation to K=1K=1–88 to systematically examine the effect of polynomial complexity on LASR. Our experiments indicate that K=6K=6 or 77 generally achieves the best performance across different model configurations.

Importantly, KK is the only LASR-specific structural hyperparameter, determining the flexibility and upper amplification envelope of the learnable spectral transformation. Increasing KK allows the spectral mapping to approach a more balanced singular-value distribution. However, excessively large KK introduces higher-order terms that predominantly affect small singular values, potentially amplifying numerical sensitivity and causing optimization fluctuations under the same learning rate. Therefore, we recommend K=6K=6 as a robust default, as it consistently achieves competitive performance across all evaluated models while maintaining a favorable balance between spectral flexibility and training stability.

Reviewer_LWgaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LASR, which learns a separate spectral transformation for each weight matrix during LLM pretraining. Earlier spectral reparameterization methods fix one mapping before training and share it across all weights, and the authors argue that different layers and weight types need different ones. Each weight is first normalized by its estimated spectral norm. Its spectrum is then amplified with a truncated polynomial expansion of the reciprocal square root. Each term has a sigmoid-bounded learnable coefficient, which keeps the amplification bounded. On LLaMA3-350M/1.3B and OLMoE-7B-A1B, LASR converges faster in terms of iterations and reaches higher average zero-shot accuracy than existing spectral reparameterization methods. My overall impression is that the method is clean and easy to adopt, but it changes little over existing polynomial preconditioning, and the paper has not yet shown that the gain holds at realistic scale or in wall-clock time.

Critical strengths
  1. The problem is important: improving LLM optimization through the spectral structure of the weights. Existing methods all fix the spectral mapping before training, so asking whether each weight should choose its own is a natural next step. The paper turns this into a concrete design that fixes only a feasible region and lets the training loss pick the mapping inside it.
  2. The transformation is mathematically motivated, and its amplification is bounded under the normalization. Because the series coefficients are positive and each gate lies in (0, 1), every coefficient pushes the amplification the same way, so terms of different orders cannot cancel. The extra cost is small: six scalars per matrix, at the same matrix-multiplication cost as PC.
  3. The experiments cover both dense and MoE models, compare against several baselines, and include ablations on coefficient sharing. The sharing ablation is the most useful result. The fixed initial mapping falls below PC, and weight-specific learning lifts it above PC, so the gain does come from learning.
Critical weaknesses
  1. The novelty is limited. The core change is adding learnable coefficients to an existing polynomial preconditioner. It is an incremental improvement. The normalization and the learnable magnitude follow PC Layer, and a fixed mapping in the new basis does not beat PC on any model. Learnable non-negative polynomial filters are also common in graph learning (e.g., BernNet), and the paper does not discuss them.
  2. There is no large-scale validation. The largest dense model is 1.3B, and the MoE model has about 1B active parameters. It is unclear whether the method still works in realistic LLM pretraining. At this scale MMLU is at chance for every method and BoolQ stays below the majority-class rate, yet BoolQ accounts for most of the gain over PC. That gain also shrinks from 350M to 1.3B.
  3. The efficiency gain is not convincing. The ~1.4× speedup is counted in iterations, while each step costs about 14% more, so the actual wall-clock benefit is unclear. The speedup is measured only against BASE, not PC, and the per-step overhead is reported only for the 350M model.
  4. The theoretical explanation is insufficient. The theory only shows that the amplification is bounded. It does not explain why the transformation should improve convergence or generalization. The claim that different weights need different mappings rests mostly on the mappings LASR itself learns. The learned coefficients also stay close to their initial value of 0.5, and the paper does not explain how changes this small produce the accuracy gain.
  5. The added complexity is not justified. There is no comparison with simpler learnable spectral scaling, such as one learnable scalar per matrix, so it is unclear whether the polynomial structure is necessary. For many weights the six coefficients move almost together during training (Figures 6 to 8), which suggests that one scalar might be enough.
Questions for authors
  1. Can the authors add experiments on larger models (e.g., dense models of 7B or above) with a longer training budget? This is my main concern.
  2. Under the same compute budget, what is the actual end-to-end wall-clock speedup over BASE and over PC?
  3. How does LASR compare with one learnable scalar per matrix, or with PC whose polynomial coefficients are made learnable?
  4. What property of a weight matrix decides how much amplification it learns, for example its condition number or stable rank?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LWga (1)

We sincerely thank the reviewer for their positive feedback and thoughtful questions. We will address each of these questions in detail in our subsequent responses.

W1: The novelty is limited. The core change is adding learnable coefficients to an existing polynomial preconditioner. It is an incremental improvement. The normalization and the learnable magnitude follow PC Layer, and a fixed mapping in the new basis does not beat PC on any model. Learnable non-negative polynomial filters are also common in graph learning (e.g., BernNet), and the paper does not discuss them.

We sincerely thank the reviewer for this insightful comment. We would like to clarify that the central contribution of LASR is not merely introducing learnable coefficients, but enabling task-driven adaptive spectral control during LLM pretraining. Our motivation is based on two considerations.

First, from an optimization perspective, standard LM pretraining minimizes f(W)f(W), whereas fixed spectral control methods optimize f(W+h(W))f(W+h(W)), where hh denotes a predefined spectral transformation. Since the optimal weight geometry is unknown, imposing a fixed transformation may introduce an optimization bias that is not necessarily aligned with the original objective. To address this limitation, LASR introduces learnable spectral control coefficients, which can be conceptually expressed as minimizing f(W+αh(W))f(W+\alpha h(W)). As learnable α\alpha approaches 0, the original parameterization is recovered, preserving access to the original solution space.

Second, the PC paper provides an important empirical observation: spectral preconditioning is selectively applied to attention output projections and FFN weights, while extending it to other modules does not consistently improve performance. This suggests that different network components may have heterogeneous spectral requirements. LASR addresses this challenge by allowing the LM objective to guide module-specific spectral transformations. Our results in Appendix C.7 further support this motivation, showing that some layers exhibit negligible spectral changes, while others undergo substantial spectral reshaping.

Importantly, many existing polynomial preconditioners involve both positive and negative coefficients, and directly learning these coefficients may produce undesirable spectral responses. LASR instead employs a structured spectral parameterization with constrained positive coefficients, enabling adaptive adjustments within a predefined family of spectral transformations. We acknowledge that spectral normalization and learnable magnitude scaling build upon PC, which we have clearly stated in our submission. However, unlike PC, which learns the overall magnitude while retaining a fixed relative spectral mapping, LASR directly learns the spectral transformation itself. LASR consistently outperforms PC in average downstream benchmark performance, with additional experimental evidence provided in our subsequent responses.

We also sincerely thank the reviewer for highlighting BernNet and related research on learnable non-negative polynomial filters. While BernNet focuses on spectral filtering of graph signals, LASR investigates task-driven adaptation of weight singular-value distributions during LLM pretraining. As our original discussion primarily focused on LLM-related literature, we will expand the Introduction to discuss this broader line of research and clarify its relationship to our work. We appreciate this suggestion and apologize for the omission.

AuthorsOpenReview ↗

Response to Reviewer LWga (2)

W2: There is no large-scale validation. The largest dense model is 1.3B, and the MoE model has about 1B active parameters. It is unclear whether the method still works in realistic LLM pretraining. At this scale MMLU is at chance for every method and BoolQ stays below the majority-class rate, yet BoolQ accounts for most of the gain over PC. That gain also shrinks from 350M to 1.3B.

We sincerely thank the reviewer for raising this important concern. Due to computational constraints, our original experiments were limited to dense models at the 1B scale and an MoE model with 7BA1B total parameters, trained for 10,000 iterations (approximately 40B tokens). To address concerns about scalability and longer training, we additionally trained a 3B dense model for 25,000 iterations (approximately 100B tokens). Given the limited time and our computational resources during rebuttal, we selected BASE and PC (the strongest spectral baseline) for comparison.

Our 3B model configuration is also based on a scaled LLaMA3 architecture:

ExpertsLayersAttention headsEmbedding dimHidden size
LLaMA3 - 3B1 / 1322025606912

Due to the increased model dimensionality, we set the peak learning rate to 2e-4 to mitigate potential loss spikes and ensure training stability, while keeping all other hyperparameters unchanged. The results are shown in the following table:

MethodMMLUARC-EBoolQHellaS.PIQASCIQWinoG.Avg.
BASE23.5557.2756.1339.2272.3477.3056.2554.58
PC23.6859.4456.8540.0473.1777.8056.8255.40
LASR23.7560.8657.5441.3573.0479.1057.6556.18

Recent studies have demonstrated the effectiveness of spectral control in neural network optimization. Furthermore, the training loss curves confirm that LASR maintains a stable advantage throughout extended pretraining, achieving 1.50× and 1.19× iteration-level speedups over BASE and PC, respectively. These results provide further evidence that the effectiveness of LASR persists at larger model scales and longer training durations. Due to space limitations, we refer the reviewer to Appendix A.1 of the revised manuscript for the complete training curves and detailed experimental results.

We also sincerely thank the reviewer for raising this important concern about evaluation. In fact, pretrained models naturally exhibit varying performance across different downstream tasks, and their overall capabilities are better reflected by evaluations across a broad range of benchmarks. Our original evaluation included several commonly used benchmark tasks adopted in prior studies. To address the reviewer's concern, we have further evaluated the 1B model on a more comprehensive benchmark suite (7 original + 12 additional).

BASEONIENFCANSIFNSOPCLASR
MMLU23.0723.0423.0523.1123.1023.0623.01
ARC-E50.5150.7351.2252.4552.8253.0554.44
BOOLQ53.3353.4653.8552.6752.4553.1056.97
HELLAS.36.2036.4436.3937.2237.1937.3337.68
PIQA68.8568.5369.3469.8869.9370.1269.59
SCIQ75.1075.3075.7076.7077.5077.3078.20
WINOG.53.8853.7254.2553.4353.6654.1254.34
WMDP.15.5615.5015.5515.3015.4515.6615.82
TMLU21.8021.7421.9521.3721.2421.6722.25
PROST.20.3920.0518.4619.1320.2020.3620.88
AEXAMS16.6517.2017.1618.2817.8417.9419.87
ACLUE.17.5517.1417.5017.4317.6517.8418.48
ANLI.26.4526.5627.1327.3527.2227.9727.66
MATHQA19.0219.4719.2520.6619.3520.3721.20
HAERAE14.2014.3514.1214.2214.4714.2914.65
COMQA.17.4518.3317.6917.8818.2118.4618.82
LOGIQA18.1018.2118.2218.3718.2918.6519.06
MULTIRC40.4843.2442.4543.3944.1044.1445.77
RTE.46.2346.2546.7347.4048.2147.9448.95
AVG33.4133.6533.6834.0134.1534.3935.14

The expanded evaluation across 19 benchmarks provides stronger evidence of the generalization benefits of LASR. Specifically, LASR outperforms BASE and PC by 1.73 and 0.75 percentage points on average, achieving improvements on 18/19 and 16/19 tasks, respectively. Importantly, even after excluding BoolQ, where LASR exhibits a relatively large gain, the average improvements remain 1.62 and 0.58 percentage points over BASE and PC. While individual task results may exhibit statistical uncertainty, the expanded evaluation provides complementary evidence of the effectiveness of adaptive spectral control.

AuthorsOpenReview ↗

Response to Reviewer LWga (3)

W3: The efficiency gain is not convincing. The ~1.4× speedup is counted in iterations, while each step costs about 14% more, so the actual wall-clock benefit is unclear. The speedup is measured only against BASE, not PC, and the per-step overhead is reported only for the 350M model.

We sincerely thank the reviewer for raising this important concern. We have additionally conducted wall-clock training time comparisons on the 1B model, including all baseline methods. Our original intention in Figure 3 was to highlight the optimization differences between training with and without spectral control. To provide a more comprehensive comparison, we have included the wall-clock training loss curves for all methods in the revised manuscript (Appendix A.2), and summarize the results in tabular form below.

BASEONIENFCANSIFNSOPCLASR
5h3.3853.4723.4113.5473.5273.6423.546
10h3.0573.0973.0823.1293.1013.2343.123
15h2.9372.9672.9412.9902.9633.0232.962
20h2.8622.8872.8692.9052.8872.9152.871
25h2.8032.8322.8102.8472.8322.8382.812
30h2.7742.7932.7702.8042.7832.7842.763
35h2.7452.7622.7472.7652.7572.7572.732
40h2.7262.7352.7252.7452.7332.7222.708
45h2.7072.7172.7102.7192.7122.6952.681
50h2.6902.6982.6922.7032.6942.6692.665
55h-2.6862.6742.6972.6852.6562.647
60h-2.677-2.6752.6692.6422.632
65h---2.667---

The iteration-level speedup exceeds 1.3×1.3\times, while the wall-clock speedup about 1.2×1.2\times. We appreciate the reviewer's important point and have added a corresponding clarification to the revised manuscript.

Regarding computational overhead, as shown in Table 1, these spectral methods share the same asymptotic computational complexity, with their costs scaling proportionally with model dimensions. Since the overhead is primarily dominated by matrix multiplication, the relative computational costs are expected to remain stable across model scales. To further validate this, we additionally report the measured runtime overhead of different methods on the 1B model below.

WTPowerFwdProxyGradAccVJP
ONI - 350M1123.15.0 (0.4%)299.2 (26.6%)7.6 (0.7%)136.3 (12.1%)675.0 (60.1%)
ENF - 350M678.75.5 (0.8%)165.5 (24.4%)8.5 (1.3%)143.7 (21.2%)355.5 (52.4%)
CANS - 350M1407.04.9 (0.3%)383.9 (27.3%)7.9 (0.6%)135.2 (9.6%)875.2 (62.2%)
IFNSO - 350M1328.85.3 (0.4%)362.6 (27.3%)8.2 (0.6%)150.3 (11.3%)802.4 (60.4%)
PC - 350M1143.8432.1 (37.8%)184.5 (16.1%)8.1 (0.7%)138.7 (12.1%)380.5 (33.3%)
LASR - 350M1283.9457.5 (35.6%)216.9 (16.9%)8.5 (0.7%)140.1 (10.9%)460.9 (35.9%)
ONI - 1B2529.25.8 (0.2%)635.3 (25.1%)10.2 (0.4%)289.5 (11.4%)1588.4 (62.8%)
ENF - 1B1528.95.2 (0.3%)324.7 (21.2%)12.4 (0.8%)307.4 (20.1%)879.2 (57.5%)
CANS - 1B2998.15.0 (0.2%)725.6 (24.2%)11.1 (0.4%)313.8 (10.5%)1942.6 (64.8%)
IFNSO - 1B3148.45.1 (0.2%)709.7 (22.5%)13.3 (0.4%)335.2 (10.6%)2085.1 (66.2%)
PC - 1B2508.9962.4 (38.4%)355.2 (14.2%)13.2 (0.5%)295.6 (11.8%)882.5 (35.2%)
LASR - 1B2621.6980.1 (37.4%)384.0 (14.6%)12.9 (0.5%)299.3 (11.4%)945.3 (36.1%)

We thank the reviewer for highlighting the computational cost of power iteration. This difference arises because PC and LASR employ a pre-normalization strategy, where the normalization factors are computed in advance and treated as constants during gradient computation. In contrast, other spectral baselines perform the corresponding computations during backpropagation, with their costs primarily reflected in the VJP component. Consequently, the combined Power and VJP overhead remains relatively stable across model scales. As shown in the table, the relative computational costs also remain broadly consistent when scaling from 350M to 1B, aligning with our complexity analysis in Table 1. The table is updated in Appendix A.3.

Reviewer_jiYmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LASR lets each weight matrix learn its own spectral mapping during pretraining, instead of using one fixed mapping for all weights as PC, CANS, IFNSO, ENF, and ONI do, with the mapping bounded by construction between the identity and full equalization. On LLaMA3-350M, LLaMA3-1.3B, and OLMoE-7B-A1B trained on C4-en, it gets the best average zero-shot accuracy of the methods compared, at a measured 14.0% per-iteration overhead on the 350M model.

Critical strengths
  • The idea is clean and the motivation is real. The learned mappings are not uniform at all: Q and K move only a little, while V, O, and the MLP projections reshape much more, especially in deeper layers (Figure 5).
  • Table 4 tests the main claim directly. Fully weight-specific mappings beat layer-shared, weight-type-shared, globally shared, and frozen mappings on both loss and accuracy at all three scales, so the gain does seem to come from per-weight adaptivity, not from spectral reshaping alone.
  • The evaluation is broad and the costs are measured, not hand-waved. Two backbones, two optimizers, and five fixed-mapping baselines, with LASR best on average in every setting; Table 7 reports a 14.0% iteration-time overhead and a 1.004x memory ratio on the 350M model.
Critical weaknesses
  • The scale is small for a pretraining claim, and everything is on a single corpus. The biggest dense model is 1.3B at about 42B tokens on C4-en, and the average gains over BASE are roughly 1.5 to 2 points. I do not know if they survive at larger dense scale or over longer runs, which is the regime the paper talks about.
  • The speedup is counted in iterations, not wall-clock time, and only against the plain baseline. Figure 3 shows 1.37x to 1.45x fewer iterations versus BASE, but each iteration costs 14.0% more, so the wall-clock gain is closer to 1.2x, and there are no loss curves against PC or the other spectral methods. The profiling is also only on the 350M model, where power iteration alone is 35.6% of LASR's Weight Transformer time versus 0.3 to 0.8% for the fixed-mapping baselines, so the overhead share may grow with model size.
  • The average gains are concentrated in a few tasks. BoolQ gains 8.68 points at OLMoE scale, and BoolQ, SciQ, and ARC-E gain the most at the smaller scales, while the rest of the suite moves much less and MMLU sits at about 23%, basically chance, for every method. So the averages make the improvement look broader than it is, and I also could not tell whether each baseline's own hyperparameters were tuned, which matters because the authors reproduced all five baselines themselves.
Questions for authors
  1. What is the wall-clock time for LASR and for PC to reach the baseline's final loss at each scale, once the 14.0% overhead is counted? Can you add loss curves for the spectral baselines, not just BASE, to Figure 3?
  2. A few tasks carry the average. What do the averages look like with BoolQ left out, and why do BoolQ, SciQ, and ARC-E move so much more than the rest?
  3. Do you have any evidence at larger scale or on longer runs, for example a dense model above 3B or well beyond 42B tokens, that the gain does not shrink as training goes on?
4 comments
AuthorsOpenReview ↗

Response to Reviewer jiYm (1)

We sincerely thank the reviewer for their positive feedback and thoughtful questions. We will address each of these questions in detail in our subsequent responses.

W1: The scale is small for a pretraining claim, and everything is on a single corpus. The biggest dense model is 1.3B at about 42B tokens on C4-en, and the average gains over BASE are roughly 1.5 to 2 points. I do not know if they survive at larger dense scale or over longer runs, which is the regime the paper talks about.

We sincerely thank the reviewer for raising this important concern. Due to computational constraints, our original experiments were limited to dense models at the 1B scale and an MoE model with 7BA1B total parameters, trained for 10,000 iterations (approximately 40B tokens). To address concerns about scalability and longer training, we additionally trained a 3B dense model for 25,000 iterations (approximately 100B tokens). Given the limited time and our computational resources during rebuttal, we selected BASE and PC (the strongest spectral baseline) for comparison.

Our 3B model configuration is also based on a scaled LLaMA3 architecture:

ExpertsLayersAttention headsEmbedding dimHidden size
LLaMA3 - 3B1 / 1322025606912

Due to the increased model dimensionality, we set the peak learning rate to 2e-4 to mitigate potential loss spikes and ensure training stability, while keeping all other hyperparameters unchanged. The results are shown in the following table:

MethodMMLUARC-EBoolQHellaS.PIQASCIQWinoG.Avg.
BASE23.5557.2756.1339.2272.3477.3056.2554.58
PC23.6859.4456.8540.0473.1777.8056.8255.40
LASR23.7560.8657.5441.3573.0479.1057.6556.18

Recent studies have demonstrated the effectiveness of spectral control in neural network optimization. Furthermore, the training loss curves confirm that LASR maintains a stable advantage throughout extended pretraining, achieving 1.50× and 1.19× iteration-level speedups over BASE and PC, respectively. These results provide further evidence that the effectiveness of LASR persists at larger model scales and longer training durations. Due to space limitations, we refer the reviewer to Appendix A.1 of the revised manuscript for the complete training curves and detailed experimental results.

AuthorsOpenReview ↗

Response to Reviewer jiYm (2)

W2: The speedup is counted in iterations, not wall-clock time, and only against the plain baseline. Figure 3 shows 1.37x to 1.45x fewer iterations versus BASE, but each iteration costs 14.0% more, so the wall-clock gain is closer to 1.2x, and there are no loss curves against PC or the other spectral methods. The profiling is also only on the 350M model, where power iteration alone is 35.6% of LASR's Weight Transformer time versus 0.3 to 0.8% for the fixed-mapping baselines, so the overhead share may grow with model size.

We sincerely thank the reviewer for raising this important concern. We have additionally conducted wall-clock training time comparisons on the 1B model, including all baseline methods. Our original intention in Figure 3 was to highlight the optimization differences between training with and without spectral control. To provide a more comprehensive comparison, we have included the wall-clock training loss curves for all methods in the revised manuscript (Appendix A.2), and summarize the results in tabular form below.

BASEONIENFCANSIFNSOPCLASR
5h3.3853.4723.4113.5473.5273.6423.546
10h3.0573.0973.0823.1293.1013.2343.123
15h2.9372.9672.9412.9902.9633.0232.962
20h2.8622.8872.8692.9052.8872.9152.871
25h2.8032.8322.8102.8472.8322.8382.812
30h2.7742.7932.7702.8042.7832.7842.763
35h2.7452.7622.7472.7652.7572.7572.732
40h2.7262.7352.7252.7452.7332.7222.708
45h2.7072.7172.7102.7192.7122.6952.681
50h2.6902.6982.6922.7032.6942.6692.665
55h-2.6862.6742.6972.6852.6562.647
60h-2.677-2.6752.6692.6422.632
65h---2.667---

The iteration-level speedup exceeds 1.3×1.3\times, while the wall-clock speedup about 1.2×1.2\times. We appreciate the reviewer's important point and have added a corresponding clarification to the revised manuscript.

Regarding computational overhead, as shown in Table 1, these spectral methods share the same asymptotic computational complexity, with their costs scaling proportionally with model dimensions. Since the overhead is primarily dominated by matrix multiplication, the relative computational costs are expected to remain stable across model scales. To further validate this, we additionally report the measured runtime overhead of different methods on the 1B model below.

WTPowerFwdProxyGradAccVJP
ONI - 350M1123.15.0 (0.4%)299.2 (26.6%)7.6 (0.7%)136.3 (12.1%)675.0 (60.1%)
ENF - 350M678.75.5 (0.8%)165.5 (24.4%)8.5 (1.3%)143.7 (21.2%)355.5 (52.4%)
CANS - 350M1407.04.9 (0.3%)383.9 (27.3%)7.9 (0.6%)135.2 (9.6%)875.2 (62.2%)
IFNSO - 350M1328.85.3 (0.4%)362.6 (27.3%)8.2 (0.6%)150.3 (11.3%)802.4 (60.4%)
PC - 350M1143.8432.1 (37.8%)184.5 (16.1%)8.1 (0.7%)138.7 (12.1%)380.5 (33.3%)
LASR - 350M1283.9457.5 (35.6%)216.9 (16.9%)8.5 (0.7%)140.1 (10.9%)460.9 (35.9%)
ONI - 1B2529.25.8 (0.2%)635.3 (25.1%)10.2 (0.4%)289.5 (11.4%)1588.4 (62.8%)
ENF - 1B1528.95.2 (0.3%)324.7 (21.2%)12.4 (0.8%)307.4 (20.1%)879.2 (57.5%)
CANS - 1B2998.15.0 (0.2%)725.6 (24.2%)11.1 (0.4%)313.8 (10.5%)1942.6 (64.8%)
IFNSO - 1B3148.45.1 (0.2%)709.7 (22.5%)13.3 (0.4%)335.2 (10.6%)2085.1 (66.2%)
PC - 1B2508.9962.4 (38.4%)355.2 (14.2%)13.2 (0.5%)295.6 (11.8%)882.5 (35.2%)
LASR - 1B2621.6980.1 (37.4%)384.0 (14.6%)12.9 (0.5%)299.3 (11.4%)945.3 (36.1%)

We thank the reviewer for highlighting the computational cost of power iteration. This difference arises because PC and LASR employ a pre-normalization strategy, where the normalization factors are computed in advance and treated as constants during gradient computation. In contrast, other spectral baselines perform the corresponding computations during backpropagation, with their costs primarily reflected in the VJP component. Consequently, the combined Power and VJP overhead remains relatively stable across model scales. As shown in the table, the relative computational costs also remain broadly consistent when scaling from 350M to 1B, aligning with our complexity analysis in Table 1. The table is updated in Appendix A.3.

AuthorsOpenReview ↗

Response to Reviewer jiYm (3)

W3: The average gains are concentrated in a few tasks. BoolQ gains 8.68 points at OLMoE scale, and BoolQ, SciQ, and ARC-E gain the most at the smaller scales, while the rest of the suite moves much less and MMLU sits at about 23%, basically chance, for every method. So the averages make the improvement look broader than it is, and I also could not tell whether each baseline's own hyperparameters were tuned, which matters because the authors reproduced all five baselines themselves.

We sincerely thank the reviewer for raising these two important concerns. In fact, pretrained models naturally exhibit varying performance across different downstream tasks, and their overall capabilities are better reflected by evaluations across a broad range of benchmarks. Our original evaluation included several commonly used benchmark tasks adopted in prior studies. To address the reviewer's concern, we have further evaluated the 1B model on a more comprehensive benchmark suite (7 original + 12 additional).

BASEONIENFCANSIFNSOPCLASR
MMLU23.0723.0423.0523.1123.1023.0623.01
ARC-E50.5150.7351.2252.4552.8253.0554.44
BOOLQ53.3353.4653.8552.6752.4553.1056.97
HELLAS.36.2036.4436.3937.2237.1937.3337.68
PIQA68.8568.5369.3469.8869.9370.1269.59
SCIQ75.1075.3075.7076.7077.5077.3078.20
WINOG.53.8853.7254.2553.4353.6654.1254.34
WMDP.15.5615.5015.5515.3015.4515.6615.82
TMLU21.8021.7421.9521.3721.2421.6722.25
PROST.20.3920.0518.4619.1320.2020.3620.88
AEXAMS16.6517.2017.1618.2817.8417.9419.87
ACLUE.17.5517.1417.5017.4317.6517.8418.48
ANLI.26.4526.5627.1327.3527.2227.9727.66
MATHQA19.0219.4719.2520.6619.3520.3721.20
HAERAE14.2014.3514.1214.2214.4714.2914.65
COMQA.17.4518.3317.6917.8818.2118.4618.82
LOGIQA18.1018.2118.2218.3718.2918.6519.06
MULTIRC40.4843.2442.4543.3944.1044.1445.77
RTE.46.2346.2546.7347.4048.2147.9448.95
AVG33.4133.6533.6834.0134.1534.3935.14

The expanded evaluation across 19 benchmarks provides a more comprehensive assessment of pretrained model performance. Our results demonstrate that LASR achieves consistent improvements in average generalization performance, outperforming BASE and PC by 1.73 and 0.75, respectively. We emphasize that performance variations across tasks mainly arise from differences in task difficulty. While improvements on challenging tasks are inherently harder to achieve, overall generalization across diverse tasks remains a key criterion for evaluating pretrained models.

We also thank the reviewer for raising the concern regarding hyperparameter tuning. Our experimental framework is built upon the standard pretraining pipeline of Megatron-LM, where we extend the linear layers to support additional mathematical transformations of their parameters. For training hyperparameters, we consistently follow the officially recommended settings of the LLaMA3 and OLMoE technical reports and the PC paper. For method-specific hyperparameters, we show the selection as follows:

MethodPolynomial / Spectral MappingMethod-specific Hyperparameters
ONINewton iteration: Bt=1.5Bt−1−0.5Bt−13SB_t = 1.5B_{t-1} - 0.5B_{t-1}^3SIterations = 4; coefficients = (1.5, -0.5); epsilon = 1e-5
ENFp1(x)=x−αx3p_1(x) = x - \alpha x^3, p2(x)=x+αx3p_2(x) = x + \alpha x^3α = 0.1; spectral scale = 1.0
CANSp(x)=a1x+a3x3+a5x5p(x) = a_1x + a_3x^3 + a_5x^5Order = 5; iterations = 4; coefficients listed below
IFNSOP(R)=I+∑l=1LαlR2l−1P(R) = I + \sum_{l=1}^{L} \alpha_l R^{2^{l-1}}L = 14; fixed offline-fitted coefficients are [0.487649, 0.481625, 0.719537, 1.246460, 1.156044, 3.124816, 0.998430, 9.335488, -5.110362, 34.735029, -44.114691, 130.881855, -161.856947, 237.302668]
PC2g(x)=2.083x−1.643x3+0.560x5g(x) = 2.083x - 1.643x^3 + 0.560x^5Polynomial level = 2; power iterations = 10; gamma init = 1.0
PC4g(x)=3.625x−9.261x3+14.097x5−10.351x7+2.890x9g(x) = 3.625x - 9.261x^3 + 14.097x^5 - 10.351x^7 + 2.890x^9Polynomial level = 4; power iterations = 10; gamma init = 1.0

Our extensive experiments demonstrate that LASR consistently improves generalization performance across a broad range of downstream tasks. By adopting a unified training framework and consistent hyperparameter settings, we ensure fair comparisons across all methods. These comprehensive evaluations further validate the effectiveness and broad generalizability of our adaptive spectral control method. We will incorporate these additional results and discussions into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer jiYm (4)

Q1: What is the wall-clock time for LASR and for PC to reach the baseline's final loss at each scale, once the 14.0% overhead is counted? Can you add loss curves for the spectral baselines, not just BASE, to Figure 3?

See W2 and additional figures and results are stated in Appendix A.1 and A.2. We also report the new 3B wall-time comparisons in the revision and hope this can address the reviewer's concerns.

Q2: A few tasks carry the average. What do the averages look like with BoolQ left out, and why do BoolQ, SciQ, and ARC-E move so much more than the rest?

See answer in W3.

Q3: Do you have any evidence at larger scale or on longer runs, for example a dense model above 3B or well beyond 42B tokens, that the gain does not shrink as training goes on?

See W1 and corresponding loss curves in Appendix A.1. We add the new LLaMA3-3B training results on about 100B tokens. The spectrum control is a generally useful method on the training, which improves the geometry of models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To AC: Summary of Rebuttal and Contributions

We sincerely thank the Area Chair and all reviewers for their time and thoughtful feedback. We recognize the substantial reviewing workload in the current research community. Nevertheless, we regret that we have not yet received any feedback from the reviewers during the discussion phase. To help reduce the effort required for AC to clearly understand our work and revision of rebuttal, we summarize below the concerns addressed during rebuttal, the corresponding evidence in our revision, and the motivation and contributions of our work. More details are shown in reponses to each reviewer.

Rebuttal Process

We conducted additional experiments and provided clarifications in response to all concerns. The table below summarizes these updates.

ConcernjiYmk1Q7LWgaLocation in revision / rebuttalMain result or clarification
Larger models and longer pretraining\sqrt{}\sqrt{}Appendix A.1Added dense 3B pretraining on approximately 100B C4-en tokens
Actual wall-clock acceleration and comparisons with spectral baselines\sqrt{}\sqrt{}\sqrt{}Appendices A.1 and A.2Added wall-clock loss curves on 1B and 3B models
Computational overhead and power-iteration costs across scales\sqrt{}\sqrt{}\sqrt{}Appendix A.3Added runtime breakdowns at 1B model
Breadth of downstream improvements\sqrt{}\sqrt{}\sqrt{}Appendix A.4Expanded evaluation from 7 to 19 tasks
Some tasks dominate the average gains\sqrt{}\sqrt{}\sqrt{}Appendices A.1, A.4; rebuttalCalculate the average accuracy on non-dominant
Baseline configurations and fairness of learning-rate and degree selection\sqrt{}\sqrt{}Appendices A.6, C; rebuttalClarified configurations and added PC with K=5 and K=6 across four learning rates
Comparisons with simpler scalar adaptation and shared mappings\sqrt{}Table 4; Appendix A.5Fully adaptive LASR outperforms scalar-only and shared-mapping variants across the evaluated model scales
Whether directly learning PC coefficients is sufficient\sqrt{}Appendix A.5Naively gated PC slightly degrades performance and can produce negative signed spectral responses; LASR maintains bounded amplification.

Motivation and Core Contributions

The importance of LASR centers on three points:

  1. Different weights on different layers require different spectral transformations. Existing polynomial spectral methods for LLM training largely approximate mappings prescribed before training. However, prior module-wise ablations and our observation indicate that different layers and parameter types favor different transformations. LASR allows each weight matrix to learn its mapping directly from the LM objective, providing finer control over parameter geometry.

  2. Adaptive shaping retains access to the original parameterization. Fixed spectral constraints change the optimization geometry and, when they restrict attainable weights, may exclude solutions of the original LM objective. LASR can approach the identity transformation as its gates approach zero, placing the original parameterization and its solutions in the closure of the learnable family. This does not compromise the original optimization objective.

  3. A structured positive basis makes adaptation controllable. Existing polynomial spectral methods compute fixed coefficients to approximate a predefined target mapping, often yielding both positive and negative coefficients. These coefficients are jointly designed for that fixed approximation; introducing independent learnable gates can disrupt their cancellations and produce negative spectral responses, so bounded adaptive shaping is not guaranteed. LASR constructs a structured polynomial expansion with positive coefficients and bounded learnable gates, keeping normalized spectral responses within the prescribed feasible region while enabling weight-specific adaptation.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.