- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Polynomial weight preconditioning methods, such as [1], reshapes the spectra of weight matrices with a fixed polynomial transformation. This paper proposes learning a spectral transformation, one for each weight matrix, trained together with the model via the language-modeling objective. It uses a taylor expansion with bounded learnable coefficients.
The experiments test whether learning these transformations improves pretraining and downstream performance. LASR reaches the same training loss in fewer steps and improves average zero-shot accuracy over the standard and spectral baselines with both AdamW and Muon. An ablation finds that separate transformations for individual matrices outperform shared coefficients.
Critical strengths
-
(S1) End-to-end spectral shaping. The main appeal is learning spectral shaping end to end under the LM objective. This offers a way to reduce manual choices about target spectra and shaping strengths for individual matrices. The parameterization used by LASR gives a simple, bounded way to do this with few learned parameters per matrix.
-
(S2) Benefits of layer-specific adaptation. The sharing ablation provides evidence that adaption across different layers is useful. The gains over PC also support the usefulness of the approach in the studied studied in the paper.
Critical weaknesses
-
(W1) Unclear hyperparameter tuning. My main concern is the fairness of hyperparameter selection. All methods share a learning-rate schedule, but the paper doesn't explain how the learning rates were selected or whether they were tuned for each method. Additionaly, the authors perform a polynomial-degree sweep for LASR, while the baselines' hyperparameter tuning protocol is not reported. E.g., PC [1] offers different spectra shaping strengths. A common setting can suit different methods differently, so it's hard to tell whether the advantage survives with tuned baselines. See next point.
-
(W2) Modest gains and unclear uncertainty. LASR's average downstream gains over PC are less than one percentage point, and some tasks get worse. Additionally, given the sensitivity of downstream evaluations, I'd want to see that the gains survive variation across repeated evaluations of the same trained models.
Questions for authors
- (Q1) LM objective and spectral shaping. Why do you expect minimizing the LM loss to steer the spectra toward better conditioning and generalization? The end to end concept is appealing, but I'd appreciate further motivation on this.
- (Q2) Hyperparameter selection. How were the learning rates and hyperparameters selected for LASR and each baseline? Please clarify the search budgets and data used for tuning, including the degree sweep for LASR. I'd want to see whether the advantage holds with comparably tuned baselines.
- (Q3) Evaluation repeatability. Do the gains presist after repeated evaluations with different evaluation seeds (given the noisiness of these downstream evals)?
- (Q4) Wall-clock acceleration. After accounting for the extra time per iteration, does LASR reach the same loss faster than the base method and PC?.
Given convincing answers to the above questions I'd consider raising the score.
[1] Wang et al. PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training. arXiv:2606.06470, 2026.
6 comments
Response to Reviewer k1Q7 (2)
W2: Modest gains and unclear uncertainty. LASR's average downstream gains over PC are less than one percentage point, and some tasks get worse. Additionally, given the sensitivity of downstream evaluations, I'd want to see that the gains survive variation across repeated evaluations of the same trained models.
We sincerely thank the reviewer for raising this important concern. In fact, pretrained models naturally exhibit varying performance across different downstream tasks, and their overall capabilities are better reflected by evaluations across a broad range of benchmarks. Our original evaluation included several commonly used benchmark tasks adopted in prior studies. To address the reviewer's concern, we have further evaluated the 1B model on a more comprehensive benchmark suite (7 original + 12 additional), demonstrating the benefits of spectral control in model pretraining and further validating the effectiveness of our proposed adaptive spectral control method.
| BASE | ONI | ENF | CANS | IFNSO | PC | LASR | |
|---|---|---|---|---|---|---|---|
| MMLU | 23.07 | 23.04 | 23.05 | 23.11 | 23.10 | 23.06 | 23.01 |
| ARC-E | 50.51 | 50.73 | 51.22 | 52.45 | 52.82 | 53.05 | 54.44 |
| BOOLQ | 53.33 | 53.46 | 53.85 | 52.67 | 52.45 | 53.10 | 56.97 |
| HELLAS. | 36.20 | 36.44 | 36.39 | 37.22 | 37.19 | 37.33 | 37.68 |
| PIQA | 68.85 | 68.53 | 69.34 | 69.88 | 69.93 | 70.12 | 69.59 |
| SCIQ | 75.10 | 75.30 | 75.70 | 76.70 | 77.50 | 77.30 | 78.20 |
| WINOG. | 53.88 | 53.72 | 54.25 | 53.43 | 53.66 | 54.12 | 54.34 |
| WMDP. | 15.56 | 15.50 | 15.55 | 15.30 | 15.45 | 15.66 | 15.82 |
| TMLU | 21.80 | 21.74 | 21.95 | 21.37 | 21.24 | 21.67 | 22.25 |
| PROST. | 20.39 | 20.05 | 18.46 | 19.13 | 20.20 | 20.36 | 20.88 |
| AEXAMS | 16.65 | 17.20 | 17.16 | 18.28 | 17.84 | 17.94 | 19.87 |
| ACLUE. | 17.55 | 17.14 | 17.50 | 17.43 | 17.65 | 17.84 | 18.48 |
| ANLI. | 26.45 | 26.56 | 27.13 | 27.35 | 27.22 | 27.97 | 27.66 |
| MATHQA | 19.02 | 19.47 | 19.25 | 20.66 | 19.35 | 20.37 | 21.20 |
| HAERAE | 14.20 | 14.35 | 14.12 | 14.22 | 14.47 | 14.29 | 14.65 |
| COMQA. | 17.45 | 18.33 | 17.69 | 17.88 | 18.21 | 18.46 | 18.82 |
| LOGIQA | 18.10 | 18.21 | 18.22 | 18.37 | 18.29 | 18.65 | 19.06 |
| MULTIRC | 40.48 | 43.24 | 42.45 | 43.39 | 44.10 | 44.14 | 45.77 |
| RTE. | 46.23 | 46.25 | 46.73 | 47.40 | 48.21 | 47.94 | 48.95 |
| AVG | 33.41 | 33.65 | 33.68 | 34.01 | 34.15 | 34.39 | 35.14 |
The expanded evaluation across 19 benchmarks provides stronger evidence of the generalization benefits of LASR. Specifically, LASR outperforms BASE and PC by 1.73 and 0.75 percentage points on average, achieving improvements on 18/19 and 16/19 tasks, respectively. Importantly, even after excluding BoolQ, where LASR exhibits a relatively large gain, the average improvements remain 1.62 and 0.58 percentage points over BASE and PC. These results suggest that the improvements are not driven by a single benchmark but are broadly observed across diverse downstream tasks. While individual task results may exhibit statistical uncertainty, the expanded evaluation provides complementary evidence of the effectiveness of adaptive spectral control.
Response to Reviewer k1Q7 (3)
Q1: LM objective and spectral shaping. Why do you expect minimizing the LM loss to steer the spectra toward better conditioning and generalization? The end to end concept is appealing, but I'd appreciate further motivation on this.
We sincerely thank the reviewer for this very insightful question. Our motivation is based on two considerations.
First, from an optimization perspective, standard LM pretraining minimizes , whereas fixed spectral control methods optimize , where denotes a predefined spectral transformation. Since the optimal weight geometry is unknown, imposing a fixed transformation may introduce an optimization bias that is not necessarily aligned with the original objective. To address this limitation, LASR introduces learnable spectral control coefficients, which can be conceptually expressed as minimizing . As learnable approaches 0, the original parameterization is recovered, preserving access to the original solution space.
Second, our motivation is further supported by an empirical observation in the PC paper. PC selectively applies spectral preconditioning to the attention output projection and FFN weights, while its ablation studies reveal that extending spectral control to other modules does not consistently improve performance. This suggests that different network components may have heterogeneous spectral requirements, making predefined spectral transformations potentially suboptimal, particularly as model architectures evolve. Motivated by this observation, LASR allows the LM objective to directly guide layer-specific spectral transformations rather than relying on manually prescribed spectral structures. Our results in Appendix C.7 further support this motivation: some layers exhibit negligible spectral changes, whereas others undergo substantial spectral reshaping, indicating that the network learns distinct spectral preferences across layers. Thus, LASR leverages task-driven optimization to adapt spectral control to the actual requirements of different network components.
Compared with the first challenge, we consider the second more fundamental. Our experimental results suggest that the LM objective can effectively guide the search for favorable spectral configurations within a predefined spectral control space in the pretraining, rather than relying on fixed spectral transformations.
Q2: Hyperparameter selection. How were the learning rates and hyperparameters selected for LASR and each baseline? Please clarify the search budgets and data used for tuning, including the degree sweep for LASR. I'd want to see whether the advantage holds with comparably tuned baselines.
We sincerely thank the reviewer for this important question. As discussed in W1, our experimental protocol prioritizes fairness and consistency. Specifically, the general training hyperparameters, including the learning rate and optimizer settings, are adopted from established LLM pretraining practices reported in leading industry technical reports and the baseline papers, rather than being optimized for LASR.
Regarding the polynomial degree , we follow the experimental protocol of the PC paper, which evaluates different polynomial configurations from to . We extend this investigation to – to systematically examine the effect of polynomial complexity on LASR. Our experiments indicate that or generally achieves the best performance across different model configurations.
Importantly, is the only LASR-specific structural hyperparameter, determining the flexibility and upper amplification envelope of the learnable spectral transformation. Increasing allows the spectral mapping to approach a more balanced singular-value distribution. However, excessively large introduces higher-order terms that predominantly affect small singular values, potentially amplifying numerical sensitivity and causing optimization fluctuations under the same learning rate. Therefore, we recommend as a robust default, as it consistently achieves competitive performance across all evaluated models while maintaining a favorable balance between spectral flexibility and training stability.
Response to Reviewer k1Q7 (1)
We sincerely thank you for reviewing our manuscript and providing valuable suggestions. We will address each of your concerns in detail in our rebuttal.
W1: Unclear hyperparameter tuning. My main concern is the fairness of hyperparameter selection. All methods share a learning-rate schedule, but the paper doesn't explain how the learning rates were selected or whether they were tuned for each method. Additionaly, the authors perform a polynomial-degree sweep for LASR, while the baselines' hyperparameter tuning protocol is not reported. E.g., PC [1] offers different spectra shaping strengths. A common setting can suit different methods differently, so it's hard to tell whether the advantage survives with tuned baselines. See next point.
We sincerely thank the reviewer for raising this important concern. Our framework is built upon the standard Megatron-LM pretraining pipeline, with general hyperparameters, including learning rates and optimizer settings, following established practices from baseline studies and technical reports of leading industry labs. We prioritize fairness and consistency by adopting the same training hyperparameters across all methods. Method-specific tuning based on limited training stages may overfit to intermediate performance, as improvements observed early in pretraining do not necessarily persist until convergence. Therefore, we adopt standardized pretraining configurations to ensure fair comparisons and avoid potential biases introduced by individually tuned learning rates.
For method-specific hyperparameters, we follow the configurations reported in the original papers wherever applicable, including polynomial coefficients, approximation degrees, and iteration counts, with necessary adaptations to our unified framework.
Coefficients of four iterations in the CANS method:
Most baseline methods employ fixed spectral control functions. Regarding the different spectral shaping configurations of PC mentioned by the reviewer, we strictly follow the settings recommended in the original paper, which adopts PC4 for AdamW and PC2 for Muon based on their experimental ablations.
For general hyperparameters, the learning rate is one of the most critical factors affecting pretraining performance. We therefore summarize the learning-rate configurations reported in the official technical reports of LLaMA3 and OLMoE, as well as the PC Layer paper, in the following table.
Based on these established pretraining configurations, we selected a balanced set of learning rates and general training hyperparameters that aligns with existing practices across different model scales and optimizers, ensuring a consistent and fair comparison among methods. Importantly, we deliberately avoided tuning these hyperparameters specifically for the performance at 10,000 iterations, as such short-horizon optimization could lead to overfitting to a limited training budget and may not generalize to longer pretraining runs. Instead, our choices follow established large-scale pretraining practices, aiming to ensure that the observed improvements reflect the effectiveness of the proposed method rather than hyperparameter choices tailored to a particular training horizon.