acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Large Language Models with Chained LMOs

Abstract

Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective. We introduce chained linear minimization oracles (chained LMOs), which cast these methods as compositions of LMOs. Despite their empirical success, many chains fall outside the standard LMO framework and can diverge on smooth convex objectives. To explain why composition can nevertheless help, we turn to linear associative memory and show that chaining can improve over Muon under anisotropic embeddings. Empirically, we propose TensorChain, a novel optimizer within the framework that stacks compatible weight matrices across different layers and normalizes the 3d tensor across its axes. In Qwen3 0.6B and 1.7B pretraining, TensorChain outperforms all chained baselines in average token efficiency, with average token savings of 9.6% over Muon at matched validation loss.

Primary area:
optimization
Keywords:
Optimization, Large Language Models, Muon, Linear Minimization Oracles, Normalized Updates

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_V2bGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents chained linear minimization oracles (LMOs) as a unified framework for understanding Muon variants that compose multiple normalization operations. It shows that such compositions need not remain LMOs over any fixed constraint set: some admit dimension-dependent convergence guarantees, while others can diverge even on smooth convex objectives. To demonstrate how chaining can nevertheless help, the paper establishes a one-step retrieval advantage over Muon and gradient descent in a linear associative-memory model with anisotropic answer embeddings.

The paper further introduces TensorChain, which stacks compatible momentum matrices across layers and applies normalization along the row, column, and layer axes before and after layerwise orthogonalization. Experiments on Qwen3-0.6B and Qwen3-1.7B cover multiple batch sizes and learning-rate schedules, with training budgets up to 112B tokens. TensorChain consistently improves validation loss over Muon and achieves average token savings of 9.6% across the four cosine-schedule settings.

Critical strengths
  1. The framework provides a useful perspective on existing optimizers, supported by substantive theoretical results.

  2. The associative-memory analysis establishes a concrete setting where chaining improves over Muon.

  3. Experiments cover two model scales, multiple training configurations, and extensive learning-rate sweeps, showing consistent improvements over Muon.

Critical weaknesses
  1. The grouping ablations do not isolate the contribution of layer-axis normalization from the other normalization operations.

  2. Single-seed experiments make it difficult to assess small performance differences between the strongest methods.

  3. How does TensorChain compare with Muon in wall-clock time to a target loss and GPU memory?

  4. Do the validation-loss improvements translate into consistent gains on downstream evaluation benchmarks?

  5. How well does TensorChain generalize beyond Qwen3, particularly to other dense Transformer and MoE architectures?

Questions for authors

See Weaknesses above.

2 comments

We appreciate the constructive feedback and have done our best to address the weaknesses you pointed out, as detailed below.

Response to Weakness 1: In Table 4 of Section 5.2 we had an ablation of TensorChain's performance across different grouping strategies, but we agree that it does not isolate the cross-layer normalization itself. To address your concern, we isolate the effect of cross-layer normalization in [common response 1. Isolating the effect of cross-layer normalization and ablation on iteration number kk]. We find that the iteration count is not crucial for final validation loss, but using cross-layer normalization consistently outperforms the version without it.

Response to Weakness 2: We address this point in [common response 5. Multiseed experiment]. Due to timing and compute constraints we could only perform repeated experiments with three seeds (42, 43, 44) for Qwen3 0.6B model, with 1M batch size and 1x chinchilla. The seed noise was of scale ≈\approx 0.001 and we believe the conclusions on the 0.6B model, ranking across methods with same number of NewtonSchulz operations, and average token efficiency improvement would not change.

Response to Weakness 3: We address this point in [common response 2. Practical cost of tensorchain.] We optimize Muon, Aurora-LN and TensorChain and in 8-way DP 8-way FSDP setup, the overhead of TensorChain is <2% for most cases (except for MoE 3B/A1.1B throughput for A100s), hence there is overall wall-clock time improvement in addition to token efficiency improvement.

Response to Weakness 4: We address the lack of downstream task evaluation in [common response 4. Downstream Evaluation]. We compare three optimizers, Muon, Aurora-LN and TensorChain, and both Aurora-LN and TensorChain improves over Muon by >1% on average for 8 downstream tasks, whereas Aurora-LN and TensorChain have different ranking for 0.6B and 1.7B across different metrics.

Response to Weakness 5: We show larger scale/more diverse architecture experiments in [common response 3. Generalization of TensorChain to different architecture and model size]. We experiment with Llama 1B, Qwen3 3B/A1.1B and Qwen3 30B/A3B models, and show that TensorChain consistently improves over both Muon and Aurora-LN for both token efficiency and overall wall-clock time improvement to reach same validation loss. These results suggest TensorChain generalizes beyond the Qwen3 dense model family.

Thanks for the rebuttal; my main concerns are resolved. I will raise to 3.

Reviewer_3mPzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces chained linear minimization oracles (LMOs) as a unified framework for understanding recent Muon variants. The authors provide theoretical analyses showing that compositions of LMOs may lose standard convergence guarantees, while also establishing conditions under which chaining can improve learning in linear associative memory. Building on this framework, they propose TensorChain, which introduces cross-layer normalization by jointly processing compatible weight matrices. Experiments on Qwen3 0.6B and 1.7B demonstrate consistent improvements over Muon and several existing variants.

Critical strengths
  1. Meaningful theoretical contributions. The chained-LMO framework provides a useful unified perspective on existing Muon variants, supported by nontrivial theoretical results on convergence and associative memory learning.

  2. Strong empirical results. TensorChain demonstrates consistent improvements over Muon across multiple pretraining settings, with an average token-efficiency gain of 9.6%. The comparisons with recent Muon variants further strengthen the empirical contribution.

Critical weaknesses

The main weakness is the lack of a direct ablation isolating the contribution of cross-layer normalization. While the paper compares different layer-grouping strategies, it remains unclear how much of TensorChain's improvement comes specifically from the proposed layer-axis normalization rather than the existing row/column normalization operations.

Questions for authors

Could the authors provide an ablation that removes the layer-axis normalization while keeping the other normalization operations unchanged? I believe this would help establish the contribution of the key design choice in TensorChain.

Overall, I find the paper interesting and technically substantial, and I am positive about its acceptance. A convincing ablation addressing this question would further strengthen my assessment.

2 comments

We sincerely appreciate the positive, constructive feedback and have done our best to address the weaknesses you pointed out, as detailed below.

Response to Weakness 1: In Table 4 of Section 5.2 we had an ablation of TensorChain's performance across different grouping strategies, but we agree that it does not isolate the cross-layer normalization itself. To address your concern, we isolate the effect of cross-layer normalization by optimizing with a variant of Muon, where row/column normalization is applied before and after Newton-Schulz and cross-layer normalization is not applied. We denote this variant as Sink-NS-Sink, as this could be understood as applying iterated row/column normalization before and after NewtonSchulz. We test for k∈{1,2,3,4,5}k \in \lbrace 1,2,3,4,5 \rbrace, where kk denotes the number of iterated normalizations. From the experiments we see that across different kk, the final validation loss does not differ that much, but comparing TensorChain and Sink-NS-Sink shows that TensorChain consistently achieves benefit from cross-layer normalization compared to its counterpart.

Table 1: Ablation of cross-layer normalization and iteration count kk on Qwen3-0.6B (1M-token batch, 1×1\times Chinchilla). Each entry reports final validation loss.

ScheduleMethodk=1k=1k=2k=2k=3k=3k=4k=4k=5k=5
WSDTensorChain2.89922.90032.90052.90022.9000
Sink–NS–Sink2.92102.92182.92142.92162.9213
CosineTensorChain2.83682.83812.83862.83822.8379
Sink–NS–Sink2.86242.86302.86262.86242.8627

Thanks for the discussion regarding cross-layer normalization. I am happy to maintain my positive score.

Reviewer_NWCKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compositions of matrix normalizations through chained linear minimization oracles. It identifies when such compositions lose standard LMO guarantees, proves a one-step retrieval advantage in a structured associative-memory model, and proposes TensorChain to normalize jointly across layers. Its main value is a clearer theoretical account of optimizer compositions, complemented by a controlled comparison of Muon variants.

Critical strengths
  1. The framework connects several recent Muon variants while carefully separating exact LMOs, approximate LMOs, and chains that can have unfavorable gradient alignment.
  2. The theoretical contribution goes beyond intuition: explicit counterexamples support the negative results, and the associative-memory separation uses conditional Gaussian laws and concentration bounds rather than assuming away dependencies.
Critical weaknesses
  1. The theory has limited relevance to practical training. The positive result assumes a single update from zero initialization, coordinate-aligned power-law anisotropy, and answer dimensions growing superquadratically with query dimensions; it does not directly explain TensorChain’s cross-layer normalization. The stateless, exact-matrix-sign analysis also leaves a gap to practical optimizers with internal EMA statistics and finite Newton–Schulz iterations.

  2. Practical gains need stronger validation. Token savings are not accompanied by wall-clock measurements, despite TensorChain’s additional cross-layer communication under FSDP. Experiments use models up to 1.7B, with small gaps among the strongest methods. Larger-scale runs, variability estimates, and downstream evaluations would clarify the robustness and value of these gains. The contribution of cross-layer normalization is not sufficiently isolated.

Questions for authors

N/A.

2 comments

We appreciate the constructive feedback and have done our best to address the weaknesses you pointed out, as detailed below.

Response to Weakness 1: We would like to respectfully disagree that the theory has limited relevance to practical training, because our theoretical contribution is identifying a problem in current theory that fails to explain a practical improvement, and coming up with a possible explanation. Existing theory (steepest descent / LMO theory) fails to elaborate why successive normalization should work well in practice, and within their theoretical framework successive normalization (even established methods such as AdaMuon [1]'s stationary counterpart or the recent Aurora optimizer [2]) is a worst-case diverging method. Our theoretical contribution identifies this and explains why they may work well by introducing different anisotropy axes. Hence we would like to argue that this is an attempt to explain a practical gain, not a theory that is irrelevant from practical training.

[1] Si, Chongjie, Debing Zhang, and Wei Shen. "Adamuon: Adaptive muon optimizer." arXiv preprint arXiv:2507.11005 (2025).

[2] Dewulf, Alec, et al. "Aurora: A Leverage-Aware Spectral Optimizer." arXiv preprint arXiv:2606.27715 (2026).

Nevertheless it is true that TensorChain is not directly motivated from the theory or internal EMA statistics / finite NewtonSchulz iterations are not considered. We would like to leave this to future work; we believe that there could be different layerwise anisotropy that TensorChain can benefit from, and theory of EMA can be extended by letting momentum as a noisy estimation of the gradient. Extension to finite NewtonSchulz is also deemed feasible as we can bound the error between exact and finite NewtonSchulz.

Response to Weakness 2: In Table 4 of Section 5.2 we had an ablation of TensorChain's performance across different grouping strategies, but we agree that it does not isolate the cross-layer normalization itself. To address your concern, we isolate the effect of cross-layer normalization in [common response 1. Isolating the effect of cross-layer normalization and ablation on iteration number kk]. We see that cross-layer normalization consistently improves the version without it for all iteration number kk, whereas the value of kk does not matter that much.

Second, we show that the overhead of TensorChain is minimal compared to Muon, in addition to the theoretical FLOP analysis in Section 4.1. We address this point in [common response 2. Practical cost of tensorchain.] We optimize Muon, Aurora-LN and TensorChain and in 8-way DP 8-way FSDP setup, the overhead of TensorChain is <2% for most cases (except for MoE 3B/A1.1B throughput for A100s), hence there is overall wall-clock time improvement in addition to token efficiency improvement.

Third, we address the lack of downstream task evaluation in [common response 4. Downstream Evaluation]. We compare three optimizers, Muon, Aurora-LN and TensorChain, and both Aurora-LN and TensorChain improves over Muon by >1% on average for 8 downstream tasks, whereas Aurora-LN and TensorChain have different ranking for 0.6B and 1.7B across different metrics.

Fourth, the variability of results can be found in [common response 5. Multiseed Experiment]. We repeat training Qwen3 0.6B model with 1M batch size and 1x chinchilla with three seeds 42, 43, 44 and report the mean/stddev of the final validation loss. We see that the seed noise is ≈\approx 0.001 and our conclusions on 0.6B ranking, comparison with methods with same number of NewtonSchulz, and average token efficiency does not change.

At last, we show larger scale/more diverse architecture experiments in [common response 3. Generalization of TensorChain to different architecture and model size]. We experiment with Llama 1B, Qwen3 3B/A1.1B and Qwen3 30B/A3B models, and show that TensorChain consistently improves over both Muon and Aurora-LN for both token efficiency and overall wall-clock time improvement to reach same validation loss. The MoE 30B/A3B experiments in specific show that TensorChain can scale to larger models.

I sincerely thank the authors for their response, particularly for the additional scaling experiments on larger models, which have helped me better assess the proposed method. I will therefore raise my score.

Reviewer_MaghOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces chained linear minimization oracles (LMOs) as a framework for representing Muon variants that compose several normalization operations. It shows that many idealized chains are not LMOs of any fixed constraint set, derives approximate-LMO guarantees for some chains, and constructs divergence examples for others. A separate analysis establishes a one-step retrieval advantage for row normalization followed by the matrix sign in a particular anisotropic associative-memory setting. The paper also proposes TensorChain, which stacks compatible momentum matrices across layers and applies axis-wise normalization before and after slice-wise Newton–Schulz updates. Experiments on Qwen3-0.6B and Qwen3-1.7B report improved validation loss and an average 9.6% token saving over Muon across four cosine-schedule settings.

Critical strengths
  1. The framework addresses a relevant theoretical question about composing optimizer updates. The distinction between individual LMO steps and their composition is useful. The analysis identifies both convergence limitations and conditions under which approximate-LMO guarantees remain available, rather than assuming that the guarantees of Muon automatically extend to its variants. The associative-memory result also provides a concrete setting in which composition offers a learning advantage.

  2. The experiments provide encouraging evidence for the complete TensorChain optimizer under a common training protocol.The comparison controls several important factors, including model architecture, data, Newton–Schulz approximation, and update scaling. Table 2 reports improvements over Muon and MuonEq in all six settings, covering two model sizes and two learning-rate schedules. The full small-model learning-rate sweeps, grouping experiments, and unsuccessful attempts to replace Newton–Schulz are useful additions.

Critical weaknesses
  1. The independent contribution of the new layer-axis normalization is not sufficiently isolated. Section 4.1 and Algorithm 1 combine repeated row, column, and layer normalization on both sides of the Newton–Schulz step. Table 4 and Appendix F.5 compare different cross-layer groupings, but all of these variants retain layer-axis normalization. These results show that grouping affects the complete method, but do not establish how much improvement comes from adding the layer axis itself. The existing Muon variants also differ in normalization placement, optimizer state, or the number of Newton–Schulz applications. A matched variant that retains the same row/column normalization before and after Newton–Schulz while removing only the layer-axis operation is not reported. Moreover, Section 3 analyzes a single-matrix row-normalization chain, so its positive result does not independently validate the new cross-layer operation. The current evidence supports the complete optimizer, but leaves the benefit of its central new component insufficiently identified.

  2. The robustness of the ranking among chained optimizers is not sufficiently established across training settings.The full learning-rate comparisons in Appendix F.1 are performed on Qwen3-0.6B with a 1M-token batch. Appendix F.2 then reports tuning for Muon and uses the selected learning rates for the other methods. The shared optima observed in the small-model experiments motivate this controlled comparison, but do not establish that the different methods retain similar optima at larger model sizes, batch sizes, and training horizons. The distinct optimum found for AdaMuon illustrates that this assumption can matter. In addition, Section 5.1 uses one fixed random seed. Some differences are small: TensorChain and Aurora-LN differ by only 0.0006 validation loss in the 1.7B WSD setting. The consistent gains over Muon are encouraging, but the relative ranking of the stronger chained baselines would be more convincing with comparable local tuning and repeated runs in representative configurations.

  3. The practical cost of cross-layer normalization is not quantified. Section 4.1 argues that axis-wise normalization introduces lower-order arithmetic cost relative to Newton–Schulz, but also acknowledges additional communication under FSDP. Both model sizes in Section 5.1 use FSDP, making this cost relevant to the reported experiments. Table 3 and Appendix F.6 measure tokens required to reach a common validation loss; they do not report throughput, step time, communication overhead, or time to that loss. The 9.6% result is explicitly a token saving and is supported as such, but it does not establish the end-to-end training-efficiency trade-off of coupling layers. Measurements under the actual distributed configurations would clarify whether the token savings translate into a practical training-time benefit.

  4. Important TensorChain configuration details and result correspondence need clarification. Algorithm 1 uses a repetition count kk before and after Newton–Schulz, but Section 5.1 and Appendix F do not specify the value used for TensorChain or how it was selected. The iteration counts in Appendix E concern replacing Newton–Schulz and do not resolve this omission. This parameter determines both the normalization chain and its cost. There is also an ambiguity in the reported grouping: Section 5.1 specifies grouping all compatible weights, and Table 2 reports 2.9000 for the 0.6B/1M/WSD setting, whereas Table 15 reports 2.8990 for the same labeled setting. These values coincide with the All and Stride results in Table 18, respectively. The difference does not overturn the comparison with Muon, but the configuration underlying each table and trajectory should be stated consistently.

  5. Minor presentation issue: the introduction would benefit from clearer motivation before the formal setup. Equations (1), (2), and (4) repeat the momentum recursion, while the general composition in Eq. (3) appears before a concrete example of a normalization chain. The main symbols and sign correction are explained, but the reader must process substantial formalism before receiving an intuitive explanation of what composition changes. A simple example and a diagram distinguishing cross-layer normalization from slice-wise Newton–Schulz would improve accessibility. A concise main-text explanation of the distinctions from the closely related unifying frameworks discussed in Appendix A.1 would also clarify the contribution. This is a presentation concern rather than an objection to the section order or to including formulas in the introduction.

Questions for authors
  1. How does TensorChain compare with a matched variant that removes only layer-axis normalization while retaining the row/column operations, their order and repetition count, the Newton–Schulz approximation, and the update scaling? Can this comparison be performed with a comparable tuning budget?
  2. Do the rankings persist when the stronger chained baselines receive comparable local learning-rate searches in a representative larger configuration? What variation is observed across repeated seeds for final validation loss and matched-loss token efficiency?
  3. What are the throughput, step time, communication overhead, and time to matched validation loss for TensorChain and Muon under the reported FSDP configurations? How do these costs change with the grouping strategy?
  4. What value of k is used for TensorChain, and was it tuned? Which grouping is used for each main result and trajectory, and how should the discrepancy between Tables 2 and 15 be reconciled?
1 comment

We appreciate the constructive feedback and have done our best to address the weaknesses you pointed out, as detailed below. We pointed you to the common response if there were relevent points addressed in the common response.

Response to Weakness 1 (lack of cross-layer normalization ablation): We address this point in [common response 1. Isolating the effect of cross-layer normalization and ablation on iteration number kk]. We find that the iteration count is not crucial for final validation loss, but using cross-layer normalization consistently outperforms the version without it.

Response to Weakness 2 (reliability issue of ranking between optimizers due to single-seed experiment): We address this point in [common response 5. Multiseed experiment]. Due to timing and compute constraints we could only perform repeated experiments with three seeds (42, 43, 44) for Qwen3 0.6B model, with 1M batch size and 1x chinchilla. The seed noise was of scale ≈\approx 0.001 and we believe the conclusions on the 0.6B model, ranking across methods with same number of NewtonSchulz operations, and average token efficiency improvement would not change.

Response to Weakness 3 (Practical Cost of TensorChain): We address this point in [common response 2. Practical cost of tensorchain.] We optimize Muon, Aurora-LN and TensorChain and in 8-way DP 8-way FSDP setup, the overhead of TensorChain is <2% for most cases (except for MoE 3B/A1.1B throughput for A100s), hence there is overall wall-clock time improvement in addition to token efficiency improvement.

Response to Weakness 4 (TensorChain iteration specification and mismatch in the appendix): Thank you so much for reading the paper in detail. We used k=5k = 5 for the main experiments and will clarify it. The value was not tuned, motivated from 5 NewtonSchulz steps and with the belief that the iteration count will not incur in huge overhead because it is only knmknm FLOPs. However, our experiments in Table 1 shows that k=1k = 1 works as good as k=5k = 5 and Table 3 shows that k=1k = 1 can incur in almost zero overhead, so we used k=1k = 1 for rebuttal experiments.

Also there indeed is a mismatch in the numbers. By reviewing the raw data we figured out that the number in the appendix (Table 15) is wrong and should be corrected to 2.9000 (the stride grouping's number was logged). We will correct the number in the revised version. Thank you for pointing out the inconsistency!

Response to Weakness 5 (Minor presentation issues): Thank you for your valuable suggestions regarding the presentation of the paper. We fully agree that certain parts of the paper can be made clearer to improve its presentation. We plan to make the paper more accessible by:

  • Introducing an example of a normalization chain in the introduction.
  • Adding a figure illustrating the process of TensorChain and how it differs from slicewise NewtonSchulz.
  • Adding a discussion of the differences between existing frameworks and ours after the introduction.

Answer to Question 1: We address this point in our response to Weakness 1.

Answer to Question 2: We address this point in our response to weakness 2 (lack of repeated runs) for multiseed result of final validation loss. The variation of matched-loss token efficiency is reported in Table 1.

Table 1.: Training-token savings (%) of TensorChain relative to baselines on Qwen3-0.6B (1M batch, 8.8B tokens). Mean ± std over three paired seeds (42, 43, 44).

TensorChain vs.CosineWSD
Muon12.9% ± 0.11.8% ± 0.1
Aurora12.3% ± 0.21.6% ± 0.1
MuonEq10.4% ± 0.21.3% ± 0.1
Aurora-LN8.0% ± 0.21.0% ± 0.1
Muon$^2$9.0% ± 0.00.9% ± 0.2

Comments on the paper

7 comments
AuthorsOpenReview ↗

[Common Response to Reviewers]

Table 8: Multiseed experiment result for Qwen3 0.6B, validation loss mean ± std over three runs

MethodCosineWSD
Muon2.8642 ± 0.00072.9232 ± 0.0016
Aurora2.8618 ± 0.00112.9205 ± 0.0012
MuonEq2.8542 ± 0.00102.9164 ± 0.0005
Aurora-LN2.8467 ± 0.00082.9136 ± 0.0004
Muon$^2$2.8495 ± 0.00032.9124 ± 0.0014
TensorChain2.8376 ± 0.00042.9008 ± 0.0007
AuthorsOpenReview ↗

[Common Response to Reviewers]

4. Downstream Evaluation

Two reviewers (reviewer NWCK, reviewer V2bG) asked whether the improvement in validation loss leads to empirical downstream performance improvement. Following [2] we evaluate our trained models in 8 downstream tasks, ARC-C, E, HellaSwag, OpenBookQA, PIQA, LAMBADA, WinoGrande, MMLU. We evaluated our Qwen3 0.6B and 1.7B models trained with 4M batch size and 4x Chinchilla and compared the accuracy, as well as chance - normalized scores that are defined as (accuracy - chance) / (1 - chance), where chance is defined as the probability that the task can be correct just by chance. For instance, for MMLU the chance is 1/41/4. If the chance-normalized score is negative we clipped it to 0, following the OpenLLM leaderboard.

The results can be found in Table 6 and Table 7 for chance-normalized version. In both cases, TensorChain and Aurora-LN beats Muon by average by approximately 1% accuracy, but between Aurora-LN and TensorChain the order is flipped for 0.6B and 1.7B. For 0.6B the gap is less than 0.5% and TensorChain has better average accuracy but worse chance-normalized scores, and sometimes Muon is better than both for PIQA. For 1.7B, TensorChain has higher accuracy than both Muon and Aurora-LN for 6/8 tasks and wins both methos by more than 1% for both accuracy and chance-normalized accuracy.

Overall, the result shows that the improvement in validation loss actually leads to downstream performance improvement compared to Muon for both methods, and TensorChain and Aurora-LN is comparable for 0.6B and TensorChain is actually better for 1.7B.

Table 6: Downstream accuracy (%) over 8 tasks. Best per task is bold for each model size.

SizeOptimizerARC-CARC-EHSwagLMB.MMLUOBQAPIQAWino.Avg
0.6BMuon27.1350.3450.5448.0724.5332.6071.7652.8844.73
Aurora-LN26.8854.2150.7748.9626.5733.2071.3854.6245.82
TensorChain27.4753.3251.2950.1126.2932.8071.7153.8345.85
1.7BMuon36.1865.9167.2059.2725.3435.8076.9961.0153.46
Aurora-LN35.4963.8067.0660.9425.0238.6077.5362.6753.89
TensorChain37.2967.3467.6761.2826.4438.4077.2662.7554.80

Table 7: Chance-normalized downstream accuracy (%) for 8 tasks. Best per task is bold for each model size.

SizeOptimizerARC-CARC-EHSwagLMB.MMLUOBQAPIQAWino.NormAvg
0.6BMuon2.8433.7834.0548.070.0010.1343.535.7622.27
Aurora-LN2.5038.9534.3648.962.0910.9342.769.2323.72
TensorChain3.3037.7735.0650.111.7110.4043.427.6623.68
1.7BMuon14.9054.5556.2659.270.4514.4053.9722.0234.48
Aurora-LN13.9951.7456.0860.940.0218.1355.0625.3435.16
TensorChain16.3856.4556.8961.281.9217.8754.5225.4936.35

5. Multiseed Experiment

Three reviewers (reviewers Magh, NWCK, V2bG) suggested that repeated experiments would be necessary for a more reliable experimental result and comparison between the algorithms. We would like to clarify our conclusion is not that TensorChain is better than Aurora-LN for all cases; in 0.6B there is a clear benefit but for 1.7B the token efficiency is comparable (specifically for 4M batch case) with difference ≈\approx seed noise. The comparison was made between methods with the same NewtonSchulz iterations, where we see improvement of TensorChain compared to MuonEq or Muon where the token efficiency improvmenet is larger than 5%, and Aurora-LN improves over Aurora with ≈\approx 10%, and the averaged token efficiency across all experiments.

Due to timing and compute constraints, we could only do multiseed experiment for 0.6B model training with 1M batch size and 1x chinchilla. We used seeds 42,43,44 for both WSD and cosine learning rate scheduling, for six algorithms, Muon, Muon$^2$, MuonEq, Aurora, Aurora-LN and TensorChain. The seed variance is about ≈0.001\approx 0.001 and the experimental conclusion that TensorChain improves over existing methods does not change for 0.6B. The seed noise scale also suggests that for 1.7B methods the conclusion of comparison with same-NewtonSchulz methods most likely would not change.

[2] Dewulf, Alec, et al. "Aurora: A Leverage-Aware Spectral Optimizer." arXiv preprint arXiv:2606.27715 (2026).

AuthorsOpenReview ↗

[Common Response to Reviewers]

Table 2 shows that for k=1k = 1, the overhead is less than 1% (except for the MoE case with A100 which is 1.9%) and even for k=5k = 5 the overhead is less than 2% (except for the MoE case with A100 which is 3.1%), whereas for Aurora-LN the overhead can go up to 6%. Nevertheless, when the model size grows or the batch size grows, we can see that the overhead tends to decrease; this is because the bottleneck then becomes the forward/backward pass of optimization rather than the computation or communication done in the optimizer step. Hence, Table 2 shows that the overhead of these methods are minimal when optimized and will further diminish for larger scale experiments.

3. Generalization of TensorChain to different architecture and model size

Reviewer NWCK suggested larger scale experiments and reviewer V2bG suggested experiments on different architecture rather than Qwen3 dense model family. We demonstrate how TensorChain can benefit for larger models and different architecture by three additional experiments: Qwen3 MoE experiments for 3B/A1.1B model and 30B/A3B model, and Llama1B model pretraining experiment. We compared Muon with two most competitive baselines, Aurora-LN and TensorChain. For 3B/A1.1B and Llama1B we used the same protocol, first tuning the learning rate of Muon and transferring the learning rate to other methods. The batch size used was 1M. For MoE 30B/A3B model we fixed the learning rate as 9e−39e-3 for all experiments and used batch size of 2M. The models were trained with 8-way FSDP and 8-way DP on 8 nodes each equipped with 8 H200s. For MoE models qkvo weights were grouped with each weight type, experts were grouped only for those that were in the same GPU, and headwise Muon was applied following the Kimi team [1]. As we cannot share figures in the rebuttal we share step versus validation loss table for three methods, and we will add the training curves in the revised manuscript.

The MoE 3B/A1.1B results are shown in Table 3 and TensorChain improves over Aurora-LN by 10.7% and Muon by 11.8% in token efficiency and 13.4% and 11.0% in wall-clock time respectively, for validation loss.

The Llama 1B results are shown in Table 4; here TensorChain and Aurora-LN ties in token efficiency but due to throughput differences TensorChain is 3.4% faster than Aurora-LN, and for Muon there is 7.5% token saving and 7.1% wall-clock time saving.

At last for large scale experiments, the MoE 30B/A3B results are shown in Table 5. Here TensorChain improves over Aurora-LN by 11% and Muon by 16% in token efficiency and 16% and 15.3% in wall-clock time respectively, for validation loss. With the MoE results and the Llama1B result we can see that the result holds outside the Qwen3 dense model family and for larger models.

Table 3: MoE 3B-A1B validation loss. Lowest loss at each step in bold.

Step1k2k3k4k5k6k7k8k9kFinal (10k)
Tokens1.0B2.1B3.1B4.2B5.2B6.3B7.3B8.4B9.4B10.5B
Muon3.50913.26753.18173.12273.06192.99582.92172.85062.79982.7862
Aurora-LN3.50803.26413.17973.11843.05792.99182.91772.84642.79562.7821
TensorChain3.49613.24883.16273.10323.04342.97752.90182.83042.77942.7658

Table 4: Llama-3 1B validation loss. Lowest loss at each step in bold.

Step2k4k6k8k10k12k14k16k18kFinal (18.6k)
Tokens2.1B4.2B6.3B8.4B10.5B12.6B14.7B16.8B18.9B19.5B
Muon3.35403.16823.10643.04922.98952.91922.84162.77082.73792.7370
Aurora-LN3.33713.15693.09583.03962.98082.91092.83352.76282.73002.7291
TensorChain3.34463.15983.09643.03942.98082.91072.83352.76282.73012.7292

Table 5: Qwen3-30B-A3B validation loss. Lowest loss at each step in bold.

Step1k2k3k4k5k6k7k8k9kFinal (9.5k)
Tokens2.1B4.2B6.3B8.4B10.5B12.6B14.7B16.8B18.9B19.9B
Muon3.20002.96162.87872.81572.75512.68102.60672.53992.50512.5028
Aurora-LN3.19032.95642.87072.80462.73752.66452.58692.51782.48182.4793
TensorChain3.18032.94252.85652.79022.72232.64922.57132.50162.46562.4633

[1] Team, Kimi, et al. "Kimi k3: Open frontier intelligence." arXiv preprint arXiv:2607.24653 (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.