- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents chained linear minimization oracles (LMOs) as a unified framework for understanding Muon variants that compose multiple normalization operations. It shows that such compositions need not remain LMOs over any fixed constraint set: some admit dimension-dependent convergence guarantees, while others can diverge even on smooth convex objectives. To demonstrate how chaining can nevertheless help, the paper establishes a one-step retrieval advantage over Muon and gradient descent in a linear associative-memory model with anisotropic answer embeddings.
The paper further introduces TensorChain, which stacks compatible momentum matrices across layers and applies normalization along the row, column, and layer axes before and after layerwise orthogonalization. Experiments on Qwen3-0.6B and Qwen3-1.7B cover multiple batch sizes and learning-rate schedules, with training budgets up to 112B tokens. TensorChain consistently improves validation loss over Muon and achieves average token savings of 9.6% across the four cosine-schedule settings.
Critical strengths
-
The framework provides a useful perspective on existing optimizers, supported by substantive theoretical results.
-
The associative-memory analysis establishes a concrete setting where chaining improves over Muon.
-
Experiments cover two model scales, multiple training configurations, and extensive learning-rate sweeps, showing consistent improvements over Muon.
Critical weaknesses
-
The grouping ablations do not isolate the contribution of layer-axis normalization from the other normalization operations.
-
Single-seed experiments make it difficult to assess small performance differences between the strongest methods.
-
How does TensorChain compare with Muon in wall-clock time to a target loss and GPU memory?
-
Do the validation-loss improvements translate into consistent gains on downstream evaluation benchmarks?
-
How well does TensorChain generalize beyond Qwen3, particularly to other dense Transformer and MoE architectures?
Questions for authors
See Weaknesses above.
2 comments
Thanks for the rebuttal; my main concerns are resolved. I will raise to 3.
We appreciate the constructive feedback and have done our best to address the weaknesses you pointed out, as detailed below.
Response to Weakness 1: In Table 4 of Section 5.2 we had an ablation of TensorChain's performance across different grouping strategies, but we agree that it does not isolate the cross-layer normalization itself. To address your concern, we isolate the effect of cross-layer normalization in [common response 1. Isolating the effect of cross-layer normalization and ablation on iteration number k]. We find that the iteration count is not crucial for final validation loss, but using cross-layer normalization consistently outperforms the version without it.
Response to Weakness 2: We address this point in [common response 5. Multiseed experiment]. Due to timing and compute constraints we could only perform repeated experiments with three seeds (42, 43, 44) for Qwen3 0.6B model, with 1M batch size and 1x chinchilla. The seed noise was of scale ≈ 0.001 and we believe the conclusions on the 0.6B model, ranking across methods with same number of NewtonSchulz operations, and average token efficiency improvement would not change.
Response to Weakness 3: We address this point in [common response 2. Practical cost of tensorchain.] We optimize Muon, Aurora-LN and TensorChain and in 8-way DP 8-way FSDP setup, the overhead of TensorChain is <2% for most cases (except for MoE 3B/A1.1B throughput for A100s), hence there is overall wall-clock time improvement in addition to token efficiency improvement.
Response to Weakness 4: We address the lack of downstream task evaluation in [common response 4. Downstream Evaluation]. We compare three optimizers, Muon, Aurora-LN and TensorChain, and both Aurora-LN and TensorChain improves over Muon by >1% on average for 8 downstream tasks, whereas Aurora-LN and TensorChain have different ranking for 0.6B and 1.7B across different metrics.
Response to Weakness 5: We show larger scale/more diverse architecture experiments in [common response 3. Generalization of TensorChain to different architecture and model size]. We experiment with Llama 1B, Qwen3 3B/A1.1B and Qwen3 30B/A3B models, and show that TensorChain consistently improves over both Muon and Aurora-LN for both token efficiency and overall wall-clock time improvement to reach same validation loss. These results suggest TensorChain generalizes beyond the Qwen3 dense model family.