acceptodds
Under review as a conference paper at ICLR 2027

SPARQ: A Subspace-Preserving Mixed Precision LMO Optimizer

Abstract

Optimizer-state memory remains a major bottleneck in large language model training. Existing low-bit state-compression methods do not explicitly account for the geometric sensitivity of linear-minimization-oracle (LMO) updates. This limitation is particularly important for Scion, whose spectral updates depend on the singular subspaces of persistent exponential-moving-average (EMA) states. We study Subspace-Preserving Adaptive Representation Quantization (SPARQ), a Scion-oriented mixed-precision state-compression method. SPARQ stores a more accurate low-rank component of each matrix-shaped EMA state and compresses the residual more aggressively, while applying the spectral LMO to matrix-shaped parameters and the root-mean-square (RMS) LMO to one-dimensional parameters. Our analysis shows that bounded reconstruction error preserves the decaying terms of the full-precision convergence bound while introducing an explicit error floor. Experiments with models up to 3B parameters show that SPARQ remains close to full-precision Scion, improves over uniform 4-bit integer (INT4) state storage, and reduces Scion optimizer-state memory by up to . By reducing optimizer-state memory while preserving training quality, SPARQ can facilitate the training of larger models under memory-constrained settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
low-precision training, training efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KhjqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-bit optimizer-state storage for Scion. SPARQ applies the scaled spectral LMO to all matrix-shaped parameters and the RMS LMO to one-dimensional parameters, whose EMA states remain in FP32. For matrix-shaped EMA states, it adopts the warm-started low-rank-plus-residual representation of MuonGRASP, storing the two low-rank factors in INT8 and the dense residual in INT4. The authors report that SPARQ remains close to full-precision Scion with the same routing, improves over uniform INT4 state storage.

Critical strengths

1.Systematic theoretical analysis: The paper analyzes uniform spectral routing, EMA storage drift, convergence with constant momentum, and spectral-LMO sensitivity, providing conditional theoretical support with explicit assumptions and limitations. 2.Controlled comparisons and memory savings: Controlled comparisons under the same routing and ablations support the method’s effectiveness. The authors report training performance close to full-precision Scion, improvements over uniform INT4 storage, and up to 6.9× optimizer-state compression.

Critical weaknesses

1.Methodological contributions could be clarified: The low-rank-plus-residual representation, warm start, and INT8/INT4 allocation build on MuonGRASP, while the optimizer framework draws from Scion. The main contributions appear to lie in broader application and error analysis. A clearer discussion of how the proposed method extends these foundations would help highlight its methodological contributions. 2.Missing direct comparison of routing choices: Scion’s original GPT configuration uses Sign for the input and output layers and Spectral for the hidden layers. The paper replaces this with uniform matrix spectral routing, which is also used by its baselines. The experiments therefore do not isolate the effect of routing changes on training quality, and conditional convergence guarantees do not establish equivalent performance.

Questions for authors

1.Comparison with the original Scion routing: In one representative setting, please compare the original Scion routing with uniform spectral routing using full-precision storage, uniform INT4 storage, and the proposed compression scheme, under matched training configurations and comparable tuning budgets. Please also add full-precision Scion results with the same routing as SPARQ to Table 1 to distinguish routing effects from compression effects. 2.Clarify memory reporting and measure peak GPU memory: The two descriptions of Figure 9 on page 29 report conflicting state-memory values for the same configurations. Appendix D.2 specifies LLaMA 124M but reports 1.98 GB and 0.53 GB for full-precision and compressed states, respectively, compared with 465 MB and 67 MB in the main text. Please reconcile and correct them. After clarifying the accounting, please report measured peak GPU memory over complete training iterations under matched configurations. This would substantiate the claim that SPARQ enables larger models under a fixed GPU memory budget.

Reviewer_jNWxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SPARQ compresses Scion’s momentum states using higher-precision low-rank factors and a more aggressively quantized residual. It aims to preserve spectral update directions while reducing storage, with experiments up to 3B parameters and an analysis of accumulated quantization error.

Critical strengths
  1. The paper identifies a relevant issue: small momentum reconstruction errors can become much larger errors after the spectral update.
  2. Controlled Scion comparisons keep the update rule and routing fixed, helping isolate the effect of state compression.
Critical weaknesses
  1. The advance beyond MuonGRASP is limited. The paper explicitly transfers its compression construction to Scion. The convergence bounds depend on a generic reconstruction-error bound and do not establish why the proposed subspace representation is better than alternative compressors.
  2. The evidence for preserving quality and practical efficiency is incomplete. Table 1 omits full-precision Scion, the central reference for the quality-preservation claim. Although small-model step times are provided, peak-memory and runtime measurements for the main larger-model experiments are missing
Questions for authors

See weaknesses.

Reviewer_4cJqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the optimizer state memory bottleneck of LMO-based methods in LLM training. Inspired by the existing work Muon-GRASP [1], this paper extends the mixed-precision and subspace-preserving compression technique to the Scion optimizer, which is another existing work [2]. Furthermore, this paper provides a preliminary theoretical study of the convergence rate by considering the reconstruction error produced by low-precision state storage. Numerical experiments are conducted to validate the proposed algorithm, indicating that state memory can indeed be reduced while training performance is not affected much compared to the baseline.

[1] Wu, H., Li, B., Yang, Y., Tu, Y., Zhou, Z., Chen, J., & Yan, J. (2026). Achieving low-bit Muon through subspace preservation and grid quantization. In The Fourteenth International Conference on Learning Representations.

[2] Pethick, T., Xie, W., Antonakopoulos, K., Zhu, Z., Silveti-Falls, A., & Cevher, V. (2025). Training deep learning models with norm-constrained LMOs. In Proceedings of the 42nd International Conference on Machine Learning (Vol. 267, pp. 49069–49104). Proceedings of Machine Learning Research.

Critical strengths

This paper shows that the existing mixed-precision compression technique used in Muon-GRASP can be generalized to the Scion method. It also provides a convergence analysis that considers the error introduced by low-precision state storage under a relatively strong bounded-error assumption.

Critical weaknesses
  1. The novelty of this paper is limited.

The motivation of this paper, i.e., the memory bottleneck, is already addressed in [1]. SPARQ appears to directly reuse the mixed-precision low-rank decomposition, warm-started subspace iteration, and 2D grid quantization proposed in [1], without introducing a new compression technique. The main difference between SPARQ and [1] is that SPARQ applies the same technique to a different optimizer, Scion, which itself is an existing method [2]. The paper does not explain why this extension introduces any new technical challenge.

  1. The theoretical analysis technique is classical and provides limited new insight.

For the theoretical part, the analysis follows a standard framework for stochastic first-order methods on a Lipschitz-smooth and nonconvex objective function. [2] has already provided the result without considering the bias error. The new work of this paper is the consideration of the bias error. However, the reconstruction error is directly assumed to have a bounded second moment (Assumption 4.2), which is a relatively strong assumption. The paper does not establish how this error bound follows from the proposed compression technique. More importantly, the analysis does not explain why subspace preservation should lead to a better convergence guarantee.

  1. The experimental results are not sufficiently convincing.

For the experimental part, the conducted experiments are very similar to those in [1]. The reported performance is also broadly comparable to Muon-GRASP. When comparing 4-bit Scion and 4-bit SPARQ (Table 1), the most noticeable improvement seems to be on LAMBADA for LLaMA 124M, where the accuracy increases from 19.4% to 30.6%. However, the improvements on most other tasks are relatively small or mixed. Notice that 4-bit Scion requires less memory. Therefore, part of the performance improvement may simply come from the additional memory budget rather than subspace preservation itself.

  1. The presentation of this paper needs substantial improvement.

Some theoretical statements are not sufficiently self-contained. For example, Proposition 4.4 only refers to "the preceding conditions" and the existing results in [2], without clearly specifying the required assumptions in the statement itself. On page 29, the discussion of Figure 9 is repeated before and after the figure, and the two descriptions even report different memory values.

Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.