- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies low-bit optimizer-state storage for Scion. SPARQ applies the scaled spectral LMO to all matrix-shaped parameters and the RMS LMO to one-dimensional parameters, whose EMA states remain in FP32. For matrix-shaped EMA states, it adopts the warm-started low-rank-plus-residual representation of MuonGRASP, storing the two low-rank factors in INT8 and the dense residual in INT4. The authors report that SPARQ remains close to full-precision Scion with the same routing, improves over uniform INT4 state storage.
Critical strengths
1.Systematic theoretical analysis: The paper analyzes uniform spectral routing, EMA storage drift, convergence with constant momentum, and spectral-LMO sensitivity, providing conditional theoretical support with explicit assumptions and limitations. 2.Controlled comparisons and memory savings: Controlled comparisons under the same routing and ablations support the method’s effectiveness. The authors report training performance close to full-precision Scion, improvements over uniform INT4 storage, and up to 6.9× optimizer-state compression.
Critical weaknesses
1.Methodological contributions could be clarified: The low-rank-plus-residual representation, warm start, and INT8/INT4 allocation build on MuonGRASP, while the optimizer framework draws from Scion. The main contributions appear to lie in broader application and error analysis. A clearer discussion of how the proposed method extends these foundations would help highlight its methodological contributions. 2.Missing direct comparison of routing choices: Scion’s original GPT configuration uses Sign for the input and output layers and Spectral for the hidden layers. The paper replaces this with uniform matrix spectral routing, which is also used by its baselines. The experiments therefore do not isolate the effect of routing changes on training quality, and conditional convergence guarantees do not establish equivalent performance.
Questions for authors
1.Comparison with the original Scion routing: In one representative setting, please compare the original Scion routing with uniform spectral routing using full-precision storage, uniform INT4 storage, and the proposed compression scheme, under matched training configurations and comparable tuning budgets. Please also add full-precision Scion results with the same routing as SPARQ to Table 1 to distinguish routing effects from compression effects. 2.Clarify memory reporting and measure peak GPU memory: The two descriptions of Figure 9 on page 29 report conflicting state-memory values for the same configurations. Appendix D.2 specifies LLaMA 124M but reports 1.98 GB and 0.53 GB for full-precision and compressed states, respectively, compared with 465 MB and 67 MB in the main text. Please reconcile and correct them. After clarifying the accounting, please report measured peak GPU memory over complete training iterations under matched configurations. This would substantiate the claim that SPARQ enables larger models under a fixed GPU memory budget.