- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SONU, a communication-efficient optimizer for distributed LLM training that approximates Muon-style orthonormalized updates using two-sided gradient sketching. SONU constructs a low-rank momentum surrogate through generalized Nyström approximation, extracting paired singular directions without dense orthonormalization. To improve optimization quality, it introduces evolving sketching subspaces, momentum alignment, and residual accumulation. Experiments on Llama-3.2-1B/3B across reasoning, function-calling, and instruction-following tasks demonstrate competitive perplexity with substantially reduced gradient communication, achieving performance comparable to Dion under the evaluated setting.
Critical strengths
SONU provides an interesting approach to gradient compression tailored to orthonormalized optimizers. By leveraging two-sided sketching and generalized Nyström approximation, the method constructs low-rank orthonormalized updates without requiring dense gradient communication or storage on workers. The LoRA-based implementation further enables efficient gradient sketch computation. This approach is practically relevant for distributed LLM training under communication and worker-memory constraints.
SONU demonstrates competitive perplexity across multiple tasks and model scales while substantially reducing communication volume. Particularly, it achieves convergence trajectories comparable to Dion with server-held momentum, despite communicating only low-dimensional gradient sketches. The ablation studies also demonstrate the contributions of evolving sketches, momentum alignment, and progressive orthonormalization, providing empirical support for the proposed design.
Critical weaknesses
-
The paper evaluates reasoning, function calling, and instruction following solely through validation perplexity, without directly measuring reasoning accuracy, function-calling success, or instruction-following quality. Consequently, it is unclear whether the reported perplexity improvements translate into meaningful downstream performance.
-
Despite communication efficiency being the primary motivation, the paper does not report wall-clock training time, communication latency, or peak memory consumption. The additional overhead introduced by sketch reconstruction, orthonormalization, and momentum alignment may offset some communication savings.
-
Dion already supports compressed synchronization, as acknowledged in Appendix D.1. However, the paper primarily compares SONU against a dense-gradient synchronization configuration of Dion. A direct comparison including compressed Dion, accounting for both communication and memory costs, is necessary to establish SONU's practical advantage.
-
Experiments are restricted to Llama-3.2-1B/3B with a fixed sketch dimension of 16. It remains unclear whether such low-dimensional approximations maintain optimization quality as model size and matrix dimensions increase, or whether higher sketch ranks become necessary, reducing the communication advantage.
-
Although the authors acknowledge approximate momentum transport, the two separately maintained momentum sketches are not guaranteed to remain consistent with a common underlying momentum matrix. The paper does not characterize this discrepancy or its potential effect on optimization.
-
Although SONU constructs paired singular directions from its low-rank surrogate, the paper does not directly evaluate how closely these directions approximate those of the original dense momentum matrix. Quantitative comparisons of update-direction alignment or singular-subspace similarity against full Muon and alternative compressors would help substantiate the proposed spectral-preservation motivation.
Questions for authors
-
Could the authors evaluate task-specific metrics, such as reasoning accuracy and function-calling success rate, beyond validation perplexity?
-
Could the authors report wall-clock training time, communication latency, and memory usage to demonstrate practical efficiency gains?
-
How does SONU compare with Dion's compressed synchronization in terms of communication, memory, and training efficiency?
-
Does SONU maintain its performance on larger models and higher-dimensional weight matrices? How does the sketch rank affect the performance–communication tradeoff?
-
How does evolving the sketching subspaces affect momentum consistency and approximation error? Could the authors quantify its impact on convergence?
-
How closely do SONU's reconstructed update directions align with those obtained from full Muon? Could the authors provide quantitative comparisons?