acceptodds
Under review as a conference paper at ICLR 2027

SONU: Communication-Efficient Orthonormalized Updates via Sketching

Abstract

As Large Language Model (LLM) training scales across distributed systems, gradient synchronization can incur substantial communication overhead, particularly in bandwidth-constrained settings. Gradient compression reduces this cost by communicating compact approximations of the gradient. Existing approaches, including sparsification and quantization, are typically designed to preserve the gradient under entrywise or Euclidean reconstruction criteria. These criteria, despite effective in coordinate-wise preconditioning optimizers, need not preserve the information used by matrix-orthonormalized optimizers such as Muon, whose update depends on the gradient’s spectral geometry rather than its individual entries. Such discrepancy makes preserving the geometry an ideal target of the gradient compressor. We propose Sketched Orthonormalized Updates (SONU), a communication-efficient optimizer built on two-sided gradient sketching that facilitates synchronizing gradients in a low-dimensional space. SONU forms a generalized Nyström surrogate of the gradient from the sketches, which coincides with the dense gradient up to the sketched subspaces, and extracts the paired singular vectors without dense orthonormalization. To prevent a fixed complement from being persistently omitted, SONU maintains an evolving sketch, in which the sketching subspace is updated with the orthonormalized directions and are gradually transported across iterations. Empirically, on chain-of-thought, agentic function-calling, and instruction-following tasks, SONU attains lower perplexity per communicated byte than the other communication-efficient orthonormalized update algorithms. Furthermore, SONU matches the perplexity trajectory of Dion, an efficient Muon variant which synchronizes the full gradients before applying orthonormalized updates.

Primary area:
optimization
Keywords:
communication-efficient, orthonormalized updates

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_s5heOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SONU, a communication-efficient optimizer for distributed LLM training that approximates Muon-style orthonormalized updates using two-sided gradient sketching. SONU constructs a low-rank momentum surrogate through generalized Nyström approximation, extracting paired singular directions without dense orthonormalization. To improve optimization quality, it introduces evolving sketching subspaces, momentum alignment, and residual accumulation. Experiments on Llama-3.2-1B/3B across reasoning, function-calling, and instruction-following tasks demonstrate competitive perplexity with substantially reduced gradient communication, achieving performance comparable to Dion under the evaluated setting.

Critical strengths

SONU provides an interesting approach to gradient compression tailored to orthonormalized optimizers. By leveraging two-sided sketching and generalized Nyström approximation, the method constructs low-rank orthonormalized updates without requiring dense gradient communication or storage on workers. The LoRA-based implementation further enables efficient gradient sketch computation. This approach is practically relevant for distributed LLM training under communication and worker-memory constraints.

SONU demonstrates competitive perplexity across multiple tasks and model scales while substantially reducing communication volume. Particularly, it achieves convergence trajectories comparable to Dion with server-held momentum, despite communicating only low-dimensional gradient sketches. The ablation studies also demonstrate the contributions of evolving sketches, momentum alignment, and progressive orthonormalization, providing empirical support for the proposed design.

Critical weaknesses
  • The paper evaluates reasoning, function calling, and instruction following solely through validation perplexity, without directly measuring reasoning accuracy, function-calling success, or instruction-following quality. Consequently, it is unclear whether the reported perplexity improvements translate into meaningful downstream performance.

  • Despite communication efficiency being the primary motivation, the paper does not report wall-clock training time, communication latency, or peak memory consumption. The additional overhead introduced by sketch reconstruction, orthonormalization, and momentum alignment may offset some communication savings.

  • Dion already supports compressed synchronization, as acknowledged in Appendix D.1. However, the paper primarily compares SONU against a dense-gradient synchronization configuration of Dion. A direct comparison including compressed Dion, accounting for both communication and memory costs, is necessary to establish SONU's practical advantage.

  • Experiments are restricted to Llama-3.2-1B/3B with a fixed sketch dimension of 16. It remains unclear whether such low-dimensional approximations maintain optimization quality as model size and matrix dimensions increase, or whether higher sketch ranks become necessary, reducing the communication advantage.

  • Although the authors acknowledge approximate momentum transport, the two separately maintained momentum sketches are not guaranteed to remain consistent with a common underlying momentum matrix. The paper does not characterize this discrepancy or its potential effect on optimization.

  • Although SONU constructs paired singular directions from its low-rank surrogate, the paper does not directly evaluate how closely these directions approximate those of the original dense momentum matrix. Quantitative comparisons of update-direction alignment or singular-subspace similarity against full Muon and alternative compressors would help substantiate the proposed spectral-preservation motivation.

Questions for authors
  • Could the authors evaluate task-specific metrics, such as reasoning accuracy and function-calling success rate, beyond validation perplexity?

  • Could the authors report wall-clock training time, communication latency, and memory usage to demonstrate practical efficiency gains?

  • How does SONU compare with Dion's compressed synchronization in terms of communication, memory, and training efficiency?

  • Does SONU maintain its performance on larger models and higher-dimensional weight matrices? How does the sketch rank affect the performance–communication tradeoff?

  • How does evolving the sketching subspaces affect momentum consistency and approximation error? Could the authors quantify its impact on convergence?

  • How closely do SONU's reconstructed update directions align with those obtained from full Muon? Could the authors provide quantitative comparisons?

Reviewer_FqyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets communication in data-parallel training with Muon-type optimizers. Each worker sends two gradient sketches, GPRGP^R and G⊤PLG^\top P^L, obtained in one backward pass as LoRA factor gradients. The server forms a generalized Nyström surrogate (Eq. 2), reads its paired singular vectors off a d×dd \times d core, and broadcasts U,VU, V for a rank-dd step. The sketches move along UU or VV in alternating phases, and projected momentum is transported between sketch bases with a dense server-side residual (Eqs. 10 to 12). There is no convergence analysis. On attention-only fine-tuning of Llama-3.2-1B and 3B with 64 workers and d=16d = 16, SONU reaches the lowest or tied-lowest validation perplexity per communicated byte on CoT, function-calling and instruction-following data, and it tracks dense-uplink Dion per step.

Critical strengths
  1. The two-sided sketch with a generalized Nyström core yields paired left and right directions from a single backward pass, with d(I+J)d(I+J) numbers per matrix in each direction and no dense gradient on workers. The LoRA realization with base-weight compensation (Appendix C, Eqs. 19 to 28) keeps the effective weights exactly on the dense SONU trajectory. The method is cheap to add to existing LoRA code, and the 2.03% and 1.30% ratios in Sec. 3.1 match this accounting.

  2. The ablation chain in Fig. 1c, 2c and 3c (with Fig. 5b to 7b at 1B) adds one component at a time. The gap between fixed and evolving sketches is large and shows up on all three tasks at both scales. This supports the claim that a fixed sketch keeps missing the same complement. Fig. 4c also shows that progressive orthonormalization is what keeps large α\alpha from diverging, which justifies that extra step.

Critical weaknesses
  1. The headline claim (Abstract, Sec. 3.2, p. 9 lines 432 to 433) is that SONU matches Dion with far less communication. The Dion run is the dense-uplink variant with server-held momentum, and it appears only in step-wise plots. Appendix D.1 states that Dion's compressed synchronization (Ahn et al., 2025, Sec. 3.3) exchanges only projected factors. As I understand Ahn et al., that mode gives the same iterates as dense-uplink Dion at a per-step cost of the same order as SONU's. What SONU then saves is one dense momentum buffer and the dense gradient on each worker, and no memory, step-time or wall-clock number is reported for any method. Please add compressed Dion to the bytes plots in Figs. 1 to 3 and report per-worker peak memory and time per step.

  2. The evidence for lower perplexity per byte is thin. Every result is validation perplexity, apparently from one run per method (three seeds exist only for SONU in Fig. 9), on runs of roughly 800 to 1,200 steps that update only the attention projections at d=16d = 16. Several gaps are in the second or third decimal (Fig. 2a, 2b, 3b), where PoLoRA, iMuon or EF21-Muon sit within the plotted noise of SONU. SONU also searched α\alpha, SS and γ\gamma (Table 2) on top of the learning-rate grid given to the baselines (Tables 1 and 4), and Dion's learning rate is not reported. Seeds for all methods, one downstream metric per task, and a sweep over dd would make the comparison convincing.

Questions for authors

See weakness.

Reviewer_ToJUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SONU, a communication-efficient optimizer for LoRA fine-tuning that updates dense weights using only low-dimensional, two-sided gradient sketches, obtained cheaply as LoRA factor gradients. The authors' experiments fine-tune 1B and 3B Llama models across 3 SFT datasets, showing that SONU achieves strong performance relative to the optimizers considered.

Critical strengths
  • Applying sketching to reduce the communication costs of optimizers like Muon is an interesting direction
  • The authors compare their method on relevant fine-tuning tasks
Critical weaknesses
  • While the method is applicable to pre-training, the authors only show fine-tuning experiments. Does the method still work there? Communication efficiency is more important in pre-training.
  • The authors state

SONU attains lower perplexity per communicated byte than the other communication-efficient orthonormalized update algorithms

However, no comparison to DiLoCo is made in the paper. Given the focus on "communication-efficient orthonormalized update algorithms" it seems like the authors should at least compare with [1,2], both of which leverage orthonormalized updates in communication-efficient settings.

  • The comparison to Dion only evaluates a rank-16 Dion baseline. It would be worth exploring more ranks to get a full comparison between the methods.
  1. MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo
  2. Can Model Merging Improve Aggregation in DiLoCo?
Questions for authors
  • Does SONU work for pre-training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.