acceptodds
Under review as a conference paper at ICLR 2027

Dobb: Shard-Local Preconditioning in Block-Diagonal Eigenbases

Abstract

Training language models whose parameters and optimizer state exceed a single node's memory requires sharding these tensors across devices. Under tensor parallelism (TP), matrix-aware optimizers such as Muon introduce communication to perform global matrix operations across shards. We propose Dobb (Distributed Optimizer with Block-diagonal Bases), a matrix-aware optimizer designed to compute updates locally within parameter shards. Dobb approximates the input-activation second moment with diagonal blocks and tracks a complete basis within each block. It projects momentum into these bases, applies sign normalization to the coefficients, and reconstructs the update, retaining every direction within each block. When each block lies within one shard under TP or fully sharded data parallelism (FSDP), Dobb applies its matrix update locally, requiring no communication beyond data-parallel synchronization of the activation second moments. We benchmark Dobb against Dion, a distributed matrix-aware optimizer designed to retain the benefits of orthonormalized updates while reducing communication. On GPT-2 models with 124M, 354M, and 772M parameters, Dobb with 64 blocks achieves validation losses of 3.037, 2.742, and 2.600, respectively, compared with 3.265, 2.879, and 2.696 for Dion at rank fraction 1/64. In a Qwen3-32B timing benchmark with FSDP shard degree 32 and TP degree 2, Dobb reduces the amortized optimizer-call time from Dion's 1307.7 ms to 168.0 ms, a 7.8 speedup.

Primary area:
optimization
Keywords:
communication-efficient distributed training, matrix-aware optimizers, preconditioning, Muon, large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nYDZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors present a method to remove optimizer-specific communication over tensor-parallel ranks during training with matrix-aware optimization. They do this by projecting the momentum into a block-diagonal basis, built from the input-activation second moment. Each block lies within a single tensor-parallel shard, so the update can be computed locally. They motivate the selection of the basis and derive the model updates. Finally they validate their approach on GPT-2 against AdamW, Muon and Dion, and show the speedup relative to Dion on Qwen3 models.

Critical strengths

The authors present a simple approach to an important distributed training bottleneck - the additional communication required by matrix-aware optimizers under tensor parallelism. The paper is well written, and the motivation and implementation of the design are clear. The authors also provide a thorough ablation study in the appendix.

Critical weaknesses

My main concern is mostly limited evaluations and comparisons:

  • The speedup of the method is overstated in the abstract - practitioners mostly care about total iteration time.
  • While Dobb consistently beats adamw, the margin is very small (For example, for B=16). Can the authors rerun training with additional seeds, to add error approximations and show that the difference is significant? Furthermore, testing a different model architecture can further strengthen the accuracy results.
  • Can the authors run evaluations on downstream tasks, and not only validation loss? I understand that due to limited computational resources the models that are trained are small and will not be performant on complex tasks, but perhaps perplexity on wikitext or accuracy on easy tasks like hellaswag can be a valid test at this model size?
Questions for authors
  • Can the authors explain why normalizing all weight updates to have the same RMS constitutes a fair comparison? Can they supply evidence for this claim?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nYDZ

We thank the reviewer for the careful reading and the concrete suggestions.

Speedup and total iteration time. We agree that the abstract should report the effect on total iteration time. We removed the 7.8× optimizer-call claim. The revised abstract reports that, on GPT-2 XL with 32 H100 GPUs, Dobb reduces pooled iteration time from 1184.0 ms for Dion to 1069.1 ms, a reduction of 114.9 ms (9.7%). The corresponding optimizer-call times are 124.1 ms and 9.2 ms. Pooled iteration time is defined in Appendix G.1. Table 3 reports pooled iteration times across the main timing configurations; Dobb's is comparable with AdamW's and below Muon's and Dion's in every GPT-2 configuration (General Response, item 3 and continued).

Margin over AdamW: seeds and architecture. On GPT-2 Small the margin holds across seeds. At the reviewer's example of B=16B=16, Dobb reaches 3.028 ± 0.001 against 3.045 ± 0.002 for AdamW, a statistically significant difference (paired t-test over the three seeds, p=0.001p=0.001), and Dobb is below AdamW at every block count in every seed. We reran all eleven GPT-2 Small configurations with two additional seeds; Table 2 reports the means and standard deviations, and Table 10 lists each seed (General Response, item 1).

We also added GPT-2 XL (1.56B parameters), where Dobb at B=16B=16 reaches 2.476 against 2.497 for AdamW, a reduction of 0.021. Additional seeds at Large and XL and training on a different architecture were beyond our compute budget for the discussion period. Our quality evaluation therefore covers the GPT-2 family at four sizes, from 124M to 1.56B parameters.

Downstream tasks. WikiText perplexity agrees with the validation-loss comparison: Dobb's is lower than AdamW's at every block count. HellaSwag points the same way, but by only 0.3 to 0.5 points. We also evaluated ARC-Easy and PIQA. Appendix F.6 and Table 19 report zero-shot evaluation of the final GPT-2 Small checkpoints for all eleven configurations. Accuracies are in percent.

HellaSwag ↑ARC-Easy ↑PIQA ↑WikiText ppl ↓
AdamW29.239.660.043.2
Muon30.038.361.239.6
RMNP28.838.460.944.5
Dion ρ=1\rho=130.239.661.039.4
Dion ρ=1/4\rho=1/429.338.960.043.0
Dion ρ=1/16\rho=1/1628.537.058.445.5
Dion ρ=1/64\rho=1/6427.637.159.356.2
Dobb B=1B=129.738.260.940.2
Dobb B=4B=429.638.560.840.5
Dobb B=16B=1629.538.859.941.2
Dobb B=64B=6429.539.160.641.5

Dobb's WikiText perplexity rises from 40.2 to 41.5 as BB increases from 1 to 64, while HellaSwag accuracy remains between 29.5% and 29.7%. For Dion, perplexity rises from 39.4 to 56.2 and HellaSwag accuracy falls from 30.2% to 27.6% as the rank fraction decreases from 1 to 1/64. ARC-Easy and PIQA show no consistent ordering among the optimizers.

Fairness of the shared update scale. The optimizers share one scale formula, which gives low-rank Dion a smaller update RMS than the others. Neither raising its update RMS to 0.2 nor tuning its learning rate closes the gap: Dion at ρ=1/16\rho=1/16 and 1/641/64 stays above every Dobb entry on GPT-2 Small. Our submission described Dion's scale incorrectly as the rank-dependent τmn/r\tau\sqrt{mn/r}; Section 6 and Appendix E.1 now state the shared scale that the runs used (General Response, item 2).

The scale formula is 0.2max⁡(m,n)0.2\sqrt{\max(m,n)}. Muon, RMNP, Dobb, and full-rank Dion have update RMS values in the range 0.2 to 0.4, whereas low-rank Dion has update RMS 0.2ρ0.2\sqrt{\rho} (Table 11).

The comparison evaluates the methods under a common learning rate and weight decay transferred from AdamW, without optimizer-specific retuning. Muon achieves lower validation loss than AdamW at all four model sizes under this scheme. AdamW's selected rate is also close to its optimum: in our AdamW sweeps, a rate off by a factor of 2\sqrt{2} raises AdamW's final loss by about 0.002 (Appendix E.2), below every margin of Dobb over AdamW in Table 2 (0.006 or more).

We separately tested whether low-rank Dion's smaller update RMS explains its loss increase. Table 12 raises its update RMS to 0.2 using a rank-corrected scale while keeping the learning rate and weight decay unchanged. On GPT-2 Small at ρ=1/16\rho=1/16, this improves validation loss from 3.108 to 3.089, recovering 0.019 of the 0.106 gap to full-rank Dion (3.002).

We also tuned Dion's learning rate directly at ρ=1/16\rho=1/16 and 1/641/64 (Table 13). The best rates lower its loss from 3.108 to 3.071 and from 3.265 to 3.227; both remain above every Dobb entry on GPT-2 Small (at most 3.036), with Dobb not retuned.

Taken together, the margin over AdamW holds across seeds, WikiText perplexity agrees with validation loss, and Dion's higher loss persists when its update RMS is raised to 0.2 or its learning rate is tuned.

Thank you for your response to my concerns. I increased my soundness score and maintained my overall score. I think this is a solid paper and recommend its acceptance.

Dear Reviewer nYDZ,

Thank you for reviewing our response and for raising your soundness score! We appreciate your recommendation for acceptance and your helpful comments throughout the review process.

Best regards, The Authors of Submission 49151

Reviewer_hGCrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Dobb, a distributed matrix-aware optimization method that gets rid of optimizer-specific cross-TP communication under specific sharding conditions by utilizing block-diagonal input-activation second moment bases and sign normalization. Experiments on GPT-2 models indicate that Dobb reduces the validation loss deterioration under block partitioning as compared to Dion under corresponding rank reductions. Also, distributed tests on Qwen3-8B and Qwen3-32B revealed lower times per optimizer-call and iteration when working with Dobb as compared to Dion.

Critical strengths
  1. Communication-efficient optimizer design. The paper elaborates on an intelligent way of combining sign normalization with shard-aligned block-diagonal activation second-moment bases so that no optimizer-specific cross-TP communication is needed when each block lies within one shard. This is an insightful system and algorithm co-design that addresses an important problem in distributed LLM training.
  2. Robust quality under block partitioning, supported by thorough ablations. Under the shared update-RMS protocol, Dobb shows much lower validation loss deterioration under block partitioning (0.007–0.021 nats from B = 1 to 64) than Dion under corresponding rank reductions (0.128–0.263 nats) across all three GPT-2 scales (Table 3). The extensive ablations in Appendix E support several key design choices, such as the activation-side basis, sign normalization, and a single warm-started orthogonal iteration per refresh.
Critical weaknesses
  1. Lack of evidence for the advantages of Dobb’s design. The paper does not provide an identity-basis (Signum) control or a block-local Muon baseline. As a result, it becomes unclear whether Dobb’s eigenbasis design turns out to be really beneficial compared to alternatives that also require no cross-TP communication. Moreover, at B = 64, Dobb’s improvements over AdamW are small (0.006–0.019 nats) and were obtained with one seed only, against an estimated seed noise of σ≈0.003\sigma \approx 0.003 nats, while the comparison between Dobb’s design and Dion relies on applying the AdamW-selected learning rate and a rank-dependent RMS multiplier to low-rank Dion without any optimizer-specific tuning.
  2. Confounded distributed performance measuring. Dobb and Dion were analyzed in different cluster allocations, with large unexplained variations in the forward and backward times and incomplete hardware specifications. While the paper reports a 7.8× optimizer-call speedup on Qwen3-32B, the associated overall iteration speedup is only around 1.13×, because the optimizer call accounts for at most 7.1% of Dion’s iteration time; moreover, part of this 1.13× comes from unexplained forward/backward differences (943.5 ms of the 2118.2 ms total gap). As a result, it is difficult to attribute the magnitude of the observed performance improvements to Dobb.
Questions for authors

Q1. Contribution of the activation eigenbasis. The paper uses the activation eigenbasis throughout, but have the authors also assessed Dobb’s performance using an identity basis (Ub=I\mathbf{U}_b = \mathbf{I}, or equivalently B=nB = n, which reduces the update to the Signum-style τ⋅sign(Mt)\tau \cdot \mathrm{sign}(\mathbf{M}_t)) with the exact same training protocol as in Table 3? If so, how does the validation loss of the identity-basis variant on GPT-2 Small and Large compare to that of Dobb with B=64B = 64 blocks? This would indicate whether or not the activation eigenbasis yields any advantage over this simpler method, which also requires no cross-TP communication. Q2. Calibration of low-rank Dion. All optimizers use the AdamW-selected learning rate, and low-rank Dion updates are scaled by τmn/r\tau\sqrt{mn/r}. Was error feedback enabled in the Table 3 Dion runs, and does this multiplier match the scaling of Dion’s reference implementation? Does Dobb retain its advantage over low-rank Dion when Dion’s learning rate is optimized separately, specifically for the rank fraction ρ=1/16\rho = 1/16 and/or ρ=1/64\rho = 1/64? Determining this would assist in ascertaining whether there truly are different performance characteristics of these optimizers, or whether the differences arise from the hyperparameters selected for them.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hGCr (1/2): Evidence for Dobb's Design

We thank the reviewer for the detailed and specific review. We reply in two comments: this one covers the evidence for Dobb's design, and the next covers Dion's calibration and the timing measurements.

Activation eigenbasis against the identity basis (Signum). The eigenbasis lowers the loss most with few blocks, and its benefit shrinks as the block count grows, as it must: Signum is Dobb at B=nB=n, where each block is a single coordinate. With GPT-2 XL now our largest model (General Response, item 1), we trained this endpoint under the training protocol of the main table at both ends of the model range: GPT-2 Small (three seeds) and GPT-2 XL (Appendix F.7, Table 20). Final validation loss:

BBSmall (mean ± SD, three seeds)XL
13.017 ± 0.0022.461
43.024 ± 0.0012.471
163.028 ± 0.0012.476
643.036 ± 0.0022.482
nn (Signum)3.049 ± 0.0062.485

Dobb has lower loss than Signum at every block count on both sizes. Within each size, the eigenbasis lowers the loss by 0.032 and 0.024 with one block, and by 0.013 and 0.003 at B=64B=64, where a block spans 12 to 48 input coordinates on Small and 25 to 100 on XL, so Dobb is close to its B=nB=n limit by construction. Locality needs only one block per shard, so BB need not exceed the sharding degree, and the eigenbasis contributes most when blocks are wide. As the sharding degree grows, Dobb approaches Signum, which is also shard-local.

Signum also varies more across seeds: its standard deviation on Small is 0.006, against 0.001 to 0.002 for Dobb at every block count (Table 10). Sign descent is sensitive to gradient noise: the probability that the sign of a stochastic gradient is correct is determined by the gradient's relative variance (Balles & Hennig, 2018), and sign descent performs poorly at small batch sizes (Kunstner et al., 2023).

Block-local Muon. Dobb's loss increases less with the block count than that of block-local Muon, which applies Muon independently to each block of a weight matrix and uses the same block partition as Dobb. Its update scale accounts for the block layout (Appendix F.5). GPT-2 Small, three seeds:

BBDobbBlock-local Muon
13.017 ± 0.0023.006 ± 0.001 (Muon)
43.024 ± 0.0013.022 ± 0.001
163.028 ± 0.0013.035 ± 0.002
643.036 ± 0.0023.050 ± 0.002

Block-local Muon is 0.002 lower at B=4B=4, a difference that is not statistically significant over the three paired seeds (paired t-test, p=0.20p=0.20). Dobb is 0.007 and 0.014 lower at B=16B=16 and B=64B=64, and both differences are significant (p=0.010p=0.010 and p<0.001p<0.001). From B=1B=1 to B=64B=64, Dobb's loss rises by 0.019 and block-local Muon's by 0.044 (Appendix F.5, Table 18).

Improvement over AdamW at B=64B=64. On GPT-2 Small it holds in every seed: Dobb reaches 3.036 ± 0.002 against 3.045 ± 0.002 for AdamW over three seeds. On GPT-2 XL (1.56B parameters), which the submission did not include, Dobb at B=64B=64 reaches 2.482 against 2.497 for AdamW, 0.015 lower (General Response, item 1).

Taken together, among the shard-local methods we tested on GPT-2 Small (AdamW, Signum, and block-local Muon), Dobb has the lowest validation loss at B=16B=16 and B=64B=64, and its loss rises less than half as much as block-local Muon's as blocks are added (0.019 against 0.044).

AuthorsOpenReview ↗

Response to Reviewer hGCr (2/2): Dion Calibration and Timing

Error feedback. Yes, it was enabled in every Dion run of Table 2. Our implementation keeps the residual of the rank-rr approximation in the momentum, Mnew←M−βUW⊤M_\text{new}\leftarrow M-\beta UW^\top, the same step as Algorithm 2 of Ahn et al. (2025). Section 6 now states this.

Dion's update scale. Our Dion runs follow the reference. In their comparison across model sizes and rank fractions, Ahn et al. (2025) use a scale that depends on the matrix shape and not on the rank, and one learning rate for every rank fraction; they tune neither per rank. The runs of Table 2 do the same: they use the shared scale τmax⁡(m,n)\tau\sqrt{\max(m,n)}, which gives Dion an update RMS of 0.2ρ0.2\sqrt{\rho} (Table 11), and the shared learning rate. The τmn/r\tau\sqrt{mn/r} in our submission misdescribed them; Section 6 and Appendix E.1 are corrected (General Response, item 2).

We then tested the Dion baseline beyond the reference in two ways. The first matches its update RMS: the rank-corrected scale τmn/r\tau\sqrt{mn/r} holds Dion's update RMS at 0.2 at every rank, with the learning rate and weight decay unchanged (Table 12). On GPT-2 Small, Dobb has the lower loss at each corresponding block count under either scale. The second tunes Dion's learning rate, described next.

GPT-2 Small, validation loss (seed 5000)ρ=1/4\rho=1/4ρ=1/16\rho=1/16ρ=1/64\rho=1/64
Dion, shared scale (Table 2, RMS = 0.2ρ0.2\sqrt{\rho})3.0383.1083.265
Dion, rank-corrected scale (RMS = 0.2)3.0343.0893.267
Dobb at B=1/ρB=1/\rho3.0243.0263.037

Dion with its learning rate tuned. Dobb retains its advantage at both rank fractions the reviewer names. We swept Dion's learning rate on GPT-2 Small, keeping the shared scale and the weight decay. We swept upward from the shared rate because the shared scale gives low-rank Dion a smaller update RMS than the others (0.2ρ0.2\sqrt{\rho}); at both rank fractions, the loss is lowest inside the grid. Final validation loss (seed 5000), by multiple of the shared rate; bold marks the two values compared in each row, Dion at its best rate and Dobb:

ρ\rho1×1.41×2×2.83×4×5.66×8×Dobb at B=1/ρB=1/\rho
1/163.1083.0963.0833.0833.0713.0733.0863.026
1/643.2653.2463.2313.2273.232n/an/a3.037

The best rates lower Dion's loss to 3.071 and 3.227. Tuned Dion at ρ=1/16\rho=1/16 is 0.045 above Dobb at B=16B=16 (3.026) and 0.034 above Dobb at B=64B=64 (3.037). Tuned Dion at ρ=1/64\rho=1/64 is 0.190 above Dobb at B=64B=64. The sweep tunes Dion alone; Dobb is not retuned (Appendix E.1, Table 13).

Optimizer-call speedup and iteration time. We agree that the 7.8× optimizer-call speedup overstated the gain, and we removed it. For GPT-2 XL on 32 H100 GPUs, the abstract now reports the optimizer call (9.2 ms for Dobb against 124.1 ms for Dion) and the pooled iteration (1069.1 ms against 1184.0 ms).

Allocations and forward-backward variation. We agree. In the submission, each method's iteration time came from its own runs, so differences in forward-backward time between runs, which do not depend on the optimizer, entered the comparison. This is the gap the reviewer identifies. The revision reduces it with pooled iteration times (Appendix G.1; General Response, item 3).

What pooling does. For each configuration, we average the forward-backward time over all runs of that configuration, whatever the optimizer, and add each method's own optimizer-call time. Two methods then differ only by their optimizer calls.

Why it is justified. The forward-backward pass performs the same model computation for every optimizer, and it is the variable part of an iteration. In a 48-run study on GPT-2 Large, in three allocations with the same node geometry, its run-to-run SD is 15.3 to 26.9 ms, against 0.8 ms for Dobb's optimizer call and 10.6 and 10.3 ms for Muon's and Dion's. Dobb's second-moment capture also runs during forward-backward, and its forward-backward mean (7740 ms) still lies within the span of the baselines' means (7713 to 7788 ms).

What it solves. The comparison no longer depends on the allocation a method ran in. New benchmarks compare AdamW, Muon, Dion, and Dobb in one implementation, on GPT-2 Large and XL and on scaled-up GPT-2-style models up to 8B parameters; Dobb's iteration time is comparable with AdamW's and below Muon's and Dion's in every GPT-2 configuration (Table 3; General Response, continued).

Hardware specifications. Appendix G now gives them. The A5000 nodes have two RTX A5000 24 GB GPUs each, connected by PCIe, with 25 Gbps Ethernet between nodes. The H100 nodes have eight H100 80 GB GPUs each, connected by NVLink, with 400 Gbps Ethernet between nodes.

Taken together, Dion's scale in our runs matches the reference, Dobb keeps its advantage when Dion's learning rate is tuned, and the timing comparison no longer depends on forward-backward variation.

Dear Reviewer hGCr,

We appreciate the time you have spent on our paper. As the discussion period closes on November 18 (AoE), we would like to briefly summarize how we addressed your main concerns:

  • Identity-basis (Signum) control: Dobb has lower validation loss than Signum at every block count on both GPT-2 Small (three seeds) and GPT-2 XL, and Signum varies more across seeds (Appendix F.7, Table 20).
  • Block-local Muon baseline: With the same block partition, Dobb has significantly lower validation loss at 16 and 64 blocks (p = 0.010 and p < 0.001 over three paired seeds), and its loss rises less than half as much from 1 to 64 blocks (0.019 vs. 0.044; Appendix F.5).
  • Dion calibration: Error feedback was enabled in all Dion runs, and Dion’s update scale follows the reference implementation (Ahn et al., 2025). Even with its learning rate tuned, Dion’s validation loss remains higher than Dobb’s at the matching block count: 3.071 for Dion at rank fraction 1/16 vs. 3.026 for Dobb with 16 blocks, and 3.227 at 1/64 vs. 3.037 with 64 blocks (Table 13). Dobb was not retuned.
  • Timing confounds: We removed the 7.8× optimizer-call speedup claim from the abstract, which now reports total iteration time: 1069 ms for Dobb vs. 1184 ms for Dion on GPT-2 XL with 32 H100s. To address the run-to-run variation in forward-backward time you identified, we now average the forward-backward time over all runs of a configuration regardless of optimizer, so methods differ only in their optimizer-call time (Appendix G.1). Hardware and interconnect details are in Appendix G.

We would be grateful to know whether these responses resolve your concerns, and we are happy to provide further clarification if needed. If you find our responses satisfactory, we would appreciate it if you could consider updating your assessment.

Best regards, The Authors of Submission 49151

Reviewer_quYbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed Dobb, a matrix-aware optimizer for sharded training. It approximates the input-activation second moment with diagonal blocks, and keeps a complete eigenbasis per block. When each block lies within one TP shard, the optimizer step needs no cross-TP communication. Only the block second moments are synchronized across data-parallel ranks. In the benchmark against Dion, Dobb achieved significant lower validations losses on GPT-2, and a speedup for amortized optimizer-call time on Qwen with FSDP.

Critical strengths
  • A clean locality design to save communication in LLM training: Deriving the basis from the activation second moment and aligning blocks to shards gives zero optimizer-specific cross-TP communication with no periodic global step. This is a clean and novel design.
  • Careful ablations and reporting: The appendix comprehensively isolates each design choice, and states the limitations openly. This makes the results interpretable even when they are weak.
Critical weaknesses

The advantage over AdamW, the natural zero-communication baseline, is marginal. At B=64, Dobb beats AdamW by only 0.007, 0.019, and 0.006 nats on Small/Medium/Large (from Table 3), with a single seed. However, the paper reported an estimated seed noise of about 0.003 nats. It's difficult to tell whether the performance improvements are real, or just from noise.

Questions for authors
  • Can you report multi-seed results (at least 3 seeds) for AdamW, Dobb B=64, and Dion 1/4 at each scale? The current Dobb’s improvements over AdamW are close to the stated seed noise.
  • The learning rate was tuned for AdamW and transferred to all other optimizers (from Section 6). Were Dobb, Muon, and Dion given any learning-rate or momentum tuning? Should we further tune them to compare their best performances?
2 comments
AuthorsOpenReview ↗

Response to Reviewer quYb

We thank the reviewer for the careful reading and concrete requests. We added three-seed results for every GPT-2 Small configuration and for the three requested configurations on GPT-2 Medium, and clarify below which hyperparameters were selected for the main comparison.

Advantage over AdamW. On GPT-2 Small and Medium, under the shared scale of Table 2, Dobb at B=64B=64 has lower validation loss than AdamW, and than Dion at ρ=1/4\rho=1/4, in each of three seeds. We ran two additional seeds for all eleven GPT-2 Small configurations, including the three requested by the reviewer, and for these three configurations on GPT-2 Medium; mean validation loss and sample standard deviation:

OptimizerSmallMedium
AdamW3.045 ± 0.0022.756 ± 0.004
Dobb B=64B=643.036 ± 0.0022.742 ± 0.000
Dion ρ=1/4\rho=1/43.042 ± 0.0032.754 ± 0.001

On GPT-2 Small, Dobb is also below AdamW at every tested block count in every seed. Table 10 lists the individual results at both sizes (General Response, item 1).

Additional seeds at Large and XL were beyond the discussion-period compute budget. To test the advantage at the largest scale, we extended the quality comparison to GPT-2 XL (1.56B parameters): Dobb reaches 2.482 at B=64B=64 and 2.476 at B=16B=16, compared with 2.497 for AdamW, reductions of 0.015 and 0.021. Across the four model sizes, Dobb has lower validation loss than AdamW in all sixteen block-count comparisons in Table 2.

Tuning of the optimizers. Dobb, Muon, and Dion received no optimizer-specific learning-rate or momentum-coefficient tuning in the main comparison. We selected the learning rate using AdamW and transferred it to the other optimizers; momentum coefficients remained fixed. AdamW learning-rate sweeps cover four model sizes up to 354M parameters, with the rates for Large and XL determined from the fitted size-scaling law in Appendix E.2 (General Response, item 2).

We agree that comparing each optimizer's best tuned performance requires optimizer-specific sweeps. Running them for every configuration at every scale was beyond our compute budget.

Within that budget, three results address this question:

AdamW's learning rate. In our AdamW sweeps, a learning rate off by a factor of 2\sqrt{2} raises AdamW's final loss by about 0.002 (Appendix E.2), below every margin of Dobb over AdamW in Table 2 (0.006 or more).

Low-rank Dion's update scale and learning rate. Neither a rank-corrected scale, which raises Dion's update RMS to 0.2, nor a tuned learning rate brings Dion at ρ=1/16\rho=1/16 or 1/641/64 below Dobb at the corresponding block count (Tables 12 and 13). Dobb keeps the shared rate.

GPT-2 Small, validation loss (seed 5000)ρ=1/16\rho=1/16ρ=1/64\rho=1/64
Dion, shared scale (Table 2, RMS = 0.2ρ0.2\sqrt{\rho})3.1083.265
Dion, rank-corrected scale (RMS = 0.2)3.0893.267
Dion, best learning rate3.0713.227
Dobb at B=1/ρB=1/\rho3.0263.037

Dobb's second-moment decay. Dobb's own settings are not tuned either. Matching the per-capture decay to the nine optimizer steps between captures, 0.9590.95^{9} in place of 0.950.95, lowers its loss at B=1B=1 and raises it at no block count (Appendix F.1). Table 2 keeps the default decay.

GPT-2 Small, validation loss (seed 5000)B=1B=1B=4B=4B=16B=16B=64B=64
Dobb, decay 0.950.95 (Table 2 setting)3.0173.0243.0263.037
Dobb, decay 0.9590.95^{9}3.0093.0203.0263.035

These results show that tuning can improve either method: a tuned learning rate lowers low-rank Dion's loss to 3.071 and 3.227, and a matched decay lowers Dobb's by 0.008 at B=1B=1. With this limited tuning, Dobb remains ahead of low-rank Dion, although only Dion's learning rate was tuned. Tuning every hyperparameter of every optimizer is what our compute budget rules out, which is why Table 2 compares all optimizers untuned, under the settings transferred from AdamW.

Taken together, Dobb's advantage over AdamW on GPT-2 Small and Medium holds in every seed, and none of the three results changes its comparison with low-rank Dion.

Dear Reviewer quYb,

We appreciate the time you have spent on our paper. As the discussion period closes on November 18 (AoE), we would like to briefly summarize how we addressed your two main concerns:

  • Margin over AdamW vs. seed noise: We added three-seed results. Dobb with 64 blocks has lower validation loss than AdamW in every seed on both GPT-2 Small (3.036 ± 0.002 for Dobb vs. 3.045 ± 0.002 for AdamW, mean ± SD) and GPT-2 Medium (2.742 ± 0.000 vs. 2.756 ± 0.004). On the newly added GPT-2 XL (1.56B parameters, one seed), Dobb with 64 blocks reaches 2.482 vs. 2.497 for AdamW.
  • Optimizer-specific tuning: In the main comparison, the learning rate was tuned for AdamW and transferred unchanged to Dobb, Muon, and Dion. To test whether this disadvantaged the baselines, we swept low-rank Dion’s learning rate. Even with its learning rate tuned, Dion’s validation loss remains higher than Dobb’s at the matching block count: 3.071 for Dion at rank fraction 1/16 vs. 3.026 for Dobb with 16 blocks, and 3.227 at 1/64 vs. 3.037 with 64 blocks (Table 13). Dobb was not retuned. We also found that AdamW’s loss changes by only about 0.002 when its learning rate is off by a factor of 2\sqrt{2}, smaller than every margin of Dobb over AdamW (0.006 or more; Appendix E.2).

We would be grateful to know whether these responses resolve your concerns, and we are happy to provide further clarification if needed. If you find our responses satisfactory, we would appreciate it if you could consider updating your assessment.

Best regards, The Authors of Submission 49151

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers and the area chair for their careful reading. The reviews raise three common concerns. We answer each once here, and the individual replies refer to these items. Two further comments give the timing results and list the changes in the revision.

1. Seeds and the margin over AdamW (all reviewers)

The margin over AdamW on GPT-2 Small holds across three seeds: Dobb is below AdamW at every block count in every seed.

AdamWDobb B=1B=1Dobb B=4B=4Dobb B=16B=16Dobb B=64B=64
Final validation loss, mean ± SD over three seeds3.045 ± 0.0023.017 ± 0.0023.024 ± 0.0013.028 ± 0.0013.036 ± 0.002

We also trained a larger model that the submission did not include, GPT-2 XL (1.56B parameters). There, Dobb reaches 2.476 at B=16B=16 and 2.482 at B=64B=64 against 2.497 for AdamW (0.021 and 0.015 lower; Table 2). On GPT-2 Medium, two added seeds for AdamW, Dion at ρ=1/4\rho=1/4, and Dobb at B=64B=64 give 2.756, 2.754, and 2.742; Dobb is lowest in each seed (Table 10).

2. Comparison scheme: learning rate and update RMS (all reviewers)

The learning rate is selected for AdamW from sweeps at four model sizes up to 354M, with a fitted power law giving the rate at Large and XL (Appendix E.2), and every other optimizer reuses it under one update-scale formula. This follows Liu et al. (2025), who scale Muon's update by 0.2max⁡(m,n)0.2\sqrt{\max(m,n)} so that its update RMS falls in the range of 0.2 to 0.4 that they observe for AdamW, and who reuse the AdamW-optimal hyperparameters for Muon.

We apply this scale, τmax⁡(m,n)\tau\sqrt{\max(m,n)} with τ=0.2\tau=0.2, to every matrix optimizer. The resulting update RMS is 0.2 to 0.4 for Muon, RMNP, Dobb, and full-rank Dion, and 0.2ρ0.2\sqrt{\rho} for Dion at rank fraction ρ\rho (Table 11). Our submission described the scales incorrectly, as τmn/r\tau\sqrt{mn/r} for Dion, τm\tau\sqrt{m} for Dobb, and τn\tau\sqrt{n} for RMNP; every run in Table 2 used the shared scale. A rank-independent scale for Dion follows the practice of Ahn et al. (2025), whose scale depends only on the matrix shape and who use one fixed learning rate across rank fractions.

  • The scheme of Liu et al. (2025) transfers AdamW's learning rate and weight decay to the matrix optimizers without a separate sweep for each. Tuning every optimizer configuration at every scale was beyond our compute budget, and one shared, AdamW-tuned setting keeps the comparison even across the matrix optimizers: none receives tuning that the others lack.
  • The rate is chosen for AdamW at every scale, and no other optimizer's rate is adjusted. In our AdamW sweeps, a rate off by a factor of 2\sqrt{2} raises AdamW's final loss by about 0.002 (Appendix E.2), below every margin of Dobb over AdamW in Table 2 (0.006 or more). Other optimizers can be more sensitive; Dion's sweep is in Table 13. Under this comparison scheme, Muon is below AdamW at all four scales, consistent with Liu et al. (2025), who find Muon more compute-efficient than AdamW when it reuses AdamW's hyperparameters.
  • Raising low-rank Dion's update RMS to 0.2 with the rank-corrected scale recovers at most 0.019 of its 0.106 gap to full rank on GPT-2 Small (Table 12).
  • Tuning Dion's learning rate lowers its GPT-2 Small loss to 3.071 (ρ=1/16\rho=1/16) and 3.227 (ρ=1/64\rho=1/64), still above every Dobb entry (at most 3.036), although Dobb is not retuned (Table 13).

3. Timing and the speedup claim (Reviewers hGCr, nYDZ)

We agree that the optimizer-call speedup overstated the gain, and we removed the 7.8× headline. For GPT-2 XL on 32 H100 GPUs, the abstract now gives the optimizer call (9.2 ms for Dobb against 124.1 ms for Dion) and the pooled iteration (1069.1 ms against 1184.0 ms).

We also agree that forward and backward times varied between runs. This variation affects every model and configuration we timed. It comes from the system, mainly the network, and grows with run size even when runs share one geometry: the same hardware, number of nodes, GPUs per node, and sharding degrees. The revision handles it in three ways.

  • Pooled iteration times share one forward-backward estimate per configuration, so they differ only by the optimizer call (Appendix G.1). Across 12 runs per optimizer, the optimizer call's run-to-run SD is 0.8 ms for Dobb and 10.6 and 10.3 ms for Muon and Dion, against 15.3 to 26.9 ms for forward-backward.
  • Dobb's second-moment capture runs during forward-backward. Even so, over 48 runs in three allocations on GPT-2 Large, its forward-backward mean (7740 ms) lies within the span of the baselines' means (7713 to 7788 ms), and the baselines do all their optimizer work in the optimizer call.
  • The revision adds matched timing benchmarks of AdamW, Muon, Dion, and Dobb on GPT-2 Large and XL and on scaled-up GPT-2-style models up to 8B parameters, which support the timing claim on their own. Dobb's iteration time is comparable with AdamW's and below Muon's and Dion's in every GPT-2 configuration (Table 3).
AuthorsOpenReview ↗

General Response (continued): Timing Results

This comment gives the main timing results of the revision (Table 3), to which item 3 of the General Response and our replies to Reviewers hGCr and nYDZ refer. All runs use one implementation under FSDP, or FSDP with TP, and all times are in milliseconds.

Pooled iteration time

Model, GPUsAdamWMuonDionDobb
GPT-2 Large, 20 A50007764837689777760
GPT-2 XL, 32 A500011,64912,57714,15311,646
GPT-2 XL, 32 H1001069114311841069
GPT-2-style 3B, 32 H1002388279335002346
GPT-2-style 6B, 32 H1004823691656394820
GPT-2-style 8B, 64 H1005337810465015332

Optimizer-call time

Model, GPUsAdamWMuonDionDobb
GPT-2 Large, 20 A500011.3623.01224.16.9
GPT-2 XL, 32 A500018.7946.52523.216.3
GPT-2 XL, 32 H1009.382.7124.19.2
GPT-2-style 3B, 32 H10049.4454.31161.97.6
GPT-2-style 6B, 32 H10013.92106.8829.710.9
GPT-2-style 8B, 64 H10015.22782.11179.610.0

Dobb's pooled iteration time is comparable with AdamW's in every GPT-2 configuration we timed. Muon takes 1.07× to 1.52× as long per iteration as Dobb, and Dion 1.11× to 1.49×. Dobb's optimizer call takes 6.9 to 16.3 ms, against 82.7 to 2782.1 ms for Muon and 124.1 to 2523.2 ms for Dion. The XL (H100) row is the comparison in the abstract, Dobb at B=16B=16 against Dion at rank fraction 1/16.

AuthorsOpenReview ↗

Summary of Changes in the Revision

This comment lists the changes from the submitted version. Table and section numbers refer to the revised PDF; the submission's Table 3 (validation loss) is now Table 2, and its Appendix E (ablations) is now Appendix F. The revision also writes "cross-shard" where the submission wrote "cross-TP", since the claim covers shards under both TP and FSDP.

Quality results

  • Table 2 adds GPT-2 XL (1.56B parameters), where Dobb with 16 blocks reaches 2.476 against 2.511 for Dion at rank fraction 1/16; the abstract now reports this comparison.
  • Table 2 reports GPT-2 Small as the mean and standard deviation over three seeds, and Table 10 lists each seed, with three seeds on GPT-2 Medium for AdamW, Dion at ρ=1/4\rho=1/4, and Dobb at B=64B=64.
  • Appendix F.6 reports zero-shot HellaSwag, ARC-Easy, PIQA, and WikiText results for the GPT-2 Small checkpoints. WikiText perplexity orders the eleven configurations of Table 2 as their validation losses do.
  • Appendix F.1 ablates the decay of Dobb's second-moment average.

Protocol

  • Section 6 and Appendix E.1 correct our description of the update scales: every matrix optimizer used the shared scale τmax⁡(m,n)\tau\sqrt{\max(m,n)}, where the submission stated τmn/r\tau\sqrt{mn/r} for Dion, τm\tau\sqrt{m} for Dobb, and τn\tau\sqrt{n} for RMNP. Table 11 gives the update RMS of each optimizer, and Table 12 tests the rank-corrected scale for Dion.
  • Appendix E.1 and Table 13 sweep the learning rate of low-rank Dion on GPT-2 Small.
  • Section 6 states that error feedback was enabled in all Dion runs.

Timing

  • The abstract no longer states the 7.8× optimizer-call speedup. It reports, for GPT-2 XL on 32 H100 GPUs, the optimizer-call time and the pooled iteration time.
  • Table 3 and Appendix G add timing benchmarks of AdamW, Muon, Dion, and Dobb on GPT-2 Large and XL and on scaled-up GPT-2-style models of 3B, 6B, and 8B parameters.
  • Appendix G.1 defines the pooled iteration time and reports a 48-run study of the run-to-run variation of forward-backward time.
  • Appendix G states the network bandwidth and the interconnect between GPUs for both clusters.
AuthorsOpenReview ↗

Summary of Reviews and Author Responses

Dear Area Chair and Reviewers,

We thank all reviewers for their feedback. As the discussion period closes in less than four hours (November 18, AoE), we summarize the reviews and our responses. Reviewer nYDZ replied to our response, raised the soundness score from 2 to 3, and wrote: "I think this is a solid paper and recommend its acceptance." Reviewers quYb and hGCr have not yet replied.

Strengths recognized by the reviewers

  • Design: "a clean and novel design" that saves communication in LLM training (quYb); "an insightful system and algorithm co-design that addresses an important problem in distributed LLM training" (hGCr); "a simple approach to an important distributed training bottleneck" (nYDZ).
  • Ablations: "careful ablations and reporting" (quYb), "thorough ablations" (hGCr), and "a thorough ablation study in the appendix" (nYDZ).
  • Clarity: "The paper is well written, and the motivation and implementation of the design are clear" (nYDZ).

Reviewer hGCr (rating 2). The reviewer asked for two controls, a calibrated Dion baseline, and unconfounded timing. We added all of them.

  • Identity basis (Q1). Signum, which is Dobb at B=nB=n, was trained under the protocol of Table 2. Dobb has lower validation loss than Signum at every block count on both sizes. On GPT-2 Small (mean of three seeds): 3.017, 3.024, 3.028, and 3.036 for Dobb with 1, 4, 16, and 64 blocks vs. 3.049 for Signum. On GPT-2 XL (one seed): 2.461, 2.471, 2.476, and 2.482 for Dobb with 1, 4, 16, and 64 blocks vs. 2.485 for Signum (Appendix F.7, Table 20).
  • Block-local Muon, which applies Muon independently to each block. With the same block partition, Dobb has lower loss at B=16B=16 and B=64B=64 (p=0.010p=0.010 and p<0.001p<0.001, paired t-test over three seeds), and its loss rises by 0.019 from B=1B=1 to B=64B=64, against 0.044 for block-local Muon (Appendix F.5, Table 18).
  • Dion calibration (Q2). Error feedback was enabled in all Dion runs, and Dion's update scale follows the reference (Ahn et al., 2025). With its learning rate tuned, Dion reaches a validation loss of 3.071 at ρ=1/16\rho=1/16 and 3.227 at ρ=1/64\rho=1/64, still above Dobb's loss at B=16B=16 (3.026) and B=64B=64 (3.037), even though Dobb was not tuned (Table 13).
  • Timing. We removed the 7.8× optimizer-call speedup claim from the abstract, which now reports pooled total iteration time (1069.1 vs. 1184.0 ms for Dobb and Dion on GPT-2 XL, 32 H100 GPUs). Pooled iteration times remove the run-to-run forward-backward variation the reviewer identified (Appendix G.1, supported by a 48-run study), new benchmarks compare AdamW, Muon, Dion, and Dobb in one implementation up to 8B parameters (Table 3), and Appendix G gives the hardware.

Reviewer nYDZ (rating 3; soundness raised from 2 to 3).

  • Speedup claim. The abstract reports iteration time instead of the optimizer-call ratio.
  • Seeds. At the reviewer's example of B=16B=16, Dobb reaches 3.028 ± 0.001 against 3.045 ± 0.002 for AdamW over three seeds, and Dobb is below AdamW at every block count in every seed (Table 10).
  • Downstream tasks. WikiText perplexity agrees with validation loss: Dobb's is lower than AdamW's at every block count (Appendix F.6, Table 19).
  • Fairness of the update scale. All matrix optimizers share the scale of Liu et al. (2025). Two checks test it for low-rank Dion: a rank-corrected scale that raises its update RMS to 0.2 (Table 12) and a learning-rate sweep (Table 13). Neither brings Dion below Dobb.

Reviewer quYb (rating 3).

  • Seeds. Three seeds for AdamW, Dobb at B=64B=64, and Dion at ρ=1/4\rho=1/4 on GPT-2 Small (3.045 ± 0.002, 3.036 ± 0.002, 3.042 ± 0.003) and GPT-2 Medium (2.756 ± 0.004, 2.742 ± 0.000, 2.754 ± 0.001). Dobb is lowest in every seed at both sizes (Table 10). On GPT-2 XL, Dobb at B=64B=64 reaches 2.482 against 2.497 for AdamW.
  • Tuning. Following Liu et al. (2025), the learning rate is tuned for AdamW and transferred to all optimizers. A rate off by a factor of 2\sqrt{2} changes AdamW's loss by about 0.002, below every margin of Dobb over AdamW in Table 2 (0.006 or more; Appendix E.2), and the Dion sweep above shows that tuning Dion does not change its comparison with Dobb.

Manuscript updates addressing the reviewers' concerns

  • A GPT-2 XL (1.56B) column in Table 2, and multi-seed results on GPT-2 Small and Medium (Table 10).
  • Signum and block-local Muon comparisons (Appendices F.5 and F.7), and the Dion learning-rate sweep and rank-corrected scale (Appendix E.1, Tables 12 and 13).
  • Zero-shot downstream evaluation (Appendix F.6).
  • Timing benchmarks with AdamW and Muon up to 8B parameters, pooled iteration times with a 48-run variation study, and hardware specifications (Table 3, Appendix G).

We would be grateful if Reviewers quYb and hGCr could let us know whether our responses address their concerns, and we are happy to clarify anything that remains.

Best regards, The Authors of Submission 49151

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.