acceptodds
Under review as a conference paper at ICLR 2027

Dion3: Full-Stack Orthogonal Updates

13 views

Abstract

The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this computational cost, eroding the benefits of Muon in many settings. We present Dion3, a revision of Muon that targets this overhead at every level of the stack. Our Gram Newton-Schulz algorithm reduces the FLOP cost of orthogonalization, our CuteDSL kernels accelerate it by exploiting symmetry, and our megabatching strategy reduces communication overhead. Moreover, we propose a simple change to the update rule that cuts costs even further: selecting only a fraction of the momentum matrix's rows to orthogonalize at each step. This update rule improves on Dion (another "compressed" version of Muon), in both speed and performance. Overall, Dion3 matches or improves on the loss achieved by Muon but reduces optimizer step time by up to 6x. Dion3 is available via the dion package as a drop-in replacement for Muon.

Primary area:
optimization
Keywords:
optimization, pretraining, muon, dion, gram, newton schulz, kernel, cutedsl, cutlass

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 4 reviews, above 99% of the others.
Reviewer_tHnoOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to reduce the computation and communication costs of the Muon optimizer. Its contributions can be summarized as follows:

  1. Gram Newton-Schulz that decouples non-symmetric ratio α\alpha and the number of Newton-Schulz iterations TT, reducing the computation cost.
  2. CuteDSL symmetric GEMM kernels on Hopper and Blackwell, for faster Gram Newton-Schulz.
  3. subsampling rows or columns of the momentum matrix for orthogonalization to reduce the computation and communication cost.
  4. megabatched communication to reduce the communication cost.

Experiments suggest that these techniques bring the optimizer step time close to that of AdamW while matching or exceeding the performance of standard Muon.

Critical strengths
  • suggested techniques reduce the computation and communication cost of Muon to that of AdamW, regarded as tolerable in general.
  • it provides pip-installable packages dion and gram-newton-schulz, that are expected to have a strong impact on the community.
Critical weaknesses

None

Questions for authors
  • convergence behavior of EF: theoretically, error feedback aims to preserve the convergence guarantee for biased compression methods. Is there any theoretical reason why the method in the paper is named EF? In other words, does it guarantees the same convergence behavior?
1 comment

Thank you for the kind feedback!

As for your question, we don't have a theoretical analysis, but the name "error feedback" (which we adopt from the Dion paper of Ahn et al.) is indeed a reference to a technique used in federated learning to combat biased compression. We are doing the same thing here, if you think of our "compression" scheme as selecting some rows and ignoring the rest. Error feedback helps ensure that the other rows get their due. (And it does help; see Figure 25.)

Reviewer_FArZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Dion3 reduces Muon's optimizer overhead through four complementary components: Gram Newton–Schulz (GNS), symmetry-aware GPU kernels, fractional row/column updates with selective momentum decay, and megabatched communication. GNS reproduces the corresponding Newton–Schulz polynomial computation in exact arithmetic, while its implementation addresses finite-precision instability through restarting and arithmetic refinements. Experiments include GNS-only controls across four architectures, 1B models trained on 100B tokens, and models up to 14B trained on 10B tokens. The paper reports comparable or slightly improved training quality and up to 6.5× optimizer-step acceleration, rather than an equivalent end-to-end training speedup.

Critical strengths
  • S1. Practical acceleration independent of fractional updates. GNS and symmetric kernels accelerate the same exact-arithmetic Newton–Schulz computation. The paper identifies concrete numerical failures and addresses them through restarting, precision choices, and arithmetic rearrangement. GNS-only experiments preserve training quality across four architectures and two coefficient families, establishing a useful contribution independently of Dion3's fractional-update rule.

  • S2. Complementary and carefully measured systems improvements. Fractional updates reduce orthogonalization work and communication volume, while megabatching reduces communication-round overhead. Staged optimizer-step benchmarks demonstrate compounding speedups, and separate communication measurements and GNS-only multi-architecture experiments strengthen the systems evidence.

Critical weaknesses
  • W1. Superiority over the closest alternatives is not established. The abstract claims improvements over Dion in both speed and performance, but no matched comparison against Dion, TRION, or block-orthogonalization methods is reported. Appendix A.2 also suggests greater stability than the predecessor Gram-based method without a controlled comparison. Since competing update rules could also benefit from GNS and the proposed kernels, the incremental advantages of Dion3's update rule and revised Gram recurrence remain insufficiently isolated.

  • W2. The reported quality advantage is not sufficiently isolated. The approximately 0.01–0.027 validation-loss improvements lack seed-level uncertainty. Table 4 transfers the 1B-selected settings to larger models trained on only 10B tokens, without independent tuning at those scales. More importantly, Appendix G.1 assigns auxiliary AdamW groups the base learning rate, so increasing Dion3's learning rate may also change these groups. The experiments therefore do not establish whether the small quality gains arise specifically from the fractional matrix-update rule.

  • W3. The mechanism and operating regime of fractional updates remain unclear. Although Section 5 mentions preliminary random-selection experiments, no quantitative ablation isolates top-ℓ1\ell_1 selection from random selection or selective momentum decay from ordinary all-row decay. Dion3 initially lags on FineWeb but outperforms NorMuon throughout the reported ClimbMix curves, while the learning-rate transfer rule fails in the smaller-scale setting. Data, architecture, training recipe, and weight-update precision also differ across these pipelines, leaving the causes of the contrasting behavior unresolved.

Questions for authors
  • Q1 (W1). Closest-prior comparisons and end-to-end benefit. Can the authors compare Dion3 against Dion and at least one of TRION or MuonBP using comparable tuning budgets and optimized orthogonalization backends, reporting quality, runtime, and actual communication volume? Given Figure 3's early token penalty, please also report measured end-to-end wall-clock time to prespecified validation-loss targets, including forward/backward computation and communication. Separately, does the revised GNS recurrence improve runtime or stability over its predecessor when coefficients, precision, and restart schedules are controlled?

  • Q2 (W2). Robustness and attribution of quality gains. For a prespecified selection fraction, what is the paired mean and variability of Dion3's advantage across independent seeds under comparable tuning budgets? Appendix G.1 appears to imply that Table 4's higher Dion3 base learning rate also increases the auxiliary AdamW rates. Can the authors confirm this and test whether the advantage persists when auxiliary rates are held fixed or tuned independently? Please distinguish transferred 1B settings from independently tuned larger-scale results.

  • Q3 (W3). Mechanism and precision controls. Under a fixed training recipe, can a top-ℓ1\ell_1/random-selection × selective/all-row-decay ablation identify which components are necessary at the same fraction and with comparable learning-rate tuning? Does the advantage persist under a higher-precision weight-update control instead of MXFP8? Please clarify the master-weight precision and rounding scheme in each pipeline.

Required Corrections

  • Proof and domain assumptions (p. 3, L109–113; p. 4, L165–166; p. 16, L810–823). Theorem 2's proof uses qt=xt/x0q_t=x_t/x_0, which is undefined at x0=0x_0=0; prove qtx0=xtq_tx_0=x_t instead. The expression (XX⊤)−1/2X(XX^\top)^{-1/2}X requires full row rank unless a pseudoinverse convention is specified. Also qualify the approximation-to-one claim on [0,1][0,1], since every odd-polynomial composition vanishes at zero.

  • Stability argument (p. 18, L963–969; pp. 20–21, L1078–1083). Restrict the negative-input divergence claim to the analyzed polynomial coefficients. In Appendix B.2.1, bounded κ2(Qt)\kappa_2(Q_t) alone does not imply bounded ∣∣Qt∣∣2||Q_t||_2; an additional scale bound is needed.

  • Equation and momentum errors (p. 18, L938–940; p. 22, L1182–1187; p. 24, L1290–1295). Figure 7's caption should use bt=−10/8b_t=-10/8, not +10/8+10/8. Appendix B.2.3 should use (atI+btA+ctA2)X(a_tI+b_tA+c_tA^2)X, rather than X(atI+btA+ctA2)X(a_tI+b_tA+c_tA^2), for A=XX⊤A=XX^\top. Footnote 3 should clarify that, at f=1f=1, the orthogonalization inputs coincide in exact arithmetic, while the stored post-step momentum buffers differ by a factor of μ\mu.

  • Reporting inconsistencies (p. 25, L1340; p. 26, L1365–1375; p. 35, L1851–1854). Clarify the hardware and rank mapping in Table 1: the caption states four GH200s per node, the rows list 8/32 FSDP shards on one/four nodes, and Appendix E describes these configurations as 8/32 GPUs. Figure 24's caption should identify NorMuon, not Muon, as the baseline.

  • Typographical errors (p. 10, L488–490; p. 31, L1657–1658). Correct “occassionally” to “occasionally,” “1B-paramter” to “1B-parameter,” and “trainin data” to “training data.”

3 comments

Thank you for the thorough review! We have added additional experiments that we hope will answer some of your questions.

W1 / Q1

Comparison to Dion We added experiments that directly compare Dion3 to Dion in both training quality and runtime.

First, Figure 3 (top panel) shows that Dion is intermediate in quality between Dion3 and the baseline at low token budgets. At high budgets it is worse. We tuned Dion's learning rate the same way we chose Dion3's: by a sweep at smaller scale. That showed η=0.028\eta=0.028 to be optimal. However, unlike Dion3, Dion's optimal learning rate is sensitive to budget. Therefore, we reran with η=0.01\eta=0.01 (matching the other optimizers), which was better at 10B and 20B tokens. Therefore, it is possible that Dion is not strictly worse at 10B and 20B, but only harder to tune than Dion3. Either way, it is unlikely to meaningfully beat Dion3 in that setting, where Dion3 already matches Muon.

Second, Figure 6 compares the runtime savings of Dion3's update rule directly to those of Dion. For a controlled comparison, we run on a single GPU and omit Gram Newton-Schulz and our custom kernels. This just tests which type of compression is faster. The results show that Dion3 is far faster than Dion, often by an order-of-magnitude or more. In particular, Dion is not applicable when the hidden dimension is 2048 or 4096, but Dion3 is.

Comparison to Trion / MuonBP We were not able to run these additional comparisons during the rebuttal period. However, we argue that Dion3's update rule has some clear advantages over these methods. As described in Appendix A.3, Trion is conceptually almost the same as our rule. Both rules build a low rank approximation to M\mathbf M by selecting vectors from a fixed basis. While we use the standard basis vectors, Trion uses the columns of the DCT matrix. We have no reason to think that one basis is better than the other in terms of approximation quality. However, our choice is simpler; it avoids the need to compute the DCT of each row of M\mathbf M and to repeatedly multiply by the selected basis matrix. As for MuonBP, it may be harder to use in practice because it orthogonalizes each shard of M\mathbf M separately, so its update rule is a function of its sharding configuration. For instance, while MuonBP may work well with tensor parallelism of 8 shards, it is unlikely to work with modern frameworks that use 64 or more shards. (See Nemotron's generalized tensor parallelism: https://docs.nvidia.com/megatron-core/developer-guide/0.20.0/api-guide/core/generalized_tensor_parallel.html)

End-to-end wall-clock time In the setting of our experiments, end-to-end wall-clock time is basically the same for all optimizers. Across all our 225M and 418M runs, Dion3 was between 3% faster and 1% slower than NorMuon. This is because in our small-scale setting, the wall-clock time of the forward pass and backward pass dominate the optimizer step time, so the differences between optimizers are not so significant. As described in Section 8 and Appendix H, this is not the case in many real-world settings.

Does the revised GNS recurrence improve runtime or stability over its predecessor? Yes. We experimented with Appendix J of Amsel et al. early in the project and found that switching to our version was more stable. We got better accuracy, more robustness to ill-conditioned inputs, and fewer blowups. The stability of our version is supported by analysis of analogous methods in Higham (2008, chapter 7), which characterizes stability in terms of the Frechet derivative. Compared to the predecessor, runtime is identical, because both versions do the same number of multiplications with matrices of the same dimensions.

W2 / Q2

Variance across seeds We found small seed-to-seed variability throughout our experiments. We have added explicit results in Appendix G.5 ("Variability across seeds" on page 35).

Does Dion3's base learning rate affect the auxiliary AdamW rates? Yes, in all experiments, we used a common learning rate for the Muon and AdamW parameters. You ask specifically about the "larger-scale runs" described at the end of Section 7.1 and the learning rate transfer rule of Appendix G.2. Unfortunately, we do not have the resources to redo these larger-scale runs during the rebuttal period. Instead, we added an experiment that tunes a separate AdamW rate for the smaller-scale runs; see the discussion in Appendix G.5, "Tuning the baseline" and Table 8 on page 35. Note that in these smaller-scale experiments, the learning rate does not need to change with ff; in fact we find that the transfer rule does not help in those runs regardless of whether it is applied to the Muon parameters, the AdamW parameters, or both.

W3 / Q3

Top vs. random selection; selective vs. all-row decay Good suggestion! We ran this experiment and added the results to Appendix G.5; see "Ablating Error Feedback and Selection Strategy" and Figure 25 on page 35-36. In this ablation, selective decay ("error feedback") is clearly better than all-row decay. Random selection is actually slightly better than top-k selection. We did not retune the learning rate.

Clarify the use of MXFP8 In the "larger-scale runs", weights are stored in mxfp8 (with mxfp8 matmuls) but activations and gradients were stored in bf16. Unfortunately, we are not able to redo those runs during the rebuttal period. In the smaller-scale experiments, mxfp8 is not used at all, only bf16.

Required Corrections

  1. Yes, we assume full rank (added at line 166). Updated the proof of Theorem 2 to cover x0=0x_0 = 0. The approximation on [0,1][0, 1] claim is informal, describing the motivation for picking these polynomials. Technically, the approximation is only good on a smaller interval like [10−3,1][10^{-3}, 1].
  2. These two claims might not be well explained or technically proven, but they are basically correct. (1) The Newton-Schulz p(x)=xh(x2)p(x) = x h(x^2), so p′(x)=h(x2)+2x2h′(x2)p'(x) = h(x^2) + 2x^2 h'(x^2) so p′(0)=h(0)p'(0) = h(0). Newton Schulz surely needs p′(0)>1p'(0) > 1 so h(0)>1h(0) > 1. Equation 5 shows that small negative eigenvalues do grow exponentially if h(0)>1h(0) > 1. (2) Assuming the input has an eigenvalue > 0.5, lim⁡tQt\lim_t Q_t will have a small eigenvalue between 1 and 2. So the condition number and max eigenvalue will be the same up to a factor of two. For reasonable choices of coefficients, including all the standard choices, this basic logic works at every iteration.
  3. Thank you! We corrected these.
  4. These have been also corrected.
  5. These have been corrected too. It's four nodes of eight GPUs each.

Thank you for the detailed rebuttal and revisions. Figures 3 and 6 helpfully clarify training quality against Dion and update runtimes, though TRION, MuonBP comparisons remain absent and end-to-end times were similar in the reported 225M and 418M runs. Then, I appreciate the extra seeds and the small-scale separate learning rate ablation. However, the shared LR in large-scale runs still obscures the matrix update's exact contribution. Also, Figure 25 effectively addresses my main question on W3 by showing that selective decay outperforms all-row decay, and random selection slightly beats selection by ℓ1\ell_1 norm. I also appreciate the correction to the Theorem 2 proof. Overall, the Gram Newton–Schulz method remains valuable, and I am maintaining my score of 3.

Reviewer_3uQSOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Dion3, a collection of methods to speed-up Muon optimization. Dion3 consists of four distinct contributions, which each speed-up Muon and compound when used together. These contributions are:

  1. Specialized GPU kernels for symmetric matrices that skip redundant computations
  2. Switching Newton-Schulz to Gram-Newton Schulz, a specialized version in particular for rectangular matrices
  3. Sub-selecting rows from Muon for the update step
  4. Batching transformer layers of the same shape together in multi-GPU settings, which in turn speeds up communication by saturating the communication link better

2 is proven to be equivalent to Newton-Schulz, and 3 is shown empirically to achieve similar performance as vanilla Muon for larger token budgets. 1,2,4 do not change the Muon optimization trajectory, which is also verified through experiments.

Critical strengths
  1. Large potential practical impact. The results of the paper show impressive practical results (over 4x improvement in speed) while the implementation seems to be easy to use for a practitioner. 3 out of 4 proposed methods produce the same optimization trajectory as Muon, making it a drop-in replacement for potentially many use cases.
  2. Soundness. The methods that the authors propose are very well supported by experiments and theorems. The extended appendix covers many additional questions that would pop up during reading.
Critical weaknesses

None

Questions for authors

From my understanding of the algorithm presentation, the weight decay is influenced by the learning rate. As Dion3 sets the learning rate to be η/f\eta / \sqrt f, the weight decay rate is Dion3 is also different than in NorMuon, and as weight decay is enabled for all matrix rows, not just the selected ones, the total decay of the weight matrix is increased. Is there an ablation on this hyperparameter, for example where the weight decay rate for NorMuon is also scaled by η/f\eta / \sqrt f?

1 comment

Thank you for the lovely review!

Good point about weight decay; your understanding is correct. We remember that in some of our earlier experiments, we tried turning weight decay off and it did not have such a large effect, but we do not have a controlled comparison like you suggest. However, note that in our "smaller-scale runs", Dion3 transfers the learning rate from NorMuon unchanged.

Reviewer_P46QOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles an important problem. The Muon optimizer has a significant cost related to orthogonalization via NS or some SVD-like operation. This is especially problematic when the weights are sharded, when we use a fancy distribution strategy beyond distributed data parallel.

The authors introduce a way to reduce this overhead, which is not present for optimizers like AdamW, and they reduce the flop cost. They also introduce kernels which allow exploitation of the speedup, and they also use ideas from Dion.

The authors present substantial experiments to compare their new method against the regular Muon optimizer and also AdamW, and show that speedups are real and most of the overhead of AdamW is gone. I also like the fact that the code for this has already been open-sourced.

Overall, this paper is solid, and the contributions are easy to understand. I was a bit disappointed with the experiments, but it's still sufficiently novel and useful, and I'm hoping it gets widespread adoption.

Critical strengths
  • The challenge of scaling Muon in Section 2 is very well written. It sets up the problem nicely. For instance, it clearly calls out that Muon must gather the shards of M in order to compute some stuff on M.
  • The runtime analysis section was also nice. It clearly calls out the different issues with standard NS.
  • I also really appreciated Appendix B and how detailed it is. The entire appendix is actually quite substantial and convincing, and it's quite nice to read.
  • The sequencing of sections 3 and 4 is quite nice. It goes into the right amount of detail to introduce all the different issues that the algorithm solves.
  • The first figure in the paper also is quite self-explanatory and tells the reader the key contribution of the paper.
  • The systems comparison, especially the timing work, is quite good and convincing.
Critical weaknesses
  • “for the 418M model, Dion3 takes 20% longer to achieve loss ≤ 3.2, but both optimizers reach loss = 3.0 at the same time” - This really concerns me, and I don't know whether the rest of the paper explains this.
  • Also, where are the direct comparisons with Dion? Why is only non-Muon compared? I feel this comparison is too narrow in terms of quality, at least.
  • Why are large-scale runs done with a 1B model and 100B tokens? Why not use something with a Chinchal-optimal? This is the big weakness in the paper.
  • Why are there no post-training experiments? This would be very useful. A lot of people post-train.
  • Although this is not a critical weakness, it would be nice to contrast algorithm 3 with the Dion or the Muon update rule so that people can remember the delta.
Questions for authors

See weaknesses.

2 comments

Thank you so much for the kind and thorough feedback! We have added additional experiments that we hope will address some of your questions.

Weakness #1

Your concern seems to be that you expect the comparison between optimizers to be consistent, regardless of model size or training budget. In our experience---even apart from Dion3---this is not always the case. The literature on Muon includes lively debates about the true size of its advantage over AdamW in different settings, showing that these factors can significantly affect the gap; see Wen et al., "Fantastic Pretraining Optimizers and Where to Find Them" and follow-up work. You are right to point out that we do not provide an explanation for why the gap exists in some settings but not in others. (Perhaps future work will!) However, we believe we have demonstrated the following, particularly in Figure 3: (1) the gap between Dion3 and Muon consistently closes as the training budget increases (2) the gap between Dion3 and Muon consistently closes as the target loss decreases. Thus, we are hopeful that downstream applications are in the regime where the gap is small.

Weakness #2

We have added additional experiments that directly compare to Dion and to plain Muon (as opposed to NorMuon). See Figure 3 (top panel) and Figure 6 on pages 8 and 10.

First of all, Figure 3 now shows the quality of Muon and the corresponding Muon flavor of Dion alongside NorMuon and the NorMuon flavor of Dion. The findings are consistent with our previous experiments: Dion3 (Muon flavor) is to Muon as Dion3 (NorMuon flavor) is to NorMuon. As expected, NorMuon is slightly better than Muon, and Dion3 (NorMuon flavor) is slightly better than Dion3 (Muon flavor).

Second, Figure 3 shows that Dion is intermediate in quality between Dion3 and the baseline at low token budgets. At high budgets it is worse. We tuned Dion's learning rate the same way we chose Dion3's: by a sweep at smaller scale. That showed η=0.028\eta=0.028 to be optimal. However, unlike Dion3, Dion's optimal learning rate is sensitive to budget. Therefore, we reran with η=0.01\eta=0.01 (matching the other optimizers), which was better at 10B and 20B tokens. Therefore, it is possible that Dion is not strictly worse at 10B and 20B, but only harder to tune than Dion3. Either way, it is unlikely to meaningfully beat Dion3 in that setting, where Dion3 already matches Muon.

Third, Figure 6 compares the runtime savings of Dion3's update rule directly to those of Dion. For a controlled comparison, we run on a single GPU and omit Gram Newton-Schulz and our custom kernels. This just tests which type of compression is faster. The results show that Dion3 is far faster than Dion, often by an order-of-magnitude or more. In particular, Dion is not applicable when the hidden dimension is 2048 or 4096, but Dion3 is.

Finally, we note that runtime benchmarks of Muon and Dion3 (Muon flavor) are given in Figure 26. The results closely match those for NorMuon given in the main text (Figure 5, Figure 1).

Weakness 3

Figure 3 and the "Smaller-Scale Runs" section of 7.1 describe runs that are below, at, and above Chinchilla optimal. In particular, for a 1B-parameter model, Chinchilla optimal would be 20B tokens. That run is shown in green in Figure 3.

In the section "large scale runs", we experimented with increasing either the data or the model size. So as you point out, the runs reported in Fig 4 use 1B parameters but extend to 100B tokens, which is above Chinchilla-optimal. The complementary results in Appendix G.3 (Table 4 and Fig 22) instead grow the number of parameters to 3B, 4B, 7B, and 14B, and are thus below Chinchilla-optimal. Resource constraints prevent us from doing runs that are simultaneously larger than 1B parameters and longer than 20B tokens. Figs 4 and 22 give some indication that, in this codebase at least, Dion3 wins throughout training anyway (though longer runs can give a misleading view on shorter runs, since the cooldown phase doesn't happen at the right time).

However, we believe that training longer than Chinchilla optimal is actually the most realistic setting. The Chinchilla-optimal scaling rule gives the optimal training compute to reach a given loss, but it does not take into account the fact that larger models are costlier at inference time. Therefore, model providers typically prefer a smaller, "over-trained" model to a larger, "Chinchilla-optimal" one. For this reason, it is common in practice to train well beyond Chinchilla.

Weakness 4

This is a valid point! Optimizers may behave differently in post-training settings, and the best optimizer for pre-training may not be the best for post-training. However, we think pre-training is a sufficiently important application in itself. Indeed, most of the literature on Muon so far has focused on pre-training, so we limit our scope to pre-training too.

Weakness 5

Thanks for the feedback. We compare the three optimizers more thoroughly in Appendix A.3 than in the main text. Do you suggest moving some of this discussion to the main text? The pseudocode of Dion is probably too complicated for a direct comparison with Algorithm 3. However, Algorithm 3 reduces to NorMuon if we simply set f=1f=1, so that S\mathcal S is the set of all rows. It further reduces from NorMuon to Muon if we remove lines 4, 5, and 6 and replace O^\widehat{\mathbf O} by O\mathbf O in line 8. We can update the lines before Algorithm 3 and the paragraph labeled "Weight Update" to clarify this, if you suggest.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised draft

Thank you all for your reviews! We have uploaded a revision of the paper with some extra experiments and minor corrections. Changes are marked with red text. The main additions are in section 7 (Figures 3 and 6) and in Appendix G.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.