acceptodds
Under review as a conference paper at ICLR 2027

Evie-KF: Risk-Sensitive Kronecker-Factored Preconditioning for Adaptive Optimizers

Abstract

Adaptive optimizers such as Adam adapt each update to the magnitude of its coordinate, but this diagonal view discards correlations in gradient noise. Those correlations contain information about which directions are more reliable. We propose Evie-KF, an adaptive optimizer derived from a risk-sensitive formulation of stochastic optimization. Evie-KF extends Adam's diagonal normalization with a matrix preconditioner obtained by solving the risk-sensitive Riccati equation in coordinates whitened by Adam. The resulting operator, , uses the centred covariance of gradient noise so that correlated directions are treated jointly. It recovers AdamW exactly at and adds one tuned parameter. Across vision, language, and finance tasks, Evie-KF improves over AdamW and other common adaptive optimizers on most tasks tested across multiple seeds. On CIFAR-100, it improves top-1 accuracy over AdamW by 5.6%, winning in all five seeds. On WikiText-103 language modeling, it improves over AdamW by 11.5% and over SGD by 40.7%, again in all five seeds. On a 33.7M-parameter OpenWebText model, the largest we train, Evie-KF improves over AdamW by 13.1% on three of three seeds at matched wall time. Ablating the Kronecker structure on its own, against a diagonal gate at matched normalization, it gains 6.75% on a finance panel and 6.67% on WikiText-2, on nine and ten of ten seeds, both surviving Holm correction. [Our code is available](https://github.com/rpextra2026-afk/evie-kf-optim), with Evie-KF released as the [`evie-kf`](https://pypi.org/project/evie-kf/) Python package.

Primary area:
optimization
Keywords:
Optimization, Adaptive Optimization, Deep Learning Optimization, Gradient Noise, Stochastic Optimization, Second-Order Optimization, Kronecker-Factored Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yDcQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Evie-KF, an AdamW-based optimizer using Kronecker-factored gradient-noise covariance. Across 12 tasks, it outperforms AdamW on 8 but shows no statistically significant overall advantage. Kronecker ablations yield significant gains of 6.75% on Finance-Tech and 6.67% on WikiText-2. On OpenWebText (33.7M parameters), Evie-KF improves over AdamW but trails Muon.

Critical strengths
  • The method has a well-specified AdamW nesting property, explicit Kronecker algebra, and numerical implementation checks.

  • The ten-seed structural ablation compares Kronecker and diagonal gates under matched spectral normalization, showing Holm-significant gains on Finance-Tech (+6.75%) and WikiText-2 (+6.67%).

Critical weaknesses

The coupling-bias claim is stronger than Proposition 4 proves.

Cov⁡(g,δ)\operatorname{Cov}(g,\delta)=0 does not imply independence of the nonlinear gate based on δδ⊤\delta\delta^\top and the gradient, particularly for skewed noise, same-step Adam statistics, and gradient-dependent clipping.

Fairness and selection may alter the ranking.

Evie-KF searches 49–99 hyperparameter points per task versus approximately 7–9 for most baselines, while the fixed-γ\gamma study covers only four selected hosts. Finance further conditions the test sample on every arm completing every seed, retaining only 23/49 Finance tickers and 37/49 Tech tickers. Full-budget-matched and full-panel sensitivity analyses are needed.

The risk-sensitive interpretation is not established for the implemented update.

The reported gate uses the optimistic process-noise sign branch, while the separate update-noise risk criterion does not derive the matrix gate; norm preservation may increase directional gains .

Questions for authors

Split-batch independence: Proposition 4 establishes Cov⁡(g,δ)=0\operatorname{Cov}(g,\delta)=0, but this does not generally imply independence of the nonlinear preconditioner. Can the authors clarify under what assumptions the coupling-bias claim remains valid, particularly with same-step Adam whitening and gradient clipping?

Fairness and statistical robustness: Using the existing experimental results, can the authors report the Evie-KF versus AdamW comparison excluding seed 42 (used for tuning), with paired uncertainty estimates? Can they also clarify how the unequal tuning budgets (49–99 versus 7–9 trials) affect the conclusions?

2 comments

Thank you for your careful review. We appreciate your recognition of the exact recovery of AdamW, the Kronecker formulation, and the matched-normalization ablations.

The gains over AdamW persist without the tuning seed and at a matched budget

We excluded tuning seed 42 from the original comparison. We also reran Evie-KF on all 12 tasks with fixed =30 and the baselines’ seven-point learning-rate budget.

Evie-KF versus AdamWTasks wonMedian relative gainWilcoxon p
Original comparison, all seeds8/12+3.3%0.20
Original comparison, seed 42 excluded10/12+4.0%0.13
Fixed , paper’s complete-case tickers8/12—0.30
Fixed , all paired finance tickers7/12——

The p-values are two-sided and unadjusted. Excluding seed 42 strengthens the observed result, with one near-tie among the 10 wins. At the matched budget, Evie-KF improves on AdamW on all four vision tasks and three of four language tasks under either ticker rule.

  • Expanded baseline search. We gave AdamW, AdaBelief, Lion, and Muon 49 trials each on WikiText-2. Their median test perplexities changed by at most 1.6 points on the same three seeds.
  • Finance ticker selection. The sole fixed- finance win reverses when all paired tickers are included. Tuned Evie-KF beats AdamW on 42/49 Tech, 37/49 Healthcare, 31/49 Financials, and 30/50 Consumer/Industrials tickers.

We will report the fixed- comparison in the main results, include both finance ticker rules, and provide paired uncertainty estimates with and without seed 42.

Split-batch sampling gives zero covariance, and the lagged gate preserves the gains

We agree that zero covariance alone is insufficient to establish independence. Let g be the half-batch mean and the half-batch difference divided by two. Exchangeability gives E[δ∣g]=0\mathbb{E}[\delta\mid g]=0. With independent Gaussian half-batch noise, g and are independent. With skewed noise, a gate built from can still couple to g. Using earlier steps to estimate the gate removes its direct dependence on the current noise sample. Same-step Adam whitening and gradient-dependent clipping remain possible sources of coupling in the full update.

The lagged gate closely matched Evie-KF on both ablation hosts, suggesting that the gains did not require the current split-batch noise sample.

Ablation hostEvie-KFLagged gateRandom eigenbasis
WikiText-2, perplexity99.6599.6599.85
Finance-Tech, MSE3.77810-43.78010-43.76810-4

We will replace “provably decoupled” with the precise split-batch statement and add the lagged control to the ablation table.

The gate has a variational derivation and reduces noise variance at fixed step length

The optimistic process-noise branch gives a derivation of the gate. The revised analysis fixes the reference direction and noise covariance in whitened coordinates. At the same step length, the variance of the first-order loss change is at most that of the ungated step and is nonincreasing in . Norm preservation can amplify some directions. The gain in the noisiest eigendirection remains at most one.

The gate also has a variational characterization independent of the Riccati model. It is the unique positive-definite minimizer of the AdaGrad-style objective

tr⁡(K−1)+tr⁡ ⁣[K(I+γΣz)].\operatorname{tr}(K^{-1})+\operatorname{tr}\!\left[K(I+\gamma\Sigma_z)\right].

Its solution is K=(I+z)-1/2, where z is the centered gradient-noise covariance in Adam-whitened coordinates.

In the revision, we will present the variational characterization alongside the Riccati derivation, state the variance guarantee for the norm-preserving update, and clarify the scope of the separate update-noise criterion.

The new ablations show where the learned basis and wider Kronecker coverage help

Exact recovery of AdamW at =0 gives a clear reference for evaluating the gate. The matched-normalization ablations show that the matrix gate can improve on a diagonal gate.

A random eigenbasis gave similar results on WikiText-2 and Finance-Tech. On CIFAR-100, with the original loader and the learning rate selected for the learned basis, Evie-KF achieved 55.36% accuracy versus 49.19% with a random basis and was ahead on all five seeds. At 124M parameters, increasing Kronecker coverage from 6.0% to 68.8% improved test perplexity on all three seeds, lowering the median from 56.41 to 53.21 at the same and learning rate.

We will include the random-basis controls and the 124M coverage comparison in the revision. The anonymous repository’s guide at benchmark12/results/rebuttal/README.md indexes the scripts, per-seed results, and analysis commands.

Overall, I appreciate the authors' additional results and clarifications. The matched-budget comparisons and lagged-gate experiments address several of my requests. However, the coupling-bias claim remains unresolved for the full implementation; the overall improvement over AdamW is still not statistically significant, and the risk-sensitive interpretation remains only partially resolved.

I am maintaining my original rating (2: Weak rejection).

Reviewer_AhnzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Evie-KF is an optimizer that dampens noisy gradient directions by accounting for their correlations; when the compression strength is set to zero, it reduces to AdamW. Across 12 tasks, it ranked second out of eight, significantly outperformed Sophia, and on several tasks, achieved performance comparable to AdamW in fewer steps.

Critical strengths
  1. The idea seems interesting, with potential for reuse and further improvement.

  2. I liked some aspects of the paper’s presentation—for instance, Table 2 in section F; I would even recommend including it in the main body of the text, space permitting.

  3. The set of experiments is extensive.

Critical weaknesses
  1. The optimization theory (Theorem 6) appears quite simple and relies on strong assumptions.
  • The convergence theorem is proven in a fairly straightforward manner; the proof technique is similar to, for instance, the one used here:

[1] Doubly adaptive scaled algorithm for machine learning using second-order information

[2] Stochastic gradient methods with preconditioned updates

  • The assumption regarding the boundedness of vtv_t, combined with the smoothness assumption, seems strong (it is in fact boundness of the stochastic gradient). For example, in the convergence proof for Adam, one can dispense with the assumption that the gradient norm is bounded and rely solely on the smoothness property:

[3] Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed

  1. While the method does show a performance advantage over Adam, it does not outperform existing optimization methods; in particular, it lags behind Muon.

I would suggest that the authors consider combining their method with matrix-based optimizers. For instance, since Muon utilizes Adam in certain layers, it might be worth replacing Adam with this new method.

  1. The experimental results appear somewhat questionable, given how far they fall behind the state-of-the-art. While I understand this is due to a fixed training protocol (number of iterations, batch size, etc.), the gap is, in my view, too large. For instance, the authors report 55% accuracy for ResNet + CIFAR-100, whereas it is possible to achieve over 75+%. This discrepancy is so significant that it is difficult to draw any meaningful conclusions: all the methods perform poorly, and one can only say that one method is the "best among the worst." I would recommend making some adjustments to the experimental protocol.

  2. Additional computational costs arise both in theory and in practice. For language models, the layers are often rectangular—that is, n>mn > m. So 2nm2nm is better than n2+m2n^2 + m^2. In the experiments, overheads are also observed compared to AdamW. Do these overheads exist when compared to Muon?

To sum up:

  • Consequently, the theoretical component does not introduce any novel or compelling proof techniques, nor does it provide state-of-the-art convergence guarantees. While the paper’s contribution to theory is complementary, I do not find it sufficient to warrant acceptance.

  • Despite extensive experiments, the method’s performance appears weak compared to existing alternatives.

I like the core idea of ​​the paper; I would suggest that the authors consider how to "sell" it more effectively, as, in its current form, the paper offers only an idea that requires further refinement in several aspects.

Questions for authors

See Weaknesses

3 comments

Thank you for the detailed review. Your references informed our convergence analysis and your suggestions prompted a loader audit and a Muon hybrid experiment.

The gains over AdamW persist after correcting the CIFAR training protocol

The loader replayed first-epoch batches, including their random crops and flips. We fixed it and reran ResNet-18 for 30,000 steps, searching learning rate and weight decay for each optimizer. The table reports median test accuracy over three seeds.

DatasetEvie-KFAdamWMuonSGD
CIFAR-10071.93%70.12%72.62%72.31%
CIFAR-1093.35%92.64%94.21%93.36%

Evie-KF improves on AdamW on all six dataset-seed pairs, with median gains of 1.81 and 0.71 percentage points. With fresh shuffling on WikiText-2, Evie-KF improves on AdamW on all five seeds (median perplexity 90.47 vs. 94.94).

Standalone gains motivate further work on the Muon hybrid

We kept Muon on matrix parameters and replaced its auxiliary AdamW updates with Evie-KF at the same tuning budget.

WorkloadMetricMuonMuon with Evie-KF
WikiText-2Perplexity83.6987.21
WikiText-103Perplexity32.6832.71
enwik8Bits per character2.86782.8692
OpenWebText, 124MPerplexity53.3654.63

The hybrid was close on WikiText-103 and enwik8 and trailed on the other two tasks. Evie-KF is in its first implementation, while AdamW has a long history of refinement. With fixed and a matched learning-rate search, standalone Evie-KF beats AdamW on all four vision tasks and three of four language tasks. At 124M, it lowers median test perplexity by 25%, from 70.90 to 53.21, with gains on all three seeds. Tuning the gate for Muon’s auxiliary parameters could improve the hybrid.

Theorem 6 quantifies the gate’s advantage in the high-noise regime

The revised theorem gives a descent bound in the preconditioned norm without uniform bounds on stochastic gradients or conditional variance. It assumes a smooth objective bounded below, conditionally unbiased gradients with finite conditional covariance, and finite expected loss and noise terms. The idealized update omits momentum and norm rescaling. The gate and base preconditioner PtI use earlier steps, with 0</Ls. Any base preconditioner satisfying these conditions is covered.

The gate never increases total whitened noise before norm rescaling. Its strongest guarantee is in the high-noise regime. Let Vt=tr⁡(Σz,t)V_t=\operatorname{tr}(\Sigma_{z,t}). Since 0GtI, transmitted noise is at most Vt. For an nm block with exact marginal factors and >0, we obtain tr⁡(GtΣz,tGt)≤min⁡{Vt,nm2Vt/γ}\operatorname{tr}(G_t\Sigma_{z,t}G_t) \leq \min\{V_t, \frac{nm}{2}\sqrt{V_t/\gamma}\}.

For Vt>(nm)2/(4), the square-root bound is strictly tighter. At fixed , it grows as Vt while ungated variance grows linearly. In the correlated-noise quadratic experiment at =30, the Kronecker gate lowered final loss by 14.5% relative to the ungated update across 20 seeds, versus 2.0% for the diagonal gate. A batch-size sweep with noise measurements could test the predicted regime directly.

A variational derivation shows how the gate uses correlations that diagonal methods miss

In Adam-whitened coordinates, G=(I+z)-1/2 uniquely minimizes tr⁡(K−1)+tr⁡[K(I+γΣz)]\operatorname{tr}(K^{-1})+\operatorname{tr}[K(I+\gamma\Sigma_z)] over positive-definite K. In the whitened quadratic model with an isotropic gradient prior and correlated noise, the best matrix preconditioner achieves strictly greater expected one-step progress than the best diagonal one, even with oracle covariance and a tuned learning rate for the diagonal method.

Limited coverage improves on AdamW, and wider coverage lowers perplexity further

Kronecker factors use n2+m2 entries versus (nm)2 for a dense covariance, and exceed 2nm when nm. At 124M, 6.0% coverage yields 56.41 perplexity, 20% below AdamW at the same step budget for 15% more time per step (0.90 vs. 0.78 s). Muon takes 0.83 s. Peak memory is 15.3 GB for Evie-KF, 14.5 GB for AdamW, and 14.2 GB for Muon. At fixed and learning rate, covering every transformer-block matrix further improves perplexity on all three seeds (median 53.21), at 1.19 s/step and 17.5 GB.

The revision will foreground the corrected results and the gate’s guarantees

We will move Appendix F’s Table 2 into the main text alongside the corrected CIFAR and 124M results, disclose the loader issue, report the hybrid and both coverage settings with the corrected memory comparison, and revise Theorem 6 with explicit assumptions. We will lead with the gate’s variational characterization and exact AdamW recovery at =0, summarize the high-noise bound in the abstract, and identify cheaper wide coverage as a direction for further work.

Thanks to the authors for their response!

I will attempt to outline how the authors' response addressed the weaknesses I pointed out in my original review:

  1. I opened the paper and looked at Theorem 6. The assumptions for Theorem 6 still include an assumption that vt<Ginftyv_t < G_{infty} is bounded, which implies that the stochastic gradient is bounded. It is possible that I am mistaken or looking in the wrong place. It is also possible that the authors have not updated the paper.

At present, this weakness has not been resolved.

  1. Thanks to the authors for trying out my idea. Unfortunately, it didn’t work. It turns out that the superiority of the new method remains quite limited.

At present, this weakness has not been resolved.

  1. The results for CIFAR100 look better now. It’s strange that they aren’t included in the document (Table 1).

This weakness has been resolved.

  1. The authors agree with the fourth drawback. It is argued that the additional costs are small. It was clear to me from the first reading of the paper that the extra charge wouldn't amount to a doubling of the price—but the extra charge is there, and the improvements it brings are severely limited (see weakness 2).

At present, this weakness has not been resolved.

Two main weaknesses: 1 and 2, have not been addressed. I am maintaining the rating.

Thank you for your reply and for taking the time to revisit the paper. We appreciate your clarifying which concerns remain. We currently haven’t uploaded the revised manuscript because we are collecting any remaining reviewer questions so we can address them together and finally upload the revised PDF.

  1. You are looking at the correct theorem. The submitted PDF still contains the boundedness assumption and the replacement outlined is in our rebuttal which gives a descent bound in the preconditioned norm without uniform bounds on stochastic gradients or conditional variance. It requires finite conditional covariance and applies to an idealized update with a preconditioner formed from earlier steps, without momentum or norm rescaling.

We will replace Theorem 6 in the final revised PDF.

  1. The hybrid experiment was comparable to Muon. The clearest direct advantage we observed over Muon was on the Financials forecasting panel. Tuned Evie-KF completed 245/245 ticker-seed runs, compared with 213/245 for Muon. Evie-KF also achieved lower forecast error than Muon on the Financials tickers where all optimizers completed. AdamW completed all 245 runs, so the practical question is how many forecasts each optimizer can deliver at an acceptable error level for the same compute.

We will compare Evie-KF, Muon, AdamW, and SGD on a held-out forecasting window with matched tuning budgets and a common compute budget. In the revised PDF, we will report how many runs meet a prespecified forecast-error target, counting failed runs and retries against that budget.

  1. Thank you for noting that the corrected CIFAR-100 results address this concern. Those results were generated during the rebuttal and are currently reported in our response.

We will add the corrected CIFAR results to Table 1 in the final revised PDF.

  1. Your follow-up prompted us to check whether the extra work per step pays off at a common quality target. In the recorded 124M OpenWebText runs, Evie-KF with 6.0% Kronecker coverage reached validation perplexity 80 sooner than AdamW on all three seeds. Median times to the first recorded checkpoint at or below that target were 46.7 minutes for Evie-KF, 55.6 for AdamW, and 26.4 for Muon. At this target, Evie-KF saved about nine minutes relative to AdamW despite costing more per step. Muon reached the target fastest.

We will report this time-to-target comparison alongside the per-step costs in the revised PDF.

Reviewer_jWBUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel adaptive optimizer, Evie-KF, designed to address the limitation of traditional AdamW optimizers in effectively capturing correlations between parameters in gradient noise. The proposed method combines risk-sensitive control theory with Kronecker factorization to construct a matrix preconditioner based on gradient noise covariance, dynamically adjusting gradient update directions to improve training performance. Experimental results demonstrate that Evie-KF outperforms AdamW on most tasks and achieves faster convergence on several tasks.

Critical strengths
  1. The proposed method is interesting, combining risk-sensitive control theory with Kronecker factorization. By incorporating gradient noise covariance modeling into AdamW, it captures correlations in gradient noise across different parameters, thereby improving gradient update directions.

  2. The paper provides a relatively comprehensive theoretical analysis, including a closed-form solution to the Riccati equation, theoretical analysis of the Kronecker factorization, and a convergence proof for the idealized algorithm. Moreover, the proposed method reduces exactly to AdamW when the risk parameter is set to zero.

Critical weaknesses
  1. Limited validation at larger model scales. The experiments are primarily conducted on relatively small-scale models, and the effectiveness and scalability of the proposed method for large language model pretraining remain unverified.

  2. Limited statistical significance of performance improvements. Although Evie-KF outperforms AdamW on 8 out of 12 tasks, the overall improvement is not statistically significant after correction for multiple comparisons.

  3. Higher computational overhead and potential concerns about comparison fairness. Evie-KF requires more wall-clock time than AdamW in the main vision and language experiments. Additionally, Evie-KF uses a larger hyperparameter search budget, which may affect the fairness of comparisons with other optimizers.

Questions for authors
  1. Although Evie-KF outperforms AdamW on 8 out of 12 tasks, the overall improvement is not statistically significant after correction for multiple comparisons. Could the authors further explain this result and provide additional evidence demonstrating that Evie-KF consistently outperforms AdamW?

  2. The largest model evaluated in the paper has only 33.7M parameters. Could the authors further discuss the scalability of Evie-KF to the pretraining of larger language models?

1 comment

Thank you for your careful review. We appreciate your recognition of the gradient-noise model, the closed-form Kronecker preconditioner, and the exact recovery of AdamW at =0. We agree that the original submission would benefit from evaluation at a larger model size, a matched tuning budget, and a fuller account of the variability and cost of the gains over AdamW.

The gains over AdamW extend to 124M parameters

We extended the evaluation from 33.7M to 124M parameters, training GPT-2-small on OpenWebText for 6,000 steps. With fixed at 30, Evie-KF reduced median test perplexity from 70.90 to 53.21, a 25% relative reduction. Evie-KF improved on AdamW on each of the three seeds.

OptimizerWeights covered by the Kronecker gateMedian test perplexity ↓
AdamW—70.90
Evie-KF, factor cap 10246.0%56.41
Evie-KF, factor cap 307268.8%53.21

We held and the selected learning rate fixed while raising the factor-size cap. This extended the Kronecker gate to every transformer-block matrix. Perplexity improved on all three seeds.

We will add the 124M results to the paper, including the training budget, the fraction of weights covered by the gate, and the measured time and memory costs.

The gains persist across seeds and with fresh data loading

Excluding seed 42, which was used for tuning, increases the observed advantage from 8 to 10 wins out of 12 tasks and the median relative gain from 3.3% to 4.0%. One of the 10 wins is a near-tie. The two-sided task-level Wilcoxon test gives p=0.13 before multiple-comparison correction.

Evie-KF improves on AdamW on every evaluated seed in each comparison below.

WorkloadTest metricEvie-KFAdamWSeeds favoring Evie-KF
CIFAR-100, fresh augmentation, 30k stepsAccuracy ↑71.93%70.12%3/3
CIFAR-10, fresh augmentation, 30k stepsAccuracy ↑93.35%92.64%3/3
WikiText-2, fresh shufflingPerplexity ↓90.4794.945/5
OpenWebText, 124M parametersPerplexity ↓53.2170.903/3

The aggregate test uses the task as its unit of analysis, giving it 12 observations. With more time and compute, we could evaluate a larger, prespecified set of tasks and run more seeds per task. More seeds would improve the precision of individual task estimates, and more tasks would increase the power of the aggregate comparison. A study of that scale is beyond the time and compute available to us currently.

In the revision, we will report the results with and without the tuning seed and include paired uncertainty estimates.

A matched tuning budget preserves gains on vision and language

We agree that Evie-KF’s larger search budget made the original comparison harder to interpret.

  • Matched learning-rate search across all 12 tasks. We fixed at 30 and used the baselines’ seven-point learning-rate budget. Evie-KF improved on AdamW on all four vision tasks and three of four language tasks. It won 8/12 under the paper’s complete-case ticker rule and 7/12 using all paired finance tickers.
  • Expanded baseline search on WikiText-2. We gave four baselines 49 trials each. Their median test perplexities changed by at most 1.6 points on the same three seeds.

We will include the matched-budget comparison in the main results and report both finance ticker rules.

Wider Kronecker coverage improves perplexity at a measured cost

We agree that the additional compute is material. At 124M parameters, Evie-KF takes 1.19 seconds per step versus 0.78 for AdamW and uses 3.0 GB more peak memory. In return, it delivers 25% lower median test perplexity at the same training step and data budget, with gains on all three seeds. It also gives practitioners a way to control how much of the model uses structured gradient-noise information.

We will report both settings in the revision so readers can choose where to trade additional compute for better model quality.

Structured gradient-noise modeling builds on AdamW

The exact recovery of AdamW at =0 provides a clear reference for assessing the addition of the gradient-noise gate. The gains over AdamW persist with fresh data loading on CIFAR and WikiText-2. At 124M, expanding the gate to every transformer-block matrix improves perplexity on every seed at the same and learning rate.

We will emphasize these findings in the revision because they quantify the benefit of wider Kronecker coverage.

The anonymous repository’s rebuttal guide at benchmark12/results/rebuttal/README.md indexes the training scripts, per-seed results, tuning records, and commands for reproducing the analyses.

Comments on the paper

3 comments

Thank you to all three reviewers for the detailed reviews and follow-up discussion. We appreciate the time taken to examine the method, suggest additional experiments, and identify where the analysis and presentation needed greater precision.

We have uploaded a revised manuscript incorporating the additional experiments, corrected protocols, and expanded analysis. The paper now makes Evie-KF’s contribution more explicit, a structured gradient-noise extension of AdamW that recovers AdamW exactly at = 0.

The revision strengthens this contribution in three ways.

  • Demonstrated improvements over AdamW. With a fixed mean-normalized =30 and a matched learning-rate search budget, Evie-KF improves on AdamW on all four vision tasks and three of four language tasks. Fresh augmentation on CIFAR and fresh shuffling on WikiText-2 preserve gains across every evaluated seed.

  • Benefits that extend to 124M parameters. On OpenWebText, Evie-KF reduces median test perplexity from 70.90 to 53.21, a 25% reduction, at the same 6,000-step training and data budget. Increasing Kronecker coverage improves every seed at fixed learning rate and . The smaller-cap setting also reaches a common validation target sooner than AdamW despite its additional per-step work.

  • A principled operator with direct implementation evidence. The gate has a unique variational characterization, population noise-transmission guarantees, and an exact AdamW reference at zero gating strength. The new checkpoint diagnostic adds empirical evidence of lower directional variance in the implemented normalized update.

Together, these results support Evie-KF as a useful extension of AdamW and identify a concrete direction for further development, making wider structured covariance coverage cheaper.

Our accompanying comments describe the manuscript changes and summarize how the rebuttal addressed each review. The anonymous repository’s guide at benchmark12/results/rebuttal/README.md indexes the released experiments and checkpoint reproduction files.

AuthorsOpenReview ↗

Revisions from the rebuttal

We have incorporated the following changes into the revised manuscript.

  • Larger-scale results and wider covariance coverage — Section 5, Table 3. We added GPT-2-small experiments with 124M parameters on OpenWebText. At the same 6,000-step training and data budget, Evie-KF reduces median test perplexity from AdamW’s 70.90 to 53.21, a 25% reduction, with improvements on all three seeds. Increasing Kronecker coverage from 6.0% to 68.8% improves perplexity on every seed while holding the learning rate and fixed. This provides a controlled demonstration of the benefit of extending the structured gate to more of the model.

  • Training quality and measured computational cost — Section 5, Table 3 and Figure 6. We report both per-step costs and validation progress against elapsed training time. At 6.0% coverage, Evie-KF reaches validation perplexity 80 sooner than AdamW on all three seeds. Median recorded times are 46.7 versus 55.6 minutes, an 8.9-minute saving despite the additional work per step. The wider setting further improves final test perplexity. The revision identifies the H100 MIG hardware and distinguishes the checkpoint-based time comparison from final test performance.

  • Corrected data loading and matched tuning budgets — Section 5, Table 2 and Figure 5. With fresh augmentation and 30,000 training steps, Evie-KF improves on AdamW on all three seeds for both CIFAR datasets. Median accuracies are 93.35% versus 92.64% on CIFAR-10 and 71.93% versus 70.12% on CIFAR-100. Fresh shuffling on WikiText-2 preserves the improvement on all five seeds, with median perplexity 90.47 versus 94.94. With mean-normalized =30 and the same seven-point learning-rate search budget, Evie-KF improves on AdamW on all four vision tasks and three of four language tasks. Section 5 reports the aggregate tests, including the nonsignificant overall comparison with AdamW.

  • Direct measurements of the implemented update — Section 5 and Appendix G.5, Table 14. At six checkpoints covering eigensystem-refresh and cached steps on a fresh WikiText-2 trajectory, the gated directions have 35.0–63.4% lower measured first-order loss variance than the gamma-zero directions at the same model and moment state, at equal whitened step length. All six pointwise 95% bootstrap intervals favor gating. Independent probe batches evaluate the directions separately from the batches supplying the gate. Projected mean shifts under an independent gate-input control are 0.26–0.47% of baseline step length at the sampled refresh steps. The appendix states the remaining dependence, inactive clipping in the probe batches, and single-trajectory scope.

  • Revised theory — Section 3.5, Theorem 6 and Appendix C.10. The replacement theorem is now in the PDF. It gives a descent bound in the preconditioned norm without uniform bounds on stochastic gradients or conditional variance. It states finite-covariance and integrability conditions for an idealized update with predictable preconditioning. The gate-specific analysis includes its unique variational characterization in Section 3.2 and Appendix C.4, and the population noise-transmission bound from exact covariance marginals in Appendix C.13. Appendix C.10 discusses the preconditioning and clipping references suggested in the review.

  • Promised presentation and supporting analyses — Table 1 and Appendices G.6–G.8. We moved the optimizer-positioning table from the original Appendix F into the main text as Table 1, as suggested by Ahnz. Appendix G.6, Tables 15–17 report tuning-seed sensitivity, paired uncertainty and expanded baseline searches. Appendix G.7, Table 18 reports the lagged-gate and random-basis controls. Appendix G.8, Table 19 reports the suggested Muon-hybrid experiment. Financials follow-ups appear in Appendix G.1, Tables 6 and 7. Appendix H describes the released code and reproduction procedures.

These changes make the empirical benefit, computational trade-offs, and scope of the guarantees explicit while retaining the controlled comparisons that demonstrate what structured gradient-noise information adds to AdamW.

For completeness, we summarize how the rebuttal and revised manuscript address the three reviews.

Reviewer jWBU

We extended the language-model evaluation to 124M parameters, added measured coverage, time and memory comparisons, and tested Evie-KF with the baselines’ learning-rate search budget. The corrected CIFAR and fresh-shuffling WikiText-2 runs improve on AdamW on every evaluated seed.

Excluding tuning seed 42 from the original benchmark strengthens the observed advantage from 8/12 to 10/12 tasks, with one near-tie, and increases the median relative gain from 3.3% to 4.0%. The two-sided task-level Wilcoxon value is p=0.13 before multiple-comparison correction. Appendix G.6, Tables 15–17 reports the tuning-seed comparison, paired uncertainty and expanded baseline searches. These analyses support repeatability on the demonstrated workloads while defining the scope of the aggregate claim.

Reviewer Ahnz

The replacement Theorem 6 is now in Section 3.5, with its proof and discussion of the three suggested references in Appendix C.10. The optimizer-positioning table is now Table 1 in the main text.

Corrected CIFAR-100 results reach 71.93% for Evie-KF versus 70.12% for AdamW, improving on every seed. We also ran the suggested Muon auxiliary-group substitution and report it in Appendix G.8, Table 19. That tested combination did not improve Muon. The direct AdamW extension provides the clearest demonstrated benefit.

To address the follow-up on computational cost, we added the recorded time-to-target comparison showing an 8.9-minute median saving over AdamW at validation perplexity 80. Appendix G.1, Tables 6 and 7 reports the matched-budget Financials follow-ups, including completed-run forecast errors and the number of usable forecasts obtained within the common training-compute budget. The paper also corrects the distinction between unusually high-error Muon runs and training failures, and accounts for the additional factor and eigenbasis memory on rectangular layers.

Reviewer yDcQ

We replaced the overly broad decoupling claim with the conditional-centering statement and the assumptions needed for independence. The measured lagged-gate and random-basis controls appear in Appendix G.7, Table 18. The lagged gate preserves the observed performance on WikiText-2 and Finance-Tech.

Expanded 49-trial baseline searches on WikiText-2 change median perplexity by at most 1.6 points on the same three seeds. Appendix G.6, Tables 15–17 reports these searches alongside tuning-seed sensitivity and paired uncertainty. The matched-budget comparison in Section 5 reports both finance ticker rules.

In response to the follow-up about the full implementation, the new Appendix G.5, Table 14 directly measures its update directions. It finds lower directional variance at all six checkpoints and quantifies residual coupling at refresh steps. The revision also distinguishes the Riccati derivation, the variational characterization and the variance guarantees. The overall task-level AdamW comparison remains nonsignificant, and the implementation’s remaining dependence is stated explicitly.

The resulting case for Evie-KF rests on complementary findings: matched-budget gains over AdamW in vision and language, gains that persist under corrected data loading, improved quality from wider Kronecker coverage, and measured variance reduction in implemented directions.

We hope the revised manuscript and additional evidence address the concerns raised during review. We would be grateful if the reviewers would reconsider their assessments in light of these changes, particularly the matched-budget gains, the 124M results, and the direct measurements of the implemented update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.