- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Evie-KF, an AdamW-based optimizer using Kronecker-factored gradient-noise covariance. Across 12 tasks, it outperforms AdamW on 8 but shows no statistically significant overall advantage. Kronecker ablations yield significant gains of 6.75% on Finance-Tech and 6.67% on WikiText-2. On OpenWebText (33.7M parameters), Evie-KF improves over AdamW but trails Muon.
Critical strengths
-
The method has a well-specified AdamW nesting property, explicit Kronecker algebra, and numerical implementation checks.
-
The ten-seed structural ablation compares Kronecker and diagonal gates under matched spectral normalization, showing Holm-significant gains on Finance-Tech (+6.75%) and WikiText-2 (+6.67%).
Critical weaknesses
The coupling-bias claim is stronger than Proposition 4 proves.
=0 does not imply independence of the nonlinear gate based on and the gradient, particularly for skewed noise, same-step Adam statistics, and gradient-dependent clipping.
Fairness and selection may alter the ranking.
Evie-KF searches 49–99 hyperparameter points per task versus approximately 7–9 for most baselines, while the fixed- study covers only four selected hosts. Finance further conditions the test sample on every arm completing every seed, retaining only 23/49 Finance tickers and 37/49 Tech tickers. Full-budget-matched and full-panel sensitivity analyses are needed.
The risk-sensitive interpretation is not established for the implemented update.
The reported gate uses the optimistic process-noise sign branch, while the separate update-noise risk criterion does not derive the matrix gate; norm preservation may increase directional gains .
Questions for authors
Split-batch independence: Proposition 4 establishes , but this does not generally imply independence of the nonlinear preconditioner. Can the authors clarify under what assumptions the coupling-bias claim remains valid, particularly with same-step Adam whitening and gradient clipping?
Fairness and statistical robustness: Using the existing experimental results, can the authors report the Evie-KF versus AdamW comparison excluding seed 42 (used for tuning), with paired uncertainty estimates? Can they also clarify how the unequal tuning budgets (49–99 versus 7–9 trials) affect the conclusions?
2 comments
Overall, I appreciate the authors' additional results and clarifications. The matched-budget comparisons and lagged-gate experiments address several of my requests. However, the coupling-bias claim remains unresolved for the full implementation; the overall improvement over AdamW is still not statistically significant, and the risk-sensitive interpretation remains only partially resolved.
I am maintaining my original rating (2: Weak rejection).
Thank you for your careful review. We appreciate your recognition of the exact recovery of AdamW, the Kronecker formulation, and the matched-normalization ablations.
The gains over AdamW persist without the tuning seed and at a matched budget
We excluded tuning seed 42 from the original comparison. We also reran Evie-KF on all 12 tasks with fixed =30 and the baselines’ seven-point learning-rate budget.
The p-values are two-sided and unadjusted. Excluding seed 42 strengthens the observed result, with one near-tie among the 10 wins. At the matched budget, Evie-KF improves on AdamW on all four vision tasks and three of four language tasks under either ticker rule.
We will report the fixed- comparison in the main results, include both finance ticker rules, and provide paired uncertainty estimates with and without seed 42.
Split-batch sampling gives zero covariance, and the lagged gate preserves the gains
We agree that zero covariance alone is insufficient to establish independence. Let g be the half-batch mean and the half-batch difference divided by two. Exchangeability gives E[δ∣g]=0. With independent Gaussian half-batch noise, g and are independent. With skewed noise, a gate built from can still couple to g. Using earlier steps to estimate the gate removes its direct dependence on the current noise sample. Same-step Adam whitening and gradient-dependent clipping remain possible sources of coupling in the full update.
The lagged gate closely matched Evie-KF on both ablation hosts, suggesting that the gains did not require the current split-batch noise sample.
We will replace “provably decoupled” with the precise split-batch statement and add the lagged control to the ablation table.
The gate has a variational derivation and reduces noise variance at fixed step length
The optimistic process-noise branch gives a derivation of the gate. The revised analysis fixes the reference direction and noise covariance in whitened coordinates. At the same step length, the variance of the first-order loss change is at most that of the ungated step and is nonincreasing in . Norm preservation can amplify some directions. The gain in the noisiest eigendirection remains at most one.
The gate also has a variational characterization independent of the Riccati model. It is the unique positive-definite minimizer of the AdaGrad-style objective
tr(K−1)+tr[K(I+γΣz)].
Its solution is K=(I+z)-1/2, where z is the centered gradient-noise covariance in Adam-whitened coordinates.
In the revision, we will present the variational characterization alongside the Riccati derivation, state the variance guarantee for the norm-preserving update, and clarify the scope of the separate update-noise criterion.
The new ablations show where the learned basis and wider Kronecker coverage help
Exact recovery of AdamW at =0 gives a clear reference for evaluating the gate. The matched-normalization ablations show that the matrix gate can improve on a diagonal gate.
A random eigenbasis gave similar results on WikiText-2 and Finance-Tech. On CIFAR-100, with the original loader and the learning rate selected for the learned basis, Evie-KF achieved 55.36% accuracy versus 49.19% with a random basis and was ahead on all five seeds. At 124M parameters, increasing Kronecker coverage from 6.0% to 68.8% improved test perplexity on all three seeds, lowering the median from 56.41 to 53.21 at the same and learning rate.
We will include the random-basis controls and the 124M coverage comparison in the revision. The anonymous repository’s guide at
benchmark12/results/rebuttal/README.mdindexes the scripts, per-seed results, and analysis commands.