acceptodds
Under review as a conference paper at ICLR 2027

Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss

Abstract

Many modern applications of deep learning involve training a neural network via a one-step prediction loss (e.g., regression, cross-entropy), but deploy the network by rolling out along its own predictions. Key examples include autoregressive language modeling, flow-based generative modeling, and robot policy learning. It is well-documented that these settings induce a phenomenon we call **test-time feedback** (TTF): the mismatch between the training/validation loss and downstream metrics of interest, such as task success rate and generation quality, which grows with task length. While data curation, architecture, and objective design have been proposed to combat train-test shift in TTF settings, this paper proposes *optimization* as a new design axis to mitigate error accumulation. Specifically, we introduce a new optimization paradigm called **double-preconditioning** () uniquely tailored to the challenges of TTF. combines gradient-wise preconditioning, as in and , with activation-wise preconditioning (), such as in . We show that the addition of yields a *drop-in intervention* for increasing downstream model performance across a range of TTF settings. Interestingly, these gains in test-time performance *do not consistently accompany improvements in validation loss*, opening new questions about how to properly evaluate models trained with one-step supervised objectives.

Primary area:
optimization
Keywords:
deep learning, optimizers, optimization, preconditioning, KFAC, feature learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GXiiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the distribution drift problem common in neural odes, learning to control and sequence modeling problems, where the one step training objective does not match the actual performance metrics at test time, and error accumulation can be catastrophic when the prediction horizon is long. Unlike most existing work that tries to address the issue by introducing new training objectives, the paper takes an interesting approach by studying the role of optimization. The claim made here is that activation conditioning plays a crucial role in test time performance, and poor conditioning in the activations can lead to good in-distribution loss but bad rollout results. The proposed fix is an additional activation covariance preconditioning step, essentially encouraging the learned features to spread out more evenly. The idea therefore shares some intuition with DRO in my opinion and can potentially mitigate online prediction issues without fundamentally changing the loss. Numerical results seem consistent with the proposal.

Critical strengths

The strongest aspect of the work in my view is the empirical results part. The method is tested across several substantially different settings and optimizers, and there is a fairly consistent improvement in test time performance. The main claim of the paper does seem to validate itself even though the performance gain can be marginal at times, The computation overhead and limitations discussion is also very much appreciated and I thank the authors for their honesty.

Critical weaknesses

To me the theoretical results are the weakness of the work and several places need more justification than what's currently given, and I think the actual novelty needs to be separated much better. A couple things here:

A major portion of the results is essentially analyzed in a linear quadratic setting. Can this simplification be justified? I personally don't see how such trivialization can be made naively as a blanket statement, especially as written in Propositions 3.1 and 3.2. I actually believe the claim to be reasonable even for the actual nonlinear problems, but the argument is more heuristic than proof to me.

I also think the authors should make it clear in terms of novelty. Much of the LQ problem derivation is fairly standard in the control literature, and I don't see much new there. A key contribution could be the bridge between the standard LQ setup and the actual nonlinear setting the paper tries to tackle, yet it's largely missing here.

Similarly, Proposition 4.1 is explicitly adapted from prior feature learning results and establishes a local, one step contraction under fairly restrictive assumptions. What is the genuinely new theoretical statement obtained by combining these pieces? Saying that the test time performance is a mismatch to the training objective and showing the proposed scheme is reasonable in a linear setting are both okay, but not really of much value. The key question to me is actually not answered here, despite the extensive setup.

Questions for authors

While the numerical results of the paper are very much self-consistent, I cannot help but notice that these improvements are often quite marginal. The paper acknowledges the existence of a common line of work targeting the same issue through modifications to the training objective, and these methods would usually lead to much more significant improvements in test-time results. I do not think it is fair to ask for a side-by-side comparison given the limited time, but I would like to see some insight into the differences. My rough guess is that while the additional preconditioning can change the optimization, it does not have the necessary information about πideal\pi_{ideal}, and even according to the paper's own proposition, that discrepancy can be essentially unbounded.

I also have a question regarding the paper's proposed method and how it relates to the idea of natural gradient methods, which have seen an abundance of applications and adoption in modern machine learning. If I remove the language on the mismatch between test time metrics and the training objective,, since we never have access to π∣ideal\pi_|{ideal}, doesn't the proposed method essentially default to a natural gradient method? What remains novel in this paper from a methodological perspective? I think the paper made some comments on this in the appendix, but I would like to see it spelled out more explicitly, as I do not think this is merely an issue of related work and I do have some concerns on the claimed novelties of the paper.

2 comments

We thank you for your review: we address your questions sequentially below.

Purpose and novelty of the linear analysis. We emphasize that we chose a simple linear system intentionally to provide a familiar setting that achieves the following in a readable manner: 1. demonstrate what TTF looks like precisely and quantiatively, 2. explicitly show that feature learning can be extremely suboptimal even in this simple setting, 3. show how DoPr naturally arises from this setting. We also remark that presenting these concepts involves some novel contributions, for example the existence of two policies where one achieves much lower val loss than the other, but leads to much worse test-time error on the same system.

Standard extensions, e.g., to generalized linear models, can be performed, but we did not think these meaningfully provided to our conceptual contributions. Notably, we emphasize that this paper is not intended as a theoretical contribution, and our various results are purely in service of making TTF-induced distribution shift and impaired feature learning concrete. However, we believe we are the first to quantitatively characterize TTF as an optimization phenomenon.

Novelty of Proposition 4.1 As aforementioned, the core purpose of our paper is not theoretical generality. We explicitly tie Proposition 4.1 to prior work in Zhang et al. (2024). The purpose of Proposition 4.1 is to make explicit that raw (S)GD is a poor feature learner, and to impart the correct equivariance to TTF-induced distribution shift is AP.

However, we do note that this two-layer set-up is conceptually rich. For example, multi-layer settings can be viewed as performing linear regression from the current layer's activations to the loss derivative. This viewpoint has been explored in prior optimization literature, which we documented in the Extended Related Work, and is in fact the exact same motivation for linear-attention mechanisms such as DeltaNet and MesaNet, the latter of which can be viewed as "AP"-ed DeltaNet. However, in the optimizer literature, we find that methods motivated from layer-wise regression alone (such as AP-only) tend to be numerically unstable, which is what motivates our core practical contribution in Double Preconditioning.

Comparison with objective-based interventions. This is an interesting question; firstly, we remark that we present an optimizer-based intervention as it is in principle compatible with any training objective. Secondly, in large-scale LLM training, many regularization tricks aimed at reducing exposure bias (a form of TTF) from earlier sequence modeling literature did not survive the test of scale, and modern LLM training pipelines tend to have maintain the simplest training objective (e.g. next-token cross-entropy). Furthermore, our aim is to provide an application-agnostic intervention, where objective modifications tend to be application or even set-up dependent.

We remark that spiritually algorithms like DAgger (i.e. OPD or OPSD in LLM post-training parlance) can be viewed as an implicit modification to the loss objective, as it changes the distribution the loss is evaluated on. However, these come at an extremely significant overhead during training due to having to iteratively sample trajectories during training---this may not even be feasible depending on application, such as robotics. Thus, it is hard to perform apples-to-apples comparisons in terms of test-time gains from changing the sampling distribution versus changing the optimizer.

However, we remark that for our robot policy learning experiments, the performance gain on the hard environments tool-hang and transport from switching to DoPr in fact outsizes the gain from switching between the best and next-best architectures and loss objectives in the paper we based our set-up on [1].

Limits without access to the deployment distribution. Indeed, preconditioning cannot recover arbitrary information absent from the offline data, and we do not claim that it can avoid worst-case compounding error. For example, [2] constructs continuous-control settings in which smooth, deterministic policies learned solely from expert data can exhibit exponential-growing TTF no matter how the policy is parameterized/optimized. In fact, the core counterexample comes from a linear system construction. In this case, there is no recourse but to modify where the training data is sampled, e.g. training-time corrective sampling via DAgger.

Our claim is that optimizer-induced feature-learning failures are an additional, avoidable source of poor deployment behavior. DoPr targets that source using the information already available in the training activations. It does not require knowledge of πideal\pi_{\mathrm{ideal}}, nor does it provide a universal bound on the discrepancy from that distribution.

Relationship to natural gradient and methodological novelty.

This is another great, nuanced question. There are many conceptual connections to the Natural Gradient Method, and we document the long lineage in our extended related works. Firstly, full-matrix natural gradient should in principle address the same feature learning issue as AP. We make this connection evident in Proposition 4.2, where AP provides a targeted parameterization-invariance that can be achieved by a layer-wise operation---recall that full natural gradient is invariant to any smooth re-parameterization. This said, all modern computationally-efficient approximations of natural gradient are motivated from a training convergence perspective. Furthermore, only the KFAC line of work (which is in modern practice unstable or suboptimal) performs a faithful approximation to the true natural gradient, whereas all other modern gradient-based optimizers (Adam, Shampoo, Muon, SOAP) can be viewed as approximations to the empirical natural gradient. Our work can be viewed as providing a mediation of these perspectives: activation-side-only preconditioning is a valid approximation to the true Fisher Information that does not require the re-sampling/additional backwards passes that KFAC does, while preserving precisely the parameterization-invariance relevant to TTF: since TTF causes a distribution shift in the inputs, distribution shift propagates layerwise through the activations. The combination with GP then inherits the well-conditioning and training stability afforded by gradient-based optimizers.

We also remark that some works have demonstrated that full natural gradient can confusingly underperform its factored counterparts in neural network training [3], further motivating practical developments such as the combination of AP and GP presented in our work.

[1] Pan et al. “Much Ado About Noising: Dispelling the Myths of Generative Robotic Control” 2026.

[2] Simchowitz et al. “The Pitfalls of Imitation Learning when Actions are Continuous” 2025.

[3] Benzing et al. "Gradient descent on neurons and its link to approximate second-order optimization" 2022.

Reviewer_2q4fOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch between training loss and performance when models are deployed over multiple steps for example in autoregressive language generation and robot control. It proposes Double Preconditioning (DoPr) that combines activation-based preconditioning with existing optimizers such as AdamW and Muon. The authors further provide a theoretical motivation based on feature learning and distribution shift. They evaluate the method on continuous control, robot imitation learning, LLM fine-tuning and image generation.

Critical strengths
  1. Lower validation loss does not always mean better performance. This is especially true when a model is deployed and using its own predictions. In particular, LLMs and robot policies are places where this holds and errors can accumulate over time. The authors provide a mathematical analysis connecting poor feature learning to distribution shift during deployment.
  2. The discussion of activation covariance and feature learning offers a useful explanation as to why standard optimizers can produce models with similar validation loss but different downstream performance. The proposed method is relatively simple and can be combined with existing optimizers without the need to change objectives, architecture or dataset. The theoretical analysis also motivates extension to different layer types.
  3. I think another really strong positive is that the method has been evaluated across substantially different tasks rather than focusing on a single benchmark. The robotics experiments show clear improvements in downstream tasks. The Transport experiment demonstrates the method can improve on a trained policy even when training loss does not improve. The LLM experiments are also encourgaing with meaningful improvements. Further the method can improve stability across learning rates and this is very useful in practice as optimizers are often sensitive to hyperparameter choices.
Critical weaknesses
  1. There needs to be a clearer distinguishing of the work with prior work. Combining activation-based preconditioning with gradient-based optimization is closely related to existing work on KFAC and other optimizers. Whilst the motivation is interesting, the contribution is not fully clear to me. In particular, the Newton-Muon optimizer [1] independently derives the same core update as DoPr-Muon. The authors acknowledge this in the appendix, but there is no comparison of the methods experimentally. There are also other recent works such as SOAP (ICLR 2025) [2] and From SGD to Muon [3] that study ways of improving gradient updates through more structured optimization. Although the objectives and update rules differ, a more detailed comparison would help clarify what is new about the proposed method.
  2. Whilst the experiments cover multiple domains, the evidence or gains is not of the same strength in all settings. In the LLM experiments, the improvements are modest especially on GSM8K and MATH-500. Some learning rates also perform worse with DoPr. The large gains at specific learning rates therefor mainly demonstrate improved robustness to aggressive learning rates rather than consistently large gains. The additional computational cost also needs more attention. A comparison based on equal wall-clock time or total compute would help establish the practical advantages.

[1] https://arxiv.org/pdf/2604.01472 [2] https://openreview.net/pdf?id=IDxZhXrpNf [3] https://arxiv.org/pdf/2605.19781

Questions for authors
  1. How does the method compare directly with Newton-Muon and activation-preconditioning-only methods under the same experimental setup?
  2. Can authors report downstream performance at matched wall-clock training time or total compute, including overhead of activation preconditioning?
  3. Are improvements on the 8B LLM experiments consistent across seeds? The gains are relatively marginal. How do these results compare when using the best validation-selected learning rate for each method?
  4. Could the authors clarify the Wasserstien-2 identity in proposition 3.1? The equality involving the Frob norm of covariance square roots generally needs addtional conditions such as commuting covariance matrices.
2 comments

We thank you for your review: we address your questions sequentially below.

Novelty relative to KFAC, SOAP, and Schatten-pp methods. We highly encourage the reviewer to refer to our extended related works in Appendix A, namely the "Fisher Information, KFAC, and prior occurrences of Activation Preconditioning" and "Adaptive gradient optimizers in deep learning" sections. We summarize how KFAC, Newton-Muon, and SOAP relate to our work here; we appreciate the pointer to [3] and will add this to our related works.

Our novel contributions are a few-fold:

  1. Across all the aformentioned optimizers, they were all designed with accelerating training loss convergence in mind. First and foremost, DoPr is a conceptual contribution demonstrating that feature learning robust to TTF is a key design consideration, which can demonstrably be orthogonal to improving loss convergence.

  2. Regarding KFAC in particular, DoPr is a contribution two-fold: first we find that only the "right-side" activation preconditioner is necessary, and we do not need "left-side" derivative preconditioner in practice. This is supported by our Proposition 4.2, where we show distribution shift propagating layer-wise from the input can be addressed by activation-preconditioning. Furthermore, DoPr crucially introduces gradient preconditioning; KFAC alone has been long known to be a suboptimal and numerically unstable optimizer in modern applications.

  3. Regarding Newton-Muon, we first remark that we are in contact with the authors, and our works are concurrent. Secondly, we remark that our generalized DoPr construction is meaningful. For example, we ran Adam versus DoPr-Adam in the NanoGPT speedrun set-up of their paper, and the gain is very significant, which we show below. However, we reiterate that our paper is not intended for improved validation loss performance. Lastly, we could not reliably reproduce the validation loss improvements documented in Newton-Muon; notably, the performance is sensitive to a heuristic "refresh" hyperparameter dictating the frequency of re-estimating the activation preconditioner. Under the theoretically correct setting of Newton-Muon's (and our DoPr-Muon) construction of refresh=1, the validation loss performance of Newton/DoPr-Muon is slightly worse than baseline Muon, which is consistent with our message that DoPr need not improve in-distribution val loss.

  4. Regarding SOAP, and the Schatten-pp optimizers in "From SGD to Muon...", these are precisely within the class of Gradient Preconditioners as coined in our paper: they adjust the update direction using solely layer-wise batch gradient information. Thus, these are in principle valid candidates to combine with DoPr, e.g. DoPr-SOAP! In fact, the Schatten-pp optimizers are derived from the same Linear Minimization Oracle (LMO) framework that we adapt in Section 4. We think this synergy between Activation- and Gradient-preconditioning is worth further exploration. We note a practical caution for SOAP is its already large memory requirement, as it stores approximately 3x the memory of Adam due to its left- and right-side covariances and the corresponding eigenbases.

Relationship to Newton-Muon and direct comparisons.

Regarding Newton-Muon [1], we first remark that we are in contact with the authors and have verified our works were concurrent. Secondly, we remark that our generalized DoPr construction is meaningful. For example, we ran Adam versus DoPr-Adam in the NanoGPT speedrun set-up of their paper, and the gain is very significant, which we show below. However, we reiterate that our paper is not intended for improved validation loss performance. For example, we could not reliably reproduce the validation loss improvements documented in Newton-Muon; notably, the performance is sensitive to a heuristic "refresh" hyperparameter dictating the frequency of re-estimating the activation preconditioner. Under the theoretically correct setting of Newton-Muon's (and our DoPr-Muon) construction of refresh=1, the validation loss performance of Newton/DoPr-Muon is slightly worse than baseline Muon. These results reinforce the distinction between improving pretraining loss and improving downstream behavior under feedback.

OptimizerlrVal loss, mean ± sd (n=3)
AdamW5.5e-43.4643 ± 0.0060
DoPr-AdamW5.5e-43.3874 ± 0.0022
Muon0.053.2788 ± 0.0014
Newton-Muon / DoPr-Muon (refresh=1)0.053.2884 ± 0.0013

LLM gains, learning-rate robustness, and validation-based selection.

We note that the primary purpose of the LLM experiments is not to show per-LR dominance of DoPr's downstream performance. For example, typical SFT pipelines use a very conservative learning rate --- it is well-documented that small learning rate SFT equalizes performance across optimizers, e.g. Adam vs Muon. This is due to the empirical observation that large-LR SFT causes catastrophic damage to prior capabilities. Our results reproduce this effect for the baseline: notably, the performance collapse is not due to unstable training: large LR training does lead to optimal validation loss. However,

Computational overhead and matched-compute performance.

We refer the reviewer to Appendix B.6 for derivations of the computational overhead. In summary, the added cost of doing activation preconditioning amounts to around ~1.25x for large batch training, which is in practice smaller (e.g. we documented around ~1.18x overhead in LLM training), and can be made even smaller for modern LLM architectures that share activation streams, e.g. multi-head attention. For the continuous control and robot policy experiments, we remark that the peak performance is reached early in training, and thus DoPr performance is dominant even when adjusting for computational overhead. For the SFT experiments, we remark that they are typically considered data-constrained rather than compute-constrained, and most methods are benchmarked by token or epoch count (e.g. 2 epochs in our set-up).

Wasserstein-2 identity in Proposition 3.1. We thank the reviewer for pointing this out. The correct statement should read $W_2^2(N(0,\Sigma_1),N(0,\Sigma_2)) = \mathrm{tr}(\Sigma_1+\Sigma_2-2(\Sigma_1^{1/2}\Sigma_2\Sigma_1^{1/2})^{1/2})$, substituting in the stationary covariances for Σ1,Σ2\Sigma_1, \Sigma_2. The Wasserstein-2 identity is purely for illustrating how TTF in the our constructed setting can be directly characterized in terms of the covariance matrices. As the result is not involved in any other derivations, we have opted to omit it from Proposition 3.1 for visual clarity.

[1] Du and Su. “The Newton-Muon Optimizer,” 2026.

[2] Vyas et al. “SOAP: Improving and Stabilizing Shampoo using Adam” 2025.

[3] Massena et al. “From SGD to Muon: Adaptive Optimization via Schatten-p Norms” 2026.

Reviewer_ejmyOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to improve model performance under repeated feedback during deployment by changing the training optimizer. The authors propose DoPr, which first preconditions gradients using the covariance of each layer’s inputs and then applies a conventional optimizer. They motivate the approach through linear-model analyses and report performance improvements in control, robotic manipulation, mathematical reasoning, and image generation.

Critical strengths
  • The authors separate activation-based preconditioning from the subsequent gradient optimizer, making the modification conceptually straightforward and easy to reuse.

  • The paper provides a wealth of experimental results and robustness tests.

Critical weaknesses
  • The proofs require further verification because some theoretical results conflict. For example, Lemma B.7 and Proposition B.8 contradict each other regarding the ordering of Lval(Kψ)L_{\mathrm{val}}(K_\psi) and Lval(Kθ)L_{\mathrm{val}}(K_\theta).

  • The theoretical analysis mainly considers AP alone. The authors should further explain the combined effects when using adaptive algorithms such as Adam, or narrow their theoretical claims.

Questions for authors
  • The authors should validate the proposed mechanism by directly testing whether feedback-induced distribution shift is reduced in actual sequential tasks.

  • Some figures need to be checked or explained more clearly, such as the arrow labeled “Better” in the first panel of Figure 8.

1 comment

We thank you for your review: we address your questions sequentially below.

Verify theoretical consistency between Lemma B.7 and Proposition B.8. Lemma B.7 is correct as written. The intended conditions in Proposition B.8 are ϵ≪α\epsilon \ll \alpha and sin⁡(θ)<α2/ϵ2\sin(\theta) < \alpha^2/\epsilon^2, rather than α≪ϵ\alpha \ll \epsilon and sin⁡(θ)<ϵ2/α2\sin(\theta) < \epsilon^2/\alpha^2. We have corrected this in the revision.

To reiterate, the proposition illustrates a reversal between validation and deployment performance on the same system: a policy can satisfy Lval(Kθ)≪Lval(Kψ)\mathcal{L}_{\mathrm{val}}(\mathbf{K}_\theta) \ll \mathcal{L}_{\mathrm{val}}(\mathbf{K}_\psi) while Lideal(Kθ)≫Lideal(Kψ)\mathcal{L}_{\mathrm{ideal}}(\mathbf{K}_\theta) \gg \mathcal{L}_{\mathrm{ideal}}(\mathbf{K}_\psi). Its purpose is to make the mismatch induced by test-time feedback (TTF) explicit.

Combining AP and gradient-based preconditioners Section 4 separates two roles: activation preconditioning (AP) addresses the feature-learning pathology identified in the analysis, while gradient preconditioning (GP) supplies practical update normalization and stability. The activation-covariance-weighted geometry provides the construction for incorporating existing GPs such as Adam and Muon; the practical motivation is discussed in Section 4.3 and lines 265–266.

Our feature-learning guarantees are restricted to the stated linear-model setting. This setting makes both the feedback-induced distribution shift and the optimal feature subspace explicit, allowing a direct comparison with gradient descent. The nonlinear experiments evaluate the resulting DoPr algorithms; they are not presented as consequences of a general convergence theorem for Adam or Muon.

We note that all our theoretical claims about feature learning are restricted to the illustrative linear system setting. This is intentional, as this setting allows us to concisely illustrate what TTF looks like in closed-form, and also allows us to compare to a ground-truth optimal set of features. This allows us to illustrate suboptimality of SGD most clearly.

Measuring TTF-induced distribution shift We make a few remarks:

  1. Feedback-induced distribution shift as a phenomenon is well-documented in various literature, which we have documented thoroughly in our Extended Related Works (Appendix A).
  2. However, typically the effect of TTF distribution shift is hard to isolate: a better optimized model (from a given recipe) typically leads to better performance, thus reduced distribution shift is hard to extract from overall better optimization. Direct estimation of distributional measures, especially in continuous input/latent space, is notoriously difficult to perform in practice. This motivates our two-fold approach: a. We introduced a simple theoretical model of a linear system to directly show which predictor best reduces distribution shift, and why AP is the natural solution without access to on-policy trajectories. b. We strongly emphasize throughout the paper how DoPr variants do not consistently training/validation loss, which means the improved performance comes from reduced TTF, and not from improved loss. Therefore, this directly ablates the explanation that DoPr is improving model performance solely due to better training performance, which in turn provides performance under TTF as the primary explanation, matching our illustrative theoretical constructions.

This said, we also remark that an implication of our theoretical set-up when extended to the multi-layer case is that TTF can occur not only across timesteps, but along the layers of the neural network itself: poorly learned features at an early layer cause a loss of information that cannot be recovered at later layers. We added a new section Appendix D.3 to illustrate this.

In particular, we demonstrate that a base GP versus its DoPr variant both achieve the exact same "in-distribution" coverage of layer-wise activations, but DoPr yields better coverage of activations under TTF. This is further evidence that DoPr is indeed learning better/robust features layer-wise via a measurable quantity of feedback-induced distribution shift.

Figure clarity The "Better" arrow indicates that points in that direction are dominant, i.e. better in-distribution validation loss, and better downstream performance. Please let us know if there are any further clarifications you'd like to see.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.