- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the distribution drift problem common in neural odes, learning to control and sequence modeling problems, where the one step training objective does not match the actual performance metrics at test time, and error accumulation can be catastrophic when the prediction horizon is long. Unlike most existing work that tries to address the issue by introducing new training objectives, the paper takes an interesting approach by studying the role of optimization. The claim made here is that activation conditioning plays a crucial role in test time performance, and poor conditioning in the activations can lead to good in-distribution loss but bad rollout results. The proposed fix is an additional activation covariance preconditioning step, essentially encouraging the learned features to spread out more evenly. The idea therefore shares some intuition with DRO in my opinion and can potentially mitigate online prediction issues without fundamentally changing the loss. Numerical results seem consistent with the proposal.
Critical strengths
The strongest aspect of the work in my view is the empirical results part. The method is tested across several substantially different settings and optimizers, and there is a fairly consistent improvement in test time performance. The main claim of the paper does seem to validate itself even though the performance gain can be marginal at times, The computation overhead and limitations discussion is also very much appreciated and I thank the authors for their honesty.
Critical weaknesses
To me the theoretical results are the weakness of the work and several places need more justification than what's currently given, and I think the actual novelty needs to be separated much better. A couple things here:
A major portion of the results is essentially analyzed in a linear quadratic setting. Can this simplification be justified? I personally don't see how such trivialization can be made naively as a blanket statement, especially as written in Propositions 3.1 and 3.2. I actually believe the claim to be reasonable even for the actual nonlinear problems, but the argument is more heuristic than proof to me.
I also think the authors should make it clear in terms of novelty. Much of the LQ problem derivation is fairly standard in the control literature, and I don't see much new there. A key contribution could be the bridge between the standard LQ setup and the actual nonlinear setting the paper tries to tackle, yet it's largely missing here.
Similarly, Proposition 4.1 is explicitly adapted from prior feature learning results and establishes a local, one step contraction under fairly restrictive assumptions. What is the genuinely new theoretical statement obtained by combining these pieces? Saying that the test time performance is a mismatch to the training objective and showing the proposed scheme is reasonable in a linear setting are both okay, but not really of much value. The key question to me is actually not answered here, despite the extensive setup.
Questions for authors
While the numerical results of the paper are very much self-consistent, I cannot help but notice that these improvements are often quite marginal. The paper acknowledges the existence of a common line of work targeting the same issue through modifications to the training objective, and these methods would usually lead to much more significant improvements in test-time results. I do not think it is fair to ask for a side-by-side comparison given the limited time, but I would like to see some insight into the differences. My rough guess is that while the additional preconditioning can change the optimization, it does not have the necessary information about , and even according to the paper's own proposition, that discrepancy can be essentially unbounded.
I also have a question regarding the paper's proposed method and how it relates to the idea of natural gradient methods, which have seen an abundance of applications and adoption in modern machine learning. If I remove the language on the mismatch between test time metrics and the training objective,, since we never have access to , doesn't the proposed method essentially default to a natural gradient method? What remains novel in this paper from a methodological perspective? I think the paper made some comments on this in the appendix, but I would like to see it spelled out more explicitly, as I do not think this is merely an issue of related work and I do have some concerns on the claimed novelties of the paper.
2 comments
Relationship to natural gradient and methodological novelty.
This is another great, nuanced question. There are many conceptual connections to the Natural Gradient Method, and we document the long lineage in our extended related works. Firstly, full-matrix natural gradient should in principle address the same feature learning issue as AP. We make this connection evident in Proposition 4.2, where AP provides a targeted parameterization-invariance that can be achieved by a layer-wise operation---recall that full natural gradient is invariant to any smooth re-parameterization. This said, all modern computationally-efficient approximations of natural gradient are motivated from a training convergence perspective. Furthermore, only the KFAC line of work (which is in modern practice unstable or suboptimal) performs a faithful approximation to the true natural gradient, whereas all other modern gradient-based optimizers (Adam, Shampoo, Muon, SOAP) can be viewed as approximations to the empirical natural gradient. Our work can be viewed as providing a mediation of these perspectives: activation-side-only preconditioning is a valid approximation to the true Fisher Information that does not require the re-sampling/additional backwards passes that KFAC does, while preserving precisely the parameterization-invariance relevant to TTF: since TTF causes a distribution shift in the inputs, distribution shift propagates layerwise through the activations. The combination with GP then inherits the well-conditioning and training stability afforded by gradient-based optimizers.
We also remark that some works have demonstrated that full natural gradient can confusingly underperform its factored counterparts in neural network training [3], further motivating practical developments such as the combination of AP and GP presented in our work.
[1] Pan et al. “Much Ado About Noising: Dispelling the Myths of Generative Robotic Control” 2026.
[2] Simchowitz et al. “The Pitfalls of Imitation Learning when Actions are Continuous” 2025.
[3] Benzing et al. "Gradient descent on neurons and its link to approximate second-order optimization" 2022.
We thank you for your review: we address your questions sequentially below.
Purpose and novelty of the linear analysis. We emphasize that we chose a simple linear system intentionally to provide a familiar setting that achieves the following in a readable manner: 1. demonstrate what TTF looks like precisely and quantiatively, 2. explicitly show that feature learning can be extremely suboptimal even in this simple setting, 3. show how DoPr naturally arises from this setting. We also remark that presenting these concepts involves some novel contributions, for example the existence of two policies where one achieves much lower val loss than the other, but leads to much worse test-time error on the same system.
Standard extensions, e.g., to generalized linear models, can be performed, but we did not think these meaningfully provided to our conceptual contributions. Notably, we emphasize that this paper is not intended as a theoretical contribution, and our various results are purely in service of making TTF-induced distribution shift and impaired feature learning concrete. However, we believe we are the first to quantitatively characterize TTF as an optimization phenomenon.
Novelty of Proposition 4.1 As aforementioned, the core purpose of our paper is not theoretical generality. We explicitly tie Proposition 4.1 to prior work in Zhang et al. (2024). The purpose of Proposition 4.1 is to make explicit that raw (S)GD is a poor feature learner, and to impart the correct equivariance to TTF-induced distribution shift is AP.
However, we do note that this two-layer set-up is conceptually rich. For example, multi-layer settings can be viewed as performing linear regression from the current layer's activations to the loss derivative. This viewpoint has been explored in prior optimization literature, which we documented in the Extended Related Work, and is in fact the exact same motivation for linear-attention mechanisms such as DeltaNet and MesaNet, the latter of which can be viewed as "AP"-ed DeltaNet. However, in the optimizer literature, we find that methods motivated from layer-wise regression alone (such as AP-only) tend to be numerically unstable, which is what motivates our core practical contribution in Double Preconditioning.
Comparison with objective-based interventions. This is an interesting question; firstly, we remark that we present an optimizer-based intervention as it is in principle compatible with any training objective. Secondly, in large-scale LLM training, many regularization tricks aimed at reducing exposure bias (a form of TTF) from earlier sequence modeling literature did not survive the test of scale, and modern LLM training pipelines tend to have maintain the simplest training objective (e.g. next-token cross-entropy). Furthermore, our aim is to provide an application-agnostic intervention, where objective modifications tend to be application or even set-up dependent.
We remark that spiritually algorithms like DAgger (i.e. OPD or OPSD in LLM post-training parlance) can be viewed as an implicit modification to the loss objective, as it changes the distribution the loss is evaluated on. However, these come at an extremely significant overhead during training due to having to iteratively sample trajectories during training---this may not even be feasible depending on application, such as robotics. Thus, it is hard to perform apples-to-apples comparisons in terms of test-time gains from changing the sampling distribution versus changing the optimizer.
However, we remark that for our robot policy learning experiments, the performance gain on the hard environments tool-hang and transport from switching to DoPr in fact outsizes the gain from switching between the best and next-best architectures and loss objectives in the paper we based our set-up on [1].
Limits without access to the deployment distribution. Indeed, preconditioning cannot recover arbitrary information absent from the offline data, and we do not claim that it can avoid worst-case compounding error. For example, [2] constructs continuous-control settings in which smooth, deterministic policies learned solely from expert data can exhibit exponential-growing TTF no matter how the policy is parameterized/optimized. In fact, the core counterexample comes from a linear system construction. In this case, there is no recourse but to modify where the training data is sampled, e.g. training-time corrective sampling via DAgger.
Our claim is that optimizer-induced feature-learning failures are an additional, avoidable source of poor deployment behavior. DoPr targets that source using the information already available in the training activations. It does not require knowledge of πideal, nor does it provide a universal bound on the discrepancy from that distribution.