- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Summary
This paper proposes ROAM (Riemannian Optimization with Adaptive Metrics), a task-conditioned optimization framework for time-series forecasting. The method constructs block-wise Riemannian metrics based on six task statistics to adapt the optimization geometry to different forecasting tasks. It further introduces a state-triggered switching mechanism between momentum-based and quasi-Newton optimization methods, together with a population-based outer loop that adaptively adjusts the optimization controls. The authors provide a local theoretical analysis relating metric alignment to improved conditioning and convergence behavior. Experiments on seven datasets, five forecasting backbones, and multiple prediction horizons demonstrate consistent improvements in forecasting accuracy. Additional optimizer comparisons and ablation studies investigate the effectiveness of the task-conditioned metrics, optimizer switching, and outer-loop adaptation.
Critical strengths
-
The paper presents an interesting research perspective.
-
The experiments cover a broad range of settings and demonstrate relatively consistent improvements.
Critical weaknesses
1. There remains a gap between the theoretical analysis and the proposed optimization framework.
The paper analyzes the spectral alignment between Riemannian metrics and local curvature, establishing theoretical results on improved conditioning and local convergence properties under certain assumptions. However, the authors do not demonstrate that the metrics constructed from the six task statistics satisfy the proposed spectral alignment conditions. Moreover, the paper does not sufficiently establish a theoretical connection among task statistics, the resulting Riemannian metrics, and improvements in practical optimization performance. Therefore, the current theoretical analysis does not fully explain why the proposed task-adaptive optimization framework improves the actual training process.
2. The novelty of the proposed method compared with existing optimization approaches is not sufficiently established.
The proposed framework incorporates several well-established techniques, including Gauss–Newton/Fisher preconditioning, gradient second-moment estimation, and existing optimization methods such as NAdam and L-BFGS. Although the authors integrate task-conditioned metric construction, optimizer switching, and outer-loop adaptation into a unified framework, its distinctive algorithmic contributions compared with existing methods remain insufficiently clarified.
Furthermore, existing research on Learning to Optimize on Riemannian Manifolds has also explored learning-based mechanisms for adapting optimization strategies to different tasks. The paper does not sufficiently discuss its relationship to such approaches or clearly explain the distinctive advantages of task-statistics-driven metric construction over learned optimization methods.
Questions for authors
1. Relationship between task statistics and optimization geometry.
The paper constructs task-conditioned Riemannian metrics using six task statistics. However, these statistics mainly characterize data distributions, temporal structures, and initial gradient noise, whereas the optimization geometry of the parameter space also depends on the model architecture, parameterization, and loss function. Could the authors further explain why these six statistics can effectively characterize the optimization geometry of the parameter space? In particular, is there a more explicit connection between these task statistics and the curvature alignment properties of the resulting metrics?
2. Necessity of the optimizer switching mechanism.
The proposed framework switches between NAdam and L-BFGS based on gradient noise and curvature reliability. However, why is a discrete switching mechanism preferred over a continuous weighted combination of the two update directions? Could the authors further clarify the distinctive advantages of discrete switching and discuss the potential differences between these two strategies in terms of optimization stability, convergence speed, and final performance?
6 comments
Response (2/3): Relationship to existing and learned optimizers
W2: What distinguishes ROAM from existing and learned optimizers?
The revised S̲e̲c̲t̲i̲o̲n̲ 2̲ attributes the established ingredients and states the contribution at the level of task-conditioned control.
Gauss–Newton/Fisher preconditioning, second-moment estimation, momentum, L-BFGS, manifold operations, and PBT are not new individually. The revised text cites preconditioned product-manifold metrics by Gao, Peng, and Yuan [1], gradient-noise-scale work by McCandlish et al. [2], and stochastic quasi-Newton safeguards by Bollapragada et al. [3].
Learning to Optimize on Riemannian Manifolds [4] parameterizes a Riemannian optimizer with a recurrent network and trains it through meta-optimization, including an implicit-differentiation scheme. ROAM instead retains a prescribed metric/update family and fits a compact affine map from explicit task statistics to its control coordinates, followed by derivative-free population adaptation. Source fitting is itself a learned stage, with its calibration cost explicitly included in the first-task comparison and amortized analysis.
The practical distinction is inspectability. Descriptor inputs have stated data dependencies; control coordinates have meanings such as block scale, damping, curvature mixture, and phase threshold; the realized metric also uses current model derivatives. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ now gives the fit and descriptor-to-control sensitivity. Individual descriptor removal, removal of the descriptor term while retaining the source intercept, and direct search over the same family are explicit interventions on this interface.
T̲a̲b̲l̲e̲ 3̲ reports the largest mean marginal effect for the metric (.01025 versus .00475 for switching and .00525 for outer adaptation). The source-mean/direct-search results in S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ further delimit the role of task information: conditioning improves on source mean on both forecasting targets, but direct search can match or exceed it and is stronger in the first-task cost comparison.
Thus our contribution is a transparent task-conditioning mechanism for forecasting optimization, supported by competitive prediction results and controlled attribution tests. The comparison with recurrent learned optimizers concerns the parameterized interface and learning procedure; empirical performance comparisons are restricted to the optimizers actually evaluated.
References
[1] Gao, Peng, and Yuan (2025). Optimization on Product Manifolds under a Preconditioned Metric. SIAM J. Matrix Anal. Appl.
[2] McCandlish et al. (2018). An Empirical Model of Large-Batch Training.
[3] Bollapragada et al. (2018). A Progressive Batching L-BFGS Method for Machine Learning. ICML.
[4] Gao et al. (2023). Learning to Optimize on Riemannian Manifolds. IEEE TPAMI.
Response to gg28 (3/3): Discrete switching and a continuous-blending control
Q2: Why use discrete switching rather than continuous blending?
We have added a safeguarded continuous comparator, summarized in the revised S̲e̲c̲t̲i̲o̲n̲ 5̲.̲7̲. A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲4̲ defines it, and T̲a̲b̲l̲e̲ 2̲5̲ and F̲i̲g̲u̲r̲e̲ 1̲5̲ report the comparison.
Why use a gate? The default rule waits for low relative gradient disagreement and acceptable directional curvature for three checks before using L-BFGS. A metric refresh clears memory and requires renewed eligibility. This gives an interpretable decision about when the available quasi-Newton state is usable and can avoid its direction/line-search work during Phase I. Hysteresis is a persistence rule, not a statistical certificate.
Continuous alternative. Using the same metric and accepted pairs, we evaluate
\frac{\max(\rho_t,0)}{\max(\rho_t,0)+\tau_\rho},\qquad p_t=(1-a_t)p_t^I+a_tp_t^{II}.$$ The first-order direction includes its prescribed scaling. Missing valid memory disables the quasi-Newton contribution. The combined direction is checked for descent and uses the same Armijo safeguard; a non-descent direction falls back to the metric gradient. Refresh and pair-acceptance policies are shared. Conditioned initialization, ordinary DLinear, data, population, metric estimator, and validation selection are fixed. Each population has the same 20-second target cap and 20 validation calls over ten seeds; source calibration is shared and separately accounted for. | Policy | ETTh2-96 MSE | ETTm2-96 MSE | ETTh2 total updates | |---|---:|---:|---:| | Discrete switch | .312 +/- .010 | .174 +/- .004 | 2,400 | | Continuous blend | .314 +/- .011 | .173 +/- .005 | 2,150 | | Phase I only | .319 +/- .012 | .179 +/- .006 | 3,000 | Update counts sum across the four members; deviations are seed standard deviations. **Switching is slightly stronger on ETTh2, blending on ETTm2.** These small differences are not paired significance claims. The figure plots validation progress by evaluation round, without relabeling rounds as elapsed time. If two directions descend for the same objective at the same point, their convex combination also descends. We therefore do not claim blending is intrinsically unstable. Momentum and stochastic estimates need not satisfy that premise, so both policies need safeguards. The result supports a work–accuracy trade-off for this specified blend, not universal superiority of switching. The revised text presents the discrete policy as a practical default with an explicit eligibility rule.
Response (1/3): Task statistics, model dependence, and measured geometry
Dear Reviewer, thank you for recognizing the breadth of the forecasting evaluation and for asking for a clearer mechanism. We address your concerns in three responses.
W1/Q1: How do task statistics connect to realized geometry?
The revised S̲e̲c̲t̲i̲o̲n̲s̲ 3̲–̲4̲ distinguish metric construction from the conditional alignment result. F̲i̲g̲u̲r̲e̲ 1̲0̲ makes the information flow explicit.
Information path. The descriptor initializes control coordinates. The metric also uses model-dependent operators Fb(θ) and Db(θ) computed from the instantiated model and training loss. For general losses, Gauss–Newton curvature contains both the model Jacobian and output-loss curvature. Initial gradient disagreement is already model-dependent, and the block partition and initializer dimensions follow the backbone. Six scalars are not intended to replace these dependencies.
For a centered linear lag predictor, K=2E[xx⊤] gives a precise connection between lag covariance and curvature; stationarity connects that covariance to the full spectrum. Spectral summaries can be useful without identifying extreme eigenvalues or eigendirections. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ gives the source-fitting objective and the explicit derivative of positive controls with respect to descriptor coordinates.
What is measured. S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ compares conditioned, source-mean, and direct-search initialization within the same model, metric family, and target budget. For each of ten seeds, the same-state diagnostic uses common initial weights and a shared batch of 64 training windows to estimate the initial metric, with the second-moment term initially zero. It then evaluates each metric against the exact full-training Hessian, before training trajectories diverge. The entries below are means of per-seed positive-spectrum ratios.
F̲i̲g̲u̲r̲e̲ 4̲ and A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲3̲ report this diagnostic without treating the redundant DLinear Hessian as positive definite. With a full-rank identifiable design, the two-map model's positive rank and nullity are each H(T+1), or 9,312 for T=H=96. This is a conditional dimension calculation; the ETTh2 design-rank measurement (97 at tolerance 10−10) is stated separately in the exact-solve experiment.
The lower positive-spectrum ratios support improved geometry on observable directions in these settings. They do not prove the spectral sandwich in Proposition 1 or explain the entire test-error gain. The revised S̲e̲c̲t̲i̲o̲n̲ 4̲ attributes the classical result and makes its positive-definiteness/coupling assumptions explicit; it makes no convergence claim for the complete switched, stochastic, population-adapted algorithm. Together, the controls and diagnostics connect the task-conditioned mechanism to measured geometry and prediction.