acceptodds
Under review as a conference paper at ICLR 2027

ROAM: Task-Conditioned Metrics for Forecasting Optimization

Abstract

Can a forecasting task inform how its model should be optimized, without changing the forecasting architecture? We study this question through ROAM (Riemannian Optimization with Adaptive Metrics), which maps six training-derived task statistics to a block-wise positive-definite metric. A slow population controller adapts the metric and a two-signal transition between momentum-based and quasi-Newton updates. The central mechanism is task-conditioned preconditioning on ordinary parameters; explicit orthogonality and positive-definite parameter constraints are separate, optional interventions. Across seven datasets and five backbones, ROAM improves all 35 dataset–backbone MSE aggregates relative to standard training recipes, with approximately 4.8%-7.3% lower macro-average MSE by backbone. Four matched-budget optimizer comparisons and a complete three-component factorial provide more targeted evidence: the metric has the largest average marginal contribution, while switching and outer adaptation also help. A local analysis relates metric alignment and cross-block coupling to conditioning, without assuming that task descriptors guarantee alignment. Together, these results support task information as a useful input to training geometry in the evaluated forecasting regimes.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Riemannian Optimization, Task-Conditioned Metrics, Adaptive Optimization, Population-Based Training, Optimization Geometry

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_gg28OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary

This paper proposes ROAM (Riemannian Optimization with Adaptive Metrics), a task-conditioned optimization framework for time-series forecasting. The method constructs block-wise Riemannian metrics based on six task statistics to adapt the optimization geometry to different forecasting tasks. It further introduces a state-triggered switching mechanism between momentum-based and quasi-Newton optimization methods, together with a population-based outer loop that adaptively adjusts the optimization controls. The authors provide a local theoretical analysis relating metric alignment to improved conditioning and convergence behavior. Experiments on seven datasets, five forecasting backbones, and multiple prediction horizons demonstrate consistent improvements in forecasting accuracy. Additional optimizer comparisons and ablation studies investigate the effectiveness of the task-conditioned metrics, optimizer switching, and outer-loop adaptation.

Critical strengths
  1. The paper presents an interesting research perspective.

  2. The experiments cover a broad range of settings and demonstrate relatively consistent improvements.

Critical weaknesses

1. There remains a gap between the theoretical analysis and the proposed optimization framework.

The paper analyzes the spectral alignment between Riemannian metrics and local curvature, establishing theoretical results on improved conditioning and local convergence properties under certain assumptions. However, the authors do not demonstrate that the metrics constructed from the six task statistics satisfy the proposed spectral alignment conditions. Moreover, the paper does not sufficiently establish a theoretical connection among task statistics, the resulting Riemannian metrics, and improvements in practical optimization performance. Therefore, the current theoretical analysis does not fully explain why the proposed task-adaptive optimization framework improves the actual training process.

2. The novelty of the proposed method compared with existing optimization approaches is not sufficiently established.

The proposed framework incorporates several well-established techniques, including Gauss–Newton/Fisher preconditioning, gradient second-moment estimation, and existing optimization methods such as NAdam and L-BFGS. Although the authors integrate task-conditioned metric construction, optimizer switching, and outer-loop adaptation into a unified framework, its distinctive algorithmic contributions compared with existing methods remain insufficiently clarified.

Furthermore, existing research on Learning to Optimize on Riemannian Manifolds has also explored learning-based mechanisms for adapting optimization strategies to different tasks. The paper does not sufficiently discuss its relationship to such approaches or clearly explain the distinctive advantages of task-statistics-driven metric construction over learned optimization methods.

Questions for authors

1. Relationship between task statistics and optimization geometry.

The paper constructs task-conditioned Riemannian metrics using six task statistics. However, these statistics mainly characterize data distributions, temporal structures, and initial gradient noise, whereas the optimization geometry of the parameter space also depends on the model architecture, parameterization, and loss function. Could the authors further explain why these six statistics can effectively characterize the optimization geometry of the parameter space? In particular, is there a more explicit connection between these task statistics and the curvature alignment properties of the resulting metrics?

2. Necessity of the optimizer switching mechanism.

The proposed framework switches between NAdam and L-BFGS based on gradient noise and curvature reliability. However, why is a discrete switching mechanism preferred over a continuous weighted combination of the two update directions? Could the authors further clarify the distinctive advantages of discrete switching and discuss the potential differences between these two strategies in terms of optimization stability, convergence speed, and final performance?

6 comments
AuthorsOpenReview ↗

Response (1/3): Task statistics, model dependence, and measured geometry

Dear Reviewer, thank you for recognizing the breadth of the forecasting evaluation and for asking for a clearer mechanism. We address your concerns in three responses.

W1/Q1: How do task statistics connect to realized geometry?

The revised S̲e̲c̲t̲i̲o̲n̲s̲ 3̲–̲4̲ distinguish metric construction from the conditional alignment result. F̲i̲g̲u̲r̲e̲ 1̲0̲ makes the information flow explicit.

Information path. The descriptor initializes control coordinates. The metric also uses model-dependent operators F^b(θ)\widehat F_b(\theta) and Db(θ)D_b(\theta) computed from the instantiated model and training loss. For general losses, Gauss–Newton curvature contains both the model Jacobian and output-loss curvature. Initial gradient disagreement is already model-dependent, and the block partition and initializer dimensions follow the backbone. Six scalars are not intended to replace these dependencies.

For a centered linear lag predictor, K=2E[xx⊤]K=2\mathbb E[xx^\top] gives a precise connection between lag covariance and curvature; stationarity connects that covariance to the full spectrum. Spectral summaries can be useful without identifying extreme eigenvalues or eigendirections. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ gives the source-fitting objective and the explicit derivative of positive controls with respect to descriptor coordinates.

What is measured. S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ compares conditioned, source-mean, and direct-search initialization within the same model, metric family, and target budget. For each of ten seeds, the same-state diagnostic uses common initial weights and a shared batch of 64 training windows to estimate the initial metric, with the second-moment term initially zero. It then evaluates each metric against the exact full-training Hessian, before training trajectories diverge. The entries below are means of per-seed positive-spectrum ratios.

Initial metricETTh2-96 positive-spectrum ratioETTm2-96 positive-spectrum ratio
Identity800410
Default controls760385
Source mean720390
Conditioned640375

F̲i̲g̲u̲r̲e̲ 4̲ and A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲3̲ report this diagnostic without treating the redundant DLinear Hessian as positive definite. With a full-rank identifiable design, the two-map model's positive rank and nullity are each H(T+1)H(T+1), or 9,312 for T=H=96T=H=96. This is a conditional dimension calculation; the ETTh2 design-rank measurement (97 at tolerance 10−1010^{-10}) is stated separately in the exact-solve experiment.

The lower positive-spectrum ratios support improved geometry on observable directions in these settings. They do not prove the spectral sandwich in Proposition 1 or explain the entire test-error gain. The revised S̲e̲c̲t̲i̲o̲n̲ 4̲ attributes the classical result and makes its positive-definiteness/coupling assumptions explicit; it makes no convergence claim for the complete switched, stochastic, population-adapted algorithm. Together, the controls and diagnostics connect the task-conditioned mechanism to measured geometry and prediction.

AuthorsOpenReview ↗

Response (2/3): Relationship to existing and learned optimizers

W2: What distinguishes ROAM from existing and learned optimizers?

The revised S̲e̲c̲t̲i̲o̲n̲ 2̲ attributes the established ingredients and states the contribution at the level of task-conditioned control.

Gauss–Newton/Fisher preconditioning, second-moment estimation, momentum, L-BFGS, manifold operations, and PBT are not new individually. The revised text cites preconditioned product-manifold metrics by Gao, Peng, and Yuan [1], gradient-noise-scale work by McCandlish et al. [2], and stochastic quasi-Newton safeguards by Bollapragada et al. [3].

Learning to Optimize on Riemannian Manifolds [4] parameterizes a Riemannian optimizer with a recurrent network and trains it through meta-optimization, including an implicit-differentiation scheme. ROAM instead retains a prescribed metric/update family and fits a compact affine map from explicit task statistics to its control coordinates, followed by derivative-free population adaptation. Source fitting is itself a learned stage, with its calibration cost explicitly included in the first-task comparison and amortized analysis.

The practical distinction is inspectability. Descriptor inputs have stated data dependencies; control coordinates have meanings such as block scale, damping, curvature mixture, and phase threshold; the realized metric also uses current model derivatives. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ now gives the fit and descriptor-to-control sensitivity. Individual descriptor removal, removal of the descriptor term while retaining the source intercept, and direct search over the same family are explicit interventions on this interface.

T̲a̲b̲l̲e̲ 3̲ reports the largest mean marginal effect for the metric (.01025 versus .00475 for switching and .00525 for outer adaptation). The source-mean/direct-search results in S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ further delimit the role of task information: conditioning improves on source mean on both forecasting targets, but direct search can match or exceed it and is stronger in the first-task cost comparison.

Thus our contribution is a transparent task-conditioning mechanism for forecasting optimization, supported by competitive prediction results and controlled attribution tests. The comparison with recurrent learned optimizers concerns the parameterized interface and learning procedure; empirical performance comparisons are restricted to the optimizers actually evaluated.

References

[1] Gao, Peng, and Yuan (2025). Optimization on Product Manifolds under a Preconditioned Metric. SIAM J. Matrix Anal. Appl.

[2] McCandlish et al. (2018). An Empirical Model of Large-Batch Training.

[3] Bollapragada et al. (2018). A Progressive Batching L-BFGS Method for Machine Learning. ICML.

[4] Gao et al. (2023). Learning to Optimize on Riemannian Manifolds. IEEE TPAMI.

AuthorsOpenReview ↗

Response to gg28 (3/3): Discrete switching and a continuous-blending control

Q2: Why use discrete switching rather than continuous blending?

We have added a safeguarded continuous comparator, summarized in the revised S̲e̲c̲t̲i̲o̲n̲ 5̲.̲7̲. A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲4̲ defines it, and T̲a̲b̲l̲e̲ 2̲5̲ and F̲i̲g̲u̲r̲e̲ 1̲5̲ report the comparison.

Why use a gate? The default rule waits for low relative gradient disagreement and acceptable directional curvature for three checks before using L-BFGS. A metric refresh clears memory and requires renewed eligibility. This gives an interpretable decision about when the available quasi-Newton state is usable and can avoid its direction/line-search work during Phase I. Hysteresis is a persistence rule, not a statistical certificate.

Continuous alternative. Using the same metric and accepted pairs, we evaluate

\frac{\max(\rho_t,0)}{\max(\rho_t,0)+\tau_\rho},\qquad p_t=(1-a_t)p_t^I+a_tp_t^{II}.$$ The first-order direction includes its prescribed scaling. Missing valid memory disables the quasi-Newton contribution. The combined direction is checked for descent and uses the same Armijo safeguard; a non-descent direction falls back to the metric gradient. Refresh and pair-acceptance policies are shared. Conditioned initialization, ordinary DLinear, data, population, metric estimator, and validation selection are fixed. Each population has the same 20-second target cap and 20 validation calls over ten seeds; source calibration is shared and separately accounted for. | Policy | ETTh2-96 MSE | ETTm2-96 MSE | ETTh2 total updates | |---|---:|---:|---:| | Discrete switch | .312 +/- .010 | .174 +/- .004 | 2,400 | | Continuous blend | .314 +/- .011 | .173 +/- .005 | 2,150 | | Phase I only | .319 +/- .012 | .179 +/- .006 | 3,000 | Update counts sum across the four members; deviations are seed standard deviations. **Switching is slightly stronger on ETTh2, blending on ETTm2.** These small differences are not paired significance claims. The figure plots validation progress by evaluation round, without relabeling rounds as elapsed time. If two directions descend for the same objective at the same point, their convex combination also descends. We therefore do not claim blending is intrinsically unstable. Momentum and stochastic estimates need not satisfy that premise, so both policies need safeguards. The result supports a work–accuracy trade-off for this specified blend, not universal superiority of switching. The revised text presents the discrete policy as a practical default with an explicit eligibility rule.
Reviewer_esdHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ROAM is an optimizer for long-horizon forecasting that leaves the backbone unchanged. Six statistics of the training data are mapped by an affine map γ(0)=P(Wc~+b)\gamma^{(0)}=P(W\tilde c+b) to the controls of a block-diagonal preconditioner Qb=(λb+δ)I+ωbF^b+(1−ωb)DbQ_b=(\lambda_b+\delta)I+\omega_b \hat F_b+(1-\omega_b)D_b, where F^b\hat F_b is a Gauss-Newton proxy and DbD_b is the diagonal of an EMA of squared gradients, and the update is G−1∇LG^{-1}\nabla L. Training starts in a NAdam-style phase and switches to L-BFGS with Armijo backtracking when a two-batch disagreement statistic χt\chi_t is below a threshold and the curvature ratio ⟨s,y⟩/∥s∥2\langle s,y\rangle/\Vert s\Vert^2 is above its threshold for three consecutive checks. A population of four individuals perturbs the controls and is ranked by validation loss plus time and memory penalties. Section 4 gives local bounds. On ETTh1, ETTh2, ETTm1, ETTm2, Weather, ECL and Exchange with iTransformer, PatchTST, TimeMixer, ModernTCN and DLinear at horizons 96 to 720, ROAM is reported to lower MSE in all 35 dataset-backbone cells by 4.8% to 7.3% per backbone, to beat Lion, SAM, Sophia-G and internal controls in four settings, and in a 232^3 factorial the metric gets the largest marginal gain.

Critical strengths

∙\bullet The preconditioner is positive definite by construction: Qb⪰(λb+δ)IQ_b\succeq(\lambda_b+\delta)I for any values of the two proxies, so the metric solve is always well posed. The L-BFGS phase is safeguarded appropriately for mini-batch gradients, with the pair test ⟨s,y⟩≥ccurv∥s∥2\langle s,y\rangle\ge c_{\text{curv}}\Vert s\Vert^2 and three consecutive checks in both directions, and the paper says itself that this is hysteresis, not a certificate.

∙\bullet The paper keeps a change of metric apart from a change of model (Section 3.4, Appendix C.3) and reports the Stiefel and SPD results separately from the metric on ordinary weights.

∙\bullet I went through Appendix A.1 to A.6 by hand, the congruence step, the whitened recursion, the Markov bound and the BFGS argument, and found no algebraic errors. The paper is open about what the theory does not cover.

Critical weaknesses

W1. Some reported numbers look suspicious, and there is no code to check them

First, Figure 6 contradicts Table 20. The heatmap gives 0.0 for ETTm1 to Weather, Weather to ECL and ECL to Exchange, while Table 20 gives 5.2%, 6.0% and 5.4% for the same pairs. The three zeros are located on the diagonal of the matrix as if source and target were the same dataset, although the row and column labels differ. The caption only says that the 12-transfer mean is not an average of the displayed cells, which does not explain a disagreement in single cells.

Second, some per-horizon numbers coincide with four-horizon averages. For Weather-192 with TimeMixer (Table 2, column C) ROAM reports 0.232, which is exactly the ROAM Weather four-horizon average in Table 10, and Lion at 0.243 is within 0.002 of the baseline four-horizon average of 0.245 in the same table. Table 13 never names the backbone for its three settings, and per-horizon, per-seed results are not reported outside Tables 2 and 13.

Third, none of this can be checked, because there is no code. No repository, supplementary material or release statement appears anywhere, and the Reproducibility Statement points only to the appendices.

W2. DLinear is a convex problem and shows the largest gain

If I understand DLinear correctly, it splits the input window into a moving-average trend and a residual, passes each through its own linear map from RT\mathbb{R}^{T} to RH\mathbb{R}^{H} and sums the two outputs, so the forecast is y^=Wtrendxtrend+Wseasxseas+b\hat y=W_{\text{trend}}x_{\text{trend}}+W_{\text{seas}}x_{\text{seas}}+b, with no nonlinearity and weights shared across variables. Then the training problem under mean squared error is convex and quadratic in the weights, so the minimizer is unique up to a flat set. The training is stochastic, so no optimizer lands on it exactly: each converges to a neighborhood whose size is set by the step size and the preconditioner, which is the noise floor of the paper's own Appendix A.4. A metric can therefore change how fast the iterates approach the minimizer and at which point early stopping ends training, but not the quality of the solution they approach. The paper reports its largest macro gain on this backbone (7.3%, Table 1) and a 10% gain on ETTh2 (0.469 to 0.421, Table 12). On a convex problem such a large gap can only come from a baseline that stops far from the minimizer, from a different early-stopping or implicit-regularization effect, or from a changed model. None of these is task-conditioned geometry, and the paper does not say which one it is.

W3. The theory is overstated

Proposition 1 is the standard spectral-equivalence bound for a preconditioner together with the optimal-step rate of gradient descent on a quadratic and a first-order local convergence argument under a retraction, all of which are in [1] and [2]. Proposition 2 is the submultiplicativity κ(G−1K)≤κ(G−1D)κ(D−1K)\kappa(G^{-1}K)\le\kappa(G^{-1}D)\kappa(D^{-1}K) with the block-Jacobi spectral bound, and its hypothesis ν<1\nu<1 is impossible whenever KK is singular: if Kv=0Kv=0 then v⊤Dv>0v^\top Dv>0, so D−1/2KD−1/2D^{-1/2}KD^{-1/2} has a zero eigenvalue, λmin⁡(E)=−1\lambda_{\min}(E)=-1 and ν≥1\nu\ge1. Some statements in the main text are contradicted by the appendix: Appendix A.3 says the condition ν<1\nu<1 "is restrictive and is not asserted for the benchmark networks", while Section 4 says Equation 8 "explains why module-aligned blocks can help".

W4. The method appears to be a combination of existing papers, none of which is cited

The gradient-disagreement statistic χt\chi_t (Equation 5) and the descriptor gns (Equation 14), the squared norm of the difference of two mini-batch gradients over the squared norm of their mean, are the gradient noise scale of [3]. The Phase II design in Section 3.3, a variance test that gates L-BFGS, the pair-acceptance rule y⊤s≥ε∥s∥2y^\top s\ge\varepsilon\Vert s\Vert^2 (Equation 4) and Armijo backtracking on mini-batches, is the method of [4]. The block-wise metric on a product manifold built from approximations of the diagonal Hessian blocks, with a Gauss-Newton option (Equations 1 and 2), is the construction of [5]. None of these appears in the related works or references. Apart from these components, the contribution is the map from task descriptors to metric controls (Equation 3), and the paper does not describe how its parameters WW and bb are fitted.

[1] J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 2nd edition, 2006.

[2] P.-A. Absil, R. Mahony and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008.

[3] S. McCandlish, J. Kaplan, D. Amodei and the OpenAI Dota Team. An Empirical Model of Large-Batch Training. arXiv:1812.06162, 2018.

[4] R. Bollapragada, D. Mudigere, J. Nocedal, H.-J. M. Shi and P. T. P. Tang. A Progressive Batching L-BFGS Method for Machine Learning. ICML 2018.

[5] B. Gao, R. Peng and Y. Yuan. Optimization on Product Manifolds under a Preconditioned Metric. SIAM Journal on Matrix Analysis and Applications, 2025. arXiv:2306.08873.

Questions for authors

If all my concerns are addressed, I will be happy to raise my score.

7 comments
AuthorsOpenReview ↗

Response (1/4): Numerical consistency and reproducibility

Dear Reviewer, thank you for your detailed checks of the numerical presentation and mathematical assumptions. We address these points in four responses.

W1: Numerical consistency and reproducibility.

You are correct that different aggregation scopes cannot explain conflicting values for the same named transfer pair. We have corrected the transfer figure using the four individually reported pairs. In the revised paper, F̲i̲g̲u̲r̲e̲ 8̲ and T̲a̲b̲l̲e̲ 1̲9̲ now share these keyed entries:

Source to targetStandard MSEROAM MSERecomputed reduction
ETTh1 to ETTh2.421.3926.89%
ETTm1 to Weather.286.2715.24%
Weather to ECL.201.1895.97%
ECL to Exchange.462.4375.41%

Each reduction is 100(1−MSER/MSEB)100(1-\mathrm{MSE}_R/\mathrm{MSE}_B). Other cells are marked NR. The separate 12-transfer aggregate, .342/.322, does not determine the eight unreported individual pairs. F̲i̲g̲u̲r̲e̲ 1̲3̲ has likewise been regenerated from T̲a̲b̲l̲e̲ 1̲4̲ using the common .312 reference.

Aggregation and setting labels. The Weather-192/TimeMixer .232 in T̲a̲b̲l̲e̲ 2̲ is horizon-specific; the .232 in T̲a̲b̲l̲e̲ 9̲ is a four-horizon average. The revision states these scopes and the macro-aggregation formula in A̲p̲p̲e̲n̲d̲i̲x̲ E̲. Lion .243 and the standard-recipe .245 are also distinct optimizer summaries. T̲a̲b̲l̲e̲ 1̲2̲ now names ETTh1-192/iTransformer, ETTm1-336/PatchTST, and Weather-192/TimeMixer; tournament setting A instead uses ETTh1-96. The reporting inputs preserve these scopes without reconstructing individual horizons from a mean. The broad comparison covers 35 dataset–backbone aggregates; the four optimizer-comparison settings are separately identified.

Implementation and reporting artifacts. The revised Reproducibility Statement and A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲6̲ link the anonymous release. Its evidence map connects each result to its configuration, budget, aggregation rule, and implementation. The release includes roam/controls.py and roam/optimizer.py, plus entry points for source fitting, direct search, exact DLinear, classification, and same-state curvature. aggregate_results.py, audit_records.py, and verify_checkpoints.py check generated run records. Optional forecasting-backbone adapters provide integration entry points; their smoke tests check those interfaces.

The supplied reporting/figures/figure_data.json and figure generator reproduce the added numerical plots and corrected table-derived percentages. Reporting summaries and per-run training artifacts are distinguished: documented training commands generate logs, fitted coefficients, and checkpoints under results/. The aggregate inputs support the reported mean comparisons.

A̲p̲p̲e̲n̲d̲i̲x̲ D̲.̲3̲ states the supplementary environment, model blocks, source/target split, loss normalization, refresh/reset policy, and checkpoint selection. The target cap counts all members, probes, diagnostics, line searches, and validation; source fitting and other excluded costs are accounted for separately. The original benchmark and supplementary protocols retain their respective scopes, and plateau times are not used as population-wide compute ledgers.

AuthorsOpenReview ↗

Response (2/4): Convex DLinear and the exact training reference

W2: DLinear has a convex objective—what can an optimizer improve?

We agree. With fixed moving-average decomposition SS and ordinary affine maps,

=W_{\rm eff}x+b_{\rm eff}.$$ Unregularized MSE is a convex quadratic with redundant parameters. **An invertible SPD metric preserves its stationary set and attainable training minimum.** The revised **S̲e̲c̲t̲i̲o̲n̲ 5̲.̲6̲** states this and examines finite-budget trajectories and validation-selected stopping separately from the exact optimum. **Same-objective experiment.** **A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲2̲** fixes ordinary DLinear, context 96, all seven ETT channels, decomposition, chronological splits, training-only normalization, and zero regularization for both optimizers. Both select a learning rate from $\\{10^{-4},3\times10^{-4},10^{-3},3\times10^{-3},10^{-2}\\}$ on validation seed 2027, then use ten reporting seeds. ROAM receives 2,000 updates; AdamW receives 2,000 or 10,000. These are update allocations, not equal wall-clock budgets. **T̲a̲b̲l̲e̲ 2̲2̲** reports ETTh2-96 below. Training losses use the final iterate; test losses use validation-selected checkpoints. | Method | Final training MSE | Validation-selected test MSE | |---|---:|---:| | AdamW, 2,000 updates | .2270 +/- .0015 | .330 +/- .012 | | AdamW, 10,000 updates | .2205 +/- .0003 | .322 +/- .010 | | ROAM, 2,000 updates | .2213 +/- .0006 | .312 +/- .010 | | Exact least squares | .2200 | .326 | | Validation-selected ridge | .2220 | .310 | Iterative deviations are seed standard deviations. Exact/ridge solutions are deterministic for the fixed split and solver. The exact float64 solve pools windows and channels for the equivalent identifiable shared affine predictor, including an intercept. Its design rank is 97 at relative tolerance $10^{-10}$; $\\|(A^\top A/n)U-A^\top Y/n\\|_F=2.1\times10^{-13}$ is an absolute residual of sample-normalized equations. Ridge changes the objective and serves as a separate predictive reference. **The results separate training convergence from prediction.** At 2,000 updates, ROAM is closer to the exact training minimum than AdamW. Extended AdamW reduces that gap further, while ROAM retains lower selected test MSE; ridge is stronger still on ETTh2-96. Across horizons 96/192/336/720, the test-MSE means are .41425 for AdamW 2k, .40325 for AdamW 10k, and .39650 for ROAM 2k. The individual horizon results are in **T̲a̲b̲l̲e̲ 2̲3̲**, and **F̲i̲g̲u̲r̲e̲ 5̲** separates full-training gaps from validation trajectories. This evidence supports a finite-time trajectory/checkpoint interpretation, including the possible roles of stochastic noise and stopping, without claiming a better convex optimum or optimal test risk. The supplementary zero-decay protocol is distinct from the original **T̲a̲b̲l̲e̲ 1̲1̲** benchmark and its .469/.421 values. The original 7.3% macro improvement is therefore not attributed to changing the predictor or attaining a lower quadratic minimum.
AuthorsOpenReview ↗

Response (3/4): Classical analysis and singular-Hessian applicability

W3: What does the analysis establish, particularly with singular Hessians?

The revised S̲e̲c̲t̲i̲o̲n̲ 4̲ states the classical attribution and conditional scope alongside the main-text results; A̲p̲p̲e̲n̲d̲i̲x̲ A̲ gives the proofs.

Classical ingredients. Proposition 1 applies spectral equivalence, the optimal constant-step quadratic contraction factor, and a local expansion. Nocedal and Wright [1] and Absil et al. [2] are now cited at the start of the main-text analysis as well as in the proofs. These ingredients explain the metric mechanism; they are not claimed as new optimization theorems.

The singular-Hessian limitation. The relevant operator is the symmetric G−1/2KG−1/2G^{-1/2}KG^{-1/2}. If KK is singular and D≻0D\succ0, then D−1/2KD−1/2=I+ED^{-1/2}KD^{-1/2}=I+E is singular, so EE has eigenvalue −1-1 and ν=∣E∣2≥1\nu=\\|E\\|_2\ge1. If DD is singular, the inverse square root in that expression is not defined. Thus the coupled-block result cannot be applied directly to redundant DLinear or a singular neural-network Hessian.

The main text now interprets the bound under its stated positive-definiteness and coupling assumptions: it separates within-block alignment from neglected cross-block coupling. The singular-case argument follows immediately, so applicability is stated alongside the result.

Damping and positive-spectrum diagnostics. Damping makes the metric GG invertible; the loss Hessian KK can remain singular. F̲i̲g̲u̲r̲e̲ 4̲ and A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲3̲ report a positive-spectrum ratio while retaining flat directions in the full parameterization. It is labeled as a diagnostic, not as verification of the nondegenerate-minimum theorem. A regularized operator or a subspace convergence theorem would require its own assumptions.

Purpose of the analysis. It identifies alignment, cross-block coupling, and noise amplification as distinct mechanisms, and states what a positive-curvature pair guarantees in a common metric space. It does not prove convergence of NAdam, stochastic L-BFGS, the metric estimator, or the population controller. The direct-search and same-objective controls test empirical questions separately from those conditional implications. The analysis thus explains the mechanisms under explicit assumptions, while the controlled experiments assess the implemented adaptive procedure.

References

[1] Nocedal and Wright (2006). Numerical Optimization, 2nd ed. Springer.

[2] Absil, Mahony, and Sepulchre (2008). Optimization Algorithms on Matrix Manifolds. Princeton University Press.

Reviewer_BKGGOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Riemannian Optimization with Adaptive Metrics (ROAM), a task-conditioned optimization framework for time-series forecasting. ROAM uses six task descriptors to initialize a set of optimizer controls, including block-wise metric parameters, phase-transition thresholds, and exploration settings for an outer adaptation loop.

With these controls, ROAM (1) adjusts the optimization geometry through task-conditioned preconditioning, (2) dynamically switches between momentum-based and quasi-Newton updates based on gradient disagreement and curvature reliability, and (3) adapts the optimizer controls through a population-based outer loop using validation feedback.

The paper provides a local theoretical analysis of how metric alignment can improve optimization conditioning. Experiments on seven datasets and five forecasting backbones show improvements over standard training. The authors further conduct optimizer comparisons and ablation studies to evaluate the contributions of the proposed components and task descriptors.

Critical strengths
  1. ROAM achieves a lower mean squared error (MSE) than standard training recipes across all 35 dataset-backbone aggregates. Experiments covering seven datasets, five forecasting backbones and four prediction horizons provide broad empirical evidence of ROAM's effectiveness.
  2. The paper includes factorial ablations and descriptor analyses to investigate the contributions of ROAM's different components.
Critical weaknesses
  1. ROAM assumes that task-level statistics can be used to determine suitable optimization metrics. However, the paper does not sufficiently explain why the selected descriptors, such as SNR, spectral entropy, and seasonality, should be informative about the curvature of the model's loss landscape.
  2. Some details of the method are missing:
  • The authors define the curvature proxy F^b\hat{F}_b in Eq. (2), but its actual computation is not sufficiently specified.
  • In Eq. (3), the paper does not clearly explain how the weight matrix (WW) and bias vector (bb) are determined.
  1. Table 3 is hard to interpret. The authors should clearly describe the details of different configurations in the 232^3 factorial, for readers to understand more about the proposed method.
Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response (1/3): Descriptor information and optimization geometry

Dear Reviewer, thank you for identifying the details needed to follow the method. We address your three concerns below.

W1: Why are data-level descriptors relevant to parameter-space geometry?

The revised S̲e̲c̲t̲i̲o̲n̲s̲ 3̲.̲1̲–̲3̲.̲2̲ explain the descriptor-to-control connection. F̲i̲g̲u̲r̲e̲ 1̲ gives the system overview, and F̲i̲g̲u̲r̲e̲ 1̲0̲ separates source fitting, target descriptors, model derivatives, and validation adaptation.

For a centered linear lag predictor with squared loss, K=2E[xx⊤].K=2\mathbb E[xx^\top]. Under stationarity, entries of this lag covariance are autocovariances, which have a spectral representation. Spectral entropy and seasonal concentration can therefore summarize information related to conditioning. Cross-variable correlation density summarizes another part of the input covariance structure. The SNR proxy and initial gradient disagreement instead concern the reliability of stochastic updates, and H/TH/T describes the prediction task.

These are informative summaries, not sufficient statistics for a general Hessian. ROAM uses them to select block scales, damping, curvature mixtures, and phase controls. The actual metric also contains online prediction derivatives and gradient second moments from the instantiated model. Thus model architecture, parameterization, and loss remain part of the geometry even when the data descriptors are unchanged.

The revised initializer fit predicts source-validation-selected controls. Its affine form makes descriptor influence inspectable: before safety clipping, a positive control αb=softplus⁡(zb)\alpha_b=\operatorname{softplus}(z_b) satisfies ∂αb∂ci=sigmoid⁡(zb)Wbisi,\frac{\partial\alpha_b}{\partial c_i}=\operatorname{sigmoid}(z_b)\frac{W_{bi}}{s_i}, where sis_i is the source-feature scale. This expresses the direction and local sensitivity of a control to each descriptor; it is not a theorem that the resulting metric must align with curvature.

The empirical evidence tests separate links. T̲a̲b̲l̲e̲ 1̲4̲ reports .328 mean MSE for random descriptors versus .312 for the six descriptors; individual removals give .315–.319. The new source-mean/direct-search experiment in S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ isolates task-specific initialization within the same metric family. F̲i̲g̲u̲r̲e̲ 4̲ then compares the realized positive-spectrum conditioning at the same initial model state. This connects interpretable inputs, model-dependent operators, and observed optimization behavior.

AuthorsOpenReview ↗

Response (2/3): Numerical definitions and source fitting

W2: How are the initializer and curvature estimate computed numerically?

The revised S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ gives the source-fitting objective, and A̲p̲p̲e̲n̲d̲i̲x̲ D̲.̲3̲ gives its full protocol. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲1̲ and A̲p̲p̲e̲n̲d̲i̲x̲ D̲ specify the curvature realization and metric solve.

Initializer. The supplementary DLinear source set is ETTh1/ETTm1 at horizons 96/192/336/720. Each of eight sources compares 16 control candidates trained for 1,000 updates, with validation every 200 updates. The selected latent controls zkz_k are fitted from featurewise standardized source descriptors by min⁡W,b∑k∣Wc~k+b−zk∣22+∣W∣F2.\min_{W,b}\sum_k\\|W\widetilde c_k+b-z_k\\|_2^2+\\|W\\|_F^2. The ridge coefficient is one; the intercept is unpenalized. Coefficients and source normalization are frozen for held-out ETTh2/ETTm2. The initializer export records coefficients, scales, source identities, and block count. The target population updates controls, not this fitted map.

Curvature and solve. For MSE over mm output scalars, J=2/m,∂fθ/∂θJ=\sqrt{2/m}\\,\partial f_\theta/\partial\theta. Two Rademacher probes yield

\mathbb E[\widehat f]=\operatorname{diag}(J^\top J).$$ The stored proxy is diagonal and computed by reverse-mode products. The metric solve divides each ordinary gradient coordinate by $$\alpha_b[\lambda_b+10^{-6}+\omega_b\widehat f_{b,j}+(1-\omega_b)D_{b,j}],$$ with $D=\operatorname{EMA}(h^2)$ and decay .999. This needs $O(n)$ metric storage/application, plus the differentiation cost of the probes. The applied metric refreshes initially and every 100 updates; metric/control changes reset curvature memory and restart Phase I. Same-batch pairs and two fresh diagnostic batches keep curvature and disagreement measurements distinct. The appendix specifies these choices for the supplementary realization, including its hardware/software environment and data access. `roam/controls.py`, `roam/optimizer.py`, and `docs/IMPLEMENTATION.md` in the [anonymous release](https://anonymous.4open.science/r/ROAM-D935/) provide the corresponding implementation.
AuthorsOpenReview ↗

Response (3/3): Complete factorial and active configurations

W3: What does each factorial configuration activate?

The revised T̲a̲b̲l̲e̲ 3̲ states the active configuration in all eight rows; A̲p̲p̲e̲n̲d̲i̲x̲ F̲ names the task–backbone settings and distinguishes the structural controls.

Switch definitions. M=1M=1 enables the task metric; M=0M=0 uses identity. S=1S=1 enables state-triggered Phase I/L-BFGS transitions; S=0S=0 retains Phase I. O=1O=1 enables validation-driven population replacement and perturbation of active controls; O=0O=0 freezes initialized controls. Freezing online adaptation retains descriptor-based initialization.

MSOActive configurationMean MSE
000Identity; Phase I; fixed controls.333
100Task metric; Phase I; fixed controls.323
010Identity; switching; fixed controls.329
001Identity; Phase I; population selection.327
110Task metric; switching; fixed controls.318
101Task metric; Phase I; outer adaptation.319
011Identity; switching; outer adaptation.324
111Task metric; switching; outer adaptation.312

When M=0M=0, metric coordinates are inactive; when S=0S=0, switching thresholds are inactive. The O-only row denotes population selection, not learning-rate schedule evolution. A population search over unused metric coordinates is not counted as a metric intervention.

Marginal effects. For reported cell mean f(m,s,o)f(m,s,o),

=\tfrac14(.010+.011+.008+.012)=.01025.$$ The switching and outer effects are .00475 and .00525, so **the metric has the largest average marginal effect**. They average four on/off contrasts; they are neither percentages nor additive pieces that must sum to the .021 full-versus-base reduction. **F̲i̲g̲u̲r̲e̲ 6̲** visualizes these descriptive effects without inferring uncertainty from rounded means. **Named settings and scope.** The settings are ETTh1-192/iTransformer, ETTm1-336/PatchTST, and Weather-192/TimeMixer. The first horizon is 192, whereas tournament setting A in **T̲a̲b̲l̲e̲ 2̲** uses 96. Both captions now make this explicit. The separate fixed-identity-plus-PBT control rounds to .326 and is not substituted for the .327 O-only cell. **T̲a̲b̲l̲e̲ 6̲** separately reports ordinary-weight, Stiefel, and SPD-eligible interventions; changing parameter constraints is a distinct structural intervention, not an implicit consequence of changing the optimizer metric. These distinctions keep the factorial interpretation specific to its declared components and populations.
AuthorsOpenReview ↗

Follow-up Letter

Dear Reviewer,

Thanks for your valuable review, which has inspired us to improve our paper further. We kindly wonder if our responses have addressed your concerns.

Following your suggestions, we have implemented the following updates:

  • Clarified the relationship between task descriptors and optimization geometry, distinguishing descriptor-based control initialization from model-dependent curvature estimation in Sections 3.1–3.2‾\underline{\text{Sections 3.1–3.2}}. The information-flow diagram in Figure 10‾\underline{\text{Figure 10}} further separates source fitting, target initialization, and online adaptation.
  • Specified the numerical procedures needed to understand and reproduce the method, including the initializer’s fitting objective, diagonal curvature estimator, coordinate-wise solve, computational cost, and refresh/reset policy in Appendix D‾\underline{\text{Appendix D}}.
  • Expanded the complete factorial into an explicit eight-configuration comparison, identifying the active components, task–backbone settings, and interpretation of the average marginal effects in Table 3‾\underline{\text{Table 3}} and Appendix F‾\underline{\text{Appendix F}}.

The revised presentation follows the full path from task descriptors to metric controls, model-dependent geometry, and optimization behavior. These changes have been incorporated into the revised paper‾\underline{\text{revised paper}} to make the method and its empirical evidence easier to follow.

Thank you again for your dedication and feedback. We look forward to hearing your thoughts on our revisions.

Best regards, The Authors

Reviewer_p4jQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on forecasting optimization and proposes ROAM (Riemannian Optimization with Adaptive Metrics). This method takes specified task into consideration by using six training-derived task statistics to initialize a block-wise positive-definite metric, which guides parameter updates through task-conditioned preconditioning. It further introduces a state-dependent switching mechanism between momentum-based and quasi-Newton optimization and a population-based outer loop to adapt the optimization controls according to validation performance and computational costs. The paper provides a theoretical analysis relating metric alignment to local conditioning and convergence behavior. Experiments on seven forecasting datasets with five backbones demonstrate improvements over standard training strategies.

Critical strengths
  1. This paper shows an interesting perspective on forecasting optimization by incorporating task-specific information into the optimization process. As forecasting tasks often have heterogeneous characteristics, it's practically meaningful to adapt the optimization geometry according to task statistics. The proposed method adjusts parameter updates without modifying the model architecture, and its effectiveness has been proven by comparison with common optimizers.

  2. The theoretical analysis is comprehensive and well-developed. This paper provides a detailed analysis of how metric alignment affects optimization conditioning and local convergence. It also considers cross-block coupling, stochastic gradient noise, and curvature-related stability, which further support the design of the proposed method. The mathematical derivations and proofs are relatively complete and provide a solid theoretical foundation for the main optimization mechanisms.

  3. The experimental evaluation is relatively comprehensive for multivariate time-series regression tasks. This paper conducts experiments on seven datasets with different forecasting horizons and five backbone architectures, demonstrating consistent improvements over standard optimization methods. It also includes comparisons with stronger optimizers, component ablation studies, task-descriptor analyses, and optimization efficiency evaluations. These experiments provide sufficient empirical evidence to support most of the paper's main claims within the evaluated forecasting tasks.

Critical weaknesses
  1. The mapping from task statistics to the optimization metric is not sufficiently explained. The paper uses six task descriptors and an affine transformation parameterized by (W) and (b) to initialize the metric controls. However, it remains unclear how (W) and (b) are learned or determined, and how these task statistics are related to the curvature properties of the loss landscape. More details on the learning objective and the relationship between task descriptors and metric alignment would help clarify the core task-conditioned mechanism.

  2. The effectiveness of task-conditioned metric initialization requires further verification. Although the paper compares ROAM with several optimization methods and includes descriptor ablations, it is unclear whether directly tuning the metric parameters on the validation set under the same computational budget could achieve comparable performance. Such a comparison would help distinguish the contribution of task-specific information from the benefits of additional hyperparameter optimization.

  3. The practical implementation of the metric matrix is not sufficiently specified. The paper defines a block-wise metric using Fisher/Gauss–Newton-type curvature and gradient second-moment approximations. However, it does not clearly explain how these matrices are represented and computed in practice, such as whether diagonal, low-rank, or other structured approximations are adopted. More implementation details on metric construction, inversion, and computational complexity are needed to better understand the practical feasibility and reproducibility of the proposed method.

  4. The experimental evaluation is limited to regression tasks. Although the paper considers multiple datasets and backbone architectures, all experiments focus on multivariate time-series forecasting. It remains unclear whether the proposed task-conditioned optimization framework can generalize to other learning paradigms, such as classification. Additional experiments on different task types would provide stronger evidence for the general applicability of ROAM.

Questions for authors

As mentioned in the weaknesses, I would appreciate further clarification and discussion of these issues.

4 comments
AuthorsOpenReview ↗

Response (1/4): Descriptor-to-metric mechanism and source fitting

Dear Reviewer, thank you for your thoughtful assessment of task-aware optimization. We address your four concerns below.

W1: How do descriptors determine the metric, and how is the initializer fitted?

The revised S̲e̲c̲t̲i̲o̲n̲s̲ 3̲.̲1̲–̲3̲.̲2̲ distinguish control selection from model-dependent curvature estimation; A̲p̲p̲e̲n̲d̲i̲x̲ D̲.̲3̲ gives the calibration protocol.

What the descriptor determines. For block bb, Gb=αb[(λb+δ)I+ωbF^b+(1−ωb)Db].G_b=\alpha_b[(\lambda_b+\delta)I+\omega_b\widehat F_b+(1-\omega_b)D_b]. The descriptor initializes block scales, damping, curvature mixtures, phase thresholds, and exploration settings. The current model, parameterization, loss, and batch enter through F^b\widehat F_b and DbD_b. The same data descriptors can yield different parameter-space metrics for different models. Initial gradient disagreement is itself model-dependent.

How the initializer is fitted. The supplementary DLinear calibration uses eight source tasks: ETTh1 and ETTm1 at horizons 96, 192, 336, and 720. Each evaluates 16 candidates for 1,000 updates, with source validation every 200 updates. Let zkz_k be the selected controls in latent coordinates and c~k\widetilde c_k the source-standardized descriptors. The fit is min⁡W,b∑k∣Wc~k+b−zk∣22+∣W∣F2.\min_{W,b}\sum_k\\|W\widetilde c_k+b-z_k\\|_2^2+\\|W\\|_F^2. The ridge coefficient is one and the intercept is unpenalized. Domain transforms and safety bounds yield feasible controls. Coefficients and feature normalization are frozen for held-out ETTh2/ETTm2; the target population adapts controls, not W,bW,b. The initializer dimension follows the model partition. This objective predicts source-validation-selected controls, rather than minimizing the theorem's alignment bound.

Why the statistics can help. For a centered linear lag predictor under squared loss, K=2E[xx⊤]K=2\mathbb E[xx^\top]. Under stationarity, lag covariance is determined by autocovariances and their spectral representation. Spectral concentration and seasonality can therefore be informative without identifying the full spectrum. Correlation density summarizes cross-variable dependence, while the SNR proxy and gradient disagreement concern update reliability. S̲e̲c̲t̲i̲o̲n̲ 3̲.̲2̲ also gives the explicit sensitivity of a positive control to a descriptor through the fitted affine coefficients.

Empirical assessment. T̲a̲b̲l̲e̲ 1̲4̲ reports .312 mean MSE with the six descriptors and .328 with random descriptors. The new source-mean/direct-search controls in S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ and same-state curvature diagnostic in F̲i̲g̲u̲r̲e̲ 4̲ test more specific links in the mechanism. Proposition 1 remains a conditional implication from alignment to conditioning; it does not assert that the affine map always achieves alignment.

AuthorsOpenReview ↗

Response (2/4): Direct metric search and calibration cost

W2: What does task conditioning add over direct metric search?

We have added same-family controls in the revised S̲e̲c̲t̲i̲o̲n̲ 5̲.̲4̲ and F̲i̲g̲u̲r̲e̲ 3̲, with the full protocol in A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲1̲.

Matched target comparison. All arms use ordinary DLinear, context/horizon 96/96, the same metric family and approximation, initialization-seed pool, switching rule, four-member population, data split, and validation-only checkpoint selection. Only initial controls differ:

  • Conditioned: evaluate the frozen source-fitted affine map on the target descriptor.
  • Source mean: retain the source-fitted latent intercept and remove the descriptor-dependent term.
  • Direct search: initialize four candidates from the same feasible metric-control domains, without the descriptor map.

All three retain the same target population adaptation. Each population receives a 20-second training/validation cap, five evaluation rounds, and 20 model-level validation calls. Entries are mean +/- seed standard deviation over ten seeds.

Initial controlsETTh2-96 MSEETTm2-96 MSE
Conditioned.312 +/- .010.174 +/- .004
Source mean.318 +/- .011.177 +/- .005
Direct search.315 +/- .012.173 +/- .005

Conditioning improves on source mean on both targets. Direct search is slightly stronger on ETTm2, so the result supports a task-dependent initialization benefit rather than universal superiority to metric search. The marginal deviations alone do not establish paired significance.

Calibration-inclusive comparison. Source calibration costs 600 seconds. With target adaptation of 20 seconds, the source-plus-target subtotal is 620 seconds for the first task, 80 seconds per task over ten targets, and 26 over 100. Loading, descriptor construction, and final reporting are outside these subtotals. More generally, C‾(N)=Csource/N+Cdescriptor+Ctarget.\overline C(N)=C_{\mathrm{source}}/N+C_{\mathrm{descriptor}}+C_{\mathrm{target}}. Under an equal 620-second subtotal allocation on ETTh2-96, direct search obtains .303 versus .312 for conditioning, averaged over the three matched seeds 0–2. These are separate from the ten-seed target comparison above. Direct search is stronger for the first task; source calibration can be shared across repeated deployments.

Source mean retains calibration investment while removing task-specific initialization; direct search tests the same metric family under either the target cap or the calibration-inclusive allocation. These controls separate the value of task information from the compute invested in selecting controls.

AuthorsOpenReview ↗

Response (3/4): Curvature representation, solve, and implementation cost

W3: What is stored, how is the metric solved, and what does it cost?

The revised S̲e̲c̲t̲i̲o̲n̲ 3̲.̲1̲ defines the estimator and coordinate-wise solve; A̲p̲p̲e̲n̲d̲i̲x̲ D̲ specifies the supplementary implementation and refresh policy.

Diagonal prediction-Jacobian estimator. For MSE averaged over mm output scalars, set J=2/m,∂fθ/∂θJ=\sqrt{2/m}\\,\partial f_\theta/\partial\theta. Two independent Rademacher output probes give f^=12∑r=12(J⊤vr)⊙(J⊤vr),F^b=diag⁡(f^b).\widehat f=\tfrac12\sum_{r=1}^{2}(J^\top v_r)\odot(J^\top v_r),\qquad \widehat F_b=\operatorname{diag}(\widehat f_b). At fixed parameters and batch, E[f^]=diag⁡(J⊤J)\mathbb E[\widehat f]=\operatorname{diag}(J^\top J). Reverse-mode vector–Jacobian products avoid materializing the Jacobian or a dense Gram matrix. The stored approximation is diagonal; two probes do not make it a rank-two metric.

The separate term is Dj=EMA⁡(hj2)D_j=\operatorname{EMA}(h_j^2) with decay .999 and ordinary loss gradient hh. It is not the same estimator as the prediction-Jacobian construction.

Explicit solve and cost. gb,j=hb,jαb[λb+10−6+ωbf^b,j+(1−ωb)Db,j].g_{b,j}=\frac{h_{b,j}}{\alpha_b[\lambda_b+10^{-6}+\omega_b\widehat f_{b,j}+(1-\omega_b)D_{b,j}]}. The damped denominator is strictly positive. Metric storage and application are O(n)O(n), excluding network differentiation. Each refresh adds a forward computation and two reverse-mode probes. L-BFGS separately requires O(Mn)O(Mn) storage and recursion work with M=10M=10; the population has four model/optimizer states.

Refreshes and consistent memory. The applied metric is refreshed initially and every 100 updates, and frozen between refreshes while DD accumulates. A metric or control change clears curvature memory and restarts Phase I. Curvature pairs use the same batch before and after a step under the same frozen metric. Diagnostics use two additional independent batches every ten updates, with three successful checks for entry and three failures for exit.

The supplementary forecasting runs use ordinary weights, additive retraction, and identity coordinate transport. They activate no Stiefel or SPD parameter constraints. A̲p̲p̲e̲n̲d̲i̲x̲ C̲ separately describes the restricted tangent-space solve needed for constrained configurations.

The added implementation specification is identified with the supplementary experiments. The original benchmark cost table and the supplementary population cap retain separate scopes; a per-epoch measurement or plateau time is not substituted for population-wide compute. Code is available in roam/optimizer.py and docs/IMPLEMENTATION.md in the anonymous repository.

AuthorsOpenReview ↗

Response (4/4): Subject-disjoint temporal classification

W4: Does the framework extend beyond regression?

We have added a subject-disjoint UCI HAR classification study [1], summarized in the revised S̲e̲c̲t̲i̲o̲n̲ 5̲.̲7̲. A̲p̲p̲e̲n̲d̲i̲x̲ L̲.̲5̲ specifies the loss interface and splits, with results in T̲a̲b̲l̲e̲ 2̲6̲ and F̲i̲g̲u̲r̲e̲ 1̲6̲.

Loss and descriptor changes. For softmax cross-entropy, logit curvature is diag⁡(p)−pp⊤\operatorname{diag}(p)-pp^\top, giving F=J⊤[diag⁡(p)−pp⊤]J.F=J^\top[\operatorname{diag}(p)-pp^\top]J. The implementation estimates its nonnegative diagonal using sampled softmax labels and independent signs across examples. SNR, spectral entropy, seasonal concentration, and correlation density are averaged over training windows; gradient disagreement uses the classification loss. There is no forecast horizon, so H/TH/T is set to zero and a separate initializer is fitted.

Subject separation and model. We use the official held-out test subjects and a fixed CNN with two 32-channel convolutional layers, GELU, temporal averaging, and a six-class head. Twenty percent of official training subjects form target validation. Six internal source splits, each comparing eight candidates for 500 updates, use only the remaining training subjects. Neither target validation nor test subjects enter source calibration. The model has three optimizer blocks and ordinary weights.

Results. Four population arms share a 30-second target cap, five rounds, 20 model-level validation calls, batch size 64, and ten reporting seeds. Source calibration is separate. AdamW uses a distinct 1,500-update allocation. Accuracy deviations are percentage points.

MethodAccuracy (%)Cross-entropyBudget
Conditioned ROAM92.8 +/- .6.215 +/- .01830 s / 20 calls
Source mean92.3 +/- .7.232 +/- .02030 s / 20 calls
Direct metric search93.1 +/- .6.207 +/- .01730 s / 20 calls
Continuous blend92.9 +/- .6.212 +/- .01830 s / 20 calls
AdamW reference93.5 +/- .5.194 +/- .0161,500 updates

Conditioning improves over source mean in both metrics; direct search and blending have higher mean accuracy. The separately budgeted AdamW reference is also stronger. The extension demonstrates the loss interface and descriptor convention on temporal classification, using internal subject splits for calibration. Its scope is this dataset and classifier; the table identifies the stronger reference outcomes and their budgets.

References

[1] Reyes-Ortiz et al. (2013). Human Activity Recognition Using Smartphones. UCI Machine Learning Repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.