acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Optimizer in Spectral Time-Series Forecasting

Abstract

Multi-branch architectures in spectral time-series forecasting are commonly presumed to expand representational capacity. This paper demonstrates that such structural additions often merely counteract a hidden optimization pathology rather than enrich expressive power. To isolate this phenomenon, we utilize a modular Phase-Structured Predictor (PSP) that decouples phase, frequency, and input-conditioned pathways, establishing a controlled diagnostic testbed. Through this architecture, we reveal that standard adaptive optimizers (e.g., AdamW) leave an acute update-scale attenuation in forecast space due to the inverse Fourier transform, severely throttling terminal kernel adaptation and corrupting upstream representations. We prove that under whole-model closure, multi-branch dynamics can be rigorously collapsed into a single effective kernel. Grounded in this equivalence, we introduce SpecScale (Spectral Scale Optimizer): theoretically, its Exact formulation strictly reproduces multi-branch training dynamics step-for-step with zero trajectory divergence; practically, it yields an efficient, branch-free rule that applies an analytic frequency multiplier directly to base-kernel AdamW updates. On a four-cell diagnostic panel under closure, SpecScale recovers of the multi-branch empirical gain. On the full 32-cell weighted benchmark, SpecScale reduces test MSE by (improving 27 of 32 settings under equal-budget selection) while matching a 15-setting hyperparameter grid search with fewer trials and zero inference overhead. Furthermore, readout corrections consistently transfer to three prominent external backbones, confirming that readout scale attenuation is an architecture-agnostic optimization bottleneck.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Post-Hoc Correction, Residual Calibration, Spectral Analysis, Model Selection, Shrinkage Estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o1isOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how spectral readout parameterization affects optimization in time-series forecasting. It introduces a foldable Quadrature Reparameterization and derives SpecScale-Exact, a single-kernel optimizer that reproduces the two-branch AdamW trajectory under whole-model closure. It then develops a practical Fixed-Bin update multiplier and evaluates the approach through forecasting benchmarks, trajectory checks, and interventions intended to identify the source of the gains.

Critical strengths
  1. SpecScale-Exact is the strongest technical contribution. It extends function equivalence to whole-model training trajectory equivalence by accounting for Adam’s moment states, finite epsilon, matched decay, and virtual global clipping.

  2. The experiments investigate the mechanism rather than relying only on benchmark accuracy. Frozen-feature controls, bidirectional rate interventions, and simple multiplier controls support the importance of readout update scale.

Critical weaknesses
  1. The theoretical distinction from prior work needs sharper positioning. Section 2 characterizes RepOptimizers [1] as primarily addressing SGD, although its paper and official implementation already include AdamW, and the manuscript does not discuss the stateful optimizer trajectory equivalence in [2]. The contribution should be identified more precisely as moment-state compression for the particular coordinate structure, together with finite-epsilon and global-clipping treatment.

  2. The overall $ O(1/n) $ attenuation claimed in Section 3.1 does not follow from the per-bin $ c_b/n $ factor alone, because prediction displacement also depends on spectral amplitudes, the number of contributing bins, and update directions. A fixed-amplitude cosine under an unnormalized FFT can have a spectral coefficient proportional to n, cancelling this factor.

[1] Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Kaiqi Huang, Jungong Han, and Guiguang Ding. Re-parameterizing Your Optimizers rather than Architectures. ICLR, 2023. [2] Lucas Torroba-Hennigen, Hunter Lang, Han Guo, and Yoon Kim. On the Duality between Gradient Transformations and Adapters. ICML, 2025.

Questions for authors
  1. Relative to the AdamW implementation of [1] and the trajectory-equivalence theorem of [2], which parts of Proposition 1 are new? Please distinguish general linear reparameterization equivalence from compression to one moment pair under the specific structure used here.
Reviewer_eTLsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies an optimization-scale problem in spectral time-series forecasting and propose SpecScale. It establishes exact equivalence between certain multi-branch and single-kernel training dynamics under a closure assumption. Experiments with 8 datasets show improved forecasting accuracy without additional inference overhead.

Critical strengths

The exact trajectory equiv result under whole model closure is interesting and supported by numerical verification. The extensive experiments and controlled optimization studies provide useful evidence that parameterization can affect learning even without increasing representational capacity.

Critical weaknesses

The main theoretical concern is the claimed O(1/n) attenuation in Eqn 2. Since the number of frequency bins also grows with n the stated scaling does not immediately follow for the complete forecast update. The argument needs clearer assumptions about spectral magnitudes and aggregation across frequencies.

The practical novelty is also somewhat limited. Appendix D.2 shows that uniform or tuned learning-rate scaling matches or exceeds SpecScale on the diagnostic panel. The Strong Host baseline achieves nearly identical performance with additional tuning, while improvements on external architectures are very small. Ergo, the evidence for a uniquely beneficial frequency-dependent optimization rule is limited.

Questions for authors

Can the authors rigorously justify the O(1/n) attenuation for the complete prediction rather than individual frequency contributions? What advantage does the proposed frequency-dependent multiplier provide over a uniform readout learning-rate multiplier, particularly given the results in Appendix D.2?

Reviewer_9UjxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the benefits of auxiliary branches in spectral time-series forecasting arise from improved optimization rather than increased representational capacity. The authors argue that inverse Fourier scaling attenuates parameter updates in prediction space, creating an optimization bottleneck at the spectral readout. They introduce the Phase-Structured Predictor (PSP) as a controlled testbed and use Quadrature Reparameterization (QR) to examine this effect. Under a whole-model closure condition, they derive a single-kernel optimizer that reproduces the multi-branch training trajectory. They also propose a practical SpecScale rule that applies frequency-dependent multipliers to the readout kernel’s AdamW updates. Experiments demonstrate improvements on PSP, while additional correction-branch experiments examine transfer to existing forecasting models.

Critical strengths
  1. The effort to distinguish representational capacity from optimization effects is valuable. Frozen-feature experiments and update-scale interventions provide useful evidence about the source of the improvements within PSP.

  2. The conditional analysis of training-trajectory equivalence accounts for AdamW’s moment states, finite epsilon, and gradient clipping, and motivates a simple practical update rule.

Critical weaknesses
  1. The generality of the central explanation remains insufficiently established. Prediction-space updates depend on the input spectrum and FFT normalization, not solely on the inverse transform’s (1/n) factor. The analysis does not clearly establish when this factor produces a general optimization bottleneck.

  2. The practical advantage beyond learning-rate tuning remains limited. Tuning the readout learning rate largely closes the performance gap on PSP. Improvements on existing models are small and primarily obtained through auxiliary corrections, leaving the direct applicability of branch-free SpecScale insufficiently demonstrated.

Questions for authors
  1. Under what conditions does the claimed update attenuation hold after accounting for input-spectrum magnitude and FFT normalization? Although the inverse transform contains a (1/n) factor, spectral coefficients may also scale with (n), potentially offsetting it. Clarifying these conditions would help distinguish a general bottleneck associated with the inverse transform from an issue specific to the chosen parameterization and training configuration.

  2. When applied directly to existing spectral forecasting models, what advantage does SpecScale offer over readout learning-rate tuning under the same search budget? On PSP, sufficiently tuning the readout learning rate achieves similar performance, while the external-model experiments primarily add correction branches. A direct comparison would help establish whether SpecScale generalizes beyond PSP and whether its main practical benefit is improved accuracy or reduced tuning cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.