acceptodds
Under review as a conference paper at ICLR 2027

Excess Spectral Suppression: Why Deep Forecasters Sacrifice Predictable Tail Modes

Abstract

Deep forecasters are widely reported to over-smooth their targets. Yet any MSE-optimal predictor must discard unpredictable variance, so comparing output spectra with data spectra cannot separate legitimate shrinkage from genuine bias, and no existing definition of that bias is anchored to a Bayes reference. We supply one: excess suppression (ES), the per-mode skill gap to the Bayes reference, measured in the SVD mode space of the target matrix. We measure it on a synthetic family with an analytic Bayes predictor and on eight real benchmarks under a conservative attainable-skill proxy, across rank and time budgets, four intervention families (one applied at inference time), and three zero-shot foundation forecasters. The sacrifice is all-or-nothing: under a tight rank budget, models realize none of the attainable tail skill (median unrealized-to-attainable ratio over all skill-bearing channels), and tail-ES falls monotonically as the bottleneck widens. The effective rank of the prediction operator is the only axis that tracks the sacrifice across every intervention (Spearman ); feature-spectrum entropy dissociates from it. Per-mode loss reweighting closes – of the tail gap while improving overall MSE on all four datasets with attainable tail skill, and the foundation forecasters show tail-ES within of zero on all eight benchmarks. Tail sacrifice is thus an artifact of narrow budgets and the MSE objective, not of the data: it is measurable, diagnosable at inference time, and correctable in the loss. Code and run outputs are available at https://anonymous.4open.science/r/excess-spectral-suppression-6980.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, over-smoothing, spectral bias, excess suppression, Bayes reference, reduced-rank regression, effective rank, rank collapse, loss reweighting, time series foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1m2XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces excess spectral suppression (ES), a mode-wise prediction-skill gap relative to a Bayes or attainable reference. Through synthetic experiments and eight forecasting benchmarks, it studies how capacity and training budgets affect predictable tail modes. It also investigates operator effective rank and proposes predictability-aware loss reweighting to reduce tail errors.

Critical strengths

Evaluating mode-wise errors against an attainable reference provides a useful alternative to diagnosing forecasting bias from output smoothness alone.

The controlled synthetic experiments and interventions on width, regularization, and head rank provide informative evidence about the relationship between capacity constraints and tail prediction errors.

Critical weaknesses

ES does not uniquely identify suppression. Underprediction and over-amplification can produce identical ES and its decomposition, weakening the interpretation of “complete sacrifice” on real data.

The noise-floor calibration is not statistically convincing. Appendix B acknowledges that the DLinear-based floor mainly reflects architecture performance gaps, so falling below it cannot establish the absence of suppression in modern forecasters.

Questions for authors

For a centered mode, predicting 00 or 2f⋆(x)2f^\star(x) yields identical ES and “unrealized skill.” What additional evidence distinguishes actual suppression from miscalibration?

Do the conclusions about modern forecasters hold when uncertainty is estimated using repeated strong reference models and independent reference selection, rather than the weakest member’s performance gap?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1m2X: Direction Diagnostics, Cross-Fitted Reference & Seed-Noise Calibration

We thank the reviewer for two precise soundness objections. Both pointed at real gaps, and the resulting experiments changed the paper. Sections and tables refer to the revised PDF.

W1 / Q1. Can ES tell suppression from miscalibration (predicting 00 vs. 2f⋆(x)2f^\star(x))?

The reviewer is right that R2R^2-based ES cannot: on a centered mode, both predictions give Rk2=0R^2_k=0. The two cases differ in information content, so we now report two scale-free diagnostics per tail mode (re-run measurement study, 26 skill-bearing channels, modes 25–48):

  • the squared correlation between predicted and target mode projections. A 2f⋆2f^\star forecaster attains the reference value; one without the information attains ≈0\approx 0;
  • the R2R^2 after a per-mode affine recalibration fitted on validation windows. This recovers all the skill of a miscalibrated forecaster.
modelamplitude ratiosquared corr.recalibrated R2R^2raw R2R^2
MLP, r=8r{=}80.5530.0010.000−0.993
deep linear, r=8r{=}80.5760.0020.000−0.680
RRR, r=8r{=}8 (closed form)0.0200.0010.0000.001
MLP, r=96r{=}960.2510.0520.045−0.026
RRR, r=32r{=}320.2960.1350.1240.118

The attainable reference is R⋆2=0.185R^2_\star=0.185. At r=8r{=}8 the tail predictions are uncorrelated with the target, and no recalibration recovers any skill. The MLP's tail energy (amplitude 0.55) is uncorrelated with the target, which is exactly our "injected error" term. This mirrors the amplitude/decorrelation split of Subich et al. (ICML 2025), now cited. (App. C.3, Table 9.)

One related change: the closed-form RRR-8 baseline shows that the r=8r{=}8 endpoint is budget-mandated, so "complete sacrifice" is no longer a headline. The new headline is that trained r=32r{=}32 networks leave 2.0–3.5× more tail skill unrealized than the rank-32 optimum (§3.4, Table 2). Their tail predictions are also nearly uncorrelated with the target (MLP-32: 0.017, vs. 0.135 for RRR-32), so the same diagnosis holds.

W2 / Q2. Noise floor; do the modern-forecaster conclusions survive strong repeated references and independent reference selection?

We agree the DLinear-twin floor mostly measured an architecture gap, and adopted the two controls the reviewer proposes:

  • Independent reference selection (cross-fitting). The per-mode best member is selected on one chronological half of the test windows and scored on the other, and the halves are then swapped. Winner's-curse inflation is small: the pooled skill-bearing tail reference drops from 0.198 to 0.185.
  • Repeated strong references. The ensemble member is re-trained with 3 new seeds. Its tail R2R^2 moves by 0.006–0.008 on Traffic and Electricity, and by ≤ 0.026 on any dataset with skill-bearing channels. Swapping every member for its re-trained copy shifts the reference by ≤ 0.001. The old floors (0.19–0.25) were therefore architecture gap, not noise.

Modern forecasters under this calibration (8 benchmarks; new: shared-weight PatchTST, dmodel∈{16,…,128}d_{\text{model}}\in\{16,\dots,128\}). Values are tail ES / unrealized component, all measured channels; seed noise is 0.007 (Traffic) and 0.006 (Electricity):

modelTraffic (R⋆2R^2_\star 0.248)Electricity (R⋆2R^2_\star 0.240)
iTransformer−0.021 / −0.0310.025 / 0.001
shared-weight PatchTST, dd 16–128−0.031 to −0.060 / −0.040 to −0.0640.013–0.026 / −0.013 to −0.004
per-channel PatchTST (the §6 configuration)0.084 / 0.0270.060 / 0.012

So yes: the conclusion does not rest on a loose floor. On Traffic these models beat the reference; on Electricity the unrealized part is ≤ 0.001 for the shared-weight and cross-channel models, and the remaining ES is injected error. The per-channel PatchTST is above seed noise on Traffic (0.027) but at about a tenth of the attainable 0.248: small, not absent. For the zero-shot foundation models (already in [−0.028,+0.046][{-}0.028, {+}0.046] under the stricter un-cross-fitted reference), we likewise drop the floor comparison and state "within ±0.05" (§6).

The tighter calibration also cuts the other way. On Traffic and Electricity, full-rank (r=96r{=}96) networks leave 0.129–0.209 of tail skill unrealized, 18–35× seed noise. Previously "at floor level", this is now claimable.

Changes: §3.1 ("lower bound" withdrawn; both bias directions stated); Table 1 (cross-fitted, decomposed, seed-noise column); App. C.1, C.4.

Presentation. We restructured the paper:

  • formal definition moved from the introduction to §3.1; propositions moved to §3.3, before their first use;
  • "Phase A/B/C" replaced by named measurement, intervention and loss studies, with a roadmap and transitions opening §4–§5;
  • a Conclusion (§8); channel-selection rules in App. B; all new analyses in App. C.

Thank you for the response. Most of my concerns have been addressed, so I am considering raising my score by one point. However, the manuscript’s overall quality, including its explanation of the underlying mechanisms, may still fall short of the standard required for acceptance.

Thank you for your reply and for considering raising your score.

Since our initial response focused on your two soundness questions, we briefly summarize how the revised manuscript establishes the underlying mechanism (§§3.3–5):

  1. Why MSE sacrifices predictable low-energy modes first (§3.3): Under MSE, both a hard rank ceiling (Proposition 1) and gradient-descent learning speed (Proposition 2, learning time τk∝1/(σkρk)\tau_k \propto 1/(\sigma_k\sqrt{\rho_k})) prioritize modes by sk=σkρks_k = \sigma_k\sqrt{\rho_k}, where σk\sigma_k is mode energy and ρk\rho_k is predictability. Because σk\sigma_k spans orders of magnitude while ρk≤1\sqrt{\rho_k} \le 1, low-energy modes receive tiny MSE gradients and are dropped first — even when highly predictable.
  2. Separating hard capacity limits from optimization/objective bias (§3.4, §5): Comparing trained networks against closed-form Reduced-Rank Regression (RRR, the exact MSE-optimal rank-rr linear predictor) separates the two on real data. At rank r=8r{=}8, tail loss is mandated by the rank ceiling (RRR-8 also drops the tail). At r=32r{=}32, however, RRR-32 recovers over half the tail skill while trained networks leave 2.0–3.5× more tail skill unrealized — an optimization/objective effect that fixed-capacity inverse-energy loss reweighting (§5) recovers (closing 31–64% of the gap while also improving overall MSE).
  3. Pinpointing the bottleneck inside the model (§4): Prior work attributes spectral loss to collapse of internal feature rank, but §4 shows feature rank is only a proxy: dropout crushes internal feature rank without hurting tail skill, whereas inference-time SVD truncation of the output head sacrifices tail skill without touching internal features. Across all four intervention families, what actually governs tail sacrifice is the effective rank of the end-to-end input-to-prediction operator (Spearman −0.89-0.89 with unrealized tail skill, vs. −0.40-0.40 for feature rank).

We appreciate your engagement and are glad to clarify any point further.

Reviewer_MBA4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates the phenomenon that deep forecasting models tend to smooth their targets. This is explained with a nice analogy to popularity bias in recommender systems, where niche items that are easy to predict might be poorly predicted due to models focusing on the popular items. The paper shows that when the forecasting model is constrained, data modes with low energy but high predictability are the first to be sacrificed. The paper proposes a protocol, shows the effect on synthetic and real data, and shows how it can be corrected in practice.

Critical strengths

The paper provides a protocol, actual experimental evidence, and algorithmic ideas.

The question asked in the paper is valuable and can lead to interesting insights.

Critical weaknesses

The main results are presented in the context of multilayer perceptron architectures, which raises the question how the insights extrapolate to current forecasting pipelines. The authors have looked into this (Section 6) but it does limit the significance of the work.

The structure of the paper needs to be improved. Mathematical contributions are already presented in the introduction. Proposition 1 is referenced before it is introduced. The paper talks about a Phase A and Phase C without explaining what they are. There is a lack of connecting text between sections, and also within sections, that makes the paper hard to follow. There are no conclusions.

Questions for authors

Section 3.2 describes how the authors create the synthetic model, but the motivation for this exact choice is limited. Can the authors comment on that?

Can the authors comment more on the impact of not having the actual Bayes reference of the real data sets?

1 comment
AuthorsOpenReview ↗

Response to Reviewer MBA4: Modern/Lightweight Pipelines, Paper Restructuring, Synthetic Design & Reference Biases

We thank the reviewer for a constructive review that led to two main changes: a narrowed scope with new evidence beyond MLPs, and a restructured paper. Sections and tables refer to the revised PDF.

W1. Main results are on MLPs; how do they extrapolate to current pipelines?

A clarification: the intervention study (§4) already used PatchTST-style Transformers, and §6 covered PatchTST, iTransformer and three foundation forecasters; still, the concern is right in substance. New experiments (8 benchmarks, cross-fitted reference, seed-noise calibration; App. C.4):

  • Standard-capacity modern pipelines show no claimable sacrifice. We tested a shared-weight, channel-independent PatchTST at four widths (dmodeld_{\text{model}} 16–128, official hyperparameters), and iTransformer. On Traffic their tail ES is −0.021 to −0.060 (they beat the reference); on Electricity their unrealized tail skill is ≤ 0.001. The per-channel PatchTST leaves a small 0.027 on Traffic.
  • The sacrifice appears where a budget binds, including in a published lightweight model. Lowering the cut-off frequency of FITS, its own compression knob, monotonically raises unrealized tail skill (Spearman −0.94 / −1.00 on Traffic / Electricity); on Electricity it ranges from ≈ 0 to 97% of the attainable skill while total MSE changes by under 0.6%. At its default cut-off FITS shows no sacrifice, and neither does SparseTSF.
  • Trained networks exceed their rank budget. At rank 32, bottlenecked networks leave 2.0–3.5× more attainable tail skill unrealized than the MSE-optimal rank-32 linear predictor (Table 2).

This limits the significance for standard-capacity pipelines, so we narrowed the title and abstract: "Excess Spectral Suppression: Why Budget-Constrained Forecasters Sacrifice Predictable Tail Modes", with a new "Scope" paragraph. The significance that remains is for compressed and lightweight forecasters, plus a use of the diagnostic. Standard models look over-smoothed (tail amplitude 48–57% of the target) yet lose no attainable skill; a spectrum comparison would flag them, and restoring amplitude is costly (an AMSE-style loss raises our MLPs' MSE by 11–49%). ES separates this shrinkage from genuine loss in bottlenecked networks and narrowly cut FITS (tail predictions nearly uncorrelated with the target). In practice: low tail amplitude with ES ≈ 0 needs no fix; after compressing a model, check unrealized tail skill, which aggregate MSE does not show.

W2. Structure. We agree with each point. The formal definition moves from the introduction (which keeps a verbal one) to §3.1, and the propositions to a new §3.3, before their first use. "Phase A/B/C" become the measurement, intervention and loss studies, with a roadmap. Transitions open §4 and §5, and a new §8 Conclusion is added. The main text stays within 10 pages; new analyses are in App. C.

Q1. Motivation for the synthetic family (§3.2). It is the simplest family with three properties that real data cannot offer together:

  1. An exact Bayes reference: Rk2(f⋆)=ρkR^2_k(f^\star)=\rho_k, so ES is measured without any proxy.
  2. Energy and predictability set independently per mode (on real data they are correlated). The alternating ρk∈{0.9,0.05}\rho_k\in\{0.9,0.05\} puts a predictable and an unpredictable mode at every energy level, so all four buckets exist; geometric energies fix the energy ordering.
  3. Closed forms that apply exactly. With orthonormal A,VA,V and whitened inputs, reduced-rank regression and deep-linear dynamics (Propositions 1–2) hold exactly, so trained networks are checked against them (Fig. 1).

The price is linear-Gaussian structure; the real-data study tests whether the ordering transfers. These reasons are now in §3.2.

Q2. Impact of not having the Bayes reference on real data. We use the per-mode best of a fixed model family (deep ensemble, OLS, DLinear). It can err both ways; we now control each (§3.1, App. C.1):

  • Upward bias (picking the luckiest model on the evaluation windows): removed by cross-fitting (select on one half of the test windows, score on the other); the pooled tail reference drops only from 0.198 to 0.185.
  • Downward bias (all members are MSE-trained): reweighted losses exceed the reference on at most 5.2% of tail pairs, and adding three non-MSE foundation models raises it by only +0.006 to +0.049.
  • Noise. Re-training the strongest member changes tail R2R^2 by 0.006–0.008 on Traffic and Electricity.

Now stated in the paper:

  1. Real-data ES is an estimate with known biases, not a bound ("lower bound" withdrawn).
  2. The injected-error component, max⁡(−Rk2(f),0)\max(-R^2_k(f),0), does not depend on the reference; only the unrealized component does, so we claim sacrifice only where attainable skill is verified externally (foundation models realize it on Traffic and Electricity) and lies far above seed noise.
  3. Structure that no current model realizes would be missed, so ES can understate the gap there (a stated limitation).
Reviewer_2SPeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether the "over-smoothing" of deep forecasters is a real bias or just correct shrinkage. Some smoothing is optimal: if part of the future is unpredictable, the best forecast should leave it out. The real question is whether a model leaves out structure it could have predicted. The authors take the SVD of the target matrix (future windows × horizon). For each mode, they compare the model's skill (per-mode R²) with the Bayes-optimal skill and call the gap excess suppression (ES). Classical results (reduced-rank regression, learning dynamics of deep linear networks) suggest that a model with limited rank or training time keeps high-energy modes and drops low-energy ones, even when these are predictable. The authors confirm this on a synthetic family with an analytic Bayes predictor.

On eight real benchmarks, the reference is the best per-mode score over several models. The authors report that rank-limited models (r=8) lose all attainable skill in the "tail", defined as the lower half of the 48 highest-energy modes. Across interventions (width, weight decay, dropout, inference-time head truncation), the effective rank of the prediction operator tracks this loss best (Spearman −0.89).

Critical strengths
  1. A useful, well-posed definition. ES measures each mode's skill against a Bayes reference, so correct smoothing of unpredictable variance (Eq. 1) is not counted as bias. On synthetic data with an analytic Bayes predictor, the pipeline recovers the reference to within 0.012, and trained networks match the reduced-rank closed form (Fig. 1). Splitting real-data ES into unrealized skill and injected error (Eq. 3) is also helpful.

  2. Clear and honest. The paper reports results that go against its own story: standard and foundation forecasters show no sacrifice, and reweighting barely moves PatchTST. It calibrates claims with a twin-seed noise floor and a random-walk negative control, and withdraws claims that fall below the floor (DLinear on ETT).

Critical weaknesses
  1. The "tail" result rests on an unjustified choice, and the headline "complete sacrifice" is close to automatic.

    • The tail is really the middle of the spectrum. With horizon H=96, the target matrix has 96 modes. The paper keeps only K=48 and calls modes 25–48 the "tail" (§3.3, App. B), so the lowest-energy half (modes 49–96) is never analysed. No reason is given for K=48 or for splitting at K/2, and neither is varied; even the horizon sweep keeps K=48 at H=720. The title and every headline number depend on this choice.
    • "Complete sacrifice" is close to automatic. At r=8 the model has only 8 output directions, and 24 higher-energy modes sit ahead of the tail, so by Proposition 1 the tail goes first. Eq. 3 then clips negative R² to zero, so the unrealized-to-attainable ratio is exactly 1 whenever a mode's R² ≤ 0. Since r=8 models have positive tail R² in only 13 of 1440 (channel, mode) pairs, the median ratio of 1.000 follows from the setup rather than being a finding.
    • Small modes inflate the scores. Per-mode R² divides by the mode's energy, so small absolute errors on low-energy modes give very large ES. At r=8, tail-ES reaches 13.84 on ETTm2, 10.2 on Weather and 225 on Exchange. On ETT, Weather and Solar, 49–105% of tail-ES is this injected error, not lost predictable structure.
    • The reference can be biased both ways. The proxy takes the best model per mode on the same test windows, which favours the luckiest model and inflates ES. This conflicts with the claim in §3.1 that reported ES is a lower bound. The paper's own twin floors are large (0.08–1.80), and at r=96 Traffic tail-ES (0.17–0.19) is below its floor (0.25). The proxy can also be too low, because all its members are MSE-trained; the foundation-model augmentation addresses this only partly.
  2. The problem appears only in models made small on purpose, so the title and the fix overreach.

    • Normal models do not show it. Under the same protocol, standard PatchTST and iTransformer never exceed the noise floor on any of the eight datasets, and three foundation forecasters stay within ±0.05 of zero (§6, Table 5). A PatchTST with d=64 already realizes nearly all attainable tail skill (§5). Sacrifice appears only with narrow width (r=8, d=8) or strong weight decay (10⁻²). This contradicts the title, which says deep forecasters in general sacrifice tail modes.
    • The fix is shown mainly where it is expected to work. A loss that upweights small modes should improve them. Overall MSE does improve for the shrunk MLPs, but these models are deliberately weakened, and the paper does not report whether they reach the accuracy of a standard model.
    • For PatchTST, reweighting moves tail R² by at most 0.024, and its 14–34% MSE gain on ETTh1/ETTh2 comes from reducing injected error, not from recovering predictable tail structure.
    • On three of four datasets, the best variant is plain inverse-energy weighting (γ=0, Table 3), so the predictability part, the method's main idea, mostly does not help on real data. It is therefore not shown that these modes matter for normal forecasters.
    • Few series, no selection rule. Phase A uses 7–8 channels per dataset (Traffic has 862, Electricity 321), Phase C uses 4 channels on Traffic and Electricity, and the intervention study rests on two "representative" channels. No rule for choosing these channels is given.
    • Related work on loss-based fixes for over-smoothing is missing, for example Subich et al. (ICML 2025), who modify the MSE loss to remove over-smoothing in data-driven weather models, and FreDF (Wang et al., ICLR 2025), which moves the forecasting loss to the frequency domain.

Overall. The question is good, ES is a useful way to separate correct smoothing from real bias, and the paper is clear and honest. However, three things weaken it: (a) the "tail" is an unjustified and untested choice that the title and headline numbers depend on; (b) "complete sacrifice" follows largely from the rank cap and the clipping in Eq. 3; (c) by the paper's own §6, standard and foundation forecasters do not show the problem.

My recommendation is weak reject. I would consider raising my score if the authors:

  • justify K and the head/tail split, and show the results hold under other splits, including modes 49–96;
  • show measurable sacrifice in a standard-size model;
  • compare PredReg with existing loss-based fixes.
Questions for authors

Clarifications (answers could change my score):

  1. Why keep K=48 of the 96 modes and split head/tail at K/2? Do the main results (rising tail-ES, complete sacrifice, PredReg gains) hold for other K and split points, including the lowest-energy modes 49–96?
  2. At r=8, how many tail modes have R² ≤ 0? How much of the median ratio of 1.000 comes from the clipping in Eq. 3 rather than from a measured loss of skill?
  3. The per-mode maximum is taken on the same test windows, and the twin floors are large. In what sense is reported ES a lower bound (§3.1)?
  4. Is there any standard-size supervised model in which tail sacrifice exceeds the noise floor? If not, should the title be limited to budget-constrained models?
  5. How were the 7–8 channels per dataset, and the two "representative" channels in §4.2, chosen?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2SPe (Part 1/3): Summary, Tail Definition & Full Basis (Q1), Beyond-Budget Sacrifice (Q2)

We thank the reviewer for a careful review with concrete conditions. We ran the requested analyses; some support the paper, others led us to concede and narrow it. In brief:

  • the tail band is now justified and tested, including modes 49–96 and the full basis;
  • "complete sacrifice" is withdrawn as a headline, because the rank-8 optimum shows it is budget-mandated, and replaced by a beyond-budget result at r=32r{=}32;
  • "lower bound" is withdrawn; the reference is now cross-fitted and calibrated against seed noise;
  • Q4: sacrifice is measurable where a budget binds (full-rank MLPs, FITS' own cut-off) but not in standard-capacity shared-weight Transformers, so the title is narrowed as suggested;
  • PredReg is compared with FreDF and an AMSE-style loss.

Sections and tables refer to the revised PDF.

[Q1] K = 48 and the head/tail split. We agree that this was neither justified nor varied. We re-ran the measurement study with the full K=96K{=}96 basis (same windows and seeds; it reproduces the submitted numbers exactly).

  • Where modes 25–48 sit. On every benchmark they lie beyond the 90% cumulative-variance point (pooled mean: mode 9; maximum: mode 23, on Traffic), and on all but Traffic beyond the 95% point (pooled: mode 16): a low-energy tail, though not the lowest modes.
  • Data-adaptive tails. Unrealized skill on the 26 skill-bearing channels:
bandR⋆2R^2_\starMLP-8MLP-32MLP-96RRR-32
modes 25–480.1850.1850.1730.1410.053
beyond 90% cum. var.0.1250.1230.1010.0800.039
beyond 95% cum. var.0.1040.1030.0940.0790.039
modes 49–960.0470.0470.0470.0460.044
  • Rising ES. Over all 96 modes, ESk\mathrm{ES}_k increases with the mode index for every trained network (Spearman 0.94–0.98, channel-mean curve) and decreases for RRR.
  • Modes 49–96 hold little variance (99% is reached by mode 43 on average, by 65 at most, on Traffic) and little attainable skill (0.047 on skill-bearing channels vs. 0.185 on 25–48; 0.011 over all channels); ES there is mostly injected error (MLP-8: 7.36). The sacrificed structure thus sits past the 90% point, ahead of an injection-dominated end. K=48 was not chosen for this reason; it is what the full basis now shows.
  • Other split points within K=48K{=}48 (modes 13–48 and 37–48): r=8>r=96r{=}8>r{=}96 holds, and r=8r{=}8 stays above the original floor on 7 of 8 datasets (ETTm1 at the floor, as before).
  • PredReg gains hold for every band inside the 48 reweighted modes. Tail R2R^2 at r=96r{=}96, MSE → inverse-energy, for modes 13–48 / 37–48: Traffic +0.165 → +0.350 / −0.109 → +0.133; Electricity +0.127 → +0.249 / −0.028 → +0.058; ETTh1 and ETTh2 also improve. They do not hold on modes 49–96. The loss leaves that residual subspace at unit weight, and there inverse-energy weighting is worse than MSE (Traffic −0.985 → −1.677; Electricity −0.150 → −0.386), while FreDF is better (−0.110 / −0.060). With little attainable skill there (0.011 over all channels), the effect is added injected error in the unweighted residual subspace, while overall MSE still falls on all four datasets (Part 3; §5, Discussion).

(§3.4; App. C.2 with Table 8; App. C.5.)

[Q2] Is "complete sacrifice" automatic? Largely, yes. At r=8r{=}8, 1427 of 1440 tail (channel, mode) pairs have R2≤0R^2\le 0 (deep linear: 1424). The median ratio of 1.000 thus follows from the clipping. More decisively, the closed-form RRR-8 baseline (the MSE-optimal rank-8 linear predictor) also leaves all attainable tail skill unrealized. The r=8r{=}8 endpoint confirms Proposition 1 but is not a separate finding; we withdrew it as the headline. Two unclipped quantities show the skill is missing, not miscalibrated: squared correlation 0.001 (reference 0.185) and validation-recalibrated R2R^2 0.000 (App. C.3).

The RRR baseline reveals a stronger result, now the headline (Table 2; unrealized tail skill, cross-fitted reference, skill-bearing channels):

dataset (channels)R⋆2R^2_\starRRR-8MLP-8RRR-32MLP-32MLP-96seed noise
Traffic (8)0.2810.2810.2800.1340.2490.1560.008
Electricity (8)0.3200.3200.3200.1120.3050.2780.007
Traffic (32)0.2210.2200.2200.0950.1910.1090.005
Electricity (32)0.2660.2660.2660.0670.2360.1820.006

At r=32r{=}32 the budget allows more than half of the tail skill to be realized. Yet MLP-32 leaves 2.0× (Traffic) and 3.5× (Electricity) as much unrealized as RRR-32 on 32 channels (1.9× and 2.7× on the original 8). This is sacrifice beyond the rank budget, consistent with the time budget of Proposition 2 and with the loss study, where reweighting at fixed capacity recovers much of it.

AuthorsOpenReview ↗

Response to Reviewer 2SPe (Part 2/3): Mode Energy, Reference Bias & Seed Noise (Q3), Standard vs. Budgeted Models & Title (Q4)

Small modes inflate the scores. We agree that raw tail ES on ETT, Weather and Solar is mostly injected error divided by small energies. Table 1 now reports the decomposed components under the cross-fitted reference; the original table moves to the appendix. Every sacrifice claim now rests on the unrealized component on skill-bearing channels.

We also concede the point on absolute size. Modes 25–48 hold 1–7% of the variance of the first 48 modes. Weighted by energy, MLP-8's excess error on them equals 2.6% of the target variance (19% over all modes). We do not argue that the tail matters for aggregate MSE. Our point is the converse: because it costs little MSE, budgeted MSE training drops it even where it is predictable, and aggregate MSE cannot see the loss. FITS (below) shows this directly. Low amplitude is also not the same as lost skill: shared-weight PatchTST and iTransformer output only 48–57% of the target's tail amplitude, yet leave nothing attainable unrealized.

[Q3] Reference bias and the "lower bound". We agree that the claim conflicted with the same-window maximum, and we have withdrawn it. §3.1 now states both bias directions and how each is controlled.

  • Upward (winner's curse): cross-fitting. The member is selected per mode on one chronological half of the test windows and scored on the other, and the halves are swapped. The pooled skill-bearing reference drops from 0.198 to 0.185. All new numbers are cross-fitted.
  • Downward: reweighted losses (including FreDF) exceed the reference on at most 5.2% of tail pairs, and adding three non-MSE foundation models raises the tail reference by only +0.006 to +0.049. Structure that no current model realizes cannot be ruled out, so the Discussion now calls ES an estimate with known biases.
  • Floors: the DLinear twin is replaced by the seed noise of the re-trained ensemble (3 seeds): 0.008 / 0.007 on Traffic / Electricity. The r=96r{=}96 Traffic case the reviewer flagged (0.17–0.19 vs. a floor of 0.25) is now claimable: it leaves 0.156 unrealized on the skill-bearing channels, about 20× seed noise. The old floor was an architecture gap (on ch300, DLinear's tail R2R^2 is 0.22 vs. 0.80 for OLS).

[Q4] Standard-size models; the title. It depends on whether a budget binds. We have narrowed the title as the reviewer suggests: "Excess Spectral Suppression: Why Budget-Constrained Forecasters Sacrifice Predictable Tail Modes." New evidence on all 8 benchmarks, under the new protocol:

  • Yes where a budget binds, even without a rank bottleneck. The full-rank (r=H=96r{=}H{=}96) MLP and deep linear network leave 0.129–0.209 of tail skill unrealized on Traffic and Electricity, 18–35× seed noise (Table 1). These are modest per-channel models (the MLP has two hidden layers of width 256), not standard-size forecasters; the excess is consistent with the training-time budget of Proposition 2, and reweighting at fixed capacity recovers much of it (Part 3).
  • A published model's own knob: the FITS cut-off. We used the official implementation with lookback 336. Lowering the cut-off monotonically raises unrealized tail skill (Spearman −0.94 on Traffic, −1.00 on Electricity). On Electricity, total MSE varies by less than 0.6% across the sweep while unrealized tail skill ranges from ≈ 0 to 0.232 of an attainable 0.240; at cut-off 100, about half is unrealized at a total MSE of 1.237e6, vs. 1.240e6 untruncated. FITS at its default cut-off and SparseTSF show no sacrifice.
  • No claimable sacrifice in standard-capacity shared-weight Transformers. We tested a shared-weight, channel-independent PatchTST with dmodel∈{16,32,64,128}d_{\text{model}}\in\{16,32,64,128\} (official hyperparameters), and iTransformer. On Traffic their tail ES is −0.021 to −0.060 (they beat the reference, with a mild monotone drift from −0.060 at d=128d{=}128 to −0.031 at d=16d{=}16); on Electricity their unrealized component is ≤ 0.001. The only residual above the much tighter seed noise in the tail band is the per-channel PatchTST of §6, which leaves 0.027 on Traffic (about 4× seed noise, 11% of the attainable 0.248) and 0.012 on Electricity; we report it as small rather than absent, and do not build a claim on it.
  • Positive reading. By spectral standards these models look over-smoothed (tail amplitude 0.48–0.57), yet they leave no attainable tail skill unrealized. ES identifies this as legitimate shrinkage, the distinction it was built to make.

The abstract, the introduction (new "Scope" paragraph), §6 and the limitations are revised accordingly (App. C.4, Tables 10–11).

AuthorsOpenReview ↗

Response to Reviewer 2SPe (Part 3/3): PredReg Scope & Channel Rules (Q5), Comparison with FreDF/AMSE & Stated Conditions

The fix is shown where expected; accuracy vs. a standard model; PatchTST. We agree that reweighting does not make a narrow model competitive with a standard one, and now say so. On ETTh1 / ETTh2, where the channel sets are identical, the overall MSE of the reweighted r=96r{=}96 MLP is 8.78 / 23.28, vs. 9.53 / 24.79 for iTransformer and 8.26 / 18.81 for the best-width shared-weight PatchTST (single seed). It beats iTransformer, not PatchTST. We position the loss as:

  1. a mechanistic test: at fixed capacity, changing only the objective closes 31–64% of the tail gap while overall MSE also improves, which shows that the sacrifice is an artifact of the objective under a budget;
  2. a tool for budget-constrained forecasters.

The PatchTST result (tail R2R^2 moves by ≤ 0.024) is the expected null: when the operator is not constrained, there is nothing to reallocate. As the reviewer notes, its 14–34% MSE gains come from reduced injection, and the paper says so. We did not apply PredReg to FITS: its cut-off removes frequencies from input and output, a hard architectural limit like r=8r{=}8 in the synthetic study, where no reweighting helps (§5, "soft vs. hard budgets"). PredReg targets soft budgets (r=32r{=}32, 9696), where the rank leaves room for the skill (RRR-32 realizes over half); for FITS, ES is the diagnostic.

γ = 0 is best on 3 of 4 datasets. Agreed. We now recommend γ = 0 as the default and present the predictability term as conditional. It helps when ρ^\hat\rho is estimable: on the synthetic family, PredReg removes the LH sacrifice (0.506 → 0.001; γ = 0 reaches 0.077). It does not help with an MSE-family probe that is blind on real tails. Contribution (4) and §5 are restated.

[Q5] Channel selection. The rules are now collected in App. B:

  • Measurement study: all channels for ETT; for the other datasets, 8 channels at approximately evenly spaced indices fixed before running.
  • Intervention study: the three channels with the largest attainable tail skill in each skill-bearing dataset (Traffic 300, 0, 100; Electricity 150, 100, 200). Sacrifice can only be measured where there is skill; the two "representative" channels are the top channel of each dataset.
  • Loss study: the four channels with the largest attainable tail skill on Traffic and Electricity, and all channels on ETTh1 and ETTh2.

We also extended Traffic and Electricity to 32 channels each, at indices round(linspace(0, C−1, 32)), and the conclusions are unchanged (Table 2; 31/32 and 27/32 channels are skill-bearing).

Loss-based fixes (Subich et al.; FreDF). Both are now discussed in related work and compared (MLP r=96r{=}96, 3 seeds). For FreDF, α is selected on validation. The AMSE-style loss follows Subich et al.'s modified form on data-mode bands; this is an adaptation, since it was designed for weather models. Entries are tail R2R^2 (modes 25–48) / overall MSE:

lossETTh1ETTh2TrafficElectricity
MSE−0.117 / 9.60−0.317 / 26.55+0.016 / 7.67e-4+0.030 / 2336
inverse-energy (ours, γ=0)−0.017 / 8.78−0.207 / 23.28+0.239 / 7.37e-4+0.182 / 2165
FreDF−0.026 / 8.64−0.108 / 21.89+0.145 / 7.47e-4+0.135 / 2052
AMSE-style−1.052 / 12.03−2.048 / 29.5−1.118 / 1.14e-3−0.782 / 3449
  • Skill-bearing datasets. Inverse-energy weighting recovers more tail skill than FreDF; the seed ranges do not overlap, and the ordering is the same at r=32r{=}32.
  • ETTh1 / ETTh2. FreDF is better or comparable on tail R2R^2, and it has the lowest MSE on ETTh1, ETTh2 and Electricity.
  • Amplitude. FreDF raises tail amplitude only modestly (Traffic / Electricity: 0.36 / 0.31, vs. 0.32 / 0.22 for MSE and 0.46 / 0.43 for ours). It is also better on modes 49–96 (see Q1).
  • AMSE-style loss. It restores amplitude (0.63–0.99) but not phase: tail R2R^2 turns strongly negative and MSE rises by 11–49%. Matching amplitude is not the same as recovering predictable structure, which is why ES is defined against a Bayes reference.
  • Exceeding the reference. No loss, including FreDF, exceeds the reference on more than 5.2% of tail pairs.

(§2, §5, App. C.5.)

On the stated conditions.

  1. K and the split. Now justified by the full-basis analysis, and tested on other splits and on modes 49–96. The measurement conclusions hold. PredReg's gains hold within the reweighted modes, not on 49–96, which carry little attainable skill.
  2. A standard-size model with measurable sacrifice. Not in standard-capacity shared-weight Transformers (only a small residual of 0.027 for the per-channel PatchTST on Traffic). Where a budget binds it is measurable: full-rank MLPs at 18–35× seed noise, and the published FITS once its cut-off narrows. The title is narrowed accordingly.
  3. Comparison with existing loss-based fixes. Done: mixed on aggregate MSE, better at recovering tail skill where it exists.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: New Experiments, Scope Refinement, and Paper Revision

TL;DR. Under tighter controls, forecasters sacrifice predictable tail skill wherever a budget binds (trained rank-32 networks leave 2.0–3.5× more unrealized than the rank-32 optimum), while standard-capacity shared-weight Transformers show no claimable sacrifice; we narrowed the title and claims to budget-constrained forecasters.

We thank all reviewers. The reviews converged on four questions:

  1. Is the measured "sacrifice" real, or an artifact of the reference, the noise floor, the tail definition and the clipping?
  2. Does it occur in standard models?
  3. How does the correction compare with existing loss-based fixes?
  4. Is the paper's structure clear?

We ran new experiments and revised the paper (changes marked in the revised PDF). Where results did not support our claims, we narrowed them.

New experiments (paper's protocol; numbers in the individual replies)

  1. Measurement study re-run with the full 96-mode basis, three re-trained copies of the strongest reference member, and closed-form reduced-rank regression (RRR) at rank 8 and 32. It reproduces the submitted numbers exactly.
  2. Cross-fitted reference and seed-noise calibration. These replace the same-window maximum and the DLinear-twin floor. The winner's-curse inflation is 0.013, and seed noise is 0.006–0.008 on Traffic and Electricity.
  3. Direction diagnostics (squared correlation; validation-recalibrated R2R^2). At rank 8, tail predictions are uncorrelated with the target, so what is lost is information, not calibration.
  4. Budget-optimum baseline. RRR-8 also loses all tail skill, so the rank-8 endpoint is budget-mandated. At rank 32, trained networks leave 2.0–3.5× more tail skill unrealized than RRR-32. Confirmed on 32 channels per dataset.
  5. Tail definition: full basis, cumulative-variance tails, modes 49–96, and alternative split points.
  6. Modern and lightweight forecasters: shared-weight PatchTST at four widths; iTransformer and per-channel PatchTST re-scored; a FITS cut-off sweep; SparseTSF.
  7. Loss comparison: FreDF and an AMSE-style loss after Subich et al., against our inverse-energy weighting.

What changed in the claims

  • Title and scope. Standard-capacity shared-weight PatchTST, iTransformer and foundation forecasters show no claimable sacrifice (the per-channel PatchTST shows a small residual), so the paper is now about budget-constrained forecasters. New title: "Excess Spectral Suppression: Why Budget-Constrained Forecasters Sacrifice Predictable Tail Modes."
  • Withdrawn: "complete sacrifice" as a standalone finding (it is the budget's), and "ES is a lower bound" (the reference can err both ways; both directions are now controlled).
  • Strengthened: sacrifice beyond the rank budget, and sacrifice in full-rank (r=96r{=}96) networks on Traffic and Electricity, now 18–35× seed noise where it was previously at floor level.
  • Conceded: the tail band holds a small share of absolute MSE; the predictability term of PredReg rarely beats plain inverse-energy weighting on real data; and the loss's gains do not extend to modes 49–96, which it does not reweight and which hold little attainable skill (0.011 over all channels).
  • Made explicit: what the diagnostic is for. On Traffic and Electricity, standard-capacity models look over-smoothed (tail amplitude 48–57% of the target) yet leave no attainable tail skill unrealized. A spectrum comparison would flag them, and restoring amplitude is costly (the AMSE-style loss raises our MLPs' MSE by 11–49%). ES separates this legitimate shrinkage from the information loss of bottlenecked networks and narrowly cut FITS, whose tail predictions are nearly uncorrelated with the target. In practice: low tail amplitude with ES ≈ 0 needs no fix; compressing a model (rank, width, cut-off) calls for checking unrealized tail skill, which aggregate MSE does not show.

Structure. Definitions leave the introduction; propositions precede their first use; "Phase" code names become named studies with a roadmap; transitions, a Conclusion, rewritten limitations and the channel-selection rules are added (details: replies to 1m2X, MBA4). The main text stays within 10 pages; new analyses are in Appendix C.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.