acceptodds
Under review as a conference paper at ICLR 2027

When Does Learned Normalization Actually Adapt? A Diagnostic Analysis of Adaptive Centring in Time Series Forecasting

Abstract

Adaptive normalization schemes for time series forecasting introduce learnable coefficients intended to let models discover data-dependent centring behavior, but it is rarely established whether these coefficients capture genuine structure or merely reparameterize training around their initialization. We study one such mechanism, Adaptive Cross-Scale Normalisation (ACSN), across a factorial benchmark of 3,200 runs spanning four forecasting architectures, five datasets, four forecast horizons, and ten random seeds. The learned coefficient departs from its initialization in an architecture- and dataset-specific manner: on Electricity and Weather it shifts meaningfully, while on Exchange Rate and National Illness it remains close to initialization. Where the coefficient fails to move, a fixed normalization baseline reliably outperforms the adaptive variant; where it moves, adaptive normalization is more often competitive. Because the coefficient is sigmoid-parameterised and restricted to , it can approach but never equal the Fixed baseline (). This constraint induces an inherent adaptive-versus-fixed performance gap whenever Fixed is near-optimal, complicating a direct interpretation of that comparison. Pooled performance comparisons show extreme heterogeneity ( up to 99.9%), so we report conditional, dataset- and horizon-specific effects rather than a single aggregate number. On the evidence gathered here – one controlled-centring scalar, five datasets, and ten seeds per configuration – we do not claim ACSN is a general-purpose normalization improvement; we present it, and the analysis pipeline built around it, as a diagnostic method for identifying when a normalization coefficient is and is not doing meaningful work.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Transformer Architectures, Normalization, Distribution Shift, Long-Term Forecasting, Benchmarking, Representation Learning, Adaptive Cross-Scale Normalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6jLPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Adaptive Cross-Scale Normalisation (ACSN), a RevIN-style normalization with learnable centring coefficients constrained to (0, 1) through a sigmoid. Using a full factorial benchmark of 3200 runs (four backbones, five datasets, four horizons, four normalization modes, ten seeds), the authors examine whether this coefficient adapts at all, and whether adaptation improves forecasting. They report that the coefficient moves only slightly from its initialization but does so consistently, with movement structured by architecture and dataset. Standard full centering (fixed) outperforms the adaptive variant on all primary metrics when pooled. The authors attribute part of this gap to the sigmoid bound and propose the learned coefficient as a diagnostic of when adaptation occurs, rather than as a better normalization method.

Critical strengths
  • The benchmark is complete and seed-matched across all factors, and the analysis uses paired comparisons, effect sizes, multiplicity correction, and heterogeneity estimates rather than single-run averages.
  • The authors report a negative result candidly, stating that 'Fixed' remains the strongest option and that they cannot recommend ACSN as a drop-in replacement for accuracy.
  • The limitations are disclosed openly. The authors acknowledge that the current evidence cannot distinguish a near-optimal initial coefficient from insufficient optimization.
Critical weaknesses
  • The contribution is limited. Although the paper's intended contribution is diagnostic, the current experiments do not establish what coefficient movement reveals beyond what the direct comparison against 'Fixed' already shows.
  • The key baseline is missing. Without a fixed coefficient at the initial value of 0.5, or a dense sweep of fixed values, the paper cannot show whether learning the coefficient changes forecasting accuracy at all.
  • The paper supports dataset-associated coefficient movement under its training protocol, but it does not establish that the coefficients reflect dataset-specific preferred or converged centering strengths. The coefficient shares the backbone's small learning rate and a limited number of epochs, and the datasets with the fewest optimizer updates show the least movement (Table 6).
  • Proposition 1 does not explain the observed gap. The learned coefficients stay between roughly 0.44 and 0.59, far from sigmoid saturation. The experiments therefore do not support attributing the gap to the excluded endpoint rather than to the coefficient remaining near its initial value.
Questions for authors
  • How does Adaptive compare with a matched model whose coefficient is frozen at 0.5?
  • Do coefficients move further with a larger coefficient learning rate or more updates, and do different initializations converge?
  • How are PatchTST and iTransformer embedded in the multi-scale refinement framework, and how many scale-specific coefficients does each learn? Figure 7 presents scale profiles only for Autoformer-MS and Informer-MS. How is the reported coefficient aggregated across scales and checkpoints?
  • What evidence attributes the observed gap to endpoint exclusion, given that Proposition 1 states the infimum equals the Fixed loss? Could a fixed-α sweep or an endpoint-capable parameterization distinguish this explanation from optimization effects?
Reviewer_as2COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores when normalization in time series prediction adapts through an adaptive cross scale normalization method with learnable coefficients and conducts a large number of experiments.

Critical strengths

1.This article focuses on an interesting issue related to deep learning, namely the normalization problem. 2.This article conducted a large number of experiments to explore the normalization issue that this article focuses on.

Critical weaknesses
  1. The paper focuses on an interesting normalization problem, but the innovation of the research method is relatively low.
  2. The introduction section of the paper mentions that the introduction of multi-scale methods to improve temporal prediction has led to a mismatch between the input and the representations generated by the model propagating between scales. However, not all existing multi-scale time series prediction methods propagate inputs between scales, which makes the research motivation of the paper not solid.
  3. This paper uses the method of adding learnable coefficients to explore the normalization problem, and the final coefficients are concentrated around the initial value of 0.5. This experimental result has low persuasiveness.
  4. The author is exploring normalization issues, but the experimental results do not perform as well as in the case of fixed coefficients of 1. The experimental conclusion cannot support the research question.
Questions for authors
  1. For the normalization problem studied in the paper, using only learnable coefficients has low interpretability and innovation. Can further innovation be carried out?
  2. The motivation for the paper mainly comes from the distribution difference between real observations and model generated predictions in coarse to fine sequential refinement. However, the original architectures of PatchTST and iTransformer do not naturally adopt this scale by scale prediction transfer mechanism. How can the author prove that their proposed cross scale mismatch is equally applicable to all experimental architectures, and whether the actual mechanism of ACSN in these architectures is consistent?
  3. Has the author explored the impact of selecting different initial initialization values on the resulting outcomes? Is there a separate optimizer or loss function set for this learnable coefficient? Why hasn't an initialization sensitivity analysis been conducted or a complete report on coefficient trajectories been submitted.
  4. In terms of writing, more methods should be presented and the analysis should be based on the research problem for a fundamental analysis.
Reviewer_YjsXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a single learnable centring coefficient (ACSN) inside RevIN-style normalisation for multi-scale time-series forecasting. At each scale the model subtracts α·μ instead of μ, with α = sigmoid(a) ∈ (0, 1) initialised at 0.5; variance scaling is unchanged. Four modes are compared: None (α=0), Adaptive (learned), Fixed (α=1, standard RevIN) and Forced (α=1.5). The evaluation is a 3,200-run factorial: 5 datasets × 4 backbones × 4 horizons × 4 modes × 10 seeds, paired by seed, with paired Cohen's d, FDR correction, power analysis and random-effects meta-analysis. The headline findings are that α barely moves (mean 0.5115, mean absolute shift 0.018), that its movement is reproducible across seeds and structured by architecture and dataset, and that Fixed beats Adaptive by ~17% MSE pooled with I² ≈ 99.8%. The two datasets where α does not move (Exchange Rate, Illness) are where Adaptive loses worst. A proposition notes that sigmoid(a) < 1 so Adaptive can never equal Fixed. The authors reposition ACSN as a diagnostic probe of "when adaptation happens" rather than a forecasting improvement.

Detailed review: The finding is that one scalar passed through a sigmoid and initialised at 0.5 ends up at 0.5115 on average after training with AdamW at lr 1e-4 for at most 10 epochs with patience 3. Figure 3 is basically a spike at 0.50. The authors read this as something about the datasets and architectures. I think it is just as likely that the coefficient was never given enough optimisation to move. There is no learning-rate ablation on a. There is no training-time trajectory of a. There is no alternative initialisation and no fixed-α sweep. Section 7 admits the data cannot tell "0.5 is near-optimal" apart from "the optimiser did not move it". That admission sits right under the central claim and the paper never resolves it. Proposition 1 does not help much. Sigmoid(a) < 1 is a one-line observation. Also it does not explain the 17% gap. With a = 5 you get α = 0.993 and that is the same as Fixed for any practical purpose. The gap is there because α sat at 0.5 and not because 1.0 is out of reach. The asymmetry is self-inflicted and easy to remove. Initialise near 1 or use α = 1 + a with no sigmoid. The paper spots this and lists it as limitation (1) and then builds the discussion around it. The paper calls itself a benchmark but it compares one method against three ablations of that same method. There is no DAIN or SAN or Dish-TS or any other adaptive normaliser. There are also no absolute MSE or MAE numbers anywhere so I cannot check that the four backbones are at their published accuracy. With L_input = 96 for PatchTST and iTransformer and a 10 epoch cap I suspect they are not. The statistics are heavier than the question needs. A random-effects meta-analysis across five datasets the authors picked themselves with I² at 99.9% only tells me the datasets differ. Figure 5 already shows that. A p-value of 7e-31 for a 0.0115 shift in α and a Cohen's d of 31 on Illness are signs of almost no seed variance and not of strong effects. On Illness that is one more hint the coefficient was frozen. The diagnostic signal claim rests on two datasets that did not move and lost and three that moved and gave mixed results. A coefficient stuck at half centring losing to full centring is not a mechanistic insight. It follows from where it was initialised.

Critical strengths

Strengths: S1. 3,200 completed runs, 10 seeds, paired by seed. This is rare in long-horizon forecasting and the paired-d / FDR / power chain is the right instinct. S2. The negative headline is not buried. Fixed wins, and the paper says so in the abstract. S3. Asking "does the coefficient move at all?" before "does it help?" is a good question, and separating the two is sensible. S4. The discriminability-ratio analysis (seed ≈ 0, architecture 0.67, dataset 0.48) is a cheap, reusable check that others could adopt.

Critical weaknesses

Major Weaknesses:

MW1. I cannot tell if the coefficient stayed near 0.5 because the data wanted it there or because it was never trained enough to move. One sigmoid scalar at lr 1e-4 for 10 epochs is unlikely to go anywhere and there is no lr or init ablation and no fixed-α sweep to rule that out. Section 7 admits as much. MW2. Proposition 1 is just sigmoid < 1 and does not explain the 17% gap since a = 5 already gives α = 0.993. The asymmetry is the authors' own choice. Start near 1 or drop the sigmoid and it goes away. They list it as a limitation instead of fixing it. MW3. This is one method against three ablations of itself with no outside normaliser like DAIN or SAN. There are also no absolute MSE or MAE numbers so I cannot check the backbones are properly trained. L_input = 96 and a 10 epoch cap make me doubt it.

Minor Weaknesses:

mW1. The paper never says how PatchTST and iTransformer were made multi-scale. Neither is a multi-scale model so the setup cannot be reproduced from the text. mW2. The Illness numbers (d = 31 and Forced beating Adaptive) and the 250 or so runs sitting at exactly 0.50 in Fig. 3 both look like a frozen coefficient. The fraction that moved less than 0.001 should be in the text. mW3. The diagnostic signal claim rests on five datasets. The two that did not move lost and the three that moved were mixed. A coefficient stuck at half centring losing to full centring follows from the init and is not much of a mechanism. Figure 6 needs a correlation with a CI across the 80 cells rather than a scatter.

Questions for authors

Questions: Q1: What learning rate did a get and was it the same 1e-4 as the backbone? What happens at 10x or 100x that or with a started at 2 or 4? And how many of the 800 runs moved α by less than 0.001? Q2: Could you run a fixed α sweep over 0.25, 0.5, 0.75, 0.9, 0.99 and 1.0 for one backbone on Exchange Rate and Illness? If fixed 0.5 matches Adaptive then Adaptive is not adapting. Q3: Why the sigmoid at all? What does α = 1 + a with no bound give on the same five datasets? Q4: Please report absolute MSE and MAE for the Fixed mode per dataset and backbone and horizon and compare them to the published PatchTST and iTransformer numbers. Q5: How exactly were PatchTST and iTransformer made multi-scale? Q6: Across the 80 cells in Figure 6 what is the correlation between |α − 0.5| and the Adaptive vs Fixed MSE difference and what is its CI? Does it hold if Exchange Rate is dropped?

Suggestions: Q1 to Q3 are the ones that matter to me and they all point at MW1. The paper rests on the coefficient not moving and right now I cannot tell whether that is a property of the data or a property of the training. A learning rate sweep on a and a run or two from a different init would show whether the coefficient can move at all. A fixed α sweep on Exchange Rate and Illness would show whether the value it lands on matters.

If the coefficient moves once it is given a chance and Adaptive then matches or beats Fixed the paper has a positive result and a stronger story than the one it currently tells. If fixed α = 0.5 reproduces the Adaptive numbers then the adaptation claim has to go and the paper should be reframed around the fact that a learned centring coefficient does not learn under standard training. Q4 is about trust in the backbones. All the results are relative so I have no way to tell whether a 17% gap is 17% of a well trained model or 17% of an undertrained one. The per seed tables are released so the absolute numbers exist and a table in the appendix next to the published PatchTST and iTransformer numbers would settle it. Q5 is a reproducibility gap that a paragraph can fix. Q6 would turn the diagnostic claim from a reading of Figure 6 into a number. A DAIN baseline would address MW. The gap in the paper is one specific experiment and more of the same kind of evidence does not close it.

1 comment

Thank you for the detailed review. We ran the controls behind Q1 and Q2 and answer Q4 and Q6 with numbers; the dense sweep and alpha = 1 + a were not run. Results: Sec. 5 and the appendix.

MW1 / Q1 (learning rate of a; 10x / 100x; start at 2 or 4; runs moving < 0.001). The coefficient had no learning rate of its own: it sits in the backbone's AdamW group (lr 1e-4, halved each epoch). Study 2 gives its raw parameter 10x and 100x that rate, and starts it at raw about -2 and +2 (alpha = 0.12 and 0.88; raw 4 was not run). On ER the mean final alpha is 0.543 (x10) and 0.678 (x100); on IL it stays at 0.500 and 0.497. At the shared rate it ends within 0.0072 of its start in every run. Runs within 0.001 of 0.5 in Study 1: 192/800 (24.0%): Electricity 1.9%, Traffic 5.0%, Weather 5.6%, ER 26.3%, IL 81.3% (Table 2). More epochs would add little: with the rate halved each epoch, 20 epochs instead of 10 add 0.1% to the summed learning rate (Sec. 3).

Q2 (fixed-alpha sweep over 0.25, 0.5, 0.75, 0.9, 0.99, 1.0). Of these points we have 0.5 (frozen, Study 2) and 1.0 (the Fixed mode of Study 1); 0.25, 0.75, 0.9 and 0.99 were not run. The comparison your question turns on, fixed 0.5 against Adaptive, is now in the paper: pairing the 240 Study 1 Adaptive runs on ER and IL (three backbones) with the Study 2 frozen-0.5 runs, Adaptive's MSE advantage is +0.44% (90% CI -0.45 to +1.34), equivalent to zero within +-2% (TOST p = 0.0023; not within +-1%); for MAE within +-1%. So at the shared learning rate fixed 0.5 matches Adaptive to within 2% in MSE, your second branch, and the paper is organised around it (separate job arrays; Comment 1b). On ER the start matters: 0.12 gives MSE 61.9% higher than frozen 0.5 and 0.88 gives 46.2% lower. Your first branch appears in part with a larger coefficient learning rate: the coefficient moves to the better side and lowers ER MSE, relative to frozen 0.5, by 10.5% (x10) to 26.7% (x100), but only part of the better-start gain. We did not re-run Adaptive against Fixed with a boosted coefficient, so we make no claim that Adaptive matches or beats Fixed.

MW2 / Q3 (alpha = 1 + a). Not run (Limitation 3). The bound does not explain the gap in our data: alpha = 0.88 is representable and lowers ER MSE by 46.2% (40.8 to 51.1). Proposition 1 is now a remark (Sec. 3).

MW3 / Q4 (benchmark wording; absolute MSE and MAE; published numbers). The paper now says "internal ablations"; DAIN, Dish-TS and SAN are a stated limitation, not compared (Limitation 4). On 6 Nov we said we were preparing absolute Fixed-mode MSE and MAE for all cells beside published results; the paper has this only for Autoformer-MS. Table 4 (MSE) and Table 15 (MSE and MAE, appendix) compare its Fixed mode with the published Autoformer (input length 96) in 16 dataset-horizon cells: MSE is lower than published in 14 of 16 (all 12 Electricity, Traffic and Weather cells, by 3.6% to 31.1%) and MAE in 12 of 16; on ER it is lower at 96 and 720 but higher at 192 (+40.6%) and 336 (+19.6%). This is comparable, not exact: optimiser, schedule, batch size and repeats differ (Sec. 5). We did not compare PatchTST, iTransformer or Informer-MS with published numbers (published PatchTST uses look-backs of 336 or 512; ours is 96; Limitation 6). The Fixed-mode MSE and MAE of all four backbones (80 cells, ten seeds each) are in the anonymous repository (folder rebuttal_evidence/), not in the paper.

mW1 / Q5 (PatchTST and iTransformer). They were not made multi-scale: each has one scalar alpha applied once, so the cross-scale motivation does not apply to them (Table 1, Sec. 3). PatchTST applies it inside RevIN and iTransformer to the input window; both also divide by sigma, whereas Autoformer-MS and Informer-MS (one coefficient per scale: five, four on IL) only centre and restore (a correction to Eq. 1; Comment 1b).

mW2. The share within 0.001 is in the text and Table 2; the paper now says the very large d_z on IL reflects near-zero seed variance, not large adaptation.

mW3 / Q6 (correlation over the 80 cells, CI, with and without ER). Over the 80 Study 1 cells (Informer-MS included) the Pearson correlation between a cell's mean |alpha - 0.5| and its Adaptive-versus-Fixed MSE benefit is r = 0.33 (95% CI 0.15 to 0.51, cluster bootstrap over the 20 dataset-backbone clusters); Spearman 0.37. The benefit is positive when Adaptive has the lower MSE, so against the vertical axis of Figure 6 (positive favours Fixed) the sign reverses. Dropping ER strengthens it: r = 0.41 (cluster CI 0.17 to 0.63; Spearman 0.46); leaving out each dataset in turn gives r between 0.26 and 0.41. The association is moderate and observational; it does not show that movement causes a smaller gap. Without Informer-MS it was not computed (Limitation 8).

Comments on the paper

1 comment

We have uploaded a revised PDF: the main text is within 10 pages, followed by the appendix. Red marks new or substantively changed content that responds to your reviews (new results, corrected claims, new disclosures and definitions, new headings, and the captions of new or changed tables and figures). Comment 1b says what is not coloured, what we are correcting and what was removed; Comments 2-4 answer each reviewer point by point.

New runs (Study 2). 1,200 runs on Exchange Rate (ER) and National Illness (IL), three backbones (Autoformer-MS, PatchTST, iTransformer), four horizons, ten seeds. Five conditions of 240 runs each: the coefficient frozen at 0.5 (the reference); learned from a start of 0.12 or 0.88 at the shared learning rate; learned from 0.5 with its own learning rate x10 or x100. Comparisons are paired by seed with the frozen reference and pooled over the 12 cells of a dataset (DerSimonian-Laird; Holm-adjusted p-values). Gain = 100(1 - exp(mean log MSE ratio)); positive means lower MSE than frozen 0.5.

New analyses of existing results. (i) The 240 Study 1 Adaptive runs on ER and IL with the same three backbones, matched by dataset, backbone, horizon and seed index with the Study 2 frozen runs. (ii) Over the 80 Study 1 cells (5 datasets x 4 backbones x 4 horizons), the association between a cell's mean |alpha - 0.5| and its Adaptive-versus-Fixed MSE benefit. (iii) Fixed-mode (alpha = 1) Autoformer-MS against the published Autoformer.

What we found.

  1. At the shared learning rate the coefficient ends within 0.0072 of its start in every Study 2 run (Study 1: 24.0% of runs end within 0.001 of 0.5).
  2. Ordinary Adaptive is equivalent to the frozen coefficient: mean MSE improvement +0.09% (bootstrap 95% CI -0.96 to +1.12), Adaptive better in 75.0% of the 240 pairs. The paired t-test (p = 0.33) and the Wilcoxon test (p = 2.2e-4) use the 240 pairs of raw MSE values (many very small wins, a few larger losses); the equivalence tests (TOST) use the paired log MSE ratios: advantage +0.44% (90% CI -0.45 to +1.34), equivalent to zero within +-2% (p = 0.0023) but not within +-1%; for MAE, within +-1% (p = 0.0021). The two sets of runs come from separate job arrays, which the paper states.
  3. On ER the start alone matters: 0.88 lowers MSE by 46.2% (95% CI 40.8 to 51.1) relative to the frozen control and 0.12 raises it by 61.9% (45.2 to 80.5).
  4. With its own larger learning rate the coefficient moves to the side the displaced starts identify as better in 24 of 24 cells at x100 and 22 of 24 at x10 (sign tests, p = 1.2e-7 and 3.6e-5). On ER the gain is +10.5% at x10 and +26.7% at x100 (mean final value 0.543 and 0.678), only part of the better-start gain (21-82% at x100, by backbone). On IL every effect is practically small.
  5. Over the 80 Study 1 cells, those where the coefficient moves more have a smaller Adaptive-versus-Fixed gap: Pearson r = 0.33 (cluster-bootstrap 95% CI 0.15 to 0.51); r = 0.41 without ER (0.17 to 0.63); positive whichever dataset is left out (0.26 to 0.41). It is observational.
  6. Absolute accuracy: the Fixed mode of Autoformer-MS has lower MSE than the published Autoformer in 14 of 16 dataset-horizon cells (all 12 Electricity, Traffic and Weather cells, by 3.6% to 31.1%) and lower MAE in 12 of 16; on ER it is lower at 96 and 720 but higher at 192 (+40.6%) and 336 (+19.6%). The comparison is comparable, not exact (optimiser, schedule, batch size and repeats differ), and only for Autoformer-MS.

Our reading: on these datasets the learned coefficient mostly records its initialisation and optimisation budget; with a larger learning rate it moves in the direction accuracy rewards, but not all the way. We do not claim that ACSN improves forecasting: Fixed remains the strongest centring in Study 1 (Adaptive's pooled MSE is 16.9% above Fixed's) and Study 2 has no alpha = 1 runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.