acceptodds
Under review as a conference paper at ICLR 2027

Scale-Invariant Training for Time Series Foundation Models

Abstract

Time series foundation models (TSFMs) are trained on large collections of time series datasets that span various morphologies and domains. This setting exposes models to series whose scales – typical magnitudes of their values – can differ substantially. Affine scaling methods such as Reversible Instance Normalization (ReVIN) scale model inputs and reverse the transform before computing the loss. We show that this inversion multiplies each series' gradient by relative to loss on scaled targets, where is the scaling denominator (e.g., standard deviation) and is the loss degree. We call this *scale-contaminated training* (**ScaleCon**), because the scale of each series consequently becomes an importance weight, causing high-scale series to dominate training. For any scale-equivariant scaler and residual loss that is homogeneous of degree , including MSE, MAE, and Quantile Loss, we prove that computing loss on scaled targets makes every mini-batch gradient and, consequently, the full optimization trajectory invariant to arbitrary independent rescaling of the training series, yielding *scale-invariant training* (**ScaleIn**). Notably, existing TSFMs use both objectives, with neither consistent reporting nor a common convention on how to compute training loss. We isolate the convergence disparity induced by **ScaleCon** and its correction under **ScaleIn** in controlled studies on synthetic and real data. In pretraining across four TSFM architectures, **ScaleIn** lowers MASE in all 24 architecture-benchmark comparisons, with average reductions across TSFMs of 18.8% on GIFT-Eval and 21.9% on the M-competitions. The gains extend to supervised neural forecasting, where it lowers MASE in 16 of 20 matched settings. Most existing time series forecasting pipelines can adopt **ScaleIn** with a one-line code change.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series forecasting, Time series foundation models, Normalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mSgoOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work compares SCALEIN, which computes the loss against scaled targets, with SCALECON, which first restores predictions to their original scale and then computes the loss. It shows that SCALECON implicitly prioritizes the optimization of high-scale series by introducing scale-dependent weights into the gradients. The work shows, both theoretically and empirically, that SCALEIN removes this scale-driven training bias under the stated scaling assumptions and generally improves forecasting performance as measured by MASE and WQL, which account for differences in scale.

Critical strengths
  • The paper is clearly written, and its figures, together with the explanations, make the scale-driven training bias and the effect of SCALEIN easy to understand and follow.
  • The controlled synthetic and real-data scale-swap experiments are well designed and provide convincing empirical support for the claims.
  • Comparisons across architectures and benchmarks, covering both time series foundation model pretraining and supervised forecasting baselines, generally show improvements in MASE and WQL, which account for differences in scale. These results show that the proposed loss-space choice is not only theoretically motivated but also empirically effective on real-world forecasting benchmarks.
Critical weaknesses
  • The experiments on time series foundation models use small model configurations and a fixed training budget, leaving the benefits with larger models and longer pretraining unverified. Thereby, the comparisons do not fully distinguish faster convergence from better forecasting performance after sufficient optimization. It remains unclear whether SCALECON could narrow the gap with longer training.
  • The normalized-target training strategy and its scale-balancing motivation have already been explored in prior work, as acknowledged in the paper. The main contribution therefore lies in a systematic analysis of the mechanism and broader empirical validation. While these contributions are important, the novelty of the training method itself is limited.
  • The empirical validation in multivariate settings is limited. Although the theory allows cross-channel dependencies under channel-wise scaling, the practical benefits in joint multivariate forecasting are not separately demonstrated.
Questions for authors

Overall, the clear explanations and systematic analysis make me lean toward acceptance. The weaknesses above are my main concerns, and addressing them would have the greatest impact on my assessment. I also have the following additional questions for clarification:

  • The evaluation primarily relies on scale-normalized metrics. Could the authors also report original-scale MAE and RMSE to clarify the benefits and limitations of SCALEIN under these metrics?
  • I agree that SCALEIN addresses an important issue, which led me to look at how tabular foundation models are trained for regression. Based on the open-source training code I have seen, most models already compute regression losses on normalized targets. Do the authors expect scale-invariant training to be similarly important for tabular regression, and could the paper’s analysis be extended to that setting?
4 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for their feedback and are glad that they found our controlled synthetic and real-data scale-swap experiments are well designed and provide convincing empirical support for the claims. We have addressed the comments below and paper edits are reflected in red in the updated paper.

W1

Our models range from 1.5 to 49.6M parameters and are pretrained with an 80k optimization-step budget. We did not test larger model configurations, but the bpb^p weighting (Lemma 3.1) still applies to them. Beyond the empirical results, our analysis identifies an implicit weighting scheme that should be explicitly considered in TSFM design.

To address the reviewer's concern about faster convergence, we continued pretraining for another 420k steps, raising the total budget ~sixfold to 500k steps. Reported below is the relative percentage reduction in MASE for ScaleIN over ScaleCon, averaged over the six benchmarks (lower is better). ScaleIN continues to outperform ScaleCON on every benchmark and at every budget. The gap narrows for Chronos-2 (from 16.2% to 5.5%) and stays stable for Moirai 2.0 (from 7.1% to 6.0%), and in both cases it plateaus after about 250k steps. TimesFM-2.5 and PatchTST runs are still in progress (est. ~1.5 day). We will update Section 4.3 of the paper with the full 500k results and notify the reviewer afterwards.

steps80k125k250k375k500k
Chronos-2-16.2-11.7-5.1-4.5-5.5
Moirai 2.0-7.1-6.7-6.9-5.6-6.0
TimesFM-2.5-14.0-12.5------
PatchTST-35.8-32.6------

W2

We fully agree that computing the loss on normalized targets is not a new method. We aimed to be transparent about this by providing Tables 3 and 4 in Appendix A that survey the loss space choice (ScaleIn vs ScaleCon) across 24 TSFMs and popular forecasting codebases, which we also summarize in the introduction. To make this more explicit, we have moved Tables 3 and 4 (previously in Appendix A) to the main text. We would like to clarify that the aim of this work is to provide theoretical and empirical evidence and bring awareness to the fact that the choice of loss space has critical consequences for training and generalization in TSFMs and should be reported as a design decision to enable reproducibility. As represented by Table 3, the choice of loss space is only explicitly documented for 9 of 24 TSFMs, cannot be determined from the paper or code for 3 TSFMs, and no surveyed forecasting codebase makes it configurable.

Beyond prior empirical observations (Berthelier et al., 2026, Ahmed et al., 2026), we contribute the following conclusions, which do not follow from prior empirical comparisons. We have revised the related work to state these distinctions explicitly:

  1. An exact formal characterization of the weighting as bpb^p for any residual loss homogeneous of degree pp, including MSE, MAE, and pinball loss;
  2. A proof that ScaleIN leaves the full optimization trajectory invariant to arbitrary series-wise rescaling;
  3. Evidence that common remedies do not generally eliminate scale-dependent weighting, including: global gradient-norm matching (Fig. 3d), Adam (Appendix E.2), and global gradient clipping, which is a uniform rescale and is used in all our TSFM pretraining runs where the gap persists;
  4. A scale-swap intervention that isolates the effect causally on real data;
  5. Matched pretraining comparisons across four TSFM architectures with different normalization schemes.
AuthorsOpenReview ↗

Response (2/2)

W3

We run an offline analysis of 8 multivariate datasets that we used in Section 4.2 by computing per-channel normalization statistics and comparing their scale across channels. For every training window, we compute the per-channel standard normalization scale bb (std. dev.) and measure each channel's share of the total ScaleCON loss weight b2b^2 (MSE) across all windows. Under ScaleIn each channel gets the same weight, but under ScaleCON per-channel loss weights can widely differ. For example, the top 10% of channels account for the following percentages of the scaling weight (offline analysis): Electricity (321 ch.) 100%, FRED-MD (107 ch.) 100%, Wind Farms (308 ch.) 62%, Taxi (1214 ch.) 44%, KDD Cup 2018 (269 ch.) 40%, Pedestrian (26 ch.) 37%, Solar (137 ch.) 35%, Traffic (862 ch.) 32%. In other words, in a jointly forecast multivariate series, ScaleCON concentrates a large part of its loss weight on a small subset of channels.

For a brief empirical evaluation, we trained PatchTST (channel-independent) and iTransformer (channel-mixing) on Electricity (321 channels) with both losses (context 64, horizon 16, 3 seeds). We report the relative median MASE change of ScaleIN with respect to ScaleCON (Δ\Delta MASE (%), lower is better), and the share of the gradient signal that goes to the 10% of channels with the largest gradients. For ScaleCON, these channels receive almost the entire gradient signal (99.6%), starving the remaining 90%. ScaleIN weights all channels equally a priori. However, in practice, harder-to-forecast channels receive more gradient (in this example 28%), so the gradient is allocated by forecast error rather than by scale.

ModelΔ\Delta MASE (%)ScaleIN grad shareScaleCON grad share
PatchTST-12.628%99.6%
iTransformer-15.528%99.6%

Q1

We thank the reviewer for this suggestion. We computed the original-scale MAE and RMSE for all runs in Table 1, aggregated per benchmark in the same way as MASE and WQL. ScaleIN achieves lower MAE in 23/24 architecture-benchmark comparisons and lower RMSE in 18/24. We added the full table to the appendix.

We note that metrics in raw scale should be interpreted with care. Averaging errors across series of heterogeneous scales weights each series by its magnitude, so a few high-scale series may dominate the result (e.g., series concerning electricity demand forecasting in Wh units will have much higher magnitude than temperature forecasting series). Even though these metrics apply scale-dependent weighting, ScaleIN still wins most comparisons under them, which suggests that its benefits are not an artifact of scale-free evaluation.

Q2

Our focus is TSFMs, but Lemma 3.1 holds for any affine target normalization with a homogeneous loss, including per-dataset target standardization in tabular regression. The bias arises only when the training targets span heterogeneous scales. Forecasting PFNs (e.g., ForecastPFN, TabPFN-TS, TempoPFN, ApolloPFN, Nori) train solely on synthetic data, where the scale range is a design choice of the prior and can be bounded during generation, perhaps making the loss-space choice less consequential. In contrast, all TSFMs in Table 3 train on real-world data, alone or mixed with synthetic data, so scale heterogeneity is inherited from the corpus and cannot be controlled at generation time. We have expanded Appendix A with this discussion on forecasting PFNs.

We thank the reviewer for suggestions, which have helped us strengthen the empirical evaluation. We will share the remaining results concerning longer PatchTST and TimesFM as they become available in about two days. We would be happy to discuss any further questions.

AuthorsOpenReview ↗

Full 500k-step TSFM results

Dear Reviewer mSgo,

We're following up with full 500k steps results for the TSFM experiment. We revised the paper to report the completed 500k-step runs as the main TSFM results, replacing the 80k runs (changes in blue).

ScaleIN lowers MASE in 23/24 comparisons (was 24/24) and WQL in 21/24 (was the same). Average reductions are 10.4% on GIFT-Eval and 12.7% on the M-competitions (was 18.8% and 21.9% at 80k).

Below is the relative MASE percentage change over the six benchmarks (ScaleIN relative to ScaleCon, lower is better):

80k125k250k375k500k
Chronos-2−16.2−11.7−5.1−4.5−5.5
Moirai 2.0−7.1−6.7−6.9−5.6−6.0
TimesFM-2.5−14.0−12.5−11.5−9.3−7.4
PatchTST−35.8−32.6−28.1−23.8−23.0

As evidenced, ScaleIn converges faster (Figure 5b) and retains a consistent advantage at a six times longer budget and ScaleCon does not catch up.

Thanks for running the longer experiments. The training budget was my biggest concern, and the new results address it. It would be great to note in the main paper that the gap narrows with more training. Even with a smaller gap, I think reaching good forecasting accuracy sooner is valuable in its own right, given the cost of pretraining foundation models.

I’m raising my score. I like papers that take a familiar design choice and show why and how it matters. This one made me think more carefully about how the choice of loss space affects which series drive training.

For the area chair, my confidence rating of 2 reflects my limited direct experience with time-series foundation models. I’ve worked on time-series models and on foundation models in other domains.

Reviewer_35PUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines where training loss is computed in forecasting pipelines that normalize their inputs. For an affine transform with denominator b and a residual loss homogeneous of degree p, computing the loss after inverse scaling weights each example’s parameter gradient by b^p. Computing the loss on scaled targets removes this factor and, under scale-equivariant normalization and matched optimization conditions, makes the training trajectory invariant to independent positive rescaling of the series. Synthetic experiments and a real-data scale-swap intervention examine this mechanism. In matched pretraining runs across four architectures and six benchmarks, SCALEIN improves mean MASE in all 24 comparisons and WQL in 21. Supervised forecasting experiments show smaller and less consistent gains. The contribution is a formal and empirical account of how loss placement affects training.

Critical strengths

S1. The mechanism is supported by targeted controls. Lemma 3.1 and Theorem 3.2 appear correct under their respective assumptions, including scaling statistics that are fixed with respect to model parameters and scale equivariance for trajectory invariance. The equal-scale and gradient-norm controls in Section 4.1 help separate relative sample weighting from overall update magnitude. The scale-swap experiment in Section 4.2 is especially informative because it changes the scale of the same forecasting data. These results give convincing evidence that input normalization alone does not ensure scale-invariant training. S2. The paired pretraining results show a useful practical effect across architectures. Within each architecture, the runs share initialization, sampled batches, and optimization settings. Table 1 reports lower mean MASE in all 24 architecture-benchmark comparisons over four seeds, despite differences in scaling statistics and training objectives across models. This supports treating loss space as a training choice that should be specified explicitly. The smaller supervised gains and the WQL regressions also provide a useful picture of where the benefit is less consistent.

Critical weaknesses

W1. The paper’s core contribution is relatively weak. The main approach is to normalize prediction targets using statistics from the historical context window and compute the training loss in normalized space. This procedure has already been used in prior work, such as Ahmed et al. (2026), which analyzed the gradient bias of original-scale losses toward high-amplitude series and used normalized-space losses combined with clipping for training. The novelty of this paper appears insufficient. W2. The reliability of the paper’s core claim is questionable. The paper’s analysis shows that samples with larger scales receive higher weights during training. The authors consider this weighting unreasonable and attempt to eliminate it, but do not provide a sufficient explanation. Time series data are inherently diverse, and the rationale for imposing equal training weights on data from different domains and at different scales remains debatable.

Questions for authors

Q1. What is the core novelty of this paper relative to existing research? Which conclusions cannot be derived from prior work? Q2. Is treating series at all scales equally a reasonable choice? Is ScaleIn a better choice in every scenario? Are there any criteria for making this choice? Q3. In real-world settings, some series exhibit different scales across different time windows, i.e., nonstationarity within a single series. Does Theorem 3.2 apply to such cases? The theorem considers multiplying a series by a fixed scaling factor, which does not seem to cover nonstationarity within a single series. Q4. Why are some models, such as Moirai-2.0, classified inconsistently in Table 3 of Appendix A? Is it actually ScaleIn or ScaleCon? What is the basis for this classification? This is unclear to me.

2 comments

We sincerely thank the reviewer for reading our paper and providing thoughtful feedback. Below, we provide answers to all four raised questions. We have revised the paper and all new edits are reflected in red.

Q1/W1

We fully agree that computing the loss on normalized targets is not a new method. Our contribution is theoretical and empirical evidence that the choice of loss space has critical consequences for training and generalization and should be reported as a design decision to enable reproducibility. We aimed to be transparent about this by providing Tables 3 and 4 in Appendix A that survey the loss space choice (ScaleIn vs ScaleCon) across 24 TSFMs and forecasting codebases, which we also summarize in the introduction. To make this more explicit, we moved these tables to the main text. As shown in the survey, the choice of loss space is only explicitly documented for 9 of 24 TSFMs, cannot be determined from the paper or code for 3 TSFMs, and no surveyed codebase makes it configurable.

Ahmed et al. (2026) compare input normalization methods across TSFMs and observe that their effectiveness depends on the scale sensitivity of the training objective. Our work is complementary and contributes the following conclusions, which do not follow from prior empirical comparisons. We have revised the related work to state these distinctions explicitly:

  1. an exact formal characterization of the weighting as bpb^p for any residual loss homogeneous of degree pp, including MSE, MAE, and pinball loss;
  2. a proof that ScaleIn leaves the full optimization trajectory invariant to arbitrary series-wise rescaling;
  3. evidence that common remedies do not generally eliminate scale-dependent weighting, including global gradient-norm matching (Fig. 3d), Adam (Appendix E.2), and global gradient clipping used in all our TSFM pretraining runs where the gap persists;
  4. a scale-swap intervention that isolates the effect causally on real data;
  5. matched pretraining comparisons across four TSFM architectures with different normalization schemes.

Q2

We thank the reviewer for raising this point and agree that the appropriate weighting of diverse series is debatable -- our aim is not to settle it. We do not argue that scale-based weighting is an inherently incorrect choice (if errors in original units represent the actual cost, scale-sensitive weighting may be preferable). Rather, we show that the loss space implicitly determines this weighting, that this choice has measurable consequences for training dynamics and generalization, and that it is rarely reported or supported as a configurable option in TSFM papers and forecasting codebases (current Tables 1 and 2). Under ScaleCon, each example is weighted by bpb^p, where bb is a statistic of its context window. This weighting depends on measurement units (the same series in cents rather than dollars receives 100$\times$ the weight under MAE) and on local volatility rather than on any notion of series importance. ScaleIn removes this implicit weighting and practitioners who want to prioritize certain series could add explicit weights on top of ScaleIn, making the choice intentional and independent of units.

Q3

Theorem 3.2 applies to nonstationary series, although the invariance guarantees concern fixed series-wise rescaling. However, Lemma 3.1 directly addresses this issue as it applies to each training example independently, with b computed over its context window. We think that the most intuitive explanation is that two windows from the same series can have different scale-dependent weights. For example, consider one window with standard deviation b = 0.1 and another relatively more volatile window from the same series with b = 2.0. For the same scaled error (z^−z=0.1)(\hat z - z = 0.1), ScaleCon MSE gives the volatile window a 400× larger scale-dependent weight (b2b^2 ratio of 4.0 / 0.01). ScaleIn removes this difference in weighting. We have expanded our methods section to address this result more clearly with regard to nonstationary series across forecast creation dates (Fig. 1e).

Q4

The two columns classify the paper description and released implementation independently, thus this inconsistent classification is not an error. We have manually surveyed 24 TSFM papers and their open-source code releases and assigned ScaleIn/ScaleCon labels. Moirai 2.0 is a paper where this inconsistent classification appears. Specifically, the description in the paper suggests ScaleCon (precisely, Fig. 1 has an explicit de-scaling module before loss computation), but the code suggests ScaleIn (the model class in training mode returns (preds, scaled_target)). We believe that this example shows the importance of our work as it can lead to clearer definitions of loss space in future papers. We added a note in the Table description to clarify this.

We hope these clarifications address the reviewer’s concerns and would be happy to discuss any remaining questions.

I thank the authors for their detailed response to my review. My concerns have largely been addressed. Although I still have reservations about the paper’s novelty, particularly its limited incremental contribution over existing work, I believe the topic is meaningful, and the authors have made several improvements to the paper. I have therefore decided to raise my score.

I also suggest improving the layout of the appendix. For example, page 21 contains only a pseudocode block, and the final page contains only a table. There is also substantial blank space in several places, so the space could be used more efficiently. This is just a minor suggestion.

Reviewer_swdFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the effect of loss computation space on training time series foundation models with heterogeneous series scales. It compares computing losses after reversing input normalization (SCALECON) with computing losses directly on normalized targets (SCALEIN). For homogeneous residual losses of degree pp, the authors show that inverse affine scaling introduces an example-dependent factor bpb^p into the loss and parameter gradient. They further establish that, under scale-equivariant normalization, normalized-space training preserves gradients and optimization trajectories under independent positive rescaling of training series. Validation includes synthetic experiments, a real-data scale-swap intervention, four-architecture pretraining, and supervised forecasting experiments. SCALEIN achieves lower MASE in all 24 aggregated pretraining comparisons and 16 of 20 supervised comparisons, supporting its benefits under the evaluated settings.

However, based on my understanding of time series data and experience implementing forecasting training pipelines, I am not convinced that the issue is as practically significant as the paper suggests. It may already be mitigated in commonly used pipelines, or its remaining impact may be limited. My concern primarily arises from the implementations in TSLib and Chronos-2. The conventional TSLib workflow applies per-channel standardization using training-set statistics before model input, followed by RevIN-style instance normalization, model prediction, reversal of instance normalization, and loss computation. Chronos-2 instead accepts inputs at their original numerical scales, applies instance normalization, generates predictions, computes the loss against correspondingly normalized targets, and subsequently reverses normalization for the returned predictions. Although the loss locations differ, reversing instance normalization in TSLib returns predictions to the already standardized data space, rather than directly to the original physical scale. Consequently, this operation does not inherently produce extremely large or small values, nor does it by itself establish a problem of exploding or vanishing gradients. Both workflows appear reasonable. In my view, Chronos-2’s design is particularly convenient for heterogeneous inputs from many datasets, avoids the need for separately fitted dataset-level preprocessing, and naturally accommodates zero-shot inference. Some variation in numerical loss values is also expected during training and is not, by itself, evidence of impaired optimization. For a uniform multiplicative rescaling of the entire loss, its effect under ordinary SGD can be absorbed into the learning rate without changing the gradient direction; whether the remaining example-dependent weighting causes a substantial practical disadvantage requires further evidence.

For example, suppose dataset-level standardization uses μD=1000\mu_D=1000 and sD=500s_D=500. A target value of 15001500 therefore becomes x=1x=1. Suppose the subsequent instance normalization uses a window mean of a=0.2a=0.2 and a standard deviation of b=0.8b=0.8. The normalized target is z=1−0.20.8=1.z=\frac{1-0.2}{0.8}=1. If the model predicts z^=0.9\hat z=0.9, computing MSE before reversing instance normalization gives Lbefore=(1−0.9)2=0.01.\mathcal L_{\mathrm{before}}=(1-0.9)^2=0.01. Reversing instance normalization gives x^=0.8×0.9+0.2=0.92\hat x=0.8\times0.9+0.2=0.92, so computing MSE afterward gives Lafter=(1−0.92)2=0.0064.\mathcal L_{\mathrm{after}}=(1-0.92)^2=0.0064. The latter prediction remains in the dataset-standardized space. Thus, the two losses differ by a moderate factor of 0.640.64, despite the original observations being on the order of thousands. This example does not establish that all windows have comparable weights, but illustrates why loss computation after instance de-normalization need not create extreme numerical scales when dataset-level standardization has already been applied.

Of course, there are exceptions. For example, for a monotonically increasing sequence, after inverse standardization, the gradient value calculated for the first half will definitely be smaller than that for the second half. However, the time series data we deal with is rarely monotonically increasing; it usually exhibits significant linear or non-linear periodicity. This is evident in current TSFMs, where models tend to predict cycles well but are less effective at predicting trends. In other words, current TSFMs primarily learn to fit the periodicity.

Critical strengths
  1. The paper is well organized, with a clearly stated motivation, a well-defined research question, and a coherent connection between its analysis and experiments.
  2. The empirical evaluation is broad, covering controlled synthetic experiments, real-data interventions, and multiple forecasting architectures, with repeated runs and additional controls.
  3. The proposed training modification is simple to implement, introduces little computational overhead, and can be incorporated into existing normalization-based forecasting pipelines.
Critical weaknesses
  1. I think the practical significance of the central issue is still not clear enough. Overall, the paper is complete and well-organized. However, as discussed in my abstract, my main reservation concerns the practical significance of its motivations. The mere existence of scale-related example weights does not materially limit commonly used prediction pipelines. Dataset-level standardization may have mitigated large-scale variance, and normalized spatial loss calculations have been adopted by models like Chronos-2. Therefore, I still do not believe the paper adequately identifies when residual weights become important and why existing preprocessing practices are insufficient. This concern about practical significance is the main reason for my low score.

  2. The preprocessing process needs to be clarified and actual controls strengthened. Appendices G.1 and H describe context-based standardization, but do not explicitly determine whether additional dataset-level or per-channel normalization is applied before model input. The corresponding preprocessing details for the MOMENT experiment are also limited. Additionally, Section 4.2 covers global standard scaling, while Appendix F.1 describes window-based normalization, followed by specified rescaling; Their relationship should be clarified. These details determine the numerical space for actual calculation of so-called raw space loss. Paired comparisons based on training sets, after per-channel normalization (as commonly used in TSLib), will help determine SCALEIN's additional advantages over traditional preprocessing pipelines.

  3. Chronos-2 comparison does not separate scale weights from nonlinear residual geometry. Appendix B.2 correctly acknowledges that, in addition to introducing the (b^p) factor, inverting the arcsine transform also changes the residual geometry. Therefore, performance differences between the reported conditions cannot be entirely attributed to the weights related to proportion. Using the intermediate control of dividing the original space loss by (b^p) will retain the nonlinear inverse transformation while removing the explicit scaling factor. Comparing this control with SCALEIN and SCALECON will clarify the respective contributions of scale-weighted and nonlinear loss geometry.

Questions for authors

See the Paper Summary and Critical Weaknesses sections for specific questions. I currently believe this issue is not crucial, but I would be happy to increase my rating if the authors can provide strong evidence.

6 comments
AuthorsOpenReview ↗

Response (1/2)

We sincerely thank the reviewer for an in-depth analysis of the paper and for providing a constructive review. In the following, our aim is to concisely address the concerns. Should any more details remain unclear, we would be happy to promptly follow up.

W1

We agree that dataset-level standardization can help alleviate scale differences and promote more stable training. However, it does not remove the example-dependent weighting, and we respectfully note that the reviewer's own example confirms that the two losses still differ by an example-dependent factor after standardization.

As discussed in the introduction and illustrated in Fig. 1, scale varies across training examples in two ways: across series (Fig. 1d) and within a single series across forecast creation dates (Fig. 1e). Dataset-level standardization mitigates loss bias introduced by heterogeneous scales across series. However, it does not remove the within-series bias and can still result in loss function bias on the order of magnitudes.

Formally, let β\beta denote the global standard deviation and btb_t the t-th context-window's standard deviation before global standardization. For MSE, the ScaleCon loss after global standardization satisfies LContam,t=(btβ)2LInv,tL_{\text{Contam},t}=\left(\frac{b_t}{\beta}\right)^2L_{\mathrm{Inv},t} assuming scale-equivariant normalization (e.g., standardization). This shows precisely that global standardization does not remove scale differences in local variance, as btb_t can still vary.

Moreover, this effect does not necessarily have to be of small magnitude as in the example provided by the reviewer. For example, consider a globally standardized retail series (mean 0, std 1) with intermittent demand. One window has instance std b=0.1b = 0.1, and the second, relatively more volatile window, has b=2.0b = 2.0. For the same normalized error (z^−z=0.1\hat{z} - z = 0.1), ScaleCon MSE gives the volatile window 400×400\times the scale-dependent weight (b2b^2 ratio: 4.0/0.014.0 / 0.01). ScaleIn gives both equal weight. We agree that this weighting may sometimes be negligible when, e.g., window variances are approximately constant, as can occur with stationary periodic series. However, trends, regime changes, level shifts, heteroscedasticity, non-stationarity, seasonal peaks, and intermittent demand can produce substantial variation in bt/βb_t/\beta. Importantly, all of these phenomena/effects are common in any time-series data and can still lead to large disparities.

To further back this empirically we run two experiments:

  1. We sampled 1k series from our TSFM pretraining corpus (GIFT-Eval Pretrain), z-scored each series by its own statistics (as in TSLib), and computed the ScaleCon weight b2b^2 (MSE loss) of 50 randomly sampled context windows per series. Within a single series, the window at the 90th weight percentile still receives a median of 2.5×2.5\times the weight of the window at the 10th percentile. Furthermore, in 10% of series this ratio exceeds 53×53\times. As evidenced, per-series standardization removes scale differences between series but not within them, and these within-series differences can still be large.

  2. We repeated the controlled synthetic experiment of Section 4.1, but with the three sources as alternating regimes within each series. Each series was multiplied by a random scale in [1,1000][1, 1000] and then z-scored by its own statistics (TSLib style). Under ScaleIn, all regimes reach nMSE <0.1<0.1 within 157 to 183 steps. Under ScaleCon, the high-variance regime does so in 59 steps and the low-variance regime only after 8k. Swapping the regime variances reverses this ordering, so it follows the scale rather than the difficulty of the pattern.

We added the extra formal analysis of the global norm and these two empirical experiments with visualizations to a new Appendix J.

Finally, we would like to clarify that the aim of this work is to formalize and raise awareness of the existence of this implicit weighting, which we believe is a practically significant consideration for anyone training a TSFM. We do not aim to identify a threshold at which this weighting becomes harmful because it is application-driven. We note that global norm + ScaleCon can perhaps be used for pre-training, but the sole fact of having an implicit curriculum/scale bias is worth knowing and accounting for. Intuitively speaking, by default we do not see the reason why the unit of the series (e.g., cents vs dollars) should decide the weight of a particular series, and practitioners should know that if they use ScaleCon or ScaleCon + global norm, this weighting exists. Moreover, as mentioned in the paper, the choice of loss space is only explicitly documented for 9 of 24 TSFMs we surveyed, and the field has no dominant convention: 14/21 TSFMs use ScaleCon while the remaining 7 use ScaleIn. We argue that loss space should be an explicit design choice, consistently reported in the literature.

AuthorsOpenReview ↗

Response (2/2)

W2

Section 4.2 does not use global standard scaling, and our phrasing was incorrect. In fact, any series-level standardization there would be redundant because of later per-window normalization. Each context window is scaled to zero mean and unit standard deviation, and then multiplied by its assigned b∈1,10b \in {1, 10} (Appendix F.1), so the only scale difference between windows is the controlled factor bb. We corrected it in the revised paper. All other experiments do not apply any "global" standardization before instance normalization.

Appendix I (MOMENT): We have now added more pre-training details.

W3

We agree that Chronos-2's nonlinear arcsinh transformation introduces an additional difference in residual geometry, as already acknowledged in Appendix B.2. We still attribute the Chronos-2 ScaleIn/ScaleCon performance gap to implicit gradient weighting, but we note that the weighting was not bpb^p exactly. Furthermore, it is easy to show that the arcsinh transformation does not remove scale-dependent gradient weighting, and for quantile loss, each prediction's ScaleCon gradient contribution is exactly bcosh⁡(u^)b\cosh(\hat u) times its ScaleIn counterpart, where u^=arcsinh(z^)\hat u = \text{arcsinh}(\hat z) is the transformed prediction.

To add empirical evidence, we repeated the controlled synthetic experiment of Section 4.1 with and without Chronos-2's arcsinh after instance normalization, for both loss spaces (added to Appendix E.3). To briefly summarize the setup of the experiment, we have 3 synthetic sources with controlled scales (10σ,10σ,σ10\sigma, \sqrt{10}\sigma, \sigma) and train a small PatchTST-style model with Adam. ScaleCon reaches nMSE <0.1<0.1 on the sources after 10, 20, and 106 steps without the arcsinh and after 14, 25, and 139 steps with it, respectively. ScaleIn converges in parallel in both cases (between 9–10 and 19–20 steps). The results show that scale-driven convergence disparities persist with arcsinh

Most importantly, our paper is not specifically about Chronos-2. There are other architectures that tweak the standard affine normalization. Our central contribution is broad and characterizes the implicit scale-dependent weighting introduced by loss computation and demonstrates its consequences for TSFM training. We establish the (bp)(b^p) weighting analytically for affine scaling (Lemma 3.1), isolate its optimization consequences through controlled synthetic and real-data scale-swap experiments (Sections 4.1 and 4.2), and validate ScaleIn across common forecasting architectures. Importantly, Moirai-2.0, TimesFM-2.5, and PatchTST do not employ the additional nonlinear transformation, yet ScaleIn improves MASE across the board. Thus, our central findings do not depend on the Chronos-2-specific comparison.


We again thank the reviewer for the comprehensive review. We hope that we managed to accurately address the issues raised. If we did not provide the expected explanation, we would be happy to promptly provide more details over the next few days.

I sincerely thank the author for their reply, which indeed resolved some of my concerns. However, after much reflection, I believe the issue raised in your paper is neither particularly targeted nor profound. For example, in my personal experience, I initially used SCALEON for normalization in the TSLib, a method I used for a long time. It wasn't until last year, when Chronos-2 open sourced their model pipeline, that I discovered they were using SCALEIN to calculate the quantile regression loss. But the moment I saw their code, I immediately understood why they did it that way. To a large extent, the foundation model encounters different types of representation data at each step during training, which may come from different datasets and have different scale information. Therefore, using SCALEIN for real-time normalization is the best approach (although I acknowledge that this method might provide a more stable and robust loss in eliminating time scale bias). Otherwise, my colleagues who also study time series immediately understood why this was the approach. Our subsequent work will also consciously adopt SCALEIN, without any instruction from anyone or anything. Because we can subconsciously tell which method is more appropriate. Therefore, I believe that most people working on TSA are aware of this issue, and it's not a particularly important or noteworthy problem to specifically raise awareness about. So, I have decided to maintain my rating for now.

If the author has any other opinions or views, please discuss them with me.

Comments on the paper

1 comment
Area_Chair_N8KWOpenReview ↗

Reminder: Please Participate Actively in the Discussion

Dear Reviewers,

Please actively participate in the discussion by responding to the authors’ replies and engaging with your fellow reviewers. This discussion is essential for clarifying concerns, resolving differences, and reaching a well-supported consensus on the final recommendation.

Thank you for your time and valuable contributions.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.