acceptodds
Under review as a conference paper at ICLR 2027

Robust Time Series Forecasting via Bounded Kernel Loss

Abstract

Time series forecasting models predominantly rely on Mean Squared Error (MSE) optimization, which fundamentally assumes independent, identically distributed Gaussian residuals. However, in real-world scenarios, the complex interplay between inherent data noise (aleatoric uncertainty) and model capacity limits (epistemic uncertainty) frequently yields asymmetric, heavy-tailed error distributions. Under MSE's unbounded quadratic penalty, these unlearnable extreme residuals disproportionately dominate the gradient updates, forcing the model to overfit noise and severely degrade generalization. To resolve this fundamental misalignment, we propose a robust optimization framework centered on Bounded Kernel Loss (BKLoss). Rather than penalizing all errors uniformly, BKLoss evaluates prediction quality through localized Gaussian-kernel similarity. This mechanism naturally caps the gradient contribution of extreme outliers, allowing the model to dynamically assign lower weights to unlearnable deviations and focus its capacity on generalizable patterns. We further develop a Top- Mean Adaptive Strategy that dynamically calibrates the batch-wise kernel bandwidth based on the residual statistics. Supported by theoretical analysis of the loss formulation and gradient boundedness, extensive experiments across multiple benchmark datasets demonstrate that our method consistently outperforms strong baselines, achieving highly stable forecasting performance in highly uncertain environments. The code is provided in the supplementary material.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Robust Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ThToOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper investigates the limitations of MSE-based optimization in time series forecasting, arguing that large residuals arising from aleatoric and epistemic uncertainty can dominate gradient updates and impair generalization. To address this issue, the authors propose Bounded Kernel Loss (BKLoss), which suppresses the influence of extreme residuals through Gaussian-kernel similarity, together with a Top-β Mean strategy for adaptive bandwidth selection. Experiments across multiple datasets and forecasting architectures demonstrate consistent improvements in many settings, particularly for models with limited representation capacity.

Critical strengths
  1. Practical and relevant problem. The paper addresses the sensitivity of MSE-based optimization to extreme residuals, a meaningful challenge in real-world time series forecasting. The proposed loss is model-agnostic and can be integrated into existing forecasting architectures without architectural modifications.
  2. Simple and interpretable design. BKLoss provides a straightforward mechanism for reducing the influence of large residuals, while the adaptive bandwidth strategy attempts to accommodate different residual scales across datasets and training stages.
  3. Broad empirical evaluation. The experiments cover multiple forecasting architectures, real-world benchmarks, and synthetic scenarios. The comparisons between adaptive and fixed bandwidths, as well as soft downweighting and hard truncation, provide useful evidence for the effectiveness of the proposed design.
Critical weaknesses
  1. Limited novelty and insufficient differentiation from existing robust losses. The proposed BKLoss closely resembles classical correntropy-based or Welsch-type robust losses, which already suppress extreme residuals through bounded influence functions. Although the adaptive bandwidth strategy may provide an additional contribution, the paper lacks direct comparisons with established robust objectives such as Huber, Cauchy, and Welsch losses. Consequently, it is unclear whether the improvements arise from a genuinely new optimization principle or from applying an existing robust loss to time series forecasting.
  2. The distinction from Selective Learning is not sufficiently established. The paper motivates its method using aleatoric and epistemic uncertainty, which appear conceptually related to the anomaly and uncertainty factors considered in Selective Learning. Both approaches aim to reduce the interference of unreliable or difficult-to-learn observations during training. However, the authors mainly distinguish their method by contrasting continuous downweighting with hard masking, without sufficiently clarifying the conceptual differences in their uncertainty assumptions or demonstrating why the proposed formulation is fundamentally preferable. A more explicit comparison of the underlying motivations and optimization mechanisms is needed.
  3. Insufficient evidence for the central uncertainty-driven optimization hypothesis. The paper implicitly associates large residuals with unlearnable patterns, although large errors may also arise from informative but underfitted observations or rare yet predictable events. The experiments do not independently control aleatoric and epistemic uncertainty, nor do they directly demonstrate that extreme residuals dominate training gradients and that BKLoss mitigates this effect. Thus, the reported performance gains do not fully validate the proposed explanation of why the method works.
  4. The adaptive bandwidth strategy lacks sufficient justification. The Top-β Mean strategy estimates the bandwidth using the largest residuals, potentially making it sensitive to the very outliers the method seeks to suppress. Its advantage over more robust alternatives, such as quantile-based or median-based estimators, is not established. Moreover, the theoretical gradient analysis primarily considers a fixed bandwidth and does not fully characterize the optimization behavior when the bandwidth depends on batch residuals.
Questions for authors

See weakness.

5 comments

A3.1

We thank the reviewer for their question.

Regarding comparisons with classical robust losses such as Huber, Tukey, Cauchy, and Student-t, Appendix B.4 already reports experimental results on synthetic data. In A2.2, we further provide comparisons on real-world datasets and specify the thresholds, scales, and common training settings used for these robust losses. These results show that our method outperforms classical robust loss functions.

Our design explicitly considers the characteristics of time series data. Time series distributions are non-stationary and contain both hard-to-learn and easy-to-learn components. These distributions vary not only across datasets but also across batches. Through the Top-β\beta strategy, we adapt the Welsch-type robust loss to time series forecasting and achieve strong performance. Please refer to A1.1 for a detailed explanation.

A3.2

We thank the reviewer for their comments. We explicitly distinguish between aleatoric uncertainty and epistemic uncertainty. Our basic assumption is that samples in the data can be divided into easy-to-learn and hard-to-learn categories. Hard-to-learn samples are not necessarily devoid of information, so we continuously downweight them according to their residual magnitudes rather than directly removing them. The hard-truncation experiment in Figure 4(b) shows that completely removing samples with large errors substantially degrades forecasting performance, indicating that large residuals may still contain useful, predictable information.

From an optimization perspective, Selective Learning performs hard sample selection through uncertainty and anomaly masks, but does not further distinguish between aleatoric and epistemic uncertainty or explicitly model the aleatoric component. In contrast, BKLoss treats the residual as a continuous indicator of learnability. Its weight decreases smoothly as the residual increases while retaining nonzero gradients, thereby suppressing interference from unlearnable patterns during optimization while preserving useful information in hard-to-learn samples. Most importantly, Selective Learning does not distinguish between data from different batches, whereas BKLoss adapts to each individual batch. This is a distinctive advantage of our method.

To further compare optimization behavior under the two types of uncertainty, we conduct additional experiments on synthetic trend data. The trend signal contains long-period components with periods of 168 and 336, both exceeding the prediction window of 96, to construct hard-to-learn structures. We then consider Clean, Drift, Gaussian, and Laplace conditions to examine how different perturbations affect the three methods. All experiments use the same iTransformer backbone, prediction length of 96, and random seed of 2021, while the test set remains clean. Each cell below reports test MSE/MAE; lower is better. SL denotes Selective Learning.

DatasetConditionMSEBKLossSL
trend_linearClean0.00007290/0.006768580.00004880/0.005298850.00226445/0.02493423
trend_linearDrift0.00013644/0.009234530.00006312/0.006452370.00300110/0.03114157
trend_linearGaussian0.00769161/0.072541680.00702716/0.068423890.00874804/0.07628339
trend_linearLaplace0.00755157/0.072217400.00716223/0.069621560.00785655/0.07322811
trend_quadraticClean0.00124635/0.026933530.00095023/0.023051020.00625199/0.04953751
trend_quadraticDrift0.00082619/0.021807750.00054309/0.017500120.00587884/0.04315763
trend_quadraticGaussian0.01111832/0.082505190.01025336/0.077731180.01070959/0.08393519
trend_quadraticLaplace0.01104065/0.079960770.01103054/0.084290990.01430890/0.09544443

On clean trend data, BKLoss achieves substantially better test MSE and MAE than Selective Learning, suggesting that Selective Learning's hard selection mechanism may mistakenly discard samples containing predictable structures. Under Drift, Gaussian, and Laplace perturbations, BKLoss still achieves the best results in most conditions.

A3.3

We thank the reviewer for their comments. We also agree that large residuals cannot simply be equated with useless noise. The hard-truncation experiment in Figure 4(b) shows that completely removing samples with large errors substantially degrades forecasting performance, indicating that these samples still contain useful information. BKLoss continuously downweights large residuals rather than discarding them entirely, thereby preserving this information. We additionally provide warm-up experiments as further indirect evidence, as shown in the figure. MSE decreases rapidly during the early stages of training, indicating that the model has already learned most of the main patterns at this stage. Subsequently switching to BKLoss improves performance, indicating that continuing to force the model to fit the high-error tail may impair learning on regular samples. Since the early MSE updates have already altered the optimization trajectory, switching at a later stage cannot fully reverse this effect. This is also why we ultimately use BKLoss from the beginning of training.

Regarding the two types of uncertainty, as stated on line 51 of the paper, from the optimizer's perspective, “unlearnable” patterns behave in the same way as purely random noise. Since real-world data lack per-sample ground-truth annotations, we cannot directly determine whether a large residual arises from random noise or an insufficiently fitted predictable event. Therefore, in Section 4.4, we explicitly control the learnable structures and perturbation components in synthetic data to examine the two types of uncertainty separately.

Regarding the influence of extreme residuals on gradients, Figure 6 provides a direct analysis. The gradient magnitude of MSE increases linearly with the absolute residual, whereas the gradient of BKLoss is

∣∂LBK∂e∣=∣e∣σ2exp⁡(−e22σ2).\left|\frac{\partial \mathcal{L}_{\mathrm{BK}}}{\partial e}\right| = \frac{|e|}{\sigma^2}\exp\left(-\frac{e^2}{2\sigma^2}\right).

Once ∣e∣>σ|e| > \sigma, the gradient magnitude decreases as the residual magnitude increases further. Therefore, BKLoss continuously downweights extreme residuals while retaining their nonzero gradients, avoiding the complete information loss caused by hard truncation.

A3.4

We thank the reviewer for their comments. MAD, the 80th percentile, and Top-β\beta all estimate an adaptive bandwidth from the residual distribution of the current batch and achieve comparable forecasting performance in our experiments. This suggests that the effectiveness of BKLoss does not depend on one particular bandwidth estimator. Top-β\beta averages the largest 20% of absolute residuals, incorporating scale information from multiple tail samples rather than relying solely on a single maximum or a particular quantile.

We believe that averaging multiple tail residuals helps smooth bandwidth fluctuations caused by small changes in individual residuals. Meanwhile, when extreme residuals increase the upper-tail scale, the estimated σ\sigma also increases accordingly. This expands the kernel's non-saturated range and reduces the risk of excessively downweighting normal or otherwise learnable samples because of an overly small bandwidth. These considerations motivate our choice of Top-β\beta.

We compare MAD, the 80th percentile, and Top-β\beta on iTransformer. Lower values are better, and the best results are shown in bold.

DatasetMADQuantile (Q0.8Q_{0.8})Top-β\beta
ETTh20.3853640.3830980.378750
ETTm10.4099980.4050780.398500
ETTm20.2875200.2847320.284000
Electricity0.1915930.1843880.175750
Weather0.2591630.2569620.257500

Thank you for the rebuttal and the additional experiments. They strengthen the empirical evaluation. My main concerns remain the modest methodological novelty and the limited evidence for the proposed optimization mechanism, so I will keep my score.

Reviewer_wNm4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies robust training objectives for time-series forecasting and proposes BKLoss, which minimizes one minus the average Gaussian-kernel similarity of prediction residuals. It combines this bounded loss with a batch-adaptive bandwidth estimated from the mean of the largest absolute residuals. The theoretical analysis establishes fixed-bandwidth properties, including bounded residual scores, a rescaled MSE limit, and a sufficient condition for improved clean-test population MSE under mean-shift training contamination. Experiments cover eight real-world datasets and four forecasting backbones, with five-seed results, comparisons with alternative objectives, synthetic corruption experiments, bandwidth ablations, and residual analyses.

Critical strengths
  1. The empirical evaluation is broad and includes meaningful reproducibility evidence. The main experiments cover eight datasets and four forecasting architectures, with five-run means and corresponding standard deviations reported in the appendix. Using the same β=0.2 across these tasks is a practical strength, as it reduces the possibility that the improvements depend on extensive task-specific tuning. Some gains are substantial, such as the reduction in DLinear’s average MSE on ETTh2 from 0.561 to 0.464.

  2. The paper provides useful analysis beyond aggregate forecasting scores. It includes controlled training-corruption experiments, comparisons with classical robust losses in Appendix B.4, fixed-versus-adaptive bandwidth comparisons, and residual-tail statistics. The fixed-bandwidth theoretical results are also explicitly scoped: in particular, Theorem 1 states sufficient conditions for improvement under mean-shift training contamination and clean test targets, rather than claiming universal superiority over MSE.

Critical weaknesses
  1. The contribution needs clearer positioning relative to established correntropy-based robust regression. For fixed bandwidth, Equation (2) is equivalent, up to bandwidth parameterization and positive scaling, to the Gaussian maximum correntropy/Welsch loss. The paper acknowledges inspiration from Liu et al. (2007), but describing the underlying loss as a novel objective or a paradigm shift overstates this component. Related regression analysis is also available in Feng et al., “Learning with the Maximum Correntropy Criterion Induced Losses for Regression,” JMLR 2015. The main methodological contribution therefore appears to be the Top-β bandwidth rule and its forecasting evaluation. Although Figure 4(a) compares against fixed bandwidths and Appendix B.4 includes Huber, Tukey, Cauchy, and Student-t losses, the manuscript does not sufficiently specify the robust baselines’ scale settings or establish the advantage over competitive bandwidth-selection strategies on real-world datasets.

  2. The fixed-bandwidth analysis does not establish the behavior of the adaptive method used in the experiments. Equation (3) is the derivative with σ held fixed, but Equation (4) makes σ depend on the current residuals, without specifying whether this dependence is detached during backpropagation. If gradients pass through σ, additional terms arise, and the composite loss is scale-invariant whenever the bandwidth floor is inactive: L(cE;σ(cE))=L(E;σ(E)). It can even decrease when a single residual increases because that residual enlarges the bandwidth for the entire batch. If σ is detached, this particular gradient issue changes, but the Top-β mean remains sensitive to the extreme residuals the method is intended to handle. The fixed-bandwidth score bound alone does not resolve this concern. This missing implementation detail and the limited analysis of bandwidth robustness make it difficult to assess the central adaptive mechanism.

Questions for authors
  1. Implementation clarification: Is the adaptive bandwidth computed under stop-gradient/no_grad, or do gradients propagate through the Top-β mean? Please provide the loss implementation and clarify how Equation (3) relates to the update actually used. If gradients propagate through σ, how is the scale invariance of the composite objective addressed?

  2. Contribution and comparison: Can the authors explicitly distinguish the proposed contribution from Gaussian maximum correntropy regression, and compare the Top-β rule with a validation-selected fixed bandwidth and a robust scale-based bandwidth rule? Appendix B.4 already provides useful robust-loss comparisons; please specify the thresholds, scales, and tuning budgets for those baselines and clarify whether the advantage persists on representative real-world datasets.

  3. Robustness evidence: How does the method behave as contamination amplitude and fraction vary, especially when a very small number of extreme residuals dominates the Top-β mean? A controlled comparison with predictable rare events would also help determine whether the method suppresses corruption without discarding useful high-error examples. Theorem 1 concerns nonzero-mean contamination at fixed bandwidth, whereas the symmetric spike construction in Appendix A.3 does not directly instantiate those assumptions.

  4. Experimental clarification: Are the residual statistics in Tables 4 and 10–12 computed from the same checkpoints, seeds, normalization, and evaluation protocol as Table 1? For example, DLinear on ETTm1 at horizon 96 has BKLoss MSE 0.332 in Table 1, whereas Table 4 reports RMSE 0.5867, whose square is approximately 0.3442. Please explain the aggregation or experimental differences, since these tables are used to support the proposed mechanism.

4 comments

A2.1

We thank the reviewer for their question. We clarify that the adaptive bandwidth σ\sigma is computed under stop-gradient/no_grad, and the Top-β\beta mean does not participate in backpropagation. It is defined as

σb=sg⁡[max⁡(ϵ,1k∑i∈TopK⁡(∣r∣)∣ri∣)].\sigma_b = \operatorname{sg}\left[\max\left(\epsilon,\frac{1}{k}\sum_{i\in\operatorname{TopK}(|r|)}|r_i|\right)\right].

Therefore, within each mini-batch, σb\sigma_b is first computed and then held fixed. The actual update corresponding to Eq. (3) is

∂LBK∂y^i=−riNσb2exp⁡(−ri22σb2).\frac{\partial \mathcal{L}_{\mathrm{BK}}}{\partial \hat y_i} = -\frac{r_i}{N\sigma_b^2}\exp\left(-\frac{r_i^2}{2\sigma_b^2}\right).

Since gradients do not flow through σb\sigma_b, no additional chain-rule term involving ∂σb/∂ri\partial\sigma_b/\partial r_i appears during backpropagation. Therefore, the scale invariance of the composite objective pointed out by the reviewer does not carry over to the actual parameter updates.

A2.2

We thank the reviewer for their question. The contribution of this work lies in proposing an entirely new perspective on residual learning: residuals contain both patterns that the model can continue to learn and hard-to-learn components arising from noise, sudden disturbances, and limitations in model capacity. We also introduce correntropy to time series forecasting for the first time and address both types of uncertainty. Please refer to A1.1 for a detailed explanation. Different datasets have different data distributions, and even different batches may have different noise distributions. Consequently, a fixed bandwidth cannot adapt well to all data, which is an important reason for using Top-β\beta. See A1.2 for detailed comparisons.

Appendix B.4 already reports comparisons with Huber, Tukey, Cauchy, and Student-t on synthetic data. To further verify whether these results hold for real-world time series, we provide additional results for iTransformer and DLinear on ETTh1, ETTh2, ETTm1, ETTm2, Solar, and ILI. The prediction length is 96, except for ILI, which uses the corresponding prediction length of 24. Huber uses δ=1.345\delta=1.345, Tukey uses c=4.685c=4.685, Cauchy uses c=2.385c=2.385, and Student-t uses ν=4\nu=4 and scale = 1. All methods use a residual scale of 1.

iTransformer (MSE/MAE)

DatasetHuberTukeyCauchyStudent-tBKLoss
ETTh10.393/0.4080.393/0.4070.393/0.4070.392/0.4060.391/0.404
ETTh20.298/0.3460.297/0.3450.298/0.3460.297/0.3450.296/0.342
ETTm10.336/0.3710.336/0.3700.334/0.3680.332/0.3660.325/0.353
ETTm20.179/0.2610.182/0.2650.178/0.2600.178/0.2600.177/0.255
Solar0.201/0.2360.198/0.2330.202/0.2330.204/0.2320.196/0.218
ILI3.088/1.1853.082/1.1833.081/1.1823.076/1.1802.993/1.150

DLinear (MSE/MAE)

DatasetHuberTukeyCauchyStudent-tBKLoss
ETTh10.394/0.4080.393/0.4070.393/0.4070.392/0.4060.390/0.403
ETTh20.313/0.3700.308/0.3650.310/0.3660.308/0.3650.306/0.359
ETTm10.341/0.3680.341/0.3680.339/0.3660.338/0.3640.332/0.354
ETTm20.186/0.2810.184/0.2770.184/0.2780.184/0.2760.182/0.265
Solar0.287/0.3680.286/0.3650.285/0.3610.284/0.3570.283/0.336
ILI2.805/1.1462.795/1.1422.786/1.1402.767/1.1322.738/1.112

These results show that BKLoss also outperforms commonly used robust loss functions on real-world datasets.

A2.3

We thank the reviewer for their detailed comments. The performance of the methods under varying contamination magnitudes and proportions is shown in the figure. BKLoss maintains good performance as either the contamination magnitude or proportion increases. Moreover, its advantage over MSE tends to widen as the contamination magnitude and proportion increase.

We agree that the global contamination construction should be distinguished from the batch-level theoretical conditions. Appendix A.3 applies symmetric spike contamination across the entire dataset, so the global contamination mean can be zero. However, Theorem 1 concerns a single training batch, and a random mini-batch does not necessarily contain equal numbers of positive and negative spikes. Its empirical contamination mean may therefore remain nonzero. We further constructed a seven-channel synthetic dataset with multiple short periods by superimposing four periodic components with periods of 12, 24, 48, and 96. We randomly injected symmetric +3/−3+3/-3 spike noise with magnitude A=3A=3 at a contamination proportion of p=0.15p=0.15 and examined all 10,600 target batches across DLinear and iTransformer trained with MSE and BKLoss. The results show that 99.33% of batches have a nonzero noise mean, while only 0.67% have a noise mean of exactly zero.

We also agree that large residuals cannot simply be equated with useless noise. The hard-truncation experiment in Figure 4(b) shows that completely removing samples with large errors substantially degrades forecasting performance, indicating that these samples still contain useful information. BKLoss continuously downweights large residuals rather than discarding them entirely, thereby preserving this information. We additionally provide warm-up experiments as further indirect evidence, as shown in the figure. MSE decreases rapidly during the early stages of training, indicating that the model has already learned most of the main patterns at this stage. Subsequently switching to BKLoss improves performance, suggesting that continuing to force the model to fit the high-error tail may impair learning on regular samples. Since the early MSE updates have already altered the optimization trajectory, switching at a later stage cannot fully reverse this effect, which further highlights the importance of BKLoss.

Therefore, we will narrow the manuscript's wording that associates large residuals with unlearnable noise. We will explicitly clarify that the current results support the distinction between continuous downweighting and hard removal, but cannot, on their own, establish that all large residuals arise from unpredictable contamination. Further distinguishing predictable rare events requires controlled experiments with and without available input cues.

A2.4

We thank the reviewer for pointing this out. We would like to clarify that all results reported in this paper were obtained from actual experiments, and our code is included in the supplementary materials. Upon checking, we found that the residual table reports results from a single random seed, whereas the main table reports averages over five random seeds. We will promptly update the residual table in the revised manuscript using results from the same five random seeds as the main table.

Thank you for the clarifications and additional experiments. The stop-gradient implementation resolves my gradient-related concern, and the robust-loss and bandwidth comparisons strengthen the empirical evidence.

Two points remain: nonzero empirical batch means alone do not establish the population assumptions and sufficient condition in Theorem 1; please clarify this distinction. Please also update the residual statistics using the same five seeds and evaluation settings as the main results.

While the methodological novelty remains modest, the practical simplicity and broad empirical evaluation support weak acceptance. I therefore maintain my rating of 3.

First, thank you for pointing out that the relationship between Theorem 1 and the spike contamination experiment needed further clarification. We added a one-sided spike experiment using the same contamination locations, contamination rate (approximately 15%), and training configuration as in the symmetric setting, with all spike values set to +3.

Under both types of contamination, BKLoss reduced MSE and MAE on the clean test set for both DLinear and iTransformer across three random seeds. One-sided contamination introduced a positive prediction bias. For iTransformer, the mean bias decreased from +0.43326 with MSE training to +0.21358 with BKLoss training. In the symmetric setting, the mean prediction bias was close to zero for both training losses, and BKLoss still achieved lower test errors.

Symmetric spikes have zero mean at the population level. One possible explanation for the observed improvement is that large residuals in finite samples can still substantially affect parameter updates under MSE. When positive and negative spikes occur in different input windows, the gradients from those windows may also fail to cancel each other out. The results in the symmetric setting are consistent with the soft downweighting of large residuals by BKLoss.

Theorem 1 addresses the population-level mean shift under a fixed bandwidth when its sufficient conditions are satisfied. These two experiments provide empirical evidence on the performance of adaptive BKLoss under both types of training contamination.

ModelTraining contaminationTest MSE (MSE training → BKLoss training)Test MAE (MSE training → BKLoss training)
DLinearSymmetric spikes (±3)0.00720 → 0.00364 (↓49.5%)0.06803 → 0.04791 (↓29.6%)
DLinearOne-sided spikes (+3)0.01997 → 0.01451 (↓27.3%)0.11585 → 0.10008 (↓13.6%)
iTransformerSymmetric spikes (±3)0.00191 → 0.00105 (↓45.2%)0.03441 → 0.02511 (↓27.0%)
iTransformerOne-sided spikes (+3)0.19461 → 0.04743 (↓75.6%)0.43326 → 0.21358 (↓50.7%)

The reported MSE and MAE values are averages over three random seeds, evaluated on the same clean test set at the end of the third epoch.

Second, we have updated the values in the residual table in the revised manuscript.

Thank you again for the time and care you devoted to reviewing our paper and for recognizing the contributions of our work.

Reviewer_TSZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that MSE, the commonly used objective in time series forecasting, implicitly assumes i.i.d. Gaussian residuals, whereas aleatoric and epistemic uncertainty make the error distribution asymmetric and heavy-tailed, so that extreme residuals dominate the gradients under the unbounded quadratic penalty. Inspired by Information Theoretic Learning, the authors propose the Bounded Kernel Loss (BKLoss), which measures the local similarity between predictions and ground truth with a Gaussian kernel. The loss is bounded, and its gradient decays when residuals become too large, thereby softly down-weighting extreme residuals. The authors further propose a Top-β Mean Adaptive Strategy, which sets the kernel bandwidth to the mean of the largest fraction of residuals in each batch. They also provide a theoretical analysis under a fixed bandwidth. In the experiments, the authors replace the original losses with BKLoss on four backbones and eight datasets, compare it with recent training objectives, use synthetic data to analyze different types of uncertainty, and examine the effects of the bandwidth, β, and the kernel function.

Critical strengths
  1. The method is simple and plug-and-play: with only one more hyperparameter β, the bandwidth can be calculated in O(N) time using Quickselect class algorithms with almost no additional overhead

  2. Extensive experimental coverage: 4 types of backbone × 8 datasets, averaged over 5 runs, and the standard deviation is also given.

Critical weaknesses
  1. Limited originality. The formulation of BKLoss is the existing correntropy (Welsch) loss, and the authors themselves state that their method is inspired by Information Theoretic Learning and the correntropy work of Liu et al. (2007). Propositions 1 and 2 are known properties of this type of loss. Moreover, the related work section does not discuss closely related robust losses (e.g., Huber and correntropy-based losses) or explain how BKLoss differs from them. The only genuinely new component is the Top-β adaptive bandwidth.

  2. The benefit of the adaptive bandwidth is limited. Adaptive and fixed bandwidths are compared only in a single radar chart (Fig. 4(a)), only on iTransformer, and no numerical results are reported. Judging from the axis ticks, the differences are mostly in the third decimal place: σ = 2 performs better on Weather and Electricity, and almost coincides with the adaptive bandwidth on ETTh2, ILI, and Solar. The adaptive bandwidth is clearly better only on ETTm1 and ETTm2. Moreover, each axis of the radar chart is scaled separately, which visually exaggerates these small differences.

  3. Limited improvements in the main experiments. The significant improvements are concentrated on weaker backbones such as DLinear and FITS. On stronger backbones such as iTransformer and PatchTST, the gains are mostly marginal, and on Electricity the performance is unchanged or even worse.

Questions for authors
  1. Apart from the Top-β adaptive bandwidth, what is the essential difference between BKLoss and the existing correntropy (Welsch) loss?

  2. In Fig. 4(a), a fixed bandwidth (e.g., σ = 2) performs comparably to or even better than the adaptive bandwidth on most datasets. What, then, is the practical advantage of the Top-β strategy?

  3. The paper attributes the large gains on DLinear to its limited representational capacity. Does this imply that the benefit of BKLoss diminishes as backbones become stronger? If so, what is its value for current state-of-the-art models?

3 comments

A1.1

We thank the reviewer for their comments. The contribution of this work lies in proposing an entirely new perspective on residual learning: residuals contain both patterns that the model can continue to learn and hard-to-learn components arising from noise, sudden disturbances, and limitations in model capacity. Traditional MSE continuously imposes substantial optimization pressure on all residuals, whereas BKLoss exploits the saturation property of the kernel function to reduce the gradient contributions of large residuals. These hard-to-learn components are not directly removed; instead, they are retained in the objective function and downweighted, which fundamentally differs from simply truncating samples or merely removing outliers. Some hard-to-learn components may still be learnable and should therefore be retained.

To the best of our knowledge, existing correntropy/Welsch losses have not been systematically analyzed and applied within a general deep time series forecasting framework around this distinction between “learnable residuals and hard-to-learn residuals.” Our main algorithmic component is the Top-β\beta adaptive bandwidth:

σt=Tβ(∣ei∣i∈Bt),\sigma_t = \mathcal{T}_{\beta}\left(\\{|e_i|\\}_{i \in \mathcal{B}_t}\right),

It dynamically adjusts the effective range according to the residual scale of the current mini-batch, enabling the bounded kernel objective to adapt to residual distributions that vary across datasets, training stages, and batches. Therefore, the novelty of this work lies not in introducing a previously nonexistent Gaussian kernel or re-establishing the basic properties of the Welsch loss, but in reinterpreting existing bounded kernel ideas and systematically developing them into an uncertainty-aware training framework for deep time series forecasting.

A1.2

(1) A fixed σ=2\sigma=2 is not universally optimal, and Top-β\beta achieves better overall performance. The table below reports iTransformer test MSE averaged over the four prediction lengths. Compared with fixed σ=2\sigma=2, Top-β\beta performs better on four datasets and equally well on three, with a slightly higher MSE only on Electricity.

Datasetσ=0.05\sigma=0.05σ=0.1\sigma=0.1σ=0.2\sigma=0.2σ=0.5\sigma=0.5σ=2\sigma=2σ=6\sigma=6Adaptive BKLoss
ETTh10.5520.4790.4540.4480.4570.4620.455
ETTh20.4020.3940.3890.3850.3790.3820.379
ETTm10.4270.4190.4140.4080.4020.4070.399
ETTm20.3000.2930.2890.2840.2880.2910.284
Electricity0.2070.2020.1940.1800.1740.1770.176
Weather0.2700.2620.2590.2570.2580.2590.258
Solar0.6730.4130.2900.2510.2340.2350.234
ILI4.7183.9683.3182.8632.8762.9292.831

(2) Varying residual distributions limit the applicability of a globally fixed σ\sigma. Residual distributions and scales may differ across datasets, prediction lengths, and even mini-batches. A fixed bandwidth therefore requires task-specific searches and cannot adjust to changes in residual scales across batches. In contrast, Top-β\beta dynamically estimates the bandwidth from the current batch's residual distribution, avoiding a separate fixed-σ\sigma grid search for each task.

A1.3

We observe substantial performance improvements on weaker backbones, which is expected from our design because BKLoss can reduce the model's epistemic uncertainty. The higher the model's epistemic uncertainty, the more pronounced the benefits of BKLoss. Meanwhile, BKLoss also improves performance on stronger backbones. For example, on ETTm2 with iTransformer, the Avg-4 MSE decreases from 0.292 to 0.284. To further demonstrate the effectiveness of our method, we additionally evaluate two more recent backbones, TimePro and TimeFilter, with results reported below.

Test MSE/MAE; parentheses indicate the training loss.

DatasetTimePro (MSE)TimePro (BKLoss)TimeFilter (MSE)TimeFilter (BKLoss)
ETTh20.384/0.4080.373/0.3960.373/0.4020.364/0.391
ETTm20.293/0.3350.287/0.3260.272/0.3210.272/0.316
Weather0.283/0.3040.278/0.2940.240/0.2700.235/0.260
Solar0.331/0.3320.325/0.3170.233/0.2680.226/0.239

Furthermore, as shown in the figure, MSE decreases rapidly during the early stages of training, indicating that the model has already learned most of the main patterns at this stage. Subsequently switching to BKLoss improves performance, suggesting that continuing to force the model to fit the high-error tail may impair learning on regular samples. Since the early MSE updates have already altered the optimization trajectory, switching at a later stage cannot fully reverse this effect, which further highlights the importance of BKLoss.

Thank the authors for their detailed responses to the concerns. However, the response did not reduce my concerns about the limited contribution and originality of this paper (as they did not convince me that there is a clear difference between their approach and the existing paradigm), so I chose to maintain the original score.

Thank you for your response. We understand your concerns and would like to clarify two points.

First, we make no claim of novelty for the functional form of BKLoss. Our claim is that time-series data contain both easy-to-learn and hard-to-learn components, and that the hard-to-learn components can still carry useful information. Previous work has generally treated hard-to-learn and unlearnable components alike in two ways.

  1. Many studies have made progress in backbone design. They treat both hard-to-learn and unlearnable components as learnable and seek to improve the model's ability to learn them.
  2. Other studies typically discard both hard-to-learn and unlearnable components.

We believe that treating these two components alike is problematic. BKLoss downweights them, retaining useful information from the hard-to-learn components while limiting how strongly the model fits them.

Second, we believe that time-series data fluctuate and change over time. Residual distributions vary across datasets, models, and even batches. We therefore use the Top-β\beta design to adapt to these different distributions, and it has yielded good results.

We would especially like to draw your attention to the warm-up experiment figure. Comparing BKLoss and MSE during the first three epochs, we observe two patterns.

  1. The test loss typically drops the most during the first three epochs, with the steepest drop occurring in the first epoch. As a rough interpretation, this suggests that the model learns most of what it will learn early in training.
  2. BKLoss gains an early advantage over MSE. Even when the model switches from MSE to BKLoss after the third epoch, it struggles to close the gap that opened during the first three epochs. We interpret this as evidence that training with MSE leads the model to fit noise or uncertainty, which in turn biases its predictions on normal samples. The effects of fitting this noise persist, so switching to BKLoss later fails to close the gap.

We would be grateful if you could reconsider your score in light of these clarifications and the warm-up results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.