- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of MSE-based optimization in time series forecasting, arguing that large residuals arising from aleatoric and epistemic uncertainty can dominate gradient updates and impair generalization. To address this issue, the authors propose Bounded Kernel Loss (BKLoss), which suppresses the influence of extreme residuals through Gaussian-kernel similarity, together with a Top-β Mean strategy for adaptive bandwidth selection. Experiments across multiple datasets and forecasting architectures demonstrate consistent improvements in many settings, particularly for models with limited representation capacity.
Critical strengths
- Practical and relevant problem. The paper addresses the sensitivity of MSE-based optimization to extreme residuals, a meaningful challenge in real-world time series forecasting. The proposed loss is model-agnostic and can be integrated into existing forecasting architectures without architectural modifications.
- Simple and interpretable design. BKLoss provides a straightforward mechanism for reducing the influence of large residuals, while the adaptive bandwidth strategy attempts to accommodate different residual scales across datasets and training stages.
- Broad empirical evaluation. The experiments cover multiple forecasting architectures, real-world benchmarks, and synthetic scenarios. The comparisons between adaptive and fixed bandwidths, as well as soft downweighting and hard truncation, provide useful evidence for the effectiveness of the proposed design.
Critical weaknesses
- Limited novelty and insufficient differentiation from existing robust losses. The proposed BKLoss closely resembles classical correntropy-based or Welsch-type robust losses, which already suppress extreme residuals through bounded influence functions. Although the adaptive bandwidth strategy may provide an additional contribution, the paper lacks direct comparisons with established robust objectives such as Huber, Cauchy, and Welsch losses. Consequently, it is unclear whether the improvements arise from a genuinely new optimization principle or from applying an existing robust loss to time series forecasting.
- The distinction from Selective Learning is not sufficiently established. The paper motivates its method using aleatoric and epistemic uncertainty, which appear conceptually related to the anomaly and uncertainty factors considered in Selective Learning. Both approaches aim to reduce the interference of unreliable or difficult-to-learn observations during training. However, the authors mainly distinguish their method by contrasting continuous downweighting with hard masking, without sufficiently clarifying the conceptual differences in their uncertainty assumptions or demonstrating why the proposed formulation is fundamentally preferable. A more explicit comparison of the underlying motivations and optimization mechanisms is needed.
- Insufficient evidence for the central uncertainty-driven optimization hypothesis. The paper implicitly associates large residuals with unlearnable patterns, although large errors may also arise from informative but underfitted observations or rare yet predictable events. The experiments do not independently control aleatoric and epistemic uncertainty, nor do they directly demonstrate that extreme residuals dominate training gradients and that BKLoss mitigates this effect. Thus, the reported performance gains do not fully validate the proposed explanation of why the method works.
- The adaptive bandwidth strategy lacks sufficient justification. The Top-β Mean strategy estimates the bandwidth using the largest residuals, potentially making it sensitive to the very outliers the method seeks to suppress. Its advantage over more robust alternatives, such as quantile-based or median-based estimators, is not established. Moreover, the theoretical gradient analysis primarily considers a fixed bandwidth and does not fully characterize the optimization behavior when the bandwidth depends on batch residuals.
Questions for authors
See weakness.
5 comments
A3.3
We thank the reviewer for their comments. We also agree that large residuals cannot simply be equated with useless noise. The hard-truncation experiment in Figure 4(b) shows that completely removing samples with large errors substantially degrades forecasting performance, indicating that these samples still contain useful information. BKLoss continuously downweights large residuals rather than discarding them entirely, thereby preserving this information. We additionally provide warm-up experiments as further indirect evidence, as shown in the figure. MSE decreases rapidly during the early stages of training, indicating that the model has already learned most of the main patterns at this stage. Subsequently switching to BKLoss improves performance, indicating that continuing to force the model to fit the high-error tail may impair learning on regular samples. Since the early MSE updates have already altered the optimization trajectory, switching at a later stage cannot fully reverse this effect. This is also why we ultimately use BKLoss from the beginning of training.
Regarding the two types of uncertainty, as stated on line 51 of the paper, from the optimizer's perspective, “unlearnable” patterns behave in the same way as purely random noise. Since real-world data lack per-sample ground-truth annotations, we cannot directly determine whether a large residual arises from random noise or an insufficiently fitted predictable event. Therefore, in Section 4.4, we explicitly control the learnable structures and perturbation components in synthetic data to examine the two types of uncertainty separately.
Regarding the influence of extreme residuals on gradients, Figure 6 provides a direct analysis. The gradient magnitude of MSE increases linearly with the absolute residual, whereas the gradient of BKLoss is
Once , the gradient magnitude decreases as the residual magnitude increases further. Therefore, BKLoss continuously downweights extreme residuals while retaining their nonzero gradients, avoiding the complete information loss caused by hard truncation.
A3.4
We thank the reviewer for their comments. MAD, the 80th percentile, and Top- all estimate an adaptive bandwidth from the residual distribution of the current batch and achieve comparable forecasting performance in our experiments. This suggests that the effectiveness of BKLoss does not depend on one particular bandwidth estimator. Top- averages the largest 20% of absolute residuals, incorporating scale information from multiple tail samples rather than relying solely on a single maximum or a particular quantile.
We believe that averaging multiple tail residuals helps smooth bandwidth fluctuations caused by small changes in individual residuals. Meanwhile, when extreme residuals increase the upper-tail scale, the estimated also increases accordingly. This expands the kernel's non-saturated range and reduces the risk of excessively downweighting normal or otherwise learnable samples because of an overly small bandwidth. These considerations motivate our choice of Top-.
We compare MAD, the 80th percentile, and Top- on iTransformer. Lower values are better, and the best results are shown in bold.
| Dataset | MAD | Quantile () | Top- |
|---|---|---|---|
| ETTh2 | 0.385364 | 0.383098 | 0.378750 |
| ETTm1 | 0.409998 | 0.405078 | 0.398500 |
| ETTm2 | 0.287520 | 0.284732 | 0.284000 |
| Electricity | 0.191593 | 0.184388 | 0.175750 |
| Weather | 0.259163 | 0.256962 | 0.257500 |
Thank you for the rebuttal and the additional experiments. They strengthen the empirical evaluation. My main concerns remain the modest methodological novelty and the limited evidence for the proposed optimization mechanism, so I will keep my score.
A3.1
We thank the reviewer for their question.
Regarding comparisons with classical robust losses such as Huber, Tukey, Cauchy, and Student-t, Appendix B.4 already reports experimental results on synthetic data. In A2.2, we further provide comparisons on real-world datasets and specify the thresholds, scales, and common training settings used for these robust losses. These results show that our method outperforms classical robust loss functions.
Our design explicitly considers the characteristics of time series data. Time series distributions are non-stationary and contain both hard-to-learn and easy-to-learn components. These distributions vary not only across datasets but also across batches. Through the Top-β strategy, we adapt the Welsch-type robust loss to time series forecasting and achieve strong performance. Please refer to A1.1 for a detailed explanation.
A3.2
We thank the reviewer for their comments. We explicitly distinguish between aleatoric uncertainty and epistemic uncertainty. Our basic assumption is that samples in the data can be divided into easy-to-learn and hard-to-learn categories. Hard-to-learn samples are not necessarily devoid of information, so we continuously downweight them according to their residual magnitudes rather than directly removing them. The hard-truncation experiment in Figure 4(b) shows that completely removing samples with large errors substantially degrades forecasting performance, indicating that large residuals may still contain useful, predictable information.
From an optimization perspective, Selective Learning performs hard sample selection through uncertainty and anomaly masks, but does not further distinguish between aleatoric and epistemic uncertainty or explicitly model the aleatoric component. In contrast, BKLoss treats the residual as a continuous indicator of learnability. Its weight decreases smoothly as the residual increases while retaining nonzero gradients, thereby suppressing interference from unlearnable patterns during optimization while preserving useful information in hard-to-learn samples. Most importantly, Selective Learning does not distinguish between data from different batches, whereas BKLoss adapts to each individual batch. This is a distinctive advantage of our method.
To further compare optimization behavior under the two types of uncertainty, we conduct additional experiments on synthetic trend data. The trend signal contains long-period components with periods of 168 and 336, both exceeding the prediction window of 96, to construct hard-to-learn structures. We then consider Clean, Drift, Gaussian, and Laplace conditions to examine how different perturbations affect the three methods. All experiments use the same iTransformer backbone, prediction length of 96, and random seed of 2021, while the test set remains clean. Each cell below reports test MSE/MAE; lower is better. SL denotes Selective Learning.
trend_lineartrend_lineartrend_lineartrend_lineartrend_quadratictrend_quadratictrend_quadratictrend_quadraticOn clean trend data, BKLoss achieves substantially better test MSE and MAE than Selective Learning, suggesting that Selective Learning's hard selection mechanism may mistakenly discard samples containing predictable structures. Under Drift, Gaussian, and Laplace perturbations, BKLoss still achieves the best results in most conditions.