- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how imputation errors propagate through normalization to alter observed-value representations, derives error bounds, and examines their effects on frozen time series foundation models.
Critical strengths
-
Provides theoretical bounds with explicit assumptions and a tightness proof.
-
Extensive controls, including fixed-statistics and coordinate interventions, clearly distinguish forecast displacement from forecast error.
Critical weaknesses
-
Most amplification evidence concerns constant or low-variance segments under artificial missingness; applicability to natural missingness remains unverified.
-
Learned imputers are trained on the current history for only 3–10 epochs, potentially affecting method rankings and quantitative conclusions about selection regret.
Questions for authors
-
Does the mismatch between reconstruction and forecasting rankings persist with larger training budgets or pretrained imputers?
-
How frequently does amplification occur, and how does it affect forecast error, under natural missingness and native missing-value handling?
2 comments
Response to Reviewer yN6q (Part 2 of 2)
2. Small perturbations can produce large responses at natural gaps, while actual forecast error can increase or decrease.
We retained the original gaps in household-power and Beijing air-quality data and used 512-step histories to forecast the next 128 steps. After imputation, we perturbed only missing values in the first 32-point segment containing a gap. The perturbation RMS was times the observed-history standard deviation. A large response was recorded when forecast RMS displacement, averaged over three directions, exceeded 1% of that standard deviation. The table reports event frequencies among gap-containing configurations:
| Data | Gap-containing configurations | TimesFM events | Toto events |
|---|---|---|---|
| Household power | 27 | 2 (7.41%) | 2 (7.41%) |
| Beijing, single station | 205 | 1 (0.49%) | 1 (0.49%) |
| Beijing, multiple stations, evaluated per channel | 12,587 | 110 (0.87%) | 107 (0.85%) |
Perturbation amplification also occurs at natural gaps and is concentrated in statistical segments that become constant after imputation. Across these gap-containing configurations, the large-response frequencies are 0.88% for TimesFM and 0.86% for Toto. The results connect sensitivity to the dispersion within the normalization support and identify constant segments as a practical condition to inspect.
Forecast changes can also reduce error. Among TimesFM's 113 large responses, RMSE increased in 50 and decreased in 63; both directions were also observed for Toto. The magnitude of a forecast change measures sensitivity, while forecast quality is evaluated against the observed future values.
The effect of observed-position representations also occurs with positive observed variance. For SAITS, we examined 775 histories whose first statistical support containing an actual imputation error has . After equal weighting across data sources, replacing only the observed representations changed RMSE by more than 1%, in either direction, in 79.15% of cases for TimesFM and 72.92% for Toto. Actual imputation errors can therefore affect forecasting through these representations in positive- supports as well.
For native missing-value handling, we compared Toto's native mask input with linear imputation using windows whose last 32 steps were observed in all input channels, as required by the interface. The native path directly receives the missingness mask. Its pooled RMSE was 0.21% lower on household power and 1.72% higher on the single-station Beijing data. These processing options should therefore also be selected using historical performance on the target forecasting task.
We have revised the manuscript in response to your comments. Revised text is marked in blue, with purple margin labels identifying changes addressing your questions.
Response to Reviewer yN6q (Part 1 of 2)
Thank you for identifying training budget and natural missingness as the two issues to test further. We increased training budgets and added a pretrained imputer and controls on naturally occurring gaps.
1. More training improves reconstruction, but reconstruction and forecasting rankings remain different.
On the original 19 series, we increased SAITS's training cap from 10 epochs to 50 and 200, and the other four learned methods' caps from 3 epochs to 15 and 60. All comparisons use 30% block missingness and the original 720-step forecast horizon. Each budget selects its checkpoint by validation error using only the visible history.
The table compares the 20× and 1× budgets. Reconstruction reductions are calculated within each series and then summarized by their median. The 38 forecasting conditions consist of 19 series and two forecasters.
Increasing training improves imputation quality: all five methods have lower seed-averaged reconstruction MAE and RMSE on all 19 series. Forecasting also improves in most conditions, but some conditions do not improve despite the lower reconstruction error. Improvements in reconstruction and forecasting therefore do not correspond one-to-one.
We also added pretrained MOMENT to the highest-budget candidate set. The following retrospective comparison uses all evaluation windows. Here, ρ is the Spearman correlation between reconstruction and forecasting rankings, and excess RMSE is the additional forecast error of the reconstruction-selected method relative to the best forecasting candidate in the same set.
At 20× budget and after adding MOMENT, median excess RMSE remains 2.41% and 2.48%, respectively. Ranking differences also occur among the methods requiring no training. Thus, increasing the training budget and adding a pretrained method do not eliminate the ranking difference in this evaluation. Improved reconstruction still needs to be assessed against the target forecasting task.
Selecting by historical forecasting performance improves average performance on later data. We selected an imputer on earlier windows and evaluated it on later windows. At 20× budget, reconstruction-based selection produced 2.83% higher later RMSE on average than forecast-based selection. Historical forecast error therefore provides a useful basis for choosing an imputer.