- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose the SCOPE: model-agnostic time-series imputation for fixing incomplete historical inputs of frozen time-series foundation models (TSFM) as the main motivation. The key argument here is that pointwise accuracy may not ensure preservation of key temporal structures for downstream forecasting.
The proposed SCOPE consists of component-wise numerical initialization and structure-aware refinement. The component M3 will provide information for trend seasonality residual. Then M1 provides estimation of 6 structural targets based on the observed data. M2 will estimate positional importance structural leverage. M4 will do a bounded optimization to refine the initialized values. An unnecessary modification can be prevented by a gate that is based on calibration, with a fallback mechanism. Also, an optional M5 is to provide model-aware alignment.
The experiments are done with 27 datasets. For main TSFMs, 4. Also 9 combinations of missing rate and contiguous block length are tested. As a result, SCOPE shows a good average rank to classical/deep imputation counterparts. They also show a 6.97% decrease in downstream MAE from linear interpolation.
Ablation studies indicate that the component-wise initialization brings the most contribution to the improvement for forecasting. For structural projection, average 6-dim structural error is decreased by 5.49%. Also, a paired study is conducted for block length 50. With a prespecified 0.5% MAE range, average forecasting non-inferiority is shown.
Critical strengths
- Large empirical experiments across forecasting models and missingness settings.
27 datasets from different applications, 4 frozen TSFMs, 9 missingness settings.
SCOPE vs. both classical imputation and various deep baselines, with 2 budget levels.
Also direct comparison against native/default missing handling. Shows strong wins for 3 TSFMs, with the caveat of stronger native performance for Chronos-2.
Results by datasets. Module ablations. Structural faithfulness analysis. Sensitivity.
All this demonstrates that the numerical reconstruction has a practical role.
- The clear disentanglement of numerical recovery and structural refinement.
The paper very clearly separates the component-wise initialization and structural projection.
The authors clearly admits that the main forecasting improvement comes from the M3. The M1, M2, M4 add much lesser downstream improvement.
For structural refinement experiment, the end sequences are compared to clean historical references that are not used for optimization. Moreover, structural projection is constrained to be observation consistent and with bounded change with a fallback to original initialization.
Critically, the paper conducts formal non-inferiority analysis instead of taking a small forecasting difference (close to zero) as a statistically proven improvement.
This design and reporting helps to enhance the credibility.
Critical weaknesses
- The separate methodological and practical contribution of structural projection is not clear enough.
The key novelty that the paper proposes is an importance-aware structural projection. Which utilizes structural targets from observations, positional importance, and bounded optimization.
But the results show that most of the downstream forecasting improvement comes from component-wise initialization.
Dropping M3 alone increases the average forecasting MAE by 7.24%. While the drop in M1/M2/M4 causes much less change. More importantly, for direct pair-wise comparison between M3 and complete projected approach, the average MAE improvement is only 0.0198%. And 90% CI includes zero.
While the authors correctly point out that the result shows average forecasting non-inferiority/equivalence for their specified margin. But it does beg the question on the practical merit of introducing all these extra estimation/optimization components.
Even the stated 5.49% improvement in terms of structural fidelity is not trivial. But it's measured for the same structural dimensions which are used for the optimization. But compared to independently held-out clean references.
Further assessment (e.g. with separate metrics of temporal structure or downstream task-based evaluation) would help make the case that these benefits reflect an interesting behavior distinct from the proxy-optimized representation.
Interestingly, also residual autocorrelation is somewhat worse (and 19 out of 97 activated inputs have lower clean-ref structure fidelity) suggesting some non-uniform reliability of the refinement.
Last, to better establish the need of downstream-aware imputation, it'd be helpful to more explicitly contrast to existing task-aware imputation approaches (e.g. Wang et al. NeurIPS 2024).
- Still quite unknown are the practical efficiency, baseline fairness and robustness of the refinement.
The compute analysis omits the precomputed SCOPE gating calibration, but includes per-input compute budget for the deep baselines. With Gaussian process fitting, sensitivity estimation and iterative optimization included, the reported runtimes don't give us a complete picture of the deployment cost of your proposed framework.
Further, the baseline comparison to deep imputation models depends on training/tuning choices. Two budget for training does give us a nice piece of evidence; but is insufficient to support that these baselines have achieved representative performance for the intended deployment environment. More explanation on model selection. Maybe other well-known baselines like SAITS?
The forecast non-inferiority for structural projection is shown for block 50 only, for fixed imputations and gates. The range of equivalence for the subset of activated inputs is close to the pre-defined margin. The supplemental experiments reveal non-negligible sensitivity wrt imputation seeds.
So this does not yet show robust forecast preservation for the larger missing blocks used in the main benchmark.
And focusing on artificially created sequential holes in the retained historical data. To make this more useful, it should also include missing block assessments in the vicinity of the forecast origination point, especially endpoint missingness (unusable future observations for interpolation).
Questions for authors
Q1. Independence of structural projection.
Are there any M3 vs SCOPE comparisons that you can make for all 9 missingness scenarios (e.g., multiple indep. imputation seeds)?
Are you able to show improvements to independent structural characteristics (e.g., season phase, trend turning points, other downstream tasks)? In addition to the 6 optimized structural parameters by M4.
What makes you sure that this structural improvement is practically useful, given that downstream forecasting performance is statistically identical as the M3 initialization?
This is crucial to understanding the methodological impact of your paper.
Q2. Verifying importance of position/leveraging structure.
How well do your model independent importance of position scores agree with the sensitivity of forecasting for the considered TSFMs?
Could you evaluate the proposed importance distributions vs straightforward recency weights, random position weights, uniform reg. under otherwise the same projection goals?
With such small M2 impact on forecasting it is critical to see if the proposed importance / leverage ration has a consistent benefit over simpler approaches.
Q3. Complete computational load and base training settings.
Could you provide the full computational cost for GP fitting, two block gate calibration, structural sensitivity analysis, projection optimization (end-to-end)?
For the deep imputation baselines, how do you choose the training budget, hyper-parameters?
Is their original training preserved? How would this change with more established imputation techniques such as SAITS?
If the calibration outputs are reusable over multiple inference calls, then amortized cost analysis would also be useful.
Q4. Robustness of the method in challenging missingness patterns:
Is the forecast non-inferiority property of structural projection still holding when the block sizes get longer (e.g., 100 and 150) and the seeds of imputation change?
How does SCOPE fare if the entire missing block is contiguous just before the origin of the forecast (i.e. we don't have any right-side observations to interpolate?)
This is key to understanding how useful the method can be in sensor outage scenarios or other patterns relevant to deployment.
4 comments
Response to Reviewer Cqb8(Part 2 of 4)
To Q2 — positional weighting. At 20% missingness and length 100 we perturbed 12 missing positions by observed standard deviations in each direction and measured forecast sensitivity by central differences. Median within-series Spearman correlations range from -0.08 to +0.23 for the use-importance score and +0.26 to +0.51 for recency. The composite cost is often constant (four defined correlations per model). In 972 forced-projection pairs with the target, initialization, gap basis and solver fixed, the original weighting yields gains of -0.53%, -0.38% and -0.40% over recency, shuffled recency and uniform weighting. All 95% intervals span zero. This comparison tests the combined weights, and a separate leverage ablation is still needed (Appendix G.5, Tables 44-45).
For leverage alone, the median correlations with forecast sensitivity are +0.01 to +0.15 across the four backbones. Recency has a higher median correlation than the use-importance score for each backbone (Table 44).
Response to Reviewer Cqb8(Part 3 of 4)
To Q3 (W2a-W2b) — full costs and SAITS. Across seven settings, SCOPE's end-to-end median is 1.163-3.142 s, including calibration, completion and forecasting. For 1,536-step histories and 50-step gaps, calibration medians for GP fitting, sampling, M2 and M4 are 0.766, 0.025, 0.069 and 0.558 s. The median paired overhead over M3 is 2.773 s. With reusable calibration, per-call medians are 1.198 s over 10 calls and 1.084 s over 100 (Appendix G.6, Tables 46-47).
The original deep baselines use fixed univariate PyPOTS adapters, with 96-step windows, stride 48 and batch size up to 32. These are benchmark configurations. The larger budget triples each epoch cap with architecture and other settings fixed (Table 48). The exploratory Sundial-SAITS comparison uses the same 126 pairs at both caps, across 42 datasets at the middle origin and 20% missingness (Appendix G.7, Table 49). Both confidence intervals span zero.
Table C1. SCOPE forecast gain over SAITS on Sundial (revision Table 49).
| Epoch cap | Gain [95% CI] | Wins / losses |
|---|---|---|
| 10 | +2.76% [-2.93%, +8.66%] | 72 / 54 |
| 30 | +2.49% [-2.65%, +7.86%] | 72 / 54 |
Response to Reviewer Cqb8(Part 4 of 4)
To Q4 (W2c-W2d) — long and terminal gaps. At the 0.5% margin, only the 50-step internal gap meets average non-inferiority in the three-origin aggregates (90% CI [-0.40%, +0.25%]). At that length, all three middle-origin GP seeds meet it when pooled across rates. Internal gaps of 100 and 150 steps have lower endpoints of -0.69% and -0.86%. For terminal gaps, the imputer sees only the left history. Lower endpoints at lengths 50, 100 and 150 are -1.90%, -2.24% and -0.54%, respectively, so none of the terminal aggregates meets the margin. Historical terminal-point reconstruction still improves (Section 4.6, Table 4, Appendix G.2-G.3, Tables 37-39 and 41).
Response to Reviewer Cqb8(Part 1 of 4)
Overall response. Thank you for the detailed questions about the projection and its use in practice. We now report all nine missingness settings, with separate seed checks and measurements outside the projection objective. The clearest improvement is in recovery of hidden historical values. At the 0.5% margin, only 50-step internal gaps meet average forecast non-inferiority in the three-origin aggregates. We also test simpler position weights, report the full cost with and without calibration reuse, and compare SAITS at both epoch caps on the same inputs.
To Q1 (W1) — nine settings and independent measures. The nine rate-length settings use 2,916 forecast pairs from 27 datasets and four frozen backbones at three origins. The pooled gain over M3 is -0.22% (95% CI [-0.70%, +0.14%]). At the middle origin each setting has 108 matched forecasts per GP seed (0, 1, 2), with masks and gates held fixed. All 27 seed estimates and intervals are reported (Appendix G.2, Tables 36-38).
The original six-dimensional structural error fell 5.49%, with 19 of 97 activated inputs worsening and a slight deterioration in residual autocorrelation (Section 4.4, Table 3). Independent tests show lower masked-point MAE for internal and terminal gaps (reductions of 0.0056 and 0.0128 observed standard deviations, Holm p=0.0118 and p=0.0323). Internal phase and turning-point intervals span zero, and NAB detection shows no consistent improvement (Appendix G.3, Tables 41-42). This supports a use when recovered historical values matter. Section 2 distinguishes the pseudo-gap calibration signal from the downstream gain estimates of Wang et al. (2024b).