- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies spectral distortions associated with inference-time SNR–t mismatch in diffusion-based forecasting and proposes Spectral-Shape SNR Correction (SSSC). As defined in the method and algorithm, SSSC estimates average target and forecast power spectra, constructs candidate frequency-dependent gains, and selects a correction using calibration-side CRPS and an abstention rule. The selected gain is applied once to final forecast samples, without retraining the backbone. The paper includes diagnostics of noise-prediction spectra, paired evaluations across diffusion forecasters, component ablations, scalar correction controls, and a comparison with repeated in-loop correction. The full results report six backbones across thirteen datasets.
Critical strengths
-
A simple correction with broad practical applicability. SSSC operates on forecast samples from frozen models and does not require modifying their training. This breadth provides useful evidence that output spectral correction can be beneficial beyond a single model.
-
Several relevant controls and transparent reporting of mixed outcomes. The paper includes scalar scaling and dispersion controls, as well as a comparison with stepwise correction. It also reports settings where CRPS and QICE move in different directions and explicitly limits what the theoretical analysis guarantees. These are helpful features for assessing a calibration method rather than relying only on favorable examples.
Critical weaknesses
-
The SNR–t diagnosis is not sufficiently connected to the final-output correction. The method uses average target and forecast spectra to select a gain applied after sampling. It does not directly measure or restore the SNR of intermediate states, a limitation acknowledged in the appendix. Appendix A gives a sufficient mechanism under an orthogonality assumption, rather than establishing that this mechanism explains the errors in the evaluated models. General output-scale or spectral errors could also benefit from the same filter. The distinction matters because the claimed contribution includes a diffusion-specific diagnosis, not only an effective post-processing rule.
-
The abstract describes held-out calibration, while the method and algorithm describe spectral estimation and selection using training data. It remains unclear whether these windows are disjoint from those used to train the backbone. This is consequential because in-sample forecast spectra and errors need not represent deployment behavior.
-
Section 4.3 reports global scaling and mean-preserving dispersion scaling for CSDI and TMDM on Electricity, which is a useful control. The stepwise comparison is also limited to two backbones on ETTh1. These results do not yet establish how consistently the frequency-dependent method adds value across the wider benchmark.
Questions for authors
-
Which empirical result distinguishes correction of SNR–t bias from generic output spectral calibration? Is there evidence that settings with a larger diagnosed mismatch are the ones that benefit from SSSC, or should the mechanistic claim be limited to a sufficient explanation rather than an established cause?
-
Are the spectral-estimation and selection windows disjoint from backbone training? Please give the temporal partitions and calibration-origin count. If training windows were reused, how do the results change under an independent temporal calibration block in a few representative settings?
-
Why does the selection rule accept the correction in the ETTm2–TimeDiff failure case? What were the calibration-side gain and abstention statistic there, and how did they change at test time? How should users interpret cases where CRPS improves but QICE deteriorates substantially?
-
Can the scalar calibration controls be evaluated on one additional dataset and a relatively strong baseline using the same calibration data and selection budget? Please also summarize the typical relative gain and its uncertainty across test origins, so that the practical conclusion does not depend primarily on the maximum improvement.
-
Please provide the actual , , , , , low-frequency cutoff, and spectral-band boundaries used in each setting, together with how they were chosen. Which TimeDiff implementation and configuration produced the reported forecasts, and are the exact selected corrections recoverable from the released configuration?
Response to Reviewer x8dS
Thank you for the detailed review. We added temporal separation, scalar controls, and failure analysis.
1. Mechanistic attribution
Real-model evidence cannot uniquely exclude generic output spectral calibration. The appendix provides a sufficient mechanism under explicit assumptions, not an established cause of all improvements. TMDM scalar controls support extra CRPS benefit from frequency information, but not unique attribution to SNR–t bias. We will clarify that SSSC acts once on final samples; diffusion diagnostics motivate it but do not measure or restore intermediate SNR.
2. Calibration independence
Original experiments reused training windows; the abstract's held-out claim requires correction. New TMDM estimation/selection/test counts are 64/64/128 (ETTh1) and 32/32/64 (Electricity), with 30 samples per origin. Electricity target periods:
Target windows crossing boundaries are excluded; inputs precede forecast origins. Separated Electricity CRPS is 0.449240 → 0.289827: 35.49% reduction [34.59%,36.59%]. Validation excludes optimization labels but contributed to early stopping. A stricter unused ETTh1 tail uses 64/64/128 origins: estimation 2018-02-21 to 2018-03-24 10:00; selection 2018-03-24 11:00 to 2018-04-24 21:00; evaluation 2018-04-24 22:00 to 2018-06-26 19:00. TMDM selects identity (0.587323 → 0.587323).
3. Original TimeDiff/ETTm2 failure
Calibration CRPS 0.378853 → 0.274413 improves 27.57%, exceeding the 3% acceptance margin. Reconstructed concentration statistic ρ=0.00118282 exceeds the source-default rejection threshold 7.5×10−5, so that rule would accept it. The original JSON does not record the threshold; we will identify its source.
DC gain is approximately 0.559; mean low/middle/high gains are 2.949/2.808/2.706. Test CRPS worsens 0.200843 → 0.557726 (+177.69%), while QICE×100 improves 13.995 → 11.811. Calibration acceptance cannot guarantee test benefit. Paired low-frequency protection suggests mean/low-frequency changes contribute, but missing full gains/origin indices prevent precise recovery of original test ρ or error attribution. We retain the failure. CRPS, point errors, QICE, and coverage should be reported together; one improved score does not establish overall calibration.
4. Scalar controls and uncertainty
We extended controls to TMDM/ETTh1 and retained Electricity. SSSC, global scaling, and mean-preserving dispersion scaling share selection windows, cached samples, nine candidates including identity, and acceptance margin; computation times are not matched.
Electricity results (SSSC uses separated calibration):
SSSC reduces CRPS relative to global/dispersion scaling by 21.97% [21.00%,23.11%]/6.91% [6.49%,7.32%], using each scalar score as denominator. Scalars have better QICE: this supports extra CRPS benefit, not superiority on all metrics. All methods select identity on ETTh1 (CRPS 0.483879). Median per-origin relative improvement is 36.47%/0% on Electricity/ETTh1. Intervals average per-origin scores over three inference seeds before 2000 moving-block bootstrap replicates. Frozen checkpoints exclude retraining variability.
5. Parameters and recoverability
Defaults: gain bounds 0.5/3.0; η∈{0.5,1.0}; protection cutoff fraction 0.25; acceptance margin 0.03; rejection threshold 7.5×10−5. For H=192, rFFT has 97 bins: protection [0,24), non-DC bands [1,24), [24,48), [48,97). For H=24: [0,3), [1,3), [3,6), [6,13). Right endpoints are excluded. These empirical defaults are retained after sensitivity analysis. Rejection uses DC and band-average powers, not summed high-frequency energy; we will give the formula.
TimeDiff uses the local CNN, x0 prediction, cosine schedule, 100 training steps and 20 native DPM sampling steps; recorded
nfe=100is not the sampling count. New runs preserve checkpoint checksums, configurations, candidate CRPS, selections, abstention reasons, and complete gains. Historical band-only records have limited recoverability.