- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EvoFIF, a framework for forecast-oriented time series imputation (FOTSI) that shifts the optimization object from a scenario-specific model to a reusable configuration of imputation inductive biases. It extracts an initial pool of explicit biases from seven classical imputation algorithms with an LLM, evolves the pool with utility-weighted Thompson Sampling and LLM-based mutation and crossover, and verifies candidates by downstream forecasting loss, an objective-alignment penalty, and computational cost. The paper formalizes FOTSI and claims that, if utility requires cross-scenario reuse without retraining, the FOTSI problem is equivalent to searching over bias configurations. Experiments use nine multivariate datasets, four missingness patterns, four forecasters, and seven imputation baselines. EvoFIF attains the lowest downstream loss on 21 of 36 dataset-pattern combinations, ranks in the top two on 33, imputes orders of magnitude faster than the baselines, and keeps one configuration across unseen forecasters and horizons without retraining.
Critical strengths
- The empirical scope is substantial. Nine datasets, four missingness patterns, four forecasters, and seven recent baselines. The result that EvoFIF is best on 21 of 36 combinations and top-two on 33 is a meaningful signal.
- The ablations are informative. Both the initial pool and the utility-weighted sampling carry most of the gain, and the utility terms contribute a smaller but consistent improvement. This supports the design choices.
- The case study adds interpretability. Reading the discovered configuration line by line and tracing it back to the initial pool is a feature that model-centric methods cannot offer.
Critical weaknesses
- No FOTSI baseline is compared. The paper's central claim is that EvoFIF is better than existing FOTSI frameworks, but no FOTSI method is in the comparison.
- The "across datasets" transfer claim is unsupported. §4.4 states that the held-out axes are the forecaster and the horizon, and it does not list the dataset. The nine datasets in the main comparison were likely all used in the search. The abstract and §5 nevertheless claim transfer across datasets, patterns, forecasters, and horizons.
- The search cost is not reported. The efficiency comparison excludes the one-off search. The number of LLM calls, tokens, and wall-clock time are not given. Without them, the amortization argument is incomplete and the efficiency claim is only about inference.
- The w/o Utility ablation is underspecified. Equation 9 removes the downstream-utility and cost terms. What remains is the conflict penalty. It is unclear what the variant optimizes. If it falls back to reconstruction error, the variant is no longer forecast-oriented, and the result that it is the least damaging variant needs explanation.
- The Table 2 outlier needs explanation. HyFAD on DLinear is 5.645 against 0.388 for EvoFIF, an order of magnitude above every other entry. The text does not discuss whether this is a method failure, a tuning issue, or a scaling artifact. A baseline that fails this badly weakens the comparison.
Questions for authors
see weakness
3 comments
To W4, the variant does fall back to reconstruction error, and Section 4.3 now says so in one sentence. It removes the downstream prediction term and the conflict penalty from the verification and keeps the cost term, so a candidate is ranked by its imputation error and the forecaster is never called during the search; what it searches is a purely reconstruction-oriented configuration over the same pool.
The relatively smaller average degradation of w/o Utility suggests that the initial pool and the evolutionary search already provide a reasonably strong starting point. Nevertheless, its substantial degradation under MCAR indicates that reconstruction-oriented selection cannot consistently preserve downstream forecasting utility. This highlights the role of the downstream objective in resolving cases where imputation accuracy and forecasting performance are misaligned. The distribution supports this reading, since only two of the 36 settings exceed 50 percent, weather and ETTm1 under MCAR, against four for each of the other two variants, and those two are precisely the settings in which a more accurate imputation and a worse forecast disagree most.
To W5, the HyFAD entry under DLinear is a method failure on one series rather than a tuning artefact. Table R4 gives the column per dataset.
Table R4. Per-dataset downstream loss under DLinear, averaged over the four missingness patterns. ILI is national_illness.
| Method | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Elec. | Exch. | ILI | Traffic | Weather | Mean | Mean w/o ILI |
|---|---|---|---|---|---|---|---|---|---|---|---|
| EvoFIF | 0.311 | 0.157 | 0.294 | 0.067 | 0.382 | 0.149 | 1.781 | 0.290 | 0.062 | 0.388 | 0.214 |
| SAITS | 0.358 | 0.167 | 0.359 | 0.093 | 0.397 | 0.151 | 2.060 | 0.298 | 0.082 | 0.441 | 0.238 |
| FIM | 0.446 | 0.196 | 0.431 | 0.142 | 0.527 | 0.186 | 2.246 | 0.368 | 0.111 | 0.517 | 0.301 |
| HyFAD | 0.451 | 0.211 | 0.484 | 0.160 | 0.526 | 0.342 | 48.138 | 0.371 | 0.122 | 5.645 | 0.333 |
National illness has 966 steps over 7 variables, the shortest and sparsest benchmark of the nine, and every method loses an order of magnitude more on it than on the others, from 1.781 for EvoFIF to 48.138 for HyFAD. A diffusion generator needs a long history to constrain the amplitude of its samples, so it is the method most exposed to a series this short, and the same weakness appears elsewhere, since HyFAD is the worst method on eight of the nine datasets. All methods, EvoFIF included, were run with their official defaults on the same GPU and the same budget, with no per-method hyperparameter search, so there is no setting tuned in our favour, and dropping national illness from the column leaves the ranking unchanged, 0.214 for EvoFIF, 0.238 for SAITS, 0.301 for FIM and 0.333 for HyFAD. Section 4.4 now explains this in the body of the paper.
The revised PDF marks these changes in the margin, labelled 8CCE.W1 to 8CCE.W5.
We would like to gently follow up on our revised manuscript. We sincerely appreciate your insightful feedback on the same-objective comparison, the scope of the reuse claim, the cost of the search, the w/o Utility ablation and the HyFAD outlier, all of which motivated additional experiments and a substantial revision of the paper.
Regarding your main concern, we have completed the comparison you asked for, under the same forecasting objective:
- Direct comparison: forecasting-guided selection applies each of the seven classical algorithms that seed our initial pool and keeps the one with the lowest forecasting error on the validation window, run once for all settings and once per missingness pattern.
- Results: EvoFIF attains the lower loss in 34 of the 36 settings against the first reference and in 32 against the second, by 9.6 and 3.1 percent, in 0.085 s against 12.6 s and 5.4 s. A single strong algorithm therefore does not reproduce the gain, which comes from evolving and composing the inductive biases.
Beyond your main concern, we have also made the following revisions:
- Scope of the reuse claim: all nine datasets take part in the search on purpose, since they span minute-level loads, sensor streams of 321 and 862 channels and a 966-step count series, so the configuration is selected against heterogeneous distributions rather than one of them. What is held out, the forecaster and the horizon, is then served without retraining.
- Cost: the whole search took 10 h 25 min 08 s, 0.8M and 4.5M tokens and CNY 32, about 17 minutes and CNY 0.9 per scenario, paid once, while deployment needs 0.085 s where a single baseline imputation needs 23 to 221 s.
- w/o Utility: the variant falls back to reconstruction and degrades 28.5 percent under MCAR against 4.9, 2.6 and 10.2 elsewhere, with only 2 of the 36 settings above 50 percent, so the downstream objective matters exactly where imputation accuracy and forecasting disagree.
- HyFAD outlier: the 5.645 of the DLinear column comes from one series, national_illness, 48.1 against 0.333 without it, and that method is the worst on eight of the nine datasets, so the entry is a method failure on a 966-step series rather than a tuning artefact, and the column ranking is unchanged.
We would be very grateful if you could take another look at our revised manuscript and reconsider your assessment in light of these additional experiments and revisions. We would also sincerely welcome any further comments or guidance you may have.
Thank you very much for your time and consideration.
Thank you for the careful reading and for the constructive report. We answer the five points below.
To W1, we agree that a FOTSI method should be in the comparison, and the difficulty is that none of them runs in our setting without becoming a different method: they target variable subset forecasting under a fixed forecaster and fit their imputation module jointly with it, so the downstream loss has to be back-propagated into the imputer, while our evaluation spans four forecasters, two of which admit no gradient for that joint fit, and every scenario would need its own fit.
What we add instead is forecasting-guided selection (FGS), which optimizes the same quantity as EvoFIF without any search. Its candidate pool is the seven classical algorithms that seed our initial pool, each imputes the series of a scenario, the imputed series pass through the same forecaster, and the algorithm with the lowest forecasting MSE on the validation window is kept, the test window never entering the selection. We run it once for all 36 settings (FGS-global), which keeps the reusability EvoFIF has, and once per missingness pattern (FGS-pattern), which is the best a single classical algorithm can provide.
Table R1. Forecasting-guided selection against EvoFIF over the 36 settings.
EvoFIF attains the lower loss in 34 of 36 settings against the reusability-preserving reference and in 32 of 36 against the per-pattern one, with the widest margin under MNAR and the narrowest under BLACKOUT, and it imputes a scenario 63 to 148 times faster than the algorithm the reference selects. Since FGS-pattern already takes the best of the seven by the downstream loss for each pattern, what EvoFIF adds over the classical methods is the composition and the evolution of inductive biases rather than the choice of one of them. Section 4.1 defines the reference and Appendix F.5 reports every setting.
To W2, you are right that the nine datasets take part in the search, and that is a deliberate design rather than an oversight. They span hourly and minute-level transformer loads, sensor streams of 321 and 862 channels, meteorological and financial series, and a spiky count series of 966 steps, so the configuration is selected against different sampling frequencies, channel counts and statistical structures instead of one distribution. What the evaluation holds out is the forecaster and the horizon, as Section 4.4 states, and neither of them takes part in the search. The abstract and Section 5 state the result as reuse across scenarios, one configuration applied to every dataset, pattern, forecaster and horizon of the evaluation without retraining, and since the discovered object carries no fitted parameter, extending it beyond this coverage needs no retraining of an imputation model as long as the new series falls within what the bias library can express and combine, the condition Section 5 already states.
To W3, the cost of the search is now reported in Appendix F.3, and Section 4.2 points to it.
Table R2. Cost of the search behind the reported configuration.
The search is one-off and returns a configuration rather than a fitted model, so none of these numbers is paid again when the configuration meets a dataset, forecaster or horizon it was not selected on. Deployment imputes a scenario in 0.085 s with no training, against 23 to 221 s for a single baseline imputation, and amortised over the 36 settings the search is about 17 minutes and CNY 0.9 per scenario. The efficiency claim therefore survives the amortisation, and the marginal cost of a further scenario stays at zero training plus 0.085 s.