- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The motivation of the work is that static time series benchmarks have narrow test distributions that can lead to overfitting / benchmark-specific tuning, and that single aggregation scores in these benchmarks cannot characterize model capabilities across specific temporal structures, and that they are not useful (enough) for model selection.
With that motivation, the authors propose CaFE - capability focused evaluation which enriches static benchmarks with structural diversity (through time series features), and further goes beyond a single aggregate score with a multi-dimensional assessment. This is done by defining structural features, profiling the benchmark and then augmenting it with samples along specific features.
They evaluate time series models on their proposed benchmark extension to give each model a per-feature capability profile, which allows quantifying the failure modes of each model better.
Critical strengths
I believe the two key points in the motivation of the paper are strong: That existing benchmarks (GIFT-EVAL, fev bench) are narrow and that it is really hard to make a model selection decision based on the aggregate scores on these benchmarks
The proposed high level idea is also strong - to profile benchmarks and give more dimensions of evaluation.
I do not think these constitute critical strengths that support acceptance, however. More details below.
Critical weaknesses
I summarize the critical weaknesses first, and then paste the raw review I wrote section-by-section as I read the paper.
SUMMARY
- Sections 3 and 4 are extremely hard to follow. They lack crucial low-level implementation details, clear rationale, and definitions for key terms. The writing style and phrasing make it strongly look like heavy AI copy-editing. This makes text impossible to understand.
- Core tables and figures are not useful enough (like Table 1 lacking citations, row/column definitions, or Table 2 being referenced in the main text but hidden in the appendix) or poorly explained. The results sections just dump low-level stats and correlations without summarizing what they actually mean. And since the benchmark section is not clearly written, I'm not able to get anything concrete from the results section.
- The model selection experiment (4.3) either is not written accurately or feels obvious based on what’s written. Other results sections - are not written to explain to a human reader in my opinion, they seem like LLM-written parts of the paper that were never checked by a human before submitting.
RAW REVIEW
Intro
- Table 2 referred to in the main text but it’s in the appendix, but it seems like a critical table to convey the points in the intro. Please either bring it up or explain why it is in the appendix.
Related Work
-
Table 1 needs to cite specific benchmarks to compare to, instead of grouping them and not citing any
- It is unclear what any of the rows actually mean as they don't refer to standardized terms
- Same with the columns - it is not explained what each of it actually means
-
The paper relies heavily on extracting structural time series features, but does not cites papers on standard feature extraction frameworks like tsfeatures [1] or catch22 [2] which are heavily used in research. It's unclear how their proposed features compare to or differ from these existing sets. They however cite papers like Kang et al (such as GRATIS), so not all relevant literature is missed.
[1] Hyndman, R., Kang, Y., Montero-Manso, P., O'Hara-Wild, M., Talagala, T., Wang, E., & Yang, Y. (2026). tsfeatures: Time Series Feature Extraction (R package version 1.2.0.0). https://pkg.robjhyndman.com/tsfeatures/ [2] Lubba, C. H., Sethi, S. S., Knaute, P., Schultz, S. R., Fulcher, B. D., & Jones, N. S. (2019). catch22: CAnonical Time-series CHaracteristics: CH Lubba et al. Data mining and knowledge discovery, 33(6), 1821-1852.
Section 3
- Figure 2 is useful but very cryptic; I did not fully understand it
- E.g. Family-Level Sifting, Salience coefficient - unsure what that means
- “The second adds intervention-response metrics that measure how faithfully a forecast tracks each induced change”
- I don’t understand what this means. "forecast tracks each induced change" - what does "forecast track" mean here - what is a forecast supposed to track? Do you mean how well a model produces a forecast of time series that has been changed that way? Further, I don't know what "intervention response metrics" means - it is not a standard term in the literature. Do you simply mean a forecasting metric on the intervened time series?
- The structural features make sense at a high level, but I know that the low-level implementation of each can vary widely. There are no standardized/accepted definitions of each. The main text (Sec 3.1) does not provide the low-level implementation / assumptions made for each; does not point to a relevant appendix section as well.
- “removing edge-of-distribution windows so that the refreshed set stays within the benchmark’s typical structural range”
- I am unable to understand the rationale behind this. Ideally one would want to keep the long tail of forecasting windows or at least report them separately
- The augmentation part of the framework does not make sense to me / it is extremely unclear from the text (Sec 3.2) what it means:
- Do you take a single time series and generate samples that change one structural feature?
- What guarantees that this generated time series is realistic / scores a realistic property? It would be better to simply filter out the existing time series based on their properties and produce a better “view” of the benchmark. This, in my opinion, would have been a solid contribution instead of the proposed augmentation approach.
- Section 3.3: Inspection and Sifting: very much unclear how this is done. Both of these steps seem like non-trivial steps
- Sentences like “Any cross-feature summary is a secondary equal-weight macro over direction-corrected feature-local ranks and does not pool raw values or define a cross-feature capability scale” are extremely cryptic (use terms not defined before in the text) and make me suspect heavy AI copy-editing of the text, which renders the text unreadable.
- I do not have enough information to really understand the figures. The captions do not summarize / help the reader enough in order to understand them.
- The results section is completely unreadable - the text does not summarize key results and is presented very verbosely
- E.g. (Reference MASE and Probe MASE are rank correlated at 0.943, 0.700, 0.900, and 0.657; Appendix C).
- “At the aggregate level the two rankings are largely consistent: the Spearman correlation between Probe-MASE rank and NRMSE rank is 0.029, 0.800, 0.900, and 0.771 on the four suites, and the refreshed set does not distort accuracy rankings”
- It is really unclear what any of this refers to at the low level. Seems like an AI dump of the results without any understanding of what’s a significant result
- Sec 4.4, 4.5, 4.6 are completely unreadable as well. Everything there onwards I don’t understand.
Questions for authors
Recommendations for the Authors
- I recommend the authors carefully audit the method and results texts so it’s clear to them what it means, and they fully understand each sentence. The goal is not to simply use an AI to build a benchmark extension and get the results, but to convey them meaningfully to other readers.
- The results section should be refreshed completely in my opinion, first introducing what measures / metrics make sense for such tasks; then gradually explaining the results. Limit the number of results you want the readers to takeaway in the main text. Fewer, well conveyed results - are much better in my opinion than dumping every low level statistic and conclusion in the main text, expecting readers to understand everything.
- More specifically, if it is a paper where a single metric doesn’t summarize everything, then the authors should make an even better effort to convey the results and provide useful takeaways.
- The figures and tables must make sense, standalone. This means, any jargon or term introduced in the figures must be explained in the caption. For figures/tables with results, the caption should “summarize” the results.
Questions:
- What are the 3-4 key takeaways that you would like readers to take from the results? These takeaways could obscure low level details, but are important for readers like me who are looking for such takeaways (instead of the correlation score between each model in each metric)
- Please clearly motivate the rationale behind each decision made in developing CaFE. E.g. why these features were chosen, why not others (tsfeatures)
- One algorithm that explains the entire procedure would really help.
2 comments
Official Response to Reviewer X1sw [2/2]
W3: About the model-selection experiment.
We have rewritten the protocol in Section 4.3, with details in Appendix C.3. Within each benchmark, we hold out the datasets of one source family, recommend the Top-3 models from the other families by aggregate Reference MASE or by SF-specific Probe MASE or Forecast-Change NRMSE, and count a hit when the recommendation contains the best-MASE model on held-out, unaugmented windows that exhibit the SF. The result is not trivial: SF-specific scores are computed on generated samples from other source families, yet they are tested on real data that never entered the ranking. Over 211 items, the Top-3 hit rate rises from 56.4% to 61.1% and 62.1% (vune:Q1(i)).
W6: About tsfeatures and catch22.
Section 2 now cites tsfeatures and catch22 and states the difference: such descriptors summarize an observed series, whereas an SF is a controlled change whose salience can be varied, which CaFE applies to the source task as a paired change.
W7: About Figure 2.
Figure 2's caption explains each stage and the salience coefficient λ. Section 3.2 defines the depicted sample family, and Appendix B.4 traces one SF1 family through the procedure.
W8: About the meaning of intervention-response metrics.
Intervention-response metrics do not simply recompute an ordinary forecast error on the intervened series. They compare the forecast for a generated sample with the forecast for its Reference and ask whether this forecast change matches the change we induced. Section 3.4 gives the NRMSE formula and how to read its values: 0 for an exact response, 1 for no response, and above 1 for worse than no response. Appendix B.4 gives a numerical example.
W9: About low-level implementation of each SF.
Section 3.1 now points to Appendices A.4–A.11 for each SF’s source evidence, transformation formula, parameter range, and checks, summarized in Table 4.
W10: About removing edge-of-distribution windows.
The removed windows are those in which the selected feature is only weakly expressed. Strengthening a structure that is barely present would produce an unreliable change, so excluding them yields References with clear evidence of the feature (Section 3.2). These windows remain References for the original task and for other SFs. We will report results on the excluded long-tail windows in the appendix of the revised version.
W11: About what augmentation does and whether it is realistic.
Yes, augmentation fixes one Reference and one SF and generates a family of K = 3 samples at randomly drawn salience coefficients, all sharing the same Reference and forecasting task (Section 3.2; Appendix A.2). For realism, see (C8qM:Q1); for filtering existing series instead of augmentation, see (vune:Q4) and (C8qM:W2).
W12: About how Inspection and Sifting are done.
Section 3.3 explains what each step checks, with feature-specific details in Appendix B.1 and Table 5:
- Structural enhancement: Inspection re-extracts the structural parameters from each augmented sample to verify that the realized gain falls within the allowed range for the selected SF.
- Source properties: it checks that the properties required by the selected transformation, such as fitted residuals, seasonal frequencies, or event support, are retained, along with observation masks and applicable nonnegativity constraints.
- Measurable forecast-horizon change: it checks that the induced change is large enough to measure the model's response reliably.
- Sifting: a family is admitted only if all K samples pass; otherwise it is rejected and the failed check is recorded.
Inspection uses no forecasting model, and Appendix B.4 shows each check on a concrete example.
W13: About the cryptic cross-feature summary sentence.
The sentence meant that models are ranked within each SF and these ranks are then averaged, without pooling raw values across SFs. We have removed it; Section 4.2 now reports per-SF ranks directly, and any cross-SF summary is stated explicitly as a mean rank, such as Toto 2.0's mean Probe MASE rank falling from 2.125 on Short to 5.375 on Long.
W15: About verbose results and the quoted rank-correlation sentence.
We have removed the quoted sentence. Section 4.2 now states its findings directly, centered on the complementarity of accuracy and response fidelity on SF1, and the remaining results are summarized in (X1sw:Q1).
W16: About Sections 4.4–4.6.
We have reorganized the experimental content of Sections 4.4–4.6. Each subsection now opens with the question it answers and closes with its finding. The main findings are summarized in our response to Q1 above (X1sw:Q1).
Official Response to Reviewer X1sw [1/2]
Thank you for the thorough review and detailed suggestions.
Section 4 now emphasizes four findings:
These conclusions are stable under benchmark refresh (Section 4.5).
Each decision serves a specific measurement need:
Algorithm 1 (Section 3.4) now traces the full procedure from References through family generation, Inspection, and Sifting to scoring. Appendices A and B give the SF constructions and checks, and Appendix B.4 walks one SF1 family through them.
We have substantially rewritten both sections and audited them sentence by sentence. Section 3 defines the key terms and adds Algorithm 1 and a worked SF1 example in Appendix B.4 (C8qM:W1). Section 4 is organized around one finding per subsection (X1sw:Q1). We removed or rewrote the quoted sentences; the specific changes are detailed below.
We revised and followed this structure: Section 3.4 explains the metrics; Section 4 previews the five analyses; Sections 4.2–4.6 present their findings and supporting statistics. Detailed numerical tables are in Appendix C.
Section 3.4 now states that Probe MASE, Forecast-Change NRMSE, and the Auxiliary-Input Gap measure accuracy, response fidelity, and input utility, so we report them side by side rather than as one score. Sections 4.2, 4.4, and 4.6 then show how accuracy and response complement each other on SF1, how NRMSE and the Gap relate, and how training on each metric affects the other; the takeaways are summarized in (X1sw:Q1).
We revised the captions to explain how to read each figure and what it tells us. Figure 2 clarifies how the stages turn a Reference into a family of augmented samples and then evaluate model forecasts. Figures 4 and 5 distinguish the rankings for forecast accuracy and response fidelity. Figure 6 explains that the radar charts show Forecast-Change NRMSE, with lower values farther out, and that models should be compared within each SF because the axes have different scales.
Figure 7 highlights the association between lower NRMSE and greater benefit from auxiliary inputs. Table 2 tests model recommendations on held-out real data and shows the advantage of selecting models by the relevant structural feature. Figure 8 shows that joint fine-tuning can improve both accuracy and response fidelity on held-out series.
Table 1 now names and cites representative benchmarks, including M4/M5, TFB, and BasicTS+, and its caption defines the four compared properties. The Introduction’s mistaken reference to Table 2 now points to Figure 3, which summarizes the eight SFs; detailed definitions remain in Appendix A, Table 4. The results are restructured as described in (X1sw:R2).