acceptodds
Under review as a conference paper at ICLR 2027

Skill4TSF: Contrastive Skill Distillation for Time Series Forecasting Agents

Abstract

Time series forecasting exhibits substantial heterogeneity across datasets, forecasting horizons, and evolving temporal regimes, making it difficult for any single forecasting model or fixed pipeline to perform reliably in all settings. We propose Skill4TSF, a skill-based forecasting agent that learns reusable, condition-specific strategies for model selection and forecast refinement. Our key component, Contrastive Skill Distillation (CSD), derives skills from matched comparisons among heterogeneous forecasters. At the model level, CSD identifies which forecaster is preferable under particular time series characteristics using statistically supported performance differences. At the strategy level, it analyzes recurrent residual patterns on hard forecasting windows, proposes targeted interventions, including adaptive preprocessing, component-wise routing, and bias correction, and retains only those validated by paired statistical tests. Each skill explicitly represents its applicability conditions, recommended strategy, empirical evidence, and explanatory rationale. At inference time, Skill4TSF combines feature-based filtering with reasoning-based applicability assessment to retrieve and compose compatible skills for each input window. Experiments on eight real-world benchmarks show that Skill4TSF achieves the lowest average MSE on seven datasets and the lowest average MAE on six, with results averaged over four forecasting horizons. Ablations support the contributions of both distillation levels, while online adaptation and cross-backbone transfer experiments further examine the utility of learned skills. These results support evidence-grounded skill learning as a practical approach to adaptive time series forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series forecasting, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HZrvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Skill4TSF, a skill-based agentic framework for time series forecasting. There are two categories of skills: model-level model selection skills and strategy-level forecast refinement skills, which are distilled from matched forecasting comparisons. During inference, these skills are retrieved through agentic reasoning and executed for each forecasting case. Empirical evidence from eight benchmark datasets demonstrates Skill4TSF's strong performance, with detailed ablation studies supporting the effectiveness of each component.

Critical strengths
  1. The paper proposes Contrastive Skill Distillation to distill reusable and condition-specific skills from matched forecasting comparisons, combining agents' strong reasoning capabilities with evidence-grounded experience, which represents a step forward for agentic time series forecasting.

  2. The method incorporates empirical skill validation and additional refinement into the skill distillation process, further ensuring the effectiveness and robustness of the framework.

Critical weaknesses
  1. The Skill4TSF framework demonstrates good transferability across models, while its transferability across datasets remains unknown. If the skills learned from one dataset are not applicable to other datasets, this may limit the generalizability and efficiency of the framework. Also, the applicability conditions of the illustrative skills in Appendix D.2 seem too restrictive to allow transfer to other cases.

  2. Skill4TSF is compared with three LLM-based approaches, while only MemCast among them is a truly agent-based approach that breaks down the forecasting process instead of simply prompting an LLM. Comparisons with more agentic baselines that feature model selection, tool utilization, or agentic RL training would make the conclusions more persuasive.

Questions for authors
  1. How do you define the fixed forecaster (M_{\text{global}}) for each dataset in Section 3.2?

  2. What is the validation stopping criterion mentioned in the last paragraph of Section 3.2?

  3. How does the LLM determine whether a preliminary forecast is consistent with the diagnosed structural patterns, as mentioned in the last paragraph of Section 3.3?

  4. Is there an error in the legend of Figure 2? It seems that the colors of the bars have been mistakenly swapped.

4 comments

We thank Reviewer HZrv for the suggestions on transferability, agentic comparisons, and implementation details.

W1: The Skill4TSF framework demonstrates good transferability across models...

We first clarify that the skills illustrated in Appendix D.2 were distilled from experiments on a single dataset. They are therefore tailored to the statistical characteristics of that dataset. These examples are meant to illustrate the format of a skill, not to restrict its scope. Following the reviewer's suggestion, we conducted transfer experiments in which skills learned on a source dataset are applied directly to a target dataset without any re-learning. Deep learning forecasters must be trained separately on each dataset, so for this experiment the model pool contains only training-free forecasters (statistical models and pretrained foundation models). In addition, we define the system prompt and the predefined rules in a dataset-agnostic manner, removing all dataset-specific designs. The results are reported below. Source-only transfer yields MSEs higher than target-only acquisition, while requiring no target-side skill learning. Combining source and target skills achieves the lowest MSE in four of the five transfer directions. These results suggest that source skills offer reusable forecasting knowledge that can complement target-specific skill acquisition.

We have added this experiment and discussion in Appendix D.8.

Transfer directionSource-onlyTarget-onlySource+Target
ETTm1 → ETTm20.2530.2480.245
ETTh1 → ETTh20.3380.3320.333
ETTm1 → ETTh10.4060.4030.401
ECL → Traffic0.3940.3860.383
Weather → Solar0.2080.2040.203

W2: 1. Skill4TSF is compared with three LLM-based approaches...

Existing agentic forecasting methods differ substantially in their LLM backbones, candidate forecaster pools, and evaluation protocols, so they cannot be compared directly. In addition, methods based on agentic RL training, such as CastR1, require fine-tuning the LLM at considerable computational cost. Skill4TSF, by contrast, is training-free for LLM: it improves the agent by distilling reusable skills from experience, without updating any LLM parameters. We therefore regard RL-based methods as an orthogonal and complementary line of work. A comprehensive comparison with them is beyond the scope of this paper.

To address the concern, we have added two recent agentic baselines, TimeSeriesScientist and TimeReasoner. To ensure a fair comparison, we re-implemented both methods under exactly the same setting as Skill4TSF: the same LLM backbone, the same candidate forecaster pool, the same diagnostic tools and the same evaluation protocol. Below reports the results. Skill4TSF achieves the lowest MSE on all eight datasets and the lowest MAE on seven. Compared with the strongest baseline on each dataset, it reduces MSE by 4.7%–11.6%. On ETTh2, TimeSeriesScientist achieves a slightly lower MAE, while Skill4TSF retains the lowest MSE. We have added this experiment and discussion in Appendix D.9.

DatasetSkill4TSFMemCastTimeSeriesScientistTimeReasoner
ETTm10.317 / 0.3660.359 / 0.3920.339 / 0.3820.364 / 0.395
ETTm20.242 / 0.3070.265 / 0.3220.263 / 0.3180.277 / 0.332
ETTh10.396 / 0.4140.439 / 0.4410.426 / 0.4320.482 / 0.473
ETTh20.326 / 0.3850.364 / 0.3980.342 / 0.3800.380 / 0.406
ECL0.154 / 0.2470.165 / 0.2610.163 / 0.2490.182 / 0.276
Traffic0.381 / 0.2590.426 / 0.2930.413 / 0.2820.422 / 0.295
Weather0.214 / 0.2500.242 / 0.2660.249 / 0.2670.248 / 0.273
Solar0.199 / 0.2780.210 / 0.2850.216 / 0.2840.243 / 0.303

Q1: How do you define the fixed forecaster...

The fixed forecaster MglobalM_{\text{global}} is defined as the single forecaster in the pool that achieves the lowest average MSE on the discovery set, and is then fixed. Level 1 skills are compared with this model. Level 2 skills instead use the fixed Level 1 policy as their reference, which may select different forecasters across windows.

Q2: What is the validation stopping criterion...

After each round, we check two conditions: (i) the skill pool remains unchanged after a full round of revision, de-duplication, and conflict resolution; or (ii) the relative improvement in validation MSE of the full skill-guided workflow, compared with the best round so far, is below ϵ=1\epsilon = 1 %. Refinement stops early once either condition holds for three consecutive rounds, and it otherwise terminates at 30 rounds. We have added this explanation to Appendix B.

(continued from above)

Q3: How does the LLM determine whether a preliminary forecast...

Reflection is an outcome-free plausibility check: it never accesses the ground truth and does not estimate the forecast error. Instead, the diagnostic tools deterministically compute four consistency diagnostics between the preliminary forecast and the input: (i) the dominant period and STL seasonal strength; (ii) the level discontinuity between the last observation and the first forecast step, normalized by the input standard deviation; (iii) the ratio of forecast amplitude to recent seasonal amplitude; and (iv) violations of the look-back value range. The selected skills may also specify additional dataset-relevant checks (e.g., non-negativity for solar power). Given these diagnostics, the forecast plot, and the selected skills, the LLM flags a mismatch only when it can attribute the mismatch to a specific diagnostic. It then applies at most one correction from a predefined action set: revising component weights, adjusting post-processing, or switching to another compatible strategy. Otherwise, the preliminary forecast is kept unchanged. We have added this explanation to Appendix B.

Q4: Is there an error in the legend of Figure 2...

We thank the reviewer for pointing this out. We have corrected this in the revised manuscript.

Thank you to the authors for the detailed response and additional experiments. The rebuttal has adequately addressed my concerns, particularly regarding cross-dataset transferability and comparisons with additional agentic baselines. I also appreciate the clarifications on the implementation details. Overall, I am satisfied with the authors' responses and am happy to maintain my score in support of acceptance.

Dear Reviewer HZrv,

Thank you for your positive follow-up. We’re glad the response addressed your concerns, and we appreciate your support of acceptance.

Sincerely, The Authors

Reviewer_kMxKOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a strategy to build skill files for a forecasting LLM/agent with skills such as model selection, preprocessing and bias correction. The goal is to tackle heterogeneity of domains. They evaluate the approach on ETT, Traffic, Weather, Solar and ECL datasets and show that they can outperform individual models.

Critical strengths

Learning arbitration of time series models which suit the task is crucial for time series forecasting. The proposed approach is well motivated and provides a good framework for this.

Critical weaknesses
  • The authors only compare against individual TS models, and do not compare against any ensemble baselines. Arbitration and ensemble of TS models has been studied [1]. They should compare against stronger baselines as it is not meaningful to beat just the individual models.
  • The analysis lacks depth. The motivation is to tackle heterogeneity of domains but analysis on which models are picked for which domains and what is learned is not provided. Also, the number of benchmarks should be expanded (datasets are pre-2021 and experiments on Qwen are not very meaningful due to potential memorization).

[1] Das, S. S. S., et al. (2026). Synapse: Adaptive Arbitration of Complementary Expertise in Time Series Foundational Models. Transactions on Machine Learning Research, 2026

Questions for authors

Can you show comparison of your approach against other ensemble or arbitration based approaches on more comprehensive and recent benchmarking?

4 comments

We thank Reviewer kMxK for recommending stronger arbitration baselines and a more detailed analysis of the learned behavior.

W1: The authors only compare against individual TS models...

Since Skill4TSF draws on multiple forecasters, we agree that ensemble and arbitration methods operating on the same model pool constitute a natural and important comparison. Following the reviewer's suggestion, we have added such baselines. All of them use exactly the same eleven-model pool and data splits as Skill4TSF: (i) mean and median ensembles; (ii) an inverse-validation-error weighted ensemble; and (iii) Synapse. The results are reported in the table below. Skill4TSF achieves the lowest MSE on all four datasets, reducing MSE by 2.9%–7.8% relative to Synapse, the strongest baseline in this comparison. These results support the additional predictive value of Skill4TSF beyond the tested ensemble and arbitration strategies when using the same forecaster pool. We have added this experiment and discussion in Section 4.3.5.

MethodETTm1 MSEECL MSEWeather MSESolar MSE
Mean ensemble0.3620.1830.2390.214
Median ensemble0.3550.1780.2460.216
Weighted ensemble0.3510.1710.2350.207
Synapse0.3420.1660.2320.205
Skill4TSF0.3170.1540.2140.199

W2: The analysis lacks depth. The motivation is to...

We analyze forecaster usage and intervention execution on ETTm1, ECL, Weather, and Solar, reporting their frequencies across forecasting windows. The most frequently used forecasters differ across datasets: PatchTST on ETTm1 (22%), iTransformer on ECL (30%), Time-MoE on Weather (19%), and TimesNet on Solar (16%). Intervention usage also varies: seasonal-amplitude correction is executed on 38% of ETTm1 windows and 43% of Solar windows, compared with 13% on Weather. Weather instead exhibits more frequent look-back adjustment (31%) and bias correction (34%). These statistics make the dataset-specific model preferences and intervention behavior explicit, complementing the aggregate forecasting results. We have added this analysis and visualization in Appendix D.6.

The datasets in our main experiments are well-established benchmarks in time series forecasting. Recent agentic forecasting methods, including MemCast, TimeSeriesScientist, and Cast-R1, still use them widely, and recently released benchmarks such as TSFM-Bench and GIFT-Eval also include them. Regarding memorization, we note that, by design, the LLM in Skill4TSF never reasons directly over the raw time series. It observes only the outputs of the diagnostic tools and uses them to trigger skills, while all numerical predictions are produced by the forecasters in the model pool. Nevertheless, we share the reviewer's concern and have added two recent benchmarks, Wind Power (WP) and Solar Power (SP). We retain our original forecaster training procedure and Qwen3.5-27B backbone, and follow the experimental setting of MemCast. The results are reported below. Skill4TSF achieves the lowest MSE and MAE on both datasets. Relative to the strongest baseline on each dataset, it reduces MSE by 6.0% on WP compared with MemCast and by 10.0% on SP compared with PatchTST. These results extend the empirical support for Skill4TSF to additional renewable-power forecasting tasks. We have added this experiment and discussion in Appendix D.7.

Skill4TSFMemCastTime-LLMAuroraSundialPatchTSTiTransformer
WP MSE2219.2482360.4563152.1182564.1503159.2732519.2242488.695
WP MAE31.94532.82336.15434.53936.92233.98233.719
SP MSE22.45228.96543.27839.81432.13124.95325.812
SP MAE1.8262.6814.9934.1673.5822.4742.667

Q1: Can you show comparison of your approach against other ensemble...

We have taken the reviewer's suggestion and added ensemble and arbitration baselines, and more recently released benchmarks, as described in W1 and W2. The results are reported in the tables above.

Area_Chair_w6hyOpenReview ↗

please engage

Dear reviewers,

Please join the discussion with the authors.

Best, AC

Thanks for adding the ensemble baselines and analysis on distribution of models picked across datasets. I have considered this and increased my score to 2. Memorization issue and potential overfitting are still my concerns given the approach.

We thank the reviewer for the constructive suggestions and are pleased that the additional experiments addressed some of your concerns. We would like to clarify the remaining concern about memorization. Our evaluation follows the established practice in agentic time series forecasting. For example, MemCast uses GPT-5, TimeReasoner uses DeepSeek-R1, and Cast-R1 uses Qwen3-8B to evaluate forecasting performance on established benchmarks, including ETT and EPF that published in 2021. This experimental setting is common in the field and is not the setting we introduced. The experiment in Appendix D.9 shows that our proposed method achieved better performance than the agentic baselines under the same setting.

As noted in our earlier response W2, the LLM in Skill4TSF does not receive the raw numerical sequence or generate forecast values directly. It uses tool-generated diagnostics, plots, and skill records to select forecasters and interventions, while the forecasters and numerical tools compute the final predictions. This design precludes the LLM from directly supplying a memorized sequence as the forecast. To examine a possible indirect use of memory, we tested whether the LLM relies on explicit names to recognize a benchmark and select a forecaster or strategy based on remembered model rankings. We masked the forecaster names, the dataset name, or both. As shown below, even masking both left MSE unchanged to three decimal places on five of the eight datasets and increased it by only 0.001 on the other three. These results suggest that the observed forecasting performance does not depend materially on access to dataset or forecaster names. We have added this experiment to Appendix D.10 in the revised manuscript.

Masking conditionETTm1ETTm2ETTh1ETTh2ECLTrafficWeatherSolar
Original0.3170.2420.3960.3260.1540.3810.2140.199
Forecaster masked0.3170.2420.3960.3260.1550.3810.2140.199
Dataset masked0.3170.2420.3950.3260.1550.3810.2140.200
Both masked0.3180.2420.3960.3260.1550.3810.2140.200

Regarding potential overfitting, we follow the standard chronological training, validation, and test splits for each benchmark. Most papers [1-3] on time series forecasting (including our baselines) that utilize these datasets follow this setup. We discover and confirm candidate skills on the training partition, refine them on the validation partition, and freeze the skill pool and selection thresholds before the main test evaluation. The future observations for each test window are unavailable to the agent when it issues its forecast. The cross-dataset experiments in Appendix D.8 provide a separate test of generalization. Across five transfer directions, skills learned on a source dataset were applied directly to a target dataset without target-side refinement. Their MSE remained close to that of skills learned on the target dataset, providing evidence that the learned strategies retain value beyond the data on which they were developed.

[1] Wang, Yuxuan, et al. "Deep time series models: A comprehensive survey and benchmark." IEEE Transactions on Pattern Analysis and Machine Intelligence (2026).

[2] Liu, Yong, et al. "itransformer: Inverted transformers are effective for time series forecasting." International conference on learning representations. Vol. 2024. 2024.

[3] Zeng, Ailing, et al. "Are transformers effective for time series forecasting?." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 9. 2023.

Reviewer_ExJFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Skill4TSF, a time-series forecasting agent that acquires reusable, condition-specific strategies as external textual skills. Its central component, Contrastive Skill Distillation (CSD), has two levels. Level 1 compares heterogeneous forecasters across historical windows to identify conditions under which one model should be preferred over a globally selected reference forecaster. Level 2 analyzes recurring forecast failures and proposes interventions, such as preprocessing, component-wise routing, and bias correction. Candidate skills are evaluated on separate confirmation windows using one-sided paired Wilcoxon tests, subsequently refined on a validation partition, and frozen for test-time use. Each skill includes applicability conditions, a strategy, supporting evidence, and a rationale. At inference time, statistical filtering and LLM-based applicability assessment select and compose skills, followed by a limited reflection step. Experiments on eight benchmarks and four forecasting horizons report strong predictive results, together with component ablations, backbone comparisons, online skill updates, and cross-backbone transfer studies.

Critical strengths
  1. Time series forecasting exhibits substantial heterogeneity across datasets and temporal patterns, making it difficult for a single model or fixed strategy to perform consistently well. Learning reusable, condition-specific skills provides a meaningful direction for improving the adaptability of forecasting agents.
  2. Contrastive Skill Distillation (CSD) integrates model-level comparisons with hypothesis-driven strategy refinement. Separating model selection from forecasting interventions is reasonable, while paired statistical tests provide an explicit mechanism for validating candidate skills.
  3. Extensive experiments across eight real-world datasets and four prediction horizons demonstrate the effectiveness of Skill4TSF.
Critical weaknesses
  1. In Section 3.2, Level 2 intervention skills are individually validated against the fixed Level 1 forecasting policy. However, Section 3.3 allows multiple compatible skills to be composed during inference. Improvements established for individual interventions do not necessarily hold when several interventions are applied together, particularly when preprocessing, component routing, and bias correction interact. The framework does not clearly explain how the effectiveness of composed strategies is verified.
  2. Equations (4) and (5) use paired Wilcoxon signed-rank tests to determine whether candidate skills should be retained. However, the forecasting windows are generated with strides of 8 or 16, which can produce substantial overlap between samples, especially for longer prediction horizons. Such dependencies may violate the independence assumptions underlying the statistical tests and lead to overly optimistic significance estimates.
  3. Level 2 discovers intervention skills by examining residual patterns on hard forecasting windows, where future observations are available. During inference, however, skill applicability must be determined entirely from the observed look-back window. It remains unclear how the framework ensures that historical residual patterns can be reliably associated with observable input characteristics, particularly when similar historical patterns may lead to different forecasting errors.
  4. Skill conditions are constructed using features such as STL-based trend strength, seasonality strength, and ADF test statistics. However, the evaluated datasets include hundreds of variables, while the description primarily focuses on window-level conditions. How channel-specific characteristics are aggregated or represented remains unclear, potentially limiting the precision of skill retrieval when different channels exhibit heterogeneous temporal patterns.
Questions for authors
  1. Why is the validation success rate in Equation (6) preferred over the average reduction in forecasting error when selecting and refining skills? How does the framework handle skills that improve most windows but introduce large errors on a small subset?
  2. When the skill-update interval is shorter than the forecasting horizon, how does the framework determine which forecasting trajectories have complete ground-truth feedback? Are updates restricted to fully observed prediction horizons?
  3. What exactly is provided to the LLM during inference? Does the LLM directly process raw time series, visualizations, or extracted statistical features? How are multivariate time series represented in the input, and what information is available to the LLM during skill retrieval, strategy composition, and forecast refinement?
  4. Why are the strongest forecasting models not included in the base forecaster pool? Although the framework evaluates against recent forecasting models such as Aurora and Sundial, these models are not included in the pool used for skill distillation and model selection. Could the authors further justify this design choice and explain how the composition of the forecaster pool affects the effectiveness of the proposed framework?
8 comments

We thank Reviewer ExJF for the detailed comments on skill validation, applicability, and execution.

W1: In Section 3.2, Level 2 intervention skills are individually validated against the fixed Level 1 forecasting policy...

Section 3.2 describes validation in terms of the success rate of each individual skill in Eq. (6), which guides skill refinement. However, the “validation stopping criterion” is determined by the validation error of the complete forecasting procedure described in Section 3.3, including retrieval, compatibility checking, composition, and reflection. Consequently, the effects of condition refinement and conflict resolution are assessed with the skills retrieved and executed together. Composition is therefore evaluated empirically during validation, and its effectiveness is not inferred solely from the significance of individual interventions. When a skill degrades the validation error of the composed pipeline on windows where it co-occurs with other skills, its evidence is updated, its condition is narrowed, or it is marked as incompatible with the co-occurring skill.

We have clarified the relationship between the per-skill success rate and the validation stopping criterion in Section 3.2.

W2: Equations (4) and (5) use paired Wilcoxon signed-rank tests to determine...

Overlapping windows are standard in time series forecasting benchmarks, which commonly use a stride of 1 for both training and evaluation. Our strides of 8 and 16 were chosen to reduce this redundancy. A larger stride alone does not resolve the issue: obtaining windows with disjoint inputs and targets would require a stride of at least L+HL+H (up to 1232 steps on ETTm1), which would leave too few confirmation windows for any meaningful test. We therefore believe the appropriate remedy is to use a test that accounts for temporal dependence, rather than to discard most of the data.

We would also like to distinguish the role of confirmation from that of final evaluation. Equations (4)–(5) screen candidate skills whose conditions and parameters are specified on discovery data and frozen before confirmation. Retained skills undergo further validation and refinement, and the final skill pool is frozen before evaluation on the chronologically held-out test partition. The reported test MSE and MAE therefore measure the forecasting performance of the resulting method, which are not inferred from the confirmation p-values. We have added the ablation about confirmation criterion. Specifically, we audited the frozen candidates using a null-centered block bootstrap of paired error reductions, preserving the original temporal grid and applicability indicators. An i.i.d. bootstrap served as a control for the change from signed ranks to mean error reduction. We then re-screened both distillation levels and applied the same validation procedure. This directly measures whether the reported forecasting gains depend on the original screening assumption.

Confirmation criterionETTm1ECLWeatherSolar
Original Wilcoxon0.3170.1540.2140.199
i.i.d. mean-loss bootstrap0.3230.1580.2160.204
Block bootstrap, block length b00.3190.1560.2180.206
Block bootstrap, block length 2b00.3190.1580.2130.203

Across the three alternative criteria, our criterion achieves the best results in most cases. Increasing the block length does not produce a consistent deterioration: ETTm1 remains unchanged, while Weather and Solar improve. The dependence-aware variants thus retain broadly comparable forecasting accuracy, suggesting that the reported performance is not driven solely by the independence assumption underlying the original screening test. We have added this experiment and discussion in the Appendix D.3.

W3: Level 2 discovers intervention skills by examining residual patterns...

Historical residuals are used during discovery to diagnose recurrent failure modes and propose candidate interventions. Each candidate pairs an intervention with an applicability condition assessed using only the observed look-back window. Confirmation then evaluates the skill on all confirmation windows satisfying the condition signature, regardless of whether they were hard or what their residuals look like. Consequently, a hypothesis inspired by residuals is retained only if its observable condition identifies windows on which the intervention yields a consistent improvement on data not used for discovery. At inference time, the framework computes look-back features and evaluates the condition signature, without access to the current window’s future observations.

(continued from above)

We agree that similar look-back windows can lead to different forecasting errors. Skill4TSF does not assume that input characteristics determine the error of each window. Instead, the objective is to identify observable conditions under which an intervention provides positive conditional benefit: $ \mathbb{E}!\left[e_{\mathrm{ref}}-e_\omega \mid C_\omega(X)=1\right]>0. $ The confirmation comparison provides empirical evidence toward this objective by evaluating all eligible windows, rather than only those exhibiting the residual pattern that motivated the intervention. Subsequent validation and refinement further assess whether the benefit persists beyond confirmation. Variation across individual windows is therefore expected: the condition is intended to identify a region of beneficial application, rather than predict the future residual of every window.

W4: Skill conditions are constructed using features such as STL-based trend strength...

In the multivariate setting, diagnostic tools first compute features separately for each channel, including trend strength, seasonal strength, and ADF diagnostics. Skill conditions are then expressed through cross-channel summaries, such as feature means and the fraction of channels whose seasonal strength exceeds a stored threshold. Prevalence predicates distinguish patterns concentrated in a small subset of channels from those shared more broadly across the window. Candidate conditions are specified on discovery data, and their usefulness is subsequently assessed through confirmation and validation.

The agent makes a window-level forecasting decision: the selected operation family and execution order are shared across channels, while channel-specific quantities, such as trends and seasonal components, are computed separately by the tools. This design supports batched or multivariate forecasting execution. For example, Traffic contains 862 channels. Running a separate retrieval, composition, and reflection workflow for each channel would introduce hundreds of channel-specific decision workflows per forecasting window, substantially increasing LLM reasoning and tool-orchestration overhead. Window-level decisions keep this orchestration compact while retaining channel-level numerical processing. Channel-specific workflows would also increase forecasting overhead. For forecasters supporting multivariate or batched execution, separate channel-level invocations can fragment batches and increase dispatch costs. We agree that heterogeneous channels may have different individually optimal strategies. Our design therefore evaluates skill usefulness through forecasting error aggregated over the evaluated channels, while cross-channel summaries inform when a shared strategy is applicable. Its benefit is assessed empirically at the whole-window level, rather than assumed from the prevalence predicate alone. We have clarified the aggregation procedure in Section 3.3

Q1: Why is the validation success rate in Equation (6) preferred over

We prefer the success rate in Eq. (6) because the error scale varies considerably across windows, for example between high-amplitude periods and windows with regime shifts. Under these conditions, the average error reduction is easily dominated by a few windows. A skill may yield large improvements on one or two high-error windows while degrading performance on most others, and still obtain a positive mean gain. Such a skill is not reusable in practice. Selecting by average reduction therefore tends to favor skills that happen to fit a few large-error windows in the validation set, which amounts to overfitting the validation data. By contrast, the success rate requires improvement on the majority of triggered windows. It thus rewards consistent, condition-aligned gains and transfers more reliably to unseen test data.

Regarding the failure scenarios you mentioned, the framework addresses this concern through condition refinement. Failure windows are analyzed separately, and the agent identifies how their predicates differ from those of successful windows (e.g., high ADF statistics, structural breaks, or weak seasonal strength). The agent then tightens and refines the skill condition to exclude these cases, and the skill's evidence is updated accordingly. As a result, windows on which a skill would cause large errors are progressively excluded from its trigger region in most cases. Although this procedure cannot strictly guarantee that a skill never increases the error, it effectively suppresses such cases in practice.

(continued from above)

We compare four validation criteria on ETTm2 and Traffic: success rate alone, mean MSE reduction alone, their conjunction (AND), and their disjunction (OR). We report overall test MSE and the mean MSE over each method’s worst-performing 5% of test windows, denoted as “5% MSE.” Success-rate-based refinement achieves the lowest overall MSE on Traffic and ties with OR on ETTm2, while obtaining the lowest worst-5% MSE on both datasets. These results support success rate as an effective refinement signal, yielding favorable average and tail forecasting performance in the evaluated settings. We have added this experiment and discussion in Appendix D.4.

Validation criterionETTm2 MSEETTm2 5% MSETraffic MSETraffic 5% MSE
Success rate only0.2420.2680.3810.436
Mean MSE reduction only0.2440.2750.3850.449
Both criteria (AND)0.2450.2810.3850.442
Either criterion (OR)0.2420.2700.3830.443

Q2: When the skill-update interval is shorter than the forecasting horizon...

Online skill updates use only forecasting trajectories whose entire prediction horizons have been observed. Specifically, a forecast issued at origin uu with horizon H becomes eligible for updating at time tt only when u+H≤tu + H ≤ t. At each scheduled update, the newly completed trajectories provide additional evidence for skill learning and refinement. Trajectories with only partially observed targets are excluded.

As noted in W2, we follow the standard practice in time series forecasting and construct samples with a sliding window of stride 1. Thus, after warm-up, SS newly completed trajectories become available every SS steps, even when S<HS < H. These trajectories were issued earlier and have just completed their horizons. Updates with no complete feedback are skipped, and updated skills are used only for subsequent forecasts. We have added this explanation to Section 4.3.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.