- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents Skill4TSF, a skill-based agentic framework for time series forecasting. There are two categories of skills: model-level model selection skills and strategy-level forecast refinement skills, which are distilled from matched forecasting comparisons. During inference, these skills are retrieved through agentic reasoning and executed for each forecasting case. Empirical evidence from eight benchmark datasets demonstrates Skill4TSF's strong performance, with detailed ablation studies supporting the effectiveness of each component.
Critical strengths
-
The paper proposes Contrastive Skill Distillation to distill reusable and condition-specific skills from matched forecasting comparisons, combining agents' strong reasoning capabilities with evidence-grounded experience, which represents a step forward for agentic time series forecasting.
-
The method incorporates empirical skill validation and additional refinement into the skill distillation process, further ensuring the effectiveness and robustness of the framework.
Critical weaknesses
-
The Skill4TSF framework demonstrates good transferability across models, while its transferability across datasets remains unknown. If the skills learned from one dataset are not applicable to other datasets, this may limit the generalizability and efficiency of the framework. Also, the applicability conditions of the illustrative skills in Appendix D.2 seem too restrictive to allow transfer to other cases.
-
Skill4TSF is compared with three LLM-based approaches, while only MemCast among them is a truly agent-based approach that breaks down the forecasting process instead of simply prompting an LLM. Comparisons with more agentic baselines that feature model selection, tool utilization, or agentic RL training would make the conclusions more persuasive.
Questions for authors
-
How do you define the fixed forecaster (M_{\text{global}}) for each dataset in Section 3.2?
-
What is the validation stopping criterion mentioned in the last paragraph of Section 3.2?
-
How does the LLM determine whether a preliminary forecast is consistent with the diagnosed structural patterns, as mentioned in the last paragraph of Section 3.3?
-
Is there an error in the legend of Figure 2? It seems that the colors of the bars have been mistakenly swapped.
4 comments
(continued from above)
Q3: How does the LLM determine whether a preliminary forecast...
Reflection is an outcome-free plausibility check: it never accesses the ground truth and does not estimate the forecast error. Instead, the diagnostic tools deterministically compute four consistency diagnostics between the preliminary forecast and the input: (i) the dominant period and STL seasonal strength; (ii) the level discontinuity between the last observation and the first forecast step, normalized by the input standard deviation; (iii) the ratio of forecast amplitude to recent seasonal amplitude; and (iv) violations of the look-back value range. The selected skills may also specify additional dataset-relevant checks (e.g., non-negativity for solar power). Given these diagnostics, the forecast plot, and the selected skills, the LLM flags a mismatch only when it can attribute the mismatch to a specific diagnostic. It then applies at most one correction from a predefined action set: revising component weights, adjusting post-processing, or switching to another compatible strategy. Otherwise, the preliminary forecast is kept unchanged. We have added this explanation to Appendix B.
Q4: Is there an error in the legend of Figure 2...
We thank the reviewer for pointing this out. We have corrected this in the revised manuscript.
Thank you to the authors for the detailed response and additional experiments. The rebuttal has adequately addressed my concerns, particularly regarding cross-dataset transferability and comparisons with additional agentic baselines. I also appreciate the clarifications on the implementation details. Overall, I am satisfied with the authors' responses and am happy to maintain my score in support of acceptance.
Dear Reviewer HZrv,
Thank you for your positive follow-up. We’re glad the response addressed your concerns, and we appreciate your support of acceptance.
Sincerely, The Authors
We thank Reviewer HZrv for the suggestions on transferability, agentic comparisons, and implementation details.
W1: The Skill4TSF framework demonstrates good transferability across models...
We first clarify that the skills illustrated in Appendix D.2 were distilled from experiments on a single dataset. They are therefore tailored to the statistical characteristics of that dataset. These examples are meant to illustrate the format of a skill, not to restrict its scope. Following the reviewer's suggestion, we conducted transfer experiments in which skills learned on a source dataset are applied directly to a target dataset without any re-learning. Deep learning forecasters must be trained separately on each dataset, so for this experiment the model pool contains only training-free forecasters (statistical models and pretrained foundation models). In addition, we define the system prompt and the predefined rules in a dataset-agnostic manner, removing all dataset-specific designs. The results are reported below. Source-only transfer yields MSEs higher than target-only acquisition, while requiring no target-side skill learning. Combining source and target skills achieves the lowest MSE in four of the five transfer directions. These results suggest that source skills offer reusable forecasting knowledge that can complement target-specific skill acquisition.
We have added this experiment and discussion in Appendix D.8.
W2: 1. Skill4TSF is compared with three LLM-based approaches...
Existing agentic forecasting methods differ substantially in their LLM backbones, candidate forecaster pools, and evaluation protocols, so they cannot be compared directly. In addition, methods based on agentic RL training, such as CastR1, require fine-tuning the LLM at considerable computational cost. Skill4TSF, by contrast, is training-free for LLM: it improves the agent by distilling reusable skills from experience, without updating any LLM parameters. We therefore regard RL-based methods as an orthogonal and complementary line of work. A comprehensive comparison with them is beyond the scope of this paper.
To address the concern, we have added two recent agentic baselines, TimeSeriesScientist and TimeReasoner. To ensure a fair comparison, we re-implemented both methods under exactly the same setting as Skill4TSF: the same LLM backbone, the same candidate forecaster pool, the same diagnostic tools and the same evaluation protocol. Below reports the results. Skill4TSF achieves the lowest MSE on all eight datasets and the lowest MAE on seven. Compared with the strongest baseline on each dataset, it reduces MSE by 4.7%–11.6%. On ETTh2, TimeSeriesScientist achieves a slightly lower MAE, while Skill4TSF retains the lowest MSE. We have added this experiment and discussion in Appendix D.9.
Q1: How do you define the fixed forecaster...
The fixed forecaster Mglobal is defined as the single forecaster in the pool that achieves the lowest average MSE on the discovery set, and is then fixed. Level 1 skills are compared with this model. Level 2 skills instead use the fixed Level 1 policy as their reference, which may select different forecasters across windows.
Q2: What is the validation stopping criterion...
After each round, we check two conditions: (i) the skill pool remains unchanged after a full round of revision, de-duplication, and conflict resolution; or (ii) the relative improvement in validation MSE of the full skill-guided workflow, compared with the best round so far, is below ϵ=1. Refinement stops early once either condition holds for three consecutive rounds, and it otherwise terminates at 30 rounds. We have added this explanation to Appendix B.