- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TimeEvolve, a self-evolving framework for agentic time-series forecasting that adapts its orchestration policy using delayed feedback from realized forecasting outcomes while keeping the underlying language model and numerical forecasting models frozen. The framework consists of three components: EvolveTrust, which adaptively weights numerical forecasting experts based on their reliability; EvolveReason, which selects among LLM-generated candidate forecasting paths using a learned error predictor; and EvolveIntervene, which determines how strongly the selected candidate should modify the numerical ensemble forecast. Through a predict–reveal–update procedure, the framework uses observed outcomes to update expert reliability, candidate selection, and intervention strength for subsequent predictions, including feedback from unselected alternatives. Experiments on eight Time-MMD domains compare TimeEvolve against 14 baselines and report the best average MSE and MAE rankings, while ablation studies demonstrate improvements over a counterpart with a frozen orchestration policy.
Critical strengths
S1. The paper introduces a structured predict–reveal–update framework that leverages realized forecasting outcomes to update expert weighting, candidate path selection, and intervention strength while keeping the underlying forecasting models and LLM frozen. A notable strength is that the framework can evaluate both selected and unselected forecasting alternatives against the same observed outcomes, providing additional supervision without requiring manual annotations. This offers a practical and computationally lightweight approach to adapting agentic forecasting systems.
S2. TimeEvolve achieves competitive results across eight Time-MMD domains, achieving the best average MSE and MAE rankings among 15 methods. The comparison against TimeEvolve-Stack, which uses the same forecasting models and pipeline without online policy updates, also shows improvements in both metrics across all eight domains. The component-wise and feedback-coverage ablations further support the effectiveness of the proposed adaptation mechanism.
Critical weaknesses
W1. A key motivation behind this paper is the idea that the effectiveness of forecasting models and agent policies can vary over time, making it essential to continually evolve these policies. However, the experiments focus mainly on reporting aggregated forecasting errors for the entire test period. The paper falls short in that it doesn't provide in-depth analyses of how each expert's reliability shifts over time, or how the learned weights respond to these changes. It also misses out on showing how the performance advantage develops as more feedback is gathered. Moreover, the proposed progressive weighting schedule hasn't been tested independently against other scheduling methods. So, while the findings indicate that outcome-driven updates can enhance overall accuracy, they don't convincingly demonstrate that the method can adapt to changing conditions over time, which is a central goal of this research.
W2. The performance boost seen with the new method is hard to pin down to just the proposed policy evolution. In six out of eight areas tested, the frozen-policy version alone shows an impressive improvement, accounting for about 70% to 97% of the MSE reduction compared to KairosAgent. For instance, in the Climate domain, KairosAgent has an MSE of 0.863, while the TimeEvolve-Stack drops that to 0.342. The full method only slightly improves upon this with an MSE of 0.314. This means that an impressive 94.9% of the improvement occurs without any online policy evolution. In the Traffic domain, the MSE goes from 0.151 to 0.089 with the frozen policy, while the full approach achieves 0.087, reflecting 96.9% of the overall improvement. Moreover, the paper doesn’t compare the proposed method to standard online ensemble-learning techniques like gradient-based weight optimization. This omission leaves us unsure whether EvolveTrust really offers any benefits beyond what established methods can provide. Even though the framework includes paths generated by LLMs and external text evidence, there are no tests removing these elements to see how much they contribute. Without these comparisons, it’s hard to determine whether the presented systematic approach is essential or more effective than simpler numerical forecasting and online ensemble methods.
Questions for authors
Q1. The paper motivates TimeEvolve by arguing that expert reliability changes over time, yet the experiments primarily report aggregate test-set performance. Can the authors provide evidence demonstrating that EvolveTrust tracks changes in expert reliability and that the benefits of policy evolution persist or increase as more feedback becomes available?
Q2. In six of eight domains, TimeEvolve-Stack already accounts for about 70%–97% of the MSE improvement over KairosAgent, despite having no online policy updates. Can the authors clarify how much of the improvement comes from the underlying numerical expert pool and initial forecasting pipeline rather than the proposed policy evolution? Are results available under a matched numerical expert pool?
Q3. The numerical forecasting models primarily operate on time-series inputs, while external textual evidence is incorporated indirectly through LLM-generated candidate paths. Can the authors provide numerical-only and LLM-removal ablations to quantify the contribution of textual evidence and LLM-generated paths? Additionally, how does EvolveTrust compare with standard online ensemble-weighting approaches, such as gradient-based optimization or exponentiated-gradient updates, using the same numerical experts?
2 comments
Thank you to the authors for the rebuttal, which clarifies the benefits of policy evolution and provides an illustrative case study. However, my main concerns remain unresolved.
First, the manuscript still lacks systematic temporal evidence that EvolveTrust tracks changing expert reliability or that its benefits persist or increase as feedback accumulates, while Table 3 reports aggregate results and the progressive weighting schedule remains untested against alternatives.
Second, although the Stack comparison isolates the incremental benefit of policy updates, no matched-expert-pool comparison with KairosAgent is provided, and Section 5.2 continues to use improvements over agentic baselines to support the orchestration mechanism without isolating its contribution.
Third, the comparisons with standard online ensemble methods using the same numerical experts, along with numerical-only and no-text ablations, are not provided.
As these concerns remain insufficiently addressed, I maintain my original rating.
Thank you for the detailed review and constructive suggestions. We appreciate your recognition of the feedback framework and your emphasis on explaining how adaptation develops over time and contributes to forecasting accuracy.
Q1. Does EvolveTrust respond to changing expert reliability and accumulating feedback?
R1. EvolveTrust updates expert weights using errors from completed forecast windows. Its progressive schedule increases the contribution of deployment feedback as more outcomes become available (Section 4.3). Table 3 provides complementary evidence that broader feedback improves forecasting: supervising all precommitted alternatives reduces MSE/MAE by 5.0%/6.3% relative to selected-only feedback. To connect this aggregate result to temporal behavior, we will add chronological plots of expert errors, issued weights, and cumulative loss differences from the evaluation records. These plots will show when allocation changes and whether subsequent predictions benefit.
Q2. How much improvement comes from the initial forecasting pipeline?
R2. The initial pipeline contributes substantially to overall performance. The adaptation increment is measured directly against Stack, which shares the expert pool and forecast-construction pipeline. In Climate, MSE decreases from 0.342 to 0.314, an 8.19% reduction. In Traffic, it decreases from 0.089 to 0.087, a 2.25% reduction. Across all eight domains, the average relative reductions are 6.3% MSE and 7.6% MAE. We will present this matched comparison as the evidence for policy adaptation and use comparisons with KairosAgent to characterize overall system performance.
Q3. What do the numerical policy, LLM paths, and textual evidence contribute?
R3. Table 3 measures the contribution of path supervision while retaining full expert feedback and intervention learning. Expanding supervision from the selected path to all precommitted paths improves MSE and MAE in every domain, averaging 1.6% and 2.2% relative reductions. In Economy, MSE decreases from 0.187 to 0.181. To isolate candidate generation and textual evidence, we will add numerical-only and no-text controls alongside a same-pool online ensemble. Comparing the final forecast with its own recorded prior will additionally quantify the net contribution of path adoption under the same trust state.
We appreciate your suggestions for strengthening the temporal analysis and contribution attribution. We will incorporate these clarifications and targeted comparisons to make the relationship between the proposed mechanism and the observed gains easier to assess.