- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CoEvoST, a dual-evolution framework for continual spatio-temporal forecasting that employs an LLM agent to dynamically determine whether to update the forecasting model, which data to use, the adaptation scope, and the adaptation mechanism based on diagnostic information such as prediction errors, distribution shifts, graph structures, and historical experience. Experimental results demonstrate improved forecasting accuracy across multiple spatiotemporal streams. However, the paper does not sufficiently evaluate the computational overhead and real-time adaptation efficiency associated with introducing an LLM, candidate strategy exploration, and online DPO.
Critical strengths
The main strength of this paper lies in its proposed dual-evolution framework, which dynamically selects adaptation strategies using an LLM and continuously improves its decision-making capability through execution feedback.
Critical weaknesses
-
The authors identify the first challenge as: "How to form an appropriate adaptation plan from heterogeneous diagnostic evidence?" However, the six categories of diagnostic indicators used in the paper are pre-designed, the risk thresholds are calibrated based on offline data, and the evidence required for each adaptation strategy is also manually specified. Given that the evidence needed for each strategy is already determined by human-defined rules, what is the added value of multi-turn LLM reasoning compared to a rule-based approach for evidence acquisition and strategy selection? Since the strategy space is both limited and predefined, the authors have not sufficiently justified why traditional, lightweight decision-making mechanisms could not accomplish the same task. Could this problem be solved directly through optimization methods, without the need to introduce an LLM?
-
The experimental evaluation is insufficient. The authors fail to analyze the training and inference overhead of their proposed LLM-based approach compared to lightweight models such as EAC or STRAP. Furthermore, since most online prediction scenarios require real-time model updates, I have concerns regarding the efficiency of both model updates and inference when an LLM is incorporated. Additionally, the paper reports only the average MAE, RMSE, and MAPE across the entire online data stream, without presenting performance metrics for individual periods. Consequently, it is difficult to determine whether the model has truly achieved stable continual learning or if the observed gains are limited to specific phases. Finally, the study lacks comparisons with some more recent works, such as STBP[1].
[1] A General Spatio-Temporal Backbone with Scalable Contextual Pattern Bank for Urban Continual Forecasting. ICLR2026
Questions for authors
- The author posits that as spatio-temporal environments constantly shift, not only must the predictor be updated, but the decision-making strategy of the LLM must also continuously evolve. Can this be verified through experimental case studies?
8 comments
For W2: Computational Efficiency, Stage-wise Performance, and Recent Baselines
(1) Computational efficiency and online feasibility.
We have expanded Appendix B.5 to quantify runtime, GPU memory usage, exploration frequency, and online DPO overhead, with all revisions highlighted in red.
CoEvoST separates Fast Predictor Adaptation from Slow Agent Evolution. Routine forecasting operates at the batch level, LLM planning occurs only at chunk-level decision points, and agent evolution runs in the background.
| Dataset | EAC (s/batch) | STRAP (s/batch) | CoEvoST (s/batch) | LLM Planning (s/chunk) |
|---|---|---|---|---|
| ENERGY | 0.000590 | 0.001446 | 0.012552 | 40.34 |
| PEMS | 0.000653 | 0.003144 | 0.024146 | 20.16 |
| AIR | 0.000961 | 0.004685 | 0.027145 | 21.03 |
| CAMELS-13 | 0.000409 | 0.000956 | 0.005694 | 20.34 |
CoEvoST, EAC, and STRAP all operate at the millisecond scale for batch-level inference, while CoEvoST's LLM planning is performed only at chunk-level decision points.
Candidate exploration occurs at only 9.61%–28.19% of decision points, and online DPO is triggered only 7–14 times per evaluation stream. Each DPO update takes approximately 61–76 minutes and runs in the background, allowing the predictor and existing agent to continue operating. GPU memory usage is approximately 18.2–18.9 GiB during online operation and 23.24 GiB during DPO training.
Importantly, even if a full DPO update were performed after every chunk, the total processing time on PEMS—including online processing (374.29 s), candidate exploration (7.53 s), and DPO (76 min)—would be approximately 82.4 minutes, substantially shorter than the 12-hour chunk interval.
These measurements indicate that CoEvoST maintains millisecond-level routine forecasting while computationally intensive agent evolution runs asynchronously without blocking online predictions, with sufficient computational capacity to complete agent updates between successive chunks under the evaluated setting.
(2) Stage-wise forecasting performance.
To evaluate whether the improvements persist throughout the online stream, we have added stage-wise MAE comparisons in Appendix B.3 (Table 12), dividing each dataset into early, middle, and late stages.
| Dataset | Method | Early | Middle | Late |
|---|---|---|---|---|
| ENERGY | STBP | 1.18 | 1.65 | 1.67 |
| STKEC | 0.93 | 1.24 | 1.40 | |
| CoEvoST | 0.86 | 0.87 | 1.38 | |
| CAMELS-13 | STBP | 20.61 | 11.89 | 11.64 |
| STKEC | 26.26 | 14.93 | 13.59 | |
| CoEvoST | 19.03 | 10.80 | 10.32 |
CoEvoST achieves the lowest MAE among the compared methods in all 3 stages, demonstrating that its improvements are not confined to particular phases of the online evaluation.
(3) Comparison with STBP.
We have included STBP (ICLR 2026) in the revised Table 2, following the same online evaluation protocol as the other baselines. Compared with STBP, CoEvoST achieves lower MAE across all four datasets: 1.04 vs. 1.50 on ENERGY, 13.69 vs. 14.24 on PEMS, 17.43 vs. 17.84 on AIR, and 13.61 vs. 14.95 on CAMELS-13.
For Q1: Case Study of Continual Agent Evolution
To verify whether online agent evolution improves adaptation decisions, we examine an actual case from CAMELS-13, comparing the agent's decisions before and after online DPO under the same diagnostic state.
(1) Environmental diagnosis. During October 28–November 26, 1995, the recent rolling MAE increased by 105.48% relative to the initial reference. The diagnostics identified substantial data distribution shift (MMD² = 0.1537) and environmental covariate shift (MMD² = 0.1678), while the graph structure remained unchanged. Historical input similarity was high (0.9806), but the corresponding future outcomes exhibited substantial dispersion (0.9166), suggesting that similar historical conditions did not necessarily imply consistent future behavior.
(2) Evolution of adaptation decisions. Given the same diagnostic state, the agent selected Standard LoRA before DPO but switched to Progressive LoRA after DPO. Both strategies perform lightweight global adaptation using recent data, but differ in their adaptation mechanisms. Standard LoRA updates existing adapter parameters, whereas Progressive LoRA freezes previous adapters and introduces a new adapter to capture emerging patterns.
Under the observed distributional and environmental shifts, Progressive LoRA may better accommodate new dynamics while reducing interference with previously learned adaptation parameters. This provides a plausible explanation for the evolved strategy selection.
(3) Execution outcomes. Starting from the same predictor checkpoint, we compare the recorded candidate outcomes on the same subsequent evaluation window (November 27–December 31, 1995):
| Adaptation Strategy | MAE | RMSE |
|---|---|---|
| No Update | 3.24997 | 3.62507 |
| Before DPO: Standard LoRA | 3.24057 | 3.61560 |
| After DPO: Progressive LoRA | 3.15991 | 3.53666 |
The strategy selected after DPO achieves a 2.49% MAE reduction compared with the pre-DPO selection, without triggering rollback.
This case demonstrates that online DPO can change the agent's adaptation mechanism under the same diagnostic state and improve its forecasting outcome. Together with the paired before-and-after DPO comparisons in Figure 4 and the SFT-only ablation in Table 3, these results provide complementary evidence for the effectiveness of experience-driven agent evolution.
Regarding presentation clarity, we have further clarified the LLM's role in adaptation planning, provided more detailed explanations of the decision-making process, and included additional experimental analyses in the appendix.
Finally, we sincerely appreciate your recognition of our dual-evolution framework and your constructive suggestions. We hope our additional experiments and clarifications have addressed your concerns and would be grateful if you could reconsider your assessment in light of these results.
Response: LLM-Based Adaptation Planning, Experimental Evaluation, and Agent Evolution
We sincerely thank the reviewer for the insightful and constructive comments, which raise important questions and provide valuable perspectives for strengthening our work. We provide the following additional experiments and analyses.
For W1: Necessity and Value of LLM-Based Adaptation Planning
The core challenge of adaptation planning lies in interpreting heterogeneous diagnostic evidence and determining an appropriate adaptation strategy under changing operating conditions.
We employ an LLM-based planner for four complementary capabilities:
To examine its added value, we conducted additional comparisons with Multi-Armed Bandit (UCB), Contextual Bandit (LinUCB), and Neural Policy Network (MLP). All methods use the same adaptation strategy space, online evaluation protocol, and reward definition. Both LinUCB and MLP receive all numerical diagnostic information from the summaries and detailed evidence.
LinUCB consistently outperforms UCB, confirming the value of state-dependent diagnostic information. More importantly, compared with the strongest conventional controller, Neural Policy Network, CoEvoST reduces MAE by 15.45%, 8.92%, and 7.54% on ENERGY, PEMS, and CAMELS-13, respectively. These results support the effectiveness of the complete LLM-based planning and evolution framework compared with conventional numerical decision policies.