acceptodds
Under review as a conference paper at ICLR 2027

CoEvoST: A Dual Evolution Agent for Continual Spatiotemporal Forecasting

Abstract

Spatiotemporal forecasting systems operate on evolving streams with changing data distributions, graph structures, observed node sets, and external conditions, requiring continual predictor adaptation. Existing continual spatiotemporal forecasting methods primarily focus on how to adapt the predictor through specialized mechanisms, while higher-level decisions about whether, where, on what data, and by which mechanism to adapt are typically prescribed rather than learned from evolving states. This raises two challenges: forming an appropriate adaptation plan from heterogeneous diagnostic evidence, and improving adaptation decisions from limited execution feedback. We therefore propose CoEvoST, a dual evolution framework that couples fast predictor adaptation with slower evolution of an LLM adaptation agent. For evidence-to-action reasoning, CoEvoST starts from compact diagnostic summaries and performs multi-round evidence retrieval and plan refinement, producing an adaptation plan that specifies whether, where, on what data, and by which mechanism to update the predictor with sufficient diagnostic support. For outcome-to-policy learning, CoEvoST executes candidate plans from the same predictor checkpoint and constructs near-optimal preference sets from their realized outcomes, accumulating them as experience for online field-weighted DPO to improve the agent's decision policy. Experiments across four spatiotemporal streams demonstrate that CoEvoST improves forecasting performance over strong fixed and continual baselines while adapting its decisions to changing operating conditions.

Primary area:
learning on time series and dynamical systems
Keywords:
Spatiotemporal Forecasting, Continual Learning, LLM Agent

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Qp6wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CoEvoST, a dual-evolution framework for continual spatio-temporal forecasting that employs an LLM agent to dynamically determine whether to update the forecasting model, which data to use, the adaptation scope, and the adaptation mechanism based on diagnostic information such as prediction errors, distribution shifts, graph structures, and historical experience. Experimental results demonstrate improved forecasting accuracy across multiple spatiotemporal streams. However, the paper does not sufficiently evaluate the computational overhead and real-time adaptation efficiency associated with introducing an LLM, candidate strategy exploration, and online DPO.

Critical strengths

The main strength of this paper lies in its proposed dual-evolution framework, which dynamically selects adaptation strategies using an LLM and continuously improves its decision-making capability through execution feedback.

Critical weaknesses
  1. The authors identify the first challenge as: "How to form an appropriate adaptation plan from heterogeneous diagnostic evidence?" However, the six categories of diagnostic indicators used in the paper are pre-designed, the risk thresholds are calibrated based on offline data, and the evidence required for each adaptation strategy is also manually specified. Given that the evidence needed for each strategy is already determined by human-defined rules, what is the added value of multi-turn LLM reasoning compared to a rule-based approach for evidence acquisition and strategy selection? Since the strategy space is both limited and predefined, the authors have not sufficiently justified why traditional, lightweight decision-making mechanisms could not accomplish the same task. Could this problem be solved directly through optimization methods, without the need to introduce an LLM?

  2. The experimental evaluation is insufficient. The authors fail to analyze the training and inference overhead of their proposed LLM-based approach compared to lightweight models such as EAC or STRAP. Furthermore, since most online prediction scenarios require real-time model updates, I have concerns regarding the efficiency of both model updates and inference when an LLM is incorporated. Additionally, the paper reports only the average MAE, RMSE, and MAPE across the entire online data stream, without presenting performance metrics for individual periods. Consequently, it is difficult to determine whether the model has truly achieved stable continual learning or if the observed gains are limited to specific phases. Finally, the study lacks comparisons with some more recent works, such as STBP[1].

[1] A General Spatio-Temporal Backbone with Scalable Contextual Pattern Bank for Urban Continual Forecasting. ICLR2026

Questions for authors
  1. The author posits that as spatio-temporal environments constantly shift, not only must the predictor be updated, but the decision-making strategy of the LLM must also continuously evolve. Can this be verified through experimental case studies?
8 comments
AuthorsOpenReview ↗

Response: LLM-Based Adaptation Planning, Experimental Evaluation, and Agent Evolution

We sincerely thank the reviewer for the insightful and constructive comments, which raise important questions and provide valuable perspectives for strengthening our work. We provide the following additional experiments and analyses.

For W1: Necessity and Value of LLM-Based Adaptation Planning

The core challenge of adaptation planning lies in interpreting heterogeneous diagnostic evidence and determining an appropriate adaptation strategy under changing operating conditions.

We employ an LLM-based planner for four complementary capabilities:

  1. Semantic Understanding and Reasoning: We provide the LLM with semantic descriptions of diagnostic indicators and adaptation strategies, rather than numerical values alone. The predefined evidence requirements also convey which diagnostic factors are relevant to each strategy, helping the LLM understand their characteristics and reason about the relationships between diagnostic evidence and adaptation actions. Unlike conventional numerical controllers that learn feature-to-strategy mappings primarily from task-specific feedback, the LLM can leverage semantic knowledge to inform decisions, particularly when execution feedback is limited.
  2. Adaptive evidence acquisition: Rather than relying exclusively on predefined evidence fields, the LLM can identify and request additional fine-grained evidence according to the current state and candidate plan. This helps the agent focus on decision-relevant information instead of being overwhelmed by numerous diagnostic indicators.
  3. Iterative plan verification and refinement: Additional evidence may confirm or challenge an initial decision, allowing the agent to reassess and revise its adaptation plan before execution. In practice, plan revisions occur at 8.64%, 2.52%, 2.73%, and 12.50% of decision points on ENERGY, PEMS, AIR, and CAMELS-13, respectively, demonstrating that iterative planning can lead to actual decision changes.
  4. Experience-driven decision evolution: The LLM incorporates historical execution experience and improves its decision policy through offline SFT and online DPO, without manually redesigning state-to-strategy mapping rules.

To examine its added value, we conducted additional comparisons with Multi-Armed Bandit (UCB), Contextual Bandit (LinUCB), and Neural Policy Network (MLP). All methods use the same adaptation strategy space, online evaluation protocol, and reward definition. Both LinUCB and MLP receive all numerical diagnostic information from the summaries and detailed evidence.

Decision PolicyENERGYPEMSCAMELS-13
Multi-Armed Bandit (UCB)2.3917.2618.87
Contextual Bandit (LinUCB)1.4115.9315.22
Neural Policy Network (MLP)1.2315.0314.72
CoEvoST1.0413.6913.61

LinUCB consistently outperforms UCB, confirming the value of state-dependent diagnostic information. More importantly, compared with the strongest conventional controller, Neural Policy Network, CoEvoST reduces MAE by 15.45%, 8.92%, and 7.54% on ENERGY, PEMS, and CAMELS-13, respectively. These results support the effectiveness of the complete LLM-based planning and evolution framework compared with conventional numerical decision policies.

For W2: Computational Efficiency, Stage-wise Performance, and Recent Baselines

(1) Computational efficiency and online feasibility.

We have expanded Appendix B.5 to quantify runtime, GPU memory usage, exploration frequency, and online DPO overhead, with all revisions highlighted in red.

CoEvoST separates Fast Predictor Adaptation from Slow Agent Evolution. Routine forecasting operates at the batch level, LLM planning occurs only at chunk-level decision points, and agent evolution runs in the background.

DatasetEAC (s/batch)STRAP (s/batch)CoEvoST (s/batch)LLM Planning (s/chunk)
ENERGY0.0005900.0014460.01255240.34
PEMS0.0006530.0031440.02414620.16
AIR0.0009610.0046850.02714521.03
CAMELS-130.0004090.0009560.00569420.34

CoEvoST, EAC, and STRAP all operate at the millisecond scale for batch-level inference, while CoEvoST's LLM planning is performed only at chunk-level decision points.

Candidate exploration occurs at only 9.61%–28.19% of decision points, and online DPO is triggered only 7–14 times per evaluation stream. Each DPO update takes approximately 61–76 minutes and runs in the background, allowing the predictor and existing agent to continue operating. GPU memory usage is approximately 18.2–18.9 GiB during online operation and 23.24 GiB during DPO training.

Importantly, even if a full DPO update were performed after every chunk, the total processing time on PEMS—including online processing (374.29 s), candidate exploration (7.53 s), and DPO (76 min)—would be approximately 82.4 minutes, substantially shorter than the 12-hour chunk interval.

These measurements indicate that CoEvoST maintains millisecond-level routine forecasting while computationally intensive agent evolution runs asynchronously without blocking online predictions, with sufficient computational capacity to complete agent updates between successive chunks under the evaluated setting.

(2) Stage-wise forecasting performance.

To evaluate whether the improvements persist throughout the online stream, we have added stage-wise MAE comparisons in Appendix B.3 (Table 12), dividing each dataset into early, middle, and late stages.

DatasetMethodEarlyMiddleLate
ENERGYSTBP1.181.651.67
STKEC0.931.241.40
CoEvoST0.860.871.38
CAMELS-13STBP20.6111.8911.64
STKEC26.2614.9313.59
CoEvoST19.0310.8010.32

CoEvoST achieves the lowest MAE among the compared methods in all 3 stages, demonstrating that its improvements are not confined to particular phases of the online evaluation.

(3) Comparison with STBP.

We have included STBP (ICLR 2026) in the revised Table 2, following the same online evaluation protocol as the other baselines. Compared with STBP, CoEvoST achieves lower MAE across all four datasets: 1.04 vs. 1.50 on ENERGY, 13.69 vs. 14.24 on PEMS, 17.43 vs. 17.84 on AIR, and 13.61 vs. 14.95 on CAMELS-13.

For Q1: Case Study of Continual Agent Evolution

To verify whether online agent evolution improves adaptation decisions, we examine an actual case from CAMELS-13, comparing the agent's decisions before and after online DPO under the same diagnostic state.

(1) Environmental diagnosis. During October 28–November 26, 1995, the recent rolling MAE increased by 105.48% relative to the initial reference. The diagnostics identified substantial data distribution shift (MMD² = 0.1537) and environmental covariate shift (MMD² = 0.1678), while the graph structure remained unchanged. Historical input similarity was high (0.9806), but the corresponding future outcomes exhibited substantial dispersion (0.9166), suggesting that similar historical conditions did not necessarily imply consistent future behavior.

(2) Evolution of adaptation decisions. Given the same diagnostic state, the agent selected Standard LoRA before DPO but switched to Progressive LoRA after DPO. Both strategies perform lightweight global adaptation using recent data, but differ in their adaptation mechanisms. Standard LoRA updates existing adapter parameters, whereas Progressive LoRA freezes previous adapters and introduces a new adapter to capture emerging patterns.

Under the observed distributional and environmental shifts, Progressive LoRA may better accommodate new dynamics while reducing interference with previously learned adaptation parameters. This provides a plausible explanation for the evolved strategy selection.

(3) Execution outcomes. Starting from the same predictor checkpoint, we compare the recorded candidate outcomes on the same subsequent evaluation window (November 27–December 31, 1995):

Adaptation StrategyMAERMSE
No Update3.249973.62507
Before DPO: Standard LoRA3.240573.61560
After DPO: Progressive LoRA3.159913.53666

The strategy selected after DPO achieves a 2.49% MAE reduction compared with the pre-DPO selection, without triggering rollback.

This case demonstrates that online DPO can change the agent's adaptation mechanism under the same diagnostic state and improve its forecasting outcome. Together with the paired before-and-after DPO comparisons in Figure 4 and the SFT-only ablation in Table 3, these results provide complementary evidence for the effectiveness of experience-driven agent evolution.


Regarding presentation clarity, we have further clarified the LLM's role in adaptation planning, provided more detailed explanations of the decision-making process, and included additional experimental analyses in the appendix.


Finally, we sincerely appreciate your recognition of our dual-evolution framework and your constructive suggestions. We hope our additional experiments and clarifications have addressed your concerns and would be grateful if you could reconsider your assessment in light of these results.

Reviewer_Te6VOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptation decisions for continual spatiotemporal forecasting. An LLM uses diagnostic evidence to determine whether to update the predictor, which data to use, and which adaptation mechanism to apply. Execution outcomes from candidate strategies are then converted into near-optimal preferences for online field-weighted DPO. Experiments on four streams show overall improvements over fixed update rules and the evaluated continual forecasting methods.

Critical strengths

S1. The problem is meaningful and empirically motivated. Different operating conditions may require different adaptation strategies. Figure 2 shows that candidate rankings change across stages, supporting the motivation for learning state-dependent adaptation decisions.

S2. The method achieves strong forecasting results across multiple domains. Compared with the best continual forecasting baseline on each dataset in Table 2, CoEvoST reduces MAE by approximately 12.6%, 14.1%, and 13.6% on ENERGY, PEMS, and CAMELS-13, respectively.

S3. Candidate comparison and online feedback are carefully designed. Candidate strategies start from the same predictor checkpoint and are evaluated on the same subsequent window. Feedback affects only future decisions, while validation and rollback limit the lasting effects of unsuccessful updates. The improvements over SFT-only in Table 3 also support the value of online policy updates.

Critical weaknesses

W1. The distinct value of the LLM over a conventional learned policy remains insufficiently established. The execution space contains only nine predefined strategies, with data selection and adaptation scope further constrained by the mechanism. The core task therefore largely amounts to selecting from a finite action set. Although a neural policy network baseline is included, its architecture, training supervision, and access to detailed diagnostics and historical experience are unclear. The current comparison cannot isolate the benefit of LLM reasoning from differences in information access and training.

W2. The progressive evidence planning ablation confounds information availability with acquisition strategy. Summary-only planning also removes detailed evidence. Its degradation therefore shows that summaries may be insufficient, rather than establishing that multi-round retrieval outperforms providing evidence in a single round. In Figure 3, the harness retrieves mandatory fields, and the selected strategy remains unchanged, providing limited evidence for active retrieval and plan refinement. A single-round planner with complete evidence, together with measurements of information overhead and strategy revision frequency, would help isolate this contribution.

W3. Computational costs are not quantified sufficiently to support online deployment. The framework combines multi-round planning with an 8B LLM, execution of multiple candidate branches, and online DPO. However, the paper does not report total runtime, GPU memory usage, exploration frequency, or the cost of individual components, nor does it compare against simpler policies under matched computational budgets. The qualitative discussion of overhead is insufficient to assess whether the forecasting gains justify the additional computation.

W4. Some improvements are small and lack uncertainty estimates. Results report only averages over three random seeds, without standard deviations or confidence intervals. For example, MAE decreases from 13.98 to 13.69 over Periodic on PEMS and from 17.56 to 17.43 over Static on AIR. Paired differences across runs or confidence intervals that account for temporal dependence are needed to establish whether these gains are reliable.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response: LLM Value, Progressive Evidence Planning, and Computational Efficiency

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the important concerns raised regarding our methodology and experimental results.


For W1: Distinct Value of the LLM over Conventional Decision Policies

We conducted additional comparisons with three representative controllers: Multi-Armed Bandit (UCB), Contextual Bandit (LinUCB), and Neural Policy Network. All methods operate within the same adaptation strategy space and follow the same online evaluation protocol and reward definition.

UCB selects strategies based on historical rewards. Both LinUCB and Neural Policy Network receive all numerical diagnostic information from the summaries and detailed evidence. LinUCB employs a linear contextual bandit, while Neural Policy Network uses MLP to learn a nonlinear mapping from diagnostic features to adaptation strategies. Both aim to maximize the expected reward of the selected strategy.

Decision PolicyENERGYPEMSCAMELS-13
Multi-Armed Bandit (UCB)2.3917.2618.87
Contextual Bandit (LinUCB)1.4115.9315.22
Neural Policy Network1.2315.0314.72
CoEvoST1.0413.6913.61

Compared with the strongest conventional controller, Neural Policy Network, CoEvoST reduces MAE by 15.45%, 8.92%, and 7.54% on ENERGY, PEMS, and CAMELS-13, respectively. These results demonstrate the effectiveness of the complete LLM-based decision framework relative to conventional learned controllers with access to the same numerical diagnostic information.

This advantage may stem from CoEvoST's ability to reason about the semantic relationships between diagnostic evidence and adaptation strategies, selectively acquire relevant information, and refine its decisions through accumulated execution experience, rather than relying solely on numerical feature-to-strategy mappings.


For W2: Contribution of Progressive Evidence Planning

We first clarify that the original Summary-only Planning ablation evaluates whether compact diagnostic summaries alone provide sufficient information for adaptation decisions, rather than isolating the contribution of multi-round evidence retrieval.

To address the reviewer's concern, we conducted an additional Single-round Full Evidence experiment (w/o Progressive). This variant uses the same Qwen3-8B agent and adaptation strategy space as CoEvoST, but provides all diagnostic summaries and detailed evidence in a single round, without subsequent retrieval or plan refinement.

We conduct this comparison on ENERGY and CAMELS-13, where the complete evidence fits within the model's native context window:

MethodENERGYCAMELS-13
Summary-only Planning1.4215.25
w/o Progressive (Single-round Full Evidence)1.2714.32
CoEvoST (Progressive Planning)1.0413.61

Single-round Full Evidence outperforms Summary-only Planning, confirming the value of detailed diagnostics. More importantly, CoEvoST further reduces MAE by 18.11% on ENERGY and 4.96% on CAMELS-13 compared with Single-round Full Evidence, demonstrating the additional benefits of progressive evidence planning beyond information availability.

For PEMS and AIR, the complete diagnostic evidence contains 33,237 and 48,669 tokens, respectively, exceeding Qwen3-8B's native 32K-token context window. In contrast, ENERGY and CAMELS-13 require only 7,323 and 2,955 tokens. We therefore restrict the untruncated full-evidence comparison to these two datasets.

To further examine actual evidence retrieval and plan refinement, we analyzed the complete online decision records:

MetricENERGYPEMSAIRCAMELS-13Total
Decision points3596351,0612962,351
Decisions with detailed evidence retrieval3586351,0382962,327
Plan revisions31162937113
Revisions after evidence retrieval12821335
Plan revision rate (%)8.642.522.7312.504.81

Across 2,351 decision points, detailed evidence was retrieved in 2,327 cases (98.98%), while plans were revised in 113 cases (4.81%), including 35 revisions following detailed evidence retrieval. These results show that progressive evidence planning supports both plan verification and refinement.

Overall, the Single-round Full Evidence comparison isolates the benefits of progressive planning beyond detailed evidence availability, while the decision records confirm that evidence retrieval supports both plan verification and actual revisions.

For W3: Computational Overhead and Online Feasibility

We have expanded Appendix B.5 to quantify runtime, GPU memory usage, exploration frequency, and online DPO overhead.

CoEvoST separates Fast Predictor Adaptation from Slow Agent Evolution. During routine forecasting, the predictor operates at the batch level, while LLM planning occurs only at chunk-level decision points. The measured runtime is summarized below:

DatasetEAC (s/batch)STRAP (s/batch)CoEvoST (s/batch)LLM Planning (s/chunk)
ENERGY0.0005900.0014460.01255240.34
PEMS0.0006530.0031440.02414620.16
AIR0.0009610.0046850.02714521.03
CAMELS-130.0004090.0009560.00569420.34

CoEvoST, EAC, and STRAP all operate at the millisecond scale for batch-level inference, while CoEvoST's LLM planning is performed only at chunk-level decision points.

Candidate exploration is triggered at only 9.61%–28.19% of decision points, and online DPO occurs only 7–14 times per evaluation stream. Each DPO update takes approximately 61–76 minutes and runs in the background, allowing the predictor and existing agent to continue operating without interruption. GPU memory usage is approximately 18.2–18.9 GiB during online operation and 23.24 GiB during DPO training.

Importantly, even if a full DPO update were performed after every chunk, the total processing time on PEMS—including online processing (374.29 s), candidate exploration (7.53 s), and DPO (76 min)—would be approximately 82.4 minutes, substantially shorter than the 12-hour chunk interval.

These results demonstrate that CoEvoST maintains millisecond-level routine forecasting while computationally intensive agent evolution runs asynchronously without blocking online predictions, with sufficient computational capacity to complete agent updates between successive chunks under the evaluated setting.


For W4: Statistical Reliability of Performance Improvements

We have added means and standard deviations across random seeds in Table 2 in the uploaded revised manuscript, with revisions highlighted in red. CoEvoST achieves MAEs of 1.04 ± 0.04, 13.69 ± 0.06, 17.43 ± 0.07, and 13.61 ± 0.16 on ENERGY, PEMS, AIR, and CAMELS-13, respectively, indicating stable performance across different random seeds. Beyond MAE, CoEvoST also achieves substantial improvements in other metrics, reducing RMSE by 13.41% on ENERGY and 9.50% on PEMS, and MAPE by 12.65% on CAMELS-13, providing additional evidence of its effectiveness across evaluation metrics.


Finally, we sincerely appreciate your positive assessment of our work and your constructive feedback. We hope our additional experiments and analyses have addressed your remaining concerns and would be grateful for your continued support.

Thank you for your detailed responses. My concerns are resolved during this rebuttal period. I stand positive on this paper.

Thank you for your positive feedback and continued support. We sincerely appreciate your thoughtful comments and are glad that our responses have addressed your concerns.

Reviewer_3g5XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoEvoST proposes an LLM agent-based adaptation method to tackle continual spatiotemporal forecasting where data streams are evolving over time. The LLM agent (Qwen3 8B, initialized with LoRA-SFT) is given a compact diagnostic summary of the performance of the current predictor model (used for forecasting) and iteratively retrieves additional information to produce a structured adaptation plan for the predictor model. This adaptation is executed on the predictor and validated on the next chunk of data, where the adaptation will be rolled back if it decreases performance. On a slower timescale, the LLM agent is finetuned with DPO using a constructed “near-optimal” preference set with multiple candidates plans for different checkpoints. Experiments were done on four different forecasting data streams (ENERGY, PEMS, AIR, CAMEL-13) and compared with different strategies with the LLM agent and seven other baselines with ablations.

Critical strengths
  1. The motivation is clear and it is sound to have dynamic adaptation strategies for evolving data streams as applying the same adaptation rule is not always appropriate (shown in Figure 2) depending on the data stream and the phase of data.
  2. While having two timescales for adaptation is not entirely novel (with many works focusing on hierarchical methods) it is rather novel in this context and done well with multiple important tweaks to make it work (DPO on collected candidate set and the harness for the LLM with different hand engineered option). One could see the LLM as a replacement for a data scientist tasked to continuously update the forecasting model while learning to get better at their job at the same time.
Critical weaknesses
  1. There are too little concrete details in the main text with most of it relegated to the appendix. Some of the information in table 5,6,7 should be in the main text to provide some clarity of how the LLM interacts with the system. Right now, the lines to explain how the LLM updates the predictor is extremely scares in line 208.
  2. There are some inconsistencies with the results
  • On AIR and CAMELS-13 static predictor from Table 1 (MAE of 17.56 and 14.11 respectively) beats all seven baselines (MAE of 17.60 to 18.45 and 15.76 to 28.10 respectively). Hence, it is hard to distinguish the performance gain of the proposed method as the static baseline already surpasses other baselines. Is the gain due to different architectures or harnesses for the baseline predictor or is it due to the dual evolution architecture.
  • On CAMELS-13, every ablation from Table 3 (MAE of 14.38 to 16.93) is worse than the static predictor from Table 1 (MAE of 14.11). Highlighting the same problem as above.
  • On PEMS, results from EAC’s paper report the MAE of EAC, TrafficStream, STKEC and PECMP as 13.53, 14.22, 14.14 and 14.85 respectively while in Table 1, the same results for EAC, TrafficStream, STKEC and PECMP are 16.82, 16,63, 16.56, 15.93 respectively while CoEvoST reports MAE of 13.69. Why is there such a stark difference in reported results for a standard benchmark?
  • Similar inconsistency with the scale and ranking of the baselines from EAC’s paper vs CoEvoST. Highlighting the same problem as above.
  1. Lack of any standard deviation statistics especially when some of the performance gains in Table 2 are marginal. E.g. AIR dataset with MAE of 17.60 from TrafficStream vs MAE of 17.43 from CoEvoST
Questions for authors

As mentioned in the weaknesses.

  1. What predictor backbone do the baselines use and why is the static version of CoEvoST stronger than almost all baselines.
  2. Why do the PEMS/ENERGY baseline numbers differ so much from the reported values from EAC?
7 comments
AuthorsOpenReview ↗

Response: Methodological Clarity, Baseline Fairness, and Statistical Reliability

We thank the reviewer for the careful evaluation and constructive feedback. We appreciate the important concerns raised regarding our methodology and experimental results.

For W1: Methodological Clarity and LLM–Predictor Interaction

We agree that the main text should provide more concrete details to clarify how the LLM interacts with the forecasting system. Accordingly, we have expanded Section 4.1 of the uploaded revised manuscript, with all modifications highlighted in red:

  1. Diagnostic information: We provide representative diagnostic metrics and explain how the LLM receives diagnostic summaries and retrieves fine-grained evidence on demand.
  2. Strategy execution: We clarify that the LLM does not directly modify predictor parameters. Instead, it generates a structured adaptation plan, which is executed by a separate module using the corresponding adaptation algorithms. We also provide a concrete strategy example.
  3. Evidence constraints: We clarify that predefined evidence requirements ensure sufficient information for decision-making rather than directly determining strategy selection.

Since the core contribution of this work lies in adaptation planning and agent evolution rather than the design of individual diagnostic metrics or adaptation algorithms, we retain the complete implementation details in Appendix, balancing main-text clarity with space constraints.

For W2.1, W2.2, W3 and Q1: Strong Static Baselines, Backbone Fairness, and Statistical Reliability

The strong performance of Static relative to some continual learning methods is not an experimental inconsistency. Rather, it reflects differences in adaptation needs across data streams and the possibility that inappropriate updates may degrade forecasting performance.

(1) Different data characteristics lead to different benefits from continual adaptation.

ENERGY and PEMS exhibit pronounced daily periodicity, with periodic-band power ratios of 33.18% and 74.49%, respectively. For these datasets, relatively short update windows can cover the dominant daily cycle, allowing conventional continual updates to benefit from recent observations and often improve forecasting performance.

On CAMELS-13, continual updates based on limited seasonal patterns may impair previously learned knowledge, potentially explaining why some methods underperform Static. To investigate this, we analyze the dataset's temporal characteristics and actual update behavior. CAMELS-13 exhibits a pronounced annual cycle, with a periodic-band power ratio of 23.91%, while the training window used by Periodic covers only approximately half a year. We further examine 18–19 actual MAE-Triggered updates per seed during 1990–2014, comparing MAE before and after each update on the same fixed historical validation set. Updates using data concentrated in particular seasons tend to be associated with larger increases in historical MAE, with Spearman correlations of 0.466, 0.494, and 0.332 across three seeds. Meanwhile, AIR exhibits relatively stable patterns, and the performance differences among continual learning methods are generally small.

To examine whether the strong Static performance depends on a particular predictor architecture, we additionally evaluated static STID and STGNN predictors:

DatasetSTD-PLM Static MAESTID Static MAESTGNN Static MAE
AIR17.5617.8417.76
CAMELS-1314.1113.9715.65

Static predictors with different architectures remain competitive on AIR and CAMELS-13, indicating that the strong Static performance is not specific to the STD-PLM backbone.

Overall, continual updates do not necessarily improve forecasting performance and may even degrade it when applied inappropriately. The key is to determine when to update, which data to use, and which adaptation mechanism to apply based on the current state of the data stream. This is precisely the central motivation of CoEvoST.

(2) CoEvoST's improvements are not attributable to differences in predictor backbones.

We clarify that TrafficStream, STKEC, EAC, PECPM, STRAP, and ST-TTC all use the same STD-PLM backbone as CoEvoST in our experiments. These methods allow the forecasting backbone to be replaced within their original frameworks, making the use of a unified STD-PLM backbone consistent with their methodological designs. Static, Periodic, and Triggered also use STD-PLM. Therefore, the performance comparisons in Table 1 are conducted under a consistent predictor architecture, eliminating backbone differences as a confounding factor.

Furthermore, Appendix B.4 demonstrates the applicability of CoEvoST to different predictor architectures. We additionally evaluated CoEvoST with three backbones on AIR and CAMELS-13:

DatasetCoEvoST (STD-PLM)CoEvoST (STID)CoEvoST (STGNN)
AIR17.4317.5217.49
CAMELS-1313.6112.9714.12

These results demonstrate that CoEvoST achieves competitive forecasting performance across different predictor architectures, supporting its applicability beyond STD-PLM.

(3) Multi-seed evaluation and statistical reliability.

We have added multi-seed results to the main experimental tables, with means and standard deviations highlighted in red in Table 2. For example, CoEvoST achieves MAEs of 1.04 ± 0.04, 13.69 ± 0.06, 17.43 ± 0.07, and 13.61 ± 0.16 on ENERGY, PEMS, AIR, and CAMELS-13, respectively, indicating stable performance across different random seeds. On AIR, where the data patterns are relatively stable, the differences among baselines are generally small, while our method remains highly competitive. Beyond MAE, CoEvoST also achieves more substantial improvements in other metrics, reducing RMSE by 13.41% on ENERGY and 9.50% on PEMS, and MAPE by 12.65% on CAMELS-13, compared with the strongest baselines for each metric.

For W2.3, W2.4 and Q2: Differences from the Results Reported in EAC

The differences between our results and those reported in the original EAC paper primarily arise from different online learning and evaluation protocols.

(1) Different data splits and online evaluation procedures.

EAC divides the data within each period into training, validation, and test subsets using a 6:2:2 ratio. The model is evaluated only on the test subset after adaptation within the corresponding period, primarily measuring forecasting performance following period-specific adaptation.

In contrast, CoEvoST focuses on how a continuously operating forecasting system autonomously decides whether, when, and how to adapt. This objective requires an evaluation protocol that preserves the chronological nature of online forecasting, rather than relying on predefined period-wise training and evaluation procedures. Accordingly, we use a 4:1:5 split only for the initial period, supporting predictor initialization, diagnostic calibration, and agent training. Subsequent data arrive sequentially, and the model follows a predict-then-adapt protocol: predictions are made before the corresponding ground-truth labels become available, and adaptation is performed afterward. All online predictions are included in the evaluation.

This protocol more comprehensively evaluates forecasting stability and autonomous adaptation during continuous operation, aligning with the objectives of our work. Further details are provided in Appendix A.1.

(2) Different forecasting evaluation metrics.

EAC reports prediction errors at 3-, 6-, and 12-step horizons, together with their average, and the values cited by the reviewer correspond to its Avg. MAE. In contrast, our evaluation uses 12-step forecasting over the entire online stream. Consequently, the reported error values and method rankings should not be directly compared without accounting for these protocol differences.

(3) Reproduction under the original EAC protocol.

To further investigate the reported discrepancies, we conducted additional experiments using exactly the same datasets and EAC implementation code as in our main experiments, changing only the online training and evaluation protocol to match the original EAC paper. We repeated each experiment with three random seeds and report the 12-step forecasting results below:

DatasetMetricOriginal EACOur Reproduction (Mean ± Std)
PEMSMAE14.9215.013 ± 0.297
RMSE24.1724.229 ± 0.469
MAPE (%)20.8221.706 ± 1.092
ENERGYMAE5.155.313 ± 0.278
RMSE5.465.733 ± 0.307
MAPE (%)50.5554.857 ± 6.684

The reproduced results are generally consistent with those reported in the original EAC paper. Since the datasets and implementation remain unchanged, these results provide direct evidence that the reported discrepancies largely stem from differences in evaluation protocols rather than baseline implementation.

(4) Consistent evaluation protocol within our study.

We have carefully re-examined and rerun all baselines to ensure that they follow the same offline initialization procedure, online data availability constraints, and evaluation scope. We have also added means and standard deviations across multiple random seeds in the revised manuscript. These controlled settings enable fair comparisons among methods on the same continual online forecasting task.


Finally, we sincerely appreciate your recognition of our work and your constructive feedback. We hope our clarifications and additional analyses have addressed your concerns and would be grateful if you could reconsider your assessment in light of these revisions.

Reviewer_SQbfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual spatiotemporal forecasting under evolving data streams, where the data distribution, graph structure, active nodes, and external conditions may change over time. The main idea is to move beyond fixed update rules and let an LLM based agent decide whether the forecasting model should be adapted, what data should be used, where the adaptation should be applied, and which adaptation mechanism should be selected.

Critical strengths

The paper tackles an important practical issue in continual spatiotemporal forecasting. In real deployments, changes in node sets, graph structure, temporal patterns, and external conditions do not necessarily call for the same update rule, so the idea of treating adaptation as a state dependent decision problem is meaningful. I also like that the method does not only adapt the predictor, but also tries to improve the adaptation policy itself from observed outcomes.

The experimental results are fairly convincing at the high level. CoEvoST is compared with static, periodic, triggered, and several continual forecasting baselines across four datasets. The gains are especially clear on ENERGY, PEMS, and CAMELS-13, and the ablation studies give useful evidence that progressive evidence retrieval, online DPO, near optimal preference construction, and field weighting each contribute to the final performance. The rollback analysis is also helpful because it shows that not every adaptation attempt is beneficial.

Critical weaknesses

Some parts of the method are still underspecified, especially around the LLM agent and the construction of candidate plans. The paper says that candidate plans are executed from the same predictor checkpoint and that near optimal preference sets are constructed, but it is not fully clear how expensive this is in practice, how many candidates are evaluated, and how this scales when the strategy space becomes larger. Since the method relies on extra candidate execution and online DPO, the computational cost is an important part of the practical story.

I also found the evaluation somewhat dependent on the particular design choices of the diagnostic evidence and strategy space. These choices seem to encode a fair amount of task knowledge, but the paper could do more to separate the benefit of the LLM based planning from the benefit of a carefully engineered adaptation space. A stronger comparison to simpler non LLM controllers using the same evidence and candidate strategies would make the claimed advantage of the agent more convincing.

Questions for authors

How many candidate plans are evaluated per decision point in the main experiments, and what is the wall clock or training cost compared with the strongest continual baselines?

Can the authors clarify how much of the diagnostic evidence design and strategy compatibility rules are dataset specific? If these rules need substantial redesign for a new forecasting domain, the generality of the approach may be more limited.

What happens if the LLM planner is replaced by a simpler bandit or reinforcement learning controller that has access to the same compact summaries, retrieved evidence, and candidate strategy set?

4 comments
AuthorsOpenReview ↗

Response: Computational Cost, Generality, and Lightweight Controllers

We sincerely thank the reviewer for the insightful and constructive comments, which raise important questions and provide valuable perspectives for strengthening our work. We provide the following additional experiments and analyses.

For W1 and Q1: Computational Efficiency and Online Feasibility

First, we would like to clarify that CoEvoST consists of two processes operating at different timescales: Fast Predictor Adaptation and Slow Agent Evolution. The former handles continuously arriving data streams, where the predictor performs forecasting and the LLM plans predictor adaptation at each online decision point. The latter uses feedback from candidate strategy executions to optimize the agent's decision-making policy over a longer timescale. Importantly, agent evolution runs asynchronously in the background without interrupting continuous forecasting, allowing the existing predictor and agent to continue operating while the decision policy is being updated. Therefore, it is important to distinguish the computational costs of routine forecasting, adaptation planning, candidate exploration, and agent training.

(1) Forecasting inference and fast adaptation costs.

The predictor performs forward inference for each forecasting batch, whereas each online chunk corresponds to one decision point and invokes one LLM planning session. Each chunk contains multiple forecasting batches. Thus, the LLM is not involved in the forward computation of individual forecasting batches.

We measured the runtime of each component on an NVIDIA A40 GPU.

DatasetChunk DurationEAC (s/batch)STRAP (s/batch)CoEvoST predictor (s/batch)LLM Planning (s/chunk)Predictor Update (s/update)
ENERGY12 h0.0005900.0014460.01255240.340.44
PEMS12 h0.0006530.0031440.02414620.160.83
AIR24 h0.0009610.0046850.02714521.030.54
CAMELS-1330 days0.0004090.0009560.00569420.340.59

The forward inference times of our predictor, EAC, and STRAP are all at the millisecond scale. The additional LLM planning cost is incurred only at the chunk-level decision stage, with the end-to-end online workflow taking 23.55–374.29 seconds per chunk, substantially less than the corresponding nominal chunk arrival intervals.

(2) Actual frequency and cost of candidate exploration.

Beyond routine adaptation planning, the system selectively triggers candidate exploration based on prediction degradation, distribution shifts, structural changes, environmental changes, and periodic exploration conditions.

MetricENERGYPEMSAIRCAMELS-13
Decision points3596351061296
Exploration triggers8117910244
Exploration frequency (%)22.5628.199.6114.86
Avg. candidates / exploration8.858.988.818.91
Total candidate evaluations7171607899392
Exploration time (s/trigger)8.197.536.035.80

Candidate exploration occurs at only 9.61%–28.19% of decision points, with approximately nine candidate strategies evaluated per exploration event. The sampled execution time per exploration event is 5.80–8.19 seconds.

(3) Actual frequency and cost of slow agent evolution.

The outcomes of candidate exploration are accumulated as preference experience. DPO is triggered only when the online optimization conditions are satisfied, updating the agent's decision-making policy.

MetricENERGYPEMSAIRCAMELS-13
Number of online DPO update triggers111478

With 64 preference pairs and four training epochs, each DPO update takes approximately 61–76 minutes. Although this cost is substantially higher than that of predictor adaptation, DPO is performed infrequently and runs in the background, allowing the predictor and the current agent to continue operating normally without interruption.

(4) Online feasibility under a conservative computational budget.

We further consider a conservative scenario in which both candidate exploration and online DPO are triggered at every chunk. Taking PEMS as an example, routine online processing takes 374.29 seconds per chunk, candidate exploration requires approximately 7.53 seconds, and a full DPO update takes up to 76 minutes. Even when these costs are added together, the total is approximately 82.4 minutes, substantially shorter than the 12-hour chunk arrival interval.

In summary, CoEvoST separates forecasting, adaptation planning, and agent evolution across different timescales, enabling continuous prediction alongside background agent optimization. The measured runtimes demonstrate its computational feasibility under the evaluated online forecasting settings.

We have updated the revised manuscript accordingly, with all modifications highlighted in red and a detailed efficiency analysis provided in Appendix B.5.

For W2 and Q2: Generality of Diagnostic Evidence and Adaptation Strategies

CoEvoST is designed to assess the model's current operational state as comprehensively as possible, rather than relying on diagnostic rules specifically tailored to individual datasets. To this end, we construct diagnostic evidence from six general perspectives: prediction performance, distribution shifts, temporal changes, structural changes, environmental changes, and historical support. By retaining information from these complementary dimensions, the agent can comprehensively assess the current adaptation needs.

Similarly, the candidate strategy space is not customized for particular datasets. Instead, it consists of representative adaptation strategies drawn from existing continual learning approaches and remains consistent across all datasets.

Therefore, CoEvoST's performance improvements do not depend on dataset-specific diagnostic designs or customized adaptation strategy spaces. Moreover, the same framework achieves strong performance across four distinct domains: traffic, hydrology, air quality, and energy, demonstrating its applicability and generalizability.


For W2 and Q3: Comparison with Lightweight Decision Controllers

To further evaluate the benefits of the LLM agent over conventional decision-making approaches, we introduce three representative lightweight controllers: Multi-Armed Bandit (UCB), Contextual Bandit (LinUCB), and a Neural Policy Network.

For a fair comparison, all methods operate within the same space of nine candidate adaptation strategies, using the same online evaluation protocol and strategy reward definition. Specifically, UCB selects strategies based on historical rewards. Both LinUCB and Neural Policy Network receive all numerical diagnostic information from the summaries and detailed evidence, using contextual bandit learning and neural policy learning, respectively, to maximize the expected reward of the selected adaptation strategy.

Decision PolicyENERGYPEMSCAMELS-13
Multi-Armed Bandit (UCB)2.3917.2618.87
Contextual Bandit (LinUCB)1.4115.9315.22
Neural Policy Network1.2315.0314.72
CoEvoST1.0413.6913.61

The results show that conventional controllers can leverage diagnostic information to improve strategy selection. Nevertheless, CoEvoST consistently outperforms the strongest Neural Policy Network baseline, reducing MAE by 15.45%, 8.92%, and 7.54% on ENERGY, PEMS, and CAMELS-13, respectively.

This advantage highlights the potential benefits of CoEvoST's decision-making mechanism, which combines (1) semantic understanding of heterogeneous diagnostics and adaptation strategies, (2) decision-dependent evidence acquisition, (3) iterative plan evaluation and refinement, and (4) continual policy improvement through historical execution feedback. Together, these capabilities support more flexible adaptation decisions beyond directly learning state-to-strategy mappings.

Overall, the results provide empirical support for CoEvoST's LLM-based decision-making and evolution framework within the same predefined adaptation strategy space.


Finally, we sincerely appreciate your positive assessment of our motivation and experimental results. We hope our additional experiments and clarifications have addressed your remaining concerns and would be grateful for your continued support.

Area_Chair_Sqz4OpenReview ↗

Author-reviewer discussion period is ending soon

Dear Reviewer SQbf,

If you haven't done so, please read through authors' response and let authors know if you need further clarification or have more questions. Beginning 11/19, authors are not allowed to respond any more, so this is good time to review the response and ask more.

Thank you.

-AC

Dear Reviewer,

Thank you again for your valuable feedback. We would greatly appreciate it if you could let us know whether our responses and revisions have adequately addressed your concerns.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.