acceptodds
Under review as a conference paper at ICLR 2027

DiagMem: Experience-Grounded Diagnostic Memory for Agentic Time-Series Forecasting

Abstract

Agentic time series forecasting extends standalone forecasting by incorporating tool use, feedback, and historical experience into the forecasting process. However, reusing this experience to diagnose a new forecast requires relating verified error feedback from completed forecasts to a current forecast whose outcome is still unknown. We introduce DiagMem, an experience-grounded diagnostic-memory framework for agentic time series forecasting. Its key idea is to interpret disagreement among heterogeneous forecasters using verified error patterns from similar diagnostic states. DiagMem organizes completed forecasting cases into diagnostic prototypes and uses a large language model to summarize their aggregated forecast, disagreement, and error evidence as structured scores and semantic descriptions. For a new window, the historical input, base forecast, and cross-model disagreement define its diagnostic state, which guides prototype retrieval. A lightweight numerical executor combines the retrieved diagnostic knowledge with current numerical inputs to estimate trend and seasonal residual corrections while keeping the base forecaster frozen. Across seven multivariate benchmarks and four forecast horizons, DiagMem delivers broad improvements over diverse frozen base forecasters, while ablations support the contributions of cross-model disagreement, state-conditioned diagnostic memory, and the joint use of structured scores and semantic descriptions.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Agentic AI, Large Language Models, Retrieval-Augmented Learning, Error Correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bp3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiagMem, an experience-grounded diagnostic memory framework for time-series forecasting. It uses disagreement among heterogeneous frozen forecasters as diagnostic evidence, links this disagreement to verified historical forecast errors, and organizes past cases into state-conditioned prototypes. An LLM converts prototype-level evidence into structured scores and semantic descriptions, which are retrieved to condition a residual correction module for the base forecast. Experiments on seven multivariate datasets, four forecast horizons, and three base forecasters show consistent improvements over the corresponding uncorrected backbones.

Critical strengths

i) The proposed framework is well motivated and provides a coherent way to connect cross-model disagreement, historical forecast errors, and residual correction. This design offers an intuitive and technically sound mechanism for using historical forecasting experience to diagnose and refine new predictions.

ii) The experimental protocol is carefully designed to preserve temporal causality and reduce the risk of information leakage. In particular, the chronological split into memory, correction, and validation sets, together with horizon-sized purge gaps, provides a clean temporal evaluation protocol.

iii) The empirical evaluation is sufficiently broad, covering multiple datasets, forecasting horizons, and base forecasters. Specifically, the method is evaluated on seven datasets, four forecasting horizons, and three different base forecasters, providing evidence for its generality.

iv) The ablation study is comprehensive and provides clear evidence for the contribution of the main components. The Full model consistently outperforms variants that remove auxiliary forecasts, state-conditioned memory, or diagnostic representations, including across multiple backbones.

Critical weaknesses

i) The current experiments do not clearly isolate the benefit of diagnostic memory from the benefit of simply using multiple forecasters. Since the correction module directly uses information from the auxiliary forecasters, comparisons with simple ensemble or stacking baselines would help clarify whether the gains are specifically attributable to DiagMem.

ii) The specific contribution of the LLM-based diagnostic representation is not fully isolated by the current ablations. Although the Full model outperforms NumericCondition and PrototypeID, these variants modify multiple parts of the representation, making it difficult to determine how much of the improvement comes from the LLM-generated semantic information itself.

iii) The comparison with standalone forecasting baselines is not fully protocol-matched. DiagMem uses the calibration split for memory construction and correction training, whereas the standalone baselines mainly use the same split for model selection, which makes the comparison in Table 1 less direct.

iv) The experimental analysis provides limited evidence regarding statistical robustness and sensitivity to key retrieval hyperparameters. The main results do not report standard deviations or confidence intervals, and important design choices such as the number of prototypes and retrieved neighbors are fixed without extensive sensitivity analysis.

Questions for authors

i) Can the authors clarify whether the performance gains are mainly due to the diagnostic memory mechanism rather than simply using multiple forecasters? A comparison with simple ensemble or stacking baselines using the same base and auxiliary forecasters would help isolate the specific contribution of DiagMem.

ii) Can the authors provide a cleaner ablation to isolate the contribution of the LLM-based diagnostic representation? For example, replacing the LLM-generated semantic representation with a capacity-matched numerical or learned encoder while keeping the remaining components unchanged would make this contribution easier to assess.

iii) Can the authors clarify the fairness of the comparison with standalone forecasting baselines in Table 1? Since DiagMem uses the calibration split for memory construction and correction training, while the standalone baselines mainly use it for model selection, it would be helpful to discuss how this difference affects the comparison.

iv) Can the authors provide additional evidence on statistical robustness and sensitivity to key retrieval hyperparameters? Reporting run-to-run variance and evaluating different numbers of prototypes or retrieved neighbors would help demonstrate the stability of the reported improvements.

1 comment

We thank the reviewer for their thoughtful feedback and constructive suggestions. In response to the concerns, we have strengthened the experiments through (1) additional ensemble and memory-free numerical correction baselines, (2) more targeted ablations of LLM-generated diagnostic representations, (3) clarification of the training and calibration protocols, and (4) run-to-run variability and retrieval hyperparameter sensitivity analyses. The main revisions are summarized below.

  1. Contribution of diagnostic memory beyond forecast ensembles (W1, Q1)

This concern also overlaps with Reviewer 6THh (Q1) and Reviewer 581i (W6, Q3). We have added Simple Average and Ridge Stacking using the same three forecasters. DiagMem achieves better forecasting performance than both ensemble baselines on several datasets (Section 4.2, Table 2). To further isolate the contribution of diagnostic memory, we introduce MFRC and MFRCFullBudget, which perform multi-forecaster numerical residual correction without diagnostic memory. MFRCFullBudget additionally uses the memory subset for correction training. Full achieves better overall forecasting performance than both variants. The NoAux and GlobalMemory ablations further examine the contributions of auxiliary forecasts and state-conditioned memory, with Full outperforming both variants (Section 4.3, Table 3). These comparisons provide evidence that DiagMem's improvements extend beyond forecast combination and memory-free numerical correction.

  1. Contribution of LLM-generated diagnostic representations (W2, Q2)

This concern also overlaps with Reviewer 6THh's comments on LLM-based diagnostic memory and Reviewer 581i (W5, Q2). We have added three representation ablations to more directly isolate the contributions of LLM-generated diagnostic information. FullyNumerical replaces both LLM-derived branches with numerical representations, while LLMScoreOnly and LLMSemanticOnly retain only the LLM-generated structured scores or semantic descriptions, respectively, using numerical representations for the other branch. These variants preserve the diagnostic memory mechanism and share the same training protocol. Full achieves better overall forecasting performance than all three variants (Section 4.3, Table 3). These results provide more direct evidence for the complementary contributions of LLM-generated structured scores and semantic descriptions.

  1. Fairness of comparison with standalone forecasting baselines (W3, Q3)

We have clarified the training and calibration protocols in Section 4.1 and Appendix A. Standalone supervised baselines use the full training split for model fitting and the calibration split for checkpoint selection, while TimesFM and Chronos-2 are evaluated in the zero-shot setting. DiagMem instead uses an internal portion of the training split for forecaster selection and reserves the calibration split for memory construction, correction training, and validation. Both are evaluated on the same held-out test sets. Table 1 provides an overall comparison of forecasting performance under these respective protocols. To more directly assess the contribution of diagnostic memory, we additionally compare DiagMem with its uncorrected Backbone and numerical correction variants using the same forecasters, data partitions, and evaluation protocol (Section 4.3, Table 3). These comparisons provide complementary evidence for the effectiveness of the proposed framework.

  1. Statistical robustness and retrieval hyperparameter sensitivity (W4, Q4)

We have extended the experimental analysis to examine run-to-run variability and retrieval hyperparameter sensitivity. We report the mean and sample standard deviation of MSE and MAE over three runs for each DiagMem configuration at each dataset–horizon setting (Appendix A, Table 6).

We further evaluate prototype counts K∈{8,16,32}K\in\{8,16,32\} and retrieved neighbors k∈{1,3,5}k\in\{1,3,5\}, varying one parameter at a time. The default configuration (K=16,k=3)(K=16,k=3) achieves the lowest errors among the evaluated settings across all seven datasets (Section 4.3, Fig. 2). These results demonstrate the influence of prototype granularity and retrieval neighborhood size on forecasting performance.

Reviewer_581iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiagMem, an experience-grounded diagnostic memory framework for time-series forecasting. The method leverages disagreements among heterogeneous forecasters and historical forecasting errors to construct diagnostic prototypes, which are summarized by an LLM into structured scores and semantic descriptions. These diagnostic memories are then retrieved to guide a lightweight residual correction module while keeping the base forecaster frozen. Experiments on seven multivariate benchmarks across four prediction horizons demonstrate improvements over various forecasting backbones.

Critical strengths
  1. The paper presents an interesting approach to leveraging historical forecasting errors and cross-model disagreement for forecast correction. The offline construction of diagnostic memory avoids additional LLM inference costs during forecasting, making the framework practically appealing.
  2. The authors evaluate DiagMem on seven multivariate datasets, four prediction horizons, and three different forecasting backbones. The results demonstrate relatively consistent improvements over the frozen backbones.
Critical weaknesses
  1. The proposed framework is essentially a fixed forecasting and residual correction pipeline. The memory is constructed offline, and there is no adaptive tool selection or online memory updating. Its distinction from conventional retrieval-augmented forecasting is not sufficiently justified.
  2. The method assumes that cross-model disagreement can provide useful diagnostic signals for forecasting errors. However, disagreement does not necessarily indicate the direction or magnitude of prediction errors, and the paper provides limited empirical evidence supporting this relationship.
  3. The framework mainly combines existing techniques, including multi-model forecasting, prototype retrieval, LLM-based summarization, and residual correction. The key methodological differences from prior work such as CTRL and RATL are not sufficiently demonstrated.
  4. The choices of K-means clustering, manually defined state statistics, and LLM-generated diagnostic scores appear largely heuristic. It remains unclear why these representations are preferable to directly learned diagnostic features.
  5. The LLM summarizes numerical statistics into discrete scores and text embeddings, but the paper does not convincingly establish what additional information this process provides. Simpler numerical or rule-based alternatives are not adequately investigated.
  6. The ablation studies do not fully isolate the effects of auxiliary forecasts, diagnostic memory, and LLM reasoning. In particular, comparisons with simple forecast averaging, learned ensembles, and multi-forecaster residual correction without memory are missing.
  7. Although multiple benchmarks are evaluated, the experiments focus on a fixed set of forecasting models and standard datasets. The robustness of the diagnostic memory under distribution shifts, different auxiliary model combinations, and evolving forecasting environments remains unexplored.
Questions for authors
  1. Can the authors provide stronger empirical evidence that cross-model disagreement reliably reflects the direction and magnitude of forecasting errors?
  2. What unique benefits do LLM-generated diagnostic scores and semantic descriptions provide over simpler numerical statistics, rule-based methods, or directly learned representations?
  3. How does DiagMem compare with simple forecast averaging, learned ensembles, or multi-forecaster residual correction without diagnostic memory? Can the authors better isolate the contributions of each component?
1 comment

We thank the reviewer for their valuable suggestions and comments. We have revised the manuscript to clarify its methodological contributions and strengthened the empirical evaluation through additional numerical correction baselines, ensemble comparisons, diagnostic representation ablations, sensitivity analyses, and correction-reliability evaluations. We address the related concerns together below.

  1. Agentic positioning and methodological distinctions (W1, W3)

DiagMem focuses on experience-grounded diagnosis within a forecasting-tool workflow, connecting cross-model disagreement with verified historical errors through state-conditioned diagnostic memory. We have clarified its distinctions from CTRL and RATL in Section 2. CTRL derives component-wise control signals from historical error analysis, while RATL combines retrieved residual trajectories through learned routing. In contrast, DiagMem retrieves diagnostic evidence to condition numerical residual estimation. Offline memory construction also avoids test-time LLM inference.

  1. Diagnostic value of cross-model disagreement (W2, Q1)

We have added a quantitative diagnostic analysis using ridge probes to estimate residual direction and magnitude from cross-model disagreement statistics. Compared with a historical-mean reference, disagreement-based diagnosis improves the overall estimation of both error direction and magnitude. Incorporating state-conditioned historical evidence further increases average directional balanced accuracy from 0.597 to 0.600 and reduces magnitude estimation MAE from 0.242 to 0.238 (Appendix, Quantitative Diagnostic Analysis). These results support the diagnostic value of cross-model disagreement when interpreted alongside verified historical errors.

  1. Diagnostic representation design and LLM contributions (W4, W5, Q2)

The diagnostic state summarizes historical dynamics, base forecast characteristics, and auxiliary disagreements using statistics available at inference time, while K-means organizes comparable historical cases into diagnostic prototypes.

To evaluate the diagnostic representations, we introduce FullyNumerical, LLMScoreOnly, and LLMSemanticOnly. Full achieves better overall forecasting performance than all three variants (Section 4.3, Table 3). These results support the complementary contributions of LLM-generated structured scores and semantic descriptions. We also evaluate prototype counts K∈{8,16,32}K\in\{8,16,32\} and retrieved neighbors k∈{1,3,5}k\in\{1,3,5\}. The default configuration (K=16,k=3)(K=16,k=3) achieves the lowest errors among the evaluated settings across all seven datasets (Section 4.3, Fig. 2).

  1. Component contributions and ensemble comparisons (W6, Q3)

This concern also overlaps with Reviewer 6THh's question on memory-free numerical correction and Reviewer bp3t (Wi, Qi). We introduce Simple Average and Ridge Stacking using the same three forecasters. DiagMem configurations achieve lower four-horizon average MSE than both ensembles on four of the seven datasets, while Simple Average performs better on ETTh1 and ETTm1, and Ridge Stacking on ECL (Section 4.2, Table 2).

We further introduce MFRC and MFRCFullBudget to evaluate multi-forecaster numerical residual correction without diagnostic memory. Full achieves better overall forecasting performance than MFRC and MFRCFullBudget. The NoAux and GlobalMemory ablations further isolate the contributions of auxiliary forecasts and state-conditioned memory, with Full outperforming both variants (Section 4.3, Table 3). Together with the representation ablations above, these comparisons clarify the contributions of the main components.

  1. Robustness across forecasting configurations and states (W7)

This concern also relates to Reviewer 6THh's comments on historical memory applicability.

We evaluate DiagMem with three heterogeneous forecasters, each serving as the base model while the other two provide auxiliary forecasts. DiagMem improves upon the corresponding frozen backbones in 78 of 84 dataset–horizon settings (Section 4.2).

We further analyze correction outcomes across retrieval-distance quartiles. Even in the farthest quartile, DiagMem maintains positive mean MSE reductions on six of the seven datasets, suggesting that historical diagnostic evidence can remain useful beyond closely matched states (Appendix, Quantitative Diagnostic Analysis). Future work will investigate adaptive memory updating, selective retrieval, and dynamic forecaster selection under evolving forecasting environments.

Reviewer_6THhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DiagMem uses historical forecasting errors to help correct new forecasts. It groups past cases into prototypes, asks an LLM to summarize their error patterns, and retrieves these summaries to guide a numerical correction network.

Critical strengths

The paper addresses a useful problem: learning from past mistakes without retraining the base forecaster.

Critical weaknesses

The novelty seems limited. The method combines clustering, retrieval, LLM-generated summaries, and residual learning. It is unclear what this adds beyond existing retrieval-based correction methods. In particular, the paper needs stronger evidence that converting numerical evidence into text and then encoding it back into vectors is better than directly learning from that evidence.

The method also assumes that past error patterns remain useful for new forecasts. This may hold for familiar situations, but it is less clear when the series changes or an unusual event occurs. The system retrieves the nearest prototypes even when none is a good match, without an explicit mechanism to reject irrelevant memories. Average test-set improvements do not show whether it avoids harmful corrections in these cases.

Questions for authors

How does DiagMem compare with a numerical correction network that uses the same history and forecasts from all three models, with the same supervision and a comparable training budget, but without LLM-based diagnostic memory? This comparison would help clarify how much the diagnostic memory adds.

1 comment

We thank the reviewer for their thoughtful comments and feedback. In response to the concerns, we have strengthened the manuscript by (1) clarifying the methodological distinctions from existing retrieval-based correction methods, (2) introducing numerical correction baselines and representation ablations, and (3) adding window-level analyses of retrieval distance and correction reliability. The main revisions are summarized below.

  1. Methodological novelty and diagnostic representations

This concern also overlaps with Reviewer 581i (W3, W5, Q2). We have clarified the methodological distinctions from CTRL and RATL in Section 2. While CTRL derives component-wise control signals from historical error analysis and RATL combines retrieved residual trajectories through learned routing, DiagMem organizes the relationships between cross-model disagreement and verified base errors into state-conditioned diagnostic prototypes. The retrieved diagnostic evidence then conditions numerical residual estimation.

To examine the contribution of LLM-generated representations, we introduce FullyNumerical, which replaces both LLM-derived diagnostic branches with numerical representations while retaining the memory mechanism. Full achieves better overall forecasting performance than FullyNumerical, with an average relative MSE reduction of 2.43% (Section 4.3, Table 3). These results support the benefit of LLM-generated diagnostic representations over the evaluated numerical alternative.

  1. Historical memory applicability and correction reliability

This concern also relates to Reviewer 581i (W7). DiagMem uses retrieved diagnostic evidence to condition a learned residual executor rather than directly applying historical residuals to new forecasts. The diagnostic evidence includes error-pattern reusability and evidence consistency scores, while a validation-selected coefficient $\beta$ controls the final correction magnitude.

We further examine correction outcomes across retrieval-distance quartiles, reporting both window-level MSE improvements and harmful correction frequencies (Appendix, Quantitative Diagnostic Analysis). Even in the farthest quartile, DiagMem achieves positive mean MSE reductions on six of the seven datasets. ETTh1 is the exception, with a mean reduction of -0.0072 and harmful corrections in 53.1% of windows. These findings suggest that historical diagnostic evidence can remain useful beyond closely matched states. Future work will explore selective retrieval and correction abstention under distribution shifts.

  1. Comparison with memory-free numerical correction

This question also overlaps with Reviewer 581i (W6, Q3) and Reviewer bp3t (Wi, Qi). We introduce two additional numerical correction baselines using the same three forecasters without diagnostic memory. MFRC performs multi-forecaster numerical residual correction, while MFRCFullBudget additionally uses the memory subset for correction training. The correction variants share the same data partitions, optimization settings, hidden width, supervision objectives, and validation criterion.

Full achieves better overall forecasting performance than MFRC and MFRCFullBudget across the evaluated dataset–horizon settings (Section 4.3, Table 3). These comparisons provide direct evidence that the gains extend beyond multi-forecaster numerical correction and the additional use of historical supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.