- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DiagMem, an experience-grounded diagnostic memory framework for time-series forecasting. It uses disagreement among heterogeneous frozen forecasters as diagnostic evidence, links this disagreement to verified historical forecast errors, and organizes past cases into state-conditioned prototypes. An LLM converts prototype-level evidence into structured scores and semantic descriptions, which are retrieved to condition a residual correction module for the base forecast. Experiments on seven multivariate datasets, four forecast horizons, and three base forecasters show consistent improvements over the corresponding uncorrected backbones.
Critical strengths
i) The proposed framework is well motivated and provides a coherent way to connect cross-model disagreement, historical forecast errors, and residual correction. This design offers an intuitive and technically sound mechanism for using historical forecasting experience to diagnose and refine new predictions.
ii) The experimental protocol is carefully designed to preserve temporal causality and reduce the risk of information leakage. In particular, the chronological split into memory, correction, and validation sets, together with horizon-sized purge gaps, provides a clean temporal evaluation protocol.
iii) The empirical evaluation is sufficiently broad, covering multiple datasets, forecasting horizons, and base forecasters. Specifically, the method is evaluated on seven datasets, four forecasting horizons, and three different base forecasters, providing evidence for its generality.
iv) The ablation study is comprehensive and provides clear evidence for the contribution of the main components. The Full model consistently outperforms variants that remove auxiliary forecasts, state-conditioned memory, or diagnostic representations, including across multiple backbones.
Critical weaknesses
i) The current experiments do not clearly isolate the benefit of diagnostic memory from the benefit of simply using multiple forecasters. Since the correction module directly uses information from the auxiliary forecasters, comparisons with simple ensemble or stacking baselines would help clarify whether the gains are specifically attributable to DiagMem.
ii) The specific contribution of the LLM-based diagnostic representation is not fully isolated by the current ablations. Although the Full model outperforms NumericCondition and PrototypeID, these variants modify multiple parts of the representation, making it difficult to determine how much of the improvement comes from the LLM-generated semantic information itself.
iii) The comparison with standalone forecasting baselines is not fully protocol-matched. DiagMem uses the calibration split for memory construction and correction training, whereas the standalone baselines mainly use the same split for model selection, which makes the comparison in Table 1 less direct.
iv) The experimental analysis provides limited evidence regarding statistical robustness and sensitivity to key retrieval hyperparameters. The main results do not report standard deviations or confidence intervals, and important design choices such as the number of prototypes and retrieved neighbors are fixed without extensive sensitivity analysis.
Questions for authors
i) Can the authors clarify whether the performance gains are mainly due to the diagnostic memory mechanism rather than simply using multiple forecasters? A comparison with simple ensemble or stacking baselines using the same base and auxiliary forecasters would help isolate the specific contribution of DiagMem.
ii) Can the authors provide a cleaner ablation to isolate the contribution of the LLM-based diagnostic representation? For example, replacing the LLM-generated semantic representation with a capacity-matched numerical or learned encoder while keeping the remaining components unchanged would make this contribution easier to assess.
iii) Can the authors clarify the fairness of the comparison with standalone forecasting baselines in Table 1? Since DiagMem uses the calibration split for memory construction and correction training, while the standalone baselines mainly use it for model selection, it would be helpful to discuss how this difference affects the comparison.
iv) Can the authors provide additional evidence on statistical robustness and sensitivity to key retrieval hyperparameters? Reporting run-to-run variance and evaluating different numbers of prototypes or retrieved neighbors would help demonstrate the stability of the reported improvements.
We thank the reviewer for their thoughtful feedback and constructive suggestions. In response to the concerns, we have strengthened the experiments through (1) additional ensemble and memory-free numerical correction baselines, (2) more targeted ablations of LLM-generated diagnostic representations, (3) clarification of the training and calibration protocols, and (4) run-to-run variability and retrieval hyperparameter sensitivity analyses. The main revisions are summarized below.
This concern also overlaps with Reviewer 6THh (Q1) and Reviewer 581i (W6, Q3). We have added Simple Average and Ridge Stacking using the same three forecasters. DiagMem achieves better forecasting performance than both ensemble baselines on several datasets (Section 4.2, Table 2). To further isolate the contribution of diagnostic memory, we introduce MFRC and MFRCFullBudget, which perform multi-forecaster numerical residual correction without diagnostic memory. MFRCFullBudget additionally uses the memory subset for correction training. Full achieves better overall forecasting performance than both variants. The NoAux and GlobalMemory ablations further examine the contributions of auxiliary forecasts and state-conditioned memory, with Full outperforming both variants (Section 4.3, Table 3). These comparisons provide evidence that DiagMem's improvements extend beyond forecast combination and memory-free numerical correction.
This concern also overlaps with Reviewer 6THh's comments on LLM-based diagnostic memory and Reviewer 581i (W5, Q2). We have added three representation ablations to more directly isolate the contributions of LLM-generated diagnostic information. FullyNumerical replaces both LLM-derived branches with numerical representations, while LLMScoreOnly and LLMSemanticOnly retain only the LLM-generated structured scores or semantic descriptions, respectively, using numerical representations for the other branch. These variants preserve the diagnostic memory mechanism and share the same training protocol. Full achieves better overall forecasting performance than all three variants (Section 4.3, Table 3). These results provide more direct evidence for the complementary contributions of LLM-generated structured scores and semantic descriptions.
We have clarified the training and calibration protocols in Section 4.1 and Appendix A. Standalone supervised baselines use the full training split for model fitting and the calibration split for checkpoint selection, while TimesFM and Chronos-2 are evaluated in the zero-shot setting. DiagMem instead uses an internal portion of the training split for forecaster selection and reserves the calibration split for memory construction, correction training, and validation. Both are evaluated on the same held-out test sets. Table 1 provides an overall comparison of forecasting performance under these respective protocols. To more directly assess the contribution of diagnostic memory, we additionally compare DiagMem with its uncorrected Backbone and numerical correction variants using the same forecasters, data partitions, and evaluation protocol (Section 4.3, Table 3). These comparisons provide complementary evidence for the effectiveness of the proposed framework.
We have extended the experimental analysis to examine run-to-run variability and retrieval hyperparameter sensitivity. We report the mean and sample standard deviation of MSE and MAE over three runs for each DiagMem configuration at each dataset–horizon setting (Appendix A, Table 6).
We further evaluate prototype counts K∈{8,16,32} and retrieved neighbors k∈{1,3,5}, varying one parameter at a time. The default configuration (K=16,k=3) achieves the lowest errors among the evaluated settings across all seven datasets (Section 4.3, Fig. 2). These results demonstrate the influence of prototype granularity and retrieval neighborhood size on forecasting performance.