- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to provide a method called Probabilistic Agentic Memory (PAM), which learns how outcomes based evidence for persistent skill support for predictive forecasting tasks. Since each skill use can impact the future prediction, PAM helps agent learn "what an outcome implies for the future reuse of a skill." They evaluate PAM over three epidemic-forecasting tasks with a 15-skill bank retrieved from 31 candidates and show that it outperforms three agent setup using the same 31-skill library on all tasks and over same-bank.
Overall, the paper tackles an interesting problem; however, in its current form, it appears premature. Unclear research direction (poorly written introduction and abstract), incomplete formulation, and lack of proper evaluation makes it hard to evaluate the proper contribution of the work.
Critical strengths
The problem identified by the authors seems interesting but the overall paper seems premature.
Critical weaknesses
Major:
-
The evaluation seems unfair. The method proposed by the authors uses RMSE as regularizer in Record step and as optimizer in the location head and contextual proposal part. Furthermore, the model checkpoints are selected using a weighted combination of standardized RMSE, WIS, and NLL on validation set. Now, if RMSE and WIS is used in selecting and training of model, it seems unfair to compare the baselines on these metrics as they were never trained on them. And more generally speaking, the optimizing objective should not be used to evaluate as that is a circular evaluation.
-
The evaluation seems limited and incomplete. The authors have compared with some old agentic setup like React. Evaluation with more newer agents, like self-evolving harness, will explain the usefulness of the method. Furthermore, the standard deviations are missing for the baselines.
-
It is also unclear as to what models were used in the agentic system and what skills look like in the author's setup. Without this, it is hard to evaluate if the proposed framework is useful for more recent models.
Minor:
- The overall writing is poor, making it hard to understand the work in one go. It took me several reads to understand the problem statement the authors are working on.
- "Sec 3: Problem Formulation" is missing. Only the problem setup is described.
Questions for authors
See Weakness.
3 comments
Response to Reviewer 7ML9, Part 2
As a separate check, we tested whether access to historical prediction errors explains the performance gap. Agent prompts included the past RMSE of all 31 models, computed from the latest four batches with available target values. The LSTM uses either its original inputs or additional inputs containing its past RMSE and a flag showing whether that error is available. Even with these inputs, PAM has lower mean RMSE and WIS on all three tasks; Table R2 reports WIS.
Table R2. Baselines with and without historical prediction errors
| Method | Configuration | Bench-A WIS ↓ | B-COVID WIS ↓ | B-FLU WIS ↓ |
|---|---|---|---|---|
| PAM | Full | 0.0914 ± 0.0008 | 0.1513 ± 0.0050 | 0.4449 ± 0.0268 |
| Agent Top-1 | Original inputs | 0.1101 ± 0.0000 | 0.2267 ± 0.0000 | 0.4898 ± 0.0000 |
| Agent Top-1 | With past RMSE | 0.1113 ± 0.0047 | 0.2403 ± 0.0000 | 0.7411 ± 0.0000 |
| Agent Rediscovery | Original inputs | 0.1099 ± 0.0006 | 0.2401 ± 0.0060 | 0.5886 ± 0.0533 |
| Agent Rediscovery | With past RMSE | 0.1201 ± 0.0003 | 0.2001 ± 0.0014 | 0.7545 ± 0.0179 |
| Agent ReAct | Original inputs | 0.1110 ± 0.0013 | 0.2405 ± 0.0044 | 0.5408 ± 0.0056 |
| Agent ReAct | With past RMSE | 0.1212 ± 0.0011 | 0.1990 ± 0.0026 | 0.8172 ± 0.0268 |
| LSTM | Original inputs | 0.1143 ± 0.0011 | 0.2510 ± 0.0008 | 0.7426 ± 0.0004 |
| LSTM | With past RMSE | 0.1143 ± 0.0011 | 0.2507 ± 0.0010 | 0.7403 ± 0.0007 |
Additional results; mean ± sample SD over three runs. Agent rows vary decoding seeds; LSTM and PAM rows vary training seeds. Both LSTM variants reserve two additional input channels: zero-filled in “Original inputs”, or containing past RMSE and its availability flag in “With past RMSE”. PAM uses the three original fits. WIS is normalized by training-target SD, as in the submitted Table 1; lower is better. WIS jointly measures prediction-interval width and missed targets.
Q2a: “The evaluation seems limited and incomplete.” “The authors have compared with some old agentic setup like React.” “Evaluation with more newer agents, like self-evolving harness, will explain the usefulness of the method.”
Answer: Good suggestion. We evaluated Agentic Context Engineering (ACE; Zhang et al., ICLR 2026). ACE updates its context playbook through generation, reflection and curation. On Bench-A, we tested all four combinations of iterative regeneration and semantic deduplication, providing feedback only after the true target values became available.
Table R3. ACE configurations on Bench-A
| Method | RMSE ↓ | WIS ↓ | Cov90 |
|---|---|---|---|
| PAM | 0.3347 ± 0.0162 | 0.0914 ± 0.0008 | 0.9230 ± 0.0137 |
| ACE basic reflection and curation | 0.4557 ± 0.0003 | 0.1361 ± 0.0002 | 0.5110 ± 0.0018 |
| ACE + iterative regeneration | 0.4242 ± 0.0564 | 0.1335 ± 0.0045 | 0.5378 ± 0.0295 |
| ACE + semantic deduplication | 0.4680 ± 0.0113 | 0.1273 ± 0.0043 | 0.6104 ± 0.0420 |
| ACE + both mechanisms | 0.4510 ± 0.0082 | 0.1336 ± 0.0044 | 0.5265 ± 0.0260 |
Additional results; mean ± sample SD over three sampled ACE runs and three PAM fits. RMSE and WIS are normalized as in Table 1. Cov90 is the fraction of targets covered by 90% prediction intervals; its target is 0.90. Bold marks lower errors or coverage closer to that target.
PAM substantially outperforms all four ACE configurations in RMSE and WIS, with coverage much closer to the 90% target (Table R3). This consistent advantage suggests a benefit from PAM’s learned probabilistic scoring, which ACE does not include. ACE updates textual memory; PAM also learns model bias and uncertainty to produce comparable log scores for updating model weights (Sections 4.3–4.4). Table 2B supports this explanation: learned scoring improves later predictions even when the forecast distributions being combined are identical.
Q2b: “Furthermore, the standard deviations are missing for the baselines.”
Answer: Thank you for highlighting the need to report variation across runs. The original LLM-agent baselines used deterministic greedy decoding: sampling was disabled, and changing the decoding seed produced identical results. The single-value reporting therefore reflects the deterministic setup, not missing repeated sampling runs (Table 1; Appendix C.2, pp. 16–17). PAM’s SD measures variation across three independent training runs. Following your suggestion, we also enabled agent sampling with temperature 0.2, top-p 0.9, top-k 20 and seeds 42, 43 and 44. Tables R2–R3 report mean ± SD for every tested agent configuration, alongside the training SDs for PAM and LSTM.
Response to Reviewer 7ML9, Part 3
Q3: “It is also unclear as to what models were used in the agentic system and what skills look like in the author's setup.” “Without this, it is hard to evaluate if the proposed framework is useful for more recent models.”
Answer: The foundation model is Qwen2.5-7B-Instruct, used for retrieval embeddings and the agent baselines. Appendix C.1 (p. 16) already names the retrieval model and specifies how 15 skills are selected from a 31-model library. Section 3 (p. 3) defines executable predictive skills; Section 4.2 (pp. 3–4) explains how descriptions guide model selection and how predictions are recorded. Appendix E (p. 21) gives individual model details and results. Each skill pairs an executable forecasting model with a description of its mechanism, inputs, fitting, updates, uncertainty and limitations. We list the complete library below and will add the full skill descriptions and agent prompts to the appendix.
Table R4. Complete predictive-model library
| Family | Count | Models |
|---|---|---|
| Statistical | 7 | Last value; seasonal naive; AutoARIMA; AutoETS; AutoTheta; AutoCES; Prophet |
| State-space | 5 | Drift; covariate drift; local level; covariate local trend; particle local level |
| Compartmental | 4 | SIR; SEIR; SEIRS; time-varying SEIR |
| Renewal | 1 | Renewal |
| Neural | 10 | RNN-style recurrence; LSTM; GRU-style recurrence; DeepAR; N-BEATS; NHITS; PatchTST; TFT; NHITS-t; residual-calibrated PatchTST |
| Pretrained time-series | 4 | Chronos-Bolt; TimesFM-2; TimesFM-3; TiRex |
Q4: “The overall writing is poor, making it hard to understand the work in one go.” “It took me several reads to understand the problem statement the authors are working on.”
Answer: We will make the forecasting task explicit from the opening paragraph. PAM learns how a model’s past prediction errors should change its contribution to later forecasts. The inputs are available histories and a library of predictive models; the outputs are combined forecasts and stored model weights (Section 3). We will state this task and its inputs and outputs at the start of the Abstract and Introduction, define skills before introducing memory terminology, and move training details out of the opening explanation.
Q5: “"Sec 3: Problem Formulation" is missing.” “Only the problem setup is described.”
Answer: Section 3, “Problem Formulation,” is present on page 3, lines 112–124, and specifies the prediction problem. It defines the inputs, prediction distributions, model weights, skill budget and objective, and states that future target values cannot change earlier forecasts. The training objectives and optimization procedure are given in Section 4.6 and Appendix A.4, Eqs. (17)–(19). Both the problem definition and its optimization are therefore included in the submission.
Thank you again for your constructive suggestions. We would appreciate reassessment of the soundness concerns in light of these definitions, the evaluation protocol and the additional comparisons above.
Response to Reviewer 7ML9, Part 1
Thank you for recognizing that the paper addresses an interesting problem and for your constructive suggestions on the evaluation. Below, we clarify the evaluation protocol and skill configuration, and provide additional baseline comparisons, including a self-evolving agent framework, together with results across multiple runs.
Q1a: “And more generally speaking, the optimizing objective should not be used to evaluate as that is a circular evaluation.”
Answer: Using RMSE for both training and held-out testing is standard forecasting practice, not circular evaluation. For example, exponential smoothing minimizes squared training errors and reports RMSE on later observations (Hyndman and Athanasopoulos, 2021, Sections 5.8, 8.1–8.2). Table 4 (Appendix C.1, p. 16) gives separate chronological training, validation and test periods. PAM’s trained parameters and hyperparameters are fixed before testing (Section 4.6; Algorithm 1, p. 15). Once an earlier target value is released, it can update memory for later forecasts, but cannot change predictions already made (Section 3, p. 3). The same metric is used on separate data; test predictions are evaluated without using their own target values to fit or select the model.
Q1b: “The method proposed by the authors uses RMSE as regularizer in Record step and as optimizer in the location head and contextual proposal part.” “Furthermore, the model checkpoints are selected using a weighted combination of standardized RMSE, WIS, and NLL on validation set.”
Answer: RMSE in Record is a validation-based model-ranking score, not a regularizer. Calling it a regularizer confuses two different operations: Section 4.2 (p. 3) uses validation RMSE to rank candidate models, while Eq. (17) separately defines the correction penalty added to the training loss (Appendix A.4, p. 14). The location-correction and contextual-weighting networks use training RMSE; the uncertainty network uses negative log likelihood (Section 4.6). Checkpoints are selected on validation data using Eq. (19). These steps are separate from evaluation on future test predictions.
Q1c: “The evaluation seems unfair.” “Now, if RMSE and WIS is used in selecting and training of model, it seems unfair to compare the baselines on these metrics as they were never trained on them.”
Answer: Several baselines have no forecasting parameters to train with RMSE or WIS; this follows from their definitions, not an omitted training step. Last-value and seasonal-naive forecasting are fixed rules, while the LLM agents select existing model predictions. Adding supervised training would change these algorithms (Section 5.1; Appendix C.2). PAM trains its own correction and weighting modules without retraining the forecasting models in the bank (Section 4.6).
DMA already uses the same model bank, stored base forecasts, test events and validation objective as PAM. Its hyperparameters are selected over a 20-value grid (Appendix C.2, p. 16; Eq. (19)). PAM achieves lower RMSE and WIS on Bench-A and B-FLU (submitted Table 1, p. 6):
Table R1. PAM and DMA on the same model bank
Submitted results; PAM reports mean ± sample SD across three training runs; DMA is deterministic. Both metrics are normalized as in submitted Table 1. On B-COVID, DMA’s mean RMSE and WIS are slightly lower than PAM’s.