acceptodds
Under review as a conference paper at ICLR 2027

Probabilistic Agentic Memory: Learning Distributional Evidence for Skill Reuse

Abstract

Repeated use of predictive skills gives an agent outcome feedback, but a numerical error alone does not specify how that experience should influence future reuse. We introduce Probabilistic Agentic Memory (PAM), which learns how outcomes become evidence for persistent skill support. PAM builds context-conditioned outcome distributions around black-box forecasts and uses their realized log scores in sequential generalized Bayesian updates. A memory-informed readout combines these same predictive distributions, linking the interpretation of past outcomes to future predictions. Training differentiates through chronological replay without gradients through the skills; at deployment, learned parameters are frozen and released outcomes update memory. Across three epidemic-forecasting tasks, PAM improves mean RMSE and weighted interval score over three agent selectors using the same skill library. It also improves both metrics over same-bank dynamic model averaging on Bench-A and B-FLU. On Bench-A, ablations support nonlinear, context-conditioned uncertainty modeling, and matched-component comparisons show that learned writes improve subsequent prediction at comparable coverage. Prospective diagnostics on both CDC tasks further show that accumulated support favors components with better subsequent predictions. Together, these results support learning the evidential meaning of experience as a mechanism for predictive-skill reuse.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Skill Reuse, Probabilistic Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_7ML9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to provide a method called Probabilistic Agentic Memory (PAM), which learns how outcomes based evidence for persistent skill support for predictive forecasting tasks. Since each skill use can impact the future prediction, PAM helps agent learn "what an outcome implies for the future reuse of a skill." They evaluate PAM over three epidemic-forecasting tasks with a 15-skill bank retrieved from 31 candidates and show that it outperforms three agent setup using the same 31-skill library on all tasks and over same-bank.

Overall, the paper tackles an interesting problem; however, in its current form, it appears premature. Unclear research direction (poorly written introduction and abstract), incomplete formulation, and lack of proper evaluation makes it hard to evaluate the proper contribution of the work.

Critical strengths

The problem identified by the authors seems interesting but the overall paper seems premature.

Critical weaknesses

Major:

  1. The evaluation seems unfair. The method proposed by the authors uses RMSE as regularizer in Record step and as optimizer in the location head and contextual proposal part. Furthermore, the model checkpoints are selected using a weighted combination of standardized RMSE, WIS, and NLL on validation set. Now, if RMSE and WIS is used in selecting and training of model, it seems unfair to compare the baselines on these metrics as they were never trained on them. And more generally speaking, the optimizing objective should not be used to evaluate as that is a circular evaluation.

  2. The evaluation seems limited and incomplete. The authors have compared with some old agentic setup like React. Evaluation with more newer agents, like self-evolving harness, will explain the usefulness of the method. Furthermore, the standard deviations are missing for the baselines.

  3. It is also unclear as to what models were used in the agentic system and what skills look like in the author's setup. Without this, it is hard to evaluate if the proposed framework is useful for more recent models.

Minor:

  1. The overall writing is poor, making it hard to understand the work in one go. It took me several reads to understand the problem statement the authors are working on.
  2. "Sec 3: Problem Formulation" is missing. Only the problem setup is described.
Questions for authors

See Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7ML9, Part 1

Thank you for recognizing that the paper addresses an interesting problem and for your constructive suggestions on the evaluation. Below, we clarify the evaluation protocol and skill configuration, and provide additional baseline comparisons, including a self-evolving agent framework, together with results across multiple runs.

Q1a: “And more generally speaking, the optimizing objective should not be used to evaluate as that is a circular evaluation.”

Answer: Using RMSE for both training and held-out testing is standard forecasting practice, not circular evaluation. For example, exponential smoothing minimizes squared training errors and reports RMSE on later observations (Hyndman and Athanasopoulos, 2021, Sections 5.8, 8.1–8.2). Table 4 (Appendix C.1, p. 16) gives separate chronological training, validation and test periods. PAM’s trained parameters and hyperparameters are fixed before testing (Section 4.6; Algorithm 1, p. 15). Once an earlier target value is released, it can update memory for later forecasts, but cannot change predictions already made (Section 3, p. 3). The same metric is used on separate data; test predictions are evaluated without using their own target values to fit or select the model.

Q1b: “The method proposed by the authors uses RMSE as regularizer in Record step and as optimizer in the location head and contextual proposal part.” “Furthermore, the model checkpoints are selected using a weighted combination of standardized RMSE, WIS, and NLL on validation set.”

Answer: RMSE in Record is a validation-based model-ranking score, not a regularizer. Calling it a regularizer confuses two different operations: Section 4.2 (p. 3) uses validation RMSE to rank candidate models, while Eq. (17) separately defines the correction penalty added to the training loss (Appendix A.4, p. 14). The location-correction and contextual-weighting networks use training RMSE; the uncertainty network uses negative log likelihood (Section 4.6). Checkpoints are selected on validation data using Eq. (19). These steps are separate from evaluation on future test predictions.

Q1c: “The evaluation seems unfair.” “Now, if RMSE and WIS is used in selecting and training of model, it seems unfair to compare the baselines on these metrics as they were never trained on them.”

Answer: Several baselines have no forecasting parameters to train with RMSE or WIS; this follows from their definitions, not an omitted training step. Last-value and seasonal-naive forecasting are fixed rules, while the LLM agents select existing model predictions. Adding supervised training would change these algorithms (Section 5.1; Appendix C.2). PAM trains its own correction and weighting modules without retraining the forecasting models in the bank (Section 4.6).

DMA already uses the same model bank, stored base forecasts, test events and validation objective as PAM. Its hyperparameters are selected over a 20-value grid (Appendix C.2, p. 16; Eq. (19)). PAM achieves lower RMSE and WIS on Bench-A and B-FLU (submitted Table 1, p. 6):

Table R1. PAM and DMA on the same model bank

DatasetMethodRMSE ↓WIS ↓
Bench-APAM0.3347 ± 0.01620.0914 ± 0.0008
Bench-ADMA0.66900.1132
B-FLUPAM1.0447 ± 0.17730.4449 ± 0.0268
B-FLUDMA1.44080.5519

Submitted results; PAM reports mean ± sample SD across three training runs; DMA is deterministic. Both metrics are normalized as in submitted Table 1. On B-COVID, DMA’s mean RMSE and WIS are slightly lower than PAM’s.

AuthorsOpenReview ↗

Response to Reviewer 7ML9, Part 2

As a separate check, we tested whether access to historical prediction errors explains the performance gap. Agent prompts included the past RMSE of all 31 models, computed from the latest four batches with available target values. The LSTM uses either its original inputs or additional inputs containing its past RMSE and a flag showing whether that error is available. Even with these inputs, PAM has lower mean RMSE and WIS on all three tasks; Table R2 reports WIS.

Table R2. Baselines with and without historical prediction errors

MethodConfigurationBench-A WIS ↓B-COVID WIS ↓B-FLU WIS ↓
PAMFull0.0914 ± 0.00080.1513 ± 0.00500.4449 ± 0.0268
Agent Top-1Original inputs0.1101 ± 0.00000.2267 ± 0.00000.4898 ± 0.0000
Agent Top-1With past RMSE0.1113 ± 0.00470.2403 ± 0.00000.7411 ± 0.0000
Agent RediscoveryOriginal inputs0.1099 ± 0.00060.2401 ± 0.00600.5886 ± 0.0533
Agent RediscoveryWith past RMSE0.1201 ± 0.00030.2001 ± 0.00140.7545 ± 0.0179
Agent ReActOriginal inputs0.1110 ± 0.00130.2405 ± 0.00440.5408 ± 0.0056
Agent ReActWith past RMSE0.1212 ± 0.00110.1990 ± 0.00260.8172 ± 0.0268
LSTMOriginal inputs0.1143 ± 0.00110.2510 ± 0.00080.7426 ± 0.0004
LSTMWith past RMSE0.1143 ± 0.00110.2507 ± 0.00100.7403 ± 0.0007

Additional results; mean ± sample SD over three runs. Agent rows vary decoding seeds; LSTM and PAM rows vary training seeds. Both LSTM variants reserve two additional input channels: zero-filled in “Original inputs”, or containing past RMSE and its availability flag in “With past RMSE”. PAM uses the three original fits. WIS is normalized by training-target SD, as in the submitted Table 1; lower is better. WIS jointly measures prediction-interval width and missed targets.

Q2a: “The evaluation seems limited and incomplete.” “The authors have compared with some old agentic setup like React.” “Evaluation with more newer agents, like self-evolving harness, will explain the usefulness of the method.”

Answer: Good suggestion. We evaluated Agentic Context Engineering (ACE; Zhang et al., ICLR 2026). ACE updates its context playbook through generation, reflection and curation. On Bench-A, we tested all four combinations of iterative regeneration and semantic deduplication, providing feedback only after the true target values became available.

Table R3. ACE configurations on Bench-A

MethodRMSE ↓WIS ↓Cov90
PAM0.3347 ± 0.01620.0914 ± 0.00080.9230 ± 0.0137
ACE basic reflection and curation0.4557 ± 0.00030.1361 ± 0.00020.5110 ± 0.0018
ACE + iterative regeneration0.4242 ± 0.05640.1335 ± 0.00450.5378 ± 0.0295
ACE + semantic deduplication0.4680 ± 0.01130.1273 ± 0.00430.6104 ± 0.0420
ACE + both mechanisms0.4510 ± 0.00820.1336 ± 0.00440.5265 ± 0.0260

Additional results; mean ± sample SD over three sampled ACE runs and three PAM fits. RMSE and WIS are normalized as in Table 1. Cov90 is the fraction of targets covered by 90% prediction intervals; its target is 0.90. Bold marks lower errors or coverage closer to that target.

PAM substantially outperforms all four ACE configurations in RMSE and WIS, with coverage much closer to the 90% target (Table R3). This consistent advantage suggests a benefit from PAM’s learned probabilistic scoring, which ACE does not include. ACE updates textual memory; PAM also learns model bias and uncertainty to produce comparable log scores for updating model weights (Sections 4.3–4.4). Table 2B supports this explanation: learned scoring improves later predictions even when the forecast distributions being combined are identical.

Q2b: “Furthermore, the standard deviations are missing for the baselines.”

Answer: Thank you for highlighting the need to report variation across runs. The original LLM-agent baselines used deterministic greedy decoding: sampling was disabled, and changing the decoding seed produced identical results. The single-value reporting therefore reflects the deterministic setup, not missing repeated sampling runs (Table 1; Appendix C.2, pp. 16–17). PAM’s SD measures variation across three independent training runs. Following your suggestion, we also enabled agent sampling with temperature 0.2, top-p 0.9, top-k 20 and seeds 42, 43 and 44. Tables R2–R3 report mean ± SD for every tested agent configuration, alongside the training SDs for PAM and LSTM.

AuthorsOpenReview ↗

Response to Reviewer 7ML9, Part 3

Q3: “It is also unclear as to what models were used in the agentic system and what skills look like in the author's setup.” “Without this, it is hard to evaluate if the proposed framework is useful for more recent models.”

Answer: The foundation model is Qwen2.5-7B-Instruct, used for retrieval embeddings and the agent baselines. Appendix C.1 (p. 16) already names the retrieval model and specifies how 15 skills are selected from a 31-model library. Section 3 (p. 3) defines executable predictive skills; Section 4.2 (pp. 3–4) explains how descriptions guide model selection and how predictions are recorded. Appendix E (p. 21) gives individual model details and results. Each skill pairs an executable forecasting model with a description of its mechanism, inputs, fitting, updates, uncertainty and limitations. We list the complete library below and will add the full skill descriptions and agent prompts to the appendix.

Table R4. Complete predictive-model library

FamilyCountModels
Statistical7Last value; seasonal naive; AutoARIMA; AutoETS; AutoTheta; AutoCES; Prophet
State-space5Drift; covariate drift; local level; covariate local trend; particle local level
Compartmental4SIR; SEIR; SEIRS; time-varying SEIR
Renewal1Renewal
Neural10RNN-style recurrence; LSTM; GRU-style recurrence; DeepAR; N-BEATS; NHITS; PatchTST; TFT; NHITS-t; residual-calibrated PatchTST
Pretrained time-series4Chronos-Bolt; TimesFM-2; TimesFM-3; TiRex

Q4: “The overall writing is poor, making it hard to understand the work in one go.” “It took me several reads to understand the problem statement the authors are working on.”

Answer: We will make the forecasting task explicit from the opening paragraph. PAM learns how a model’s past prediction errors should change its contribution to later forecasts. The inputs are available histories and a library of predictive models; the outputs are combined forecasts and stored model weights (Section 3). We will state this task and its inputs and outputs at the start of the Abstract and Introduction, define skills before introducing memory terminology, and move training details out of the opening explanation.

Q5: “"Sec 3: Problem Formulation" is missing.” “Only the problem setup is described.”

Answer: Section 3, “Problem Formulation,” is present on page 3, lines 112–124, and specifies the prediction problem. It defines the inputs, prediction distributions, model weights, skill budget and objective, and states that future target values cannot change earlier forecasts. The training objectives and optimization procedure are given in Section 4.6 and Appendix A.4, Eqs. (17)–(19). Both the problem definition and its optimization are therefore included in the submission.

Thank you again for your constructive suggestions. We would appreciate reassessment of the soundness concerns in light of these definitions, the evaluation protocol and the additional comparisons above.

Reviewer_MNdQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Probabilistic Agentic Memory, a framework for reusing predictive models through adaptive forecast combination. PAM learns context-dependent outcome distributions around black-box forecasts and uses their scores on observed outcomes to update skill weights. Experiments on three epidemic forecasting tasks compare PAM with agent selectors and dynamic model averaging, with ablations examining distribution learning and memory updates.

Critical strengths
  1. The method accounts for context-dependent prediction bias and uncertainty when updating ensemble weights.
  2. The matched component experiment helps isolate the effect of the write mechanism by keeping the predictive distributions fixed.
Critical weaknesses
  1. The presentation makes the core task unnecessarily difficult to understand. The paper introduces broad terms such as “agentic memory” and “skill reuse” before clearly explaining that the skills are forecasting models and that the task is adaptive forecast combination.

  2. The claims about agentic memory and skill reuse are broader than the evidence supports. The implementation combines a fixed bank of forecasting models through learned distribution corrections and sequential weight updates. It does not demonstrate broader agent-memory capabilities or skill reuse across different tasks. The paper should narrow its claims and clarify its methodological contribution beyond existing probabilistic postprocessing and online forecast-combination methods.

  3. The benefit of the added complexity is unclear. PAM does not outperform DMA on B-COVID, while TimesFM-3 has better RMSE and WIS point estimates on both CDC tasks.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer MNdQ, Part 1

Thank you for recognizing the value of modeling prediction bias and uncertainty, and for highlighting the fixed-component experiment as a strength. Below, we clarify the novelty and scope of our work and explain how the controlled experiments demonstrate the benefit of learning how prediction errors update memory.

Q1a: “The paper introduces broad terms such as “agentic memory” and “skill reuse” before clearly explaining that the skills are forecasting models and that the task is adaptive forecast combination.”

Answer: In PAM, each skill is a forecasting model paired with its description. The task is to update how these models are combined using feedback from their past predictions. Descriptions support retrieval; external memory stores the selected models, their predictions and the weights updated from prediction errors (Sections 3–4.2, p. 3). This follows established agent designs: HuggingGPT selects and runs models using their descriptions (Shen et al., 2023, Sections 3.2–3.3), while Voyager stores executable skills indexed by descriptions (Wang et al., 2024, Section 2.2); both are cited in our paper.

Q1b: “The presentation makes the core task unnecessarily difficult to understand.”

Answer: We agree that the task should be stated before introducing the memory terminology. Our question is how past model prediction errors should change the weights used for the next forecast. The same error can reflect systematic bias or expected variation, which should affect later model weights differently (Section 1, p. 1). Forecasting directly tests this question: predictions are recorded before their target values are known, memory is updated when those values become available, and performance is measured on later predictions (Sections 3 and 5.1). We will state this task and feedback sequence at the start of the Abstract and Introduction.

AuthorsOpenReview ↗

Response to Reviewer MNdQ, Part 2

Q2a: “The claims about agentic memory and skill reuse are broader than the evidence supports.” “It does not demonstrate broader agent-memory capabilities or skill reuse across different tasks.”

Answer: Cross-task transfer is not the claim being tested: Section 3 defines repeated predictive-skill reuse with delayed feedback, as also stated in the abstract and contributions (pp. 1–3). We test whether learning how to interpret model prediction errors improves later use of those models. Bench-A, B-COVID and B-FLU test this hypothesis across different targets, time scales and geographic settings, using strategy comparisons, controlled ablations and analyses of later prediction quality (Sections 5.1–5.4; Tables 1, 2 and 5).

Q2b: “The implementation combines a fixed bank of forecasting models through learned distribution corrections and sequential weight updates.”

Answer: “Fixed” refers to which models are available; their weights change with prediction feedback. Section 4.2 states: “Retrieval selects the bank but does not determine its memory weights.” Once target values become available, PAM compares them with the recorded predictions and updates the stored weights (Eq. (6)). Those weights determine each model’s contribution to later forecasts (Eq. (7)). The fixed bank therefore supports continual model reuse; adding new models is a separate capability.

Q2c: “The paper should narrow its claims and clarify its methodological contribution beyond existing probabilistic postprocessing and online forecast-combination methods.”

Answer: This is an important distinction to clarify. PAM learns how prediction errors should update model weights in external memory to improve later predictions. This goes beyond calibrating forecasts through postprocessing or applying a predefined error score for weighting.

The agent-memory methods reviewed here do not learn how to score numerical prediction errors. The methods in Section 2.1 store records, reflections or beliefs, but do not learn a common probabilistic score from model predictions and observed targets. Models have different biases and uncertainty estimates, and some provide no uncertainty estimate at all. The same error can be expected for one model but surprising for another. Storing that error does not determine how much it should change the model’s future weight.

PAM learns these scores for Bayesian memory updates. Shared neural networks learn bias and uncertainty corrections for black-box model predictions. The resulting predictive distributions provide comparable log scores for generalized Bayesian weight updates. Training differentiates through these updates, so the scores are learned for their effect on later predictions, without gradients through the forecasting models (Sections 4.3–4.6). Section 2.2 distinguishes this shared scoring model from the model-specific or latent-state inference in the cited DMA/BPS methods.

The controlled experiment isolates the contribution beyond forecast calibration. The Table 2B control you highlighted keeps the forecasts being combined identical and changes only the scores used to update weights. Learned scoring still improves later predictions, directly supporting the benefit of learning how feedback changes future model use.

Q3: “The benefit of the added complexity is unclear. PAM does not outperform DMA on B-COVID, while TimesFM-3 has better RMSE and WIS point estimates on both CDC tasks.”

Answer: This is a fair question about the value of the added complexity. The controlled tests in Tables 2A–B show that the added modules improve prediction quality. Removing uncertainty learning or input-dependent weighting, or replacing the correction networks with constant or linear functions, worsens both RMSE and WIS on Bench-A after retraining (Section 5.3, Table 2A, p. 7). The fixed-distribution experiment you highlighted changes only the scores used to update model weights and improves later forecasts (Section 5.4, Table 2B, p. 7).

PAM adds 7,346 trainable parameters, excluding the forecasting models and retrieval, and its weight update costs O(K) for K models (Table 3, p. 13; Appendix B, p. 16).

The rankings you cite are correct: DMA has slightly lower mean RMSE and WIS on B-COVID, and TimesFM-3 has lower values on both CDC tasks (Tables 1 and 6). PAM improves both metrics over DMA on Bench-A and B-FLU using the same model bank. On B-COVID, PAM gives narrower intervals than DMA, with coverage closer to the 90% target (Section 5.2, Table 1).

The full strategy comparison also shows lower mean RMSE and WIS than all three agent selectors on all three tasks (Table M1; submitted Table 1, p. 6).

Table M1. PAM versus all three agent selectors across all tasks

MethodBench-A RMSE / WIS ↓B-COVID RMSE / WIS ↓B-FLU RMSE / WIS ↓
PAM0.3347 / 0.09140.2984 / 0.15131.0447 / 0.4449
Agent Top-10.4729 / 0.11620.3756 / 0.24031.3888 / 0.4898
Agent Rediscovery0.3811 / 0.11060.3870 / 0.22481.2852 / 0.5901
Agent ReAct0.3905 / 0.10830.4147 / 0.24621.4309 / 0.6430

From Table 1 (p. 6). PAM values are means over three fits; agent values come from saved selections on the same test events. All agents select from the same 31-model library used to retrieve PAM’s 15 models. RMSE and WIS are normalized by training-target SD; lower is better. WIS measures both prediction-interval width and missed targets.

We would appreciate reassessment of the contribution based on these controlled results and the stated scope.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Reviewer Echs, Part 1

Thank you for recognizing the value of accounting for prediction bias and uncertainty, and for highlighting the fixed-component experiment. Your suggestions helped us strengthen the comparisons and clarify the roles of memory updates and learned scoring.

Q1a: “Does the support update include any forgetting, reset, or minimum-weight mechanism beyond Equation (6)?”

Answer: The stored weights follow Eq. (6), without an additional forgetting factor, reset, or minimum-weight floor. Repeatedly scoring below competing models accumulates evidence against a model and progressively reduces its relative weight, explaining values such as 10−3310^{-33} (Eq. (20)).

To tackle this at prediction time, PAM balances the accumulated weights with context-dependent weights based on the learned initial weights (Eqs. (7)–(8)). When α>0\alpha > 0, this allows previously downweighted models to contribute according to the current input. This balancing changes the weights used for prediction; the stored weights continue to follow Eq. (6).

Q1b: “Do the evaluated sequences contain performance reversals, and how quickly does support recover for previously downweighted skills?”

Answer: This is a good question, especially when relative model performance changes. Regaining substantial weight requires later predictive evidence to offset a model’s earlier accumulated disadvantage (Eq. (20)). Figures 2 and 5 show weights changing as new target values become available. These results support useful feedback-driven weighting; recovery time after a verified performance reversal remains unmeasured. We will clarify this distinction in the revised discussion.

Q2: “The experiments do not fully separate the benefit of memory updates from better forecast calibration.” “With learned component distributions held fixed, how does PAM compare with static learned weights, context-only mixing, and tuned alternative support updates?”

Answer: This is a good suggestion. We therefore added a comparison that holds the learned prediction distributions fixed within each checkpoint and changes only the weighting rule. This separates the benefit of model weighting from forecast calibration. Static learned W0W_0 uses PAM’s learned initial weights without updates; Context-only similarity uses context–description similarity without prediction-error feedback. PAM achieves lower mean RMSE and WIS than both controls on all three datasets (Table R5), supporting the benefit of its weighting mechanism beyond component calibration.

Table R5. Readout comparison with learned component distributions fixed.

BenchmarkMethodRMSE ↓WIS ↓Cov90Width90
Bench-APAM (full)0.3105 ± 0.00710.0891 ± 0.00120.8711 ± 0.01870.5132 ± 0.0608
Bench-AStatic learned W00.3186 ± 0.00580.0901 ± 0.00150.8764 ± 0.00710.5064 ± 0.0644
Bench-AContext-only similarity0.3191 ± 0.00620.0903 ± 0.00160.8767 ± 0.00700.5056 ± 0.0643
B-COVIDPAM (full)0.3000 ± 0.01350.1518 ± 0.00570.9648 ± 0.00211.0984 ± 0.0375
B-COVIDStatic learned W00.3141 ± 0.00670.1726 ± 0.00390.9858 ± 0.00661.4315 ± 0.0313
B-COVIDContext-only similarity0.3216 ± 0.00910.1770 ± 0.00470.9851 ± 0.00611.4599 ± 0.0278
B-FLUPAM (full)0.9438 ± 0.01620.4703 ± 0.01690.9114 ± 0.01212.2857 ± 0.1362
B-FLUStatic learned W00.9480 ± 0.00420.5110 ± 0.01590.9031 ± 0.02092.3054 ± 0.1571
B-FLUContext-only similarity0.9494 ± 0.00370.5223 ± 0.01480.8990 ± 0.02302.3010 ± 0.1553

Mean ± SD. Both controls use the learned component distributions of each selected checkpoint. Lower RMSE and WIS are better; Cov90 targets 0.90, and Width90 is interpreted alongside coverage. Bold marks the lowest mean RMSE and WIS within each dataset.

To make sure we address your intended comparison, could you clarify which specific update rules and tuning settings you have in mind for “tuned alternative support updates”?

AuthorsOpenReview ↗

Response to Reviewer Echs, Part 2

Q3: “Does the learned-versus-anchored write improvement in Table 2B hold on both CDC tasks? Are paired temporal uncertainty estimates available for these comparisons?”

Answer: Thank you for suggesting that we extend this comparison to both CDC tasks. The extended comparison shows lower WIS with learned scoring on B-FLU and numerically close WIS on B-COVID (Table R6). Table 2B in the submitted paper shows that learned scoring reduces Bench-A WIS from 0.1006 to 0.0917 while keeping the component prediction distributions fixed. Within each comparison, only the scores used to update model weights change.

Table R6. Learned versus anchored scoring on both CDC benchmarks.

BenchmarkLearned scoring WIS ↓Anchored scoring WIS ↓
B-FLU0.482 ± 0.0060.490 ± 0.005
B-COVID0.143 ± 0.0030.143 ± 0.003

Mean ± SD across runs, rounded consistently to three decimal places. Before rounding, B-COVID WIS is 0.1427 (learned) and 0.1426 (anchored).

Regarding paired temporal uncertainty, Appendix C.3 reports a paired time-block interval for the original Bench-A comparison. The updated CDC results currently report mean ± SD across runs; paired temporal intervals for these new comparisons are not included.

Together, these results suggest that PAM’s learned scoring improves later predictions beyond component calibration, with gains on Bench-A and B-FLU and comparable WIS on B-COVID. The original B-COVID weight trajectories suggest a possible explanation for the small difference: once one model dominates, changing the update scores may have less influence on the combined prediction (Figure 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.