- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how different memory forms affect an agent’s predictions in a working telescope system. The main task is to predict how many minutes remain before a cloud-covered target becomes visible again. It compares four settings: no memory, nightly text summaries, summaries with historical statistics, and an additional note about past prediction errors. The paper reports that adding statistics improves predictions, mainly by reducing overestimation, while text summaries provide limited help and can hurt some models. The study uses a 62-day archive, with two original runs evaluated on 170 prediction points across 22 observing days. This paper mainly adds historical statistics and past-error notes to prompts, with little methodological innovation. Most positive results are from one task, which does not separate added information from memory format, and several conclusions exceed the evidence. Overall, this paper is far from acceptance.
Critical strengths
None
Critical weaknesses
W1 It is unclear what actually drives the gains. L2 and L3 give the model extra statistics and error feedback. The paper does not show whether better memory representation helps, or whether simply providing that extra information explains the improvement.
W2 Key baseline implementations violate temporal causality. Some baselines use the final duration or prediction residual of an earlier anchor before its obscuration has ended and the outcome is observable. Consequently, the comparisons against simple predictors and the mechanism analysis in App E require recomputation. The magnitude of the resulting changes remains unknown, but these conclusions are not yet reliable.
W3 The evaluation mainly uses offline and single-turn prediction. It primarily tests how LLMs use contextual information, rather than agent capabilities. The memory mechanisms are also fairly conventional and have poor novelty. Moreover, the positive results are concentrated on one type of task, leaving the conclusions suggested by the title supported.
Questions for authors
Q1 Why does the sec introduction have no citations?
Q2 When computing persistence, climatology, and rolling bias correction, were historical outcomes required to have been confirmed before the current prediction time? How does the implementation enforce this when multiple anchors occur within one ongoing obscuration?
Q3 After enforcing outcome-confirmation timing, what are the corrected baseline scores and paired intervals for L2 plus arithmetic correction versus L3? Do the conclusions of Appendix E survive?
3 comments
Response to Authors
Thank you for your response and clarifications. We acknowledge your efforts and will raise our score to 2. However, our main concern regarding the limited novelty of the proposed method remains unresolved.
Thank you for your constructive feedback and for reconsidering the score.
We acknowledge that the individual memory mechanisms employed in our study are not fundamentally new, and we do not intend to claim novelty in these components.
Rather, our primary contribution is an empirical investigation of how different experience representations and their accessible information influence the utility of persistent memory in scientific agents.
Through controlled comparisons using a common underlying observation archive, cross-model replay, and writer–reader evaluations, we identify conditions under which narrative memory may be ineffective or even detrimental, while structured statistical information and historical error feedback can improve calibration.
We recognize that the evaluated representations also differ in the information explicitly made accessible, and have clarified the scope of our conclusions accordingly.
We hope this distinction better communicates the contribution of our work as an empirical finding rather than a fundamentally new memory algorithm.
Thank you. Your W2 is correct, and we are grateful it was caught. General response 1 gives the full account and the table of corrected numbers; here we answer each question directly. Section and appendix names are those of the revised PDF.
Q2: were historical outcomes required to be confirmed before the prediction time? No, not for persistence, climatology or the rolling bias correction. They selected earlier outcomes by anchor order alone, so when several anchors fell inside one ongoing obscuration, a later anchor could read the duration, or the residual, of an earlier anchor in the same obscuration before it had ended. This happened at 84 of 170 scored anchors, 50 of them probes. For example, an anchor with 22 minutes remaining received persistence 26: the duration recorded 4 minutes earlier in the same obscuration, not confirmed until 26 minutes later.
How it is enforced now. Each outcome carries the time of its confirming frame (the third consecutive unobscured frame of the truth definition), and it is usable only if uncensored and confirmed at or before the anchor's time. Persistence takes the most recently confirmed outcome, climatology their median, and the rolling correction their residuals only. A regression test builds probe anchors inside one obscuration and asserts that no baseline can read its outcome. The agent-side inputs were causal from the start, so every agent row is unchanged.
Q3: corrected scores, and Appendix E (minutes; run 1 / run 2; baseline MAEs do not depend on the run):
What survives in Appendix E: neither run separates L3 from the arithmetic baseline, and in the "Left" column L3 keeps the least remaining correction of the four rungs in both runs (−0.60 and −0.21, against L0's −3.18 and −4.13). What we changed: the two arms receive different quantities. The arithmetic arm subtracts L2's own median residual over outcomes confirmed by the current anchor, while the L3 note states L3's own residuals on previous days, with their count and median absolute error. Appendix E therefore says only that the result is consistent with a simple bias correction and that no additional MAE benefit of L3 over that baseline was detected; it claims no equivalence. Withdrawn: "the strongest model-free predictor is two lines of arithmetic", and persistence's slope as a tracking reference (corrected persistence runs low, pushing its slope to 1.26); climatology (0.43) and conditional climatology (0.33) are the fair references, and the agent's 0.22 and 0.12 stay below both. In Appendix A, GRU and attention are no longer resolvably worse than persistence.
W1: what drives the gains. The matched-content control (general response 2 (a), Section 4.1, Appendix H) changes only the statistics block of the recorded L2 prompts. Neither of two runs detected an MAE advantage of the table over the same numbers in sentences (+0.05 [−0.80, +0.82], −0.10 [−1.02, +0.59]), nor of the full table over the conditional median. We have narrowed the claim accordingly: which quantities are stored changes what a memory can carry, with no format effect detected. All arms keep the statistics, and we claim no equivalence. L3's gain over L2 does not replicate (−1.55 [−3.26, −0.19], −0.99 [−2.35, +0.18]) and is stated as a point estimate.
W3: offline, single-turn, conventional mechanisms, one task. We agree the mechanisms are conventional: the memory system was deployed before this study and is evaluated unchanged, and the contribution is the measurement. The writing side is multi-day (the agent's own nightly consolidation over 62 replayed days). The reading side is one call per anchor by design, so every configuration answers the same 170 anchors and differences can be paired; we accept that it mainly tests how a model uses its context. The positive result rests on one task at one site, and the abstract and Limitations are scoped to it.
Q1: why the introduction had no citations. There was no good reason. It now cites Park et al. (2023), Packer et al. (2023), Sumers et al. (2024), Bouthillier et al. (2021), Yuan et al. (2025) and Zhang et al. (2026).