acceptodds
Under review as a conference paper at ICLR 2027

What a Memory Can and Cannot Carry: Representation Shapes What an Experience Layer Can Hold

Abstract

Language-model agents are increasingly given persistent memory, and the usual question is how much it helps. We report an experiment in which one archive of stored observations helps, does nothing, or hurts according to what is derived from it and how that is written down. A deployed observing agent at a solar telescope accumulates nightly experience over 62 days; we ask it a quantitative question its input cannot answer — how many minutes until an obscured target is observable again — and compare memories built from that one archive: the prose its consolidation cycle generates, a stratified statistical snapshot rebuilt nightly, and the snapshot plus a note on the configuration's own past error. Against truth from future frames the prose form is indistinguishable from having no memory, while the statistical form cuts mean absolute error from 22–25 to about 16 minutes and bias from +16 to +5; the note on past error reaches 15–16 minutes and +2 to +3. Both reproduce across two runs against a pre-fixed criterion. What that rung supplies is a level, not a conditional prediction: it does not raise the slope of truth on prediction, and an arithmetic version of the correction cannot be told apart from it. Replaying the 680 recorded prompts against five models from four laboratories sharpens this: the last rung beats no memory resolvably on all five, whereas the prose form raises the bias above the memory-free level on four of five, failing a pre-specified criterion. The one model the prose does not harm is the one that wrote it, which invites an author effect; a writer-by-reader matrix does not support it — across this three-writer panel the ablest reader writes the prose least useful to every reader. A second agent supplies a zero point predicted in advance: where the memory-free configuration is already near-unbiased, every rung was run and the ladder is flat, so the effect tracks the miscalibration it corrects. No configuration reaches a one-line statistical rule, whose bias-corrected form is the strongest predictor here. Four measurement settings that each invalidated an earlier run are reported with the released harness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, memory representation, language-model agents, calibration, measurement, negative results, deployed system

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_frGTOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how different memory forms affect an agent’s predictions in a working telescope system. The main task is to predict how many minutes remain before a cloud-covered target becomes visible again. It compares four settings: no memory, nightly text summaries, summaries with historical statistics, and an additional note about past prediction errors. The paper reports that adding statistics improves predictions, mainly by reducing overestimation, while text summaries provide limited help and can hurt some models. The study uses a 62-day archive, with two original runs evaluated on 170 prediction points across 22 observing days. This paper mainly adds historical statistics and past-error notes to prompts, with little methodological innovation. Most positive results are from one task, which does not separate added information from memory format, and several conclusions exceed the evidence. Overall, this paper is far from acceptance.

Critical strengths

None

Critical weaknesses

W1 It is unclear what actually drives the gains. L2 and L3 give the model extra statistics and error feedback. The paper does not show whether better memory representation helps, or whether simply providing that extra information explains the improvement.

W2 Key baseline implementations violate temporal causality. Some baselines use the final duration or prediction residual of an earlier anchor before its obscuration has ended and the outcome is observable. Consequently, the comparisons against simple predictors and the mechanism analysis in App E require recomputation. The magnitude of the resulting changes remains unknown, but these conclusions are not yet reliable.

W3 The evaluation mainly uses offline and single-turn prediction. It primarily tests how LLMs use contextual information, rather than agent capabilities. The memory mechanisms are also fairly conventional and have poor novelty. Moreover, the positive results are concentrated on one type of task, leaving the conclusions suggested by the title supported.

Questions for authors

Q1 Why does the sec introduction have no citations?

Q2 When computing persistence, climatology, and rolling bias correction, were historical outcomes required to have been confirmed before the current prediction time? How does the implementation enforce this when multiple anchors occur within one ongoing obscuration?

Q3 After enforcing outcome-confirmation timing, what are the corrected baseline scores and paired intervals for L2 plus arithmetic correction versus L3? Do the conclusions of Appendix E survive?

3 comments

Thank you. Your W2 is correct, and we are grateful it was caught. General response 1 gives the full account and the table of corrected numbers; here we answer each question directly. Section and appendix names are those of the revised PDF.

Q2: were historical outcomes required to be confirmed before the prediction time? No, not for persistence, climatology or the rolling bias correction. They selected earlier outcomes by anchor order alone, so when several anchors fell inside one ongoing obscuration, a later anchor could read the duration, or the residual, of an earlier anchor in the same obscuration before it had ended. This happened at 84 of 170 scored anchors, 50 of them probes. For example, an anchor with 22 minutes remaining received persistence 26: the duration recorded 4 minutes earlier in the same obscuration, not confirmed until 26 minutes later.

How it is enforced now. Each outcome carries the time of its confirming frame (the third consecutive unobscured frame of the truth definition), and it is usable only if uncensored and confirmed at or before the anchor's time. Persistence takes the most recently confirmed outcome, climatology their median, and the rolling correction their residuals only. A regression test builds probe anchors inside one obscuration and asserts that no baseline can read its outcome. The agent-side inputs were causal from the start, so every agent row is unchanged.

Q3: corrected scores, and Appendix E (minutes; run 1 / run 2; baseline MAEs do not depend on the run):

  • Persistence 11.82 (bias −10.73), climatology 11.52, persistence with its rolling bias removed 11.57. The pre-fix 8.31 and 7.22 are retracted and appear in the PDF only labelled as such.
  • Persistence − L3: −2.40 [−7.17, +1.60] / −3.62 [−9.70, +0.86], unresolvable in both runs. Climatology − L3: −2.63 [−6.65, +0.69] / −3.99 [−9.40, −0.11]. Persistence with correction − L3: −2.61 [−6.55, +0.54] / −3.91 [−9.38, −0.23].
  • L2 + arithmetic correction − L3: −0.13 [−1.59, +1.38] in run 1 and −0.08 [−1.50, +1.34] in run 2, on 153 and 151 commonly answered anchors over 22 observing days.

What survives in Appendix E: neither run separates L3 from the arithmetic baseline, and in the "Left" column L3 keeps the least remaining correction of the four rungs in both runs (−0.60 and −0.21, against L0's −3.18 and −4.13). What we changed: the two arms receive different quantities. The arithmetic arm subtracts L2's own median residual over outcomes confirmed by the current anchor, while the L3 note states L3's own residuals on previous days, with their count and median absolute error. Appendix E therefore says only that the result is consistent with a simple bias correction and that no additional MAE benefit of L3 over that baseline was detected; it claims no equivalence. Withdrawn: "the strongest model-free predictor is two lines of arithmetic", and persistence's slope as a tracking reference (corrected persistence runs low, pushing its slope to 1.26); climatology (0.43) and conditional climatology (0.33) are the fair references, and the agent's 0.22 and 0.12 stay below both. In Appendix A, GRU and attention are no longer resolvably worse than persistence.

W1: what drives the gains. The matched-content control (general response 2 (a), Section 4.1, Appendix H) changes only the statistics block of the recorded L2 prompts. Neither of two runs detected an MAE advantage of the table over the same numbers in sentences (+0.05 [−0.80, +0.82], −0.10 [−1.02, +0.59]), nor of the full table over the conditional median. We have narrowed the claim accordingly: which quantities are stored changes what a memory can carry, with no format effect detected. All arms keep the statistics, and we claim no equivalence. L3's gain over L2 does not replicate (−1.55 [−3.26, −0.19], −0.99 [−2.35, +0.18]) and is stated as a point estimate.

W3: offline, single-turn, conventional mechanisms, one task. We agree the mechanisms are conventional: the memory system was deployed before this study and is evaluated unchanged, and the contribution is the measurement. The writing side is multi-day (the agent's own nightly consolidation over 62 replayed days). The reading side is one call per anchor by design, so every configuration answers the same 170 anchors and differences can be paired; we accept that it mainly tests how a model uses its context. The positive result rests on one task at one site, and the abstract and Limitations are scoped to it.

Q1: why the introduction had no citations. There was no good reason. It now cites Park et al. (2023), Packer et al. (2023), Sumers et al. (2024), Bouthillier et al. (2021), Yuan et al. (2025) and Zhang et al. (2026).

Reviewer_frGTOpenReview ↗

Response to Authors

Thank you for your response and clarifications. We acknowledge your efforts and will raise our score to 2. However, our main concern regarding the limited novelty of the proposed method remains unresolved.

Thank you for your constructive feedback and for reconsidering the score.

We acknowledge that the individual memory mechanisms employed in our study are not fundamentally new, and we do not intend to claim novelty in these components.

Rather, our primary contribution is an empirical investigation of how different experience representations and their accessible information influence the utility of persistent memory in scientific agents.

Through controlled comparisons using a common underlying observation archive, cross-model replay, and writer–reader evaluations, we identify conditions under which narrative memory may be ineffective or even detrimental, while structured statistical information and historical error feedback can improve calibration.

We recognize that the evaluated representations also differ in the information explicitly made accessible, and have clarified the scope of our conclusions accordingly.

We hope this distinction better communicates the contribution of our work as an empirical finding rather than a fundamentally new memory algorithm.

Reviewer_9CHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at how different representations of persistent memory can influence an LLM agent's ability to use accumulated experience. The authors use operational data from a solar telescope to evaluate different configurations for predicting how long an obscured target will remain unobservable. They find that structured memory reduces prediction error relative to no memory, along with other findings. They extend their investigations to five models, and find that structured statistical information is more useful than narrative summaries for this particular task.

Critical strengths
  • The authors use a careful experimental methodology (repeated runs, replication criteria, multiple model comparisons, robustness analyses) and also transparently report negative results (which is always appreciated!)
  • The author present interesting evidence that the effectiveness of memory depends on representation and write quality, which are useful insights for designing effective agents with persistent memory.
Critical weaknesses

I believe there are a couple of weaknesses that are potentially concerning, and make me lean towards rejection:

  1. The most serious issue: there is a chance that the information content and the representation are confounded. The authors' main claim is about the representation of information, but L2 adds statistical quantities that don't exist in L1, and L3 also adds past prediction-error info not contained in the original observation archive. As such, it is possible that some improvements could arise from simply supplying more task-relevant information, rather than from the representation itself. The authors partially address this with their narrative-removal experiment, but they do not directly compare equivalent info-containing representations. This weakens the paper's central claim.

  2. There is also limited generality, and while the authors frame this as more of a diagnostic probe, the practical significance, including over simple statistical methods, is unclear. The strongest memory configurations is worse than a trivial persistence baseline, so this cannot be a competitive memory architecture. Overall, it is unclear about the significance of the paper for broader LLM-agent memory design.

Questions for authors
  1. Can you add any experiments that better separate the effects of memory representation and introducing more task-relevant info? For example, comparing narrative and structured memories that encode approx. the same facts. Basically, a matched-content control.

  2. How much incremental improvement does L3 provide over L2 (rather than over L0)? And is the difference between L2 and L3 statistically significant across repeated runs?

  3. To what extent do you feel that these findings can generalize to other tasks, and its usefulness overall on top of simple statistical methods?

1 comment

Thank you for the review. The two general responses report a fixed causality error in two model-free baselines, which bears on your third question (no agent number changed), and the matched-content control you asked for. Section and appendix names are those of the revised PDF.

Q1: separating representation from added task-relevant information. We agree this is the main confound: the ladder is cumulative, L2 adding medians the narrative does not state and L3 the configuration's own past error. The control you describe is general response 2 (a), reported in Section 4.1 and Appendix H. On the recorded L2 prompts only the statistics block changes, as the table, as the same numbers in sentences, or as the conditional median only. In two runs, neither detected an MAE advantage of the table over the sentences (+0.05 [−0.80, +0.82], −0.10 [−1.02, +0.59]; the penalised variant also spans zero), nor of the full table over the conditional median. The reader was DeepSeek V4 Flash, as the MiniMax-M3 quota ran out; a partial M3 run agrees (−0.09 [−0.60, +0.48], one run).

We have narrowed the paper to what this supports. The subtitle is now "A Matched-Content Study of Memory in an Observing Agent", and contribution 1 says that which quantities are derived and stored changes what a memory can carry, with no format effect detected. Two limits stay with it: all arms keep the statistics inside the narrative context, so the control does not measure the gain over a same-session arm without statistics; and intervals containing zero in two runs do not establish equivalence.

Q2: what does L3 add over L2, and is it significant across runs? It does not replicate. Paired on the anchors both answered:

run 1run 2
L3 − L2−1.55 [−3.26, −0.19]−0.99 [−2.35, +0.18]
L3 − L2, unanswered charged 30 min−1.57 [−3.43, −0.22]−1.04 [−2.52, +0.40]

Both runs point the same way, and dropping any single observing day leaves the sign unchanged; dropping the most influential day makes run 1 unresolvable too (−0.93 [−1.96, +0.08]). Only run 1 excludes zero, so by our rule the increment is a point estimate of about 1–1.5 minutes, and the abstract, contribution 1 and Section 4 say so. L2 over L1 does not replicate either (−4.45 [−8.32, −0.80], −1.99 [−5.83, +2.40]). What replicates is L2 and L3 against L0 (Table 1). L3 also shows no detected MAE gain over L2 with its own median residual subtracted (−0.13 [−1.59, +1.38], −0.08 [−1.50, +1.34]; Appendix E).

Q3: generality, and usefulness on top of simple statistics. Corrected numbers first. The best model-free predictor is climatology at 11.52, and corrected persistence is 11.82; L3 reaches 14.78 and 15.73. Persistence − L3 is unresolvable in both runs (−2.40 [−7.17, +1.60], −3.62 [−9.70, +0.86]); climatology − L3 is resolvable in run 2 only. So "worse than a trivial persistence baseline" no longer holds as a resolvable statement, and neither does the opposite. No reasoning-on configuration reaches climatology's point estimate.

We agree the configurations are not a competitive predictor and do not propose them as one. The task is a probe in which the current observation is insufficient, and the quantity of interest is what accumulated memory changes in a general-purpose agent. On that, the paper supports three things: the memory-free agent is systematically biased (+13 to +28 minutes across five models, against a truth median of 8); narrative memory does not remove the bias and raises it on 4 of 5 models; and a statistical snapshot removes most of it, as a level correction rather than better tracking of individual cases (slope 0.22 and 0.12). For memory design this suggests storing derived quantities when later decisions are quantitative, and measuring the memory-free bias before building a ladder.

On generality, the evidence is one deployment: one target with a positive result replayed across five models, and a second agent on the same instrument whose probability ladder shows no resolvable gain (Section 7, Appendix M; observed half-widths 0.0099–0.0185 Brier on the commonly answered anchors, so a small benefit is not excluded). The cross-task contrast is exploratory, and the Limitations state the single-site scope. We do not claim the result transfers to other domains.

Reviewer_TYLTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how memory construction affects a deployed observing agent's predictions. Using a solar-telescope archive, it compares no memory, generated nightly prose, the prose supplemented with stratified historical statistics, and a further note on past prediction errors. Statistics and error feedback improve predictions of remaining obscuration time, but simple statistical rules remain more accurate. The authors use cross-model replay, a writer-by-reader matrix, narrative removal, and an arithmetic bias correction to examine what drives the improvement. They find that much of the gain is a correction of systematic overestimation rather than better tracking of individual events. A second agent provides a further task on which the tested memory configurations yield no detectable improvement.

Critical strengths
  1. The paper provides a useful measurement on an operational archive and makes the comparison with simple predictors explicit. Table 1 reports both repeated runs and shows the remaining gap to persistence. Section 4.1 and Appendix E go beyond an on/off memory ablation: they remove the narrative and compare the error-feedback note with subtracting historical bias. These controls help distinguish an improvement in the agent from a stronger prediction method, and a lower mean bias from better conditional prediction.

  2. The treatment of experimental failures is a strength. The duration results use observing-day bootstrap intervals, and Appendix A checks influential days and unanswered-anchor penalties. The cross-model analysis reports the failed monotonicity criterion rather than redefining it. The writer-by-reader matrix and the serving-fingerprint checks also show why seemingly reasonable pairwise comparisons can support the wrong explanation.

Critical weaknesses
  1. The paper clearly acknowledges that the memory configurations add derived information rather than holding the content fixed (Section 3). To help interpret the improvement, it would be useful to include a small comparison presenting the same historical statistics as prose and as a table. This would help distinguish the benefit of supplying the statistics from any additional benefit of their presentation, and strengthen the discussion in Section 8.

  2. The second-agent experiments do not yet show that memory fails to help because the starting bias is small. A model can overestimate in some cases and underestimate in others, so its average bias is near zero even though many predictions could improve. The probability task shows no detectable benefit from memory, but it also changes the task and error measure. The count task only tests L0, so it cannot show what adding memory would do. Section 7 makes this distinction, but Appendix K still uses the stopped count experiment to address whether the probability metric could hide an effect. That question remains open. Low average bias alone is therefore not enough to justify skipping the memory comparison.

  3. A smaller presentation issue is that the main text has no overview figure for the experimental pipeline. Section 3 describes L0–L3 in prose, but it takes some effort to follow how the archive is turned into narrative memories, statistical summaries, and error feedback, and how these enter the prediction task. Although Tables 1 and 2 report the main results, useful supporting views of bias, error, and paired effects are deferred to Figures 2–6 in the appendix. A pipeline diagram and a compact result plot in the main text would make the design and findings easier to assess.

Questions for authors
  1. If the same historical statistics are written as ordinary sentences instead of a table, does the improvement remain? How does this compare with giving the model only a simple historical median?

  2. On the probability task, do overestimates and underestimates cancel out in the overall bias? Please report errors across different conditions or probability ranges, and how large a benefit the confidence intervals can rule out. For the count task, is the bias threshold simply a way to limit experiment costs, or is there evidence that memory cannot help below it?

  3. Would the writer ranking stay the same if every memory record were given the same weight? The supplementary code shows that these weights vary with the writer and are included in the prompt, so the comparison changes more than the text. Also, which writer-reader and cross-model comparisons were run more than once? Appendix N says every result follows the repeated-run rule, but I could not identify those repeats for all of these comparisons.

  4. As a presentation improvement, could you add a diagram showing how L0–L3 are constructed and used in prediction and evaluation? It would also help to bring a compact bias/error plot or the key paired-effect intervals into the main text, alongside Tables 1 and 2.

3 comments

Thank you for a careful review. You identified two errors, which we concede: the weight confound and the replication sentence. The two general responses report a fixed causality error in two model-free baselines (no agent number changed) and the two experiments your questions asked for. Section and appendix names are those of the revised PDF.

Q1: the same statistics as sentences, and a median-only arm. This is the matched-content control, general response 2 (a), reported in Section 4.1 and Appendix H. On these L2 prompts neither of two runs detected an MAE advantage of the table over the same numbers written as sentences (+0.05 [−0.80, +0.82], −0.10 [−1.02, +0.59]; the 30-minute penalised variant also spans zero), nor of the full table over the conditional median alone (−0.17 [−1.07, +0.84], −0.29 [−1.18, +0.71]). The median-only arm did shift predictions down in run 2 (−1.56 [−2.48, −0.57]) without a detected change in error. All arms keep the statistics and the narrative context, and two runs do not establish equivalence. A related fact: conditional climatology (14.24) is the model-free form of "only the median", and L3 is not resolvably different from it (+0.02 [−0.98, +0.96], −0.95 [−2.48, +0.49]). The Discussion (Section 8) now rests on this.

Q2a: do over- and underestimates cancel on the probability task? We binned every rung into ten reliability bins on the anchors all rungs answered (Appendix M). This uses the Brier score pooled over anchors, not averaged over days as in the paper, so it shows the shape of the errors only. For V4 Pro and Flash the reliability term is 0.8–4.8% of the Brier score, with no large deviation hidden by cancellation. For MiniMax-M3 the memory-free bias of −0.065 is under-forecasting in 10 of 10 bins (e.g. forecasts in [0.3, 0.4) average 0.349 against an observed 0.600, n = 15), not cancellation. The structured rung lowers its reliability term from 0.0080 to 0.0049, but resolution falls from 0.0907 to 0.0794, so the Brier score does not improve.

Q2b: how large a benefit can the intervals rule out? On the anchors all four rungs answered, the paired Brier differences have 95% half-widths of 0.0099–0.0185 (7–13% of the memory-free score), and some still admit an improvement of up to 0.024 (Pro, narrative). Of nine paired intervals (three rungs, three models), eight cross zero and one is resolvably worse: Flash's calibrated rung, +0.0118 [+0.0007, +0.0246].

Q2c: is the count-task threshold a cost limit? Yes. The 0.5 gate was fixed in code before the calls, but it is a spending rule, set between two naive gaps measured in advance (0.15 and 0.72 groups) rather than derived from a power or error argument. The measured bias is +0.262 [−0.121, +0.679], which includes 0.5 (17.5% of bootstrap draws reach it). We have no evidence that memory cannot help below the gate. You were also right that the count task, which ran only L0, cannot address whether the probability metric hides an effect. Appendix M no longer argues from it: it states that the count target estimates no memory effect, that whether a memory helps only where the memory-free configuration starts substantially biased is a hypothesis the two completed ladders suggest and do not establish, and that the contrast with the duration task is exploratory.

Q3a: would the writer ranking survive equal weights? Partly; see general response 2 (b), the paragraph "The printed weights" in Section 6.1, and Appendix I. For both DeepSeek readers M3 stays the worst writer by point estimate; one of the four comparisons (Pro reader, against Flash's prose) spans zero. The Flash reader's full ordering changed: its own prose improves by −7.45 [−14.80, −2.02] once its lower self-proposed weights become 0.80. The M3 reader row is missing. We claim only which writer is worst for those two readers.

Q3b: which comparisons were run more than once? The sentence you quote was false, and Appendix P now lists the runs. Two repeated runs: the duration ladder (Table 1), the narrative-removal test (Section 4.2), and the matched-content control (Section 4.1); the arithmetic correction (Appendix E) is derived from the duration ladder's two runs. One run per model or cell: the cross-model replay, the reasoning-off control, the writer-by-reader matrix and its equal-weight rerun, the pairwise production stage, the model-version contrast, the form-sensitivity gate, and the second agent's ladders. The cross-model M3 row resends run 1's prompts and the version contrast compares two weight versions; neither is a repeat. The L3-over-L2 increment does not replicate and is stated as a point estimate.

Q4: diagram and result plot. The main text now has a diagram of how L0–L3 are built from one archive, with the block the matched-content control replaces (Section 3), and a forest plot of all fifteen paired intervals against no memory, three rungs on five models (Section 5). Table 1 carries the paired intervals of both runs.

Thank you for the detailed response. The clarifications and additional evidence have substantially addressed my main concerns and strengthened my assessment of the paper. I have therefore increased my rating. I encourage the authors to incorporate the relevant explanations and results into the revised manuscript.

Thank you very much for your careful reconsideration and constructive feedback.

We sincerely appreciate your recognition of our clarifications and additional evidence. We have incorporated the key explanations and supporting results into the revised manuscript and supplementary materials, and will ensure that these points are clearly presented in the final version.

Thank you again for your valuable suggestions and thoughtful assessment.

Reviewer_WrQ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how different memory representations affect the performance of LLM agents. Using a real-world solar telescope agent, the authors compare narrative memories, structured statistical memories, and memories containing information about the agent's previous prediction errors. The results show that statistical memories improve prediction accuracy, while narrative memories provide little benefit and can even harm performance. The paper also investigates the relationship between memory writers and readers, finding that stronger task-solving models do not necessarily produce more useful memories.

Critical strengths

I particularly liked the use of a concrete, real-world application at the center of the paper. It makes the problem easy to understand and the experiments straightforward to follow. The paper is generally well written, clearly organized, and easy to follow. The paper evaluates several relatively recent LLMs from different laboratories, providing useful evidence of how the findings transfer across models. The authors also provide extensive experimental controls, repeated runs, and statistical analyses. I appreciate their transparency regarding unexpected results and failed hypotheses.

Critical weaknesses

Some of the main findings are relatively expected, particularly that providing structured statistical information improves performance over textual summaries on a quantitative prediction task. The statistical memories explicitly provide information relevant to the prediction, making this result somewhat unsurprising. Additionally, the experiment does not completely separate the benefits of the representation itself from those of adding task-relevant statistical information. The evaluation relies primarily on a single real-world application. While keeping one concrete example at the center of the paper makes it easier to follow, including secondary examples or tasks would strengthen the generalizability of the findings.

Questions for authors

I think Contribution 2 could be extended in a particularly interesting direction: measuring a model's intuition about what information is worth remembering for future use. For example, different models could receive the same history of cloud observations and independently decide what information to preserve, without being explicitly told which future questions they will need to answer. A model might proactively identify that cloud-duration distributions and statistical patterns are important and include them in its memory, while another might produce only generic textual summaries. Evaluating which memories are more useful for future tasks would help measure the models' ability to anticipate what information should be retained. This differs from the current writer-by-reader experiment, which measures the downstream usefulness of memories produced by different models for a specific task, without directly evaluating their ability to anticipate future information needs. Could the authors include at least a preliminary experiment investigating this capability? A broader benchmark could eventually be developed around this idea, assessing memory-writing usefulness across tasks and models. I find this direction potentially more exciting than the current representation comparison, and even a preliminary experiment could raise my overall rating from 3 to 4. Could the authors include a smaller secondary application or prediction task to examine whether the main representation findings extend beyond the solar telescope setting?

1 comment

Thank you for the review and for the concrete suggestion. The two general responses report a fixed causality error in two model-free baselines (no agent number changed) and two new experiments. Section and appendix names are those of the revised PDF.

W1: an expected result; representation versus added information. We agree that the ladder is cumulative: L2 adds medians the narrative does not state, and L3 adds the configuration's own past error (Section 3). The matched-content control, general response 2 (a) and Section 4.1, holds the statistics fixed and changes only their presentation. On these L2 prompts neither of two runs detected an MAE advantage of the table over the same numbers as sentences (+0.05 [−0.80, +0.82], −0.10 [−1.02, +0.59]), nor of the full table over the conditional median alone. All arms keep the statistics, so this narrows our claim to what is stored rather than separating storage from presentation in general, and we do not claim equivalence. The subtitle is now "A Matched-Content Study of Memory in an Observing Agent".

Two parts of the result are less expected than the headline. The statistics remove most of a systematic overestimate (L0 bias +16.4 / +13.3 minutes, L3 +2.1 / +3.1) but leave the slope of truth on prediction at 0.22 / 0.12, so the gain is a level correction rather than better tracking of individual cases (Appendix E). And the narrative form raises the bias above the memory-free level on 4 of 5 models in Table 2 (one run per model).

Q1: do writers anticipate what is worth remembering without being told the question? The existing records answer part of this at no cost, because the matrix writers were in that situation. We added the analysis to Appendix I (paragraph "What the writers were told, and what they kept"). One system prompt serves all 62 nightly consolidation calls and asks for three to five insights capturing a pattern, anomaly or trend; neither it nor any user prompt names the remaining-occlusion question, a forecast, or the answer field. Twenty-two of the user prompts contain resolved occlusions, so a duration was there to be kept.

writerstates a duration (deployed criterion)stricter patterncolumn-mean MAE
MiniMax-M334 / 310 (10.97%)8.06%43.38
DeepSeek V4 Flash28 / 286 (9.79%)8.39%35.37
DeepSeek V4 Pro21 / 278 (7.55%)8.27%32.03

The deployed criterion (a number with a minute unit near an ending word) orders the writers opposite to their usefulness; the stricter one (minutes beside an occlusion or clearing word) barely separates them. So in this panel the share of duration statements does not account for the writer ordering. This is exploratory and post hoc, with three limits: one downstream question, three writers, and definitions that converge under the stricter pattern. With every printed record weight equalised, M3 remains the worst writer by point estimate for both DeepSeek readers we could re-run, with one of the four comparisons spanning zero (general response 2 (b)).

This is preliminary evidence on a single downstream question, which is the scope we claim. We think your framing describes contribution 2 better than ours: the same history, writers not told the question, memories scored by downstream use. Testing it across several future tasks, and the benchmark you describe, remain open.

Q2: a secondary task. The paper has one on the same instrument: the controller's second agent (Section 7, Appendix M), a probability forecast over 477 anchors and 53 observing days, with all four rungs on three models. Of nine paired intervals, eight cross zero and one is resolvably worse (Flash, calibrated: +0.0118 [+0.0007, +0.0246]). On the anchors all four configurations answered, the observed half-widths are 0.0099–0.0185 Brier, and some intervals still admit an improvement of up to 0.024. It is a different agent on the same instrument, not a different domain, and its result is a null, so it bounds the positive result rather than extending it. The cross-domain validation you ask for remains open; the Limitations state the single-site scope.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): a causality error in two model-free baselines, the corrected numbers, and what we retract

We thank the four reviewers. The revised PDF is uploaded. This comment covers the error Reviewer frGT found (W2, Q2, Q3) and every claim we retract; the second covers the new experiments and the revised positioning. Section and appendix names are those of the revised PDF.

What was wrong. Persistence and climatology, and the rolling bias correction built from the same history (Appendix E), chose earlier outcomes by anchor order. They never checked that the earlier occlusion had ended and been confirmed by the current anchor's time. When several anchors fall inside one ongoing occlusion (15-minute probes and intermediate state changes), a later anchor read the duration recorded at an earlier anchor of the same unfinished occlusion: its own answer plus the time between the two anchors. This happened at 84 of the 170 scored anchors, 50 of them probes.

What was not affected. Every agent-side input was causal. The L2 snapshot and the L3 note are rebuilt at night from earlier days only; the within-day "resolved so far" feature enters a prompt only once the occlusion's confirming frame precedes the anchor; the narrative is written at night; conditional climatology uses earlier days only. No agent number changed.

Fix. Each outcome carries the time of its confirming frame, the third consecutive unobscured frame of the truth definition. An outcome is usable at an anchor only if it is uncensored and confirmed at or before the anchor's time; an uncensored outcome without a confirmation time raises an error. A regression test places probe anchors inside one occlusion and asserts that no baseline can read its outcome. All baselines were recomputed with zero model calls, using the same day-clustered bootstrap (10,000 resamples, fixed seed).

Corrected numbers (minutes; r1 / r2 are the two repeated runs; the baselines do not depend on the model):

quantitybefore (retracted)after
persistence MAE (bias)8.31 (+0.0)11.82 (−10.73)
climatology MAE11.3811.52
persistence + rolling bias correction7.2211.57
persistence − L3, r1 / r2−6.06 [−9.54, −2.94] / −7.19 [−12.17, −3.12]−2.40 [−7.17, +1.60] / −3.62 [−9.70, +0.86]
climatology − L3, r1 / r2not reported−2.63 [−6.65, +0.69] / −3.99 [−9.40, −0.11]
L2 + arithmetic correction − L3 (App. E)−0.20 [−1.64, +1.31] / −0.05 [−1.52, +1.39]−0.13 [−1.59, +1.38] / −0.08 [−1.50, +1.34]
persistence calibration slope (App. E)0.721.26
persistence, supervised subset (App. A, 114 anchors)7.027.83

Where the PDF still prints 8.31 and 7.22, they are labelled as the retracted pre-fix values.

What we retract.

  1. "A one-line statistical rule is more accurate than the agent" and "the strongest model-free predictor is two lines of arithmetic". Now: no reasoning-on configuration reaches the point estimate of the best model-free predictor (climatology, 11.52, against L3's 14.78 and 15.73), but persistence − L3 is unresolvable in both runs and climatology − L3 in run 1. Two reasoning-off V4 Pro configurations reach climatology with unanswered anchors dropped (11.37 and 11.14); charged 30 minutes, the lowest reasoning-off MAE is 13.41 against 11.63.
  2. Appendix A: GRU and attention are no longer resolvably worse than persistence (+3.00 [−0.14, +5.61], +1.55 [−1.08, +4.57]).
  3. Appendix E's mechanism wording. The arithmetic arm subtracts L2's own median residual over outcomes confirmed by the current anchor; the L3 note states L3's own residuals on previous days, with their count and median absolute error. These are different quantities. The appendix now says only that the result is consistent with a simple bias correction and that no additional MAE benefit of L3 over that baseline was detected.
  4. L3 over L2 as an established increment: −1.55 [−3.26, −0.19] and −0.99 [−2.35, +0.18]. One run excludes zero, so by our rule it did not replicate; it is a point estimate.
  5. "Every result in this paper is therefore reported from repeated runs" (now Appendix P, which lists the single-run comparisons). Only the duration ladder, the narrative-removal test and the matched-content control have two runs.
  6. Appendix M: the inference from the count target, which ran only L0, to the probability target's null, and our earlier statement of the detectable effect (now the observed half-widths; general response 2).

What survives. The three baselines remain resolvably better than L0 in both runs (persistence −12.05 [−19.96, −6.44], −9.83 [−14.89, −5.96]). L2 and L3 against L0 are resolvable in both runs, L1 in neither. In Appendix E's "Left" column L3 keeps the least remaining correction in both runs (−0.60, −0.21; L0 −3.18, −4.13). The agent's slopes (0.22, 0.12) stay below climatology's 0.43 and conditional climatology's 0.33: a level correction, not conditional tracking.

AuthorsOpenReview ↗

General response (2/2): two new experiments, the narrowed claims, and where they are in the revised PDF

(a) Matched-content control (9CHu Q1, TYLT Q1, frGT W1, WrQ5 W1; Section 4.1 "Is it the quantities or the format?", Appendix H). On the recorded L2 prompts of both runs only the statistics block changes; the harness asserts that every other byte is identical. Arms: the table resent; the same snapshot as sentences, with the multiset of numbers asserted identical prompt by prompt; and the conditional median only. The rule fixed before the calls: unless sentences differ resolvably from the table in both runs, we do not claim a format effect. The completed experiment used DeepSeek V4 Flash after the MiniMax-M3 quota was exhausted (all 1,020 calls on one serving build, temperature 0, arms interleaved by anchor). The sentences are Chinese, the language of the rest of the prompt; and each run has 161 edited anchors over 21 observing days (the 9 with an empty snapshot are excluded).

run 1run 2
MAE: table / sentences / median only16.72 / 16.74 / 16.3914.76 / 15.04 / 14.95
sentences − table (n = 143 / 144)+0.05 [−0.80, +0.82]−0.10 [−1.02, +0.59]
same, unanswered charged 30 min (n = 161)+0.52 [−0.60, +1.80]+0.36 [−1.17, +1.91]
median only − table (n = 149 / 147)−0.17 [−1.07, +0.84]−0.29 [−1.18, +0.71]

What this supports is narrow: on these L2 prompts, neither run detected an MAE advantage of the table over the same content in sentences, nor of the full table over the conditional median. All three arms keep the statistics inside the L2 narrative context, so none measures the gain over a same-session arm without statistics. Two runs whose intervals contain zero do not establish equivalence, which would need a margin fixed in advance. A partial M3 run (run 1 only, about 140 pairs over 19 days) gives −0.09 [−0.60, +0.48] for sentences − table: consistent, not a replication.

(b) Writer-by-reader matrix with equal printed weights (TYLT Q3a, WrQ5 Q1; Section 6.1, paragraph "The printed weights"; Appendix I). Every printed weight was set to 0.80, with the original-weight cells resent in the same session, interleaved by anchor, as a control. Six of nine cells completed, for the two DeepSeek readers. The M3 reader row stopped on the same quota (43–45 of 170 anchors per cell) and is not reported. Pro met its published build throughout; the Flash alias has moved, so Flash is compared within this session only. The control reproduces the published ordering for both readers. M3's prose minus another writer's, equal weights, unanswered dropped:

readervs Flash's prosevs Pro's prose
Flash+11.91 [+2.98, +23.05]+10.75 [+1.29, +22.46]
Pro+10.36 [−1.80, +27.25]+18.43 [+2.92, +38.62]

For both completed readers M3 is the worst writer by point estimate, and all eight point estimates across both weight conditions are positive. One comparison, the Pro reader on M3's against Flash's prose, has an interval spanning zero. The Flash reader's full ordering changed: its own prose improves by −7.45 [−14.80, −2.02] at equal weights and moves ahead of Pro's. The Pro reader's ordering is unchanged. We claim which writer is worst for these two readers, not a complete ranking, and nothing for the missing M3 reader.

(c) What the writers kept without being told the question (WrQ5 Q1; Appendix I, paragraph "What the writers were told, and what they kept"). An exploratory, post-hoc count over the existing records, zero calls; details in our reply to WrQ5.

Second agent (Section 7, Appendix M). Of nine paired Brier intervals (three rungs against no memory on each of three models), eight cross zero and one is resolvably worse (Flash, calibrated rung: +0.0118 [+0.0007, +0.0246]). On the anchors all four configurations answered, the observed 95% half-widths are 0.0099–0.0185, and some intervals still admit an improvement of up to 0.024. The count target ran only L0, so it says nothing about the probability null; the contrast with the duration task is labelled exploratory.

Revised positioning. The subtitle is now "A Matched-Content Study of Memory in an Observing Agent" (main title unchanged). Contribution 1 reads: which quantities are derived and stored changes what a memory can carry, with no format effect detected. The abstract, the Section 4 heading ("The stored quantities change what the memory can carry") and the Discussion are narrowed to match.

Other changes: corrected baselines in Table 1, Appendix A and Appendix E; an L0–L3 construction diagram (Section 3) and a forest plot of all fifteen cross-model paired intervals (Section 5) in the main text; citations in the introduction; the replication list in Appendix P. The code is attached, with a per-anchor results bundle for (a) and (b); the full products and per-call transcripts will be released on acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.