acceptodds
Under review as a conference paper at ICLR 2027

Decision Instability Beyond Retrieval Metrics in Consolidated Agent Memory

Abstract

Language agents increasingly consolidate stored experiences to operate under bounded memory, yet consolidated memory is commonly evaluated by whether relevant information remains retrievable. We show that retrieval preservation does not imply preservation of the decisions supported by memory. Using a controlled intervention that holds target queries and outcome labels fixed while storing the same experiences in either original or consolidated form, we trace the effects of consolidation through the representation → retrieval → decision pathway. Aggregate retrieval success remains unchanged, with Hit@3 of 87.06% in both conditions, while downstream accuracy also remains nearly unchanged. Despite this aggregate stability, consolidation substantially reorganizes retrieved evidence and produces decision changes beyond ordinary repeat variability, with cross-condition disagreement exceeding same-memory disagreement by 5.20 percentage points. This instability concentrates on a reproducible subset of experiences. Vulnerable instances recur 9.43× and 10.84× above independence expectation under consumer-model and retriever substitution, respectively, even though the retrievers exhibit weak correspondence in their consolidation-induced changes. These results show that consolidation can preserve conventional retrieval and task-level performance while changing the evidence presented to the consumer and the decisions associated with particular experiences. Evaluating consolidated memory therefore requires considering not only whether relevant information remains retrievable, but whether the decision-relevant structure of stored experience is preserved.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, memory consolidation, retrieval-augmented generation, decision fidelity, evaluation, large language models, retrieval robustness, instance-level analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Bd6POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether consolidating stored experiences can change downstream decisions even when aggregate retrieval success and task accuracy remain stable. It constructs paired narrative and risk-attribute memories from 394 occupational-incident records, holds the target narratives and outcome labels fixed, excludes self-retrieval, and evaluates nine-class accident classification. The primary experiment reports identical Hit@3 of 87.06%, but a 5.20-percentage-point excess of cross-condition decision disagreement over same-memory repeat disagreement. Additional experiments substitute the consumer or retriever and examine recurrence of vulnerable targets.

Critical strengths
  1. The paired intervention is easy to interpret: the incident, target query and historical outcome remain fixed while the stored representation changes. Excluding the target from retrieval avoids a direct self-retrieval shortcut.

  2. The distinction between aggregate accuracy and instance-level decision stability is useful. The repeat control and paired bootstrap make the reported disagreement more informative than a simple comparison of two stochastic model outputs. The paper also reports both harmful and beneficial decision changes rather than equating every change with an error.

  3. The retrieval analysis goes beyond a single embedding model, and the concentration near the caught-between/struck-by boundary gives the study an interpretable observation beyond an aggregate score.

Critical weaknesses
  1. The experiment does not establish how much the task depends on memory. The consumer receives the complete target accident narrative, which may already contain most of the information needed for classification. Without a query-only baseline, it is difficult to distinguish disruption of genuinely necessary memory use from sensitivity to optional in-context examples. The paper acknowledges the single-corpus and non-sequential setting, but the central memory interpretation still needs this control. Moreover, decision preservation is a diagnostic, not automatically a desirable objective: the paper itself reports 25 incorrect-to-correct changes alongside 30 correct-to-incorrect changes across the two repeats.

  2. The intervention identifies a total effect, not which part of the representation–retrieval–decision pathway causes it. The retrieved record identities and the content shown for those records change together. A crossed control that independently fixes retrieved IDs and switches their displayed representations would distinguish retrieval-mediated effects from direct effects of compressed evidence. Also, label purity already drops from 66.92% to 57.70%, so the result is specifically a limitation of aggregate Hit@k and accuracy, not evidence that retrieval metrics in general fail to reveal a change.

  3. The cross-component localization null does not account for outcome-category difficulty. Table 4 shows that 12 of the 15 targets shared across consumers are caught-between cases. Using the category-specific vulnerable-set sizes in that table gives an expected overlap of approximately 4.30 under within-category independence, compared with 1.59 under the pooled null. The observed overlap of 15 is still substantially larger; this does not dismiss the finding, but it changes the meaning of the headline 9.43-fold enrichment. A stratified analysis is needed to distinguish transferable instance susceptibility from shared category composition. Two repeats also provide only a limited basis for labeling individual targets reproducibly vulnerable.

Questions for authors
  1. What is the query-only classification accuracy and repeat disagreement? Can the authors cross retrieved record identity with full versus consolidated evidence presentation to identify the main source of the decision changes?

  2. Does vulnerable-set overlap remain significant under a label-stratified permutation test, and how stable are the identified targets when more independent repeats are used?

  3. Can the authors provide the paired-record construction specification, complete prompts/category definitions and exact inference settings, together with the per-target predictions underlying the bootstrap and overlap analyses?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3

We thank the reviewer for the careful reading and constructive suggestions. We appreciate the recognition of the paired design and instance-level findings. The review helped us strengthen the controls and interpretation. Changes are marked in blue.

Summary of revisions: We added a query-only baseline, crossed retrieval/representation conditions, and five independent repeats across five conditions and two consumers (19,700 predictions). We added a second task, category-adjusted overlap tests, and Appendices A-D.

Responses to Critical Weaknesses:

  1. W1: We agree that the complete target narrative may already support classification. The new query-only baseline makes this clear: without retrieved memory, accuracy is 86.95% for Sol and 86.60% for Astra, compared with 88.02% and 88.68% under Full memory (Table B.1). Thus, memory improves aggregate accuracy only modestly. It does, however, affect individual predictions: adding Full memory produces repeat-controlled excess disagreement of +4.52 pp [2.89, 6.32], comparable to +5.40 pp [3.63, 7.35] for Full versus Consolidated memory (Table B.2). We have narrowed the claim to how memory changes decisions, not how much classification depends on memory. We also agree that preserving a decision is not necessarily desirable. Across five repeats, consolidation changed 16 majority decisions from correct to incorrect and 14 in the other direction under Sol (Astra: 13 and 14; Table B.3). Sections 5.3 and 6 now explicitly treat decision preservation as a diagnostic, not an accuracy objective.

  2. W2: We implemented the crossed controls the reviewer suggested. FC displays the Full-retrieved records in consolidated form; CF displays Consolidated-retrieved records in full form. This lets us compare retrieval identity while holding presentation fixed, and presentation while holding retrieved IDs fixed. Changing retrieved IDs produces larger contrasts (+5.24 and +4.37 pp) than changing only displayed content (+2.33 and +0.40 pp; Section 5.3, Table B.2). This is evidence that retrieval identity contributes more strongly in our setting, not proof of a universal causal mechanism. The reviewer is also right about retrieval metrics: label purity falls from 66.92% to 57.70%, and diversity rises from 1.57 to 1.86. Our revised claim is specifically about the limitations of aggregate Hit@k and accuracy, not all retrieval measures. Section 6 now recommends tracking neighbor identity and label composition.

  3. W3: We agree that the original pooled null overstated the enrichment. Therefore, we replaced it with a category-adjusted null preserving vulnerable-target counts within each outcome class, with exact p-values from the convolution of per-category hypergeometric distributions (Sections 3.3 and 4.5). With five repeats, 13 targets are shared across consumers versus 3.35 expected (3.88×, p = 9.35 × 10⁻⁹). The enrichment persists at thresholds 0.3 and 0.7 (4.00× and 3.28×). Within caught-between cases alone, 9 are shared versus 2.80 expected (p = 2.25 × 10⁻⁵). Under retriever substitution, 11 are shared versus 2.01 expected (5.48×, p = 1.01 × 10⁻⁸); only three are caught-between. All 13 consumer-shared targets have the same modal Full-to-Consolidated transition under both models, with each mode supported by at least four of five repeats. We retain and explicitly identify the two-repeat protocol for retriever substitution (Section 5.4).

Responses to Questions for Authors:

  1. Query-only within-condition repeat disagreement is 1.68% for Sol and 0.81% for Astra. The accuracy and crossed-condition contrasts are reported above and in Tables B.1–B.2.

  2. Yes. We use an exact category-stratified overlap test rather than relying on the pooled null. Results remain enriched across vulnerability thresholds, and all 13 consumer-shared targets retain the same modal decision transition across models (Section 5.4; Tables C.1–C.2).

  3. Appendix A specifies paired-record construction, all five conditions, classification prompts and category definitions, inference settings, normalization, and the repeat/bootstrap/overlap procedures. Appendix D details the second task; Appendices B-C provide additional results. The 19,700 per-target prediction records were used to verify the reported statistics. Individual records are not included in the appendix.

We appreciate the reviewer’s suggestions. They led to more informative controls, a fairer statistical comparison and a more precise statement of what the evidence shows.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for your careful and constructive review. We understand this is a busy period and we would welcome any further questions or comments on our response before the discussion period closes.

Best regards,

Reviewer_5bCSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this work, the authors argue that an appropriate evaluation protocol for memory "compression" methods in agentic systems (specifically, targeting external memory systems) requires measuring if the structure of past decisions is preserved, in addition to measuring aggregate retrieval quality.

The authors proceed with a controlled intervention approach: instead of tracking a complete trajectory of sequential agent decisions, they structure information such that attribute-style consolidation boils down to a chain involving representation, retrievel and decision pathway.

This measurement methodology is used to elucidate the role of external memory consolidation, by disentangling evidence from retrieval success, to study instabilities. The methodology rests on four metrics: representation fidelity, retrieval identity, retrieval success, and retrieval composition. This contrasts aggregate success metrics that might hide changes in the retrieved evidence.

Overall, the take home message from this work is that memory consolidation approaches should be evaluated by distinguishing aggregate retrieval metrics from decision preservation.

Critical strengths
  • Studying the implications of memory consolidation in external memory systems for language-agents is an important endeavor, which can potentially extend (at least methodology and protocol wise) to more general (non-language) settings. I think the authors did a good job in building upon existing literature to define a more fine-grained method to measure the impact of consolidation, going beoynd the conclusions that can be drawn by focusing on aggregate metrics alone
  • This paper is clearly written, easy to follow, and well organized. The methodology is clear, the mathematical notation is used sparingly and easy to understand. The experimental protocols are clearly spelled out, and motivated well.
  • The claim that aggregate retrieval success "hides" target-level change is clearly demonstrated through the experimental validation, supporting the claim the authors make on the need for more fine-grained metrics and analyses. The effects of retrieval depths (more pronounced purity and diversity differences) and embedding/retrievers models are interesting, because they can be helpful in defining the parameters of retrieval systems (section 5.2). This is even more evident in the decision instability measurements. While aggregate accuracy indicates no substantial decline in performance due to consolidation, individual decisions reveals a positive repeat-controlled contrast: downstream decisions differs substantially more often than expected from repeat variability.
Critical weaknesses
  • If it wasn't for section 6, which is still in my opinion too shallow, the reader is largely left with drawing her own conclusion from the result analysis. Most of the comments to section 5 results are observations and statistics about the plots and tables, with only aggregate metrics being fully developed, and not detailed ones. What are the take home messages of your study? How much such conclusions would generalize across datasets? Section 6 is a rather "timid" attempt to answers these two questions, and it is a pity.
Questions for authors
  1. In section 3.2: can you elucidate the reasoning behind the definitions of the repeat-controlled constrast did_i and its dataset-wide mean? DicrossD_i^{\textrm{cross}} is the average across repeats of the number of times decisions taken using the full or consolidated memory agree: in the worst case, this could be zero. DisameD_i^{\textrm{same}} measures the average "agreement" between decisions taken using the full memory or the consolidated memory, over the two repeats. Worst case, decisions are coherent when using the full and consolidated memories. So did_i can be negative. In line 159, you comment that a positive Δ\Delta, computed across all targets ii, indicates decision instability, but I think there is more to it, right? What I am trying to articulate is: it is possible that decisions are consistents within the full or the consolidated memory settings, but different between the full and the consolidated memories. Can you elaborate?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2

We sincerely thank the reviewer for the encouraging assessment, and for highlighting the clarity of the paper and the value of looking beyond aggregate retrieval metrics. The reviewer raised two important points: the Discussion needed stronger takeaways, and the reasoning behind our repeat-controlled measure could be clearer. We have revised the paper with both in mind. All changes are highlighted in blue.

Summary of revisions: We expanded the experiments to include a query-only baseline, two crossed conditions separating retrieval identity from displayed representation, and five independent repeats across both consumers (19,700 predictions). We also added a second labeling task with a different consolidation procedure, category-adjusted overlap tests, and additional implementation details and results (Appendices A-D).

Response to Critical Weakness:

We agree that the original Discussion did not do enough to explain what readers should take away. We substantially revised Section 6 and strengthened the interpretation throughout Section 5. The revised Discussion develops four main points: (1) First, retrieval success and decision preservation are not the same. Hit@3 remained unchanged at 87.06% and accuracy changed very little, yet consolidation produced excess decision disagreement of +5.24 pp under Sol (pooled: +5.40 pp). Changes occurred in both directions of correctness, so we treat decision preservation as a diagnostic, not a measure of correctness. (2) Second, which experiences are retrieved matters more than how they are displayed in our tested comparisons. Decision changes were larger when retrieved identities differed (+5.24 and +4.37 pp) than when only the displayed representation changed (+2.33 and +0.40 pp). This explains why Hit@k alone can miss changes in the evidence available to a consumer. (3) Third, instability is not spread evenly. Some targets repeatedly changed decisions across consumers and retrievers, with overlap 3.88× and 5.48× above category-adjusted expectation. Many lie near the caught-between/struck-by boundary, although the effect is not limited to those categories. (4) Finally, we added five practical implications: measure decision disagreement alongside Hit@k and accuracy; examine retrieved-neighbor identity and label composition; include query-only baselines; investigate full-narrative retrieval keys with compact displayed memories; and use recurring vulnerable instances to guide selective consolidation.

On generalization, we added a second task: fall-protection labeling on 167 nonoverlapping OSHA incidents, using LLM-generated factual summaries instead of structured attributes. The same pattern appeared (pooled Δ = +14.17 pp, 95% CI [10.11, 18.44]; Section 5.5). We do not interpret this as generalization across arbitrary datasets or agent environments: the dataset, task, and method changed together, and both datasets come from OSHA records. Section 6 states these boundaries explicitly.

Response to Question for Authors:

We appreciate this question, as it gets at exactly what the measure is intended to capture. The reviewer is right that decisions can be perfectly consistent within Full memory and within Consolidated memory, yet consistently different between the two. Our measure is designed to identify precisely this situation. Both Dᵢᶜʳᵒˢˢ and Dᵢˢᵃᵐᵉ measure disagreement, not agreement. Dᵢᶜʳᵒˢˢ is the fraction of Full-versus-Consolidated repeat pairs whose decisions differ, and Dᵢˢᵃᵐᵉ is the fraction of repeat pairs within the same memory condition whose decisions differ. The target-level contrast dᵢ = Dᵢᶜʳᵒˢˢ − Dᵢˢᵃᵐᵉ subtracts ordinary response variability.

For example, suppose a target is labeled struck-by in all five Full-memory repeats but caught-between in all five Consolidated-memory repeats. Then Dᵢˢᵃᵐᵉ = 0, every cross-condition pair disagrees so Dᵢᶜʳᵒˢˢ = 1, and dᵢ = 1: the maximum possible excess disagreement, even though the model is perfectly consistent within each condition. Conversely, when both conditions consistently produce the same label, dᵢ = 0. With finitely many repeats, dᵢ can be negative when within-condition disagreement exceeds cross-condition disagreement. The dataset-level Δ = (1/N) Σᵢ dᵢ therefore indicates, when positive, that decisions differ between memory conditions more often than they vary within them.

Section 3.2 now includes this worked example, and Section 6 returns to its significance. The measure captures decision changes, not necessarily errors: consolidation may turn an incorrect decision into a correct one, or vice versa.

We thank the reviewer again! These suggestions helped make the paper's main message and the contribution of the measure clearer.

Reviewer_5bCSOpenReview ↗

Thank you for your rebuttal

Dear Authors,

thank you for your rebuttal, it helped clarify my doubts.

I have checked your revised paper and will keep my positive rating. Looking forward to the discussion phase.

Dear Reviewer,

Thank you for reviewing our revised paper and for your positive assessment. We are glad the rebuttal clarified your questions and we appreciate your constructive suggestions, which improved the paper.

Best regards,

Reviewer_SkHVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether memory consolidation changes downstream decisions despite stable retrieval and classification performance. Using 394 OSHA incident records, the authors report increased decision disagreement after replacing original narratives with structured risk attributes, with some vulnerable instances persisting across model and retriever substitutions.

Critical strengths
  1. The paper highlights an interesting gap between retrieval performance and decision consistency.
  2. The paired evaluation and repeat-controlled analysis provide useful instance-level evidence.
Critical weaknesses

1.Limited experimental scope. The evaluation uses only 394 records from a single classification task. It does not involve long-horizon agent interactions, incremental memory updates, or memory-dependent actions. The broader claims would benefit from evaluation on representative agent memory benchmarks. 2.Restricted consolidation setting. Memory consolidation is reduced to extracting predefined risk attributes from individual records. This does not reflect the broader range of consolidation operations used in long-term agent memory. Only one compression strategy is tested. 3.Missing key controls. The consumer already receives the complete target narrative, yet no query-only baseline is provided. Moreover, the intervention changes both retrieved example identities and their contents, making it difficult to distinguish retrieval effects from representation effects. 4.Insufficient experimental support. Only two repeats are conducted per condition, and no supplementary experiments or detailed implementation materials are provided. The vulnerability-overlap analysis also lacks a class-stratified control despite substantial variation across outcome categories.

Questions for authors
  1. How does performance change without retrieved memory?
  2. Can the authors isolate retrieval changes from changes in memory content?
  3. Do the findings generalize to other consolidation methods and long-horizon agent memory benchmarks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1

We thank the reviewer for recognizing the distinction between retrieval performance and decision consistency and the value of our paired, repeat-controlled design. We have added the requested controls, expanded the experiments, strengthened statistical testing, and narrowed our claims. Changes are highlighted in blue in the revised paper.

Summary of revisions: We added (i) a query-only baseline (Q); (ii) crossed FC/CF conditions separating retrieval identity and displayed representation; (iii) five repeats across five conditions and two consumers (19,700 predictions); (iv) a second task using 167 nonoverlapping OSHA incidents with LLM-generated factual summaries; and (v) category-adjusted exact overlap tests and Appendices A-D.

Responses to Critical Weaknesses:

  1. W1: We now explicitly limit our claims to short-horizon, retrieval-backed decisions following one consolidation step (Section 6). We also added a second task: fall-protection annotation (Yes/No) on 167 distinct incidents. The pattern recurred (pooled Δ = +14.17 pp, 95% CI [10.11, 18.44]), while the Full–Consolidated accuracy difference was uncertain (+2.10 pp [−2.93, 7.25]; Sections 4.6, 5.5). The two tasks cover 561 distinct targets. Long-horizon agents with evolving queries and memory updates remain untested; our paired intervention instead holds target queries, candidate pools, and labels fixed to isolate consolidation-related decision changes. The query-only baseline confirms that retrieved memory influences decisions (FF–Q Δ = +4.52 pp [2.89, 6.32]).

  2. W2: The new task consolidates narratives into compact factual summaries generated without reference labels and manually reviewed for fidelity (Appendix D), rather than extracting predefined attributes. Instability recurred for Sol (+12.28 pp) and Astra (+16.07 pp), including under Astra, which was not used to generate the summaries. In the primary experiment, contrasts changing retrieved IDs were larger than those changing displayed content alone. We explicitly acknowledge that the dataset, task, and method change jointly, so this shows recurrence rather than method-specific causation. Cross-record merging and incremental updates remain future work.

  3. W3: We added both controls for each consumer with five repeats. Without memory, accuracy was 86.95% (Sol) and 86.60% (Astra), versus 88.02% and 88.68% with Full memory; pooled macro-F1 was 0.8225 vs. 0.8603 (Table B.1). Adding Full memory changed individual decisions beyond repeat variability (Δ = +4.52 pp [2.89, 6.32]). Crossed conditions isolate the two factors (Section 5.3; Table B.2): retrieved-ID changes, FF–CF +5.24 pp [3.44, 7.17] and FC–CC +4.37 [2.75, 6.15]; display-only changes, FF–FC +2.33 [1.21, 3.62] and CF–CC +0.40 [0.08, 0.80]. Retrieval identity contributed more strongly in these comparisons.

  4. W4: The five-repeat evaluation uses 25 cross-condition pairs and 10 within-condition pairs per condition, per target. The primary Δ remains +5.24 pp [3.29, 7.40] versus +5.20 pp in the original two-repeat analysis (Sections 3.2, 5.3). Appendix A details prompts, settings, conditions, and procedures; B provides condition-level results; C provides localization and sensitivity; D covers the extension. We replaced the pooled overlap null with a category-adjusted exact test (Sections 3.3, 5.4): across consumers, 13 shared vulnerable targets vs. 3.35 expected (3.88×, p = 9.35 × 10⁻⁹); within caught-between, 9 vs. 2.80 (p = 2.25 × 10⁻⁵); across retrievers, 11 vs. 2.01 (5.48×, p = 1.01 × 10⁻⁸). Threshold sensitivity is reported in Table C.1.

Responses to Questions for Authors:

  1. Query-only accuracy was 86.95% for Sol and 86.60% for Astra, versus 88.02% and 88.68% under Full memory. Pooled macro-F1 was 0.8225 without memory vs. 0.8603 with Full memory. At the decision level, adding Full memory produced Δ = +4.52 pp [2.89, 6.32] (Tables B.1-B.2). Please also see our response to W3.

  2. Yes. The crossed conditions FC and CF allow us to compare changes in retrieval identity while holding displayed representation fixed, and vice versa. Retrieval-identity contrasts (+5.24 and +4.37 pp) exceed the corresponding representation-only contrasts (+2.33 and +0.40 pp) in the tested comparisons (Section 5.3; Table B.2). Please also see our response to W3.

  3. The instability pattern recurred with LLM-generated factual summaries in the second task (pooled Δ = +14.17 pp, 95% CI [10.11, 18.44]). However, the task, dataset, and consolidation method changed jointly, so this is not a controlled demonstration of method-independent generalization. Long-horizon agent benchmarks remain untested. We have made this limitation explicit in Section 6 and identify extending the paired protocol to sequential agents as future work.

We thank the reviewer again for the detailed feedback, which substantially strengthened the paper.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for your thoughtful review. We understand this is a busy period and we would welcome any further questions or comments on our response before the discussion period closes.

Best regards,

Thank you for the response and additional experiments. I have raised my rating.

Dear Reviewer,

Thank you very much for taking the time to review our response and the revised paper and for raising your rating. We are truly grateful for your thoughtful and detailed feedback.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.