acceptodds
Under review as a conference paper at ICLR 2027

Top-k Retrieval Leaves Memory Value Undetermined

Abstract

A memory-augmented agent must decide which memories to keep, merge and retrieve, and its only evidence of their value is its retrieval log: the memories it retrieved in each episode and whether it succeeded. Existing methods assign every memory a value from this log, even when the log cannot separate one memory's contribution from another's. We treat the log as an experiment that nobody designed and ask which memory values it determines. Two counts answer this before any outcome is observed: how often each memory was retrieved, and how often each pair was retrieved together. From these counts, we prove that top- retrieval leaves every memory value undetermined, however many episodes are logged, and that a question with its own memories, asked once, determines at most one of their values. On fourteen standard memory benchmarks, no memory value is determined in any of configurations. Both causes are choices about retrieval. We introduce LEDGER, which estimates each determined Shapley value in closed form with a confidence interval and reports every other value as undetermined. It merges memories with identical histories without losing a determined value, and it retrieves so that the next log determines more. On pooled HotpotQA, LEDGER raises the fraction of memories determined from to , at a cost of accuracy points on HotpotQA and with a gain of on LoCoMo. Code: https://anonymous.4open.science/r/ledger-review-AF71/.

Primary area:
causal reasoning
Keywords:
agent memory, memory valuation, retrieval-augmented generation, identifiability, retrieval logs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YMTbOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when an agent’s retrieval logs contain enough information to determine the value of individual memories. Under a model with bounded-order interactions and an unknown baseline, it proves that retrieving a fixed number of memories leaves every individual Shapley value undetermined, regardless of the number of logged episodes. The proposed framework, LEDGER, checks which values are identifiable, estimates those values, groups memories with identical retrieval histories, and changes retrieval to collect more informative logs.

Critical strengths
  1. The paper clearly separates insufficient data from retrieval designs that cannot distinguish individual contributions. Its fixed-size retrieval result explains why collecting more episodes may not resolve the problem. The analysis of private memory stores provides another useful explanation for why common evaluation protocols offer limited evidence about individual memory values.

  2. LEDGER provides a useful check before assigning values to memories. In the additive setting, retrieval frequencies and pairwise co-retrieval counts are enough to determine which values the log can identify. Reporting other values as undetermined avoids presenting arbitrary estimates as evidence. The guarantee for grouping memories with identical retrieval histories also gives this diagnostic a concrete use.

Critical weaknesses
  1. The positive results on real data do not establish recovery of the intended Shapley values. They use an additive model, which does not capture interactions between memories. Moreover, retrieval depends on the query, so differences between queries can affect both the retrieved memories and the outcome. As the paper acknowledges (Appendix A.3), the resulting estimates may describe associations in the log rather than query-averaged memory contributions. This limits the interpretation of the positive results.

  2. The benefit for downstream memory management remains unclear. A larger identifiable fraction does not necessarily lead to better retention decisions. In the reported synthetic retention experiment, the tested LEDGER policy performs worse than reward averaging, and the planned real-data retention experiment (Appendix F.7) was not completed. Changing retrieval can also reduce answer accuracy.

Questions for authors
  1. Can you evaluate value recovery in a controlled setting where retrieval depends on the query and memories have known pairwise interactions? Comparing the current additive estimates with query-adjusted estimates and a second-order model would help show when improved identifiability translates into more accurate memory values.

  2. Can you test the implemented intervals with an estimated noise level on new settings selected before observing the results? A comparison with intervals that account for ridge bias, including cases with non-Gaussian noise or strongly correlated retrieval histories, would help clarify when the reported uncertainty is reliable.

  3. Can you complete a real-data retention experiment under a fixed memory budget, comparing LEDGER with reward averaging, retrieval frequency, and similarity-based selection? Measuring final answer accuracy together with the cost of collecting the more informative logs would show whether improved identifiability leads to a practical benefit.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YMTb

Thank you for the review and for three precise requests. We ran each one as a pre-registered experiment.

Additive model and query-dependent retrieval. The negative results hold for interactions of every order: a value that is undetermined at resolution d stays undetermined at every higher resolution (the resolution lemma, Appendix B), and the audit relies on this. For the estimates, the revision now states what they target (Section 4, "Estimation"). When the value function has higher-order or query-specific effects, the estimate is the best fit at the chosen resolution, and the interval covers that fit.

Q1. Controlled study with query-dependent retrieval and pairwise interactions (Appendix G.7). We used synthetic logs with 40 memories, eight query types, query-specific effects and interactions on 10% of the pairs, over 50 seeds. None of the three registered hypotheses holds:

  • The additive estimate tracks the Shapley values (Spearman 0.62, mean absolute error 0.09), but its intervals contain all of them at once in 0 of 50 logs.
  • Adding an intercept per query type raises the error by 0.008 ([+0.006, +0.011]).
  • The second-order model determines only 0.08 of the memories at T = 8,000, and its error is higher by 0.36 on the memories both models determine.

An exploratory follow-up shows what the interval does cover: the best additive fit to the logged episodes, in at least 95% of logs under every condition, and the Shapley value only when the model holds. Identifiability therefore yields accurate values at the right resolution, and a second-order model needs much longer logs. On real data, adding an intercept per question removes one bias, where a memory's value absorbs the difficulty of the questions that retrieve it, and leaves the ranking unchanged (Appendix G.5).

Q2. Intervals with estimated noise, on settings fixed in advance (Appendix G.8). We simulated outcomes on real retrieval designs (HotpotQA and LoCoMo) and on designs with strongly correlated histories, estimating the noise level each time. Joint coverage of the paper's interval:

DesignBinaryGaussianStudent-t (3 d.f.)
HotpotQA0.9890.9730.678
LoCoMo0.9460.9140.427
Correlated0.9590.9680.883

With binary outcomes, which is the setting of the paper, the interval reaches the coverage we registered on all three designs. With Gaussian noise on the LoCoMo designs it falls short, and adding the ridge-bias term brings it up to 0.962. Heavy-tailed Student-t noise is where the interval fails, but such noise cannot occur when outcomes lie between 0 and 1, and the revision now states this limit. The self-normalized set of Abbasi-Yadkori et al. covers every case, at the price of being 6 to 16 times as wide. Adding the bias term to our interval makes it only 0.07% wider on average, so it costs almost nothing.

Q3. Real-data retention under a fixed budget, with the cost of collecting the log (Section 5.5; Appendix G.5). We used all 10 LoCoMo conversations. Values come from twelve passes over half of each conversation's questions; each method keeps 25% or 50% of the turns; the remaining questions test the result. At the 50% budget:

MethodAccuracy
Full store0.168
LEDGER, determined values first0.131
Reward averaging0.128
Retrieval frequency0.127
Random0.121
Similarity0.119

Neither registered hypothesis holds. The LEDGER rule does about as well as retrieval frequency (+0.4 points, [−0.7, +1.4]) and reward averaging (+0.3 points, [−1.8, +2.3]), and neither difference is distinguishable from 0. Collecting the more informative log did not hurt on this corpus; it helped. While the log was being collected, 19.1% of answers were correct under LEDGER's retrieval and 15.5% under top-k. On HotpotQA, collecting it costs 2.4 points, or 0.6 when LEDGER's retrieval is applied to only 1 query in 10.

Downstream benefit. In these tests, determining more memory values did not lead to better retention, and the revised abstract, introduction and conclusion now say this plainly. Section 5.5 explains the reason. LEDGER's retrieval puts a query's most similar memories into every one of its episodes, so the log does not see the query without them, and their values remain imprecise. The paper now states the open problem directly: a retrieval rule that also varies these memories while keeping the cost low.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer YMTb,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,

Authors

Reviewer_Lbd6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

N/A

Critical strengths

N/A

Critical weaknesses

The main text exceeds 9 pages in length and violates the submission policy, therefore the manuscript is desk-rejected.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer Lbd6

Thank you for the review. The submitted main text ran four lines onto page 10 and exceeded the limit. This was our error. With the repeated paragraphs of Section 5.2 removed, the original content fits within 9 pages. The revision adds the results the other reviewers requested and ends on page 10, the limit for revisions. We would be grateful if you could assess the content of the revised paper; the general response summarizes the changes.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer Lbd6,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,

Authors

Reviewer_4DF9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether individual memory Shapley values can be identified from retrieval logs. It shows that, under its linear outcome model, fixed-size retrieval leaves every memory value undetermined however many episodes are logged, and that per-question memory stores limit how many values each question can determine. The proposed method, LEDGER, checks identifiability from retrieval counts, estimates identifiable values with confidence intervals, merges memories with identical retrieval histories, and changes retrieval to improve identification. Its retrieval rule varies the number of retrieved memories and reserves part of each selection for less frequently retrieved memories among relevant candidates. An audit of fourteen benchmarks finds no determined value in any of 55 configurations. On pooled HotpotQA, LEDGER increases the identifiable fraction from 0.1% to 79.7% at a cost of 2.6 accuracy points, while the accuracy effect is +0.8 on LongMemEval and +2.7 on LoCoMo.

Critical strengths
  1. A clear diagnostic for unsupported attribution. The paper gives a simple test, computable from retrieval counts alone, for whether a log determines each memory's value, and shows that logs produced under standard retrieval conventions leave absolute per-memory values undetermined.
Critical weaknesses
  1. Insufficient evidence that the method provides practical value. The paper builds on linear estimability (as noted in Section 2) to show that absolute memory values are often undetermined, but it does not establish when these ambiguities change memory-management decisions. Section 2 motivates the work by retention and merging decisions, yet the main text contains no experiment showing that determined values improve such decisions, and the audit in Table 2 does not quantify how often undetermined absolute values would change one. The proposed retrieval rule has mixed accuracy effects (−2.6 on HotpotQA, +0.8 on LongMemEval with an interval that includes zero, +2.7 on LoCoMo); the variant that removes the HotpotQA cost relies on an oracle with gold labels, and the conclusion leaves a query-time version as an open problem. Thus, neither the decision relevance of the diagnostic nor the practical benefit of its remedy is sufficiently established.
  2. Presentation. The main text does not clearly separate whether a log determines a memory's value from whether that value is useful for memory management, which makes it harder to judge what the experiments establish. Section 5 is also repetitive: in Section 5.2, the description of the audit and the two causes of non-identification each appear twice, and the setup of the pooled HotpotQA experiment (300 questions over 24 passes, 1,600 held-out questions) is stated at the end of Section 5.2 and again in Section 5.3.
Questions for authors
  1. (Requested analysis) Figure 1 shows a log that determines pairwise differences but not absolute values. How often does this situation arise in the logs of Table 2, and in te cases, how often would the undetermined absolute values change a retention or ranking decision compared with using the determined differences alone?
  2. (Requested analysis) Section 2 states that determined values guide which memories to keep under a budget. Can you show, on real data, that retention guided by LEDGER's determined values outperforms simple baselines such as reward averaging or retrieval frequency?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4DF9

Thank you for the review and for the two questions, which shaped the revision.

Q1. How often does the situation of Figure 1 arise, and would the undetermined absolute values change a decision?

We answer this in the revised version with rank classes (Section 3, "Which decisions a log determines"): a log orders two memories exactly when every invisible change moves their values by the same amount. Table 2 now reports the fraction of pairs whose order the log determines for every benchmark.

  • None of the 55 configurations matches Figure 1, where a shared shift is the only thing the log leaves open.
  • None of the 29 configurations with a store per question determines the order of any pair, so their rankings are as undetermined as their absolute values.
  • Stores shared within a document or a conversation determine the order of some pairs of retrieved memories: up to 0.32 on LoCoMo, 0.05 on MultiHopRAG, and 0.86 on the 115-memory NarrativeQA summaries.

When a log determines the order of two memories, a ranking decision needs only that order, and the revision now says so. Other decisions need the absolute value: whether a memory helps at all, and whether it is worth keeping. To see how much this matters, we took the HotpotQA top-k log and computed three valuations that fit its outcomes equally well. They disagree on whether 20% to 32% of the retrieved memories help or hurt, and if each kept its top half of the store, the kept sets would overlap with a Jaccard index of only 0.44 to 0.72. On LEDGER's log, the three valuations agree on every determined value (Section 5.3; Appendix G.2).

Q2. Does retention guided by determined values beat reward averaging or retrieval frequency on real data?

We ran this as a pre-registered experiment on LoCoMo (Section 5.5; Appendix G.5). Each method learns from 12 passes over half of each conversation's questions and keeps half of the turns; the other questions test the result. The 9 methods reach accuracy 0.11 to 0.13, and the full store 0.168. The rule built on determined values differs from retrieval frequency by +0.4 points ([−0.7, +1.4]), so the hypothesis is unsupported. A second pre-registered test agrees: the values rank gold memories with AUC 0.63, and retrieval frequency with 0.73 (Appendix G.4).

The cause lies in the retrieval rule (Appendix G.6). LEDGER varies the less similar part of each retrieval, so a question's two most similar memories appear in all of its episodes, and their values are estimated from little variation. A longer log under the same rule narrows the intervals slowly. The revision removes the claim that determined values guide retention and states the open problem: a rule that also varies the most similar memories at a low cost.

Practical value. The revised paper offers two things a practitioner can use. First, a check that tells them, before any outcome is used, which values and rankings their log can support; the audit shows that current benchmark logs support few or none. Second, a retrieval change whose cost is measured and adjustable, with no gold labels needed: applied to one query in 10, it costs 0.6 points ([−1.1, −0.1]) and determines 0.42 of the values after 48 passes (Section 5.3). On LoCoMo, the accuracy gain holds on a separate set of logs (+3.6 points, [+2.1, +5.2]).

Presentation. Section 3 now states which decisions a log determines, and Section 5.5 tests which determined values are useful. The repeated paragraphs of Section 5.2 are removed, and Section 5.3 states the HotpotQA setup once.

AuthorsOpenReview ↗

Response by Authors

Dear Reviewer 4DF9,

As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the reviewers. The revised PDF marks every change in blue. The anonymous repository contains the code, outputs and pre-registration of every new experiment. Each pre-registered experiment fixed its design, hypotheses and analysis in a committed file before any outcome was computed; analyses specified after earlier results are marked exploratory.

Page limit. The submitted main text ran four lines onto page 10. This was our error. Removing the repeated paragraphs of Section 5.2 brings the original content within nine pages. The revision adds the results the reviewers requested and ends on page 10, and Figure 2 moved to Appendix C.

New results

  1. Decisions each log determines (Section 3; Table 2, column "Pairs"; Appendix G.2). A log orders two memories exactly when they share a rank class, which the log's counts determine. None of the 55 audited configurations determines a memory value, and none of the 29 with a store per question determines the order of any pair of memories.
  2. Adjustable cost (Section 5.3; Figure 3a; Appendix G.3). LEDGER's retrieval costs 2.4 accuracy points on every query and 0.6 on one query in ten.
  3. What determined values support (Section 5.5; Appendices G.4 to G.6). In two pre-registered tests on real data, retrieval frequency ranks gold memories better than the determined values (AUC 0.73 against 0.63) and retains memories equally well. The cause: LEDGER's retrieval keeps a query's two most similar memories in every episode, so their values are imprecise. The abstract, introduction and conclusion now state this limit.
  4. Model checks (Section 5.4; Appendices G.7 and G.8). A controlled study with interactions and query-dependent retrieval, and an interval study with estimated noise on real retrieval designs.

Scoring correction. While rerunning the experiments, we found that our scorer accepted some answers that decline, such as "unknown". We reran every reader call, rescored all answers with a stricter scorer registered in advance, and selected the judge model by its agreement with 100 hand labels. Every comparison keeps its sign. The HotpotQA cost moves from 2.6 to 2.4 points and the LoCoMo gain from 2.7 to 3.1 (Appendix G.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.