- Rating
- 2
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when an agent’s retrieval logs contain enough information to determine the value of individual memories. Under a model with bounded-order interactions and an unknown baseline, it proves that retrieving a fixed number of memories leaves every individual Shapley value undetermined, regardless of the number of logged episodes. The proposed framework, LEDGER, checks which values are identifiable, estimates those values, groups memories with identical retrieval histories, and changes retrieval to collect more informative logs.
Critical strengths
-
The paper clearly separates insufficient data from retrieval designs that cannot distinguish individual contributions. Its fixed-size retrieval result explains why collecting more episodes may not resolve the problem. The analysis of private memory stores provides another useful explanation for why common evaluation protocols offer limited evidence about individual memory values.
-
LEDGER provides a useful check before assigning values to memories. In the additive setting, retrieval frequencies and pairwise co-retrieval counts are enough to determine which values the log can identify. Reporting other values as undetermined avoids presenting arbitrary estimates as evidence. The guarantee for grouping memories with identical retrieval histories also gives this diagnostic a concrete use.
Critical weaknesses
-
The positive results on real data do not establish recovery of the intended Shapley values. They use an additive model, which does not capture interactions between memories. Moreover, retrieval depends on the query, so differences between queries can affect both the retrieved memories and the outcome. As the paper acknowledges (Appendix A.3), the resulting estimates may describe associations in the log rather than query-averaged memory contributions. This limits the interpretation of the positive results.
-
The benefit for downstream memory management remains unclear. A larger identifiable fraction does not necessarily lead to better retention decisions. In the reported synthetic retention experiment, the tested LEDGER policy performs worse than reward averaging, and the planned real-data retention experiment (Appendix F.7) was not completed. Changing retrieval can also reduce answer accuracy.
Questions for authors
-
Can you evaluate value recovery in a controlled setting where retrieval depends on the query and memories have known pairwise interactions? Comparing the current additive estimates with query-adjusted estimates and a second-order model would help show when improved identifiability translates into more accurate memory values.
-
Can you test the implemented intervals with an estimated noise level on new settings selected before observing the results? A comparison with intervals that account for ridge bias, including cases with non-Gaussian noise or strongly correlated retrieval histories, would help clarify when the reported uncertainty is reliable.
-
Can you complete a real-data retention experiment under a fixed memory budget, comparing LEDGER with reward averaging, retrieval frequency, and similarity-based selection? Measuring final answer accuracy together with the cost of collecting the more informative logs would show whether improved identifiability leads to a practical benefit.
2 comments
Response by Authors
Dear Reviewer YMTb,
As the interactive discussion period is drawing to a close, we would like to kindly ask whether our response has adequately addressed your concerns. Please let us know if any further clarification or additional information would be helpful.
Best regards,
Authors
Response to Reviewer YMTb
Thank you for the review and for three precise requests. We ran each one as a pre-registered experiment.
Additive model and query-dependent retrieval. The negative results hold for interactions of every order: a value that is undetermined at resolution d stays undetermined at every higher resolution (the resolution lemma, Appendix B), and the audit relies on this. For the estimates, the revision now states what they target (Section 4, "Estimation"). When the value function has higher-order or query-specific effects, the estimate is the best fit at the chosen resolution, and the interval covers that fit.
Q1. Controlled study with query-dependent retrieval and pairwise interactions (Appendix G.7). We used synthetic logs with 40 memories, eight query types, query-specific effects and interactions on 10% of the pairs, over 50 seeds. None of the three registered hypotheses holds:
An exploratory follow-up shows what the interval does cover: the best additive fit to the logged episodes, in at least 95% of logs under every condition, and the Shapley value only when the model holds. Identifiability therefore yields accurate values at the right resolution, and a second-order model needs much longer logs. On real data, adding an intercept per question removes one bias, where a memory's value absorbs the difficulty of the questions that retrieve it, and leaves the ranking unchanged (Appendix G.5).
Q2. Intervals with estimated noise, on settings fixed in advance (Appendix G.8). We simulated outcomes on real retrieval designs (HotpotQA and LoCoMo) and on designs with strongly correlated histories, estimating the noise level each time. Joint coverage of the paper's interval:
With binary outcomes, which is the setting of the paper, the interval reaches the coverage we registered on all three designs. With Gaussian noise on the LoCoMo designs it falls short, and adding the ridge-bias term brings it up to 0.962. Heavy-tailed Student-t noise is where the interval fails, but such noise cannot occur when outcomes lie between 0 and 1, and the revision now states this limit. The self-normalized set of Abbasi-Yadkori et al. covers every case, at the price of being 6 to 16 times as wide. Adding the bias term to our interval makes it only 0.07% wider on average, so it costs almost nothing.
Q3. Real-data retention under a fixed budget, with the cost of collecting the log (Section 5.5; Appendix G.5). We used all 10 LoCoMo conversations. Values come from twelve passes over half of each conversation's questions; each method keeps 25% or 50% of the turns; the remaining questions test the result. At the 50% budget:
Neither registered hypothesis holds. The LEDGER rule does about as well as retrieval frequency (+0.4 points, [−0.7, +1.4]) and reward averaging (+0.3 points, [−1.8, +2.3]), and neither difference is distinguishable from 0. Collecting the more informative log did not hurt on this corpus; it helped. While the log was being collected, 19.1% of answers were correct under LEDGER's retrieval and 15.5% under top-k. On HotpotQA, collecting it costs 2.4 points, or 0.6 when LEDGER's retrieval is applied to only 1 query in 10.
Downstream benefit. In these tests, determining more memory values did not lead to better retention, and the revised abstract, introduction and conclusion now say this plainly. Section 5.5 explains the reason. LEDGER's retrieval puts a query's most similar memories into every one of its episodes, so the log does not see the query without them, and their values remain imprecise. The paper now states the open problem directly: a retrieval rule that also varies these memories while keeping the cost low.