acceptodds
Under review as a conference paper at ICLR 2027

How Much Is a Memory Worth? Efficient Credit Assignment for Memory Admission Under Unknown Demand

Abstract

Language-model agents use persistent memory to carry information across interactions, but limited storage requires deciding which memories to retain before future questions arrive. Existing admission policies rank memories using combinations of factors such as recency and confidence, or novelty relative to what is already stored. A memory's contribution to answering, however, depends on both the questions and the other available memories. Learning an admission policy from this contribution requires assigning credit when memories overlap or depend on one another. Here, we measure the utility of a set of memories by its reduction of a frozen reader's codelength for reference answers, then use the Shapley value to assign credit to individual memories. A policy, which we call Shapley Credit Assignment Policy (**SCAP**), predicts the resulting credit at write time from candidate and resident records using one regression objective over frozen embeddings. On held-out LongMemEval questions and supporting sessions, SCAP retains greater utility than the selections made using A-MAC and SAGE across capacities, with larger advantages at higher capacity pressure. The advantage also persists with a reader from another model family without retraining the policy. These findings support a simple approach to memory admission in which the learning target follows from the utility of the stored information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agent memory, memory admission, bounded-capacity retention, information-efficient memory representation, memory valuation, learned admission policy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8UZwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses memory admission for language-model agents operating under limited storage capacity and uncertain future demand. The proposed SCAP framework estimates memory utility using reference-answer codelength reduction, assigns contributions via Shapley values, and learns admission decisions through an attention-based predictor. Experiments on LongMemEval memory stores show improved retained utility over A-MAC and SAGE across storage capacities, with evidence of cross-reader generalization.

Critical strengths
  1. The paper presents a principled framework that combines reference-answer codelength reduction with Shapley attribution to quantify memory contributions while accounting for redundancy and complementarity. Unlike heuristic retention strategies based on recency, novelty, or confidence, SCAP learns retention from downstream information utility, providing a theoretically grounded and methodologically significant approach to persistent LLM memory management.

  2. SCAP consistently outperforms A-MAC and SAGE across all tested memory capacities, with especially large gains under tight memory constraints. Ablation studies validate the importance of Shapley-based supervision, record-level representations, and store-context awareness, while strong performance with Llama-3.1-8B without retraining provides preliminary evidence of transferability across reader architectures.

Critical weaknesses
  1. The main limitation is the gap between the sequential memory-admission problem and the static subset-selection evaluation. Experiments on 88 small, curated memory stores do not capture key real-world challenges such as sequential admissions, irreversible evictions, evolving user preferences, changing demand distributions, and larger memory scales. Thus, the results demonstrate effectiveness in a simplified offline setting rather than practical long-term memory management.

  2. A key concern is that Shapley-value ranking does not generally optimize utility under a fixed memory budget, and the manuscript provides neither theoretical guarantees nor conditions linking these objectives. Moreover, the gaps between the subset oracle, exact Shapley selection, and SCAP's learned predictions (e.g., at K=2) indicate substantial attribution and prediction errors.

  3. The manuscript reports only mean performance, without confidence intervals, significance tests, or multi-seed variability analyses, and store dependencies may reduce the number of independent evaluation units. Moreover, higher retained codelength utility does not consistently translate into better QA accuracy (e.g., at K=3).

Questions for authors
  1. Since Shapley values capture average marginal contributions rather than optimal fixed-capacity selections, what justifies using top-KKK Shapley rankings for memory retention? Can the authors provide theoretical justification or comparisons with capacity-aware and direct subset-optimization methods? Also, how much of the gap to the oracle is attributable to Shapley ranking itself versus errors in the learned predictor?

  2. How does SCAP perform in a realistic sequential setting where memories arrive over time and discarded information cannot be revisited? Evaluating chronological interaction histories with larger memory stores, repeated admission and eviction decisions, and evolving query distributions would better demonstrate its effectiveness for long-term memory management.

  3. Can the authors provide confidence intervals, significance tests, and variability analyses for the reported improvements, particularly in QA accuracy? How do the results compare against complete baseline workflows or matched memory representations? Also, what are the computational costs of Shapley-label generation and policy training relative to the deployment-time savings achieved?

3 comments

We thank the reviewer for the constructive feedback. Our main contribution is Shapley Credit Assignment Policy (SCAP), a learned admission policy that scores candidate and resident memories before future questions arrive, where information efficiency refers to answer information retained within a fixed memory count or reader token budget (Title, lines 0–4, Abstract, lines 11–30, and Section 1, lines 50–83). References below cite the revised manuscript. We hope the clarifications below help address the reviewer's concerns.

1. Chronological admission, irreversible eviction, and changing demand. We replayed all held-out LongMemEval histories (16401640 arrivals, 4242 to 5555 sessions each) in timestamp order without retraining the policy, keeping the KK memories with the highest scores at each arrival without question access while rejected or evicted memories never return up to K=10K=10 (K=8,10K=8,10 not seen in training) (lines 353–358 and Appendix D, Table 11, lines 863–883).

PolicyK=10K=10K=8K=8K=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
SCAP17.8013.600.810.45−0.29-0.29−4.73-4.73−4.06-4.06
A-MAC11.784.81−0.50-0.50−0.69-0.69−5.14-5.14−11.68-11.68−13.03-13.03
SAGE12.588.35−3.30-3.30−2.62-2.62−5.60-5.60−8.41-8.41−9.91-9.91

In our setup, future questions are unobserved at admission and policy parameters stay fixed at deployment (Section 1, lines 52–55). Replaying all LoCoMo conversations (272272 sessions, 200200 questions) chronologically without retraining, SCAP leads every baseline from K=5K=5 to K=1K=1 and remains positive for early, middle, and late supporting sessions at K=5K=5 (lines 359–362 and Appendix D, Table 11, lines 868–875, 890–896). Online adaptation to shifting query streams remains future work (Section 4, lines 482–484).

2. Subset selection under a fixed capacity. Ranking memories by Shapley credit is optimal at capacity KK under additive utility, whereas interactions remove any general subset guarantee (Section 3.1, lines 413–416, and Appendix E, lines 901–904). We trained a matched attention model on the same frozen record embeddings and split across 1273512735 fitting subsets to score all 1648716487 held-out subsets directly without access to questions (Table 7, lines 419–427, and Appendix E, Table 12, lines 905–917).

MethodK=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
Subset oracle28.4330.1830.3526.2319.36
Exact Shapley credit25.5026.5827.4022.7317.18
SCAP19.4516.6014.199.164.82
Direct subset predictor13.7210.467.583.620.64

3. Uncertainty, answer quality, baseline controls, and cost. We repeated training across ten random seeds for SCAP and all learned ablations (Section 2.7, lines 238–239, and Tables 3–4, lines 304–334).

Reader and policyK=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
Qwen3-8B, SCAP17.4±2.817.4\pm2.814.4±4.614.4\pm4.610.9±4.910.9\pm4.96.9±4.36.9\pm4.32.9±3.62.9\pm3.6
Qwen3-8B, A-MAC / SAGE16.6 / 16.212.8 / 11.78.5 / 7.4−0.6-0.6 / 2.5−6.5-6.5 / −4.1-4.1
Llama-3.1-8B, SCAP19.4±1.719.4\pm1.716.6±2.516.6\pm2.513.3±3.113.3\pm3.19.1±2.99.1\pm2.93.1±1.93.1\pm1.9
Llama-3.1-8B, A-MAC / SAGE17.5 / 17.514.6 / 13.310.0 / 9.15.3 / 5.3−1.3-1.3 / −0.7-0.7

Paired jackknife tt-tests omitting one test question and its stores at a time on the Table 2 selections averaged from K=5K=5 to K=1K=1 give +6.69+6.69 bits over A-MAC (p=.015p=.015) and +6.11+6.11 bits over SAGE (p=.048p=.048, lines 314–317), and pairing reader and grader across Qwen3-8B and Llama-3.1-8B places SCAP first at every capacity in both directions (lines 342–347 and Table 10, lines 828–848). Across ten seeds in Section 3.1, SCAP also leads fitted A-MAC (Table 5, lines 371–385), pooled complete memory text (Table 6, lines 388–404), and predictors trained on leave-one-out credit, single-question credit, or candidate records alone (Table 8, lines 407–454), while Section 2.8 (lines 255–258) notes that SAGE's rewriting and merging stages modify stored memories. Appendix B (lines 821–826) reports 194598194598 offline reader evaluations (12.5212.52 wall-hours on seven Quadro RTX 6000 GPUs), five minutes per fit on one GPU, and 7.97 ms7.97\text{ ms} median store scoring after one extractor and encoder pass per arrival.

To complement our response above, we answer each item in Questions for Authors directly in its numbered order (references cite the revised manuscript).

Question 1 (Justification of top-KK Shapley ranking, subset optimization, and decomposition of the oracle gap). Under additive utility across memories, top-KK Shapley ranking is exact for maximizing subset utility at any capacity KK. When memories interact, top-KK ranking carries no general subset guarantee, yet Shapley attribution converts joint reader utility across subsets into dense individual supervision that avoids combinatorial subset search at admission time (Section 2.5, lines 207–217, Section 3.1, lines 413–416, and Appendix E, lines 901–904). Comparing directly against subset optimization under the same information constraints, a matched attention model with a learned embedding for capacity KK, trained across 1273512735 fitting subsets to score all 1648716487 test subsets directly without question access, trails SCAP at every capacity from K=5K=5 to K=1K=1 (Table 12, lines 905–917, and Table 3, lines 304–312). Decomposing the gap to the subset oracle across K=5K=5 to K=1K=1, exact Shapley ranking retains 86.7%86.7\% to 90.3%90.3\% of oracle utility (Table 7, lines 419–427). At K=2K=2, for instance, exact Shapley sits 3.503.50 bits below the oracle whereas the learned policy sits 13.5713.57 bits below exact Shapley (Appendix E, lines 902–904), so 67%67\% to 85%85\% of the total gap to the oracle across capacities arises from predicting credit before questions arrive while Shapley ranking accounts for a small fraction.

Question 2 (Sequential arrival, irreversible eviction, store scale, and evolving demand). Replaying all 3434 held-out LongMemEval histories chronologically (16401640 arrivals, 4242 to 5555 sessions each) with permanent eviction and capacities extended to K=8K=8 and K=10K=10, SCAP retains higher utility than A-MAC and SAGE at every capacity from K=10K=10 down to K=1K=1 (lines 353–358 and Appendix D, Table 11, lines 863–883), while negative values at K≤3K\le 3 reflect permanent loss of early context in long streams. On shifted query distributions, chronological replay on all ten LoCoMo conversations (272272 sessions, 200200 questions) without retraining places SCAP ahead of every baseline from K=5K=5 to K=1K=1 (Table 11, lines 890–896) and preserves positive utility across early (2.532.53 bits), middle (1.751.75 bits), and late (6.876.87 bits) supporting sessions at K=5K=5 (lines 872–875), while online adaptation to nonstationary query streams remains future work (Section 4, lines 482–484).

Question 3 (Variability, QA significance, baseline workflows, and costs vs deployment). Across ten random seeds (lines 238–239), SCAP's mean retained utility leads A-MAC, SAGE, and random selection at every capacity under both Qwen3-8B (Table 3, lines 304–312) and Llama-3.1-8B (Table 4, lines 325–334), and paired jackknife tt-tests omitting one test question and its stores at a time confirm gains of +6.69+6.69 bits over A-MAC (p=.015p=.015) and +6.11+6.11 bits over SAGE (p=.048p=.048, lines 314–317). On QA accuracy specifically, differences under Qwen3-8B alone in Table 2 (lines 283–291) do not reach statistical significance because binary grades saturate once the supporting session is retained (lines 293–297), whereas pairing reader and grader across Qwen3-8B and Llama-3.1-8B yields consistent gains for SCAP at every capacity in both directions (lines 342–347 and Table 10, lines 828–848). All baselines use the identical extracted records (lines 155–166, 254–255), SCAP outperforms fitted A-MAC and pooled complete memory text across ten seeds (Tables 5–6, lines 371–404), and Section 2.8 (lines 255–258) notes that SAGE's rewriting and merging stages modify stored memories beyond admission. Offline labeling used 194598194598 reader evaluations (12.5212.52 wall-hours on seven Quadro RTX 6000 GPUs) and policy fitting took five minutes on one GPU versus 7.97 ms7.97\text{ ms} median store scoring after one extractor and encoder pass per arrival (Appendix B, lines 821–826). We claim answer-information efficiency under fixed capacity and make no claim of overall compute savings (lines 0–30, 50–83).

The rebuttal deals with almost all of my main concerns via a series of chronological admission experiments, oracle-gap decomposition, direct subset prediction comparisons, multi-seed evaluation, significance testing and cost analysis. There are still some issues to be resolved with respect to scale, online adaptation, Shapley optimality in the presence of interactions and QA significance, but the response is a significant improvement on the paper and I would go from a 3 to a 4, pending verification of the new results in the revised manuscript.

Reviewer_VpYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SCAP, a Shapley Credit Assignment Policy for memory admission under unknown demand. It defines memory-set utility using the reduction in a frozen reader’s codelength for reference answers, assigns credit to individual memories with Shapley values, and trains a policy to predict this credit at write time.

Critical strengths
  1. The paper provides a clear formulation of memory admission as credit assignment under unknown demand, explicitly accounting for interactions among memories.
  2. The proposed Shapley-based supervision is simple and well motivated, and consistently improves retained utility over A-MAC and SAGE across tested capacities.
Critical weaknesses
  1. The empirical evaluation is limited to a subset of LongMemEval with relatively small memory stores, which may limit the generality of the conclusions.
  2. Unknown demand is a central motivation of the paper, but both training and evaluation are constructed from the same LongMemEval benchmark. It remains unclear how well the policy generalizes when future demand differs from this setting.
Questions for authors
  1. Could the authors further evaluate SCAP on another memory benchmark or a larger-scale memory setting?
  2. Since unknown demand is central to the paper, could the authors clarify how SCAP performs when future demand shifts beyond the LongMemEval distribution?
2 comments

We thank the reviewer for the clear and constructive comments. We hope the clarifications below can help address the reviewer's concerns.

1. Benchmark breadth and memory scale. The initial submission evaluated completed LongMemEval stores of five to ten sessions. In the revised manuscript, we expanded both stream scale and benchmark breadth without retraining, checkpoint selection, or hyperparameter tuning (Abstract, lines 27–28, Section 1, lines 82–83, Section 3, lines 353–362, and Appendix D, Table 11, lines 859–896). First, we replayed all 3434 held-out LongMemEval histories in timestamp order across 16401640 arriving sessions (16211621 usable extractions, 4242 to 5555 arrivals per history). Arrivals enter the store until capacity KK is reached, after which each new arrival is scored alongside the KK current residents without question access, the KK memories with the highest scores are retained, and rejected or evicted memories never return, including at capacities K=8K=8 and K=10K=10 beyond the store sizes seen in training.

Chronological LongMemEvalK=10K=10K=8K=8K=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
SCAP17.8013.600.810.45−0.29-0.29−4.73-4.73−4.06-4.06
A-MAC11.784.81−0.50-0.50−0.69-0.69−5.14-5.14−11.68-11.68−13.03-13.03
SAGE12.588.35−3.30-3.30−2.62-2.62−5.60-5.60−8.41-8.41−9.91-9.91

Second, we applied the same LongMemEval policy to all ten conversations of LoCoMo (272272 sessions, 271271 usable extractions) under the identical chronological admission rule, evaluating 2020 questions per conversation (200200 total) that had standard reference answers supported by text in one session and averaging utility within each conversation before averaging across conversations (Appendix D, lines 868–872 and 890–896). We also evaluated reader token budgets from 40964096 down to 10241024 tokens on the 8888 held-out LongMemEval stores, where SCAP again leads A-MAC and SAGE at every budget (Section 3, lines 348–352, and Table 11, lines 885–889), and repeated training across ten random seeds for the main Qwen3-8B evaluation, Llama-3.1-8B reader transfer, and all learned ablations (Section 2.7, lines 238–239, Tables 3–4, lines 304–334, and Tables 5, 6, and 8, lines 379–454).

Chronological LoCoMoK=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
SCAP3.683.970.57−2.74-2.74−7.64-7.64
A-MAC1.23−3.22-3.22−3.31-3.31−6.62-6.62−10.60-10.60
SAGE−1.08-1.08−2.52-2.52−5.68-5.68−7.01-7.01−11.50-11.50
SAGE, centrality3.140.66−0.84-0.84−3.73-3.73−11.14-11.14
Random0.78−1.29-1.29−2.99-2.99−5.71-5.71−10.44-10.44

2. Transfer across demand distributions. LoCoMo conversations and questions come from a distinct dialogue corpus spanning many sessions with its own topic structure and question styles, testing whether a fixed policy trained on LongMemEval demand transfers when the query distribution changes. Grouping the 198198 valid LoCoMo questions at K=5K=5 by whether their supporting session occurs early, in the middle, or late in the stream gives 2.532.53, 1.751.75, and 6.876.87 bits for SCAP, compared with −5.49-5.49, 0.290.29, and 8.388.38 bits for A-MAC and 0.660.66, 1.131.13, and −3.23-3.23 bits for SAGE (Appendix D, lines 872–875), showing that SCAP preserves useful early and middle sessions that baselines relying on recency or novelty evict. At the same time, Section 3.1 (lines 435–444), Section 4 (lines 477–484), and Appendix E (lines 918–929) state the remaining scope boundaries transparently, noting that LoCoMo comprises ten conversations, that both SCAP and the tested baselines remain near chance when answers require combining facts across multiple sessions, and that online adaptation to nonstationary query streams remains open for future work.

To complement our response above, we answer both items in Questions for Authors directly (line and table references cite the revised manuscript).

Question 1 (Evaluation on another memory benchmark and a larger-scale memory setting). We evaluated the unchanged LongMemEval policy on both an external benchmark and longer sequential streams without retraining or tuning (Section 3, lines 348–362, and Appendix D, Table 11, lines 859–896). On LoCoMo (1010 conversations spanning 272272 sessions and 200200 questions), SCAP achieves the highest retained utility at every capacity from K=5K=5 to K=1K=1, exceeding A-MAC, SAGE, SAGE centrality, and random selection by 0.550.55 to 3.313.31 bits over the closest comparator (Table 11, lines 890–896). To test larger-scale streams and capacities, we replayed all 3434 full held-out LongMemEval histories chronologically across 16401640 arrivals (4242 to 5555 sessions per history, five to ten times longer than the completed stores) with irreversible eviction up to capacities K=8K=8 and K=10K=10, and also evaluated Qwen3-8B reader token budgets up to 40964096 tokens on the 8888 test stores, with SCAP leading A-MAC and SAGE at every tested capacity and token budget (Table 11, lines 877–889).

Question 2 (Performance when future demand shifts beyond the LongMemEval distribution). In our formulation, unknown demand means that future questions are unobserved at admission while policy parameters stay fixed at deployment (Section 1, lines 52–55). Transferring the fixed LongMemEval policy directly to LoCoMo tests a distinct dialogue and question distribution, and grouping the 198198 valid LoCoMo questions at K=5K=5 by early, middle, and late supporting sessions shows that SCAP retains positive utility in all three regimes (2.532.53, 1.751.75, and 6.876.87 bits versus −5.49-5.49, 0.290.29, and 8.388.38 for A-MAC and 0.660.66, 1.131.13, and −3.23-3.23 for SAGE, Appendix D, lines 872–875). At the same time, when shifting from questions supported by one session to 489489 stores built from questions that combine facts across two or more sessions, held-out pairwise accuracy drops near chance for both SCAP (0.5180.518) and the baselines (0.5300.530 for A-MAC with equal weights, 0.5290.529 for A-MAC with fitted weights, and 0.4860.486 for SAGE, Section 3.1, lines 435–444, and Appendix E, lines 918–926), and adapting policy weights online as query distributions evolve remains future work (Section 4, lines 477–484).

Reviewer_VtXwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCAP learns to retain memories before future questions arrive. It measures a memory set’s utility through reduced reference-answer codelength under a frozen reader, assigns individual Shapley credits, and trains a predictor over frozen record embeddings. Evaluation uses small stores constructed from LongMemEval, with an additional reader-transfer experiment.

Critical strengths
  • This paper deals with important challenges in persistent memory, where the system should proactively determine which information should be retained before the question arrives by assigning the importance of each memory.
  • Utility improvements persist with a Llama reader without retraining, suggesting that the learned ranking captures information useful beyond the training reader.
Critical weaknesses
  • Novelty and efficiency need clearer grounding. Data Shapley [1] already studies contribution-based valuation, and FastSHAP amortizes Shapley estimation through a learned predictor. Position SCAP’s contribution around memory-specific utility and prediction without the future query. Report labeling cost, prediction latency, permutation-estimate variance, and scaling beyond ten memories, including token-budget evaluation. Cheap inference alone does not establish efficient training or deployment.
  • Baseline comparisons leave representation and workflow confounds. SCAP receives individual record embeddings, whereas A-MAC and SAGE use pooled memory representations. The fitted A-MAC ablation is helpful, but stronger comparisons should give learned baselines equivalent inputs and supervision. SAGE’s novelty-only ranking also omits its adaptive gating and merging workflow, limiting conclusions about the original system.
  • The evaluation weakly tests the central interaction hypothesis. Could the authors justify evaluating exclusively on a single-session-support subset of LongMemEval? Since SCAP is motivated by interactions among memories, evaluation on multi-session questions and another benchmark would help establish whether its advantages extend beyond the constructed small-store setting. At minimum, could the authors include controlled cases requiring complementary memories and test transfer to a different question distribution?
  • Shapley attribution is not equivalent to budget-optimal selection. Shapley values average contributions across all coalition sizes, whereas deployment requires maximizing utility at a particular K. Ranking once can retain redundant memories or split complementary groups. Explain this mismatch and compare against budget-conditioned attribution, sequential selection, or direct subset-utility learning.
  • Codelength gains do not consistently translate into better answers. SCAP trails A-MAC at K=3, and its QA advantage nearly disappears at K=5. Summed log-probabilities also weight longer reference answers more heavily and depend on reference wording. Test alternative references, length-normalized targets, and an independent grader; report generated-answer accuracy for the transferred reader.

[1] Data Shapley: Equitable Valuation of Data for Machine Learning
[2] FastSHAP: Real-Time Shapley Value Estimation

Questions for authors

see weaknesses

2 comments

We thank the reviewer for the constructive points raised. We hope the clarifications below help address the reviewer's concerns. All refences point to the revised manuscript.

1. Prior Shapley work, cost, and token budgets. Section 1.1 (lines 105–109) cites Data Shapley and FastSHAP and clarifies our focus on learning admission over extracted records and resident memories when future questions are unknown. Section 2.5 (lines 219–222) reports exact credits over 2n2^n subsets for n≤6n\le 6 and 6464 sampled permutations for 7≤n≤107\le n\le 10, with mean SE 0.90±0.440.90\pm0.44 bits across 25192519 sampled labels (split counts in Appendix A.2, lines 739–742). Appendix B (lines 821–826) reports 194598194598 offline reader evaluations (12.5212.52 wall-hours on seven Quadro RTX 6000 GPUs), five minutes per fit on one GPU, and 7.97 ms7.97\text{ ms} median store scoring after one extractor and encoder pass per arrival. Appendix D (Table 11, lines 863–889) extends chronological replay to K=8K=8 and K=10K=10, where SCAP leads at every capacity, and evaluates Qwen3-8B token budgets on the 8888 test stores (lines 348–358).

Policy4096 tokens3072 tokens2048 tokens1024 tokens
SCAP18.2314.6110.180.96
A-MAC16.327.790.34−4.51-4.51
SAGE16.5610.263.74−2.93-2.93

2. Representation, supervision, and workflow controls. All policies act on the same extracted records (lines 155–166, 254–255). Across ten random seeds (lines 238–239), SCAP leads fitted A-MAC (Table 5, lines 371–385), pooled complete memory text and hierarchical pooling (Table 6, lines 388–404), and predictors trained on leave-one-out credit, single-question credit, or candidate records alone (Table 8, lines 407–454), alongside Qwen3-8B and Llama-3.1-8B evaluations (Tables 3–4, lines 304–334). Section 2.8 (lines 255–258) notes that SAGE's rewriting and merging stages modify stored memories beyond admission.

3. Multiple supporting sessions, similar distractors, and transfer. On 489489 stores supported by two or more sessions (389389 train, 100100 test, lines 435–444, 477–481, and Appendix E, lines 918–929), exact Shapley retains more utility than leave-one-out credit at tight capacities (2.262.26 versus −0.28-0.28 bits at K=1K=1), while test pairwise accuracy stays near chance across all policies (0.5180.518 for SCAP, 0.5300.530 for A-MAC with equal weights, 0.5290.529 for A-MAC with fitted weights, and 0.4860.486 for SAGE). Under distractors chosen for embedding similarity in 100100 test stores, pairwise accuracy is 0.431±0.0540.431\pm0.054 across ten SCAP seeds versus 0.5990.599, 0.5420.542, and 0.5760.576 for those baselines (lines 926–929), showing sensitivity when relying on record embeddings alone. Chronological replay on all ten LoCoMo conversations (272272 sessions, 200200 questions) without retraining places SCAP first from K=5K=5 to K=1K=1 (lines 359–362 and Table 11, lines 890–896).

4. Subset selection under a capacity limit. Top-KK Shapley ranking is optimal under additive utility and carries no general guarantee under interactions (lines 413–416, 901–904). We trained a matched attention model across 1273512735 fitting subsets to score all 1648716487 test subsets directly without question access (Table 7, lines 419–427, and Table 12, lines 905–917).

MethodK=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
Subset oracle28.4330.1830.3526.2319.36
Exact Shapley credit25.5026.5827.4022.7317.18
SCAP19.4516.6014.199.164.82
Direct subset predictor13.7210.467.583.620.64

5. Answer grading and reference sensitivity. Paired jackknife tt-tests omitting one test question and its stores at a time give +6.69+6.69 bits over A-MAC (p=.015p=.015) and +6.11+6.11 bits over SAGE (p=.048p=.048, lines 314–317). Pairing reader and grader across Qwen3-8B and Llama-3.1-8B in both directions and rescoring 3434 verified Llama-3.1-8B paraphrases in bits and bits per target token also place SCAP first from K=5K=5 to K=1K=1 (lines 319–322, 342–347, and Table 10, lines 828–856).

Reviewer has also referred to the Weaknesses under Questions for Authors. To ensure every direct question embedded within those Weaknesses is answered explicitly, we summarize them below (references cite the revised manuscript).

1. Justification for using questions supported by one session in the primary setup (Weakness 3). Starting with questions supported by a single annotated session isolates redundancy, distractor competition, and store context cleanly without confounding admission errors with synthesis failures across multiple sessions in the reader (Section 2.2, lines 138–153, and Section 3, lines 293–297). To test complementary memories directly, we then constructed 489489 LongMemEval stores from questions supported by two or more sessions (389389 train, 100100 test, Section 3.1, lines 435–444, and Appendix E, lines 918–926), where exact Shapley credit separates complementary sessions better than leave-one-out credit at tight capacities (2.262.26 versus −0.28-0.28 bits at K=1K=1) while held-out pairwise accuracy stays near chance across SCAP (0.5180.518), A-MAC with equal weights (0.5300.530), A-MAC with fitted weights (0.5290.529), and SAGE (0.4860.486). For transfer to a different question distribution, chronological replay on all ten LoCoMo conversations (272272 sessions, 200200 questions) without retraining places SCAP first at every capacity from K=5K=5 to K=1K=1 (Section 3, lines 359–362, and Appendix D, Table 11, lines 890–896).

2. Generated-answer QA accuracy for the transferred Llama-3.1-8B reader (Weakness 5). Generating answers with the transferred Llama-3.1-8B reader from the selected memories and grading them under the LongMemEval rubric with Qwen3-8B, SCAP achieves the highest QA accuracy at every capacity from K=5K=5 to K=1K=1 (Section 3, lines 342–347, and Appendix C, Table 10, lines 845–848).

Llama-3.1-8B reader, Qwen3-8B grader (%)K=5K=5K=4K=4K=3K=3K=2K=2K=1K=1
SCAP50.443.735.429.318.2
A-MAC44.536.422.611.12.8
SAGE44.329.821.012.95.9

Comments on the paper

1 comment

We thank Reviewers 8UZw, VpYt, and VtXw for their constructive feedback and have uploaded a revised manuscript alongside detailed responses to all Weaknesses and Questions for Authors.

To address the four shared themes across the reviews, our revision adds the following experiments and clarifications.

  • Sequential streaming and external transfer (8UZw, VpYt, VtXw). We evaluate the unchanged policy under chronological arrival with irreversible eviction on full LongMemEval histories at larger capacities, across reader token budgets, and on LoCoMo without retraining (Section 3, lines 348–362, Appendix D, Table 11, lines 859–896).
  • Statistical rigor and grading controls (8UZw, VpYt, VtXw). We expand the main evaluations and learned ablations to ten random seeds, add paired significance tests accounting for question reuse across stores, swap reader and grader families reciprocally, and verify independent answer paraphrases (Tables 3–6 and 8, lines 304–454, Appendix C, Table 10, lines 828–856).
  • Subset selection scope (8UZw, VtXw). We clarify the theoretical conditions for top-KK Shapley ranking, decompose the gap to the subset oracle, and compare against a matched neural predictor of complete subset utility (Section 3.1, Table 7, lines 413–427, Appendix E, Table 12, lines 899–917).
  • Empirical boundaries, framing, and reproducibility (8UZw, VpYt, VtXw). We add baseline comparisons on questions supported by multiple sessions and semantically similar distractors, align the title and framing strictly with retaining answer information under fixed capacity, and report complete split, Monte Carlo, and compute diagnostics (lines 219–222, 435–444, 477–485, 739–742, 821–826, 918–929).

We hope the extended experiments and clarifications explained in rebuttal responses help clarify the concerns regarding the submitted manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.