- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses memory admission for language-model agents operating under limited storage capacity and uncertain future demand. The proposed SCAP framework estimates memory utility using reference-answer codelength reduction, assigns contributions via Shapley values, and learns admission decisions through an attention-based predictor. Experiments on LongMemEval memory stores show improved retained utility over A-MAC and SAGE across storage capacities, with evidence of cross-reader generalization.
Critical strengths
-
The paper presents a principled framework that combines reference-answer codelength reduction with Shapley attribution to quantify memory contributions while accounting for redundancy and complementarity. Unlike heuristic retention strategies based on recency, novelty, or confidence, SCAP learns retention from downstream information utility, providing a theoretically grounded and methodologically significant approach to persistent LLM memory management.
-
SCAP consistently outperforms A-MAC and SAGE across all tested memory capacities, with especially large gains under tight memory constraints. Ablation studies validate the importance of Shapley-based supervision, record-level representations, and store-context awareness, while strong performance with Llama-3.1-8B without retraining provides preliminary evidence of transferability across reader architectures.
Critical weaknesses
-
The main limitation is the gap between the sequential memory-admission problem and the static subset-selection evaluation. Experiments on 88 small, curated memory stores do not capture key real-world challenges such as sequential admissions, irreversible evictions, evolving user preferences, changing demand distributions, and larger memory scales. Thus, the results demonstrate effectiveness in a simplified offline setting rather than practical long-term memory management.
-
A key concern is that Shapley-value ranking does not generally optimize utility under a fixed memory budget, and the manuscript provides neither theoretical guarantees nor conditions linking these objectives. Moreover, the gaps between the subset oracle, exact Shapley selection, and SCAP's learned predictions (e.g., at K=2) indicate substantial attribution and prediction errors.
-
The manuscript reports only mean performance, without confidence intervals, significance tests, or multi-seed variability analyses, and store dependencies may reduce the number of independent evaluation units. Moreover, higher retained codelength utility does not consistently translate into better QA accuracy (e.g., at K=3).
Questions for authors
-
Since Shapley values capture average marginal contributions rather than optimal fixed-capacity selections, what justifies using top-KKK Shapley rankings for memory retention? Can the authors provide theoretical justification or comparisons with capacity-aware and direct subset-optimization methods? Also, how much of the gap to the oracle is attributable to Shapley ranking itself versus errors in the learned predictor?
-
How does SCAP perform in a realistic sequential setting where memories arrive over time and discarded information cannot be revisited? Evaluating chronological interaction histories with larger memory stores, repeated admission and eviction decisions, and evolving query distributions would better demonstrate its effectiveness for long-term memory management.
-
Can the authors provide confidence intervals, significance tests, and variability analyses for the reported improvements, particularly in QA accuracy? How do the results compare against complete baseline workflows or matched memory representations? Also, what are the computational costs of Shapley-label generation and policy training relative to the deployment-time savings achieved?
3 comments
To complement our response above, we answer each item in Questions for Authors directly in its numbered order (references cite the revised manuscript).
Question 1 (Justification of top- Shapley ranking, subset optimization, and decomposition of the oracle gap). Under additive utility across memories, top- Shapley ranking is exact for maximizing subset utility at any capacity . When memories interact, top- ranking carries no general subset guarantee, yet Shapley attribution converts joint reader utility across subsets into dense individual supervision that avoids combinatorial subset search at admission time (Section 2.5, lines 207–217, Section 3.1, lines 413–416, and Appendix E, lines 901–904). Comparing directly against subset optimization under the same information constraints, a matched attention model with a learned embedding for capacity , trained across fitting subsets to score all test subsets directly without question access, trails SCAP at every capacity from to (Table 12, lines 905–917, and Table 3, lines 304–312). Decomposing the gap to the subset oracle across to , exact Shapley ranking retains to of oracle utility (Table 7, lines 419–427). At , for instance, exact Shapley sits bits below the oracle whereas the learned policy sits bits below exact Shapley (Appendix E, lines 902–904), so to of the total gap to the oracle across capacities arises from predicting credit before questions arrive while Shapley ranking accounts for a small fraction.
Question 2 (Sequential arrival, irreversible eviction, store scale, and evolving demand). Replaying all held-out LongMemEval histories chronologically ( arrivals, to sessions each) with permanent eviction and capacities extended to and , SCAP retains higher utility than A-MAC and SAGE at every capacity from down to (lines 353–358 and Appendix D, Table 11, lines 863–883), while negative values at reflect permanent loss of early context in long streams. On shifted query distributions, chronological replay on all ten LoCoMo conversations ( sessions, questions) without retraining places SCAP ahead of every baseline from to (Table 11, lines 890–896) and preserves positive utility across early ( bits), middle ( bits), and late ( bits) supporting sessions at (lines 872–875), while online adaptation to nonstationary query streams remains future work (Section 4, lines 482–484).
Question 3 (Variability, QA significance, baseline workflows, and costs vs deployment). Across ten random seeds (lines 238–239), SCAP's mean retained utility leads A-MAC, SAGE, and random selection at every capacity under both Qwen3-8B (Table 3, lines 304–312) and Llama-3.1-8B (Table 4, lines 325–334), and paired jackknife -tests omitting one test question and its stores at a time confirm gains of bits over A-MAC () and bits over SAGE (, lines 314–317). On QA accuracy specifically, differences under Qwen3-8B alone in Table 2 (lines 283–291) do not reach statistical significance because binary grades saturate once the supporting session is retained (lines 293–297), whereas pairing reader and grader across Qwen3-8B and Llama-3.1-8B yields consistent gains for SCAP at every capacity in both directions (lines 342–347 and Table 10, lines 828–848). All baselines use the identical extracted records (lines 155–166, 254–255), SCAP outperforms fitted A-MAC and pooled complete memory text across ten seeds (Tables 5–6, lines 371–404), and Section 2.8 (lines 255–258) notes that SAGE's rewriting and merging stages modify stored memories beyond admission. Offline labeling used reader evaluations ( wall-hours on seven Quadro RTX 6000 GPUs) and policy fitting took five minutes on one GPU versus median store scoring after one extractor and encoder pass per arrival (Appendix B, lines 821–826). We claim answer-information efficiency under fixed capacity and make no claim of overall compute savings (lines 0–30, 50–83).
The rebuttal deals with almost all of my main concerns via a series of chronological admission experiments, oracle-gap decomposition, direct subset prediction comparisons, multi-seed evaluation, significance testing and cost analysis. There are still some issues to be resolved with respect to scale, online adaptation, Shapley optimality in the presence of interactions and QA significance, but the response is a significant improvement on the paper and I would go from a 3 to a 4, pending verification of the new results in the revised manuscript.
We thank the reviewer for the constructive feedback. Our main contribution is Shapley Credit Assignment Policy (SCAP), a learned admission policy that scores candidate and resident memories before future questions arrive, where information efficiency refers to answer information retained within a fixed memory count or reader token budget (Title, lines 0–4, Abstract, lines 11–30, and Section 1, lines 50–83). References below cite the revised manuscript. We hope the clarifications below help address the reviewer's concerns.
1. Chronological admission, irreversible eviction, and changing demand. We replayed all held-out LongMemEval histories (1640 arrivals, 42 to 55 sessions each) in timestamp order without retraining the policy, keeping the K memories with the highest scores at each arrival without question access while rejected or evicted memories never return up to K=10 (K=8,10 not seen in training) (lines 353–358 and Appendix D, Table 11, lines 863–883).
In our setup, future questions are unobserved at admission and policy parameters stay fixed at deployment (Section 1, lines 52–55). Replaying all LoCoMo conversations (272 sessions, 200 questions) chronologically without retraining, SCAP leads every baseline from K=5 to K=1 and remains positive for early, middle, and late supporting sessions at K=5 (lines 359–362 and Appendix D, Table 11, lines 868–875, 890–896). Online adaptation to shifting query streams remains future work (Section 4, lines 482–484).
2. Subset selection under a fixed capacity. Ranking memories by Shapley credit is optimal at capacity K under additive utility, whereas interactions remove any general subset guarantee (Section 3.1, lines 413–416, and Appendix E, lines 901–904). We trained a matched attention model on the same frozen record embeddings and split across 12735 fitting subsets to score all 16487 held-out subsets directly without access to questions (Table 7, lines 419–427, and Appendix E, Table 12, lines 905–917).
3. Uncertainty, answer quality, baseline controls, and cost. We repeated training across ten random seeds for SCAP and all learned ablations (Section 2.7, lines 238–239, and Tables 3–4, lines 304–334).
Paired jackknife t-tests omitting one test question and its stores at a time on the Table 2 selections averaged from K=5 to K=1 give +6.69 bits over A-MAC (p=.015) and +6.11 bits over SAGE (p=.048, lines 314–317), and pairing reader and grader across Qwen3-8B and Llama-3.1-8B places SCAP first at every capacity in both directions (lines 342–347 and Table 10, lines 828–848). Across ten seeds in Section 3.1, SCAP also leads fitted A-MAC (Table 5, lines 371–385), pooled complete memory text (Table 6, lines 388–404), and predictors trained on leave-one-out credit, single-question credit, or candidate records alone (Table 8, lines 407–454), while Section 2.8 (lines 255–258) notes that SAGE's rewriting and merging stages modify stored memories. Appendix B (lines 821–826) reports 194598 offline reader evaluations (12.52 wall-hours on seven Quadro RTX 6000 GPUs), five minutes per fit on one GPU, and 7.97 ms median store scoring after one extractor and encoder pass per arrival.