- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MemQuilt, a memory selection framework for LLM agents operating under a context budget. It identifies missing task requirements, estimates how each candidate memory supports them, and selects memories through a submodular objective combining individual utility with concave requirement coverage. Cost-aware greedy selection and threshold stopping control context use. Experiments on MS-TOD, Mem2Act, and a single-hop subset of LoCoMo report improved task performance with fewer memory-context tokens than relevance-based selectors.
Critical strengths
-
Distinguishing task complementarity from semantic diversity is well motivated. Requirement-level support offers an interpretable basis for selecting memories that address different information needs.
-
The selection procedure explicitly enforces the context budget and requires no additional model calls during greedy construction. The theoretical analysis carefully states the scope and limitations of its guarantees.
-
Shared candidate pools, Readers, and memory ordering provide useful experimental controls. Budget sweeps and component ablations offer evidence that the method can preserve task quality with more compact memory contexts.
Critical weaknesses
-
The headline context savings do not clearly isolate the contribution of set selection. Based on Tables 9 and 10, approximately 91% of the MS-TOD token reduction comes from skipping memory for queries with no identified requirements. This rule is not applied to the pointwise baselines. Among queries with active requirements, context decreases only from 44.85 to 42.42 tokens. Although the w/o DR ablation supports diminishing returns, a pointwise baseline with the same skipping rule and a tuned stopping threshold would better establish the additional benefit of submodular selection.
-
The advantage over stronger alternatives is less convincing. SU nearly matches MemQuilt’s F1 on Mem2Act and LoCoMo, although it uses more context (Tables 7 and 8). Moreover, Direct Gemini achieves higher TaskSucc than MemQuilt, 0.507 versus 0.477, with fewer memory tokens and no budget violations (Table 4). MemQuilt retains higher SlotAcc, so neither method consistently dominates. Extending this comparison beyond the 300-query MS-TOD subset would clarify the practical advantage of the proposed approach.
-
The practical cost of memory selection remains unclear. MemTok measures only the context passed to the Reader, excluding requirement analysis and candidate assessment. These stages process 50 candidate memories per query on Mem2Act and LoCoMo, whereas the reported context savings are only tens of tokens. Total model input and output tokens, latency, and inference cost are needed to determine whether the reduced Reader context compensates for the additional selection overhead.
-
The evaluation only partly tests the central claim about complementary evidence. LoCoMo is restricted to single-hop questions, and the analyzer identifies only one requirement for 74.0% of Mem2Act queries and 85.7% of LoCoMo queries. These settings primarily test identifying useful evidence rather than composing several information needs. This matters because independently estimated, nonnegative support scores cannot explicitly distinguish mutually consistent evidence from conflicting values across memories. Multi-hop evaluation and targeted conflict analyses would better assess this limitation.
-
Hyperparameter selection is insufficiently documented. Appendix C.4 uses benchmark-specific parameters for MemQuilt while retaining globally fixed settings for MMR, Dedup, and SU. Different settings can be reasonable across benchmarks, but the paper does not describe the validation split, search ranges, or selection criterion. Comparable tuning opportunities and sensitivity analyses are needed to establish whether the reported trade-offs are robust.
Questions for authors
Please refer to the summary of weaknesses.
9 comments
Response to Reviewer dHbY (2of9)
Response to W2: MemQuilt's Advantages over Scalar Utility and Model-Directed Selection
Thank you for raising this insightful question. We would like to offer two clarifications regarding the concern.
(1) Advantages over Scalar Utility.
First of all, unlike MS-TOD, Mem2Act and LoCoMo often have only one identified requirement. Scalar usefulness can therefore already provide a strong selection signal. MemQuilt further considers what each memory adds to the selected set. In this setting, its benefit is similar F1 with less Reader memory context: 65.2% less on Mem2Act and 16.8% less on LoCoMo (Table R2).
Besides, Table 2 of the paper shows that, on multi-requirement MS-TOD tasks, stronger individual values are useful but cannot replace requirement-aware set-level optimization.
Table R2. SU and MemQuilt in the original Mem2Act and LoCoMo evaluations (Tables 7 and 8 of the paper).
| Dataset | SU F1 | MemQuilt F1 | SU MemTok | MemQuilt MemTok | Memory context reduction |
|---|---|---|---|---|---|
| Mem2Act | 0.4108 | 0.4144 | 95.68 | 33.31 | 65.2% |
| LoCoMo | 0.6292 | 0.6319 | 264.44 | 220.07 | 16.8% |
(2) Advantages over Model-Directed Selection.
We agree that a larger evaluation would help clarify MemQuilt's advantages. We therefore extended the comparison from 300 to all 982 MS-TOD queries, following the protocol in Tables 4 and 5 of the paper (Table R3). The Gemini trade-off remains consistent with the paper. With GPT-4o-mini, MemQuilt achieves higher task scores with less memory context, and its TaskSucc lead grows from 3.0 to 5.1 percentage points.
Here, the key difference is the role of the model. The Direct Set Selector (DSS) asks the model to assess candidates jointly and choose the memory set. In contrast, MemQuilt uses the model to estimate requirement-level support. An explicit algorithm then builds the set, updating marginal gains as it grows and enforcing the token budget. This design decouples semantic assessment from budget-constrained set construction.
This assigns the model a focused semantic-assessment task, supported by prior LLM-as-a-judge studies [8,9]. Specialized LLM and MLP scorers [1,2,9] also offer a practical route to lower-cost local assessment. We can replace or specialize the assessor without training a new set-selection policy. Requirement-aware optimization stays unchanged, and each choice remains traceable to marginal gain and token cost. The model-swap results support transferability across the two tested models: MemQuilt retains task performance better when the assessor changes.
Table R3. Selection-model comparison on all 982 MS-TOD queries at B = 35. MemQuilt uses weighted individual values. Raw budget violation is measured before repair; DSS task metrics and MemTok are computed after deterministic budget repair.
| Selection-stage model | Method | SlotAcc | TaskSucc | MemTok | Raw budget violation |
|---|---|---|---|---|---|
| Gemini | DSS | 0.7425 | 0.5418 | 18.89 | 0.0% |
| Gemini | MemQuilt | 0.7468 | 0.5000 | 20.17 | 0.0% |
| GPT-4o-mini | DSS | 0.6476 | 0.4369 | 29.81 | 68.2% |
| GPT-4o-mini | MemQuilt | 0.7352 | 0.4878 | 20.75 | 0.0% |
Response to Reviewer dHbY (3of9)
Response to W3: Selection Overhead and End-to-End Inference Cost
Thank you for highlighting this point. We agree that deployment cost is central to MemQuilt's practical value. In our paper, to better isolate the contribution of requirement-aware set selection, we keep individual values the same across selectors: relevance-based values in Table 1 and stronger scalar values in Table 2. Fig. 3 examines both settings across budgets. This controlled design adds support-vector assessments for MemQuilt per active query with candidates. Table 1 also adds one Analyzer call, while Table 2 shares the Analyzer and scalar assessments with the baselines. Building on these controlled comparisons, we further examine MemQuilt's practical deployment value.
(1) Practical Deployment.
In practice, agent systems commonly use specialized models trained for memory assessment or context preparation, while a separate Reader performs the task [1,2,6,7,10,11]. For example, MemGuide uses a fine-tuned LLaMA-8B filter and evaluates GPT-4o-mini as the Reader [1]. RMM uses a trained MLP to rerank memories and Gemini to generate responses [2]. Task execution and candidate assessment therefore need not use the same costly model. This makes efficient allocation of the Reader's limited memory context valuable in its own right, particularly in paid API deployments, where memory tokens contribute directly to the Reader's input cost.
With this deployment perspective in mind, we examine memory selection in MemGuide's MS-TOD confirmation-response setting [1]. At , we compare MemQuilt with usefulness-weighted Top-K, MMR, and Dedup, keeping the candidate pool, Analyzer, Reader, and evaluator fixed. MemQuilt uses relevance-based individual values here, so its vector Judge replaces the scalar Judge used by the baselines. Both pipelines therefore require one Analyzer and candidate assessments per active query.
(2) Selection Cost.
We report Judge token counts and local selection latency separately. For the local-scorer deployment considered here, let be the task state, a candidate memory, and the active requirements, with .
Table R4. Memory-selection workload per active query after shared candidate retrieval. The baseline column denotes any one of weighted Top-K, MMR, or Dedup; Reader costs are excluded.
| Top-K / MMR / Dedup | MemQuilt (relevance value) |
|---|---|
| Requirement analysis: one shared Analyzer evaluation. | Requirement analysis: the same Analyzer evaluation. |
| Judge: , one usefulness score. | Judge: , one support score per requirement. |
| Candidate assessment: scalar assessments. | Candidate assessment: vector assessments. |
| Local selection: ranking or similarity-based selection, with required preprocessing; total latency , no further Judge calls. | Local selection: marginal-gain-based selection; total latency , no further Judge calls. |
Judge Token Counts.
We count input and output tokens in reconstructed Judge prompts and cached responses across 49,608 paired candidate assessments (Table R4a).
Table R4a. Average Judge input and output tokens per candidate memory. Counts use cl100k_base; outputs include short reasons. Analyzer and Reader are excluded.
| Tokens per memory | Scalar Judge | Vector Judge |
|---|---|---|
| Input | 302.21 | 506.02 |
| Output | 15.77 | 81.64 |
Response to Reviewer dHbY (1of9)
Thank you for your careful review and for recognizing the motivation behind MemQuilt, its explicit budget control, and our experimental design. Below, we address your concerns one by one.
Response to W1: MemQuilt's Contribution Beyond the Skipping Rule and Early Stopping
Thank you for this insightful suggestion. We agree that separating submodular selection from the effects of skipping and threshold stopping helps us better examine its contribution. We therefore analyze all 645 MS-TOD queries with active requirements, where the skipping rule is inactive. On this same group, we separately disable threshold stopping and diminishing returns in MemQuilt, keeping all other settings fixed.
As shown in Table R1, removing threshold stopping leaves SlotAcc and TaskSucc almost unchanged, while MemTok increases from 42.42 to 43.71. In contrast, removing diminishing returns lowers SlotAcc by 3.10 percentage points and TaskSucc by 4.65 points, despite nearly identical memory-context use (42.39 vs. 42.42 MemTok). Diminishing returns therefore helps select a more useful memory set, rather than simply reducing context length. Threshold stopping provides a further reduction in memory context.
Table R1. Targeted component ablations on 645 MS-TOD queries with active requirements identified by the frozen Analyzer. Budgets match relevance-based Top-5 per query. Each ablation changes one component of full MemQuilt: w/o TS sets τ=0; w/o DR sets ρ=1 and retains the original threshold.