- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that memory selection for language agents should be supervised at the set level rather than solely by per-item signals. It proves a non-identification result for selectors restricted to anchored add-one and delete-one signals, proposes a set-derived per-item target obtained from optimal budgeted sets, and retains efficient Top-B item-wise selection at deployment. The paper evaluates the idea through exact subset enumeration on sealed MuSiQue pools, controlled synthetic/planning settings, and multi-hop QA experiments with gold supporting-set supervision and larger retrieval pools.
Critical strengths
-
The paper identifies a meaningful issue: evidence items can be complementary, so marginal item relevance need not recover the best budgeted evidence set. The formalization of anchored singleton signals, together with the exact-enumeration diagnostic on held-out MuSiQue questions, provides a clear and useful explanation of when simple marginal signals can fail.
-
The distinction between the supervision unit and the deployment unit is conceptually appealing. The proposal to learn per-item scores from set-level information preserves practical Top-B deployment, and the multi-hop QA results show that task-aligned supporting evidence can substantially improve downstream accuracy under the evaluated protocol.
Critical weaknesses
-
The central natural-language experiments do not cleanly isolate the effect of set-level supervision from the effect of using higher-quality human supporting-set labels. The comparison changes not only the supervision unit, but also label source, noise level, task alignment, and likely annotation cost: item labels are salience- or reader-derived, whereas set labels are dataset-provided gold supporting facts. The paper explicitly notes that label source and unit vary together in the large-pool experiment. Consequently, the results establish that high-quality, task-aligned supporting-evidence labels are effective, but do not yet establish that the reported gains are caused by set-level supervision itself. This is consequential because it is the paper's main empirical claim.
-
The path from the small-pool set oracle to scalable supervision for general long-term memory is not sufficiently established. The set-derived target p* is produced by exact search over subsets in enumerable pools, while the strongest natural-language results rely on dataset-provided gold supporting facts. The paper does not provide a practical, low-cost procedure for constructing comparable set-level labels when candidate pools are large and no human supporting-fact annotations exist, nor does it report label-generation cost or scaling behavior. Thus, the work currently reads more as a compelling high-quality-supervision setting than as a demonstrated general memory-supervision pipeline.
Questions for authors
1.The non-identification theorem is stated for selectors restricted to anchored add-one/delete-one observations. Which claims in the paper are intended to extend beyond this restricted information model to learned item-wise selectors that can read candidate text and model candidate interactions? Would an interaction-aware item-wise baseline trained under the same label source narrow the gap?
-
Can the authors provide, or point to, a fully controlled comparison in which the item-level and set-level objectives are derived from the same supervision source and have comparable annotation/computation budgets? In particular, how should readers interpret the reported 71% overlap between the gold set and the reader-defined optimal family when attributing gains to the supervision unit?
-
What are the inference calls, wall-clock cost, and scaling curve for constructing p* or an approximate set-level target as the candidate-pool size and budget increase? Is there evidence that an approximate label-generation procedure remains effective without gold supporting facts?
-
The main utility V(S) is defined by a fixed reader and the main natural-language task is multi-hop QA. How stable are optimal sets and selector gains across readers and in settings closer to persistent agent memory, such as long multi-turn histories with stale or conflicting information?
2 comments
The fixed-source comparisons carry the argument, and I'll take Section 3's scoping. But the effect only shows up at 100 candidates: nothing at 50, nothing in-distribution, and the budget isn't matched, 42 subset values against 14. So what survives is that set-level labels help top-B selection on retrieved pools, which is narrower than the title, and the unannotated long-memory case is still the open item. Backward elimination over 849 ScienceWorld steps is a diagnostic, not a pipeline.
Response to Reviewer v6eB
We thank the reviewer for the thoughtful review, and for finding the problem meaningful and the separation of supervision and deployment units appealing. We have uploaded a revised PDF, and the changes mentioned below refer to it.
We agree with the main concern. The gold-label results show that task-aligned set labels work well, but they do not show that the supervision unit causes the gain. The evidence for the unit comes from two comparisons in which both labels come from the same reader: Table 1 (oracle) and Table 4 (learned). The revised paper is organized around this distinction.
Q2. Same-source comparison, budgets, and the 71%
In Table 4, two identical selectors are trained on targets from one reader's exact values. The item target is the rank-sum of add-one and delete-one values, and the set target is p*. At 100 retrieved candidates, the set target gives 0.173 F1 versus 0.132 (Δ = +0.040 [0.021, 0.063]; per seed +0.043 / +0.041 / +0.037). At 50 candidates the gap is positive but not significant, and on small in-distribution pools the two selectors are indistinguishable (Appendix W). Both stay below dense top-B. In Table 1, where source and noise are both fixed, singleton fusion leaves 0.181 regret versus 0.006, and the best post hoc fusion still leaves 0.079 (Table 18).
The budgets are not equal, and Section 5 now says so. With six candidates and B ≤ 3, the set target uses 42 subset values, while the singleton target uses 2n+2 = 14. The 2n+2 anchored values are all the anchored information there is, and by the theorem they do not identify the optimum. A larger budget for item labels helps only if it is spent on intermediate-size subsets, which is the information the set target uses.
The 71% is not item overlap. Among the 204 sealed questions with at most three gold paragraphs, the gold set is itself reader-optimal at the same size in 71% of cases (median regret 0, mean 0.13). So gold sets are a related but different source, and we do not attribute the gold-label gains to the reader-defined target.
Q1. Scope of the theorem
Theorem 1 applies to any selector whose information about V is A(V), including any architecture trained only on add-one and delete-one labels. It does not claim that a model reading text cannot predict the optimal set, since text, priors, or structure can supply what the anchored values leave open (Remark in Section 3). The claim that goes beyond the restricted model is about supervision: an anchored label does not determine the optimum, so anything a model learns beyond it must come from elsewhere. We did not train an interaction-aware scorer on singleton labels, and we make no empirical claim about one. Section 3 now states this scope.
Q3. Cost and scaling
Exact p* needs Σ_{k≤B} C(n,k) reader calls per question: 42 at n=6 (B ≤ 3), 93 at n=8, and 1,351 at n=20 (B=3). Each call is one short greedy generation from a fixed reader, so cost grows in proportion to this count on any hardware. For this reason we report calls rather than machine-specific times. Exact p* is therefore an offline, small-pool procedure, and the paper has two routes beyond it. First, targets can be built on small pools and the trained scorer applied to 100 retrieved candidates (Table 4). Second, greedy backward elimination (Appendix C) is linear in pool size per pass and needs no evidence annotation. On 849 ScienceWorld steps, it keeps 1.8 of 8.3 observations and, by construction, preserves the next action on every step, while recency and BM25 at the same budget preserve it on 70% and 60% of steps (Appendix V). A validated low-cost pipeline for large unannotated memory is still open, and the conclusion now says so.
Q4. Readers and persistent memory
The ordering of selectors holds under three readers (Table 15; value / dense / BM25: 0.345 / 0.162 / 0.103 for Qwen3-8B, 0.438 / 0.232 / 0.153 for Tulu-3-8B, 0.450 / 0.245 / 0.153 for Gemma-2-9B). The interaction-order diagnostic shows the same pattern for all three (Figure 5, Table 19). We did not measure whether the optimal sets themselves agree across readers. ScienceWorld is our closest setting to persistent memory, but it is a replay diagnostic and does not test stale or conflicting histories. The conclusion now states both points.
We hope these answers address your concerns, and we are happy to clarify anything further.