acceptodds
Under review as a conference paper at ICLR 2027

The Unit of Memory Supervision Is the Set

Abstract

Memory selectors for language agents score each candidate item on its own, by relevance, salience, or recency. But a memory helps only through the set it forms, and a set’s value is rarely the sum of its items, so which items to keep is a question about the set. We show that item-level supervision cannot in general identify the optimal set. Two value functions can have identical add-one and delete-one effects for every fact yet have disjoint optimal budgeted sets, so no rule restricted to those add-one and delete-one signals can avoid set regret. On a sealed split of 250 MuSiQue questions with exact subset enumeration, a development-selected rank-sum of these singleton signals leaves 0.18 answer-F1 regret against the optimal set, while a set-derived item score leaves 0.006. Changing only the training label from salience to gold supporting sets, at matched architecture, data, and budget, raises exact match from 0.107 to 0.345. On 500 fresh MuSiQue questions with retrieved candidates, supporting-set supervision beats both add-one item supervision and dense retrieval. The supervision must observe value at the set level, but deployment need not: set-level value can be distilled into per-item scores, preserving item-wise selection at test time. When downstream value is interactive, the unit of supervision is the set.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, agent memory, memory selection, set-level supervision, retrieval-augmented generation, multi-hop question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_v6eBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that memory selection for language agents should be supervised at the set level rather than solely by per-item signals. It proves a non-identification result for selectors restricted to anchored add-one and delete-one signals, proposes a set-derived per-item target obtained from optimal budgeted sets, and retains efficient Top-B item-wise selection at deployment. The paper evaluates the idea through exact subset enumeration on sealed MuSiQue pools, controlled synthetic/planning settings, and multi-hop QA experiments with gold supporting-set supervision and larger retrieval pools.

Critical strengths
  1. The paper identifies a meaningful issue: evidence items can be complementary, so marginal item relevance need not recover the best budgeted evidence set. The formalization of anchored singleton signals, together with the exact-enumeration diagnostic on held-out MuSiQue questions, provides a clear and useful explanation of when simple marginal signals can fail.

  2. The distinction between the supervision unit and the deployment unit is conceptually appealing. The proposal to learn per-item scores from set-level information preserves practical Top-B deployment, and the multi-hop QA results show that task-aligned supporting evidence can substantially improve downstream accuracy under the evaluated protocol.

Critical weaknesses
  1. The central natural-language experiments do not cleanly isolate the effect of set-level supervision from the effect of using higher-quality human supporting-set labels. The comparison changes not only the supervision unit, but also label source, noise level, task alignment, and likely annotation cost: item labels are salience- or reader-derived, whereas set labels are dataset-provided gold supporting facts. The paper explicitly notes that label source and unit vary together in the large-pool experiment. Consequently, the results establish that high-quality, task-aligned supporting-evidence labels are effective, but do not yet establish that the reported gains are caused by set-level supervision itself. This is consequential because it is the paper's main empirical claim.

  2. The path from the small-pool set oracle to scalable supervision for general long-term memory is not sufficiently established. The set-derived target p* is produced by exact search over subsets in enumerable pools, while the strongest natural-language results rely on dataset-provided gold supporting facts. The paper does not provide a practical, low-cost procedure for constructing comparable set-level labels when candidate pools are large and no human supporting-fact annotations exist, nor does it report label-generation cost or scaling behavior. Thus, the work currently reads more as a compelling high-quality-supervision setting than as a demonstrated general memory-supervision pipeline.

Questions for authors

1.The non-identification theorem is stated for selectors restricted to anchored add-one/delete-one observations. Which claims in the paper are intended to extend beyond this restricted information model to learned item-wise selectors that can read candidate text and model candidate interactions? Would an interaction-aware item-wise baseline trained under the same label source narrow the gap?

  1. Can the authors provide, or point to, a fully controlled comparison in which the item-level and set-level objectives are derived from the same supervision source and have comparable annotation/computation budgets? In particular, how should readers interpret the reported 71% overlap between the gold set and the reader-defined optimal family when attributing gains to the supervision unit?

  2. What are the inference calls, wall-clock cost, and scaling curve for constructing p* or an approximate set-level target as the candidate-pool size and budget increase? Is there evidence that an approximate label-generation procedure remains effective without gold supporting facts?

  3. The main utility V(S) is defined by a fixed reader and the main natural-language task is multi-hop QA. How stable are optimal sets and selector gains across readers and in settings closer to persistent agent memory, such as long multi-turn histories with stale or conflicting information?

2 comments
AuthorsOpenReview ↗

Response to Reviewer v6eB

We thank the reviewer for the thoughtful review, and for finding the problem meaningful and the separation of supervision and deployment units appealing. We have uploaded a revised PDF, and the changes mentioned below refer to it.

We agree with the main concern. The gold-label results show that task-aligned set labels work well, but they do not show that the supervision unit causes the gain. The evidence for the unit comes from two comparisons in which both labels come from the same reader: Table 1 (oracle) and Table 4 (learned). The revised paper is organized around this distinction.

Q2. Same-source comparison, budgets, and the 71%

In Table 4, two identical selectors are trained on targets from one reader's exact values. The item target is the rank-sum of add-one and delete-one values, and the set target is p*. At 100 retrieved candidates, the set target gives 0.173 F1 versus 0.132 (Δ = +0.040 [0.021, 0.063]; per seed +0.043 / +0.041 / +0.037). At 50 candidates the gap is positive but not significant, and on small in-distribution pools the two selectors are indistinguishable (Appendix W). Both stay below dense top-B. In Table 1, where source and noise are both fixed, singleton fusion leaves 0.181 regret versus 0.006, and the best post hoc fusion still leaves 0.079 (Table 18).

The budgets are not equal, and Section 5 now says so. With six candidates and B ≤ 3, the set target uses 42 subset values, while the singleton target uses 2n+2 = 14. The 2n+2 anchored values are all the anchored information there is, and by the theorem they do not identify the optimum. A larger budget for item labels helps only if it is spent on intermediate-size subsets, which is the information the set target uses.

The 71% is not item overlap. Among the 204 sealed questions with at most three gold paragraphs, the gold set is itself reader-optimal at the same size in 71% of cases (median regret 0, mean 0.13). So gold sets are a related but different source, and we do not attribute the gold-label gains to the reader-defined target.

Q1. Scope of the theorem

Theorem 1 applies to any selector whose information about V is A(V), including any architecture trained only on add-one and delete-one labels. It does not claim that a model reading text cannot predict the optimal set, since text, priors, or structure can supply what the anchored values leave open (Remark in Section 3). The claim that goes beyond the restricted model is about supervision: an anchored label does not determine the optimum, so anything a model learns beyond it must come from elsewhere. We did not train an interaction-aware scorer on singleton labels, and we make no empirical claim about one. Section 3 now states this scope.

Q3. Cost and scaling

Exact p* needs Σ_{k≤B} C(n,k) reader calls per question: 42 at n=6 (B ≤ 3), 93 at n=8, and 1,351 at n=20 (B=3). Each call is one short greedy generation from a fixed reader, so cost grows in proportion to this count on any hardware. For this reason we report calls rather than machine-specific times. Exact p* is therefore an offline, small-pool procedure, and the paper has two routes beyond it. First, targets can be built on small pools and the trained scorer applied to 100 retrieved candidates (Table 4). Second, greedy backward elimination (Appendix C) is linear in pool size per pass and needs no evidence annotation. On 849 ScienceWorld steps, it keeps 1.8 of 8.3 observations and, by construction, preserves the next action on every step, while recency and BM25 at the same budget preserve it on 70% and 60% of steps (Appendix V). A validated low-cost pipeline for large unannotated memory is still open, and the conclusion now says so.

Q4. Readers and persistent memory

The ordering of selectors holds under three readers (Table 15; value / dense / BM25: 0.345 / 0.162 / 0.103 for Qwen3-8B, 0.438 / 0.232 / 0.153 for Tulu-3-8B, 0.450 / 0.245 / 0.153 for Gemma-2-9B). The interaction-order diagnostic shows the same pattern for all three (Figure 5, Table 19). We did not measure whether the optimal sets themselves agree across readers. ScienceWorld is our closest setting to persistent memory, but it is a replay diagnostic and does not test stale or conflicting histories. The conclusion now states both points.

We hope these answers address your concerns, and we are happy to clarify anything further.

The fixed-source comparisons carry the argument, and I'll take Section 3's scoping. But the effect only shows up at 100 candidates: nothing at 50, nothing in-distribution, and the budget isn't matched, 42 subset values against 14. So what survives is that set-level labels help top-B selection on retrieved pools, which is narrower than the title, and the unannotated long-memory case is still the open item. Backward elimination over 849 ScienceWorld steps is a diagnostic, not a pipeline.

Reviewer_iVg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory selection for language agents under a limited budget. Its central argument is that multiple pieces of information may need to work together to support a correct answer, so supervision should account for the value of the entire set. The authors theoretically show that the effects of adding or removing individual items do not always identify the optimal subset. They then construct supervision targets either by searching for optimal subsets or by using human-annotated supporting evidence to train a selector. At inference time, the selector either directly outputs the retained items or scores candidates individually and selects the top items, without exhaustive subset search. The paper evaluates this approach on multi-hop question answering, synthetic decision tasks, and out-of-distribution tests.

Critical strengths
  1. The experiments cover a broad range of settings, including multi-hop question answering, synthetic decision tasks, OOD tests, and cross-dataset transfer, with comparisons under matched selection budgets.

  2. The paper reports negative results and limitations alongside positive findings, including the lack of a significant advantage in some learning experiments and performance below direct retrieval when transferring to larger candidate pools.

Critical weaknesses
  1. The proposed approach closely resembles a standard reranker: it scores candidate passages and selects the highest-scoring ones. The main difference lies in the training labels. It remains unclear whether the improvements come from the proposed set-level supervision or simply from using higher-quality, human-annotated supporting evidence. A comparison with a standard reranker trained on the same annotations is needed to establish what additional benefit the proposed method provides.

  2. Prior work has already used feedback from downstream answering models to train retrievers or rerankers, including REPLUG and RA-DIT for retriever training, as well as SETR and OptiSet for reranking or set selection. The paper emphasizes evaluating whether multiple chunks are useful together, rather than evaluating each chunk in isolation. However, its strongest empirical results rely on human-annotated supporting evidence. Consequently, the paper does not clearly establish its specific novelty relative to these existing approaches.

  3. The paper’s organization and presentation of results lack clarity. Generating labels by enumerating optimal subsets, directly using human-annotated supporting evidence, and using a solver to find minimum sufficient subsets are three distinct approaches. The paper should explain more clearly how they relate to one another and which results correspond to each approach.

Questions for authors
  1. When the candidate pool is large and human-annotated supporting evidence is unavailable, how would the training labels be generated in practice?
2 comments
AuthorsOpenReview ↗

Response to Reviewer iVg4

We thank the reviewer for the detailed review, and for noting the breadth of the experiments and our reporting of negative results. We have uploaded a revised PDF, and the changes mentioned below refer to it.

W1. Set supervision or better labels?

We agree that the deployed selector is a standard reranker: it scores each candidate and keeps the top B. This is intentional. The paper does not propose a new architecture. It asks what the training label must observe so that top-B selection recovers the right set.

A standard reranker trained on gold supporting-set membership is, up to architecture, our Section 6 selector. That comparison would change the architecture and keep the label fixed. To separate set supervision from better labels, we need the opposite: the same architecture, and labels from the same source that differ only in unit. The paper has two such comparisons.

  • Oracle (Table 1). The reader's exact add-one and delete-one values leave 0.181 regret against the optimal set, and even the best combination chosen per question after the fact leaves 0.079 (Table 18). The set-derived score from the same reader leaves 0.006. These singleton values are exact rather than estimated, so label noise cannot explain the gap.
  • Learned (Table 4). Identical scorers trained on the two targets from one reader reach 0.132 versus 0.173 F1 on 100 retrieved candidates (Δ = +0.040 [0.021, 0.063], positive in all three seeds). On small in-distribution pools they are indistinguishable (Appendix W), and both stay below dense top-B.

We agree that source and unit are confounded in the gold-label results, and the revision now says so in the abstract, Section 6, and Section 8. The MS-MARCO rerankers in Table 2 were not trained on the same annotations, and we do not present them as that baseline.

W2. Novelty relative to prior work

Thank you for these references. REPLUG and RA-DIT are now cited, and SETR and OptiSet are discussed in the main text. REPLUG and RA-DIT train retrievers on the LM's answer likelihood given each retrieved document separately. This is an add-one signal (the value of one document alone), so these methods fall in the class that Theorem 1 covers. They work well when documents contribute independently. Our result shows when they cannot identify the right set, namely when budget-relevant value lies in interactions, and Table 1 measures this cost on multi-hop questions. SETR and OptiSet move to set-aware selection, and OptiSet labels sets by their utility. Our theorem explains why this set-level observation is needed. Our results add that once value is observed at the set level, the deployed selector does not need to be set-aware.

We do not claim downstream-feedback training or set selection as new, nor an improvement over these methods. Our contributions are the non-identification result, with a regret bound that holds even when every item's signals are distinct; its measurement on held-out MuSiQue questions by exact enumeration; and the separation of supervision unit from deployment unit.

W3. Three label routes

We agree. Section 5 now has a short paragraph that describes the three routes and the results that use each one. The table below summarizes it.

RouteLabels fromResultsRole
Reader-derived p*exact enumeration of reader valueTable 1, Table 4, Appendix Wsame-source test of the unit
Gold supporting setsdataset annotationSection 6, Section 8 (second experiment)practical gains; source differs
Solver-derived setsexact solver on controlled tasksSection 7mechanism when value and salience diverge

Q. Large pools without human evidence

Exact p* costs Σ_{k≤B} C(n,k) reader calls (42 at n=6, B ≤ 3; 1,351 at n=20, B=3), so it is a small-pool, offline procedure. The paper offers two ways to go further. Targets can be built on small pools and the trained scorer applied to large ones (Table 4). Alternatively, greedy backward elimination (Appendix C), which is linear in pool size per pass, needs only a downstream value signal and no evidence annotation. On 849 ScienceWorld agent steps, it keeps 1.8 of 8.3 observations and, by construction, preserves the next action on every step, while recency and BM25 at the same budget preserve it on 70% and 60% of steps (Appendix V). A validated low-cost pipeline for large unannotated memory is still open, and the conclusion now says so.

We hope these answers address your concerns, and we are happy to clarify anything further.

Thank you for your response. I will maintain my current rating.

Reviewer_g1P9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates memory selection for language agents, arguing that item-level supervision is insufficient to identify optimal memory sets when facts exhibit interactions. It establishes a theoretical non-identifiability result and proposes distilling set-level supervision into item-wise scores for efficient inference. Experiments on multi-hop QA and controlled decision-making tasks demonstrate the potential benefits of decision-sufficient memory selection.

Critical strengths
  1. Transparency about AI usage and negative results: The authors are commendably transparent about their use of generative AI in preparing the paper, explicitly disclosing assistance with coding, language editing, and internal consistency checks. They also honestly report negative experimental results, including cases where set-derived supervision fails to show significant improvements.

  2. Solid theoretical analysis: The paper provides a rigorous non-identifiability theorem showing that even exact item-level add-one and leave-one-out signals cannot always identify the optimal memory set. The theoretical construction and regret lower bound offer a principled justification for considering set-level information in memory selection.

Critical weaknesses
  1. Several parts of the appendix appear to retain internal implementation terminology rather than providing self-contained scientific descriptions. For example, on page 17 (Line 901), the authors state that confidence intervals for single-run LLM baselines are available in “the generated artifact files”, without clearly specifying where these files can be accessed or how reviewers can verify the reported statistics. Similarly, page 27 (Lines 1433–1444) refers to “synthetic v2-hard suite”, “written ρ in our code”, and “direct action suite”, introducing internal experiment names and code-specific notation without sufficiently explaining their relationship to the experimental setup. Taken together, these artifacts give the impression that the final manuscript, potentially assembled with substantial AI assistance, may not have been carefully read and reviewed end-to-end by the authors themselves before submission.

  2. The most directly controlled comparison in Appendix W shows no significant improvement from set-derived supervision over singleton supervision (Δ = 0.001, 95% CI [-0.031, 0.034]). This raises concerns about whether the theoretical advantage of set-level information translates into practical learnability.

  3. The strongest positive results (Sections 6 and 8) compare gold supporting-set annotations against BM25 salience or reader-derived singleton labels. Since both the supervision source and information content differ, it is unclear whether the improvements arise from set-level supervision itself or simply from stronger supervision labels.

Questions for authors

Q1. Can the authors disentangle the supervision unit from the label source?

Table 2 compares gold supporting-set labels (0.345 EM) with BM25-derived salience labels (0.107 EM), confounding supervision granularity with label quality. Could the authors provide a controlled comparison using the same reader-derived V(S)V(S) to construct both item-level add-one labels and set-derived p∗p^* labels under identical training conditions? This would clarify whether the gains arise from set-level supervision itself or stronger supervision labels.

Q2. Does the theoretical advantage of set-level supervision translate into practical learnability?

Table 1 shows near-zero regret (0.006) using oracle-derived p∗p^*, yet Appendix W reports no significant improvement when learning these targets over singleton supervision (Δ = 0.001, 95% CI [-0.031, 0.034]), even on high-interaction questions. Given that identifiability does not imply learnability, what empirical evidence supports the claim that set-derived supervision provides a generalizable advantage rather than merely an oracle-level advantage?

1 comment
AuthorsOpenReview ↗

Response to Reviewer g1P9

We thank the reviewer for the careful review, and for recognizing the theoretical analysis and our reporting of negative results. We have uploaded a revised PDF, and the changes mentioned below refer to it. We answer Q1 and Q2 first, since they also cover W2 and W3, and then the appendix wording (W1).

Q1. Same reader-derived V(S) for both labels

Yes. Table 4 is this comparison, and we should have made it clearer in the paper. Two identical selectors (Qwen3-1.7B with LoRA, same data, three seeds) are trained on targets from one deterministic reader (Qwen3-8B, greedy). The item-level target is the rank-sum of that reader's exact add-one and delete-one values, which is a stronger baseline than add-one alone. The set-level target is that reader's p*. Gold membership is not used as a label. Results on 430 held-out questions with 100 globally retrieved candidates and B=3:

F1
Item-level target0.132
Set-level target0.173
Δ, all questions+0.040 [0.021, 0.063]
Δ, questions with all gold retrieved+0.061 [0.030, 0.093]
Δ per seed+0.043 / +0.041 / +0.037

At 50 candidates the difference is +0.016 [−0.004, 0.036], positive in all seeds but not significant. Both selectors are below dense top-B (0.223), so we read this as an effect of the supervision unit, not a retrieval improvement. In the submission, the item arm was called "singleton-derived" without its formula. Appendix W now gives both targets and the full training recipe.

The oracle result (Table 1) shows the same picture without learning. There, source and noise are both fixed: the reader's exact singleton signals are compared with its exact optimal sets. Singleton fusion leaves 0.181 regret, and even the best fusion chosen per question after the fact leaves 0.079 (Table 18). The set-derived score leaves 0.006. These singleton values are exact rather than estimated, so label noise does not explain the gap.

Q2. Learnability beyond the oracle

We agree that identifiability does not imply learnability. The small-pool null (Δ = 0.001 [−0.031, 0.034]) stands, and Table 4 does not overturn it. What Table 4 adds is that the same two trained selectors do separate, consistently across seeds, once candidates come from global retrieval. The revised Section 8 states the learned evidence in these terms. We also changed the abstract. It now says that set-level value can be represented by per-item scores (the oracle result) rather than distilled, and it describes the gold-label gains as results with a different label source.

W1. Appendix wording

Thank you for pointing to the exact lines. These phrases came from the two appendix paragraphs that refer to the supplementary material, where we used its file names. dsla_bench_v2_hard is the folder with the Synthetic suite's records, rho is the key for β in the result files, and "the generated artifact files" are the per-example records of the single-run LLM baselines (outputs/*/llm_baselines_*.json). We agree that this cannot be understood from the PDF alone. The revision uses "the Synthetic suite", writes β only, and replaces "direct action suite" with "on the same Synthetic tasks". Table 8 now reports an interval for every regret: the seed range for trained selectors, and a bootstrap interval from the per-example records for single-run baselines. File and key names are explained once, in the reproducibility appendix.

The key numbers can also be checked directly from the supplement. Two scripts, code/verify_exact.py and code/verify_results.py, read the stored per-example outputs and recompute Table 1 (0.181 vs. 0.006), Table 4 (0.132 / 0.173 / 0.223), and the fresh 500-question result with its bootstrap intervals. They do not run any model, so each finishes in about a second on a CPU.

We hope these answers address your concerns, and we are happy to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.