acceptodds
Under review as a conference paper at ICLR 2027

MemSteer: Turning User Histories into Supervision for Agent Memory Optimization

Abstract

An effective agent memory system should adapt to a user's history and task requirements. Human-annotated supervision can guide this adaptation by providing feedback for evaluating and improving memory designs. Yet suitable labels are rarely available for new user histories, and manually constructing QA pairs and supporting evidence is costly. We introduce MemSteer, which turns unlabeled user histories into question–answer (QA) supervision with supporting evidence for memory-design optimization. MemSteer constructs an evidence-grounded answer plan that specifies how facts across related records jointly support a reference answer. The plan guides question generation, while source-grounded auditing and blind answer recovery check the resulting annotations.To study the utility of this supervision, we develop MemSteer-Opt, which combines QA-based design evaluation with evidence-guided failure diagnosis to revise memory store, retrieval, and context construction without updating model parameters. Experiments on ATM-Bench and RHELM show that this supervision is competitive with official-QA-guided optimization. Relative to Base-BM25, MemSteer-Opt improves ATM-Bench Question Score by 13.4 points and RHELM accuracy by 4.1 and 6.6 points without and with external data, respectively. The supervision also benefits two existing optimizers on ATM-Bench, while new-user experiments demonstrate continued design optimization using supervision generated from the target histories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Memory Systems, Synthetic Data Generation, Iterative Optimization, Personalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qXPFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MemSteer, which automatically generates evidence-grounded QA data from unlabeled user histories to guide memory system optimization. The authors also propose MemSteer-Opt, which uses QA performance and supporting evidence to diagnose errors in storage, retrieval, and answer generation, then iteratively refines the memory system design without updating model parameters. Experiments on ATM-Bench and RHELM show that the generated supervision improves memory system performance, benefits other memory optimization frameworks, and supports cross-dataset transfer and adaptation to new users.

Critical strengths
  1. The paper addresses a problem of practical value. Personal memory system optimization often lacks human supervision, and manually annotating data for each user is prohibitively expensive. By automatically constructing evidence-grounded QA data from user histories, the paper provides a scalable source of supervision with promising practical applications.

  2. The paper introduces a training-free memory optimization method. MemSteer-Opt combines QA-based evaluation, evidence-guided error diagnosis, and system design revisions to iteratively improve memory storage, retrieval, and context construction.

  3. The experimental evaluation is comprehensive. The paper evaluates the method on two benchmarks and systematically examines the effectiveness and applicability of the generated supervision through ablations of data generation and optimizer components, integration with other optimization frameworks, cross-dataset transfer, and adaptation to new users.

Critical weaknesses
  1. The reliability of synthetic supervision needs further independent verification. Answer planning, question generation, auditing, and blind answer recovery all heavily rely on LLMs. Consistency across these steps does not guarantee annotation accuracy, as shared errors or biases may go undetected. Human verification and robustness experiments that vary the generator and verifier are needed to further establish the reliability of the supervision.

  2. Error attribution based on designated evidence may be unreliable. MemSteer-Opt diagnoses failures using the evidence selected during QA generation, but this evidence may not be the only valid basis for answering the question. The system may retrieve other similar records containing the same facts or retain a summary sufficient to answer the question. Therefore, the absence of a designated piece of evidence does not necessarily mean that essential information is missing. The paper should clarify how it identifies equivalent evidence and verify that the diagnostic mechanism does not misclassify these cases as storage or retrieval errors.

  3. The benefits of optimization from a stronger starting point remain unclear. The main experiments start with a basic BM25 system and progressively add components such as hybrid retrieval, reranking, and multi-query generation. It remains unclear whether MemSteer-Opt would still deliver substantial gains when starting from a stronger, manually configured RAG system under the same computational budget.

Questions for authors

Please refer to the weaknesses section. Moreover, I have an additional question:

  1. How does the author make sure the distribution of synthetic data matches real-world user needs?

I'll raise my score if the author can address these concerns.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qXPF (Part-1)

Thank you for the thoughtful review and constructive suggestions for improving this work. We address each of your questions/comments below.

W1. The reliability of synthetic supervision needs further independent verification. Answer planning, question generation, auditing, and blind answer recovery all heavily rely on LLMs. Consistency across these steps does not guarantee annotation accuracy, as shared errors or biases may go undetected. Human verification and robustness experiments that vary the generator and verifier are needed to further establish the reliability of the supervision.

Response to W1: Thank you for this constructive suggestion. To assess the reliability of our supervision beyond the original LLM pipeline, we conducted a blinded human audit of the original generated supervision and tested additional generator–verifier combinations.

(i) Human audit of the original generated supervision. In our paper, when DeepSeek-V4-Flash was used as generator, gpt-5-mini was used as the verifier. We sampled 100 accepted QA-evidence pairs from the supervision used to optimize the ATM-Bench memory design (these pairs were generated using DeepSeek-V4-Flash (G1) and verified using gpt-5-mini (V1)). A human annotator assessed each pair against the original records. Table R14 shows that among the 100 randomly sampled QA-evidence pairs, the human annotator judged 91% of reference answers correct and complete and 90% of evidence sets sufficient.

Table R14: Human assessment of 100 sampled ATM-Bench generated QA-evidence pairs. Generator: DeepSeek-V4-Flash; Verifier: gpt-5-mini.

Audit criterionHuman pass (%)
Question validity99.0
Reference-answer correctness and completeness91.0
Evidence sufficiency90.0

(ii) Robustness across generators and verifiers. DeepSeek-V4-Flash (G1) and Kimi-K2.6 (G2) each generated 200 candidate QA-evidence pairs from ATM-Bench materials. For each generator, gpt-5-mini (V1) and gemma-4-31B (V2) assessed the same 200 candidates under matched criteria, yielding four generator–verifier configurations. As a fifth configuration, we used DeepSeek-V4-Flash (V3) to verify candidates generated by DeepSeek-V4-Flash (G1) only. For each configuration, the same blinded human annotator assessed 50 randomly sampled accepted pairs for question validity, reference-answer correctness and completeness, and evidence sufficiency.

As shown in Table R15, human-audited validity ranges from 94.0% to 100.0% across all five configurations. These results provide evidence that the verification pipeline produces high-quality supervision across the tested generator–verifier configurations, reducing concern that the results depend on a single model pairing.

Table R15: Effects of generator and verifier choices on ATM-Bench QA-evidence quality, evaluated on 50 samples each. V1 and V2 assessed the same candidates for each generator; V3 was evaluated only with G1. Human-valid pairs satisfy question validity, answer correctness and completeness, and evidence sufficiency; the 95% CI applies to the human-valid rate among assessable accepted pairs.

GeneratorVerifierAccepted / candidatesHuman-valid / assessable95% CI
DeepSeek-V4-Flash (G1)gpt-5-mini (V1)91/200 (45.5%)47/50 (94.0%)83.8%–97.9%
DeepSeek-V4-Flash (G1)gemma-4-31B (V2)103/200 (51.5%)50/50 (100.0%)92.9%–100.0%
DeepSeek-V4-Flash (G1)DeepSeek-V4-Flash (V3)106/200 (53.0%)48/50 (96.0%)86.5%–98.9%
Kimi-K2.6 (G2)gpt-5-mini (V1)133/200 (66.5%)47/50 (94.0%)83.8%–97.9%
Kimi-K2.6 (G2)gemma-4-31B (V2)124/200 (62.0%)48/50 (96.0%)86.5%–98.9%
AuthorsOpenReview ↗

Response to Reviewer qXPF (Part-2)

W2. Error attribution based on designated evidence may be unreliable. MemSteer-Opt diagnoses failures using the evidence selected during QA generation, but this evidence may not be the only valid basis for answering the question. The system may retrieve other similar records containing the same facts or retain a summary sufficient to answer the question. Therefore, the absence of a designated piece of evidence does not necessarily mean that essential information is missing. The paper should clarify how it identifies equivalent evidence and verify that the diagnostic mechanism does not misclassify these cases as storage or retrieval errors.

Response to W2: Thank you for the constructive suggestion. We have clarified the attribution process in the revision. To reduce the latency and cost of diagnosing each incorrect response (in order to identify the failure type), our error attribution uses lightweight, programmatic evidence-coverage and source-recovery checks rather than an LLM-based semantic-equivalence check. We then sample a small set of failures from each observed failure category and provide the design proposer with the question, reference answer, selected evidence, retrieval traces, and answer-context excerpts for each case. This allows the proposer to consider information beyond the initial coarse labels when proposing design revisions.

Other records. Yes, the absence of a designated piece of evidence from the answer context does not necessarily mean that the required information is unavailable: another retrieved record may contain the same facts. However, if the designated record is recoverable from memory but not retrieved, this is still a missed retrieval opportunity for that record and provides a signal for improving retrieval of the designated evidence.

Summaries. Each summary retains its source IDs in our schema. The attribution check matches these IDs against the selected evidence records, allowing a retrieved summary to be associated with its source evidence.

Empirical usefulness of the coarse labels. Although these labels may be imperfect when equivalent information is present in other records, Figure 3 in our manuscript shows that they still provide useful optimization signals: adding failure attribution to randomly sampled failures with gold evidence (Random-30 + Gold evidence → Random-30 + Diagnosis) improves ATM-Bench QS from 58.4 to 62.7 and R@10 from 67.6 to 83.7.

AuthorsOpenReview ↗

Response to Reviewer qXPF (Part-3)

W3. The benefits of optimization from a stronger starting point remain unclear. The main experiments start with a basic BM25 system and progressively add components such as hybrid retrieval, reranking, and multi-query generation. It remains unclear whether MemSteer-Opt would still deliver substantial gains when starting from a stronger, manually configured RAG system under the same computational budget.

Response to W3: Thank you for the suggestion. We evaluated whether MemSteer-Opt remains effective from strong starting designs, using the same optimization budget of 20 candidate rounds as in the Base-BM25 experiments.

ATM-Bench: optimization beyond a strong retrieval pipeline. Our fixed ATM-Bench-evolved-retrieval baseline already includes BGE-M3 hybrid retrieval (dense retrieval combined with BM25), reranking, HyDE-4, and a 25-record reader. Starting from this pipeline, MemSteer-Opt improves QS from 61.2 to 70.9 on the full set (+9.7 QS points) and from 24.7 to 30.5 on Hard (+5.8 QS points) (Table R16, single run). Thus, the gains are not limited to upgrading a basic BM25 system with these retrieval components.

RHELM: continued optimization of an already evolved design. We also start from ATM-Bench-evolved (frozen), the full design optimized on ATM-Bench and transferred to RHELM without target-domain optimization. Continued optimization improves accuracy by 4.2 percentage points in both settings, from 26.4% to 30.6% without external data and from 38.6% to 42.8% with external data (single run; Table R17). Across three runs, the resulting accuracy has a standard deviation of 0.3 percentage points in both settings.

Together, these results support the value of continued memory-design optimization even when the starting system already incorporates strong retrieval components or a previously optimized design.

Table R16: ATM-Bench performance from different starting designs. Question Score (QS) and evidence recall at 10 (R@10) are reported in percent for the full (1,013 questions) and Hard (31 questions) sets. Each group compares a starting design with its optimized counterpart (under the same computational budget of 20 candidate rounds); mean ± std summarizes 3 repeated runs.

MethodFull QSFull R@10Hard QSHard R@10
Starting point: Base-BM25
Base-BM2554.961.78.210.0
MemSteer (Base-BM25 start; single run)68.384.129.143.6
MemSteer (Base-BM25 start; 3 runs, mean ± std)68.7 ± 0.883.7 ± 0.328.1 ± 1.843.2 ± 0.4
Starting point: ATM-Bench-evolved-retrieval
ATM-Bench-evolved-retrieval61.283.824.743.6
Ragas+check*(ATM-Bench-evolved-retrieval start)67.881.524.241.0
MemSteer* (ATM-Bench-evolved-retrieval start)70.983.430.542.5

Table R17: RHELM accuracy from different starting designs. Accuracy (%) is reported for conversation records alone (without external data) and with emails and documents added (with external data). Each group compares a starting design with its optimized counterpart; mean ± std summarizes 3 repeated optimization runs.

MethodWithout external dataWith external data
Starting point: Base-BM25
Base-BM2522.333.3
MemSteer (Base-BM25 start; single run)26.439.9
MemSteer (Base-BM25 start; 3 runs, mean ± std)27.0 ± 0.840.2 ± 1.3
Starting point: ATM-Bench-evolved (frozen)
ATM-Bench-evolved (frozen)26.438.6
ATM-Bench-evolved + MemSteer (single run)30.642.8
ATM-Bench-evolved + MemSteer (3 runs, mean ± std)30.3 ± 0.342.8 ± 0.3
Reviewer_QzWsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MemSteer, which generates question–answer pairs with supporting evidence from unlabeled user histories. These pairs are then used as supervision for optimizing agent memory designs. Related records are sampled from a graph built over the user’s history. An LLM writes an answer plan that cites the supporting facts, and a separate LLM call writes the question. Each pair is checked by an LLM audit and a blind answer-recovery step, and about 50% are kept. To test the supervision, the authors build MemSteer-Opt. It runs a memory design on the generated questions and uses the evidence to label each failure as a storage, retrieval, or answering failure. An LLM then proposes revisions to storage, retrieval, and context construction, and a revision is kept only if validation accuracy improves. Starting from a BM25 baseline, MemSteer-Opt improves ATM-Bench Question Score by 13.4 points and RHELM accuracy by 4.1 and 6.6 points without and with external data. The results are close to optimizing with official benchmark QA. The generated supervision also improves two existing optimizers, M* and MemEvolve. Further experiments cover design transfer from ATM-Bench to RHELM and continued optimization on two held-out RHELM users.

Critical strengths
  1. Generated supervision matches official QA without touching evaluation data: Across three optimizers and two benchmarks, designs optimized on MemSteer’s generated QA perform about as well as designs optimized on official benchmark QA. This holds even though the Official-QA condition trains and validates on items that remain in its test set. If it holds up, memory systems could be tuned on a new user’s history without anyone writing questions for it.
Critical weaknesses
  1. It is unclear how much the generated supervision actually matters: The accepted design in Figure 5 is a sequence of standard retrieval upgrades: dense retrieval, hybrid retrieval with reranking, HyDE, question decomposition, and a larger reading budget. These would likely help on almost any QA benchmark. Table 4 supports this. Optimizing with naive random-record QA plus the paper’s checks already reaches 65.4 QS, and Ragas-generated QA reaches 63.5, against 68.3 for MemSteer. Most of the 13.4-point gain over Base-BM25 is therefore available with much weaker supervision. The paper includes no fixed strong RAG baseline built from these same components without optimization, which would show how much of the gain needs the optimization loop at all. Without that baseline, and with no variance reported for gaps of 1-3 points, the case that MemSteer’s supervision is what drives the improvement is weak.

  2. The per-user adaptation motivation is not supported by the results: The paper is framed around one memory design not fitting all users, but the evidence points the other way. ATM-Bench is a single person’s history, and RHELM uses synthetic personas. In Table 9, the shared 10-user design scores 41.8 on David R. Ellis, higher than personal continuation (39.2 in Table 3), and 42.5 on Martin Keller, close to personal continuation (44.3). Per-user optimization is tested on only these two users, each with 158-228 evaluation questions and a single run. The data are more consistent with one good general design than with a need for per-user designs.

Questions for authors

Clarifications

  1. How does a fixed pipeline built from the final accepted components (BGE-M3 hybrid retrieval, reranking, HyDE, 25-record reader), with no optimization, score on ATM-Bench and RHELM?
  2. What is the run-to-run variance of MemSteer-Opt? On Hard, the 1.3-point margin over Official-QA is smaller than one question out of 31.
  3. Was the 10-user shared design in Table 9 optimized with official QA or generated QA, and did it include Martin Keller and David R. Ellis? If they were included, how does personal continuation compare against a shared design that excludes them?

Suggestion

  1. A human audit of a sample of the generated QA would establish how often the LLM-only verification lets through wrong reference answers. DeepSeek-V4-Flash is used for generation, answering, judging, and proposing revisions in the same pipeline.
5 comments
AuthorsOpenReview ↗

Response to Reviewer QzWs(Part-1)

Thank you for the thoughtful review and constructive suggestions for improving this work. We address each of your questions/comments below.

W1. It is unclear how much the generated supervision actually matters: The accepted design in Figure 5 is a sequence of standard retrieval upgrades: dense retrieval, hybrid retrieval with reranking, HyDE, question decomposition, and a larger reading budget. These would likely help on almost any QA benchmark. Table 4 supports this. Optimizing with naive random-record QA plus the paper’s checks already reaches 65.4 QS, and Ragas-generated QA reaches 63.5, against 68.3 for MemSteer. Most of the 13.4-point gain over Base-BM25 is therefore available with much weaker supervision. The paper includes no fixed strong RAG baseline built from these same components without optimization, which would show how much of the gain needs the optimization loop at all. Without that baseline, and with no variance reported for gaps of 1-3 points, the case that MemSteer’s supervision is what drives the improvement is weak.

Q1. How does a fixed pipeline built from the final accepted components (BGE-M3 hybrid retrieval, reranking, HyDE, 25-record reader), with no optimization, score on ATM-Bench and RHELM?

Response to W1 and Q1: Thank you for the constructive suggestions. We evaluate fixed strong designs and compare supervision sources from the same strong starting point.

Fixed designs without further optimization. On ATM-Bench, (i) the fixed ATM-Bench-evolved-retrieval pipeline uses BGE-M3 hybrid retrieval (dense retrieval combined with BM25), reranking, HyDE-4, and a 25-record reader. It outperforms Base-BM25 by 6.3 QS points on the full set (54.9 → 61.2) and 16.5 QS points on Hard (8.2 → 24.7) (Table R10); (ii) compared with the fixed strong retrieval baseline, the full ATM-Bench-evolved design scores 7.1 QS points higher on the full set (61.2 → 68.3) and 4.4 QS points higher on Hard (24.7 → 29.1). Note that during optimization, MemSteer-Opt automatically introduced additional reader-side changes into the full design, including evidence-table reading and targeted evidence-selection guidance.

On RHELM, applying the ATM-Bench evolved full design (i.e., ATM-Bench-evolved (frozen)) without RHELM optimization achieves 26.4% / 38.6% accuracy without/with external data, compared with 22.3% / 33.3% for Base-BM25 (Table R11). Thus, strong reusable components already provide substantial gains.

Additional gains from optimization. Starting from ATM-Bench-evolved-retrieval, optimization with MemSteer-generated supervision improves ATM-Bench QS by 9.7 QS points on the full set (61.2 → 70.9) and 5.8 QS points on Hard (24.7 → 30.5), under the same budget of 20 candidate rounds (Table R10, single run).

On RHELM, continued optimization from the ATM-Bench-evolved (frozen) design improves accuracy by 4.2 percentage points in both settings, from 26.4% to 30.6% without external data and from 38.6% to 42.8% with external data (single run; Table R11). Across three runs, the resulting accuracy has a standard deviation of 0.3 percentage points in both settings. Beyond retrieval tuning, the evolved RHELM design introduces calendar-grouped day-level memory alongside individual records and, when external data are available, cross-source context curation to balance conversations, emails, and documents in the answer context.

AuthorsOpenReview ↗

Response to Reviewer QzWs (Part-2)

Contribution of the supervision source. To assess whether the supervision matters beyond the choice of starting components, we also optimize the same ATM-Bench-evolved-retrieval starting design using Ragas-QA+checks under the same optimization budget, which is denoted as Ragas+check*(ATM-Bench-evolved-retrieval start). From the same strong starting design and under the same optimization budget, MemSteer-generated supervision scored 3.1 QS points higher on the full set (70.9 vs. 67.8) and 6.3 QS points higher on Hard (30.5 vs. 24.2) than Ragas-QA+checks (Table R10). These results provide additional evidence that the supervision source affects the quality of the optimized design.

More broadly, in this work, we investigate an underexplored question: Can generated supervision guide memory-design optimization? MemSteer offers a practical approach: it generates evidence-grounded QA from unlabeled user histories and uses it to improve memory designs without requiring manually annotated QA. The observed differences between supervision sources also suggest that better QA generation strategies may further improve downstream performance.

Table R10: ATM-Bench performance from different starting designs. Question Score (QS) and evidence recall at 10 (R@10) are reported in percent for the full (1,013 questions) and Hard (31 questions) sets. Each group compares a starting design with its optimized counterpart (under the same computational budget of 20 candidate rounds); mean ± std summarizes 3 repeated runs.

MethodFull QSFull R@10Hard QSHard R@10
Starting point: Base-BM25
Base-BM2554.961.78.210.0
MemSteer (Base-BM25 start; single run)68.384.129.143.6
MemSteer (Base-BM25 start; 3 runs, mean ± std)68.7 ± 0.883.7 ± 0.328.1 ± 1.843.2 ± 0.4
Starting point: ATM-Bench-evolved-retrieval
ATM-Bench-evolved-retrieval61.283.824.743.6
Ragas+check*(ATM-Bench-evolved-retrieval start)67.881.524.241.0
MemSteer* (ATM-Bench-evolved-retrieval start)70.983.430.542.5

Table R11: RHELM performance from different starting designs. Accuracy (%) is reported for conversation records alone (without external data) and with emails and documents added (with external data). Each group compares a starting design with its optimized counterpart; mean ± std summarizes 3 repeated optimization runs.

MethodWithout external dataWith external data
Starting point: Base-BM25
Base-BM2522.333.3
MemSteer (Base-BM25 start; single run)26.439.9
MemSteer (Base-BM25 start; 3 runs, mean ± std)27.0 ± 0.840.2 ± 1.3
Starting point: ATM-Bench-evolved (frozen)
ATM-Bench-evolved (frozen)26.438.6
ATM-Bench-evolved + MemSteer (single run)30.642.8
ATM-Bench-evolved + MemSteer (3 runs, mean ± std)30.3 ± 0.342.8 ± 0.3
AuthorsOpenReview ↗

Response to Reviewer QzWs (Part-3)

Q2. What is the run-to-run variance of MemSteer-Opt? On Hard, the 1.3-point margin over Official-QA is smaller than one question out of 31.

Response to Q2: Thank you for highlighting the uncertainty associated with the small Hard subset. We repeated MemSteer-Opt three times from Base-BM25. Table R10 reports QS of 68.7 ± 0.8 on the full ATM-Bench set and 28.1 ± 1.8 on Hard (mean ± std).

Regarding the 31-question Hard subset, QS is an average of per-question scores that allow partial credit, rather than binary accuracy: open-ended answers receive judge-assigned scores, and list answers are scored using Jaccard similarity. Consequently, QS differences need not be multiples of 100/31 ≈ 3.23 QS points.

Our goal is to establish whether generated supervision can guide memory-design optimization, with Official-QA serving as a human-annotated reference rather than a baseline we aim to outperform. The three-run Hard result (28.1 ± 1.8, versus 8.2 for Base-BM25 and 27.9 for Official-QA) supports substantial improvement from generated supervision and performance numerically close to the Official-QA reference. We therefore treat the originally reported 1.3-QS-point margin on Hard as descriptive, not as evidence of superiority over Official-QA.


Q3. Was the 10-user shared design in Table 9 optimized with official QA or generated QA, and did it include Martin Keller and David R. Ellis? If they were included, how does personal continuation compare against a shared design that excludes them?

Response to Q3: Thank you for asking this clarifying question, and we apologize for the confusion in Table 9.

Supervision and user pool in Table 9. In Table 9, the 10-user shared design was optimized using Official-QA across all 10 users—including the target users David R. Ellis and Martin Keller themselves. In contrast, personal continuation (Table 3) started from a target-excluded 6-user shared design and adapted using automatically generated MemSteer QA from the target user's history alone (0 target-user official QAs; 6+1=7 users in total).

Thus, the two settings were not directly comparable; we have separated Table 9 into two tables in the revision for clarity. When compared against the target-excluded 6-user shared design (Table 3 in the submission), personal continuation improves David R. Ellis from 38.6% to 39.2% (+0.6 pp) and Martin Keller from 37.7% to 44.3% (+6.6 pp).

Reviewer_2P5ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemSteer, a framework that optimizes agent memory designs through generating QA–evidence supervision from unlabeled user histories. It further introduces MemSteer-Opt to iteratively refine memory store, retrieval, and context construction with the help of QA-based design evaluation and evidence-guided failure diagnosis. Authors conduct a series of experiments on ATM-Bench and RHELM with promising results.

Critical strengths
  1. By constructing questions from collected evidence, MemSteer helps ensure the correctness of generated QA pairs while supporting automatic memory-design adaptation to different users.

  2. MemSteer-Opt further introduces an iterative refinement process that uses evidence-guided failure diagnosis to evaluate and improve memory designs.

Critical weaknesses
  1. The evidence-grounded generation pipeline helps ensure that generated questions and answers are supported by historical records. However, the verification procedure does not establish that every selected evidence record is necessary to answer the question. Redundant or irrelevant records may therefore be included in the generated supervision, potentially encouraging unnecessary memory retention or inefficient use of the memory budget. It would be useful to evaluate evidence minimality and investigate whether the generated supervision leads to redundant memory representations or increased storage costs.

  2. Although generating questions from user histories grounds supervision in available evidence, the resulting questions may not reflect what real users would ask or value: generation may favor easily extractable facts while underrepresenting implicit preferences, evolving interests, and infrequent but important needs. Optimizing memory designs on synthetic questions may therefore not transfer to real user queries. The authors could investigate this gap by comparing generated questions with real queries and calibrating generation with human feedback or a small set of real queries.

Questions for authors
  1. The graph construction process relies on heuristic edge definitions, connecting records based on shared conversations, document adjacency, explicit references, or sufficient lexical overlap. However, the paper does not investigate how sensitive the quality of the generated QA–evidence supervision is to these design choices. Different edge definitions may lead to different sampled subgraphs and evidence bundles, thereby affecting the diversity, quality, and downstream utility of generated questions. A sensitivity analysis comparing alternative edge criteria or graph construction strategies would help establish the robustness of the proposed approach.

  2. The cross-user transfer and continuation experiments offer preliminary evidence that adapting a shared memory design with target-user histories can improve performance. However, with only two target users, the conclusions about the general benefits of adaptive user memory remain weak. Furthermore, these experiments do not fully separate the gains from user-specific adaptation from those attributable to additional optimization with automatically generated supervision. Stronger evidence would require a larger and more diverse user pool, together with comparisons against both a fixed shared design and an equally optimized but non-personalized baseline.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2P5Z (Part-1)

Thank you for the thoughtful review and constructive suggestions for improving this work. We address each of your questions/comments below.

W1. The evidence-grounded generation pipeline helps ensure that generated questions and answers are supported by historical records. However, the verification procedure does not establish that every selected evidence record is necessary to answer the question. Redundant or irrelevant records may therefore be included in the generated supervision, potentially encouraging unnecessary memory retention or inefficient use of the memory budget. It would be useful to evaluate evidence minimality and investigate whether the generated supervision leads to redundant memory representations or increased storage costs.

Response to W1: Thank you for this observation. We agree that our existing verification does not guarantee evidence minimality. Our new audits find no clear evidence redundancy in 87% of 100 sampled pairs assessed by a human annotator and 85% of 600 accepted QA-evidence pairs assessed by GPT-6.1-sol. In a controlled reoptimization experiment, correcting the supervision led to very similar measured storage footprint (Table R3) and similar QA performance (Table R4).

Existing safeguards. (i) Before generating a question, the planner constructs an evidence-grounded answer plan that includes the reference answer and each selected evidence record’s stated contribution. The planner is instructed to select evidence records with substantive, source-supported contributions and to avoid irrelevant records. (ii) Source-grounded verification (Section 3.2) includes a semantic audit that checks each contribution and rejects candidates if any contribution is unsupported. Together, these safeguards encourage relevant and complementary evidence, but do not guarantee that every selected record is necessary.

Human and LLM-based assessment on redundancy/irrelevancy. To quantify redundancy, a human annotator assessed 100 randomly sampled accepted ATM-Bench QA-evidence pairs against the original records. As references, DeepSeek-V4-Pro, Kimi-K2.6 and GPT-6.1-sol also assessed these pairs. The audit evaluated reference-answer correctness and completeness, evidence sufficiency, and whether any selected evidence record could be removed without losing support for the complete answer. Table R1 reports the results. For redundancy, the human annotator and GPT-6.1-sol agreed on 99 of 100 pairs.

We used GPT-6.1-sol to audit all 600 accepted ATM-Bench QA-evidence pairs. For each pair, the audit assessed whether any selected evidence record could be removed without losing support for the complete reference answer. As shown in Table R2, 510 pairs (85.0%) passed the criterion of having no clear evidence redundancy, while 90 pairs (15.0%) were flagged as containing at least one potentially redundant record.

Table R1: Human and model assessments of 100 sampled ATM-Bench QA-evidence pairs. Percentages use 100 pairs as the denominator.

Audit criterionDeepSeek-V4-Pro pass (%)Kimi-K2.6 pass (%)GPT-6.1-sol pass (%)Human pass (%)
Question validity99.0%100.0%99.0%99.0%
Reference-answer correctness and completeness92.0%93.0%93.0%91.0%
Evidence sufficiency94.0%95.0%88.0%90.0%
No clear evidence redundancy85.0%91.0%88.0%87.0%

Table R2: Quality assessment of 600 accepted ATM-Bench QA-evidence pairs using a stronger model GPT-6.1-sol. All percentages use 600 pairs as the denominator.

Audit criterionPassFail
Question validity596 (99.3%)4 (0.7%)
Reference-answer correctness and completeness562 (93.7%)38 (6.3%)
Evidence sufficiency549 (91.5%)51 (8.5%)
No clear evidence redundancy510 (85.0%)90 (15.0%)
AuthorsOpenReview ↗

Response to Reviewer 2P5Z (Part-2)

Influence on memory cost and performance. Selected evidence is used for failure attribution and design revisions. To assess its potential effect on the resulting memory design, we compared designs optimized with original and corrected supervision.

  1. Optimized with original supervision: The architecture obtained by optimizing the BM25 starting system with MemSteer on ATM-Bench using our generated QA-evidence supervision.
  2. Optimized with corrected supervision: The architecture obtained by restarting from the same BM25 system and reoptimizing with MemSteer after reviewing and removing redundant evidence and correcting flagged QA defects over generated QA-evidence based on GPT-6.1-sol.

Table R3 shows that correcting the supervision did not change the selected record-level representation or the measured storage footprint. The average context supplied per question differed slightly (46.0k vs. 46.6k characters). The two architectures achieved similar QS (68.3% with original supervision and 68.5% with corrected supervision; Table R4).

Table R3: Memory and index storage comparison between architectures optimized with original and corrected supervision on ATM-Bench. Both runs used 400 training examples and 200 validation examples, an optimization budget of 20 valid candidate configurations, and a fixed random seed of 7.

MetricOptimized with original supervisionOptimized with corrected supervision
Stored memory units11,03411,034
Memory index text size (UTF-8)12.7 MB12.7 MB
BM25 inverted index (RAM)8.3 MB8.3 MB
Vector index45.2 MB45.2 MB
Total retrieval index (RAM)53.5 MB53.5 MB
Context supplied per question46.0k characters46.6k characters

Table R4: Performance comparison between architectures optimized with original and corrected supervision on the full 1,013-question ATM-Bench evaluation set.

ArchitectureQuestion Score (QS)Evidence Recall@10
Optimized with original supervision68.3%84.1%
Optimized with corrected supervision68.5%83.0%
AuthorsOpenReview ↗

Response to Reviewer 2P5Z (Part-3)

W2. Although generating questions from user histories grounds supervision in available evidence, the resulting questions may not reflect what real users would ask or value: generation may favor easily extractable facts while underrepresenting implicit preferences, evolving interests, and infrequent but important needs. Optimizing memory designs on synthetic questions may therefore not transfer to real user queries. The authors could investigate this gap by comparing generated questions with real queries and calibrating generation with human feedback or a small set of real queries.

Response to W2: Thank you for this insightful question. We agree that grounding synthetic questions in user histories does not ensure that they reflect what users would ask in practice. Our goal is to use generated QA-evidence supervision covering a wide range of question types to generally optimize memory capabilities (how a memory system represents user data, retrieves relevant information, and organizes retrieved content for accurate answer generation). Improving these underlying capabilities is intended to generalize to a broad range of user questions.

Design strategy. To better optimize these capabilities, we construct single-hop and multi-hop questions covering a range of question types and user needs, including factual recall, multi-record information integration, temporal and causal reasoning, reasoning about explicit updates to previously recorded information, aggregation, and identifying false premises or conflicts with prior context.

Transfer to preference- and need-related questions. Using keyword search and semantic checks with gpt-5-nano, we identified 23 questions about implicit preferences or evolving interests and 107 questions about important practical needs. Without using official QA for optimization, the design optimized with MemSteer-generated supervision outperformed Base-BM25 by 24.7 QS points on the first subset and 8.1 QS points on the second (Table R5). These results support transfer to these types of benchmark questions on ATM-Bench.

Table R5: ATM-Bench Question Score (QS, %) on the preference/interest and important-needs subsets. Official-QA training and validation items remain in the evaluation subsets.

MethodPreference/interest QS (%) (23 questions)Important-needs QS (%) (107 questions)
Base-BM2547.877.6
MemSteer-Opt + Official-QA68.184.7
MemSteer-Opt + MemSteer72.585.7

Transfer to official benchmark questions. To assess transfer beyond the generated supervision, we evaluate the optimized designs on official benchmark questions. On ATM-Bench, MemSteer with our QAs achieves 68.3 QS, compared with 67.3 QS for MemSteer with Official-QA (Table 1), both substantially outperforming the baseline of 54.9 QS. On RHELM with external data, MemSteer with generated QAs achieves 39.9 accuracy, compared with 40.7 for Official-QA and 33.3 for baseline (Table 2). Note that MemSteer uses no official evaluation questions for optimization, whereas the Official-QA training and validation questions remain in the full evaluation sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.