- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MemSteer, which automatically generates evidence-grounded QA data from unlabeled user histories to guide memory system optimization. The authors also propose MemSteer-Opt, which uses QA performance and supporting evidence to diagnose errors in storage, retrieval, and answer generation, then iteratively refines the memory system design without updating model parameters. Experiments on ATM-Bench and RHELM show that the generated supervision improves memory system performance, benefits other memory optimization frameworks, and supports cross-dataset transfer and adaptation to new users.
Critical strengths
-
The paper addresses a problem of practical value. Personal memory system optimization often lacks human supervision, and manually annotating data for each user is prohibitively expensive. By automatically constructing evidence-grounded QA data from user histories, the paper provides a scalable source of supervision with promising practical applications.
-
The paper introduces a training-free memory optimization method. MemSteer-Opt combines QA-based evaluation, evidence-guided error diagnosis, and system design revisions to iteratively improve memory storage, retrieval, and context construction.
-
The experimental evaluation is comprehensive. The paper evaluates the method on two benchmarks and systematically examines the effectiveness and applicability of the generated supervision through ablations of data generation and optimizer components, integration with other optimization frameworks, cross-dataset transfer, and adaptation to new users.
Critical weaknesses
-
The reliability of synthetic supervision needs further independent verification. Answer planning, question generation, auditing, and blind answer recovery all heavily rely on LLMs. Consistency across these steps does not guarantee annotation accuracy, as shared errors or biases may go undetected. Human verification and robustness experiments that vary the generator and verifier are needed to further establish the reliability of the supervision.
-
Error attribution based on designated evidence may be unreliable. MemSteer-Opt diagnoses failures using the evidence selected during QA generation, but this evidence may not be the only valid basis for answering the question. The system may retrieve other similar records containing the same facts or retain a summary sufficient to answer the question. Therefore, the absence of a designated piece of evidence does not necessarily mean that essential information is missing. The paper should clarify how it identifies equivalent evidence and verify that the diagnostic mechanism does not misclassify these cases as storage or retrieval errors.
-
The benefits of optimization from a stronger starting point remain unclear. The main experiments start with a basic BM25 system and progressively add components such as hybrid retrieval, reranking, and multi-query generation. It remains unclear whether MemSteer-Opt would still deliver substantial gains when starting from a stronger, manually configured RAG system under the same computational budget.
Questions for authors
Please refer to the weaknesses section. Moreover, I have an additional question:
- How does the author make sure the distribution of synthetic data matches real-world user needs?
I'll raise my score if the author can address these concerns.
7 comments
Response to Reviewer qXPF (Part-2)
W2. Error attribution based on designated evidence may be unreliable. MemSteer-Opt diagnoses failures using the evidence selected during QA generation, but this evidence may not be the only valid basis for answering the question. The system may retrieve other similar records containing the same facts or retain a summary sufficient to answer the question. Therefore, the absence of a designated piece of evidence does not necessarily mean that essential information is missing. The paper should clarify how it identifies equivalent evidence and verify that the diagnostic mechanism does not misclassify these cases as storage or retrieval errors.
Response to W2: Thank you for the constructive suggestion. We have clarified the attribution process in the revision. To reduce the latency and cost of diagnosing each incorrect response (in order to identify the failure type), our error attribution uses lightweight, programmatic evidence-coverage and source-recovery checks rather than an LLM-based semantic-equivalence check. We then sample a small set of failures from each observed failure category and provide the design proposer with the question, reference answer, selected evidence, retrieval traces, and answer-context excerpts for each case. This allows the proposer to consider information beyond the initial coarse labels when proposing design revisions.
Other records. Yes, the absence of a designated piece of evidence from the answer context does not necessarily mean that the required information is unavailable: another retrieved record may contain the same facts. However, if the designated record is recoverable from memory but not retrieved, this is still a missed retrieval opportunity for that record and provides a signal for improving retrieval of the designated evidence.
Summaries. Each summary retains its source IDs in our schema. The attribution check matches these IDs against the selected evidence records, allowing a retrieved summary to be associated with its source evidence.
Empirical usefulness of the coarse labels. Although these labels may be imperfect when equivalent information is present in other records, Figure 3 in our manuscript shows that they still provide useful optimization signals: adding failure attribution to randomly sampled failures with gold evidence (Random-30 + Gold evidence → Random-30 + Diagnosis) improves ATM-Bench QS from 58.4 to 62.7 and R@10 from 67.6 to 83.7.
Response to Reviewer qXPF (Part-3)
W3. The benefits of optimization from a stronger starting point remain unclear. The main experiments start with a basic BM25 system and progressively add components such as hybrid retrieval, reranking, and multi-query generation. It remains unclear whether MemSteer-Opt would still deliver substantial gains when starting from a stronger, manually configured RAG system under the same computational budget.
Response to W3: Thank you for the suggestion. We evaluated whether MemSteer-Opt remains effective from strong starting designs, using the same optimization budget of 20 candidate rounds as in the Base-BM25 experiments.
ATM-Bench: optimization beyond a strong retrieval pipeline. Our fixed ATM-Bench-evolved-retrieval baseline already includes BGE-M3 hybrid retrieval (dense retrieval combined with BM25), reranking, HyDE-4, and a 25-record reader. Starting from this pipeline, MemSteer-Opt improves QS from 61.2 to 70.9 on the full set (+9.7 QS points) and from 24.7 to 30.5 on Hard (+5.8 QS points) (Table R16, single run). Thus, the gains are not limited to upgrading a basic BM25 system with these retrieval components.
RHELM: continued optimization of an already evolved design. We also start from ATM-Bench-evolved (frozen), the full design optimized on ATM-Bench and transferred to RHELM without target-domain optimization. Continued optimization improves accuracy by 4.2 percentage points in both settings, from 26.4% to 30.6% without external data and from 38.6% to 42.8% with external data (single run; Table R17). Across three runs, the resulting accuracy has a standard deviation of 0.3 percentage points in both settings.
Together, these results support the value of continued memory-design optimization even when the starting system already incorporates strong retrieval components or a previously optimized design.
Table R16: ATM-Bench performance from different starting designs. Question Score (QS) and evidence recall at 10 (R@10) are reported in percent for the full (1,013 questions) and Hard (31 questions) sets. Each group compares a starting design with its optimized counterpart (under the same computational budget of 20 candidate rounds); mean ± std summarizes 3 repeated runs.
| Method | Full QS | Full R@10 | Hard QS | Hard R@10 |
|---|---|---|---|---|
| Starting point: Base-BM25 | ||||
| Base-BM25 | 54.9 | 61.7 | 8.2 | 10.0 |
| MemSteer (Base-BM25 start; single run) | 68.3 | 84.1 | 29.1 | 43.6 |
| MemSteer (Base-BM25 start; 3 runs, mean ± std) | 68.7 ± 0.8 | 83.7 ± 0.3 | 28.1 ± 1.8 | 43.2 ± 0.4 |
| Starting point: ATM-Bench-evolved-retrieval | ||||
| ATM-Bench-evolved-retrieval | 61.2 | 83.8 | 24.7 | 43.6 |
| Ragas+check*(ATM-Bench-evolved-retrieval start) | 67.8 | 81.5 | 24.2 | 41.0 |
| MemSteer* (ATM-Bench-evolved-retrieval start) | 70.9 | 83.4 | 30.5 | 42.5 |
Table R17: RHELM accuracy from different starting designs. Accuracy (%) is reported for conversation records alone (without external data) and with emails and documents added (with external data). Each group compares a starting design with its optimized counterpart; mean ± std summarizes 3 repeated optimization runs.
| Method | Without external data | With external data |
|---|---|---|
| Starting point: Base-BM25 | ||
| Base-BM25 | 22.3 | 33.3 |
| MemSteer (Base-BM25 start; single run) | 26.4 | 39.9 |
| MemSteer (Base-BM25 start; 3 runs, mean ± std) | 27.0 ± 0.8 | 40.2 ± 1.3 |
| Starting point: ATM-Bench-evolved (frozen) | ||
| ATM-Bench-evolved (frozen) | 26.4 | 38.6 |
| ATM-Bench-evolved + MemSteer (single run) | 30.6 | 42.8 |
| ATM-Bench-evolved + MemSteer (3 runs, mean ± std) | 30.3 ± 0.3 | 42.8 ± 0.3 |
Response to Reviewer qXPF (Part-1)
Thank you for the thoughtful review and constructive suggestions for improving this work. We address each of your questions/comments below.
W1. The reliability of synthetic supervision needs further independent verification. Answer planning, question generation, auditing, and blind answer recovery all heavily rely on LLMs. Consistency across these steps does not guarantee annotation accuracy, as shared errors or biases may go undetected. Human verification and robustness experiments that vary the generator and verifier are needed to further establish the reliability of the supervision.
Response to W1: Thank you for this constructive suggestion. To assess the reliability of our supervision beyond the original LLM pipeline, we conducted a blinded human audit of the original generated supervision and tested additional generator–verifier combinations.
(i) Human audit of the original generated supervision. In our paper, when DeepSeek-V4-Flash was used as generator, gpt-5-mini was used as the verifier. We sampled 100 accepted QA-evidence pairs from the supervision used to optimize the ATM-Bench memory design (these pairs were generated using DeepSeek-V4-Flash (G1) and verified using gpt-5-mini (V1)). A human annotator assessed each pair against the original records. Table R14 shows that among the 100 randomly sampled QA-evidence pairs, the human annotator judged 91% of reference answers correct and complete and 90% of evidence sets sufficient.
Table R14: Human assessment of 100 sampled ATM-Bench generated QA-evidence pairs. Generator: DeepSeek-V4-Flash; Verifier: gpt-5-mini.
(ii) Robustness across generators and verifiers. DeepSeek-V4-Flash (G1) and Kimi-K2.6 (G2) each generated 200 candidate QA-evidence pairs from ATM-Bench materials. For each generator, gpt-5-mini (V1) and gemma-4-31B (V2) assessed the same 200 candidates under matched criteria, yielding four generator–verifier configurations. As a fifth configuration, we used DeepSeek-V4-Flash (V3) to verify candidates generated by DeepSeek-V4-Flash (G1) only. For each configuration, the same blinded human annotator assessed 50 randomly sampled accepted pairs for question validity, reference-answer correctness and completeness, and evidence sufficiency.
As shown in Table R15, human-audited validity ranges from 94.0% to 100.0% across all five configurations. These results provide evidence that the verification pipeline produces high-quality supervision across the tested generator–verifier configurations, reducing concern that the results depend on a single model pairing.
Table R15: Effects of generator and verifier choices on ATM-Bench QA-evidence quality, evaluated on 50 samples each. V1 and V2 assessed the same candidates for each generator; V3 was evaluated only with G1. Human-valid pairs satisfy question validity, answer correctness and completeness, and evidence sufficiency; the 95% CI applies to the human-valid rate among assessable accepted pairs.