- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that no single memory transformation, such as summarizing, adding keywords, or linking related chunks, is best for every question, and proposes MemRx to pick one memory view per question. A preliminary study on LoCoMo shows that each transformation helps some questions and hurts others, and that always picking the best view would gain 7 F1 over the best fixed view. MemRx makes this choice with a small router that looks at the question and at simple statistics of a quick retrieval over raw memory, a component called PCR, and it is trained with a mix of answer F1 and LLM judge scores, called DUS. On LoCoMo and PerLTQA, MemRx beats the best fixed view by about 2 F1.
Critical strengths
-
The preliminary study is clear and useful. It shows at the query level that memory transformations are complementary, which gives a concrete reason to route between them.
-
The method is light at inference time (one probe retrieval, one scorer pass, one reader call), and the case studies, including a failure case, explain when the probe features help.
Critical weaknesses
- Most design choices are intuitive and hand-picked, with no justification or analysis, so the method reads as engineering more than a principled routing approach.
- The set of candidate views is chosen by hand: three families, one representative each. The paper does not test what happens with more or fewer views, for example routing over all seven views of the preliminary study, or removing structuring, which MemRx selects for only 1.9% of questions.
- The seven probe features are hand-designed, with no ablation of individual features and no explanation of why the shape of a retrieval result over raw memory should tell which transformation to use.
- Each view is represented by an author-written description (Table 6) that already says when the view works well (e.g., "Good when the question uses different words than the conversation"), so part of the routing decision is written in by hand.
- The training target mixes judge scores and F1 with a fixed weight (lambda=0.5) and a fixed switching threshold (delta=0.2). As a result, it is unclear which parts of the design matter and whether the method would work for a different set of views.
- The key numbers are measured with token F1, which mostly measures wording. The paper itself says F1 is sensitive to wording, which is why it adds LLM judges for training.
- The motivating headroom (+7.03) comes from picking, for each question, the view whose answer has the highest F1. If several views give equally correct answers with slightly different wording, this picks the luckiest wording, so the headroom mixes real complementarity with wording noise.
- The main gain (about +2 F1 over the best fixed view) may also come from picking answers whose wording is closer to the reference. The paper has an answer judge but does not report judge accuracy. Judge-based accuracy and a judge-based oracle are needed to show that the headroom and the gain are real.
- The benefit of looking at the memory state (PCR), the main idea of the method, is small in the experiments. Routing on the query alone loses only 0.4 F1 with Qwen3-8B (Figure 3). On questions with a single best view, MemRx picks it 34.6% of the time, against 30.5% without PCR and 25.0% at random (Figure 4). The failure case in Appendix A also shows that when all views retrieve the right evidence, the outcome depends on how the reader uses it, which PCR cannot see.
- The paper does not show that choosing one view per question is worth it.
- Inference needs only one reader call, but all four views must be built in advance for every memory (three extra LLM calls per chunk), and training runs all four views with two judge calls each per training question.
- Once all views exist, the simplest alternative is not to choose: retrieve from every view and merge the results into the same 20 entries (for example with reciprocal-rank fusion). This is not compared.
- Existing routing methods are not compared either, including SelRoute, which the paper cites and which also routes retrieval for long-term conversational memory, and Adaptive-RAG.
Questions for authors
- Do the memory-system baselines (Mem0, A-Mem, LightMem, SimpleMem) use the same reader, encoder, and chunking as MemRx?
5 comments
W4. Do view descriptions encode hand-written routing hints? This is a great question. First, our preliminary study (Table 2) shows that view descriptions alone cannot drive effective routing: the LLM Judge router is given the exact view descriptions and the query, yet fails to beat the best fixed view (36.35 F1) because it routes 100% of queries to A-Att; adding 8 labeled examples (ICL, 35.86 F1) also fails. Second, following your suggestion, we directly tested whether MemRx's MLP relies on the hand-written hints in Table 6 by re-training MemRx with neutral placeholder names ("View A / B / C / D") and with no descriptions (one-hot vectors) (E8):
- Neutral names ("View A–D"): Achieves 39.76 ± 0.27 F1 (42.70% judge acc) on Qwen3-8B, within 0.15 F1 of Table 6 descriptions (39.91).
- No descriptions (one-hot): Achieves 39.18 ± 0.53 F1 (42.04% judge acc) on Qwen3-8B and 29.19 ± 0.37 F1 on Gemma3-4B—still above the best fixed view (37.86 / 28.77 F1). This indicates that the router learns view utility primarily from the supervised training targets rather than from author-written text hints.
W5–W7. Token F1, wording noise, and LLM-judge accuracy. We completely agree with you that evaluating with a semantic judge alongside F1 is essential to rule out wording noise. Following your recommendation, we now report LLM-judge accuracy for all methods, the 4-view oracle, and the single-view decoding noise floor (E1, E3):
| Method | Qwen3-8B F1 | Qwen3-8B Judge Acc. | Gemma3-4B F1 | Gemma3-4B Judge Acc. |
|---|---|---|---|---|
| Fixed RAW | 35.83† | 38.60 | 26.61† | 31.93 |
| Fixed CONSOLIDATE | 37.43† | 37.33 | 27.18† | 32.45 |
| Fixed AUGMENT | 37.86† | 41.85 | 28.77† | 31.74 |
| Fixed STRUCTURE | 35.05† | 35.29 | 27.81† | 29.98 |
| Multi-view Fusion (RRF, all 4 views) | 38.97 | 38.00 | 29.75 | 33.35 |
| Query-only router (w/o PCR) | 39.50† | 42.80 | 29.10† | 31.74 |
| MemRx (Ours) | 39.91† | 42.99 | 30.54† | 33.10 |
| Oracle (4-view) | 48.74 | 54.23 | 40.80 | 46.42 |
To separate true cross-view complementarity from random decoding/wording variance, we ran RAW twice on the identical retrieved context (at temperature 0.1) and took the best of the two answers (E1): this noise floor yields +3.71 F1 / +4.72 judge-accuracy points on Qwen3-8B and +4.81 F1 / +5.28 points on Gemma3-4B. By contrast, the 4-view oracle gains +12.91 F1 / +15.63 judge-accuracy points over RAW (+10.88 F1 / +12.38 points over the best fixed view) on Qwen3-8B, and +14.19 F1 / +14.49 points over RAW on Gemma3-4B—roughly 3× the decoding noise floor (2.7–3.5× depending on reader and metric). Under the judge metric, MemRx is above all four fixed views on both readers (by 1.14 points over the best fixed view on Qwen3-8B and 0.65 points on Gemma3-4B); RRF fusion is slightly higher in judge accuracy on Gemma3-4B (33.35% vs. 33.10%) but lower in F1 on both readers and in judge accuracy on Qwen3-8B.
W8. Benefit of PCR. We appreciate this nuanced point. While the overall margin over w/o PCR on Qwen3-8B is 0.41 F1 (39.91 vs. 39.50), PCR delivers larger gains where query semantics alone are ambiguous: (i) on Gemma3-4B, PCR improves overall F1 by +1.44 points (30.54 vs. 29.10) and judge accuracy by +1.36 points (33.10% vs. 31.74%), including a +4.32 F1 gain on Open-Domain questions (14.42 vs. 10.10), where w/o PCR collapses to picking Augmentation 100% of the time; (ii) on Qwen3-8B, PCR improves Multi-Hop by +1.49 F1 (32.99 vs. 31.50) and Temporal by +0.67 F1 (48.07 vs. 47.40); and (iii) on unique-winner queries (Figure 4), PCR raises exact hit accuracy from 30.5% to 34.6% (+4.1 points). We also agree with your reading of Figure 7d: when all views retrieve the right turn, the outcome depends on reader utilization, which we will highlight as an inherent boundary of retrieval-stage routing.
W9–W10. Is per-query selection worth it? Cost and comparison to RRF fusion. Thank you for suggesting the RRF fusion baseline and cost comparison. Any multi-view approach—including RRF fusion—requires constructing the views offline (1,475 LLM calls per transformed view across LoCoMo's 10 conversations; 0 for Raw). Once built, MemRx's one-time offline router training on 233 questions requires only 932 reader calls and 1,864 single-token judge calls. At inference time, MemRx is more accurate in F1 and nearly 2× faster in retrieval than RRF fusion (E1, E12):
- Online latency: RRF fusion must query all 4 indices (193.38 ms), whereas MemRx runs only the raw probe (49.61 ms), the MLP scorer (1.12 ms), and the single selected view's retrieval (49.11 ms), totaling 99.84 ms (and 1 reader LLM call with ~3,360 context tokens vs. 3,510 for RRF).
- Accuracy: On Qwen3-8B, MemRx beats RRF fusion by +0.94 F1 (39.91 vs. 38.97) and +4.99 judge-accuracy points (42.99% vs. 38.00%); on Gemma3-4B, MemRx is +0.79 F1 higher (30.54 vs. 29.75) with comparable judge accuracy (33.10% vs. 33.35%).
W11. Comparison to SelRoute and Adaptive-RAG. Following your recommendation, beyond the four query-side routers in Table 2 (Random, LLM Judge, ICL, MLP Clf.), we implemented SelRoute (both regex query-type routing and an oracle Gold-Category router using ground-truth LoCoMo question categories) and Adaptive-RAG (complexity classifier) to route over our four views under the exact same readers (E2):
- Qwen3-8B: MemRx (39.91 F1 / 42.99% Judge Acc) is above SelRoute (39.26 / 41.87%), Gold-Category SelRoute (38.99 / 41.21%), and Adaptive-RAG (38.30 / 38.60%). Notably, even a router with ground-truth question categories (38.99 F1) trails MemRx by 0.92 F1 (3.49 F1 on Multi-Hop: 29.50 vs. 32.99), suggesting that question type alone does not capture how evidence is organized in memory.
- Gemma3-4B: MemRx (30.54 F1 / 33.10% Judge Acc) is likewise above SelRoute (29.07 / 31.79%), Gold-Category SelRoute (29.23 / 32.39%), and Adaptive-RAG (29.47 / 31.91%).
Q1. Baseline settings. Yes: all four memory-system baselines (Mem0, A-Mem, LightMem, and SimpleMem) use the exact same readers (Qwen3-8B and Gemma3-4B), the same embedding model / encoder (Qwen3-Embedding-0.6B), the same chunking configuration (5 dialogue turns with a 1-turn overlap), and the same evaluation script as MemRx. Evaluated with our LLM answer judge on the Table 4 outputs (E13), Mem0, A-Mem, LightMem, and SimpleMem achieve 24.51%, 19.26%, 30.36%, and 35.40% judge accuracy on Qwen3-8B (vs. 42.99% for MemRx) and 13.88%, 15.78%, 23.81%, and 22.85% on Gemma3-4B (vs. 33.10% for MemRx). We will state these shared baseline settings explicitly in Appendix D.
We thank you again for your constructive guidance. We hope these results address your concerns, and we are happy to answer any further questions during the discussion period.
We sincerely thank you for your thorough, insightful, and constructive review. We are glad that you found our preliminary study clear and motivating, appreciated our inference efficiency and diagnostic case studies, and rated our presentation as clear (
Presentation: 3). Your suggestions directly inspired a comprehensive set of new experiments (view-set ablations, description ablations, judge-based accuracy, noise-floor analysis, RRF fusion, and routing baselines) that have made the paper much stronger.W1. Justification and sensitivity of design choices. We appreciate your point that every design choice should be backed by systematic analysis. While our components are motivated by specific failure modes (the four views represent raw memory plus the three canonical transformation families in Table 1; the probe features summarize four fundamental properties of a retrieval distribution; and DUS invokes LLM judges when F1 fails to separate views), we ran new ablations and sensitivity sweeps to check that MemRx does not rely on brittle hand-tuning:
W2. Choice of candidate views (7 views vs. 4 views vs. 3 views). Thank you for suggesting these exact view-set experiments. We initially selected one representative per family (S-Str, A-Att, G-Ent) because Figure 1b shows that intra-family variants have nearly parallel win/loss profiles, capturing most complementarity while halving offline construction cost. Following your suggestion, we tested both view-set modifications (E8):
W3. Why raw-retrieval shape indicates the right transformation & feature ablations. We thank you for asking us to make this intuition explicit. The raw probe is not meant to answer the query, but to expose structural signatures of how raw retrieval fails: (i) flat, dispersed scores across redundant chunks (low fmar, high feff,fred) indicate that summarization (CONSOLIDATE) can compress noise; (ii) low query–centroid cosine (fali) or low lexical coverage (fcov) diagnoses vocabulary mismatch, where keyword/attribute enrichment (AUGMENT) helps; and (iii) high multi-chunk coverage gap (fgap) indicates distributed evidence, where relational linking helps. While individual scalar features have modest standalone linear correlation with view gains (≤0.09), they interact non-linearly with the query and probe centroid embeddings in our MLP scorer. In a new leave-one-feature-out ablation (E7), dropping any single feature lowers F1 by 0.45–0.84 points on Qwen3-8B (39.07–39.46 vs. 39.91) and by 1.27–1.50 points on Gemma3-4B (29.04–29.27 vs. 30.54), suggesting that the features act jointly rather than independently and that the router benefits from the full set.