acceptodds
Under review as a conference paper at ICLR 2027

Different Strokes for Different Folks: Memory Transformation Routing for LLM Agents

Abstract

Long-term memory enables language agents to retain information beyond a bounded context window. Existing systems typically transform interaction histories through consolidation, augmentation, or relational structuring, applying a fixed transformation across queries. We show that these transformations offer complementary benefits: each improves some queries while degrading others, and no single transformation consistently dominates. This finding motivates MemRx, a memory-aware router that selects among raw and transformed memory views for each query. However, routing based solely on the query and answer-level supervision is insufficient: the query does not reveal how relevant evidence is organized in memory, while answer scores overlook evidence quality and can be sensitive to wording. MemRx addresses these limitations through two components. Probe-Conditioned Routing (PCR) combines query representations with memory-state signals extracted from a lightweight retrieval probe to estimate each view’s utility. Dual-Level Utility Supervision (DUS) integrates LLM judgments of evidence sufficiency and answer correctness with gold-answer F1, giving greater weight to judge signals when F1 poorly distinguishes candidate views. At inference time, MemRx executes only on the selected view and requires a single LLM call. Experiments on LoCoMo and PerLTQA demonstrate the benefits of adaptive transformation selection, with gains of up to 2.05 average F1 points over the strongest fixed transformation on LoCoMo.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Memory, Memory Transformation, Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_D11QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that no single memory transformation, such as summarizing, adding keywords, or linking related chunks, is best for every question, and proposes MemRx to pick one memory view per question. A preliminary study on LoCoMo shows that each transformation helps some questions and hurts others, and that always picking the best view would gain 7 F1 over the best fixed view. MemRx makes this choice with a small router that looks at the question and at simple statistics of a quick retrieval over raw memory, a component called PCR, and it is trained with a mix of answer F1 and LLM judge scores, called DUS. On LoCoMo and PerLTQA, MemRx beats the best fixed view by about 2 F1.

Critical strengths
  1. The preliminary study is clear and useful. It shows at the query level that memory transformations are complementary, which gives a concrete reason to route between them.

  2. The method is light at inference time (one probe retrieval, one scorer pass, one reader call), and the case studies, including a failure case, explain when the probe features help.

Critical weaknesses
  1. Most design choices are intuitive and hand-picked, with no justification or analysis, so the method reads as engineering more than a principled routing approach.
  • The set of candidate views is chosen by hand: three families, one representative each. The paper does not test what happens with more or fewer views, for example routing over all seven views of the preliminary study, or removing structuring, which MemRx selects for only 1.9% of questions.
  • The seven probe features are hand-designed, with no ablation of individual features and no explanation of why the shape of a retrieval result over raw memory should tell which transformation to use.
  • Each view is represented by an author-written description (Table 6) that already says when the view works well (e.g., "Good when the question uses different words than the conversation"), so part of the routing decision is written in by hand.
  • The training target mixes judge scores and F1 with a fixed weight (lambda=0.5) and a fixed switching threshold (delta=0.2). As a result, it is unclear which parts of the design matter and whether the method would work for a different set of views.
  1. The key numbers are measured with token F1, which mostly measures wording. The paper itself says F1 is sensitive to wording, which is why it adds LLM judges for training.
  • The motivating headroom (+7.03) comes from picking, for each question, the view whose answer has the highest F1. If several views give equally correct answers with slightly different wording, this picks the luckiest wording, so the headroom mixes real complementarity with wording noise.
  • The main gain (about +2 F1 over the best fixed view) may also come from picking answers whose wording is closer to the reference. The paper has an answer judge but does not report judge accuracy. Judge-based accuracy and a judge-based oracle are needed to show that the headroom and the gain are real.
  1. The benefit of looking at the memory state (PCR), the main idea of the method, is small in the experiments. Routing on the query alone loses only 0.4 F1 with Qwen3-8B (Figure 3). On questions with a single best view, MemRx picks it 34.6% of the time, against 30.5% without PCR and 25.0% at random (Figure 4). The failure case in Appendix A also shows that when all views retrieve the right evidence, the outcome depends on how the reader uses it, which PCR cannot see.
  2. The paper does not show that choosing one view per question is worth it.
  • Inference needs only one reader call, but all four views must be built in advance for every memory (three extra LLM calls per chunk), and training runs all four views with two judge calls each per training question.
  • Once all views exist, the simplest alternative is not to choose: retrieve from every view and merge the results into the same 20 entries (for example with reciprocal-rank fusion). This is not compared.
  • Existing routing methods are not compared either, including SelRoute, which the paper cites and which also routes retrieval for long-term conversational memory, and Adaptive-RAG.
Questions for authors
  1. Do the memory-system baselines (Mem0, A-Mem, LightMem, SimpleMem) use the same reader, encoder, and chunking as MemRx?
5 comments

We sincerely thank you for your thorough, insightful, and constructive review. We are glad that you found our preliminary study clear and motivating, appreciated our inference efficiency and diagnostic case studies, and rated our presentation as clear (Presentation: 3). Your suggestions directly inspired a comprehensive set of new experiments (view-set ablations, description ablations, judge-based accuracy, noise-floor analysis, RRF fusion, and routing baselines) that have made the paper much stronger.

W1. Justification and sensitivity of design choices. We appreciate your point that every design choice should be backed by systematic analysis. While our components are motivated by specific failure modes (the four views represent raw memory plus the three canonical transformation families in Table 1; the probe features summarize four fundamental properties of a retrieval distribution; and DUS invokes LLM judges when F1 fails to separate views), we ran new ablations and sensitivity sweeps to check that MemRx does not rely on brittle hand-tuning:

  • Sensitivity to λ\lambda (E9): Sweeping λ∈0.0,0.25,0.5,0.75,1.0\lambda \in \\{0.0, 0.25, 0.5, 0.75, 1.0\\} yields 39.12, 39.39, 39.91, 39.78, 39.77 F1 on Qwen3-8B (judge acc: 41.93%–43.00%) and 30.19, 30.13, 30.54, 30.02, 30.09 F1 on Gemma3-4B. Across λ∈[0.25,0.75]\lambda \in [0.25, 0.75], MemRx consistently beats the best fixed transformation (AUGMENT) by +1.53 to +2.05 F1 (Qwen3-8B) and +1.25 to +1.77 F1 (Gemma3-4B).
  • Sensitivity to δ\delta (Figure 5 & E9): Sweeping δ∈0,0.05,0.1,0.2,0.5,1.0\delta \in \\{0, 0.05, 0.1, 0.2, 0.5, 1.0\\} shows a wide flat plateau for δ≥0.1\delta \ge 0.1 on Qwen3-8B (39.7–39.9 F1; 42.25%–43.01% judge acc) and stability within 1 F1 point on Gemma3-4B (29.5–30.5 F1). Thus, (λ=0.5,δ=0.2)(\lambda = 0.5, \delta = 0.2) sits in a broad, insensitive region.

W2. Choice of candidate views (7 views vs. 4 views vs. 3 views). Thank you for suggesting these exact view-set experiments. We initially selected one representative per family (S-Str, A-Att, G-Ent) because Figure 1b shows that intra-family variants have nearly parallel win/loss profiles, capturing most complementarity while halving offline construction cost. Following your suggestion, we tested both view-set modifications (E8):

  1. Routing over all 7 preliminary views (V+\mathcal{V}^+): Achieves 39.40 ± 0.32 F1 (42.34% judge acc) on Qwen3-8B and 29.06 ± 0.04 F1 on Gemma3-4B, still beating the best fixed view (37.86 / 28.77 F1) but slightly below the 4-view router (39.91 / 30.54), as splitting probability mass across near-duplicate intra-family views slightly dilutes supervision while doubling offline construction calls.
  2. Removing STRUCTURE (3 views): Drops F1 from 39.91 to 39.11 ± 0.26 (judge acc from 42.99% to 42.01%) on Qwen3-8B and from 30.54 to 29.00 ± 0.14 on Gemma3-4B, indicating that keeping all three transformation families in the action space is beneficial even when Structuring is chosen selectively.

W3. Why raw-retrieval shape indicates the right transformation & feature ablations. We thank you for asking us to make this intuition explicit. The raw probe is not meant to answer the query, but to expose structural signatures of how raw retrieval fails: (i) flat, dispersed scores across redundant chunks (low fmarf_{\text{mar}}, high feff,fredf_{\text{eff}}, f_{\text{red}}) indicate that summarization (CONSOLIDATE) can compress noise; (ii) low query–centroid cosine (falif_{\text{ali}}) or low lexical coverage (fcovf_{\text{cov}}) diagnoses vocabulary mismatch, where keyword/attribute enrichment (AUGMENT) helps; and (iii) high multi-chunk coverage gap (fgapf_{\text{gap}}) indicates distributed evidence, where relational linking helps. While individual scalar features have modest standalone linear correlation with view gains (≤0.09\le 0.09), they interact non-linearly with the query and probe centroid embeddings in our MLP scorer. In a new leave-one-feature-out ablation (E7), dropping any single feature lowers F1 by 0.45–0.84 points on Qwen3-8B (39.07–39.46 vs. 39.91) and by 1.27–1.50 points on Gemma3-4B (29.04–29.27 vs. 30.54), suggesting that the features act jointly rather than independently and that the router benefits from the full set.

W4. Do view descriptions encode hand-written routing hints? This is a great question. First, our preliminary study (Table 2) shows that view descriptions alone cannot drive effective routing: the LLM Judge router is given the exact view descriptions and the query, yet fails to beat the best fixed view (36.35 F1) because it routes 100% of queries to A-Att; adding 8 labeled examples (ICL, 35.86 F1) also fails. Second, following your suggestion, we directly tested whether MemRx's MLP relies on the hand-written hints in Table 6 by re-training MemRx with neutral placeholder names ("View A / B / C / D") and with no descriptions (one-hot vectors) (E8):

  • Neutral names ("View A–D"): Achieves 39.76 ± 0.27 F1 (42.70% judge acc) on Qwen3-8B, within 0.15 F1 of Table 6 descriptions (39.91).
  • No descriptions (one-hot): Achieves 39.18 ± 0.53 F1 (42.04% judge acc) on Qwen3-8B and 29.19 ± 0.37 F1 on Gemma3-4B—still above the best fixed view (37.86 / 28.77 F1). This indicates that the router learns view utility primarily from the supervised training targets rather than from author-written text hints.

W5–W7. Token F1, wording noise, and LLM-judge accuracy. We completely agree with you that evaluating with a semantic judge alongside F1 is essential to rule out wording noise. Following your recommendation, we now report LLM-judge accuracy for all methods, the 4-view oracle, and the single-view decoding noise floor (E1, E3):

MethodQwen3-8B F1Qwen3-8B Judge Acc.Gemma3-4B F1Gemma3-4B Judge Acc.
Fixed RAW35.83†38.6026.61†31.93
Fixed CONSOLIDATE37.43†37.3327.18†32.45
Fixed AUGMENT37.86†41.8528.77†31.74
Fixed STRUCTURE35.05†35.2927.81†29.98
Multi-view Fusion (RRF, all 4 views)38.9738.0029.7533.35
Query-only router (w/o PCR)39.50†42.8029.10†31.74
MemRx (Ours)39.91†42.9930.54†33.10
Oracle (4-view)48.7454.2340.8046.42

To separate true cross-view complementarity from random decoding/wording variance, we ran RAW twice on the identical retrieved context (at temperature 0.1) and took the best of the two answers (E1): this noise floor yields +3.71 F1 / +4.72 judge-accuracy points on Qwen3-8B and +4.81 F1 / +5.28 points on Gemma3-4B. By contrast, the 4-view oracle gains +12.91 F1 / +15.63 judge-accuracy points over RAW (+10.88 F1 / +12.38 points over the best fixed view) on Qwen3-8B, and +14.19 F1 / +14.49 points over RAW on Gemma3-4B—roughly 3× the decoding noise floor (2.7–3.5× depending on reader and metric). Under the judge metric, MemRx is above all four fixed views on both readers (by 1.14 points over the best fixed view on Qwen3-8B and 0.65 points on Gemma3-4B); RRF fusion is slightly higher in judge accuracy on Gemma3-4B (33.35% vs. 33.10%) but lower in F1 on both readers and in judge accuracy on Qwen3-8B.

W8. Benefit of PCR. We appreciate this nuanced point. While the overall margin over w/o PCR on Qwen3-8B is 0.41 F1 (39.91 vs. 39.50), PCR delivers larger gains where query semantics alone are ambiguous: (i) on Gemma3-4B, PCR improves overall F1 by +1.44 points (30.54 vs. 29.10) and judge accuracy by +1.36 points (33.10% vs. 31.74%), including a +4.32 F1 gain on Open-Domain questions (14.42 vs. 10.10), where w/o PCR collapses to picking Augmentation 100% of the time; (ii) on Qwen3-8B, PCR improves Multi-Hop by +1.49 F1 (32.99 vs. 31.50) and Temporal by +0.67 F1 (48.07 vs. 47.40); and (iii) on unique-winner queries (Figure 4), PCR raises exact hit accuracy from 30.5% to 34.6% (+4.1 points). We also agree with your reading of Figure 7d: when all views retrieve the right turn, the outcome depends on reader utilization, which we will highlight as an inherent boundary of retrieval-stage routing.

W9–W10. Is per-query selection worth it? Cost and comparison to RRF fusion. Thank you for suggesting the RRF fusion baseline and cost comparison. Any multi-view approach—including RRF fusion—requires constructing the views offline (1,475 LLM calls per transformed view across LoCoMo's 10 conversations; 0 for Raw). Once built, MemRx's one-time offline router training on 233 questions requires only 932 reader calls and 1,864 single-token judge calls. At inference time, MemRx is more accurate in F1 and nearly 2× faster in retrieval than RRF fusion (E1, E12):

  • Online latency: RRF fusion must query all 4 indices (193.38 ms), whereas MemRx runs only the raw probe (49.61 ms), the MLP scorer (1.12 ms), and the single selected view's retrieval (49.11 ms), totaling 99.84 ms (and 1 reader LLM call with ~3,360 context tokens vs. 3,510 for RRF).
  • Accuracy: On Qwen3-8B, MemRx beats RRF fusion by +0.94 F1 (39.91 vs. 38.97) and +4.99 judge-accuracy points (42.99% vs. 38.00%); on Gemma3-4B, MemRx is +0.79 F1 higher (30.54 vs. 29.75) with comparable judge accuracy (33.10% vs. 33.35%).

W11. Comparison to SelRoute and Adaptive-RAG. Following your recommendation, beyond the four query-side routers in Table 2 (Random, LLM Judge, ICL, MLP Clf.), we implemented SelRoute (both regex query-type routing and an oracle Gold-Category router using ground-truth LoCoMo question categories) and Adaptive-RAG (complexity classifier) to route over our four views under the exact same readers (E2):

  • Qwen3-8B: MemRx (39.91 F1 / 42.99% Judge Acc) is above SelRoute (39.26 / 41.87%), Gold-Category SelRoute (38.99 / 41.21%), and Adaptive-RAG (38.30 / 38.60%). Notably, even a router with ground-truth question categories (38.99 F1) trails MemRx by 0.92 F1 (3.49 F1 on Multi-Hop: 29.50 vs. 32.99), suggesting that question type alone does not capture how evidence is organized in memory.
  • Gemma3-4B: MemRx (30.54 F1 / 33.10% Judge Acc) is likewise above SelRoute (29.07 / 31.79%), Gold-Category SelRoute (29.23 / 32.39%), and Adaptive-RAG (29.47 / 31.91%).

Q1. Baseline settings. Yes: all four memory-system baselines (Mem0, A-Mem, LightMem, and SimpleMem) use the exact same readers (Qwen3-8B and Gemma3-4B), the same embedding model / encoder (Qwen3-Embedding-0.6B), the same chunking configuration (5 dialogue turns with a 1-turn overlap), and the same evaluation script as MemRx. Evaluated with our LLM answer judge on the Table 4 outputs (E13), Mem0, A-Mem, LightMem, and SimpleMem achieve 24.51%, 19.26%, 30.36%, and 35.40% judge accuracy on Qwen3-8B (vs. 42.99% for MemRx) and 13.88%, 15.78%, 23.81%, and 22.85% on Gemma3-4B (vs. 33.10% for MemRx). We will state these shared baseline settings explicitly in Appendix D.

We thank you again for your constructive guidance. We hope these results address your concerns, and we are happy to answer any further questions during the discussion period.

Reviewer_KheSOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This work addresses the problem of long-term memory representation selection in LLM agents. The authors observe that existing memory transformations—specifically consolidation (summarization), augmentation (metadata/cue tagging), and structuring (graph construction)—exhibit distinct trade-offs, meaning no single transformation uniformly outperforms others across diverse user queries. To resolve this, this paper introduces MemRx, a query-adaptive memory routing framework designed to dynamically choose the optimal memory representation view per query. MemRx incorporates two main components: Probe-Conditioned Routing (PCR): A pre-retrieval step that performs lightweight probing over raw memory to compute seven scalar features characterizing evidence concentration, dispersion, alignment, and coverage. Dual-Level Utility Supervision (DUS): An offline supervision pipeline combining context-level evidence sufficiency and answer-level correctness evaluated by an external LLM judge.

Critical strengths
  1. The paper highlights the "no one-size-fits-all" bottleneck in agent memory management, empirically demonstrating a substantial 9.12 F1 gap between standard raw retrieval and an ideal Oracle router.
  2. The authors provide a clear diagnostic breakdown identifying why standard routers fail, explicitly formalizing the Information Gap (lack of visibility into memory layout) and the Supervision Bottleneck (noisy lexical surface matching).
  3. By offloading feature extraction to a lightweight vector probe and training a compact MLP router, MemRx preserves online operational efficiency with minimal added latency.
Critical weaknesses
  1. MemRx restricts its routing action space to a mutually exclusive choice among four discrete views (RAW, CONSOLIDATE, AUGMENT, STRUCTURE). In complex reasoning tasks, relevant evidence is frequently heterogeneous, requiring structural relational paths alongside global semantic summaries. Enforcing a hard partition prevents cross-view hybrid retrieval and fundamentally caps system expressiveness.
  2. The PCR component extracts memory state features solely by probing the RAW memory view. If raw vector retrieval suffers from semantic drift or vocabulary mismatch, the resulting seven probe features become noise-corrupted, causing the downstream router to make sub-optimal choices across all other views.
  3. The seven scalar probe features (fmar, fdec, feff, fred, fali, fcov, fgap) rely heavily on hand-crafted heuristic formulas (e.g., cosine margin decay and token coverage differentials). Such manual IR statistics are inherently fragile and sensitive to embedding dimensions, chunking granularities, and similarity metrics.
  4. Experimental results reveal that MemRx yields minimal gains—and in several cases, performance degradation—on multi-hop queries. For instance, under the Gemma3-4B backbone, MemRx achieves 26.85 F1 on multi-hop questions, falling below the static Structuring baseline. The framework underperforms precisely on the multi-step queries where memory transformation is most needed.
  5. Due to high offline LLM labeling costs, the dataset partitions are extremely small (e.g., 10 total conversations in LoCoMo split 2:1:7 for train/val/test), yielding a router training set of only ~230 instances. Furthermore, the experiments evaluate static, pre-packaged logs rather than non-stationary long-term memory stores subjected to real-time insertions and edits.
Questions for authors
  1. Why did the authors choose a mutually exclusive discrete selection strategy over a multi-view dense re-ranking or score-fusion approach that could combine top candidates across multiple views?
  2. How does PCR maintain feature reliability when raw memory retrieval experiences near-zero similarity scores or catastrophic vocabulary mismatch?
  3. How robust are the seven hand-engineered probe features when applied to non-stationary memory stores undergoing continuous real-time insertion, deletion, and re-consolidation over long operational periods?
4 comments

We sincerely thank you for your detailed and constructive review, and for highlighting our empirical demonstration of the 9.12 F1 oracle headroom, our diagnostic breakdown of the information gap and supervision bottleneck, and the online operational efficiency of MemRx. Your insightful questions pushed us to run new experiments on multi-view fusion, probe robustness, feature ablations, and cross-dataset transfer.

Presentation & Clarity Revisions. We also thank you for your candid feedback on presentation (Presentation: 1), which we take to heart. To make the paper much clearer and easier to follow, we are making three concrete structural revisions: (1) adding a comprehensive Notation Table in Appendix A; (2) replacing the dense inline formulas of the seven probe features in Section 3.1 with a clean, structured summary table that pairs each mathematical definition with a one-line intuition and the specific retrieval failure mode it diagnoses; and (3) tightening Section 2.3 so the logical flow from our preliminary observations to the PCR/DUS architecture is seamless.

W1 / Q1. Discrete selection vs. multi-view fusion. We thank you for raising this natural question. We initially chose discrete selection because evidence across different transformations is not guaranteed to be mutually reinforcing—in fact, it is often antagonistic. As shown in Figure 1a, for a given query, one view frequently improves the answer while another degrades it relative to raw memory. Pooling entries from all views into a fixed 20-entry context budget can inject redundant or conflicting statements (e.g., a summary resolving a date differently from a raw turn) while crowding out relevant evidence from the best view. Following your suggestion, we tested this empirically by implementing two multi-view fusion baselines that retrieve from all four views and merge candidates into the same 20-entry budget via (i) Reciprocal Rank Fusion (RRF) and (ii) Round-Robin interleaving (5 entries per view) (E1):

  • Qwen3-8B: RRF fusion achieves 38.97 F1 (38.00% judge acc) and Round-Robin achieves 39.15 F1 (42.73% judge acc), versus 39.91 F1 (42.99% judge acc) for MemRx.
  • Gemma3-4B: RRF fusion achieves 29.75 F1; Round-Robin is on par with MemRx (30.57 vs. 30.54 F1).
  • Online Latency (E12): Fusion executes all 4 view retrievals online (193.38 ms), whereas MemRx executes only the raw probe and the single chosen view (99.84 ms total).

Thus, discrete routing outperforms RRF fusion and matches or exceeds Round-Robin interleaving while querying a single view. We agree that simple interleaving is a strong baseline; we will include both fusion baselines in the main paper and discuss soft multi-view allocation as a promising extension.

W2 / Q2. Reliability of probing only RAW. We appreciate this insightful question. We would like to clarify that the purpose of the raw probe is not to retrieve the final answer, but to diagnose how candidate evidence is structured in raw memory. When raw retrieval suffers from vocabulary mismatch or semantic drift, the resulting low similarity and flat score profile are themselves informative diagnostic signals—specifically indicating that lexical/keyword expansion (AUGMENT) is required. Furthermore, the router does not rely on ϕq\phi_q in isolation; it conditions jointly on the query embedding eqe_q and probe centroid eˉPq\bar{e}_{\mathcal{P}_q} (Eq. 8). Following your suggestion, we examined robustness under degraded raw retrieval with two targeted experiments (E10):

  1. Stratifying by probe top-1 similarity: In the lowest-similarity quartile on LoCoMo (r1∈[0.361,0.618]r_1 \in [0.361, 0.618], n=288n = 288), MemRx recognizes the weak raw match, routes to AUGMENT 93% of the time (90% on Gemma3-4B), and beats RAW by +1.40 F1 on Qwen3-8B (37.68 vs. 36.28).
  2. Synthetic paraphrased queries (vocabulary mismatch): We rewrote all 1,155 test queries to replace content words while preserving meaning (reducing lexical overlap with the original queries to 0.43). Without retraining, MemRx remains at or above the fixed views on both readers (Qwen3-8B: 34.06 ± 0.41 vs. 32.97 RAW and 33.85 AUGMENT; Gemma3-4B: 26.36 ± 0.17 vs. 25.53 RAW and 26.25 AUGMENT), i.e., its advantage narrows but does not reverse under vocabulary mismatch.

W3. Hand-crafted features. We acknowledge that the seven probe features involve feature engineering, and we appreciate your push to rigorously verify their contribution. We designed them to incur zero extra model calls while capturing four distinct structural properties of a retrieval distribution: Concentration (fmar,fdecf_{\text{mar}}, f_{\text{dec}}) detects whether evidence is localized in a few top chunks; Dispersion (feff,fredf_{\text{eff}}, f_{\text{red}}) detects spread across redundant chunks (where CONSOLIDATE helps); Alignment (falif_{\text{ali}}) measures semantic query–evidence match (where low values favor AUGMENT); and Coverage (fcov,fgapf_{\text{cov}}, f_{\text{gap}}) measures whether multiple chunks are needed jointly (favoring STRUCTURE / AUGMENT). Rescaling scores by the within-query standard deviation σq\sigma_q also makes the features invariant to linear shifts in embedding similarity scale. In a new leave-one-feature-out ablation (E7), removing any individual feature reduces F1 on both Qwen3-8B (from 39.91 to 39.07–39.46, a drop of 0.45–0.84 F1) and Gemma3-4B (from 30.54 to 29.04–29.27, a drop of 1.27–1.50 F1), suggesting that the features act jointly rather than independently: the router benefits from the full set, and removing any one of them forfeits most of the probe's benefit.

W4. Multi-hop performance. We agree with your observation that MemRx is not uniformly best on every single category, and we will state this explicitly in Section 4.2. We emphasize that MemRx is a lightweight, plug-and-play router rather than a heavy new memory indexing mechanism, yet it achieves the highest overall F1, BLEU-1, and Judge Accuracy and wins 7 of 8 category-reader pairs (Table 4). On multi-hop questions, MemRx outperforms all four fixed views on Qwen3-8B (32.99 vs. 32.28 Cons. and 31.33 Struct.) and remains within 0.36 F1 of Structuring on Gemma3-4B (26.85 vs. 27.21). Because multi-hop queries often require combining a high-level summary of one session with fine-grained entity links from another, selecting a single view per query naturally bounds multi-hop gains, which we will highlight as a key motivation for multi-view composition.

W5 & Q3. Small training set, generalization, and non-stationary memory. We thank you for raising these important points regarding data scale and dynamic memory:

  1. Why ~230 training instances suffice: Although the LoCoMo training split contains 233 questions, MemRx is specifically architected for sample efficiency: the query and probe embeddings pass through fixed (non-learned) low-dimensional projections (Πq,Πp∈R32×d\Pi_q, \Pi_p \in \mathbb{R}^{32 \times d}), and the trainable scorer θ=Wx,Wd,g\theta = \\{W_x, W_d, g\\} is a compact 3-layer MLP with 32-dimensional hidden states totaling only ~10K trainable parameters, regularized with weight decay (10−410^{-4}; Section 3.1, Table 5). This tiny parameter footprint limits overfitting on small data, while conversation-level splitting ensures zero memory leakage between train and test splits.
  2. Cross-dataset evaluation & zero-shot transfer (E11): Beyond LoCoMo, we evaluate on PerLTQA (1,552 test questions across 7 characters), where MemRx beats all baselines on 3 of 4 memory types. Furthermore, to test whether a router trained on 233 LoCoMo questions generalizes across datasets, we transferred the LoCoMo-trained router zero-shot to PerLTQA: it achieves 35.64 ± 0.13 average F1, on par with, and slightly above, fixed Consolidation (35.43) and Augmentation (35.54) without any PerLTQA training data.
  3. Non-stationary / streaming memory (Q3): Static pre-built conversation logs are the standard evaluation protocol used by all compared baselines on LoCoMo—Mem0 (Chhikara et al., 2025), A-Mem (Xu et al., 2026b), LightMem (Fang et al., 2026), and SimpleMem (Liu et al., 2026). MemRx's probe features are computed online from the top-KpK_p retrieved chunks at query time (normalized by KpK_p and within-query σq\sigma_q), so they do not depend on total memory size and are recomputed after insertions, deletions, and updates. We will add a dedicated discussion of streaming updates to the revision.

We hope that our new experiments (on multi-view fusion, low-similarity/paraphrase robustness, feature ablations, and zero-shot cross-dataset transfer) alongside our concrete presentation improvements address your concerns, and we are happy to answer any further questions during the discussion period.

Thank you to the authors for the thorough rebuttal and the new experiments. My main concerns have been well addressed.

Please ensure all newly added experiments and presentation revisions are fully incorporated into the final manuscript. I am raising my rating to 3 (and updating Presentation to 2).

Thank you for your thoughtful feedback and for raising the score. We are glad that our responses and new experiments addressed your main concerns.

We have uploaded a revised PDF that incorporates all the newly added experiments and presentation improvements from the rebuttal, with changes highlighted in blue for easy reference.

Reviewer_L4tXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies whether different questions benefit from different ways of organizing long-term conversational memory. It proposes MemRx, a router that selects among raw, consolidated, augmented, and graph-structured memory views. Probe-Conditioned Routing (PCR) combines the question representation with information from an initial retrieval over raw memory. Dual-Level Utility Supervision (DUS) combines judgments of evidence sufficiency and answer correctness with reference-answer F1 to train the router. Experiments on LoCoMo and PerLTQA report improvements over several baselines.

Critical strengths
  1. The problem is practically relevant in memory adminstration which the authors reveal a more deeper view. The paper questions the assumption that a single memory transformation is suitable for every question. Its per-question analysis provides useful evidence that different views have complementary strengths and weaknesses.
  2. The online routing design is lightweight. MemRx uses retrieval-derived signals and a small scoring network rather than evaluating every candidate view with an LLM at inference time. It sounds like good generalization ability with different backbones and several retrieval methods.
  3. The overall logic of the article is clear, the main picture is distinct to understand.
Critical weaknesses
  1. The novelty relative to closely related work is insufficiently established. As I remember, there was an article several months ago that proposed a mechanism for adaptive memory which called FluxMem (Choosing How to Remember: Adaptive Memory Structures for LLM Agents). They already learns to select among complementary memory structures using interaction-level features and offline feedback combining response quality and memory utilization. This overlaps substantially with the paper’s high-level motivation and learning framework, yet FluxMem is neither cited nor compared against.
  2. I find a confused point in your comparation that raw memory uses dense retrieval, whereas attribute augmentation uses hybrid dense–sparse retrieval. Consequently, the observed differences cannot be attributed solely to memory transformation. A raw-memory baseline with the same hybrid retriever is needed to separate these effects.
  3. The supervision design needs more targeted validation. Equation (12) uses the difference between the highest and lowest F1 across all views to determine the judge weight. This fairness needs to be carefully considered. A poor view can make this difference large and disable judge supervision even when the two best views are tied. Minor issue: Writing
Questions for authors
  1. Does MemRx retain its advantage when raw memory uses the same hybrid dense–sparse retrieval strategy as the augmentation view?
  2. The implementation in this paper is inconsistent with the final settings in the paper in many aspects. These include 5/10-turn chunking, 32/64-dimensional projection, AdamW/Adam, and the models used for building the memory. The main table uses macro-average for categories, while the given summary code uses average by question; the soft scores of judge also do not strictly equal the P(Y) defined in the paper. These issues make it difficult to reliably reproduce the results.
  3. Can you provide an evaluation experiment of separately evidence-only and answer-only supervision, leaving the necessity of both components unclear that I mentioned in weakness 3.
  4. Insufficient validation of Dual-Level Utility Supervision (DUS). DUS is presented as one of the two main contributions of the paper. However, the authors rely on DeepSeek-V3.1 to evaluate evidence sufficiency and answer correctness, and then combine these judgments with F1 scores. Then the question naturally arose that the effectiveness and reliability of this supervision strategy are not sufficiently validated, specifically in how accurate is the Evidence Judge in assessing evidence sufficiency? How well do the LLM judges agree with human annotations? Whenever you introduce an external LLM as a judge, you always need to conduct such verification and provide evidence through correlation coefficients. Is should at least shown in Appendix.
10 comments

We sincerely thank you for your thoughtful review and for recognizing the practical relevance of our problem, the value of our per-question complementarity analysis, and the lightweight online design of MemRx. Your insightful questions helped us significantly strengthen our baselines, ablations, and reproducibility.

W1. Novelty relative to FluxMem. We thank you for pointing us to FluxMem (Lu et al., 2026)—this is a very relevant reference, and we will cite and discuss it prominently in Section 5. While both works share the high-level motivation that no single memory structure fits all scenarios, their core distinction lies in when and how adaptation happens: FluxMem decides how to build memory at write time by assigning one structure to a conversation segment from dialogue-level interaction features, whereas MemRx decides how to read memory at query time by maintaining candidate views and routing each individual question. This distinction matters for two fundamental reasons: (1) within the exact same conversation, different questions require different views (as shown in Section 4.2, ground-truth optimal views across questions of the same LoCoMo conversations are split across Consolidation 36.0%, Augmentation 33.3%, Raw 19.6%, and Structuring 11.1%), which a conversation-level choice made before seeing the query cannot capture; and (2) whether a transformation helps depends on how the evidence for that specific query is distributed in memory, which is only observable once the query arrives. This is why MemRx conditions on a query-time retrieval probe (PCR) rather than dialogue statistics and supervises per-query evidence sufficiency (DUS). Moreover, because MemRx is agnostic to how candidate views are constructed, it is complementary to FluxMem and can route directly over FluxMem's structures. Following your suggestion, we re-implemented FluxMem's structure selector (Sec. 3.4: 12 interaction features, 4-unit MLP picking one view per conversation) on top of our four views (E2). On Qwen3-8B it is competitive with MemRx (39.87 vs. 39.91 F1), while on Gemma3-4B it trails MemRx (29.08 vs. 30.54 F1; 31.74% vs. 33.10% judge acc), with the largest gap on Open-Domain questions (10.05 vs. 14.42 F1). We note that this comparison covers FluxMem's selection mechanism only, not its full memory hierarchy and fusion gate.

W2 / Q1. Retrieval confound between RAW and AUGMENT. We thank you for catching this important experimental control. We initially paired each transformation with the standard retrieval method adopted by its family in prior systems (e.g., attribute-augmented memories are explicitly designed to support lexical/keyword matching alongside dense embeddings, hence hybrid dense–sparse retrieval; Appendix C.3), so that each view was evaluated in its intended setting. Following your suggestion, we isolated the transformation from the retriever by re-running RAW with the exact same hybrid dense–sparse retriever (BM25 + dense RRF, Eq. 15) as AUGMENT (E5):

View & RetrieverQwen3-8B F1Qwen3-8B BLEU-1Qwen3-8B Judge Acc.Gemma3-4B F1Gemma3-4B BLEU-1Gemma3-4B Judge Acc.
RAW (dense)35.83†28.22†38.6026.61†20.19†31.93
RAW + hybrid retriever (E5)37.9330.2140.1028.2521.5233.17
AUGMENT (hybrid)37.86†30.45†41.8528.77†21.93†31.74
MemRx39.91†32.07†42.9930.54†23.92†33.10

As you anticipated, hybrid retrieval improves RAW from 35.83 to 37.93 F1 (Qwen3-8B) and from 26.61 to 28.25 F1 (Gemma3-4B), so part of the gap between dense RAW and AUGMENT in our original table is indeed attributable to the retriever. Under the identical retriever the two views are close: on Qwen3-8B they are on par in F1 (37.93 vs. 37.86), with AUGMENT higher in BLEU-1 (30.45 vs. 30.21) and judge accuracy (41.85% vs. 40.10%); on Gemma3-4B AUGMENT is higher in F1/BLEU-1 (28.77 / 21.93 vs. 28.25 / 21.52), while Hybrid RAW is higher in judge accuracy (33.17% vs. 31.74%). MemRx improves over Hybrid RAW by +1.98 F1 and +2.89 judge-accuracy points on Qwen3-8B and by +2.29 F1 on Gemma3-4B, with comparable judge accuracy on Gemma3-4B (33.10% vs. 33.17%). We will add Hybrid RAW to the main results table and revise the text so that the retriever and transformation effects are reported separately. Thank you for suggesting this control.

W3. Eq.(12) uses max–min F1 gap. We appreciate this thoughtful observation. Conceptually, we used the max–min F1 spread because the role of αq\alpha_q in Eq.(12) is to detect whether lexical F1 provides any discriminative signal across the view set as a whole: when all four views achieve nearly the same F1 (e.g., all 0 or all 1), F1 carries zero ranking information and supervision should rely on the LLM judges. We completely agree with your point that a single poor outlier view can enlarge the max–min gap while the top two views tie. However, its practical impact on training is mild because the expected regret loss (Eq. 13) still penalizes the poor outlier view accurately, leaving only the tie-break between the top two views to F1. To test your intuition directly, we evaluated a variant replacing the max–min spread in Eq.(12) with the top-2 F1 gap (E6). This variant achieves 38.91 ± 0.55 F1 (41.83% judge acc) on Qwen3-8B and 28.60 ± 0.33 F1 (31.80% judge acc) on Gemma3-4B, compared to 39.91 / 30.54 F1 (42.99% / 33.10% judge acc) for the max–min spread. Because the top two views frequently have a small F1 difference even when F1 clearly separates good views from bad ones, using the top-2 gap makes αq\alpha_q overly conservative and under-weights the gold F1 anchor. We will include both variants and this discussion in the revision.

Q2. Implementation inconsistencies. We sincerely apologize for the confusion caused by the code archive, and we thank you for inspecting our code so carefully. The supplementary zip was exported from an earlier development branch while we were running grid searches via CLI flags; when we finalized the settings for the paper, we inadvertently failed to overwrite the default constants in config.py and the standalone summary script before packaging. We have thoroughly audited the codebase and resolved every discrepancy you noted:

  1. Hyperparameters & Models: Every reported run in the paper used 5-turn chunks with 1-turn overlap (Appendix C.1; config.py has been updated from 10 to 5), 32-dim projections/hidden states, AdamW (Table 5), and Qwen3-8B (non-thinking mode) for memory construction (Appendix C.1).
  2. Aggregation: All paper tables report the macro-average across the four question categories ("Avg." in Table 4); we fixed the helper summary script, which previously printed micro-averages across questions.
  3. Judge soft scores vs. P(Y)P(Y): In our implementation, judge scores are extracted from single-token (Y/N) generation at temperature 0. To verify that minor differences between logprob extraction and hard binary verdicts do not affect our conclusions, we re-trained MemRx using strictly binary hard judge verdicts 0,1\\{0, 1\\} (E6): on Qwen3-8B, it achieves 39.89 ± 0.39 F1 and 42.97% judge accuracy (within 0.02 of 39.91 / 42.99%). We will release the cleaned repository with one locked config file per table and a one-click reproduction script.

Q3. Evidence-only vs. answer-only supervision. Thank you for suggesting this ablation to clarify the contribution of each judge in DUS. Extending Figure 3 (w/o DUS = F1-only), we ablate evidence-only, answer-only, and combined judge supervision without the F1 anchor on LoCoMo (E6):

Supervision VariantQwen3-8B F1Qwen3-8B Judge Acc.Gemma3-4B F1Gemma3-4B Judge Acc.
F1 only (= w/o DUS, Fig. 3)36.90†38.5029.50†31.81
Evidence judge only (λ=1,δ=∞\lambda = 1, \delta = \infty)39.58 ± 0.3142.7229.00 ± 0.1932.65
Answer judge only (λ=0,δ=∞\lambda = 0, \delta = \infty)37.51 ± 0.5139.2728.72 ± 0.3932.48
Evidence + answer, no F1 (λ=0.5,δ=∞\lambda = 0.5, \delta = \infty)38.63 ± 0.4541.2528.93 ± 0.4732.84
Full DUS (λ=0.5,δ=0.2\lambda = 0.5, \delta = 0.2)39.91†42.9930.54†33.10

Each judge alone improves judge accuracy over F1-only training on both readers, with the evidence judge being the stronger standalone signal (+2.68 F1 / +4.22 judge-accuracy points on Qwen3-8B, close to Full DUS). However, judge-only variants do not transfer to lexical F1 on Gemma3-4B (28.72–29.00 vs. 29.50 for F1-only), which is precisely why DUS keeps gold F1 as an anchor. Full DUS is the only variant that is best on both metrics and both readers. We will state in the revision that the evidence judge accounts for most of the gain and that the answer judge's benefit is smaller.

Q4. Reliability of the LLM judges. We completely agree that validating external LLM judges is essential. Following your advice, we sampled 200 (question, view) pairs (50 per view) and re-annotated them independently using a stronger, frontier-tier model (Claude Fable 5.2) as an external reference annotator, following the exact same grading rubrics without access to our judge outputs (E4). Because DeepSeek-V3.1 was deprecated on our compute provider during the rebuttal period, we also cross-checked against DeepSeek-V4.1-Flash:

  • Answer Correctness Judge: Shows strong agreement with the Claude Fable 5.2 reference annotations, achieving 82.0% raw agreement (κ=0.65\kappa = 0.65) for DeepSeek-V3.1 and 93.5% raw agreement (κ=0.87\kappa = 0.87) for DeepSeek-V4.1-Flash (V3.1 and V4.1-Flash also agree with each other at 85.5%, κ=0.71\kappa = 0.71).
  • Evidence Sufficiency Judge: Reaches 62.5% (κ=0.22\kappa = 0.22) and 66.0% (κ=0.34\kappa = 0.34) agreement with Claude Fable 5.2 (and 70.5%, κ=0.41\kappa = 0.41 between V3.1 and V4.1-Flash). Examining the confusion matrix shows that disagreements are almost purely one-directional (66/75 for V3.1 and 68/68 for V4.1-Flash): Claude Fable 5.2 marks 81.5% of contexts as sufficient (vs. 53.0% / 47.5% for DeepSeek) because a frontier reasoning model can connect subtle, implicit clues that our stricter rubric rejects for 4B–8B readers. Crucially, precision with respect to the Claude Fable 5.2 reference is 88.0%–100% (when our judge says evidence is sufficient, Claude Fable 5.2 almost always agrees). We will add the full correlation tables and annotation analysis to the appendix.

We hope these new experiments, ablations, and code fixes address your concerns, and we are happy to answer any further questions during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

Dear Area Chair,

We thank you and the reviewers for the time spent on our submission. Below is a brief summary of the discussion and of what we see as the contribution of the paper.

1. Status of the reviewers' concerns. All three reviewers engaged with our response and acknowledged the additional experiments (E1–E13): (1) Reviewer L4tX asked for a retrieval control for raw memory, validation of the supervision design and of the LLM judges, and a comparison with FluxMem; after our response and a follow-up on the answer judge and FluxMem, the reviewer raised the soundness score to 3 and subsequently raised the overall rating, commenting that "your quality and innovation are quite good"; (2) Reviewer KheS asked about multi-view fusion, the robustness of the raw probe, the hand-crafted features, and generalization; the reviewer raised their rating after our response and asked that the new experiments be included in the final version, which we have done in the revised PDF; (3) Reviewer D11Q asked for judge-based evaluation, a decoding-noise floor for the oracle headroom, view-set and description ablations, and routing baselines; the reviewer stated that most concerns have been resolved, with a remaining reservation about the overall contribution, which we address below.

2. Contribution. (1) An empirical finding: memory transformations are complementary at the query level. Each helps some queries and hurts others relative to raw memory, and the oracle headroom over the best fixed view persists under a wording-invariant judge metric and is 2.7 to 3.5 times the decoding-noise floor. Existing systems apply one transformation to all queries, so this headroom is currently unused; (2) A diagnosis of why the obvious solutions fall short: routing from the query alone does not beat the best fixed view, because the query does not reveal how its evidence is organized in memory, and supervision from answer F1 alone is unreliable when views differ only in wording; (3) A lightweight method that follows from this diagnosis: MemRx conditions routing on a single probe of the memory state and trains on evidence- and answer-level judgments anchored to F1. It needs one reader call per query and improves over the best fixed view by +2.05 and +1.77 F1 on the two readers, with consistent gains in judge accuracy.

3. Revision. The revised PDF (changes in blue) adds the related-work discussion of FluxMem, a notation table and a summary table of the probe features, and an appendix with all additional experiments, labeled with the identifiers used in our responses.

Thank you again for handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.