acceptodds
Under review as a conference paper at ICLR 2027

No Single Memory Mechanism Wins: A Systematic Evaluation of Memory Routing Policies for LLM Agents

Abstract

Long-horizon agents accumulate experience across sessions, and a growing number of memory mechanisms have been proposed to store it and read it back, e.g., vector indexes, knowledge graphs, summary trees, and strategy banks. These mechanisms are developed and evaluated separately, each on the regime it targets, which presumes that the type of the upcoming request is known. In deployment, however, an agent faces a mixed stream of requests and has to decide, per request, which form of memory to rely on. This decision has received little attention, and its value has never been measured against an oracle. In this work, we present the first systematic evaluation of memory routing for agents. We merge three benchmarks into a single pool of 2,614 requests and execute every request under all 12 memory substrates, which yields a complete matrix of 31,368 cells. Based on this matrix, we first show that query semantics, on which conventional routers operate, are insufficient to determine which memory is needed. We then quantify how much the choice is worth, i.e., selecting the right substrate for each request attains 64.74 against 36.82 for the best fixed one, a relative improvement of 75.85%. We finally evaluate 14 routers in five tiers under two generalization protocols. The best of them recovers only 32.8% of this headroom, and generalization to unseen memory units remains the binding constraint.

Primary area:
datasets and benchmarks
Keywords:
agent router, Memory substrate, memory router

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dwSZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper benchmark different memory mechanism on 3 different tasks. Their result claim that there are still a huge headroom between the best memory routers and the best performance achievable using an oracle.

Critical strengths
  1. The overall question itself is timely; The question of the effect of memory structure with respect to the tasks at hand is under-explored, especially with the rising number of memory paper/method in recent literatures.
    • This does not endorse the three main question of the paper or their methodology
Critical weaknesses
  1. I struggle to see what is the novelty or contribution of this paper
    • While I don't take issue with the negative result that they showed, the novelty of the research is weak; They paper describes either methodology from the literature or flawed methodology (see Review 2).
    • The novelty itself is incremental. See [1], which is one of the paper they cited and use a very similar methodology with almost no discussion on how this work differ.
  2. The paper feels AI generated with both weak readability and flawed methodology
    • The language is really bad and almost unreadable: dressing up normal language with academic sounding language (e.g. profile eligibility, fixed references), unclear definitions and structure in both the methodology, and results.
    • The methodology for similarity is weak: Weak model (all-MiniLM-L6-v2), normalized result (which again I dont think is defensible as routing model will use full result) so similar requests ended up favoring different memories is unsurprising.
    • Memory-Substrate-Based Routing by itself is I think not defensible. We see multiple memory routing based works (the authors mentioned quite a lot in the related work section), that performs better than the method in this paper (substrates).
    • Given that full context may be more than 32k, the performance (which is less than no memory and compressed KV) reduces my confidence of their results, where they just do blanket zero score for excluded request above 32k context. Also 32k context limit is arbitrary and I think it's lacking especially for long benchmarks.
    • Profile eligibility is not defined in main text. It also introduces many confounder (result is biased towards lower context methods, etc). It's really bad that the author manually set the benchmark questions that can be answered. Compressed KV and A-Mem are excluded based on heuristic which distort the result.
    • Offline selection is flawed. They run all substrates on prompts, and store it in a "matrix". However benchmark like ALFWorld is online and need actual rollout.
    • The baseline itself is weak because of ALFworld issue (see above). Most of the gain can just be traced back to ALFworld.
    • Memory routing policy discussed in section 5 is weak. Learned policies are weak and/or elementary without any discussion on SoTA methods.
    • Lookup is strong, but its trivial as it gets task metadata (which doesn't make sense for real world usage) and outcomes. "The Aggregate Gain Is Concentrated in the Task-Source Distinction"
    • They reported "Held-Out Groups Test More Than Unseen Memory." This is again a failure of the authors to establish a clear train-test split and incredibly confounded by TTL.
    • Cost discussion are insufficient. It took an unreasonable amount of time to setup all 12 substrates, a Pareto analysis is warranted to see the gains and losses.
    • More flaws are hidden in the appendix, e.g. Compressed KV does not compress at all, it re-prefill some arbitrary selected episodes instead of compressing the actual state from the attention. Furthermore the appendix is unreadable and further increases my concern of this being a fully AI written paper with minimal human supervision.
    • Overall they did not distinguish memory mechanism from their implementation and scoring choices, and use a blanked statement of evaluation of memory mechanisms. They included ReasoningBank where its only eligible for ALFWorld, etc.

[1] Huang, et al. Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents. https://arxiv.org/pdf/2608.15008

Questions for authors
  1. "giving equal weight to the three benchmarks and, within MemoryAgent-Bench, to its four competencies. This prevents LoCoMo’s 76% share of requests from dominating the aggregate." Why not report each bench separately from the start?
  2. Is there any discussion on confounders introduced (task mix, length bias, etc)?
  3. Why do the authors think offline selection for ALFworld is defensible?

Comment: Given the severity of the issue of the paper, the probablity of the score being raised barring a huge rewrite and/or methodology convincing during rebuttal will be close to none.

5 comments

We sincerely appreciate Reviewer dwSZ for the detailed and professional feedback. We have carefully considered every point and respond below, grouped by theme in the order of the review. We hope these clarifications address the concerns.

W1: Thank you for pointing this out. We would like to clarify the difference from [1]. [1] compares memory substrates as fixed design choices and reports that none is uniformly best across benchmarks. Our paper studies a follow-up question that [1] leaves open, namely whether this observation can be turned into a gain by choosing a substrate per request. To keep the two studies comparable, we adopt the substrate families and reference configurations of [1], and our contributions are the following three parts:

(1) A per-request outcome matrix for all 12 substrates on one mixed pool of 2,614 requests, with eligibility and observation masks.

(2) A decomposition of the oracle gap by granularity (from 43.66 at the benchmark level to 45.88 at the memory-unit level, leaving 18.87 points at the request level, Fig. 2(b)).

(3) A comparison of 23 routing-policy variants in five families, showing that none improves on a six-row lookup table with an interval excluding zero.

In the revised version, we will add a dedicated paragraph and a comparison table against [1] in Sec. 2.


W2-a: Thank you for the comment. For the terms, "profile eligibility" indicates whether the evaluation profile admits a substrate on a request (App. A). For example, ReasoningBank is eligible only on ALFWorld. A "fixed reference" is the score of using one single substrate for every request, i.e., without routing. We will add these explanations to Sec. 5.1 in the revised version. We will also carefully rewrite the main text and the appendix with simpler language and a clearer structure to improve the readability.


W2-b: Thank you for the comment. We would like to offer the following clarifications:

(1) The similarity analysis in Sec. 4 is a motivating probe, and the main results do not depend on it. The oracle, the headroom and all policy comparisons in Secs. 5-6 are computed from the outcome matrix.

(2) L2 normalization is the standard output of all-MiniLM-L6-v2. It only fixes the vector length, so that the inner product equals the cosine similarity used in Table 1. The direction of the embedding, which carries the semantic information, is unchanged.

(3) The observation that requests favor different substrates does not rely on the encoder. It is also measured directly on the outcome matrix: even granting the best substrate for each memory unit leaves 18.87 of the 27.93 points (Sec. 5.2).

(4) Regarding stronger models, the LLM judges (π5\pi_5) read the request directly without this encoder, and seven of eight score below the lookup (Sec. 6.2). We will further repeat the analysis of Table 1 with stronger sentence encoders in the revised version.


W2-c: Thank you for raising this point. We would like to offer the following explanations:

(1) These routers and substrate-based routing address different decisions. The routers cited in Sec. 2 (e.g., Self-RAG, Adaptive-RAG, MemFlow) decide whether or how deeply to retrieve within one memory design, whereas we choose among heterogeneous memory designs, such as a knowledge graph, a summary tree or a strategy bank. The two are therefore complementary.

(2) We choose substrates as the action space because they are the unit at which [1] reports no uniformly best choice, which motivates choosing among them per request.

(3) The results reported by these works are obtained with different backbones and benchmarks. For a direct comparison, we will add representative ones as reference columns of our matrix in the revised version.

W2-d: Thank you for the careful check. We clarify the full-context results as follows:

(1) The 32K limit is the effective context window of the backbone Qwen3.5-9B (Sec. 3), and it applies to every substrate.

(2) The low aggregate score of full context mainly comes from the zero-credit convention. Full context is excluded on ALFWorld (one third of the weight) and on histories beyond 32K, which gives the aggregate 5.74. On LoCoMo, it scores 16.47, above no memory (3.11) and Compressed KV (9.15) (Table B.3), and on the requests the oracle routes to it, it scores 62.37 (Table 3).

(3) This convention does not affect the main results. LightRAG (best fixed, 36.82) and the summary tree (36.11) have no excluded cells, so SfixS_{\mathrm{fix}} is unaffected by any eligibility rule, and the oracle only maximizes over observed cells.

In the revised version, we will report scores on covered requests beside the zero-credit scores, and add a truncation fallback so that full context is also evaluated on longer histories.


W2-e: Thanks for the comment. We explain the eligibility rules as follows:

(1) The eligibility mask consists of four rules of the evaluation profile (App. A), and no benchmark question is selected manually. Each rule applies to a whole task source or follows from the context window: ReasoningBank is restricted to ALFWorld (2,346 cells), full context must fit the context window and is excluded on ALFWorld (1,550 cells), and A-Mem and Compressed KV are excluded on test-time learning (100 cells each).

(2) These rules do not change which substrate is the best fixed one. LightRAG and the summary tree have no excluded cells. For A-Mem and Compressed KV, even a perfect score on all 200 excluded cells would give at most 17.94+8.33=26.2717.94+8.33=26.27 and 12.36+8.33=20.6912.36+8.33=20.69, both below 36.82.

(3) ReasoningBank is restricted to ALFWorld because it distills strategies from task trajectories, which the QA benchmarks do not provide. Its fixed score of 12.94 therefore mainly reflects this scope.

In the revised version, we will define profile eligibility in Sec. 5.1 and list the four rules there as a table.


W2-f & Q3: Thank you for this important comment. We would like to provide responses by separating three cases:

(1) Half of the ALFWorld requests read a static memory. The 134 offline-bank requests use a fixed bank of expert trajectories, so their outcomes do not depend on which substrate served earlier requests, and offline selection is valid for them.

(2) The task-source lookup is scored on an actual rollout. It assigns every ALFWorld request to ReasoningBank, so its ALFWorld score (38.81) is ReasoningBank's own executed column, whose online memory was updated with its own trajectories. No columns are combined, and this gives its ALFWorld contribution of (38.81−18.28)/3=+6.84(38.81-18.28)/3=+6.84.

(3) Per-request switching under online accumulation needs actual rollouts. π4\pi_4 and the oracle switch substrates within ALFWorld, so on the 134 online requests their scores are offline estimates. We will add router-in-the-loop rollouts in the revised version, and report offline-bank and online requests separately.

Regarding the baseline, the gain on ALFWorld reflects that the globally best substrate is not the locally best one there (LightRAG 18.28 vs. ReasoningBank 38.81), and a task-source policy makes use of this difference. The QA-only comparison is reported in Sec. 6.2: on the 2,346 QA requests, the lookup and π4\pi_4 score 45.69 and 46.04 against 46.08 for fixed LightRAG.

W2-g: Thanks for the comment. We have the following discussions:

(1) The lookup is intended as a simple reference, and the finding is that more complex policies do not improve on it. The best of the 23 variants differs from it by +0.60+0.60 with [−1.7,+3.1][-1.7, +3.1] (Table E.1).

(2) The policies follow the main designs of current routers. They cover category classification as in vLLM Semantic Router (π3\pi_3), nearest-neighbor retrieval (π2\pi_2), per-candidate score regression (π4\pi_4), and prompted selection with eight LLMs (π5\pi_5). We will add adapted versions of more recent routers (e.g., RouteLLM, RouterDC, GraphRouter) in the revised version.

(3) Policies without the task-source label reach nearly the same score. π3\pi_3 predicts the category from the request text only. It reaches 43.31 and makes the same choice as the lookup on 98.89% of requests (Table B.14). Removing the source and request-type inputs from π4\pi_4 changes its score by −0.04-0.04 (Table B.17). As noted in Sec. 6.1, every policy also receives the eligibility mask, which separates ALFWorld from QA requests.

(4) All learned policies, including the lookup, are fitted on training-fold cells only (Sec. 6.1), so no policy uses test outcomes.


W2-h: Thank you for pointing this out. Please kindly notice that both protocols use explicit train-test splits, and the effect of test-time learning is separated from the rest. We provide the details as follows:

(1) For the split, the first protocol evaluates new requests under known memory and keeps identical request texts in one fold. The second holds out whole groups from 29 clusters and also excludes training requests whose text occurs in the test fold (Sec. 6.1, App. B.6).

(2) For test-time learning, Table B.18 decomposes the 7.29-point drop of the grouped protocol by source: test-time learning accounts for 5.58 because its entire competency is held out, whereas LoCoMo and long-range understanding, where a memory unit is held out, account for 0.57 and 0.00.

(3) We also report an unseen-memory test without test-time learning. Leave-one-memory-unit-out on the 20 LoCoMo and long-range units gives 38.47 for π4\pi_4 against 39.22 for the fixed comparator (−0.75-0.75 with [−3.7,+1.8][-3.7, +1.8], Table B.19).


W2-i: Thanks for the constructive suggestion. (1) A gain-versus-cost analysis over substrate subsets is reported in Sec. 6.3, Fig. 3(b) and App. E.3. We grow the substrate set greedily, and report both the oracle and the best policy at each size against the construction time. (2) A small subset already covers most of the headroom. The first three substrates raise the oracle from 36.82 to 51.92, and six substrates reach 61.15 at 82.4 of the 112.1 construction hours, i.e., 94% of the full oracle at about 73.5% of the cost. The last six substrates add only 3.59 points. (3) The policies need even fewer substrates. The best policy scores 43.29 with six substrates against 44.00 with all 12. The benchmark-level rule needs only LightRAG and ReasoningBank, i.e., 9.86+0.20=10.069.86+0.20=10.06 construction hours (Table B.11), for 43.66 in-sample. In the revised version, we will present these results as an explicit score-versus-construction-hours Pareto frontier in Sec. 6.3.


W2-j: Thank you for the careful reading of the appendix. Our explanations are three-fold as follows:

(1) The construction of M11M_{11} is stated in both the main text and App. A. Table 2 lists its readout as "episode re-prefill", and App. A states that utterances are clustered into episodes and the selected episodes are re-prefilled through the backbone's own attention path, so it is not an implementation of a published key-value compression method. App. A also states that M8M_8 and M11M_{11} are built for this evaluation rather than reproduced.

(2) M11M_{11} has little influence on the results. It is not the best fixed substrate (12.36 vs. 36.82 for LightRAG), and removing it lowers the oracle by at most 2.15 points (Table 3).

(3) Our conclusions are about the substrates as configured in this evaluation. App. A states that the evaluation profile specifies the scope of the comparison rather than a substrate's intrinsic ability, and App. F.2 states that underperformance may reflect the evaluation setup as well as the substrate design.

In the revised version, we will move these statements from the appendix into Sec. 4 next to Table 2, state the scope of the conclusions in the main text, and reorganize the appendix for readability.

Reviewer_E5fvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a systematic empirical evaluation of memory routing for LLM agents. The authors evaluate 12 memory substrates across 2,614 requests from three benchmarks and compare 23 routing policy variants spanning lookup-based methods, nearest-neighbor approaches, learned classifiers and regressors, and LLM-based judges. The paper quantifies the gap between the best fixed memory substrate and an offline per-request oracle, reporting scores of 36.82 and 64.74, respectively. The best evaluated routing policy achieves 44.00, with a simple task-source lookup accounting for most of the practical improvement. The authors further investigate routing granularity, generalization to held-out groups, and the computational costs of constructing memory substrates and performing routing.

Critical strengths
  1. Broad empirical coverage. The paper evaluates a relatively diverse set of 12 memory substrates and 23 routing variants across multiple benchmarks and task categories.
  2. Analysis beyond aggregate performance. In addition to reporting overall scores, the paper examines routing granularity, held-out generalization, and routing overhead. These analyses help characterize where the observed gains arise and expose limitations of the evaluated approaches.
Critical weaknesses

Weakness 1: Overly Defensive Writing and Poor Readability

The manuscript is excessively verbose and defensive, with repetitive caveats and unnecessary explanations that undermine readability. For example, the limitations of the offline oracle, including its dependence on recorded execution states, inability to simulate online routing, and lack of generalization guarantees, are repeatedly discussed in the introduction (Lines 100–102), experimental setup (Lines 164–166), and analysis (Lines 295–296, 372–374). The paper even notes that “display rounding can change the last digit of derived quantities” (Lines 245–246). Such excessive qualifications obscure the core contributions, emphasizing what the experiments cannot establish rather than the scientific insights they provide.

Weakness 2: Limited Novelty and Scientific Contribution

The central idea of selecting memory mechanisms based on task characteristics is largely intuitive, given the distinct purposes of retrieval-based, structured, and procedural memory. More importantly, the experiments offer limited insights beyond this intuition. As shown in Section 6.2, ALFWorld accounts for 6.84 of the lookup policy's 6.58-point gain and 7.21 of the regression policy's 7.18-point gain. Across 2,346 QA requests, neither policy outperforms fixed LightRAG (45.69 and 46.04 vs. 46.08). These results suggest that the gains primarily reflect a coarse distinction between procedural and QA tasks rather than effective fine-grained memory routing. Despite evaluating 12 memory substrates and 23 routing variants, the paper offers limited novel mechanisms or generalizable insights, resembling an extensive empirical comparison rather than a hypothesis-driven investigation.

Weakness 3: Insufficient Transparency in AI Usage Disclosure

The AI Use Statement acknowledges generative AI assistance in manuscript polishing, code development, data processing, LaTeX editing, and figure preparation, but does not clarify its involvement in research formulation, experimental design, routing strategy selection, or result interpretation. Given the manuscript's highly formulaic writing and largely enumerative experimental analysis, greater transparency is warranted. The authors should clarify the extent of AI involvement in these research activities and explain how AI-assisted experiments and analyses were independently validated, consistent with the applicable ICLR 2027 AI disclosure requirements.

Questions for authors

See weaknesses.

2 comments

We sincerely thank Reviewer E5fv for the detailed feedback. Below, we provide point-by-point responses and hope these clarifications address the concerns.

W1: Thank you for pointing this issue out. Following the reviewer's suggestions, we will revise the paper as follows:

(1) State the conditions once, in a single paragraph at the start of Sec. 5, and remove the repetitions that the reviewer pointed out:

"Scope. All oracle and policy scores are computed on a recorded matrix with one execution per cell. Three consequences hold throughout: (i) the oracle is an upper reference that may include execution noise, (ii) excluded cells and run-limit failures receive zero credit, and (iii) under ALFWorld online accumulation, a policy that switches substrates per request is scored on histories it did not generate."

(2) Delete the rounding remark and similar bookkeeping sentences, keeping in table captions only what is needed to read the table.

(3) Rewrite Secs. 5-6 so that each paragraph opens with its finding and the qualification follows once.


W2: Thank you for the constructive feedback. We respectfully hope to clarify that the concentration of the gain in the procedural-versus-QA distinction is the main finding of this evaluation study, which we summarize as follows:

(1) None of the 23 policy variants in five families, including eight LLM judges, improves on a six-row lookup table with an interval excluding zero (best: +0.60+0.60 with [−1.7,+3.1][-1.7, +3.1], Table E.1).

(2) The measurable gain reduces to one decision between procedural and QA memory.

(3) On unseen QA memory units, the best learned policy shows no gain over a fixed substrate (−0.75-0.75, [−3.7,+1.8][-3.7, +1.8]).

Memory routing is an intuitive idea, so we believe it is useful for the community to know that, in this setting, its gain so far does not go beyond a coarse split. In the revised version, we will (a) restate the three questions of Secs. 4-6 as explicit hypotheses, each with a verdict (H1: query semantics suffice for memory selection, H2: per-request selection offers headroom over a fixed substrate, and H3: routing policies recover this headroom and transfer to unseen memories), (b) state this finding at the beginning of the Abstract, and (c) add repeated executions to examine how much of the remaining gap is repeatable. Thank you!


W3: Thank you for raising this point. We would like to clarify as follows: (1) All research formulation, experimental design, and routing strategy selection were designed and determined by the authors. (2) For result interpretation, all experiments were run by the authors, and the analysis and conclusions were drawn by the authors from the completed results. LLMs helped with the wording of these descriptions. (3) In addition, LLMs helped with auxiliary tasks, such as drawing figures from the experimental results and polishing the manuscript, together with the code development, data processing and LaTeX editing disclosed in the statement. (4) As for validation, all AI-assisted text, code, analyses and visualizations were reviewed by the authors, as stated in the AI Use Statement, and the reported numbers are obtained from these completed experiments. We will release the outcome matrix with its masks, the evaluation weights, the fold and group definitions, and the scoring code, so that every reported number can be recomputed.

We thank Reviewer E5fv for the review, which will help us improve the paper. We would be ready to further clarify if any point remains unclear.

I thank the authors for their response. However, the rebuttal provides limited new evidence or substantive insights that address my main concerns regarding the research significance and the scientific contributions. Therefore, I maintain my original assessment.

Reviewer_4gFWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares routing policies that choose among twelve memory mechanisms, including retrieval, graph memory, summaries, procedural memory, LoRA, and full context. It builds an outcome matrix over 2614 requests from LoCoMo, MemoryAgentBench, and ALFWorld, then evaluates routers by selecting entries from this matrix.

The best fixed mechanism scores 36.82, the per-request oracle scores 64.74, and the best tested router reaches 44.00. A simple task-source lookup already reaches 43.40. Most of the practical gain comes from choosing different memory for ALFWorld versus question answering. Finer routing and transfer to unfamiliar settings remain difficult.

Critical strengths

The simple lookup result is the strongest part for me. The paper tests many more complicated routers and finds no established advantage over this baseline. It further shows that the aggregate gain comes almost entirely from ALFWorld. This is a useful correction to the impression that memory routing already requires a sophisticated learned controller.

There is substantial experimental work here, with careful checks on duplicate question text, eligibility masks, grouped evaluation, and construction costs. The distinction between the recorded oracle and an achievable routing policy is also important. As an empirical study, the paper gives a useful picture of where current routing methods help and where they do not.

Critical weaknesses

The large oracle gap is based on one execution per cell. Taking the maximum over twelve outcomes can select execution noise as well as a genuinely better memory mechanism. Bootstrapping the same matrix does not resolve this. The paper acknowledges the issue, but the 27.93 point delta still carries much of the motivation. Without repeated executions, it is unclear how much of the remaining gap is stable and predictable.

The practical result is narrower than the mixed-stream framing suggests. On the 2346 QA requests, neither the lookup nor the best main router improves over fixed LightRAG. Also, the ALFWorld online results are scored by combining columns whose memory histories were generated separately. A deployed router would change those histories. This matters because ALFWorld supplies essentially all of the measured routing gain.

Questions for authors

On a representative subset, can you select the best substrate using one execution and evaluate that choice on independent executions? I would like to know how much request-level headroom survives this test.

Does the simple task-source policy retain its advantage in an actual ALFWorld rollout, where memory is updated using the trajectories produced by that policy?

Is there a held-out result showing a benefit from request-level routing within one task source, rather than from separating procedural tasks from QA? The aggregate results currently suggest the latter is doing nearly all the work.

1 comment

We sincerely appreciate Reviewer 4gFW for the constructive feedback, and for recognizing the task-source lookup result and our validity checks. We respond point by point below.

W1 & Q1.

Thank you for this insightful comment. As the reviewer correctly interpreted, 64.74 is a maximum over one execution per cell, and the bootstrap intervals (request level [25.27,30.11][25.27, 30.11] and memory-unit level [14.15,34.57][14.15, 34.57]) hold the cell outcomes fixed, so they do not remove the maximization bias. Regarding Q1, given the limited time of the rebuttal period, we will add this experiment in the revised version. Meanwhile, we would like to share what the current results can already tell:

(1) The part that does not rely on per-request maximization is 7-9 points. Choosing one substrate per benchmark, per request type, or per memory unit gives 43.66, 45.69 and 45.88 against 36.82 (Fig. 2(b) and App. F.1). Each choice is shared by many requests, so a single noisy cell has little influence on it.

(2) The remaining 18.87 points are at the request level, within memory units. This is the part that the repeated executions in the revised version will examine.

(3) Answer decoding is greedy, so variation mainly comes from memory construction, which calls an auxiliary LLM.

In the revised version, following the reviewer's suggestion, we will select the best substrate on one execution over a representative subset, score that choice on independent executions, and report the share of request-level headroom that survives. In the text, we will also motivate the paper with the group-level references in the Abstract and Sec. 1.


W2 & Q2.

Thank you for this important comment. We would like to address the two kinds of policy separately.

(1) For the task-source lookup, yes, its advantage is obtained from an actual rollout. The lookup assigns every ALFWorld request to ReasoningBank, so its ALFWorld score (38.81) is ReasoningBank's own column. Under online accumulation, that memory was updated with ReasoningBank's own trajectories, and no columns are combined. The lookup's ALFWorld contribution is (38.81−18.28)/3=+6.84(38.81-18.28)/3=+6.84, which accounts for the whole of its +6.58+6.58 gain.

(2) For the per-request policies (π4\pi_4 and the oracle), we agree with the reviewer. They switch substrates within ALFWorld, so on the 134 online requests they are scored on histories that a deployed router would not have produced. π4\pi_4's extra ALFWorld contribution over the lookup (+7.21+7.21 vs. +6.84+6.84) should therefore be read as an offline estimate. The 134 offline-bank requests read a static memory and are not affected. We will add router-in-the-loop ALFWorld rollouts in the revised version.

We also notice that the practical result is narrower than the mixed-stream framing: on the 2,346 QA requests, the lookup and π4\pi_4 score 45.69 and 46.04 against 46.08 for fixed LightRAG (Sec. 6.2). In revision, we will report offline-bank and online ALFWorld requests separately and mark per-request online entries as offline estimates, and modify the Abstract and Sec. 1 to say that the measured gain in our pool is one coarse decision between procedural and QA memory.


Q3.

Thank you for this insightful question. Yes, we have preliminary held-out evidence on LoCoMo, the largest single source (1,975 conversational QA requests over 10 conversations), where the procedural-versus-QA distinction is not involved.

(1) On held-out requests under known memory, π4\pi_4 exceeds the best fixed substrate on LoCoMo by +1.15+1.15 (App. B.3).

(2) On held-out conversations (leave-one-memory-unit-out), π4\pi_4 remains above the best fixed substrate on unseen LoCoMo conversations (Table B.19):

Held-out unitsBest fixedπ4\pi_4 on unseen unitDifference [95% CI]
LoCoMo (10 conversations)30.3630.83+0.47 [-0.8, +1.8]
Long-range understanding (10 corpora)74.6569.01-5.63 [-15.3, +6.3]
Both (20 units)39.2238.47-0.75 [-3.7, +1.8]

The LoCoMo gain is positive under both protocols, though modest in size. Long-range understanding has only 71 requests, and its interval is wide.

(3) Within LoCoMo, the oracle reaches 44.61 against 30.36 for the best fixed substrate. In addition, an oracle restricted to π4\pi_4's top-2 predictions reaches 52.74 against 44.00 for its top-1 choice (Sec. 6.2), i.e., choosing correctly between two candidates per request would recover up to 8.74 points.

Overall, on LoCoMo, request-level routing within a single source is already above the best fixed substrate on both held-out requests and held-out conversations, and we view strengthening it as the main direction for improvement. We will state this explicitly in the revision.

We hope our responses address Reviewer 4gFW's concerns. We remain available for any further clarifications and discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.