acceptodds
Under review as a conference paper at ICLR 2027

Routed Continual Learning

Abstract

Agents deployed at scale repeatedly encounter related problems, and it is natural to let them learn from each other's experience rather than starting over each time. Unfortunately, there is no single way to retain that experience that works well across settings, and no way to know in advance which future problems a given experience will help with. A global history preserves every detail but becomes expensive and mixes together experience that may not belong together; compressing that history into one fixed representation limits the cost but can throw away valuable details. We introduce Routed Continual Learning (Routed CL), an architecture that decides how to partition and retain experience. Past trajectories are grouped into clusters, each of which maintains state using one memory system from a fixed library. Before each run, an LLM-based router either reuses an existing cluster or creates a new one, basing its decision on the problem statement, memory system descriptions, and cluster summaries. We also introduce Context Rewriter, which mediates between the memory system and the agentic loop by augmenting the agent's initial context with the memory system's outputs and updating the selected cluster with information from the trajectory. We evaluate the architecture on a heterogeneous stream that interleaves 301 instances from the six CL-Bench tasks with 97 banking instances from -Bench, for 398 instances total. Across four language models and five matched rollouts, Routed Continual Learning raises normalized reward over a single global in-context learning (ICL) history by 15–30% of the maximum attainable normalized reward, achieves higher mean normalized reward than ICL-only clustering, and reduces estimated API cost by 65–72% relative to Global ICL. These results support approaching continual learning as a routing problem that chooses both a learning mechanism and a state cluster rather than committing to one representation globally.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, LLM agents, agent memory, in-context learning, experience reuse, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JTGTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces routed continual learning, a system where an LLM router picks both a memory mechanism and a cluster of past experience for each new task an agent handles, instead of using one fixed memory setup for everything. They test it on a mix of CL-Bench and banking tasks across four models, and it beats both "no memory" and "one shared memory" baselines while being much cheaper to run.

Critical strengths

I really appreciated the setting that has been overlooked so far. The proposed approach is very simple and works well with several models on the adopted benchmark.

Critical weaknesses
  • Single, author-constructed benchmark. All results rely on one benchmark the authors built themselves (CL-Bench's six tasks interleaved with τ³-Bench banking instances), so it's unclear whether the routing architecture's gains generalize beyond this specific constructed benchmark. This matters because three of the five memory systems (Hierarchical Regression, Hierarchical Discrete-Event State, Persistent-Entity State) are purpose-built for CL-Bench's structured-prediction tasks. On a stream drawn from other agentic benchmarks cited in the paper itself (e.g. WebArena, SWE-Bench-Verified, AppWorld, Mind2Web), Routed CL would most likely collapse to choosing between ICL and ReasoningBank only. The claim that "memory-system selection adds task-specific gains" is therefore demonstrated only on tasks engineered to make that claim true. This is compounded by CL-Bench itself being only an arXiv preprint rather than an established, peer-reviewed benchmark, which makes anchoring the entire evaluation on it more problematic.
  • Router is purely prompt-based. System performance hinges entirely on the router's prompt, provided in the appendix, and the task descriptions it's given appear detailed enough to make routing comparatively easy on this particular benchmark. Given how central the router is to the whole architecture, it would strengthen the paper to measure sensitivity to less detailed or less informative prompts with a robustness/ablation study on prompt specificity.
  • Limited conceptual novelty (less critical). Framing continual-learning memory as a routing problem over a library of heterogeneous representations is not new (cf. MemAgent, MESA, ALMA), and context injection via a prompt-level hook is also well established (MemGPT, Reflexion, ACE, ReasoningBank, Dynamic Cheatsheet). The paper should position its contribution more precisely against this line of work rather than presenting the routing framing itself as the core novelty.
Questions for authors

questions:

  • in fig 2, I wonder if price is the right metric, as prices will change over time and across countries making the experiments not reproducible

suggestions:

  • Calling it continual learning is questionable, as the system is prompt-based. In this sense, there is no learning, only memory management. It's important to clarify this point.
  • include sonnet at it obtains surprisingly good results in CL-Bench
  • I suggest basing evaluation on open-source models to achieve reproducibility
  • It's not clear what part of the pipeline needs a hig/expensive model. Ablations combining different models. For example, taking the best model, opus 5 in this benchmark, we could change the LLM only for the routing to see if the choice of the model has impact on routing. We need to understand whether we need for the task itself or the routing.
1 comment

All results rely on one benchmark the authors built themselves (CL-Bench's six tasks interleaved with τ³-Bench banking instances)

We consider this construction of a composite benchmark a strength of the paper. A typical benchmark on which models or agentic harnesses are evaluated consists of independent tasks; the literature on continuous learning observes that in practice tasks arrive sequentially, which creates opportunities for amortizing effort and accumulating knowledge. We bring the evaluation closer to reality by interleaving tasks from disjoint streams of continuous learning challenge sets.

Not only is this setup more practically relevant, it advances our understanding of the original benchmark. Asawa et al., who introduced the CL Bench, report that “naive ICL outperforms systems dedicated to memory management.” In our evaluation, this advantage disappears when tasks from different streams are interleaved. These results suggest that the original finding is conditional on the task order preferencing in-context learning, rather than reflecting a general phenomenon.

This is compounded by CL-Bench itself being only an arXiv preprint rather than an established, peer-reviewed benchmark, which makes anchoring the entire evaluation on it more problematic.

We augment CL-Bench with τ³-Bench Banking and report combined results together with per-stream metrics (Appendix C.5). We use these benchmarks to empirically validate our proposal, rather than presenting benchmark performance improvements as a distinct contribution.

Calling it continual learning is questionable, as the system is prompt-based. In this sense, there is no learning, only memory management. It's important to clarify this point.

We follow the established terminology that considers in-context learning (ICL) a form of learning (see, for instance, https://arxiv.org/pdf/2301.00234, https://arxiv.org/pdf/2410.23042).

Q: In fig 2, I wonder if price is the right metric, as prices will change over time and across countries making the experiments not reproducible

The price is a combined metric, aggregating unevenly priced input, cache read, cache write, and output tokens (reported in Table 7). Figure 2 is most useful for intra-model comparisons, as the relative costs of different models do fluctuate.

We appreciate the suggestions that provide useful directions for further exploration.

It's not clear what part of the pipeline needs a high/expensive model. Ablations combining different models. For example, taking the best model, opus 5 in this benchmark, we could change the LLM only for the routing to see if the choice of the model has impact on routing. We need to understand whether we need for the task itself or the routing.

Our results suggest that stronger models make better routing decisions. Opus 5 and Sol match Ground-truth assignment, in both cluster and memory system, for 91% and 89% of source tasks across rollouts, compared with 71% for Opus 4.7 and 66% for Gemini 3.7 Flash (Table 3).

Since each routing decision is a choice among a handful of options (an existing cluster, or a new cluster with a selected memory system), a specialized model (such as Jev or Gemini Decision) may be enough to serve as the router.

Reviewer_YJ9NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper presents Routed Continual Learning, a framework for organizing persistent experience in LLM agents across heterogeneous tasks. An LLM router selects from a fixed library of memory systems and assigns each incoming instance to an existing or newly created state-sharing cluster. The selected memory system supplies context to the agent and updates its state after execution. The framework is evaluated on interleaved tasks from CL-Bench and τ³-Bench, measuring task performance and LLM API cost.

Critical strengths

Practical insights for agent memory design.

The paper shows that the strong performance of naive ICL reported on homogeneous CL-Bench task streams does not carry over to heterogeneous interleaving: Global ICL achieves only a +0.015 mean gain relative to the matched stateless baseline. Routing histories into separate clusters substantially improves performance across all four evaluated models. The full Routed CL system also reduces estimated LLM API costs by 65–72% relative to Global ICL (Tables 1, 4, and 6). These findings provide useful practical guidance on organizing persistent memory for agents operating across heterogeneous tasks.

Critical weaknesses

W1. Limited novelty and uncertain generality.

My main concern is the originality and depth of the contribution. The paper asks how to represent experience and which instances should share state, but its answer primarily combines a fixed library of manually designed memory systems with LLM-based selection and clustering. Although this integration has practical value, the paper does not clearly establish a substantive methodological advance or derive insights beyond the expected benefits of selecting suitable representations and separating unrelated histories. The empirical breakdown reinforces this concern: partitioning histories improves normalized reward by 0.117–0.241, whereas access to multiple memory systems adds 0.002–0.076 (Section 6; Tables 1 and 5). These results demonstrate utility, but do not sufficiently distinguish the contribution of the proposed joint routing design from that of selective history access and specialized memory modules. Similarly, the deterioration of Global ICL under heterogeneous interleaving is informative, but the absence of consistent clustering gains for ReasoningBank suggests that this finding depends on how memories are accessed, limiting its generality.

The close correspondence between the numerical memory systems and the development tasks further constrains the contribution. HR, HDE, and PES closely match Sales, Cohort, and Spectrum, respectively, and the library was developed through exploration on CL-Bench itself. Freezing the design before confirmatory evaluation does not establish transfer beyond the tasks that informed its construction. Consequently, the current evidence leaves unclear whether the framework offers a broadly transferable approach to organizing experience or may primarily reflect specialized modules well matched to the development benchmark. Evaluation on unseen tasks with the library held fixed would help establish its scope, but my central concern remains the limited methodological and conceptual contribution, rather than effectiveness alone.

W2. Presentation and organization.

The presentation makes the central contribution difficult to identify. Although the Introduction states the motivating questions, substantial space is devoted to deployment details, routing procedures, memory components, and benchmark descriptions before the research gap and the rationale for the proposed design are sufficiently developed. This interrupts the progression from problem to challenge to contribution. The Introduction would benefit from a more focused explanation of what existing approaches fail to address, why the proposed combination resolves that limitation, and what new understanding the paper provides, with implementation and evaluation details moved to later sections.

Questions for authors

None.

1 comment

My main concern is the originality and depth of the contribution.

We recap the main contributions of the paper:

  1. Identifying a gap in the literature and public benchmarks with significant upside potential for large-scale agentic deployments.

  2. Introducing a modular architecture addressing this gap comprising several abstractions that can be independently implemented, incrementally deployed, and iterated upon.

  3. Proposing a methodology of evaluating an implementation on interweaving streams of heterogeneous benchmarks.

  4. Demonstrating effectiveness of one such implementation across multiple application domains and models.

We argue that these steps in toto advance the state-of-the-art and provide a useful foundation for future work.

the absence of consistent clustering gains for ReasoningBank suggests that this finding depends on how memories are accessed

We agree with this observation, which is precisely the point of including the table. Indeed, ReasoningBank is a strict improvement on ICL, and yet for some application domains it is suboptimal compared to other memory systems. The combination of memory systems with diverse capabilities and a routing algorithm performs better than either of them alone.

The close correspondence between the numerical memory systems and the development tasks further constrains the contribution.

To validate generalizability, we performed additional experiments on the recently released ReScopeBench benchmark (see the comment here). In order to reduce the risk of overfitting and better approximate real-world constraints, we did not adjust memory-system settings or customize our prompts, which are reproduced in the Appendix. Where task-specific attributes need to be mapped to memory-system schemas, a generic adapter performs the necessary translation.

Presentation and organization

We restructured the introduction and other sections of the paper to address the feedback.

Reviewer_zutJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Routed Continual Learning as a method to learn from experience over stream of tasks by creating clusters of memory systems and routing to them. Concretely, before each new instance of tasks, the router decides to create a new cluster with a memory system selected from a library of five systems or reuse an existing cluster. On a stream of 396 instances it the ICL baselines and global memory-based baseline.

Critical strengths

The problem of continual learning with multiple memory system types is interesting. The authors have done a good job at careful designing of the evaluation protocol.

Critical weaknesses

1- The claim that routing memory systems helps would require more evidence. From the results (Figure 3 and Table 1), it looks like most of the gain comes from clustering rather than memory systems (Router CL and Router ICL are almost within the statistical noise from each other with even some negative impact on Gemini and Opus 4.7. Also, the library of memory systems are chosen based on same stream of tasks for evaluation. Finally, there is no retrieval based or embedding-cluster based baselines to compare with.

2- The paper is hard to follow and most of the key results are in the appendix. The writing is generic with repetitive patterns. Table 1 never discussed in the text and the quantities the text cites (paired differences) are in appendix tables and figures, so the main paper cannot be checked on its own. I would suggest the authors to reduce the verbosity of the paper and move key results from the appendix into the main text.

Questions for authors

1- How the learned clusters/memory systems generalize to held-out stream of instances?

2- Is it possible to run the RCL approach without the exploratory phase and access to the tasks for library design?

1 comment

The claim that routing memory systems helps would require more evidence.

Figure 2 presents the Pareto curve of accuracy and cost. It shows clear and consistent gains across all models (with Opus 4.7 Routed CL being a sole outlier) of switching from Global ICL to Routed ICL to Routed CL. (The cost axis is log scale.) Between these architectures the gains are regret-free as the reward and cost improve or stay stable. In practice, both metrics matter; we expect that large scale deployments may prioritize multiplicative savings in token costs over incremental gains in task performance.

most of the gain comes from clustering rather than memory systems

The central thesis of the paper is that there is no single memory system that serves all domains well but routing between several memory systems can be done efficiently. In fact, Figure 3 demonstrates exactly this phenomenon: ReasoningBank deployed across all tasks degrades costs without improving performance. Routing some tasks to different memory systems leads to significant gains in the reward function.

the library of memory systems are chosen based on same stream of tasks for evaluation

We draw a distinction between selecting memory systems knowing the mixture of the task types vs optimizing for the individual tasks. Choosing memory systems in a manner that fits application domains is fully aligned with our core proposal. We envision a setting of agents acting on a stream of heterogeneous tasks sharing longitudinal patterns across extended time periods. Dynamic selection, deployment, and monitoring performance of memory systems become parts of the operational workflow.

Alternative approaches explored by related work, such as tuning memory system parameters or optimizing routing prompts, are indeed prone to overfitting and require separation between train and test datasets.

The paper is hard to follow and most of the key results are in the appendix.

We appreciate the reviewer’s interest in results presented in the appendix and revised the presentation to incorporate more supportive evidence in the main text.

  1. How the learned clusters/memory systems generalize to held-out stream of instances?

The clustering and memory system assignment procedure is general purpose and was not tuned to any of the tasks.

  1. Is it possible to run the RCL approach without the exploratory phase and access to the tasks for library design?

To further demonstrate generalizability of our approach we conducted a fresh set of experiments on a recently released benchmark: ReScopeBench (see the comment here). With the router, prompts, and memory-system library unchanged, Routed CL outperformed Global ICL for all four models, by 0.23 to 0.42 in mean reward at 27–60% lower cost.

Finally, there is no retrieval based or embedding-cluster based baselines to compare with.

Our evaluation does include a retrieval-based baseline. Like Mem0, which the original CL-Bench paper evaluated, ReasoningBank retrieves stored memories by embedding similarity. Global RB (Appendix C.2, Figure 3) keeps a single ReasoningBank store for the whole stream, and Routed CL outperforms it by 0.189 to 0.358 in normalized reward for every model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New benchmark

To address the reviewers comments about generalizability of our contribution, we conducted a new set of experiments on a benchmark released after the ICLR submission deadline. ReScopeBench (https://arxiv.org/abs/2611.05450v1) contains 352 instances (in our terminology) across 18 tasks and nine application domains. Applying the same evaluation methodology as before, we obtain the following:

ModelStateless ICLGlobal ICLRouted CL
Gemini 3.7 Flash0.225 [0.220, 0.230] ($15)0.325 [0.317, 0.335] ($76)0.653 [0.590, 0.702] ($56)
GPT-5.6 Sol0.219 [0.215, 0.222] ($24)0.355 [0.350, 0.358] ($352)0.707 [0.695, 0.723] ($141)
Opus 4.70.129 [0.125, 0.134] ($72)0.294 [0.292, 0.297] ($525)0.522 [0.197, 0.688] ($295)
Opus 50.250 [0.246, 0.255] ($127)0.341 [0.337, 0.344] ($562)0.764 [0.754, 0.772] ($386)

Each cell shows the mean reward over three rollouts, the range across rollouts in brackets, and the mean estimated API cost per rollout in parentheses. As with τ³-Bench, we report the benchmark’s native reward, which lies in [0, 1]. We used the router, prompts, and memory-system library from the paper without changes. Given the rebuttal timeline, we ran three rollouts instead of five and did not run Routed ICL.

Routed CL outperforms Global ICL for every model by 0.23 to 0.42 in mean reward while costing 27–60% less, and it wins in 11 of the 12 individual rollouts. The exception is one Opus 4.7 rollout in which the router assigned ReasoningBank to every cluster and scored 0.197, compared with 0.681 and 0.688 for the other two rollouts.

The gap between Routed CL and Global ICL grows as the stream progresses. On the first instance of each task, Routed CL is no better than Stateless ICL (0.12–0.27), but from the 17th instance onward it reaches 0.66–0.88. Global ICL peaks around a task’s second to fourth instance (0.32–0.41) and then slips to 0.29–0.34, presumably because its single 100K-token history is mostly taken up by the other 17 tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.