acceptodds
Under review as a conference paper at ICLR 2027

An Expert Knowledge Composition Method for Continual Reinforcement Learning

Abstract

Continual reinforcement learning requires an agent to continuously adapt to sequentially arriving tasks while retaining and reusing previously acquired knowledge. However, when past interaction data are no longer accessible, effectively organizing historical knowledge, identifying transferable knowledge components, and preventing the knowledge repository from growing unboundedly with the number of tasks remain key challenges. To address these issues, we propose an expert knowledge composition method for continual reinforcement learning (EKC-CRL). The proposed method represents each historical task as a group of expert knowledge vectors, enabling a more fine-grained characterization of task-specific knowledge than a single task-level representation. When adapting to a new task, the model performs two-level expert routing to select and compose relevant historical experts, while maintaining an independent set of current experts to acquire task-specific new knowledge. We further introduce routing-balancing and expert-diversity regularization to improve expert utilization and reduce knowledge redundancy across experts. As tasks arrive sequentially, learned experts are incorporated into a historical knowledge pool. Once the pool reaches its capacity limit, Hungarian matching is employed to align experts across tasks, followed by the merging of similar knowledge groups. In this way, the model can achieve flexible reuse of historical knowledge and effective new-task learning without storing past interaction trajectories. Experimental results demonstrate that our method consistently outperforms state-of-the-art approaches across multiple continual reinforcement learning benchmarks.

Primary area:
reinforcement learning
Keywords:
Continual Reinforcement Learning, Knowledge Composition, Dynamic Routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WvDtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EKC-CRL for continual reinforcement learning without replaying historical trajectories. The core approach is as follows: each historical task is represented as a set of expert knowledge vectors; historical experts are combined through two-level routing at the group level and the expert level, while an independent set of current experts is maintained to learn new tasks; a routing balance loss and an expert diversity loss are introduced; when the historical knowledge pool exceeds capacity P, experts are aligned via Hungarian matching and then merged by averaging, thereby keeping the knowledge pool bounded.

Critical strengths

The motivation is clearly articulated, the combination of methods is reasonable, and the writing is relatively complete.

Critical weaknesses
  1. The reasoning “finer representation granularity → more flexible composition → more effective reuse” has long been a premise rather than a novel finding in the modular learning and MoE literature: Mixture of Experts (Jacobs et al. 1991) has since its inception replaced a single monolithic model with “multiple local experts + gated composition,”
  2. The performance gains are not statistically significant, and the claim “consistently outperforms” is too strong. In particular, the improvements over the CKA-RL baseline are marginal, and Fig. 4 shows that the proposed method is less efficient than CKA-RL.
  3. The evaluation benchmark consisting of only two datasets is too narrow. Moreover, the compared baseline CKA-RL was evaluated on the Meta-World dataset, whereas this paper does not conduct such an evaluation.

In my view, this paper requires substantial improvement in terms of novelty, the significance of its empirical gains, and experimental completeness. I therefore regret to recommend rejection.

Questions for authors
  1. The necessity of two-level routing is not validated. In Section 4.1, group-level routing αg\alpha_g and expert-level routing βg,eβ_{g,e} are independently normalized via softmax, and their product is then added to the current-expert term. However, the historical and current terms are normalized independently, leaving their relative scale unconstrained. Thus, the combined weights therefore no longer form a reasonable convex combination. In addition, the advantage of two-level routing over “flattening all historical experts and applying single-level routing” is not demonstrated by ablation (the “Route” condition in Table 2 only tests routing presence/absence and does not distinguish single-level from two-level routing).

  2. Implementation details are missing, impairing reproducibility and understanding. Three issues are particularly relevant. First, key design choices are unspecified: whether hth_t in Eq. (2) comes from a shared or frozen encoder, whether routing is layer-wise or global, and at which layers (LAL_A) the composition and diversity losses in Eq. (4) are applied. Second, Eq. (2), “θk(ht)θ_k(h_t)” implies state-dependent composition of network weights. If so, recomputing network weights for every state at inference could incur substantial latency, yet Fig. 4 reports an inference latency of only 1.5 ms. If composition is applied only at specific layers, this must be stated explicitly. Third, the parameter scale of a single expert and the range of layers it covers are not provided. Moreover, Fig. 4 reports only memory curves without absolute parameter counts (M), limiting comparability. Please refer to "Critical Weaknesses".

3 comments

We extend our sincere gratitude for your thoughtful and professional insights, which are invaluable in enhancing the quality of our work. In response to your concerns, we report below the explanations and answer your questions accordingly.

1. Relationship to classical mixture-of-experts methods

We agree that using a gate to combine multiple local experts is a classical mixture-of-experts design. EKC-CRL combines expert-level knowledge reuse with the merging of historical knowledge under a fixed pool capacity. Expert groups learned on earlier tasks enter the historical pool for use on subsequent tasks. When the number of historical groups exceeds the capacity, we use the Hungarian algorithm to align independently trained experts in the two groups selected for merging. We then average the parameter offsets of each matched pair separately in each of the two policy-head layers. Subsequent tasks can assign weights to individual experts within a group, while merging operates on pairs of groups.

2. Performance claims and inference overhead

We agree that “consistently outperforms” is too strong. In Table 1, the mean differences between EKC-CRL and CKA-RL across Freeway and SpaceInvaders are 0.0054 for PERF and 0.0199 for FWT. The reported standard deviations describe variation across seeds for each method. As discussed in Section 5.5, maintaining, routing, and composing multiple experts incurs additional overhead. Figure 4 shows that EKC-CRL uses more policy parameter memory and has higher inference latency than CKA-RL. In the later portion of the plotted sequence, the active policy's parameter memory levels off and its inference latency remains relatively stable. We will make this trade-off explicit in the revision.

3. Scope of the evaluation

We evaluate historical knowledge reuse on two Atari continual learning sequences. Freeway and SpaceInvaders contain 8 and 10 task configurations, respectively. The reported comparisons and experimental conclusions are based on these two sequences.

4. Separate normalization and two-level routing

(1) We use two-level routing to reflect the organization of the historical expert pool. The group-level router assigns a total weight to each historical group, and the within-group router distributes that weight among the group's experts. This allows the within-group router to redistribute expert weights while keeping the total weight assigned to the group fixed. The routing ablation in Table 2 measures the observed change from uniform to adaptive weighting.

(2) The historical and current expert sets each form a convex combination of their respective parameter offsets. The policy head adds the resulting two offsets to its base parameters. When historical experts are available, each set's routing weights sum to one, so the two offsets enter the sum with unit coefficients. This separate normalization gives the current experts a dedicated share of routing weight early in training. The effect of each offset on the policy head also depends on its magnitude and direction.

5. Encoder, expert composition, and parameter counts

(1) In Equation (2), hₜ is the output of the CNN encoder shared by the policy head and the router. Each subsequent task initializes this encoder from the previous task's checkpoint and continues updating it during PPO training. The router computes one set of composition weights from the encoder output, and the two linear layers of the policy head share those weights. Expert composition covers the weights and biases of both layers. We compute the diversity loss separately for the current experts' weight offsets in each layer and average the two losses.

(2) At inference, expert composition is applied only to these two linear layers. For each layer, the implementation computes the experts' contributions to the layer output and sums them using the routing weights. This is mathematically equivalent to composing the parameters before computing the linear layer output, as expressed in Equation (2). The implementation can therefore compute the output without explicitly constructing a composed weight matrix for each input.

(3) Each expert spans two linear layers of dimensions 512 → 512 and 512 → A, where A is the number of discrete actions. Including the weights and biases in both layers gives 262,656 + 513A parameters per expert. Figure 4 reports policy parameter memory in MiB.

I thank the authors for their detailed response. I acknowledge that they have addressed some of my concerns, and I have therefore raised my score by one point. However, I remain concerned about the novelty of the paper, the performance of the proposed method, and the diversity of the datasets used in the experimental comparisons. In my view, these issues mean that the paper does not yet meet the acceptance threshold.

Thank you for your thoughtful feedback and for reconsidering your evaluation. We sincerely appreciate your time and constructive suggestions, and we will continue to improve our work.

Reviewer_zBLBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EKC-CRL, a continual reinforcement learning method that represents historical knowledge as expert parameter offsets and reuses them through hierarchical routing. Separate current-task experts, trained with balancing and diversity regularization, support adaptation. Hungarian matching and expert merging ensure that the size of the historical knowledge pool remains controllable. Experiments on the Atari Freeway and Space Invaders task sequences show that this method improves both final performance and forward transfer compared with the baseline methods evaluated.

Critical strengths
  1. Frozen historical parameter offsets and independent trainable experts provide a new approach to achieving knowledge reuse and adapting to new tasks.
  2. The paper reports means and standard deviations over ten seeds, achieving the highest mean PERF/FWT on both benchmarks, alongside component ablations and sensitivity analyses.
Critical weaknesses
  1. The authors need to clarify whether the previous tasks were evaluated using the final policy, a task-specific router, or saved checkpoints. This distinction is crucial for explaining knowledge retention and memory efficiency, as freezing historical expert parameters does not necessarily preserve the corresponding policies.
  2. Although Table 1 reports results for ten initial values, the average improvement relative to CKA-RL is relatively small: 0.0054 for PERF and 0.0199 for FWT. Because these differences lack uncertainty estimates, it is difficult to assess their statistical reliability.
  3. The advantages of fine-grained expert combinations do not stem solely from increased capacity and regularization. Increasing the number of experts also boosts model capacity, but this paper does not provide a comparison of capacity-matched models.
  4. Under the assumptions of the norm and cosine similarity defined in Lemma 2, bounds are established for perturbations to the original parameter offsets; however, the algorithm itself does not enforce these assumptions. Therefore, these bounds do not prove that the selected merging operation preserves the policy’s performance.
Questions for authors
  1. Guo et al. (2026) concerns graph anomaly detection rather than replay-based CRL. Please verify their relevance to the associated claims and correct any mismatched citations.
  2. Please clarify Figure 3’s environment, seeds, and aggregation, and explain how Eq. (4) handles E=1E=1.
1 comment

We extend our sincere gratitude for your thoughtful and professional insights, which are invaluable in enhancing the quality of our work. In response to your concerns, we report below the explanations and answer your questions accordingly.

1. PERF computation and task checkpoints

We follow the PERF computation in the public Atari implementation of CKA-RL. For each task, we derive a success-rate curve from the returns logged during training, average the last 100 points of the processed curve, and then average these values across tasks. The PERF results in Table 1 therefore summarize each task's performance near the end of its own training. At the end of each task, we also save its policy head and encoder, along with the router parameters and configuration where applicable. These checkpoints can restore the policy at the end of the corresponding task.

2. Mean differences relative to CKA-RL and their uncertainty

Table 1 compares the two methods using the same 10 random seeds. EKC-CRL has higher mean PERF and FWT than CKA-RL on both Freeway and SpaceInvaders. The mean differences across sequences are 0.0054 and 0.0199, respectively. The reported standard deviations describe seed-to-seed variation for each method. We interpret the gains as observed mean differences on the two evaluated sequences.

3. Number of experts, model capacity, and routing ablation

Figure 3(a) compares configurations with one or more experts per group. The default four-expert configuration achieves higher PERF and FWT than the single-expert configuration. This comparison varies both the number of experts and the parameter count. Table 2 examines routing with the same expert configuration and both regularizers disabled. On Freeway, adaptive routing yields mean PERF and FWT of 0.7437 and 0.6154, respectively, compared with 0.7348 and 0.6133 for uniform routing. These results quantify the observed mean differences between the two routing schemes at a fixed expert configuration.

4. Scope of Lemma 2

The algorithm matches experts using normalized descriptors and selects groups for merging based on the mean similarity of the matched pairs. Lemma 2 analyzes the perturbation caused by averaging the original parameter offsets of one matched pair. Under its assumptions on the norms, their difference, and cosine similarity, the lemma bounds the parameter-space distance from the merged expert to each original expert. We will clarify that this is a conditional bound on parameter perturbation and discuss policy performance in terms of the experimental results.

5. Citation of Guo et al. (2026)

Thank you for pointing out the citation mismatch. We will remove the citation to Guo et al. (2026).

6. Experimental setup for Figure 3 and the case E = 1

Both panels of Figure 3 use the Freeway continual learning sequence, with the same 10 random seeds and aggregation procedure as Table 1. For each hyperparameter configuration, we compute PERF and FWT separately for each seed and then average across seeds to obtain the plotted values. For E = 1, each group contains a single expert, so the implementation sets the diversity loss to zero. The expression in Equation (4), whose denominator contains E(E − 1), is evaluated only for E > 1.

Reviewer_ToZoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes EKC-CRL for continual reinforcement learning. It extends CKA-RL by representing each task with several expert vectors instead of one task vector. A state-dependent router combines historical and current experts. The method also uses a diversity loss and Hungarian matching when the expert pool is full. Experiments are conducted on different modes of Freeway and Space Invaders.

Critical strengths
  1. The idea of representing a task with several experts is reasonable. It may capture more diverse behaviors than a single task vector.

  2. The authors include several continual RL baselines and provide supplementary code. The Hungarian matching step is also implemented across layers rather than independently for every layer.

Critical weaknesses

1.The novelty over CKA-RL method is limited. The method mainly replaces each task vector with multiple experts and adds routing, diversity regularization, and Hungarian matching. Much of the submitted code is also directly based on the official CKA-RL implementation.

2.The experimental support is weak. The evaluation covers only two Atari games, and the gains over CKA-RL are small compared with the reported variation. No statistical significance test is provided.

3.The routing design is questionable. Historical and current experts are normalized separately, which forces both sides to contribute equal total weight. The balance loss also applies only to current experts.

Questions for authors

1.What is the main conceptual contribution beyond replacing each CKA-RL task vector with several expert vectors and adding a router? Please provide a more direct comparison and controlled ablations against CKA-RL.

2.Why is the reproduced CKA-RL performance substantially lower than the results reported in the original CKA-RL paper? Can the authors provide identical-configuration comparisons and statistical tests for the improvement over CKA-RL?

3.Why are the historical and current expert weights normalized separately? Can the authors show an ablation in which the model learns their relative contribution?

3 comments

We extend our sincere gratitude for your thoughtful and professional insights, which are invaluable in enhancing the quality of our work. In response to your concerns, we report below the explanations and answer your questions accordingly.

1. Relationship to CKA-RL and contribution of EKC-CRL

To compare the two methods under the same training setup, we used the Atari PPO training framework from the public CKA-RL implementation. EKC-CRL contributes a way to reuse and merge knowledge from previous tasks. CKA-RL reuses knowledge at the level of individual task vectors. EKC-CRL represents the knowledge from each previous task as a group of expert knowledge vectors. The router uses the encoder output to assign composition weights to experts within each group. When the number of historical expert groups exceeds the pool capacity, we use the Hungarian algorithm to align the experts in the two groups selected for merging and then merge the groups according to the resulting correspondence. This design combines expert-level knowledge reuse with merging under a fixed pool capacity.

2. CKA-RL reproduction setup and interpretation of the results

The original CKA-RL paper lists a PPO clipping coefficient of 0.2. The public implementation we used for reproduction defaults to 0.1. In Table 1, both CKA-RL and EKC-CRL use 0.1, the same 10 random seeds, the same hardware, and the same results processing pipeline. The public script available at the time calculated a common success threshold for each task from the returns of the methods being compared. We therefore ran all comparison methods. The clipping coefficient and the procedure for computing success rates are relevant when interpreting differences between the reported results. Our reproduced CKA-RL scores are lower than those in the original paper for PERF and FWT on SpaceInvaders and for FWT on Freeway. On Freeway, the reproduced PERF is 0.7994, compared with 0.7923 in the original paper. Across the two Atari sequences, EKC-CRL exceeds CKA-RL by 0.0054 in mean PERF and 0.0199 in mean FWT. The standard deviations in Table 1 describe the variation across seeds for each method. We interpret the two gains as observed mean differences under the shared experimental setup.

3. Separate normalization and the load-balancing loss

When historical experts are available, the routing weights sum to one separately for the historical experts and the current experts. The router uses the encoder output to adjust the weight distribution within each set. Separate normalization gives the current experts a dedicated share of routing weight early in training so that they can learn from interactions with the new task. The effect of each set on the policy head also depends on its weighted parameter offsets. The load-balancing loss encourages the router to distribute weight across multiple current experts within a training batch. Historical experts have already been trained on earlier tasks, so the router can concentrate historical routing weight on the experts most relevant to the current input.

The authors clarified the differences from CKA-RL and the implementation details. This addresses some of my concerns, so I am raising my score from 1 to 2. However, the novelty remains limited, and the benefits of the added components are not sufficiently demonstrated. I wish the authors the best with their continued work.

Thank you for your thoughtful feedback and for reconsidering your evaluation. We sincerely appreciate your time and constructive suggestions, and we will continue to improve our work.

Comments on the paper

1 comment
Area_Chair_n3U8OpenReview ↗

Reminder: Author–Reviewer Discussion Deadline

Dear Reviewers,

This is a friendly reminder that the author–reviewer discussion period for ICLR 2027 will end on November 18, 2026 (AOE).

As the deadline approaches, I encourage you to actively engage in discussions with the authors and carefully consider their rebuttal responses. Please address any remaining concerns, seek further clarification where necessary, and update your assessments if appropriate.

Your timely and constructive participation is essential to ensuring a fair and well-informed evaluation.

Thank you very much for your time and effort!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.