- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes EKC-CRL for continual reinforcement learning without replaying historical trajectories. The core approach is as follows: each historical task is represented as a set of expert knowledge vectors; historical experts are combined through two-level routing at the group level and the expert level, while an independent set of current experts is maintained to learn new tasks; a routing balance loss and an expert diversity loss are introduced; when the historical knowledge pool exceeds capacity P, experts are aligned via Hungarian matching and then merged by averaging, thereby keeping the knowledge pool bounded.
Critical strengths
The motivation is clearly articulated, the combination of methods is reasonable, and the writing is relatively complete.
Critical weaknesses
- The reasoning “finer representation granularity → more flexible composition → more effective reuse” has long been a premise rather than a novel finding in the modular learning and MoE literature: Mixture of Experts (Jacobs et al. 1991) has since its inception replaced a single monolithic model with “multiple local experts + gated composition,”
- The performance gains are not statistically significant, and the claim “consistently outperforms” is too strong. In particular, the improvements over the CKA-RL baseline are marginal, and Fig. 4 shows that the proposed method is less efficient than CKA-RL.
- The evaluation benchmark consisting of only two datasets is too narrow. Moreover, the compared baseline CKA-RL was evaluated on the Meta-World dataset, whereas this paper does not conduct such an evaluation.
In my view, this paper requires substantial improvement in terms of novelty, the significance of its empirical gains, and experimental completeness. I therefore regret to recommend rejection.
Questions for authors
-
The necessity of two-level routing is not validated. In Section 4.1, group-level routing and expert-level routing are independently normalized via softmax, and their product is then added to the current-expert term. However, the historical and current terms are normalized independently, leaving their relative scale unconstrained. Thus, the combined weights therefore no longer form a reasonable convex combination. In addition, the advantage of two-level routing over “flattening all historical experts and applying single-level routing” is not demonstrated by ablation (the “Route” condition in Table 2 only tests routing presence/absence and does not distinguish single-level from two-level routing).
-
Implementation details are missing, impairing reproducibility and understanding. Three issues are particularly relevant. First, key design choices are unspecified: whether in Eq. (2) comes from a shared or frozen encoder, whether routing is layer-wise or global, and at which layers () the composition and diversity losses in Eq. (4) are applied. Second, Eq. (2), “” implies state-dependent composition of network weights. If so, recomputing network weights for every state at inference could incur substantial latency, yet Fig. 4 reports an inference latency of only 1.5 ms. If composition is applied only at specific layers, this must be stated explicitly. Third, the parameter scale of a single expert and the range of layers it covers are not provided. Moreover, Fig. 4 reports only memory curves without absolute parameter counts (M), limiting comparability. Please refer to "Critical Weaknesses".
3 comments
I thank the authors for their detailed response. I acknowledge that they have addressed some of my concerns, and I have therefore raised my score by one point. However, I remain concerned about the novelty of the paper, the performance of the proposed method, and the diversity of the datasets used in the experimental comparisons. In my view, these issues mean that the paper does not yet meet the acceptance threshold.
Thank you for your thoughtful feedback and for reconsidering your evaluation. We sincerely appreciate your time and constructive suggestions, and we will continue to improve our work.
We extend our sincere gratitude for your thoughtful and professional insights, which are invaluable in enhancing the quality of our work. In response to your concerns, we report below the explanations and answer your questions accordingly.
1. Relationship to classical mixture-of-experts methods
We agree that using a gate to combine multiple local experts is a classical mixture-of-experts design. EKC-CRL combines expert-level knowledge reuse with the merging of historical knowledge under a fixed pool capacity. Expert groups learned on earlier tasks enter the historical pool for use on subsequent tasks. When the number of historical groups exceeds the capacity, we use the Hungarian algorithm to align independently trained experts in the two groups selected for merging. We then average the parameter offsets of each matched pair separately in each of the two policy-head layers. Subsequent tasks can assign weights to individual experts within a group, while merging operates on pairs of groups.
2. Performance claims and inference overhead
We agree that “consistently outperforms” is too strong. In Table 1, the mean differences between EKC-CRL and CKA-RL across Freeway and SpaceInvaders are 0.0054 for PERF and 0.0199 for FWT. The reported standard deviations describe variation across seeds for each method. As discussed in Section 5.5, maintaining, routing, and composing multiple experts incurs additional overhead. Figure 4 shows that EKC-CRL uses more policy parameter memory and has higher inference latency than CKA-RL. In the later portion of the plotted sequence, the active policy's parameter memory levels off and its inference latency remains relatively stable. We will make this trade-off explicit in the revision.
3. Scope of the evaluation
We evaluate historical knowledge reuse on two Atari continual learning sequences. Freeway and SpaceInvaders contain 8 and 10 task configurations, respectively. The reported comparisons and experimental conclusions are based on these two sequences.
4. Separate normalization and two-level routing
(1) We use two-level routing to reflect the organization of the historical expert pool. The group-level router assigns a total weight to each historical group, and the within-group router distributes that weight among the group's experts. This allows the within-group router to redistribute expert weights while keeping the total weight assigned to the group fixed. The routing ablation in Table 2 measures the observed change from uniform to adaptive weighting.
(2) The historical and current expert sets each form a convex combination of their respective parameter offsets. The policy head adds the resulting two offsets to its base parameters. When historical experts are available, each set's routing weights sum to one, so the two offsets enter the sum with unit coefficients. This separate normalization gives the current experts a dedicated share of routing weight early in training. The effect of each offset on the policy head also depends on its magnitude and direction.
5. Encoder, expert composition, and parameter counts
(1) In Equation (2), hₜ is the output of the CNN encoder shared by the policy head and the router. Each subsequent task initializes this encoder from the previous task's checkpoint and continues updating it during PPO training. The router computes one set of composition weights from the encoder output, and the two linear layers of the policy head share those weights. Expert composition covers the weights and biases of both layers. We compute the diversity loss separately for the current experts' weight offsets in each layer and average the two losses.
(2) At inference, expert composition is applied only to these two linear layers. For each layer, the implementation computes the experts' contributions to the layer output and sums them using the routing weights. This is mathematically equivalent to composing the parameters before computing the linear layer output, as expressed in Equation (2). The implementation can therefore compute the output without explicitly constructing a composed weight matrix for each input.
(3) Each expert spans two linear layers of dimensions 512 → 512 and 512 → A, where A is the number of discrete actions. Including the weights and biases in both layers gives 262,656 + 513A parameters per expert. Figure 4 reports policy parameter memory in MiB.