- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This manuscript focuses on the forward knowledge transfer problem in CRL, specifically targeting how to leverage knowledge from previous tasks to accelerate learning in new tasks under limited interaction budgets, and proposes a heuristic initialization method named NAMI. By selecting the corresponding neuron with the largest norm from previous policies on a neuron-wise basis according to the Manhattan norm of neuron parameters, and recursively maintaining a global initialization network, NAMI is evaluated on Atari and Meta-World environments without incurring additional transfer computational overhead as the number of past tasks increases.
Critical strengths
- The manuscript identifies an important and practically meaningful research problem, focusing on forward transfer efficiency in continual reinforcement learning, which is equally crucial alongside mitigating catastrophic forgetting, and the research motivation is sound.
- The design of the method is very concise and intuitive, demonstrating favorable computational efficiency and implementation feasibility.
- The experimental evaluation covers diverse Atari task sequences as well as Meta-World continuous control scenarios, and compares against a relatively comprehensive set of continual reinforcement learning baselines.
- Beyond comparative experiments, the manuscript attempts to analyze the contributions of different design choices through ablation studies and parameter reset experiments, providing preliminary empirical motivation for the proposed method design.
Critical weaknesses
- There are obvious internal inconsistencies in the experimental results in the manuscript, and I question the credibility of some experimental results. Both Table 1 and Table 3 report the performance of FT-N and NAMI in the Freeway environment, but there are substantial discrepancies between the values reported for the exact same methods.
- The fact that large-norm parameters are important for the performance of original tasks does not automatically imply that these parameters are equally beneficial for learning future new tasks, and they may even induce negative transfer when tasks are unrelated. The manuscript relies on the neuron norm to measure knowledge transferability without sufficient theoretical and causal evidence.
- The mechanism verification experiment in Figure 3 is not clean. When comparing Reset-top with Reset-random, the manuscript only matches the proportion of reset parameters without controlling for the total magnitude of parameter perturbations.
- The function of a neuron depends not only on its incoming weights but also heavily on its connectivity with downstream layers. The manuscript does not clearly specify the concrete boundaries of neuron parameter groups in Equation (9). In addition, because the selection mechanism of NAMI is monotonic, high-norm neurons generated by early tasks may permanently dominate the global initialization network, and the manuscript lacks evaluations on resisting negative transfer under conflicting task sequences or randomized task orders.
- The performance improvements of the manuscript across several primary benchmarks are marginal and lack the support of statistical significance.
- The manuscript lacks several direct and discriminative simple comparison baselines, such as directly selecting a single intact model with the largest overall norm from previous policies or random neuron selection, making it difficult to conclusively demonstrate the incremental benefits of fine-grained neuron reorganization itself.
- The details of the evaluation metrics and the reporting of computational overhead are not sufficiently complete or clear.
Questions for authors
- The authors should first clarify and resolve the severe numerical inconsistency between Table 1 and Table 3 discussed in the first weakness.
- Regarding the second, third, and fifth weaknesses, could the authors provide cleaner empirical evidence?
- Could the authors provide detailed explanations for the fourth and sixth weaknesses?
6 comments
Response to Reviewer KWaZ (2/4)
Response to W2: Effectiveness of Manhattan Norm for Forward Transfer
NAMI is designed as a low cost heuristic rather than a theoretically guaranteed criterion. As discussed in Section 4.3, NAMI-FO and NAMI-SO use first- and second-order information to estimate transferability, potentially accounting for both positive and negative transfer. However, they require additional computation and storage. In contrast, NAMI provides an interaction-free alternative suitable for CRL under limited interaction budgets. To further validate our heuristic, we compare NAMI with NAMI-min (selecting neurons with the smallest Manhattan norms) and NAMI-random (randomly selecting historical neurons). NAMI achieves the highest AP, FP, and FWT among these variants, providing additional empirical support for Manhattan-norm maximization as a simple and computationally efficient heuristic for forward transfer, without claiming a universal guarantee.
Table R6: Ablation study of NAMI on Freeway under different neuron selection strategies and parameter transformations. Results are reported as mean ± standard deviation. Best results are in bold.
| Method | FP (↑) | AP (↑) | FWT (↑) |
|---|---|---|---|
| Baseline | 0.1891 ± 0.3473 | 0.1771 ± 0.3330 | 0.0000 ± 0.0000 |
| Finetune | 0.9287 ± 0.0669 | 0.8406 ± 0.0949 | 0.7544 ± 0.0943 |
| NAMI | 0.9382 ± 0.0560 | 0.8470 ± 0.0857 | 0.7612 ± 0.0871 |
| NAMI-rescaling | 0.7309 ± 0.3499 | 0.6666 ± 0.3334 | 0.5698 ± 0.3074 |
| NAMI-permutation | 0.9016 ± 0.0726 | 0.8106 ± 0.1065 | 0.7213 ± 0.0920 |
| NAMI-min | 0.9299 ± 0.0632 | 0.8392 ± 0.0948 | 0.7525 ± 0.0938 |
| NAMI-random | 0.9302 ± 0.0668 | 0.8415 ± 0.0991 | 0.7581 ± 0.0920 |
| NAMI-L2 | 0.9179 ± 0.0721 | 0.8234 ± 0.1028 | 0.7347 ± 0.0952 |
| NAMI-all | 0.9287 ± 0.0668 | 0.8408 ± 0.0950 | 0.7546 ± 0.0946 |
- NAMI: Our proposed method that selects the neuron group with the largest Manhattan norm across historical policies to initialize the new task policy.
- NAMI-rescaling: Applies function-preserving parameter rescaling (scale=0.5) to historical policies before neuron selection.
- NAMI-permutation: Applies function-preserving neuron permutation (percent=100%) to historical policies before neuron selection.
- NAMI-min: Selects neuron groups with the smallest Manhattan norms instead of the largest.
- NAMI-random: Randomly selects neuron groups from historical policies instead of using Manhattan norms.
- NAMI-L2: Selects neuron groups with the largest L2 norms instead of Manhattan norms.
- NAMI-all: Selects the entire historical policy with the largest overall Manhattan norm instead of performing neuron-wise selection.
Response to W6: Comparison with Simple Baselines
To verify the benefits of fine-grained neuron-wise knowledge selection, we additionally compare NAMI with NAMI-random (random neuron selection) and NAMI-all. As shown in Table R6, NAMI achieves the highest FP, AP, and FWT among all compared methods. In particular, NAMI outperforms both NAMI-random and NAMI-all. These results provide additional empirical evidence that fine-grained neuron-wise recombination offers benefits beyond random selection or whole-network reuse, supporting the effectiveness of NAMI's design.
Response to W3: Controlling the Magnitude of Parameter Perturbations
To address this concern, we additionally evaluate Reset-top with controlled perturbation magnitudes at reset ratios of 10% and 50%. The results show that, at a 50% reset ratio, controlling the perturbation magnitude substantially mitigates the performance degradation caused by Reset-top, improving FP from 0.6596 to 0.9018 and FWT from 0.4715 to 0.7148. This suggests that perturbation magnitude contributes substantially to the degradation observed in the original experiment. Nevertheless, Controlled Reset-top still performs below the FT-N baseline. These findings clarify the limitations of the original reset experiment and motivate our use of the Manhattan norm as a computationally efficient heuristic rather than a theoretically guaranteed measure of transferability.
Table R7: The performance on Freeway with different reset setting.
| Method | Reset Ratio | FP (↑) | FWT (↑) |
|---|---|---|---|
| FT-N (No Reset) | 0% | 0.9287 ± 0.0669 | 0.7544 ± 0.0943 |
| Reset-top | 10% | 0.9234 ± 0.0618 | 0.7438 ± 0.0867 |
| Reset-random | 10% | 0.9138 ± 0.0644 | 0.7342 ± 0.0886 |
| Controlled Reset-top | 10% | 0.8900 ± 0.0950 | 0.7147 ± 0.1056 |
| Reset-top | 50% | 0.6596 ± 0.3066 | 0.4715 ± 0.2209 |
| Reset-random | 50% | 0.8043 ± 0.1611 | 0.6168 ± 0.1287 |
| Controlled Reset-top | 50% | 0.9018 ± 0.0732 | 0.7148 ± 0.0941 |
Response to Reviewer KWaZ (3/4)
Response to W4: Neuron Group Definition and Negative Transfer
As described in the Implementation Details (Section 5.1), following prior work [1,2], we apply NAMI only to the actor network. Specifically, our Atari-based architecture consists of a shared CNN encoder and a two-layer linear actor, with NAMI applied only to the actor. For Meta-World, we use a three-layer linear actor without a CNN encoder, and NAMI is applied to the entire actor. In Eq. (9), each neuron group consists of the weights and bias of an output neuron. Moreover, NAMI recursively maintains a global initialization network rather than combining independently initialized policies, providing consistent parameter correspondence across tasks.
To investigate whether early high-norm neurons permanently dominate the global network, we additionally track the source task distribution of selected neurons throughout continual learning. The results show that the global network is predominantly composed of neurons from recent tasks, while the proportion of early task neurons gradually decreases. This suggests that NAMI does not suffer from permanent domination by early high-norm neurons in this task sequence.
Table R8. Source task distribution of neurons in the global network on Freeway.
| current task | source 0 | source 1 | source 2 | source 3 | source 4 | source 5 | source 6 | source 7 |
|---|---|---|---|---|---|---|---|---|
| 0 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 1 | 8.42% | 91.58% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 2 | 1.72% | 2.81% | 95.47% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| 3 | 0.84% | 1.04% | 14.77% | 83.36% | 0.00% | 0.00% | 0.00% | 0.00% |
| 4 | 0.47% | 0.55% | 2.29% | 8.24% | 88.46% | 0.00% | 0.00% | 0.00% |
| 5 | 0.14% | 0.14% | 0.37% | 0.82% | 2.09% | 96.45% | 0.00% | 0.00% |
| 6 | 0.06% | 0.04% | 0.08% | 0.14% | 0.12% | 2.05% | 97.52% | 0.00% |
| 7 | 0.02% | 0.00% | 0.04% | 0.04% | 0.06% | 0.49% | 3.46% | 95.90% |
Furthermore, we provide per-task forward transfer results on CrossGame, which involves more heterogeneous tasks and potential knowledge conflicts. The results indicate that NAMI can still experience negative transfer under task conflicts, as observed on Tasks 4 and 6. Nevertheless, it achieves stronger positive transfer on several tasks and the highest average FWT among the compared methods. These findings suggest that NAMI can effectively reuse historical knowledge in heterogeneous task sequences, although it does not eliminate negative transfer entirely.
Table R9. Per-task forward transfer on CrossGame.
| Method | Task 0 | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| CKA-RL | 0.0 | 0.6222 | 0.6409 | 0.4733 | -0.3185 | 0.3096 | -0.2003 | 0.131 | 0.2369 ± 0.3553 |
| ReGraMa | 0.0 | 0.545 | 0.5563 | 0.4117 | -0.3537 | 0.4768 | -0.0637 | 0.1872 | 0.2514 ± 0.3219 |
| NAMI | 0.0 | 0.6637 | 0.5617 | 0.7079 | -0.3665 | 0.4946 | -0.1742 | 0.0904 | 0.2825 ± 0.3993 |
References
[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.
[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.
Response to W5: Statistical Significance
To assess statistical reliability, we independently calculate FP and FWT for each of the 10 random seeds using identical task sequences and seed configurations across methods. We then report the paired differences between NAMI and each baseline with 95% confidence intervals (CIs), as detailed in Tables R7–R12 of our response to Reviewer 4yQP.
The results show that NAMI achieves the highest aggregate FP and FWT across all three benchmarks in Table 1, although not all paired improvements are statistically significant at the 95% confidence level. For instance, NAMI shows positive FWT improvements over CKA-RL on Freeway and CompoNet on SpaceInvaders. On CrossGame, the slightly negative mean paired FWT difference against ReGraMa is largely influenced by an unusually poor result at seed 0. Nevertheless, NAMI achieves the highest aggregate FWT in Table 1 (0.2825), which uses a different aggregation procedure. Overall, these results provide additional statistical evidence for NAMI's effectiveness while acknowledging that its improvements over some strong baselines remain modest.
Response to Reviewer KWaZ (1/4)
We thank Reviewer KWaZ for the valuable feedback and for appreciating NAMI's clear motivation, simple and efficient design, and extensive experimental evaluation.
Response to Q1 and W1: Inconsistency Between Tables 1 and 3
The discrepancy arises from the use of different success score thresholds when calculating the evaluation metrics. As described in Section 5.1, our CRL evaluation uses success rates to calculate average performance (AP), final performance (FP), and forward transfer (FWT). However, Atari games such as Freeway, SpaceInvaders, and those in CrossGame are score-based environments without an explicitly defined binary success criterion. Moreover, reward scales vary substantially across games, making direct comparisons based on raw episodic returns inappropriate. Following the evaluation protocol of prior CRL studies [1, 2], we therefore convert episodic returns into success rates using task-specific thresholds, referred to as success scores. For each task, the success score is defined as the average final episodic return across all compared methods, with 10 random seeds per method. During evaluation, a run is considered successful if its averaged episodic return is greater than or equal to the corresponding success score, and unsuccessful otherwise. The resulting success rates are then used to compute AP, FP, and FWT.
Specifically, Table 1 uses success scores computed from all methods included in the overall comparison, whereas Table 3 uses success scores computed from the methods included in the ablation study. Since these two sets of methods differ, the resulting success scores also differ, leading to different numerical values of the evaluation metrics even for the same methods. For transparency, we provide the success scores used in Table 1 for Freeway, SpaceInvaders, and CrossGame below.
Table R1. Success scores used for Freeway in Table 1.
Table R2. Success scores used for SpaceInvaders in Table 1.
Table R3. Success scores used for CrossGame in Table 1.
In contrast, Table 3 originally used the following success scores for Freeway, calculated from the methods included in the ablation study.
Table R4. Success scores originally used for Freeway in Table 3.
To ensure a consistent and fair comparison, we have recalculated all ablation results in Table 3 using exactly the same success scores as Table 1. The updated results are reported below. As shown in Table R5, the recalculated FP and FWT values of FT-N and NAMI are now fully consistent with those reported in Table 1. More importantly, the main conclusions of the ablation study remain unchanged. These results confirm that the numerical discrepancy was caused by inconsistent success-score thresholds rather than differences in the underlying training results. We will revise Table 3 and clarify the success score calculation in Section 5.1 to ensure consistency and reproducibility.
Table R5. Revised ablation study of NAMI on Freeway using the success scores from Table 1.
References
[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.
[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.