acceptodds
Under review as a conference paper at ICLR 2027

NAMI: A Heuristic Forward Transfer Approach for Continual Reinforcement Learning

Abstract

Continual reinforcement learning (CRL) requires a reinforcement learning agent to adapt rapidly to new tasks by reusing knowledge acquired from previous ones, a capability known as forward transfer. Existing architecture-based methods, which typically maintain task-specific modules and reuse them for future tasks, often perform forward transfer at the level of entire networks. While effective, such coarse-grained transfer entangles reusable and task-specific modules, making it difficult to reuse prior knowledge flexibly under strict interaction budgets. In this paper, we propose Neuron-wise mAnhattan-norm Maximization Initialization (NAMI), a simple and efficient strategy for forward transfer in CRL. Instead of transferring the entire policy networks, NAMI initializes each neuron of a new task policy network by selecting the corresponding neuron with the largest Manhattan norm (i.e., the maximum sum of absolute weights) from the previously learned policy networks. This converts model-level transfer into neuron-level selection, allowing the new policy to inherit salient reusable components from different prior tasks while remaining a single compact network whose inference cost is independent of the number of previous tasks. Extensive experiments show that NAMI achieves superior performance over various state-of-the-art methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Reinforcement Learning, Continual Reinforcement Learning, Knowledge Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KWaZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This manuscript focuses on the forward knowledge transfer problem in CRL, specifically targeting how to leverage knowledge from previous tasks to accelerate learning in new tasks under limited interaction budgets, and proposes a heuristic initialization method named NAMI. By selecting the corresponding neuron with the largest norm from previous policies on a neuron-wise basis according to the Manhattan norm of neuron parameters, and recursively maintaining a global initialization network, NAMI is evaluated on Atari and Meta-World environments without incurring additional transfer computational overhead as the number of past tasks increases.

Critical strengths
  1. The manuscript identifies an important and practically meaningful research problem, focusing on forward transfer efficiency in continual reinforcement learning, which is equally crucial alongside mitigating catastrophic forgetting, and the research motivation is sound.
  2. The design of the method is very concise and intuitive, demonstrating favorable computational efficiency and implementation feasibility.
  3. The experimental evaluation covers diverse Atari task sequences as well as Meta-World continuous control scenarios, and compares against a relatively comprehensive set of continual reinforcement learning baselines.
  4. Beyond comparative experiments, the manuscript attempts to analyze the contributions of different design choices through ablation studies and parameter reset experiments, providing preliminary empirical motivation for the proposed method design.
Critical weaknesses
  1. There are obvious internal inconsistencies in the experimental results in the manuscript, and I question the credibility of some experimental results. Both Table 1 and Table 3 report the performance of FT-N and NAMI in the Freeway environment, but there are substantial discrepancies between the values reported for the exact same methods.
  2. The fact that large-norm parameters are important for the performance of original tasks does not automatically imply that these parameters are equally beneficial for learning future new tasks, and they may even induce negative transfer when tasks are unrelated. The manuscript relies on the neuron L1L_1 norm to measure knowledge transferability without sufficient theoretical and causal evidence.
  3. The mechanism verification experiment in Figure 3 is not clean. When comparing Reset-top with Reset-random, the manuscript only matches the proportion of reset parameters without controlling for the total magnitude of parameter perturbations.
  4. The function of a neuron depends not only on its incoming weights but also heavily on its connectivity with downstream layers. The manuscript does not clearly specify the concrete boundaries of neuron parameter groups in Equation (9). In addition, because the selection mechanism of NAMI is monotonic, high-norm neurons generated by early tasks may permanently dominate the global initialization network, and the manuscript lacks evaluations on resisting negative transfer under conflicting task sequences or randomized task orders.
  5. The performance improvements of the manuscript across several primary benchmarks are marginal and lack the support of statistical significance.
  6. The manuscript lacks several direct and discriminative simple comparison baselines, such as directly selecting a single intact model with the largest overall norm from previous policies or random neuron selection, making it difficult to conclusively demonstrate the incremental benefits of fine-grained neuron reorganization itself.
  7. The details of the evaluation metrics and the reporting of computational overhead are not sufficiently complete or clear.
Questions for authors
  1. The authors should first clarify and resolve the severe numerical inconsistency between Table 1 and Table 3 discussed in the first weakness.
  2. Regarding the second, third, and fifth weaknesses, could the authors provide cleaner empirical evidence?
  3. Could the authors provide detailed explanations for the fourth and sixth weaknesses?
6 comments
AuthorsOpenReview ↗

Response to Reviewer KWaZ (1/4)

We thank Reviewer KWaZ for the valuable feedback and for appreciating NAMI's clear motivation, simple and efficient design, and extensive experimental evaluation.

Response to Q1 and W1: Inconsistency Between Tables 1 and 3

The discrepancy arises from the use of different success score thresholds when calculating the evaluation metrics. As described in Section 5.1, our CRL evaluation uses success rates to calculate average performance (AP), final performance (FP), and forward transfer (FWT). However, Atari games such as Freeway, SpaceInvaders, and those in CrossGame are score-based environments without an explicitly defined binary success criterion. Moreover, reward scales vary substantially across games, making direct comparisons based on raw episodic returns inappropriate. Following the evaluation protocol of prior CRL studies [1, 2], we therefore convert episodic returns into success rates using task-specific thresholds, referred to as success scores. For each task, the success score is defined as the average final episodic return across all compared methods, with 10 random seeds per method. During evaluation, a run is considered successful if its averaged episodic return is greater than or equal to the corresponding success score, and unsuccessful otherwise. The resulting success rates are then used to compute AP, FP, and FWT.

Specifically, Table 1 uses success scores computed from all methods included in the overall comparison, whereas Table 3 uses success scores computed from the methods included in the ablation study. Since these two sets of methods differ, the resulting success scores also differ, leading to different numerical values of the evaluation metrics even for the same methods. For transparency, we provide the success scores used in Table 1 for Freeway, SpaceInvaders, and CrossGame below.

Table R1. Success scores used for Freeway in Table 1.

Task Index12345678
Success Score20.969515.304411.834320.048023.100711.291410.856017.6945

Table R2. Success scores used for SpaceInvaders in Table 1.

Task Index12345678910
Success Score367.0271410.1991403.8130408.0557385.9235382.7462423.7819394.4095519.8464506.5775

Table R3. Success scores used for CrossGame in Table 1.

Task Index12345678
Success Score65.8216940.2108142.54043.4980751.5892694.345897.849856.8017

In contrast, Table 3 originally used the following success scores for Freeway, calculated from the methods included in the ablation study.

Table R4. Success scores originally used for Freeway in Table 3.

Task Index12345678
Success Score21.711918.608315.000122.867824.601113.084512.804119.2184

To ensure a consistent and fair comparison, we have recalculated all ablation results in Table 3 using exactly the same success scores as Table 1. The updated results are reported below. As shown in Table R5, the recalculated FP and FWT values of FT-N and NAMI are now fully consistent with those reported in Table 1. More importantly, the main conclusions of the ablation study remain unchanged. These results confirm that the numerical discrepancy was caused by inconsistent success-score thresholds rather than differences in the underlying training results. We will revise Table 3 and clarify the success score calculation in Section 5.1 to ensure consistency and reproducibility.

Table R5. Revised ablation study of NAMI on Freeway using the success scores from Table 1.

MethodAP (↑)FP (↑)FWT (↑)MaximizationAbsoluteNeuronFirstSecondReset
FT-N0.8406 ± 0.09490.9287 ± 0.06690.7544 ± 0.0943
NAMI-Param-No-Abs0.8331 ± 0.10230.9186 ± 0.07630.7462 ± 0.0991✓
NAMI-Param0.8416 ± 0.09150.9295 ± 0.06350.7554 ± 0.0908✓✓
NAMI0.8470 ± 0.08570.9382 ± 0.05600.7612 ± 0.0871✓✓✓
NAMI-FO0.8465 ± 0.08890.9357 ± 0.05880.7608 ± 0.0908✓✓✓✓
NAMI-SO0.8320 ± 0.10050.9226 ± 0.06850.7446 ± 0.0971✓✓✓✓✓
NAMI-Reset0.8232 ± 0.08730.9208 ± 0.06580.7353 ± 0.0729✓✓✓✓

References

[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.

[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer KWaZ (2/4)

Response to W2: Effectiveness of Manhattan Norm for Forward Transfer

NAMI is designed as a low cost heuristic rather than a theoretically guaranteed criterion. As discussed in Section 4.3, NAMI-FO and NAMI-SO use first- and second-order information to estimate transferability, potentially accounting for both positive and negative transfer. However, they require additional computation and storage. In contrast, NAMI provides an interaction-free alternative suitable for CRL under limited interaction budgets. To further validate our heuristic, we compare NAMI with NAMI-min (selecting neurons with the smallest Manhattan norms) and NAMI-random (randomly selecting historical neurons). NAMI achieves the highest AP, FP, and FWT among these variants, providing additional empirical support for Manhattan-norm maximization as a simple and computationally efficient heuristic for forward transfer, without claiming a universal guarantee.

Table R6: Ablation study of NAMI on Freeway under different neuron selection strategies and parameter transformations. Results are reported as mean ± standard deviation. Best results are in bold.

MethodFP (↑)AP (↑)FWT (↑)
Baseline0.1891 ± 0.34730.1771 ± 0.33300.0000 ± 0.0000
Finetune0.9287 ± 0.06690.8406 ± 0.09490.7544 ± 0.0943
NAMI0.9382 ± 0.05600.8470 ± 0.08570.7612 ± 0.0871
NAMI-rescaling0.7309 ± 0.34990.6666 ± 0.33340.5698 ± 0.3074
NAMI-permutation0.9016 ± 0.07260.8106 ± 0.10650.7213 ± 0.0920
NAMI-min0.9299 ± 0.06320.8392 ± 0.09480.7525 ± 0.0938
NAMI-random0.9302 ± 0.06680.8415 ± 0.09910.7581 ± 0.0920
NAMI-L20.9179 ± 0.07210.8234 ± 0.10280.7347 ± 0.0952
NAMI-all0.9287 ± 0.06680.8408 ± 0.09500.7546 ± 0.0946
  • NAMI: Our proposed method that selects the neuron group with the largest Manhattan norm across historical policies to initialize the new task policy.
  • NAMI-rescaling: Applies function-preserving parameter rescaling (scale=0.5) to historical policies before neuron selection.
  • NAMI-permutation: Applies function-preserving neuron permutation (percent=100%) to historical policies before neuron selection.
  • NAMI-min: Selects neuron groups with the smallest Manhattan norms instead of the largest.
  • NAMI-random: Randomly selects neuron groups from historical policies instead of using Manhattan norms.
  • NAMI-L2: Selects neuron groups with the largest L2 norms instead of Manhattan norms.
  • NAMI-all: Selects the entire historical policy with the largest overall Manhattan norm instead of performing neuron-wise selection.

Response to W6: Comparison with Simple Baselines

To verify the benefits of fine-grained neuron-wise knowledge selection, we additionally compare NAMI with NAMI-random (random neuron selection) and NAMI-all. As shown in Table R6, NAMI achieves the highest FP, AP, and FWT among all compared methods. In particular, NAMI outperforms both NAMI-random and NAMI-all. These results provide additional empirical evidence that fine-grained neuron-wise recombination offers benefits beyond random selection or whole-network reuse, supporting the effectiveness of NAMI's design.

Response to W3: Controlling the Magnitude of Parameter Perturbations

To address this concern, we additionally evaluate Reset-top with controlled perturbation magnitudes at reset ratios of 10% and 50%. The results show that, at a 50% reset ratio, controlling the perturbation magnitude substantially mitigates the performance degradation caused by Reset-top, improving FP from 0.6596 to 0.9018 and FWT from 0.4715 to 0.7148. This suggests that perturbation magnitude contributes substantially to the degradation observed in the original experiment. Nevertheless, Controlled Reset-top still performs below the FT-N baseline. These findings clarify the limitations of the original reset experiment and motivate our use of the Manhattan norm as a computationally efficient heuristic rather than a theoretically guaranteed measure of transferability.

Table R7: The performance on Freeway with different reset setting.

MethodReset RatioFP (↑)FWT (↑)
FT-N (No Reset)0%0.9287 ± 0.06690.7544 ± 0.0943
Reset-top10%0.9234 ± 0.06180.7438 ± 0.0867
Reset-random10%0.9138 ± 0.06440.7342 ± 0.0886
Controlled Reset-top10%0.8900 ± 0.09500.7147 ± 0.1056
Reset-top50%0.6596 ± 0.30660.4715 ± 0.2209
Reset-random50%0.8043 ± 0.16110.6168 ± 0.1287
Controlled Reset-top50%0.9018 ± 0.07320.7148 ± 0.0941
AuthorsOpenReview ↗

Response to Reviewer KWaZ (3/4)

Response to W4: Neuron Group Definition and Negative Transfer

As described in the Implementation Details (Section 5.1), following prior work [1,2], we apply NAMI only to the actor network. Specifically, our Atari-based architecture consists of a shared CNN encoder and a two-layer linear actor, with NAMI applied only to the actor. For Meta-World, we use a three-layer linear actor without a CNN encoder, and NAMI is applied to the entire actor. In Eq. (9), each neuron group consists of the weights and bias of an output neuron. Moreover, NAMI recursively maintains a global initialization network rather than combining independently initialized policies, providing consistent parameter correspondence across tasks.

To investigate whether early high-norm neurons permanently dominate the global network, we additionally track the source task distribution of selected neurons throughout continual learning. The results show that the global network is predominantly composed of neurons from recent tasks, while the proportion of early task neurons gradually decreases. This suggests that NAMI does not suffer from permanent domination by early high-norm neurons in this task sequence.

Table R8. Source task distribution of neurons in the global network on Freeway.

current tasksource 0source 1source 2source 3source 4source 5source 6source 7
0100.00%0.00%0.00%0.00%0.00%0.00%0.00%0.00%
18.42%91.58%0.00%0.00%0.00%0.00%0.00%0.00%
21.72%2.81%95.47%0.00%0.00%0.00%0.00%0.00%
30.84%1.04%14.77%83.36%0.00%0.00%0.00%0.00%
40.47%0.55%2.29%8.24%88.46%0.00%0.00%0.00%
50.14%0.14%0.37%0.82%2.09%96.45%0.00%0.00%
60.06%0.04%0.08%0.14%0.12%2.05%97.52%0.00%
70.02%0.00%0.04%0.04%0.06%0.49%3.46%95.90%

Furthermore, we provide per-task forward transfer results on CrossGame, which involves more heterogeneous tasks and potential knowledge conflicts. The results indicate that NAMI can still experience negative transfer under task conflicts, as observed on Tasks 4 and 6. Nevertheless, it achieves stronger positive transfer on several tasks and the highest average FWT among the compared methods. These findings suggest that NAMI can effectively reuse historical knowledge in heterogeneous task sequences, although it does not eliminate negative transfer entirely.

Table R9. Per-task forward transfer on CrossGame.

MethodTask 0Task 1Task 2Task 3Task 4Task 5Task 6Task 7Avg.
CKA-RL0.00.62220.64090.4733-0.31850.3096-0.20030.1310.2369 ± 0.3553
ReGraMa0.00.5450.55630.4117-0.35370.4768-0.06370.18720.2514 ± 0.3219
NAMI0.00.66370.56170.7079-0.36650.4946-0.17420.09040.2825 ± 0.3993

References

[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.

[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.

Response to W5: Statistical Significance

To assess statistical reliability, we independently calculate FP and FWT for each of the 10 random seeds using identical task sequences and seed configurations across methods. We then report the paired differences between NAMI and each baseline with 95% confidence intervals (CIs), as detailed in Tables R7–R12 of our response to Reviewer 4yQP.

The results show that NAMI achieves the highest aggregate FP and FWT across all three benchmarks in Table 1, although not all paired improvements are statistically significant at the 95% confidence level. For instance, NAMI shows positive FWT improvements over CKA-RL on Freeway and CompoNet on SpaceInvaders. On CrossGame, the slightly negative mean paired FWT difference against ReGraMa is largely influenced by an unusually poor result at seed 0. Nevertheless, NAMI achieves the highest aggregate FWT in Table 1 (0.2825), which uses a different aggregation procedure. Overall, these results provide additional statistical evidence for NAMI's effectiveness while acknowledging that its improvements over some strong baselines remain modest.

Reviewer_4yQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies forward transfer in continual reinforcement learning (CRL), where an agent must learn a sequence of tasks under limited environment-interaction budgets. The authors propose Neuron-wise mAnhattan-norm Maximization Initialization (NAMI), a lightweight alternative to transferring or combining whole policy networks. For each neuron group, NAMI retains the parameter block with the largest L1 (Manhattan) norm among previously trained task policies. The selected blocks form a single global initialization network, which is updated recursively after each task and used to initialize the next task's PPO policy. This design avoids retaining or evaluating all historical policies during transfer and keeps inference complexity independent of the number of tasks.

The evaluation covers several Atari cross-mode sequences (Freeway, SpaceInvaders, Hero, and WordZapper), an eight-game cross-game sequence, and a five-task Meta-World setting. The paper compares NAMI with fine-tuning, architectural continual-learning methods, and other transfer baselines, reporting final performance (FP), forward transfer (FWT), and, for the main Atari settings, average performance (AP). Neuron-level, parameter-level, reset-based, and gradient-based variants are also examined. NAMI obtains the highest reported mean FP and FWT on the three main Atari sequences, although the improvements over the strongest alternatives are often small compared with the reported variability.

Strengths. NAMI is simple, inexpensive at inference time, and directly addresses the practical constraint of transferring knowledge without accumulating multiple policy networks. The paper includes a relatively broad collection of Atari baselines, ten-seed aggregate results, and ablations contrasting neuron-wise and parameter-wise selection.

Weaknesses. The proposed L1-norm heuristic is not convincingly connected to cross-task transferability: raw weight magnitude depends on parameterization, and corresponding neurons across independently trained policies need not be functionally aligned. The empirical advantages are generally modest relative to their standard deviations, and there is an unexplained discrepancy between the reported main Freeway results and the Freeway ablation results. These issues weaken the evidence for a generalizable neuron-level transfer mechanism.

Critical strengths

1. A simple and computationally attractive forward-transfer mechanism. NAMI replaces growing ensembles or model-level composition with a single recursively maintained parameter network. Algorithm 1 and Eq. (9) make the selection rule easy to implement and verify. Its O(P) storage/transfer-update footprint and single-policy inference are relevant under restricted interaction and deployment budgets. This makes NAMI a useful practical baseline even if the proposed selection criterion ultimately proves heuristic rather than principled.

2. Meaningful baseline coverage and informative initial ablations. The paper evaluates multiple Atari task sequences and compares against strong continual-RL methods, including FT-N, CompoNet, CKA-RL, and ReGraMa. Table 3 compares coordinate-wise versus neuron-wise selection, while Figures 2–3 investigate magnitude accumulation and the effect of resetting large-magnitude parameters. The additional Hero, WordZapper, and Meta-World results broaden the empirical picture, and reporting ten seeds for Atari is a positive aspect of the evaluation.

Critical weaknesses

1. The L1-norm selection rule is insufficiently justified as a measure of transferable knowledge. The argument moves from larger parameter magnitudes and greater sensitivity to resetting them (Figures 2–3) to the conclusion that the corresponding neurons should be imported into a different task. However, magnitude and within-network importance do not establish cross-task compatibility or positive transfer. For networks with common positively homogeneous activations, function-preserving rescaling can change a neuron's incoming-weight norm; neuron permutations can also change which units occupy corresponding indices without changing the represented function. Consequently, comparing the same-index neuron across task policies may conflate arbitrary parameterization, optimization history, and transferable features. Moreover, independently choosing neuron groups from different historical networks can break coordination between upstream features and downstream weights. The task-source visualization in Figure 5, which is dominated by relatively recent tasks, also leaves open whether the rule primarily tracks progressive magnitude growth rather than selecting complementary knowledge. A more convincing mechanism analysis would measure the actual improvement from transplanting selected neuron groups, test sensitivity to function-preserving rescaling or neuron permutation, and compare with activation-aligned matching or equally cheap alternative importance scores. These controls would directly establish what information the Manhattan norm contributes.

2. The empirical margin is often small relative to variance, and key experimental comparisons need clarification. In Table 1, NAMI improves Freeway FWT over FT-N from 0.7544 to 0.7612, and SpaceInvaders FWT over CReLUs from 0.7986 to 0.8087; both gaps are small compared with the reported standard deviations. The CrossGame FWT margin over ReGraMa is 0.0311, whereas the corresponding NAMI standard deviation is 0.3993. On Meta-World (Table 4), NAMI's FP of 0.3118 ± 0.4632 is close to fine-tuning's 0.2945 ± 0.4558, offering limited evidence of strong continuous-control generalization. Although ten-seed results are reported for Atari, paired per-seed improvements or uncertainty intervals for method differences are not provided. More importantly, the Freeway NAMI FP is 0.9382 ± 0.0560 in Table 1 but 0.8109 ± 0.0990 in Table 3, while the FT-N FP changes from 0.9287 ± 0.0669 to 0.7997 ± 0.1058. If the ablation uses a different task ordering, evaluation protocol, or training configuration, this distinction is not sufficiently explained. Clarifying these conditions and reporting paired comparisons are necessary to assess whether neuron-wise selection provides a stable advantage over a strong sequential fine-tuning baseline.

Questions for authors

1. Functional alignment and norm validity. How exactly are neuron groups defined (incoming weights, outgoing weights, biases, and shared CNN parameters), and how is compatibility across task policies ensured? Do the results remain stable under neuron permutations or function-preserving parameter rescalings? If such tests have not been run, can the authors provide existing evidence that the selected L1-large neurons have greater positive transfer value, rather than merely greater magnitude or within-task importance?

2. Consistency of the Freeway experiments. Why do Table 1 and Table 3 give markedly different FP numbers for both NAMI (0.9382 versus 0.8109) and FT-N (0.9287 versus 0.7997) on Freeway? Please identify any differences in task sequences, evaluation protocols, or hyperparameters and clarify which experimental comparison supports the ablation conclusions.

3. Statistical reliability and simple controls. Can the authors report paired seed-wise NAMI-minus-baseline differences (or confidence intervals) for FP and FWT, particularly on Freeway, SpaceInvaders, and CrossGame? It would also be helpful to clarify whether the present results include comparisons against a random-neuron selector, an L2-norm selector, or a last-task-only initializer under the same interaction budget. These are low-cost controls that would isolate the value of maximum L1 selection from the benefit of reusing previously trained parameters in general.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 4yQP (1/4)

We sincerely thank Reviewer 4yQP for the constructive feedback and recognition of NAMI's simplicity, efficiency, single-network design, comprehensive comparisons, and insightful ablation studies.

Response to W2/Q2: Inconsistency Between Tables 1 and 3

The discrepancy arises from the use of different success score thresholds when calculating the evaluation metrics. As described in Section 5.1, our CRL evaluation uses success rates to calculate average performance (AP), final performance (FP), and forward transfer (FWT). However, Atari games such as Freeway, SpaceInvaders, and those in CrossGame are score-based environments without an explicitly defined binary success criterion. Moreover, reward scales vary substantially across games, making direct comparisons based on raw episodic returns inappropriate. Following the evaluation protocol of prior CRL studies [1, 2], we therefore convert episodic returns into success rates using task-specific thresholds, referred to as success scores. For each task, the success score is defined as the average final episodic return across all compared methods, with 10 random seeds per method. During evaluation, a run is considered successful if its averaged episodic return is greater than or equal to the corresponding success score, and unsuccessful otherwise. The resulting success rates are then used to compute AP, FP, and FWT.

Specifically, Table 1 uses success scores computed from all methods included in the overall comparison, whereas Table 3 uses success scores computed from the methods included in the ablation study. Since these two sets of methods differ, the resulting success scores also differ, leading to different numerical values of the evaluation metrics even for the same methods. For transparency, we provide the success scores used in Table 1 for Freeway, SpaceInvaders, and CrossGame below.

Table R1. Success scores used for Freeway in Table 1.

Task Index12345678
Success Score20.969515.304411.834320.048023.100711.291410.856017.6945

Table R2. Success scores used for SpaceInvaders in Table 1.

Task Index12345678910
Success Score367.0271410.1991403.8130408.0557385.9235382.7462423.7819394.4095519.8464506.5775

Table R3. Success scores used for CrossGame in Table 1.

Task Index12345678
Success Score65.8216940.2108142.54043.4980751.5892694.345897.849856.8017

In contrast, Table 3 originally used the following success scores for Freeway, calculated from the methods included in the ablation study.

Table R4. Success scores originally used for Freeway in Table 3.

Task Index12345678
Success Score21.711918.608315.000122.867824.601113.084512.804119.2184

To ensure a consistent and fair comparison, we have recalculated all ablation results in Table 3 using exactly the same success scores as Table 1. The updated results are reported below. As shown in Table R5, the recalculated FP and FWT values of FT-N and NAMI are now fully consistent with those reported in Table 1. More importantly, the main conclusions of the ablation study remain unchanged. These results confirm that the numerical discrepancy was caused by inconsistent success-score thresholds rather than differences in the underlying training results. We will revise Table 3 and clarify the success score calculation in Section 5.1 to ensure consistency and reproducibility.

Table R5. Revised ablation study of NAMI on Freeway using the success scores from Table 1.

MethodAP (↑)FP (↑)FWT (↑)MaximizationAbsoluteNeuronFirstSecondReset
FT-N0.8406 ± 0.09490.9287 ± 0.06690.7544 ± 0.0943
NAMI-Param-No-Abs0.8331 ± 0.10230.9186 ± 0.07630.7462 ± 0.0991✓
NAMI-Param0.8416 ± 0.09150.9295 ± 0.06350.7554 ± 0.0908✓✓
NAMI0.8470 ± 0.08570.9382 ± 0.05600.7612 ± 0.0871✓✓✓
NAMI-FO0.8465 ± 0.08890.9357 ± 0.05880.7608 ± 0.0908✓✓✓✓
NAMI-SO0.8320 ± 0.10050.9226 ± 0.06850.7446 ± 0.0971✓✓✓✓✓
NAMI-Reset0.8232 ± 0.08730.9208 ± 0.06580.7353 ± 0.0729✓✓✓✓

References

[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.

[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer 4yQP (2/4)

Response to W1/Q1: Neuron Group Definition and Transferability

1. Neuron Group Definition and Alignment. Following CompoNet[1] and CKA-RL [2], our Atari-based architecture consists of a shared CNN encoder and a two-layer linear actor, with NAMI applied only to the actor. For Meta-World, we use a three-layer linear actor without a CNN encoder, and NAMI is applied to the entire actor. In Eq. (9), each neuron group consists of the weights and bias of an output neuron. Moreover, NAMI recursively maintains a global initialization network rather than combining independently initialized policies, providing consistent parameter correspondence across tasks.

2. Manhattan Norm and Transferability. To further validate our heuristic, we compare NAMI with minimum-norm selection, random selection, function-preserving rescaling, and neuron permutation. Rescaling and permutation are applied after historical policy training but before neuron selection. NAMI achieves the highest mean performance across all metrics. The degradation under rescaling and permutation suggests that the parameter magnitudes and neuron correspondence established through continual learning contribute to effective knowledge transfer. Although these results also reveal NAMI's sensitivity to parameterization, they provide empirical support for preserving the learned parameter structure through recursive initialization. We will clarify these details and discuss this limitation in the revised manuscript.

Table R6: Ablation study of NAMI on Freeway under different neuron selection strategies and parameter transformations. Results are reported as mean ± standard deviation. Best results are in bold.

MethodFP (↑)AP (↑)FWT (↑)
Baseline0.1891 ± 0.34730.1771 ± 0.33300.0000 ± 0.0000
Finetune0.9287 ± 0.06690.8406 ± 0.09490.7544 ± 0.0943
NAMI0.9382 ± 0.05600.8470 ± 0.08570.7612 ± 0.0871
NAMI-rescaling0.7309 ± 0.34990.6666 ± 0.33340.5698 ± 0.3074
NAMI-permutation0.9016 ± 0.07260.8106 ± 0.10650.7213 ± 0.0920
NAMI-min0.9299 ± 0.06320.8392 ± 0.09480.7525 ± 0.0938
NAMI-random0.9302 ± 0.06680.8415 ± 0.09910.7581 ± 0.0920
NAMI-L20.9179 ± 0.07210.8234 ± 0.10280.7347 ± 0.0952
NAMI-all0.9287 ± 0.06680.8408 ± 0.09500.7546 ± 0.0946
  • NAMI: Our proposed method that selects the neuron group with the largest Manhattan norm across historical policies to initialize the new task policy.
  • NAMI-rescaling: Applies function-preserving parameter rescaling (scale=0.5) to historical policies before neuron selection.
  • NAMI-permutation: Applies function-preserving neuron permutation (percent=100%) to historical policies before neuron selection.
  • NAMI-min: Selects neuron groups with the smallest Manhattan norms instead of the largest.
  • NAMI-random: Randomly selects neuron groups from historical policies instead of using Manhattan norms.
  • NAMI-L2: Selects neuron groups with the largest L2 norms instead of Manhattan norms.
  • NAMI-all: Selects the entire historical policy with the largest overall Manhattan norm instead of performing neuron-wise selection.

references

[1] Malagón, M., Ceberio, J., and Lozano, J. A. Self-Composing Policies for Scalable Continual Reinforcement Learning. ICML, 2024.

[2] Hu, J., Lian, Z. H., Wen, Z., et al. Continual Knowledge Adaptation for Reinforcement Learning. NeurIPS, 2025.

Response to Q3: Comparison with Simple Baselines

To further validate the effectiveness of neuron selection based on the Manhattan norm, we introduce several simple baselines, including NAMI-min, NAMI-random, NAMI-L2, and NAMI-all. As shown in Table R6, NAMI achieves the highest FP, AP, and FWT among all compared variants. These results provide additional empirical support for the effectiveness of maximum L1-norm selection and suggest that fine-grained neuron-wise knowledge recombination offers benefits over whole-network selection and alternative selection strategies under the same interaction budget.

AuthorsOpenReview ↗

Response to Reviewer 4yQP (3/4)

Response to Q3: Statistical Reliability and Paired-Seed Analysis

To assess statistical reliability, we independently compute FP and FWT for each of the 10 random seeds (0–9), using identical task sequences and seed configurations across methods. We then calculate the paired differences between NAMI and each baseline for every seed and report their mean differences with 95% confidence intervals (CIs).

The results show that NAMI achieves the best overall mean performance, although its improvements over some strong baselines are not statistically conclusive at the 95% confidence level. Notably, on CrossGame, NAMI exhibits a slightly negative mean FWT difference against ReGraMa (−0.01890), primarily driven by an unusually poor result at seed 0 (−0.57244), which substantially affects the seed-wise average. Nevertheless, this does not contradict the superior aggregate FWT reported in Table 1, as the two evaluations use different aggregation procedures. Overall, these results support NAMI's competitive effectiveness while highlighting the variability of forward transfer across random seeds.

Table R7: The FP on Freeway with different seeds.

ComparatorSeed 0Seed 1Seed 2Seed 3Seed 4Seed 5Seed 6Seed 7Seed 8Seed 9Mean [95% CI]
FS+0.7323+0.6885+0.6090+0.5822+0.7456+0.5680+0.7472+0.4755+0.5917+0.6736+0.6413 [+0.5869, +0.6931]
FT-N+0.0032+0.0076+0.0096+0.0153+0.0017+0.0008-0.0002-0.0003-0.0001+0.0005+0.0038 [+0.0010, +0.0071]
CReLUs+0.0102+0.0150+0.0118+0.0050-0.0008+0.0064+0.0137-0.0172+0.0058-0.0039+0.0046 [-0.0016, +0.0098]
CbpNet+0.0066+0.0051+0.0048+0.0074+0.0003+0.0110-0.0011-0.0032-0.0045+0.0071+0.0033 [+0.0003, +0.0063]
CKA-RL-0.0037+0.0171+0.0062+0.0010-0.0005-0.0019-0.0025-0.0189-0.0001+0.0126+0.0009 [-0.0050, +0.0066]
CompoNet+0.0059-0.0046+0.0116+0.0001-0.0030+0.0088+0.0070-0.0208+0.0028-0.0040+0.0004 [-0.0055, +0.0054]
MaskNet+0.8757+0.8882+0.8826+0.8862+0.8851+0.8839+0.8758+0.8626+0.8672+0.8877+0.8795 [+0.8739, +0.8843]
PackNet+0.5358+0.5842+0.5323+0.9048+0.7699+0.5459+0.5233+0.6311+0.5974+0.7831+0.6408 [+0.5690, +0.7237]
ProgNet+0.5284+0.3297+0.2879+0.3322+0.2717+0.3067+0.3362+0.2938+0.2797+0.3903+0.3356 [+0.2985, +0.3856]
C-CHAIN+0.0104-0.0025+0.0069+0.0054-0.0024-0.0043-0.0033-0.0136+0.0069+0.0104+0.0014 [-0.0033, +0.0058]
ReGraMa+0.0023+0.0115+0.0584+0.0132+0.0064+0.0124-0.0065-0.0173+0.0042+0.0055+0.0090 [-0.0010, +0.0219]
TRAC+0.9887+0.8688+0.8602+0.8600+0.8576+0.9904+0.8863+0.8382+0.9828+0.8710+0.9004 [+0.8668, +0.9385]

Table R8: The FWT on Freeway with different seeds.

ComparatorSeed 0Seed 1Seed 2Seed 3Seed 4Seed 5Seed 6Seed 7Seed 8Seed 9Mean [95% CI]
FS+0.7893+0.7717+0.7251+0.6531+0.7782+0.6199+0.7391+0.7184+0.7983+0.7512+0.7344 [+0.6977, +0.7662]
FT-N+0.0253+0.0167+0.0235+0.0048+0.0206-0.0233+0.0014+0.0759+0.0130-0.0184+0.0140 [-0.0010, +0.0312]
CReLUs+0.0719+0.0434+0.0475-0.0776+0.0949-0.1157+0.0491+0.0011+0.0540+0.0338+0.0202 [-0.0216, +0.0556]
CbpNet+0.0677+0.0569+0.2052-0.0125+0.0706-0.0179+0.0685+0.0757+0.0948+0.0551+0.0664 [+0.0329, +0.1046]
CKA-RL+0.0749+0.1081+0.0394+0.0760+0.0049-0.0583+0.0143+0.0109+0.0617+0.0095+0.0341 [+0.0054, +0.0616]
CompoNet+0.0397-0.0340+0.0293-0.0898+0.0463-0.0713+0.0152+0.0987+0.0616+0.0054+0.0101 [-0.0258, +0.0437]
MaskNet+1.3566+0.8681+1.2850+1.3327+0.9607+1.2946+0.9430+1.5293+1.2513+0.9233+1.1745 [+1.0392, +1.3073]
PackNet+0.5825+0.6305+0.6570+1.3273+0.8457+0.5340+0.5103+0.8452+0.7802+0.8288+0.7541 [+0.6293, +0.9086]
ProgNet+0.6272+0.4324+0.4393+0.4284+0.4276+0.5581+0.5046+0.5416+0.4818+0.4691+0.4910 [+0.4542, +0.5328]
C-CHAIN+0.0672+0.1070+0.0191-0.0396+0.0646-0.1070+0.1221+0.0126+0.0345+0.0400+0.0321 [-0.0099, +0.0696]
ReGraMa-0.0010+0.0498+0.1122-0.0658+0.0275-0.0953+0.0052+0.0312+0.0641+0.0216+0.0150 [-0.0212, +0.0489]
TRAC+1.3638+0.8430+1.2495+1.2864+0.9225+1.3126+0.9539+1.4910+1.2639+0.8975+1.1584 [+1.0213, +1.2903]
Reviewer_eWb3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes NAMI, a neuron-wise initialization strategy for forward transfer in continual reinforcement learning. For each new task, NAMI initializes the policy by selecting neuron groups with the largest Manhattan norms from previously learned policies. It maintains a single global network that recursively retains these selected groups, avoiding the need to store all historical policies and keeping per-task transfer and training costs, as well as inference cost, independent of the number of tasks. Experiments on Atari and Meta-World show improved forward transfer and competitive final performance compared with several baselines.

Critical strengths
  1. The paper identifies a clear limitation of existing methods, namely coarse-grained transfer at the whole-network level, and moves to neuron-level selection.
  2. The global network stores only one model, so transfer, training, and inference all remain at O(P). This is a concrete advantage over CompoNet.
Critical weaknesses
  1. The core assumption would benefit from a more direct test. The method rests on the idea that a neuron with a larger Manhattan norm carries more transferable knowledge. Figures 2 and 3 show that high-magnitude parameters are important for the task they were trained on, since resetting them hurts more than resetting random ones. The FWT gains in Table 1 and the NAMI-Reset degradation in Table 3 give indirect support, but a direct comparison is still missing. I encourage the authors to initialize with high-magnitude versus random versus low-magnitude neurons.
  2. The relationship to magnitude-based pruning should be clarified [A], [B]. Selecting parameters by magnitude is a classic idea in network pruning. The paper does not explain how NAMI differs from this line of work beyond the application setting. Stating the difference along one or two axes, such as the operating object or the mechanism, would make the contribution clearer.
  3. The Meta-World evidence is thin. In Table 4, NAMI is compared with only two baselines, Fine-tuning and CKA-RL, and the evaluation uses the first five tasks of CW10. Claiming the highest FP and FWT on this basis is not fully convincing. I suggest adding more baselines or extending the task sequence to strengthen the continuous-control result.

[A] Learning both weights and connections for efficient neural networks. NeurIPS 2015.

[B] The lottery ticket hypothesis: finding sparse, trainable neural networks. ICLR 2019.

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer eWb3 (1/2)

We sincerely thank the reviewer for the constructive feedback and for recognizing the key strengths of our work, particularly its fine-grained knowledge transfer mechanism and computational efficiency achieved by maintaining a single global network with O(P) complexity.

Response to W1: Effectiveness of Manhattan Norm for Forward Transfer

To directly evaluate the effectiveness of Manhattan-norm maximization, we additionally compare NAMI with other variants, including NAMI-min (low-magnitude) and NAMI-random (random). As shown in the following table, NAMI achieves the highest AP, FP, and FWT among all variants. These results provide additional empirical support for using the maximum Manhattan norm as a simple and computationally efficient heuristic for forward transfer.

Table R1: Ablation study of NAMI on Freeway under different neuron selection strategies and parameter transformations. Results are reported as mean ± standard deviation. Best results are in bold.

MethodFP (↑)AP (↑)FWT (↑)
Baseline0.1891 ± 0.34730.1771 ± 0.33300.0000 ± 0.0000
Finetune0.9287 ± 0.06690.8406 ± 0.09490.7544 ± 0.0943
NAMI0.9382 ± 0.05600.8470 ± 0.08570.7612 ± 0.0871
NAMI-min0.9299 ± 0.06320.8392 ± 0.09480.7525 ± 0.0938
NAMI-random0.9302 ± 0.06680.8415 ± 0.09910.7581 ± 0.0920
NAMI-rescaling0.7309 ± 0.34990.6666 ± 0.33340.5698 ± 0.3074
NAMI-permutation0.9016 ± 0.07260.8106 ± 0.10650.7213 ± 0.0920
NAMI-L20.9179 ± 0.07210.8234 ± 0.10280.7347 ± 0.0952
NAMI-all0.9287 ± 0.06680.8408 ± 0.09500.7546 ± 0.0946
  • NAMI: Our proposed method that selects the neuron group with the largest Manhattan norm across historical policies to initialize the new task policy.
  • NAMI-min: Selects neuron groups with the smallest Manhattan norms instead of the largest.
  • NAMI-random: Randomly selects neuron groups from historical policies instead of using Manhattan norms.
  • NAMI-rescaling: Applies function-preserving parameter rescaling to historical policies before neuron selection.
  • NAMI-permutation: Applies function-preserving neuron permutation to historical policies before neuron selection.
  • NAMI-L2: Selects neuron groups with the largest L2 norms instead of Manhattan norms.
  • NAMI-all: Selects the entire historical policy with the largest overall Manhattan norm instead of performing neuron-wise selection.

Response to W2: Relationship to Magnitude-Based Pruning

NAMI is truly inspired by magnitude-based pruning [1][2]. Although both methods leverage parameter magnitudes, they differ in the following aspects:

  1. Comparison: Magnitude-based pruning compares parameter magnitudes within a single network, whereas NAMI compares the Manhattan norms of corresponding neuron groups across networks learned from different tasks.
  2. Objective: Magnitude-based pruning aims to compress networks while preserving learned knowledge, whereas NAMI aims to maximize forward knowledge transfer by reusing knowledge from previous tasks.
  3. Mechanism: Magnitude-based pruning removes parameters to obtain a smaller network, whereas NAMI selects and consolidates neuron groups to initialize a new policy, which is subsequently trained on the current task, allowing the transferred parameters to be further adapted.

We will add a discussion of these distinctions and the relevant literature in the Related Work section of the revised manuscript.

[1] Learning both weights and connections for efficient neural networks. NeurIPS 2015.

[2] The lottery ticket hypothesis: finding sparse, trainable neural networks. ICLR 2019.

AuthorsOpenReview ↗

Response to Reviewer eWb3 (2/2)

Response to W3: Evaluation on Continuous Control

Response to W3: We clarify that the Meta-World experiment serves as an additional analysis to demonstrate NAMI's applicability to continuous-control tasks, rather than as our primary evaluation. To strengthen this evidence, we expand the evaluation to 6 tasks and include three additional baselines: CbpNet, CReLUs, and CompoNet. As shown in Table R2, the original NAMI suffers from negative forward transfer (-0.0160) in this more challenging continuous-control environment. However, incorporating first-order gradient information effectively mitigates this issue, enabling NAMI-FO to achieve positive forward transfer (0.0400) and the highest final performance (0.4683) and average performance (0.3727). These results highlight both the limitations of the original NAMI and the effectiveness of its first-order extension. We will incorporate these additional results and the corresponding analysis into the revised manuscript.

Table R2: The performance comparison on Meta-World (6 tasks).

MethodFP ↑FT ↑AP ↑
FS0.4300 ± 0.4951—0.3420 ± 0.4744
FT-N0.3817 ± 0.4858-0.1080 ± 0.11770.3142 ± 0.4642
CKA-RL0.3817 ± 0.4858-0.0920 ± 0.17520.3195 ± 0.4663
CompoNet0.4167 ± 0.49300.0580 ± 0.06790.3530 ± 0.4779
CbpNet0.3850 ± 0.4866-0.0060 ± 0.02580.3225 ± 0.4674
CReLUs0.4650 ± 0.4988-0.0220 ± 0.14340.3656 ± 0.4816
NAMI (ours)0.3967 ± 0.4892-0.0160 ± 0.04030.3410 ± 0.4740
NAMI-FO (ours)0.4683 ± 0.49900.0400 ± 0.07240.3727 ± 0.4835

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.