acceptodds
Under review as a conference paper at ICLR 2027

SpikeARC: Spike-Guided Routing and Selective Policy Consolidation for Continual Reinforcement Learning

Abstract

Reinforcement learning can produce high-performance control policies, but real-world deployment demands more than performance in fixed environments. As environment dynamics and task requirements change, agents must keep learning, which can disrupt acquired yet still useful behaviors. Continual reinforcement learning therefore faces a fundamental challenge: when adaptation is required, it is not immediately clear whether a behavioral correction should remain transient or eventually become part of the long-term policy. We introduce SpikeARC, a framework that organizes continual policy adaptation through state-dependent action corrections between complementary Stable and Fast pathways. The Stable branch provides the primary action, while recurrent spiking Fast and Fusion modules generate and gate state-dependent corrections. Spike-guided routing assigns sample-wise replay responsibility, dual behavior-space constraints regulate behavioral drift, and corrections that remain consistent are progressively consolidated into the Stable branch. We systematically evaluate SpikeARC through external baseline comparisons, cross-environment controlled experiments and mechanism ablations, and structured experiments with simulated quadruped robots. Across external continual RL evaluations on CARL Walker and Quadruped, SpikeARC maintains competitive adaptation while reducing forgetting relative to hidden-context continual RL baselines. In a controlled five-environment study with a PPO backbone, it retains 90.8% of the adaptation AUC of a fully shared policy while reducing forgetting by 36.2% and long-term-policy parameter drift by 35.3% on average across environments. Ablations and structured quadruped experiments further characterize the adaptation–retention trade-offs induced by routing and consolidation. Overall, SpikeARC does not require a new behavioral correction to be classified as transient or permanent at first encounter; instead, its role is resolved progressively through continued interaction while mature behavior remains protected.

Primary area:
reinforcement learning
Keywords:
Continual Reinforcement Learning, Stability–Plasticity Dilemma, Spiking Neural Networks, Gradient Routing, Catastrophic Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UGtUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes SpikeARC, a framework for continual reinforcement learning that aims to balance rapid adaptation to changing environment dynamics with the retention of previously acquired behaviors. The method combines a Stable pathway for the mature policy with recurrent spiking Fast and Fusion modules that generate and gate state-dependent action corrections. Experiments are conducted on continuous-control environments using PPO, additional CARL benchmarks using SAC.

Critical strengths
  1. Balancing adaptation to changing dynamics with the retention of previously learned control behaviors is interesting and useful for long-term robot deployment.

  2. The paper considers several continuous-control environments, two RL algorithms (PPO and SAC), and component ablations.

Critical weaknesses
  1. Presentation and formatting need improvement. The keyword list appears unusual in the ICLR format. Several figures leave substantial whitespace, and some tables contain awkward line breaks that make their contents difficult to read. The overall layout could be made more compact and consistent.

  2. The central mechanisms are largely independent of the SNNs (e.g., Stable and Fast policy decomposition, contribution-aware gradient routing, selective consolidation, and behavioral-drift constraints). GRU or LSTM modules can also replace the recurrent LIF modules. Besides, Appendix C.3 reports that the GRU variant achieves better performance. The method appears to combine several general continual RL mechanisms with an interchangeable SNN implementation with limited novelty and contributions.

Questions for authors

Does SpikeARC improve a conventional GRU/LSTM policy?

Can you compare the performance of vanilla LIF/GRU/LSTM and SpikeARC-LIF/GRU/LSTM in one table?

What is the specific contribution of SNNs to SpikeARC?

Can you include and discuss more recent SNN-RL studies in the related work section?

Can the proposed method be compared against vanilla recurrent policies and relevant SNN-RL baselines?

Reviewer_Bj96OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors attempt to reframe continual reinforcement learning as a behavioral correction problem, rather than an adaptation problem. They proposed a framework, known as SpikeARC, which relies on two complementary pathways (Stable & Fast): Stable for generating the primary action, and Fast generate corrective actions based on gating mechanisms. These corrective actions are stored in the replay buffer, and samples from the buffer are drawn to produce gradient updates to learn a routing function, which determines whether the corrections are used to update the Stable or Fast pathways. Experiments were done using 3 seeds using PPO, SAC with simulated quadruped robots, undergoing non-stationary changes such as mass, friction etc. Overall, the results from SpikeARC are comparable to the baselines proposed by the authors.

Critical strengths
  1. The authors attempted to perform standard analyses in continual RL with regard to the characteristics of adaption and forgetting.
  2. The proposed SpikeArc framework looks interesting, using adaption in the form of action corrections. To the best of my knowledge, this is probably a less common approach to addressing catastrophic interference in continual RL.
Critical weaknesses
  1. The paper is poorly written overall. There is a severe lack of readability and coherence throughout the main paper. Furthermore, the figures (while detailed), lacks sufficient depth in the captions to highlight the main message of the figures. I have provided some questions about the writing in the next section.

  2. Only 3 seeds were used for the experiments, and it is clear from Figure 3 that the results exhibit high variance when it comes to positive forgetting.

  3. More importantly, where are the performance comparable among all the models in CARL Walker and CARL Quadruped? Is the task too easy since we don't see any discriminative performances?

With these concerns, I do not think that the paper is ready for publication.

Questions for authors
  1. What does it mean by write-back (line 76)?
  2. What does it mean by “Stable remains trainable within controlled limits (line 82-83)?”
  3. What does it mean by forward action (line 119)?
  4. Why is the model named SpikeARC when nothing seems to be based on spikes?
Reviewer_3UT8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the problem of continuous learning under changing dynamics. The authors argue that when a correction to the policy is needed, the agent can't know whether it is a long term change that shuold be integrated into the stable policy, or be used temporarily. Therefore, their approach postpones the decision rather than make it right away.

The approach consists of three parts:

  1. A stable MLP which is the "main" policy
  2. A fast recurrent LIF (a spiking neural network) that reads a short state history
  3. A fusion recurrent LIF that reads the state history and functions as a gate that determines the weight that should be assigned to the #2

The claimed contributions:

  1. A framework that separates immediate behavioral correction from long-term policy formation
  2. A selection mechanism, combining routing and selective consolidation, that decides which corrections are retained
  3. Dual behavioral-drift budgets for risk-controlled Stable plasticity
Critical strengths
  1. A clear and interesting approach to a well-framed problem

  2. The experiments are well thought-of, and the use of budgets makes a lot of sense

  3. The authors present interpretable behavioral evidence regarding the changes in model behavior due to their approach.

Critical weaknesses
  1. A major part of the evaluation is missing - the whole point of the approach is to adapt to situations where changes to the policy are temporary, and handle them differently from cases where the changes need to be integrated into the Stable policy. However, all evaluated scenarios don't contain these "temporary" changes - all changes are consistent. Not evaluating such a scenario is highly problematic given teh authors' claims.

  2. The approach is not fully task-agnostic - while the agent never sees the stage ID, this information is used to compute the stage-progress variable and in the scheduling of the controller. This information is not shared with the baselines, and it reduces the fairness of the evaluation.

  3. Multiple hyperparameters, all setad-hoc, without analysis - while I didn't count exactly, there are at least 10 hyperparameters that are set. This makes the approach potentially unstable and hard to configure.

Questions for authors
  1. Please address the weaknesses I listed above

  2. What are the budget values you used in your evaluation? I couldn't find them in the paper. The same applies for the history length H, and the size of the replay buffer

  3. Can the authors explain why removing consolidation increase Stable drift by 23 pp?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.