acceptodds
Under review as a conference paper at ICLR 2027

Continual Reinforcement Learning via Behavioral Foundation Models

Abstract

Continual reinforcement learning (CRL) requires agents that adapt quickly to new tasks, reuse past knowledge, and avoid forgetting. Existing methods address parts of this trade-off but rely on restrictive assumptions, such as slow gradient-based adaptation or known task boundaries. We ask whether Behavioral Foundation Models (BFMs), a recent class of zero-shot RL methods, can serve as a backbone for CRL. BFMs are appealing: switching to a new task-conditioned policy does not require task-specific gradient updates, the unsupervised objective is reward-agnostic, and the induced policy family may mitigate forgetting if coverage is maintained. However, BFMs are designed for offline settings and fail when applied naively online: zero-shot inference assumes a reward model, and online updates on a single task cause coverage collapse. To enable BFMs as continual online agents, we introduce a generic three-phase pipeline: reward-free offline pre-training; reward-model-free online task inference, which switches policies without updating their parameters; and continual updates through a coverage-preserving buffer. We evaluate continual sequences from DM Control and Continual World to characterize where BFMs succeed, where pre-training coverage becomes a bottleneck, and how each component contributes to stability and adaptation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sXkoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work evaluates Behavioral Foundation Models (BFMs) in a more realistic continual RL setting. The authors combine existing offline pre-training methods (e.g., FB and PSM) and online task inference (e.g., OpTI) with a new, third phase for continual online updates. To prevent coverage collapse in BFMs, the authors propose a replay buffer, the coverage buffer, that maintains a mixture of offline and online transitions. Online samples are selected to reduce the number of redundant transitions stored and to prioritize underrepresented transitions (i.e., to improve coverage). The authors' goals are to:

  1. show how each pipeline stage contributes to continual learning metrics;
  2. identify where BFMs are effective and where they are not; and
  3. evaluate whether the coverage buffer improves online performance, isolating the contribution of each of its components.
Critical strengths
  • The paper addresses a clear, relevant, and underexplored area: BFMs in the continual RL (CRL) setting.
  • The proposed method is straightforward and easy to follow. The results include a solid initial set of relevant baselines, which gives a reasonable foundation for evaluating the proposed method, as well as an initial benchmark of other BFMs.
  • The proposed algorithmic contribution follows traditional CRL approaches by integrating a replay buffer to prevent forgetting and enhance continual learning abilities.
Critical weaknesses
  • Potentially limited method novelty
    • The paper extends BFMs to a CRL setting, but the proposed pipeline is mainly an incremental approach. The method seems to reuse existing work for phases 1 and 2, but it does propose a third phase for continual updates that uses a coverage buffer (i.e., an adapted replay buffer).
  • Result generalization concerns
    • The initial results are interesting, but I have concerns about how far the conclusions generalize. CW10, the most distinct environment in the set, is not included in the ablations. I'm assuming this is because all methods perform poorly on it in the baseline comparison (ACC ≤ 0.20). As a result, the ablations rely entirely on the three DMC environments. These environments are relatively similar to one another, making it unclear whether the ablation results would generalize beyond them.
    • Adding at least one additional, sufficiently different environment (both to the main and ablation results) could help bolster the results of this paper. The authors note that BFMs struggle in manipulation domains, but I wonder whether other environments could be explored. The broader CRL literature offers a range of more diverse benchmarks, such as COOM, MiniHack, and Procgen, but I'm unsure whether these can be easily reused here.
Questions for authors

My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work, as I am approaching it from a more traditional CRL background. I am open to revising my score in light of the rebuttal.

  • Is there a reason for EWC's low performance on CW10? EWC reaches an ACC of 0.12, versus ~0.60 in Wołczyk et al. (2021) (Table 1) and Wołczyk et al. (2022) (Table 16). The ACC metric here and the average performance metric in those works seem to be defined similarly, so I wonder whether the gap is due to a difference in hyperparameters.
  • What was the methodology for selecting hyperparameters for the baseline algorithms?
  • Was there any exploration of the refresh-rate variable n, which determines how often the coverage buffer updates its values? Additionally, how is n selected? I could not find much discussion of this variable.
  • The paper does present pseudocode and relevant hyperparameters in the appendix; however, there does not seem to be any attached or referenced source code. Are there any plans to make the code publicly available?
  • Is there an explanation for why some numbers in Table 1 are highlighted and others are not? Providing a clear explanation would be helpful.
  • The recent work AdaBFM (Karthikeyan et al., "Adaptive Behavioral Foundation Models: Continual Training from Reward-Free Experience," NeurIPS 2026 Workshop on From Pretrained Representations to Acting Agents) seems closely related. I wonder whether the authors have any additional thoughts on how this paper relates to their work.
4 comments

We thank the reviewer for their constructive feedback and suggestions.

W1: We acknowledge that Phases 1 and 2 build on existing methods. However, we believe that combining these techniques into a continual learning framework is itself valuable, as they were not originally designed for this setting. More importantly, our contribution addresses a specific challenge arising from continually updating BFMs: maintaining behavioral coverage while learning from a stream of online experience. Unlike conventional replay mechanisms, the coverage buffer allows the model to incorporate new experience without degrading previously acquired capabilities. A capability proven through our ablations, to isolate the contributions of these components.

W2: We conducted the coverage-buffer ablations on Quadruped, Walker, and Cheetah. We excluded CW10 because the poor performance of BFMs on manipulation tasks makes it difficult to isolate the effects of continual updates, but retained it in the main evaluation to highlight this limitation. While the ablations are limited to DMC, they support the benefits of the coverage buffer. We also added PointMass Maze to provide further evidence beyond locomotion tasks (W3).

W3: Our evaluation follows established BFM benchmarks to isolate the contribution of continual updates. We also considered COOM, MiniHack, and Procgen, but COOM's pixel observations would introduce additional complexity in representation learning, making it harder to isolate the contribution of our method, while MiniHack and Procgen's discrete actions would require adapting our implementations and baselines. Given the limited rebuttal timeline, we added PointMass Maze, which, although part of DMC, introduces goal-directed navigation rather than locomotion, closer in task structure to MiniHack. Results are reported below:

VariantBoundariesPoint-Mass Maze ACC ↑Point-Mass Maze F ↓
FT-1✓0.03 ± 0.010.52 ± 0.24
FT-L2✓0.66 ± 0.130.00 ± 0.00
EWC✓0.73 ± 0.10-0.00 ± 0.00
P&C✓0.22 ± 0.120.09 ± 0.11
CLEAR✗0.29 ± 0.060.06 ± 0.09
PT-SAC✗0.15 ± 0.120.43 ± 0.31
3RL✗0.14 ± 0.100.33 ± 0.14
VariBAD (PPO)✗0.09 ± 0.030.22 ± 0.06
FB (frozen, reward inference)✗0.12 ± 0.070.00*
OpTI-FB (frozen, online)✗0.56 ± 0.030.00*
Continual FB✗0.72 ± 0.06-0.03 ± 0.03
PSM (frozen, reward inference)✗0.11 ± 0.020.00*
OpTI-PSM (frozen, online)✗0.50 ± 0.060.00*
Continual PSM✗0.54 ± 0.070.02 ± 0.14

On PointMass, Continual FB achieves the highest accuracy among boundary-free approaches (0.72), while maintaining near-zero forgetting and improving over frozen OpTI-FB (0.56). Its performance is also comparable to EWC (0.73), which requires task boundaries.

Q1: We thank the reviewer for pointing out this discrepancy. One possible explanation is the difference in the underlying MetaWorld implementations: our experiments use MetaWorld V3, whereas the original Continual World benchmark uses an earlier version. However, to investigate further, we conducted an additional sweep over the EWC regularization coefficient λ\lambda. With a smaller coefficient, EWC performance improved from 0.12 to 0.31 ACC. We will update the results accordingly and clarify the experimental differences.

Q2: For the CRL baselines, we performed hyperparameter selection starting from the original configurations and then sweeping key parameters (especially learning rates and regularization coefficients). For BFMs, we retained the original pre-training configurations, while for OpTI we tuned β\beta and the decay parameter. The coverage-buffer ratio was selected through our ablations. We will clarify the hyperparameter selection procedure in the appendix.

Q3: The refresh rate nn controls how often the coverage buffer recomputes its covariance matrix and leverage scores. We set n=1000n=1000 to reduce computational overhead and improve online execution speed, without noticeable performance degradation in preliminary experiments. Since this parameter mainly affects computational efficiency, we omitted a dedicated ablation. We will clarify this in the paper.

Q4: Yes, we plan to publicly release the codebase, including the experimental configurations, upon acceptance.

Q5: The colored cells in Table 1 are intended as a heatmap visualization to facilitate comparisons across methods and metrics, rather than indicate statistical significance. If the reviewer thinks this compromises readability in some way we are happy to remove it.

Q6: We thank the reviewer for pointing out AdaBFM, a complementary work focusing on reward-free adaptation to changing dynamics. Combining the two approaches could be an interesting direction to address both changing dynamics and rewards. We will discuss this connection in the revised paper.

Thank you for the detailed response. I believe most of my concerns have been addressed. Before I give my final assessment, I have a few additional questions/statements given the authors response.

  1. Which task sequence did you use for Point-Mass Maze, and how did you adapt it into a continual learning setting? For example, did each new task use a different maze size or layout?
  2. Keeping the color coding in Table 1 is fine, as long as it is clearly explained, such as in the caption or a legend.

We thank the reviewer for the follow-up and feedback.

Q1: For PointMass Maze, we maintained the same assumption of stationary dynamics as in our other environments, keeping the maze layout fixed across all tasks and changing only the goal location. We adapted it to continual learning by defining a sequence of four tasks: reaching the top-left, top-right, bottom-left, and bottom-right corners, respectively. Each task lasts 500K steps, for a total of 2M steps.

Q2: We will keep the color heatmap and clarify in the table caption that it represents relative performance across methods and metrics. Thank you for the suggestion.

My primary reservation continues to be directed toward the paper’s claim that the work can “identify where BFMs are effective and where they are not,” given the limited diversity of environments. Adding Point-Mass Maze is a welcome step toward supporting this claim. However, open questions remain about how well the findings transfer to more general CRL benchmarks, performance across multiple cycles, and discrete action spaces. More broadly, the empirical evaluation and environment coverage could be further strengthened with additional time. I do recognize that some of these points could reasonably be left to future work, and I am therefore inclined to slightly increase my score.

Reviewer_G9wKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether successor-feature-based Behavioral Foundation Models (BFMs)—typically designed for offline zero-shot reinforcement learning—can serve as effective backbones for online, task-boundary-free Continual Reinforcement Learning (CRL). To overcome the need for a reward model during inference and prevent representation collapse during online single-task updates, the authors propose a three-phase pipeline: (1) reward-free offline pre-training, (2) online optimistic task inference (OpTI) to switch policies in latent space without parameter updates, and (3) slow-timescale continual representation updates via a coverage-preserving buffer that mixes frozen offline rehearsal data with leverage-scored online transitions. Empirical evaluations on DM Control and Continual World (CW10) demonstrate that when initial pre-training coverage is sufficient, Continual BFMs adapt effectively with near-zero forgetting across up to 20M steps, though they cannot overcome poor initial coverage or manipulation bottlenecks.

Critical strengths
  1. By pairing zero-shot latent switching (via OpTI) with slower unsupervised representation updates, the framework naturally separates rapid task adaptation from parameter updates, avoiding the task-boundary assumptions required by regularization methods like EWC or FT-L2.

  2. Rather than overclaiming state-of-the-art dominance, the authors transparently highlight where the paradigm fails—specifically when pre-training state-space coverage is artificially restricted (k-means cluster ablation) or when the underlying BFM struggles with complex manipulation (CW10).

Critical weaknesses
  1. As shown in the Cheetah, CW10, and reduced-coverage Walker experiments, continual online updates only refine existing representations; they fail to discover missing behaviors when initial offline coverage is poor, because policy execution remains restricted to the span of ϕ\phi and the agent remains anchored to Doffline\mathcal{D}_{\text{offline}}. This makes the method not ideal for real online learning scenario - we couldn't expect online experiences are similar to previously collected offline data.

  2. Asymmetric comparison: Continual BFMs leverage massive offline pre-training datasets (ExoRL for DMC; MT50 expert trajectories + 10M RND transitions for CW10), whereas boundary-free CRL baselines (PT-SAC, 3RL, VariBAD) appear to be trained online from scratch without warm-starting on Dpretrain\mathcal{D}_{\text{pretrain}}.

Questions for authors
  1. Equation 3 and Algorithm 2 show VtV_t and z^t\hat z_t accumulating from i=0i=0 to tt without discounting. In a boundary-free setting where the reward RkR_k changes abruptly, how does OpTI forget old task rewards, and how do you prevent stale features ϕold(si)\phi_{\text{old}}(s_i) from corrupting VtV_t as ϕ\phi is updated online?

  2. Perhaps you may find this work relevant "Continual Reinforcement Learning by Planning with Online World Models" and could discuss in the paper?

1 comment

We thank the reviewer for the positive score, and the thoughtful comments.

W1: We partially disagree with the reviewer. The poor performance on Cheetah and CW10 primarily reflects limitations of the underlying BFMs, already visible before continual updates. Since our framework is BFM-agnostic, improvements in the underlying models could also benefit continual adaptation. Importantly, we intentionally left exploration outside the scope of this work to focus on enabling stable continual updates, while designing a framework that can be extended with exploration strategies. The reduced-coverage ablation was intended to test this stability under poor initial coverage, rather than demonstrate the discovery of missing behaviors. Even without dedicated exploration, continual updates slightly improve performance over frozen BFMs. We left the addition of an exploration framework for future work.

W2: While BFMs benefit from offline pre-training, we view this as an intrinsic capability of the approach rather than an artificial experimental advantage. The CRL baselines considered are not designed to directly exploit reward-free datasets , as their learning objectives depend on rewards. Even providing these baselines with additional training steps on the first task would not be equivalent, as they would optimize that specific reward rather than acquire reward-agnostic behaviors. Our comparison therefore aims to evaluate the approaches in their intended settings, rather than establish superiority under an equal total training budget. Furthermore, in practical applications, pre-training is a one-time cost that can be amortized across multiple tasks, enabling rapid adaptation to new rewards without requiring task-specific policy updates.

Q1: We thank the reviewer for pointing out this omission. Equation 3 reports the non-discounted estimator, while our implementation uses the non-stationary variant of OpTI, which exponentially decays previous observations. This acts as a tunable sliding window, allowing the agent to progressively forget previous rewards without requiring task boundaries. Regarding stale features ϕold\phi_{\mathrm{old}}, we use two mechanisms to mitigate their impact: (i) BFM parameters are updated more slowly, with offline rehearsal limiting representation drift, and (ii) exponential decay progressively reduces the influence of features computed using older representations. Empirically, our long-horizon experiments show stable performance over 20M steps without cumulative degradation, supporting the practical stability of our approach in the evaluated settings. We will clarify the estimator and these mechanisms in the revised paper.

Q2: We thank the reviewer for pointing out this relevant work. We agree that it is closely related to our setting and will include a discussion in the revised paper.

We also invite the reviewer to consider the additional CLEAR and P&C baselines reported in our response to reviewer 3djK, and the new PointMass results in our response to reviewer sXko, which provide further empirical evidence supporting our approach.

Reviewer_3djKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper posits that behavioral foundation models (BFMs) may be a general approach to continual learning scenarios with fixed transition dynamics but non-stationary rewards. They envision that with a special training methodology, BFMs can be adapted for online learning. Besides the usual BFM pre-training, they propose two online additions in parallel; first, the task latent is continually inferred online so the BFM can react to changes. Second, an interaction buffer with sufficient coverage is maintained, such that a diversity measure for the saved interactions is maximized. They test this methodology on four RL tasks, and show that this approach is mostly capable of combatting catastrophic forgetting without needing explicit task boundaries.

Critical strengths

I'd like to thank the authors for submitting their work. I find the idea in this paper very interesting. I believe:

  • The idea of pre-trained BFMs being used for continual learning seems very intuitive. I am convinced by this paper that such an approach may be more easily immune to catastrophic forgetting.
  • The technical solution to the coverage problem is quite fascinating. Creating a projection matrix based on vectorized representations of the tasks is clever, and reduces the problem to making sure the task latent is representative enough of reward-model similarities.
  • I find the experimental section to have diverse enough tasks for a good analysis of this approach.
Critical weaknesses

I believe there are certain flaws in the current version of this paper:

  • I don't think the baselines are expansive. There are numerous task-boundary free approaches in continual RL. For example, see Online EWC in "Progress & compress: A scalable framework for continual learning", MBCD in "Minimum-delay adaptation in non-stationary reinforcement learning via online high-confidence change-point detection", MBPO in "When to trust your model: Model-based policy optimization" (model-based methods are naturally immune to catastrophic forgetting because they do not have the on-policy data drift issue), and even CLEAR "Experience replay for continual learning".
  • The experimental results are mixed. It seems that for the most part, the current design is outperformed by 3RL. While I do appreciate the authors statement "The goal is not to establish that Continual BFMs are better than existing gradient-based or latent-based approaches", the authors should show that there is some merit in this approach that makes it worthwhile to research. If there is not, perhaps the current iteration needs work.
  • The writing in §4 needs work. This section is very dense and many parts can be simplified without losing rigor. For example, §4.3 and §4.4 have too much prose, and §4.2 has too little. The rest of the writing is fine, but §4 is quite critical to this paper and needs to be revised.
Questions for authors

Please see weaknesses.

1 comment

We thank the reviewer for their constructive feedback and suggestions.

W1: Our original baseline selection aimed to cover different families of CRL approaches rather than provide an exhaustive comparison. Nevertheless, we agree that additional baselines would strengthen the evaluation. In particular, CLEAR is relevant as our coverage buffer takes inspiration from its replay buffer strategy, although our method specifically targets the preservation of BFM behavioral coverage. Following the reviewer's suggestion, we added CLEAR and Progress & Compress (P&C) as additional baselines. We did not include model-based approaches, as their implementation and proper tuning would require a more flexible timeline. We also extended the evaluation to PointMass Maze as an additional environment. The new results are reported below.

VariantBoundariesQuadruped ACC ↑Quadruped F ↓Walker ACC ↑Walker F ↓Cheetah ACC ↑Cheetah F ↓Point-Mass Maze ACC ↑Point-Mass Maze F ↓
P&C✓0.76 ± 0.060.01 ± 0.030.63 ± 0.070.03 ± 0.090.54 ± 0.040.01 ± 0.030.22 ± 0.120.09 ± 0.11
CLEAR✗0.56 ± 0.100.03 ± 0.040.56 ± 0.120.00 ± 0.030.46 ± 0.040.00 ± 0.000.29 ± 0.060.06 ± 0.09
Continual FB✗0.74 ± 0.02-0.03 ± 0.020.70 ± 0.050.01 ± 0.060.22 ± 0.03-0.04 ± 0.100.72 ± 0.06-0.03 ± 0.03
Continual PSM✗0.65 ± 0.02-0.06 ± 0.050.62 ± 0.020.05 ± 0.060.19 ± 0.040.02 ± 0.060.54 ± 0.070.02 ± 0.14

The additional experiments show that Continual BFMs achieve competitive performance on Quadruped and Walker, and substantially outperform CLEAR and P&C on PointMass Maze, while maintaining low forgetting. On Cheetah, CLEAR and P&C achieve higher accuracy, highlighting the limitations already discussed in the paper.

W2: We respectfully disagree with the reviewer’s interpretation of the results, which show a more nuanced picture than a general advantage of 3RL. Regarding accuracy (ACC), Continual FB substantially outperforms 3RL on Quadruped (0.74 vs. 0.26) and achieves comparable performance on Walker (0.70 vs. 0.72). On Cheetah, 3RL performs better (0.37 vs. 0.21). However, regarding forgetting (F), Continual BFMs are consistently better, exhibiting near-zero or negative forgetting across all environments. In particular, on Quadruped, Walker, and Cheetah, Continual FB achieves F values of (-0.03, 0.01, -0.05), compared to (0.08, 0.08, 0.39) for 3RL.

We believe that maintaining competitive accuracy with substantially lower forgetting is a promising result, especially without requiring task-boundary information. Furthermore, our ablations show that continual updates improve performance over frozen BFMs while preserving previously acquired capabilities. The weaker performance on Cheetah and CW10 highlights existing limitations of BFM pre-training, particularly regarding behavioral coverage and manipulation tasks. Since our pipeline is agnostic to the underlying BFM, improvements in BFM pre-training could further benefit our continual adaptation. The additional PointMass Maze results further support the potential of Continual BFMs, with Continual FB and Continual PSM achieving ACC of 0.72 and 0.54, respectively, compared to 0.29 for CLEAR and 0.22 for P&C.

W3: We thank the reviewer for this feedback. We will revise Section 4 to reduce repetitions in the pipeline description, which is already introduced in the Introduction. In particular, we will shorten the discussion of existing techniques in Phase 1, clarify the online inference mechanism in Phase 2, and dedicate more space to our main contribution in Phase 3. We will also simplify the presentation of the coverage buffer, emphasizing its intuition and motivation before introducing the technical details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.