- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work evaluates Behavioral Foundation Models (BFMs) in a more realistic continual RL setting. The authors combine existing offline pre-training methods (e.g., FB and PSM) and online task inference (e.g., OpTI) with a new, third phase for continual online updates. To prevent coverage collapse in BFMs, the authors propose a replay buffer, the coverage buffer, that maintains a mixture of offline and online transitions. Online samples are selected to reduce the number of redundant transitions stored and to prioritize underrepresented transitions (i.e., to improve coverage). The authors' goals are to:
- show how each pipeline stage contributes to continual learning metrics;
- identify where BFMs are effective and where they are not; and
- evaluate whether the coverage buffer improves online performance, isolating the contribution of each of its components.
Critical strengths
- The paper addresses a clear, relevant, and underexplored area: BFMs in the continual RL (CRL) setting.
- The proposed method is straightforward and easy to follow. The results include a solid initial set of relevant baselines, which gives a reasonable foundation for evaluating the proposed method, as well as an initial benchmark of other BFMs.
- The proposed algorithmic contribution follows traditional CRL approaches by integrating a replay buffer to prevent forgetting and enhance continual learning abilities.
Critical weaknesses
- Potentially limited method novelty
- The paper extends BFMs to a CRL setting, but the proposed pipeline is mainly an incremental approach. The method seems to reuse existing work for phases 1 and 2, but it does propose a third phase for continual updates that uses a coverage buffer (i.e., an adapted replay buffer).
- Result generalization concerns
- The initial results are interesting, but I have concerns about how far the conclusions generalize. CW10, the most distinct environment in the set, is not included in the ablations. I'm assuming this is because all methods perform poorly on it in the baseline comparison (ACC ≤ 0.20). As a result, the ablations rely entirely on the three DMC environments. These environments are relatively similar to one another, making it unclear whether the ablation results would generalize beyond them.
- Adding at least one additional, sufficiently different environment (both to the main and ablation results) could help bolster the results of this paper. The authors note that BFMs struggle in manipulation domains, but I wonder whether other environments could be explored. The broader CRL literature offers a range of more diverse benchmarks, such as COOM, MiniHack, and Procgen, but I'm unsure whether these can be easily reused here.
Questions for authors
My questions are listed below. I welcome the authors' clarification on these and on the weaknesses above, in case I have misunderstood any aspects of the work, as I am approaching it from a more traditional CRL background. I am open to revising my score in light of the rebuttal.
- Is there a reason for EWC's low performance on CW10? EWC reaches an ACC of 0.12, versus ~0.60 in Wołczyk et al. (2021) (Table 1) and Wołczyk et al. (2022) (Table 16). The ACC metric here and the average performance metric in those works seem to be defined similarly, so I wonder whether the gap is due to a difference in hyperparameters.
- What was the methodology for selecting hyperparameters for the baseline algorithms?
- Was there any exploration of the refresh-rate variable n, which determines how often the coverage buffer updates its values? Additionally, how is n selected? I could not find much discussion of this variable.
- The paper does present pseudocode and relevant hyperparameters in the appendix; however, there does not seem to be any attached or referenced source code. Are there any plans to make the code publicly available?
- Is there an explanation for why some numbers in Table 1 are highlighted and others are not? Providing a clear explanation would be helpful.
- The recent work AdaBFM (Karthikeyan et al., "Adaptive Behavioral Foundation Models: Continual Training from Reward-Free Experience," NeurIPS 2026 Workshop on From Pretrained Representations to Acting Agents) seems closely related. I wonder whether the authors have any additional thoughts on how this paper relates to their work.
4 comments
Thank you for the detailed response. I believe most of my concerns have been addressed. Before I give my final assessment, I have a few additional questions/statements given the authors response.
- Which task sequence did you use for Point-Mass Maze, and how did you adapt it into a continual learning setting? For example, did each new task use a different maze size or layout?
- Keeping the color coding in Table 1 is fine, as long as it is clearly explained, such as in the caption or a legend.
We thank the reviewer for the follow-up and feedback.
Q1: For PointMass Maze, we maintained the same assumption of stationary dynamics as in our other environments, keeping the maze layout fixed across all tasks and changing only the goal location. We adapted it to continual learning by defining a sequence of four tasks: reaching the top-left, top-right, bottom-left, and bottom-right corners, respectively. Each task lasts 500K steps, for a total of 2M steps.
Q2: We will keep the color heatmap and clarify in the table caption that it represents relative performance across methods and metrics. Thank you for the suggestion.
My primary reservation continues to be directed toward the paper’s claim that the work can “identify where BFMs are effective and where they are not,” given the limited diversity of environments. Adding Point-Mass Maze is a welcome step toward supporting this claim. However, open questions remain about how well the findings transfer to more general CRL benchmarks, performance across multiple cycles, and discrete action spaces. More broadly, the empirical evaluation and environment coverage could be further strengthened with additional time. I do recognize that some of these points could reasonably be left to future work, and I am therefore inclined to slightly increase my score.
We thank the reviewer for their constructive feedback and suggestions.
W1: We acknowledge that Phases 1 and 2 build on existing methods. However, we believe that combining these techniques into a continual learning framework is itself valuable, as they were not originally designed for this setting. More importantly, our contribution addresses a specific challenge arising from continually updating BFMs: maintaining behavioral coverage while learning from a stream of online experience. Unlike conventional replay mechanisms, the coverage buffer allows the model to incorporate new experience without degrading previously acquired capabilities. A capability proven through our ablations, to isolate the contributions of these components.
W2: We conducted the coverage-buffer ablations on Quadruped, Walker, and Cheetah. We excluded CW10 because the poor performance of BFMs on manipulation tasks makes it difficult to isolate the effects of continual updates, but retained it in the main evaluation to highlight this limitation. While the ablations are limited to DMC, they support the benefits of the coverage buffer. We also added PointMass Maze to provide further evidence beyond locomotion tasks (W3).
W3: Our evaluation follows established BFM benchmarks to isolate the contribution of continual updates. We also considered COOM, MiniHack, and Procgen, but COOM's pixel observations would introduce additional complexity in representation learning, making it harder to isolate the contribution of our method, while MiniHack and Procgen's discrete actions would require adapting our implementations and baselines. Given the limited rebuttal timeline, we added PointMass Maze, which, although part of DMC, introduces goal-directed navigation rather than locomotion, closer in task structure to MiniHack. Results are reported below:
On PointMass, Continual FB achieves the highest accuracy among boundary-free approaches (0.72), while maintaining near-zero forgetting and improving over frozen OpTI-FB (0.56). Its performance is also comparable to EWC (0.73), which requires task boundaries.
Q1: We thank the reviewer for pointing out this discrepancy. One possible explanation is the difference in the underlying MetaWorld implementations: our experiments use MetaWorld V3, whereas the original Continual World benchmark uses an earlier version. However, to investigate further, we conducted an additional sweep over the EWC regularization coefficient λ. With a smaller coefficient, EWC performance improved from 0.12 to 0.31 ACC. We will update the results accordingly and clarify the experimental differences.
Q2: For the CRL baselines, we performed hyperparameter selection starting from the original configurations and then sweeping key parameters (especially learning rates and regularization coefficients). For BFMs, we retained the original pre-training configurations, while for OpTI we tuned β and the decay parameter. The coverage-buffer ratio was selected through our ablations. We will clarify the hyperparameter selection procedure in the appendix.
Q3: The refresh rate n controls how often the coverage buffer recomputes its covariance matrix and leverage scores. We set n=1000 to reduce computational overhead and improve online execution speed, without noticeable performance degradation in preliminary experiments. Since this parameter mainly affects computational efficiency, we omitted a dedicated ablation. We will clarify this in the paper.
Q4: Yes, we plan to publicly release the codebase, including the experimental configurations, upon acceptance.
Q5: The colored cells in Table 1 are intended as a heatmap visualization to facilitate comparisons across methods and metrics, rather than indicate statistical significance. If the reviewer thinks this compromises readability in some way we are happy to remove it.
Q6: We thank the reviewer for pointing out AdaBFM, a complementary work focusing on reward-free adaptation to changing dynamics. Combining the two approaches could be an interesting direction to address both changing dynamics and rewards. We will discuss this connection in the revised paper.