- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies efficient adaptation of behavioral foundation models (BFMs) to unseen test-time dynamics using reward-free online experience. It introduces AdaBFM, which frames offline pretraining as learning an adaptive prior over dynamics. At deployment, the policy conditions on a causal representation of interaction history and a recursively updated belief over an ensemble of successor-feature predictors. The method further supports continual fine-tuning through task-compatible exploration guided by successor-belief information gain. Experiments across navigation, locomotion, and dexterous manipulation benchmarks demonstrate performance improvements over existing baselines, particularly after continual training. Overall, the paper is clearly written and well structured, with relevant implementation details provided in the appendix.
Critical strengths
The paper addresses an important problem, which provides a high contribution to the adaptive zero-shot RL field. Despite several previous works targeting the same problem, neither of them managed to collect all required components for efficient adaptation into unified framework. The idea of performing weighted sampling of potential successor feature in order to estimate best action is interesting, resulting in improved exploration strategies, where prior baselines completely fail.
Critical weaknesses
-
Method contains lots of loss components, making it hard to tune and requires parameter changes depending on prior knowledge of the environment.
-
AdaBFM deteriorates sharply on some shifts and simpler exploration strategies sometimes perform better, limiting claims of robustness.
-
While reading text and exact details of the implementation from the Appendix, I observed several mismatches between claims made. Section 5.2 states that subsequent continual-training experiments use successor information gain, but Appendix A.5 says deployment uses random-feasible exploration. Table 5 lists the task coordinate as frozen even for manipulation, whereas Sections A.3 and B.4 describe updating it using online rewards.
-
The environments being tested are relatively simple
-
No experiments support claims regarding continual adaptation. Only adaptation to a single fixed dynamics per run is reported.
Questions for authors
-
Could authors specify the exact experiment design setting and where does the conflict between report in the main part and Appendix arises?
-
Is it possible to conduct experiments on much harder simulated environments, e.g humanoid under varying dynamics (like wind speed, gravity etc.)?
-
Does AdaBFM could continuosly adapt to dynamics shifts (i.e walls layout could change after each horizon of steps)? How fast does AdaBFM belief estimation of dynamics converges to true transition function?
-
Crucially, which new research directions could AdaBFM open?
Even though not a weakness, I would like authors to clarify whether there exists some established benchmark in the community that could be used for showcasing adaptation performance?
6 comments
Author Response (2/4)
W5. No experiments support claims regarding continual adaptation. Only adaptation to a single fixed dynamics per run is reported.
Q3. Does AdaBFM could continuously adapt to dynamics shifts (i.e., walls layout could change after each horizon of T steps)? How fast does AdaBFM belief estimation of dynamics converges to true transition function?
We thank the reviewer for the suggestion. We extend the PointMaze task to continually changing dynamics by alternating route-blocking gates every 1,000 interaction steps across 20 held-out layouts. After each switch, success requires reaching the goal before the next change. We evaluate both zero-shot adaptation and reward-free continual training, and visualize successor-belief weights, entropy, and trajectories throughout each run (Table 4, Figure 20).
Table: Continual adaptation to dynamics shifts. Goal-reaching success across continually changing PointMaze layouts. ZS and CT denote zero-shot and continual training.
| Method | ZS (%) | CT (%) |
|---|---|---|
| TD-JEPA | 0.0 ± 0.0 | 0.0 ± 0.0 |
| GRU-TD-JEPA | 26.4 ± 3.4 | 30.3 ± 4.3 |
| MetaBFM | 31.1 ± 2.1 | 38.3 ± 2.9 |
| AdaBFM | 33.3 ± 5.0 | 50.8 ± 5.5 |
AdaBFM maintains a belief over learned successor hypotheses and recursively updates their weights using Bellman residuals from observed transitions. Since these beliefs approximate uncertainty over successor predictions rather than the true transition function, we assess adaptation dynamics through belief concentration and stabilization over ensemble members. Concentration measures when a hypothesis first sustains the specified probability threshold for consecutive steps, while stabilization requires the final dominant hypothesis to retain it through the remaining rollout.
| PointMaze measurement | Mean ± SD |
|---|---|
| Time for ≥50% posterior mass on a single member for 32 consecutive steps | 700.4 ± 129.6 steps |
| Stabilization: concentration retained through step 2,000 | 929.0 ± 91.6 steps |
Belief concentration occurs after approximately 700 interactions, equivalent to roughly 2–3 PointMaze episodes, suggesting relatively rapid adaptation from online transition evidence.
During continual training, we also observe improved transition consistency. Across task-successful PointMaze runs, replay Bellman surprise and successor TD loss are approximately 50% and 45% lower, respectively, during interactions 1,001–2,000 compared with interactions 1–1,000.
We additionally present belief-evolution plots and corresponding trajectories for the continually changing dynamics task in Figure 20, illustrating how successor beliefs evolve as new transition evidence accumulates.
W1. Method contains lots of loss components, making it hard to tune and requires parameter changes depending on prior knowledge of the environment.
Most components and their hyperparameters are inherited from TD-JEPA and LCL (Tables 21 and 22), whose default settings we largely retain without per-environment tuning. We additionally examine the two AdaBFM-specific hyperparameters, ensemble size and task-value slack , in Figure 10 and Table 8, respectively. In practice, we find that ensemble members are sufficient and efficiently supported by GPU parallelization. For the slack parameter, achieves better sample-efficiency on maze tasks.
Author Response (3/4)
W2. AdaBFM deteriorates sharply on some shifts and simpler exploration strategies sometimes perform better, limiting claims of robustness.
- Reward-free continual training may result in deterioration in some challenging cases, such as Ant friction 2× (Table 13). In locomotion, pretraining uses only nominal dynamics, without meta-training across environments, this lack of dynamics diversity may contribute to these failures, we also observe that all the baselines suffer this deterioration in Ant friction 2x.
- Appendix C and Figure 17 further show that naive reward-based fine-tuning can suffer from unlearning despite dense feedback. Our task-compatible latent search provides regularization by anchoring exploration to the specified task. Combined with reward feedback, this leads to policy improvements, as shown in Fig.~17A.
- In locomotion, simpler exploration strategies, such as our task-compatible random-feasible strategy, also provide strong baselines. Exploratory actions often have short-lived effects, and subsequent actions can compensate for errors. In contrast, exploration mistakes in navigation (PointMaze and AntMaze), such as entering the wrong corridor or falling, can lead to failures that are difficult to recover from within the interaction budget.
- We additionally observe positive correlations between predicted and realized successor-belief entropy reduction in AntMaze (Figure 13), supporting the usefulness of our surrogate exploration objective. Figure 20 further visualizes how successor beliefs evolve in a continually changing environment.
W3. While reading text and exact details of the implementation from the Appendix, I observed several mismatches between claims made. Section 5.2 states that subsequent continual-training experiments use successor information gain, but Appendix A.5 says deployment uses random-feasible exploration. Table 5 lists the task coordinate as frozen even for manipulation, whereas Sections A.3 and B.4 describe updating it using online rewards.
Q1. Could authors specify the exact experiment design setting and where does the conflict between report in the main part and Appendix arises?
-
Exploration protocol. AdaBFM and AdaBFM$_{-\mathrm{LCL}}$ use successor-belief IG to select exploratory behaviors within the task-compatible feasible set. The random-feasible strategy refers to the non-AdaBFM baselines, which sample uniformly from the same feasible set. We have clarified this distinction in Appendix A.6 and Table 6.
-
Task coordinate and reward usage. The task coordinate is inferred from offline reward-labeled data and remains frozen during reward-free CT, including in manipulation and the newly added humanoid experiments. Only in the separately reported reward-based Adroit experiments is updated using online rewards through optimistic task inference. We have updated Table 6 to explicitly distinguish frozen in reward-free CT from updated in reward-based CT.
We also summarize the complete experimental protocol in our Common Response.
Author Response (1/4)
Thank you for your review and questions. We respond to your questions below, and we kindly invite you to also consult the updated manuscript and the common response.
We extend our evaluation to G1 humanoid motion tracking, similar to Li et al. (2025), adapting this setting to study reward-free CT under unseen dynamics. As in our other experiments, the task coordinates inferred under nominal dynamics remain frozen, while the policy, successor representations and context encoders are adapted using reward-free online transitions only. We evaluate 14 unseen dynamics shifts, including wind, friction, slopes, and terrain roughness (Figure 4, Tables 16–18).
Tracking success (%) across 14 dynamics shifts, before and after reward-free CT.
AdaBFM$_{-\mathrm{LCL}}$ achieves the highest mean tracking success on 13/14 shifts, both zero-shot and after reward-free CT. This is consistent with prior observations that explicit history conditioning can impair humanoid transfer and robotic policy execution (Liao et al., 2026; Park & Levine, 2026). Importantly, removing LCL does not make AdaBFM history-free, as its recursive successor beliefs continue integrating evidence from past transitions. Unlike navigation, where LCL helps distinguish layout-dependent dynamics, humanoid tracking preserves the motion objective while varying physical properties. We have expanded this discussion on the task-dependent role of memory in the revised main text (L474–509).
References
Li et al. (2025). BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning, ICLR.
Liao et al. (2026). BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion, Science Robotics.
Park, S., & Levine, S. (2026). Behavioral Cloning Mystery. arXiv preprint arXiv:2610.07056.