acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Behavioral Foundation Models: Continual Training from Reward-Free Experience

Abstract

Behavioral foundation models (BFMs) learn general-purpose policies and successor features from reward-free offline data, enabling agents to retrieve a policy for a given reward function without additional learning or planning. However, this policy can fail when test-time dynamics differ from those seen during pretraining. Memory-based approaches address this mismatch through in-context learning without updating model parameters. We study reward-free continual training for adaptation to unseen dynamics within a limited online interaction budget. We introduce AdaBFM, which frames pretraining as learning an adaptive prior distribution over test-time dynamics. At test time, it adapts this prior through continual fine-tuning on reward-free online experience. The zero-shot policy is conditioned on causal memory and a belief over successor ensemble members, allowing it to adapt as evidence accumulates, while successor-belief information gain guides continual training. We evaluate AdaBFM across an aggregate of 23 navigation, locomotion, and manipulation tasks under unseen dynamics. AdaBFM achieves the highest mean performance in 9 out of 23 tasks in the zero-shot setting and 16 out of 23 after continual training, improving by an average of 28% on harder tasks.

Primary area:
reinforcement learning
Keywords:
Unsupervised reinforcement learning, Finetuning, zero-shot reinforcement learning, behavior foundation models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TthjOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies efficient adaptation of behavioral foundation models (BFMs) to unseen test-time dynamics using reward-free online experience. It introduces AdaBFM, which frames offline pretraining as learning an adaptive prior over dynamics. At deployment, the policy conditions on a causal representation of interaction history and a recursively updated belief over an ensemble of successor-feature predictors. The method further supports continual fine-tuning through task-compatible exploration guided by successor-belief information gain. Experiments across navigation, locomotion, and dexterous manipulation benchmarks demonstrate performance improvements over existing baselines, particularly after continual training. Overall, the paper is clearly written and well structured, with relevant implementation details provided in the appendix.

Critical strengths

The paper addresses an important problem, which provides a high contribution to the adaptive zero-shot RL field. Despite several previous works targeting the same problem, neither of them managed to collect all required components for efficient adaptation into unified framework. The idea of performing weighted sampling of potential successor feature in order to estimate best action is interesting, resulting in improved exploration strategies, where prior baselines completely fail.

Critical weaknesses
  1. Method contains lots of loss components, making it hard to tune and requires parameter changes depending on prior knowledge of the environment.

  2. AdaBFM deteriorates sharply on some shifts and simpler exploration strategies sometimes perform better, limiting claims of robustness.

  3. While reading text and exact details of the implementation from the Appendix, I observed several mismatches between claims made. Section 5.2 states that subsequent continual-training experiments use successor information gain, but Appendix A.5 says deployment uses random-feasible exploration. Table 5 lists the task coordinate as frozen even for manipulation, whereas Sections A.3 and B.4 describe updating it using online rewards.

  4. The environments being tested are relatively simple

  5. No experiments support claims regarding continual adaptation. Only adaptation to a single fixed dynamics per run is reported.

Questions for authors
  1. Could authors specify the exact experiment design setting and where does the conflict between report in the main part and Appendix arises?

  2. Is it possible to conduct experiments on much harder simulated environments, e.g humanoid under varying dynamics (like wind speed, gravity etc.)?

  3. Does AdaBFM could continuosly adapt to dynamics shifts (i.e walls layout could change after each horizon of TT steps)? How fast does AdaBFM belief estimation of dynamics converges to true transition function?

  4. Crucially, which new research directions could AdaBFM open?

Even though not a weakness, I would like authors to clarify whether there exists some established benchmark in the community that could be used for showcasing adaptation performance?

6 comments
AuthorsOpenReview ↗

Author Response (1/4)

Thank you for your review and questions. We respond to your questions below, and we kindly invite you to also consult the updated manuscript and the common response.

W4. The environments being tested are relatively simple.

Q2. Is it possible to conduct experiments on much harder simulated environments, e.g., humanoid under varying dynamics (like wind speed, gravity, etc.)?

We extend our evaluation to G1 humanoid motion tracking, similar to Li et al. (2025), adapting this setting to study reward-free CT under unseen dynamics. As in our other experiments, the task coordinates inferred under nominal dynamics remain frozen, while the policy, successor representations and context encoders are adapted using reward-free online transitions only. We evaluate 14 unseen dynamics shifts, including wind, friction, slopes, and terrain roughness (Figure 4, Tables 16–18).

Tracking success (%) across 14 dynamics shifts, before and after reward-free CT.

MethodZS (%)RF CT (%)
TD-JEPA41.9 ± 28.839.6 ± 27.6
GRU-TD-JEPA21.9 ± 32.821.0 ± 29.8
MetaBFM32.2 ± 47.632.4 ± 47.0
AdaBFM24.0 ± 19.629.7 ± 16.2
AdaBFM$_{-\mathrm{LCL}}$71.9 ± 10.374.5 ± 11.2

AdaBFM$_{-\mathrm{LCL}}$ achieves the highest mean tracking success on 13/14 shifts, both zero-shot and after reward-free CT. This is consistent with prior observations that explicit history conditioning can impair humanoid transfer and robotic policy execution (Liao et al., 2026; Park & Levine, 2026). Importantly, removing LCL does not make AdaBFM history-free, as its recursive successor beliefs continue integrating evidence from past transitions. Unlike navigation, where LCL helps distinguish layout-dependent dynamics, humanoid tracking preserves the motion objective while varying physical properties. We have expanded this discussion on the task-dependent role of memory in the revised main text (L474–509).

References

Li et al. (2025). BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning, ICLR.

Liao et al. (2026). BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion, Science Robotics.

Park, S., & Levine, S. (2026). Behavioral Cloning Mystery. arXiv preprint arXiv:2610.07056.

AuthorsOpenReview ↗

Author Response (2/4)

W5. No experiments support claims regarding continual adaptation. Only adaptation to a single fixed dynamics per run is reported.

Q3. Does AdaBFM could continuously adapt to dynamics shifts (i.e., walls layout could change after each horizon of T steps)? How fast does AdaBFM belief estimation of dynamics converges to true transition function?

We thank the reviewer for the suggestion. We extend the PointMaze task to continually changing dynamics by alternating route-blocking gates every 1,000 interaction steps across 20 held-out layouts. After each switch, success requires reaching the goal before the next change. We evaluate both zero-shot adaptation and reward-free continual training, and visualize successor-belief weights, entropy, and trajectories throughout each run (Table 4, Figure 20).

Table: Continual adaptation to dynamics shifts. Goal-reaching success across continually changing PointMaze layouts. ZS and CT denote zero-shot and continual training.

MethodZS (%)CT (%)
TD-JEPA0.0 ± 0.00.0 ± 0.0
GRU-TD-JEPA26.4 ± 3.430.3 ± 4.3
MetaBFM31.1 ± 2.138.3 ± 2.9
AdaBFM33.3 ± 5.050.8 ± 5.5

AdaBFM maintains a belief over learned successor hypotheses and recursively updates their weights using Bellman residuals from observed transitions. Since these beliefs approximate uncertainty over successor predictions rather than the true transition function, we assess adaptation dynamics through belief concentration and stabilization over ensemble members. Concentration measures when a hypothesis first sustains the specified probability threshold for consecutive steps, while stabilization requires the final dominant hypothesis to retain it through the remaining rollout.

PointMaze measurementMean ± SD
Time for ≥50% posterior mass on a single member for 32 consecutive steps700.4 ± 129.6 steps
Stabilization: concentration retained through step 2,000929.0 ± 91.6 steps

Belief concentration occurs after approximately 700 interactions, equivalent to roughly 2–3 PointMaze episodes, suggesting relatively rapid adaptation from online transition evidence.

During continual training, we also observe improved transition consistency. Across task-successful PointMaze runs, replay Bellman surprise and successor TD loss are approximately 50% and 45% lower, respectively, during interactions 1,001–2,000 compared with interactions 1–1,000.

We additionally present belief-evolution plots and corresponding trajectories for the continually changing dynamics task in Figure 20, illustrating how successor beliefs evolve as new transition evidence accumulates.


W1. Method contains lots of loss components, making it hard to tune and requires parameter changes depending on prior knowledge of the environment.

Most components and their hyperparameters are inherited from TD-JEPA and LCL (Tables 21 and 22), whose default settings we largely retain without per-environment tuning. We additionally examine the two AdaBFM-specific hyperparameters, ensemble size KK and task-value slack ϵ\epsilon, in Figure 10 and Table 8, respectively. In practice, we find that K=10K=10 ensemble members are sufficient and efficiently supported by GPU parallelization. For the slack parameter, ϵ=0.5\epsilon=0.5 achieves better sample-efficiency on maze tasks.

AuthorsOpenReview ↗

Author Response (3/4)

W2. AdaBFM deteriorates sharply on some shifts and simpler exploration strategies sometimes perform better, limiting claims of robustness.

  • Reward-free continual training may result in deterioration in some challenging cases, such as Ant friction 2× (Table 13). In locomotion, pretraining uses only nominal dynamics, without meta-training across environments, this lack of dynamics diversity may contribute to these failures, we also observe that all the baselines suffer this deterioration in Ant friction 2x.
  • Appendix C and Figure 17 further show that naive reward-based fine-tuning can suffer from unlearning despite dense feedback. Our task-compatible latent search provides regularization by anchoring exploration to the specified task. Combined with reward feedback, this leads to policy improvements, as shown in Fig.~17A.
  • In locomotion, simpler exploration strategies, such as our task-compatible random-feasible strategy, also provide strong baselines. Exploratory actions often have short-lived effects, and subsequent actions can compensate for errors. In contrast, exploration mistakes in navigation (PointMaze and AntMaze), such as entering the wrong corridor or falling, can lead to failures that are difficult to recover from within the interaction budget.
  • We additionally observe positive correlations between predicted and realized successor-belief entropy reduction in AntMaze (Figure 13), supporting the usefulness of our surrogate exploration objective. Figure 20 further visualizes how successor beliefs evolve in a continually changing environment.

W3. While reading text and exact details of the implementation from the Appendix, I observed several mismatches between claims made. Section 5.2 states that subsequent continual-training experiments use successor information gain, but Appendix A.5 says deployment uses random-feasible exploration. Table 5 lists the task coordinate as frozen even for manipulation, whereas Sections A.3 and B.4 describe updating it using online rewards.

Q1. Could authors specify the exact experiment design setting and where does the conflict between report in the main part and Appendix arises?

  • Exploration protocol. AdaBFM and AdaBFM$_{-\mathrm{LCL}}$ use successor-belief IG to select exploratory behaviors within the task-compatible feasible set. The random-feasible strategy refers to the non-AdaBFM baselines, which sample uniformly from the same feasible set. We have clarified this distinction in Appendix A.6 and Table 6.

  • Task coordinate and reward usage. The task coordinate z∗z^{\ast} is inferred from offline reward-labeled data and remains frozen during reward-free CT, including in manipulation and the newly added humanoid experiments. Only in the separately reported reward-based Adroit experiments is z∗z^{\ast} updated using online rewards through optimistic task inference. We have updated Table 6 to explicitly distinguish frozen z∗z^{\ast} in reward-free CT from updated z∗z^{\ast} in reward-based CT.

We also summarize the complete experimental protocol in our Common Response.

Reviewer_TFkAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Behavioural foundation models (BFMs) often fail at test time because of differences between training and test-time dynamics. This work proposes a reward-free continual training framework for BFMs to adapt to new dynamics at test time. The method, AdaBFM, has a memory module that encodes interaction history into a context. An ensemble of successor predictors takes the context, state/task encodings and actions as input, and a belief over the ensemble members is updated based on how well each member’s Bellman residual fits the observed transitions. The policy learns to output actions given the belief, context and task. The proposed approach collects informative data during test time by exploring the latent policy space and filtering candidates based on their predicted values and information gain on successor beliefs. Using the explored candidates and the frozen test-time task coordinate, the memory module, successor ensemble and policy are updated. The proposed approach matches baselines in terms of zero-shot success and achieves strong continual training results across different tasks.

Critical strengths
  • The idea is well explained and nicely motivated.
  • The method is sound and makes sense.
  • Restricting exploration to task-compatible candidates is a nice idea.
  • Experimental coverage and ablation studies are extensive.
Critical weaknesses
  • Results in Table 3b and Figure 4 show other methods work as well or better in several settings. Can you please elaborate on these, or extend these experiments so that the difference is clearer?
  • Main results in Figure 3, especially in locomotion and Adroit Pen, look inconclusive since MetaBFM performs fairly close to AdaBFM.
  • The main-text related work never positions AdaBFM relative to MetaBFM, and the difference is only mentioned briefly in the Baselines paragraph. Given the small gaps to MetaBFM in the results and the shared memory module (LCL), the benefit of AdaBFM over MetaBFM is not clearly established. A more detailed discussion and more examples similar to Figure 2b would be helpful.
  • In the Adroit Pen tasks, z* is refined using online rewards, and AdaBFM only outperforms the baselines in this reward-based setting while underperforming in the paper’s main (reward-free) setting.
Questions for authors

See also weaknesses.

  • MetaBFM also works in the unseen-reward setting, although AdaBFM is never tested in such settings. Can you elaborate whether AdaBFM can be applied to such settings?
6 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for your review and questions. We respond to your questions below, and we kindly invite you to also consult the updated manuscript and the common response.

W3. The main-text related work never positions AdaBFM relative to MetaBFM, and the difference is only mentioned briefly in the Baselines paragraph. Given the small gaps to MetaBFM in the results and the shared memory module (LCL), the benefit of AdaBFM over MetaBFM is not clearly established. A more detailed discussion and more examples similar to Figure 2b would be helpful.

Qualitative successor representations. We have added extensive qualitative visualizations (Figures 18–19, Appendix B.5) showing how successor representations reflect MDP dynamics and long-horizon goal reachability. AdaBFM better distinguishes maze-dependent transition constraints, whereas MetaBFM often exhibits uniformly high successor–goal similarities, suggesting overly optimistic reachability predictions and potentially averaged representations across different dynamics.

Distinction from MetaBFM. Although AdaBFM and MetaBFM both use LCL for causal context learning, they differ in how they represent and resolve uncertainty under unseen dynamics. Prior contextual BFM approaches (BFB and MetaBFM) use random policies or off-the-shelf intrinsic exploratory behavior to obtain coverage, which becomes harder to achieve as task complexity increases. AdaBFM instead maintains multiple successor hypotheses, preserving counterfactual explanations of the observed dynamics, and recursively reweights them using observed transitions. It further selects task-compatible exploratory behaviors through predicted successor-belief entropy reduction, targeting transitions that distinguish competing hypotheses. This distinction is particularly evident in challenging AntMaze navigation (Figure 3b, Table 14), while our new humanoid results with AdaBFM$_{-\mathrm{LCL}}$ (Figure 4, Tables 16–18) further demonstrate benefits beyond the shared LCL module.

Additional humanoid evaluation. We extend our evaluation to 14 unseen G1 humanoid dynamics shifts (Figure 4, Tables 16–18).

Tracking success (%) across 14 dynamics shifts, before and after reward-free continual training.

MethodZS (%)RF CT (%)
TD-JEPA41.9 ± 28.839.6 ± 27.6
GRU-TD-JEPA21.9 ± 32.821.0 ± 29.8
MetaBFM32.2 ± 47.632.4 ± 47.0
AdaBFM24.0 ± 19.629.7 ± 16.2
AdaBFM$_{-\mathrm{LCL}}$71.9 ± 10.374.5 ± 11.2

Role of context. AdaBFM−LCL\mathrm{AdaBFM}_{-\mathrm{LCL}} achieves the highest mean tracking success on 13/14 shifts, both zero-shot and after reward-free CT, exceeding the strongest baseline TD-JEPA by absolute margins of approximately 30% and 35%, respectively. These results highlight adaptation capabilities beyond the LCL module shared with MetaBFM. In navigation, LCL helps distinguish layout-dependent dynamics (Table 10), whereas in humanoid control, removing LCL improves transfer. Importantly, AdaBFM$_{-\mathrm{LCL}}$ remains history-dependent through recursive successor-belief updates. We have expanded this discussion in the revised main text (L474–509).

AuthorsOpenReview ↗

Author Response (2/3)

W1. Results in Table 3b and Figure 4 show other methods work as well or better in several settings. Can you please elaborate on these, or extend these experiments so that the difference is clearer?

We refer the reviewer to Tables 12-13 and Figure 12 for the full locomotion results and training progression, beyond the relative improvements shown in Figure 5. AdaBFM achieves the highest mean return on 6/12 configurations zero-shot and 8/12 after reward-free CT. Importantly, we introduce task-compatible random-feasible exploration as a strong ensemble-free strategy (Table 11), which also enables baselines such as TD-JEPA to benefit from continual training.

In locomotion, exploratory mistakes are often recoverable through subsequent actions, making simpler exploration effective. In contrast, navigation errors can leave agents trapped in corridors, making belief-IG-guided exploration particularly beneficial in challenging AntMaze layouts, where uncertainty about unseen dynamics has longer-lasting consequences.

Dexterous manipulation (Table 3b, Table 15). Broken joints and shrunk fingers can invalidate pretrained manipulation skills, making behavioral recovery challenging. Prior work suggests that residual and recurrent control can use action-response history to compensate for broken joints [1,2]. To better understand these adaptation mechanisms, we extend our evaluation with Ada-GRU, which replaces LCL with successor-trained recurrent memory, and AdaBFM$_{-\mathrm{LCL}}$, which removes the explicit context encoder while retaining recursive successor-belief inference. Importantly, the latter remains history-dependent through its belief updates.

MethodZS (%)RF CT (%)R CT (%)
TD-JEPA8.8 ± 4.914.2 ± 8.224.0 ± 1.9
BFB4.7 ± 1.96.7 ± 4.716.0 ± 3.8
GRU-TD-JEPA25.0 ± 4.720.2 ± 9.222.7 ± 4.2
MetaBFM4.2 ± 4.59.7 ± 5.222.8 ± 2.1
AdaBFM7.0 ± 9.016.7 ± 4.725.5 ± 0.7
Ada-GRU16.0 ± 0.314.9 ± 1.816.3 ± 2.5
AdaBFM$_{-\mathrm{LCL}}$21.9 ± 1.819.9 ± 2.820.8 ± 4.3

Under broken-joint shifts, GRU-TD-JEPA achieves the highest zero-shot success among the compared methods, consistent with recurrent memory helping compensate for actuator faults. AdaBFM$_{-\mathrm{LCL}}$ achieves comparable aggregate reward-free CT performance to GRU-TD-JEPA (19.9% vs. 20.2%), with higher success on easy broken joints (43.3% vs. 37.0%) and medium shrunk fingers (38.7% vs. 32.0%).

Under the hardest shrunk-finger shift, full AdaBFM improves from 9.0% zero-shot to 30.0% after reward-free CT, compared with TD-JEPA (25.0%), GRU-TD-JEPA (16.0%), and AdaBFM$_{-\mathrm{LCL}}$ (15.3%) after CT. This suggests that explicit context learning can provide additional benefits under severe changes in finger geometry, while recurrent memory and recursive successor beliefs help with correcting broken joints or actuator faults. We additionally evaluate reward-based CT with sparse binary feedback to study recovery under these challenging manipulation shifts.

Dexterous manipulation remains a challenging adaptation problem, with even the strongest methods achieving only around 20% aggregate success under reward-free CT. We have expanded the discussion of these contrasting adaptation mechanisms and limitations in the revised main text (L463–473) and Appendix B.4.

References

[1] Linan Deng, Xing Liu, Lin Hong, Feng Hua, Guijun Ma, Zuogong Yue, and Fumin Zhang. Residual fault adaptation for dexterous in-hand manipulation under runtime joint faults. arXiv preprint arXiv:2609.17404, 2026

[2] OpenAI et al. (2019). Solving Rubik's Cube with a Robot Hand.

AuthorsOpenReview ↗

Author Response (3/3)

W2. Main results in Figure 3, especially in locomotion and Adroit Pen, look inconclusive since MetaBFM performs fairly close to AdaBFM.

W4. In the Adroit Pen tasks, z∗z^* is refined using online rewards, and AdaBFM only outperforms the baselines in this reward-based setting while underperforming in the paper’s main (reward-free) setting.

Locomotion (W2). In locomotion, AdaBFM achieves the highest mean return on 6/12 dynamics shifts zero-shot and 8/12 after reward-free CT (Tables 12–13, Figure 12). Here, exploratory actions are often recoverable, as subsequent actions can compensate for temporary deviations, making simpler exploration strategies competitive. Even memoryless TD-JEPA remains competitive in locomotion, despite achieving near-zero success on unseen PointMaze layouts. In navigation, competing successor hypotheses can imply distinct routes, and incorrect exploration can leave agents trapped in corridors, making belief-guided exploration more consequential.

  • The AdaBFM−LCL\mathrm{AdaBFM}_{-\mathrm{LCL}} results (Tables 12–13) further clarify this distinction: removing LCL yields higher post-CT returns on all three HalfCheetah friction shifts and marginally on easy Ant morphology, but substantially underperforms full AdaBFM on HalfCheetah morphology and medium/hard Ant morphology. This suggests that recursive successor beliefs alone can suffice for more recoverable shifts, while explicit context learning can benefit more challenging structural changes.

Adroit Pen (W2) presents a challenging adaptation problem, where disabled joints and shortened fingers can invalidate pretrained manipulation skills, requiring the discovery of new behaviors. Even the strongest methods achieve only around 20% aggregate success after reward-free CT. Nevertheless, full AdaBFM improves over MetaBFM in aggregate success both zero-shot (7.0% vs. 4.2%) and after reward-free CT (16.7% vs. 9.7%). AdaBFM$_{-\mathrm{LCL}}$ achieves 21.9% zero-shot and 19.9% after reward-free CT, outperforming MetaBFM on 6/6 shifts zero-shot and 5/6 after CT (Table 15). These results further highlight the benefit of successor-belief adaptation beyond the LCL module shared with MetaBFM.

Reward usage (W4). In reward-free CT, the task coordinate z∗z^{\ast} remains frozen. Only the separately reported reward-based CT uses sparse binary online rewards to refine z∗z^{\ast} through optimistic task inference. For example, a disabled thumb joint can cause the agent to drop the pen, making recovery difficult. Since pretraining covers only nominal dynamics, the offline data provide limited coverage of these severe actuator and morphology shifts, making reward-free recovery difficult and sometimes leading to performance degradation. We therefore also evaluate reward-based CT as a diagnostic to examine whether task feedback can facilitate the discovery of new manipulation behaviors under severe dynamics shifts.

Q1. MetaBFM also works in the unseen-reward setting, although AdaBFM is never tested in such settings. Can you elaborate whether AdaBFM can be applied to such settings?

  • Yes. BFMs generally support adaptation to unseen reward objectives through task inference from offline reward-labeled samples, such as specifying a new goal within the same maze. The extent of generalization depends in part on the coverage and behavioral diversity of the pretraining data. OpTI-BFM extends this capability by inferring the task coordinate (z) online from reward feedback collected during deployment. MetaBFM adopts optimistic task inference in its BFM/Zero-shot RL mode. It also supports an optional supervised meta-RL mode, which uses extrinsic rewards during training to learn behavior conditioned on reward history. Generalization in this mode depends on the diversity and coverage of the reward-labeled training tasks.
  • We present results on adaptation to non-stationary rewards and present results in Appendix C.1 and in Figure 16 and Table 20
Reviewer_pW1ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AdaBFM, a behavioral foundation model that adapts to unseen dynamics through reward-free online experience. It builds on TD-JEPA and latent context learning, adding an ensemble of successor predictors whose weights are updated using Bellman errors. An information-gain objective selects exploratory behaviors that retain high predicted value for the target task. After initial task inference from reward-labeled data, online transitions update the context encoder, successor model, and policy while keeping the task representation fixed. Experiments cover navigation, locomotion, and dexterous manipulation. The clearest evidence comes from navigation, where controlled ablations support both the exploration strategy and parameter adaptation.

Critical strengths
  1. The exploration ablation is useful. It compares collection strategies under the same training conditions and interaction budget, providing evidence that the proposed exploration rule improves adaptation on hard PointMaze tasks (Table 2).
  2. The results on unseen AntMaze layouts show a meaningful benefit over memory-based adaptation alone (Table 13). This supports the practical value of continuing to train BFMs without reward feedback.
Critical weaknesses
  1. The experimental protocol is unclear in an important place. Section 5.2 says the main experiments use information-gain exploration, while Appendix A.5 says they use random-feasible exploration. The headline comparison also appears to include reward-based manipulation results. These inconsistencies make it difficult to assess how much of the reported improvement supports the proposed reward-free method.
  2. I am less convinced by the interpretation of the exploration objective as information gain. It treats successor predictions as hypothetical observations, while the actual belief update uses observed transitions and Bellman errors (Eqs. 5–9). The connection between these updates needs explanation. The ablation shows that the rule can help, but does not establish that it reduces uncertainty in the claimed way.
Questions for authors
  1. Which exploration strategy was used for the main results, and which headline results include online rewards?
  2. What justifies the hypothetical belief update used in the information-gain objective? Is there evidence that its predictions agree with the actual belief updates?
1 comment
AuthorsOpenReview ↗

Author Response

Thank you for your review and questions. We respond to your questions below, and we kindly invite you to also consult the updated manuscript and the common response.

W1. The experimental protocol is unclear in an important place. Section 5.2 says the main experiments use information-gain exploration, while Appendix A.5 says they use random-feasible exploration. The headline comparison also appears to include reward-based manipulation results. These inconsistencies make it difficult to assess how much of the reported improvement supports the proposed reward-free method.

Q1. Which exploration strategy was used for the main results, and which headline results include online rewards?

AdaBFM and AdaBFM$_{-\mathrm{LCL}}$ use successor-belief IG exploration within the task-compatible feasible set. Other baselines use random-feasible exploration, the strongest non-IG strategy in our PointMaze ablation (47.5% vs. 46.7% for successor disagreement and 45.0% for LCL-error exploration, Table 2a), which also requires no ensemble or context learning. We have clarified this distinction in Appendix A.6 and Table 6.

Evaluation settingOnline rewardsTask coordinate (z∗)(z^*)
NavigationNoFrozen
LocomotionNoFrozen
Humanoid trackingNoFrozen
Adroit, reward-free CTNoFrozen
Adroit, reward-based CTYesUpdated through optimistic task inference

In the main comparison, only Adroit results labeled R CT use online rewards (Figure 3d, Tables 3b and 15). These additional experiments examine behavioral recovery under extreme dynamics and morphology shifts.

The reward-based locomotion experiments in Appendix C serve as a diagnostic, showing that task-compatible latent constraints can mitigate policy unlearning compared with direct online RL, even with dense reward feedback.


W2. I am less convinced by the interpretation of the exploration objective as information gain. It treats successor predictions as hypothetical observations, while the actual belief update uses observed transitions and Bellman errors (Eqs. 5-9). The connection between these updates needs explanation. The ablation shows that the rule can help, but does not establish that it reduces uncertainty in the claimed way.

Q2. What justifies the hypothetical belief update used in the information-gain objective? Is there evidence that its predictions agree with the actual belief updates?

Similar to Eq. (9) of Ghosh et al. (2022), which uses squared Bellman residuals as a surrogate negative log-likelihood of observed transitions, we extend this construction from scalar value functions to vector-valued successor predictions. Each ensemble member's successor Bellman residual provides a proxy for its compatibility with observed transitions, allowing us to update the belief without explicitly modeling transition likelihoods (Eqs. 5-6).

For exploration, we treat each ensemble member's predicted successor features as a hypothetical observation and apply the same compatibility kernel to estimate how alternative hypotheses would be reweighted (Eq. 8). We then select task-compatible exploratory candidates with the highest predicted entropy reduction (Eq. 9), collecting transitions that help distinguish competing successor hypotheses and refine successor predictions and policies during reward-free CT. Actual belief updates remain grounded in observed transitions.

We compare the selected behavior's predicted successor-belief entropy reduction at the start of each window with the realized reduction over the subsequent 64 and 128 interactions in AntMaze (Figure 13), obtaining Spearman correlations of 0.47 ± 0.14 and 0.61 ± 0.18, respectively. These correlations support predicted entropy reduction as a useful surrogate for realized uncertainty reduction in the learned successor belief. Figure 20 further visualizes belief evolution as new transition evidence accumulates in the continually changing dynamics task.

  1. Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning, pp. 7513–7530. PMLR, 2022.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response

We thank all reviewers for their feedback and questions on AdaBFM.

We have individually addressed the reviewers' concerns and updated the manuscript, with changes highlighted in red. Below, we highlight the main revisions and address concerns shared across reviewers.

Summary of revisions

Reward-free humanoid motion tracking (TFkA, Tthj). We extend our evaluation to 14 unseen G1 humanoid dynamics shifts with reward-free CT (Figure 4, Tables 16–18).

MethodZS (%)RF CT (%)
TD-JEPA41.9 ± 28.839.6 ± 27.6
GRU-TD-JEPA21.9 ± 32.821.0 ± 29.8
MetaBFM32.2 ± 47.632.4 ± 47.0
AdaBFM24.0 ± 19.629.7 ± 16.2
AdaBFM$_{-\mathrm{LCL}}$71.9 ± 10.374.5 ± 11.2

AdaBFM$_{-\mathrm{LCL}}$ leads in 13/14 shifts, outperforming TD-JEPA by an absolute margin of approximately 30% zero-shot and 35% after CT.

History-dependent adaptation beyond LCL (TFkA, Tthj). AdaBFM$_{-\mathrm{LCL}}$ remains history-dependent through recursive successor-belief updates, despite removing the explicit context encoder. Full AdaBFM additionally learns a causal dynamics context ctc_t, which helps distinguish layout-dependent transition patterns when pretraining across diverse mazes (Table 10). In humanoid tracking, however, dynamics shifts vary physical properties while preserving the motion objective, making explicit context encoding less beneficial. These results distinguish recursive belief-based adaptation from explicit context learning (L479–L509).

Continually changing dynamics (Tthj). We extend PointMaze to dynamics shifts every 1,000 interactions, evaluating zero-shot and reward-free CT (Section 5.3, Table 4). Figure 20 visualizes the evolution of successor beliefs and corresponding trajectories throughout these shifts.

MethodZS (%)CT (%)
TD-JEPA0.0 ± 0.00.0 ± 0.0
GRU-TD-JEPA26.4 ± 3.430.3 ± 4.3
MetaBFM31.1 ± 2.138.3 ± 2.9
AdaBFM33.3 ± 5.050.8 ± 5.5

Qualitative successor representations (TFkA). We add extensive visualizations (Figures 18–19, Appendix B.5) showing how successor representations reflect MDP dynamics and long-horizon goal reachability. AdaBFM better distinguishes maze-dependent reachability, while MetaBFM often exhibits overly optimistic predictions.

Experimental protocol and reward usage (pW1R, Tthj, TFkA). We clarify that both AdaBFM and AdaBFM$_{-\mathrm{LCL}}$ use successor-belief IG exploration, while non-AdaBFM baselines use random-feasible exploration. We have also updated the manuscript to reflect these protocol clarifications (Appendix A.6, Table 6).

Evaluation settingOnline rewardsTask coordinate (z∗)(z^*)
NavigationNoFrozen
LocomotionNoFrozen
Humanoid trackingNoFrozen
Adroit, reward-free CTNoFrozen
Adroit, reward-based CTYesUpdated through optimistic task inference
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Updates

We thank all reviewers for their feedback and questions. We would like to highlight AdaBFM's central contribution.

Reward-free continual adaptation under unseen dynamics. Existing BFMs have primarily addressed dynamics mismatch through zero-shot or in-context adaptation, without updating parameters of pretrained policy and successor representations at deployment. AdaBFM takes a different approach: we frame pretraining as learning an adaptive prior (belief-weighted pretraining) over test-time dynamics, enabling zero-shot adaptation with a history-dependent policy through recursive successor belief updates, while allowing this prior to be rapidly and continually refined from reward-free online experience.

We are encouraged that the reviewers recognize the significance of these contributions:

  • pW1R: "The results on unseen AntMaze layouts show a meaningful benefit over memory-based adaptation alone."
  • TFkA: "The method is sound and makes sense." The reviewer also notes that "Experimental coverage and ablation studies are extensive."
  • Tthj: "The paper addresses an important problem, which provides a high contribution to the adaptive zero-shot RL field."

Comments on the ODRL benchmark. The ODRL benchmark (Lyu et al., 2024) explicitly identifies AntMaze as extremely challenging, reporting that existing methods struggle to achieve meaningful returns under dynamics shifts (Section 5.1, Observation 3). This difficulty extends to large AntMaze layouts (Appendix E.2, Figure 18), where most evaluated methods achieve zero normalized scores despite access to offline target-layout trajectories. ODRL similarly identifies dexterous manipulation as challenging due to high-dimensional, sparse-reward control, with most methods struggling even with expert offline data (Section 4.1, Appendix E.2, Figure 16). In contrast, AdaBFM achieves 40.8% success on unseen large AntMaze layouts after reward-free CT, without access to offline test-layout data.

Additional experiments. We have extended our experiments with 14 G1 humanoid dynamics shifts, continually changing PointMaze dynamics, extensive qualitative visualizations of successor-representation, additional LCL and recurrent-memory ablations, and empirical validation of predicted entropy reduction.

To summarize the results, we report below the number of configurations where either full AdaBFM or AdaBFM$_{-\mathrm{LCL}}$ achieves the highest reported mean:

EnvironmentZero-shotReward-free CT
PointMaze2/33/3
AntMaze1/22/2
HalfCheetah5/64/6
Ant locomotion4/64/6
Adroit Pen2/63/6
G1 humanoid13/1413/14
Continually changing PointMaze1/11/1
Overall28/3830/38

Complementary roles of context learning and successor beliefs. Our ablations reveal that the benefits of explicit context learning depend on the nature of the dynamics shift. In diverse-layout navigation and challenging morphology changes in HalfCheetah, Ant, and Adroit Pen, LCL (context learning) helps summarize interaction history and distinguish unseen dynamics. In contrast, under more locally recoverable shifts, such as friction changes and many humanoid tracking configurations, recursive successor beliefs alone can provide effective history-dependent adaptation without a separate context encoder (LCL). These findings highlight the complementary roles of explicit context learning and recursive belief updates, offering a useful design insight for adapting BFMs to different dynamics shifts.

Broader significance. Recent work such as FADA (Xie et al., 2026) independently highlights the importance of reward-free target-domain adaptation for robust humanoid control. AdaBFM complements this direction by improving zero-shot adaptation through recursive successor beliefs while enabling further adaptation through reward-free continual training.

We believe these results open a new research direction in reward-free continual adaptation for behavioral foundation models, moving beyond static zero-shot policies toward agents that can continuously improve their representations and behavior from deployment experience, without requiring newly engineered reward functions.

References

Lyu, J., Xu, K., Xu, J., Yan, M., Yang, J.-W., Zhang, Z., Bai, C., Lu, Z., & Li, X. (2024). ODRL: A benchmark for off-dynamics reinforcement learning. Advances in Neural Information Processing Systems, 37 (Datasets and Benchmarks Track), 59859–59911. https://openreview.net/forum?id=ap4x1kArGy

Xie, A., Sobanbabu, N., Shikhare, I., Wang, A., Simchowitz, M., & Shi, G. (2026). FADA: Few-shot domain adaptation via dynamics alignment for humanoid control. In Conference on Robot Learning (CoRL 2026). https://arxiv.org/abs/2606.28476

AuthorsOpenReview ↗

Additional updates and clarifications.

  • We have updated the supplementary material to include the code for our humanoid experiments.

  • As detailed in our response to reviewer TFkA, we study and demonstrate reward-free continual training for adapting BFMs to unseen dynamics, independent of the particular BFM variant. We deliberately equip competing baselines with our proposed task-compatible random-feasible exploration strategy (Eq. 7), which achieves higher mean performance than task-only variant in 11/12 comparisons across locomotion and Adroit Pen.

  • Role of explicit LCL conditioning. Our experiments reveal a simple design rule. Explicit context learning (LCL) is particularly beneficial for diverse navigation layouts and challenging morphology changes, whereas recursive successor-belief updates alone can be effective for more recoverable dynamics shifts, including friction and humanoid tracking. Importantly, AdaBFM−LCL\mathrm{AdaBFM}_{-\mathrm{LCL}} remains history-dependent through these belief updates. We will clarify this distinction in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.