acceptodds
Under review as a conference paper at ICLR 2027

Ensemble Sampling with Optimistic Refreshes for Behavior Foundation Models

Abstract

Online task inference with behavior foundation models (BFMs) aims to retrieve a high-performing policy for a target reward function through interaction with the environment, without updating the pretrained model. However, collecting reward-labeled data while achieving high returns at low computational cost remains challenging. In this paper, we propose Ensemble Sampling with Optimistic Refreshes for BFMs (ESOR-BFM), an online task inference method that combines ensemble sampling with optimistic recentering. When the regularized Gram matrix built from state features changes substantially, ESOR-BFM refreshes the ensemble perturbations and recenters the ensemble around an optimistic task embedding estimate. ESOR-BFM samples ensemble members for policy selection between refreshes and performs optimistic search only at refresh times, amortizing the search cost over multiple interactions. We theoretically establish a sublinear regret guarantee for an episodic variant of ESOR-BFM with well-trained BFMs, matching the state-of-the-art regret rate for ensemble sampling in stochastic linear contextual bandits. Extensive experiments on ExORL show that ESOR-BFM consistently outperforms existing online task inference methods and achieves the highest mean returns across settings with stationary rewards, selective reward labeling, and inaccurate successor feature estimates.

Primary area:
reinforcement learning
Keywords:
Behavior Foundation Model, Ensemble sampling, Optimistic exploration

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2rsoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ESOR-BFM, an online task inference method for pretrained BFMs that combines ensemble sampling with optimistic recentering. The method periodically performs UCB-based search to identify a promising task embedding estimate and refreshes the ensemble around this optimistic estimate. Between refreshes, it relies on randomized ensemble sampling for policy selection, amortizing the computational cost of optimistic search. The authors establish a sublinear regret bound for an episodic variant and demonstrate improved online adaptation, reward-label efficiency, and robustness to successor feature estimation errors on ExORL benchmarks.

Critical strengths
  1. The paper addresses the trade-off between exploration effectiveness and computational efficiency in online task inference without updating pretrained BFMs.
  2. Combining ensemble sampling with periodic optimistic recentering offers a straightforward way to incorporate UCB-based guidance into randomized exploration while reducing the frequency of expensive optimistic searches.
  3. The paper establishes a sublinear regret guarantee for an episodic variant of the proposed algorithm, matching the standard regret rate of linear ensemble sampling.
Critical weaknesses
  1. The paper's main contribution is to recenter the ensemble around an optimistic estimate rather than the ridge regression estimate. While this is a reasonable design choice, the paper does not sufficiently establish why this particular combination is necessary or offer a fundamentally different exploration mechanism.
  2. The theoretical algorithm constructs the optimistic center wk,tw_{k,t} by maximizing the predicted return within a confidence ellipsoid, using successor features and uncertainty estimates. However, the practical implementation replaces this theoretically motivated update with a heuristic shift toward the UCB-selected task embedding uk,tu_{k,t} controlled by a scaling factor ρ\rho. These two updates are not generally equivalent, and the heuristic update does not necessarily preserve the optimism properties underlying the theoretical analysis. Moreover, the regret guarantee applies only to an episodic variant that updates the task embedding at the beginning of each episode, whereas the practical implementation selects task embeddings at every interaction step.
  3. Despite computational efficiency being a key motivation, ESOR-BFM achieves only a modest runtime improvement over OpTI-BFM (approximately 8.5%) and remains slower than OpTI-BFM-TS and ES-BFM. This weakens the claimed computational advantages of the proposed method.
Questions for authors
  1. While the ES-BFM ablation demonstrates the benefit of optimistic recentering over standard ensemble sampling, it does not establish why incorporating UCB guidance through ensemble recentering is necessary. In particular, a simpler alternative would be to periodically perform UCB search and directly execute the selected policy between refreshes, which could similarly amortize the computational cost of optimistic optimization without maintaining an ensemble. The paper lacks such a controlled comparison, leaving it unclear whether persistent ensemble sampling provides meaningful benefits beyond periodically reusing optimistic policy selections.
7 comments

We address your questions below and appreciate the chance to strengthen our results.


Benefit of Optimistic Recentering with Ensemble Sampling

To investigate whether ensemble sampling (ES) provides benefits beyond reusing an optimistic policy between refreshes, we introduce UCB-Reuse-BFM and compare it with ESOR-BFM under stationary rewards, different refresh thresholds, and perturbed successor feature estimates.

UCB-Reuse-BFM updates the reward posterior whenever a reward label is acquired and performs UCB search when the same log-determinant refresh criterion used by ESOR-BFM is met (Section 4.3). At each refresh, UCB-Reuse-BFM selects the task embedding that maximizes the UCB objective in Equation (3) among the sampled candidates and keeps this embedding fixed until the next refresh, without ES. ESOR-BFM uses the UCB search result to optimistically recenter its ensemble. Between refreshes, it updates the ensemble whenever new reward labels are acquired and samples an ensemble member at each environment step to select task embeddings. UCB-Reuse-BFM and ESOR-BFM use the same pretrained BFM, initial reward posterior, initial task embedding, and UCB search settings. Actual refresh times can differ because the methods collect different trajectories.

We first compare the methods under stationary rewards using five random seeds. Figure 12 in Appendix C.4 presents the learning curves over ten episodes of 1,000 steps each. All tables report returns in episode 10 and refresh rates computed over all ten episodes, as mean ±\pm standard deviation across seeds.

EnvironmentESOR-BFM refresh rate (%)ESOR-BFM returnUCB-Reuse-BFM refresh rate (%)UCB-Reuse-BFM return
Walker14.72 ±\pm 0.13671.2 ±\pm 4.113.03 ±\pm 0.19577.8 ±\pm 31.7
Cheetah18.62 ±\pm 0.24583.5 ±\pm 17.218.95 ±\pm 0.31583.5 ±\pm 14.7
Quadruped15.50 ±\pm 0.30698.0 ±\pm 31.815.20 ±\pm 0.37695.4 ±\pm 16.9
Jaco Arm8.91 ±\pm 0.03150.9 ±\pm 35.49.13 ±\pm 0.0445.1 ±\pm 14.7
PointMass Maze25.76 ±\pm 0.16109.4 ±\pm 26.825.84 ±\pm 0.1497.7 ±\pm 14.5

Following the hyperparameter settings in Table 3 of our paper, we use αER=0.1\alpha_{\mathrm{ER}}=0.1 for all environments except Jaco Arm, where we use 0.50.5. These thresholds result in frequent refreshes for both methods. ESOR-BFM achieves 16.2% higher mean final return in Walker and approximately 3.35×3.35\times the return in Jaco Arm. The improvement in Jaco Arm occurs with a lower refresh rate. The methods achieve similar final returns in Cheetah and Quadruped, while ESOR-BFM also achieves a higher mean final return in PointMass Maze.

To examine whether the benefits persist when UCB searches become infrequent, we vary αER∈{0.1,1,10,100}\alpha_{\mathrm{ER}}\in\{0.1,1,10,100\} while keeping the other settings fixed. Both methods use the refresh criterion log⁡det⁡Vt−log⁡det⁡Vτ>log⁡(1+αER)\log\det V_{t}-\log\det V_{\tau}>\log(1+\alpha_{\mathrm{ER}}) and are evaluated using the same three seeds. Figure 13 in Appendix C.4 presents the learning curves for each αER\alpha_{\mathrm{ER}}. The table below reports the results at αER=100\alpha_{\mathrm{ER}}=100.

EnvironmentESOR-BFM refresh rate (%)ESOR-BFM returnUCB-Reuse-BFM refresh rate (%)UCB-Reuse-BFM return
Walker0.760 ±\pm 0.003638.6 ±\pm 35.50.754 ±\pm 0.009518.0 ±\pm 26.0
Cheetah0.814 ±\pm 0.001535.0 ±\pm 33.20.838 ±\pm 0.004511.1 ±\pm 22.4
Quadruped0.768 ±\pm 0.006689.3 ±\pm 15.10.781 ±\pm 0.007705.2 ±\pm 15.7
Jaco Arm0.902 ±\pm 0.007156.7 ±\pm 33.00.893 ±\pm 0.00211.3 ±\pm 15.7
PointMass Maze0.918 ±\pm 0.00550.1 ±\pm 16.50.885 ±\pm 0.00917.2 ±\pm 15.9

At this threshold, both methods refresh on fewer than 1% of environment steps. ESOR-BFM achieves higher mean final returns in Walker, Cheetah, Jaco Arm, and PointMass Maze, while UCB-Reuse-BFM achieves a similar performance to ESOR-BFM in Quadruped. These results show that ESOR-BFM can achieve higher mean final returns even when refreshes are infrequent.

Finally, we examine whether the benefits persist under SF estimation errors in Walker, Cheetah, and Quadruped using the perturbation in Equation (8), ψ~(s,z)=ψ(s,z)+αgθ(z)\widetilde{\psi}(s,z)=\psi(s,z)+\alpha g_{\theta}(z), where gθg_{\theta} is a fixed random network whose outputs have unit Euclidean norm. We vary the SF perturbation strength α∈{0,100,300,1000}\alpha\in\{0,100,300,1000\} using five seeds and retain the stationary hyperparameters, including the refresh threshold αER=0.1\alpha_{\mathrm{ER}}=0.1. Figure 14 in Appendix C.4 presents the learning curves for each α\alpha. The table below reports the results at α=1000\alpha=1000.

EnvironmentESOR-BFM refresh rate (%)ESOR-BFM returnUCB-Reuse-BFM refresh rate (%)UCB-Reuse-BFM return
Walker15.12 ± 0.09679.2 ± 16.814.16 ± 0.28632.5 ± 24.8
Cheetah20.02 ± 0.51451.4 ± 52.220.59 ± 0.36401.3 ± 54.7
Quadruped18.65 ± 0.63652.9 ± 27.720.33 ± 0.94637.3 ± 43.5

At this perturbation strength, ESOR-BFM achieves higher mean final returns in all three environments. In Cheetah and Quadruped, these improvements occur with lower refresh rates, providing further evidence of benefits beyond more frequent UCB search.

The ES-BFM ablation supports the benefit of optimistic recentering within ensemble sampling. The comparison with UCB-Reuse-BFM shows that ESOR-BFM can achieve higher mean final returns than reusing the selected policy between refreshes. These improvements occur in several environments with comparable or fewer UCB searches, including settings with infrequent refreshes or perturbed SF estimates. Together, these results support the practical value of combining ensemble sampling with optimistic recentering for online task inference in BFMs.


Runtime Analysis

We provide a component-wise runtime analysis on Quadruped to clarify the runtime differences between ESOR-BFM and the baselines. The approximately 8.5% reduction reported in Table 1 refers to total agent runtime per interaction step, including both action selection and online task inference updates. Since all methods share the same pretrained policy architecture and have similar action selection costs, we focus on the update component to understand their computational differences.

In an additional profiling experiment, we evaluate each method over three seeds, including UCB-Reuse-BFM. We use Ncand=128N_{\mathrm{cand}}=128 and m=128m=128 where applicable, employ JIT-compiled updates, and exclude the first episode of each task to avoid compilation and warm-up overhead. The following table reports the mean and standard deviation across seeds.

Runtime componentOpTI-BFMUCB-Reuse-BFM (ablation)OpTI-BFM-TSES-BFM (ablation)ESOR-BFM (ours)
Action selection (ms/step)0.752 ± 0.0100.773 ± 0.0250.716 ± 0.0020.743 ± 0.0080.773 ± 0.014
Update (ms/step)3.375 ± 0.0492.596 ± 0.0382.192 ± 0.0092.505 ± 0.0272.757 ± 0.017
Total (ms/step)4.127 ± 0.0583.369 ± 0.0632.907 ± 0.0123.248 ± 0.0213.530 ± 0.028

In this additional profiling experiment, ESOR-BFM reduces the mean update time by 18.3% and the total agent runtime by 14.5% relative to OpTI-BFM. The ordering of update times is consistent with the operations performed by each method. OpTI-BFM generates and evaluates 128 candidate task embeddings to obtain the optimistic embedding at every update, resulting in the highest update time. OpTI-BFM-TS instead draws a single posterior sample as the task embedding and has the lowest update time. ES-BFM maintains an ensemble of 128 perturbation vectors and evaluates a refresh criterion based on the log determinant at each step. In our measurements, ES-BFM’s update time is approximately 0.31 ms higher than that of OpTI-BFM-TS. ESOR-BFM additionally performs optimistic recentering when a refresh occurs, with a mean update time approximately 0.25 ms higher than that of ES-BFM. UCB-Reuse-BFM performs UCB search only when the same refresh criterion is met and keeps the selected task embedding fixed between refreshes, without maintaining an ensemble. Compared with UCB-Reuse-BFM, ESOR-BFM incurs an additional 0.161 ms of update time per step.

Although ESOR-BFM is slower than OpTI-BFM-TS and ES-BFM, it achieves a higher mean final return than both methods in Quadruped (Figure 1). UCB-Reuse-BFM is also faster in this profiling experiment and achieves a similar mean final return in stationary Quadruped, while our broader evaluation shows higher mean final returns for ESOR-BFM in several other settings, including Walker and Jaco Arm. These results support the practical value of the additional ensemble computation and a favorable balance between task performance and computational cost.


Theory and Practical Implementation

Our regret guarantee is established for the episodic variant, following the setting analyzed in OpTI-BFM (Section 3.3 in [1]). Fixing the task embedding within each episode connects successor features to the expected discounted state features along the executed trajectory, enabling linear contextual bandit analysis. Our practical implementation selects task embeddings at every interaction step to incorporate newly acquired information, and its performance is evaluated empirically.

We first show that, under the conditions below, the policies conditioned on the UCB-selected task embedding uk,tu_{k,t} and the theoretical optimistic estimate wk,tw_{k,t} have the same successor features at the refresh state. This correspondence provides a rationale for incorporating uk,tu_{k,t} into the ensemble center in the practical implementation.

Following OpTI-BFM’s theoretical analysis (Assumption (A1) in [1]), we assume that successor features equal the expected discounted sum of state features under the corresponding policy. At a refresh state sk,ts_{k,t}, let CkH+t\mathcal C_{kH+t} be the confidence ellipsoid defined in our algorithm, and suppose CkH+t⊆Z\mathcal C_{kH+t}\subseteq\mathcal Z which also assumed in OpTI-BFM [1]. For this comparison, we additionally assume that the BFM pretraining property discussed in Section 4 of OpTI-BFM [1] holds exactly, so that ⟨ψ(sk,t,w),w⟩=max⁡z∈Z⟨ψ(sk,t,z),w⟩\langle\psi(s_{k,t},w),w\rangle=\max_{z\in\mathcal Z}\langle\psi(s_{k,t},z),w\rangle for every w∈CkH+tw\in\mathcal C_{kH+t}. Let uk,tu_{k,t} be an exact UCB maximizer in Equation (3), and let wk,t∈arg⁡max⁡w∈CkH+t⟨ψ(sk,t,uk,t),w⟩w_{k,t}\in\arg\max_{w\in\mathcal C_{kH+t}}\langle\psi(s_{k,t},u_{k,t}),w\rangle be its corresponding optimistic estimate.

Applying the assumed optimality property at wk,tw_{k,t} gives ⟨ψ(sk,t,uk,t),wk,t⟩≤⟨ψ(sk,t,wk,t),wk,t⟩\langle\psi(s_{k,t},u_{k,t}),w_{k,t}\rangle\leq\langle\psi(s_{k,t},w_{k,t}),w_{k,t}\rangle. Since wk,t∈CkH+tw_{k,t}\in\mathcal C_{kH+t}, the latter is at most max⁡w∈CkH+t⟨ψ(sk,t,wk,t),w⟩\max_{w\in\mathcal C_{kH+t}}\langle\psi(s_{k,t},w_{k,t}),w\rangle. Finally, the optimality of uk,tu_{k,t} and the definition of wk,tw_{k,t} imply max⁡w∈CkH+t⟨ψ(sk,t,wk,t),w⟩≤⟨ψ(sk,t,uk,t),wk,t⟩\max_{w\in\mathcal C_{kH+t}}\langle\psi(s_{k,t},w_{k,t}),w\rangle\leq\langle\psi(s_{k,t},u_{k,t}),w_{k,t}\rangle. Combining these inequalities yields ⟨ψ(sk,t,uk,t),wk,t⟩≤⟨ψ(sk,t,wk,t),wk,t⟩≤max⁡w∈CkH+t⟨ψ(sk,t,wk,t),w⟩≤⟨ψ(sk,t,uk,t),wk,t⟩\langle\psi(s_{k,t},u_{k,t}),w_{k,t}\rangle\leq\langle\psi(s_{k,t},w_{k,t}),w_{k,t}\rangle\leq\max_{w\in\mathcal C_{kH+t}}\langle\psi(s_{k,t},w_{k,t}),w\rangle\leq\langle\psi(s_{k,t},u_{k,t}),w_{k,t}\rangle. Since the first and last quantities are identical, every inequality must hold with equality. Thus, wk,tw_{k,t} is also a maximizer of the UCB objective and maximizes both linear objectives associated with ψ(sk,t,uk,t)\psi(s_{k,t},u_{k,t}) and ψ(sk,t,wk,t)\psi(s_{k,t},w_{k,t}) over CkH+t\mathcal C_{kH+t}.

To establish equality of the SF vectors, we further assume ⟨ψ(sk,t,uk,t),wk,t⟩≠0\langle\psi(s_{k,t},u_{k,t}),w_{k,t}\rangle\neq0. The equalities above then imply that both SF vectors are nonzero. The closed-form solution of the inner maximization then gives wk,t=z^kH+t+βkH+tVkH+t−1ψ(sk,t,uk,t)/∥∣ψ(sk,t,uk,t)∥∣VkH+t−1w_{k,t}=\hat z_{kH+t}+\beta_{kH+t}V_{kH+t}^{-1}\psi(s_{k,t},u_{k,t})/\||\psi(s_{k,t},u_{k,t})\||_{V_{kH+t}^{-1}} and wk,t=z^kH+t+βkH+tVkH+t−1ψ(sk,t,wk,t)/∥∣ψ(sk,t,wk,t)∥∣VkH+t−1w_{k,t}=\hat z_{kH+t}+\beta_{kH+t}V_{kH+t}^{-1}\psi(s_{k,t},w_{k,t})/\||\psi(s_{k,t},w_{k,t})\||_{V_{kH+t}^{-1}}. Equating these expressions shows that the SF vectors normalized by their VkH+t−1V_{kH+t}^{-1}-norms are identical. Taking the inner product of each normalized vector with wk,tw_{k,t} and using the equal, nonzero inner products established above yields ∥∣ψ(sk,t,uk,t)∥∣VkH+t−1=∥∣ψ(sk,t,wk,t)∥∣VkH+t−1\||\psi(s_{k,t},u_{k,t})\||_{V_{kH+t}^{-1}}=\||\psi(s_{k,t},w_{k,t})\||_{V_{kH+t}^{-1}}. Consequently, ψ(sk,t,uk,t)=ψ(sk,t,wk,t)\psi(s_{k,t},u_{k,t})=\psi(s_{k,t},w_{k,t}). Since the successor features are exact, this equality implies that the two policies have the same value from the refresh state under any common linear reward.

Motivated by this correspondence, our practical implementation follows OpTI-BFM’s search procedure and selects uk,tu_{k,t} as the candidate with the highest UCB score among a finite set of sampled task embeddings. At each refresh, we incorporate the selected embedding into the ensemble center through wk,t=z^kH+t+ρ(uk,t−z^kH+t)w_{k,t}=\hat z_{kH+t}+\rho(u_{k,t}-\hat z_{kH+t}). This incorporates the UCB search result into the ensemble and allows its influence to persist between refreshes, while the perturbed estimates continue to be updated and sampled. With 0<ρ≤10<\rho\leq1, as used in our experiments, the center interpolates between the ridge regression estimate and the selected embedding. The practical update also admits an explicit bound on its displacement from the ridge regression estimate. The candidate sampler uses a Mahalanobis radius of 2β2\beta, where β\beta is the practical UCB coefficient. Hence, ∥∣uk,t−z^kH+t∥∣VkH+t≤2β\||u_{k,t}-\hat z_{kH+t}\||_{V_{kH+t}}\leq2\beta implies ∥∣wk,t−z^kH+t∥∣VkH+t≤2ρβ\||w_{k,t}-\hat z_{kH+t}\||_{V_{kH+t}}\leq2\rho\beta. Thus, ρ\rho controls the displacement in the same uncertainty geometry used to bound the common recentering component in Lemma 3.

The SF correspondence under the stated conditions and the bound on the recentering displacement provide a rationale for the practical update. The exact identity in Equation (6) holds for the theoretical optimistic estimate, and the optimism and regret guarantees are established for the episodic variant under the stated assumptions. Empirically, ESOR-BFM achieves higher mean final returns than ES-BFM, which centers the ensemble at the ridge regression estimate, in all five environments under stationary rewards (Figure 1). This supports the contribution of practical recentering alongside the theoretical analysis.


Reference

[1] Rupf, Thomas, et al. "Optimistic task inference for behavior foundation models." International Conference on Learning Representations, 2026.

Reviewer_kSFcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes ESOR-BFM, a method for online task inference, based on ensemble sampling: The method maintains and samples from an ensemble of reward estimates at each step and introduces a trick for achieving lower computational cost: When the Gram matrix has grown enough, the ensemble is re-centered on an UCB-style estimate. The authors provide theoretical regret guarantees for their method, and achieve the highest mean return and sample efficiency across five environment among four baselines.

Critical strengths

The manuscript is clearly written and concise, the proposed method is supported by sublinear regret guarantee, and the experiments are thorough, covering noisy successor features, runtime comparison, and ablation. I particularly appreciate section 4.3 regarding the practical implementation.

Critical weaknesses

The reported runtime savings (Table 1) over OpTI-BFM are modest, despite lower computational cost being among the state central motivation of the work, and the runtime analysis is of moderate scope at best. More generally, while the results are convincing empirically, I'd appreciate a more thorough analysis of the observed trends.

The manuscript lacks a clear discussion of the strengths and weaknesses of the proposed method, as well as a limitations section.

Questions for authors

Tables 3 and 4 show different hyperparameters for ESOR-BFM and ES-BFM. These imply some degree of hyperparameter tuning, while the other baselines reportedly use default settings. How extensive was that hyperparameter tuning and have you given a equal amount of effort to tuning the baselines?

5 comments

We thank the reviewer for the careful reading and positive feedback. We address the questions and concerns below.


Runtime Analysis

We provide a component-wise runtime analysis on Quadruped to clarify the runtime differences between ESOR-BFM and the baselines. Table 1 reports the total runtime per interaction step, including both action selection and online task inference updates. Since all methods share the same pretrained policy architecture and have similar action selection costs, we focus on the update component to understand their computational differences.

In an additional profiling experiment, we evaluate each method over three seeds. With Ncand=128N_{\mathrm{cand}}=128 and m=128m=128, we use JIT-compiled updates and exclude the first episode of each task to avoid compilation and warm-up overhead. The following table reports the mean and standard deviation across seeds.

Runtime componentOpTI-BFMOpTI-BFM-TSES-BFM (ablation)ESOR-BFM (ours)
Action selection (ms/step)0.752 ± 0.0100.716 ± 0.0020.743 ± 0.0080.773 ± 0.014
Update (ms/step)3.375 ± 0.0492.192 ± 0.0092.505 ± 0.0272.757 ± 0.017
Total (ms/step)4.127 ± 0.0582.907 ± 0.0123.248 ± 0.0213.530 ± 0.028

ESOR-BFM reduces the mean update time by 18.3% and the total agent runtime by 14.5% relative to OpTI-BFM. The ordering of update times is consistent with the operations performed by each method. OpTI-BFM generates and evaluates 128 candidate task embeddings to obtain the optimistic embedding at every update, resulting in the highest update time. OpTI-BFM-TS instead draws a single posterior sample as the task embedding and has the lowest update time. ES-BFM maintains an ensemble of 128 perturbation vectors and evaluates a refresh criterion based on the log determinant at each step. In our measurements, ES-BFM’s update time is approximately 0.31 ms higher than that of OpTI-BFM-TS. ESOR-BFM additionally performs optimistic recentering when a refresh occurs. Its mean update time is approximately 0.25 ms higher than that of ES-BFM. To examine the contribution of refresh operations, we separate refresh and non-refresh steps below.

MeasurementESOR-BFM
Refresh steps (%)9.02%
Update time on refresh steps (ms/step)3.772
Non-refresh steps (%)90.98%
Update time on non-refresh steps (ms/step)2.656

For ESOR-BFM, a refresh update takes approximately 42% longer than a non-refresh update. Its update time on refresh steps, 3.772 ms, also exceeds OpTI-BFM’s average update time of 3.375 ms. On these steps, ESOR-BFM performs ensemble maintenance together with optimistic candidate evaluation and recentering. However, refresh occurs in only 9.02% of measured steps, yielding an average update time of approximately 2.757 ms. These results highlight the computational advantage of performing optimistic recentering only at refresh steps, reducing the average update time relative to OpTI-BFM. Although ESOR-BFM is slower than OpTI-BFM-TS, it achieves a higher mean final return in Quadruped (Figure 1). Together with its 14.5% lower total agent runtime than OpTI-BFM, these results demonstrate a favorable balance between task performance and computational cost.


Hyperparameters Tuning

We tuned the ESOR-BFM hyperparameters αER\alpha_{\mathrm{ER}}, ν\nu, and ρ\rho, using 0.1,0.5,1.0\\{0.1, 0.5, 1.0\\} as the initial candidate set for each parameter. The selected configurations for stationary rewards are reported in Table 3. For ES-BFM, we reused the corresponding ESOR-BFM values for the shared hyperparameters. For LoLA, OpTI-BFM, and OpTI-BFM-TS, we used the evaluation hyperparameters recommended in the official OpTI-BFM implementation [1], with no additional hyperparameter search for any of these three baselines.

Under selective reward labeling, where reward feedback is not available at every step, we observed that larger values of ρ\rho could reduce ESOR-BFM’s performance. We therefore additionally evaluated ρ∈0.01,0.05\rho\in\\{0.01,0.05\\} for this setting and adjusted αER\alpha_{\mathrm{ER}} and ν\nu within their original candidate sets. The selected configurations are reported in Table 4. For the experiments with perturbed successor feature estimates and nonlinear rewards, we reused the stationary hyperparameter configurations in Table 3 without further tuning and achieved strong performance in both settings.


Discussion of the Strengths, Weaknesses, and Limitations

We provide a more detailed discussion of the strengths, weaknesses, and limitations of our work.

We propose ESOR-BFM, which combines ensemble sampling (ES) with optimistic recentering for efficient online task inference in BFMs. When the regularized Gram matrix changes substantially, ESOR-BFM refreshes the ensemble perturbations and recenters the ensemble around an optimistic estimate of the task embedding. To the best of our knowledge, this is the first theoretical and empirical study of ES with optimistic recentering for online task inference in BFMs. We establish a new regret guarantee for ES with optimistic recentering, proving a sublinear regret bound of O~(d3/2n)\widetilde{O}(d^{3/2}\sqrt{n}) for an episodic variant, matching the standard dependence of linear ES regret on the feature dimension dd and the number of episodes nn. In contrast, OpTI-BFM’s theoretical guarantee is obtained by adapting existing UCB-based regret results to the BFM setting.

Extensive numerical experiments on the ExORL benchmark demonstrate that ESOR-BFM outperforms existing online task inference methods across a range of experimental settings, including stationary settings, downstream tasks, limited reward feedback (costly reward labeling), and errors in successor feature estimates (imperfect models). Our empirical analysis shows that ESOR-BFM achieves higher mean returns than OpTI-BFM while reducing total runtime per interaction step by 14.5%. These results show that ESOR-BFM achieves a better trade-off between exploration effectiveness and computational efficiency in online task inference.

In practice, we have some hyperparameters, including the refresh threshold αER\alpha_{\mathrm{ER}}, perturbation inflation factor ν\nu, and recentering strength ρ\rho, to adapt the exploration procedure to different environments and feedback settings. However, the reported results were obtained with limited hyperparameter tuning. In particular, our additional sensitivity analysis in Figure 11 of the revised paper shows that performance in Walker and Quadruped remains robust across the tested range of αER\alpha_{\mathrm{ER}}.

Some components of Algorithm 1 are approximated in the practical implementation. Nevertheless, our work demonstrates that ES with optimistic recentering can be effectively adapted to online task inference in BFMs through a simple and computationally efficient procedure. Our regret guarantee is established for an episodic variant under idealized assumptions, including linear rewards and exact successor features, as in OpTI-BFM’s theoretical analysis. In future work, we aim to extend this analysis to task embedding selection at every interaction step and to settings where rewards are not linear in the learned features.


Reference

[1] Rupf, Thomas, et al. "Optimistic task inference for behavior foundation models." International Conference on Learning Representations, 2026.

Dear Reviewer kSFc,

Thank you for your thoughtful review. As the rebuttal period is ongoing, we would like to follow up on our response.

  • We have provided a detailed runtime breakdown, separating action selection from task inference updates and distinguishing refresh from non-refresh steps. The results show that ESOR-BFM reduces mean update time by 18.3% and total agent runtime per interaction step by 14.5% relative to OpTI-BFM, highlighting the computational benefit of performing optimistic recentering only at refresh steps.
  • We have clarified our hyperparameter tuning procedure and the configurations used for the baselines. In particular, ESOR-BFM achieves strong performance under perturbed successor feature estimates and nonlinear rewards using the stationary hyperparameter configurations without further tuning.
  • We have also expanded our discussion of the method’s novelty, strengths, and limitations. We clarify the contribution of combining ensemble sampling with optimistic recentering and the new sublinear regret guarantee for an episodic variant, while explicitly discussing the assumptions underlying the guarantee.

Please let us know if the rebuttal has answered your questions. If the Reviewer does not have any additional concerns or questions, we would sincerely ask that the score be adjusted accordingly.

Yours sincerely,

The Authors

Reviewer_vstBOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ESOR-BFM which is a novel method for online reward inference with pretrained BFMs built on Ensemble Sampling. With added Optimistic Recentering, the method outperforms prior work (OpTI-BFM) in terms of computational cost and performance while retaining a comparable sublinear regret bound under similar assumptions. The authors validate their approach in a common zero-shot RL benchmark: Deepmind Control Suite (DMC) tasks with sufficient data for good BFM pretraining. ESOR-BFM consistently beats the relevant baselines in terms of maximizing return, collecting informative data, sample efficiency, handling misspecification, and computational cost. Most gains seem to be small but outside of standard deviations.

Critical strengths

This paper presents a clean improvement over prior methods in the field. While it uses a well-established Bandit algorithm in ES, it does add the optimistic recentering which they empirically show to make a big difference while still enjoying the EV-style regret bound. Given that this online BFM task inference setting suffers from very correlated features, refreshes based on information gain are a well-motivated idea to reduce computational cost in this setting.

Critical weaknesses

The authors should provide more information on how their methods behaves in practice. Specifically, how often refreshes are triggered and how much computational cost varies between steps with and without refreshes. See questions 3. and 4. below.

Questions for authors
  1. How do all online inference methods compare to zero-shot “oracle” reward inference with the same BFM? I.e. is their return converging to the return of the policy conditioned on the zero-shot inference (least squares) latent?
  2. What happens in Jaco and PointMass Maze after the 10 episodes? Seeing at which return the methods converge (if they do) in the next 20-40 episodes would be interesting.
  3. Real-time applications like robotics do not only value average computational cost but also consistency. How much does ESOR-BFMs computational cost vary between steps? How much slower is the refresh step compared to a non-refresh step?
  4. How often does ESOR-BFM refresh in practice? A plot simultaneously showing the (cumulative) number of refreshes and the (cumulative) return over episodes would give some good intuition of how the algorithm behaves in practice.
5 comments

Thank you for your positive evaluation of our paper and your insightful feedback. We address your questions below.


Comparison with an "Oracle" Baseline

In the revised paper, we add a zero-shot "Oracle" baseline to Figure 1, evaluated using the same pretrained BFM. Similar to OpTI-BFM [1], we estimate z^r\hat{z}_{r} by solving the least-squares problem in Equation (1) using 50k states sampled from the pretraining dataset and labeled with the target reward function. The baseline then executes the pretrained policy πz^r\pi_{\hat{z}_{r}} without updating the BFM parameters. Note that the oracle baseline serves as an offline least-squares reference rather than a guaranteed upper bound on return.

ESOR-BFM achieves the highest final mean return among the compared online methods across all five environments. Within only 10 episodes of online interaction (10k steps), ESOR-BFM reaches a return comparable to the oracle baseline in Walker and exceeds the oracle’s mean return in Cheetah and Quadruped, despite the oracle’s privileged access to 50k reward-labeled samples. In Jaco Arm and PointMass Maze, ESOR-BFM comes closest to the oracle baseline among the online methods and continues to improve through episode 10.


Performance Beyond 10 Episodes on Jaco Arm and PointMass Maze

In the revised paper, we extend the evaluation horizon from 10 to 50 episodes in Jaco Arm and PointMass Maze, keeping each method’s hyperparameters unchanged from the original 10-episode experiments. The results are shown in Figure 10 in Appendix C.3. As shown in Figure 10, ESOR-BFM maintains the highest mean return among the compared online methods beyond episode 10 in both environments. In Jaco Arm, ESOR-BFM continues to improve, achieving mean returns well above the oracle baseline. In PointMass Maze, ESOR-BFM reaches near-oracle performance around episodes 15–20 and maintains a substantial advantage over the online baselines through episode 50, despite fluctuations in return. We thank the reviewer for suggesting this extended evaluation, which further demonstrates the sustained performance advantage of ESOR-BFM.


Refresh vs. Non-Refresh Runtime

We compare the mean runtime per interaction step of ESOR-BFM and the online task inference baselines on Quadruped. We exclude the first episode of each task from timing to avoid JIT compilation overhead. To examine computational consistency in more detail, we separately analyze the agent-update time, excluding action selection, and distinguish between refresh and non-refresh steps.

For ESOR-BFM, we use Ncand=128N_{cand} = 128 and m=128m = 128 across five seeds and four Quadruped tasks, yielding 180,000180,000 measured steps. The full update function is JIT-compiled. The table below reports statistics pooled over individual steps. SD denotes the standard deviation of per-step update latency.

Step typeFraction of stepsMean ± SD (ms)
Non-refresh90.98%2.656 ±\pm 0.134
Refresh9.02%3.772 ±\pm 0.192
All steps100%2.757 ±\pm 0.349

Refresh steps are 1.116 ms slower on average, corresponding to 1.42$\times$ the update latency of non-refresh steps. Since refresh occurs in only 9.02% of measured steps, the overall mean update latency is 2.757 ms.


Cumulative Return and Number of Refreshes

We examine refresh frequency using the our experimental settings for all five environments, varying only the refresh threshold αER∈0.1,1,10,100\alpha_{\mathrm{ER}} \in \\{0.1, 1, 10, 100\\}. A larger αER\alpha_{\mathrm{ER}} raises the threshold for triggering a refresh and reduces refresh frequency. In the revised paper, Figure 11 presents cumulative refresh counts, cumulative returns, and episodic returns over 10 episodes, with means and standard deviations across five seeds.

Refreshes occur more frequently early in adaptation. For example, with αER=0.1\alpha_{\mathrm{ER}}=0.1, ESOR-BFM performs approximately 1,475-2,718 refreshes per 10,000 step run, depending on the environment. The mean refresh count ranges from 683-972 in episode 1 to 33-55 in episode 10, with 1,000 steps per episode. Walker and Quadruped maintain similar cumulative returns despite large differences in refresh counts. Increasing αER\alpha_{\mathrm{ER}} from 0.1 to 100 reduces refresh counts by approximately 95%, while cumulative returns decrease by 4.1% and 2.9%, respectively. The same threshold increase reduces mean cumulative return by 14.2% in Cheetah and 51.1% in PointMass Maze. Jaco Arm achieves its highest observed mean cumulative return at αER=100\alpha_{\mathrm{ER}}=100, 19.9% higher than at αER=0.1\alpha_{\mathrm{ER}}=0.1. More frequent refreshing therefore does not uniformly lead to higher returns. These results show that substantially fewer optimistic refreshes can preserve most of the cumulative return in Walker and Quadruped.


Reference

[1] Rupf, Thomas, et al. "Optimistic task inference for behavior foundation models." International Conference on Learning Representations, 2026.

Dear Reviewer vstB,

Thank you for your thoughtful review. As the rebuttal period is ongoing, we would like to follow up on our response.

  • As requested, we have added an “Oracle” baseline and extended the evaluation of all methods from 10 to 50 episodes on Jaco Arm and PointMass Maze. These additional results demonstrate that ESOR-BFM outperforms the compared online baselines in terms of final mean return across all five environments and maintains its advantage in the extended 50-episode evaluations.
  • We have also provided detailed analyses of ESOR-BFM’s computational efficiency, including the mean update time at refresh and non-refresh steps and the relationship between cumulative return and the number of refreshes. These analyses show that refreshes have a limited impact on average update time and that substantially fewer refreshes can preserve most of the cumulative return.

Please let us know if the rebuttal has answered your questions. If the Reviewer does not have any additional concerns or questions, we would sincerely ask that the score be adjusted accordingly.

Yours sincerely,

The Authors

Dear Reviewer vstB,

We would like to once more sincerely ask for the feedback regarding the rebuttal. As discussed above, we do believe that we have thoroughly addressed all Reviewer's concerns (thank you very much for the insightful feedback). However if the Reviewer has any remaining questions, we would be very happy to provide additional explanation. We just want to make sure that we have enough time to provide any extra clarifications if needed, especially given the fact that the authors-reviewers discussion period will end soon. If the Reviewer does not have any additional concerns, we would like to sincerely ask the Reviewer to update the score accordingly.

Thank you very much !

Yours sincerely,

The Authors

Comments on the paper

1 comment
Area_Chair_sn3dOpenReview ↗

Reviewer-Author Discussion Phase is Ending

Dear Reviewers,

Thank you for your service in the review process. Now that we are approaching the deadline of the author-reviewer discussion period. Please take time to read authors' responses and let them know if it addresses your concerns. Thank you.

Best,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.