- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ESOR-BFM, an online task inference method for pretrained BFMs that combines ensemble sampling with optimistic recentering. The method periodically performs UCB-based search to identify a promising task embedding estimate and refreshes the ensemble around this optimistic estimate. Between refreshes, it relies on randomized ensemble sampling for policy selection, amortizing the computational cost of optimistic search. The authors establish a sublinear regret bound for an episodic variant and demonstrate improved online adaptation, reward-label efficiency, and robustness to successor feature estimation errors on ExORL benchmarks.
Critical strengths
- The paper addresses the trade-off between exploration effectiveness and computational efficiency in online task inference without updating pretrained BFMs.
- Combining ensemble sampling with periodic optimistic recentering offers a straightforward way to incorporate UCB-based guidance into randomized exploration while reducing the frequency of expensive optimistic searches.
- The paper establishes a sublinear regret guarantee for an episodic variant of the proposed algorithm, matching the standard regret rate of linear ensemble sampling.
Critical weaknesses
- The paper's main contribution is to recenter the ensemble around an optimistic estimate rather than the ridge regression estimate. While this is a reasonable design choice, the paper does not sufficiently establish why this particular combination is necessary or offer a fundamentally different exploration mechanism.
- The theoretical algorithm constructs the optimistic center by maximizing the predicted return within a confidence ellipsoid, using successor features and uncertainty estimates. However, the practical implementation replaces this theoretically motivated update with a heuristic shift toward the UCB-selected task embedding controlled by a scaling factor . These two updates are not generally equivalent, and the heuristic update does not necessarily preserve the optimism properties underlying the theoretical analysis. Moreover, the regret guarantee applies only to an episodic variant that updates the task embedding at the beginning of each episode, whereas the practical implementation selects task embeddings at every interaction step.
- Despite computational efficiency being a key motivation, ESOR-BFM achieves only a modest runtime improvement over OpTI-BFM (approximately 8.5%) and remains slower than OpTI-BFM-TS and ES-BFM. This weakens the claimed computational advantages of the proposed method.
Questions for authors
- While the ES-BFM ablation demonstrates the benefit of optimistic recentering over standard ensemble sampling, it does not establish why incorporating UCB guidance through ensemble recentering is necessary. In particular, a simpler alternative would be to periodically perform UCB search and directly execute the selected policy between refreshes, which could similarly amortize the computational cost of optimistic optimization without maintaining an ensemble. The paper lacks such a controlled comparison, leaving it unclear whether persistent ensemble sampling provides meaningful benefits beyond periodically reusing optimistic policy selections.
7 comments
| Environment | ESOR-BFM refresh rate (%) | ESOR-BFM return | UCB-Reuse-BFM refresh rate (%) | UCB-Reuse-BFM return |
|---|---|---|---|---|
| Walker | 15.12 ± 0.09 | 679.2 ± 16.8 | 14.16 ± 0.28 | 632.5 ± 24.8 |
| Cheetah | 20.02 ± 0.51 | 451.4 ± 52.2 | 20.59 ± 0.36 | 401.3 ± 54.7 |
| Quadruped | 18.65 ± 0.63 | 652.9 ± 27.7 | 20.33 ± 0.94 | 637.3 ± 43.5 |
At this perturbation strength, ESOR-BFM achieves higher mean final returns in all three environments. In Cheetah and Quadruped, these improvements occur with lower refresh rates, providing further evidence of benefits beyond more frequent UCB search.
The ES-BFM ablation supports the benefit of optimistic recentering within ensemble sampling. The comparison with UCB-Reuse-BFM shows that ESOR-BFM can achieve higher mean final returns than reusing the selected policy between refreshes. These improvements occur in several environments with comparable or fewer UCB searches, including settings with infrequent refreshes or perturbed SF estimates. Together, these results support the practical value of combining ensemble sampling with optimistic recentering for online task inference in BFMs.
Runtime Analysis
We provide a component-wise runtime analysis on Quadruped to clarify the runtime differences between ESOR-BFM and the baselines. The approximately 8.5% reduction reported in Table 1 refers to total agent runtime per interaction step, including both action selection and online task inference updates. Since all methods share the same pretrained policy architecture and have similar action selection costs, we focus on the update component to understand their computational differences.
In an additional profiling experiment, we evaluate each method over three seeds, including UCB-Reuse-BFM. We use and where applicable, employ JIT-compiled updates, and exclude the first episode of each task to avoid compilation and warm-up overhead. The following table reports the mean and standard deviation across seeds.
| Runtime component | OpTI-BFM | UCB-Reuse-BFM (ablation) | OpTI-BFM-TS | ES-BFM (ablation) | ESOR-BFM (ours) |
|---|---|---|---|---|---|
| Action selection (ms/step) | 0.752 ± 0.010 | 0.773 ± 0.025 | 0.716 ± 0.002 | 0.743 ± 0.008 | 0.773 ± 0.014 |
| Update (ms/step) | 3.375 ± 0.049 | 2.596 ± 0.038 | 2.192 ± 0.009 | 2.505 ± 0.027 | 2.757 ± 0.017 |
| Total (ms/step) | 4.127 ± 0.058 | 3.369 ± 0.063 | 2.907 ± 0.012 | 3.248 ± 0.021 | 3.530 ± 0.028 |
In this additional profiling experiment, ESOR-BFM reduces the mean update time by 18.3% and the total agent runtime by 14.5% relative to OpTI-BFM. The ordering of update times is consistent with the operations performed by each method. OpTI-BFM generates and evaluates 128 candidate task embeddings to obtain the optimistic embedding at every update, resulting in the highest update time. OpTI-BFM-TS instead draws a single posterior sample as the task embedding and has the lowest update time. ES-BFM maintains an ensemble of 128 perturbation vectors and evaluates a refresh criterion based on the log determinant at each step. In our measurements, ES-BFM’s update time is approximately 0.31 ms higher than that of OpTI-BFM-TS. ESOR-BFM additionally performs optimistic recentering when a refresh occurs, with a mean update time approximately 0.25 ms higher than that of ES-BFM. UCB-Reuse-BFM performs UCB search only when the same refresh criterion is met and keeps the selected task embedding fixed between refreshes, without maintaining an ensemble. Compared with UCB-Reuse-BFM, ESOR-BFM incurs an additional 0.161 ms of update time per step.
Although ESOR-BFM is slower than OpTI-BFM-TS and ES-BFM, it achieves a higher mean final return than both methods in Quadruped (Figure 1). UCB-Reuse-BFM is also faster in this profiling experiment and achieves a similar mean final return in stationary Quadruped, while our broader evaluation shows higher mean final returns for ESOR-BFM in several other settings, including Walker and Jaco Arm. These results support the practical value of the additional ensemble computation and a favorable balance between task performance and computational cost.
Theory and Practical Implementation
Our regret guarantee is established for the episodic variant, following the setting analyzed in OpTI-BFM (Section 3.3 in [1]). Fixing the task embedding within each episode connects successor features to the expected discounted state features along the executed trajectory, enabling linear contextual bandit analysis. Our practical implementation selects task embeddings at every interaction step to incorporate newly acquired information, and its performance is evaluated empirically.
We first show that, under the conditions below, the policies conditioned on the UCB-selected task embedding and the theoretical optimistic estimate have the same successor features at the refresh state. This correspondence provides a rationale for incorporating into the ensemble center in the practical implementation.
Following OpTI-BFM’s theoretical analysis (Assumption (A1) in [1]), we assume that successor features equal the expected discounted sum of state features under the corresponding policy. At a refresh state , let be the confidence ellipsoid defined in our algorithm, and suppose which also assumed in OpTI-BFM [1]. For this comparison, we additionally assume that the BFM pretraining property discussed in Section 4 of OpTI-BFM [1] holds exactly, so that for every . Let be an exact UCB maximizer in Equation (3), and let be its corresponding optimistic estimate.
Applying the assumed optimality property at gives . Since , the latter is at most . Finally, the optimality of and the definition of imply . Combining these inequalities yields . Since the first and last quantities are identical, every inequality must hold with equality. Thus, is also a maximizer of the UCB objective and maximizes both linear objectives associated with and over .
To establish equality of the SF vectors, we further assume . The equalities above then imply that both SF vectors are nonzero. The closed-form solution of the inner maximization then gives and . Equating these expressions shows that the SF vectors normalized by their -norms are identical. Taking the inner product of each normalized vector with and using the equal, nonzero inner products established above yields . Consequently, . Since the successor features are exact, this equality implies that the two policies have the same value from the refresh state under any common linear reward.
Motivated by this correspondence, our practical implementation follows OpTI-BFM’s search procedure and selects as the candidate with the highest UCB score among a finite set of sampled task embeddings. At each refresh, we incorporate the selected embedding into the ensemble center through . This incorporates the UCB search result into the ensemble and allows its influence to persist between refreshes, while the perturbed estimates continue to be updated and sampled. With , as used in our experiments, the center interpolates between the ridge regression estimate and the selected embedding. The practical update also admits an explicit bound on its displacement from the ridge regression estimate. The candidate sampler uses a Mahalanobis radius of , where is the practical UCB coefficient. Hence, implies . Thus, controls the displacement in the same uncertainty geometry used to bound the common recentering component in Lemma 3.
The SF correspondence under the stated conditions and the bound on the recentering displacement provide a rationale for the practical update. The exact identity in Equation (6) holds for the theoretical optimistic estimate, and the optimism and regret guarantees are established for the episodic variant under the stated assumptions. Empirically, ESOR-BFM achieves higher mean final returns than ES-BFM, which centers the ensemble at the ridge regression estimate, in all five environments under stationary rewards (Figure 1). This supports the contribution of practical recentering alongside the theoretical analysis.
Reference
[1] Rupf, Thomas, et al. "Optimistic task inference for behavior foundation models." International Conference on Learning Representations, 2026.
We address your questions below and appreciate the chance to strengthen our results.
Benefit of Optimistic Recentering with Ensemble Sampling
To investigate whether ensemble sampling (ES) provides benefits beyond reusing an optimistic policy between refreshes, we introduce UCB-Reuse-BFM and compare it with ESOR-BFM under stationary rewards, different refresh thresholds, and perturbed successor feature estimates.
UCB-Reuse-BFM updates the reward posterior whenever a reward label is acquired and performs UCB search when the same log-determinant refresh criterion used by ESOR-BFM is met (Section 4.3). At each refresh, UCB-Reuse-BFM selects the task embedding that maximizes the UCB objective in Equation (3) among the sampled candidates and keeps this embedding fixed until the next refresh, without ES. ESOR-BFM uses the UCB search result to optimistically recenter its ensemble. Between refreshes, it updates the ensemble whenever new reward labels are acquired and samples an ensemble member at each environment step to select task embeddings. UCB-Reuse-BFM and ESOR-BFM use the same pretrained BFM, initial reward posterior, initial task embedding, and UCB search settings. Actual refresh times can differ because the methods collect different trajectories.
We first compare the methods under stationary rewards using five random seeds. Figure 12 in Appendix C.4 presents the learning curves over ten episodes of 1,000 steps each. All tables report returns in episode 10 and refresh rates computed over all ten episodes, as mean ± standard deviation across seeds.
Following the hyperparameter settings in Table 3 of our paper, we use αER=0.1 for all environments except Jaco Arm, where we use 0.5. These thresholds result in frequent refreshes for both methods. ESOR-BFM achieves 16.2% higher mean final return in Walker and approximately 3.35× the return in Jaco Arm. The improvement in Jaco Arm occurs with a lower refresh rate. The methods achieve similar final returns in Cheetah and Quadruped, while ESOR-BFM also achieves a higher mean final return in PointMass Maze.
To examine whether the benefits persist when UCB searches become infrequent, we vary αER∈{0.1,1,10,100} while keeping the other settings fixed. Both methods use the refresh criterion logdetVt−logdetVτ>log(1+αER) and are evaluated using the same three seeds. Figure 13 in Appendix C.4 presents the learning curves for each αER. The table below reports the results at αER=100.
At this threshold, both methods refresh on fewer than 1% of environment steps. ESOR-BFM achieves higher mean final returns in Walker, Cheetah, Jaco Arm, and PointMass Maze, while UCB-Reuse-BFM achieves a similar performance to ESOR-BFM in Quadruped. These results show that ESOR-BFM can achieve higher mean final returns even when refreshes are infrequent.
Finally, we examine whether the benefits persist under SF estimation errors in Walker, Cheetah, and Quadruped using the perturbation in Equation (8), ψ(s,z)=ψ(s,z)+αgθ(z), where gθ is a fixed random network whose outputs have unit Euclidean norm. We vary the SF perturbation strength α∈{0,100,300,1000} using five seeds and retain the stationary hyperparameters, including the refresh threshold αER=0.1. Figure 14 in Appendix C.4 presents the learning curves for each α. The table below reports the results at α=1000.