acceptodds
Under review as a conference paper at ICLR 2027

Task Inference Beyond Least Squares in Behavioral Foundation Models

Abstract

Behavioral Foundation Models (BFMs) aim to solve a wide range of downstream tasks without test-time policy learning by inferring a task vector from the reward function. While efficient, the retrieved policies are often suboptimal because of how this task vector is inferred, typically with ordinary least squares (OLS). OLS minimizes reward reconstruction error but leaves the ordering of rewards unconstrained, which can bias the successor measure of the retrieved zero-shot policy away from that of the optimal policy. In this work, we propose BLS, an efficient test-time inference method that balances minimizing reward reconstruction error with reducing successor-measure mismatch. Theoretically, we provide a suboptimality gap upper bound characterized by both successor-measure and reward-function residuals. Empirically, we evaluate BLS on top of state-of-the-art BFMs across benchmarks for locomotion, manipulation, and humanoid control. BLS outperforms existing task inference baselines with negligible computational overhead.

Primary area:
reinforcement learning
Keywords:
Behavioral Foundation Models, Zero-Shot Reinforcement Learning, Task Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7psVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper investigates the innate problem of the ordinary least squares (OLS) extraction of policy representation at inference time in the BFMs. Since OLS aim at minimizing the expected mean across test-time reward labeled states, this could drift or bias the zero-shot extracted policy successor measure away from the optimal one. To mitigate this, authors propose BLS, which empirically performs significantly better to OLS. Moreover, theoretical analysis supports claims made on suboptimality gap.

Critical strengths

The method is very intuitive and simple to understand. It solves a crucial and important problem of extracting a near optimal policy at test time. Empirical results support made claims and robustness of the approach.

Critical weaknesses
  1. I would expect comparison with much simpler baselines, like ridge or reward-weighted regression.
  2. Preserving the ordering of immediate rewards does not generally preserve optimal policies: discounted returns depend on reward magnitudes, transition dynamics, reachability, and the duration and frequency of visits. Rewards with identical state rankings can induce different optimal policies. Moreover, Eq. (3) encourages separation between two threshold-defined groups and it does not explicitly preserve rankings within either group.
  3. The proposed method is limited to zero-shot RL algorithms, which extract policy representations only via OLS. For example, it is not clear how comparison with FB was made, since its extraction differs.
  4. Another limitation I see is that BLS is limited by original OLS extraction (since BLS initializes zz from OLS). So if OLS fails to extract good policy, BLS will not recover from this.
Questions for authors
  1. Vanilla FB infers the task vector as z=Eμ[Br]z = \mathbb{E}_{\mu}[Br]. From Eq. 2, if ϕ=B\phi = B,then z=CB−1E[Br]z =C^{-1}_B \mathbb{E}[Br] with C=E[B(s)B(s)T]C = \mathbb{E}[B(s)B(s)^T] Please specify the relationship between ϕ\phi and BB, any covariance correction and coordinate conversion, and whether the reported FB baseline implements native FB inference.

  2. Why should separating high- and low-reward states preserve optimal behaviour when returns also depend on reward magnitudes and dynamics?

  3. Does BLS outperform ridge regression, reward-weighted regression, and regression on thresholded rewards under matched conditions?

  4. How important initialization from OLS for BLS to succeed? Some ablations would improve the argument

7 comments

We thank the reviewer for the insightful feedback. We address the concerns below.

[W1, Q3] Does BLS outperform ridge regression, reward-weighted regression, and regression on thresholded rewards under matched conditions?

We compare BLS against five additional regression baselines:

  • Ridge Regression: Least squares with L2 regularization.

  • Reward-balanced LS (RBLS): Assigns equal total weights to high- and low-reward samples, with p=0.5p=0.5.

  • Group-weighted LS (G-WLS): Assigns total weights pp and 1−p1-p to high- and low-reward samples, respectively.

  • Reward-weighted Regression (R-WLS): Weights samples proportionally to r−min⁡(r)+ϵr-\min(r)+\epsilon, normalized to have mean weight one.

  • Thresholded Regression: Regresses onto binary reward targets indicating whether a sample belongs to the maximum-reward group.

We evaluate all five representations on the nine proprioceptive OGBench task groups, each containing five tasks, using three evaluation seeds (0, 1, 2) and 50 episodes per task and seed.

Laplacian

TaskOLSBLSRidgeRBLSG-WLSR-WLSThresh.
antmaze-mn48.1 ± 2.967.3 ± 8.134.0 ± 4.942.7 ± 0.845.3 ± 2.743.1 ± 2.032.3 ± 3.4
antmaze-ln7.6 ± 1.810.9 ± 2.911.6 ± 3.89.6 ± 3.13.9 ± 3.111.2 ± 3.99.9 ± 2.9
antmaze-ms15.1 ± 7.834.7 ± 3.49.7 ± 5.07.3 ± 6.28.1 ± 7.17.5 ± 7.29.9 ± 4.2
antmaze-ls15.2 ± 3.915.3 ± 0.613.5 ± 1.917.3 ± 1.217.3 ± 0.918.0 ± 1.114.1 ± 1.6
antmaze-me3.7 ± 3.37.6 ± 1.78.5 ± 0.53.5 ± 1.62.8 ± 0.76.3 ± 2.07.1 ± 1.8
cube-single19.9 ± 1.318.3 ± 4.613.3 ± 4.69.2 ± 1.68.5 ± 0.58.3 ± 0.813.2 ± 4.6
cube-double1.3 ± 0.62.0 ± 1.81.3 ± 0.81.7 ± 0.81.3 ± 0.81.6 ± 0.41.3 ± 0.5
scene4.3 ± 1.44.0 ± 1.73.7 ± 0.23.9 ± 0.53.9 ± 1.24.4 ± 1.14.0 ± 0.4
puzzle-3×31.5 ± 0.61.6 ± 0.82.5 ± 0.62.3 ± 0.62.4 ± 1.21.1 ± 0.52.1 ± 0.6
Average13.0 ± 0.518.0 ± 1.710.9 ± 0.110.8 ± 1.410.4 ± 1.111.3 ± 1.310.4 ± 0.7

ICVF

TaskOLSBLSRidgeRBLSG-WLSR-WLSThresh.
antmaze-mn66.3 ± 1.269.2 ± 6.762.7 ± 1.578.8 ± 1.277.6 ± 2.168.5 ± 5.362.9 ± 1.6
antmaze-ln56.0 ± 4.165.5 ± 4.954.4 ± 2.856.1 ± 6.053.7 ± 3.352.9 ± 2.652.3 ± 5.6
antmaze-ms28.8 ± 5.636.5 ± 9.129.2 ± 1.429.3 ± 1.028.0 ± 2.527.7 ± 1.728.3 ± 0.9
antmaze-ls5.5 ± 0.84.0 ± 0.74.0 ± 0.42.1 ± 0.23.1 ± 1.54.3 ± 1.65.2 ± 0.4
antmaze-me0.3 ± 0.50.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.00.0 ± 0.0
cube-single19.5 ± 1.213.3 ± 0.517.7 ± 2.918.0 ± 2.819.7 ± 2.617.9 ± 1.418.3 ± 2.0
cube-double3.1 ± 1.53.5 ± 0.84.0 ± 0.72.1 ± 0.83.1 ± 0.64.1 ± 0.52.9 ± 0.6
scene39.3 ± 3.247.7 ± 3.739.5 ± 1.434.9 ± 1.836.5 ± 3.244.0 ± 3.922.7 ± 2.3
puzzle-3×313.6 ± 0.715.3 ± 1.412.7 ± 0.812.8 ± 1.614.0 ± 1.112.9 ± 1.315.1 ± 1.2
Average25.8 ± 0.628.3 ± 2.324.9 ± 0.326.0 ± 0.426.2 ± 0.625.8 ± 0.123.1 ± 0.8

HILP

TaskOLSBLSRidgeRBLSG-WLSR-WLSThresh.
antmaze-mn74.3 ± 1.487.7 ± 1.565.9 ± 1.673.6 ± 1.880.8 ± 0.874.3 ± 4.773.6 ± 2.9
antmaze-ln49.2 ± 2.163.7 ± 1.848.1 ± 2.954.1 ± 4.148.5 ± 8.348.3 ± 9.455.3 ± 2.3
antmaze-ms26.3 ± 9.131.3 ± 2.335.3 ± 3.438.4 ± 2.638.8 ± 1.439.5 ± 2.630.1 ± 3.2
antmaze-ls11.9 ± 1.213.5 ± 0.89.6 ± 1.410.0 ± 2.29.2 ± 2.19.5 ± 2.68.9 ± 2.8
antmaze-me2.4 ± 2.66.7 ± 1.50.5 ± 0.20.8 ± 0.85.5 ± 7.42.7 ± 1.90.9 ± 0.5
cube-single84.5 ± 5.782.0 ± 6.781.6 ± 3.980.0 ± 3.274.4 ± 3.073.5 ± 1.682.5 ± 2.2
cube-double2.8 ± 3.125.6 ± 2.11.9 ± 1.22.8 ± 1.81.1 ± 1.20.0 ± 0.024.4 ± 2.5
scene26.3 ± 2.736.8 ± 2.235.6 ± 2.018.0 ± 0.419.3 ± 1.20.1 ± 0.246.3 ± 4.8
puzzle-3×32.4 ± 0.04.5 ± 1.57.6 ± 0.41.7 ± 0.61.3 ± 0.21.3 ± 0.23.6 ± 1.2
Average31.1 ± 2.139.1 ± 1.131.8 ± 0.731.1 ± 1.231.0 ± 2.027.7 ± 1.836.2 ± 0.7

FB

TaskOLSBLSRidgeRBLSG-WLSR-WLSThresh.
antmaze-mn77.5 ± 2.688.8 ± 4.278.4 ± 2.979.9 ± 2.781.7 ± 1.378.4 ± 2.679.9 ± 1.2
antmaze-ln52.4 ± 4.063.5 ± 9.251.6 ± 1.756.3 ± 1.352.1 ± 2.055.6 ± 3.551.6 ± 2.8
antmaze-ms69.9 ± 1.779.6 ± 2.169.9 ± 3.671.5 ± 1.871.7 ± 1.371.5 ± 1.270.0 ± 2.1
antmaze-ls51.5 ± 2.945.7 ± 4.643.5 ± 3.744.8 ± 11.943.2 ± 14.644.7 ± 6.141.9 ± 5.1
antmaze-me50.3 ± 2.049.2 ± 4.952.3 ± 0.248.0 ± 1.748.3 ± 0.550.1 ± 0.550.3 ± 1.3
cube-single50.4 ± 0.854.0 ± 1.849.5 ± 1.851.7 ± 3.650.3 ± 1.951.3 ± 4.451.5 ± 1.8
cube-double1.9 ± 1.03.1 ± 1.24.0 ± 1.12.7 ± 1.32.8 ± 0.42.9 ± 0.52.5 ± 0.2
scene7.6 ± 0.79.6 ± 0.46.7 ± 1.222.7 ± 4.822.5 ± 3.47.5 ± 2.046.7 ± 3.7
puzzle-3×35.7 ± 1.29.6 ± 1.86.4 ± 0.713.6 ± 1.412.4 ± 2.27.1 ± 0.215.3 ± 2.2
Average40.8 ± 0.344.8 ± 0.940.2 ± 1.243.5 ± 2.242.8 ± 2.141.0 ± 1.545.5 ± 1.4

TD-JEPA

TaskOLSBLSRidgeRBLSG-WLSR-WLSThresh.
antmaze-mn63.1 ± 0.883.2 ± 7.763.9 ± 2.467.2 ± 3.866.8 ± 3.066.0 ± 0.465.2 ± 4.2
antmaze-ln50.7 ± 2.464.3 ± 7.052.1 ± 1.652.8 ± 1.155.5 ± 4.453.5 ± 2.053.3 ± 0.5
antmaze-ms54.5 ± 4.859.5 ± 2.452.4 ± 3.049.3 ± 2.850.5 ± 3.050.9 ± 3.652.8 ± 4.6
antmaze-ls39.2 ± 5.945.1 ± 5.936.5 ± 3.128.9 ± 1.930.5 ± 5.230.7 ± 1.436.7 ± 4.4
antmaze-me15.1 ± 5.529.2 ± 7.023.9 ± 1.423.2 ± 1.223.6 ± 1.624.8 ± 3.523.7 ± 2.7
cube-single20.3 ± 2.955.9 ± 3.021.5 ± 4.332.4 ± 4.225.2 ± 4.038.4 ± 3.921.3 ± 3.8
cube-double1.7 ± 0.52.1 ± 1.20.9 ± 0.20.8 ± 0.41.1 ± 0.51.5 ± 1.51.6 ± 1.6
scene37.9 ± 3.254.0 ± 6.841.6 ± 1.143.6 ± 3.742.3 ± 2.241.3 ± 1.260.4 ± 3.5
puzzle-3×311.2 ± 0.414.1 ± 2.111.9 ± 2.012.3 ± 0.812.0 ± 1.212.5 ± 1.812.8 ± 3.5
Average32.6 ± 0.545.3 ± 1.933.9 ± 0.834.5 ± 0.534.2 ± 1.635.5 ± 0.736.4 ± 1.7

Overall Average

MethodSuccess Rate (%)
OLS28.7 ± 0.3
Ridge28.3 ± 0.5
Reward-balanced LS29.2 ± 0.4
Group-weighted LS28.9 ± 0.7
Reward-weighted Regression28.3 ± 0.1
Thresholded Regression30.3 ± 0.7
BLS35.1 ± 0.7

BLS outperforms all five additional regression baselines on average in four of five representations. The exception is FB, where Thresholded Regression achieves 45.5%, slightly above BLS at 44.8%, largely due to its stronger performance on scene. Overall, these results show that the improvements from BLS are not generally recovered by simpler regularized, weighted, balanced, or thresholded regression methods.

[W2, Q2] Preserving the ordering of immediate rewards does not generally preserve optimal policies. Eq. (3) encourages separation between two threshold-defined groups and it does not explicitly preserve rankings within either group. Why should separating high- and low-reward states preserve optimal behaviour when returns also depend on reward magnitudes and dynamics?

Reward magnitudes and reward ranking are two entangled factors for an optimal policy. Same ranking but different magnitudes may induce different optimal policies. Therefore, if considering soft-margin loss alone (ranking only, i.e., λ=0\lambda=0), as shown in Figure 4, the performance does not improve over total loss. Eq (3) does not explicitly preserve rankings within either group and it can only be ranked by trust-region loss for current total loss. We find that it is enough to show the gain over OLS in Figure 5. Furthermore, to build intuition on how Theorem 4.1 connects with the successor measure learning, we consider the binary-reward case (which is common in goal-conditioned RL) and further bound the sub-optimality gap by LsmL_{sm} in the following Proposition and we leave extension to more general reward functions as future work.

Proposition (Binary rewards). Let r=1Gr = \mathbb{1}_G, N=S∖GN = \mathcal{S}\setminus G, r^(s)=ϕ(s)⊤z\hat r(s) = \phi(s)^\top z with ∣z∣2=1\\|z\\|_2 = 1 and Lsm(z)=log⁡∑s∈Gexp⁡(−r^(s))+log⁡∑s∈Nexp⁡(r^(s))\mathcal{L}_{\mathrm{sm}}(z) = \log \sum_{s \in G} \exp\big(-\hat r(s)\big) + \log \sum_{s \in N} \exp\big(\hat r(s)\big) be the soft-margin loss. If Lsm<0\mathcal{L}_{\mathrm{sm}} < 0, then

SubOpt(r,r^);≤;11−γ⋅2max⁡s∣ϕ(s)∣22max⁡s∣ϕ(s)∣2−Lsm.\mathrm{SubOpt}(r, \hat r) \\;\le\\; \frac{1}{1-\gamma}\cdot\frac{2\max_s \\|\phi(s)\\|_2}{2\max_s \\|\phi(s)\\|_2 - \mathcal{L}_{\mathrm{sm}}} .

We further show that soft-margin loss can emprically reduce successor-measure mismatch by conducting a controlled experiment on finite MDPs, where both can be computed exactly.

We evaluate 500 goal-reaching tasks across 20 stochastic grid environments and 25 goals. Lower soft-margin loss correlates with lower successor-measure mismatch, with a mean Spearman correlation of 0.280 increasing to 0.497 using Eq. (3).

We further evaluate BLS optimization directly. The normalized successor-measure mismatch decreases from 0.145 to 0.062, while the true discounted return improves from 16.30 to 17.15 (optimal: 17.19).

These results provide empirical evidence that the soft-margin objective is associated with reduced successor-measure mismatch, beyond improving reward-group separation.

[W3]The proposed method is limited to zero-shot RL algorithms, which extract policy representations only via OLS. For example, it is not clear how comparison with FB was made, since its extraction differs.

We clarify that our method is limited to zero-shot RL algorithms, but not limited to extracting policy representations only via OLS. It only requires the policies of the BFM to be indexed by task vectors zz through rewards of the form ϕ(s)⊤∗z\phi(s)^\top*z, which is the basic structure of BFM. We show how to compare with FB in Q1.

[Q1] For FB, please specify the relationship between ϕ and B, any covariance correction and coordinate conversion, and whether the reported FB baseline implements native FB inference. Following the FB–SF correspondence, we use ϕ(s)=CB−1B(s)\phi(s)=C_B^{-1}B(s) as the feature map, where CB=Eρ[B(s)B(s)⊤]C_B=\mathbb{E}{\rho}[B(s)B(s)^\top]. As established by Section 4 in Touati et al.[1], OLS task inference reduces to native FB inference z=Eρ[B(s)r(s)]z=\mathbb{E}{\rho}[B(s)r(s)], so the covariance correction is already accounted for. Let BB denote the empirical FB inference operator and B†B^{\dagger} its Moore-Penrose pseudoinverse. In practice, we reconstruct rewards from zz as r^=B†z\hat{r}=B^{\dagger} z, using the corresponding least-squares inverse of native FB inference.

[1] Touati et al. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations (ICLR), 2023.

[W4, Q4] If OLS fails to extract good policy, BLS will not recover from this. How important is initialization from OLS for BLS to succeed? Some ablations would improve the argument.

We agree that the quality of the initialization affects BLS, but BLS can still recover when OLS fails. In Table 4, OLS collapses for large feature dimensions (e.g., 9.2% at d=500) while BLS initialized from the same OLS solution reaches 79.6%. To further study the role of the initialization, we extend Table 9 on HumEnv with a random initialization.

InitializationBefore BLSAfter BLS
Random0.075 ± 0.0070.494 ± 0.014
OLS0.496 ± 0.0180.648 ± 0.022
Weighted regression0.613 ± 0.0380.700 ± 0.023

BLS improves performance from all three initializations, and the gain is larger when the initialization is weaker. Starting from a random task vector, BLS raises the expert-normalized return from 0.075 to 0.494. This indicates that the soft-margin objective alone recovers substantial task information. However, BLS from random initialization does not reach the performance of BLS from OLS or weighted regression, and the final performance increases with the quality of the initialization (0.494 → 0.648 → 0.700). This also shows that by preserving the reward ordering alone does not generally preserve optimal policies.

Reviewer_R76wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BLS, a procedure that refines the zero-shot latent of successor-feature-based BFMs with gradient updates. Notably, BLS does not need access to the environment or extra data and is very lightweight, making it still a zero-shot reward inference method. The authors motivate the loss of BLS by recognizing that the error between inferred and true reward affects the final policy performance differently depending on how often the state with this error is visited by the policy, giving great intuition in text and with a theoretical upper bound which the authors acknowledge does not give any guarantees for BLS. The paper then validates BLS’ effectiveness across a large range of common zero-shot RL benchmarks and methods, showing that their method improves downstream task performance in most cases. Notably, they also show that their method is not overly sensitive to the two hyperparameters it additionally depends on. This is specifically important for the threshold that is used to separate the states into a low and high reward region as this step is motivated by intuition alone but seems to work well across all settings and a large range of values.

Critical strengths

The authors provide ample evidence that BLS improves performance in a wide range of zero-shot RL scenarios. By showing that OLS is not the best reward inference in the zero-shot setting (i.e. no online adaptation), this paper potentially opens up a new line of research whose inference improvements are transferable to any BFM algorithm.

Critical weaknesses

The main weakness of this paper is that BLS is not directly by theoretical analysis. While the authors bound a suboptimality gap between inferred and true reward, the bound depends on intractable values, eg. the successor measure of the optimal policy. The authors do not attempt to bound their theoretical result further until it can be turned into a tractable loss directly but rather use this result only as intuition that justifies their method.

Questions for authors

I have no clarifying questions to the authors. My main suggestion for future work would be to develop the theoretical analysis they have further until it becomes a tractable loss function. This may be only possible by introducing additional assumptions on the true reward function.

3 comments

We thank the reviewer for the insightful comments. We address the concerns below.

[W1, Q1] The suboptimality gap depends on intractable values. Would it be turned into a tractable loss directly in the future work?

We agree that Theorem 4.1 cannot be optimized directly, since it involves the successor measure of the unknown optimal policy. To build intuition on how Theorem 4.1 connects with the successor measure learning, we consider the binary-reward case (which is common in goal-conditioned RL) and further bound the sub-optimality gap by LsmL_{sm} in the following Proposition.

Proposition (Binary rewards). Let r=1Gr = \mathbb{1}_G, N=S∖GN = \mathcal{S}\setminus G, r^(s)=ϕ(s)⊤z\hat r(s) = \phi(s)^\top z with ∣z∣2=1\\|z\\|_2 = 1 and Lsm(z)=log⁡∑s∈Gexp⁡(−r^(s))+log⁡∑s∈Nexp⁡(r^(s))\mathcal{L}_{\mathrm{sm}}(z) = \log \sum_{s \in G} \exp\big(-\hat r(s)\big) + \log \sum_{s \in N} \exp\big(\hat r(s)\big) be the soft-margin loss. If Lsm<0\mathcal{L}_{\mathrm{sm}} < 0, then

SubOpt(r,r^);≤;11−γ⋅2max⁡s∣ϕ(s)∣22max⁡s∣ϕ(s)∣2−Lsm.\mathrm{SubOpt}(r, \hat r) \\;\le\\; \frac{1}{1-\gamma}\cdot\frac{2\max_s \\|\phi(s)\\|_2}{2\max_s \\|\phi(s)\\|_2 - \mathcal{L}_{\mathrm{sm}}} .

We further show that soft-margin loss can emprically reduce successor-measure mismatch by conducting a controlled experiment on finite MDPs, where both can be computed exactly.

We evaluate 500 goal-reaching tasks across 20 stochastic grid environments and 25 goals. Lower soft-margin loss correlates with lower successor-measure mismatch, with a mean Spearman correlation of 0.280 increasing to 0.497 using Eq. (3).

We further evaluate BLS optimization directly. The normalized successor-measure mismatch decreases from 0.145 to 0.062, while the true discounted return improves from 16.30 to 17.15 (optimal: 17.19).

These results provide empirical evidence that the soft-margin objective is associated with reduced successor-measure mismatch.

Finally, we thank the reviewer for the suggestion that turning the bound into a tractable objective is an important direction for future work. The main challenge is that the successor measure of the optimal policy depends jointly on the reward function and the environment dynamics. Making it tractable has to consider the reward and the dynamics together. Beyond the standard assumption that the reward lies in the span of the learned features, a natural choice is to restrict the reward class to tasks whose optimal policy visits the highest-reward states with at least a minimum discounted frequency. We expect that soft-margin loss increases the probability that the retrieved policy visits high-reward states, and thus reduces the successor-measure mismatch.

Proof of Proposition. For any policy π\pi, ⟨Mˉπ,1G⟩ρ\langle \bar M^\pi, \mathbb{1}_G \rangle_\rho and ⟨Mˉπ,1N⟩ρ\langle \bar M^\pi, \mathbb{1}_N \rangle_\rho are the successor measures of π\pi on GG and NN, and they sum to 11−γ\frac{1}{1-\gamma}. Moreover, since the range of r^\hat r within GG or NN is at most 2∣r^∣∞2\\|\hat r\\|_\infty, we have max⁡Gr^≤min⁡Gr^+2∣r^∣∞\max_G \hat r \le \min_G \hat r + 2\\|\hat r\\|_\infty and min⁡Nr^≥max⁡Nr^−2∣r^∣∞\min_N \hat r \ge \max_N \hat r - 2\\|\hat r\\|_\infty.

The value Vr^πr^V^{\pi_{\hat r}}_{\hat r} is at most what πr^\pi_{\hat r} would collect if it received max⁡Gr^\max_G \hat r whenever it visits GG and max⁡Nr^\max_N \hat r whenever it visits NN:

V^{\pi_{\hat r}}_{\hat r} &\le \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho \max_G \hat r + \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_N \rangle_\rho \max_N \hat r \\\\ &\le \tfrac{\max_N \hat r}{1-\gamma} + \big(\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty\big) \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho . \end{aligned} \tag{R1}$$ Similarly, the value $V^{\pi_r}_{\hat r}$ is at least what $\pi_r$ would collect if it received $\min_G \hat r$ whenever it visits $G$ and $\min_N \hat r$ whenever it visits $N$: $$\begin{aligned} V^{\pi_r}_{\hat r} &\ge \langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho \min_G \hat r + \langle \bar M^{\pi_r}, \mathbb{1}_N \rangle_\rho \min_N \hat r \\\\ &\ge \tfrac{\max_N \hat r}{1-\gamma} + \big(\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty\big) \langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho - \tfrac{2\\|\hat r\\|_\infty}{1-\gamma} . \end{aligned} \tag{R2}$$ Since $\pi_{\hat r}$ is optimal for $\hat r$, we have $V^{\pi_{\hat r}}_{\hat r} \ge V^{\pi_r}_{\hat r}$. Combining this with (R1) and (R2) gives $$\langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho - \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho \le \frac{1}{1-\gamma}\cdot\frac{2\\|\hat r\\|_\infty}{\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty} . \tag{R3}$$ Since $r = \mathbb{1}_G$, we have $V^\pi_r = \langle \bar M^\pi, \mathbb{1}_G \rangle_\rho$ for any $\pi$, so the left-hand side of (R3) equals $V^{\pi_r}_r - V^{\pi_{\hat r}}_r = \mathrm{SubOpt}(r, \hat r)$. Next, since $\log\sum_i e^{x_i} \ge \max_i x_i$, the soft-margin loss gives $$\mathcal{L}_{\mathrm{sm}} \ge \max_N \hat r - \min_G \hat r , \tag{R4}$$ and since $\mathcal{L}_{\mathrm{sm}} < 0$, this implies $\min_G \hat r - \max_N \hat r \ge -\mathcal{L}_{\mathrm{sm}} > 0$. Moreover, by the Cauchy–Schwarz inequality and $\\|z\\|_2 = 1$, $$\\|\hat r\\|_\infty = \max_s |\phi(s)^\top z| \le \\|z\\|_2 \max_s \\|\phi(s)\\|_2 = \max_s \\|\phi(s)\\|_2 . \tag{R5}$$ Since $x \mapsto \frac{x}{a + x}$ is increasing in $x$ and decreasing in $a$ for $a, x > 0$, substituting (R4) and (R5) into (R3), we conclude that $$\mathrm{SubOpt}(r, \hat r) \\;\le\\; \frac{1}{1-\gamma}\cdot\frac{2\max_s \\|\phi(s)\\|_2}{2\max_s \\|\phi(s)\\|_2 - \mathcal{L}_{\mathrm{sm}}} . \qquad \blacksquare$$ In practice, a temperature $\tau > 0$ can be added to the soft-margin loss, i.e., $\tau \log\sum_{s\in G} e^{-\hat r(s)/\tau} + \tau \log\sum_{s\in N} e^{\hat r(s)/\tau}$. Since this loss still satisfies (R4), the proposition continues to hold, and whenever $\min_G \hat r > \max_N \hat r$, any $\tau < (\min_G \hat r - \max_N \hat r) / (\log|G| + \log|N|)$ ensures $\mathcal{L}_{\mathrm{sm}} < 0$. Empirically, we found that $\mathcal{L}_{\mathrm{sm}}$ is indeed negative.

Dear Reviewer R76w,

As the discussion period will end soon, we wanted to kindly follow up in case you have not yet had a chance to review our rebuttal. If you have any remaining questions or concerns, we would be very happy to provide further clarification.

To briefly summarize our main update addressing your concern on the gap between the theoretical analysis and the empirical study, we derived a bound for the binary-reward setting that connects our theoretical analysis to the empirical results, and further measured empirically that the ranking loss reduces the successor-measure mismatch.

We hope this additional analysis provides stronger evidence for the effectiveness of BLS and addresses your concern regarding the connection between theory and practice. If our response has addressed your concern, we would be grateful if you could reconsider your evaluation.

We sincerely appreciate your time and constructive suggestions, and look forward to hearing from you.

Reviewer_jJd5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies task inference in Behavioral Foundation Models, where a frozen policy is conditioned on a task vector inferred from reward-labeled states. It argues that ordinary least squares (OLS) can reconstruct rewards reasonably well while retrieving a suboptimal policy. The proposed BLS method refines the task vector using a soft-margin objective that separates high-reward from low-reward states, together with a regularizer that keeps reconstructed rewards close to those produced by an initial inference method. The policy and representation remain frozen, and refinement requires no additional environment interaction.

Critical strengths

This paper works on a useful and practically relevant problem. Improving task inference without retraining a large behavioral model is valuable. BLS requires only reward-feature samples and optimizes a small task vector, making it applicable across different representation-learning methods.

Critical weaknesses

Limited connection between theory and algorithm. The suboptimality bound is valid under its assumptions, but the paper does not establish that minimizing the proposed ranking loss reduces successor-measure mismatch. Reward ordering alone does not generally determine optimal behavior.

Questions for authors
  1. Does reducing the soft-margin loss empirically reduce successor-measure mismatch, beyond improving reward-group separation?
  2. How does BLS compare with ridge regression and reward-balanced least squares, especially in Table 4?
  3. How were the OGBench regularization coefficients selected, and how is the dense-task cosine target enforced? Does the reported runtime include that procedure?
  4. Why do ReLA and LoLA frequently underperform their initialization, and how do BLS and ZOL compare on DMC or HumEnv?
5 comments

We thank the reviewer for the insightful comments. We address the concerns below.

[W1] Reward ordering alone does not generally determine optimal behavior. The paper does not establish that minimizing the proposed ranking loss reduces successor-measure mismatch.

Reward magnitudes and reward ranking are two entangled factors for an optimal policy. Same ranking but different magnitudes may induce different optimal policies. Therefore, if considering soft-margin loss alone (ranking only, i.e., λ=0\lambda=0), as shown in Figure 4, the performance does not improve over total loss.

We agree that Theorem 4.1 cannot be optimized directly, since it involves the successor measure of the unknown optimal policy. To build intuition on how Theorem 4.1 connects with the successor measure learning, we consider the binary-reward case (which is common in goal-conditioned RL) and further bound the sub-optimality gap by LsmL_{sm} in the following Proposition and we leave extension to more general reward functions as future work.

Proposition (Binary rewards). Let r=1Gr = \mathbf{1}_G be the indicator of a goal set G⊆SG \subseteq \mathcal{S}, let N=S∖GN = \mathcal{S} \setminus G, and let r^(s)=ϕ(s)⊤z\hat{r}(s) = \phi(s)^\top z with ∥z∥2=1\Vert z \Vert_2 = 1. Recap the soft-margin loss

Lsm(z)=log⁡∑s∈Gexp⁡(−r^(s))+log⁡∑s∈Nexp⁡(r^(s))\mathcal{L}_{\mathrm{sm}}(z) = \log \underset{s \in G}{\sum} \exp\big(-\hat{r}(s)\big) + \log \underset{s \in N}{\sum} \exp\big(\hat{r}(s)\big)

If Lsm(z)<0\mathcal{L}_{\mathrm{sm}}(z) < 0, then

SubOpt(r,r^)≤2max⁡s∥ϕ(s)∥2(1−γ)⋅(−Lsm(z)).\mathrm{SubOpt}(r, \hat{r}) \le \frac{2 \max_{s} \Vert \phi(s) \Vert_2}{(1-\gamma) \cdot \left(-\mathcal{L}_{\mathrm{sm}}(z)\right)}.

[Q1] Does reducing the soft-margin loss empirically reduce successor-measure mismatch, beyond improving reward-group separation?

We further show that soft-margin loss can empirically reduce successor-measure mismatch by conducting a controlled experiment on finite MDPs, where both can be computed exactly.

We evaluate 500 goal-reaching tasks across 20 stochastic grid environments and 25 goals. Lower soft-margin loss correlates with lower successor-measure mismatch, with a mean Spearman correlation of 0.280 increasing to 0.497 using Eq. (3).

We further evaluate BLS optimization directly. The normalized successor-measure mismatch decreases from 0.145 to 0.062, while the true discounted return improves from 16.30 to 17.15 (optimal: 17.19).

These results provide empirical evidence that the soft-margin objective is associated with reduced successor-measure mismatch, beyond improving reward-group separation.

[Q2] How does BLS compare with ridge regression and reward-balanced least squares, especially in Table 4?

We compare BLS against Ridge Regression and Reward-balanced LS across six feature dimensions using HILP on antmaze-medium-navigation.

  • Ridge Regression: Least squares with L2 regularization.

  • Reward-balanced LS (RBLS): Assigns equal total weights to high- and low-reward samples, with p=0.5p=0.5.

Feature dimensionOLSBLSRidgeRBLS
2570.984.719.120.9
5074.387.765.973.9
10059.287.520.531.6
20025.382.159.522.9
5009.279.682.117.6
100022.082.773.39.6

BLS maintains success rates between 79.6% and 87.7% across all six feature dimensions, whereas OLS, Ridge, and RBLS vary by more than 60 percentage points, showing that BLS is robust to the choice of feature dimension.

[Q3] How were the OGBench regularization coefficients selected, and how is the dense-task cosine target enforced? Does the reported runtime include that procedure?

For OGBench, we empirically explored a small set of λ\lambda values, including 0,0.3,1,2,3,10,20\\{0, 0.3, 1, 2, 3, 10, 20\\}, and selected a fixed coefficient for each representation and observation modality across all tasks, as reported in Appendix B. As discussed in Section 5.5, λ\lambda balances reward separation and preserving the initial reward reconstruction. Smaller values generally work better for sparse rewards, while larger values favor dense rewards. Although λ\lambda may require tuning across different settings, BLS improves average performance across all five representations without task-specific tuning.

For dense-reward tasks, we select λ\lambda to achieve a cosine similarity closest to 0.95 between the refined and anchor task vectors. We evaluate 13 positive λ\lambda candidates and a λ=0\lambda=0 control, each using 500 optimization steps, and select the candidate closest to the cosine target.

The runtime reported in the paper measures BLS optimization with a fixed λ\lambda and does not include λ\lambda selection. In our FB timing experiment on an NVIDIA RTX 6000 Ada GPU, a single 500-step optimization takes 0.52 seconds, while the complete λ\lambda selection takes 7.17 seconds per task on average (range: 6.70–7.93 seconds across three seeds). This includes optimization of the selected task vector, which is reused without additional optimization. The timing excludes data loading, feature extraction, and policy rollout.

[Q4] Why do ReLA and LoLA frequently underperform their initialization, and how do BLS and ZOL compare on DMC or HumEnv?

This behavior is consistent with prior observations. Sikchi et al. [1] report that ReLA exhibits non-monotonic performance with a noticeable initial drop, and there are two reasons. First, ReLA trains its residual critic purely on online samples, whose distribution differs from the pretraining data. Second, TD3-based critic learning struggles with exploration in long-horizon tasks. We expect both issues to be more severe on OGBench, whose tasks are long-horizon with sparse rewards. ReLA drops from 28.7% to 20.8% on average.

In contrast, LoLA does not learn a critic and updates the task vector with lookahead returns, so it is not affected by the critic distribution shift. Its average success rate (27.7% ± 0.7) remains within about one percentage point of the OLS initialization (28.7% ± 0.5) so we attribute this small gap to the high variance of LoLA's gradient estimates under sparse rewards.

We compare BLS and ZOL with FB representations on the 16 DMC tasks (average episode return) and on the 45 HumEnv tasks (average expert-normalized return).

BenchmarkZOLBLS
DMC627.2 ± 0.9642.0 ± 1
HumEnv0.688 ± 0.010.700 ± 0.023

The results show that BLS achieves higher final performance than ZOL on both benchmarks.

[1] Sikchi et al., Fast adaptation with behavioral foundation models. RLC, 2025

Proof of Proposition. For any policy π\pi, ⟨Mˉπ,1G⟩ρ\langle \bar M^\pi, \mathbb{1}_G \rangle_\rho and ⟨Mˉπ,1N⟩ρ\langle \bar M^\pi, \mathbb{1}_N \rangle_\rho are the successor measures of π\pi on GG and NN, and they sum to 11−γ\frac{1}{1-\gamma}. Moreover, since the range of r^\hat r within GG or NN is at most 2∣r^∣∞2\\|\hat r\\|_\infty, we have max⁡Gr^≤min⁡Gr^+2∣r^∣∞\max_G \hat r \le \min_G \hat r + 2\\|\hat r\\|_\infty and min⁡Nr^≥max⁡Nr^−2∣r^∣∞\min_N \hat r \ge \max_N \hat r - 2\\|\hat r\\|_\infty.

The value Vr^πr^V^{\pi_{\hat r}}_{\hat r} is at most what πr^\pi_{\hat r} would collect if it received max⁡Gr^\max_G \hat r whenever it visits GG and max⁡Nr^\max_N \hat r whenever it visits NN:

V^{\pi_{\hat r}}_{\hat r} &\le \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho \max_G \hat r + \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_N \rangle_\rho \max_N \hat r \\\\ &\le \tfrac{\max_N \hat r}{1-\gamma} + \big(\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty\big) \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho . \end{aligned} \tag{R1}$$ Similarly, the value $V^{\pi_r}_{\hat r}$ is at least what $\pi_r$ would collect if it received $\min_G \hat r$ whenever it visits $G$ and $\min_N \hat r$ whenever it visits $N$: $$\begin{aligned} V^{\pi_r}_{\hat r} &\ge \langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho \min_G \hat r + \langle \bar M^{\pi_r}, \mathbb{1}_N \rangle_\rho \min_N \hat r \\\\ &\ge \tfrac{\max_N \hat r}{1-\gamma} + \big(\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty\big) \langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho - \tfrac{2\\|\hat r\\|_\infty}{1-\gamma} . \end{aligned} \tag{R2}$$ Since $\pi_{\hat r}$ is optimal for $\hat r$, we have $V^{\pi_{\hat r}}_{\hat r} \ge V^{\pi_r}_{\hat r}$. Combining this with (R1) and (R2) gives $$\begin{aligned} \langle \bar M^{\pi_r}, \mathbb{1}_G \rangle_\rho - \langle \bar M^{\pi_{\hat r}}, \mathbb{1}_G \rangle_\rho &\le \frac{2\\|\hat r\\|_\infty}{(1-\gamma)\big(\min_G \hat r - \max_N \hat r + 2\\|\hat r\\|_\infty\big)} \\\\ &\le \frac{2\\|\hat r\\|_\infty}{(1-\gamma)\big(\min_G \hat r - \max_N \hat r\big)} . \end{aligned} \tag{R3}$$ Since $r = \mathbb{1}_G$, we have $V^\pi_r = \langle \bar M^\pi, \mathbb{1}_G \rangle_\rho$ for any $\pi$, so the left-hand side of (R3) equals $V^{\pi_r}_r - V^{\pi_{\hat r}}_r = \mathrm{SubOpt}(r, \hat r)$. Next, since $\log\sum_i e^{x_i} \ge \max_i x_i$, the soft-margin loss gives $$\mathcal{L}_{\mathrm{sm}} \ge \max_N \hat r - \min_G \hat r , \tag{R4}$$ and since $\mathcal{L}_{\mathrm{sm}} < 0$, this implies $\min_G \hat r - \max_N \hat r \ge -\mathcal{L}_{\mathrm{sm}} > 0$. Moreover, by the Cauchy–Schwarz inequality and $\\|z\\|_2 = 1$, $$\\|\hat r\\|_\infty = \max_s |\phi(s)^\top z| \le \\|z\\|_2 \max_s \\|\phi(s)\\|_2 = \max_s \\|\phi(s)\\|_2 . \tag{R5}$$ Substituting (R4) and (R5) into (R3), we conclude that $$\mathrm{SubOpt}(r, \hat r) \\;\le\\; \frac{2\max_s \\|\phi(s)\\|_2}{(1-\gamma)\\,(-\mathcal{L}_{\mathrm{sm}})} . \qquad \blacksquare$$ In practice, a temperature $\tau > 0$ can be added to the soft-margin loss, i.e., $\tau \log\sum_{s\in G} e^{-\hat r(s)/\tau} + \tau \log\sum_{s\in N} e^{\hat r(s)/\tau}$. Since this loss still satisfies (R4), the proposition continues to hold, and whenever $\min_G \hat r > \max_N \hat r$, any $\tau < (\min_G \hat r - \max_N \hat r) / (\log|G| + \log|N|)$ ensures $\mathcal{L}_{\mathrm{sm}} < 0$. Empirically, we found that $\mathcal{L}_{\mathrm{sm}}$ is indeed negative.

Comments on the paper

1 comment

Dear Area Chair,

We thank you and all reviewers for their time and constructive feedback. Below, we summarize the main concerns raised and how we addressed them in the rebuttal and revised paper.

1. Connection between theoretical analysis and empirical study (Reviewers jJd5, R76w, 7psV) All three reviewers asked how our theoretical analysis relates to the empirical results. We derived a bound for the binary-reward setting that connects the theory to the empirical objective, and further measured empirically that the ranking loss reduces the successor-measure mismatch, providing direct evidence that the mechanism identified in our analysis is at work in practice.

2. Comparison with other regression baselines (Reviewers jJd5, 7psV) We added five regression baselines: ridge regression (Ridge), reward-balanced LS (RBLS), group-weighted LS (G-WLS), reward-weighted regression (R-WLS), and regression on thresholded rewards (Thresh.). BLS outperforms all five on average in four of the five representations. Moreover, BLS maintains success rates between 79.6% and 87.7% across all six feature dimensions, whereas OLS, Ridge, and RBLS vary by more than 60 percentage points, showing that BLS is robust to the choice of feature dimension.

3. Applicability beyond OLS (Reviewer 7psV) We clarified that BLS is not limited to OLS-based task inference and provided a comparison with FB.

4. Initialization of BLS (Reviewer 7psV) We compared random, OLS, and reward-weighted initializations, and found that better initialization leads to better final performance.

5. Comparison with other adaptation methods (Reviewer jJd5) We analyzed the reasons for the performance drop of other adaptation methods and added a comparison with ZOL on an additional benchmark.

6. Experimental settings (Reviewer jJd5) We provided a detailed description of the experimental settings to improve clarity.

Regarding Reviewer 7psV's remaining concerns

We completed all experiments requested by Reviewer 7psV during the discussion. In their final response, the reviewer raised two further points, which we briefly address:

(a) Magnitude of improvement. BLS is an inference-time method applied to the same pretrained BFM as OLS. Since the pretraining is shared, the performance difference between BLS and OLS is attributable to task inference. BLS achieves this improvement without any additional training, data, or environment interaction, and the same loss also improves FB-based task inference. In the feature dimension experiment, OLS varies by more than 60 percentage points, whereas BLS stays within 79.6–87.7%. Regarding randomness, BLS exceeds OLS by more than one standard deviation in most of the settings across three seeds, providing further evidence that the gains are not merely due to seed variance.

(b) Dependence on the sampled batch. All zero-shot task inference methods in BFMs, including OLS and FB, estimate the task embedding from a sampled reward-labeled batch. BLS uses exactly the same batch as OLS and introduces no additional sampling requirements. To directly address the reviewer's concern, we conducted a sample-size ablation on HILP AntMaze-Large across five tasks and three seeds:

Inference samplesOLS (%)BLS (%)
10,00055.60 ± 2.8861.33 ± 5.14
50,00060.13 ± 0.6166.80 ± 4.92
100,00056.27 ± 4.0966.27 ± 1.67

BLS consistently outperforms OLS across all evaluated sample sizes. Its performance also remains relatively stable as the sample size increases from 10,000 to 100,000, suggesting that BLS is not particularly sensitive to the number of inference samples within this range.

We believe these additions substantially strengthen the paper's theoretical grounding, empirical validation, and clarity. We sincerely appreciate your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.