- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper investigates the innate problem of the ordinary least squares (OLS) extraction of policy representation at inference time in the BFMs. Since OLS aim at minimizing the expected mean across test-time reward labeled states, this could drift or bias the zero-shot extracted policy successor measure away from the optimal one. To mitigate this, authors propose BLS, which empirically performs significantly better to OLS. Moreover, theoretical analysis supports claims made on suboptimality gap.
Critical strengths
The method is very intuitive and simple to understand. It solves a crucial and important problem of extracting a near optimal policy at test time. Empirical results support made claims and robustness of the approach.
Critical weaknesses
- I would expect comparison with much simpler baselines, like ridge or reward-weighted regression.
- Preserving the ordering of immediate rewards does not generally preserve optimal policies: discounted returns depend on reward magnitudes, transition dynamics, reachability, and the duration and frequency of visits. Rewards with identical state rankings can induce different optimal policies. Moreover, Eq. (3) encourages separation between two threshold-defined groups and it does not explicitly preserve rankings within either group.
- The proposed method is limited to zero-shot RL algorithms, which extract policy representations only via OLS. For example, it is not clear how comparison with FB was made, since its extraction differs.
- Another limitation I see is that BLS is limited by original OLS extraction (since BLS initializes from OLS). So if OLS fails to extract good policy, BLS will not recover from this.
Questions for authors
-
Vanilla FB infers the task vector as . From Eq. 2, if ,then with Please specify the relationship between and , any covariance correction and coordinate conversion, and whether the reported FB baseline implements native FB inference.
-
Why should separating high- and low-reward states preserve optimal behaviour when returns also depend on reward magnitudes and dynamics?
-
Does BLS outperform ridge regression, reward-weighted regression, and regression on thresholded rewards under matched conditions?
-
How important initialization from OLS for BLS to succeed? Some ablations would improve the argument
7 comments
FB
| Task | OLS | BLS | Ridge | RBLS | G-WLS | R-WLS | Thresh. |
|---|---|---|---|---|---|---|---|
| antmaze-mn | 77.5 ± 2.6 | 88.8 ± 4.2 | 78.4 ± 2.9 | 79.9 ± 2.7 | 81.7 ± 1.3 | 78.4 ± 2.6 | 79.9 ± 1.2 |
| antmaze-ln | 52.4 ± 4.0 | 63.5 ± 9.2 | 51.6 ± 1.7 | 56.3 ± 1.3 | 52.1 ± 2.0 | 55.6 ± 3.5 | 51.6 ± 2.8 |
| antmaze-ms | 69.9 ± 1.7 | 79.6 ± 2.1 | 69.9 ± 3.6 | 71.5 ± 1.8 | 71.7 ± 1.3 | 71.5 ± 1.2 | 70.0 ± 2.1 |
| antmaze-ls | 51.5 ± 2.9 | 45.7 ± 4.6 | 43.5 ± 3.7 | 44.8 ± 11.9 | 43.2 ± 14.6 | 44.7 ± 6.1 | 41.9 ± 5.1 |
| antmaze-me | 50.3 ± 2.0 | 49.2 ± 4.9 | 52.3 ± 0.2 | 48.0 ± 1.7 | 48.3 ± 0.5 | 50.1 ± 0.5 | 50.3 ± 1.3 |
| cube-single | 50.4 ± 0.8 | 54.0 ± 1.8 | 49.5 ± 1.8 | 51.7 ± 3.6 | 50.3 ± 1.9 | 51.3 ± 4.4 | 51.5 ± 1.8 |
| cube-double | 1.9 ± 1.0 | 3.1 ± 1.2 | 4.0 ± 1.1 | 2.7 ± 1.3 | 2.8 ± 0.4 | 2.9 ± 0.5 | 2.5 ± 0.2 |
| scene | 7.6 ± 0.7 | 9.6 ± 0.4 | 6.7 ± 1.2 | 22.7 ± 4.8 | 22.5 ± 3.4 | 7.5 ± 2.0 | 46.7 ± 3.7 |
| puzzle-3×3 | 5.7 ± 1.2 | 9.6 ± 1.8 | 6.4 ± 0.7 | 13.6 ± 1.4 | 12.4 ± 2.2 | 7.1 ± 0.2 | 15.3 ± 2.2 |
| Average | 40.8 ± 0.3 | 44.8 ± 0.9 | 40.2 ± 1.2 | 43.5 ± 2.2 | 42.8 ± 2.1 | 41.0 ± 1.5 | 45.5 ± 1.4 |
TD-JEPA
| Task | OLS | BLS | Ridge | RBLS | G-WLS | R-WLS | Thresh. |
|---|---|---|---|---|---|---|---|
| antmaze-mn | 63.1 ± 0.8 | 83.2 ± 7.7 | 63.9 ± 2.4 | 67.2 ± 3.8 | 66.8 ± 3.0 | 66.0 ± 0.4 | 65.2 ± 4.2 |
| antmaze-ln | 50.7 ± 2.4 | 64.3 ± 7.0 | 52.1 ± 1.6 | 52.8 ± 1.1 | 55.5 ± 4.4 | 53.5 ± 2.0 | 53.3 ± 0.5 |
| antmaze-ms | 54.5 ± 4.8 | 59.5 ± 2.4 | 52.4 ± 3.0 | 49.3 ± 2.8 | 50.5 ± 3.0 | 50.9 ± 3.6 | 52.8 ± 4.6 |
| antmaze-ls | 39.2 ± 5.9 | 45.1 ± 5.9 | 36.5 ± 3.1 | 28.9 ± 1.9 | 30.5 ± 5.2 | 30.7 ± 1.4 | 36.7 ± 4.4 |
| antmaze-me | 15.1 ± 5.5 | 29.2 ± 7.0 | 23.9 ± 1.4 | 23.2 ± 1.2 | 23.6 ± 1.6 | 24.8 ± 3.5 | 23.7 ± 2.7 |
| cube-single | 20.3 ± 2.9 | 55.9 ± 3.0 | 21.5 ± 4.3 | 32.4 ± 4.2 | 25.2 ± 4.0 | 38.4 ± 3.9 | 21.3 ± 3.8 |
| cube-double | 1.7 ± 0.5 | 2.1 ± 1.2 | 0.9 ± 0.2 | 0.8 ± 0.4 | 1.1 ± 0.5 | 1.5 ± 1.5 | 1.6 ± 1.6 |
| scene | 37.9 ± 3.2 | 54.0 ± 6.8 | 41.6 ± 1.1 | 43.6 ± 3.7 | 42.3 ± 2.2 | 41.3 ± 1.2 | 60.4 ± 3.5 |
| puzzle-3×3 | 11.2 ± 0.4 | 14.1 ± 2.1 | 11.9 ± 2.0 | 12.3 ± 0.8 | 12.0 ± 1.2 | 12.5 ± 1.8 | 12.8 ± 3.5 |
| Average | 32.6 ± 0.5 | 45.3 ± 1.9 | 33.9 ± 0.8 | 34.5 ± 0.5 | 34.2 ± 1.6 | 35.5 ± 0.7 | 36.4 ± 1.7 |
Overall Average
| Method | Success Rate (%) |
|---|---|
| OLS | 28.7 ± 0.3 |
| Ridge | 28.3 ± 0.5 |
| Reward-balanced LS | 29.2 ± 0.4 |
| Group-weighted LS | 28.9 ± 0.7 |
| Reward-weighted Regression | 28.3 ± 0.1 |
| Thresholded Regression | 30.3 ± 0.7 |
| BLS | 35.1 ± 0.7 |
BLS outperforms all five additional regression baselines on average in four of five representations. The exception is FB, where Thresholded Regression achieves 45.5%, slightly above BLS at 44.8%, largely due to its stronger performance on scene. Overall, these results show that the improvements from BLS are not generally recovered by simpler regularized, weighted, balanced, or thresholded regression methods.
[W2, Q2] Preserving the ordering of immediate rewards does not generally preserve optimal policies. Eq. (3) encourages separation between two threshold-defined groups and it does not explicitly preserve rankings within either group. Why should separating high- and low-reward states preserve optimal behaviour when returns also depend on reward magnitudes and dynamics?
Reward magnitudes and reward ranking are two entangled factors for an optimal policy. Same ranking but different magnitudes may induce different optimal policies. Therefore, if considering soft-margin loss alone (ranking only, i.e., ), as shown in Figure 4, the performance does not improve over total loss. Eq (3) does not explicitly preserve rankings within either group and it can only be ranked by trust-region loss for current total loss. We find that it is enough to show the gain over OLS in Figure 5. Furthermore, to build intuition on how Theorem 4.1 connects with the successor measure learning, we consider the binary-reward case (which is common in goal-conditioned RL) and further bound the sub-optimality gap by in the following Proposition and we leave extension to more general reward functions as future work.
Proposition (Binary rewards). Let , , with and be the soft-margin loss. If , then
We further show that soft-margin loss can emprically reduce successor-measure mismatch by conducting a controlled experiment on finite MDPs, where both can be computed exactly.
We evaluate 500 goal-reaching tasks across 20 stochastic grid environments and 25 goals. Lower soft-margin loss correlates with lower successor-measure mismatch, with a mean Spearman correlation of 0.280 increasing to 0.497 using Eq. (3).
We further evaluate BLS optimization directly. The normalized successor-measure mismatch decreases from 0.145 to 0.062, while the true discounted return improves from 16.30 to 17.15 (optimal: 17.19).
These results provide empirical evidence that the soft-margin objective is associated with reduced successor-measure mismatch, beyond improving reward-group separation.
[W3]The proposed method is limited to zero-shot RL algorithms, which extract policy representations only via OLS. For example, it is not clear how comparison with FB was made, since its extraction differs.
We clarify that our method is limited to zero-shot RL algorithms, but not limited to extracting policy representations only via OLS. It only requires the policies of the BFM to be indexed by task vectors through rewards of the form , which is the basic structure of BFM. We show how to compare with FB in Q1.
[Q1] For FB, please specify the relationship between ϕ and B, any covariance correction and coordinate conversion, and whether the reported FB baseline implements native FB inference. Following the FB–SF correspondence, we use as the feature map, where . As established by Section 4 in Touati et al.[1], OLS task inference reduces to native FB inference , so the covariance correction is already accounted for. Let denote the empirical FB inference operator and its Moore-Penrose pseudoinverse. In practice, we reconstruct rewards from as , using the corresponding least-squares inverse of native FB inference.
[1] Touati et al. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations (ICLR), 2023.
[W4, Q4] If OLS fails to extract good policy, BLS will not recover from this. How important is initialization from OLS for BLS to succeed? Some ablations would improve the argument.
We agree that the quality of the initialization affects BLS, but BLS can still recover when OLS fails. In Table 4, OLS collapses for large feature dimensions (e.g., 9.2% at d=500) while BLS initialized from the same OLS solution reaches 79.6%. To further study the role of the initialization, we extend Table 9 on HumEnv with a random initialization.
| Initialization | Before BLS | After BLS |
|---|---|---|
| Random | 0.075 ± 0.007 | 0.494 ± 0.014 |
| OLS | 0.496 ± 0.018 | 0.648 ± 0.022 |
| Weighted regression | 0.613 ± 0.038 | 0.700 ± 0.023 |
BLS improves performance from all three initializations, and the gain is larger when the initialization is weaker. Starting from a random task vector, BLS raises the expert-normalized return from 0.075 to 0.494. This indicates that the soft-margin objective alone recovers substantial task information. However, BLS from random initialization does not reach the performance of BLS from OLS or weighted regression, and the final performance increases with the quality of the initialization (0.494 → 0.648 → 0.700). This also shows that by preserving the reward ordering alone does not generally preserve optimal policies.
We thank the reviewer for the insightful feedback. We address the concerns below.
[W1, Q3] Does BLS outperform ridge regression, reward-weighted regression, and regression on thresholded rewards under matched conditions?
We compare BLS against five additional regression baselines:
Ridge Regression: Least squares with L2 regularization.
Reward-balanced LS (RBLS): Assigns equal total weights to high- and low-reward samples, with p=0.5.
Group-weighted LS (G-WLS): Assigns total weights p and 1−p to high- and low-reward samples, respectively.
Reward-weighted Regression (R-WLS): Weights samples proportionally to r−min(r)+ϵ, normalized to have mean weight one.
Thresholded Regression: Regresses onto binary reward targets indicating whether a sample belongs to the maximum-reward group.
We evaluate all five representations on the nine proprioceptive OGBench task groups, each containing five tasks, using three evaluation seeds (0, 1, 2) and 50 episodes per task and seed.
Laplacian
ICVF
HILP