- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper considers online RL with a reference base policy. It identifies issues with behavior cloning and support constraints, and proposes a geometric regularization term that imposes zero penalty within the base policy support and a distance-based penalty outside it, encouraging the policy to explore beyond the base policy support while preventing it from visiting regions where the critic has large error. To estimate the support, it uses a reference-action neighborhood method. Empirical results show that the proposed method outperforms behavior cloning and support constraint baselines.
Critical strengths
-
The paper is well-motivated and proposes a simple yet effective algorithm.
-
The empirical results look promising, and most of the claims are well-supported.
Critical weaknesses
-
The paper assumes that distance from the base-policy support is a useful proxy for critic unreliability. However, critic error also depends on local state-action coverage, and distance alone cannot determine critic error.
-
The proposed regularizer can be implicitly dependent on the action density. A dense and frequently sampled mode can determine , while other modes can be sparse and more spread out. In addition, estimating support with this method in high-dimensional systems can be difficult.
Questions for authors
-
Why is the LIBERO regularizer (17)-(19) different from the one proposed in Section 4.2?
-
How often does GSBR sample out-of-support actions after convergence (e.g., when the regularizer has non-zero values), and how does this compare with other methods? Reporting this can help explain whether GSBR achieves better performance because of frequently sampling these actions.
-
In Table 2, why does a larger have lower performance?
-
Figure 5 compares the estimated support with a 95% KDE region, which is itself an estimate of a high-probability region, and only visualizes two action dimensions. Could you provide a quantitative evaluation in the full action space, for example, the fraction of samples from the base policy that fall within/outside the estimated support region? How does this coverage change with action dimensionality?
6 comments
[2/3]
Frequency of final actor outside support
Thank you for the suggestion. We present how often the converged actor for GSBR and other methods (with the best hyperparameter value) predicts actions outside the estimated support region in the table below. We estimate the support using the GSBR method with N=8 and average over 128 observation samples from different episodes and 16 actor samples per observation.
Surprisingly, we see that GSBR deviates outside the support region the most, and the lower-performing methods generally stay within the support more. This does not suggest that all baseline methods are over-constrained to the base policy (since we sweep lower regularizer weights, which perform worse although it allows the policy to deviate more. Instead, these results indicate that GSBR offers more stable and structured exploration outside the base policy support, allowing the actor to achieve better performance while deviating out of the support.
| Task | Selected quantile | TD3+BC | RLT | Residual | BFN | DSRL | EXPO | DiceRL | GSBR |
|---|---|---|---|---|---|---|---|---|---|
| Can | 0.5 | 87.4 ± 2.1 | 91.0 ± 1.2 | 94.3 ± 1.2 | 49.6 ± 1.0 | 87.7 ± 1.3 | 88.7 ± 1.3 | 89.5 ± 1.3 | 99.7 ± 0.1 |
| Square | 0.9 | 45.1 ± 3.2 | 68.7 ± 2.8 | 81.5 ± 2.7 | 16.0 ± 0.8 | 58.5 ± 2.6 | 54.9 ± 2.6 | 44.5 ± 2.8 | 78.4 ± 2.7 |
| Transport-mh | 0.1 | 63.1 ± 2.7 | 91.0 ± 0.8 | 86.9 ± 0.6 | 81.0 ± 0.8 | 82.6 ± 1.4 | 81.0 ± 0.8 | 77.3 ± 0.9 | 70.0 ± 1.8 |
| Tool Hang | 0.5 | 83.0 ± 2.4 | 85.7 ± 1.4 | 75.6 ± 1.3 | 49.6 ± 1.0 | 72.2 ± 2.1 | 97.2 ± 0.8 | 91.0 ± 1.2 | 96.6 ± 0.8 |
| Cube 1 | 0.7 | 100.0 ± 0.0 | 100.0 ± 0.0 | 88.1 ± 1.7 | 39.4 ± 1.1 | 62.8 ± 2.7 | 91.8 ± 1.8 | 99.3 ± 0.5 | 100.0 ± 0.0 |
| Cube 2 | 0.7 | 91.4 ± 1.7 | 95.1 ± 1.2 | 84.0 ± 2.2 | 37.6 ± 1.2 | 59.5 ± 2.5 | 46.8 ± 1.5 | 97.7 ± 0.8 | 99.0 ± 0.4 |
| Cube 3 | 0.9 | 37.3 ± 3.7 | 56.0 ± 2.9 | 79.0 ± 2.5 | 20.0 ± 0.9 | 16.1 ± 2.0 | 57.3 ± 2.9 | 71.5 ± 2.7 | 63.7 ± 2.2 |
| Cube 4 | 0.9 | 80.1 ± 3.0 | 85.6 ± 2.3 | 81.9 ± 2.4 | 20.8 ± 1.0 | 43.0 ± 2.7 | 56.2 ± 2.8 | 66.3 ± 2.8 | 97.4 ± 0.7 |
| Cube 5 | 0.7 | 69.4 ± 0.7 | 60.8 ± 2.6 | 56.7 ± 1.1 | 37.9 ± 1.1 | 24.3 ± 2.4 | 74.5 ± 2.4 | 99.0 ± 0.4 | 55.3 ± 2.4 |
| Block 1 | 0.9 | 100.0 ± 0.0 | 100.0 ± 0.0 | 73.9 ± 3.4 | 23.7 ± 1.0 | 35.1 ± 2.5 | 77.5 ± 3.0 | 66.3 ± 2.8 | 100.0 ± 0.0 |
| Block 2 | 0.3 | 99.8 ± 0.2 | 99.2 ± 0.2 | 91.4 ± 1.2 | 67.9 ± 0.9 | 96.1 ± 0.6 | 92.8 ± 1.3 | 100.0 ± 0.0 | 99.3 ± 0.2 |
| Chamber 1 | 0.1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 98.9 ± 0.5 | 85.9 ± 0.5 | 96.8 ± 0.4 | 98.4 ± 0.6 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| Chamber 2 | 0.1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 98.2 ± 0.5 | 87.1 ± 0.5 | 95.9 ± 0.5 | 98.0 ± 0.5 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| Chamber 3 | 0.5 | 100.0 ± 0.0 | 100.0 ± 0.0 | 96.3 ± 1.1 | 62.6 ± 1.1 | 85.0 ± 1.4 | 98.0 ± 0.8 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| LIBERO 14 | 0.9 | 100.0 ± 0.0 | 100.0 ± 0.0 | 81.7 ± 2.7 | 69.9 ± 1.3 | 99.1 ± 0.3 | 92.1 ± 1.8 | 88.3 ± 1.6 | 100.0 ± 0.0 |
| LIBERO 29 | 0.1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| LIBERO 54 | 0.5 | 100.0 ± 0.0 | 100.0 ± 0.0 | 99.2 ± 0.2 | 98.6 ± 0.1 | 100.0 ± 0.0 | 99.4 ± 0.1 | 99.8 ± 0.1 | 100.0 ± 0.0 |
| LIBERO 60 | 0.9 | 100.0 ± 0.0 | 100.0 ± 0.0 | 89.2 ± 2.4 | 66.0 ± 1.1 | 92.5 ± 1.5 | 92.0 ± 1.8 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| LIBERO 64 | 0.1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| Average | — | 87.2 ± 0.4 | 91.2 ± 0.3 | 87.2 ± 0.4 | 58.6 ± 0.2 | 74.1 ± 0.4 | 84.0 ± 0.4 | 89.0 ± 0.3 | 92.6 ± 0.2 |
[3/3]
Quantitative metrics for support estimation
Thank you for this suggestion. We quantitatively evaluated the GSBR support region estimation coverage for all 19 tasks. For each observation, we construct the estimated support region from N=8 base-policy reference actions and different quantile values {0.1, 0.3, 0.5, 0.7, 0.9}. We then sample 256 base-policy actions and compute the fraction of them inside each estimated support region. We report the average over 128 observations from distinct episodes per task. We also include the action dimension for each task: H is the chunk length, d is the primitive action dimension, and H × d is the total dimension tested.
We find that for a fixed quantile value, the fraction of base policy samples inside the estimated support region is across different tasks and action dimensions is within 1.5%.
| Benchmark — tasks | H × d (total) | q=0.1 | q=0.3 | q=0.5 | q=0.7 | q=0.9 |
|---|---|---|---|---|---|---|
| RoboMimic - Can | 5 × 7 (35) | 22.44% | 39.71% | 53.32% | 68.30% | 84.43% |
| RoboMimic - Square | 5 × 7 (35) | 22.71% | 39.73% | 53.18% | 67.64% | 84.12% |
| RoboMimic - Transport | 5 × 14 (70) | 22.59% | 39.45% | 52.73% | 67.37% | 84.03% |
| RoboMimic - Tool Hang | 5 × 7 (35) | 22.20% | 39.65% | 52.98% | 67.67% | 84.45% |
| OGBench cube-triple-task{1-5} | 5 × 5 (25) | 22.15% | 39.27% | 52.39% | 67.62% | 84.51% |
| OCBench - block-double-task{1,2} | 3 × 7 (21) | 22.34% | 40.63% | 53.41% | 68.40% | 85.22% |
| OCBench - chamber-easy-task{1,2,3} | 3 × 7 (21) | 21.91% | 39.92% | 53.07% | 68.08% | 85.05% |
| LIBERO π0.5 | 10 × 7 (70) | 22.49% | 40.45% | 53.17% | 67.97% | 84.57% |
[1/3]
Overview
Thank you for acknowledging that our algorithm is “well-motivated” and “simple yet effective”. We address the weaknesses you posed and explain why GSBR is the right regularizer in the online RL setting. We also consolidated the regularizer implementation across LIBERO and the other state-only tasks. We present quantitative metrics for how well GSBR estimates the base policy support and how often the trained actors of different methods deviate from the support.
Critic error
We acknowledge that distance to the base policy support is not a proxy for critic unreliability. We believe that critic unreliability is a bigger issue in the offline RL setting since estimating Bellman backup targets on actions with high Q uncertainty can lead to Q explosion. However, since we are operating in the online RL setting, such Q uncertainty or overestimation can be resolved by collecting counterfactual data in the environment. Instead, the main role of the behavior regularizer we propose is to guide exploration to efficiently search for optimal actions.
Furthermore, a number of prior works have also observed that Euclidean distance can serve as a reasonable penalty for behavior regularization (RLT, TD3+BC, FQL, ReBRAC). Typically, such methods use distance to the base policy sample or mean, and GSBR improves over these prior approaches.
Choice of rB
We agree that rB depends on the action density, and a densely sampled mode can determine rB. In fact, this is exactly the desired behavior since sampling many nearby actions indicates that the support region around these actions is already well covered by the samples. Thus, the support radius around each sample should be small so as not to include too much area outside the support. While the concern about high-dimensional space is valid, our method performs well on tasks with up to 70-dimensional action space such as LIBERO and transport.
Regularizer for LIBERO
Sorry for the confusion. The regularizer for LIBERO was implemented to compute a different support radius for each action dimension, instead of a scalar radius as used for the other three state-based benchmarks. The per-action-dimension radii implementation wouldn’t suffer from the high-dimensionality concern discussed above, but new comparisons below show that scalar support radius performs approximately the same as per-action-dimension radii. Therefore, we have consolidated our implementation to use the scalar support radius for all tasks and updated the results in the manuscript.
Performance of larger N
Thank you for pointing this out. In the original Table 2 experiments, we independently swept q for N=16 and N=32, but reused the BC weights selected for N=8. To assess whether this disadvantaged larger N, we ran 300 additional experiments, extending each task’s BC-weight grid to {λ/3, λ, 3λ}, with q∈{0.1, 0.3, 0.5, 0.7, 0.9} and three seeds per setting. After selecting hyperparameters independently for each task and N, the five-task averages are:
All results use the 2M-step checkpoint and 50 evaluation episodes per seed. Throughput is normalized per task by the highest three-seed mean among N∈{8, 16, 32}; all rows use this shared normalization. Values are mean ± SEM across three seed averages, each averaged over the five tasks.
Retuning improves the success averages from 97.6 ± 0.0% to 98.0 ± 0.2% for N=16 and from 97.6 ± 0.5% to 98.0 ± 0.2% for N=32 and increases the normalized throughput of N=32 from 0.925 to 0.958, partially closing the gap. The remaining difference is small in success rate and is not monotonic in N. While we don’t have an explanation for why larger N performs slightly worse, these results suggest that the benefit of additional references largely saturates around N=8 on these tasks. We will include the retuned results in the revision.