acceptodds
Under review as a conference paper at ICLR 2027

GSBR: GEOMETRIC SUPPORT BEHAVIOR REGULARIZATION FOR ONLINE RL

Abstract

Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies - typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets - capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy’s support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance yet still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method.

Primary area:
reinforcement learning
Keywords:
Online RL, Behavior Constraint, Flow RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fUS7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers online RL with a reference base policy. It identifies issues with behavior cloning and support constraints, and proposes a geometric regularization term that imposes zero penalty within the base policy support and a distance-based penalty outside it, encouraging the policy to explore beyond the base policy support while preventing it from visiting regions where the critic has large error. To estimate the support, it uses a reference-action neighborhood method. Empirical results show that the proposed method outperforms behavior cloning and support constraint baselines.

Critical strengths
  1. The paper is well-motivated and proposes a simple yet effective algorithm.

  2. The empirical results look promising, and most of the claims are well-supported.

Critical weaknesses
  1. The paper assumes that distance from the base-policy support is a useful proxy for critic unreliability. However, critic error also depends on local state-action coverage, and distance alone cannot determine critic error.

  2. The proposed regularizer can be implicitly dependent on the action density. A dense and frequently sampled mode can determine rBr_\mathcal{B}, while other modes can be sparse and more spread out. In addition, estimating support with this method in high-dimensional systems can be difficult.

Questions for authors
  1. Why is the LIBERO regularizer (17)-(19) different from the one proposed in Section 4.2?

  2. How often does GSBR sample out-of-support actions after convergence (e.g., when the regularizer has non-zero values), and how does this compare with other methods? Reporting this can help explain whether GSBR achieves better performance because of frequently sampling these actions.

  3. In Table 2, why does a larger NN have lower performance?

  4. Figure 5 compares the estimated support with a 95% KDE region, which is itself an estimate of a high-probability region, and only visualizes two action dimensions. Could you provide a quantitative evaluation in the full action space, for example, the fraction of samples from the base policy that fall within/outside the estimated support region? How does this coverage change with action dimensionality?

6 comments

[1/3]

Overview

Thank you for acknowledging that our algorithm is “well-motivated” and “simple yet effective”. We address the weaknesses you posed and explain why GSBR is the right regularizer in the online RL setting. We also consolidated the regularizer implementation across LIBERO and the other state-only tasks. We present quantitative metrics for how well GSBR estimates the base policy support and how often the trained actors of different methods deviate from the support.

Critic error

We acknowledge that distance to the base policy support is not a proxy for critic unreliability. We believe that critic unreliability is a bigger issue in the offline RL setting since estimating Bellman backup targets on actions with high Q uncertainty can lead to Q explosion. However, since we are operating in the online RL setting, such Q uncertainty or overestimation can be resolved by collecting counterfactual data in the environment. Instead, the main role of the behavior regularizer we propose is to guide exploration to efficiently search for optimal actions.

Furthermore, a number of prior works have also observed that Euclidean distance can serve as a reasonable penalty for behavior regularization (RLT, TD3+BC, FQL, ReBRAC). Typically, such methods use distance to the base policy sample or mean, and GSBR improves over these prior approaches.

Choice of rBr_{\mathcal{B}}

We agree that rBr_{\mathcal{B}} depends on the action density, and a densely sampled mode can determine rBr_{\mathcal{B}}. In fact, this is exactly the desired behavior since sampling many nearby actions indicates that the support region around these actions is already well covered by the samples. Thus, the support radius around each sample should be small so as not to include too much area outside the support. While the concern about high-dimensional space is valid, our method performs well on tasks with up to 70-dimensional action space such as LIBERO and transport.

Regularizer for LIBERO

Sorry for the confusion. The regularizer for LIBERO was implemented to compute a different support radius for each action dimension, instead of a scalar radius as used for the other three state-based benchmarks. The per-action-dimension radii implementation wouldn’t suffer from the high-dimensionality concern discussed above, but new comparisons below show that scalar support radius performs approximately the same as per-action-dimension radii. Therefore, we have consolidated our implementation to use the scalar support radius for all tasks and updated the results in the manuscript.

TaskScalar radius — Success rate (%)Scalar radius — Throughput (Normalized by best method per task)Per-action-dimension radii — Success rate (%)Per-action-dimension radii — Throughput (Normalized by best method per task)
LIBERO task 1480.7 ± 2.474.14 ± 5.8986.7 ± 2.491.42 ± 7.70
LIBERO task 2960.0 ± 5.863.40 ± 10.3760.7 ± 6.463.86 ± 10.71
LIBERO task 5496.0 ± 1.298.83 ± 3.6794.7 ± 2.494.79 ± 8.59
LIBERO task 6098.7 ± 1.393.74 ± 5.3698.0 ± 1.288.92 ± 5.44
LIBERO task 6484.7 ± 3.788.90 ± 8.2984.7 ± 2.490.94 ± 4.35
Average84.0 ± 1.283.80 ± 2.4984.9 ± 1.385.99 ± 3.07

Performance of larger N

Thank you for pointing this out. In the original Table 2 experiments, we independently swept q for N=16 and N=32, but reused the BC weights selected for N=8. To assess whether this disadvantaged larger N, we ran 300 additional experiments, extending each task’s BC-weight grid to {λ/3, λ, 3λ}, with q∈{0.1, 0.3, 0.5, 0.7, 0.9} and three seeds per setting. After selecting hyperparameters independently for each task and N, the five-task averages are:

NSuccess rate (%)Normalized throughput
8 (ours)98.7 ± 0.51.00 ± 0.05
1698.0 ± 0.20.91 ± 0.01
3298.0 ± 0.20.96 ± 0.02

All results use the 2M-step checkpoint and 50 evaluation episodes per seed. Throughput is normalized per task by the highest three-seed mean among N∈{8, 16, 32}; all rows use this shared normalization. Values are mean ± SEM across three seed averages, each averaged over the five tasks.

Retuning improves the success averages from 97.6 ± 0.0% to 98.0 ± 0.2% for N=16 and from 97.6 ± 0.5% to 98.0 ± 0.2% for N=32 and increases the normalized throughput of N=32 from 0.925 to 0.958, partially closing the gap. The remaining difference is small in success rate and is not monotonic in N. While we don’t have an explanation for why larger N performs slightly worse, these results suggest that the benefit of additional references largely saturates around N=8 on these tasks. We will include the retuned results in the revision.

[2/3]

Frequency of final actor outside support

Thank you for the suggestion. We present how often the converged actor for GSBR and other methods (with the best hyperparameter value) predicts actions outside the estimated support region in the table below. We estimate the support using the GSBR method with N=8 and average over 128 observation samples from different episodes and 16 actor samples per observation.

Surprisingly, we see that GSBR deviates outside the support region the most, and the lower-performing methods generally stay within the support more. This does not suggest that all baseline methods are over-constrained to the base policy (since we sweep lower regularizer weights, which perform worse although it allows the policy to deviate more. Instead, these results indicate that GSBR offers more stable and structured exploration outside the base policy support, allowing the actor to achieve better performance while deviating out of the support.

TaskSelected quantileTD3+BCRLTResidualBFNDSRLEXPODiceRLGSBR
Can0.587.4 ± 2.191.0 ± 1.294.3 ± 1.249.6 ± 1.087.7 ± 1.388.7 ± 1.389.5 ± 1.399.7 ± 0.1
Square0.945.1 ± 3.268.7 ± 2.881.5 ± 2.716.0 ± 0.858.5 ± 2.654.9 ± 2.644.5 ± 2.878.4 ± 2.7
Transport-mh0.163.1 ± 2.791.0 ± 0.886.9 ± 0.681.0 ± 0.882.6 ± 1.481.0 ± 0.877.3 ± 0.970.0 ± 1.8
Tool Hang0.583.0 ± 2.485.7 ± 1.475.6 ± 1.349.6 ± 1.072.2 ± 2.197.2 ± 0.891.0 ± 1.296.6 ± 0.8
Cube 10.7100.0 ± 0.0100.0 ± 0.088.1 ± 1.739.4 ± 1.162.8 ± 2.791.8 ± 1.899.3 ± 0.5100.0 ± 0.0
Cube 20.791.4 ± 1.795.1 ± 1.284.0 ± 2.237.6 ± 1.259.5 ± 2.546.8 ± 1.597.7 ± 0.899.0 ± 0.4
Cube 30.937.3 ± 3.756.0 ± 2.979.0 ± 2.520.0 ± 0.916.1 ± 2.057.3 ± 2.971.5 ± 2.763.7 ± 2.2
Cube 40.980.1 ± 3.085.6 ± 2.381.9 ± 2.420.8 ± 1.043.0 ± 2.756.2 ± 2.866.3 ± 2.897.4 ± 0.7
Cube 50.769.4 ± 0.760.8 ± 2.656.7 ± 1.137.9 ± 1.124.3 ± 2.474.5 ± 2.499.0 ± 0.455.3 ± 2.4
Block 10.9100.0 ± 0.0100.0 ± 0.073.9 ± 3.423.7 ± 1.035.1 ± 2.577.5 ± 3.066.3 ± 2.8100.0 ± 0.0
Block 20.399.8 ± 0.299.2 ± 0.291.4 ± 1.267.9 ± 0.996.1 ± 0.692.8 ± 1.3100.0 ± 0.099.3 ± 0.2
Chamber 10.1100.0 ± 0.0100.0 ± 0.098.9 ± 0.585.9 ± 0.596.8 ± 0.498.4 ± 0.6100.0 ± 0.0100.0 ± 0.0
Chamber 20.1100.0 ± 0.0100.0 ± 0.098.2 ± 0.587.1 ± 0.595.9 ± 0.598.0 ± 0.5100.0 ± 0.0100.0 ± 0.0
Chamber 30.5100.0 ± 0.0100.0 ± 0.096.3 ± 1.162.6 ± 1.185.0 ± 1.498.0 ± 0.8100.0 ± 0.0100.0 ± 0.0
LIBERO 140.9100.0 ± 0.0100.0 ± 0.081.7 ± 2.769.9 ± 1.399.1 ± 0.392.1 ± 1.888.3 ± 1.6100.0 ± 0.0
LIBERO 290.1100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
LIBERO 540.5100.0 ± 0.0100.0 ± 0.099.2 ± 0.298.6 ± 0.1100.0 ± 0.099.4 ± 0.199.8 ± 0.1100.0 ± 0.0
LIBERO 600.9100.0 ± 0.0100.0 ± 0.089.2 ± 2.466.0 ± 1.192.5 ± 1.592.0 ± 1.8100.0 ± 0.0100.0 ± 0.0
LIBERO 640.1100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Average—87.2 ± 0.491.2 ± 0.387.2 ± 0.458.6 ± 0.274.1 ± 0.484.0 ± 0.489.0 ± 0.392.6 ± 0.2

[3/3]

Quantitative metrics for support estimation

Thank you for this suggestion. We quantitatively evaluated the GSBR support region estimation coverage for all 19 tasks. For each observation, we construct the estimated support region from N=8 base-policy reference actions and different quantile values {0.1, 0.3, 0.5, 0.7, 0.9}. We then sample 256 base-policy actions and compute the fraction of them inside each estimated support region. We report the average over 128 observations from distinct episodes per task. We also include the action dimension for each task: H is the chunk length, d is the primitive action dimension, and H × d is the total dimension tested.

We find that for a fixed quantile value, the fraction of base policy samples inside the estimated support region is across different tasks and action dimensions is within 1.5%.

Benchmark — tasksH × d (total)q=0.1q=0.3q=0.5q=0.7q=0.9
RoboMimic - Can5 × 7 (35)22.44%39.71%53.32%68.30%84.43%
RoboMimic - Square5 × 7 (35)22.71%39.73%53.18%67.64%84.12%
RoboMimic - Transport5 × 14 (70)22.59%39.45%52.73%67.37%84.03%
RoboMimic - Tool Hang5 × 7 (35)22.20%39.65%52.98%67.67%84.45%
OGBench cube-triple-task{1-5}5 × 5 (25)22.15%39.27%52.39%67.62%84.51%
OCBench - block-double-task{1,2}3 × 7 (21)22.34%40.63%53.41%68.40%85.22%
OCBench - chamber-easy-task{1,2,3}3 × 7 (21)21.91%39.92%53.07%68.08%85.05%
LIBERO π0.510 × 7 (70)22.49%40.45%53.17%67.97%84.57%
Reviewer_ysmaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GSBR (Geometric Support Behavior Regularization), a behavior regularization algorithm for online RL fine-tuning with a frozen base policy. The core issue is how to design a regularization term when using a pre-trained robot foundation model (VLA/flow policy/BC policy) as a behavioral prior, one that lets the actor discard bad action patterns (mode selection) while enabling local refinements near the base policy's support. The answer of GSBR is: use limited sampling of the base policy to estimate its action support set. The experiment covers 4 benchmarks and 19 tasks (RoboMimic, OGBench, OCBench, LIBERO). GSBR improves the average success rate from 33% of the baseline strategy to 95%, achieving the best performance in both success rate and throughput.

Critical strengths
  • The considered problem, i.e., fine-tuning diffusion policies, is important in the community.

  • Theorem 4.1 gives a pointwise bound, so approximating the support by a union of sample balls is more than a heuristic.

  • The experiments show the effectiveness of GSBR.

Critical weaknesses
  • The actor is designed with the action buffer condition B\mathcal{B}. So, during inference time, the method requires sampling N reference actions from the base policy, which increases computational cost and can delay robot operation. So I'm curious: with the same training loss, why not directly fine-tune the actor π(⋅∣s)\pi(\cdot|s) rather than π(⋅∣s,B)\pi(\cdot|s, \mathcal{B})?

  • If there are some real-world robot experiments showing that the time delay during inference will not affect a lot, this work will be more solid.

  • Table 6 shows that GSBR chooses different hyperparameters for different tasks. For a fair comparison (I guess that the hyperparameters of baselines are not cherry-picked), it is better to choose the same hyperparameters for different tasks and show whether GSBR really outperforms baselines.

  • Lack of discussion of some works on RL for diffusion policies, such as [1-3].

Reference:

[1] Efficient online reinforcement learning for diffusion policy

[2] Exploratory diffusion model for unsupervised reinforcement learning

[3] Diffusion-based reinforcement learning via q-weighted variational policy optimization

Questions for authors

See weaknesses above

4 comments

[1/3]

Overview

Thank you for your valuable feedback. We present a table of inference latency below to show that sampling multiple base policy actions at test time has negligible effect. We also discuss the reasoning for conditioning the actor on the base policy samples and present results without passthrough during test-time and train-time. Regarding hyperparameter value selection, we present our hypothesis for why different tasks need different values and reiterate fair comparison with prior methods as we tune each method per task. We appreciate the additional related work and updated the manuscript.

Inference Latency

We measured reference-action sampling speed on an RTX 5090 with 1 and 8 samples on each of the four benchmarks and found that sampling 8 base policy actions adds negligible latency at test-time using basic optimization.

For the base policy with MLP networks on the three state-based benchmarks, we apply batched inference by repeating the observation 8 times, supplying IID-sampled initial noises, and denoising them together to get 8 action samples at once. On these MLP networks, sampling 8 actions adds less than 0.5ms of latency over a single sample. For π0.5 base policy on LIBERO, we compute the KV cache on the input observation once, then call batched inference with 8 IID noise samples to obtain 8 base policy actions. Using this simple optimization, sampling 8 actions during inference time adds approximately 9ms (+15%) of latency. In real-world deployment where inference latency is critical, standard asynchronous inference methods such as RTC[1] could be used to further reduce latency.

The table below shows the average test-time base policy inference latency averaged over 100 trials.

Base policyN=1 (ms)N=8 (ms)Added (ms)
RoboMimic - Can0.5600.896+0.336
RoboMimic - Square0.5510.877+0.326
RoboMimic - Transport0.5980.981+0.383
RoboMimic - Tool Hang0.5851.032+0.447
OGBench cube-triple-task{1-5}0.5810.939+0.358
OCBench - block-double-task{1,2}0.8531.198+0.346
OCBench - chamber-easy-task{1,2,3}0.8450.774−0.071
LIBERO π0.561.02070.005+8.985

[2/3]

Why condition the actor on base policy samples

We follow the design choice in RLT[2] to condition the actor on the base policy actions. This provides a stronger prior for the actor, allowing it to produce the full base policy distribution without re-learning it in the network parameters. We present two new ablation results on reference action conditioning to show that conditioning provides a meaningful boost in performance. The variants are:

  • A. Our standard recipe trains the actor with conditioning (and 50% dropout for the state-only tasks) and deploys with conditioning.

  • B. Keeps the same training-time conditioning as A, but evaluates the policy without conditioning.

  • C. No base policy action conditioning during both train- and test-time.

Our results show that reference action conditioning is helpful overall, achieving a higher success rate (A=95%, B=87%, and C=79%) and 20% higher normalized throughput compared to B and C. On certain tasks, such as tool-hang-mh and chamber-easy-task1, removing conditioning at only test-time or both train- and test-time has little effect on performance.

Success Rate (%)

TaskA. Ours: Train and test conditioningB. Only Train ConditioningC. No conditioning
cube-triple-task1100.0 ± 0.0100.0 ± 0.099.3 ± 0.7
cube-triple-task299.3 ± 0.796.7 ± 2.493.3 ± 1.3
cube-triple-task3100.0 ± 0.0100.0 ± 0.099.3 ± 0.7
cube-triple-task498.7 ± 1.394.7 ± 1.890.0 ± 3.1
cube-triple-task595.3 ± 1.394.0 ± 3.597.3 ± 0.7
Can98.0 ± 1.296.0 ± 2.394.0 ± 1.2
Square99.3 ± 0.796.0 ± 2.097.3 ± 0.7
Transport96.0 ± 1.287.3 ± 1.888.0 ± 2.3
Tool Hang98.0 ± 0.098.0 ± 1.298.0 ± 1.2
block-double-task197.3 ± 1.392.7 ± 0.795.3 ± 2.7
block-double-task298.7 ± 0.788.7 ± 0.794.7 ± 1.8
chamber-easy-task1100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
chamber-easy-task298.0 ± 1.296.7 ± 0.796.0 ± 1.2
chamber-easy-task3100.0 ± 0.099.3 ± 0.796.7 ± 1.3
LIBERO task 1486.7 ± 2.456.7 ± 3.70.7 ± 0.7
LIBERO task 2960.7 ± 6.434.7 ± 1.832.0 ± 1.2
LIBERO task 5494.7 ± 2.454.7 ± 10.92.7 ± 1.3
LIBERO task 6098.0 ± 1.292.0 ± 1.298.0 ± 1.2
LIBERO task 6484.7 ± 2.472.0 ± 6.035.3 ± 4.1
Overall94.9 ± 0.486.84 ± 0.9479.37 ± 0.43

Throughput (normalized by best variant per task)

TaskA. Ours: Train and test conditioningB. Only Train ConditioningC. No conditioning
cube-triple-task199.97 ± 0.54100.00 ± 0.3281.12 ± 14.75
cube-triple-task2100.00 ± 8.5485.06 ± 16.4062.63 ± 8.85
cube-triple-task3100.00 ± 5.6293.50 ± 1.8285.71 ± 3.71
cube-triple-task4100.00 ± 16.5970.88 ± 10.7646.30 ± 7.66
cube-triple-task597.41 ± 8.5194.68 ± 18.61100.00 ± 3.00
Can100.00 ± 3.9492.64 ± 7.1187.43 ± 4.09
Square100.00 ± 2.5386.13 ± 7.1088.56 ± 3.72
Transport100.00 ± 2.4079.84 ± 2.8679.16 ± 4.57
Tool Hang98.98 ± 1.78100.00 ± 7.3398.67 ± 5.05
block-double-task1100.00 ± 3.7887.39 ± 0.4792.76 ± 7.80
block-double-task2100.00 ± 6.6158.20 ± 3.2479.67 ± 8.96
chamber-easy-task199.44 ± 0.78100.00 ± 2.2494.13 ± 4.51
chamber-easy-task2100.00 ± 9.2293.58 ± 7.7683.93 ± 3.81
chamber-easy-task395.80 ± 5.94100.00 ± 5.5280.82 ± 6.00
LIBERO task 14100.00 ± 8.4237.68 ± 4.960.26 ± 0.26
LIBERO task 29100.00 ± 16.7747.42 ± 3.4041.48 ± 1.94
LIBERO task 54100.00 ± 9.0630.36 ± 11.080.84 ± 0.42
LIBERO task 60100.00 ± 6.1283.09 ± 3.2899.99 ± 3.18
LIBERO task 64100.00 ± 4.7879.24 ± 9.7529.72 ± 3.98
Overall99.56 ± 1.6879.98 ± 2.5970.17 ± 0.79

[3/3]

Hyperparameter selection

Thanks for raising the point about the hyperparameter selection. You are right that we chose different hyperparameters for GSBR on different tasks. We hypothesize that the choice of behavior regularizer weight depends on many factors such as the performance of the base policy, the reward/Q-value scale, and action dimensionality. For a fair comparison, we also extensively sweep the hyperparameter values with the same resolution for all other methods on each task individually and report the best one.

Additional related work

Thank you for bringing to our attention additional related work on RL for flow policies. We have incorporated these into the manuscript.

[1] Real-Time Execution of Action Chunking Flow Policies.

[2] RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Dear Reviewer ysma, we have addressed all of your feedback with additional experiments and detailed reporting. Please let us know if you have additional questions before the discussion deadline, and kindly consider raising your score.

Reviewer_oL5HOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Geometric Support Behavior Regularization (GSBR) for online reinforcement learning with a frozen pretrained base policy. GSBR approximates the base policy’s support using neighborhoods around sampled reference actions, with a radius determined by nearest-neighbor distances. It applies no penalty within these neighborhoods and a squared-distance penalty outside them, allowing mode selection and local refinement. The paper provides approximation guarantees and evaluates the method on 19 simulated robotic tasks across four benchmarks, reporting improved aggregate success rate and task throughput over the compared baselines.

Critical strengths
  • Simple and interpretable regularization. The zero-penalty region allows the actor to favor useful behaviors without matching their original frequencies, while the finite penalty permits departures from the prior. The method requires only sample access to the base policy, making it applicable without access to its parameters or gradients.

  • Encouraging empirical results. The evaluation covers both state-based policies and a pretrained VLA, with comparisons against seven baselines. GSBR achieves strong aggregate performance, and the zero-radius ablation provides direct evidence that allowing unpenalized neighborhoods around reference actions is beneficial.

Critical weaknesses
  • The paper motivates GSBR as a way to leverage pretrained policy priors for sample-efficient online RL. However, the new actor begins online learning without imitation- or distillation-based pretraining to inherit the base policy’s initial capabilities. This creates tension with the motivation of exploiting existing capabilities and may incur avoidable interaction costs. A comparison with an actor pretrained to imitate the base policy would help justify this initialization choice.

  • As online experience accumulates, the policy may discover well-supported, high-return actions beyond the base policy’s behavior. However, GSBR maintains a fixed regularization weight against the frozen base policy, potentially limiting further improvement. The authors should evaluate a decaying regularization weight and compare fixed, gradually reduced, and late-stage removal of regularization in terms of stability and final performance.

Questions for authors
  • Q1. Computational overhead from multiple reference actions. GSBR requires multiple reference actions from the base policy, potentially introducing additional sampling and computation compared with methods using a single reference action. How much does this increase training time and deployment latency?

  • Q2. egularization decay Could decaying the regularization weight, or removing the regularizer late in training, improve final performance while maintaining stability?

  • Q3.Placement of the regularizer Could the geometric penalty be incorporated into the critic’s TD updates as a negative auxiliary reward, rather than applied only to the actor objective?

  • Q4. Uncertainty reporting Several SEM values in Table 1 round to zero. Could the authors report greater numerical precision and per-task, per-seed results to clarify variability and the reliability of performance differences?

6 comments

[1/3]

Overview

Thank you for your feedback. We have conducted additional experiments to address the weaknesses around network initialization and regularizer weight decay. While these ideas could be helpful in principle, we did not find either to improve performance in our extensive experiments with two different network initialization strategies and two regularizer decay strategies. For simplicity and performance, we retain our original method’s implementation with random initialization and constant regularizer weight. We also answer the four questions for the authors below.

Network initialization

As requested, we extensively experimented with two network initialization strategies on the five OGBench cube-triple tasks, and found both methods performed worse in terms of both success rate and throughput compared to the random network initialization used in GSBR (ours).

  • Actor init: As requested by the reviewer, the first method is actor-pretraining with distillation loss to imitate the base policy. The actor network is pretrained for 100k steps with a pure BC distillation objective. For each batch sampled from the replay buffer, we sample reference actions from the base policy, condition the actor on the sampled reference actions, and train the actor network with L2 regression towards the reference actions. After the 100k distillation steps, we start RL training with the original GSBR algorithm.

  • Actor + critic init: The second method is WSRL-style initialization, where we warm up the actor with distillation loss and also warm up the critic on the frozen actor. We follow the actor initialization scheme in the previous method for the first 100k steps, then freeze the actor and update the critic using the frozen actor for another 100k steps. After a total of 200k warm-up steps, we unfreeze both networks and proceed with GSBR training.

We follow the same protocol as previous experiments in the paper and run three seeds and evaluate over 50 episodes. We summarize the average performance in the table below. While this experiment does not conclusively prove that no better initialization method exists, it does indicate that random initialization is a strong choice for its simplicity and performance. We defer further improvements to the network initialization strategy to future work, since it can be explored independently from the choice of behavior regularizer.

Success Rate

TaskGSBR (ours)Actor initActor + critic init
cube-triple-task1100.0 ± 0.0100.0 ± 0.01.3 ± 1.3
cube-triple-task299.3 ± 0.798.7 ± 0.724.0 ± 24.0
cube-triple-task3100.0 ± 0.098.7 ± 0.799.3 ± 0.7
cube-triple-task498.7 ± 1.392.7 ± 1.823.3 ± 4.1
cube-triple-task595.3 ± 1.338.0 ± 29.58.0 ± 8.0
Average98.7 ± 0.585.6 ± 5.631.2 ± 4.8

Throughput (successes/hour)

TaskGSBR (ours)Actor initActor + critic init
cube-triple-task11582.74 ± 8.531569.14 ± 23.940.50 ± 0.50
cube-triple-task2328.89 ± 28.09309.52 ± 11.4421.71 ± 21.71
cube-triple-task3261.97 ± 14.71235.28 ± 4.23207.94 ± 10.53
cube-triple-task4344.38 ± 57.15190.05 ± 18.9810.34 ± 2.12
cube-triple-task5174.25 ± 15.2259.18 ± 55.463.49 ± 3.49
Average538.44472.6348.80

Regularizer decay

We applied two regularizer weight decay schemes suggested by the reviewer and found both to deteriorate performance compared to a fixed regularizer weight on the five OGBench cube-triple tasks.

  • Linear decay: As requested by the reviewer, we first apply a linear decay schedule. The regularizer weight starts at 0 steps with the previous best weight on each task, and linearly decays to 0 over the course of 2M online steps.

  • Late-stage removal: We use the same fixed regularizer weight as the previous best on each task for the first 1.6M steps. In the final 400k steps, the regularizer is removed (weight set to 0).

We report average performance over three seeds and 50 eval episodes per seed below. These comparisons justify our original choice of fixed regularizer weight since it is more performant and requires no tuning.

Success Rate

TaskGSBR (ours)Linear DecayLate-stage removal
cube-triple-task1100.0 ± 0.099.3 ± 0.7100.0 ± 0.0
cube-triple-task299.3 ± 0.748.0 ± 24.042.7 ± 21.8
cube-triple-task3100.0 ± 0.049.3 ± 9.462.0 ± 8.1
cube-triple-task498.7 ± 1.340.7 ± 4.154.7 ± 10.5
cube-triple-task595.3 ± 1.31.3 ± 1.325.3 ± 4.4
Average98.7 ± 0.547.7 ± 6.756.9 ± 1.5

[2/3]

Throughput (successes/hour)

TaskGSBR (ours)Linear DecayLate-stage removal
cube-triple-task11582.74 ± 8.531301.32 ± 237.981573.51 ± 8.21
cube-triple-task2328.89 ± 28.0944.58 ± 22.6636.44 ± 20.12
cube-triple-task3261.97 ± 14.7128.62 ± 7.9048.23 ± 11.15
cube-triple-task4344.38 ± 57.1522.92 ± 3.6745.56 ± 18.39
cube-triple-task5174.25 ± 15.220.50 ± 0.5011.93 ± 2.74
Average538.44279.59343.13

Q1: Compute overhead

We measured and reported reference-action sampling speed on an RTX 5090 with 1 and 8 samples on each task. At test-time, sampling 8 base policy actions adds negligible latency using basic optimization. At training-time, sampling 8 actions increases time per gradient step by 20-130%, but this can be optimized by caching a set of base policy samples as we did for LIBERO experiments.

Test-time latency: For the base policy with MLP networks on the three state-based benchmarks, we apply batched inference by repeating the observation 8 times, supplying IID-sampled initial noises, and denoising them together to get 8 action samples at once. On these MLP networks, sampling 8 actions adds less than 0.5ms of latency over a single sample. For π0.5 base policy on LIBERO, we compute the KV cache on the input observation once, then call batched inference with 8 IID noise samples to obtain 8 base policy actions. Using this simple optimization, sampling 8 actions during inference time adds approximately 9ms (+15%) of latency. The table below shows the average test-time base policy inference latency averaged over 100 calls.

Base policyN=1 (ms)N=8 (ms)Added (ms)
RoboMimic - Can0.5600.896+0.336
RoboMimic - Square0.5510.877+0.326
RoboMimic - Transport0.5980.981+0.383
RoboMimic - Tool Hang0.5851.032+0.447
OGBench cube-triple-task{1-5}0.5810.939+0.358
OCBench - block-double-task{1,2}0.8531.198+0.346
OCBench - chamber-easy-task{1,2,3}0.8450.774−0.071
LIBERO π0.561.02070.005+8.985

Training speed: We also report the training time per gradient step on an RTX 5090 with batch size 256 for each of the environments. Since training already applies batched inference over different states, sampling 8 actions does increase compute and slows down training by 20-130% depending on the size of the base policy networks. Luckily, we can optimize this by caching a set of reference actions for each state (as we did on LIBERO) to remove the need for repeated base policy sampling during each gradient step, which still produces good performance with negligible increase in training time.

WorkloadN=1 (ms)N=8 (ms)Increase
RoboMimic Can5.136.2722.2%
RoboMimic Square5.206.2820.6%
RoboMimic Transport5.296.4221.3%
RoboMimic Tool Hang5.186.2520.6%
OGBench cube-triple-task{1-5}5.176.1118.3%
OCBench - block-double-task{1,2}6.8814.46110.3%
OCBench - chamber-easy-task{1,2,3}6.4115.05134.7%
LIBERO π0.53.934.022.3%

Q2: Regularization decay

As shown above, we did not find gradual regularizer decay or late-stage removal to improve performance.

Q3: Placement of the regularizer

In principle, the geometric penalty could also be incorporated into the critic’s TD target as a negative auxiliary reward. A related precedent is ReBRAC, which includes a behavior penalty in the bootstrapped target while retaining actor regularization, with independently controlled strengths. However, this is a distinct design choice since incorporating the penalty into TD updates propagates future geometric penalties through the value function, whereas actor-only regularization directly constrains policy improvement while retaining a critic trained on task rewards. Our formulation follows the latter approach, consistent with the actor-only placement of behavior regularization in TD3+BC and RL Token.

Q4: Uncertainty reporting

Thank you for raising the question about numerical precision. We have updated Table 1 in the manuscript to include higher numerical precision and attached a copy below. We also present the per-seed and per-task performance for each method. For space reasons, please refer to the Google Sheet linked below. Per-seed and per-task results

[3/3]

Updated Table 1. Success rate with higher numerical precision.

MethodRoboMimicOGBenchOCBenchLIBEROAll tasks
TD3+BC95.5 ± 1.872.5 ± 1.398.4 ± 0.454.3 ± 2.279.4 ± 1.2
RLT95.7 ± 1.690.0 ± 5.497.7 ± 0.385.5 ± 1.692.0 ± 1.2
Residual Policy94.2 ± 1.768.3 ± 9.695.1 ± 0.378.0 ± 1.383.3 ± 1.8
BFN86.2 ± 1.768.8 ± 5.188.1 ± 1.565.5 ± 1.176.7 ± 0.6
DSRL96.5 ± 0.667.1 ± 5.186.8 ± 1.881.1 ± 1.982.1 ± 1.7
EXPO96.5 ± 0.579.5 ± 0.997.5 ± 0.582.4 ± 2.388.6 ± 0.5
DiceRL94.7 ± 1.439.5 ± 0.697.2 ± 0.673.7 ± 1.375.3 ± 0.6
GSBR (ours)97.8 ± 0.298.7 ± 0.598.8 ± 0.485.1 ± 1.694.9 ± 0.4

Comments on the paper

1 comment
AuthorsOpenReview ↗

Cover note on rebuttal response

Cover Note

Dear Area Chair and reviewers, we thank everyone for the thoughtful feedback. We summarize the additional experiments and clarifications point by point here. We have addressed all comments from the reviewers, and we kindly ask you to consider raising the score for our submission.

Responses to reviewer concerns

  • Reviewer oL5H — Actor initialization: We tested actor distillation on the five OGBench cube-triple tasks with three seeds each. We separately tested actor-plus-critic warm-up. Random initialization retained the strongest aggregate final performance among these variants.

  • Reviewer oL5H — Regularization schedule: We tested gradual regularization decay. We separately tested late-stage removal. A fixed regularization weight retained the strongest aggregate final performance among the tested schedules.

  • Reviewer ysma — Reference-action conditioning: We evaluated removing reference-action conditioning at test time. We separately evaluated removing conditioning during both training and testing. The standard conditioning recipe performed best overall.

  • Reviewers oL5H and ysma — Inference latency: We measured the cost of sampling multiple reference actions at inference time, which adds negligible delay on most tasks with batched inference.

  • Reviewer oL5H — Training cost: We measured the training-time overhead of multiple reference actions. We explain how reference-action caching reduces repeated sampling.

  • Reviewer fUS7 — Critic reliability: We clarify that distance from the base-policy support does not by itself determine critic error. We explain GSBR’s role in guiding exploration during online learning.

  • Reviewer oL5H — Regularizer placement: We explain the choice of actor-only regularization. We discuss how adding the penalty to the critic’s TD target would change the learning objective.

  • Reviewer ysma — Hyperparameter fairness: We clarify that each method was tuned separately per task with the same hyperparameter search density for fair comparison. We also explain why suitable regularization strengths can vary across tasks.

  • Reviewer fUS7 — LIBERO formulation: We compared scalar-radius and per-dimension regularization on LIBERO. Their aggregate success rates were similar on the shared tuning grid. We unified the implementation around a scalar support radius and updated the results in the manuscript.

  • Reviewer fUS7 — Number of reference actions: We independently retuned the N=16 and N=32 configurations, and found them to slightly underperform N=8.

  • Reviewer fUS7 — Support-estimation coverage: We quantified the fraction of base-policy actions within the estimated support in the full action space across all 19 tasks. We report how coverage varies with the radius quantile. We also compare coverage across action dimensions.

  • Reviewer fUS7 — Trained-policy support violations: We found that the best trained GSBR actor predicts actions outside the support the most compared to other methods at their best regularizer strength. This suggests that the geometry of the GSBR regularizer allows more structured exploration outside the support and yields better performance.

Changes to the paper

  • Revise motivation: Revise the main motivation for the distance-based regularizer to structured exploration outside the base policy support, where no priors exist.

  • LIBERO results: Unified the scalar-radius formulation across benchmarks. Updated the LIBERO results in Figure 4 and Table 4.

  • Uncertainty reporting: Increased numerical precision in Tables 1 and 3.

  • Related work: Added the suggested work on reinforcement learning for diffusion policies.

  • Larger reference sets: Updated the N=16 and N=32 results with independently retuned hyperparameters.

  • Train and inference compute cost: Added inference-latency and training-speed measurements to the appendix.

  • Base policy support coverage analysis: Added quantitative full-action-space support-coverage analysis.

  • Trained actor out-of-support analysis: Added out of support analysis comparing GSBR with other methods.

  • Reference action conditioning: Added the test-time and train-time base policy conditioning-removal ablation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.