- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses behavior-policy entropy to scale regularization in offline RL, allowing weaker constraints where the behavior policy is more stochastic. It derives bounds on regularized Bellman fixed points, a closed-form KL update, and convergence and sample-complexity results for tabular gated Q-learning. Experiments cover two tabular environments and a continuous-action version applied to CQL, TD3+BC, and ReBRAC on three D4RL medium datasets.
Critical strengths
The gate is simple to compute and depends only on the behavior policy. For KL regularization and a positive gate, Eq. (13) gives a closed-form policy update within the behavior policy's action support, without an additional inner optimization.
Table 1(a) gives useful evidence under heterogeneous behavior. At , the best mean reward among the tested fixed regularization coefficients is , compared with for the gated method; both occur at . The latter is close to the optimal reward of .
Critical weaknesses
-
Theorem 2.1 orders penalized values, , rather than the actual returns of the resulting policies. With nonnegative regularization, this ordering holds for any admissible gate and does not establish that entropy identifies where regularization should be relaxed. The return drop in Table 1(a) (Q1) shows why this distinction matters. The convergence results concern tabular fixed-point estimation under known-behavior and ergodicity assumptions, so they do not establish a return benefit from entropy-based gating either.
Two repairable statement issues remain. Definition 2.1 needs , which the proofs of Proposition 2.2 and Theorem 2.1 use. Theorem 3.2 must include the proof's conditions and (pp. 33 and 36).
-
The experiments do not separate state adaptation from a general reduction in regularization. The continuous experiments lack a smaller fixed-coefficient control. For Table 2, the experiment code constructs a deterministic using
argmax, thenpolicy_modificationmixes it with uniform noise at the same in every state. This gives identical entropy across states and a constant gate. In the linked ReBRAC implementation,compute_actor_lossmultiplies both the Q and BC terms by the same per-sample gate. This changes how samples contribute to training while preserving the relative coefficients of the two terms. Itscompute_targetleaves the critic's BC penalty ungated. The ReBRAC results therefore need clarification before they can support the proposed regularization mechanism.
Questions for authors
-
What explains the drop in Table 1(a), at , from for the regularized agent to for the gated agent? How should this qualify the claimed benefit?
-
On Hopper, how does the gate compare with a constant gate matched to its dataset mean and a tuned fixed regularization coefficient, under the same training and evaluation budget? Evidence that state dependence still helps, together with clarification of ReBRAC, would make me reconsider the weak rejection.
-
Does the released ReBRAC code correspond to the figures? Why gate both actor terms while leaving the critic penalty unchanged, and what happens when only the actor's BC term is gated?
-
Which set of Dirichlet concentrations generated Table 1: in Appendix C.1 or in the released
policy_modificationcode?
4 comments
Response (2/2)
Questions
- What explains the drop in Table 1(a), at , ...
This can be explained by zero initialization of the -values combined with an instance of not enough samples. When mixing the policy with a very small-concentration-parameter Dirichlet noise vector we can end up with a behavioral policy vector that has tiny action probabilities, e.g., . The gate returns a similar signal on a distribution with, e.g., or and while the signal itself is correct, the closed form greedy update step, that restricts learned policies to the support of the behavioral policy, masks out the actions with zero probability but does not mask out the actions with tiny probability, since they are in the support of the behavioral policy. In that setting a Q-value of zero, that has not received any updates due to the tiny sampling chance of such an action seems promising during a greedy update step and hence results in the wrongfully optimistic action. As the number of available samples increases, this issue will eventually disappear. Since the theoretic sample requirement is dependent on , such "epsilon-supported actions" result in expensive data requirements or the need of a higher regularization weight. That holds true in general since both, the gated as well as the regularized method suffer from this issue, however the gate suffers more from this when the state's behavioral policy also has high entropy. Since this instance of the sample-requirement failure mode is less obvious, we have extended the limitations section of the revised manuscript to contain this limitation and its explanation and a more detailed dissection in the Appendix's context of the table.
- On Hopper, how does the gate compare with a constant gate matched to its dataset mean and a tuned fixed regularization coefficient...
Since the revised gate (14) has unit batch mean, the constant gate matched to its dataset mean is the baseline. As smaller-coefficient controls, we have added a constant and a random gate with mean , approximately the mean of the original gate (Figures 5 and 6), and CQL with a halved conservative weight (Figures 7 and 8) to Appendix D.2, all with the same training and evaluation budget. For TD3+BC, both controls remain significantly below the gated variant, which indicates that the state dependence of the gate contributes to its improvement. For CQL, most of the gain is explained by the smaller coefficient, and for ReBRAC, both controls are worse than the baseline. Instead of a sweep, we matched the average coefficient of the original gate, since this directly tests whether its gains stem from a lower average regularization.
- Does the released ReBRAC code correspond to the figures...
Yes, the figures of the original submission were produced by the released code. Multiplying the -term of the actor loss by the gate was a bug, which we have fixed. Leaving the critic penalty ungated followed our integration into CQL and TD3+BC, where we gate the explicit regularization term of the loss and leave the bootstrapped target unchanged. In ReBRAC, however, the critic penalty is part of the explicit regularization and enters through the target. The revised ReBRAC variant therefore gates both the BC term of the actor and the critic penalty, and the released code has been updated.
The results of this variant replace the previous ones in Figure 1 (bottom row) and are contained in Figures 2-4 and Table 4. On Hopper, the gated variant reaches a lower final reward than the baseline, which already exceeds the D4RL expert reference score with small penalty weights. Appendix D.2 (paragraph “ReBRAC on Hopper”) analyzes this interaction between the gate and ReBRAC’s penalties, including the ablations in Figures 5 and 6 and a lower-bounded gate that recovers the final reward of the baseline (Figure 9).
Since the revised variant gates both the BC term of the actor and the critic penalty, we did not run a variant in which only the BC term of the actor is gated. We expect such a variant to perform worse, since the actor and the critic would then be regularized with different weights in each state and therefore disagree on how far the learned policy may deviate from the behavioral policy. The previous variant most likely performed well because multiplying the entire actor loss by the gate scales the per-sample gradient of the actor, which acts as a state-dependent damping of the actor update rather than as gated regularization.
- Which set of Dirichlet concentrations generated Table 1: in Appendix C.1 or in the released
policy_modificationcode?
Thank you for pointing out this typo. Table 1 was generated with for , as in the released code. Appendix D.1 has been corrected.
Thank you for the revisions. The theoretical clarifications and implementation fixes address several concerns. However, the random control has mean 0.5 while the revised gate has mean 1, so this comparison does not isolate the benefit of state dependence. The reported TD3+BC gains are encouraging, but CQL’s gains largely come from a smaller regularization coefficient, and corrected ReBRAC performs worse than the baseline on Hopper. I will maintain my current score.
We thank you for your response and for acknowledging the clarifications, and especially for pointing out the bug in the ReBRAC implementation.
The random control with mean , approximately the mean of the originally submitted gate, was designed to test whether the gains result from a lower average regularization weight. For completeness, we have now also added a random control with mean , as Figure 19 on the last page of the revised manuscript for ease of access.
Response (1/2)
We thank you for the time spent evaluating our work. Below, we address your comments.
While fixing the bug you pointed out in Q3, we noticed that the continuous gate did not account for actions saturating at the boundaries of MuJoCo's [−1,1] action range. We have updated the gate accordingly and, in the same step, normalized it by its batch mean, such that it is independent of the scale of the behavioral standard deviations and only redistributes the regularization weight of the respective algorithm across states (Equations (13) and (14)). We have rerun all continuous experiments with the updated gate. In the updated manuscript, changes are marked in red, and the new Appendices C.1 and E contain additional material referenced in our responses.
Weaknesses
The comments to the weaknesses will address those points that are not covered by answers to the questions below.
We agree with both distinctions: an ordering of penalized fixed points does not order the returns of the corresponding policies, and a result that holds for any admissible gate is not specific to the entropy gate.
To address the first, we have added Proposition C.1 (Appendix C.1), which bounds the return of the gated policy. The gated policy never performs worse than the behavioral policy and retains at least a fraction λ∈[0,1] of the improvement of the regularized policy over the behavioral policy, where λ depends only on the smallest and the largest value of the gate. If the gate attains the same value below one in every state, λ=1 and VπR≤VπΦ is guaranteed. For a gate whose value differs between states it is not, since the penalties of the two policies accumulate under different state distributions. The return drop in Table 1(a) is therefore not excluded by our results (see our response to Q1). That these results hold for any admissible gate is intended. They establish that the regularization can be relaxed, wherever and however strongly, without the return falling below Vb. They do not identify where it should be relaxed.
Thank you for pointing that out. Both points are resolved in the revised manuscript. Definition 2.1 now maps into R+∪{+∞}, and Theorem 3.2 now states cα≥324e, which corresponds to cα≥27ec0c1 with c0=12/c1 and implies cα≥11.
The assessment of the uniform noise tables is correct. They serve as a consistency check since they show the expected effect of the regularization-weight and gate-level combinations and thereby illustrate the shift from behavioral cloning to off-policy learning.