acceptodds
Under review as a conference paper at ICLR 2027

Gated Regularization for Offline Reinforcement Learning

Abstract

This paper proposes a regularization method that carefully balances greedy policy optimization with the risks of out-of-distribution actions in offline reinforcement learning. The method introduces a simple and computationally efficient module, called the gate, which adaptively scales the strength of classical behavioral policy regularization based on the entropy of the behavioral policy. The behavioral policy's entropy serves as a proxy for how well the dynamics of a given state are represented in the available offline dataset: high entropy (i.e., more uniform behavior) permits aggressive off-policy updates, while low entropy enforces imitation-style conservatism. We show that gated regularization yields a value function that is provably sandwiched between those of classical regularized and optimal policies, while preserving the computational benefits of standard regularization methods. Furthermore, we prove that the resulting gated Q-learning algorithm converges almost surely, with non-asymptotic sample complexity no worse than that of classical Q-learning. Empirical experiments validate the theoretical findings and demonstrate gated regularization as a simple yet useful tool for achieving effective offline reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Q-learning, Regularized Markov Decision Process

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YrY5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses behavior-policy entropy to scale regularization in offline RL, allowing weaker constraints where the behavior policy is more stochastic. It derives bounds on regularized Bellman fixed points, a closed-form KL update, and convergence and sample-complexity results for tabular gated Q-learning. Experiments cover two tabular environments and a continuous-action version applied to CQL, TD3+BC, and ReBRAC on three D4RL medium datasets.

Critical strengths

The gate is simple to compute and depends only on the behavior policy. For KL regularization and a positive gate, Eq. (13) gives a closed-form policy update within the behavior policy's action support, without an additional inner optimization.

Table 1(a) gives useful evidence under heterogeneous behavior. At κ=0.8\kappa=0.8, the best mean reward among the tested fixed regularization coefficients is −19.05-19.05, compared with −13.93-13.93 for the gated method; both occur at β=2\beta=2. The latter is close to the optimal reward of −13-13.

Critical weaknesses
  1. Theorem 2.1 orders penalized values, VR≤VΦ≤V⋆V^{\mathcal R}\leq V^\Phi\leq V^\star, rather than the actual returns of the resulting policies. With nonnegative regularization, this ordering holds for any admissible gate and does not establish that entropy identifies where regularization should be relaxed. The return drop in Table 1(a) (Q1) shows why this distinction matters. The convergence results concern tabular fixed-point estimation under known-behavior and ergodicity assumptions, so they do not establish a return benefit from entropy-based gating either.

    Two repairable statement issues remain. Definition 2.1 needs R≥0\mathcal R\geq0, which the proofs of Proposition 2.2 and Theorem 2.1 use. Theorem 3.2 must include the proof's conditions cα≥11c_\alpha\geq11 and cα≥27ec0c1c_\alpha\geq27e c_0c_1 (pp. 33 and 36).

  2. The experiments do not separate state adaptation from a general reduction in regularization. The continuous experiments lack a smaller fixed-coefficient control. For Table 2, the experiment code constructs a deterministic π⋆\pi^\star using argmax, then policy_modification mixes it with uniform noise at the same κ\kappa in every state. This gives identical entropy across states and a constant gate. In the linked ReBRAC implementation, compute_actor_loss multiplies both the Q and BC terms by the same per-sample gate. This changes how samples contribute to training while preserving the relative coefficients of the two terms. Its compute_target leaves the critic's BC penalty ungated. The ReBRAC results therefore need clarification before they can support the proposed regularization mechanism.

Questions for authors
  1. What explains the drop in Table 1(a), at β=1,κ=0.8\beta=1,\kappa=0.8, from −32.32-32.32 for the regularized agent to −100.00-100.00 for the gated agent? How should this qualify the claimed benefit?

  2. On Hopper, how does the gate compare with a constant gate matched to its dataset mean and a tuned fixed regularization coefficient, under the same training and evaluation budget? Evidence that state dependence still helps, together with clarification of ReBRAC, would make me reconsider the weak rejection.

  3. Does the released ReBRAC code correspond to the figures? Why gate both actor terms while leaving the critic penalty unchanged, and what happens when only the actor's BC term is gated?

  4. Which set of Dirichlet concentrations generated Table 1: {10i:i=−3,…,3}\{10^i:i=-3,\ldots,3\} in Appendix C.1 or {10i:i=−2,…,4}\{10^i:i=-2,\ldots,4\} in the released policy_modification code?

4 comments

Response (1/2)

We thank you for the time spent evaluating our work. Below, we address your comments.

While fixing the bug you pointed out in Q3, we noticed that the continuous gate did not account for actions saturating at the boundaries of MuJoCo's [−1,1][-1,1] action range. We have updated the gate accordingly and, in the same step, normalized it by its batch mean, such that it is independent of the scale of the behavioral standard deviations and only redistributes the regularization weight of the respective algorithm across states (Equations (13) and (14)). We have rerun all continuous experiments with the updated gate. In the updated manuscript, changes are marked in red, and the new Appendices C.1 and E contain additional material referenced in our responses.

Weaknesses

The comments to the weaknesses will address those points that are not covered by answers to the questions below.

  1. Theorem 2.1 orders penalized values, VR≤VΦ≤V⋆V^{\mathcal{R}}\leq V^{\Phi} \leq V^{\star}...

We agree with both distinctions: an ordering of penalized fixed points does not order the returns of the corresponding policies, and a result that holds for any admissible gate is not specific to the entropy gate.

To address the first, we have added Proposition C.1 (Appendix C.1), which bounds the return of the gated policy. The gated policy never performs worse than the behavioral policy and retains at least a fraction λ∈[0,1]\lambda\in[0,1] of the improvement of the regularized policy over the behavioral policy, where λ\lambda depends only on the smallest and the largest value of the gate. If the gate attains the same value below one in every state, λ=1\lambda=1 and VπR≤VπΦV^{\pi^{\mathcal{R}}}\leq V^{\pi^{\Phi}} is guaranteed. For a gate whose value differs between states it is not, since the penalties of the two policies accumulate under different state distributions. The return drop in Table 1(a) is therefore not excluded by our results (see our response to Q1). That these results hold for any admissible gate is intended. They establish that the regularization can be relaxed, wherever and however strongly, without the return falling below VbV^{b}. They do not identify where it should be relaxed.

  1. Two repairable statement issues remain. Definition 2.1 needs R≤0\mathcal{R}\leq0, which the proofs of Proposition 2.2 and Theorem 2.1 use. Theorem 3.2 must include the proof's conditions cα≥11c_{\alpha}\geq 11 and cα≥26ec0c1c_{\alpha}\geq26e c_{0} c_{1} (pp. 33 and 36).

Thank you for pointing that out. Both points are resolved in the revised manuscript. Definition 2.1 now maps into R+∪{+∞}\mathbb{R}_{+} \cup \lbrace +\infty \rbrace, and Theorem 3.2 now states cα≥324ec_\alpha \geq 324e, which corresponds to cα≥27ec0c1c_\alpha \geq 27ec_0c_1 with c0=12/c1c_0 = 12/c_1 and implies cα≥11c_\alpha \geq 11.

  1. The experiments do not separate state adaptation from a general reduction in regularization...

The assessment of the uniform noise tables is correct. They serve as a consistency check since they show the expected effect of the regularization-weight and gate-level combinations and thereby illustrate the shift from behavioral cloning to off-policy learning.

Response (2/2)

Questions

  1. What explains the drop in Table 1(a), at β=1\beta=1, κ=0.8\kappa=0.8...

This can be explained by zero initialization of the QQ-values combined with an instance of not enough samples. When mixing the policy with a very small-concentration-parameter Dirichlet noise vector we can end up with a behavioral policy vector that has tiny action probabilities, e.g., 10−810^{-8}. The gate returns a similar signal on a distribution with, e.g., [0.8,0.2,0,0][0.8, 0.2, 0, 0] or [0.8−ϵ−ε,0.2,ϵ,ε][0.8-\epsilon-\varepsilon, 0.2, \epsilon, \varepsilon] and while the signal itself is correct, the closed form greedy update step, that restricts learned policies to the support of the behavioral policy, masks out the actions with zero probability but does not mask out the actions with tiny probability, since they are in the support of the behavioral policy. In that setting a Q-value of zero, that has not received any updates due to the tiny sampling chance of such an action seems promising during a greedy update step and hence results in the wrongfully optimistic action. As the number of available samples increases, this issue will eventually disappear. Since the theoretic sample requirement is dependent on min⁡s,aηπb(s,a)\min_{s,a}\eta_{\pi^{b}}(s,a), such "epsilon-supported actions" result in expensive data requirements or the need of a higher regularization weight. That holds true in general since both, the gated as well as the regularized method suffer from this issue, however the gate suffers more from this when the state's behavioral policy also has high entropy. Since this instance of the sample-requirement failure mode is less obvious, we have extended the limitations section of the revised manuscript to contain this limitation and its explanation and a more detailed dissection in the Appendix's context of the table.

  1. On Hopper, how does the gate compare with a constant gate matched to its dataset mean and a tuned fixed regularization coefficient...

Since the revised gate (14) has unit batch mean, the constant gate matched to its dataset mean is the baseline. As smaller-coefficient controls, we have added a constant and a random gate with mean 0.50.5, approximately the mean of the original gate (Figures 5 and 6), and CQL with a halved conservative weight (Figures 7 and 8) to Appendix D.2, all with the same training and evaluation budget. For TD3+BC, both controls remain significantly below the gated variant, which indicates that the state dependence of the gate contributes to its improvement. For CQL, most of the gain is explained by the smaller coefficient, and for ReBRAC, both controls are worse than the baseline. Instead of a sweep, we matched the average coefficient of the original gate, since this directly tests whether its gains stem from a lower average regularization.

  1. Does the released ReBRAC code correspond to the figures...

Yes, the figures of the original submission were produced by the released code. Multiplying the QQ-term of the actor loss by the gate was a bug, which we have fixed. Leaving the critic penalty ungated followed our integration into CQL and TD3+BC, where we gate the explicit regularization term of the loss and leave the bootstrapped target unchanged. In ReBRAC, however, the critic penalty is part of the explicit regularization and enters through the target. The revised ReBRAC variant therefore gates both the BC term of the actor and the critic penalty, and the released code has been updated.

The results of this variant replace the previous ones in Figure 1 (bottom row) and are contained in Figures 2-4 and Table 4. On Hopper, the gated variant reaches a lower final reward than the baseline, which already exceeds the D4RL expert reference score with small penalty weights. Appendix D.2 (paragraph “ReBRAC on Hopper”) analyzes this interaction between the gate and ReBRAC’s penalties, including the ablations in Figures 5 and 6 and a lower-bounded gate that recovers the final reward of the baseline (Figure 9).

Since the revised variant gates both the BC term of the actor and the critic penalty, we did not run a variant in which only the BC term of the actor is gated. We expect such a variant to perform worse, since the actor and the critic would then be regularized with different weights in each state and therefore disagree on how far the learned policy may deviate from the behavioral policy. The previous variant most likely performed well because multiplying the entire actor loss by the gate scales the per-sample gradient of the actor, which acts as a state-dependent damping of the actor update rather than as gated regularization.

  1. Which set of Dirichlet concentrations generated Table 1: 10i:i=−3,…,310^{i}: i=-3, \dots, 3 in Appendix C.1 or 10i:i=−2,…,410^{i}:i=-2,\dots,4 in the released policy_modification code?

Thank you for pointing out this typo. Table 1 was generated with 10i10^{i} for i=−2,…,4i=-2,\dots,4, as in the released code. Appendix D.1 has been corrected.

Thank you for the revisions. The theoretical clarifications and implementation fixes address several concerns. However, the random control has mean 0.5 while the revised gate has mean 1, so this comparison does not isolate the benefit of state dependence. The reported TD3+BC gains are encouraging, but CQL’s gains largely come from a smaller regularization coefficient, and corrected ReBRAC performs worse than the baseline on Hopper. I will maintain my current score.

We thank you for your response and for acknowledging the clarifications, and especially for pointing out the bug in the ReBRAC implementation.

The random control with mean 0.50.5, approximately the mean of the originally submitted gate, was designed to test whether the gains result from a lower average regularization weight. For completeness, we have now also added a random control with mean 11, as Figure 19 on the last page of the revised manuscript for ease of access.

Reviewer_jxnGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a gate function for behavior-regularized offline reinforcement learning. The gate weakens the regularization in states where the behavior policy is stochastic, measured by its entropy in the tabular case and by its standard deviation for Gaussian policies in the continuous case. The intuition is that in such states the learned policy can deviate further from the behavior policy to improve the value. Theoretically, the paper shows that the fixed point of the gated operator lies between those of the regularized and the optimal operators, and that gated Q-learning converges with a sample complexity matching that of classical Q-learning. Empirically, the gate is added to CQL, TD3+BC, and ReBRAC on D4RL medium datasets, using the behavior policy recovered from the dataset metadata.

Critical strengths

The paper theoretically shows the fixed point of behavior-regularized offline RL with gated operator and is clearly presented. The paper demonstrates its efficacy in the Finite domain experiment. The method is simple to implement when provided with prior knowledge other than transitions within the data.

Critical weaknesses

Limited improvement compared to baseline: In the Hopper and HalfCheetah domain experiments, even if CQL and TD3+BC experienced improvement, the improvements were visible on offline RL baselines with sub-optimal performances. ReBRAC did not improve the final performance from using the gate. In this situation, I think the authors should present at least one of the two: 1. If ReBRAC’s performance has already saturated in the Hopper domain, the authors should test ReBRAC on a more complex domain and show that ReBRAC + Gate can outperform the original ReBRAC. 2. Or analyze the difference between the sub-optimal offline RL approaches (CQL and TD3+BC) and ReBRAC, and how the gate approach alleviated the sub-optimality.

The cost of obtaining the gate: Computing the gate requires privileged information on the behavior policy. In the MuJoCo domain, the standard deviation of the behavior policy was used. However, the gate was easier to compute based on the assumption that the behavior policy is Gaussian, which is not guaranteed in general offline RL settings in larger domains. If such privileged information is not available, computing the gate introduces additional complexity. Even if the increased complexity may not be significant, the approximation error could affect the performance of the approach.

Gate difference across states: It would have been better if the authors showed which states require less regularization and how such weakened regularization contributes to the actual behavior of the offline RL policy. In the discrete domain, the effect of the gate is straightforward to understand, since the entropy of the behavior policy directly reflects how actions are distributed at each state. In the continuous domain, however, it is not clear how the state-dependent standard deviation translates into the learned policy’s behavior. For example, one should demonstrate: “Without the gate, the learned policy at state s remained as stochastic as the behavior policy; after applying the gate, the behavior at that state shifted closer to the optimal policy while staying in the coverage of the data distribution.”

Questions for authors

Q1. In the MuJoCo domain, it seems the final offline RL performance has been saturated. This means that without distinguishing the stochasticity of the behavior policy, a well-performing offline RL policy is obtainable. When would the gate be critical to the improvement of the offline RL policy? Would it be when the offline data has more noisy actions, or in domains larger than MuJoCo?

Q2. In the continuous domain, it is fundamentally difficult to exploit the stochasticity of the behavior policy. Since the state space is continuous, the same state rarely appears twice in the dataset; states can be close to each other, but they are different states. Each state is therefore observed with a single action, and the stochasticity of the behavior policy at that state cannot be identified from the data without additional assumptions. Even if the behavior policy follows a Gaussian distribution, its state-dependent standard deviation cannot be recovered per state from the transitions alone. In this sense, the information on the behavior policy taken from the dataset metadata is privileged. How does the method perform when the behavior policy is estimated from the data rather than given?

Q3. The improvement in CQL and TD3+BC with the gate would have originated from the states where the behavior policy is more stochastic. Could you show how the actions shifted before and after applying the gate to clearly demonstrate the impact of the gate?

4 comments

Response (1/2)

We thank you for the time spent evaluating our work. Below, we address your comments.

While fixing a bug pointed out by Reviewer YrY5, we noticed that the continuous gate did not account for actions saturating at the boundaries of MuJoCo's [−1,1][-1,1] action range. We have updated the gate accordingly and, in the same step, normalized it by its batch mean, such that it is independent of the scale of the behavioral standard deviations and only redistributes the regularization weight of the respective algorithm across states (Equations (13) and (14)). We have rerun all continuous experiments with the updated gate. In the updated manuscript, changes are marked in red, and the new Appendices C.1 and E contain additional material referenced in our responses.

Weaknesses

The comments to the weaknesses will address those points that are not covered by answers to the questions below.

  1. Limited improvement compared to baseline...

We agree that the continuous-action experiments do not show a consistent improvement in final performance. As stated in Section 4, they are intended as an outlook. The theoretical guarantees are derived for the tabular setting, and the experiments test whether the gate can be ported to continuous-action methods.

Regarding your first suggestion, ReBRAC on Hopper has saturated with a normalized score of about 102102, and on Ant and HalfCheetah, ReBRAC and its gated variant reach comparable final rewards (Table 4). Regarding your second suggestion, the gate only rescales the explicit regularization term, so its benefit depends on how strongly this term limits policy improvement (Section 5). In ReBRAC, the penalty weights are small and its critic design adds further conservatism. For TD3+BC, the gate yields the largest improvements, which the ablations of Appendix D.2 attribute to the state-dependent profile rather than to a lower average regularization. Consistent with this, the improvement is large on Hopper and HalfCheetah, where the effective behavioral standard deviation varies strongly across states, and much smaller on Ant, where the data are more homogeneous (Table 3). Section 4 and Appendix D.2 discuss ReBRAC on Hopper in detail, and our response to Q3 shows how the gate changes the learned actions.

We do not claim that the gate is required in every setting. Just as off-policy methods suffice when a dataset covers the state-action space exhaustively, the gate is a cheap addition to the toolbox whose benefit depends on the structure of the data and on the algorithm it is integrated into.

  1. The cost of obtaining the gate...

While the gate was constructed for Gaussian behavioral policies, as is common for continuous-action policies, knowing, assuming, or estimating properties of the distribution underlying the data is a common requirement across offline and inverse reinforcement learning and not specific to the gate. The continuous gate is one instance of the concept, which only requires a gate based on the entropy of the behavioral policy. For other policy classes, a different gate would have to be constructed accordingly. The effect of an estimated behavioral policy is discussed in our response to Q2.

  1. Gate difference across states...

The response to this is merged into the answer to Q3.

Response (2/2)

Questions

Q1. In the MuJoCo domain, it seems the final offline RL performance has been saturated...

The gate matters most when the behavioral policy is spatially inhomogeneous, i.e., when data coverage differs across regions of the state space. Consider a driver who has explored every route in some districts of a city but always takes the same road in others. In the former, the data supports greedy deviation from the behavioral policy, and the gate relaxes the regularization. In the latter, only one action is covered, and the gate keeps the learned policy close to the behavioral policy. A fixed weight is either too conservative in the first case or too optimistic in the second. The deciding factor is therefore this heterogeneity, not the size of the domain. Noisier actions help only if the noise level varies across states. With uniform noise, the gate is constant, and gated regularization reduces to classical regularization. This is consistent with the MuJoCo results, where the improvement of TD3+BC is much smaller on Ant, whose behavioral standard deviation varies least across states (Table 3).

Q2. In the continuous domain, it is fundamentally difficult to exploit the stochasticity of the behavior policy...

We argue that neither the statement that the behavioral policy is always available nor that it is never available holds in general. In settings such as autonomous driving, robotics, or medicine, it can be recorded or annotated, whereas in other settings this may not be possible. We have not evaluated the gate with an estimated behavioral policy, since the estimation quality would then affect the results in ways that are outside the scope of establishing the behavioral policy’s entropy as a proxy for coverage. Section 5 discusses how an estimated behavioral policy may be a safer regularization target, but this assumes an estimate of good quality. While out of scope for this work, analyzing the effect of the estimation quality on the gate is an interesting question for future research.

Q3. The improvement in CQL and TD3+BC with the gate would have originated from the states where the behavior policy is more stochastic...

Appendix E.2 of the revised manuscript compares the final policies with and without the gate at states with high and low gate values (Figures 13-18 and Table 5). For all algorithms and environments, the gated variant deviates further from the behavioral mean than its baseline in low-gate states and stays closer to it in high-gate states. Table 5 additionally reports the fraction of actions inside the interval containing approximately 95% of the behavioral probability mass. As you point out, the connection between the gate and the behavior of the learned policy is less direct in the continuous setting. The gate mainly shifts the mean of the learned policy, while its spread changes considerably less. For TD3+BC and ReBRAC, this is expected, since both build on deterministic actor-critic methods and therefore only learn the mean of the action distribution. For CQL, which learns a stochastic policy, the spread is also affected considerably less than the mean.

Thank you for the detailed response and the new analysis.

Appendix E.2 addresses my question on how the gate changes the learned policy, and I acknowledge the theoretical results in the tabular setting. However, their significance depends on whether the gate is applicable and meaningful in continuous domains, and my concerns there remain.

First, the response attributes the benefit of the gate to heterogeneity of the behavioral stochasticity across states. If this is the deciding factor, an experiment that directly controls this heterogeneity, for example datasets whose noise level varies across regions of the state space, would be needed to support the claim; the comparison across Hopper, HalfCheetah, and Ant is only indirect evidence.

Second, in continuous domains each state is typically observed with a single action, so the empirical behavioral policy is deterministic at every state. Assuming the form of the behavioral distribution, such as a Gaussian, is common in offline RL, but assuming that its state-dependent standard deviation is given is not. Any nonzero estimate of this quantity would come from the smoothness of the function approximator rather than from the data, and this is precisely the quantity on which the gate relies. Without an evaluation using an estimated behavioral policy, it remains unclear whether the gate is practical beyond settings where the behavioral policy is recorded.

I will keep my score.

We thank you for your response and for acknowledging the analysis in Appendix E.2.

However, we respectfully disagree that the significance of the results depends on their applicability in continuous domains. As with primarily theoretical results in general, we see their significance in the insights they provide for the setting in which they are derived, rather than in their transfer to another setting. The continuous experiments merely illustrate one way in which the gate could be integrated into established continuous-action algorithms.

Similarly, the gate applies in settings where the behavioral policy, or its entropy, is recorded or can be estimated. With an estimate, the performance of the gate also depends on the estimator, and the interaction between the two is a separate question from whether the entropy of the behavioral policy is a useful signal for regularization.

Reviewer_b7yxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The main risk in offline RL is out-of-distribution actions, and a common remedy is to regularize the policy toward the behavior policy πb\pi_b. The regularization strength β\beta is, however, global: too conservative where data coverage is good, too optimistic where it is poor. The paper proposes using the entropy of the behavior policy as a proxy for how well the dynamics after a given state are covered by the offline data, and scales β\beta accordingly per state.

Concretely, a gate Φ:Δ(A)→[0,1]\Phi:\Delta(\mathcal A)\to[0,1] is defined to be 0 at the uniform distribution and 1 at a point mass, instantiated as Φ(πs)=1−H(πs)/log⁡∣A∣\Phi(\pi_s)=1-H(\pi_s)/\log|\mathcal A|, and the regularized operator becomes (TΦ,πV)(s)=(TπV)(s)−β,Φ(πsb),R(πs,πsb),(TΦV)(s)=max⁡π(TΦ,πV)(s).(T_{\Phi,\pi}V)(s)=(T^{\pi}V)(s)-\beta,\Phi(\pi_s^{b}),R(\pi_s,\pi_s^{b}),\qquad (T_\Phi V)(s)=\max_{\pi}(T_{\Phi,\pi}V)(s).

The gate depends only on πb\pi_b (hence decoupled from learning) and is a single scalar in front of the penalty. The authors prove: (i) the fixed points are sandwiched, VR(s)≤VΦ(s)≤V⋆(s)V^{R}(s)\le V^{\Phi}(s)\le V^{\star}(s); (ii) gated Q-learning converges almost surely to QΦQ^{\Phi} under Robbins–Monro stepsizes; (iii) with a piecewise-constant schedule, the non-asymptotic sample complexity matches tabular Q-learning. For R=KLR=\mathrm{KL} the policy maximization is available in closed form, πˉ(a∣s)∝ez(s,a)πb(a∣s)\bar\pi(a\mid s)\propto e^{z(s,a)}\pi_b(a\mid s) with z=(Q−max⁡a′Q)/(βΦ(πsb))z=(Q-\max_{a'}Q)/(\beta\Phi(\pi_s^b)), so the maximizer automatically lies in the support of πb\pi_b. Experiments cover tabular CliffWalking/Taxi (validating Theorem 2.1) and MuJoCo Hopper/Ant/HalfCheetah (inserting the gate into CQL, TD3+BC and ReBRAC).

Critical strengths
  1. The gate is clean, nearly free, and decoupled from learning. Its coefficient depends only on πb\pi_b and does not evolve with training, so it can be reused as a single scalar in front of any explicit regularizer. Under the KL divergence it admits a closed form, πˉ∝ezπb\bar\pi\propto e^{z}\pi_b, making each update as cheap as standard Q-learning and—for free—restricting the maximizer to supp(πb)\mathrm{supp}(\pi_b), a property that usually requires dedicated machinery. This is the most valuable engineering observation in the paper.
  2. The theory is modest but carefully assembled. Monotonicity/distributivity/contraction of the gated operator, almost-sure convergence, and a non-asymptotic rate are all stated with explicit assumptions and proved; Appendix A restates the cited results of Geist, Tsitsiklis and Li et al. in a single notation, which makes verification straightforward and is good scholarly practice.
Critical weaknesses

The mechanism amounts to a state-dependent β\beta, and the decisive control experiments are missing. What actually enters the algorithm is the product βs=β,Φ(πsb)\beta_s=\beta,\Phi(\pi_s^{b}): for any c>0c>0, (β,Φ)(\beta,\Phi) and (cβ,Φ/c)(c\beta,\Phi/c) are equivalent state by state; with homogeneous data Φ≈\Phi\approx constant and the gate degenerates to lowering β\beta. Whether the gate helps therefore hinges entirely on two empirical premises—that entropy predicts coverage, and that the data are inhomogeneous—and neither is tested directly. There is no comparison against an ungated variant with per-dataset tuned β\beta at a matched budget, no gate-shape ablation (constant cc, Φp\Phi^{p}, linear-in-entropy, mean-matched random profile), and no plot relating Φ(πsb)\Phi(\pi_s^b) to measured coverage (visit counts or density). The empirical base is also thin: three MuJoCo environments, D4RL medium only, mean curves without seed variance or significance, and the authors themselves describe effects as varying strongly across algorithms/environments with many cases of "no significant change". Finally, when πb\pi_b is unavailable it must be estimated (the policy object stored in D4RL need not be the data-collecting policy), and the resulting error propagates directly into the coefficient without being measured; the continuous gate also introduces a new parameter ρ\rho, so the "no additional hyperparameters" claim should be qualified as "no additional uncertainty-specific hyperparameters".

Questions for authors
  1. Could you compare against an ungated variant with per-dataset tuned β\beta under an equal tuning budget? If the advantage disappears, the gain is merely a smaller effective β\beta.
  2. Could you ablate the gate shape—constant Φ≡c\Phi\equiv c, Φp\Phi^{p}, a linear-in-entropy version, and a random profile matched in mean? If "constant gate + rescaled β\beta" performs the same, the entropy profile itself contributes nothing.
  3. Could you report the correlation between Φ(πsb)\Phi(\pi_s^{b}) and measured coverage (state-action visit counts or density estimates)? This is the core assumption behind "entropy as a coverage proxy".
4 comments

Response (1/2)

We thank you for the time spent evaluating our work. Below, we address your comments.

While fixing a bug pointed out by Reviewer YrY5, we noticed that the continuous gate did not account for actions saturating at the boundaries of MuJoCo's [−1,1][-1,1] action range. We have updated the gate accordingly and, in the same step, normalized it by its batch mean, such that it is independent of the scale of the behavioral standard deviations and only redistributes the regularization weight of the respective algorithm across states (Equations (13) and (14)). We have rerun all continuous experiments with the updated gate. In the updated manuscript, changes are marked in red, and the new Appendices C.1 and E contain additional material referenced in our responses.

Weaknesses

The comments to the weaknesses will address those points that are not covered by answers to the questions below.

The mechanism amounts to a state-dependent...

  • The scale equivalence, the comparison with a tuned β\beta, the gate-shape ablations, and the relation to coverage are addressed in our responses to Q1–Q3.
  • The learning curves report standard-error bands across 55 training seeds, where each seed is first averaged over 1010 evaluation episodes per checkpoint, so that within-seed rollout variance is not conflated with seed variance. The revision additionally reports the final rewards with standard errors in Table 4.
  • When πb\pi^{b} is not available it is required to be estimated, that is correct, however we argue that, in general, neither statement "the behavioral policy is always available" nor "the behavioral policy is never available" holds. In some settings such as autonomous driving, robotics, or medicine it can be recorded or annotated whereas in other settings this might not be possible. Because of that the limitation is indeed real, but does not apply homogeneously across all possible settings.
  • The manuscript does not claim that the gate introduces no additional hyperparameters (the contributions state no additional computational overhead), but we agree that ρ\rho is one. We selected ρ\rho based on the analytical shape of the resulting gate exp⁡(−x/ρ)\exp(-x/\rho) rather than by tuning it on any algorithm or environment, and use ρ=0.5\rho=0.5 throughout the revision. Since the effective standard deviation (13) and the gate (14) are normalized by their batch means, ρ\rho no longer depends on the scale of the standard deviations of an environment or on the regularization weight of an algorithm. It only sets how strongly the gate redistributes the regularization across states, with ρ→∞\rho\to\infty recovering the baseline.

Response (2/2)

Questions

  1. Could you compare against an ungated variant with per-dataset tuned under an equal tuning budget? If the advantage disappears, the gain is merely a smaller effective β\beta.

As you point out, only the product βΦ(πsb)\beta\Phi(\pi^{b}_{s}) enters the algorithm. The revised gate (14) fixes the scale cc by normalizing the gate to unit batch mean, so the gated variant applies the same average weight as the baseline and only redistributes it across states. To further separate the gate from a smaller β\beta, Appendix D.2 now contains ablations on Hopper in which the gate is replaced by a constant gate of 0.50.5 (Figure 5) and by a gate drawn uniformly at random from [0,1][0,1] (Figure 6), which approximately match the average weight of the unnormalized gate of the original submission (≈0.513\approx0.513), as well as CQL with a halved conservative weight (Figures 7 and 8), all with the same training and evaluation budget. For TD3+BC, both ablations remain significantly below the gated variant, whereas for CQL, most of the gain is explained by the smaller average weight. We matched the average weight instead of running a full sweep over β\beta, since this directly tests whether the gains stem from a lower average regularization.

  1. Could you ablate the gate shape-constant Φ≡c\Phi\equiv c, Φp\Phi^{p}, a linear-in-entropy version...

The constant and the random ablation are described in our response to Q1. Since the gate has unit batch mean, the baseline and the constant ablation are constant gates with β\beta rescaled by 11 and 0.50.5, and for TD3+BC both remain below the gated variant. Since the continuous experiments are intended as an outlook on whether the concept transfers to established algorithms, we use a single simple gate whose shape is similar to that of the entropy gate. As stated in Section 3, a continuous gate can be constructed in many different ways, and we do not claim that (14) is the optimal choice.

  1. Could you report the correlation between Φ(πsb)\Phi(\pi_{s}^{b}) and measured coverage ...

The relation between the entropy of the behavioral policy and coverage is motivated by the law of large numbers. In a sufficiently large offline dataset, the empirical action frequencies in a state approach the behavioral policy, and so do the corresponding entropies. To verify that this applies at the dataset sizes used in our experiments, we have added Appendix E.1 to the revised manuscript. Figure 10 compares the gate computed from the behavioral policy with the gate computed from the empirical action frequencies in the tabular experiments, and Figure 11 does the same for local state neighborhoods of the continuous datasets. The two quantities closely agree in the tabular setting and for Hopper and HalfCheetah, and less so for Ant. We have adjusted the wording in the introduction accordingly.

Thanks for your answer, it solved the issue I was worried about, I'll try to improve my score.

We thank you for your response and are glad that our answers resolved your concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.