acceptodds
Under review as a conference paper at ICLR 2027

Regulating Actor-Critic Coupling with Uncertainty in Offline Reinforcement Learning

10 views

Abstract

Offline reinforcement learning (RL) is particularly vulnerable to extrapolation errors induced by out-of-distribution (OOD) actions. While existing methods typically address this problem through policy constraints or conservative value estimation, we argue that the key difficulty in offline actor-critic learning lies in how estimation errors are amplified through the coupling between policy improvement and boostrapped value evaluation. In this work, we introduce Uncertainty-Regulated Actor-Critic (URAC), which uses uncertainty as a shared reliability signal to regulate both stages of this coupling. URAC learns a data-conditioned value anchor together with an uncertainty scale. On the critic side, it suppresses uncertain optimistic bootstrap values before they propagate through Bellman updates; on the actor side, it adaptively controls how much value-guided residual deviation from a behavior anchor is accepted. We further show that these two mechanisms provide complementary contractions of actor-side error admission and critic-side error transmission, yielding a multiplicative reduction in closed-loop error propagation. Across Gym-MuJoCo and NeoRL2 benchmarks, URAC achieves the highest aggregate performance among the evaluated methods, while remaining robust when offline data are limited or corrupted by noise. These results suggest that explicitly regulating actor-critic coupling provides an effective way to stabilize policy improvement under uncertain offline supervision.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Uncertainty Estimation, Out-of-Distribution Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8emwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Uncertainty-Regulated Actor-Critic (URAC), a method for offline reinforcement learning. URAC uses a shared, state-action-dependent reliability signal to regulate both critic-side bootstrap target correction and actor-side policy improvement. Specifically, the authors employ a consistency-model-based estimator to learn a value anchor and an uncertainty scale from truncated-return samples. The critic shrinks value estimates above the value anchor according to the uncertainty scale, while a gated residual actor controls the deviation of the learned policy from a behavior-cloning policy anchor. The paper interprets these two mechanisms as complementary means of controlling the admission and propagation of out-of-distribution (OOD) errors. URAC is evaluated on the Gym-MuJoCo and NeoRL2 benchmarks, with additional ablation studies and experiments involving limited data and noisy data. The reported results demonstrate competitive aggregate performance, with URAC achieving the best results among the compared methods in several cases. However, the central multiplicative contraction claim relies on conditions that are not adequately reflected in the main theorem statement. Furthermore, the experiments leave open questions regarding uncertainty calibration, baseline comparability, and statistical robustness.

Critical strengths
  1. A coherent perspective on actor–critic coupling. The paper distinguishes between the actor-side acceptance of value estimates that lack sufficient data support and the critic-side propagation of errors through bootstrapping. This distinction provides a clear conceptual rationale for controlling both stages, rather than relying solely on conservative value estimation or behavior-policy regularization.

  2. Broad experimental evaluation with targeted ablation studies. The paper evaluates URAC on the Gym-MuJoCo and NeoRL2 benchmarks and compares variants that remove the value anchor, use uncertainty only on the critic side, or use uncertainty only on the actor side. In addition, the paper investigates settings with reduced data and noisy data, which are relevant to practical challenges in offline reinforcement learning.

Critical weaknesses
  1. The central theoretical contraction claim requires a more careful statement and further justification. Theorem 4.4 provides a bound in terms of ∣ΔQstd(z)∣|\Delta Q^{\mathrm{std}}(z)|, but Appendix C.4 relies on the condition K(z,x)≥0K(z,x) \geq 0 along the relevant propagation path to establish this comparison. The paper also acknowledges that the empirical neural tangent kernel (NTK) can take both positive and negative values. In this case, Appendix C.4 derives an alternative bound based on the magnitude proxy
Bstd(z)=γη Ex∼D[∣K(z,x)∣[δ(s′)]+].B^{\mathrm{std}}(z)=\gamma\eta\, \mathbb{E}_{x\sim\mathcal{D}} \left[ |K(z,x)|[\delta(s')]_+ \right].

Since signed terms in ΔQstd(z)\Delta Q^{\mathrm{std}}(z) can cancel each other out, Bstd(z)B^{\mathrm{std}}(z) is generally neither equal to nor universally bounded by ∣ΔQstd(z)∣|\Delta Q^{\mathrm{std}}(z)|. The authors should explicitly state the required kernel condition in the theorem or consistently use the magnitude-based bound throughout the paper. Furthermore, ρmin⁡\rho_{\min} can be zero when the relevant queried points do not have values above the value anchor. Therefore, the critic-side factor does not necessarily yield a strict contraction in general.

  1. The connection between the theoretical analysis, the learned actor gate, and the uncertainty signal remains insufficiently established. The paper does not establish that the learned gate realizes the theoretically derived optimum, nor does it sufficiently explain how the actor-side contraction factor βA\beta_A is guaranteed to satisfy βA<1\beta_A < 1 in practice. The dispersion of samples from a conditional distribution learned from truncated returns does not automatically constitute a calibrated measure of epistemic uncertainty or data support, particularly for candidate actions outside the dataset's support. Direct validation of uncertainty calibration and gate behavior would strengthen the paper's central claims.

  2. In Table 1, the reported total score of URAC on Gym-MuJoCo is 866.0, whereas summing the nine individual task scores yields 869.4. Furthermore, the Walker2d-medium-expert score differs between Table 1 (115.9) and Table 2 (115.5), despite both reporting results for the full URAC model.

Questions for authors

See Weaknesses above.

1 comment

We thank the reviewer for recognizing the coherent perspective on actor–critic coupling and the targeted experimental evaluation. We clarify the theoretical conditions, uncertainty interpretation, and reporting below.

W1: Signed kernels and the contraction guarantee.

The theory is intentionally a local one-step sensitivity analysis, rather than a global convergence guarantee for nonlinear networks (Section 4, Appendix C). Our critic correction Q~=Qˉ−w[Qˉ−μ]∗+\widetilde Q=\bar Q-w[\bar Q-\mu]*+ is nonexpansive in Qˉ\bar Q for fixed (μ,u)(\mu,u): its local derivative is 1−ρ1-\rho, with ρ=w1Qˉ>μ\rho=w\mathbf 1{\bar Q>\mu}. Thus it strictly attenuates local perturbations *when above-anchor correction is active. When ρ∗min⁡=0\rho*{\min}=0, the method does not guarantee a uniform strict critic contraction; this is intentional because values not exceeding the anchor should not be indiscriminately penalized.

Importantly, Appendix C.4 already contains the signed-kernel-safe formulation:

∣ΔQURAC(z)∣≤(1−ρmin⁡)βABstd(z)+O(η2)+O(ηϵμ),|\Delta Q^{\mathrm{URAC}}(z)|\le(1-\rho_{\min})\beta_A B^{\mathrm{std}}(z)+O(\eta^2)+O(\eta\epsilon_\mu),

where Bstd(z)=γηE[∣K(z,x)∣estd(s′)]B^{\mathrm{std}}(z)=\gamma\eta\mathbb E[|K(z,x)|e_{\mathrm{std}}(s')]. The reviewer is correct that BstdB^{\mathrm{std}} is not generally identical to ∣ΔQstd∣|\Delta Q^{\mathrm{std}}|, because signed contributions can cancel. We will move this magnitude-based formulation into the main theorem and explicitly reserve the direct comparison for the no-cancellation case. A strict reduction of the leading-order envelope requires (1−ρmin⁡)βA<1(1-\rho_{\min})\beta_A<1, with anchor-error and higher-order terms retained. These clarifications strengthen the formal statement while preserving the multiplicative combination of two distinct local controls.

W2: Actor gate, admission factor, and uncertainty quality.

Theorem 4.3 directly establishes ∣yπ−yb∣≤γMg∣Δa∣|y_\pi-y_b|\le\gamma M g|\Delta a|: reducing the accepted fraction of a fixed residual proposal limits the maximum target departure from the behavior anchor. Appendix C.3 derives the optimal accepted magnitude for a fixed-direction quadratic surrogate, showing why the preferred step size decreases as uncertainty rises. We do not require the jointly learned gate to attain that surrogate optimum exactly; rather, it motivates the actor design. The additional condition eURAC≤βAestde_{\mathrm{URAC}}\le\beta_Ae_{\mathrm{std}} with βA≤1\beta_A\le1 will be emphasized explicitly as the admission condition in the combined bound, not a universal consequence of smaller residual magnitude.

Our uu is a data-conditioned predictive-dispersion signal, not a claim of globally calibrated epistemic uncertainty. The consistency-model value estimator follows QDQ; Appendix A.1 states that truncated-return targets are proxies, and Appendix A.2 already restricts the reliability assumption to the neighborhood of behavior-anchored proposals, not arbitrary far-OOD actions. Freezing and detaching the estimator keeps this reference from being optimized to match the changing actor's preferences. The local anchor accuracy condition appears explicitly in Appendix C, with its effect quantified by the O(ηϵμ)O(\eta\epsilon_\mu) term. We will emphasize the intended local role of the signal in the main text.

The existing evidence already aligns with the design: Fig. 2 shows why fixed acceptance strength is task-dependent; Figs. 3–4 show strong returns alongside systematic gate adjustments under reduced/noisy data. They are useful evidence of adaptive coupling regulation, even though they are not direct calibration measurements. As Reviewer Ty9z recognizes, the empirical contribution is broader than small gains against one baseline. To strengthen the mechanism-level link, we plan additional uncertainty-reliability diagnostics using held-out return proxies, together with accepted residual norms and correction statistics. These will complement the existing performance and robustness results rather than serve as prerequisites for the algorithmic contribution.

W3: Arithmetic and Table 1–2 reporting.

The printed 866.0 is a simple addition error: the nine URAC means listed in Table 1 sum to 869.4. We apologize and will correct the total. The Walker2d-medium-expert values have different meanings: 115.9±1.4 in Table 1 is the mean±standard deviation over five seeds, whereas 115.5 in Table 2 is from one run of the full model in the ablation study. We will explicitly label Table 2 as single-run results to prevent this reporting ambiguity. The method and underlying experimental outcomes are unchanged.

Reviewer_Ty9zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-Regulated Actor-Critic (URAC), an offline RL method that addresses error propagation through the coupling between policy improvement and bootstrapped value estimation. URAC learns a data-conditioned value anchor and uncertainty estimate, which are used both to suppress uncertain optimistic bootstrap values on the critic side and to regulate deviations from a behavior-policy anchor on the actor side. The paper provides a theoretical analysis of these two mechanisms and evaluates the approach on Gym-MuJoCo and NeoRL2 benchmarks, including comparisons with a range of offline RL baselines and ablation studies of the main components.

Critical strengths

The paper provides a clear and compelling perspective on a central problem in offline RL. Rather than viewing OOD errors only as a critic/value-estimation problem, the paper explicitly analyzes how errors can propagate through the feedback loop between policy improvement and bootstrapped value estimation. This actor-side admission versus critic-side transmission perspective is intuitive and provides a coherent motivation for the proposed method.

The proposed method is well aligned with this motivation. URAC uses a shared uncertainty signal on both sides of the actor-critic loop: the critic suppresses uncertain optimistic bootstrap values, while the actor controls deviations from a behavior-policy anchor based on uncertainty. The individual ingredients are relatively intuitive, and their combination follows naturally from the paper's analysis rather than appearing as an arbitrary collection of components.

The empirical evaluation is broad and generally strong. The main experiments cover both Gym-MuJoCo and NeoRL2 and compare against a substantial collection of established offline-RL approaches spanning behavior regularization, conservative value learning, distributional/uncertainty-based methods, and robust offline RL. The additional ablations explicitly investigate the value anchor and actor-only/critic-only variants, which is particularly useful for testing the paper's central claim that both sides of the coupling matter.

Despite the concerns about the exact comparison with RORL, the results suggest that URAC is highly competitive across a diverse set of tasks. In particular, some improvements over established uncertainty-based approaches such as PBRL are sufficiently large that the empirical contribution does not rest solely on marginal improvements over one strong baseline.

Critical weaknesses
  1. The novelty relative to prior uncertainty-based offline RL methods could be established more clearly.

The paper positions URAC as distinct from prior methods by jointly using uncertainty to regulate actor-side policy deviation and critic-side bootstrap propagation. However, some closely related uncertainty-aware actor-critic methods receive only limited discussion. In particular, UWAC is mentioned only briefly as using uncertainty to improve value robustness, without an in-depth comparison of how its uncertainty-weighted actor-critic updates differ from URAC's uncertainty-gated actor and uncertainty-corrected critic. Given the conceptual proximity, I think a more explicit methodological comparison would help establish the novelty of URAC. It may also be worthwhile to include UWAC as an experimental baseline

  1. The statistical reporting in the main experiments is incomplete.

Table 1 states that results are averaged over five random seeds and reports URAC results with ± values, but it does not specify whether these correspond to standard deviations, standard errors, confidence intervals, or another statistic. Moreover, no uncertainty estimates are provided for any of the baselines. This makes it difficult to assess whether relatively small improvements over strong baselines such as RORL are meaningful. For example, on Walker2D-medium the reported scores are 92.5 ± 0.3 for URAC and 92.4 for RORL, while on Hopper-medium-expert they are 113.2 ± 0.7 and 112.7, respectively. Comparable uncertainty statistics should therefore be reported for reproduced baselines, and the meaning of the URAC error bars should be stated explicitly.

  1. More importantly, some of the reported RORL results appear inconsistent with the originally published results, potentially affecting one of the paper's central empirical claims.

The paper states that baseline numbers are obtained either from the original experiments or by reproducing official code, but does not identify the source used for individual results. In particular, Table 1 reports RORL at 92.4 on Walker2D-medium, whereas the original RORL publication reports a substantially higher result for this benchmark. There are also discrepancies for some of the medium-expert tasks. This deserves clarification, especially because the paper explicitly claims that URAC obtains the highest aggregated Gym-MuJoCo performance. Using the originally published RORL results rather than the values currently reported in Table 1 appears to overturn that aggregate ranking.

Questions for authors

see weaknesses

1 comment

We thank the reviewer for recognizing the compelling motivation, coherent design, and broad evaluation. We appreciate the suggestions for making the methodological comparison and reporting more transparent.

W1: Novelty relative to UWAC.

UWAC is already discussed in Sec. 2, but we agree that a more explicit mechanism-level comparison will help readers see why URAC is not simply another uncertainty-weighted actor-critic. Both methods exploit uncertainty; importantly, they intervene at different places in the actor-critic feedback loop:

  • Signal: UWAC estimates uncertainty using MC dropout on critic predictions. URAC samples from a separately pretrained, frozen conditional return model to obtain a data-conditioned value reference μ\mu and dispersion uu. Freezing this estimator prevents its targets from co-adapting with the actor-critic updates.
  • Critic: UWAC reweights the TD regression loss according to uncertainty. URAC instead corrects the value *entering* the Bellman target, Q~=Qˉ−uu+c[Qˉ−μ]+\widetilde Q=\bar Q-\frac{u}{u+c}[\bar Q-\mu]_+. The reference μ\mu matters: it selectively suppresses above-anchor optimism rather than uniformly penalizing all uncertain values.
  • Actor: UWAC weights the policy objective and retains a behavior-support regularizer. URAC explicitly separates residual proposal and acceptance: aπ=ab+gΔaa_\pi=a_b+g\Delta a, with gg conditioned on critic features and (μp,up)(\mu_p,u_p), alongside a penalty on the accepted deviation. This regulates which proposed improvements the actor can admit before those actions participate in later backups.

Thus, the novelty is not merely "uncertainty on both sides," but coordinating an anchor-relative bootstrap correction with gated residual acceptance at two distinct stages of error propagation. This rationale is also recognized by Reviewer 8emw, who highlights the distinction between actor-side acceptance and critic-side propagation as a clear conceptual contribution. We will expand the UWAC comparison in Related Work and plan to include a direct UWAC benchmark comparison in the revision to complement the substantial set of existing baselines. We do not rely on an unreported UWAC result for our current claims.

W2: Statistical reporting.

The ±\pm values in Table 1 are standard deviations across five random seeds. We will make this explicit in the caption. We also agree that a table combining results from published sources and official-code reproductions should identify the provenance and statistical protocol of each baseline.

On Walker2d-medium, URAC's 92.5±0.392.5\pm0.3 and reproduced RORL's 92.492.4 are indeed very close, as are the reported Hopper-medium-expert scores. We do not base the contribution on claiming statistical superiority from these narrow gaps. Our broader evidence spans medium/medium-replay tasks, NeoRL2, and robustness under reduced/noisy data. The reviewer similarly notes that improvements over uncertainty-based baselines such as PBRL are large enough that the empirical contribution is not contingent on small gains against one strong method.

W3: RORL provenance and aggregate comparison.

The RORL numbers in Table 1 are from our reproduction using the official implementation, not a transcription of the original paper's table. Sec. 5.1 already states that baselines are drawn from original papers or replications of official code; we will label RORL explicitly as "RORL (reproduced)" and document its evaluation settings and source. These are distinct experimental results; their difference does not imply a transcription error.

We agree, however, that the aggregate claim needs a precise comparison scope. We will explicitly distinguish the original published results from our reproduced comparison rather than silently mixing protocols. We will also reconcile the Walker2d-medium-expert value across Tables 1 and 2 from the existing records.

These reporting refinements do not change the motivation or the central evidence: URAC is competitive across two benchmark families and exhibits consistent robustness trends, while its design targets a different intervention point from uncertainty-weighted optimization alone.

Reviewer_aQD8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Uncertainty-Regulated Actor-Critic (URAC) to address error amplification caused by out-of-distribution actions in offline reinforcement learning. The method uses a consistency-model-based value estimator to obtain a value anchor and uncertainty scale, which jointly regulate optimistic Bellman backups and behavior-anchored residual policy improvement. The authors further analyze the effects of these mechanisms on actor-critic error propagation and evaluate URAC on Gym-MuJoCo and NeoRL2 benchmarks. The reported results demonstrate competitive performance and improved robustness under limited and noisy offline data.

Critical strengths
  1. The paper provides a meaningful perspective on error amplification in offline actor-critic learning by considering both actor-side policy improvement and critic-side Bellman bootstrapping. The proposed uncertainty-regulated design offers a coherent approach to addressing these coupled sources of error.
  2. The experimental evaluation covers Gym-MuJoCo and NeoRL2 benchmarks, including ablation studies and robustness tests under limited and noisy data. The reported results demonstrate competitive aggregate performance and provide encouraging empirical support for the proposed framework.
Critical weaknesses
  1. The theoretical justification for the claimed multiplicative error contraction remains insufficient. Theorem 4.4 relies on an actor-side reduction factor βA whose validity is not directly established by Theorem 4.3. Moreover, the bound depends on ρmin, which may vanish when critic correction is inactive, while the empirical neural tangent kernel may contain negative values. These conditions raise concerns about whether the claimed multiplicative contraction holds for general nonlinear actor-critic networks.
  2. The experimental evidence does not sufficiently isolate or validate the proposed coupling-regulation mechanisms. The learned gate and uncertainty-weighted residual penalty in Equation (8) may have overlapping effects, while the ablation variants in Table 2 introduce additional architectural or objective changes. Furthermore, the improved returns and reduced gate coefficients in Figures 3 and 4 do not directly establish suppressed actor-critic error propagation. These limitations weaken the attribution of the observed improvements to the proposed mechanisms.
Questions for authors
  1. Equations (6) and (8) evaluate uncertainty at the proposed action, although the final accepted action may differ substantially. Why is uncertainty not evaluated at the final action? Could this affect the reliability of the gating mechanism?
  2. Equation (8) restricts policy deviation through both a learned gate and an uncertainty-weighted residual penalty. How are their individual contributions distinguished? Could the authors clarify whether either mechanism alone accounts for most of the improvement?
  3. Theorem 4.4 introduces an actor-side error reduction factor βA. Under what conditions can βA ≤ 1 be guaranteed? Does the reduction in accepted residual magnitude established in Theorem 4.3 necessarily imply a proportional reduction in OOD value error?
  4. The multiplicative bound in Equation (13) depends on ρmin, which may be zero when critic correction is inactive. Appendix C also notes that the empirical neural tangent kernel may contain negative values. Under what conditions does the claimed contraction remain valid for general nonlinear networks?
  5. In Table 2, the critic-only and actor-only variants introduce a behavior-regularized actor and a CQL-style critic penalty, respectively. Why were these additional changes introduced instead of disabling only the corresponding uncertainty-regulation mechanisms?
  6. Figures 3 and 4 show improved returns and reduced gate coefficients under degraded data conditions. Is there any direct evidence that these changes correspond to reduced value estimation bias, OOD action selection, or bootstrap error propagation?
  7. Several presentation issues require clarification. The manuscript contains repeated expressions such as "Eq. equation 9," and the Walker2D-med-expert score differs between Table 1 (115.9) and Table 2 (115.5). The notation in Equation (1) and the terminology for variance and standard deviation should also be checked.
1 comment

We thank the reviewer for recognizing the value of the actor–critic coupling perspective and URAC's broad empirical evaluation. We address the related concerns together below.

W1, Q3–Q4: Theoretical conditions.

Our analysis isolates two stages of a local error pathway: the actor determines which target perturbations are admitted, while the critic determines how they propagate through parameter sharing. Theorem 4.3 proves ∣yπ−yb∣≤γMg∣Δa∣|y_\pi-y_b|\le\gamma M g|\Delta a|, directly bounding the target's deviation from the behavior anchor. It does not, by itself, assert a proportional bound on true OOD value error. In Theorem 4.4, eURAC≤βAestde_{\mathrm{URAC}}\le\beta_A e_{\mathrm{std}} with βA≤1\beta_A\le1 is the explicit admission condition for comparing the two methods, not a guarantee for every learned policy. We will state this condition and the distinction more prominently.

The signed-kernel issue is already handled in Appendix C.4 by the magnitude-safe result

∣ΔQURAC(z)∣≤(1−ρmin⁡)βABstd(z)+O(η2)+O(ηϵμ),|\Delta Q^{\mathrm{URAC}}(z)|\le(1-\rho_{\min})\beta_A B^{\mathrm{std}}(z)+O(\eta^2)+O(\eta\epsilon_\mu),

where Bstd(z)=γηE[∣K(z,x)∣estd(s′)]B^{\mathrm{std}}(z)=\gamma\eta\mathbb E[|K(z,x)|e_{\mathrm{std}}(s')]. We will use this formulation in the main theorem; comparison to realized ∣ΔQstd∣|\Delta Q^{\mathrm{std}}| additionally requires no cancellation. If ρmin⁡=0\rho_{\min}=0, a uniform strict critic factor does not follow. Importantly, this is consistent with our design: correction is intentionally inactive when the critic does not exceed the anchor, avoiding indiscriminate pessimism. Strict improvement in the leading-order bound requires (1−ρmin⁡)βA<1(1-\rho_{\min})\beta_A<1. The multiplicative structure of admission and transmission therefore remains meaningful under explicit local conditions, without claiming global convergence of arbitrary nonlinear networks.

Q1: Why query uncertainty at the proposal?

The actor first evaluates ap=ab+Δaa_p=a_b+\Delta a and then decides how much to accept through aπ=ab+gΔaa_\pi=a_b+g\Delta a. Querying uncertainty at aπa_\pi to produce that same gate would create an implicit dependency or require iterative refinement. Proposal-level uncertainty assesses the intended improvement before acceptance, whereas Eq. (8) penalizes the accepted deviation. Crucially, critic bootstrapping does evaluate (μ,u)(\mu,u) at the final next-state action π(s′)\pi(s') (Appendix A.3). We will distinguish these two query locations and clarify that proposal uncertainty need not upper-bound uncertainty at the accepted action.

W2, Q2 and Q5: Gate, penalty, and ablations.

The gate and residual penalty have different algorithmic roles. The gate parameterizes the executed action and dynamically admits a fraction of the proposal using critic features and (μp,up)(\mu_p,u_p); the penalty shapes learning by charging up∣aπ−ab∣2u_p|a_\pi-a_b|^2 for the deviation actually accepted. Their effects reinforce one another rather than representing duplicate terms.

Table 2 deliberately compares URAC against meaningful single-side designs that retain conventional safeguards: behavior regularization with the corrected critic, or a CQL-style conservative critic with the gated actor. Simply removing all regularization from one side would risk an unstable and less informative comparison. Full URAC performs best on all six reported tasks, supporting coordinated control over either alternative alone. As Reviewer Ty9z also notes, the combined design follows naturally from the coupling analysis and the experiments are broad. To isolate individual contributions more precisely, we plan to add gate-only and penalty-only ablations in the revision.

Q6: Evidence for the mechanism.

Figures 3–4 already show that URAC maintains stronger returns while the learned gate adjusts downward as data become sparse or noisy. This is relevant behavioral evidence of adaptive admission control, not merely an aggregate-score comparison. We agree that average gg is not itself a direct measurement of value bias or g∣Δa∣g|\Delta a|. We will make that distinction clear and plan targeted diagnostics of accepted residual norms and above-anchor bootstrap corrections, with a value-bias/OOD proxy where reliably measurable, to further connect observed robustness to the proposed mechanism.

Q7: Presentation and table entries.

We will correct repeated equation references and describe uu in Eq. (1) as standard deviation. In Table 1, 115.9±1.4115.9\pm1.4 on Walker2d-medium-expert is the mean and standard deviation over five seeds; in Table 2, 115.5115.5 is a single-run result from the ablation evaluation. These are different reporting granularities, not conflicting estimates of the same statistic. We will label Table 2 explicitly to avoid this confusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.