- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Uncertainty-Regulated Actor-Critic (URAC), a method for offline reinforcement learning. URAC uses a shared, state-action-dependent reliability signal to regulate both critic-side bootstrap target correction and actor-side policy improvement. Specifically, the authors employ a consistency-model-based estimator to learn a value anchor and an uncertainty scale from truncated-return samples. The critic shrinks value estimates above the value anchor according to the uncertainty scale, while a gated residual actor controls the deviation of the learned policy from a behavior-cloning policy anchor. The paper interprets these two mechanisms as complementary means of controlling the admission and propagation of out-of-distribution (OOD) errors. URAC is evaluated on the Gym-MuJoCo and NeoRL2 benchmarks, with additional ablation studies and experiments involving limited data and noisy data. The reported results demonstrate competitive aggregate performance, with URAC achieving the best results among the compared methods in several cases. However, the central multiplicative contraction claim relies on conditions that are not adequately reflected in the main theorem statement. Furthermore, the experiments leave open questions regarding uncertainty calibration, baseline comparability, and statistical robustness.
Critical strengths
-
A coherent perspective on actor–critic coupling. The paper distinguishes between the actor-side acceptance of value estimates that lack sufficient data support and the critic-side propagation of errors through bootstrapping. This distinction provides a clear conceptual rationale for controlling both stages, rather than relying solely on conservative value estimation or behavior-policy regularization.
-
Broad experimental evaluation with targeted ablation studies. The paper evaluates URAC on the Gym-MuJoCo and NeoRL2 benchmarks and compares variants that remove the value anchor, use uncertainty only on the critic side, or use uncertainty only on the actor side. In addition, the paper investigates settings with reduced data and noisy data, which are relevant to practical challenges in offline reinforcement learning.
Critical weaknesses
- The central theoretical contraction claim requires a more careful statement and further justification. Theorem 4.4 provides a bound in terms of , but Appendix C.4 relies on the condition along the relevant propagation path to establish this comparison. The paper also acknowledges that the empirical neural tangent kernel (NTK) can take both positive and negative values. In this case, Appendix C.4 derives an alternative bound based on the magnitude proxy
Since signed terms in can cancel each other out, is generally neither equal to nor universally bounded by . The authors should explicitly state the required kernel condition in the theorem or consistently use the magnitude-based bound throughout the paper. Furthermore, can be zero when the relevant queried points do not have values above the value anchor. Therefore, the critic-side factor does not necessarily yield a strict contraction in general.
-
The connection between the theoretical analysis, the learned actor gate, and the uncertainty signal remains insufficiently established. The paper does not establish that the learned gate realizes the theoretically derived optimum, nor does it sufficiently explain how the actor-side contraction factor is guaranteed to satisfy in practice. The dispersion of samples from a conditional distribution learned from truncated returns does not automatically constitute a calibrated measure of epistemic uncertainty or data support, particularly for candidate actions outside the dataset's support. Direct validation of uncertainty calibration and gate behavior would strengthen the paper's central claims.
-
In Table 1, the reported total score of URAC on Gym-MuJoCo is 866.0, whereas summing the nine individual task scores yields 869.4. Furthermore, the Walker2d-medium-expert score differs between Table 1 (115.9) and Table 2 (115.5), despite both reporting results for the full URAC model.
Questions for authors
See Weaknesses above.
We thank the reviewer for recognizing the coherent perspective on actor–critic coupling and the targeted experimental evaluation. We clarify the theoretical conditions, uncertainty interpretation, and reporting below.
W1: Signed kernels and the contraction guarantee.
The theory is intentionally a local one-step sensitivity analysis, rather than a global convergence guarantee for nonlinear networks (Section 4, Appendix C). Our critic correction Q=Qˉ−w[Qˉ−μ]∗+ is nonexpansive in Qˉ for fixed (μ,u): its local derivative is 1−ρ, with ρ=w1Qˉ>μ. Thus it strictly attenuates local perturbations *when above-anchor correction is active. When ρ∗min=0, the method does not guarantee a uniform strict critic contraction; this is intentional because values not exceeding the anchor should not be indiscriminately penalized.
Importantly, Appendix C.4 already contains the signed-kernel-safe formulation:
∣ΔQURAC(z)∣≤(1−ρmin)βABstd(z)+O(η2)+O(ηϵμ),
where Bstd(z)=γηE[∣K(z,x)∣estd(s′)]. The reviewer is correct that Bstd is not generally identical to ∣ΔQstd∣, because signed contributions can cancel. We will move this magnitude-based formulation into the main theorem and explicitly reserve the direct comparison for the no-cancellation case. A strict reduction of the leading-order envelope requires (1−ρmin)βA<1, with anchor-error and higher-order terms retained. These clarifications strengthen the formal statement while preserving the multiplicative combination of two distinct local controls.
W2: Actor gate, admission factor, and uncertainty quality.
Theorem 4.3 directly establishes ∣yπ−yb∣≤γMg∣Δa∣: reducing the accepted fraction of a fixed residual proposal limits the maximum target departure from the behavior anchor. Appendix C.3 derives the optimal accepted magnitude for a fixed-direction quadratic surrogate, showing why the preferred step size decreases as uncertainty rises. We do not require the jointly learned gate to attain that surrogate optimum exactly; rather, it motivates the actor design. The additional condition eURAC≤βAestd with βA≤1 will be emphasized explicitly as the admission condition in the combined bound, not a universal consequence of smaller residual magnitude.
Our u is a data-conditioned predictive-dispersion signal, not a claim of globally calibrated epistemic uncertainty. The consistency-model value estimator follows QDQ; Appendix A.1 states that truncated-return targets are proxies, and Appendix A.2 already restricts the reliability assumption to the neighborhood of behavior-anchored proposals, not arbitrary far-OOD actions. Freezing and detaching the estimator keeps this reference from being optimized to match the changing actor's preferences. The local anchor accuracy condition appears explicitly in Appendix C, with its effect quantified by the O(ηϵμ) term. We will emphasize the intended local role of the signal in the main text.
The existing evidence already aligns with the design: Fig. 2 shows why fixed acceptance strength is task-dependent; Figs. 3–4 show strong returns alongside systematic gate adjustments under reduced/noisy data. They are useful evidence of adaptive coupling regulation, even though they are not direct calibration measurements. As Reviewer Ty9z recognizes, the empirical contribution is broader than small gains against one baseline. To strengthen the mechanism-level link, we plan additional uncertainty-reliability diagnostics using held-out return proxies, together with accepted residual norms and correction statistics. These will complement the existing performance and robustness results rather than serve as prerequisites for the algorithmic contribution.
W3: Arithmetic and Table 1–2 reporting.
The printed 866.0 is a simple addition error: the nine URAC means listed in Table 1 sum to 869.4. We apologize and will correct the total. The Walker2d-medium-expert values have different meanings: 115.9±1.4 in Table 1 is the mean±standard deviation over five seeds, whereas 115.5 in Table 2 is from one run of the full model in the ablation study. We will explicitly label Table 2 as single-run results to prevent this reporting ambiguity. The method and underlying experimental outcomes are unchanged.