acceptodds
Under review as a conference paper at ICLR 2027

The Critic Is Only a Proxy: Regret Optimization for Reinforcement Learning

Abstract

When direct evaluation is costly, learning systems can turn limited observations into a reusable source of feedback. Actor-critic reinforcement learning follows this approach, but its critic is only an informative, imperfect proxy for environmental return. The actor therefore faces an intrinsic robustness problem even without environment shift. We propose to protect against the worst plausible regret: the gap between the current policy and the best action under the same plausible critic. To develop a practical algorithm, we further replace hard hindsight search by a soft comparison and develop Distributionally Robust Relaxed Regret Optimization (DR3O), which can be efficiently estimated from counterfactual action samples. This principle can serve as a general framework when setting up the objective against which the actor optimizes. As an example, we use this approach to indicate "which to give to the actor" in the well-known twin critics. As a result, across MuJoCo, MetaWorld, and the DeepMind Control Suite, changing this actor-facing signal improves matched training backbones without any additional efforts. A data-dependent ellipsoidal bonus provides further gains, supporting regret optimization as a practical way to learn from imperfect feedback.

Primary area:
reinforcement learning
Keywords:
deep reinforcement learning, actor-critic methods, robust optimization, minimax regret, critic uncertainty

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gPsZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes minimizing worst-case regret to guide actor updates when critics are not accurate. It derives a soft regret objective and a further relaxation that can be estimated through action sampling. For twin critics, the canonical soft-regret formulation yields a simple rule without the further relaxation: update the actor using the critic with the larger estimated gap between its soft best-action value and the current policy's value. The paper also develops an ellipsoidal uncertainty correction and evaluates both approaches across continuous-control benchmarks.

Critical strengths
  1. The paper combines duality, entropy smoothing, and a Jensen relaxation to obtain a sampling-based objective for worst-case regret. It proves that the relaxed objective is bounded below by canonical soft regret and above by hard worst-case regret.

  2. The paper gives a principled justification for the critic-selection rule. When the uncertainty model is a convex hull of two critics, selecting the critic with the larger canonical soft regret exactly solves the inner critic-selection problem when the regrets are known. The practical method estimates these regrets from samples and bounds selection error. The resulting update fits naturally into existing actor-critic algorithms.

Critical weaknesses
  1. The contribution of the critic-selection result needs clearer positioning. The rule can be used in other actor-critic settings, but its exactness result is true under the proposed regret objective and convex-hull uncertainty model. This does not establish that the selected critic gives the best improvement in general environmental return. The position of Section 4 in this paper is not clear enough.

  2. The experiments do not sufficiently isolate the benefit of regret-based critic selection. Comparisons with other critic selections, e.g., mean-Q, would help establish whether the proposed criterion explains the gains. Learning from imperfect critics is a broad topic, and part of the paper's contribution rests on how it chooses critics to update the actor.

  3. I suggest moving the main algorithm pseudocode from Appendix into the main text.

Questions for authors
  1. Table 1 reports the largest average gain for Adaptive Ellip in each setting and this ellipsoidal variants train a single critic. How much of the gain comes from the feature-ellipsoidal correction versus the difference in critic number? Could the authors provide a matched single-critic baseline without the correction?

  2. Why does Figure 1 use TD3 for HalfCheetah-v5 and SAC for Swimmer-v5? Could the authors explain how these task-backbone pairs were selected and show whether the learning-curve improvements also hold with SAC on HalfCheetah-v5 and TD3 on Swimmer-v5?

1 comment

Thank you for your appreciation of our work and for the constructive feedback. Thanks for asking us to distinguish the twin-critic result from its implications for environmental return, and for suggesting the complementary learning curves. Both points helped us make the scope of the paper clearer.

W1 and W3. What is exact, and how is the update used? Proposition 3 solves the inner regret problem over the convex hull of the two fitted critics. At each state we estimate each critic's soft best value minus its value under the current policy. The critic with the larger gap guides that actor update. Exactness applies to this regret problem. We evaluate the resulting policy's environmental return separately. Section 5 makes that distinction explicit. Section 5.2 now introduces the ellipsoidal actor update through a concrete RKHS example, with the replay metric defined in Example 2 and the update beside Eq. (20). Algorithms 1 and 2 in Appendix C give the full training loops.

W2. Mean-Q and other actor rules. We compared all five rules under the same twin-critic training protocol on six SAC tasks with ten fresh seeds (500-509) (Appendices G.1 and G.3, Table R2). DR3O’s paired 95% intervals over Min-Q are positive on five tasks. Compared with Mean-Q, DR3O gains 4.2 success points [0.8, 7.6] on push-v3, 4.4 [0.6, 8.2] on drawer-open-v3 and 10.0 return points [0.4, 19.6] on cartpole-swingup. Mean-Q has the higher mean on Hopper (DR3O minus Mean-Q −33 [−415, 349]). To check whether DR3O simply follows one preferred critic, we identified the more frequently selected critic separately in each seed. Its average share was about 52-53% on each task. Against a rule that picks a critic uniformly at random, DR3O improves push-v3 success from 16.2% to 24.6% (paired gain 8.4 points [4.4, 12.4]). The comparison favors DR3O on Hopper as well. Proposition 3 identifies the critic with the larger regret at each state. These controls show where using that choice benefits the learned policy (Section 6.2; Appendix G.3, Tables R2 and R2b).

Q1. Separating the DR3O bonus from critic number. Without a bonus, single-critic SAC has a lower mean than twin-critic SAC on each of the six tasks (Appendices G.2–G.3, Tables R1–R2; Appendix F.2, Table 5). On push-v3 the respective success rates are 4.6% and 18.2%. Keeping the single critic and Bellman backup fixed, adding Ellip raises success to 21.4% and Adaptive Ellip to 28.7%. The fixed correction thus achieves 16.8 of the adaptive method's 24.1-point gain over no bonus [15.9, 32.3]. Adaptive Ellip's paired gain over no bonus is positive on all six tasks. Table R1 gives the matched controls, and Table 5 identifies which backbones retain twin critics.

Q2. Why these pairs in Figure 1? The original panels showed one stochastic and one deterministic backbone on two tasks. We selected those two only due to space limit. We now give readers the full comparison, and thanks for your suggestion. Figure 1 now places SAC and TD3 in separate rows, with HalfCheetah-v5 and Swimmer-v5 in separate columns. In the original twenty-seed evaluation, fixed and Adaptive Ellip gain +836.5 [709.3, 963.7] and +818.4 [554.4, 1,082.4] on HalfCheetah-SAC. On Swimmer-TD3 the gains are +9.5 [4.1, 14.9] and +9.7 [0.1, 19.3]. DR3O finishes close to baseline on both, with differences of −56.7 [−172.2, 58.7] and +0.7 [−7.2, 8.7]. Tables 17 and 18 give the paired endpoints.

Reviewer_tjAyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript proposes an alternative to pessimistic learning in the actor-critic setting, where one normally uses the worse of two or multiple critics to guide the actor, and suggests instead to minimize the regret over the best action as dictated by the plausible critics. The authors rigorously develop an approach to make this objective tractable, and instantiate their method with twin critics and an alternative feature-based ellipsoidal approach. Extensive experiments on MuJoCo, MetaWorld and DMD across several algorithms demonstrate the effectiveness of regret minimization.

Critical strengths
  1. The alternative framing of minimizing regret, rather than pessimistic evaluation, to handle uncertainty in the critics is well-motivated. The authors propose a practical algorithm that makes this approach tractable and justify their solution theoretically.

  2. Algorithmically, one of the proposed implementations is relatively simple and generally applies to most RL algorithms involving more than one critic. It is effective, with the ellipsoidal approach specifically demonstrating significant gains across the extensive empirical evaluation.

Critical weaknesses

1, The paper is very dense, and comprises primarily a theoretical justification of the algorithm which makes regret minimization tractable. While I appreciate theory, the exposition is difficult to follow as it consists of several different specialized terms of results which are likely to be unfamiliar to the reader while intuition for these results is limited. I did not completely follow all the arguments beyond section 4, and would recommend the authors focus more on intuition in the main text while deferring the details to the appendix.

  1. I have some concerns regarding the experiments. Are all methods tuned equally? It seems that the baselines use default settings while the authors' approach is tuned more heavily with different τ\tau and δ\delta. How are results reported across the tuning? Also, the work compares to TD3 and SAC primarily but fails to compare to similar actor-critic uncertainty methods such as optimistic exploration.
Questions for authors

Please see weaknesses.

  1. Could the authors add additional ablations for randomly selected critics, mean of Q1/Q2 and max of Q1/Q to their results. It would also be interesting to see the relative frequency of use for Q1/Q2 (averaged over seeds) for each of these cases, in addition to the proposed approach. It is unclear otherwise whether the method of selection, to maximize regret, is truly responsible for the improved performance.

  2. Can the authors clarify the difference between their Ellip bonus in (19) and optimistic exploration? The actor bonus seems to encourage the actor to pursue uncertain, high-advantage actions which aligns with a typical exploration bonus. Can the authors baseline against such optimistic/risk-seeking approaches?

  3. How were hyper parameters chosen, and do they need to be tuned per task. Are the results significantly sensitive to the choice of δ\delta?

2 comments

Thank the reviewer for the constructive suggestions in performing further baselines and report more measurements. Also, we appreciate the reviewer for directly pointing out the part with which the reviewer is confused. We have conducted further experiments and revised the draft accordingly. Here are some detailed explanations.

W2 and Q2. Optimism and the OAC baseline. Thank you for asking whether the ellipsoidal correction amounts to optimism. That question helped us clarify what the actor is encouraged to do and motivated a direct comparison with Optimistic Actor-Critic (OAC). OAC shifts its behavior policy toward an upper confidence bound for exploration. In Ellip the relevant direction is x(z)−Eπxx(z)-\mathbb{E}\pi x, where zz is a candidate and the second term is the current policy’s mean feature. A candidate at the policy mean has zero uncertainty in its advantage over that policy. The actor receives the average witness correction cπ⊤x(a)c\pi^\top x(a), which can have either sign.

We ran OAC with its published settings in our evaluation framework on the same six tasks and ten fresh seeds (500-509). On push-v3, Adaptive Ellip reaches 28.7% success and OAC 2.0%, a paired difference of +26.7 percentage points [17.5, 35.9]. On cartpole-swingup the difference is +46.0 return points [18.2, 73.8]. The other four paired 95% intervals include zero. OAC has the higher mean on Hopper and cheetah-run. Table R3b gives all six comparisons and Appendix G.4 describes the implementation.

Q1. Simpler selection rules and critic frequencies. Your question suggests a way to check whether DR3O always follows one preferred critic, even if its index changes across runs. Within each seed we measure the absolute difference between the two selection frequencies. Averaged over ten seeds per task, the gaps span 0.29-0.35 percentage points for Random-Q, 5.6-11.6 for Min-Q, 6.2-11.5 for Max-Q and 4.5-6.1 for DR3O. Mean-Q weights both critics equally at every update. DR3O has a smaller mean gap than Min-Q on all six tasks, with paired p<0.05p<0.05 on Hopper, push-v3 and drawer-open-v3. The two rules choose different critics on 27.5-36.8% of sampled states. On push-v3, median ∣Q1−Q2∣|Q_1-Q_2| is 2.7 on those states versus 1.8 overall. In the online comparison, DR3O gains 8.4 success points [4.4, 12.4] over Random-Q on push-v3 and 919 return points [189, 1,649] on Hopper. This supports the observation that DR3O doesn't choose one critic dominantly over the other but to choose the critic according to the state. DR3O also has higher means than Max-Q on all six tasks, and its paired intervals over Mean-Q are positive on push-v3, drawer-open-v3 and cartpole-swingup (Section 6.2, Tables R2 and R2b).

W2 and Q3. Hyperparameter tuning and sensitivity. We select task-specific regret hyperparameters by mean final-checkpoint return on seeds 200-203 and report the locked settings on separate seeds. The external baselines use published defaults, so the hyperparameter-search effort differs across families. Halving or doubling the ellipsoidal temperature or radius gives 48 paired intervals, all including zero. The largest change in success is 3.1 points. Changing the twin-selector temperature by the same factors shifts success by at most 0.6 points and mean return by less than 1% on these tasks. Appendix F.2 lists the presets. Tables R4a-R4d give the local sweeps, sample-budget results and offline diagnostics.

W1. Readability. Thank you for pointing out where the derivation became difficult to follow. Section 5.2 now introduces a concrete example of the actor update. We start from an RKHS model of critic error, show how a linear Mahalanobis kernel yields the ellipsoid, and define its metric in Example 2. The matrix H=Ct+λIH=C_t+\lambda I uses the running second moment of replay features (for example, the critic’s penultimate-layer activations). The ellipsoid is therefore narrow along directions well represented in replay. Eq. (20) shows how this geometry changes the actor update. Appendix D starts with the simpler LpL_p example and then gives the Wasserstein case. We will carry this emphasis on intuition into the final edit.

Thank you for your detailed response. I don't have further questions. Given my positive rating, I will maintain my score.

Reviewer_1qEPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies policy optimization with imperfect critics in actor–critic reinforcement learning and proposes worst-case regret, rather than worst-case value, as the actor’s decision criterion. Using function-space duality, the authors express worst-case regret as a maximization over candidate actions of nominal advantage plus an uncertainty allowance. Entropic regularization and a Jensen relaxation then yield the Distributionally Robust Relaxed Regret Optimization (DR3O) objective, which can be estimated through counterfactual action sampling. The paper develops two main implementations. The first considers the convex hull of two critics and selects the critic with the larger canonical soft regret to guide the actor update. The second uses feature-ellipsoidal ambiguity geometry to construct a policy-dependent actor bonus, with an adaptive variant that adjusts the ambiguity radius using Bellman mean-squared error. Experiments span three continuous-control benchmarks, four actor–critic backbones, and 68 task–backbone settings. The authors report aggregate performance improvements, using disjoint tuning and evaluation seed sets, 20 training seeds for evaluation, paired confidence intervals, and wall-clock measurements.

Critical strengths
  1. Beyond proposing a worst-case regret objective, the paper distinguishes canonical soft regret from its further relaxation and derives the corresponding gradients. For the convex hull of two critics, Proposition 3 exactly reduces the inner optimization to a comparison between the two endpoints, yielding a simple state-dependent critic selection rule. Appendix A.9 further establishes that this rule is invariant to action-independent shifts of either critic. These results give the method a clear decision-theoretic interpretation beyond a heuristic approach to combining critics.
  2. Experiments across three benchmarks and multiple backbones provide evidence that the approach generalizes across algorithmic settings. Disjoint tuning and evaluation seed sets, 20 training seeds per setting, and paired confidence intervals help assess the reliability of the improvements. The authors also report variation across tasks, limited gains on nearly saturated tasks, and wall-clock measurements, rather than relying solely on a few favorable learning curves. Together, these results support the method’s practical potential.
Critical weaknesses
  1. The experiments do not sufficiently distinguish between the benefit of regret-guided critic selection and the benefit of simply changing the original actor-facing signal. This concern is more pronounced for Ellip and Adaptive Ellip. Section 6 explicitly states that these variants train only a single critic, whereas Appendix F.1 states that architectures and critic-training rules remain unchanged and that only the actor-facing decision rule is modified. If one critic is indeed removed, the number of networks, any resulting changes to the bootstrapped targets, and the computational costs require clarification. Without a baseline that uses the same single-critic configuration but omits the proposed correction, the gains of Adaptive Ellip—the strongest-performing variant—cannot be clearly attributed to its actor bonus. This also complicates the interpretation of the claim that the method incurs almost no additional computational cost.
  2. Section 5 states that the reference distribution is uniform and that the actor itself serves as the sampling proposal, whereas Appendix F.2 states that all regret-aware variants include a uniform mixture component with weight 0.1. For deterministic actors, this distinction directly affects whether the support conditions required for importance sampling are satisfied. Appendix C.2 further explains that the initial TD3 implementation estimates regret using a local stochastic policy while updating the actor at its deterministic action. The authors correctly acknowledge that this differs from the exact deterministic-policy objective in Algorithm 2, but do not clearly identify which implementation was used for the reported experimental results.
Questions for authors
  1. Please clarify the number of critics trained in Ellip and Adaptive Ellip and specify how the bootstrapped targets are constructed for each backbone (SAC, TD3, CrossQ, and TD7). If these variants use a single critic, this configuration appears inconsistent with the statement that architectures and critic-training rules remain unchanged. A baseline using the same critic configuration with the actor correction disabled is needed to isolate the contribution of the proposed bonus.
  2. Comparisons with mean-critic aggregation and random critic selection, using identical twin critics, training configurations, and comparable hyperparameter search budgets, would help establish whether the improvements arise specifically from the regret criterion. Such controls are particularly relevant to assessing the benefit of the proposed selector beyond changing the actor-facing critic signal.
  3. Please specify whether the reported TD3 results use the exact deterministic-policy comparison in Algorithm 2 or the local stochastic-policy comparison described in Appendix C.2. Please also clarify whether all experiments use the mixture proposal specified in Appendix F.2. An explicit mapping between the reported results and the corresponding implementation choices would resolve ambiguities concerning the policy being compared and the support conditions required for importance sampling.
  4. The paper would benefit from an empirical analysis of the reliability of its sampling estimators. Reporting effective sample size, critic-selection agreement with a high-sample reference, and sensitivity to candidate count and temperature on representative tasks would connect the finite-sample analysis to practical performance.
  5. Although the experiments cover several established off-policy algorithms, comparisons with distributional RL methods such as TQC, and DSAC are still missing. Given that these methods also address value estimation errors and overestimation bias, could the authors include representative comparisons to better demonstrate the effectiveness and competitiveness of the proposed approach?
1 comment

Thank you for distinguishing some necessary controlled experiments and for flagging the mismatch in our implementation descriptions. We clarified both and ran the requested controls on six SAC tasks with ten fresh seeds (500-509); All intervals reported below are paired 95% intervals.

W1 and Q1. Does the correction help with one critic? We trained three versions of SAC on six tasks with the same ten seeds (500–509). All three used one critic and the same rule for training it. Only the policy update changed. On push-v3 we observed the following success rates (see Section 6.2 and Appendices G.2–G.3, Tables R1–R2).

  • 4.6% success without the correction. The policy used the critic’s action scores directly.
  • 21.4% with Ellip. We added a fixed-scale correction for uncertainty in those scores.
  • 28.7% with Adaptive Ellip. We let the scale depend on the critic’s recent prediction errors. Standard two-critic SAC reached 18.2% on the same seeds. Removing one critic alone lowered success. Adding Adaptive Ellip to the one-critic control raised success by 24.1 percentage points (paired 95% confidence interval [15.9, 32.3]). Across six tasks, the gain over that control was significant on five for Ellip and all six for Adaptive Ellip. See Section 6.2 and Appendices G.2–G.3, Tables R1–R2 for more details

W1 and Q1. Checking implementation details. In SAC and TD3, each Ellip method trains one critic. A slowly updated copy (the target critic) supplies the next-state value in its training target (Appendix F.2, Table 5). SAC includes its usual entropy term and TD3 perturbs the next action. CrossQ and TD7 keep two critics and take the lower estimate, from the current critics in CrossQ and their delayed copies in TD7. Section 6.4 reports total training time.

Q2. Mean-Q and Random-Q. Thanks for the advice for setting up more meaningful controls. We conducted experiments with Random-Q, Mean-Q, Max-Q, and Min-Q (Appendix G.3, Table R2). For the specifically mentioned Random-Q and Mean-Q, here are the results.

  • With the twin-critic training protocol fixed, DR3O has higher means than Random-Q on all six tasks and Mean-Q on four. Compared with uniform random selection, DR3O gains 8.4 success points [4.4, 12.4] on push-v3 and 919 return points [189, 1,649] on Hopper. The other four intervals include zero. Against Mean-Q, DR3O gains 4.2 points [0.8, 7.6] on push-v3, while Mean-Q has the higher mean on Hopper (DR3O minus Mean-Q −33 [−415, 349]). The controls introduce no tunable parameter. DR3O’s temperature was fixed using seeds 200-203 (Appendices G.1 and G.3, Table R2).

  • We also counted the selection ratio for each critic in DR3O. For each seed we subtracted the less frequently selected critic's share from the more frequently selected critic's share (Appendix G.3, Table R2b). The mean gap is 4.5-6.1 percentage points across tasks, so the more frequently used critic accounts for about 52-53% of decisions on average in each task. This brings us a more insightful observation that the gain of DR3O does not come from consistently selecting one critic as "dominantly better".

W2 and Q3. Proposal and deterministic actors. All runs use p=(1−ρ)g+ρνp=(1-\rho)g+\rho\nu with fixed uniform ν\nu, so dν/dp≤1/ρd\nu/dp\leq1/\rho. For SAC and CrossQ, gg is the actor. TD3 and TD7 estimate the critic or feature mean from a local tanh-Gaussian gg and differentiate the actor at its deterministic action (Appendix C.2). The uniform mass is 0.1 except for SAC HalfCheetah DR3O (0.25). The local standard deviation is 0.1 for TD3 and 0.2 for TD7, with 0.2 for TD3 Swimmer DR3O. Table 6 gives L/K=128/64L/K=128/64 on SAC/TD3 and 64/3264/32 on CrossQ/TD7.

Q4. Sampling reliability. Thanks for the advice, and we have conducted experiments with required measurements (Appendices G.5–G.6, Tables R4b–R4d). At L=128L=128 candidates and K=64K=64 policy samples, the selected critic index agrees with a high-budget reference (L=4096L=4096, K=2048K=2048) on 85.3-95.8% of 360 held-out state evaluations per task. HalfCheetah agreement is 84.2%, 88.3% and 91.7% at L=64,128,256L=64,128,256. Fixed Ellip’s mean effective sample size is 6.3-31.8 of 128 across tasks. On HalfCheetah its 10th percentile is 2.3, so some updates put most of the weight on very few candidates. For DR3O the task means are 12.9-22.3 for the reference weights and 9.5-22.7 after tilting (Table R2b). Halving or doubling the selector temperature changes mean success by at most 0.6 points (Tables R1b and R4b-R4d).

Q5. Distributional baselines. We used published TQC and DSAC v1 defaults with matched seeds and interaction budgets (Appendix G.4, Table R3). All six intervals for Adaptive Ellip minus TQC include zero. Against DSAC v1, four of six intervals are positive, including HalfCheetah (+1,352 [242, 2,462]). Table R3 reports all endpoints and intervals.

Comments on the paper

2 comments

We thank the reviewers for their positive feedback of our work and the constructive comments on improving the presentation and the soundness of the manuscript. As the end of the rebuttal is approaching, we are still actively revising the draft and running experiments (e.g., on hyper-parameter tuning and controlled groups). We will keep you updated whenever a refined response is available.

Sincerely,

Authors of Submission46752

AuthorsOpenReview ↗

General response for comments and the updated PDF

We thank the reviewers for their positive feedback and for pushing us to dig deeper.

During the rebuttal phase, we conducted extensive additional experiments to address the reviewers' concerns and requests. We also revised the manuscript to improve the clarity and accessibility of the theoretical presentation, as some reviewers found the original exposition overly dense. In particular, we added matched-control experiments across six SAC tasks using ten fresh random seeds (500–509), distinct from those used for hyperparameter tuning (200–203) and the original evaluation (100–119).

We would like to highlight the following updates and results:

Which critic guides the actor? Could DR3O gain by finding one consistently better critic and always following it? Within each training seed, we measured how much more often the more frequently selected critic was used. Averaging this difference over ten seeds per task gives 4.5-6.1 percentage points across the six tasks. Thus even the more frequently chosen critic accounts for only about 52-53% of decisions on average in each task. To see whether it matters when each critic is used, we compared DR3O with uniform random selection. On push-v3, DR3O reaches 24.6% success versus 16.2% for Random-Q, a paired gain of 8.4 percentage points [4.4, 12.4]. Paired 95% intervals favor DR3O over Random-Q on push-v3 and Hopper, and over Min-Q on five of six tasks. These comparisons support letting the critic that sees the larger missed opportunity at the current state guide the actor (see Section 6.2, Tables R2 and R2b in the updated PDF).

What does the ellipsoidal correction add? The original comparison was between "double critic + backbone update" and "single critic + DR3O update", and we thank the reviewers for pointing out that "single critic + backbone update" was missing. We conducted the required controlled group. With one SAC critic and the same Bellman backup, push-v3 success rises from 4.6% without a bonus to 21.4% with fixed Ellip and 28.7% with Adaptive Ellip. The fixed correction already gives 16.8 of the adaptive method’s 24.1-point gain over no bonus [15.9, 32.3]. The paired intervals for improvement over no bonus are positive on five tasks for fixed Ellip and all six for Adaptive Ellip (see Table R1 in the updated PDF).

Revising draft. We trimmed down the amount of theory introduced in Section 5. In Section 5.2, we now only introduce a worked example, which is derived from a linear Mahalanobis kernel on critic features and later appeared as the ellipsoidal actor update in our experiments. We postponed other examples into the Appendix. We also completed Figure 1 with all four task-backbone pairings following reviewers' advice, with one backbone per row and one task per column. For implementation details, we have checked carefully, and now Appendix G reports them faithfully (including full controls, comparisons with TQC, DSAC v1 and OAC, and sampling diagnostics). We marked the revisions in blue and will continue to use these examples to guide the exposition.

Again, we sincerely thank the reviewers for their time in reading our manuscript and posting those constructive comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.