- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes minimizing worst-case regret to guide actor updates when critics are not accurate. It derives a soft regret objective and a further relaxation that can be estimated through action sampling. For twin critics, the canonical soft-regret formulation yields a simple rule without the further relaxation: update the actor using the critic with the larger estimated gap between its soft best-action value and the current policy's value. The paper also develops an ellipsoidal uncertainty correction and evaluates both approaches across continuous-control benchmarks.
Critical strengths
-
The paper combines duality, entropy smoothing, and a Jensen relaxation to obtain a sampling-based objective for worst-case regret. It proves that the relaxed objective is bounded below by canonical soft regret and above by hard worst-case regret.
-
The paper gives a principled justification for the critic-selection rule. When the uncertainty model is a convex hull of two critics, selecting the critic with the larger canonical soft regret exactly solves the inner critic-selection problem when the regrets are known. The practical method estimates these regrets from samples and bounds selection error. The resulting update fits naturally into existing actor-critic algorithms.
Critical weaknesses
-
The contribution of the critic-selection result needs clearer positioning. The rule can be used in other actor-critic settings, but its exactness result is true under the proposed regret objective and convex-hull uncertainty model. This does not establish that the selected critic gives the best improvement in general environmental return. The position of Section 4 in this paper is not clear enough.
-
The experiments do not sufficiently isolate the benefit of regret-based critic selection. Comparisons with other critic selections, e.g., mean-Q, would help establish whether the proposed criterion explains the gains. Learning from imperfect critics is a broad topic, and part of the paper's contribution rests on how it chooses critics to update the actor.
-
I suggest moving the main algorithm pseudocode from Appendix into the main text.
Questions for authors
-
Table 1 reports the largest average gain for Adaptive Ellip in each setting and this ellipsoidal variants train a single critic. How much of the gain comes from the feature-ellipsoidal correction versus the difference in critic number? Could the authors provide a matched single-critic baseline without the correction?
-
Why does Figure 1 use TD3 for HalfCheetah-v5 and SAC for Swimmer-v5? Could the authors explain how these task-backbone pairs were selected and show whether the learning-curve improvements also hold with SAC on HalfCheetah-v5 and TD3 on Swimmer-v5?
Thank you for your appreciation of our work and for the constructive feedback. Thanks for asking us to distinguish the twin-critic result from its implications for environmental return, and for suggesting the complementary learning curves. Both points helped us make the scope of the paper clearer.
W1 and W3. What is exact, and how is the update used? Proposition 3 solves the inner regret problem over the convex hull of the two fitted critics. At each state we estimate each critic's soft best value minus its value under the current policy. The critic with the larger gap guides that actor update. Exactness applies to this regret problem. We evaluate the resulting policy's environmental return separately. Section 5 makes that distinction explicit. Section 5.2 now introduces the ellipsoidal actor update through a concrete RKHS example, with the replay metric defined in Example 2 and the update beside Eq. (20). Algorithms 1 and 2 in Appendix C give the full training loops.
W2. Mean-Q and other actor rules. We compared all five rules under the same twin-critic training protocol on six SAC tasks with ten fresh seeds (500-509) (Appendices G.1 and G.3, Table R2). DR3O’s paired 95% intervals over Min-Q are positive on five tasks. Compared with Mean-Q, DR3O gains 4.2 success points [0.8, 7.6] on push-v3, 4.4 [0.6, 8.2] on drawer-open-v3 and 10.0 return points [0.4, 19.6] on cartpole-swingup. Mean-Q has the higher mean on Hopper (DR3O minus Mean-Q −33 [−415, 349]). To check whether DR3O simply follows one preferred critic, we identified the more frequently selected critic separately in each seed. Its average share was about 52-53% on each task. Against a rule that picks a critic uniformly at random, DR3O improves push-v3 success from 16.2% to 24.6% (paired gain 8.4 points [4.4, 12.4]). The comparison favors DR3O on Hopper as well. Proposition 3 identifies the critic with the larger regret at each state. These controls show where using that choice benefits the learned policy (Section 6.2; Appendix G.3, Tables R2 and R2b).
Q1. Separating the DR3O bonus from critic number. Without a bonus, single-critic SAC has a lower mean than twin-critic SAC on each of the six tasks (Appendices G.2–G.3, Tables R1–R2; Appendix F.2, Table 5). On push-v3 the respective success rates are 4.6% and 18.2%. Keeping the single critic and Bellman backup fixed, adding Ellip raises success to 21.4% and Adaptive Ellip to 28.7%. The fixed correction thus achieves 16.8 of the adaptive method's 24.1-point gain over no bonus [15.9, 32.3]. Adaptive Ellip's paired gain over no bonus is positive on all six tasks. Table R1 gives the matched controls, and Table 5 identifies which backbones retain twin critics.
Q2. Why these pairs in Figure 1? The original panels showed one stochastic and one deterministic backbone on two tasks. We selected those two only due to space limit. We now give readers the full comparison, and thanks for your suggestion. Figure 1 now places SAC and TD3 in separate rows, with HalfCheetah-v5 and Swimmer-v5 in separate columns. In the original twenty-seed evaluation, fixed and Adaptive Ellip gain +836.5 [709.3, 963.7] and +818.4 [554.4, 1,082.4] on HalfCheetah-SAC. On Swimmer-TD3 the gains are +9.5 [4.1, 14.9] and +9.7 [0.1, 19.3]. DR3O finishes close to baseline on both, with differences of −56.7 [−172.2, 58.7] and +0.7 [−7.2, 8.7]. Tables 17 and 18 give the paired endpoints.