- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RDA2C, a replay-based actor-critic algorithm that learns a dual-score function through historical advantage regression. The method connects policy dual averaging with practical deep RL and provides theoretical analysis and experiments on MuJoCo and Atari.
Critical strengths
The algorithmic formulation is interesting and provides a novel perspective on actor optimization. The theoretical analysis offers useful insights into replay-based dual averaging.
Critical weaknesses
The MuJoCo evaluation primarily compares against PPO, while the twin-Q variant achieves only comparable performance to SAC. Recent studies on MuJoCo-v5 report substantially higher returns even for established baselines such as TD3 and SAC [1, 2]. Although cross-paper comparisons are not directly conclusive, these results raise concerns about the strength of the experimental evaluation. Moreover, comparisons with more recent continuous-control methods are missing, making it difficult to assess the practical competitiveness of RDA2C.
[1] Distributional Reinforcement Learning with Diffusion Bridge Critics. 2026. https://arxiv.org/abs/2602.05783
[2] AEAP: A Reinforcement Learning Actor Ensemble Algorithm with Adaptive Pruning. TMLR, 2025. https://openreview.net/forum?id=I5ymMVdmaR
Questions for authors
Can the authors provide direct comparisons with stronger modern continuous-control baselines under matched experimental settings?
4 comments
The point of the parity is that one actor update covers settings that SAC does not. As discussed in Section 4.3, "The same actor update can use either GAE labels from a value critic or advantage labels from twin-Q critics, and applies to both discrete and continuous action spaces." In contrast, standard continuous-action SAC relies on a soft Q-critic and reparameterized actions. Its actor update does not directly accommodate GAE labels, and extending SAC to discrete actions requires a separate formulation [Christodoulou, 2019] and stabilization techniques [Zhou et al., 2024]. RDA2C uses the same actor update across these settings. In settings where the standard SAC update is not directly applicable, RDA2C also outperforms PPO, demonstrating its broader applicability across different critic formulations and action spaces.
3. "Stronger modern continuous-control baselines"
The review does not name a single one. Its only examples, Ding et al. [2026] and Zhang and Sharon [2025], are a distributional critic and an ensemble of TD3-style actors. Neither is an alternative actor update, and either could be combined with RDA2C. A request for unnamed baselines cannot be answered or checked. If the reviewer has specific methods in mind, please name them, and we will run them under the paper's settings.
Again, it is important to distinguish improvements from policy optimization from those arising from critic design. Recent advances in continuous-control performance have largely come from improvements to the critic and network architecture rather than the actor update itself. The two modern baselines suggested by the reviewer [Ding et al., 2026; Zhang and Sharon, 2025] focus on different aspects of the actor-critic framework. Specifically, Ding et al. [2026] introduce a distributional critic based on diffusion bridges, while Zhang and Sharon [2025] propose an ensemble of TD3-style actors with adaptive pruning. Neither directly isolates the effect of replacing the actor update while holding the critic and other components fixed, which is the primary focus of RDA2C.
Furthermore, we ran TD3 using the paper's twin-Q setup. TD3 at its released defaults lands within a few percent of our SAC on every task except HumanoidStandup. RDA2C with twin-Q labels is higher than TD3 on five of eight tasks: suite +6.0% (paired-seed bootstrap 95% CI [+2.7%, +9.4%]). We used the same evaluation as the paper's SAC table (Table 5 of the paper): gymnasium MuJoCo v5, 10 seeds, evaluations every 50k steps with 20 deterministic episodes, and the best evaluation within 1M steps averaged over seeds.
Table 2. Best deterministic evaluation return within 1M steps (mean ± seed std, 10 seeds). The last column is RDA2C relative to TD3 with the paired-seed bootstrap 95% CI.
| Environment | SAC (paper) | TD3, released defaults (new) | RDA2C, twin-Q labels (paper) | RDA2C vs TD3 |
|---|---|---|---|---|
| Ant-v5 | 4389.60 ± 1100.19 | 4701.97 ± 736.31 | 4240.94 ± 877.82 | -9.8% [-23.6%, +8.1%] |
| HalfCheetah-v5 | 10118.79 ± 894.32 | 10148.21 ± 1386.25 | 9524.52 ± 1388.40 | -6.1% [-16.9%, +6.3%] |
| Hopper-v5 | 3345.29 ± 204.84 | 3240.20 ± 103.23 | 3408.27 ± 121.76 | +5.2% [+1.7%, +8.7%] |
| Humanoid-v5 | 5376.44 ± 175.72 | 5163.20 ± 154.30 | 5195.96 ± 314.71 | +0.6% [-3.6%, +4.5%] |
| HumanoidStandup-v5 | 158309.50 ± 4088.25 | 108280.81 ± 12189.50 | 158784.44 ± 6868.90 | +46.6% [+36.7%, +57.2%] |
| InvertedPendulum-v5 | 1000.00 ± 0.00 | 1000.00 ± 0.00 | 1000.00 ± 0.00 | 0.0% |
| Pusher-v5 | -29.58 ± 1.55 | -28.35 ± 1.37 | -28.05 ± 1.51 | +1.0% [-3.4%, +5.4%] |
| Walker2d-v5 | 4079.87 ± 371.70 | 4067.83 ± 541.09 | 4504.91 ± 788.33 | +10.7% [-4.4%, +26.3%] |
| Suite average (paired-seed bootstrap 95% CI) | +6.0% [+2.7%, +9.4%] |
References
[Bhatt et al., 2024] (https://arxiv.org/abs/1902.05605)
[Christodoulou, 2019] (https://arxiv.org/abs/1910.07207)
[Ding et al., 2026] (https://arxiv.org/abs/2602.05783)
[Fujimoto et al., 2023] (https://arxiv.org/abs/2306.02451)
[Kuznetsov et al., 2020] (https://arxiv.org/abs/2005.04269)
[Shahriari et al., 2022] (https://arxiv.org/abs/2204.10256)
[Tomar et al., 2022] (https://arxiv.org/abs/2005.09814v5)
[Zhang and Sharon, 2025] (https://openreview.net/forum?id=I5ymMVdmaR)
[Zhou et al., 2024] (https://arxiv.org/abs/2209.10081)
Thank you for the additional experiments and clarifications. While matching SAC demonstrates that RDA2C is a viable alternative, it remains unclear what advantages justify adopting it over SAC. More relevant comparisons with methods such as MDPO-KL, or a matched replay-based advantage regression baseline without dual averaging, would help establish the specific contribution of the proposed actor update. My main concern regarding the practical benefit of RDA2C therefore remains.
We appreciate the clarification of your concern. The primary advantage of our algorithm over SAC is its flexibility in the selection of the critic and the method by which the critic is learned. SAC's actor update relies on a soft-Q function, which is typically learned using Bellman targets. In contrast, PPO learns a state-value function using mean squared error and constructs advantage estimates using generalized advantage estimation (GAE); these estimates cannot directly substitute for the soft-Q function in SAC's standard actor update. Our algorithm accommodates both SAC-style soft-Q supervision and PPO-style value-function and GAE supervision within the same actor update, eliminating the need to switch algorithms.
Regarding the comparisons, we agree that a comparison with MDPO-KL is relevant. However, the MDPO paper states that off-policy MDPO is equivalent to SAC when the KL regularizer uses the uniform policy, rather than the previous policy, as its reference [Tomar et al., 2022]. This equivalence motivated our decision to compare against SAC rather than both methods. We do include a comparison with the recent dual-averaging algorithm AAPDA [Gao et al., 2026], proposed in 2026, and RDA2C outperforms it on six of eight MuJoCo tasks. The principal benefit of our algorithm is that, by learning the dual variable, it facilitates replay-based dual-variable actor-critic learning without requiring importance sampling in the actor update. To clarify your suggested comparison, we would appreciate more details on the definition of a matched replay-based advantage regression baseline and the specific actor update you propose we compare against.
If your suggestion pertains to the impact of retaining past advantage estimates on performance, we address this in Appendix D.1.3, "Current-label control." This control changes label retention and nothing else: with W = 2,048 the buffer keeps exactly one rollout, so every regression target comes from the current policy, while the critic, regression loss, readout, architecture and training budget stay the same. Retaining the labels of about 24 recent rollouts (W = 50,000) is higher on seven of eight tasks, and using only current labels changes the suite return by -32.8% (paired-seed bootstrap 95% CI [-37.1%, -28.3%]). This finding is consistent with the policy-as-data interpretation because the stored data represent our policy; retaining the labels of several recent policies preserves more of the information used to construct it.
References
[Gao et al., 2026] (https://arxiv.org/abs/2603.10199)
[Tomar et al., 2022] (https://arxiv.org/abs/2005.09814v5)
We thank the reviewer for their feedback. However, we respectfully disagree with several of their concerns, which we address below.
1. MuJoCo-v5 higher baselines
Table 1. SAC and TD3 returns on MuJoCo v5 at 1M steps, from Table 2 of Zhang and Sharon [2025] and from our runs, with RDA2C (twin-Q labels) for reference.
Zhang and Sharon [2025] report SAC and TD3 performance comparable to our baselines. Specifically, their SAC results differ from ours by at most 11% on every task (higher on four tasks and lower on Ant), while their TD3 results differ by at most 8% (lower on four of five tasks). Moreover, RDA2C with twin-Q labels is within 10% of their SAC on every task. Thus, although the reviewer cites Zhang and Sharon [2025] as evidence of "substantially higher returns," their reported SAC and TD3 results are comparable to those in our paper.
Ding et al. [2026] use a modified SAC implementation and a different evaluation setup, making direct comparisons difficult. Specifically, they report the average of the best-performing results across three seeds (Table 1), whereas our experiments use ten seeds. The paper does not specify the evaluation frequency, number of evaluation episodes, or whether evaluation actions are deterministic, making it difficult to establish a comparable evaluation setup. Furthermore, Ding et al. [2026] implement SAC with a three-layer actor with gradient clipping of 1.0 and a 512-unit clipped double-Q critic with gradient clipping (Appendix F, Tables 7–8). In contrast, the standard SAC implementation uses two 256-unit hidden layers without gradient clipping. These architectural and evaluation differences prevent us from attributing differences in reported returns solely to the learning algorithm.
More importantly, Ding et al. [2026] propose an improved distributional critic rather than a new actor update. The authors explicitly describe their approach as a plug-and-play critic that can be integrated into existing RL algorithms without modifying their policy optimization procedure. RDA2C instead focuses on improving the actor update while holding the critic fixed. Thus, the two approaches address complementary components of the actor-critic framework, and RDA2C could in principle use value estimates from their critic.
2. What the twin-Q comparison claims
Parity with SAC is the paper's stated hypothesis, not a shortfall. Section 4.3 mentions, "Despite these differences, we hypothesize comparable performance under matched critic learning." SAC's actor step is an entropy-regularized follow-the-regularized-leader update [Tomar et al., 2022, Section 4.2], the same form as the regularized dual averaging step behind RDA2C, so with the critic shared the two methods differ only in how the cumulative score is formed. The twin-Q experiment confirms the hypothesis. Off-policy MDPO-KL similarly performs on par with SAC across all tasks [Tomar et al., 2022, Table 1].
Furthermore, it is important to distinguish improvements from policy optimization from those arising from critic design. Recent advances in continuous-control performance have largely come from improvements to the critic and network architecture rather than the actor update itself. For example, TD3 and SAC differ by approximately 4% on average at 1M steps [Fujimoto et al., 2023, Table 1], while methods such as TQC [Kuznetsov et al., 2020], CrossQ [Bhatt et al., 2024], and TD7 [Fujimoto et al., 2023] introduce distributional critics, critic normalization, and learned state-action representations. Moreover, these improvements are not necessarily tied to a particular actor update algorithm. CrossQ reports that its results transfer to TD3 [Bhatt et al., 2024], and Shahriari et al. [2022] find that their critic improvements are largely independent of the policy optimization method. Since RDA2C primarily introduces a new actor update, matching SAC under the same critic provides evidence that the proposed update is competitive, while improvements from more advanced critics remain complementary.