acceptodds
Under review as a conference paper at ICLR 2027

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Abstract

Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin- labels, RDA2C matches SAC at matched batch size and update frequency.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, policy optimization, dual averaging, mirror descent, actor-critic, experience replay

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nfjEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RDA2C, a replay-based actor-critic algorithm that learns a dual-score function through historical advantage regression. The method connects policy dual averaging with practical deep RL and provides theoretical analysis and experiments on MuJoCo and Atari.

Critical strengths

The algorithmic formulation is interesting and provides a novel perspective on actor optimization. The theoretical analysis offers useful insights into replay-based dual averaging.

Critical weaknesses

The MuJoCo evaluation primarily compares against PPO, while the twin-Q variant achieves only comparable performance to SAC. Recent studies on MuJoCo-v5 report substantially higher returns even for established baselines such as TD3 and SAC [1, 2]. Although cross-paper comparisons are not directly conclusive, these results raise concerns about the strength of the experimental evaluation. Moreover, comparisons with more recent continuous-control methods are missing, making it difficult to assess the practical competitiveness of RDA2C.

[1] Distributional Reinforcement Learning with Diffusion Bridge Critics. 2026. https://arxiv.org/abs/2602.05783

[2] AEAP: A Reinforcement Learning Actor Ensemble Algorithm with Adaptive Pruning. TMLR, 2025. https://openreview.net/forum?id=I5ymMVdmaR

Questions for authors

Can the authors provide direct comparisons with stronger modern continuous-control baselines under matched experimental settings?

4 comments

We thank the reviewer for their feedback. However, we respectfully disagree with several of their concerns, which we address below.

1. MuJoCo-v5 higher baselines

Table 1. SAC and TD3 returns on MuJoCo v5 at 1M steps, from Table 2 of Zhang and Sharon [2025] and from our runs, with RDA2C (twin-Q labels) for reference.

EnvironmentSAC in Zhang and Sharon [2025]SAC (paper)TD3 in Zhang and Sharon [2025]TD3 (new)RDA2C, twin-Q labels (paper)
HalfCheetah-v510,49310,11910,09710,1489,525
Hopper-v53,4873,3453,4993,2403,408
Walker2d-v54,5144,0803,9984,0684,505
Ant-v54,0984,3904,6304,7024,241
Humanoid-v55,3895,3765,0045,1635,196

Zhang and Sharon [2025] report SAC and TD3 performance comparable to our baselines. Specifically, their SAC results differ from ours by at most 11% on every task (higher on four tasks and lower on Ant), while their TD3 results differ by at most 8% (lower on four of five tasks). Moreover, RDA2C with twin-Q labels is within 10% of their SAC on every task. Thus, although the reviewer cites Zhang and Sharon [2025] as evidence of "substantially higher returns," their reported SAC and TD3 results are comparable to those in our paper.

Ding et al. [2026] use a modified SAC implementation and a different evaluation setup, making direct comparisons difficult. Specifically, they report the average of the best-performing results across three seeds (Table 1), whereas our experiments use ten seeds. The paper does not specify the evaluation frequency, number of evaluation episodes, or whether evaluation actions are deterministic, making it difficult to establish a comparable evaluation setup. Furthermore, Ding et al. [2026] implement SAC with a three-layer actor with gradient clipping of 1.0 and a 512-unit clipped double-Q critic with gradient clipping (Appendix F, Tables 7–8). In contrast, the standard SAC implementation uses two 256-unit hidden layers without gradient clipping. These architectural and evaluation differences prevent us from attributing differences in reported returns solely to the learning algorithm.

More importantly, Ding et al. [2026] propose an improved distributional critic rather than a new actor update. The authors explicitly describe their approach as a plug-and-play critic that can be integrated into existing RL algorithms without modifying their policy optimization procedure. RDA2C instead focuses on improving the actor update while holding the critic fixed. Thus, the two approaches address complementary components of the actor-critic framework, and RDA2C could in principle use value estimates from their critic.

2. What the twin-Q comparison claims

Parity with SAC is the paper's stated hypothesis, not a shortfall. Section 4.3 mentions, "Despite these differences, we hypothesize comparable performance under matched critic learning." SAC's actor step is an entropy-regularized follow-the-regularized-leader update [Tomar et al., 2022, Section 4.2], the same form as the regularized dual averaging step behind RDA2C, so with the critic shared the two methods differ only in how the cumulative score is formed. The twin-Q experiment confirms the hypothesis. Off-policy MDPO-KL similarly performs on par with SAC across all tasks [Tomar et al., 2022, Table 1].

Furthermore, it is important to distinguish improvements from policy optimization from those arising from critic design. Recent advances in continuous-control performance have largely come from improvements to the critic and network architecture rather than the actor update itself. For example, TD3 and SAC differ by approximately 4% on average at 1M steps [Fujimoto et al., 2023, Table 1], while methods such as TQC [Kuznetsov et al., 2020], CrossQ [Bhatt et al., 2024], and TD7 [Fujimoto et al., 2023] introduce distributional critics, critic normalization, and learned state-action representations. Moreover, these improvements are not necessarily tied to a particular actor update algorithm. CrossQ reports that its results transfer to TD3 [Bhatt et al., 2024], and Shahriari et al. [2022] find that their critic improvements are largely independent of the policy optimization method. Since RDA2C primarily introduces a new actor update, matching SAC under the same critic provides evidence that the proposed update is competitive, while improvements from more advanced critics remain complementary.

The point of the parity is that one actor update covers settings that SAC does not. As discussed in Section 4.3, "The same actor update can use either GAE labels from a value critic or advantage labels from twin-Q critics, and applies to both discrete and continuous action spaces." In contrast, standard continuous-action SAC relies on a soft Q-critic and reparameterized actions. Its actor update does not directly accommodate GAE labels, and extending SAC to discrete actions requires a separate formulation [Christodoulou, 2019] and stabilization techniques [Zhou et al., 2024]. RDA2C uses the same actor update across these settings. In settings where the standard SAC update is not directly applicable, RDA2C also outperforms PPO, demonstrating its broader applicability across different critic formulations and action spaces.

3. "Stronger modern continuous-control baselines"

The review does not name a single one. Its only examples, Ding et al. [2026] and Zhang and Sharon [2025], are a distributional critic and an ensemble of TD3-style actors. Neither is an alternative actor update, and either could be combined with RDA2C. A request for unnamed baselines cannot be answered or checked. If the reviewer has specific methods in mind, please name them, and we will run them under the paper's settings.

Again, it is important to distinguish improvements from policy optimization from those arising from critic design. Recent advances in continuous-control performance have largely come from improvements to the critic and network architecture rather than the actor update itself. The two modern baselines suggested by the reviewer [Ding et al., 2026; Zhang and Sharon, 2025] focus on different aspects of the actor-critic framework. Specifically, Ding et al. [2026] introduce a distributional critic based on diffusion bridges, while Zhang and Sharon [2025] propose an ensemble of TD3-style actors with adaptive pruning. Neither directly isolates the effect of replacing the actor update while holding the critic and other components fixed, which is the primary focus of RDA2C.

Furthermore, we ran TD3 using the paper's twin-Q setup. TD3 at its released defaults lands within a few percent of our SAC on every task except HumanoidStandup. RDA2C with twin-Q labels is higher than TD3 on five of eight tasks: suite +6.0% (paired-seed bootstrap 95% CI [+2.7%, +9.4%]). We used the same evaluation as the paper's SAC table (Table 5 of the paper): gymnasium MuJoCo v5, 10 seeds, evaluations every 50k steps with 20 deterministic episodes, and the best evaluation within 1M steps averaged over seeds.

Table 2. Best deterministic evaluation return within 1M steps (mean ± seed std, 10 seeds). The last column is RDA2C relative to TD3 with the paired-seed bootstrap 95% CI.

EnvironmentSAC (paper)TD3, released defaults (new)RDA2C, twin-Q labels (paper)RDA2C vs TD3
Ant-v54389.60 ± 1100.194701.97 ± 736.314240.94 ± 877.82-9.8% [-23.6%, +8.1%]
HalfCheetah-v510118.79 ± 894.3210148.21 ± 1386.259524.52 ± 1388.40-6.1% [-16.9%, +6.3%]
Hopper-v53345.29 ± 204.843240.20 ± 103.233408.27 ± 121.76+5.2% [+1.7%, +8.7%]
Humanoid-v55376.44 ± 175.725163.20 ± 154.305195.96 ± 314.71+0.6% [-3.6%, +4.5%]
HumanoidStandup-v5158309.50 ± 4088.25108280.81 ± 12189.50158784.44 ± 6868.90+46.6% [+36.7%, +57.2%]
InvertedPendulum-v51000.00 ± 0.001000.00 ± 0.001000.00 ± 0.000.0%
Pusher-v5-29.58 ± 1.55-28.35 ± 1.37-28.05 ± 1.51+1.0% [-3.4%, +5.4%]
Walker2d-v54079.87 ± 371.704067.83 ± 541.094504.91 ± 788.33+10.7% [-4.4%, +26.3%]
Suite average (paired-seed bootstrap 95% CI)+6.0% [+2.7%, +9.4%]

References

[Bhatt et al., 2024] (https://arxiv.org/abs/1902.05605)

[Christodoulou, 2019] (https://arxiv.org/abs/1910.07207)

[Ding et al., 2026] (https://arxiv.org/abs/2602.05783)

[Fujimoto et al., 2023] (https://arxiv.org/abs/2306.02451)

[Kuznetsov et al., 2020] (https://arxiv.org/abs/2005.04269)

[Shahriari et al., 2022] (https://arxiv.org/abs/2204.10256)

[Tomar et al., 2022] (https://arxiv.org/abs/2005.09814v5)

[Zhang and Sharon, 2025] (https://openreview.net/forum?id=I5ymMVdmaR)

[Zhou et al., 2024] (https://arxiv.org/abs/2209.10081)

Thank you for the additional experiments and clarifications. While matching SAC demonstrates that RDA2C is a viable alternative, it remains unclear what advantages justify adopting it over SAC. More relevant comparisons with methods such as MDPO-KL, or a matched replay-based advantage regression baseline without dual averaging, would help establish the specific contribution of the proposed actor update. My main concern regarding the practical benefit of RDA2C therefore remains.

We appreciate the clarification of your concern. The primary advantage of our algorithm over SAC is its flexibility in the selection of the critic and the method by which the critic is learned. SAC's actor update relies on a soft-Q function, which is typically learned using Bellman targets. In contrast, PPO learns a state-value function using mean squared error and constructs advantage estimates using generalized advantage estimation (GAE); these estimates cannot directly substitute for the soft-Q function in SAC's standard actor update. Our algorithm accommodates both SAC-style soft-Q supervision and PPO-style value-function and GAE supervision within the same actor update, eliminating the need to switch algorithms.

Regarding the comparisons, we agree that a comparison with MDPO-KL is relevant. However, the MDPO paper states that off-policy MDPO is equivalent to SAC when the KL regularizer uses the uniform policy, rather than the previous policy, as its reference [Tomar et al., 2022]. This equivalence motivated our decision to compare against SAC rather than both methods. We do include a comparison with the recent dual-averaging algorithm AAPDA [Gao et al., 2026], proposed in 2026, and RDA2C outperforms it on six of eight MuJoCo tasks. The principal benefit of our algorithm is that, by learning the dual variable, it facilitates replay-based dual-variable actor-critic learning without requiring importance sampling in the actor update. To clarify your suggested comparison, we would appreciate more details on the definition of a matched replay-based advantage regression baseline and the specific actor update you propose we compare against.

If your suggestion pertains to the impact of retaining past advantage estimates on performance, we address this in Appendix D.1.3, "Current-label control." This control changes label retention and nothing else: with W = 2,048 the buffer keeps exactly one rollout, so every regression target comes from the current policy, while the critic, regression loss, readout, architecture and training budget stay the same. Retaining the labels of about 24 recent rollouts (W = 50,000) is higher on seven of eight tasks, and using only current labels changes the suite return by -32.8% (paired-seed bootstrap 95% CI [-37.1%, -28.3%]). This finding is consistent with the policy-as-data interpretation because the stored data represent our policy; retaining the labels of several recent policies preserves more of the information used to construct it.

References

[Gao et al., 2026] (https://arxiv.org/abs/2603.10199)

[Tomar et al., 2022] (https://arxiv.org/abs/2005.09814v5)

Reviewer_8uQGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes RDA2C, an actor–critic method that stores historical advantage-labeled state–action samples and regresses a dual score model on replay. The policy is obtained directly from this score through an entropy mirror map, using a softmax for discrete actions and a quadratic-score Gaussian construction for continuous actions. The paper provides a conditional finite-time value-gap decomposition and evaluates GAE and twin-Q variants against PPO, SAC, and AAPDA on MuJoCo and Atari.

Critical strengths

The method provides a practical actor formulation that accepts different sources of advantage labels. Using the same score-regression and policy-readout construction with both GAE and twin-Q critics, and with discrete and continuous actions, is a useful implementation result.

The empirical evaluation is reasonably substantial, with ten training seeds, several baselines, and ablations of the replay window, regularization, regression loss, and network capacity. The current-label control is particularly informative: retaining historical labels improves performance on seven of eight MuJoCo tasks.

The analysis explicitly distinguishes the ideal historical score from the behavior-weighted target produced by replay regression. Keeping fitting error, critic bias, replay concentration, and coverage visible makes the limitations easier to inspect than an analysis that assumes exact historical averaging.

Critical weaknesses

The contribution is not adequately positioned against a directly relevant prior work. Lazic et al. (ICML 2021), “Improved Regret Bound and Experience Replay in Regularized Policy Iteration,” already propose training a single network on historical replay labels to approximate the average of past Q-functions and construct a Boltzmann policy. The paper discusses the original POLITEX but omits this replay-based extension. The additional contributions of advantage labels, windowing, regularization, and the practical readout need to be distinguished from this existing construction before the central novelty claim can be assessed.

The theoretical guarantee remains conditional on properties that are not established for the implemented algorithm. Assumption F.2 requires uniformly decreasing policy movement at a Cπ/k rate, which Appendix F.8 explicitly says was not verified. The replay concentration rate is also assumed rather than derived for the adaptive, trajectory-based regression setting. Fixed replay windows, clipped regression, label normalization, continuous actions, and autotuned temperatures further limit the direct coverage of the theorem.

Two explanations used to connect the implementation to the assumptions are mathematically incorrect. Appendix F.8 states that clipped regression bounds prediction movement by the clip radius, but the loss in Appendix A.4 is not a hard prediction constraint and permits minimizers outside that interval. It also states that positive replay mass at every comparator-supported state–action pair implies finite coverage. This is insufficient on an infinite state space, where the density ratio can be unbounded despite full support. These errors do not directly invalidate the conditional theorem, but weaken its claimed applicability.

The experiments do not fully isolate the proposed actor mechanism. In MuJoCo, RDA2C has state-dependent standard deviations while PPO uses global standard deviations. On Atari, the methods differ in shared versus separate feature trunks and several optimization settings. The existing controls are helpful, but do not remove all these differences. Moreover, the strongest practical results use clipped regression, while the main theoretical construction analyzes squared-loss regression. The evidence therefore supports a useful algorithm configuration more clearly than the specific theoretical explanation offered for it.

Questions for authors

How does RDA2C differ substantively from the replay-based POLITEX implementation of Lazic et al. (ICML 2021)? Which theoretical or empirical contribution remains after accounting for that construction?

Can the Cπ/k policy-path stability and replay concentration conditions be established or empirically checked for the reported implementation? Which parts of the practical training procedure are covered by the resulting guarantee?

What precisely is meant by the claim that clipped regression bounds prediction movement? The stated loss permits predictions outside the clipping interval; does correcting this claim change the justification for path stability?

Is the coverage discussion intended to assume a finite state space? Otherwise, what uniform density-ratio or exploration conditions ensure that the coverage coefficient is finite?

Do the gains remain when PPO uses state-dependent standard deviations and the Atari comparisons match trunk sharing and optimization settings? How much improvement is attributable to historical score regression under these controls?

Reviewer_GPAQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RDA2C, which stores advantage-labeled samples, fits a dual score by regression, and obtains the policy through an entropy-based readout. It connects this procedure to policy dual averaging and evaluates variants using GAE and twin-Q labels. Experiments cover eight MuJoCo tasks and twelve Atari games, with comparisons against PPO, SAC, and AAPDA.

Critical strengths
  1. Representing policy updates through regression on stored advantage labels is an interesting approach. The connection to dual averaging gives a clear motivation for retaining historical feedback, and the same framework accommodates both GAE and twin-Q labels.

  2. The empirical results are encouraging: RDA2C improves over PPO and AAPDA on most evaluated tasks and performs comparably to SAC. Ten-seed experiments, uncertainty estimates, and ablations on replay size, temperature, and regression loss provide useful support.

Critical weaknesses
  1. In continuous control, the Gaussian readout requires a concave diagonal quadratic score, but the advantage function being fitted need not have this shape. Minimizing regression error under the replay distribution is not generally equivalent to finding the best Gaussian policy. The paper provides limited evidence on how this approximation affects policy quality, particularly for multimodal advantages or interactions between action dimensions.

  2. Some comparisons do not fully isolate the proposed update. On MuJoCo, RDA2C uses state-dependent policy variance while PPO uses a global standard deviation. On Atari, network sharing and several training settings also differ. These differences make it difficult to attribute the reported gains specifically to the proposed policy update.

Questions for authors
  1. Can you provide evidence that the quadratic score approximation preserves useful policy improvements when the advantage is not well approximated by a concave diagonal quadratic? A controlled continuous-action example would help clarify this limitation.

  2. How much of the improvement over PPO remains when policy variance parameterization is matched on MuJoCo, and network sharing and training settings are matched on Atari?

3 comments

We thank the reviewer for their feedback. We ran the experiments both weaknesses ask for.

1. The quadratic score when the advantage is not a concave diagonal quadratic

This is a good point, and it is important to distinguish between the restrictions imposed by the policy class and those introduced by how the policy is learned. PPO and SAC, like any method with a diagonal Gaussian policy, restrict the policy to the same class as RDA2C: diagonal Gaussians. Both algorithms make that restriction at the level of the policy. PPO searches within the Gaussian class by policy gradient, and SAC fits its Gaussian policy to the softmax of its Q-function by minimizing a KL divergence [Haarnoja et al., 2018]. RDA2C makes the same restriction at the level of the score function. It fits the advantage with quadratic functions by least squares. Doing it at the score function level makes the approximation explicit.

To investigate whether the concave diagonal quadratic assumption limits policy quality, we designed several controlled bandit experiments where the true advantage function is known. We consider three examples in a one-state bandit setting with 2-D actions in [−1,1]2[-1,1]^2, each with a known advantage function designed to test a different degree of misspecification:

  • (a) Diagonal quadratic, A(a)=−2(a1−0.3)2−(a2+0.4)2A(a) = -2(a_1 - 0.3)^2 - (a_2 + 0.4)^2, with maximum 0 at (0.30, -0.40). This is a setting where the diagonal quadratic readout can represent the advantage exactly.

  • (b) Non-diagonal quadratic, A(a)=−(a1−0.5a2−0.3)2−0.05(0.5a1+a2−0.6)2A(a) = -(a_1 - 0.5a_2 - 0.3)^2 - 0.05(0.5a_1 + a_2 - 0.6)^2, where the best a1a_1 depends on a2a_2. This function contains interactions between action dimensions that cannot be represented by a diagonal quadratic.

  • (c) Bimodal nonquadratic, A(a)=0.6exp⁡(−∥a−(−0.6,−0.6)∥2/(2⋅0.32))+exp⁡(−∥a−(0.6,0.6)∥2/(2⋅0.32))A(a) = 0.6\exp(-\|a - (-0.6,-0.6)\|^2 / (2 \cdot 0.3^2)) + \exp(-\|a - (0.6,0.6)\|^2 / (2 \cdot 0.3^2)), with two bumps of height 0.6 and 1.0 and maximum 1 at (0.6, 0.6). This tests whether the quadratic readout can identify a high-advantage mode despite being unable to represent the multimodal landscape.

The RDA2C arm uses the readout, log-std interval, temperature schedule (λ=0.05\lambda = 0.05, β=1\beta = 1), 24-round window, regression budget and clipped loss of our MuJoCo code. Labels are exact, or exact plus Gaussian noise with standard deviation equal to half the spread of AA over uniformly random actions. We compare against a full-covariance quadratic score, a plain squared-loss variant, and PPO with a global standard deviation and entropy coefficient of 0.05. All methods are initialized from the same policy and evaluated over 10 seeds, with 200 rounds per seed.

Table 1. Policy after 200 rounds with noisy labels, means over 10 seeds. E[A]E[A] is the expected advantage of the induced policy (maximum 0 on the non-diagonal quadratic, 1 on the bimodal nonquadratic).

Advantage functionArmFinal meanE[A]E[A]Seeds at the higher mode
Non-diagonal quadratic (0.48, 0.36)RDA2C, diagonal score(0.42, 0.25)-0.017
Non-diagonal quadraticRDA2C, plain squared loss(0.43, 0.27)-0.017
Non-diagonal quadraticPPO, global std, entropy 0.05(0.42, 0.23)-0.016
Non-diagonal quadraticRDA2C, full-covariance score(0.46, 0.34), correlation 0.72-0.011
Bimodal nonquadratic (0.60, 0.60)RDA2C, diagonal score(0.60, 0.60)0.99310 of 10
Bimodal nonquadraticRDA2C, plain squared loss(0.60, 0.60) in 9 seeds, (-0.59, -0.61) in 10.9539 of 10
Bimodal nonquadraticRDA2C, full-covariance score(0.60, 0.60)0.99310 of 10
Bimodal nonquadraticPPO, global std, entropy 0.05(0.60, 0.60)0.99310 of 10

Diagonal quadratic. All four arms (RDA2C, plain squared loss, PPO and full covariance) reach the argmax (0.30, -0.40).

Non-diagonal quadratic. With exact labels, all four arms, PPO included, reach the argmax (0.48, 0.36) by round 100. The diagonal quadratic explains only 68–74% of the variance in the advantage labels under the replay distribution, compared with 100% for the full quadratic. After 200 rounds with noisy labels, the diagonal arm and PPO with a global standard deviation and the same entropy bonus end at nearly the same expected advantage (-0.017 and -0.016), and the full-covariance score is closer to the argmax (-0.011).

Bimodal nonquadratic. None of our bimodal runs, for any arm including PPO, ended between the two bumps. The RDA2C fit starts at the origin, where the function is nearly flat. It first widens until both bumps show up in the replay, then moves to one bump and sharpens. The default configuration chose the higher bump in 10 of 10 seeds, with exact and with noisy labels, for every window from one round to the full history at entropy floors 0.01 and 0.05, and in at least 9 of 10 seeds at 0.2. With noisy labels, plain squared loss sent one seed to the lower bump.

Comparing the techniques. When the diagonal class can represent the advantage (a), all four arms reach the argmax. When the action dimensions interact (b), the full-covariance score, which can represent the interaction, ends closest to the argmax (-0.011). The diagonal score and PPO, whose Gaussian policies are both diagonal, end at about the same level (-0.017 and -0.016), and plain squared loss matches the clipped loss (-0.017). On the bimodal advantage (c), the diagonal score, the full-covariance score and PPO find the higher mode in all 10 seeds, and plain squared loss finds it in 9. The diagonal restriction therefore costs RDA2C about as much as it costs PPO with the same policy class, and only when the action dimensions interact.

2. Separating the effect of the state-dependent actor log-std

We agree that isolating the difference in policy variances is important for comparing our proposed algorithm. To isolate the state-dependent standard deviation factor, we introduce a PPO baseline with state-dependent standard deviation. All other experimental settings remain unchanged, including the eight MuJoCo tasks, 10 seeds, and 1M environment steps per seed.

Table 2. Final running return (mean ± seed std, 10 seeds), MuJoCo GAE setting.

EnvironmentPPO, global std (paper)PPO, state-dependent std (new)RDA2C (paper)
Ant-v51953 ± 3722710 ± 3732831 ± 583
HalfCheetah-v51864 ± 784078 ± 20627099 ± 645
Hopper-v52333 ± 5901617 ± 7412607 ± 272
Humanoid-v52370 ± 976893 ± 2293768 ± 1290
HumanoidStandup-v5170412 ± 8967146107 ± 20176156068 ± 9614
InvertedPendulum-v5989 ± 36947 ± 167999.2 ± 2.6
Pusher-v5-33.5 ± 2.2-34.5 ± 3.4-30.6 ± 3.0
Walker2d-v53626 ± 6042231 ± 5163546 ± 659

RDA2C outperforms this new PPO baseline on all eight tasks. Furthermore, the new PPO baseline with state-dependent standard deviation outperforms the original PPO baseline with global standard deviation on two tasks, Ant and HalfCheetah, but performs worse on the remaining six tasks. Overall, the two PPO variants achieve comparable suite-average performance (+0.6%), although the state-dependent variant has a lower median performance across tasks (−9.2%).

3. Separating the effect of actor and critic network sharing

We agree that isolating the effect of network sharing and training settings is important for comparing our proposed algorithm. We ran PPO on twelve games in two configurations. The first uses two independent Nature-CNN trunks for actor and critic, as in RDA2C. The second keeps the shared trunk with RDA2C's rollout and optimization settings (4 environments x 256 steps, 2 epochs, GAE 0.85, gradient-norm clip 2, no value-loss clipping). It keeps PPO's ratio clip and entropy bonus, since RDA2C has no counterpart. We compare both with the default PPO's shared trunk (Table 9).

Table 3. PPO final running return (mean ± seed std). The first column is the PPO column of the paper's Table 9.

GamePPO (paper)PPO, separate trunks (new)PPO at RDA2C's settings (new)
Asterix4061 ± 10143217 ± 3925638 ± 2001
BeamRider2540 ± 316687 ± 722137 ± 1453
Breakout422 ± 24385 ± 30251 ± 208
Enduro980 ± 135558 ± 841208 ± 526
Freeway32.7 ± 0.617.3 ± 9.128.1 ± 10.4
Gravitar686 ± 1871097 ± 492274 ± 62
MsPacman2085 ± 2681857 ± 3192198 ± 325
Pong20.6 ± 0.7-8.2 ± 19.320.0 ± 0.8
PrivateEye70.1 ± 47.199.0 ± 2.00.0 ± 0.0
Qbert15804 ± 199316662 ± 98713166 ± 1707
Seaquest1541 ± 4271542 ± 382904 ± 36
SpaceInvaders998 ± 187847 ± 109623 ± 71

Relative to the default PPO configuration, separate trunks lower PPO performance by 21.0% [-28.5%, -13.1%] (fixed-suite mean change, paired-seed bootstrap 95% CI; median -13.0%). With RDA2C's settings, PPO performance is lower by 21.8% [-28.7%, -15.0%] (median -16.3%). Thus, modifying PPO to use either separate trunks or RDA2C's optimization settings reduces its suite-average performance by approximately 20% relative to the default PPO configuration. RDA2C stays above PPO on at least eight of twelve games under every PPO configuration we ran, with median per-game improvements of +32.9% over the paper's PPO and +68.3% over separate trunks.

In the revision, we will add the state-dependent-std column to the paper's Table 4. We will also add the Atari PPO controls as a new table in Appendix B, with a sentence in Section 6 pointing to it.

References

[Haarnoja et al., 2018] (https://arxiv.org/abs/1801.01290)

Thank the authors for the detailed responses and analysis.

My concerns have been resolved. I'm keeping my score.

Comments on the paper

7 comments

We thank the reviewer for their feedback. We appreciate your openness to discussing these points. Your review engaged closely with the analysis, noting that keeping fitting error, critic bias, replay concentration and coverage visible makes the limitations easier to inspect, and that the current-label control is particularly informative. Your questions helped us sharpen both the positioning and the analysis. Sections 1 to 3 answer your main concerns, Q1 to Q3, in order: the relation to Lazic et al. [2021], what the guarantee covers, and what the clip does. Sections 4 and 5 answer Q4 and Q5.

1. Lazic et al. [2021] and what is new in RDA2C (Q1)

Thank you for pointing us to this paper. Both methods share the idea of fitting a single network to accumulated historical labels and extracting a policy through a softmax transformation. However, they differ in their derivations, regression targets, and implementations. Furthermore, the empirical evaluation of Lazic et al. [2021] is limited to small-scale environments.

Derivation. Lazic et al. derive their method as an approximation to mirror descent, introducing replay-based regression through the heuristic of minimizing the average of past squared losses. In contrast, RDA2C is derived directly from dual averaging [Nesterov, 2009], where the optimization objective depends on the average of past functional gradients. This motivates fitting a single model to the average of past advantage estimates using replay. Although both approaches recover the same policy with exact Q-functions in the tabular setting, they lead to different regression objectives under function approximation. Furthermore, RDA2C uses regularized dual averaging [Xiao, 2010], which introduces a persistent regularizer and yields an entropy floor through the temperature schedule αt=λ+β/t\alpha_t=\lambda+\beta/t. In contrast, the temperature in Lazic et al., 1/(ηk)1/(\eta k), decays to zero.

Implementation. RDA2C differs from Lazic et al. and POLITEX [Abbasi-Yadkori et al., 2019] in how it uses replay, constructs regression targets, and extends to continuous actions. RDA2C fits a single score model using a sliding window of replayed advantage labels, with the window size motivated by our drift-versus-concentration bound (Wt∝t2/3W_t \propto t^{2/3}). In contrast, Lazic et al. consider retaining the full replay history, uniformly discarding samples, or subsampling data from each phase. POLITEX's Atari implementation instead maintains the ten most recently fitted policy networks, rather than a sliding window of data used to train a single model. RDA2C also supports advantage estimates from different critic formulations, including GAE and twin-Q critics, both of which we evaluate experimentally, whereas Lazic et al. use Monte Carlo returns. Although exact advantage and Q values induce the same softmax policy, their estimated labels can differ in variance and in the critic formulations used to obtain them. Finally and most importantly, RDA2C extends the regression-based policy update to continuous actions through a quadratic score function and provides an error decomposition that accounts for critic bias and replay coverage. Our ablations further demonstrate the empirical importance of both the entropy floor and replay window.

Scope. Lazic et al.'s experiments use discretized Cart-pole and Ball-in-cup with Fourier features and one-hidden-layer networks. Ours cover eight continuous-action MuJoCo tasks and twelve Atari games from pixels, against PPO, SAC and TD3. Furthermore, in Appendix E, xgb_boosting is equivalent to POLITEX's explicit sum and xgb_dagger is Lazic et al.'s single-model fit.

To explore the impact of window size, we swept W on the GAE MuJoCo configuration of the paper's Table 4. As predicted by our theoretical analysis (part of our paper's contribution), the full history, as used in POLITEX and Lazic et al., is far from optimal. W = 1,000,000 never drops a sample in a 1M-step run, so it is full-history uniform replay in our configuration. Relative changes are measured against W = 50,000.

Table 1a. Final running return (mean ± seed std, 10 seeds), GAE MuJoCo. We will put the W = 200,000 column in the revised appendix.

EnvironmentW = 2,048W = 8,192W = 50,000 (paper)W = 1,000,000 (full history)
Ant-v5884 ± 4122721 ± 6112831 ± 5831358 ± 229
HalfCheetah-v55162 ± 12986717 ± 8037099 ± 6452829 ± 449
Hopper-v52112 ± 8022634 ± 2982607 ± 272517 ± 225
Humanoid-v5500 ± 753544 ± 10723768 ± 1290698 ± 25
HumanoidStandup-v594062 ± 24669141485 ± 27703156068 ± 9614115788 ± 15560
InvertedPendulum-v51000.0 ± 0.0999.9 ± 0.3999.2 ± 2.6938.7 ± 175.0
Pusher-v5-31.9 ± 1.3-31.6 ± 2.4-30.6 ± 3.0-68.0 ± 8.7
Walker2d-v52973 ± 7072899 ± 10833546 ± 6591150 ± 331

Table 1b. Suite statistics relative to W = 50,000 (the paper's paired-seed bootstrap, 95% CI).

WindowSuite-average change (95% CI)Tasks above W = 50,000Seeds above the same-seed W = 50,000 run
W = 2,048-32.8% [-37.1%, -28.3%]1 of 815 of 80
W = 8,192-5.6% [-11.0%, -0.3%]2 of 833 of 80
W = 200,000-31.2% [-35.0%, -27.0%]1 of 89 of 80
W = 1,000,000 (full history)-62.0% [-66.0%, -57.8%]0 of 82 of 80

A replay window of W=50,000W=50{,}000 achieves the highest mean return on six of eight MuJoCo tasks. In contrast, retaining the full replay history performs worse on all eight tasks and outperforms matched PPO on only one task, compared with six for W=50,000W=50{,}000.

To more closely match Lazic et al., we also evaluate a full-history variant without the entropy floor, so that the temperature decays as β/k\beta/k, analogous to their 1/(ηk)1/(\eta k). We additionally scale the regression budget with the buffer size to maintain the same number of regression updates per stored label as with W=50,000W=50{,}000. To isolate the effect of replay history, we compare this variant against W=50,000W=50{,}000 with the entropy floor also removed. Full-history replay reduces suite-average performance by 42.0% (95% CI: [-52.4%, -32.1%]) and performs worse on all eight tasks. Keeping the regression budget fixed produces a similar reduction of 42.0% (95% CI: [-51.2%, -33.3%]), indicating that increasing the regression budget does not recover the lost performance (difference: -0.6%, 95% CI: [-8.5%, +6.7%]).

The entropy floor is also important independently. With the replay window fixed, removing the floor reduces suite-average performance by 45.5% (95% CI: [-51.0%, -39.9%]), as reported in Appendix D.3. Together, these results demonstrate that both the finite replay window and persistent entropy regularization contribute substantially to RDA2C's performance, with removing either component reducing suite-average returns by more than 40%.

In the revision, we will cite the paper in Sections 1 and 4.3 and Appendices E and G.2 and add the sweep as a new subsection of Appendix D.1.

2. Path stability and what the guarantee covers (Q2)

Like other guarantees for policy optimization with function approximation, the theorem is conditional. Each condition is a standard one, and path stability can be enforced through the step size. The gap between the analyzed update and the code is also the usual one. POLITEX analyzes fixed phases and the full sum of past estimates, while its Atari implementation keeps the 10 most recent networks, with phases that grow over training. Lazic et al. prove their bound for linear features, and their single-network replay version, the one closest to RDA2C, is motivated by that analysis, not covered by it.

We give more detail about each of our assumptions and why we believe they are reasonable.

Fitting error. Assumption F.8 (item 1) asks only that the fitting error stay bounded on average over rounds, and the bound carries it as a floor. Analyses of policy optimization with function approximation take the same input. Agarwal et al. [2021] assume a bound on the regression error at each iteration of Q-NPG and derive it only for linear features. We know of no result that bounds the fitting error of a network of practical size trained by SGD on RL data, so any guarantee for a deep actor has to assume it in some form.

Concentration. Assumption F.8 (item 2) is the usual N−1/2N^{-1/2} rate for NN bounded labels, and adaptive, trajectory-based data change only its constant. The data of round kk are collected by πk\pi_k, which is fixed by the data before that round. Let Fk−1\mathcal F_{k-1} hold everything up to round k−1k-1, including πk\pi_k and the critic that labels round kk. For fixed score functions Z,Z′Z, Z' bounded by BB, labels bounded by GAG_A, and the nn samples (sk,i,ak,i,yk,i)(s_{k,i}, a_{k,i}, y_{k,i}) of round kk, let

hk=1n∑i=1n[(Z(sk,i,ak,i)−yk,i)2−(Z′(sk,i,ak,i)−yk,i)2],Dk=hk−E[hk∣Fk−1].h_k = \frac{1}{n}\sum_{i=1}^{n}\Big[\big(Z(s_{k,i},a_{k,i})-y_{k,i}\big)^2-\big(Z'(s_{k,i},a_{k,i})-y_{k,i}\big)^2\Big], \qquad D_k = h_k-\mathbb E[h_k \mid \mathcal F_{k-1}].

Then E[Dk∣Fk−1]=0\mathbb E[D_k \mid \mathcal F_{k-1}] = 0 and ∣Dk∣≤8B(B+GA)|D_k| \le 8B(B+G_A), however strongly the samples inside the round depend on each other, and

Var(hk∣Fk−1)≤4(B+GA)2 ∥Z−Z′∥2,μk2.\mathrm{Var}(h_k \mid \mathcal F_{k-1}) \le 4(B+G_A)^2\,\|Z-Z'\|_{2,\mu_k}^2.

Freedman's inequality [Freedman, 1975] and a union bound over a cover of the function class then give

∥Z~t−Z^t∥2,μt=O(log⁡t/Wt)\|\widetilde Z_t-\widehat Z_t\|_{2,\mu_t} = O(\sqrt{\log t / W_t})

when the class contains the target Z^t\widehat Z_t. Agarwal et al. [2012] use the same argument for least squares on adaptively collected bandit data, where this realizability condition is the standard assumption. Counting rounds instead of samples moves the dependence along a trajectory into the constant. That constant depends on the capacity of the network class, and known capacity bounds for networks of practical size make it far too large to use, which is why we keep CvarC_{\text{var}} as an assumption.

Path stability. A limit on how fast the learned policy moves is the usual way to handle data generated by the learned policies themselves. Two-timescale actor-critic analyses impose it through actor step sizes that shrink faster than the critic's [Konda and Tsitsiklis, 2003], and trust-region and mirror-descent methods bound every step by construction. The Cπ/kC_\pi/k rate in Assumption F.2 is stronger than required by the theorem and corresponds to the special case of full-history replay within a more general convergence rate. If

sup⁡s∥πk+1−πk∥1≤Ck−a\sup_s \|\pi_{k+1}-\pi_k\|_1 \le Ck^{-a}

with a>0a>0, the same proof with Wt∝t2a/3W_t \propto t^{2a/3} gives a rate of t−a/3t^{-a/3}, and the drift term goes to zero. (a=1a=1 gives back t−1/3t^{-1/3}. For a<3/2a<3/2, t2a/3t^{2a/3} grows more slowly than tt, so the window stays shorter than the full history.) With the window of the analysis, the concentration term also goes to zero. The implementation's constant window of WW rounds has a fixed concentration floor, O(Cvar/W)O(C_{\text{var}}/\sqrt{W}).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.