acceptodds
Withdrawn from ICLR 2027

AdaPPO: Resolving Dynamics Asymmetry in LLM Alignment via Adaptive Gating

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Proximal policy optimization (PPO) is widely used for aligning large language models, but it typically updates the actor and critic at every step despite their different optimization timescales. We observe and analyze a timescale asymmetry: under a local Polyak–Łojasiewicz condition, the critic converges exponentially, whereas the actor admits a sublinear stochastic convergence bound. This suggests that synchronous updates can incur redundant critic computation once the critic enters a low-gradient regime, while freezing it for too long can make its value estimates stale and degrade actor optimization. We therefore bound critic error accumulation under bounded value drift, which motivates a staleness-aware freeze schedule. Based on this analysis, AdaPPO adaptively refreshes the critic using three signals—performance, KL divergence, and entropy—evaluated jointly, with a maximum-freeze safeguard. We further introduce AdaPPO*, which augments the critic with privileged information. Experiments on four RLVR benchmarks show that AdaPPO reduces FLOPs by up to 25% and reaches target performance up to faster in training steps. Our code is available at https://anonymous.4open.science/r/AdaPPO-E375.

Primary area:
optimization
Keywords:
Proximal Policy Optimization, Large Language Models, Optimization Theory, Adaptive Critic Scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZH49OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The gives theoretical results regarding an asymmetry in the convergence rates of actor-critic in RL: the critic converges exponentially while the actor converges linearly. They propose to leverage this by updating the critic less often and thus increasing training efficiency. They provide experiments.

Critical strengths
  • The empirical observation that the critic converges faster than the actor, and that this can be leveraged to improve training efficiency, in the context of llm post training is, to the best of my knowledge, both novel and relevant for practitioners.
Critical weaknesses
  1. The theoretical results do not justify the asymmetry but rather assume it. That is, the convergence rates are standard/general results that stem from the PL assumption in the case of the critic and from the smoothness in the case of the actor. In my understanding the same results apply to any training objective satisfying these assumptions and has nothing to do with actor-critic. Therefore, the paper fails to address where this assymetry in actor and critic convergence stems from, but rather presents assumptions that imply it, rendering most results tautological. If the theoretical analysis of assymetry in actor critic is a major contribution of the manuscript, and the theory plays such an important role in its presentation, then why the PL condition would hold for the critic (which is deferred to an appendix) but not for the actor (as far as I know, missing from the paper) should be justified and discussed in the main body.

  2. Given their popularity, I would find the inclusion of at least one critic-free method in the experimental section pertinent

  3. Regarding presentation, I think that the description of the proposed method intertwined with the theoretical results lacks clarity. Having a separate proposed method / algorithmic section in the body could help.

  4. All of the experiment results in section 5 all focus on accuracy vs flops, whereas the main result pertains actor and critic convergence, so some additional convergence plots (apart from the one in the first page) and analysis could be helpful.

Questions for authors

How do you justify the assymetry in assumptions for the critic and actor and how these relate to their learning objectives (see weakness 1)?

Reviewer_ZA1HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces AdaPPO, a variant of PPO that keeps the critic frozen unless a heuristic (an AND of performance-stagnation, KL, and entropy conditions) is satisfied or a maximum limit is hit. The rule is motivated by a claimed timescale asymmetry between critic and actor convergence and a bound on critic staleness. They also propose a variant AdaPPO* that provides the critic with privileged information. The experiments report that their method reduces FLOPs and achieves faster convergence on 4 benchmarks.

Critical strengths
  • The paper’s core question, when the critic actually needs to be updated, is relevant to the compute cost of critic-based RL methods such as PPO. Reducing critic FLOPs would narrow the cost gap between critic-based methods and critic-free alternatives such as GRPO.
  • The proposed heuristic for determining if critic should be updated or not is simple and easy to implement.
Critical weaknesses
  • Theory

    • The Theory is just for motivation, as also said by the authors (lines 225-226), and is not critical or relevant to the proposed method. In fact, the theory can be entirely removed without changing anything about the proposed method.
    • Two time-scale actor-critic analysis is classical and exists in the RL literature [1,2]. Furthermore, the derived convergence rates follow from mismatched assumptions (PL for critic and actor under nonconvex stochastic optimization), not for anything specific to actors and critics.
    • A lot of theoretical assumptions are questionable and do not transfer to the actual setting in which their AdaPPO method was applied
      • "we assume the critic regression target is the ideal value function V∗V^*" and "GAE or TD targets do not affect the convergence rate". The actual targets used are noisy and bootstrapped, which makes these claims questionable
      • "Under the pretrained backbone, the residual term decays rapidly". This only happens when close to convergence.
      • The analysis (Sec 2.1) drops clipping and further claims in Assumption 2.2, that the gradient of this unclipped objective has bounded variance. This disregards the importance ratio that can blow up as the data becomes more and more off-policy. Furthermore, in thier analysis, the notation for this unclipped objective (J~A\tilde{J}_{A}) is switched back to the notation for the original objective (JAJ_{A}), which seems misleading (line 165).
      • Theorem 3.1 models staleness only through variance, but with GAE, a stale critic mainly introduces bias, which the paper acknowledges but doesn’t control. Also, Its quantities (ϵ0\epsilon_{0}, δsafe\delta_{safe}, ρ\rho) are never estimated or used.
  • Method

    • The performance gate measures the performance of the policy and not the critic (Sec. 3.3). It is unclear how this tells anything about the staleness of the critic. A stall in performance improvement cannot distinguish a converged policy (no refresh needed) from a stale critic (refresh needed).
    • In fact, none of the three gates measures critic staleness. The KL gate tracks per-step policy change rather than drift accumulated since the last refresh, and the paper concedes it does not bound the value drift (Sec. 3.3); the performance and entropy gates are unrelated to the critic. Combined with the AND rule and TmaxT_{max}, they act as a throttle on refresh frequency rather than a staleness detector.
    • prediction error of the critic on fresh rollouts would have served as a more principled gating mechansim, that accounts for the staleness of the critic.
    • Decoupled update frameworks already exist (TD3, PPG, DCPG). So the papers novelty is limited to the proposed heuristic gating mechanism.
  • Experiments

    • Number of seeds used not specified.
    • No baselines with fixed refresh or random refresh, that have a matched update ratio with AdaPPO. This will inform if the heuristic gating scheme does something non trivial over simpler gating mechanisms (fixed/random refresh), given that the heuristics do not directly tackle the staleness of the critic.
    • AdaPPO* confounds privileged information with critic scheduling. There is no PPO-PI variant to address this confound.
    • Gate ablation is missing. Its unclear what the importance of each of three heuristics in the gating mechanism is.
  • Misc

    • Appendix A is empty and has wrongly been referenced in multiple segments in the paper

[1] Stochastic approximation with two time scales, Borkar et al

[2] Actor-critic algorithms, Konda and Tsitsiklis

Questions for authors

See Weaknesses

Reviewer_5N4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the scheduling of actor and critic updates in PPO training for LLMs. The authors argue that the critic reaches a stage of limited additional benefit faster than the actor, making critic updates at every training step potentially wasteful. They propose AdaPPO, which jointly monitors performance changes, policy KL divergence, and policy entropy to decide when to update the critic. Additional safeguards limit freezing and trigger updates under specified conditions. An extension, AdaPPO*, provides privileged information to the critic to improve value estimation.

Experiments on four RLVR benchmarks report improved trade-offs between performance and computation. On GSM8K, AdaPPO updates the critic in approximately 54% of training steps while reaching accuracy comparable to PPO with fewer estimated actor–critic FLOPs. The theoretical analysis motivates selective updating through a local PL assumption for critic optimization, a stochastic convergence bound for the actor, and a bound on critic error accumulation during freezing.

Critical strengths

Strength 1: The problem has practical relevance. Critic training contributes substantially to PPO’s computational cost. Understanding when critic updates are necessary could improve training efficiency while preserving the benefits of value estimation.

Strength 2: The method is straightforward and the experiments have reasonable breadth. AdaPPO can be incorporated into an existing PPO pipeline as an update scheduling mechanism. Results across four tasks, multiple models and scales, and gating ablations provide promising empirical evidence.

Strength 3: 1. The paper acknowledges important limitations of its theory. The authors explicitly state that the staleness bound does not establish convergence of the actual GAE-based PPO algorithm and that the gating signals are operational proxies. These qualifications are appropriate.

Critical weaknesses

Weaknesses 1: The experiments do not sufficiently establish the benefit of adaptive scheduling over simply updating the critic less frequently.

Reducing critic update frequency has been explored previously. For example, DCPG combines less frequent critic optimization with accumulated training data. The central empirical question is therefore whether adaptively selecting update times provides additional value over simpler schedules.

In the GSM8K comparison, AdaPPO updates the critic in approximately 54% of steps, whereas the simplified DCPG baseline uses approximately 70%. This comparison changes both update timing and the total update budget, making it difficult to isolate the contribution of adaptive gating. Furthermore, the appendix states that DCPG’s value regularization is simplified, limiting what the comparison establishes about the original method.

The most decisive control would compare AdaPPO against periodic and random update schedules with matched critic update budgets and a reasonable tuning budget for the periodic baseline. The existing gate ablations do not fully replace this comparison.

The appendix also introduces independent refresh conditions based on the maximum freeze interval, a KL ceiling, a sharp KL increase, and an entropy floor. The paper should report how many updates each mechanism triggers and evaluate a version using only these safeguards. Otherwise, the independent contribution of the joint three-gate mechanism remains unclear.

Weaknesses 2: The theoretical analysis does not establish the practical optimization timescale asymmetry emphasized by the paper.

Section 3.1 derives an exponential convergence bound for the critic under a local PL condition, a fixed ideal regression target, and gradient descent. Section 3.2 derives a sublinear bound for the actor under general stochastic nonconvex optimization assumptions. These results rely on different structural assumptions and optimization conditions.

Crucially, a sublinear upper bound for the actor does not establish that the actor actually converges only at a sublinear rate. A substantially faster process can also satisfy that upper bound. Consequently, comparing these bounds provides motivation for scheduling but does not demonstrate a general difference in actual optimization speeds during LLM PPO training. The critic’s noisy, policy-dependent targets further limit the explanatory power of the fixed-target analysis.

The value-drift argument underlying Theorem 3.1 is reasonable under its assumptions. However, the implemented scheduler does not directly estimate or control the initial critic error and value drift required by the bound. The relationship between these quantities and the three observable gating signals is also insufficiently validated.

I acknowledge that the authors do not claim a convergence guarantee for the complete PPO algorithm. My concern is that the analysis currently offers qualitative motivation without adequately explaining why the specific gating rules identify useful update times. The paper should narrow the corresponding claims or provide direct evidence connecting gate activations to critic error and the benefit of refreshing the critic.

Questions for authors

Question 1: With matched critic update budgets and reasonable baseline tuning, does AdaPPO consistently outperform periodic and random update schedules? This result would directly affect my assessment of the central contribution.

Question 2: What fraction of critic updates is triggered by the joint three-gate condition versus the independent safeguards? How does a safeguard-only variant perform?

Question 3: How does AdaPPO* compare against standard PPO using identical privileged information? Are the additional critic input tokens included in the computational accounting?

Question 4: How many independent training runs support the main results? Can the authors provide variability across seeds and elapsed time to matched accuracy on representative tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.