acceptodds
Under review as a conference paper at ICLR 2027

EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training

Abstract

Reinforcement learning (RL) for LLM post-training faces a fundamental design choice: whether to use a learned critic as a baseline for policy optimization. Classical theory favors critic-based methods such as PPO for variance reduction, yet critic-free alternatives like GRPO have gained widespread adoption due to their simplicity and competitive performance. We show that in sparse-reward settings, a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance. By casting baseline selection as a Kalman filtering problem, we unify PPO and GRPO as two extremes of the Kalman gain and prove that explained variance (EV), computable from a single training batch, identifies the exact boundary: positive EV indicates the critic reduces variance, while zero or negative EV signals that it inflates variance. Building on this insight, we propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation, provably achieving no greater variance than the better of the two at every step. Across four tasks spanning classical control, agentic interaction, and mathematical reasoning, EVPO consistently outperforms both PPO and GRPO regardless of which fixed baseline is stronger on a given task. Further analysis shows that adaptive gating tracks critic maturation and reveals an approximately concave threshold-sensitivity profile. Code for all experiments is available at https://anonymous.4open.science/r/EVPO-9785.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM, Critic Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MEqYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Explained Variance Policy Optimization (EVPO), a method to dynamically choose while training between critic-based (PPO) and critic-free (GRPO) advantage estimation in LLM post training, in sparse reward settings. The proposed method uses the explained variance (ev) of the critic. Depending on the sign of the ev, it changes between a learned value baseline (ev>0) and a group-mean baseline (ev<0). The paper also provides a Kalman-filtering interpretation and theoretical analysis, claiming that ev=0 is the exact boundary between useful and harmful critics. EVPO is then evaluated against PPO, GRPO, DAPO and StarPO-S on four different tasks: Sokoban, FrozenLake, Webshop and MATH, where it shows improvements across all four tasks.

Critical strengths
  • This is a simple and well motivated approach: this paper addresses the problem of how the usefulness of a learned critic varies throughout training in LLM RL post training. The method is straightforward and integrate well into PPO.
  • Promising empirical results: EVPO outperforms both PPO and GRPO on the benchmarked tasks. The paper also includes ablations comparing adaptive gating against its individual critic and batch-mean modes, as well as sensitivity studies.
  • Include theoretical perspective: The paper tries to show the connection between explained variance and baseline selection. It provides an intuitive motivation to adaptively choose whether to use or not a critic.
Critical weaknesses
  • Experimental evidence are slightly weak. The proposed results are encouraging but its missing some comparison, such as a simple scheduled switching strategy (to test if adaptive switching is better). One thing the authors could try is for instance using group-mean baseline for a fixed warmup period before transitioning to PPO. The preliminary results already show that such interventions can improve performance, but the paper doesn't compare EVPO against a properly tuned scheduled switch across the main benchmarks. Thus, it makes it difficult to know whether EV-based adaptation provides more benefits beyond a simple schedule.
  • Also related to evaluation, MATH only uses 32 held-out problems (with 16 generations per problem). Given that EVPO improves over GRPO by only approximately 1.3%, this evaluation appears insufficient to establish robust improvement in mathematical reasoning, even more when the reported metric is the peak of the mean validation curve.
  • The theoretical guarantees seem a bit overstated. The main result shows that ev=0 is the boundary between using the critic and a constant baseline but this is only shown at the population level. In practice, EVPO estimates ev from finite batches so the guarantee in Eq. 9 does not necessarily hold. The Kalman gain derivation also seems to rely on assumptions about uncorrelated estimation errors that are not fully justified. Finally, lower advantage variance does not necessarily imply lower policy-gradient variance or better RL performance, as the paper sometimes suggests.
Questions for authors

1 - How does the population-level variance guarantee extend to the actual finite-batch implementation?

2 - What assumptions justify the optimal Kalman gain derivation?

3 - How does EVPO compare against a tuned fixed warmup followed by PPO?

4 - Could the authors provide evaluations on a larger held-out MATH benchmark (for statistical reasons)?

Reviewer_rwBXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when a learned critic helps or hurts advantage estimation in sparse-reward RL post-training of LLMs (RLVR). It uses the critic’s explained variance (EV), the fraction of return variance accounted for by the critic as a real-time proxy for critic quality. Preliminary experiments on Sokoban and FrozenLake link low critic quality to elevated gradient norms and slow learning, and test noise-injection, cold-start and critic-warmup interventions (Sec. 2.2, Fig. 2–3). The authors then cast baseline selection as fusing two noisy estimators of the true state value, the critic and the batch-mean return, in a Kalman-filter analogy (Kalman, 1960 in paper). In this view, PPO and GRPO are the two extreme settings of the Kalman gain: full trust in the critic and no trust in the critic. They show that the sign of EV is determined by whether the critic’s estimation noise exceeds the state-value signal it captures, so EV equal to zero marks the boundary between the two regimes (Eq. 4, Theorem 1). The proposed method, EVPO, computes a per-group batch EV at each training step and uses the critic baseline when this estimate is positive and the batch-mean baseline otherwise (Algorithm 1, Eq. 8). It comes with a population-level guarantee that the advantage variance of EVPO is no greater than that of the better of the two fixed baselines (Eq. 9).

Critical strengths
  1. A Simple, Essentially Free Drop-in Modification (Section 3, Alg. 1, Eq. 6, Eq. 8) EVPO changes only the baseline subtracted from the return. The gating statistic êvB is "computed from the critic values already obtained for advantage estimation, with no additional forward passes" (Sec. 3.1, below Eq. 6), and the method requires "no additional learned parameters or forward passes beyond PPO" (contribution 2). Because the critic keeps training in both modes (Appendix E.1), switching back to critic mode costs nothing. This makes the idea easy to adopt in existing PPO-style pipelines built on RAGEN/verl (Wang, 2025 ), and the code link supports reproducibility.

  2. A Useful Unifying Lens on the PPO-vs-GRPO Choice (Sec. 2.3, Sec. 3.2, Eq. 4, Eq. 7) Writing both estimators as Ât = G − b(st) and placing them at the two ends of a gain K ∈ [0, 1] is a clean way to explain why critic-free methods can be competitive in sparse-reward settings. The paper states that PPO "implicitly assumes PA ≪ PB" while GRPO "implicitly assumes PA ≫ PB", and that "neither assumption holds uniformly across training" (Section 3.2). Eq. (4) also makes explicit, as the paper notes, that the sampling noise R "compresses the magnitude of EV but preserves its sign" (Section 2.3). The paper's analysis of why the samplable continuous gain K̃ = (PA + R)/(PA + PB + 2R) is "systematically pulled toward ½ " (Eq. 12, App. A.1) reasonably motivates a hard gate over naive interpolation.

  3. A Testbed Where the Stronger Fixed Baseline Changes Across Tasks (Section 4.2, Table 1, Fig. 4) The task selection suits an adaptive claim well. The paper reports that critic-based methods "clearly outperform" critic-free methods on Sokoban, FrozenLake and WebShop, while critic-free methods "are more competitive on MATH" (Section 4.2, Table 1). Results are averaged over five runs with across-run standard deviations, and EVPO's curves track or exceed the best competitor through most of training (Fig. 4), which is more informative than a single endpoint.

Critical weaknesses
  1. The Central Theorem Follows Directly From the Paper's Own Definitions (Eq. 3, Eq. 4, Eq. 9, Theorem 1) The paper defines ev = 1 − Var(G − V̂ϕ(s))/Var(G) (Eq. 3) and rewrites it as ev = (PB − PA)/(PB + R) (Eq. 4). The text following Eq. (9) gives Var(ÂPPO) = R + PA and Var(ÂGRPO) = R + PB. Theorem 1's equivalences (ev ≤ 0 ⇔ PA ≥ PB ⇔ K ≥ ½) are therefore restatements of Eq. (4) and the definition K = PA/(PA + PB), and the paper itself states that the guarantee holds "by construction": "EVPO selects the lower-variance baseline at each step" (Section 3.3, page 6). This matters because the abstract states that the authors "prove that explained variance … identifies the exact boundary" and that EVPO "provably" achieves no greater variance than the better of the two baselines, and contribution 1 presents "ev = 0 is the exact boundary" as the main theoretical result. The paper also describes EV as "a standard indicator from regression analysis" (Schulman, 2016 in paper) (Section 2.2). Once the theorem is seen as a consequence of the definitions, the remaining contribution is a sign-based gate on an existing diagnostic, which is a modest step relative to the ICLR bar.

  2. The Population-Level Result Is Not Connected to the Batch Gate (Eq. 6, Sec. 4.4, App. E.2, App. E.5, Limitations) The Limitations state: "Theorem 1 concerns population EV, whereas EVPO uses batch EV, for which we provide no finite-sample concentration or calibration bounds." The gate uses êvB from M = 16 rollouts of a single prompt (Eq. 6, App. D.1). App. E.2 adds that with fewer samples "the batch-level EV estimate êvB has higher variance and is more likely to fall below zero," which confirms that the gate's decisions depend on estimation noise the theory does not cover. The paper itself concludes that "zero is a theoretically motivated default, not a guaranteed empirical optimum" and that the cause of the offset "remains unverified" (Limitations). This sits uneasily with the claim that ev = 0 is "the exact boundary" (contribution 1).

  3. The Motivating Narrative Conflicts With the Paper's Own Figures (Sec. 1, Sec. 2.2, Sec. 4.3, App. E.2) Several textual claims are not supported by the plotted data (figure readings below are approximate): (a) Section 1 states that "EV remains negative during the early stage of training". Observation 1 states that during the first ~150 steps "the critic's predictions are less accurate than a simple constant baseline", and App. D.1 says êvB "remains near or below 0" in that phase. However, Fig. 2a shows EV starting near 0 and rising to roughly 0.2 by step ~100. Fig. 3a (right column) shows non-negative EV throughout on both Sokoban and FrozenLake, and Fig. 5a shows P(use critic) of roughly 0.7 from the first steps on Sokoban. By the paper's own criterion ("ev > 0 favors the critic", Section 2.2), the critic is mostly preferred during the phase described as harmful. App. D.2 also states that the cold-start lengths were "chosen to cover the EV-negative phase of Fig. 2a", a phase that is not visible in that figure. (b) App. E.2 states that under M = 8 "the critic-usage rate decreases consistently across tasks". In Fig. 8b (FrozenLake), however, the Batch-8 curve overtakes Batch-16 around step 90 and stays higher (≈0.85 vs. ≈0.45 at step 200). (c) Finding 3 states that gating frequency "increases monotonically" in most tasks, but Fig. 5d (MATH) declines from ≈0.7 near step 120 to ≈0.6 at step 200.

  4. The Evaluation Protocol Is Likely to Overstate the Gains (Table 1, App. B, Tables 3–4) Table 1 reports "the peak of the mean validation curve over five runs", with the standard deviation "at the same training step" (Table 1 caption). EVPO's Sokoban peak is 0.362 ± 0.088 against PPO's 0.270 ± 0.026. Each per-run success rate is "computed from 512 evaluation results, obtained by sampling 16 times for each of the 32 queries" (Table 1 caption), and for MATH the paper uses "a fixed subset of 32 problems" (App. B), where the margin over GRPO is 0.386 vs. 0.373. No final-step results, area-under-curve, confidence intervals or significance tests are reported. The Cont. K comparison uses "existing single-run references" against five-run peaks (Table 3 caption), and the EMA entries are "controlled single runs" (Table 4 caption). The paper itself notes that the Cont. K comparison "is descriptive rather than a claim of statistical superiority" (App. E.3), and Cont. K still exceeds EVPO on WebShop (0.242 vs. 0.224).

Questions for authors
  1. States used in the batch EV (Eq. 6, Alg. 1 lines 5–6): Alg. 1 computes token-level critic values vm,t (line 5) but uses a single vm per trajectory in êvB (line 6), and Eq. (6) is written with V̂ϕ(sm). Which state(s) sm enter Eq. (6) for single-turn MATH and for the multi-turn tasks?
  2. Degenerate groups (Eq. 6): Eq. (6) divides by VarB(Gm). How is the gate applied when all M returns in a group are identical, and what fraction of groups falls into this case on each task?
  3. Threshold offset (Sec. 4.4, Limitations): The Limitations state that estimation error "may contribute to the threshold offsets" but that their cause "remains unverified". Could the authors repeat the threshold sweep under both M = 8 and M = 16, given App. E.2's statement that êvB "has higher variance" at smaller group size?
  4. Early-phase EV (Sec. 1, Sec. 2.2, App. D.1): Can the authors reconcile the statements that EV is negative or near zero early in training with Fig. 2a, Fig. 3a and Fig. 5a, which show positive EV and mostly critic-mode gating from the start on Sokoban?
  5. Evaluation protocol (Table 1): Can the authors report final-step and area-under-curve success with confidence intervals, so that the 0.386 vs. 0.373 MATH result on 32 problems can be assessed?
Reviewer_Amp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Explained Variance Policy Optimization (EVPO), which dynamically switches between critic-based and batch-mean advantage estimation using explained variance (EV). The authors present a Kalman filtering interpretation and identify EV = 0 as the theoretical switching boundary. Experiments on Sokoban, FrozenLake, WebShop, and MATH show improvements over PPO, GRPO, and other baselines.

Critical strengths
  1. The paper addresses an important issue in LLM reinforcement learning, where an inaccurate critic may introduce additional noise into policy optimization. Adaptive critic utilization is a practical and meaningful direction.

  2. EVPO is easy to implement within an existing PPO framework. The experiments cover several task settings and provide initial evidence that adaptive baseline selection can improve training performance.

Critical weaknesses
  1. The proposed method mainly uses an existing statistical metric to switch between two established advantage estimators. More importantly, the theoretical analysis focuses on advantage variance rather than the variance of the actual policy gradient, which also depends on the policy score function. Therefore, the claim that EVPO always achieves the better optimization behavior of PPO and GRPO is not sufficiently justified.

  2. The reported PPO and GRPO results are relatively low on several tasks, particularly the critic-free baselines on Sokoban and FrozenLake. It is unclear whether these results reflect inherent limitations of the algorithms or differences in implementation and hyperparameter choices. Moreover, the MATH improvement over GRPO is only 1.3 percentage points, and the evaluation uses a small fixed validation set of 32 problems. These limitations weaken the evidence for broad effectiveness in LLM post-training.

Questions for authors
  1. Does the variance guarantee in Theorem 1 apply only to advantage estimates, or can it be extended to the actual policy gradient? What assumptions are required?

  2. Could the authors explain why GRPO and DAPO perform substantially worse than PPO on Sokoban and FrozenLake? Are these results consistent with independently reported or reproduced implementations?

  3. Could the authors provide existing final-checkpoint results or statistical evidence showing that EVPO's improvements, particularly on MATH, are robust beyond the reported peak validation scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.