- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Explained Variance Policy Optimization (EVPO), a method to dynamically choose while training between critic-based (PPO) and critic-free (GRPO) advantage estimation in LLM post training, in sparse reward settings. The proposed method uses the explained variance (ev) of the critic. Depending on the sign of the ev, it changes between a learned value baseline (ev>0) and a group-mean baseline (ev<0). The paper also provides a Kalman-filtering interpretation and theoretical analysis, claiming that ev=0 is the exact boundary between useful and harmful critics. EVPO is then evaluated against PPO, GRPO, DAPO and StarPO-S on four different tasks: Sokoban, FrozenLake, Webshop and MATH, where it shows improvements across all four tasks.
Critical strengths
- This is a simple and well motivated approach: this paper addresses the problem of how the usefulness of a learned critic varies throughout training in LLM RL post training. The method is straightforward and integrate well into PPO.
- Promising empirical results: EVPO outperforms both PPO and GRPO on the benchmarked tasks. The paper also includes ablations comparing adaptive gating against its individual critic and batch-mean modes, as well as sensitivity studies.
- Include theoretical perspective: The paper tries to show the connection between explained variance and baseline selection. It provides an intuitive motivation to adaptively choose whether to use or not a critic.
Critical weaknesses
- Experimental evidence are slightly weak. The proposed results are encouraging but its missing some comparison, such as a simple scheduled switching strategy (to test if adaptive switching is better). One thing the authors could try is for instance using group-mean baseline for a fixed warmup period before transitioning to PPO. The preliminary results already show that such interventions can improve performance, but the paper doesn't compare EVPO against a properly tuned scheduled switch across the main benchmarks. Thus, it makes it difficult to know whether EV-based adaptation provides more benefits beyond a simple schedule.
- Also related to evaluation, MATH only uses 32 held-out problems (with 16 generations per problem). Given that EVPO improves over GRPO by only approximately 1.3%, this evaluation appears insufficient to establish robust improvement in mathematical reasoning, even more when the reported metric is the peak of the mean validation curve.
- The theoretical guarantees seem a bit overstated. The main result shows that ev=0 is the boundary between using the critic and a constant baseline but this is only shown at the population level. In practice, EVPO estimates ev from finite batches so the guarantee in Eq. 9 does not necessarily hold. The Kalman gain derivation also seems to rely on assumptions about uncorrelated estimation errors that are not fully justified. Finally, lower advantage variance does not necessarily imply lower policy-gradient variance or better RL performance, as the paper sometimes suggests.
Questions for authors
1 - How does the population-level variance guarantee extend to the actual finite-batch implementation?
2 - What assumptions justify the optimal Kalman gain derivation?
3 - How does EVPO compare against a tuned fixed warmup followed by PPO?
4 - Could the authors provide evaluations on a larger held-out MATH benchmark (for statistical reasons)?