acceptodds
Withdrawn from ICLR 2027

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage–value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.

Primary area:
reinforcement learning
Keywords:
policy gradient, advantage function, control variate, variance reduction

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_2kp7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Advantage-Based Control Variates (ABC), an actor-critic method for reinforcement learning with verifiable rewards (RLVR). The method reformulates trajectory-wise control variates using learned advantage and value functions, allowing the policy gradient to remain unbiased even with an imperfect critic. The authors further connect ABC with Direct Advantage Estimation (DAE) and propose an offline-to-online training framework, where the critic is pretrained on offline trajectories and updated during online RL. Experiments on mathematical reasoning benchmarks with Qwen3 models show reduced gradient variance and performance competitive with Dr. GRPO using fewer online actor updates.

Critical strengths
  1. I like the general motivation of the paper, the paper revisits an important question: whether learned critics can improve RLVR despite the strong performance of critic-free methods such as GRPO. The proposed ABC estimator combines learned advantages with an unbiased correction, providing a principled way to reduce variance without relying entirely on critic predictions.

  2. The experiments show that ABC can reduce gradient variance and achieve competitive performance with Dr. GRPO across several math benchmarks. The comparison with directly using learned advantages is also informative, highlighting the importance of correcting critic-induced bias.

Critical weaknesses
  1. The paper emphasizes that ABC achieves competitive performance with only 100 actor updates, compared with approximately 6,000 updates for Dr. GRPO. However, ABC requires pretraining an 8B critic using approximately 8.2 million offline trajectories, which introduces substantial additional computation. The online batch sizes also differ significantly. Therefore, comparing the number of optimizer steps alone is not sufficient to establish training efficiency.

  2. The experiments focus only on mathematical reasoning with models up to 4B, making it unclear whether the method generalizes to other RLVR tasks or larger models. I think the proposed method may be more beneficial for long-horizon tasks involving multiple consequential decisions, where trajectory-level rewards provide limited credit assignment. Although mathematical reasoning can involve long token sequences, it remains unclear whether the method can effectively capture the benefits of fine-grained advantage estimation in more complex settings, such as multi-step code generation, debugging, or agentic tasks. I would like to see experiments on these tasks, or at least an analysis of how trajectory length affects gradient variance, critic estimation error, and final performance. This would help establish when the proposed control variates are most useful compared with standard GRPO.

  3. I like the motivation of this paper, and I think revisiting critic-based methods for RLVR is an interesting direction. However, I feel that the overall presentation and experimental evaluation still need further improvement. Some design choices and empirical observations are not sufficiently explained and deserve deeper investigation. For example, why does ABC fail to consistently match Dr. GRPO in the larger 4B model setting, particularly on MATH500 and AIME pass@32? Does this suggest that the benefit of learned advantages becomes less significant as model size increases? How does the performance change as pass@K is evaluated with increasing values of K?

I would like to see a more detailed analysis of when and why ABC provides advantages over existing methods, rather than primarily demonstrating competitive performance.

Questions for authors

See weakness

Reviewer_pWgvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ABC combines learned value and advantage functions in a trajectory-level policy-gradient control variate, with an analytic correction intended to preserve unbiasedness. The paper relates the estimator to Direct Advantage Estimation (DAE), derives a covariance result, and tests an offline-to-online RLVR implementation on mathematical reasoning tasks with Qwen3 actors and an 8B critic.

Critical strengths

The estimator and its centering condition are explained clearly. The covariance analysis provides a useful account of how critic quality affects variance.

Critical weaknesses

The paper states that ABC is algebraically equivalent to TrajCV under Q = A + V (Sec. 3.1). The new contribution is therefore mainly the interpretation, DAE training, and RLVR application; the estimator's novelty should be framed accordingly. Covariance optimality is proved for individual gradient contributions. The practical algorithm sums contributions across time, where additional covariance terms arise (Appendix B). The theorem does not establish optimal variance for that trajectory estimator. Unbiasedness depends on policy-specific centering. The paper does not make sufficiently clear how full-vocabulary centering and the analytic correction are computed or what they cost when the actor changes. The comparison with Dr. GRPO is not matched for total resources: ABC uses about 8.2 million offline trajectories and an 8B critic. Fewer online updates alone do not demonstrate better compute or sample efficiency. Main results also lack multiple-seed uncertainty and cover only mathematics.

Questions for authors
  1. How are advantage centering and the analytic correction implemented over the vocabulary, and how often are they recomputed?
  2. What is the wall-clock time, total generated-token count, and critic-training compute relative to Dr. GRPO?
  3. What do value-only, no-offline-pretraining, and different-critic-size ablations show?
  4. How does ABC compare directly with TrajCV under the same training and evaluation budget?
Reviewer_odJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work authors propose a new gradient estimator to be used in reinforcement learning specifically applied to LLMs with verifiable reward. The authors introduce some general RL and policy gradient ideas including how control variates can mitigate the effect of gradient variance before proposing a new estimator. Their new estimator takes into account the estimated future advantage should the trajectory continues, and adjusts with a correction term involving the advantage at the starting state. The authors show this estimator is unbiased by leveraging a centering property of the estimated advantage function. The authors additionally provide some theoretical arguments about the error of the covariance of their approximate value and advantage functions, showing they are equivalent up to a defined error term and that error is bounded by Fisher information of the learned policy. Finally, the authors evaluate their estimator on a model/dataset pair at a few different scales. They note that their methods requires training an approximate value and advantage function offline, followed by using their estimator to train a policy online. The authors provide a few ablations regarding the variance of their estimator relative to prior work and show downstream results in comparison to that prior work.

Critical strengths

The paper is well written and provides a significant amount of background regarding the concepts introduced and studies. This area is definitely relevant in the current state of the field, where finding lower variance or more efficient methods could potentially save significant computational costs. The method is simple enough for the reader to get a good intuition of how it arises. As far as I can tell the derivations in the paper and appendix are without issue. The authors argue that their method does indeed have lower variance relative to other policy gradient estimators, and show some strong performance relative to baselines when measured in terms of updates to the actor.

Critical weaknesses

While likely an interesting and potentially good idea, I do feel this paper is limited in providing convincing evidence that their method does in fact outperform baseline methods. While I am somewhat averse to simply asking for more experiments, it is the case and the authors acknowledge in their limitations that they are only evaluating one class of models on one type of dataset. In order to claim a method for practitioners to adopt, I would expect to see more evaluation of the method and comparison to more baseline methods. I am also somewhat concerned about the claim of higher efficiency. While the authors claim that the method is very efficient at optimizing the policy, it seems to me that the saved compute is just being amortized into training the value and advantage function, which the compared baselines do not need. The authors would need to find a more reasonable way of comparing computational efficiency, figure 3 and figure 5 compare these costs individually but they are not helpful without being considered together. Even on the strait forward comparisons between this work and two prior on table 1, it is not clear why the proposed method is better in some cases, and notably there are many cases where that is not true. While it is likely too strong to ask for a method to always be state of the art, it does beg the question what the advantage of this estimator is if the results provide no clear indication of when a practitioner may want to use it.

Overall I do think this is an interesting idea and worth pursuing further, but the current state of the paper feels somewhat underdone in terms of an ICLR publication. I believe a more thorough evaluation relative to other estimators along with further ablation to determine what part of the estimator is actually making the improvement would significantly improve the paper.

Questions for authors
  • What is the significance of panel 2 in figure 5? How is a reader supposed to get a meaningful interpretation of completion length when we don’t know the quality or correctness of that completion.
  • If a compute budget is fixed both for policy optimization and solving for the value/advantage function, how does ABC compare to the other baselines listed?
  • Would there be any way to use this estimator in a strictly online setting? The requirement of an offline phase could be limiting in settings without a pre-existing dataset
  • What are the details of Figure 4? Is this variance ratio at a certain state? Initialization? A mean over training? The authors could clarify what variance is being reduced more precisely.
  • Could the authors further explain the setting of figure 3? For example, are the dashed lines trying to solve the DAE problem (equation 4) but with a fixed A^\hat A? or is a different MSE problem being solved. If it is a fixed A^\hat A, what is it fixed to?
Reviewer_BdEyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies policy gradient variance reduction for RL with verifiable rewards and proposes ABC, an advantage-based control variate estimator. The main idea is to use estimated advantages along the future trajectory together with a value estimate, while adding a correction term so that the gradient remains unbiased. The paper connects this estimator to Direct Advantage Estimation and uses DAE to train the critic. The practical setup is offline-to-online: the critic is first trained on a large offline set of trajectories and is then updated together with the actor during online RL. Experiments are done on Qwen3 models of several sizes on math reasoning benchmarks. The results show lower gradient variance in most settings and competitive final performance compared with Dr. GRPO while using relatively few actor updates.

Critical strengths
  • The main estimator is reasonably intuitive once the paper gets to Section 3. Using future advantage estimates as a control variate makes sense, and the correction term gives a clean way to avoid simply trusting a learned critic.
  • I think the connection between ABC and the DAE objective is one of the better parts of the paper. The variance bound in the deterministic case gives some justification for why return-prediction quality should matter for the usefulness of the control variate, instead of DAE feeling like an unrelated critic training choice.
  • The comparison to directly using the learned advantage is useful. Figure 6 shows that the biased estimator can initially work well but stops improving as much, while ABC continues improving. This is a nice empirical example of why keeping the policy gradient unbiased can matter even when the critic is already fairly informative.
  • The authors are also fairly transparent about a case where ABC does not reduce variance. For the 1.7B model, they identify large critic residual outliers as a likely cause. That is more informative than only showing the successful cases.
Critical weaknesses
  • I am not convinced the empirical section establishes a clear practical advantage over existing RLVR methods. The strongest statement is that ABC reaches competitive performance with only around 100 actor updates, but this ignores the substantial work done before those updates. The critic is trained using about 8.2 million offline trajectories and is an 8B model, so actor update count by itself is not a very meaningful efficiency metric.
  • Related to this, the paper does not report a total-compute comparison. It would be much easier to judge the method with generated tokens, GPU hours, wall-clock time, or FLOPs including offline critic training and online critic updates. Right now it is hard to tell whether ABC is actually cheaper, or whether it mostly moves computation from policy optimization into critic training.
  • The practical gains from the advantage component are not very clean across model sizes. Figure 4 shows that the value baseline already gives a large variance reduction, and ABC only gives an additional improvement for some actors. For the 1.7B actor it is actually worse than value-only because of critic error. This makes me wonder how much of the final gain is really coming from the trajectory-level advantage term versus simply having a strong value baseline.
  • I would like to see a direct value-only actor-critic baseline in the final downstream results, not only variance measurements. That seems like a particularly important missing ablation. If value-only gets most of the performance improvement, then the case for the more complicated ABC estimator becomes weaker.
  • The theoretical optimality result is somewhat narrower than the presentation initially suggests. The appendix notes that the covariance result applies to individual gradient contributions, while the actual algorithm sums token-level contributions over the trajectory. Cross-time covariance terms then appear, so the theorem does not imply that the full trajectory gradient used in practice has minimum variance. This does not invalidate the method, but I think it should be discussed more prominently in the main text.
  • The paper says ABC is algebraically equivalent to TrajCV after writing Q = A + V. This makes the novelty of the estimator itself less clear. My reading is that the more specific contribution is the advantage-value interpretation, the DAE connection, and the RLVR implementation. I think the paper should be more explicit about this distinction.
  • The experiments are all math RLVR using the same model family. I do not think every paper needs a huge benchmark suite, but here the usefulness of the method seems quite dependent on critic quality, action entropy, and trajectory structure. The paper itself observes that the advantage term becomes less useful as the actor becomes more deterministic. It would help to test at least one setting where longer-horizon credit assignment is more clearly important, or at minimum provide a deeper analysis of when ABC should be expected to help.
  • There is also no uncertainty reported over multiple training runs. Since differences between ABC and Dr. GRPO are fairly small on some benchmark/model combinations, it is difficult to know whether those differences are meaningful. Table 1 has several cases where ABC is better on one metric and worse on another, especially for the 4B model.
Questions for authors
  • How much of the final performance gain comes from the advantage term specifically? Could you report downstream results for a value-only version using the same critic and online training setup?
  • What is the total training cost of ABC including offline trajectory generation, offline critic training, and online critic updates? A comparison in generated tokens and GPU hours against Dr. GRPO would be useful.
  • How sensitive is ABC to critic quality? In particular, what happens if the critic is smaller than 8B or is trained on substantially less offline data?
  • Since the appendix says covariance optimality does not necessarily extend to the summed trajectory gradient, have you measured the variance of the actual full gradient used by Algorithm 1 rather than only per-sample/per-step contributions?
  • Since ABC is equivalent to TrajCV under a reparameterization, what empirical benefit comes specifically from using the advantage-value formulation and DAE rather than training a Q-based TrajCV critic directly?
  • Can the method work from scratch without offline critic pretraining? Even if performance is worse, I think this would help clarify whether the offline phase is essential to the method or mainly an efficiency choice.
  • Figure 5 shows completion length changing substantially over training. Is there any relationship between completion length and critic error or ABC variance reduction? It might help explain when the learned advantages are actually contributing useful credit assignment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.