acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning with Conditional Expectation Reward

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models, particularly in domains such as mathematics where reliable rule-based verifiers can be constructed. However, the reliance on handcrafted, domain-specific verification rules substantially limits the applicability of RLVR to general reasoning domains with free-form answers, where valid answers often exhibit significant variability, making it difficult to establish complete and accurate rules. To address this limitation, we propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier, and is therefore applicable to general domains and eliminates the need for external verifiers or auxiliary models. CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer. In contrast to rule-based verifiers that yield binary feedback, CER provides a soft, graded reward signal that reflects varying degrees of correctness, making it better suited to tasks where answers vary in correctness. Experimental results demonstrate that CER is effective across a wide range of reasoning tasks, spanning both mathematical and general domains, indicating that CER serves as a flexible and general verification mechanism.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning with verifiable rewards, Conditional expectation reward

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zq5uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Conditional Expectation Reward (CER), a model-intrinsic reward for reinforcement learning with verifiable rewards (RLVR) in tasks where rule-based verification is difficult. CER uses the likelihood of the generated answer to reweight sampled reasoning paths and then measures their support for the reference answer, producing a continuous reward without an additional trained verifier. The paper derives a Monte Carlo estimator and analyzes several theoretical properties, including value equivalence with exact-match reward in expectation. Experiments with Qwen3-4B and Qwen3-8B compare CER against rule-based, model-based, and likelihood-based rewards on mathematical and general-domain reasoning benchmarks. The reported results are competitive, and additional experiments examine open-ended writing performance, computational cost, and failure cases.

Critical strengths
  1. The paper addresses a meaningful limitation of RLVR: obtaining useful reward signals when exact-match or domain-specific verification rules are insufficient, particularly for answers with diverse valid expressions.

  2. CER provides a clear way to construct continuous rewards from the model's own likelihoods. Conditioning on the generated answer introduces a distinct reasoning-path weighting mechanism without requiring an additional trained verifier.

  3. CER performs competitively across two model scales and several relevant reward baselines, with encouraging results under general-domain training. The WritingBench evaluation and the discussion of failure cases are useful additions that help characterize its practical potential and limitations.

Critical weaknesses
  1. The claimed ability to reward semantic and partial correctness needs more direct validation. CER assigns rewards using the reference-answer likelihood under reasoning paths conditioned on the generated answer. However, this probabilistic relationship does not necessarily reflect semantic correctness. Theorem 2 establishes equality in expected value with exact-match reward, but does not guarantee that CER assigns higher scores to more correct answers.

  2. The benefit of conditional reasoning-path reweighting is not clearly isolated. The comparisons with VeriFree, RLPR, and Nover are relevant, but they do not directly establish whether this conditional weighting is responsible for the reported gains. Table 3 varies the number of sampled paths, which affects estimation and computation, but does not isolate the effect of their relative weights.

  3. The value-equivalence result does not directly justify the implemented policy-gradient update. Differentiating the full CER objective would also account for the reward's dependence on the policy parameters, whereas the implemented update omits this term. This does not necessarily invalidate the practical method, but value equivalence alone does not establish gradient equivalence or explain the observed optimization benefits.

Questions for authors
  1. What evidence supports the reliability of CER as a correctness signal for free-form answers? A broader assessment of reward quality could substantially strengthen the paper.

  2. Is there evidence that this conditional weighting provides a benefit beyond using the reference-answer likelihood alone?

3.Theorem 2 proves equality of expected objective values, while Eq. (7) uses CER as a detached reward. How should this difference affect the interpretation of the theoretical result, and what formal relationship, if any, remains between the implemented gradient and the exact-match objective?

1 comment

Thanks for the response. I have decided to maintain my original score.

Reviewer_Z9hVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve on the rule-based verifiers used in RLVR. By identifying that handcrafted, domain-specific verification rules are costly and not always informative for the RL training, the authors propose Conditional Expectation Reward (CER), which leverages the policy model’s own generation probability as an implicit verifier. Specifically, CER is defined as the expected likelihood of generating the reference answer conditioned on the model generated answer, and this conditional design essentially separates CER from other perplexity-based methods by considering that the model should output a higher probability to reproducing the reference answer given the generation when the originally generated answer and reference answer are close.

To evaluate the effectiveness of CER, the authors train Qwen3-4B/8B-Base on WebInstruct and MATH datasets with RLOO, and evaluate on four mathematical and two general-domain benchmarks against six verifier baselines and the base model. The results show that CER achieves the best average performance with general-domain training and is comparable to rule-based rewards with math-domain training, while other baselines remain close.

Critical strengths
  1. CER’s conditional reward design is novel and effective. Existing perplexity-based rewards usually scores π_θ(a*|s,q) and can ignore the answer the model actually outputs. This could make a wrong answer end up receiving a misleadingly high reward. Conditioning on the generated answer mitigates this problem, and the results in Table 1 and 2 show that CER outperforms other verifier baselines.
  2. Comprehensive experimentation. The main experiment is performed on 2 training datasets and on 6 benchmarks spanning math and general domains. Beyond the main experiment, the authors provide additional experiments including runtime analysis, length-normalization ablations, and open-ended WritingBench evaluations. These results support the claims in the paper comprehensively and clearly outline the effectiveness scope of CER.
Critical weaknesses
  1. CER could be biased by the surface form, and the paper does not fully addresses it. The authors acknowledge that the surface form could influence the CER in Appendix A.3, but they never quantify the severity or scope of this problem. The direct incorporation of rule-based verifier signal does not fully address this fundamental limit and can even lower CER’s own significance, since the paper’s central claim is that CER removes the need for domain-specific rules.
  2. The gains compared to rule-based verifier are not significant. Under mathematical training, CER does not improve on rule-based rewards on every benchmark and has several regressions. For the general-domain training, the improvements are larger, but the numbers are still close. Furthermore, the best numbers in both experiments come from CER+Rule, which requires the rule-based verifier that CER is motivated to omit.
Questions for authors
  1. Do authors think CER is coupled to RLOO? If not, could the authors provide the experiment results with CER on other RLVR algorithms such as GRPO and Dr. GRPO?
  2. Were VeriFree, RLPR, Nover, and General-verifier all trained with RLOO and the same hyperparameters as CER?
  3. Do the authors think CER training datasets need the reference (gold) answers to be short as those in MATH and WebInstruct?
Reviewer_KJ72OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Conditional Expectation Reward (CER), a model-intrinsic reward mechanism that extends RLVR to general reasoning domains without external verifiers or handcrafted rules. CER provides continuous rewards based on the expected likelihood of the reference answer conditioned on generated answers. The authors establish its theoretical properties, including equivalence in expected value to exact-match rewards, and develop an efficient Monte Carlo estimator. Experiments on Qwen3-4B/8B across mathematical and general reasoning benchmarks demonstrate CER's effectiveness over several existing reward methods and its complementarity with rule-based verification.

Critical strengths
  1. Simple and broadly applicable reward design. CER provides a principled way to construct continuous rewards using the model's own conditional probabilities. Its theoretical properties further strengthen the methodological foundation.

  2. Empirical effectiveness. Experiments across mathematical and general reasoning benchmarks demonstrate that CER outperforms several existing reward methods on Qwen3-4B/8B.

Critical weaknesses
  1. Insufficient theoretical justification for the optimization mechanism: Although CER is shown to be equivalent in expected value to exact-match rewards, this does not directly justify the practical policy-gradient update, which treats the policy-dependent reward as a fixed scalar. The optimization implications of this discrepancy remain insufficiently analyzed.

  2. Limited validation of reward reliability and generalizability: Although the experiments cover mathematical and general reasoning tasks, the main general-domain evaluations rely on multiple-choice benchmarks, with limited evaluation on complex open-ended responses. Moreover, the reported failure cases suggest that CER may assign misleading rewards to semantically equivalent or incorrect answers, raising concerns about its reliability across diverse tasks.

Questions for authors
  1. Optimization mechanism: Since CER depends on the current policy parameters, how is the stop-gradient treatment in Eq. (7) theoretically justified? How does this update relate to the expected-value equivalence established in Theorem 2?

  2. Reward reliability: Given the reported failure cases where CER assigns misleading rewards, how frequently do such errors occur in practice, and to what extent do they affect training performance?

  3. Generalization to agentic tasks. Can the authors provide evidence of CER's effectiveness on more complex agentic tasks, such as multi-step tool-augmented search and reasoning? Such results would help assess the generalizability of CER beyond the existing reasoning benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.