- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Conditional Expectation Reward (CER), a model-intrinsic reward for reinforcement learning with verifiable rewards (RLVR) in tasks where rule-based verification is difficult. CER uses the likelihood of the generated answer to reweight sampled reasoning paths and then measures their support for the reference answer, producing a continuous reward without an additional trained verifier. The paper derives a Monte Carlo estimator and analyzes several theoretical properties, including value equivalence with exact-match reward in expectation. Experiments with Qwen3-4B and Qwen3-8B compare CER against rule-based, model-based, and likelihood-based rewards on mathematical and general-domain reasoning benchmarks. The reported results are competitive, and additional experiments examine open-ended writing performance, computational cost, and failure cases.
Critical strengths
-
The paper addresses a meaningful limitation of RLVR: obtaining useful reward signals when exact-match or domain-specific verification rules are insufficient, particularly for answers with diverse valid expressions.
-
CER provides a clear way to construct continuous rewards from the model's own likelihoods. Conditioning on the generated answer introduces a distinct reasoning-path weighting mechanism without requiring an additional trained verifier.
-
CER performs competitively across two model scales and several relevant reward baselines, with encouraging results under general-domain training. The WritingBench evaluation and the discussion of failure cases are useful additions that help characterize its practical potential and limitations.
Critical weaknesses
-
The claimed ability to reward semantic and partial correctness needs more direct validation. CER assigns rewards using the reference-answer likelihood under reasoning paths conditioned on the generated answer. However, this probabilistic relationship does not necessarily reflect semantic correctness. Theorem 2 establishes equality in expected value with exact-match reward, but does not guarantee that CER assigns higher scores to more correct answers.
-
The benefit of conditional reasoning-path reweighting is not clearly isolated. The comparisons with VeriFree, RLPR, and Nover are relevant, but they do not directly establish whether this conditional weighting is responsible for the reported gains. Table 3 varies the number of sampled paths, which affects estimation and computation, but does not isolate the effect of their relative weights.
-
The value-equivalence result does not directly justify the implemented policy-gradient update. Differentiating the full CER objective would also account for the reward's dependence on the policy parameters, whereas the implemented update omits this term. This does not necessarily invalidate the practical method, but value equivalence alone does not establish gradient equivalence or explain the observed optimization benefits.
Questions for authors
-
What evidence supports the reliability of CER as a correctness signal for free-form answers? A broader assessment of reward quality could substantially strengthen the paper.
-
Is there evidence that this conditional weighting provides a benefit beyond using the reference-answer likelihood alone?
3.Theorem 2 proves equality of expected objective values, while Eq. (7) uses CER as a detached reward. How should this difference affect the interpretation of the theoretical result, and what formal relationship, if any, remains between the implemented gradient and the exact-match objective?
Thanks for the response. I have decided to maintain my original score.