- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoVER (Contrastive Verification Reward), which combines programmatic feedback with relative self-verification. Tackling previous label-free rewards that include model biases or do not care about the answer's correctness (e.g., TTRL's majority-vote reward, which reinforces the model's existing beliefs), the authors use label-free self-verification and a programmatic checker. The self-verification is extended to a relative version: the consensus count of each candidate answer is multiplied by a softmax over the verification scores of all distinct candidates. The programmatic checker, which scores a candidate against the constraints stated in the prompt with partial credit, is added after a handover step and corrects errors that self-verification systematically misses. The method is evaluated on three synthetic tasks (Arithmetic, Countdown, 2×2×2 Rubik's Cube), LiveCodeBench, and AIME 2024 against TTRL, Intuitor, EMPO, and Co-rewarding. CoVER is the best on every task.
Critical strengths
- The paper is generally well-written and easy to follow.
- The motivation is clear: agreement-based rewards measure what the model believes rather than whether it is correct. The method follows directly from it: make self-verification a relative judgment between candidates, and ground the reward with a checker. The design is simple yet effective.
- On all five tasks evaluated, CoVER outperforms the four label-free baselines, which shows the method's usefulness in the settings it targets.
Critical weaknesses
[W1] The method is restricted to tasks where a programmatic checker is available, so the evaluation is mostly on simple synthetic tasks
- CoVER relies on a task-specific programmatic checker, and the evaluation is accordingly concentrated on simple synthetic tasks where such a checker is easy to write
- These are the tasks with by far the largest gains.
- On AIME and LiveCodeBench, where the checker is indirect, the margin over the strongest baseline drops
- The paper's own ablation on Arithmetic shows that the checker alone reaches 0.675, against CoVER's 0.688
- The authors acknowledge that open-ended outputs remain unexplored. Still, the label-free setting is motivated precisely by inputs for which no external supervision is available.
- Therefore, it is unclear how much of CoVER's benefit would remain where no such checker can be written, which raises doubt about the method's general usefulness.
[W2] The baselines are compared without the checker that CoVER receives, which is effectively ground truth on Countdown and Rubik's Cube
- All four baselines are pure self-reward methods, whereas CoVER additionally receives the programmatic checker after the handover step.
- The authors themselves explain why the baselines fail on Countdown: the base model is wrong on more than 95% of samples, so agreement-based baselines "have almost nothing correct to reinforce".
- The comparison on these tasks therefore mostly measures access to the checker rather than the quality of CoVER's label-free reward relative to the baselines (e.g., TTRL + r_chk)
[W3] The evidence beyond the synthetic tasks is thin
- The results come from the 30 problems of AIME 2024, which serve as both the training and the evaluation set.
- I recommend to report other math benchmark as well.
- On LiveCodeBench, the test set is large (1,055 problems; l.317, l.665), but the margin over EMPO is only 0.66 and Table 4 reports no variance or seed count.
- As a result, the claim that CoVER "consistently outperforms existing label-free rewards" is supported mainly by the synthetic tasks.
[W4] The backbone and the mixing mode change per benchmark without justification
- The backbone model changes across benchmarks (Qwen2.5, 0.5B to 1.5B, instruction-tuned, base, math-tuned)
- The backbone is reported only in the appendix
- None of the setup paragraphs or Tables state it.
- No reason is given for the choice, e.g., why LiveCodeBench uses the base rather than the Instruct model.
- No task is evaluated on more than one backbone, and all backbones come from a single small-scale family. The authors acknowledge that larger models are untested.
- The mixing mode also changes per task, from reward-space mixing to advantage-space mixing to replacement.
- The Limitations section notes that it is set per task, but the paper does not explain how it was chosen.
- It is therefore unclear why the backbone and mixing mode change per benchmark, and whether the gains hold for a fixed model and a fixed mixing mode.
[W5] No code or data release for the synthetic tasks
- Arithmetic, Countdown, and Rubik's Cube are synthetic tasks generated by the authors, yet the paper neither releases code or data nor states that it will.
- The Reproducibility Statement points to Section 4 and Appendix A (l.515–518), but the text alone is not sufficient to regenerate the tasks or rerun CoVER
Questions for authors
- Can you report results on at least one task where no programmatic checker is available, using contrastive verification alone? How much of CoVER's gain remains there?
- Can you report the baselines with the same checker added (e.g., TTRL + r_chk, EMPO + r_chk), and checker-only training on Countdown and Rubik's Cube?
- Can you report results on additional math benchmarks (beyond the 30 AIME 2024 problems used for both training and evaluation), and the variance and seed count for LiveCodeBench?
- Why does the backbone change per benchmark? Can you report (i) all five tasks with a single fixed backbone and (ii) cross-model experiments on each task, including a larger model and a model outside the Qwen2.5 family?
If the computational costs are too burden, providing the justifications and/or rationales would support them.
2 comments
Q4. Backbones and mixing mode
Why the backbone differed per task. The backbone was not chosen by a systematic criterion; we picked a reasonably sized Qwen2.5 model for each task. We will state the backbone in each setup paragraph and table of the main text. To test whether the gains depend on this choice, we ran additional backbones, including a larger model (3B) and one outside the Qwen2.5 family (Llama-3.2):
Countdown
| Backbone | Base | TTRL | CoVER |
|---|---|---|---|
| Qwen2.5-1.5B-Instruct | 0.046 | 0.053 | 0.471 ± 0.221 |
| Qwen2.5-3B-Instruct | 0.104 | 0.163 ± 0.016 | 0.545 ± 0.239 |
| Llama-3.2-3B-Instruct | 0.035 | 0.101 ± 0.061 | 0.767 ± 0.034 |
Arithmetic
| Backbone | Base | TTRL | CoVER |
|---|---|---|---|
| Qwen2.5-0.5B-Instruct | 0.513 | 0.629 | 0.688 ± 0.009 |
| Qwen2.5-1.5B-Instruct | 0.305 | 0.683 ± 0.055 | 0.705 ± 0.009 |
| Qwen2.5-1.5B-Instruct, 3–4 digits | 0.145 | 0.608 ± 0.004 | 0.611 ± 0.005 |
LiveCodeBench
| Backbone | Base | TTRL | CoVER |
|---|---|---|---|
| Qwen2.5-1.5B | 10.83 | 11.72 | 13.34 |
| Qwen2.5-1.5B-Instruct | 7.63 | 9.65 | 12.22 |
Fixed mixing mode:
| Task | Reward-space | Advantage-space | Replacement |
|---|---|---|---|
| Arithmetic | 0.688 ± 0.009 | 0.673 ± 0.016 | 0.674 ± 0.015 |
| Countdown | 0.260 ± 0.036 | 0.471 ± 0.221 | 0.370 ± 0.109 |
| Rubik's | 0.078 ± 0.012 | 0.096 ± 0.036 | 0.167 ± 0.037 |
Result: CoVER is above TTRL on every new backbone on Countdown, Arithmetic and LiveCodeBench, including Llama-3.2-3B. On Arithmetic the margin is small (0.705 vs 0.683 and 0.611 vs 0.608 on Qwen2.5-1.5B). The mixing mode moves Arithmetic by ≤1.5pp, and Countdown and Rubik's by up to a factor of two. With any single fixed mixing mode, CoVER stays above TTRL on all three tasks (e.g., advantage-space: 0.673 vs 0.629, 0.471 vs 0.053, 0.096 vs 0.027), but the best mode differs per task, which we acknowledge as a limitation.
W5. Code release
We will release the code, task generators and data upon the acceptance of this work.
We thank the reviewer for the careful review and the concrete questions.
Q1. Contrastive verification with no checker
Yes, partly. We do not yet have a task without any programmatic checker, so we instead removed the checker from CoVER and trained with the self-verification reward for the whole run (CV only):
Result: Without a checker, self-verification is clearly above frequency-based rewards on Countdown and Rubik's, where the base model is almost always wrong. CV only keeps about 40–50% of the full CoVER accuracy (0.193 vs 0.471 on Countdown, 0.040 vs 0.089 on 4-number Countdown, 0.082 vs 0.167 on Rubik's), so a substantial part of the gain remains without the checker, and the checker contributes the rest. We agree that open-ended tasks without any checker remain untested.
Q2. Baselines given the same checker; checker-only
Yes. We added the same checker to TTRL and EMPO, and trained with the checker alone:
Result: On Countdown, adding the checker to TTRL or EMPO leaves them at the base level, while CoVER and checker-only learn. On Rubik's, TTRL + checker matches CoVER (0.165 vs 0.167) and exceeds it with the exact solve check (0.169 vs 0.148). Checker-only matches or exceeds CoVER when the checker is exact, and is below it when the checker is weak or partial (0.050 vs 0.295 with r_form on Countdown). We agree with the reviewer that, with an exact checker, the gain over the baselines comes mainly from the checker; the contribution of self-verification is largest when the available checker is weak.
Q3. LiveCodeBench variance
LiveCodeBench in the submission is single seed. We ran it with 3 seeds:
Result: CoVER has the highest mean and is above every baseline in each of the three runs. The margin over EMPO (0.80) is small and of the same order as the seed standard deviation, so we agree that the evidence on LiveCodeBench is modest. We did not evaluate on math benchmarks beyond AIME 2024, so we agree that the math evidence is limited to these 30 problems.