acceptodds
Under review as a conference paper at ICLR 2027

CoVER: Contrastive Self-Verification for Label-Free Reinforcement Learning

Abstract

A model that improves itself on unlabeled inputs would reduce its reliance on external supervision. Test-time reinforcement learning takes a step toward this goal by deriving rewards from the model's own predictions. However, agreement-based rewards only reflect the model’s answer distribution: they can sharpen an existing belief, but cannot tell whether that belief is correct. This creates a fundamental form of confirmation bias. A programmatic checker provides an external correctness signal without requiring labeled training examples, but it is often sparse, task-specific, or unavailable until a useful partial solution is produced. We propose CoVER (COntrastive VErification Reward), a hybrid label-free reward that combines programmatic feedback with relative self-verification. \mname first uses contrastive verification to distinguish promising candidate solutions and maintain a useful distribution of trajectories, then incorporates programmatic feedback to provide grounded credit for valid partial progress. This combination allows the model to exploit executable evidence when available while retaining a model-derived signal when programmatic feedback is insufficient. Across diverse reasoning tasks, CoVER consistently outperforms existing label-free rewards.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Reinforcement Learning, Verification, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zCZCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoVER (Contrastive Verification Reward), which combines programmatic feedback with relative self-verification. Tackling previous label-free rewards that include model biases or do not care about the answer's correctness (e.g., TTRL's majority-vote reward, which reinforces the model's existing beliefs), the authors use label-free self-verification and a programmatic checker. The self-verification is extended to a relative version: the consensus count of each candidate answer is multiplied by a softmax over the verification scores of all distinct candidates. The programmatic checker, which scores a candidate against the constraints stated in the prompt with partial credit, is added after a handover step and corrects errors that self-verification systematically misses. The method is evaluated on three synthetic tasks (Arithmetic, Countdown, 2×2×2 Rubik's Cube), LiveCodeBench, and AIME 2024 against TTRL, Intuitor, EMPO, and Co-rewarding. CoVER is the best on every task.

Critical strengths
  • The paper is generally well-written and easy to follow.
  • The motivation is clear: agreement-based rewards measure what the model believes rather than whether it is correct. The method follows directly from it: make self-verification a relative judgment between candidates, and ground the reward with a checker. The design is simple yet effective.
  • On all five tasks evaluated, CoVER outperforms the four label-free baselines, which shows the method's usefulness in the settings it targets.
Critical weaknesses

[W1] The method is restricted to tasks where a programmatic checker is available, so the evaluation is mostly on simple synthetic tasks

  • CoVER relies on a task-specific programmatic checker, and the evaluation is accordingly concentrated on simple synthetic tasks where such a checker is easy to write
    • These are the tasks with by far the largest gains.
    • On AIME and LiveCodeBench, where the checker is indirect, the margin over the strongest baseline drops
    • The paper's own ablation on Arithmetic shows that the checker alone reaches 0.675, against CoVER's 0.688
    • The authors acknowledge that open-ended outputs remain unexplored. Still, the label-free setting is motivated precisely by inputs for which no external supervision is available.
  • Therefore, it is unclear how much of CoVER's benefit would remain where no such checker can be written, which raises doubt about the method's general usefulness.

[W2] The baselines are compared without the checker that CoVER receives, which is effectively ground truth on Countdown and Rubik's Cube

  • All four baselines are pure self-reward methods, whereas CoVER additionally receives the programmatic checker after the handover step.
  • The authors themselves explain why the baselines fail on Countdown: the base model is wrong on more than 95% of samples, so agreement-based baselines "have almost nothing correct to reinforce".
    • The comparison on these tasks therefore mostly measures access to the checker rather than the quality of CoVER's label-free reward relative to the baselines (e.g., TTRL + r_chk)

[W3] The evidence beyond the synthetic tasks is thin

  • The results come from the 30 problems of AIME 2024, which serve as both the training and the evaluation set.
    • I recommend to report other math benchmark as well.
  • On LiveCodeBench, the test set is large (1,055 problems; l.317, l.665), but the margin over EMPO is only 0.66 and Table 4 reports no variance or seed count.
  • As a result, the claim that CoVER "consistently outperforms existing label-free rewards" is supported mainly by the synthetic tasks.

[W4] The backbone and the mixing mode change per benchmark without justification

  • The backbone model changes across benchmarks (Qwen2.5, 0.5B to 1.5B, instruction-tuned, base, math-tuned)
  • The backbone is reported only in the appendix
    • None of the setup paragraphs or Tables state it.
    • No reason is given for the choice, e.g., why LiveCodeBench uses the base rather than the Instruct model.
  • No task is evaluated on more than one backbone, and all backbones come from a single small-scale family. The authors acknowledge that larger models are untested.
  • The mixing mode also changes per task, from reward-space mixing to advantage-space mixing to replacement.
    • The Limitations section notes that it is set per task, but the paper does not explain how it was chosen.
  • It is therefore unclear why the backbone and mixing mode change per benchmark, and whether the gains hold for a fixed model and a fixed mixing mode.

[W5] No code or data release for the synthetic tasks

  • Arithmetic, Countdown, and Rubik's Cube are synthetic tasks generated by the authors, yet the paper neither releases code or data nor states that it will.
  • The Reproducibility Statement points to Section 4 and Appendix A (l.515–518), but the text alone is not sufficient to regenerate the tasks or rerun CoVER
Questions for authors
  • Can you report results on at least one task where no programmatic checker is available, using contrastive verification alone? How much of CoVER's gain remains there?
  • Can you report the baselines with the same checker added (e.g., TTRL + r_chk, EMPO + r_chk), and checker-only training on Countdown and Rubik's Cube?
  • Can you report results on additional math benchmarks (beyond the 30 AIME 2024 problems used for both training and evaluation), and the variance and seed count for LiveCodeBench?
  • Why does the backbone change per benchmark? Can you report (i) all five tasks with a single fixed backbone and (ii) cross-model experiments on each task, including a larger model and a model outside the Qwen2.5 family?

If the computational costs are too burden, providing the justifications and/or rationales would support them.

2 comments

We thank the reviewer for the careful review and the concrete questions.

Q1. Contrastive verification with no checker

Yes, partly. We do not yet have a task without any programmatic checker, so we instead removed the checker from CoVER and trained with the self-verification reward for the whole run (CV only):

TaskBaseTTRLCV only
Countdown, 3 numbers0.0460.0530.193
Countdown, 4 numbers0.0190.0180.040
Rubik's0.0230.0270.082

Result: Without a checker, self-verification is clearly above frequency-based rewards on Countdown and Rubik's, where the base model is almost always wrong. CV only keeps about 40–50% of the full CoVER accuracy (0.193 vs 0.471 on Countdown, 0.040 vs 0.089 on 4-number Countdown, 0.082 vs 0.167 on Rubik's), so a substantial part of the gain remains without the checker, and the checker contributes the rest. We agree that open-ended tasks without any checker remain untested.

Q2. Baselines given the same checker; checker-only

Yes. We added the same checker to TTRL and EMPO, and trained with the checker alone:

Task / checkerCoVERChecker-onlyTTRL + checkerEMPO + checker
Countdown, exact r_valid0.471 ± 0.2210.524 ± 0.1810.062 ± 0.0300.055 ± 0.022
Countdown, weak r_form0.295 ± 0.0780.050 ± 0.0060.052 ± 0.0210.047 ± 0.006
Rubik's, r_progress0.167 ± 0.0370.151 ± 0.0490.165 ± 0.0580.041 ± 0.010
Rubik's, exact solve check0.148 ± 0.0370.179 ± 0.0750.169 ± 0.0600.039 ± 0.008

Result: On Countdown, adding the checker to TTRL or EMPO leaves them at the base level, while CoVER and checker-only learn. On Rubik's, TTRL + checker matches CoVER (0.165 vs 0.167) and exceeds it with the exact solve check (0.169 vs 0.148). Checker-only matches or exceeds CoVER when the checker is exact, and is below it when the checker is weak or partial (0.050 vs 0.295 with r_form on Countdown). We agree with the reviewer that, with an exact checker, the gain over the baselines comes mainly from the checker; the contribution of self-verification is largest when the available checker is weak.

Q3. LiveCodeBench variance

LiveCodeBench in the submission is single seed. We ran it with 3 seeds:

MethodPer seedMean
Base—10.83
TTRL11.72 / 9.19 / 11.3610.76 ± 1.37
Intuitor10.97 / 10.99 / 10.3510.77 ± 0.36
EMPO12.68 / 11.69 / 11.8612.08 ± 0.53
Co-rewarding11.89 / 11.76 / 12.0611.90 ± 0.15
CoVER13.34 / 12.39 / 12.9112.88 ± 0.48

Result: CoVER has the highest mean and is above every baseline in each of the three runs. The margin over EMPO (0.80) is small and of the same order as the seed standard deviation, so we agree that the evidence on LiveCodeBench is modest. We did not evaluate on math benchmarks beyond AIME 2024, so we agree that the math evidence is limited to these 30 problems.

Q4. Backbones and mixing mode

Why the backbone differed per task. The backbone was not chosen by a systematic criterion; we picked a reasonably sized Qwen2.5 model for each task. We will state the backbone in each setup paragraph and table of the main text. To test whether the gains depend on this choice, we ran additional backbones, including a larger model (3B) and one outside the Qwen2.5 family (Llama-3.2):

Countdown

BackboneBaseTTRLCoVER
Qwen2.5-1.5B-Instruct0.0460.0530.471 ± 0.221
Qwen2.5-3B-Instruct0.1040.163 ± 0.0160.545 ± 0.239
Llama-3.2-3B-Instruct0.0350.101 ± 0.0610.767 ± 0.034

Arithmetic

BackboneBaseTTRLCoVER
Qwen2.5-0.5B-Instruct0.5130.6290.688 ± 0.009
Qwen2.5-1.5B-Instruct0.3050.683 ± 0.0550.705 ± 0.009
Qwen2.5-1.5B-Instruct, 3–4 digits0.1450.608 ± 0.0040.611 ± 0.005

LiveCodeBench

BackboneBaseTTRLCoVER
Qwen2.5-1.5B10.8311.7213.34
Qwen2.5-1.5B-Instruct7.639.6512.22

Fixed mixing mode:

TaskReward-spaceAdvantage-spaceReplacement
Arithmetic0.688 ± 0.0090.673 ± 0.0160.674 ± 0.015
Countdown0.260 ± 0.0360.471 ± 0.2210.370 ± 0.109
Rubik's0.078 ± 0.0120.096 ± 0.0360.167 ± 0.037

Result: CoVER is above TTRL on every new backbone on Countdown, Arithmetic and LiveCodeBench, including Llama-3.2-3B. On Arithmetic the margin is small (0.705 vs 0.683 and 0.611 vs 0.608 on Qwen2.5-1.5B). The mixing mode moves Arithmetic by ≤1.5pp, and Countdown and Rubik's by up to a factor of two. With any single fixed mixing mode, CoVER stays above TTRL on all three tasks (e.g., advantage-space: 0.673 vs 0.629, 0.471 vs 0.053, 0.096 vs 0.027), but the best mode differs per task, which we acknowledge as a limitation.

W5. Code release

We will release the code, task generators and data upon the acceptance of this work.

Reviewer_RAT7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduce a test-time reinforcement learning that combine majority voting, self-verification, and external programmatic checker. The most appealing component is the programmatic checker as it offer an source of feedback available at time and harder to be hacked as training proceeds. The authors use separate settings to evaluate their methods on five domains and show significant improvement compared with baseline. Further ablation decompose the improvement for each components and order of applying self verifier and programmatic helper.

Critical strengths

The paper manage to introduce a external source of feedback into test-time training that is available at test time. Individually constructed on specific domains, the methods significantly outperform baseline. The ablation further improve the understanding of where the gain comes from. The paper seriously tuned the hyperparameters of baseline according to the appendix and the reported results, making the comparison stronger and more convincing.

Critical weaknesses

Most concerns originates from the varying setting of the proposed method. The choices of the mixing strategy of self verification and the programmatic checker, handover step, and mixing parameter vary across tasks, making the stability under different hyperparameters of the method concerning. Additionally, the paper doesn't disclose how the tuning of the proposed method happens nor study the sensitivity of CoVER. Figure one lacks clarity. It is not informative of how the method differs from other TTT methods and can be confusing without the context of methodology of the paper.

Questions for authors
  1. Can the authors disclose the tuning of CoVER. Is the budget matching the other baseline?
  2. How sensitive is CoVER's performance to its hyperparameter settings, the choice of mixing method, and the choice of programmetric checker?
  3. Can the authors provide the checker only result in the Figure 3 too?
  4. Could the authors include a standard RLVR baseline with ground-truth rewards to provide an approximate performance ceiling for CoVER?
2 comments

We thank the reviewer for the thoughtful review and constructive questions. 

Q1. Tuning procedure and budget

No, the budgets are not matched. For CoVER we varied the switch step S, the checker weight c and the mixing mode, so it usually received more tuning than the baselines. The table below lists the number of configurations tried per method:

TaskCoVERTTRLIntuitorEMPOCo-rewarding
Arithmetic229665
Countdown354126966
Rubik's203112018
LiveCodeBench251232
AIME181354

CoVER had the largest budget on Arithmetic, LiveCodeBench and AIME. On Countdown, EMPO (69) and Co-rewarding (66) were tuned more than CoVER (35), and on Rubik's EMPO matched it (20). TTRL received the fewest configurations on every task except Arithmetic. The comparison therefore favors CoVER in tuning effort on three of the five tasks, and the sensitivity results in Q2 show how much of the result depends on this tuning.

Q2. Sensitivity

CoVER is robust on Arithmetic but sensitive on Countdown and Rubik's. We ran CoVER with different configurations to measure its sensitivity to the hyperparameters, the choice of mixing method, and the choice of programmatic checker.

Arithmetic:

SettingAccuracy
S = 10, c = 0.3 / 0.6 / 1.00.688 / 0.682 / 0.672
S = 20, c = 0.60.684
S = 1, c = 1.00.672
Mixing: reward / advantage / replacement0.688 ± 0.009 / 0.673 ± 0.016 / 0.674 ± 0.015
lr 5e-7 / 1e-60.607 ± 0.082 / 0.458 ± 0.292

On Arithmetic, all S, c and mixing-mode settings stay between 0.672 and 0.688, a range of 1.6pp. The learning rate matters more: a larger learning rate lowers accuracy and makes it unstable across seeds (0.458 ± 0.292 at 1e-6).

Countdown:

SettingAccuracy
Advantage mixing, c = 8 / 16 / 32 / 640.301 / 0.463 / 0.586 / 0.320
Reward mixing, c = 1 / 4 / 160.21 / 0.28 / 0.29
Checker: exact r_valid / graded r_dist / weak r_form0.471 / 0.414 / 0.295
Verification normalization: softmax / absolute (CV only)0.193 ± 0.064 / 0.178 ± 0.079
Verification query: recompute / Yes-No / joint multiple-choice (CV only)0.193 / 0.043 / 0.045

On Countdown, the checker weight c is the most sensitive setting. With advantage mixing, accuracy ranges from 0.301 to 0.586 as c varies, and both too small and too large values hurt. Reward mixing stays at or below 0.29 for all c tested. The checker quality also matters: the exact checker r_valid gives 0.471, the graded r_dist 0.414, and the weak format checker r_form 0.295. For the self-verification part alone (CV only), the normalization choice makes no clear difference, but the query type does: the recompute query gives 0.193, while Yes/No and joint multiple-choice queries give about 0.04.

Rubik's:

SettingAccuracy
Replacement0.167
Additive (c = 32)0.096
Other S / c settings≤ 0.065

On Rubik's, only replacement mixing works well. Additive mixing reaches 0.096, and the other S and c settings stay at or below 0.065.

Result: No single mixing mode is best on all tasks: reward mixing is best on Arithmetic, advantage mixing on Countdown, and replacement on Rubik's. CoVER is robust on Arithmetic (≤1.6pp across c, S and mixing mode). On Countdown and Rubik's, the result depends on c and the mixing mode, by about a factor of two on Countdown and more on Rubik's. We agree with the reviewer that this sensitivity is a limitation of CoVER, and that these settings need to be chosen per task.

Q3. Checker-only in Figure 3

Yes. The table below adds the checker-only scorers to the Figure 3 setting. It reports how often each scorer ranks the correct minority answer first (top-1) on the 365 frozen Countdown groups in which the majority answer is wrong. Majority vote scores 0.000 by construction.

ScorerTop-1
Majority vote0.000
Yes/No self-verification0.139
CoVER (recompute × frequency)0.688
Checker-only, exact r_valid1.000
Checker-only, weak r_form0.344

Result: An exact checker selects the correct minority in every group, as expected, since it checks the answer directly. A weak checker does not: the format checker r_form selects it in only 34.4% of groups, about half the rate of CoVER (68.8%). Self-verification therefore adds the most when the available checker is weak.

Q4. Ground-truth RLVR ceiling

Yes. We trained GRPO on the ground-truth reward as an approximate ceiling for CoVER:

TaskCoVERRLVR
Arithmetic0.688 ± 0.0090.716 ± 0.121
Countdown0.471 ± 0.2210.524 ± 0.181
Countdown, 4 numbers0.089 ± 0.0620.180 ± 0.034
Rubik's0.167 ± 0.0110.179 ± 0.075
LiveCodeBench13.3414.21
AIME23.5225.24

RLVR is higher on every task, as expected for a ceiling. The gap is small on Arithmetic (2.8pp), Countdown (5.3pp), Rubik's (1.2pp), LiveCodeBench (0.87 points) and AIME (1.72 points), and on the first three it lies within the seed variance. The largest gap is on the 4-number Countdown subset, where CoVER reaches about half of the RLVR accuracy (0.089 vs 0.180). CoVER therefore comes close to the ground-truth ceiling on most tasks, without using ground-truth labels.

Figure 1

We agree that Figure 1 does not show how CoVER differs from other TTT methods. We will redraw it as a side-by-side comparison with the TTRL pipeline, showing where each reward reads its signal:

  • TTRL: answer frequency only.
  • CoVER: frequency × verification score, and a checker after step S.
Reviewer_52YLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

oVER combines self-verification with programmatic feedback for reinforcement learning without reference-answer labels. Sampled responses are grouped into answer-equivalence classes, and each answer receives a reward combining its class frequency with the model’s verification preference:

rCV(a)=n(a)vˉ(a).r_{\mathrm{CV}}(a)=n(a)\bar v(a).

Training initially uses this contrastive verification reward. After a specified number of steps, a programmatic checker is introduced through reward-space mixing, advantage-space mixing, or replacement, depending on the task. The intended division is that self-verification provides discrimination when executable feedback is sparse, while the checker supplies grounded corrective information.

The paper reports improvements across five task categories. Arithmetic accuracy increases from 0.513 to 0.688, Countdown accuracy from 0.046 to 0.471, and AIME 2024 avg@8 from 10.42 to 23.52. The AIME experiment adapts on the same unlabeled problems used for evaluation, whereas the other main experiments use separate training and evaluation sets. These results therefore involve different adaptation and generalization settings.

Critical strengths
  1. The combination of model-based assessment and executable constraints is well motivated and has encouraging cross-task results.

    When initial success rates are low, consensus rewards can reinforce incorrect answers, whereas executable feedback can provide information beyond the model’s answer frequencies. The substantial improvements on Countdown and Rubik’s Cube suggest that the combination has practical value, although they do not establish which component produces the gains.

  2. The paper includes targeted diagnostic experiments.

    Tables 7–8 compare individual reward components and alternative handover schedules. Appendix D evaluates candidate selection on fixed rollouts when the majority answer is wrong and a correct minority answer is available. These analyses help distinguish reward-selection behavior from optimization dynamics and provide a useful starting point for investigating the mechanism.

Critical weaknesses
  1. Key experiments do not consistently satisfy the stated supervision definition, and the AIME checker cannot be adequately assessed from the description.

    Definition 1 on page 4 requires the checker to depend only on the prompt and candidate, without accessing dataset fields absent from the prompt. Appendix C on page 20 nevertheless states that LiveCodeBench training uses two stored input–output tests per problem. Even without reference code, expected outputs provide external supervision and do not satisfy the stated definition unless those tests are included in the prompt. For AIME, the checker is described as determining whether each claim is executable and entailed by its cited premises, but the paper does not specify how natural-language problem statements become admissible premises, which inference or execution system is used, or how fabricated premises and irrelevant true statements are rejected. The example in Figure 11 includes 4s=94s=9 and t=24t=24, which do not follow from the displayed problem. It is important to establish whether these are merely flawed prompt examples that the checker rejects. These issues directly affect whether the complex-task results support the grounded, label-free interpretation.

  2. The controls do not establish whether contrastive verification contributes beyond additional programmatic supervision and task-specific design choices.

    In the arithmetic ablation, checker-only training already achieves 0.675±0.0150.675 \pm 0.015, compared with 0.688±0.0090.688 \pm 0.009 for CoVER. The reversed schedule reaches 0.686±0.0150.686 \pm 0.015, so the reported means alone do not establish the importance of the proposed ordering. Equivalent checker-only controls are missing for the other key tasks, including Countdown, where exact executable correctness feedback is available. Moreover, the main text describes joint candidate comparison, while several task-specific implementations in Appendix B verify candidates individually before normalization. The correct-minority analysis compares a Yes/No prompt with a recomputation prompt, confounding the verification procedure with the contrastive structure. The contribution of contrastive verification therefore remains uncertain without controls matching checker access, verification prompts, and additional computation.

Questions for authors
  1. Clarification: Are the two stored LiveCodeBench tests fully included in the input problem statements? If not, do you agree that this experiment does not satisfy Definition 1? Please distinguish prompt-derived feedback, supervision from stored tests, and evaluation tests, and clarify whether the compared methods receive equivalent access to supervision.

  2. Assessment-critical correctness validation: Please provide executable rules or sufficiently detailed pseudocode for the AIME claim checker. How does it establish which premises follow from the problem statement, and does it reject the incorrect premises in Figure 11, fabricated intermediate quantities, and irrelevant true equations? This is necessary to assess whether the reward is genuinely grounded in the problem constraints.

  3. Assessment-critical contribution validation: How does CoVER compare with checker-only training on Countdown, AIME, and code generation under matched supervision and compute budgets? With the recomputation prompt held fixed, does changing only joint candidate presentation or normalization retain the correct-minority selection advantage? These comparisons would directly affect the assessment of the proposed contribution.

1 comment

Thank you for the thoughtful review.

Q1 / W1. LiveCodeBench stored tests vs. Definition 1

No, not fully, and we agree that the affected part of this experiment does not satisfy Definition 1. Both stored tests appear in the problem statement for 2,362 of the 2,978 training problems (79.3%), and at least one for 2,940 (98.7%). For the remaining 616 problems (20.7%) the checker reads at least one expected output that is absent from the prompt; this is external supervision from stored tests, distinct from the evaluation tests. We will detail this in the revised version and no longer describe the LiveCodeBench result as fully label-free.

Q2 / W1. AIME claim checker

Below is the pseudocode of the AIME claim checker, which is implemented in Python with SymPy.

claims = last <claims> block, ≤24 lines "k. [premises: P] e0 = e1 = ... = em"
env = {}                                   # symbol -> value, bound by passed claims
for claim t:
  fail if P empty, or any p ∉ {problem} ∪ {1..t-1}
  subs  = values bound by the cited claims only
  exprs = sympy(e_i).subs(subs)            # fail if any side does not parse
  if exprs[0] is an unbound symbol: drop it (definition "x = ...")
  holds  = every adjacent pair simplifies to 0 difference
  closed = no free symbols remain
  pass   = holds                  if closed
         = holds or 'problem'∈P   otherwise
  if pass and the claim pins one symbol to a number: bind it
r_trace = (#passed) / (#claims)

The checker in the submission cannot establish which premises follow from the problem statement. A claim that cites problem and still has free symbols passes, so the checker accepts the incorrect premises in Figure 11 (4s=9, t=24). By the same rule, it does not reject fabricated intermediate quantities. It also passes irrelevant true equations, since any closed equation that holds is accepted. We will revise the text to describe the AIME checker as a consistency check rather than a grounded verifier.

Q3a / W2. Checker-only under matched supervision

TaskCheckernBaseChecker-onlyCoVER
Countdown, 3 numbersexact r_valid120.0440.524 ± 0.1810.471 ± 0.221
Countdown, 4 numbers (no re-tuning)exact r_valid60.0190.180 ± 0.0340.089 ± 0.062
Countdown, 3 numbersweak r_form (never reads the target)120.0440.050 ± 0.0060.295 ± 0.078
Rubik'sgraded r_progress120.0240.151 ± 0.0490.167 ± 0.037
Arithmetic (0.5B)partial120.4950.664 ± 0.0160.679 ± 0.009
Arithmetic (1.5B)partial30.3050.686 ± 0.0010.705 ± 0.009

Results. With an exact checker, checker-only is at least as good as CoVER. CoVER is ahead where the checker is partial or weak. CoVER is proposed for the regime where the checker is absent or weak.

Q3b / W2. “With the recomputation prompt held fixed, does changing only joint candidate presentation or normalization retain the correct-minority selection advantage? “

No. On the Countdown task, the correct-minority advantage comes from the recomputation query. With recomputation fixed, competition normalization and absolute scoring are not distinguishable. We therefore attribute this advantage to the recomputation query rather than to the contrastive structure.

W2. Ordering

We reran the ablation on Countdown and Arithmetic with more seeds (PC: Programmatic Checker; CV: Contrastive Verification).

Arithmetic (0.5B):

Schedule3 seeds (paper)12 seeds
PC only0.675 ± 0.0150.664 ± 0.016
PC → CV0.686 ± 0.0150.683 ± 0.009
CV → PC (CoVER)0.688 ± 0.0090.679 ± 0.009

Countdown with the weak checker r_form:

ScheduleAccuracy
r_form only0.050 ± 0.006
CV only0.193 ± 0.064
r_form → CV0.148 ± 0.094
CV → r_form (CoVER)0.295 ± 0.078

Result: Both the proposed schedule and the reversed schedule score above checker-only. On Arithmetic, a self-verification phase helps in either order, and we agree that the ordering itself is not distinguishable there. On Countdown with a weak checker, the CoVER order is the better one.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.