acceptodds
Under review as a conference paper at ICLR 2027

Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers

Abstract

Test-time scaling via solution sampling and aggregation has become a key paradigm for improving the reasoning performance of Large Language Models (LLMs). While reward model selection is commonly employed in this approach, it often fails to identify minority-yet-correct answers, which limits its effectiveness beyond that of simple majority voting. We argue that this limitation stems from a lack of informative critique signals during verifier training. To bridge this gap, we introduce Mirror-Critique, a framework that trains a verifier with informative critiques. Our key insight is to leverage the rich critique signal by contrasting model-generated solutions with ground-truth solutions. We deploy a small instruction-tuned model to synthesize high-quality critique data with rejection sampling that teaches the verifier not only what is wrong, but also why. The synthetic data is used to cold-start the LLMs in the RLVR process to further improve the verification ability. The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution, aggregating them into a verify score used for weighted voting or selective abstention. The experimental results show that our Mirror-Verifier significantly outperforms majority voting in terms of solution accuracy and also improves the solver's honesty to recognize and abstain from answering beyond its capability boundaries.

Primary area:
reinforcement learning
Keywords:
LLM, Reasoning, Verify

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AkJTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mirror-Critique, a framework for training generative verifiers to improve mathematical reasoning at test time. The approach generates reference-guided critiques using an instruction-tuned model, filters them based on judgment correctness, and uses the resulting data for supervised fine-tuning followed by reinforcement learning with verifiable rewards. At inference time, the trained verifier generates multiple critiques per candidate solution and aggregates their judgments for weighted voting and selective abstention. Experiments on five mathematical reasoning benchmarks with Qwen2.5-Math 1.5B and 7B solvers demonstrate improvements in answer accuracy and an honesty metric over majority voting and several learned-verifier baselines.

Critical strengths
  1. Practical training pipeline. The paper presents a straightforward framework combining reference-guided critique generation, supervised fine-tuning, and verifier reinforcement learning. The approach leverages existing reference solutions and open-source models without requiring manually written critiques.
  2. Improvements across multiple benchmarks. The proposed method improves overall accuracy from 55.7% to 58.2% for the 1.5B solver and from 61.8% to 63.0% for the 7B solver compared with majority voting. The evaluation across five mathematical reasoning benchmarks demonstrates the effectiveness of the proposed approach.
  3. Joint consideration of accuracy and abstention. Beyond answer selection, the paper investigates selective abstention through an explicit utility-based honesty metric. The comparison between Mirror-SFT and Mirror-Verifier provides evidence that reinforcement learning can improve both answer selection and abstention decisions.
Critical weaknesses
  1. High inference cost for relatively modest improvements. The proposed method generates 16 critiques for each of 16 candidate solutions, requiring 256 additional critique generations per question. However, the improvements over the strongest learned-verifier baselines are only 0.9 and 0.7 percentage points for the 1.5B and 7B solvers, respectively. The absence of compute-matched comparisons makes it difficult to assess the practical efficiency of the approach.
  2. Insufficient ablation studies and limited methodological novelty. The framework combines several existing techniques, including critique-based SFT, balanced sampling, difficulty filtering, and RLVR. Although the paper compares Mirror-SFT with Mirror-Verifier, it does not isolate the individual contributions of these components. In particular, the necessity of reference-guided critique initialization over direct outcome-supervised RLVR remains unclear.
  3. Limited robustness analysis and evaluation of verification quality. The reported improvements over strong baselines are relatively small, without statistical significance analysis or uncertainty estimates. Furthermore, the synthetic critique quality assessment uses only 30 examples per dataset, and the honesty evaluation relies on a fixed abstention threshold despite different verifier score distributions. These limitations weaken the evidence for robust improvements in verification reliability.
Questions for authors
  1. Inference efficiency: Can the authors provide compute-matched comparisons with majority voting and existing verifier baselines? Specifically, under the same token or FLOP budget, would generating more solver candidates or using a scalar reward model achieve comparable performance?
  2. Ablation studies: What is the performance of a verifier trained directly with RLVR without critique-based SFT initialization? Additionally, how much do balanced sampling and difficulty-based filtering individually contribute to the final improvements?
  3. Robustness and calibration: Are the reported improvements statistically significant across multiple runs or bootstrap evaluations? When abstention thresholds are independently optimized on held-out validation data, does Mirror-Verifier still outperform competing methods at matched coverage levels?
Reviewer_stkxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about improving test-time scaling by training a verifier to actually check solutions instead of just relying on majority voting.

  • The main idea is to generate critiques by comparing solver outputs to reference solutions. They filter these critiques based on whether the final judgment is correct, use them for SFT, and then train the verifier further with RLVR.
  • At inference, they generate multiple critiques for each solution. They use the resulting scores for weighted voting and also let the model abstain when it's not confident enough.
  • They evaluate on five math benchmarks with 1.5B and 7B Qwen models. Overall accuracy improves from 55.7% to 58.2% for 1.5B and 61.8% to 63.0% for 7B over majority voting. They also show gains in their honesty metric, which rewards correct answers, penalizes wrong ones, and gives zero for abstaining.
Critical strengths

I like the way they approach the verifier training problem. A big issue with majority voting is that models can repeatedly produce the same wrong answer. Just sampling more doesn't necessarily fix that. Training a verifier to look at individual reasoning trajectories is a pretty reasonable way to deal with this. The reference-guided critique generation is also a nice idea. Instead of trying to get a base model to learn verification entirely from binary rewards, they first give it examples of what checking a solution should look like. I think this is one of the more useful parts of the paper. It also avoids needing a closed model to generate all the critique data. The experimental results are pretty encouraging, especially the SFT vs. RLVR comparisons. They evaluate on five different math benchmarks and two model sizes. The improvements over majority voting aren't huge, but they're consistent in aggregate. For the 1.5B model, RLVR improves accuracy from 55.5% with SFT alone to 58.2%. The honesty score also increases quite a bit, from 20.5 to 32.7. I thought that was a pretty meaningful result since it suggests the RL stage is helping beyond just teaching the model to produce critiques. It's also nice that they evaluate abstention instead of only forced-choice accuracy. Being able to recognize when not to answer is a useful capability, and the paper actually measures that tradeoff.

Critical weaknesses

The biggest issue for me is the inference cost relative to the gains. The main setup uses 16 solver candidates and generates 16 critiques for every candidate. That's 256 generated critiques for a single question, on top of generating the original solutions. This seems like a lot of additional work for improvements of 2.5 percentage points at 1.5B and 1.2 points at 7B over majority voting. The gains over the strongest learned-verifier baselines are even smaller, at 0.9 and 0.7 points. I don't think this necessarily means the method is inefficient, but it's hard to judge without an equal-compute comparison. For example, what if the baseline could use that same compute budget to generate more solver samples or use a different verifier setup? The paper shows that the method improves accuracy at its chosen settings, but it doesn't really establish whether this is a good use of inference compute. Since test-time scaling is such a central part of the motivation, I think this matters quite a bit. I'm also not fully convinced which parts of the training pipeline are actually responsible for the gains. The paper shows that adding RLVR after SFT helps, which is good, but several design choices are bundled together. They balance correct and incorrect examples, filter for questions with less than 60% solver accuracy, and use reference-guided synthetic critiques for initialization. The appendix gives some evidence that balancing helps avoid an always-False verifier, but I would still want a more systematic ablation of these components. In particular, it's not clear how much the reference-guided critiques themselves contribute compared to a simpler SFT or RLVR baseline. There's also the issue that the critique filtering only checks the final True/False judgment, not whether the actual explanation is right. Their human evaluation finds only 80.0% and 76.7% critique correctness on samples of 30 examples per dataset. That's a pretty small audit, and it suggests there is still a decent amount of noise in the training data. I think more analysis here would make the central contribution much more convincing.

Questions for authors
  • Have you tried varying the number of critiques per solution? I'm curious how much performance drops if you use 1, 4, or 8 critiques instead of 16. Also, how would this compare to the baselines under the same inference compute budget? This would help me better understand whether the accuracy gains are worth the additional cost.
  • Could you provide ablations separating reference-guided SFT, class balancing, and the hard-question filtering? In particular, would a simpler verifier trained with the same RLVR setup get similar results?
  • Do you have a larger evaluation of the actual critique quality? Since RL only rewards the final judgment, I'm curious whether the model is genuinely learning to identify reasoning errors or sometimes producing incorrect explanations alongside correct labels.
Reviewer_N93DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a pipeline for training generative verifiers using RL. The main idea is to collect training trajectories that involve critiques which led to the correct answers. These trojectories are used as SFT data. The verifier is then trained with GRPO-alike method with balanced data. The effectiveness of the trained verifier is verified using ttt scaling performance. The paper also shows that an honesty utility (+1 correct /-1 wrong/ 0 abstain) also improves the performance of the 1.5B model.

Critical strengths

The paper proposes a cost-effective way to generate the sft data using open models and reference-guided tasks.

The authors did careful ablations between the sft and rl stages to attribute the performance improvement.

Clear perofmrance improvements for the 1.5B model.

Critical weaknesses
  1. The performance improvement at 7B is very marginal compared to Skywork PRM.

  2. The proposed approach seems to use more critique generations per question? Compared to majority voting and other PRMs.

  3. The comparison with the baselines seem a bit unfair as the other approaches are not given the abstain option.

  4. Missing baselines, at least the reviewer is aware of DeepSeek-GRM to compare with.

Questions for authors

Please refer to the weakness section.

The reviewer is also curious how the verifier performs on larger reasoning models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.