acceptodds
Under review as a conference paper at ICLR 2027

From-Base RLVR for Verify-Only Judges: A Rule-Based Reward Turns Base Models into Trajectory Auditors

Abstract

Reinforcement learning from verifiable rewards (RLVR) is the standard recipe for improving how language models solve tasks. We ask whether the same rule-based binary signal can instead train an effective verify-only judge directly from a base model. Casting judging as a balanced accept/reject task and rewarding the model only when its parsed verdict matches the ground truth—no reward model, preference pairs, or human rationales—we optimize with GRPO on two verification-hard domains: auditing GAIA agent trajectories and verifying MuSiQue multi-hop factual answers. The payoff on trajectory auditing is large and replicates across three model families (Qwen3-8B, Qwen2.5-7B, Llama-3.1): e.g. verify accuracy improves (McNemar ), transfers to a real out-of-distribution set of multi-agent failures, and increases with task difficulty. On multi-hop factual verification, a compact 8B judge attains a lower miss rate than four frontier zero-shot judges, albeit with a false-alarm trade-off. We further report two honest negatives—tripling the data gives no gain (the recipe is sample-efficient), and the multi-hop factual payoff does not replicate on one base—that sharpen when the payoff holds. From-base RLVR is a simple, reproducible route to trustworthy verify-only judges.

Primary area:
reinforcement learning
Keywords:
RLVR, verifiable rewards, LLM-as-a-judge, verification, GRPO, trajectory auditing, multi-hop factual verification, from-base reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_qUHtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether reinforcement learning with verifiable rewards (RLVR) can train a binary verification judge directly from an existing language model. The proposed approach applies GRPO with a binary reward determined by whether the model's final CORRECT/WRONG verdict matches the ground-truth label. The authors evaluate this approach on GAIA-derived agent trajectory auditing and MuSiQue multi-hop factual verification. The experiments report improvements over the original models, including substantial gains on trajectory auditing across multiple backbones and some transfer to real out-of-distribution agent failures. Additional analyses examine false-alarm and miss rates, task difficulty, training data scale, and reward design.

Critical strengths
  1. Empirical improvements across different model backbones.

The paper reports substantial improvements in trajectory verification accuracy across Qwen and Llama models.

  1. Analysis of different verification errors.

The paper separately reports false-alarm and miss rates and acknowledges cases where accuracy improvements mainly result from changes in decision bias rather than improved error detection.

Critical weaknesses
  1. A lack of innovation is a flaw sufficient to warrant the rejection of a manuscript.
Questions for authors

Why did you write such a long "Related Work" section?

Reviewer_jVpGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to train models to act as reliable judges with reinforcement learning. Usually RL requires expensive human feedback, the paper proposes to utilize rule-based binary reward i.e. accept or reject. This enables to train base models to work on multi-step tasks without expensive training data.

Critical strengths
  • It proposes an interesting approach to use binary reward to train base models to be a judge, and it is shown that the proposed method is sample efficient to address multi-step tasks. I believe the sample efficiency is a critical topic in training LLMs and approaching that limitation is important.
Critical weaknesses
  • I doubt if the major part of the paper is written by humans. There are signs, e.g. em-dash(—) appears 43 times in pdf.
  • The presentation is suboptimal. For example, in Fig 2., the labels on x-axis are overlapping and not readable. The entire main figure (Fig 1) looks also AI-generated, because no text is embedded (they cannot be selected by dragging) and pixel noises around some characters and icons (e.g. the C in red box, left bottom).
Questions for authors
  • How can the proposed approach be extensively applied to train models on true verification reasoning rather than just updating the baseline towards accept/reject bias?
Reviewer_mD4dOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the question whether RLVR can directly train verify-only judge from base model using binary clasification signal. The experiments trained 3 models using GRPO, and run evaluation on GAIA trajectory auditng and MuSiQue multi-hop factual verification. The results shows the model performance increase on the in-distribution and OOD setup.

Critical strengths
  • This work shows RLVR approach can improve the verification classification capability.
Critical weaknesses
  • Motivation is hard to understand.
    • This work takes verification as binary classification, however, it does not show why RLVR is needed instead of simple SFT. As the reward label is simply available, using RLVR with higher cost and lower reward density is irrational.
    • This work focus on the RL from base model, but it makes no sense to study from a base model instead of a post-trained model, or at least under initial SFT. The advantage of this setup starting from base model is specious.
  • Experiment setup needs justification.
    • The so-called base model setup for Qwen3-8B and Llama-3.1-8B-Instruct is actually post-trained model checkpoints. Also, if the author take any checkpoint before RL as 'base' model, the claim of 'from-base' RLVR is not valid at all.
  • It is unclear why the tasks claimed as verification hard is hard. Specifically, the prompting for MuSiQue is not even given a reference answer, the challenge for judge is more towards answering instead of towards verification. Even for the GAIA task, the model is supposed to known the grounding to judge the result.
  • Result lacks supportive evidance.
    • The result reduces false positive but does not improve true positive rate. It is more alike lable calibration, instead of truly capability increase.
  • Writing is in extremely low quality, hard to read and follow.
Questions for authors

NA

Comments on the paper

2 comments

Dear Reviewers,

The authors have replied to your initial comments. Please have a look and engage in the discussion.

Thanks!

Your AC

Thank you for your response. Having reviewed the questions raised by myself and the other reviewers, along with your replies, I have decided to keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.