acceptodds
Under review as a conference paper at ICLR 2027

When the Answer Extractor Is Part of RLVR Evaluation

Abstract

In closed-set multiple-choice RLVR, one answer extractor often assigns reward and reports test accuracy. A score change under this shared reader can reflect a better answer choice, a more readable answer format, or both. We audit this coupling with frozen outputs and an independent output-contract intervention. On 1,270 common MedQA items, the shared parser reports a +5.62 point training gain, while two independent model readers report −0.52 points (DeepSeek; 95% CI [−1.71, +0.66]) and −1.94 points (Llama; [−3.33, −0.63]). With the same model weights and a 512-token budget, retaining the explanation but requiring a terminal Answer: X changes the shared-parser gain from +5.87 to +0.47 points; the training-by-prompt interaction is −5.40 points (95% CI [−8.37, −2.10]). The answer marker appears in 97.8–99.6% of completions, although exact final-line compliance varies by checkpoint. These results do not determine semantic truth from any single reader. They show that the reported RLVR gain depends materially on the interface that reads the answer, and that the shared reader should not be the sole validator.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, large language model post-training, answer extraction, evaluation reliability, evaluation confounds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P8vDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the effect of answer extraction in RLVER. Using Qwen2.5 backbone and two datasets (MedQA, ARC), the authors find that different answer extractor could lead to very different results: when the answer is extracted by the default parser, RLVR leads to a +5.62 performance gain; when extracted by LLM judges (deepseek-v4-pro and llama-3.3-70b-instruct), RLVR leads to performance drops instead. The authors also find that changing the answer format could lead to different evaluation results.

Critical strengths

None

Critical weaknesses
  • The paper is hard to understand. It never defines what a "reader" is. From the context I guess that the authors use "extractor" and "reader" interchangeably, but I cannot find any evidence. It should be noted that "reader" is more commonly used to refer to the policy LLM in RAG literature.

  • The experimental scope is very small, with two little-used datasets in RLVR (MedQA and ARC) and one outdated model (Qwen2.5-7B).

  • No detail is given on how the default answer parser is implemented.

  • Inconsistent results: in Table 2 the baseline score is 53.15, yet in Table 3 it is 52.79.

  • The paper lacks in-depth analysis, e.g., why do the extraction results differ between the parser and the LLM judges?

Questions for authors
  • How is the answer parser implemented?

  • When the answer parser and LLM judges disagree, which one tends to be correct? This is probably what the audience care about most in this paper.

Reviewer_kxHKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper audits multiple-choice RLVR when the same answer extractor supplies training rewards and test scores. It rereads frozen base and trained outputs with two model readers and tests an explicit answer-field prompt. On MedQA, the shared parser reports a 5.62-point gain that neither reader reproduces. The prompt intervention reduces the measured gain from 5.87 to 0.47 points.

Critical strengths

None

Critical weaknesses
  1. The methodological contribution is incremental. Molfese et al. (2025), already cited by the paper, study extraction errors, prompt constraints, and agreement with human judgments. The distinctive addition here is applying alternative readers and an answer-format intervention to the before-and-after RLVR training contrast. This is a useful extension, but the conceptual and methodological advance over existing extraction audits is modest.
  2. The experimental scope is narrow. The main evidence comes from one Qwen2.5-7B-Instruct setup and MedQA. The three training seeds strengthen this case, while ARC provides a smaller check drawn from an existing summary. Applying the same audit to another policy backbone or another commonly used reward/evaluation parser would establish whether the finding extends beyond this particular setup. Presentation problems also make the evidence harder to assess, particularly the broken citation and table links and the scattered explanation of scoring rules and denominators.
Questions for authors
  1. Can you validate the model readers on actual benchmark responses, including disagreement cases and a sample of agreement cases? The 12 synthetic cases check explicit choices but do not cover ambiguous prose, competing choices, or implicit answers. A blinded human audit would clarify which readings are correct when both model readers disagree with the parser.
  2. What exact extraction, fallback, and abstention rules produce the large baseline disagreement? Appendix E describes a label-only release, which supports aggregation checks but not verification of extraction decisions. Please provide the parser, generation prompts, and representative completions where both readers disagree with the parser.
  3. Can the two independent readers be applied to both conditions of the answer-contract experiment? The matched checkpoints and token cap do not prevent the prompt from changing the explanation or selected answer. Comparing readers across both conditions would help distinguish reduced extraction disagreement from changes in the answers themselves.
  4. Please fix the citation and table hyperlinks. Section 3 could state standard accuracy and the gained-minus-lost counting identity more briefly while retaining the paired gain and prompt interaction, leaving space for concrete extraction examples. Before Tables 3-5, a compact overview of shared-parser accuracy, strict accuracy, and accuracy among accepted responses, with their rules, denominators, and a worked example, would make Section 5.2 easier to follow.
Reviewer_k5yKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether gains from reinforcement learning depend on how answers are read and scored. It scores saved answers with independent model readers, then tests a prompt that asks for an explicit final answer. On MedQA, the gain measured by the parser used during training does not carry over to either check. The authors treat this as evidence that the measured gain depends on the evaluation setup.

Critical strengths

I think the paper provides useful evidence that the apparent benefit of training can depend on how answers are scored. In the MedQA experiment, the gain disappears when the same saved responses are read by other evaluators. Using identical responses isolates the effect of the evaluator. The separate experiment requesting an explicit final answer provides another check of how much the result depends on the evaluation setup.

The authors are careful about what these comparisons show. They acknowledge that AI readers can make mistakes and that changing the prompt can change the answer itself. The analysis compares the same questions across conditions and accounts for the reuse of base-model answers when estimating uncertainty.

Critical weaknesses

My concern is that the paper gives limited explanation of why the graders disagree. Most disagreements concern the base model, but the supplied materials omit the shared and strict parser rules and the actual responses. IProviding the rules and examples of disputed responses would help explain how much the finding depends on the particular parsers used.

However I think this concern is not critical, and hence I recommended for acceptance.

Questions for authors

The supplement shows almost no training gain under the original prompt when using the strict parser. What does it read differently from the shared parser in the base model's answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.