- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the effect of answer extraction in RLVER. Using Qwen2.5 backbone and two datasets (MedQA, ARC), the authors find that different answer extractor could lead to very different results: when the answer is extracted by the default parser, RLVR leads to a +5.62 performance gain; when extracted by LLM judges (deepseek-v4-pro and llama-3.3-70b-instruct), RLVR leads to performance drops instead. The authors also find that changing the answer format could lead to different evaluation results.
Critical strengths
None
Critical weaknesses
-
The paper is hard to understand. It never defines what a "reader" is. From the context I guess that the authors use "extractor" and "reader" interchangeably, but I cannot find any evidence. It should be noted that "reader" is more commonly used to refer to the policy LLM in RAG literature.
-
The experimental scope is very small, with two little-used datasets in RLVR (MedQA and ARC) and one outdated model (Qwen2.5-7B).
-
No detail is given on how the default answer parser is implemented.
-
Inconsistent results: in Table 2 the baseline score is 53.15, yet in Table 3 it is 52.79.
-
The paper lacks in-depth analysis, e.g., why do the extraction results differ between the parser and the LLM judges?
Questions for authors
-
How is the answer parser implemented?
-
When the answer parser and LLM judges disagree, which one tends to be correct? This is probably what the audience care about most in this paper.