- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper did a series of experiments on logic and planning puzzles. They found that a verifier that checks the final answer cannot guarantee the correctness of the reasoning process. When such accepted solutions are used for training models, such behavior will be inherited by the student model. If such solutions with wrong reasoning are filtered before training, the behavior will also be mitigated.
Critical strengths
The experiment and the findings are clearly explained. The experiment covers different model families and sizes.
Critical weaknesses
-
Many expressions in the paper are not formal, which makes it hard to read. For example, lines 459-460 will be easier to follow if it is paraphrased as “An outcome verifier evaluates only the correctness of the final answer, leaving intermediate claims unchecked and potentially allowing false claims to accumulate.” It would be better to polish the paper with AI models.
-
The findings are not new and mostly intuitive. It is well known that if a verifier only checks the final answer, the correctness of the reasoning cannot be guaranteed. That’s why we need process reward model.
-
The experiments only cover small models, up to 32B, and mostly smaller than 14B. Such findings may not be valid for strong LLMs, which should also be included in the analysis.
Questions for authors
Why do you choose the three LLM judges? Especially gpt-oss-120b, which is an old model.
2 comments
With the discussion period closing soon, we wanted to gently follow up in case you have any remaining questions about our responses. We would really appreciate your feedback. If we have resolved your concerns, we would be very grateful if you would please consider raising your score. Thank you again for your time.
Response to Reviewer FsxU
W1. Writing. L459-460 now reads, as you suggested: "An outcome verifier evaluates only the correctness of the final answer, leaving intermediate claims unchecked and potentially allowing false claims to accumulate." It opens the conclusion. L408-411 is replaced by a definition of alpha and beta in the main text. We also rewrote the abstract, the results list and closing paragraphs of the introduction, Section 4, the openings of Sections 5 and 6, and the conclusion; replaced the slogan-style section headings with descriptive ones; and each results paragraph in Sections 5 and 6 now opens with the conclusion its numbers support. All changes are in blue.
W2. "The findings are known; that is why we need process reward models." That a verifier checking only the final answer cannot guarantee the reasoning is established, and the revision cites the work that established it (Bansal et al., 2025; Wang et al., 2025). The paper's contribution is to measure the consequences exactly, in domains chosen so that every intermediate claim of every accepted solution can be checked by a solver, simulator or executor. One of the consequences measured is the remedy you name. An off-the-shelf process reward model (Qwen2.5-Math-PRM-7B) separates flawed from clean accepted solutions at AUROC 0.74 in Knights-and-Knaves and 0.60 in Blocksworld; at the precision of a simple string match it flags 1 of 500 flawed solutions, and its lowest-scored step lands on the first false claim in under 1% of them (App. N). A stronger test was run with GRPO: a reward requiring every claim to pass the exact checker gives 0.3-1.1% false claims on newly solved problems at pass@1 65.5-65.8, against 87-93% at 70.6-71.7 for the outcome reward alone, and reaches it by assuming the correct roles and confirming them: 99.9-100% of its newly solved problems open on the true role and 0.3-0.7% ever reject a hypothesis. A process reward, even an exact one, removes false claims without restoring derivation; checker filtering has the same limit. The pipelines in use (STaR, rejection-sampling fine-tuning, ReST-EM, hint-guided RL) filter on the answer alone, and what that filter admits is a measured question. The general response summarizes what those measurements show: rates that vary fivefold across models of similar size (10.8% for Qwen3-4B, 58.8% for Llama-3.1-8B) and persist at 72B to 400B, inheritance at unchanged accuracy, an 8-point pass@1 gain from checker filtering on Llama-3.1-8B's own solutions, and a 25.8% against 8.1% solve rate in Sokoban.
W3. Model scale. The submission spans 1.7B to 32B in three families, and within Qwen3 and OLMo-3 the rate falls with size without vanishing (81.4% at Qwen3-1.7B to 10.1% at 14B; 88.6% at OLMo-3-7B to 30.9% at 32B). We have now measured four larger models: Llama-4-Maverick (about 400B), GLM-4.7 and Qwen3-235B-A22B through an API on the same 300 hard Knights-and-Knaves puzzles (16 samples per condition, balanced examples), and Qwen2.5-72B-Instruct locally as an AWQ checkpoint on the 310 puzzles it cannot solve unaided (32 samples, both prompts). Unaided accepted solutions contain a false claim 12.8%, 11.4%, 18.8% and 16.8% of the time. For the three API models, first steps give 0.7-3.3% and the hint lowers the rate below unaided by 9-12 points, every CI excluding zero. For Qwen2.5-72B neither comparison separates: first steps give 28-30% and the hint-minus-unaided difference includes zero under balanced examples, on 40-54 paired puzzles. These are quantized instruct checkpoints, used for measurement only. The training half of the paper needs models with unsolved problems to recover and weights that can be fine-tuned, which excludes API models; LoRA at 70B is the practical ceiling for that half and was not completed in the discussion period. The phenomenon is present in the largest models we could measure, at a lower rate than in small ones.
Q. Choice of judges. The LLM judges play two roles, neither of which sets any rate in the paper. First, they validate the checker: on the planning domains GPT-6 Sol agrees with the checker line for line (App. G), which checks the claim extraction. Second, they test whether a judge could replace the checker, since most domains have no checker. GPT-6 Sol is the strongest judge available to us. gpt-oss-120b is the open-weight reference point: it runs on one 80 GB GPU with no API; DeepSeek-V4-Pro is a second frontier judge. On whether a solution contains a false claim, all three judges nearly match the checker; GPT-6 Sol matches it on every line in Blocksworld and Sokoban. Where they differ is in locating claims: gpt-oss-120b flags 154 of 267 true Knights-and-Knaves lines and misses 32 of 72 false ProntoQA claims, so it could not replace the checker as a claim-level filter, while a strong judge could, at a per-solution cost. Every rate in the paper comes from the checker; the judges are only compared against it.