acceptodds
Under review as a conference paper at ICLR 2027

Right Answer, Wrong Reasoning in Verifier-Filtered Self-Training

17 views

Abstract

Self-training for reasoning keeps a sampled solution only if its final answer is verified correct, and recovers problems the model cannot solve by showing it the answer. In logic and planning puzzles where every intermediate claim can be checked exactly, a verified answer turns out to say little about the reasoning behind it. The claims in a solution are of two kinds: forced claims, which the answer depends on, such as the moves of a plan, and unforced claims, which it does not, such as descriptions of the state along the way. Any solution the verifier accepts has its forced claims right almost by construction; its unforced claims go unchecked, and in accepted solutions written with an answer hint up to 74% of them are false. On Knights-and-Knaves problems the model could not solve, 67.1% of these solutions contain a false claim; giving the same information as the opening steps of a solution lowers this to 0.4%. The false claims do not stay in the data. Models trained on accepted but unsound solutions give correct answers resting on a false claim 53–75 percentage points more often than models trained on sound solutions to the same problems, with no consistent difference in accuracy, and the same inheritance appears in planning. Filtering the accepted solutions with the checker before training removes almost all of it. The effect appears in all three model families measured (Qwen, Llama, and OLMo) and, in Qwen3 and OLMo-3, shrinks with scale without vanishing. An outcome verifier certifies answers, not reasoning, and a model trained on what it certifies can learn to be right for the wrong reasons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-training, rejection sampling, chain-of-thought faithfulness, process supervision, reasoning, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FsxUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper did a series of experiments on logic and planning puzzles. They found that a verifier that checks the final answer cannot guarantee the correctness of the reasoning process. When such accepted solutions are used for training models, such behavior will be inherited by the student model. If such solutions with wrong reasoning are filtered before training, the behavior will also be mitigated.

Critical strengths

The experiment and the findings are clearly explained. The experiment covers different model families and sizes.

Critical weaknesses
  1. Many expressions in the paper are not formal, which makes it hard to read. For example, lines 459-460 will be easier to follow if it is paraphrased as “An outcome verifier evaluates only the correctness of the final answer, leaving intermediate claims unchecked and potentially allowing false claims to accumulate.” It would be better to polish the paper with AI models.

  2. The findings are not new and mostly intuitive. It is well known that if a verifier only checks the final answer, the correctness of the reasoning cannot be guaranteed. That’s why we need process reward model.

  3. The experiments only cover small models, up to 32B, and mostly smaller than 14B. Such findings may not be valid for strong LLMs, which should also be included in the analysis.

Questions for authors

Why do you choose the three LLM judges? Especially gpt-oss-120b, which is an old model.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FsxU

W1. Writing. L459-460 now reads, as you suggested: "An outcome verifier evaluates only the correctness of the final answer, leaving intermediate claims unchecked and potentially allowing false claims to accumulate." It opens the conclusion. L408-411 is replaced by a definition of alpha and beta in the main text. We also rewrote the abstract, the results list and closing paragraphs of the introduction, Section 4, the openings of Sections 5 and 6, and the conclusion; replaced the slogan-style section headings with descriptive ones; and each results paragraph in Sections 5 and 6 now opens with the conclusion its numbers support. All changes are in blue.

W2. "The findings are known; that is why we need process reward models." That a verifier checking only the final answer cannot guarantee the reasoning is established, and the revision cites the work that established it (Bansal et al., 2025; Wang et al., 2025). The paper's contribution is to measure the consequences exactly, in domains chosen so that every intermediate claim of every accepted solution can be checked by a solver, simulator or executor. One of the consequences measured is the remedy you name. An off-the-shelf process reward model (Qwen2.5-Math-PRM-7B) separates flawed from clean accepted solutions at AUROC 0.74 in Knights-and-Knaves and 0.60 in Blocksworld; at the precision of a simple string match it flags 1 of 500 flawed solutions, and its lowest-scored step lands on the first false claim in under 1% of them (App. N). A stronger test was run with GRPO: a reward requiring every claim to pass the exact checker gives 0.3-1.1% false claims on newly solved problems at pass@1 65.5-65.8, against 87-93% at 70.6-71.7 for the outcome reward alone, and reaches it by assuming the correct roles and confirming them: 99.9-100% of its newly solved problems open on the true role and 0.3-0.7% ever reject a hypothesis. A process reward, even an exact one, removes false claims without restoring derivation; checker filtering has the same limit. The pipelines in use (STaR, rejection-sampling fine-tuning, ReST-EM, hint-guided RL) filter on the answer alone, and what that filter admits is a measured question. The general response summarizes what those measurements show: rates that vary fivefold across models of similar size (10.8% for Qwen3-4B, 58.8% for Llama-3.1-8B) and persist at 72B to 400B, inheritance at unchanged accuracy, an 8-point pass@1 gain from checker filtering on Llama-3.1-8B's own solutions, and a 25.8% against 8.1% solve rate in Sokoban.

W3. Model scale. The submission spans 1.7B to 32B in three families, and within Qwen3 and OLMo-3 the rate falls with size without vanishing (81.4% at Qwen3-1.7B to 10.1% at 14B; 88.6% at OLMo-3-7B to 30.9% at 32B). We have now measured four larger models: Llama-4-Maverick (about 400B), GLM-4.7 and Qwen3-235B-A22B through an API on the same 300 hard Knights-and-Knaves puzzles (16 samples per condition, balanced examples), and Qwen2.5-72B-Instruct locally as an AWQ checkpoint on the 310 puzzles it cannot solve unaided (32 samples, both prompts). Unaided accepted solutions contain a false claim 12.8%, 11.4%, 18.8% and 16.8% of the time. For the three API models, first steps give 0.7-3.3% and the hint lowers the rate below unaided by 9-12 points, every CI excluding zero. For Qwen2.5-72B neither comparison separates: first steps give 28-30% and the hint-minus-unaided difference includes zero under balanced examples, on 40-54 paired puzzles. These are quantized instruct checkpoints, used for measurement only. The training half of the paper needs models with unsolved problems to recover and weights that can be fine-tuned, which excludes API models; LoRA at 70B is the practical ceiling for that half and was not completed in the discussion period. The phenomenon is present in the largest models we could measure, at a lower rate than in small ones.

Q. Choice of judges. The LLM judges play two roles, neither of which sets any rate in the paper. First, they validate the checker: on the planning domains GPT-6 Sol agrees with the checker line for line (App. G), which checks the claim extraction. Second, they test whether a judge could replace the checker, since most domains have no checker. GPT-6 Sol is the strongest judge available to us. gpt-oss-120b is the open-weight reference point: it runs on one 80 GB GPU with no API; DeepSeek-V4-Pro is a second frontier judge. On whether a solution contains a false claim, all three judges nearly match the checker; GPT-6 Sol matches it on every line in Blocksworld and Sokoban. Where they differ is in locating claims: gpt-oss-120b flags 154 of 267 true Knights-and-Knaves lines and misses 32 of 72 false ProntoQA claims, so it could not replace the checker as a claim-level filter, while a strong judge could, at a per-solution cost. Every rate in the paper comes from the checker; the judges are only compared against it.

With the discussion period closing soon, we wanted to gently follow up in case you have any remaining questions about our responses. We would really appreciate your feedback. If we have resolved your concerns, we would be very grateful if you would please consider raising your score. Thank you again for your time.

Reviewer_6xuUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the phenomenon where the models provide right answers with incorrect reasoning traces. To get a deeper understanding of this phenomenon, the authors introduce a distinction between forced claims, whose correctness is necessary for the final answer to be valid, and unforced claims, which may be incorrect without affecting outcome verification. Using exact checkers across four synthetic reasoning and planning domains, the authors find that incorrect claims are concentrated in the latter category. Furthermore, the paper concerns whether these errors are inherited through training. Through controlled fine-tuning experiments, the paper demonstrates that training on unsound solutions substantially increases the frequency of incorrect intermediate claims in subsequently generated correct solutions, often without a consistent change in final-answer accuracy.

Critical strengths

This paper studies an important problem in practical deployment: models may produce incorrect reasoning while still arriving at correct answers. This issue is highly relevant in real-world applications.

Moreover, the experiments are carefully designed with good control of variables, demonstrating that models can easily exhibit this failure mode. The authors further analyze the potential impact of self-training, showing how such incorrect reasoning may be inherited during training.

This topic is also highly aligned with the current frontier of self-evolving systems and self-improving models, making the findings timely and relevant.

Critical weaknesses
  1. Limited novelty and unclear significance of the research problem. The central observation of the paper — that a model can produce a correct final answer while relying on incorrect intermediate reasoning — has been extensively studied in prior work on chain-of-thought faithfulness, process supervision, and answer-conditioned reasoning. Although the paper introduces the distinction between forced and unforced claims, it remains unclear whether this perspective provides a genuinely new understanding of the problem or is mainly a re-framing of the known limitation of outcome-based verification.
  2. The overall narrative and experimental motivation are difficult to follow. The paper contains many experimental components, but the motivation and logical connections between them are not sufficiently clear. For example, Section 3.1 introduces “self-training and recovery,” but the purpose of including both concepts is not immediately obvious. The paper does not clearly explain (e.g. why recovery from unsolved problems is necessary for studying the research question). Similarly, Section 3.4 (“Giving part of the answer”) introduces multiple intervention settings (answer hints, first steps, unaided solutions), but the rationale for choosing these specific conditions is not clearly developed.
  3. Results are presented with many quantitative observations but insufficient synthesis and conceptual interpretation. The paper reports a large number of experimental results, percentages, and comparisons across tasks, models, and conditions. While the empirical evidence is extensive, the presentation often lacks a clear synthesis of what general conclusions should be drawn from these numbers.
Questions for authors
  1. (Novelty and significance of the research.) Could the authors clarify what new conceptual insight is provided by the forced/unforced claim framework beyond the existing observation that outcome verification does not guarantee reasoning correctness? In particular, what new predictions, mechanisms, or practical implications does this framework enable that could not be obtained from prior studies of incorrect reasoning traces in correct answers?

  2. (Motivation and clarification of the experimental setup.) In Section 3.1, the paper introduces both self-training and recovery from unsolved problems, but it is unclear why both components are necessary for studying the main research question. Could the authors clarify the intended role of the recovery setting and how it connects to the later training experiments? Similarly, Section 3.4 introduces answer hints, first steps, and unaided settings. While these conditions appear to study the effect of how information is provided, the motivation for choosing these specific interventions and the hypotheses they are designed to test are not explicitly stated.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6xuU

W1 and Q1. Novelty and what follows from the split. Bansal et al. (2025), Wang et al. (2025) and Li et al. (2025) establish that answer-filtered data can contain correct answers with wrong reasoning, and the revision cites them; that is the paper's starting point. The contribution is to measure the consequences exactly, in domains where every claim of every accepted solution can be checked by a solver, simulator or executor, and to train on what the verifier admits. The split between claims the grader consumes and claims it does not is an account that this checking made visible and that the code-tracing result refined. It predicts where false claims sit, what training passes on, and when accuracy will show it, and each prediction was tested. Within Blocksworld and Sokoban solutions, at most 1.4% of grader-consumed claims are false against up to 73.5% of the others (Table 2); in Countdown, where every line is graded, the hint changes the rate by about one point (Table 2, App. P). Models trained on admitted solutions reproduce the false claims and leave the graded ones correct, and in Knights-and-Knaves now on the model's own unaided solutions, four seeds, at unchanged accuracy (Section 6). In Sokoban, where an ungraded false state steers an irreversible move, models trained on unsound solutions solve 8.1% of new puzzles against 25.8% for sound ones.

Two things follow. On mechanism: the false claims enter where the model skips rejecting a wrong hypothesis and writes "no contradiction" beside it, a claim the grader never consumes; the same skip appears in models trained by outcome-only GRPO, whose newly solved problems carry false claims at 87-93%. On practice: the split says which claims to check before training, and checking them lowers the false-claim share from 67% to 4% on Llama-3.1-8B's own accepted solutions, across four seeds, while an off-the-shelf process reward model (AUROC 0.74 and 0.60) and an exact claim-level GRPO reward (satisfied by assuming the answer and confirming it) do not substitute. Prior studies report a rate. This paper locates it and tests what removes it.

W2 and Q2. Why recovery, and what the conditions test. Self-training learns only from problems the model already solves. Recovery (STaR's rationalization, and hint- or prefix-guided methods such as QuestA and ExPO) is how pipelines extend to the rest, and recovered solutions are the training data of Section 6. The conditions answer three questions in order: whether solutions that pass outcome verification contain false claims in the claims the grader does not consume (the unaided condition and the within-solution split, Sections 4 and 5); whether training on them passes the false claims on at unchanged accuracy (the Sound/Unsound and filtered/unfiltered designs, Section 6); and whether checking those claims before training removes the inheritance (the filtering experiments, Section 6). Unaided is the baseline; the answer hint is the recovery practice in use; first steps supply the same information as the opening of a derivation, which holds the opening fixed and controls for difficulty. Under balanced worked examples the hint does not raise the rate over unaided on puzzles the model already solves (-4.1 and -0.3 points, two families), and on unsolved puzzles its effect depends on the model (Section 4 and App. S), so hint and first steps are reported as recovery conditions beside the unaided baseline. The revised Section 3 gives each condition's role, and the introduction states the three results.

W3. Synthesis. Outcome-accepted solutions contain false claims without any hint, at rates set by the model: 11-19% at 72B-400B, 59% for Llama-3.1-8B, 3-76% in planning. The claims the grader consumes are correct and the rest are not, and the task says in advance which is which. Training on admitted solutions preserves the rate at unchanged accuracy; checking the unprotected claims before training removes it, with an accuracy effect ranging from a small cost to a gain. The introduction states these three results with the section that supports each, and the conclusion restates them. Each results paragraph in Sections 5 and 6 (within a solution, inheritance, hint-free inheritance, filtering, GRPO) now opens with the conclusion its numbers support.

Thank you for the detailed rebuttal and extensive revisions. I appreciate the authors' efforts to address my concerns.

Although the revised manuscript provides a clearer narrative and better experimental motivation, the changes are substantial, to the extent that the paper has been significantly restructured and rewritten rather than simply clarified. While these revisions address several of my original concerns, I find it difficult to fully assess the substantially revised contribution within the rebuttal process.

Therefore, while I acknowledge the improvements, I am inclined to maintain my original assessment.

Thank you for reading the revision. To make it easier to compare with the version you reviewed, here is what changed and what did not.

Unchanged: the domains, checkers and claim definitions (Section 3, App. A, F); the Sound/Unsound inheritance experiment and its results (Section 6, Figure 3, Table 12); the filtering experiment, with its three original seeds unchanged and two added (Table 16); and the process-reward-model comparison (App. N).

Changed in substance: one result and five statements. The Knights-and-Knaves hint result was rerun at Reviewer 4VSF's request and the abstract and Section 4 restated accordingly; the five restated sentences are listed in the general response.

Changed in presentation: the introduction states the three results, Section 3 explains what recovery is for and what each condition tests, and each results paragraph opens with its conclusion. These are the changes your review asked for.

Added: hint-free inheritance, GRPO, four larger models and a code-tracing domain, in blue; most of these were requested by other reviewers. They extend results already in the submission.

Since the inheritance and filtering results can be checked against the submitted version directly, one question: do those results, as originally reported, address the concerns in your review? If not, which one is still open?

Reviewer_4VSFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what an outcome verifier lets into the training data of self-training pipelines when unsolved problems are recovered by hinting part of the answer. The authors use puzzle domains with exact per-claim checkers (Knights-and-Knaves, Blocksworld, Sokoban, Countdown) and split each solution into forced claims, which the answer depends on, and unforced claims, which it does not. They find that 1) accepted hinted solutions are almost always correct in forced claims but up to 74% wrong in unforced claims; 2) providing the same information as first derivation steps instead of a hint eliminates most of the issue; 3) fine-tuning on sound versus unsound accepted solutions transfers the unsoundness to the trained model with little change in accuracy. Results cover Qwen3, Qwen3.5, Llama and OLMo-3 models, with checkers validated against three LLM judges.

Critical strengths
  1. The question is timely, as self-training and RLVR pipelines filter on final answers only. Using puzzle domains where every claim is decidable gives claim-level ground truth instead of LLM judges, and the checkers are validated carefully (Table 8, 10).
  2. The hint-vs-first-steps control is clean and actionable. The two conditions reveal the same two identities, yet 67.1% vs 0.4% of accepted solutions contain a false claim, and first steps even recover more problems (Table 1).
  3. The Sound/Unsound training design, which holds problems, answers and data size fixed, is the right way to isolate inheritance, and the result (53-75 points more unsound correct answers) is large.
Critical weaknesses
  1. The framing overstates what is new. That answer-filtered data contains correct answers with wrong reasoning, and that accuracy filters cannot see it, was established by Bansal et al. [1], who defined the false positive rate for STaR-style data, and Li et al. [2] showed that corrupting step content barely changes fine-tuned accuracy. Neither is cited. What is new is the claim-level checking, the forced/unforced split and the controlled inheritance experiment; finding 1 should be presented as a refinement of [1], and methods built on answer-conditioned chains such as ExPO [3] should be discussed.
  2. The headline numbers are partly definitional and partly prompt artifacts. "Unsoundness is confined to unforced claims" is true by construction for accepted solutions (Line 171, 309). The 67.1% in Knights-and-Knaves comes from a knight-first default opening colliding with knave-first problems (Table 4, Appendix O), and all worked examples in Appendix D open with "Assume X is a knight"; reweighting gives about 47%. Please rerun with balanced worked examples and present the by-construction half as a sanity check rather than a finding.

[1] Bansal et al. Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling. ICLR 2025.

[2] Li et al. Language Models Can Easily Learn to Reason from Demonstrations. Findings of EMNLP 2025.

[3] Zhou et al. ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning. NeurIPS 2025.

Questions for authors
  1. Line 196-202: Please confirm that the round-one Qwen3-4B and Llama-3.1-8B starting points are also base checkpoints, and state the exact model identifiers and whether thinking mode is disabled.
  2. Table 2: Please add the unaided false-claim rate next to the first-steps column, so that readers can see how much the hint adds.
  3. Line 202-203: The shared 523-problem set is defined by what Qwen3-4B's hint recovered. Could you report Figure 2 on a model-agnostic set?
  4. Line 186-192: In planning the hint is the complete plan, so forced claims are copied from the prompt. How often does the accepted plan deviate from the hinted plan?
  5. Table 14: For Qwen3-4B, both Sound and Unsound models fall below the starting model in in-distribution accuracy (37.8 and 37.2 vs 43.0), also in the quarter-dose run of Appendix K. Is there a training dose or a larger problem pool at which accuracy matches the starting model while the inheritance gap persists?
  6. Line 404-406: The Filtered vs Mixture difference is -6.6, -6.6 and +8.5 across three seeds, which cannot support "no clear effect on accuracy". Could you run more seeds and at least one planning domain?
  7. Line 408-411: "the Sound model learned to let its mistakes show" is hard to understand on first reading. Please define α\alpha and β\beta from Appendix L in the main text.
  8. Line 36: The paper is framed around RLVR but no policy is trained with GRPO. Could you try one GRPO run with and without the claim-level reward?
  9. Line 535: Llms gaming verifiers: Rlvr → LLMs ... RLVR
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4VSF (1/4)

W1. Related work and framing. Bansal et al. (ICLR 2025), Li et al. (EMNLP 2025 Findings) and ExPO (NeurIPS 2025) are now cited, together with Wang et al. (EMNLP 2025) on false positives under inference scaling. These papers establish that answer-filtered data can contain correct answers with wrong reasoning and that accuracy does not reveal it, which is our paper's starting point. Our contribution that follows is to measure that phenomenon exactly in the setting of verifier-filtered self-training. The domains are chosen so that every intermediate claim in every accepted solution can be checked by a solver, simulator or executor; every rate is a count over checked claims. What that makes measurable is new: where false claims sit within a solution (only claims the grader itself consumes are protected; Section 5, Table 2, and W2a below), that training on verifier-admitted solutions passes them on at unchanged accuracy (Section 6, now also on hint-free pools in Knights-and-Knaves, four seeds), and what removes them. Bansal et al. estimate a false-positive rate on MATH and GSM-8K from 50 author-rated solutions per model and a Gemini-1.5-Pro judge on 500; Li et al. inject synthetic corruptions into traces and measure accuracy after training. Neither has per-claim ground truth on every solution, and the results above need it: locating false claims by grader consumption, the counterfactual tests of App. L, and size-matched Sound and Unsound pools the verifier itself cannot distinguish all require a label on every claim of tens of thousands of solutions. The strongest judge we tested reproduces the checker line for line in the planning domains; the weaker ones disagree with it in both directions (Table 10), and the checker is what decides between them. The checker also shows what judge-based measurement misses: an off-the-shelf PRM separates flawed from clean accepted solutions at AUROC 0.74 and 0.60 (App. N), and a reward requiring every claim to pass the checker is satisfied by assuming the correct roles and confirming them, which outcome and judge metrics do not detect. Li et al.'s finding that accuracy is insensitive to step content is the reason the inheritance in Section 6 is invisible to outcome metrics. ExPO generates positives by conditioning on the ground-truth answer; Section 2 now discusses it and related answer-conditioned methods as the pipelines this audit applies to.

W2a. The forced half. Because every claim is checked, the split between protected and unprotected claims is measured. The submission states at L266-268 that forced claims are correct in accepted solutions by construction, and the finding is stated about the other half: how often claims the verifier never checks are false, and that the task identifies those claims before any model runs. The revision presents forced-claim correctness as a check on extraction and simulation and says so in the opening of Section 5. New evidence refined the condition. In code tracing (CRUXEval, execution as the checker), values on the data path to the output are false in 53-68% of hinted correct traces when the grader checks only the output. A claim is protected when the grader consumes it directly, as it does plan moves and Countdown lines; the answer depending on a claim is not enough. The forced/unforced split is restated with that condition.

AuthorsOpenReview ↗

Response to Reviewer 4VSF (2/4)

W2b. Where the unsoundness comes from. The paper's claim is about what an outcome verifier admits into self-training data and what training on it does. The K&K hint experiment was one route into that, and the rerun you proposed is an exact test of whether it was a special one. With two knight-first and two knave-first worked examples and everything else unchanged, Qwen3-4B's hinted solutions on the 833 puzzles unsolved under the submission's knight-first examples contain a false claim 6.8% [5.7, 8.1] of the time (first steps 0.7%; 69.0% under knight-first examples). The balanced examples also let the model solve 499 of those 833 unaided; on the 334 it still cannot solve, the hinted rate is 8.7% [5.9, 12.1] and first steps 0.8% [0.2, 1.8] (accepted solutions on 200 and 207 of the 334 puzzles). On the 499 now-solved puzzles the hint does not raise the rate over the model's own unaided solutions: -4.1 [-6.2, -1.9] points for Qwen3-4B and -0.3 [-3.9, +2.9] for Llama-3.1-8B on its own set (unaided 10.8% and 58.8%). Llama's hinted rate on its still-unsolved puzzles is 78.7% [74.7, 82.6] against 10.7% under first steps. On unsolved puzzles, then, the hint's effect depends on the model: small for Qwen3-4B, large for Llama-3.1-8B, whose unaided solutions are already 58.8% unsound on puzzles it can solve. First steps lower the rate in both families by replacing the model's opening claims with correct ones; whether the hint raises the model's own rate on unsolved puzzles cannot be measured, since there is no unaided solution to compare. The 67.1% measured the collision described in Section 4, App. A (Table 4) and App. O between the induced knight-first opening and knave-first hints; removing the opening removes it. The abstract's K&K sentence is restated accordingly.

False claims do not need a hint. Outcome-accepted solutions carry false claims without any hint: 11-19% in Llama-4-Maverick, GLM-4.7, Qwen3-235B and Qwen2.5-72B, 58.8% in Llama-3.1-8B (balanced examples, measurement set), 3-76% in Blocksworld and Sokoban. The verifier admits them, and training on them passes them on. On Llama's own unaided accepted solutions, across four seeds, outcome-filtered training preserves the starting model's rate (67% vs 63% on the training experiment's evaluation set) at unchanged accuracy, while checker-filtered training removes it (4%, -56 points on shared puzzles) and raises pass@1 by 8 points; unfiltered training restricted to the filtered set's own puzzles still gives 30-49%, so coverage does not explain the gap. Sokoban (three seeds) and code tracing (two families) show the same. The result does not depend on the prompt that first exposed it, and the number that did has been removed.

Q1. Starting checkpoints. All models are base checkpoints: Qwen/Qwen3-{1.7B,4B,8B,14B}-Base, Qwen/Qwen3.5-{2B,4B,9B}-Base, meta-llama/Llama-3.2-3B, meta-llama/Llama-3.1-8B, allenai/Olmo-3-1025-7B and allenai/Olmo-3-1125-32B. The round-one Knights-and-Knaves models are LoRA adapters on Qwen3-4B-Base and Llama-3.1-8B. Every prompt is a raw few-shot completion with no chat template, so thinking mode never applies. Appendix E now lists each identifier.

Q2. Unaided rate in Table 2. Added for every planning cell. The revised Table 2 reports the share of accepted solutions with at least one false state claim; the submitted table reported the share of state claims that are false, and App. T gives both units for every cell; the per-claim column reproduces the submitted Table 2 cells. Hint / unaided / first steps: Blocksworld, Qwen3-8B 84.0 / 73.6 / 41.3, Llama-3.1-8B 55.8 / 35.4 / 18.0, OLMo-3-32B 71.4 / 49.9 / 40.9, Qwen3.5-4B 88.9 / 75.8 / 40.0; Sokoban 96.5 / 65.3 / 38.4, 91.1 / 43.8 / 44.9, 11.0 / 2.9 / 0.2, 48.7 / 39.2 / 13.1. Unaided solutions are already unsound at these rates; the hint adds 8-47 points in planning, and unaided sits between hint and first steps in seven of eight cells (Sokoban Llama-3.1-8B is the exception, where first steps and unaided are within 1.1 points). In Knights-and-Knaves under balanced examples the hint does not raise the rate over unaided on puzzles the model already solves (W2b).

AuthorsOpenReview ↗

Response to Reviewer 4VSF (3/4)

Q3. A model-agnostic set. Figure 2 was recomputed on each model's own unsolved set and on the 330 puzzles every model leaves unsolved (knight-first examples). All eleven models stay positive on hint minus first steps and close to the shared-523 values (own set / intersection): Qwen3-1.7B +79.3 / +82.4, Qwen3-4B +51.7 / +80.9, Qwen3-8B +42.8 / +46.9, Qwen3-14B +9.9 / +8.1, Qwen3.5-2B +35.2 / +29.0, Qwen3.5-4B +20.8 / +22.7, Qwen3.5-9B +25.6 / +25.6, Llama-3.2-3B +88.9 / +93.1, Llama-3.1-8B +81.1 / +87.7, OLMo-3-7B +86.5 / +93.4, OLMo-3-32B +30.8 / +30.8. Given W2b, this quantity compares the model's own claims with a solver scaffold and does not isolate a hint effect; the revision adds each model's unaided false-claim rate beside it, and the balanced-example version of this figure is reported in App. C.

Q4. Deviation from the hinted plan. Accepted hinted plans reproduce the hinted plan in 99.8-100% of Blocksworld solutions and 85-98% of Sokoban solutions, so the hinted forced column in Table 2 mainly checks extraction, as you say. The false state claims do not depend on copying: in Sokoban the rates for solutions that copy the plan versus deviate are 96.8 vs 85.1, 91.0 vs 93.5, 10.9 vs 14.7 and 49.0 vs 47.3 across the four models, with the difference in both directions. Where the forced claims are the model's own, they stay correct: at most 1.4% false under first steps, and 0.00-0.51% among valid plans across the 28 trained planning models of Section 6, all evaluated without hints.

Q5. Accuracy below the starting model. The drop comes from the 118-problem pool. The All-accepted model, trained on 442 problems, matches the starting model (44.8 vs 43.0). The new hint-free experiment is the direct answer: on Llama-3.1-8B's own unaided accepted solutions, across four seeds, checker-filtered training raises pass@1 from 28.7 to 36.3 and lowers the false-claim share from 63% to 4%, while outcome-filtered training leaves pass@1 at 27.3 and the share at 67%. The inheritance gap persists at and above the starting model's accuracy.

Q6. Filtering: seeds and domains. Agreed that three seeds do not resolve the accuracy effect; the sentence is replaced. The cleanest accuracy comparison is Sokoban (Qwen3-8B, three seeds), where checker-filtered training is compared with unfiltered training on the same problems: pass@1 23.3 / 27.8 / 22.8 against 16.3 / 21.2 / 12.1, a gain of 7.0, 6.6 and 10.7 points in every seed, with false state claims at 0.3 / 0.0 / 0.3% against 50.4 / 44.7 / 44.4%. Against outcome filtering on the full pool, pass@1 is 23.9 / 28.5 / 15.0 with 82.4 / 83.3 / 58.6% false claims. Knights-and-Knaves now has five seeds: the outcome-filtered model exceeds the checker-filtered model's false-claim share on shared problems by +53.1, +60.3, +62.3, +45.1 and +31.4 points, every CI above zero, while the checker-filtered model's pass@1 is 1-7 points lower in four of five seeds. In the hint-free experiment (Q5) the checker-filtered model is 8 points higher. The soundness effect holds in every seed and domain; the accuracy effect ranges from a small cost to a gain.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author response: summary of changes and new experiments (1/2)

We thank the reviewers for their careful and detailed reviews. In response we ran the experiments requested across the three reviews and extended the paper in three directions: the measurements now cover unaided solutions, four models at 72B-400B, and a code-tracing domain with execution as the checker; the training results now include a hint-free inheritance design with a same-problems control over four seeds, filtering over five seeds and a second domain, and GRPO under three reward designs; and the prompt-sensitivity check Reviewer 4VSF requested is reported in Section 4. The revised PDF is uploaded, changes marked in blue, structure and numbering kept. Below we summarize what the paper contributes, what we changed, and what the new experiments show; detailed replies follow under each review.

What the paper contributes. That answer-filtered data can contain correct answers with wrong reasoning is established (Bansal et al., 2025; Wang et al., 2025; Li et al., 2025), and the revision cites this work. The paper measures the consequences exactly. The domains are chosen so that every intermediate claim of every accepted solution can be checked by a solver, simulator or executor, so every rate is a count over checked claims, with no judge error and at the scale the training experiments need. That checking gives four results: false claims sit in the claims the grader does not itself consume, and the task says in advance which those are; models trained on verifier-admitted solutions reproduce the false claims at unchanged accuracy; checking the unconsumed claims before training removes the inheritance; and the obvious substitutes fall short, an off-the-shelf process reward model at AUROC 0.74 and 0.60, and an exact claim-level reward, which removes false claims the same way checker filtering does: the model assumes the answer and confirms it, and neither restores search.

What changed in the paper. Five statements are restated. The sentence that one round of recovery "roughly doubles" the share of unsound correct answers is replaced by a decomposition: the rate on problems the starting model already solved barely moves, and the raw rise comes from newly solved problems whose solutions are mostly unsound. The claim that rejected solutions carry the corrective signal is restated as what the GRPO runs show: negative advantage on failures preserves soundness on problems already solved and does not produce it on new ones. "Sound" is defined as containing no refuted claim, since sound-trained models reach their answers largely by assuming the given roles and confirming them. The filtering recommendation is qualified: it removes false claims but does not restore the hypothesis-rejecting search of the starting model, which falls from 63% to 1-13% of solutions under every training condition. The introduction's RLVR framing is narrowed to what was run. Finally, at Reviewer 4VSF's suggestion we reran Knights-and-Knaves with worked examples balanced between knight-first and knave-first openings. The hinted false-claim rate on Qwen3-4B's 833 unsolved puzzles falls from 69.0% to 6.8%; on puzzles a model already solves the hint does not raise the rate over its own unaided solutions; on unsolved puzzles the hint's effect depends on the model (Section 4). The abstract now leads with the unaided rates, and hint and first steps are reported as recovery conditions beside the unaided baseline rather than as a separate finding.

AuthorsOpenReview ↗

Author response: summary of changes and new experiments (2/2)

What the new experiments show. Outcome-accepted solutions contain false claims without any hint, at rates set by the model: in Knights-and-Knaves under balanced worked examples, 10.8% for Qwen3-4B and 58.8% for Llama-3.1-8B; on harder puzzles, 11-19% for Llama-4-Maverick, GLM-4.7, Qwen3-235B and Qwen2.5-72B; and 3-76% in Blocksworld and Sokoban. Training on them inherits the rate. On Llama-3.1-8B's own unaided accepted solutions, across four seeds, outcome-filtered training leaves the false-claim share at 67% against the starting model's 63% and leaves pass@1 unchanged; checker-filtered training lowers it to 4% and raises pass@1 by 8 points; unfiltered training restricted to the filtered set's own problems still gives 30-49%, so coverage does not explain the gap. Filtering holds in Knights-and-Knaves with five seeds and in Sokoban with three, and the Sound/Unsound design holds in code tracing in two families. Outcome-only GRPO raises pass@1 from 43 to 71 while its newly solved problems carry false claims at 87-93% under knight-first examples (the submission's prompt) and 22% against ~4% under balanced examples. In code tracing, values the output depends on are false in 53-68% of hinted correct traces, so what protects a claim is the grader checking it; the answer depending on it is not enough.

New experiments and where they appear. Balanced worked examples (Section 4, App. S); unaided rates in Table 2 and Figure 2 (Sections 4 and 5, App. C); plan deviation (Section 5, App. T); hint-free inheritance with same-problems control, four seeds (Section 6, App. K); filtering, five seeds and Sokoban (Section 6, App. K); GRPO under three rewards (Section 6, App. H); four large models (Section 4, App. C); code tracing (Section 6, App. R); step-swap and activation-patching extensions (App. L); model identifiers (App. E).

Presentation. The revision states the three results in the introduction, explains the role of recovery and of each condition in Section 3, defines alpha and beta in the main text, opens each results paragraph in Sections 5 and 6 with the conclusion its numbers support, and rewrites the informal passages the reviewers cited.

Dear Reviewer FsxU,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.