acceptodds
Withdrawn from ICLR 2027

Decoupling Verification and Generation: Reasoning Content Governs What Fine-Tuning Teaches

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models are increasingly asked to both generate solutions and verifythem, and a single model is often fine-tuned to learn both at once — on the assumption that judging and solving are one competence that improves together. We show they are separately learned: a single fine-tuning run can move the two abilities in different directions. Training on a bare correctness verdict collapses generation almost entirely (math accuracy 63.8% to 0.0%) while leaving verification near the base level, and supervised fine-tuning on solutions (SFT), by contrast, improves generation while leaving verification weak. They are separable because they are learned from different parts of the signal: verification from the judgment it expresses, generation from the reasoning content it retains. A bare verdict carries only the judgment, so it leaves verification intact but collapses generation; an SFT target carries only the reasoning, so it does the reverse; a critique carries both, which is why it appears to improve both at once — not because critique is a uniquely effective objective, but because it is the one signal that contains both ingredients. Varying this content across model sizes and families, we find that whether verification training preserves generation is governed by the reasoning content of the signal, not its quantity. When generation collapses it does so by disengagement rather than lost skill: the model stops attempting problems, a failure we trace to the single instruction framing shared by all critique examples and confirm causally by mixing in direct-solve instructions. The practical consequence is that a verifier cannot be assumed to remain a capable generator; training one that still solves requires keeping reasoning content in the verification signal, and a bare verdict — the cheapest signal — is the most destructive.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Large language models, critique fine-tuning, generation-verification separability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p7F1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates whether fine-tuning a language model for solution verification preserves its ability to generate solutions, finding that these capabilities are separable and that the reasoning content of the training signal determines whether both can be retained. The authors examine how different fine-tuning targets, varying in the amount of reasoning information they provide, affect verification and generation capabilities. Their results show that training on binary correctness verdicts can maintain verification performance while almost entirely degrading generation, whereas reasoning-rich critiques can preserve or even improve both capabilities.

Critical strengths
  1. This paper identifies a consequential and underexplored distinction between verification and generation in fine-tuning. This finding challenges the assumption that training models to evaluate solutions necessarily improves their ability to produce them, providing a useful conceptual contribution to reasoning-model training.

  2. Evaluations across multiple model families and sizes, different supervision formats, and training-data scales establish that supervision content matters beyond the quantity of training examples. Although some results are affected by model-family differences and limited seed replication, the combination of broad empirical comparisons and mechanistic investigation provides meaningful support for the paper’s conclusions.

Critical weaknesses
  1. This paper provides evidence that generation collapse is partly associated with task disengagement, supported by output-level failure analysis and instruction-mixing experiments. However, instruction mixing addresses only framing-induced critique-mode disengagement, while repetition collapse remains unresolved, particularly in larger models. This work therefore does not fully explain the distinct failure modes or their differing responses to the intervention, limiting the completeness of its mechanistic analysis.
Questions for authors
  1. This paper identifies repetition collapse as a failure mode not resolved by instruction mixing. Could the authors clarify whether their current analysis provides evidence for a distinct mechanism underlying repetition collapse, or whether its relationship to critique-mode disengagement remains unclear?
Reviewer_CtLnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how different verification training signals affect LLM generation and verification abilities. By comparing binary verdicts, structured critiques, free form critiques, and solution SFT across multiple models, the authors argue that the two abilities are learned separately and that reasoning content determines whether verification training preserves generation.

Critical strengths
  1. The paper constructs different verification targets from the same source data and evaluates their effects across multiple model sizes and families.

  2. The authors examine model outputs rather than relying only on accuracy, and use instruction mixing to investigate why verification-trained models stop attempting generation tasks.

Critical weaknesses
  1. The dramatic generation collapse under binary-verdict training mainly reflects a change in response behavior: the model outputs correctness judgments instead of attempting to solve problems. This does not directly establish that generation and verification are independently learned capabilities. Moreover, binary-verdict training does not improve verification balanced accuracy over the base model (60.5% → 56.9% for Qwen-2B; 72.6% → 73.0% for Qwen-9B), while solution-only SFT substantially improves verification on Qwen-9B (72.6% → 81.2%). These results do not clearly support the proposed explanation that verification is learned from judgment signals while generation is learned from reasoning content.

  2. Binary verdicts and critiques differ not only in reasoning content but also in output length, format, and supervision density. The instruction-mixing experiment introduces 30% direct-solve examples, changing both the instruction distribution and the supervised task. Therefore, the recovered generation performance cannot be attributed solely to instruction framing. In addition, the verification benchmark has a known difficulty imbalance between correct and incorrect solutions, making it difficult to determine whether the reported verification scores reflect genuine solution checking.

Questions for authors
  1. Since binary-trained models mainly output correctness judgments instead of attempting solutions, how do the authors distinguish task-format specialization from a genuine loss of generation ability? Can generation performance be recovered through inference-time prompting without additional training?

  2. Binary-verdict training does not improve verification over the base model, while solution-only SFT substantially improves it on Qwen-9B. How do these results support the claim that verification and generation are learned separately from judgment and reasoning signals?

  3. The training targets differ in reasoning content, output length, and format, while instruction mixing also introduces direct-solve supervision. How can the authors isolate reasoning content and instruction framing as the causes of the observed effects?

Reviewer_daaVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies two abilities of LLMs, generation and verification, and questions whether improving one necessarily improves or preserves the other. The authors start from base models and fine-tune them using different training settings, including binary correctness verdicts, step-wise critiques, free-form critiques, and solution SFT as a reference. They then evaluate the trained models on generation and verification tasks. The experiments show that binary-verdict training can almost completely collapse generation performance while keeping verification near its original level. For example, on Qwen-9B, generation performance decreases by 50.3 percentage points on average, while verification accuracy remains nearly unchanged. In contrast, critique training that contains step by step reasoning preserves generation behaviour. Increasing the amount of binary verdict training data does no good to prevent the collapse. The authors claim that reasoning content, rather than data quantity or the verification objective itself, is important for preserving generation.

The paper further studies whether generation collapse is caused by a loss of solving ability or by disengagement(model stops attempting to solve problems). They replace 30% of critique training examples with direct solution SFT examples. This setting helps restore solving behavior, but binary verdict still weaker than critiques in generation correctness. This shows that while the generation behaviour can be recovered, the ability to solve the problem still related to the reasoning content in training data.

Overall, the paper demonstrates that generation and verification performance can be dissociated through fine tuning, and calims that instruction framing causes disengagement, while reasoning content in the training data is important to preserving solving ability

Critical strengths

The paper designs experiments and identifies a striking phenomenon: binary verdict finetuning can almost completely collapse generation performance while leaving verification performance unchanged. This finding shows that when the training target consists only of yes/no verification judgments, generation behavior can be harmed, suggesting that generation and verification do not necessarily improve together during finetuning. This finding is interesting and could be helpful to the community.

Critical weaknesses

My main concern is that the claims about abilites separability and the causal role of reasoning content are not sufficiently supported by the experiments.

First, about abilities separability, the Qwen-9B results show that binary-verdict training barely changes verification accuracy (72.6% → 73.0%), while solution SFT improves it to 81.2%, wtihout any verification supervision. This contradicts with authors' claim that two ability are learned seperately.

Second, the experiments do not sufficiently isolate the effect of reasoning content. Binary verdicts and step-wise critiques differ not only in reasoning content, but also in output length, structure, and the generation behavior they encourage (directly answer versus step by step answer). These factors make it difficult to attribute the observed differences to reasoning content.

Third, the 30% SFT-mixing experiment introduces additional confounding factors. Replacing critique examples with direct-solve examples changes the instruction distribution (verification to generation), the training objective, and the quality and amount of reasoning supervision(the added solutions are regenerated as clean CoT). Moreover, adding direct-solve supervision would naturally be expected to improve generation performance, making it difficult to attribute the recovery to instruction diversity. The results are also inconsistent across models. In Appendix I (Table 4), Gemma-12B binary-mix achieves an Avg Δ of −14.1, slightly outperforming structured-mix (−14.2) and varied-mix (−16.2), contradict to the authors' claim that reasoning rich critique mixing outperform binary-verdict mixing.

Overall, the experiments demonstrate interesting behavioral differences, but the current design is not convincing.

Questions for authors

Q1: Is generation collapse caused by the absence of task-relevant reasoning content, or by the loss of step by step behavior training?

Could the authors conduct a control experiment using an unrelated, non-reasoning task, such as everyday scene description? Specifically, two models could be fine-tuned on the same data, with one trained to output short, direct answers and the other trained to output step-by-step descriptions. No training target contain mathematical reasoning or verification supervision. Both models would then be evaluated on the original mathematical generation benchmarks.

If direct answer training causes generation collapse while step by step training preserves mathematical generation performance, this would suggest that the key factor may be maintaining step by step output behavior rather than learning task relevant reasoning content. Such a control would help clearify the role of reasoning content.

Q2: Are the step-level content consistent across critique settings?

Appendix B states that structured and varied critiques preserve the same step-level judgments and differ only in surface format. However, the examples appear inconsistent: Structured Critique labels the application of standard power-series formulas (Step 3) as "Correct", while Varied Critique states "Incorrect—the closed forms are not applied consistently." Could the authors clarify whether this is an error in the example?

1 comment

Thank you for catching this. The inconsistency in the Appendix B is genuine. The structured and varied targets were produced by a LLM rewriting the original free-form critiques, and the rewriting occasionally changed the judgement. We audited all training examples. The final verdict of the rewritten critique differs from the original in 0.23% of 4K examples, 0.27% of 50K examples. Using a LLM-as-a-judge, we checked that roughly half of the changes moved toward the correct verdict and half away from it, suggesting the changes are random rather than systematic. This audit covers final verdicts only.

Comments on the paper

1 comment

We thank the reviewers for their careful and constructive feedback, and we have decided to withdraw the submission to substantially revise it.

We agree with the main concerns raised. First, the separability claim is overstated: solution-only SFT improves verification on Qwen3.5-9B without any supervision on verification which contradicts the account that generation and verification are learned from different signals. Second, reasoning content is confounded with output length, format, and step-by-step behaviour, and simple instruction-mixing experiment does not isolate them enough.

In revising this work, we plan to reframe the findings around asymmetric transfer between generation and verification, and add controls that hold formats and length fixed while removing reasoning contents only, including non-mathematical examples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.