- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates whether fine-tuning a language model for solution verification preserves its ability to generate solutions, finding that these capabilities are separable and that the reasoning content of the training signal determines whether both can be retained. The authors examine how different fine-tuning targets, varying in the amount of reasoning information they provide, affect verification and generation capabilities. Their results show that training on binary correctness verdicts can maintain verification performance while almost entirely degrading generation, whereas reasoning-rich critiques can preserve or even improve both capabilities.
Critical strengths
-
This paper identifies a consequential and underexplored distinction between verification and generation in fine-tuning. This finding challenges the assumption that training models to evaluate solutions necessarily improves their ability to produce them, providing a useful conceptual contribution to reasoning-model training.
-
Evaluations across multiple model families and sizes, different supervision formats, and training-data scales establish that supervision content matters beyond the quantity of training examples. Although some results are affected by model-family differences and limited seed replication, the combination of broad empirical comparisons and mechanistic investigation provides meaningful support for the paper’s conclusions.
Critical weaknesses
- This paper provides evidence that generation collapse is partly associated with task disengagement, supported by output-level failure analysis and instruction-mixing experiments. However, instruction mixing addresses only framing-induced critique-mode disengagement, while repetition collapse remains unresolved, particularly in larger models. This work therefore does not fully explain the distinct failure modes or their differing responses to the intervention, limiting the completeness of its mechanistic analysis.
Questions for authors
- This paper identifies repetition collapse as a failure mode not resolved by instruction mixing. Could the authors clarify whether their current analysis provides evidence for a distinct mechanism underlying repetition collapse, or whether its relationship to critique-mode disengagement remains unclear?
Thank you for catching this. The inconsistency in the Appendix B is genuine. The structured and varied targets were produced by a LLM rewriting the original free-form critiques, and the rewriting occasionally changed the judgement. We audited all training examples. The final verdict of the rewritten critique differs from the original in 0.23% of 4K examples, 0.27% of 50K examples. Using a LLM-as-a-judge, we checked that roughly half of the changes moved toward the correct verdict and half away from it, suggesting the changes are random rather than systematic. This audit covers final verdicts only.