- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ThinkTwice is an RLVR framework with self-refinement that trains an LLM to do two things: solve reasoning problems and revise its own answers. Its main contribution is replacing every other GRPO step with a refinement step. In that step, the model sees one of its own earlier solutions with a generic "review and improve" instruction, and then the responses are verified and trained with the same binary reward. They test the method on Qwen3-4B and OLMo3-7B across five math benchmarks, comparing against GRPO, Dr. GRPO, DAPO and the training-free methods Self-Refine and Reflexion. ThinkTwice gets the highest average pass@4 for both direct reasoning and self-refinement. On Qwen3-4B it beats GRPO on AIME by 5 points before refinement and by 11.5 points after one refinement step.
Critical strengths
- Although Thinktwise requires two training steps for each question, it surpasses other RLVR methods with fewer training steps and fewer input questions. Thus, the method requires almost no additional overhead.
- It needs no critique data, process labels, teacher model, or hints about whether the first answer was correct. That matters most at the frontier, where no stronger supervisor exists.
Critical weaknesses
- Only 4B and 7B instruct models are tested in non-thinking mode, with a 3000-token response cap. It's unclear whether the gains hold for larger models, long-CoT thinking models, and pretrined base models.
- AIME means AIME 2022–2024. These problems may appear in the 2025 models' training data. No AIME 2025/2026 results are reported.
- The training-free baselines are weak: Reflexion is admittedly ill-suited to this setting. The paper also lacks direct empirical comparisons with trained self-correction methods such as SCoRe, S2R, or PAG, even though it discusses them in related work.
Questions for authors
- Is the method still effective under these two model settings:
- Pretrained base models (the R1-Zero-style setting). Here self-checking and multi-turn instruction following are weak. The refinement phase might then add little or no learning signal at first, since early refinements would rarely fix anything. It might also need a warm-up or SFT stage first. The paper never tests this.
- Long-CoT "thinking" models. These already do substantial implicit self-checking, so an explicit refinement turn may add less.
- Can this method also achieve stable improvements on AIME25/26 and benchmarks in other domains?
3 comments
Authors Comment
Dear Reviewer BoSn,
Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.
Best regards,
The Authors
Thank you for your reply. Most of my questions have been resolved, so I will raise the score to 3.
Authors' Rebuttal
We thank the reviewer for the careful review.
We use instruction-tuned models because the refinement phase is a multi-turn instruction-following task (Section 4.1). All methods follow the Dr. GRPO protocol with a 3,000-token budget. As the reviewer notes, a pretrained base model would first need to learn to follow the review instruction. Long-CoT thinking models generate many times more tokens per response, which is beyond our compute budget, so we leave them and models larger than Qwen3-4B and OLMo3-7B to future work.
Yes. On AIME 2025 and 2026, ThinkTwice keeps its self-refinement lead. We evaluated Qwen3-4B on the 60 problems of AIME 2025 and 2026 with the protocol of Tables 1 and 2. AIME 2026 took place after Qwen3-4B-Instruct-2507 was released, so it cannot appear in its training data.
ThinkTwice's reasoning gains 7 points over the base model and matches DAPO. After one refinement step, it leads DAPO by 5.4 points and the base model by 8.2. As the conclusion notes, other verifiable domains such as code generation are a natural direction for future work.
Against a SCoRe-style reward in our framework, ThinkTwice leads by about 4 points in self-refinement. Among the training-free baselines, Self-Refine improves on one refinement step of the base model by 2.7 points on Qwen3-4B and 3.5 on OLMo3-7B (Table 2), so it is a reasonable reference. The stronger references are the trained baselines and the cross-model evaluation of Figure 3.
SCoRe trains Gemini models, and S2R and PAG train other base models. We therefore trained our framework with SCoRe's stage-II reward [1], which adds a bonus for improving over the first attempt. We also trained it with the second step re-solving the original problem instead of refining:
The SCoRe-style reward stays at the level of training twice, and ThinkTwice leads it by 4.0 points in self-refinement, with a 95% bootstrap interval of [+3.0, +5.5]. Rewarding progress over the first attempt therefore does not improve on ThinkTwice's plain correctness reward. S2R and PAG add verification objectives, which Appendix A compares in detail.
We thank the reviewer again for the constructive review, and in particular for the suggestion to test on AIME 2025 and 2026. We have added all of the results and changes above to the revised manuscript and would be glad to address any further questions or suggestions.
[1] Kumar et al., Training Language Models to Self-Correct via Reinforcement Learning, ICLR 2025.