acceptodds
Under review as a conference paper at ICLR 2027

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models (LLMs) to the point where frontier models match or exceed human experts on many tasks. Self-refinement, where a model reviews and revises its own solutions, can recover problems that its first attempt narrowly misses. However, prompt-only refinement remains brittle even for frontier models, and existing training-based methods rely on external guidance such as critique annotations or explicit signals indicating whether the initial answer is correct. In this work, we introduce ThinkTwice, a simple two-phase RLVR framework that jointly optimizes an LLM to solve reasoning problems and refine its own answers, without any external signals. Concretely, ThinkTwice replaces every even GRPO step with a refinement step, which optimizes the model to revise its own solutions from the preceding step under the same binary correctness reward, so training takes only 3% longer than GRPO. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Reasoning, Self-Refinement, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BoSnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ThinkTwice is an RLVR framework with self-refinement that trains an LLM to do two things: solve reasoning problems and revise its own answers. Its main contribution is replacing every other GRPO step with a refinement step. In that step, the model sees one of its own earlier solutions with a generic "review and improve" instruction, and then the responses are verified and trained with the same binary reward. They test the method on Qwen3-4B and OLMo3-7B across five math benchmarks, comparing against GRPO, Dr. GRPO, DAPO and the training-free methods Self-Refine and Reflexion. ThinkTwice gets the highest average pass@4 for both direct reasoning and self-refinement. On Qwen3-4B it beats GRPO on AIME by 5 points before refinement and by 11.5 points after one refinement step.

Critical strengths
  1. Although Thinktwise requires two training steps for each question, it surpasses other RLVR methods with fewer training steps and fewer input questions. Thus, the method requires almost no additional overhead.
  2. It needs no critique data, process labels, teacher model, or hints about whether the first answer was correct. That matters most at the frontier, where no stronger supervisor exists.
Critical weaknesses
  1. Only 4B and 7B instruct models are tested in non-thinking mode, with a 3000-token response cap. It's unclear whether the gains hold for larger models, long-CoT thinking models, and pretrined base models.
  2. AIME means AIME 2022–2024. These problems may appear in the 2025 models' training data. No AIME 2025/2026 results are reported.
  3. The training-free baselines are weak: Reflexion is admittedly ill-suited to this setting. The paper also lacks direct empirical comparisons with trained self-correction methods such as SCoRe, S2R, or PAG, even though it discusses them in related work.
Questions for authors
  • Is the method still effective under these two model settings:
  1. Pretrained base models (the R1-Zero-style setting). Here self-checking and multi-turn instruction following are weak. The refinement phase might then add little or no learning signal at first, since early refinements would rarely fix anything. It might also need a warm-up or SFT stage first. The paper never tests this.
  2. Long-CoT "thinking" models. These already do substantial implicit self-checking, so an explicit refinement turn may add less.
  • Can this method also achieve stable improvements on AIME25/26 and benchmarks in other domains?
3 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the careful review.

W1 & Q: Larger, thinking and pretrained base models

We use instruction-tuned models because the refinement phase is a multi-turn instruction-following task (Section 4.1). All methods follow the Dr. GRPO protocol with a 3,000-token budget. As the reviewer notes, a pretrained base model would first need to learn to follow the review instruction. Long-CoT thinking models generate many times more tokens per response, which is beyond our compute budget, so we leave them and models larger than Qwen3-4B and OLMo3-7B to future work.

W2 & Q: Does ThinkTwice achieve stable improvements on AIME 2025 and 2026?

Yes. On AIME 2025 and 2026, ThinkTwice keeps its self-refinement lead. We evaluated Qwen3-4B on the 60 problems of AIME 2025 and 2026 with the protocol of Tables 1 and 2. AIME 2026 took place after Qwen3-4B-Instruct-2507 was released, so it cannot appear in its training data.

Pass@4, AIME 2025 and 2026ReasoningSelf-refinement
Base model32.043.8
DAPO39.246.6
ThinkTwice39.052.0

ThinkTwice's reasoning gains 7 points over the base model and matches DAPO. After one refinement step, it leads DAPO by 5.4 points and the base model by 8.2. As the conclusion notes, other verifiable domains such as code generation are a natural direction for future work.

W3: Training-free baselines and trained self-correction methods

Against a SCoRe-style reward in our framework, ThinkTwice leads by about 4 points in self-refinement. Among the training-free baselines, Self-Refine improves on one refinement step of the base model by 2.7 points on Qwen3-4B and 3.5 on OLMo3-7B (Table 2), so it is a reasonable reference. The stronger references are the trained baselines and the cross-model evaluation of Figure 3.

SCoRe trains Gemini models, and S2R and PAG train other base models. We therefore trained our framework with SCoRe's stage-II reward [1], which adds a bonus for improving over the first attempt. We also trained it with the second step re-solving the original problem instead of refining:

Pass@4, averageReasoningSelf-refinement
Base model56.0164.10
Training twice61.9467.81
SCoRe-style reward62.3767.89
ThinkTwice65.5771.88

The SCoRe-style reward stays at the level of training twice, and ThinkTwice leads it by 4.0 points in self-refinement, with a 95% bootstrap interval of [+3.0, +5.5]. Rewarding progress over the first attempt therefore does not improve on ThinkTwice's plain correctness reward. S2R and PAG add verification objectives, which Appendix A compares in detail.

We thank the reviewer again for the constructive review, and in particular for the suggestion to test on AIME 2025 and 2026. We have added all of the results and changes above to the revised manuscript and would be glad to address any further questions or suggestions.

[1] Kumar et al., Training Language Models to Self-Correct via Reinforcement Learning, ICLR 2025.

AuthorsOpenReview ↗

Authors Comment

Dear Reviewer BoSn,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thank you for your reply. Most of my questions have been resolved, so I will raise the score to 3.

Reviewer_BVNFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ThinkTwice, a two-phase RLVR framework for jointly training mathematical reasoning and self-refinement. The method alternates between a standard reasoning phase and a refinement phase. In the reasoning phase, the model samples multiple direct solutions to each problem and is updated using GRPO with a final-answer correctness reward. In the refinement phase, one base solution from the previous phase is inserted into a multi-turn prompt with a generic review instruction, and the model generates refined solutions that are again optimized with the same final-answer correctness reward. The method does not use critique annotations, process supervision, or explicit signals indicating whether the initial solution was correct. Experiments on Qwen3-4B and OLMo3-7B across AIME, AMC, MATH500, Minerva Math, and OlympiadBench show improvements over GRPO, DrGRPO, DAPO, and training-free refinement baselines. The paper also analyzes fix-wrong and damage-correct rates, response length, formatting behavior, cross-model refinement, and wall-clock cost.

Critical strengths
  1. The problem is important. Training models to refine their own reasoning without critique annotations or process supervision is a useful direction.
  2. The reported self-refinement gains are meaningful, especially on Qwen3-4B AIME, where ThinkTwice substantially outperforms GRPO after refinement.
  3. The paper is clearly written and well organized. The method, prompts, implementation sketch, and hyperparameters are described in enough detail to make the approach understandable.
  4. The cross-model refinement evaluation is useful because it attempts to separate the quality of the refiner from the quality of the base reasoning model.
Critical weaknesses
  1. The novelty is limited. ThinkTwice is close to a natural alternating-task GRPO or two-turn prompt-augmentation recipe. Relative to existing self-refinement, multi-turn RL, self-correction RL, and RLVR methods, the conceptual contribution is modest.
  2. The paper lacks key ablations needed to isolate the source of the gains. It does not compare against a baseline that trains twice on the same problem without including the previous solution, a refinement-only variant, a matched-token or matched-generation variant, a DAPO-backed ThinkTwice variant, or stratified base-solution selection.
  3. The direct reasoning gains are not very robust-looking. On Qwen3-4B, ThinkTwice averages 65.57 while DAPO averages 65.00. Without significance tests, this small difference is hard to interpret.
  4. The paper does not report sufficient statistical uncertainty. RLVR training and pass@k evaluation are noisy, yet the main tables provide point estimates without seed variance, confidence intervals, or paired significance tests.
Questions for authors
  1. Please report mean and standard deviation over multiple training seeds, as well as paired bootstrap or other significance tests comparing ThinkTwice with DAPO, GRPO, and DrGRPO.
  2. How exactly are checkpoints selected? If the fixed 20% subsets come from the final evaluation benchmarks, can the authors provide results with a strictly held-out validation set or train-only validation set?
  3. Can the authors add a matched two-pass baseline where each problem is trained twice, but the second pass uses a standard direct-solve prompt instead of a previous-solution refinement prompt?
  4. Can the authors evaluate ThinkTwice on top of DAPO or DrGRPO rather than only standard GRPO? This would test whether refinement is complementary to stronger RLVR backbones.
3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (Part 1)

We thank the reviewer for the careful review.

W1: Novelty relative to an alternating-task GRPO recipe

The novelty is showing that self-refinement can be learned from the same sparse outcome reward that RLVR already uses, and that learning it also strengthens direct reasoning. ThinkTwice jointly optimizes reasoning and self-refinement using the same binary correctness reward in both phases. The families the reviewer lists differ as follows (Section 2 and Table 4):

  • RLVR methods such as GRPO, Dr. GRPO and DAPO train only first-pass reasoning.
  • Training-free self-refinement such as Self-Refine and Reflexion does not learn to refine, and ThinkTwice outperforms both (Table 2).
  • Multi-turn and self-correction RL such as SCoRe, S2R and PAG adds fine-tuning on correction traces (S2R), an explicit verification step (S2R, PAG) or a bonus for improving over the first attempt (SCoRe, PAG). ThinkTwice uses none of these.

We see this minimal design as a feature, since it lets ThinkTwice run inside a standard GRPO loop at 3% extra wall-clock time. Training ThinkTwice also gives rise to an implicit rectify-then-fortify curriculum: refinement first corrects errors and later preserves correct solutions, without any designed schedule (Section 5.1). The controls in W2 & Q3 below show that the gain comes from revising a prior solution, not from seeing each problem twice.

W2 & Q3: Does a matched two-pass baseline reproduce the gains?

No. Training each problem twice without the previous solution does not reproduce ThinkTwice's gains. We trained the same pipeline with the second step re-solving each problem with the direct prompt, so it never sees its previous solution. The table also includes a SCoRe-style reward, which adds a bonus for improving over the first attempt:

Pass@4, averageReasoningSelf-refinement
Base model56.0164.10
Training twice61.9467.81
SCoRe-style reward62.3767.89
ThinkTwice65.5771.88

ThinkTwice leads training twice by 3.6 points in reasoning and 4.1 in self-refinement, with paired bootstrap intervals of [+1.5, +4.1] and [+2.9, +5.6].

W2 & Q4: Is refinement complementary to stronger RLVR backbones?

Yes. ThinkTwice on top of DAPO improves DAPO by 1.7 points in reasoning and 1.5 in self-refinement. We trained ThinkTwice with DAPO and evaluated it with the protocol of Tables 1 and 2:

Pass@4, averageReasoningSelf-refinement
DAPO65.0069.87
ThinkTwice65.5771.88
ThinkTwice + DAPO66.6771.41

Compared with ThinkTwice on GRPO, it adds 1.1 points in reasoning, while self-refinement stays on par. Figure 3 points the same way: DAPO's solutions rise from 62.71 to 67.07 when ThinkTwice refines them instead of DAPO.

W2: Matched compute, stratified selection and a refinement-only variant

First, compute is matched to GRPO in training and at inference time. All methods use the same number of policy updates, rollouts per update and generation budget (Section 4.1). At eight generations per problem, self-refinement pass@4 beats reasoning pass@8 by 3.2 points on Qwen3-4B and 1.8 on OLMo3-7B (Appendix C.2).

Second, random selection of the base solution beats stratified selection. Balanced takes half correct and half incorrect base solutions, and Wrong-first prefers incorrect ones:

Pass@4, averageReasoningSelf-refinement
Wrong-first63.8370.11
Balanced64.9471.46
Random, ThinkTwice65.5771.88

Third, we did not train a refinement-only variant, because our claim is that one policy trained on both tasks improves at both, and the two-pass baseline tests the role of refinement more directly. Figure 3 also isolates refinement capability from base solution quality: ThinkTwice as the refinement model achieves the highest score regardless of which model produced the base solution.

AuthorsOpenReview ↗

Authors' Rebuttal (Part 2)

W3, W4 & Q1: Training seeds and paired significance tests

Over three training seeds, the standard deviation of ThinkTwice's pass@4 is about 0.2 points, and its self-refinement lead over every baseline is significant. As in GRPO [1] and DAPO [2], the paper reports single training runs. Following the reviewer's request, we trained ThinkTwice with two more seeds, and the mean stays above every baseline in both reasoning and self-refinement:

Pass@4, averageReasoningSelf-refinement
Seed 1, in the paper65.5771.88
Seed 265.2772.05
Seed 365.3471.64
Mean ± std65.39 ± 0.1671.86 ± 0.21

In reasoning, ThinkTwice significantly outperforms GRPO and Dr. GRPO and is on par with DAPO, the strongest reasoning baseline, as Section 4.2 notes at pass@1. Each pass@4 is an unbiased estimate from 32 samples per problem over 1,526 problems. A paired bootstrap over problems gives these 95% intervals for ThinkTwice minus each baseline on Qwen3-4B:

ThinkTwice − baseline, pass@4ReasoningSelf-refinement
GRPO[+1.33, +3.82][+2.87, +5.62]
Dr. GRPO[+0.60, +3.36][+2.56, +5.24]
DAPO[−0.95, +1.22][+1.47, +3.48]

DAPO reaches its reasoning level at a much higher cost. It trains on as many rollouts per update as the other methods, but its dynamic sampling discards prompts whose rollouts are all correct or all wrong and samples again, so it generates 2.5 to 3 times as many rollouts as it trains on. To reach its best checkpoint, DAPO generates nearly five times as many rollouts as ThinkTwice and takes about twice the training time.

Q2: Do the results hold on problems that checkpoint selection never used?

Yes. Reporting only on problems that no checkpoint selection used keeps ThinkTwice first in both settings. For every method, we keep the checkpoint with the highest greedy accuracy on a fixed random 20% subset of each benchmark (Appendix B.1). To separate selection from reporting, we evaluated the selected checkpoints of Qwen3-4B on the remaining 80% of each benchmark, which no method used for training or selection:

Pass@4 on the 80% split, averageReasoningSelf-refinement
GRPO62.867.8
Dr. GRPO63.067.8
DAPO64.469.8
ThinkTwice65.171.9

Taken together, this response adds the matched two-pass baseline, a SCoRe-style reward, ThinkTwice on top of DAPO, stratified selection of the base solution, paired bootstrap intervals against every baseline and results on the 80% split that no checkpoint selection used, and we have added all of them to the revised manuscript. It also reports ThinkTwice over three training seeds. We thank the reviewer again, in particular for asking for the matched two-pass baseline, which shows where ThinkTwice's gains come from, and would be glad to address any further questions or suggestions.

[1] Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.

[2] Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025.

AuthorsOpenReview ↗

Authors Comment

Dear Reviewer BVNF,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Reviewer_rZvvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ThinkTwice, a new RLVR method for LLM fine-tuning augmented with an inner self-refinement phase. More specifically, each training iteration has two phases: (1) a normal GRPO step with on-policy samples, and (2) a GRPO step using self-refined answers that are generated by prompting the LLM to refine a previously generated answer from the first phase. The proposed method is validated on 5 mathematical reasoning tasks using Qwen3-4B and OLMo3-7B models.

Critical strengths
  1. The strongest part is that the idea is very simple, and it still shows its effectiveness in the considered benchmark. Thanks to its simplicity and generality, I think this work could potentially be used in many different contexts for RL fine-tuning of LLMs.
Critical weaknesses

1. About the checkpoints selection

The authors selected the checkpoint with the highest average performance on a 20% subset of each benchmark (B.1.3). This raises two concerns:

(1) The subset is from the test data (and I guess the subset is even included in the test data). I believe that, after fine-tuning, we expect the LLM to learn useful skills for solving a range of problems beyond the training set. However, since the authors select the checkpoint based on the test set, the generalisation capability is never tested.

(2) Selecting the checkpoint based on test set performance is more favorable for a method whose performance fluctuates more. This matters here because Figure 5 (e) shows that ThinkTwice’s avg. accuracy peaks around step 220 and then declines to below the GRPO by step 300. That is, ThinkTwice's performance fluctuates more than GRPO’s, so the gains (e.g., Table 1) may therefore be partly due to favorable checkpoint selection.

2. About the benchmark.

Only math benchmarks are considered, and I think the refinement instruction slightly exploits this. In the instruction, you have "1. Go through each calculation step-by-step. Check if there are any errors in calculations, logic, or problem understanding.", which seems tailored for math problems. So I want to see the results on the same problems with a more general instruction (e.g., "1. Go through each step and check if there are any errors"), as well as results on tasks in other domains, e.g., code generation as the authors mentioned in the conclusion.

Minor comments/suggestions

  • In Algorithm 1, line 8 "Update πθ\pi_{\theta} to πθ′\pi_{\theta}'" seems a little bit imprecise. Something like θ←θ′\theta \leftarrow \theta', where θ′\theta' is from the GRPO update, might be more accurate. Same for line 16.
  • In the related work section, it might be clearer if the citations are placed right after the method names rather than at the end of each sentence.
  • Please add citations in Table 4.
Questions for authors
  1. I think the two-phase idea of ThinkTwice is orthogonal to the objective function, so it can be combined with DAPO or Dr. GRPO. I wonder how much improvement ThinkTwice would bring over DAPO (which was the strongest baseline).
  2. Have you tried a strategy other than random selection for ybasey_{\text{base}}? How sensitive is the proposed method to the choice of ybasey_{\text{base}} selection strategy?
  3. It seems the idea is similar to the SCoRe paper [1]. Could you discuss the key differences between ThinkTwice and SCoRe? (Ideally, a detailed comparison not only with SCoRe but also with other closely related prior works would be very helpful.)

[1] Kumar, Aviral, et al. "Training language models to self-correct via reinforcement learning." ICLR 2025.

6 comments
AuthorsOpenReview ↗

Authors' Rebuttal

We thank the reviewer for the careful review.

W1: Are the gains partly due to favorable checkpoint selection?

No. Reporting only on problems that no checkpoint selection used keeps ThinkTwice first in both settings. For every method, we keep the checkpoint with the highest greedy accuracy on a fixed random 20% subset of each benchmark (Appendix B.1). To separate selection from reporting, we evaluated the selected checkpoints of Qwen3-4B on the remaining 80% of each benchmark, which no method used for training or selection:

Pass@4 on the 80% split, averageReasoningSelf-refinement
GRPO62.867.8
Dr. GRPO63.067.8
DAPO64.469.8
ThinkTwice65.171.9

On fluctuation, Figure 5(e) shows that ThinkTwice improves faster in early and middle training and stays above GRPO for most of the trajectory. Because ThinkTwice stays first on the 80% split, the gains do not come from a favorable choice of checkpoint.

W2: Does the gain depend on an instruction tailored to math?

No. ThinkTwice's self-refinement lead holds under general review instructions. Following the reviewer's suggestion, we reran self-refinement on Qwen3-4B with two general instructions, for ThinkTwice and for DAPO, the strongest baseline:

Self-refinement pass@4, averageDAPOThinkTwice
Minimal instruction70.571.9
General step-by-step instruction70.572.1

Under both instructions, a paired bootstrap over problems gives a lead over DAPO whose 95% interval excludes zero, and ThinkTwice scores about the same as with the paper's instruction.

As the conclusion notes, other verifiable domains such as code generation are a natural direction for future work.

Minor comments

We have rewritten lines 8 and 16 of Algorithm 1 as θ′ ← θ + Δθ with Δθ from the GRPO update, placed each citation right after its method name in Section 2, and added citations to Table 4.

Q1: How much does ThinkTwice improve DAPO?

ThinkTwice on top of DAPO improves DAPO by 1.7 points in reasoning and 1.5 in self-refinement. We trained ThinkTwice with DAPO and evaluated it with the protocol of Tables 1 and 2:

Pass@4, averageReasoningSelf-refinement
DAPO65.0069.87
ThinkTwice65.5771.88
ThinkTwice + DAPO66.6771.41

Compared with ThinkTwice on GRPO, it adds 1.1 points in reasoning, while self-refinement stays on par.

Q2: Have you tried other selection strategies for the base solution?

Yes. Random selection, our default, gives the best average in both settings. As Section 3.3 explains, it is the simplest choice and gives rise to an emergent curriculum, from correcting errors early in training to polishing correct solutions as accuracy improves. Two strategies that condition on correctness score lower, which supports this design. Balanced takes half correct and half incorrect base solutions, and Wrong-first prefers incorrect ones:

Pass@4, averageReasoningSelf-refinement
Wrong-first63.8370.11
Balanced64.9471.46
Random, ThinkTwice65.5771.88

Q3: Key differences from SCoRe

The main difference is the reward: ThinkTwice rewards only the correctness of the refined answer, whereas SCoRe adds a bonus that depends on whether the first attempt was correct. SCoRe [1] needs this bonus, together with a first stage that constrains the first attempt, to keep multi-turn RL from collapsing to repeating the first answer. ThinkTwice uses neither, and its fix-wrong rate stays above GRPO's throughout training (Section 5.1). Other trained self-correction methods add verification or critique objectives, process supervision, or synthetic correction data, which Appendix A compares in detail.

We trained our framework with SCoRe's stage-II reward, and also with the second step re-solving the original problem instead of refining, so each problem is trained twice without the previous solution:

Pass@4, averageReasoningSelf-refinement
Base model56.0164.10
Training twice61.9467.81
SCoRe-style reward62.3767.89
ThinkTwice65.5771.88

The SCoRe-style reward stays at the level of training twice, and ThinkTwice leads it by 4.0 points in self-refinement, with a 95% bootstrap interval of [+3.0, +5.5]. Rewarding progress over the first attempt therefore does not improve on ThinkTwice's plain correctness reward.

We thank the reviewer again, in particular for the suggestions on the refinement instruction and on combining ThinkTwice with DAPO. We have added all of the results and changes above to the revised manuscript and would be glad to address any further questions or suggestions.

[1] Kumar et al., Training Language Models to Self-Correct via Reinforcement Learning, ICLR 2025.

AuthorsOpenReview ↗

Authors Comment

Dear Reviewer rZvv,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thank you for your rebuttal, and I'm sorry for the late reply.

  1. One of my concerns, W1 (a), remains unresolved. The authors' response to W1 addresses only W1 (b).
  2. I still think results on tasks in domains other than math are necessary, as I noted in my W2.
  3. Please check line 1628 of the updated manuscript ("Review your previous answer ...").

For now, I'd like to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals and Paper Revisions

We thank the three reviewers for their careful reviews. We have uploaded a revised PDF with the main revisions marked in blue.

New results in the responses

  1. Training each problem twice without the previous solution does not reproduce ThinkTwice's gains, and neither does a SCoRe-style reward (all reviewers). ThinkTwice leads each of them by about 4 points in self-refinement. The gains therefore come from revising a prior solution, and rewarding progress over the first attempt does not improve on the plain correctness reward.
  2. ThinkTwice stays first on problems that no checkpoint selection used (rZvv, BVNF). On the 80% of each benchmark that no method used for training or selection, ThinkTwice has the highest average in both reasoning and self-refinement, so the gains do not come from a favorable choice of checkpoint.
  3. Over three training seeds, the standard deviation of ThinkTwice's pass@4 is about 0.2 points, and its self-refinement lead over every baseline is significant (BVNF). Paired bootstrap intervals over problems exclude zero against GRPO, Dr. GRPO and DAPO. In reasoning, ThinkTwice significantly outperforms GRPO and Dr. GRPO and is on par with DAPO, which generates nearly five times as many rollouts as ThinkTwice to reach its best checkpoint.
  4. ThinkTwice on top of DAPO improves DAPO in both reasoning and self-refinement (rZvv, BVNF). Refinement training is therefore complementary to a stronger RLVR backbone.
  5. The self-refinement lead holds under general review instructions and on AIME 2025 and 2026 (rZvv, BoSn). Under two general instructions, ThinkTwice still leads DAPO and scores about the same as with the paper's instruction. On AIME 2025 and 2026, which include problems released after Qwen3-4B-Instruct-2507, it leads DAPO by 5.4 points after one refinement step. The self-refinement lead therefore depends neither on the paper's instruction nor on the original benchmarks.

Changes to the paper

  • The abstract and introduction state that the two ablations do not reproduce the gains.
  • The new Section 5.3 and Table 4 report both ablations on every benchmark.
  • The new Section 5.4 states how ThinkTwice differs from multi-turn self-correction RL and states its limitations, including model scale.
  • The new Appendix E reports the evaluation on the 80% split, bootstrap intervals against every baseline, ThinkTwice on top of DAPO, the selection of the base solution, the general review instructions and AIME 2025 and 2026.
  • Algorithm 1 writes the parameter updates as θ′ ← θ + Δθ, Section 2 places each citation right after its method name, and the related-work table, now Table 5, adds citations.

Our responses cite tables by their numbers in the original submission. In the revision, the new Table 4 holds the two ablations, so former Tables 4 to 11 are now Tables 5 to 12.

We thank the reviewers again and would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.