- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
REFLECT+ trains a model to interrupt its reasoning, produce a structured reflection, and resume from the problem and reflection while discarding the previous reasoning prefix. It combines supervised initialization with modified GRPO rewards and advantages, and reports improvements across several reasoning benchmarks and model sizes.
Critical strengths
The idea is practical, and the gains on the 1.5B model are encouraging. Figure 4 provides a useful comparison between retaining and discarding the faulty prefix, while the reflection-source and fixed-reflection continuation experiments go beyond reporting aggregate accuracy.
Critical weaknesses
-
The paper has not separated self-correction from a fresh start. Figure 4 supports removing the faulty prefix, but does not compare against restarting from the problem alone at the same trigger point and with the same remaining budget. The reflection may preserve useful progress, or the model may simply benefit from abandoning a failed attempt. These are different explanations, and the current experiments do not tell them apart. This leaves the central claim of corrective state replacement unresolved.
-
The RL objective encourages reflection, but its corrective value is not established. A failed reflective rollout receives 0 while a direct failure receives -0.25; when all rollouts fail and both types are present, the failed reflection still gets positive advantage. This is understandable as exploration shaping, but it does not establish suppression of spurious reflection. The paper also changes several components together without isolating the proposed baseline and advantage floor. I cannot tell whether the gains require this credit-assignment scheme or mainly come from the initialization, reward shaping, and context reset.
-
The main baselines do not establish an advance over existing self-correction methods. Base, SFT, and vanilla GRPO are useful training comparisons, but they are not a substitute for competing self-correction approaches. The paper discusses methods such as Self-Refine and ReActR without comparing against them. A matched-backbone, matched-budget feedback-and-retry baseline and a closely related training-based method would make the contribution much clearer. As it stands, outperforming vanilla GRPO does not show that REFLECT+ is a better way to learn self-correction.
-
The general diagnosis goes beyond the models tested. The complete method is evaluated on 0.6B, 1.5B, and 4B Qwen-family backbones. This does not distinguish a general limitation of reasoning models from small models struggling to recover in the presence of misleading context. Larger models used as external continuers provide useful evidence, but do not test the complete learned policy. The paper acknowledges capacity-dependent gains, yet its broader claims about existing self-correction still need stronger evidence or a narrower scope.
Questions for authors
- Does the reflection actually help beyond a fresh restart from the problem alone, using the same model and remaining token budget?
- With the same supervised initialization, clean-slate rollout, and reward, what do the proposed baseline and advantage floor add over ordinary GRPO? What evidence shows they reward useful correction rather than merely encourage reflection?
- How does REFLECT+ compare with an existing feedback-and-retry method and a closely related self-correction training method under matched conditions?
- Does clean-slate reflection remain necessary for stronger, well-trained reasoners, or are the gains mainly compensating for limited correction abilities in small models?
5 comments
Weakness 3. How does REFLECT+ compare with existing self-correction methods?
Thank you for pointing out that comparisons with Base, SFT, and vanilla GRPO leave open how REFLECT+ performs relative to existing self-correction methods. We added three inference baselines on five datasets: Self-Refine, feedback-only restart, and ReActR-style. All methods use the same questions, four samples per question, and the same per-question cap on total generated tokens, up to 3,072. Self-Refine and feedback-only restart use the same base model and share the initial answer and feedback. They differ in whether the initial answer is retained in the context for revision.
| Dataset | Base + Self-Refine | Base + Feedback-Only Restart | Base + ReActR-style | REFLECT+ |
|---|---|---|---|---|
| MATH500 | 26.300 | 28.150 | 32.000 | 64.350 |
| GSM8K | 41.547 | 46.323 | 45.792 | 80.402 |
| AMC23 | 8.750 | 11.875 | 14.375 | 36.250 |
| GPQA | 22.348 | 22.854 | 21.970 | 26.263 |
| LogiQA | 27.385 | 27.433 | 25.747 | 26.272 |
| Average | 25.266 | 27.327 | 27.977 | 46.707 |
The table reports Mean@4 (%), with an unweighted average across the five datasets. REFLECT+ achieved an average accuracy of 46.707%, 18.730 percentage points above ReActR-style, the highest-scoring inference baseline on average. The gains were larger on the mathematics tasks. On LogiQA, both Self-Refine and feedback-only restart scored slightly higher than REFLECT+, showing that the relative performance varies by task.
ReActR-style is a training-free inference baseline. It uses the base model to review its initial answer, decide whether revision is needed, and answer again when appropriate. The original ReActR training pipeline also uses a teacher model to generate and filter multi-turn correction trajectories, followed by SFT and GRPO. The authors report processing over 100 million tokens for data synthesis alone, so reproducing the training pipeline from scratch requires substantial additional data construction and training. The table compares feedback and retry strategies at inference time. ReActR, Appendix A We plan to reproduce the full ReActR training pipeline, compare it with REFLECT+ under matched training and evaluation conditions, and report the results in the final version of the paper.
Weakness 4. Do the conclusions extend to stronger reasoning models?
We agree that the current results do not establish whether stronger reasoning models still need a Clean-Slate restart. We evaluated the full REFLECT+ policy on Qwen backbones at 0.6B, 1.5B, and 4B. On Qwen3-4B, REFLECT+ increased average Pass@32 across six benchmarks from 88.96% to 92.21% relative to vanilla GRPO. This shows that the improvement in solution coverage also extends to the 4B backbone we tested.
Figure 4 provides complementary evidence about the role of context. With the reflection generated by REFLECT+ held fixed and Qwen2.5-7B-Instruct used for continuation in both conditions, removing the prior reasoning prefix increased average Mean@8 across three mathematics benchmarks from 46.1% to 55.1%. This suggests that the old prefix can still affect subsequent reasoning at these fixed reflection points, even with a larger continuation model. Because the experiment fixes both the trigger point and the reflection content, it tests how the larger model uses a given reflection.
In the revision, we will limit our conclusions to the models and protocols tested. In future work, we will evaluate the full policy on stronger reasoning models and compare three conditions using the same model, trigger point, and remaining budget: retaining the old prefix, restarting from the question alone, and restarting from the question and the reflection.
Question 1. Does reflection help beyond restarting from the question alone with the same model and remaining budget?
Under the settings reported in Weakness 1, retaining the reflection yielded higher Mean@8 point estimates on all five datasets, with gains of 4.26 and 6.41 percentage points on MATH500 and GSM8K, respectively.
Question 2. What do the advantage baseline and floor add over vanilla GRPO, and what evidence shows that training improves correction?
With the original question and reflection held fixed, the full RL stage improved average continuation accuracy from 34.6% to 38.1%, indicating that the trained model makes better use of a given reflection. The positive advantage for failed reflective rollouts encourages exploration. Matched ablations are needed to quantify each component’s contribution to correction.
Question 3. How does REFLECT+ compare with existing feedback-and-retry methods and related training-based methods?
Across the five datasets in the inference comparison, REFLECT+ achieved an unweighted average Mean@4 of 46.707%, exceeding the strongest inference baseline by 18.730 percentage points. REFLECT+ achieved the highest accuracy on four of the five datasets, with the largest gains on the mathematics tasks.
Question 4. Do stronger reasoning models still need a Clean-Slate restart?
The current experiments do not establish whether it is necessary. The 4B results show that the full method can improve solution coverage, and the 7B comparison shows that removing the old prefix can help a larger model use a given reflection. The full policy still needs to be evaluated directly on stronger reasoning models.
Thank you for taking the time to explain what evidence would help you assess our contribution. Your feedback guided the additional comparisons and helped us state more precisely what our results support. If these experiments and clarifications address your concerns, we would be grateful if you would reconsider your rating in light of the new evidence.
Thank you for identifying four issues: the need for a question-only restart control, the attribution of gains to individual training components, comparisons with related methods, and the scope of our conclusions. We address your four weaknesses and four questions separately below.
Weakness 1. Does retaining the reflection help beyond restarting from the question alone?
To distinguish the effect of retaining the reflection from that of restarting, we conducted a paired comparison using the same REFLECT+ checkpoint. Both conditions discard the prior reasoning at the same reflection trigger point and use the same continuation budget. Clean-Slate restarts from the original question and the reflection, while Reset-only restarts from the original question alone.
Under the evaluation settings shown in the table, Clean-Slate achieved higher Mean@8 point estimates than Reset-only on all five datasets. The gains were 4.26 and 6.41 percentage points on MATH500 and GSM8K, respectively, and 0.31, 1.36, and 3.10 percentage points on AMC23, BBH, and DROP. Since both conditions discard the prior reasoning, these results show that retaining the reflection can improve continuation accuracy beyond restarting alone.
Weakness 2. The corrective value of reward shaping and the contributions of the advantage baseline and floor
We agree with your analysis. In a group where all rollouts fail and both direct and reflective rollouts are present, failed direct rollouts receive −0.25 and failed reflective rollouts receive 0. A failed reflective rollout therefore receives a raw advantage of +0.25. This design gives the model an incentive to explore reflection. In an early training run where both kinds of failure received 0, the reflection trigger rate fell from 32.2% to 0.8%, and the model increasingly answered directly.
To assess corrective ability, Table 2 holds the original question and the reflection fixed. Average continuation accuracy rose from 34.6% to 38.1% after the full RL stage, indicating that the trained model makes better use of a given reflection. The paired comparison in Weakness 1 also shows that retaining the reflection can yield higher accuracy than restarting from the question alone under the reported settings.
In the revision, we will describe the positive advantage as an exploration incentive and limit our claims about corrective ability to the continuation gains observed in these experiments. The individual contributions of the direct-response baseline and the advantage floor still need to be tested while holding the SFT initialization, restart policy, and reward fixed.