- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Long chains of thought interleave multiple reasoning abilities, yet existing training objectives typically supervise entire trajectories or rely on final-answer correctness, making it difficult to identify and improve individual abilities. To address this limitation, the paper proposes StruCtured Reasoning (SCR), which organizes reasoning into explicit units with distinct functional roles. It further introduces ability-specific supervision objectives to enhance the model's reasoning and self-correction capabilities.
Critical strengths
The paper identifies an interesting problem: SFT and RL do not explicitly distinguish between distinct reasoning abilities when optimizing long chains of thought.
Critical weaknesses
- The paper emphasizes atomic reasoning abilities, but in practice only evaluates three coarse-grained stages: generation, verification, and revision. The solutions produced during generation and revision may themselves involve multiple distinct reasoning abilities. Moreover, the reward is simply a weighted sum of generation accuracy, verification accuracy, and format reward, without providing independent training signals for individual abilities. This design does not align well with the paper's central claims.
- The paper should more clearly distinguish its approach from existing self-correction methods and provide experimental comparisons with representative methods, such as , SCoRe, and CURE, which are already discussed in the related work.
- In Stage II, according to Eq. (12) and the default coefficients in Appendix C.3, for a sample with an incorrect initial answer, directly accepting the incorrect answer yields a reward of 0.1, whereas correctly identifying and correcting the error yields a reward of -0.2. Would this reward design inadvertently encourage the model to avoid self-correction when errors occur? Is there evidence of such reward hacking in practice?
Questions for authors
- Some symbols in the theoretical analysis are not clearly defined, making the analysis difficult to follow.
- Why does the paper not conduct experiments with o1- or R1-style large reasoning models (LRMs)? LRMs exhibit longer and more complex chains of thought, making them more suitable for studying atomic reasoning abilities. The models used in this paper are outdated and may not adequately reflect the challenges faced by modern reasoning models.
9 comments
Response to Reviewer 6LKy (2/3)
Q2: How does SCR differ from representative self-correction methods such as SCoRe and CURE, and how does it compare experimentally?
A2: We thank the reviewer for the careful comment. SCR shares some common self-correction components, but it differs in research focus and training objectives. Below we (1) clarify the technical differences, (2) report a direct experimental comparison, and (3) summarize the revisions.
(1) Differences from existing self-correction methods. Research focus. SCR is a structured reasoning method rather than a self-correction method. It targets the mainstream paradigm of long-CoT SFT followed by outcome-reward RL (e.g., DeepSeek-R1), which supervises interleaved reasoning abilities only through the final answer. SCR instead decomposes reasoning into explicit units and evaluates and trains each unit separately. GVR is one instantiation, with its units selected by the trajectory analysis in Section 3.1.
Training objectives. Even as an instantiation, GVR differs from existing methods in its training objectives. S²R [1] re-solves the problem when verification fails and rewards each attempt only by its own correctness, without penalizing additional attempts. GVR revises the previous solution, is rewarded by the resulting change in correctness, and is penalized for unnecessary revisions. SCoRe [2] always answers twice without explicit verification and optimizes the second attempt first. GVR revises only when its own verification flags an error, and reverses the stage order: it trains generation and verification before revision. CURE [3] jointly trains solving, critiquing, and re-solving without modeling the cost of revision.
(2) Comparison with self-correction baselines. We add S²R and CURE as baselines. S²R-RL and CURE are trained on the same DAPO-MATH pool as SCR for 300 RL steps, and each method follows its original initialization: S²R-SFT is trained on the official S²R SFT data, S²R-RL continues from S²R-SFT, CURE starts directly from the Instruct model, and SCR starts from SCR-SFT. As shown below, SCR achieves the highest average accuracy on all three backbones, with the largest margins on GPQA and ARC. Moreover, CURE's average is lower than Base on both Qwen backbones and only marginally higher on Llama3.1-8B (24.71% → 24.81%), so adding self-correction does not consistently improve overall accuracy.
| Method | MATH500 | AIME24 | AIME25 | Olympiad | AMC | ARC | MMLU | GPQA | AVG |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | |||||||||
| Base | 66.60 | 7.00 | 2.33 | 29.93 | 27.95 | 69.20 | 39.60 | 29.80 | 34.05 |
| S²R-SFT | 60.40 | 2.00 | 2.33 | 28.04 | 27.35 | 77.82 | 53.29 | 12.12 | 32.92 |
| S²R-RL | 68.40 | 6.00 | 5.33 | 37.98 | 37.35 | 78.84 | 55.30 | 22.22 | 38.93 |
| CURE | 65.20 | 4.67 | 3.33 | 30.42 | 31.45 | 66.30 | 47.96 | 9.09 | 32.30 |
| SCR (Ours) | 70.40 | 7.33 | 4.00 | 37.03 | 36.02 | 79.35 | 44.50 | 39.39 | 39.75 |
| Qwen2.5-7B-Instruct | |||||||||
| Base | 75.60 | 11.00 | 6.00 | 39.70 | 42.05 | 84.39 | 55.80 | 33.84 | 43.55 |
| S²R-SFT | 75.80 | 11.33 | 4.67 | 41.39 | 44.70 | 83.79 | 55.40 | 26.77 | 42.98 |
| S²R-RL | 79.80 | 15.33 | 7.67 | 47.33 | 50.72 | 84.81 | 57.42 | 32.83 | 46.99 |
| CURE | 78.20 | 13.00 | 9.67 | 42.43 | 45.54 | 61.52 | 48.27 | 14.14 | 39.10 |
| SCR (Ours) | 80.00 | 13.67 | 13.67 | 45.63 | 50.12 | 86.43 | 56.30 | 38.38 | 48.03 |
| Llama3.1-8B-Instruct | |||||||||
| Base | 42.80 | 2.00 | 0.00 | 15.85 | 12.77 | 57.08 | 42.40 | 24.75 | 24.71 |
| S²R-SFT | 55.00 | 5.67 | 0.67 | 19.58 | 22.05 | 65.70 | 43.94 | 10.10 | 27.84 |
| S²R-RL | 61.00 | 9.67 | 3.67 | 27.60 | 30.00 | 67.66 | 46.96 | 18.18 | 33.09 |
| CURE | 30.60 | 1.33 | 0.00 | 6.68 | 13.13 | 73.98 | 56.06 | 16.67 | 24.81 |
| SCR (Ours) | 60.80 | 5.33 | 4.67 | 23.70 | 24.46 | 84.56 | 43.20 | 27.27 | 34.25 |
(3) Summary of revisions. All changes are highlighted in dark rose and stay within the page limit:
-
Section 4.1: added S²R and CURE as baselines.
-
Table 1 and Section 4.2: added their results and the corresponding analysis.
-
Appendix B: added the reproduction details, training data, and key hyperparameters of S²R and CURE.
[1] S²R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning
[2] SCoRe: Self-Correction for Reasoning in Large Language Models
[3] CURE: Critical Reasoning for Enhanced Reasoning in Large Language Models
Response to Reviewer 6LKy (3/3)
Q3: Does Stage II reward accepting an incorrect answer (0.1) more than correcting it (−0.2), thereby encouraging reward hacking? What empirical evidence addresses this concern?
A3: Thank you for this careful question. The −0.2 reward goes to an unsuccessful revision, not to a successful correction. The table below lists the total reward for each outcome under Eq. (12) and the coefficients in Appendix C.3.
| Initial answer | Outcome | Format | Final acc. | Revision | Total |
|---|---|---|---|---|---|
| Incorrect | Corrected by revision (F→T) | 0.1 | 1.0 | −0.1 | 1.0 |
| Incorrect | Accepted without revision | 0.1 | 0 | 0 | 0.1 |
| Incorrect | Revised but still incorrect (F→F) | 0.1 | 0 | −0.3 | −0.2 |
| Correct | Accepted without revision | 0.1 | 1.0 | 0 | 1.1 |
| Correct | Unnecessary revision (T→T) | 0.1 | 1.0 | −0.5 | 0.6 |
| Correct | Harmful revision (T→F) | 0.1 | 0 | −0.5 | −0.4 |
Our reward design prioritizes successful correction while accounting for the cost of ineffective revision. For an incorrect initial answer, successful correction earns the highest reward. Revisions without correctness gain (F→F and T→T) are penalized because they add computation without improving the answer.
Empirical check for reward hacking. We test whether this penalty discourages revision in practice by comparing how SCR-Stage I and SCR handle incorrect initial answers on MATH500 with Qwen2.5-7B-Instruct. Percentages are computed over each model's incorrect initial answers.
| Model | Falsely accepted | Sent to revision | Successfully repaired |
|---|---|---|---|
| SCR-Stage I | 36.9% | 63.1% | 6.9% |
| SCR | 25.0% | 75.0% | 14.9% |
After Stage II, the model sends more of its errors to revision (63.1% → 75.0%) and more than doubles the repair rate (6.9% → 14.9%). These results have been added to Appendix E.2.
Q4: Please clarify the undefined symbols in the theoretical analysis.
A4: Thank you for this helpful suggestion. We have added a notation table in Appendix A.1 summarizing the key symbols in the theoretical analysis, and clarified key definitions in Section 2 to make the analysis easier to follow.
Q5: Why not evaluate SCR on modern o1/R1-style reasoning models, whose longer, more complex reasoning may better test atomic abilities?
A5: Thank you for this question. The prevailing approach to improving reasoning, represented by DeepSeek-R1, starts from a non-reasoning backbone and applies long-CoT SFT followed by RL (i.e., SFT+GRPO in Table 1). We argue that long CoT cannot effectively measure a model's individual atomic reasoning abilities (Figure 2). This observation comes from our pilot analysis of four o1/R1-style reasoning models. We therefore adopt structured reasoning to optimize these atomic abilities, which in turn improves overall reasoning performance. To ensure a fair comparison with this mainstream approach, we use the same non-reasoning backbones rather than o1/R1-style reasoning models.
Response to Reviewer 6LKy (1/3)
A1: We thank the reviewer for this comment. Below we explain why GVR focuses on these three abilities, how the rewards and stages target each one, and how a new experiment isolates the effect of the staged schedule.
(1) Why generation, verification, and revision? SCR is not limited to these three abilities; GVR is one instantiation of it. Figure 2 shows that reasoning models exhibit diverse atomic behaviors. Verification and revision are frequent but rarely effective, since most revisions leave answer correctness unchanged. We therefore focus on these two abilities and instantiate SCR as GVR (Section 3.1).
(2) How does the reward design reflect ability-specific optimization? Each reward term evaluates a distinct ability, and staged training further restricts where the policy loss applies. Stage I masks revision tokens and updates only generation and verification. Stage II adds final-answer and revision-quality rewards and unmasks revision. Although the rewards share an advantage, this design achieves selective optimization rather than fully independent credit assignment.
This design also controls cost. A conditional baseline with 8 verifications per initial answer raises rollouts per problem from 8 to 64, and adding 8 revisions per verification raises it to 512. Such nested sampling grows exponentially with depth, so staged optimization balances ability separation against sampling cost.
(3) Additional experiment. We add a Mix baseline that trains on all signals in a single stage. SCR outperforms Mix on all four benchmarks for both models (accuracy, %; see also Section 4.5 and Figure 6):