acceptodds
Under review as a conference paper at ICLR 2027

Structured Reasoning for Large Language Models

Abstract

Large reasoning models (LRMs) achieve strong performance by producing long chains of thought (CoT). However, a single trajectory entangles functionally different reasoning behaviors, making it difficult to assess or optimize each behavior on its own merits. This entanglement is reinforced by prevailing training recipes, where supervised fine-tuning imitates all tokens with uniform weight and outcome-level reinforcement learning compresses the entire trajectory into a single outcome reward, leaving the model without ability-specific credit and unable to optimize individual behaviors. To address this, we propose StruCtured Reasoning (SCR), which reorganizes reasoning into stages that are explicit, evaluable, and independently trainable, and realizes them through a Generate-Verify-Revise paradigm. Specifically, SCR introduces two trajectory-aware supervision mechanisms, where Dynamic Termination Supervision ties the end of reasoning to the self-verification outcome, and Selective Loss Masking excludes incorrect initial answers from the imitation loss while preserving them as context. A progressive two-stage reinforcement learning schedule then jointly optimizes generation and verification, and turns to revision once the verifier becomes sufficiently reliable. Across three backbone models and multiple reasoning benchmarks, SCR consistently improves reasoning quality and self-verification accuracy while reducing output length by up to 50%. Code is available at https://anonymous.4open.science/r/SCR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6LKyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Long chains of thought interleave multiple reasoning abilities, yet existing training objectives typically supervise entire trajectories or rely on final-answer correctness, making it difficult to identify and improve individual abilities. To address this limitation, the paper proposes StruCtured Reasoning (SCR), which organizes reasoning into explicit units with distinct functional roles. It further introduces ability-specific supervision objectives to enhance the model's reasoning and self-correction capabilities.

Critical strengths

The paper identifies an interesting problem: SFT and RL do not explicitly distinguish between distinct reasoning abilities when optimizing long chains of thought.

Critical weaknesses
  1. The paper emphasizes atomic reasoning abilities, but in practice only evaluates three coarse-grained stages: generation, verification, and revision. The solutions produced during generation and revision may themselves involve multiple distinct reasoning abilities. Moreover, the reward is simply a weighted sum of generation accuracy, verification accuracy, and format reward, without providing independent training signals for individual abilities. This design does not align well with the paper's central claims.
  2. The paper should more clearly distinguish its approach from existing self-correction methods and provide experimental comparisons with representative methods, such as S2RS^2R, SCoRe, and CURE, which are already discussed in the related work.
  3. In Stage II, according to Eq. (12) and the default coefficients in Appendix C.3, for a sample with an incorrect initial answer, directly accepting the incorrect answer yields a reward of 0.1, whereas correctly identifying and correcting the error yields a reward of -0.2. Would this reward design inadvertently encourage the model to avoid self-correction when errors occur? Is there evidence of such reward hacking in practice?
Questions for authors
  1. Some symbols in the theoretical analysis are not clearly defined, making the analysis difficult to follow.
  2. Why does the paper not conduct experiments with o1- or R1-style large reasoning models (LRMs)? LRMs exhibit longer and more complex chains of thought, making them more suitable for studying atomic reasoning abilities. The models used in this paper are outdated and may not adequately reflect the challenges faced by modern reasoning models.
9 comments
AuthorsOpenReview ↗

Response to Reviewer 6LKy (1/3)

Q1: How do three coarse GVR stages and aggregated rewards support the claim of atomic, ability-specific optimization without independent training signals?

A1: We thank the reviewer for this comment. Below we explain why GVR focuses on these three abilities, how the rewards and stages target each one, and how a new experiment isolates the effect of the staged schedule.

(1) Why generation, verification, and revision? SCR is not limited to these three abilities; GVR is one instantiation of it. Figure 2 shows that reasoning models exhibit diverse atomic behaviors. Verification and revision are frequent but rarely effective, since most revisions leave answer correctness unchanged. We therefore focus on these two abilities and instantiate SCR as GVR (Section 3.1).

(2) How does the reward design reflect ability-specific optimization? Each reward term evaluates a distinct ability, and staged training further restricts where the policy loss applies. Stage I masks revision tokens and updates only generation and verification. Stage II adds final-answer and revision-quality rewards and unmasks revision. Although the rewards share an advantage, this design achieves selective optimization rather than fully independent credit assignment.

This design also controls cost. A conditional baseline with 8 verifications per initial answer raises rollouts per problem from 8 to 64, and adding 8 revisions per verification raises it to 512. Such nested sampling grows exponentially with depth, so staged optimization balances ability separation against sampling cost.

(3) Additional experiment. We add a Mix baseline that trains on all signals in a single stage. SCR outperforms Mix on all four benchmarks for both models (accuracy, %; see also Section 4.5 and Figure 6):

ModelMethodMATHAIME25OlympiadAMC
Qwen2.5-7B-InstructSCR-Stage I77.009.0042.0744.46
Mix76.009.0040.4448.20
SCR80.0013.6745.6350.12
Qwen2.5-3B-InstructSCR-Stage I65.202.0029.0429.16
Mix60.003.3332.0034.34
SCR70.404.0037.0336.02
AuthorsOpenReview ↗

Response to Reviewer 6LKy (2/3)

Q2: How does SCR differ from representative self-correction methods such as SCoRe and CURE, and how does it compare experimentally?

A2: We thank the reviewer for the careful comment. SCR shares some common self-correction components, but it differs in research focus and training objectives. Below we (1) clarify the technical differences, (2) report a direct experimental comparison, and (3) summarize the revisions.

(1) Differences from existing self-correction methods. Research focus. SCR is a structured reasoning method rather than a self-correction method. It targets the mainstream paradigm of long-CoT SFT followed by outcome-reward RL (e.g., DeepSeek-R1), which supervises interleaved reasoning abilities only through the final answer. SCR instead decomposes reasoning into explicit units and evaluates and trains each unit separately. GVR is one instantiation, with its units selected by the trajectory analysis in Section 3.1.

Training objectives. Even as an instantiation, GVR differs from existing methods in its training objectives. S²R [1] re-solves the problem when verification fails and rewards each attempt only by its own correctness, without penalizing additional attempts. GVR revises the previous solution, is rewarded by the resulting change in correctness, and is penalized for unnecessary revisions. SCoRe [2] always answers twice without explicit verification and optimizes the second attempt first. GVR revises only when its own verification flags an error, and reverses the stage order: it trains generation and verification before revision. CURE [3] jointly trains solving, critiquing, and re-solving without modeling the cost of revision.

(2) Comparison with self-correction baselines. We add S²R and CURE as baselines. S²R-RL and CURE are trained on the same DAPO-MATH pool as SCR for 300 RL steps, and each method follows its original initialization: S²R-SFT is trained on the official S²R SFT data, S²R-RL continues from S²R-SFT, CURE starts directly from the Instruct model, and SCR starts from SCR-SFT. As shown below, SCR achieves the highest average accuracy on all three backbones, with the largest margins on GPQA and ARC. Moreover, CURE's average is lower than Base on both Qwen backbones and only marginally higher on Llama3.1-8B (24.71% → 24.81%), so adding self-correction does not consistently improve overall accuracy.

MethodMATH500AIME24AIME25OlympiadAMCARCMMLUGPQAAVG
Qwen2.5-3B-Instruct
Base66.607.002.3329.9327.9569.2039.6029.8034.05
S²R-SFT60.402.002.3328.0427.3577.8253.2912.1232.92
S²R-RL68.406.005.3337.9837.3578.8455.3022.2238.93
CURE65.204.673.3330.4231.4566.3047.969.0932.30
SCR (Ours)70.407.334.0037.0336.0279.3544.5039.3939.75
Qwen2.5-7B-Instruct
Base75.6011.006.0039.7042.0584.3955.8033.8443.55
S²R-SFT75.8011.334.6741.3944.7083.7955.4026.7742.98
S²R-RL79.8015.337.6747.3350.7284.8157.4232.8346.99
CURE78.2013.009.6742.4345.5461.5248.2714.1439.10
SCR (Ours)80.0013.6713.6745.6350.1286.4356.3038.3848.03
Llama3.1-8B-Instruct
Base42.802.000.0015.8512.7757.0842.4024.7524.71
S²R-SFT55.005.670.6719.5822.0565.7043.9410.1027.84
S²R-RL61.009.673.6727.6030.0067.6646.9618.1833.09
CURE30.601.330.006.6813.1373.9856.0616.6724.81
SCR (Ours)60.805.334.6723.7024.4684.5643.2027.2734.25

(3) Summary of revisions. All changes are highlighted in dark rose and stay within the page limit:

  • Section 4.1: added S²R and CURE as baselines.

  • Table 1 and Section 4.2: added their results and the corresponding analysis.

  • Appendix B: added the reproduction details, training data, and key hyperparameters of S²R and CURE.

[1] S²R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning

[2] SCoRe: Self-Correction for Reasoning in Large Language Models

[3] CURE: Critical Reasoning for Enhanced Reasoning in Large Language Models

AuthorsOpenReview ↗

Response to Reviewer 6LKy (3/3)

Q3: Does Stage II reward accepting an incorrect answer (0.1) more than correcting it (−0.2), thereby encouraging reward hacking? What empirical evidence addresses this concern?

A3: Thank you for this careful question. The −0.2 reward goes to an unsuccessful revision, not to a successful correction. The table below lists the total reward for each outcome under Eq. (12) and the coefficients in Appendix C.3.

Initial answerOutcomeFormatFinal acc.RevisionTotal
IncorrectCorrected by revision (F→T)0.11.0−0.11.0
IncorrectAccepted without revision0.1000.1
IncorrectRevised but still incorrect (F→F)0.10−0.3−0.2
CorrectAccepted without revision0.11.001.1
CorrectUnnecessary revision (T→T)0.11.0−0.50.6
CorrectHarmful revision (T→F)0.10−0.5−0.4

Our reward design prioritizes successful correction while accounting for the cost of ineffective revision. For an incorrect initial answer, successful correction earns the highest reward. Revisions without correctness gain (F→F and T→T) are penalized because they add computation without improving the answer.

Empirical check for reward hacking. We test whether this penalty discourages revision in practice by comparing how SCR-Stage I and SCR handle incorrect initial answers on MATH500 with Qwen2.5-7B-Instruct. Percentages are computed over each model's incorrect initial answers.

ModelFalsely acceptedSent to revisionSuccessfully repaired
SCR-Stage I36.9%63.1%6.9%
SCR25.0%75.0%14.9%

After Stage II, the model sends more of its errors to revision (63.1% → 75.0%) and more than doubles the repair rate (6.9% → 14.9%). These results have been added to Appendix E.2.

Q4: Please clarify the undefined symbols in the theoretical analysis.

A4: Thank you for this helpful suggestion. We have added a notation table in Appendix A.1 summarizing the key symbols in the theoretical analysis, and clarified key definitions in Section 2 to make the analysis easier to follow.

Q5: Why not evaluate SCR on modern o1/R1-style reasoning models, whose longer, more complex reasoning may better test atomic abilities?

A5: Thank you for this question. The prevailing approach to improving reasoning, represented by DeepSeek-R1, starts from a non-reasoning backbone and applies long-CoT SFT followed by RL (i.e., SFT+GRPO in Table 1). We argue that long CoT cannot effectively measure a model's individual atomic reasoning abilities (Figure 2). This observation comes from our pilot analysis of four o1/R1-style reasoning models. We therefore adopt structured reasoning to optimize these atomic abilities, which in turn improves overall reasoning performance. To ensure a fair comparison with this mainstream approach, we use the same non-reasoning backbones rather than o1/R1-style reasoning models.

Reviewer_TpD8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structured Reasoning (SCR), a framework that decomposes reasoning trajectories into explicit functional units to enable ability-specific supervision. Motivated by the limitations of outcome-based supervision and the inefficiency of existing self-verification and self-correction behaviors, the authors instantiate SCR as Generate-Verify-Revise (GVR). The proposed training procedure combines selective supervised fine-tuning with staged reinforcement learning to improve generation, verification, and revision capabilities. Experiments across three backbone models demonstrate improvements in reasoning accuracy over several baselines.

Critical strengths
  • The paper addresses an important and practical problem. Improving the reliability and effectiveness of reasoning processes, particularly through better verification and correction capabilities, is highly relevant to advancing large language models.
  • The authors present a relatively complete solution, combining theoretical analysis, structured reasoning representations, and ability-specific training strategies.
  • The paper provides experiments across multiple backbone models and reasoning benchmarks, along with additional analyses supporting the effectiveness of the proposed approach.
Critical weaknesses
  • The relationship between SCR and existing self-verification and self-correction methods could be discussed more thoroughly, particularly S2R (Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning). Both works address highly similar problems and share several technical ideas. However, the manuscript does not sufficiently clarify the fundamental methodological differences or the unique contributions of SCR relative to S²R. A more detailed technical comparison, ideally supported by direct experimental results, would help establish the novelty and effectiveness of the proposed approach.

  • The theoretical analysis in Section 2 provides useful insights and helps motivate the overall framework. However, its relationship to existing theoretical studies on process-based supervision is not sufficiently discussed. For example, ``Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective'' investigates closely related questions regarding the theoretical relationship between these supervision paradigms. A more thorough comparison would help clarify which insights are already established and what new theoretical understanding is contributed by this work.

Questions for authors

My main concerns are related to the novelty of the proposed method and its differentiation from existing research, as discussed above. I would appreciate further clarification and, where appropriate, additional comparisons addressing these points. I am open to raising my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TpD8 (1/2)

Q1: What are SCR's unique contributions beyond S²R? Please support the distinction with direct experimental comparisons.

A1: We thank the reviewer for the careful comment. SCR shares some common self-correction components, but it differs in research focus and training objectives. Below we (1) clarify the technical differences, (2) report a direct experimental comparison, and (3) summarize the revisions.

(1) Differences from existing self-correction methods. Research focus. SCR is a structured reasoning method rather than a self-correction method. It targets the mainstream paradigm of long-CoT SFT followed by outcome-reward RL (e.g., DeepSeek-R1), which supervises interleaved reasoning abilities only through the final answer. SCR instead decomposes reasoning into explicit units and evaluates and trains each unit separately. GVR is one instantiation, with its units selected by the trajectory analysis in Section 3.1.

Training objectives. Even as an instantiation, GVR differs from existing methods in its training objectives. S²R [1] re-solves the problem when verification fails and rewards each attempt only by its own correctness, without penalizing additional attempts. GVR revises the previous solution, is rewarded by the resulting change in correctness, and is penalized for unnecessary revisions. SCoRe [2] always answers twice without explicit verification and optimizes the second attempt first. GVR revises only when its own verification flags an error, and reverses the stage order: it trains generation and verification before revision. CURE [3] jointly trains solving, critiquing, and re-solving without modeling the cost of revision.

(2) Comparison with self-correction baselines. We add S²R and CURE as baselines. S²R-RL and CURE are trained on the same DAPO-MATH pool as SCR for 300 RL steps, and each method follows its original initialization: S²R-SFT is trained on the official S²R SFT data, S²R-RL continues from S²R-SFT, CURE starts directly from the Instruct model, and SCR starts from SCR-SFT. As shown below, SCR achieves the highest average accuracy on all three backbones, with the largest margins on GPQA and ARC. Moreover, CURE's average is lower than Base on both Qwen backbones and only marginally higher on Llama3.1-8B (24.71% → 24.81%), so adding self-correction does not consistently improve overall accuracy.

MethodMATH500AIME24AIME25OlympiadAMCARCMMLUGPQAAVG
Qwen2.5-3B-Instruct
Base66.607.002.3329.9327.9569.2039.6029.8034.05
S²R-SFT60.402.002.3328.0427.3577.8253.2912.1232.92
S²R-RL68.406.005.3337.9837.3578.8455.3022.2238.93
CURE65.204.673.3330.4231.4566.3047.969.0932.30
SCR (Ours)70.407.334.0037.0336.0279.3544.5039.3939.75
Qwen2.5-7B-Instruct
Base75.6011.006.0039.7042.0584.3955.8033.8443.55
S²R-SFT75.8011.334.6741.3944.7083.7955.4026.7742.98
S²R-RL79.8015.337.6747.3350.7284.8157.4232.8346.99
CURE78.2013.009.6742.4345.5461.5248.2714.1439.10
SCR (Ours)80.0013.6713.6745.6350.1286.4356.3038.3848.03
Llama3.1-8B-Instruct
Base42.802.000.0015.8512.7757.0842.4024.7524.71
S²R-SFT55.005.670.6719.5822.0565.7043.9410.1027.84
S²R-RL61.009.673.6727.6030.0067.6646.9618.1833.09
CURE30.601.330.006.6813.1373.9856.0616.6724.81
SCR (Ours)60.805.334.6723.7024.4684.5643.2027.2734.25

(3) Summary of revisions. All changes are highlighted in dark rose and stay within the page limit:

  • Section 4.1: added S²R and CURE as baselines.

  • Table 1 and Section 4.2: added their results and the corresponding analysis.

  • Appendix B: added the reproduction details, training data, and key hyperparameters of S²R and CURE.

[1] S²R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning

[2] SCoRe: Self-Correction for Reasoning in Large Language Models

[3] CURE: Critical Reasoning for Enhanced Reasoning in Large Language Models

AuthorsOpenReview ↗

Response to Reviewer TpD8 (2/2)

Q2: What new theoretical insights does SCR add beyond prior work on process supervision, particularly "Do We Need to Verify Step by Step"?

A2: We thank the reviewer for the pointer. Jia et al. and our work address different questions. Jia et al. ask whether outcome supervision can learn a high-return policy as efficiently as process supervision. We ask whether outcomes reveal the quality of a specific local ability, and when improving that ability raises accuracy.

Part of our analysis builds on established results. Jia et al. show that outcome supervision is statistically no harder than process supervision under standard coverage assumptions, and that the advantage function can serve as an optimal process reward. Our Prop. 2 is likewise a performance-difference decomposition, and we do not claim it as a new principle.

Our new insights are twofold. First, outcomes identify local quality if and only if the ability label is affine in continuation success (Prop. 3). The advantage meets this condition, which is why Jia et al. can recover process signals from outcomes. Verification correctness generally does not, since equally correct verdicts with different critiques lead to different repair rates. The two results are therefore complementary. Second, higher local quality does not guarantee higher accuracy, because its value depends on how the ability is used downstream. In GVR, the accuracy gain (1−pg)ρpr−pgεqh(1-p_g)\rho p_r-p_g\varepsilon q_h shows that error detection pays off only when followed by successful revision, which motivates Stage II.

We have revised Section 2 to state this identifiability condition explicitly, and we now cite Jia et al. and the performance difference lemma.

AuthorsOpenReview ↗

Sincere Invitation to Discussion (7h Remaining)

Dear Reviewer TpD8,

Thank you for your valuable feedback. We have worked carefully to address both points in our previous responses. As there are only 7 hours left before the discussion deadline, we would be very grateful if you could let us know whether our previous responses have satisfactorily addressed your concerns, or if any questions remain.

We would be honored and grateful to hear your thoughts.

Warm regards,

All authors

Reviewer_ErTjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Structured Reasoning (SCR) splits a reasoning trajectory into explicit units supervised by a checker. For math, Generate–Verify–Revise (GVR) uses a single policy to solve, verify, and revise only on a negative verdict. Training combines masked SFT, verdict-dependent stopping, and two GRPO stages. The theory shows why outcome supervision can miss local quality. Across backbones, average accuracy beats SFT+GRPO by 0.77–1.31 points, and Qwen2.5-3B outputs are 18–49% shorter on four math benchmarks.

Critical strengths
  • The accuracy–length trade-off is real: Table 1 shows gains over both GRPO baselines across backbones, and Figure 4 shows 18–49% shorter outputs on Qwen2.5-3B.

  • The theory is honestly scoped: Theorem 1 and Proposition 2 show the limits of outcome supervision without claiming to justify the training order.

Critical weaknesses
  • My main worry is novelty. S²R (Ma et al., 2025, §2.1–2.3) already combines solve/verify actions, verdict-based stopping, masked imitation, and action-level correctness rewards, and SCoRe already uses staged RL with correctness-transition shaping, though the objectives differ. Section 5's distinction—explicit units plus targeted supervision—doesn't separate SCR from S²R by itself. The schedule and revision penalties may help, but the experiments don't isolate their contributions.

  • The mechanism is unclear: Appendix B gives 200 GRPO steps but no per-stage split, and Figure 6's outcome-only control is never shown to match initialization or total budget. Stage I vs. full SCR changes duration and objective, and Figure 5 only ablates SFT. Table 2 lacks final-model verification and repair/corruption stats, and E.3 concedes a correct verdict need not be a useful critique. Gains are small; some tasks regress against SFT+GRPO (7B MMLU 56.30 vs. 57.80; Llama AMC 24.46 vs. 28.67), and variance is unreported.

Questions for authors
  • What distinguishes SCR from S²R with matched backbones, initialization, and budget? Does Figure 6's outcome-only control match initialization and total steps, and would a joint-training or Stage-II-only run isolate the schedule?

  • Can the authors report initial/final accuracy and repair/corruption rates for Stage I and final SCR, by round? Proposition 10 covers one round; how is the second round started, stopped, and supervised?

  • For Llama Stage-I AMC, the Table 2 metrics imply ~83.7% correct initial answers if computed from a pooled confusion matrix. Under identical samples and correct verdict-based stopping, correct acceptances alone would imply at least ~78.5% final accuracy, versus 26.51% in Table 1. The Base row gives 12.70%, close to Table 1's 12.77%. Are different evaluation pools or protocols used?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ErTj (1/3)

Q1: How does SCR differ from S²R and SCoRe under matched initialization and budgets? Please isolate the effects of staged training and revision penalties using suitable controls.

A1: We thank the reviewer for this comment. SCR shares some components with existing self-correction methods, but it differs in both research focus and training objectives. Below, we (1) clarify these differences, (2) compare SCR directly with these methods, (3) isolate the effects of staged training and the revision penalty, and (4) summarize our revisions.

(1) Differences from existing self-correction methods. Research focus. SCR is a structured reasoning method rather than a self-correction method. It targets the mainstream paradigm of long-CoT SFT followed by outcome-reward RL (e.g., DeepSeek-R1), which supervises interleaved reasoning abilities only through the final answer. SCR instead decomposes reasoning into explicit units and evaluates and trains each unit separately. GVR is one instantiation, with its units selected by the trajectory analysis in Section 3.1.

Training objectives. Even as an instantiation, GVR differs from existing methods in its training objectives. S²R [1] re-solves the problem when verification fails and rewards each attempt only by its own correctness, without penalizing additional attempts. GVR revises the previous solution, is rewarded by the resulting change in correctness, and is penalized for unnecessary revisions. SCoRe [2] always answers twice without explicit verification and optimizes the second attempt first. GVR revises only when its own verification flags an error, and reverses the stage order: it trains generation and verification before revision. CURE [3] jointly trains solving, critiquing, and re-solving without modeling the cost of revision.

(2) Comparison with self-correction baselines. We add S²R and CURE as baselines. S²R-RL and CURE are trained on the same DAPO-MATH pool as SCR for 300 RL steps, and each method follows its original initialization: S²R-SFT is trained on the official S²R SFT data, S²R-RL continues from S²R-SFT, CURE starts directly from the Instruct model, and SCR starts from SCR-SFT. As shown below, SCR achieves the highest average accuracy on all three backbones, with the largest margins on GPQA and ARC. Moreover, CURE's average is lower than Base on both Qwen backbones and only marginally higher on Llama3.1-8B (24.71% → 24.81%), so adding self-correction does not consistently improve overall accuracy.

MethodMATH500AIME24AIME25OlympiadAMCARCMMLUGPQAAVG
Qwen2.5-3B-Instruct
Base66.607.002.3329.9327.9569.2039.6029.8034.05
S²R-SFT60.402.002.3328.0427.3577.8253.2912.1232.92
S²R-RL68.406.005.3337.9837.3578.8455.3022.2238.93
CURE65.204.673.3330.4231.4566.3047.969.0932.30
SCR (Ours)70.407.334.0037.0336.0279.3544.5039.3939.75
Qwen2.5-7B-Instruct
Base75.6011.006.0039.7042.0584.3955.8033.8443.55
S²R-SFT75.8011.334.6741.3944.7083.7955.4026.7742.98
S²R-RL79.8015.337.6747.3350.7284.8157.4232.8346.99
CURE78.2013.009.6742.4345.5461.5248.2714.1439.10
SCR (Ours)80.0013.6713.6745.6350.1286.4356.3038.3848.03
Llama3.1-8B-Instruct
Base42.802.000.0015.8512.7757.0842.4024.7524.71
S²R-SFT55.005.670.6719.5822.0565.7043.9410.1027.84
S²R-RL61.009.673.6727.6030.0067.6646.9618.1833.09
CURE30.601.330.006.6813.1373.9856.0616.6724.81
SCR (Ours)60.805.334.6723.7024.4684.5643.2027.2734.25

[1] S²R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning

[2] SCoRe: Self-Correction for Reasoning in Large Language Models

[3] CURE: Critical Reasoning for Enhanced Reasoning in Large Language Models

AuthorsOpenReview ↗

Response to Reviewer ErTj (2/3)

(3) Isolating staged training and the revision penalty. Staged training. Figure 6 shows that SCR outperforms outcome-only RL, but that comparison changes the rewards and the schedule together. To isolate the schedule, we add a Mix baseline: it starts from the same SCR-SFT checkpoint and optimizes the rewards of both stages in a single stage for the same total number of steps. Full SCR outperforms Mix on all four benchmarks for both backbones (accuracy, %):

ModelMethodMATHAIME25OlympiadAMC
Qwen2.5-7B-InstructSCR-Stage I77.009.0042.0744.46
Mix76.009.0040.4448.20
SCR80.0013.6745.6350.12
Qwen2.5-3B-InstructSCR-Stage I65.202.0029.0429.16
Mix60.003.3332.0034.34
SCR70.404.0037.0336.02

Revision penalty. Starting from the same Stage I checkpoint, we remove the revision penalty (γ′=0) and keep all other settings unchanged. Average accuracy drops from 30.96% to 29.02%, and increasing the weight to γ′=2 also lowers accuracy. The penalty therefore helps, but its weight must be balanced against the final-answer reward.

SettingMATH500AIME24AIME25OlympiadAMCAVG
γ′=0 (new)64.006.674.3334.5035.6029.02
γ′=1 (default)70.407.334.0037.0336.0230.96
γ′=268.207.674.0032.3037.1129.86

(4) Summary of revisions. All changes are highlighted in dark rose and stay within the page limit:

  • Section 4.1: added S²R and CURE as baselines.

  • Table 1 and Section 4.2: added the results of S²R and CURE and the corresponding analysis.

  • Section 4.5: added the Mix baseline to isolate the effect of staged training.

  • Appendix B: added the training data and key hyperparameters of S²R and CURE.

  • Appendix E.1 and Table 4: added the γ′=0 ablation and its analysis.

Q2: Please clarify stage-wise budgets and matched controls, report initial/final accuracy, and repair/corruption rates by round, and explain how the second round starts, stops, and is supervised. Do correct verdicts yield useful critiques, and what variance estimates support the small gains and task regressions?

A2: Thank you for the detailed comments. We address each point below.

(1) Per-stage budget and the outcome-only control. Stage I runs 100 GRPO steps, and Stage II continues from the Stage I checkpoint for 200 steps (300 in total). The outcome-only control in Figure 6 matches SCR in initialization and budget. It starts from the same SCR-SFT checkpoint and trains for 300 steps with the same number of rollouts, batch size, and learning rate. It differs from SCR only in its reward, which contains just the format and final-answer terms.

(2) Stage I vs. full SCR. The gain of full SCR comes from the Stage II objective, not from longer training. We extend Stage I to 300 steps on Qwen2.5-3B-Instruct to match the total budget of full SCR. Longer Stage I training barely helps (25.14 → 25.46), while SCR reaches 29.69 under the same budget.

Model (Qwen2.5-3B-Instruct)StepsMATHAIME24AIME25OlympiadAvg.
SCR-Stage I10065.204.332.0029.0425.14
SCR-Stage I30066.004.333.3328.1925.46
SCR100 + 20070.407.334.0037.0329.69

(3) RL-stage ablation beyond Figure 5. Beyond the outcome-only control in Figure 6, the Mix baseline (A1-(3)) keeps the initialization, rewards, and total steps fixed and only merges the two stages into one. SCR outperforms Mix on all four benchmarks for both backbones, so the staged schedule itself contributes to the gain.

AuthorsOpenReview ↗

Response to Reviewer ErTj (3/3)

(4) Corruption statistics of the final model. We track how correctness changes from the initial to the final answer on MATH500 with Qwen2.5-7B-Instruct (%). Stage II turns revision from harmful to beneficial. With Stage I alone, revision corrupts more correct answers than it repairs, so final accuracy falls below initial accuracy. Stage II nearly eliminates corruption (5.1% → 0.8%) and doubles the repair rate, so revision now improves accuracy (Appendix E.3).

ModelT→TT→FF→TF→FInitial Acc.Final Acc.Repair RateCorruption Rate
SCR-Stage I75.64.11.418.979.777.06.95.1
SCR76.60.63.419.477.280.014.90.8

(5) Correct verdicts and effective revision. We examine whether correctly detected errors are subsequently repaired (MATH500, Qwen2.5-7B-Instruct). The first two columns use each model's incorrect initial answers as the denominator; the last is the repair rate among those sent to revision (%).

ModelFalsely acceptedSent to revisionRepaired / sent to revision
SCR-Stage I36.963.110.9
SCR25.075.019.9

Among correctly flagged errors, the repair rate nearly doubles (10.9% → 19.9%). Stage II therefore improves repair after error detection, although this does not directly establish critique quality or whether revision uses the critique, which is the limitation discussed in Appendix E.5 (originally E.3). Details are in Appendices E.2–E.3.

(6) Small gains and regressions against SFT+GRPO. SCR does not win on every task, but it wins 16 of 24 comparisons against SFT+GRPO (1 tie) and achieves a higher average on all three backbones. Following the standard protocol, AIME24/25 and AMC report the mean accuracy over 10 samples, and the other benchmarks use greedy decoding.

(7) Summary of revisions. All changes are highlighted in dark rose:

  • Appendix B: added the per-stage training steps.

  • Section 4.5 and Figure 6: added the Mix comparison and clarified the matched settings of both RL baselines.

  • Appendix E.3 and Table 5: added correctness transitions, repair/corruption rates, and the repair rate among correctly flagged errors.

Q3: Why do the Llama Stage-I AMC metrics in Table 2 imply at least 78.5% final accuracy, while Table 1 reports 26.51%? Are the evaluation pools or protocols different?

A3: Thank you for the careful check. We have corrected the Llama Stage-I AMC entry in Table 2 to Acc. 82.91 / Pre. 72.02 / Rec. 67.80 / F1 69.85. The evaluation pools also differ. Table 2 measures self-verification only on responses whose initial answer and verdict can be parsed, whereas Table 1 reports accuracy over all samples in each benchmark. We now state this in the caption of Table 2. We have also re-checked the other entries in Table 2 and confirmed that all values in the revised version are correct.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their constructive feedback. A few days ago, we posted point-by-point responses and a revised manuscript but have not yet received any follow-up. As the discussion period is limited, we summarize them below and would greatly appreciate knowing whether they address your concerns.

A. Strengths. Reviewers noted the following strengths.

  • The paper addresses an important problem in reliable reasoning, since long-CoT training does not separate reasoning abilities (TpD8, 6LKy).
  • SCR is a coherent framework combining structured representations, ability-specific training, and carefully scoped theory (TpD8, ErTj).
  • Experiments show improved accuracy and efficiency across backbones, with 18–49% shorter outputs on Qwen2.5-3B (TpD8, ErTj).

B. Concerns and revisions. All revisions are highlighted in dark rose.

ReviewerMain concernResponse and revisions
ErTjW1: Novelty vs. S²R/SCoRe; effects of staged training and revision penaltySCR is a structured reasoning framework rather than a self-correction method. Its GVR instantiation gates revision on verification and penalizes unnecessary revisions. We added S²R and CURE as new baselines. Under the same data and budget, SCR outperforms both on average across all three backbones. New Mix and penalty ablations further support both designs. These appear in Sections 4.1–4.2 and Table 1, Section 4.5, and Appendix E.1.
W2: Unclear budgets and mechanism; missing final-model statistics; small gains and regressionsStages I and II run 100 and 200 steps, matching the outcome-only control's budget. On MATH500 with Qwen2.5-7B, Stage II reduces corruption from 5.1% to 0.8% and doubles repair (6.9% → 14.9%). We acknowledge task regressions and the critique-quality limitation. We clarified the budgets in Appendix B and added transition statistics to Appendix E.3.
Q3: Inconsistent Llama Stage-I AMC metrics in Tables 1 and 2The tables use different evaluation pools: Table 2 measures verification on parsable responses, while Table 1 evaluates all samples. We corrected the Llama Stage-I AMC entry in Table 2 and revised its caption to make this distinction explicit.
TpD8W1: Unclear differentiation from S²R; no direct comparisonSCR is a structured reasoning framework rather than a self-correction method. Its GVR instantiation gates revision on verification and penalizes unnecessary revisions. We added S²R and CURE as new baselines. Under the same data and budget, SCR outperforms both on average across all three backbones. Results and reproduction details are in Sections 4.1–4.2, Table 1, and Appendix B.
W2: Theoretical contribution beyond prior process-supervision theoryJia et al. compare the efficiency of outcome and process supervision. We ask whether outcomes reveal a specific ability's quality and when improving it raises accuracy. Error detection helps only when followed by successful revision, which motivates Stage II. We revised Section 2 to state the identifiability condition and cite Jia et al.
6LKyW1: Coarse stages and shared rewards do not show ability-specific optimizationEach reward term evaluates one ability, and token masking restricts each stage to its target abilities, with revision trained only in Stage II. Fully independent signals need exponentially growing nested sampling (8 → 64 → 512 rollouts), so staged training balances separation and cost. SCR outperforms single-stage Mix on all four benchmarks for both backbones. We added this comparison to Section 4.5 and Figure 6.
W2: Unclear distinction from SCoRe/CURE; no comparisonSCR is a structured reasoning framework rather than a self-correction method. Its GVR instantiation gates revision on verification and penalizes unnecessary revisions. Under the same data and budget, SCR outperforms S²R and CURE in average accuracy on all three backbones. We added these baselines to Sections 4.1–4.2 and Table 1.
W3: Stage-II rewards may encourage accepting errorsThe −0.2 reward applies to unsuccessful revision, while successful correction earns the highest reward (1.0). Stage II also sends more errors to revision (63.1% → 75.0%) on MATH500. We added this analysis to Appendix E.2.
Q1: Undefined theoretical symbolsWe added a notation table to Appendix A.1 and clarified key definitions in Section 2.
Q2: No experiments on o1/R1-style modelsOur pilot study analyzes four o1/R1-style models (Section 3.1). For a fair comparison with long-CoT SFT + GRPO, both use the same non-reasoning backbones.

We truly appreciate your time and effort. We believe these revisions have substantially strengthened the paper, and we respectfully ask the reviewers and AC to reconsider their assessment. If any of our responses are unclear or any concerns remain, we would be glad to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.