acceptodds
Under review as a conference paper at ICLR 2027

Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLM

Abstract

Human beings solve complex problems through critical thinking, where reasoning and evaluation are intertwined to converge toward correct solutions. However, most existing large language models (LLMs) treat the reasoning and verification as separate processes: they either generate reasoning without explicit self-checking or rely on external verifiers to detect errors post hoc. The former lacks immediate feedback, while the latter increases system complexity. Motivated by human critical thinking, we propose Stepwise Think-Critique (STC), an end-to-end trainable framework in which a single LLM emits a structured, step-level critique inline with each reasoning step. STC is trained with reinforcement learning that complements reasoning rewards with a critique-consistency reward derived from final-answer correctness, jointly optimizing reasoning correctness and critique reliability. On five mathematical reasoning benchmarks, STC retains the reasoning gains of standard RL (+7.2 Pass@1 points over its 1.5B base model) while substantially improving its step-level self-critique (F1 from 46.4% to 67.4%). In judging its own reasoning steps, it on average outperforms seven dedicated process reward models of up to 8B parameters, without any threshold tuning. STC takes a step toward LLMs with built-in critical thinking.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, reasoning, self-critique, interpretable reasoning, reinforcement learning, chain-of-thought, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VoY1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Stepwise Think-Critique (STC), a framework in which a single model alternates between reasoning steps and self-critiques. Each critique includes an explanation and a binary correctness label. The model is first fine-tuned on synthetic interleaved reasoning–critique data, then trained with GRPO using answer correctness, critique consistency, and format rewards. The critique reward is derived from final-answer correctness and applied only to critique tokens. Experiments on mathematical reasoning benchmarks show improved self-critique performance while maintaining reasoning accuracy comparable to the variant trained without the critique reward.

Critical strengths
  1. STC integrates reasoning and step-level critique into a single model, providing explicit correctness judgments and explanations without a separate verifier.

  2. The ablations show that the critique reward improves self-evaluation while largely preserving reasoning accuracy. They help distinguish the contributions of the reasoning and critique objectives.

  3. The comparison with seven external PRMs uses the same reasoning trajectories and allows per-dataset oracle thresholds for the PRMs. This provides a useful assessment of STC’s self-critique performance.

Critical weaknesses
  1. The main evidence for step-level critique quality relies on GPT-5-mini labels rather than human annotations. The reported agreement with ground truth is measured at the final-answer level, which does not establish the reliability of its judgments on intermediate steps.

  2. The PRM comparison uses only STC-generated trajectories, with 30 problems per benchmark. Evaluation on independently generated trajectories or an established process-verification benchmark would help assess whether the reported advantage extends beyond STC’s own outputs.

  3. The practical benefit of improved critique remains unclear. Adding the critique reward does not improve reasoning accuracy over the ablated model, and the paper does not demonstrate that the critiques reliably support correction or answer selection. Experiments are also limited to a single 1.5B backbone.

Questions for authors
  1. Could you validate the intermediate-step labels on a human-annotated subset? Agreement on final-answer correctness may not reflect the judge’s reliability on individual reasoning steps.

  2. How does STC perform on trajectories generated by other models? This would clarify whether its advantage over external PRMs extends beyond its own outputs.

  3. Can STC’s critiques improve error correction or answer selection? Such an experiment would help establish their practical value beyond the reported critique metrics.

Reviewer_WavHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Stepwise Think-Critique (STC), a framework in which one LLM alternates between generating a reasoning step and generating a critique of that step. Each critique contains a natural-language justification and a binary correctness score. The method first uses SFT on synthetic interleaved reasoning-critique traces, then applies GRPO with three rewards: answer correctness, critique consistency, and output format. A key design is that the critique advantage is masked to critique-token positions. Experiments on five mathematical reasoning benchmarks show that STC improves reasoning performance over the 1.5B base model and improves process-level critique F1 from 46.4% to 67.4%. The paper also compares STC with several external PRMs and reports higher average step-level F1 without threshold calibration.

Critical strengths
  • The framework is technically coherent, and the separation between reasoning reward, critique-consistency reward, and format reward is well motivated. The reward design and token-masking mechanism are simple and easy to understand.

  • The ablation study is useful: it clearly shows that reasoning improvement mostly comes from the reasoning reward, while the critique reward mainly improves error detection.

Critical weaknesses
  1. The RL critique reward directly supervises only the final critique against final-answer correctness. It does not directly supervise whether each intermediate critique is correct.
  2. Process-level critique evaluation relies on GPT-5-mini pseudo-labels. Agreement at the final-answer level does not fully validate fine-grained intermediate-step labels.
  3. The comparison with external PRMs is difficult to interpret because STC uses its own newline-delimited atomic steps, while external PRMs are often trained at different step granularities.
  4. A matched pure-reasoning RL baseline would be needed to isolate whether the reasoning gain is due to RL in general or to the interleaved STC formulation.
Questions for authors
  1. Can the authors evaluate on a benchmark with human step-level annotations, or manually annotate a representative subset of trajectories?
  2. Can the authors compare against a pure reasoning-only RL baseline with matched data, compute, and training steps?
  3. Can the authors compare final-answer-based critique reward with direct intermediate-step reward, at least on a small labeled subset?
Reviewer_ySJROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explicitly adds critiques into reasoning trajectories during LLM generation tasks, resulting in interleaved reasoning and critique at the step level. Staring with some annotated data, the base model first learns the interleaved reasoning and critique style. Then the authors design a joint learning reward for both reasoning and critique in the framework of GRPO. Experiments with Qwen 1.5B model, the authors show that their approach performs better than SFT baseline.

Critical strengths
  1. The interleaved reasoning and critique at the step level looks interesting, and we are able to judge the quality of each reasoning step.
Critical weaknesses
  1. The annotations are not well-designed, for example, the t denotes the number of steps, but latter it was used as t-th token.

  2. In experiments, STC (compact) and STC (full) are not mentioned at all. It’s really hard to know what to compare.

  3. Experiments are not well-designed. See the section of Questions For Authors. Experiments are only done with a small model, which is Qwen 1.5B model.

Questions for authors
  1. In experiments, STC (compact) and STC (full) are not mentioned at all. It’s really hard to know what to compare.

  2. The optimization in Section 3.4.2 is not clear to me. For each step, you do have critic prediction c_t, but you only use the final step R_crit(c_T) in reward, then do you just simply distribute the final critique reward to all critic tokens? Then I don’t get the point of having c_t for each step, all critic tokens are assigned with the same reward no matter this step is correct or wrong.

  3. Experiments in Table 1, I don’t see significant differences among all STC RL varieties on P@1. And I’m not convinced the results of Table 2, we need run a baseline with RL over SFT model with reasoning CoT only.

  4. I’m not convinced that the step-level self-critique improves F1 from 46.4% to 67.4%, as your step-level critique is trained on those critique labeled data, which will definitely learn the behavior of the labeling model, I’m also thinking it is possible that the STC model may overfit to the critique style and may have reward hacking issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.