acceptodds
Under review as a conference paper at ICLR 2027

ParadigmCoT: Paradigm-Guided Chain-of-Thought in Large Language Models via Policy Optimization

Abstract

Chain-of-Thought (CoT) prompting may exhibit reasoning-path drift, logical inconsistency, and fabricated explanations, limiting the reliability of LLM reasoning. While structured prompting and process supervision improve local correctness and factuality, they provide limited holistic, principle-level guidance for the overall reasoning trajectory. Drawing on established instructive paradigms in both general and domain-specific fields, we propose ParadigmCoT: a general and flexible framework that injects instructive paradigms into the CoT process via policy optimization. At its core is an automated paradigm localization and evaluation module, which identifies paradigm-triggering steps and evaluates step-wise adherence throughout the reasoning chain. These paradigm alignment signals, combined with factuality and answer correctness, are incorporated into a fine-grained advantage adjustment in reinforcement learning to promote paradigm-aligned reasoning. ParadigmCoT is instantiated with general and medical paradigms, and evaluate it across diverse benchmarks and three LLM backbones. It achieves substantial gains over base models (up to 40.2% on general tasks and 34.5% on medical tasks), and outperforms RL-based baselines by 2.5%–3.4% and 5.8%–7.7% on average, respectively. This framework offers a new perspective on improving the quality and reliability of LLM-generated CoT, contributing to more reliable AI.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Chain-of-Thought, Large Language Models, Instructive Paradigm

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_D4ebOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ParadigmCoT, a reinforcement learning framework that guides chain-of-thought reasoning through general and medical reasoning paradigms. It combines localized paradigm verification with factuality and outcome signals to adjust token-level learning advantages. Experiments across three language models demonstrate improvements on mathematical, hallucination, and medical benchmarks.

Critical strengths
  1. This paper investigates a limitation of traditional chain-of-thought reasoning, lacking principled guidance throughout the reasoning process. It introduces established reasoning paradigms as process-level supervision to improve the reasoning capabilities of language models.
  2. This paper conducts experiments on mathematical reasoning, hallucination-related benchmarks, and medical reasoning tasks using three different language model backbones. Comparisons with structured prompting methods and representative reinforcement learning baselines demonstrate the effectiveness of the proposed method.
Critical weaknesses
  1. The proposed ParadigmCoT primarily combines established techniques, including process-level reward, semantic verification, and GRPO-based policy optimization. Although their integration is practically meaningful, the individual components offer limited methodological novelty.
  2. The paper emphasizes holistic guidance over the entire reasoning trajectory. However, the proposed evaluation module only examines a fixed window of subsequent reasoning steps, with q=4. Such a local evaluation may overlook long-range logical dependencies and inconsistencies, weakening the claim that the framework provides global guidance throughout the reasoning process.
Questions for authors
  1. When multiple paradigm triggers appear in the same sentence, the current implementation uses a first-match rule. Could the authors explain whether this design introduces information loss and investigate whether allowing multiple paradigms to jointly contribute to the reward would improve performance?
  2. The framework operates on sentence-level reasoning units, but language model outputs may contain mathematical expressions, incomplete sentences, or multiple logical operations within a single sentence. Could the authors clarify the segmentation process and explain how sentence-level rewards are mapped to the corresponding tokens during policy optimization?
  3. The first term of the minimum operator uses GRPO advantages, while the clipped term uses adjusted advantages. Explain the optimization implications of this asymmetric design?
Reviewer_QyGaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ParadigmCoT, which uses instructive paradigms to guide the whole CoT process, achieving holistic, principled reasoning. These paradigm alignment signals are incorporated into a fine-grained advantage adjustment in reinforcement learning to promote paradigm-aligned reasoning. It achieves gains over base models and outperforms RL-based baselines across several benchmarks and three LLM backbones.

Critical strengths
  1. The proposed ParadigmCoT improve the performance on several benchmarks.
  2. The presentation of the paper is clear and easy-to-follow.
Critical weaknesses
  1. The motivation is unsupported. This paper would like to use instructive paradigms to guide the whole CoT process. Did those paradigms target the weakness shown in the CoT process of current models? Experiments results or literatures should be added for this point.
  2. The Paradigm-triggering machanism is naive and weak. The current version is using key-words match such as "because", "but" to split the sentences and key points in the CoT, which isn't reliable. As lite weight LLM can complete this task both effectively and cost-efficiently.
  3. The choose of the Paradigms is casual and incomplete. The author should clarify the reason why Attributional Reasoning, Causal Mediated Reasoning, and Counterfactual Reasoning are used as the paradigms but not other paradigms.
  4. How to score those dimensions is still weak and unclear. HHEM-2.1 is used for computing the scores, which is the key source of the supervision signals. However, the validity of this method has not been evaluated and compared with other methods. The reason why HHEM-2.1 is used other than simple LLM-as-a-Judge with more recent and powerful LMs is unclear.
Questions for authors
  1. What's the principles for choosing and designing those Paradigms?
  2. How can the improvements on benchmarks be traced back to those specific Paradigms?
Reviewer_rG2pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ParadigmCoT, an RL framework with an additional paradigm reward based on trigger words and a factuality reward based on whether the reasoning is supported by the question, motivated by the fact that chain-of-thought reasoning is sometimes unclear and inconsistent.

Critical strengths
  1. The motivation is justified. The reasoning process requires further refinement to maintain consistency throughout.

  2. The evaluation is rather broad.

Critical weaknesses
  1. The method is very limited in novelty. Keyword-based RL rewards are already established in [1], which uses reasoning markers (“based on,” “since,” “suppose”) that closely mirror the paper’s trigger words. Additionally, keyword pools are common for monitoring reasoning behavior during RL training. For example, [3] tracks words such as “check,” “verify,” and “however” during training. Also, the factuality reward is directly from FSPO [2], and [1] already rewards alignment between the reasoning and the question. Overall, the method is mostly a combination of existing method and the novelty is minimal.

[1] Jiajun Fan, Roger Ren, Jingyuan Li, Rahul Pandey, Prashanth Gurunath Shivakumar, Ivan Bulyko, Ankur Gandhe, Ge Liu, and Yile Gu. Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards. arXiv:2510.20867, 2025.

[2] Junyi Li and Hwee Tou Ng. Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models. In NeurIPS, 2025.

[3] Mingyang Song and Mao Zheng. Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning. arXiv:2505.21178, 2025.

Questions for authors
  1. The paper is motivated by reasoning drift, but there are no metrics showing how severe it actually is in the base model or how much the method reduces it. Showing only accuracy isn't convincing.

  2. Why pick these six specific paradigms? Also, how does this method compare to simple prompt engineering?

  3. HHEM-2.1 just checks whether one text segment is consistent with another. Why would consistency between consecutive sentences prove that a paradigm was followed?

  4. Why does the advantage table give a positive update to some wrong answers (+0.8 for “paradigm persistence”) and a negative update to some correct ones (−0.8)?

  5. Since the triggers in Table 6 are common words, how do you prevent repetitive reasoning, which could also be harmful?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.