acceptodds
Under review as a conference paper at ICLR 2027

From Generating Reasoning to Relying on It: Direct Counterfactual Optimization for Chain-of-Thought

Abstract

A model can give a correct answer and a plausible chain-of-thought (CoT) even when that CoT has little causal influence on the answer. We examine whether training to improve answer accuracy or favor low-redundancy rationales reduces the CoT's causal redundancy with respect to the answer. We assess training effects through complementary behavioral and activation-based probes: CORD and all-layer activation patching, respectively. Outcome-based RLVR improves accuracy, and DPO changes generated CoTs, but neither consistently lowers held-out CORD. This leaves open whether causal redundancy is difficult to change or these training signals fail to reach the relevant answer behavior. We compare trajectory-level scalar rewards with direct gradients on counterfactual answer likelihood. Under the tested protocols, direct optimization substantially lowers CORD, whereas scalar credit does not consistently do so. Naive direct optimization, however, harms normal-task accuracy. We propose Direct Counterfactual Optimization (DCO), a training objective that discourages premature recovery of the original answer from insufficient reasoning while protecting the model's behavior on intact question-CoT paths. It uses a reference-relative target to limit suppression under early, question-masked CoT prefixes and a KL budget to allow limited changes on intact inputs. Across three held-out mathematical-reasoning sets, DCO achieves matched of to , with accuracy changes of to percentage points. All-layer patching shows that DCO and naive direct optimization reduce sensitivity to question replacement and increase sensitivity to CoT replacement, unlike DPO and auxiliary-reward RLVR.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-Thought Reasoning, Causal Redundancy, Causal Reliance, Reasoning faithfulness, Post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9nSwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether training improves a model’s reliance on its generated chain-of-thought, rather than merely its answer accuracy or rationale quality. Using CORD as the primary behavioral measure, the authors find that outcome-based RLVR, rationale-level DPO, and auxiliary-reward training do not consistently reduce measured causal redundancy under the tested settings. They propose Direct Counterfactual Optimization (DCO), which suppresses answer likelihood under question-masked early reasoning prefixes while constraining changes on intact inputs. Experiments primarily on Qwen2.5-7B show substantial CORD reductions with small accuracy changes across three mathematical reasoning datasets, complemented by truncation-based and question/CoT substitution analyses.

Critical strengths
  • Targeted objective with accuracy preservation. DCO directly optimizes counterfactual answer likelihood while combining reference-relative suppression with normal-path KL protection. Across three evaluation datasets and three training seeds, it substantially reduces measured CORD with small changes in task accuracy; component ablations support the importance of bounded suppression and normal-behavior protection.

  • Multiple evaluations and informative controls. The paper complements CORD with CIR and question/CoT substitution analyses, including a same-problem CoT control. Additional checks address proxy-based preference scoring, decoding variability, and auxiliary-reward learning, providing useful evidence about the scope and consistency of the observed training effects.

Critical weaknesses
  • Uncertain distinction between reasoning reliance and intervention sensitivity. DCO directly suppresses answer likelihood under question-masked early prefixes, which closely resemble the B-axis evaluation conditions. The improvements may therefore partly reflect sensitivity to missing questions or truncated contexts rather than greater reliance on the logical content of reasoning. The complementary CoT substitution analysis does not fully resolve this concern because it also replaces concluding answer expressions; interventions that separate reasoning content from answer cues would strengthen the interpretation.

  • Confounded comparison of optimization routes. Reward-mediated methods train from the base model using online RLVR, whereas direct methods start from an RLVR checkpoint and use fixed correct trajectories. Although the paper acknowledges these differences and includes a detached scalar control, initialization, training stage, and data construction remain entangled with the gradient route. A comparison starting from the same checkpoint with more closely matched data and training budgets would better support attributing the difference to optimization access.

Questions for authors
  • Appendix C.1 excludes 80% and 100% prefixes because they may legitimately suffice to recover the answer, but what evidence establishes that the targeted 0–60% prefixes contain insufficient reasoning? Could the authors clarify whether the reported reductions distinguish premature recovery from suppression of valid inference on already sufficient prefixes?

  • Given the acknowledged differences in initialization, training stage, and data construction, what evidence isolates optimization access as the explanation for the contrast between scalar-credit and direct methods? Could the authors clarify which alternative explanations the detached scalar control rules out and which remain unresolved?

  • Could the authors provide a CoT-substitution control that holds concluding answer expressions fixed or excludes them from the replaced span? Since Appendix E.1 acknowledges that these expressions are currently replaced, this control would help determine whether the increased sensitivity reflects reliance on intermediate reasoning rather than answer cues.

6 comments

We thank the reviewer for the careful evaluation and constructive feedback. We particularly appreciate the recognition of DCO's effectiveness in reducing causal redundancy while preserving accuracy, as well as the value of our complementary evaluations and control experiments.


W1 & Q3

We thank the reviewer for raising the question about reasoning reliance and intervention sensitivity.

We understand the concern that, because DCO's training objective is closely related to the B-axis of CORD, a reduction in CORD alone may not be sufficient to establish increased reliance on reasoning content. However, our paper also evaluates CIR, which truncates CoT while keeping the original question visible. DCO achieved ΔCIR values of +0.227, +0.180, and +0.163 on GSM8K, MATH500, and DAPO-test, respectively (Table 8). Thus, the observed changes are not limited to question-masked settings.

Furthermore, to directly examine the influence of concluding answer expressions highlighted by the reviewer, we conducted an additional controlled experiment on GSM8K and MATH500.

Experimental Setup

We partition each CoT into the following components:

  • CtC_t, CdC_d: The bodies of the target and donor CoTs, respectively, containing all content before the sentence beginning with the first answer-bearing cue (e.g., "Therefore", "Thus").
  • AtA_t, AdA_d: The suffixes of the target and donor CoTs, respectively, extending from the first answer-bearing cue to the end of the CoT.
  • ∅\emptyset: Removal of the corresponding suffix.
  • mask⁡(At)\operatorname{mask}(A_t): Attention masking of the suffix while preserving its token positions.

We selected problems for which the base model, RLVR, and all three DCO seeds produced correct answers, and for which the boundaries of the first answer-bearing cue could be reliably identified (682 GSM8K and 214 MATH500 problems).

Across all conditions, we fixed the target question and the external final-answer cue. Following Section 7 and Appendix E.1, we measured the token-averaged log-likelihood of the original correct answer (y∗y^*) under teacher forcing. For each policy, we computed the absolute change in the answer score before and after each intervention and averaged these changes across problems to obtain the sensitivity SS. We then computed ΔS=SDCO−SRLVR\Delta S=S_{\mathrm{DCO}}-S_{\mathrm{RLVR}} using the policy-level paired-difference approach in Eq. (9). A positive ΔS\Delta S indicates that DCO exhibits a larger change in the correct-answer score than RLVR under the same type of CoT substitution.

Experimental Results

Table 1. Answer-cue-controlled CoT substitution (Qwen2.5-7B). We report ΔS=SDCO−SRLVR\Delta S=S_{\mathrm{DCO}}-S_{\mathrm{RLVR}}, where SS is the mean absolute change in the teacher-forced gold-answer token-averaged log-likelihood under each intervention. Values are mean ± sample standard deviation across three DCO seeds, with the RLVR checkpoint held fixed.

InterventionBefore → AfterGSM8K ΔSMATH500 ΔS
Full CoT substitutionCtAt→CdAdC_t A_t \rightarrow C_d A_d+3.362 ± 0.413+2.671 ± 0.438
Target suffix fixedCtAt→CdAtC_t A_t \rightarrow C_d A_t−0.301 ± 0.026−0.102 ± 0.044
Donor suffix fixedCtAd→CdAdC_t A_d \rightarrow C_d A_d+3.023 ± 0.309+2.321 ± 0.293
Suffix-only substitutionCtAt→CtAdC_t A_t \rightarrow C_t A_d+0.337 ± 0.108+0.397 ± 0.131
Suffix removedCt∅→Cd∅C_t\varnothing \rightarrow C_d\varnothing+2.104 ± 0.201+1.629 ± 0.229
Suffix attention-maskedCtmask⁡(At)→Cdmask⁡(At)C_t\operatorname{mask}(A_t) \rightarrow C_d\operatorname{mask}(A_t)+2.603 ± 0.203+2.015 ± 0.210

The paired-bootstrap 95% confidence intervals exclude zero for all comparisons. This experiment uses natural-length CoT substitutions and is therefore distinct from the length-matched, all-layer activation patching in the original paper.

First, DCO's greater sensitivity under full CoT substitution cannot be explained solely by the replacement of the concluding answer suffix. When we removed the suffixes from both CoTs and substituted only the CoT body, ΔS remained positive at +2.104 on GSM8K and +1.629 on MATH500. Similarly, when we preserved the suffix token positions but blocked attention to them, ΔS remained positive at +2.603 and +2.015, respectively. These results show that DCO is more sensitive than RLVR to substitutions of the CoT body even when the original question remains visible and the concluding answer suffix is not replaced.

Second, sensitivity to body substitution depends on which concluding answer information is preserved. When the correct target suffix was held fixed (CtAt→CdAtC_t A_t \rightarrow C_d A_t), both policies exhibited relatively small changes in the correct-answer score, and DCO was less sensitive than RLVR (GSM8K: 0.291 vs. 0.592; MATH500: 0.229 vs. 0.330). In contrast, when the donor suffix was held fixed (CtAd→CdAdC_t A_d \rightarrow C_d A_d), DCO exhibited greater sensitivity (ΔS = +3.023 and +2.321, respectively). These results indicate that the effect of body substitution depends strongly on the concluding answer information retained in the context.

Taken together, DCO's greater CoT substitution sensitivity cannot be explained solely by question masking or the replacement of the concluding answer suffix. However, the effect of body substitution becomes small when the correct target conclusion is preserved.

We will incorporate this additional experiment and its interpretation into the revised manuscript. Thank you.

W2

We thank the reviewer for pointing out that differences in training initialization and training stages may affect the comparison of optimization routes.

We first clarify why the optimization routes in our original experiments have different starting points.

Equation (3) shows that even when outcome-based RLVR improves answer accuracy, it does not determine whether the improvement arises from changes in the reasoning trajectory distribution or the answer conditional.

Pθ(Y=y∗∣X)=∑mPθ(M=m∣X)Pθ(Y=y∗∣X,M=m)P_\theta(Y=y^* \mid X)=\sum_m P_\theta(M=m\mid X)P_\theta(Y=y^*\mid X,M=m)

Based on this distinction, we compared three optimization routes:

  • DPO (rationale-distribution route): Outcome RLVR → Rationale-level preference optimization.
    • Starting from the same RLVR checkpoint, DPO is trained to prefer reasoning trajectories with lower proxy CORD. This approach primarily provides supervision targeting Pθ(M∣X)P_\theta(M\mid X).
  • Scalar credit (reward-mediated route): Outcome RLVR + Counterfactual auxiliary reward.
    • Starting from the base model, we incorporate A/B-axis counterfactual scores as trajectory-level scalar rewards during RLVR training to examine whether such reward signals can reduce causal redundancy.
  • DCO (direct conditional route): Outcome RLVR → Direct counterfactual optimization.
    • Starting from the same RLVR checkpoint as DPO, DCO directly optimizes the likelihood of the original correct answer under question-masked CoT prefixes, providing direct counterfactual supervision on Pθ(Y∣X,M)P_\theta(Y\mid X,M).

This design allows us to compare the effects of DPO and DCO from the same outcome-optimized policy, while also examining the distinction between providing counterfactual supervision through scalar rewards during RLVR and applying it directly to answer likelihood.

However, as the reviewer correctly points out, this comparison alone cannot determine whether DCO's effectiveness depends on RLVR initialization or the stage at which it is applied. To investigate this, we conducted an additional experiment in which DCO was applied before RLVR.

Table 2. Effects of DCO training order (Qwen2.5-7B, single seed). All differences are measured relative to the base model. Δacc is reported in percentage points (pp), and matched ΔCORD is computed on items answered correctly by both policies. * indicates that the paired-bootstrap 95% confidence interval excludes zero.

MethodGSM8K Δacc ↑GSM8K ΔCORD ↓MATH500 Δacc ↑MATH500 ΔCORD ↓DAPO-test Δacc ↑DAPO-test ΔCORD ↓
Base → RLVR → DCO (paper)+9.9*−0.297*+13.4*−0.347*+5.0*−0.249*
Base → DCO+1.4−0.278*−0.4−0.334*−2.4*−0.203*
Base → DCO → RLVR+8.1*−0.258*+12.2*−0.307*+5.8*−0.206*

The Base → RLVR → DCO results are not numerically identical to those in Table 2 of the paper because the present analysis uses the base model as the reference (resulting in different common-correct item sets) and reports a single seed rather than three-seed averages.

First, DCO achieved substantial CORD reductions even when applied directly to the base model without prior RLVR training. Base → DCO yielded ΔCORD values of −0.278, −0.334, and −0.203 on GSM8K, MATH500, and DAPO-test, respectively. These reductions are comparable in magnitude to those achieved by the original Base → RLVR → DCO route, demonstrating that DCO's effectiveness does not depend on RLVR-trained initialization.

Second, the effects of DCO were largely preserved after subsequent RLVR training. Under Base → DCO → RLVR, approximately 92–101% of the CORD reductions achieved immediately after DCO were retained. At the same time, subsequent RLVR improved accuracy by +6.7, +12.6, and +8.2 percentage points on GSM8K, MATH500, and DAPO-test, respectively. These results show that DCO remains effective when applied before RLVR and that its effects are compatible with subsequent outcome optimization.

Taken together, DCO's effectiveness is not limited to initialization from an RLVR checkpoint or application after RLVR training. Therefore, the effects of direct optimization observed in our original comparison cannot be readily explained by RLVR initialization or the post-RLVR training stage alone.

We will incorporate these additional experiments and their interpretation into the revised manuscript. We will also clarify the rationale behind the optimization-route comparisons. Thank you.

Reviewer_Ja7aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the causal role of CoT reasoning in LLM's final answers to questions. The paper introduces one metric for this problem, CORD, and borrows another metric, CIR, from related work. Experiments test the effect of RLVR on causal importance scores, finding that RLVR often does not influence causal importance scores, matching prior work. Next, the paper explores the role of credit assignment and objective design, finding that directly optimizing condiitonal answer probabilities under CoT interventions does affect CORD. Some additional supporting analyses are done concerning the effect of activation patching on the question text or CoT text.

Critical strengths

Strengths:

  • the paper addresses a critical topic, which is CoT faithfulness, or the role of CoT in models obtaining their final answers. This question has significant downstream implications for CoT monitoring
Critical weaknesses
  • The paper does not sincerely engage with relevant prior work, of which there is much. The introduction reviews prior work (Paul et al., 2024; Swaroop et al., 2025; Yu et al., 2026; Jia et al., 2026), then dismisses all of these papers with the claim: “Yet it remains unclear whether these training interventions also change the answer’s reliance on the CoT.” The paper would be better off explaining why 4-5 related works which directly study the question of causal importance of CoT actually fail to provide evidence about the answer’s “reliance” on the CoT. The related work later reviews the contributions of past work in slightly greater detail, then states: “These studies target different aspects of reasoning...Improvements in those objectives do not necessarily establish that an answer relies on its generated CoT: a policy could produce a correct answer and a desirable rationale while reaching that answer through another path.” This claim is blatantly disingenous. That is the research question addressed in those papers. This issue is not just a matter of writing the related work section. The paper does not test any of the methods from those papers. So there are 4-5 missing baselines immediately. The paper should compare to baselines.
  • Core hypotheses of the paper should be presented more precisely. The intro asks whether “tested training interfaces provide insufficient access to the target behavior.” What does this mean? This is a vague sentence by colloquial standards. By ML conference standards, it is completely untestable as written.
Questions for authors
  • see weaknesses above
3 comments

We thank the reviewer for the evaluation and feedback. We appreciate the recognition of the importance of studying the role of CoT in model predictions and its implications for CoT monitoring.


W1

We respectfully disagree with the claim that the paper lacks 4–5 relevant baselines. The claimed absence of baselines is factually incorrect.

Our submission already includes:

  • Outcome-based RLVR and CORD-based DPO (Section 4, Table 1; Appendix A–B).
  • CIR and SR auxiliary-reward methods from Yu et al. (2026) (Appendix C.2, Figures 2–3).
  • Four structural optimization methods from Jia et al. (2026): Update masking, Gradient masking, CoT-gradient, and FACT (Appendix C.3, Table 12).

These are direct experimental comparisons under our evaluation protocol, not merely discussions of prior work.

Therefore, the assertion that "the paper does not test any of the methods from those papers" is demonstrably incorrect. We ask the reviewer to reconsider this criticism in light of the experiments already present in the submitted manuscript.

W2

The hypothesis is neither undefined nor untestable. It is explicitly operationalized and experimentally tested in Section 5.

Section 5 defines optimization access as how target information enters the gradient. Section 5.1 then formalizes two optimization routes:

  • Reward-mediated route (Eq. 4): Counterfactual behavior contributes trajectory-level scalar credit through a detached advantage.
  • Direct conditional route (Eqs. 5–6): Counterfactual answer likelihood is directly differentiated under the intervened context.

Our hypothesis is that these two routes produce different changes in held-out causal redundancy under the tested protocols. Table 2 directly tests this hypothesis, demonstrating substantial CORD reductions under direct optimization but not consistent reductions under scalar credit.

Thus, the criticized sentence refers to a precisely defined and empirically testable distinction. We respectfully disagree with the characterization that the hypothesis is "completely untestable."

Thank you for your feedback. We have uploaded a revised manuscript. We would appreciate your reconsideration.

With approximately 30 hours remaining in the discussion period, we would appreciate your feedback on our rebuttal and revised manuscript. Thank you for your time and consideration.

Reviewer_8EqwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether the chain-of-thought (CoT) generated by a model genuinely influences its final answer and how to reduce shortcut reasoning. The authors find that although RLVR improves answer accuracy and DPO changes generated reasoning traces, neither consistently reduces causal redundancy. To address this issue, they propose Direct Counterfactual Optimization (DCO), which combines bounded counterfactual suppression with KL regularization to increase the model's reliance on CoT while preserving normal-task performance. Experiments on three mathematical reasoning datasets demonstrate that DCO substantially reduces CORD while largely maintaining answer accuracy. Activation patching experiments provide further evidence of its effectiveness.

Critical strengths
  1. The paper investigates an interesting and important research question: improvements in final-answer accuracy and changes in generated reasoning traces do not necessarily imply greater reliance on CoT. This suggests that evaluating LLM training solely based on prediction accuracy may overlook important aspects of model behavior.

  2. The two motivating experiments clearly support the authors' motivation, although their presentation in the introduction is somewhat confusing.

  3. The proposed method effectively encourages the model to rely more heavily on its generated CoT.

Critical weaknesses
  1. The introduction suffers from a disorganized and discontinuous logical flow. Specifically, the first paragraph introduces shortcut reasoning as the central problem of the paper. The second paragraph then discusses related work on shortcut reasoning and introduces the proposed approach. However, the third paragraph returns to related work before introducing the paper's research question. This organization feels disjointed and makes the introduction unnecessarily difficult to follow.

  2. Although the experiments show that DCO reduces causal redundancy, how do the authors establish that the model actually follows the intermediate reasoning steps in the CoT? The model might instead learn to extract answer-relevant information from the CoT without relying on its intermediate logical steps. Such behavior could also reduce CORD while leaving the underlying reasoning process largely unchanged.

  3. The evaluation of general model capabilities is insufficient. Since DCO is a post-training method for LLMs, why do the authors not systematically evaluate changes in prediction accuracy before and after training on a broader range of standard benchmarks, such as MMLU, GPQA, HumanEval, BBH, GSM8K, and MATH, as is commonly done in LLM training research?

Questions for authors
  1. When CORD is first mentioned in the abstract and introduction, its full name is not provided.

  2. The introduction does not adequately explain the meaning of ΔCORD and proxy CORD. Without at least a brief explanation of these concepts, readers may struggle to understand the reported experimental observations and assess the significance of the corresponding conclusions. The authors should either briefly define these terms when they first appear or avoid introducing these results before providing the necessary context. Otherwise, the readability of the introduction is significantly affected.

  3. The motivation for the central research question is not logically well established. In the introduction, the authors state that the preceding findings leave an open question: whether causal redundancy itself is difficult to change or whether the tested training methods fail to effectively target the relevant behavior. However, it is unclear how the preceding experimental observations and conclusions logically lead to this particular question. The authors should clarify the reasoning that motivates this distinction and explain why these are the two hypotheses that should be investigated.

7 comments

We thank the reviewer for the thoughtful evaluation and constructive feedback. We particularly appreciate the positive assessment of the importance of our research question, the clarity of our motivating experiments, and the effectiveness of DCO in improving reliance on CoT.


Clarity and Presentation (W1, Q1, Q2, Q3)

We particularly appreciate the reviewer's recognition that our research question is important and that the two motivating experiments involving RLVR and DPO provide clear support for our research motivation.

The central point of our study is that the causal redundancy of CoT is defined in terms of counterfactual answer behavior. Both our primary metric, CORD, and secondary metric, CIR, measure the relationship to the original answer when CoT is disrupted or incomplete. Therefore, reducing causal redundancy requires changing answer behavior in these counterfactual contexts, rather than merely changing CoT content.

From this perspective, we first examined whether existing training methods also reduce causal redundancy (Section 4). Outcome-based RLVR substantially improved answer accuracy but did not consistently reduce CORD. Similarly, DPO failed to produce consistent improvements, even when trained to prefer reasoning trajectories with lower causal redundancy.

These findings motivate our subsequent research question because existing training methods do not provide direct gradients on counterfactual answer behavior. As described in Equation 3, the probability of a correct answer can be expressed through the joint contribution of the reasoning trajectory distribution and the answer probability conditioned on each trajectory. RLVR optimizes final-answer accuracy, while DPO learns preferences constructed using counterfactual measurements. However, neither method directly applies gradients to the conditional likelihood of the original correct answer under disrupted CoT contexts. Consequently, improvements in answer accuracy or changes in reasoning trajectories alone do not guarantee improvements in the targeted counterfactual answer behavior.

Thus, these two experiments alone cannot determine whether causal redundancy is inherently difficult to change through training or whether existing optimization methods fail to provide sufficiently effective learning signals for the target behavior. This distinction motivated us to investigate two possibilities: target movability and optimization access.

To investigate this distinction, Section 5 compares trajectory-level scalar rewards with direct gradients on counterfactual answer scores, using the same type of counterfactual target. Under the evaluated conditions, direct optimization reduced causal redundancy, whereas scalar-reward optimization did not achieve consistent reductions. These results demonstrate that causal redundancy is indeed modifiable and that the optimization mechanism used to access counterfactual answer behavior matters.

Building on these findings, Section 6 introduces DCO to address the accuracy degradation associated with direct optimization. DCO combines reference-relative counterfactual suppression with normal-behavior protection, achieving substantial reductions in CORD while largely preserving the original model performance.

Based on this explanation, we will improve the clarity of the Introduction as follows:

  • W1: The two related-work paragraphs in the current Introduction serve distinct purposes: the first introduces methods for measuring shortcut reasoning, while the second discusses training methods intended to mitigate it. We will improve the transition between these paragraphs and adjust the placement of our research questions to make this distinction clearer.
  • Q1: We will explicitly spell out CORD as Causal Redundancy of Reasoning via Disruption at its first occurrence in both the Abstract and Introduction.
  • Q2: We will briefly define matched ΔCORD and proxy CORD at their first occurrences and clarify that all held-out ΔCORD results are based on the full CORD evaluation.
  • Q3: We will explicitly describe the relationship between counterfactual answer behavior and optimization gradients in the Introduction, making it clearer why the two motivating experiments lead to our subsequent investigation of target movability and optimization access.

W2

We thank the reviewer for the positive assessment of the importance of our study and the effectiveness of DCO in increasing answer reliance on CoT.

To address the reviewer's question, we conducted an additional experiment distinguishing the point at which the correct answer is first explicitly derived in an intermediate step of CoT from the point at which it is restated in the final natural-language answer cue.

Experimental Setup

We identified four boundaries in CoT, distinguishing the first explicit occurrence of the correct answer in an intermediate step from its restatement in the final natural-language conclusion:

  • I1 (Before intermediate answer): CoT ending immediately before the correct answer value is explicitly revealed in an intermediate calculation.
  • I2 (After intermediate answer): CoT ending immediately after the correct answer value is first revealed in the same intermediate calculation.
  • I3 (Before final answer cue): CoT containing the intermediate calculation result but ending before the final natural-language answer cue.
  • A (Full CoT): The complete CoT, including the final natural-language answer cue.

For example, a MATH500 case presented in Appendix F is segmented as follows:

x = 8/2 = [I1] 4 [I2] [I3] Thus, the value of x is 4. [A]

This setup allows us to isolate two effects:

  1. Intermediate answer exposure (I1→I2): The effect of first revealing the correct answer value in an intermediate calculation.
  2. Final answer cue and subsequent content (I3→A): The effect of adding the final natural-language answer cue and subsequent content when the computed answer has already been revealed.

We evaluated 50 problems each from GSM8K and MATH500. We restricted the evaluation to problems for which the RLVR model and all three DCO training seeds produced correct answers with their original CoTs, and for which all four models also correctly regenerated the answer when given the original question, the full CoT (A), and a shared answer-generation prompt. We further selected problems for which the intermediate answer derivation and the final natural-language answer cue could be clearly distinguished. To control for differences in CoT length across models, we separately compared the full CoT lengths and the lengths up to the point immediately before the final answer cue (I3) for RLVR and all three DCO seeds on each problem. We included only problems for which the ratio of the maximum to minimum length was at most 1.1 in both comparisons. This criterion limits potential differences in overall reasoning length and the relative position of the final answer cue across models.

This design allows us to examine how answer recovery changes depending on which portions of CoT are provided, using the same problems for which all models can correctly generate the answer under the full-context condition.

Experimental Results

Table 1. Changes in answer recovery rate under the question-visible condition (percentage points).

DatasetComparisonRLVRDCO
GSM8KI1 → I2+4.0+14.0 ± 7.2
GSM8KI3 → A0.0+0.7 ± 1.2
MATH500I1 → I2+8.0+25.3 ± 6.1
MATH500I3 → A0.0+2.0 ± 0.0

First, DCO's answer recovery rate increased when the correct answer value was first revealed in an intermediate calculation. For I1→I2, the recovery rate increased by 14.0 percentage points for DCO, compared with 4.0 for RLVR on GSM8K, and by 25.3 percentage points for DCO, compared with 8.0 for RLVR on MATH500. These results show that DCO's answer recovery responds to correct-answer information explicitly provided in intermediate calculations, even when the original question remains visible.

Second, adding the final natural-language answer cue and subsequent content produced only small changes in recovery rate when the correct answer had already been revealed in an intermediate calculation. For DCO, the I3→A changes were only +0.7 percentage points on GSM8K and +2.0 percentage points on MATH500. This indicates that, for the evaluated samples, most correct answers could already be recovered before the final natural-language conclusion appeared.

Together, these additional results provide concrete evidence that DCO's answer behavior does not depend solely on the presence of the final natural-language conclusion, but also responds to the point at which the correct answer is explicitly derived in intermediate CoT calculations.

We will incorporate these additional analyses and experimental details into the revised manuscript. Thank you.

W3

We thank the reviewer for suggesting additional evaluations of general model capabilities. This suggestion has helped us further assess the impact of our proposed method, DCO.

Our study aims to improve the causal reliance of answers on CoT while preserving the original model's problem-solving capabilities. In the original submission, we evaluated the preservation of mathematical reasoning performance using GSM8K, MATH500, and DAPO-test. Following the reviewer's suggestion, we conducted additional evaluations on four standard benchmarks to examine the impact of DCO on general model capabilities beyond mathematical reasoning.

Benchmarks

  • MMLU-Pro [1]: A challenging benchmark consisting of ten-option multiple-choice questions across 14 disciplines, including mathematics, physics, engineering, history, law, and psychology. We use MMLU-Pro, which includes more challenging, reasoning-intensive questions than the original MMLU.
  • GPQA [2]: A dataset of 448 challenging multiple-choice questions authored by domain experts in biology, physics, and chemistry.
  • BBH [3]: A benchmark consisting of 23 challenging tasks selected from BIG-Bench.
  • HumanEval+ [4]: An extended version of HumanEval with substantially expanded test cases for evaluating code generation.

Evaluation Protocol

  • We evaluate two models, πRLVR\pi_{\mathrm{RLVR}} and πDCO\pi_{\mathrm{DCO}}, where πDCO\pi_{\mathrm{DCO}} is obtained by applying DCO training to the πRLVR\pi_{\mathrm{RLVR}} checkpoint.
  • Both models are checkpoints from our study and were trained exclusively on mathematical reasoning tasks.
  • We follow the zero-shot evaluation protocol used in General-Reasoner [5].

Experimental Results

The additional evaluations showed no substantial accuracy degradation following DCO training across the four benchmarks (mean ΔAcc ranging from −0.6 to +0.9 pp).

Table 2. General capability evaluation. DCO accuracy and ΔAcc are reported as mean ± standard deviation over three training seeds. RLVR denotes the shared initialization checkpoint. ΔAcc is measured in percentage points relative to RLVR.

BenchmarkπRLVR\pi_{\mathrm{RLVR}} (%)πDCO\pi_{\mathrm{DCO}} (%)ΔAcc (pp)
MMLU-Pro49.550.4 ± 0.3+0.9 ± 0.3
GPQA31.932.1 ± 2.0+0.1 ± 2.0
BBH60.560.9 ± 0.2+0.3 ± 0.2
HumanEval+72.672.0 ± 0.0−0.6 ± 0.0

These additional results support that DCO can be applied without substantial performance degradation on the evaluated general capability benchmarks, even when training is conducted exclusively on mathematical reasoning tasks.

We will incorporate these additional results and detailed evaluation settings into the revised manuscript. Thank you.

[1] Wang, Yubo, et al. “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark.” Advances in Neural Information Processing Systems, vol. 37, 2024. https://doi.org/10.52202/079017-3018.

[2] Rein, David, et al. “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.” First Conference on Language Modeling (COLM), 2024. https://arxiv.org/abs/2311.12022.

[3] Suzgun, Mirac, et al. “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them.” Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, 2023, pp. 13003–13051. https://doi.org/10.18653/v1/2023.findings-acl.824.

[4] Liu, Jiawei, et al. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” Advances in Neural Information Processing Systems, vol. 36, 2023. https://doi.org/10.52202/075280-0943.

[5] Ma, Xueguang, et al. “General-Reasoner: Advancing LLM Reasoning Across All Domains.” Advances in Neural Information Processing Systems 38, Neural Information Processing Systems Foundation, 2025, pp. 63009–31, https://doi.org/10.52202/085713-1893.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.