- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether training improves a model’s reliance on its generated chain-of-thought, rather than merely its answer accuracy or rationale quality. Using CORD as the primary behavioral measure, the authors find that outcome-based RLVR, rationale-level DPO, and auxiliary-reward training do not consistently reduce measured causal redundancy under the tested settings. They propose Direct Counterfactual Optimization (DCO), which suppresses answer likelihood under question-masked early reasoning prefixes while constraining changes on intact inputs. Experiments primarily on Qwen2.5-7B show substantial CORD reductions with small accuracy changes across three mathematical reasoning datasets, complemented by truncation-based and question/CoT substitution analyses.
Critical strengths
-
Targeted objective with accuracy preservation. DCO directly optimizes counterfactual answer likelihood while combining reference-relative suppression with normal-path KL protection. Across three evaluation datasets and three training seeds, it substantially reduces measured CORD with small changes in task accuracy; component ablations support the importance of bounded suppression and normal-behavior protection.
-
Multiple evaluations and informative controls. The paper complements CORD with CIR and question/CoT substitution analyses, including a same-problem CoT control. Additional checks address proxy-based preference scoring, decoding variability, and auxiliary-reward learning, providing useful evidence about the scope and consistency of the observed training effects.
Critical weaknesses
-
Uncertain distinction between reasoning reliance and intervention sensitivity. DCO directly suppresses answer likelihood under question-masked early prefixes, which closely resemble the B-axis evaluation conditions. The improvements may therefore partly reflect sensitivity to missing questions or truncated contexts rather than greater reliance on the logical content of reasoning. The complementary CoT substitution analysis does not fully resolve this concern because it also replaces concluding answer expressions; interventions that separate reasoning content from answer cues would strengthen the interpretation.
-
Confounded comparison of optimization routes. Reward-mediated methods train from the base model using online RLVR, whereas direct methods start from an RLVR checkpoint and use fixed correct trajectories. Although the paper acknowledges these differences and includes a detached scalar control, initialization, training stage, and data construction remain entangled with the gradient route. A comparison starting from the same checkpoint with more closely matched data and training budgets would better support attributing the difference to optimization access.
Questions for authors
-
Appendix C.1 excludes 80% and 100% prefixes because they may legitimately suffice to recover the answer, but what evidence establishes that the targeted 0–60% prefixes contain insufficient reasoning? Could the authors clarify whether the reported reductions distinguish premature recovery from suppression of valid inference on already sufficient prefixes?
-
Given the acknowledged differences in initialization, training stage, and data construction, what evidence isolates optimization access as the explanation for the contrast between scalar-credit and direct methods? Could the authors clarify which alternative explanations the detached scalar control rules out and which remain unresolved?
-
Could the authors provide a CoT-substitution control that holds concluding answer expressions fixed or excludes them from the replaced span? Since Appendix E.1 acknowledges that these expressions are currently replaced, this control would help determine whether the increased sensitivity reflects reliance on intermediate reasoning rather than answer cues.
6 comments
Second, sensitivity to body substitution depends on which concluding answer information is preserved. When the correct target suffix was held fixed (), both policies exhibited relatively small changes in the correct-answer score, and DCO was less sensitive than RLVR (GSM8K: 0.291 vs. 0.592; MATH500: 0.229 vs. 0.330). In contrast, when the donor suffix was held fixed (), DCO exhibited greater sensitivity (ΔS = +3.023 and +2.321, respectively). These results indicate that the effect of body substitution depends strongly on the concluding answer information retained in the context.
Taken together, DCO's greater CoT substitution sensitivity cannot be explained solely by question masking or the replacement of the concluding answer suffix. However, the effect of body substitution becomes small when the correct target conclusion is preserved.
We will incorporate this additional experiment and its interpretation into the revised manuscript. Thank you.
W2
We thank the reviewer for pointing out that differences in training initialization and training stages may affect the comparison of optimization routes.
We first clarify why the optimization routes in our original experiments have different starting points.
Equation (3) shows that even when outcome-based RLVR improves answer accuracy, it does not determine whether the improvement arises from changes in the reasoning trajectory distribution or the answer conditional.
Based on this distinction, we compared three optimization routes:
- DPO (rationale-distribution route): Outcome RLVR → Rationale-level preference optimization.
- Starting from the same RLVR checkpoint, DPO is trained to prefer reasoning trajectories with lower proxy CORD. This approach primarily provides supervision targeting .
- Scalar credit (reward-mediated route): Outcome RLVR + Counterfactual auxiliary reward.
- Starting from the base model, we incorporate A/B-axis counterfactual scores as trajectory-level scalar rewards during RLVR training to examine whether such reward signals can reduce causal redundancy.
- DCO (direct conditional route): Outcome RLVR → Direct counterfactual optimization.
- Starting from the same RLVR checkpoint as DPO, DCO directly optimizes the likelihood of the original correct answer under question-masked CoT prefixes, providing direct counterfactual supervision on .
This design allows us to compare the effects of DPO and DCO from the same outcome-optimized policy, while also examining the distinction between providing counterfactual supervision through scalar rewards during RLVR and applying it directly to answer likelihood.
However, as the reviewer correctly points out, this comparison alone cannot determine whether DCO's effectiveness depends on RLVR initialization or the stage at which it is applied. To investigate this, we conducted an additional experiment in which DCO was applied before RLVR.
Table 2. Effects of DCO training order (Qwen2.5-7B, single seed). All differences are measured relative to the base model. Δacc is reported in percentage points (pp), and matched ΔCORD is computed on items answered correctly by both policies. * indicates that the paired-bootstrap 95% confidence interval excludes zero.
| Method | GSM8K Δacc ↑ | GSM8K ΔCORD ↓ | MATH500 Δacc ↑ | MATH500 ΔCORD ↓ | DAPO-test Δacc ↑ | DAPO-test ΔCORD ↓ |
|---|---|---|---|---|---|---|
| Base → RLVR → DCO (paper) | +9.9* | −0.297* | +13.4* | −0.347* | +5.0* | −0.249* |
| Base → DCO | +1.4 | −0.278* | −0.4 | −0.334* | −2.4* | −0.203* |
| Base → DCO → RLVR | +8.1* | −0.258* | +12.2* | −0.307* | +5.8* | −0.206* |
The Base → RLVR → DCO results are not numerically identical to those in Table 2 of the paper because the present analysis uses the base model as the reference (resulting in different common-correct item sets) and reports a single seed rather than three-seed averages.
First, DCO achieved substantial CORD reductions even when applied directly to the base model without prior RLVR training. Base → DCO yielded ΔCORD values of −0.278, −0.334, and −0.203 on GSM8K, MATH500, and DAPO-test, respectively. These reductions are comparable in magnitude to those achieved by the original Base → RLVR → DCO route, demonstrating that DCO's effectiveness does not depend on RLVR-trained initialization.
Second, the effects of DCO were largely preserved after subsequent RLVR training. Under Base → DCO → RLVR, approximately 92–101% of the CORD reductions achieved immediately after DCO were retained. At the same time, subsequent RLVR improved accuracy by +6.7, +12.6, and +8.2 percentage points on GSM8K, MATH500, and DAPO-test, respectively. These results show that DCO remains effective when applied before RLVR and that its effects are compatible with subsequent outcome optimization.
Taken together, DCO's effectiveness is not limited to initialization from an RLVR checkpoint or application after RLVR training. Therefore, the effects of direct optimization observed in our original comparison cannot be readily explained by RLVR initialization or the post-RLVR training stage alone.
We will incorporate these additional experiments and their interpretation into the revised manuscript. We will also clarify the rationale behind the optimization-route comparisons. Thank you.
We thank the reviewer for the careful evaluation and constructive feedback. We particularly appreciate the recognition of DCO's effectiveness in reducing causal redundancy while preserving accuracy, as well as the value of our complementary evaluations and control experiments.
W1 & Q3
We thank the reviewer for raising the question about reasoning reliance and intervention sensitivity.
We understand the concern that, because DCO's training objective is closely related to the B-axis of CORD, a reduction in CORD alone may not be sufficient to establish increased reliance on reasoning content. However, our paper also evaluates CIR, which truncates CoT while keeping the original question visible. DCO achieved ΔCIR values of +0.227, +0.180, and +0.163 on GSM8K, MATH500, and DAPO-test, respectively (Table 8). Thus, the observed changes are not limited to question-masked settings.
Furthermore, to directly examine the influence of concluding answer expressions highlighted by the reviewer, we conducted an additional controlled experiment on GSM8K and MATH500.
Experimental Setup
We partition each CoT into the following components:
We selected problems for which the base model, RLVR, and all three DCO seeds produced correct answers, and for which the boundaries of the first answer-bearing cue could be reliably identified (682 GSM8K and 214 MATH500 problems).
Across all conditions, we fixed the target question and the external final-answer cue. Following Section 7 and Appendix E.1, we measured the token-averaged log-likelihood of the original correct answer (y∗) under teacher forcing. For each policy, we computed the absolute change in the answer score before and after each intervention and averaged these changes across problems to obtain the sensitivity S. We then computed ΔS=SDCO−SRLVR using the policy-level paired-difference approach in Eq. (9). A positive ΔS indicates that DCO exhibits a larger change in the correct-answer score than RLVR under the same type of CoT substitution.
Experimental Results
Table 1. Answer-cue-controlled CoT substitution (Qwen2.5-7B). We report ΔS=SDCO−SRLVR, where S is the mean absolute change in the teacher-forced gold-answer token-averaged log-likelihood under each intervention. Values are mean ± sample standard deviation across three DCO seeds, with the RLVR checkpoint held fixed.
The paired-bootstrap 95% confidence intervals exclude zero for all comparisons. This experiment uses natural-length CoT substitutions and is therefore distinct from the length-matched, all-layer activation patching in the original paper.
First, DCO's greater sensitivity under full CoT substitution cannot be explained solely by the replacement of the concluding answer suffix. When we removed the suffixes from both CoTs and substituted only the CoT body, ΔS remained positive at +2.104 on GSM8K and +1.629 on MATH500. Similarly, when we preserved the suffix token positions but blocked attention to them, ΔS remained positive at +2.603 and +2.015, respectively. These results show that DCO is more sensitive than RLVR to substitutions of the CoT body even when the original question remains visible and the concluding answer suffix is not replaced.