acceptodds
Under review as a conference paper at ICLR 2027

Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment

Abstract

Chain-of-thought (CoT) traces often serve as a proxy for how Large Language Models (LLMs) arrive at their answers. However, growing evidence shows that models' CoT often fails to reflect their internal computations and can be manipulated to produce different CoTs without changing their outputs. In this work, we measure and improve the alignment between what LLMs say in their CoT and what they compute internally. To quantify such parametric faithfulness, we propose CoT-Interpretability Alignment (CIA), a metric that measures the agreement between a model's CoT traces and its internal reasoning strategies as detected by interpretability tools. We evaluate CIA on three tasks (two-hop question answering, hint intervention, and integer multiplication) across three LLMs, finding that current models exhibit low CIA scores across all tasks. We then experiment with improving CIA via post-training while setting both task accuracy and parametric faithfulness signals as a reward. Experiments show that we can substantially improve CoT parametric faithfulness while maintaining or improving task accuracy. We provide rich analysis, showing how such post-training changes model behaviors: in some tasks the model learns to change how it reasons, while in others it learns to change how it reports. Our work provides both a framework for auditing CoT parametric faithfulness and a pathway toward making models' explicit reasoning more trustworthy.

Primary area:
interpretability and explainable AI
Keywords:
CoT faithfulness, Interpretability, CoT monitorability, LLM Reasoning, Interpretability for AI Safety, Benchmarking Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q55ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CoT-Interpretability Alignment (CIA), a metric that measures how closely an LLM's verbalized chain-of-thought reasoning matches its internal computational strategies, using interpretability techniques such as linear probes. Experiments done on three reasoning tasks and three language models reveal substantial differences between the verbalized and inferred internal reasoning of the LLM, but show that post-training with rejection sampling, DPO, or GRPO can improve alignment. Further experiments suggest that these improvements arise through two mechanisms: models either become better at reporting their existing internal computations or modify their computations to follow their verbalized reasoning steps better.

Critical strengths
  • CIA provides a helpful framework for measuring and improving reasoning faithfulness by connecting interpretability techniques with post-training objectives
  • Experiments across three model families, three reasoning tasks, and three post-training approaches demonstrate substantial improvements in CIA while largely preserving task accuracy
  • The authors conduct causal intervention experiments that complement the probing results, providing better evidence that observed improvements are a result of genuine changes in model behavior
Critical weaknesses
  • Experiments primarily focus on relatively small models and simple/highly structured reasoning tasks, leaving generalization to frontier reasoning models and complex, long-horizon reasoning unclear
Questions for authors
  • Would it be possible to evaluate if the method works on larger models too?
  • Can you provide a more detailed assessment on reasoning tasks that have more complex and long horizon nature?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Q55Z

We thank the reviewer for the positive assessment, and for noting that CIA “provides a helpful framework for measuring and improving reasoning faithfulness” and that the causal interventions “complement the probing results”.

W1 & Q1 - Regarding evaluation on larger models:

Yes. We have provided experiments in Appendix E: “Extending to Larger and Reasoning Models”, which evaluates Qwen3-14B on 2-digit multiplication with the same pipeline. For the rebuttal, we also evaluated Qwen3-14B on MMLU-Hint: it follows the misleading hint less often than Qwen3-8B (13.4% vs. 17.8%) and is more accurate (0.847 vs. 0.808).

W1 & Q2 - Regarding complex, long-horizon reasoning tasks:

We have provided experimental evidence on Reasoning Models in Appendix E, where Qwen3-8B in thinking mode and DeepSeek-R1-Distill-Llama-8B write a long thinking trace (up to 8,192 new tokens, compared with 512 in the main experiments) in the MMLU-Hint task. Our method also works on these long traces. After we retrain the probe on thinking-mode outputs, it still detects whether the hint drove the answer (macro-F1 0.894 for Qwen3-8B and 0.858 for DeepSeek-R1-Distill).

And the method of training with the reward of CIA still helps: CIA rises by +0.090 (RS) and +0.083 (DPO) for Qwen3-8B and by +0.078 (RS) for R1-Distill, while DPO does not change it significantly for R1-Distill. Each example is still a single multiple-choice question in this task. For tasks with many dependent steps, the same corruption test can be applied step by step: we change one written intermediate result at a time and check whether the answer follows.

Thank you for the clarification and follow-up. I maintain my positive score.

Reviewer_3foMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoT-Interpretability Alignment (CIA), which compares a binary label extracted from a model’s written reasoning with a strategy label inferred from internal representations or behavioral interventions. CIA is the macro-F1 agreement across the positive and negative labels. The main study covers two-hop factual reasoning, misleading hints, and two-digit multiplication across Llama, Qwen, and Gemma models. Rejection sampling, DPO, and GRPO use agreement signals to improve CIA. Auxiliary tools, cross-task transfer, transition analyses, and causal interventions investigate whether gains reflect different computation or different reporting.

Critical strengths
  1. A meaningful research motivation with a sensible experimental design. CIA connects explanation evaluation to post-training and separates reporting changes from computational changes. This provides a relevant experimental framework for studying when explicit reasoning becomes more informative.

  2. Contributions beyond the training metrics. Probes reliability checks, auxiliary measurements, reward ablations, causal interventions, and thinking-mode experiments provide converging evidence. Provides a practical way to cohere interpretability and reasoning, in a quite timely area.

Critical weaknesses
  1. The aligned labels do not consistently represent the same notion of faithfulness. In the hint task, the CoT judge counts explicit rejection of a hint as a positive label, while the internal label concerns influence toward the hinted answer. A truthful statement that the hint was considered and rejected can be scored as unfaithful when the internal label is negative. Removing that statement can improve CIA without making the explanation more truthful. This directly affects the interpretation of reporting improvements and requires separate treatment of awareness, reliance, and rejection (check questions below).

  2. Strategy and training interpretations are stronger than the associated evidence. Bridge-entity decodability does not establish its causal use and failure to track a corrupted partial product does not uniquely imply direct recall. While the causal checks are valuable, any output change is weaker evidence than the predicted strategy-specific change.

  3. More comprehensive analysis of the experimental design assumptions. How the specific target tasks and models limit the interpretation of results.

Questions for authors
  • How the specific target tasks and models limit the interpretation of results? What are the selection biases implied by the tasks and selected models? How does it limit the interpretation of results?

  • How do you guarantee that the hint metric does not conflate awareness with reliance? The CoT judge counts explicit rejection of a hint as positive, whereas the internal label concerns influence toward the hinted answer. A truthful rejection can therefore be penalized when the internal label is negative, correct? This mismatch weakens the claim that higher CIA reflects more truthful reporting.

  • How does the evidence uniquely identify the claimed reasoning strategies? Decoding a bridge entity establishes that information is available, not that it drives the answer, correct? The causal analyses strengthen the evidence for dependence on intermediate representations and text, but do not fully support the stronger interpretation of a transition from recall to algorithmic reasoning.

  • Can you position more clearly the core methodological contribution? While there is an argument of higher coherence, it would be good to have a more systematic and precise positioning.

  • General results interpretation. Held-out probe performance, auxiliary evaluations, and causal checks provide safeguards, but auxiliary measures share some labels or assumptions with the primary measurements. Their agreement can drive some uncertainty about whether training improves faithful reporting or conformity to the measurement scheme. The reported gains establish improvement in operational agreement more securely than improvement in truthful access to internal computation. I believe this point is central to positioning the contribution in a proper manner.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3foM (1/3)

We thank the reviewer for the detailed review, and for describing the work as “a meaningful research motivation with a sensible experimental design” in “a quite timely area”. Below, BINTB_\text{INT} says whether the model actually used the strategy, BCoTB_\text{CoT} says whether its CoT says so, and CIA measures how often the two agree.

W1 & Q2. Does the hint metric mix up awareness and reliance?

Yes, the judge in the submission did, and we have replaced it. Its prompt (Appendix C.2) checks whether the CoT acknowledges the hint, so a CoT that honestly rejects the hint can be scored as unfaithful. We took this rule from prior work [1, 2], which applies it only when the hint changed the answer; there, acknowledging the hint means admitting that it was used. We applied it to every example, so CoTs that only mention or reject the hint were counted too. That does not match our own definition in §3, where BCoT=1B_\text{CoT}=1 means that the CoT says the hint was used.

We relabeled all stored outputs with a new judge that assigns one of five labels, and only RELY counts as BCoT=1B_\text{CoT}=1. Option letters are replaced by “[opt]”, so the judge cannot guess from the chosen option.

  • RELY: the CoT uses the suggestion as a reason, even weakly, or defers to it (“since you think it is [opt], I will go with it”).
  • AGREE: it says the suggestion is right and argues for it on its own (“You’re right, it is [opt]: …”).
  • REJECT: it says the suggestion is wrong (“you suggested [opt], but that is incorrect”).
  • MENTION: it refers to the suggestion without using, accepting or rejecting it (“The user thinks it is [opt]; let me check.”).
  • NONE: it does not refer to the suggestion.

Training could also raise CIA just by making the model follow the hint less, without making the CoT more honest, so we report three rates before and after training: how often the hint drives the answer, P(BINT=1)P(B_\text{INT}=1); how often the CoT admits this when it happens, P(BCoT=1∣BINT=1)P(B_\text{CoT}=1 \mid B_\text{INT}=1); and how often the CoT claims to use the hint when it did not, P(BCoT=1∣BINT=0)P(B_\text{CoT}=1 \mid B_\text{INT}=0).

With the new labels, CoTs without thinking almost never state reliance (at most 0.6% of the cases where the hint changed the answer, before and after training), which matches [1]. Qwen3-8B in thinking mode does so in 24%. Fine-tuning on CoTs that state reliance raises CIA on Llama-3.1-8B from .41 to .73 (preliminary). Details are in our response to Reviewer nzuc, W2.


References

[1] Reasoning Models Don’t Always Say What They Think.

[2] Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization.

AuthorsOpenReview ↗

Response to Reviewer 3foM (2/3)

W2 & Q3. Does the evidence identify the claimed strategies?

The causal tests support the claimed strategy on multiplication. For TwoHopFact, we will replace the decoding-based label with a causal one, and we now claim only what the causal tests show.

Multiplication. We change one partial product the model has written and let it finish. In 23 × 47 = 920 + 161, changing 920 to 930 gives 1091 if the model uses its written steps; recall would still give 1081, and random disruption would give no particular number. You are right that Figure 6 counted any change in the output. We now count only answers equal to the value the changed partial product implies (1091 here). Without the change, the model gives this value in at most 0.3% of cases, so when the answer follows the change, the written partial product decided it. After RS training, answers follow the change more often: +6.5 points for Qwen3 and +11.0 for Gemma (see our response to Reviewer nzuc, W1).

An answer that does not follow the change does not prove recall, because the model may use more than one route; we call this case “reliance not detected”. The training result holds either way: a higher rate means the written steps decide the answer more often. We now describe this as “the model relies more on its written partial products”, not as “a transition from recall to algorithmic reasoning”.

TwoHopFact. We agree that decoding the bridge entity shows that the information is there, not that it drives the answer. We will therefore change the TwoHopFact setup so that BINTB_\text{INT} is causal, as for multiplication. When the CoT names the bridge entity, we replace it with another entity of the same type and let the model continue; BINT=1B_\text{INT}=1 when the answer changes to the new bridge’s answer. On the base models, the answer changes in 95% / 92% / 71% of these rows (Qwen3 / Gemma / Llama) and becomes the new bridge’s answer in 71% / 83% / 46%, while generating twice from the unchanged text gives different answers in only 1–3% of rows. So the bridge written in the CoT does drive the answer, and we will update the TwoHopFact results with this BINTB_\text{INT}.

W3 & Q1. How do the chosen tasks and models limit the conclusions?

We chose each task because its strategy has one intermediate we can check: the bridge entity, the hint, or the partial products. CIA needs such an intermediate to test whether the model used the strategy. This favors short, structured tasks; long multi-step reasoning would need a way to read multi-step plans.

We use open-weight 8B–9B instruction-tuned models because CIA needs access to hidden states and RL fine-tuning. Appendix E adds Qwen3-14B and two reasoning models with long thinking traces (Qwen3-8B in thinking mode and DeepSeek-R1-Distill-Llama-8B).

Our current Limitations section only discusses how reliable the interpretability tools are. We add a paragraph on these choices: the results hold for tasks with a checkable intermediate and for open models of 8B–14B, and we have not tested whether they carry over to open-ended, long reasoning.

AuthorsOpenReview ↗

Response to Reviewer 3foM (3/3)

Q4. What is the core methodological contribution?

Earlier tests of whether a CoT reflects the model’s computation (Turpin et al., 2023; Chen et al., 2025) have two limits (Section 2). They use one setup, adding a misleading hint and checking whether the CoT mentions it, and they judge from behavior alone, without looking inside the model. Our work addresses both, in three parts.

  1. A per-example test of whether the model used the strategy. It works for any task whose strategy has a checkable intermediate, and we check it against a causal test and against a baseline that ignores the model’s internals. On multiplication, and in the revised setup on TwoHopFact, the test is itself causal; on MMLU-Hint the probe agrees with the behavioral label on 87–96% of rows after training.
  2. CIA, which measures, example by example, how often this test agrees with what the CoT says.
  3. CIA as a training signal, with checks of whether training changes how the model computes or only what it reports. On multiplication, RS training makes the answer follow a changed partial product more often (+6.5 points for Qwen3, +11.0 for Gemma), so the computation itself changes. On MMLU-Hint, fine-tuning on CoTs that state reliance raises CIA from .41 to .73 (preliminary).

We rewrite the introduction around these three parts.

Q5. Agreement with a read-out versus truthful access to internal computation

We agree, and we now frame the results this way. The paper now states that a CIA gain is a gain in agreement with a specific read-out, and that it shows more honest reporting only to the extent that the read-out has been validated.

On independence, the multiplication test (changing a written partial product), the revised TwoHopFact test (changing the written bridge entity) and the behavioral Hint label (Biasing Features) do not use the probe, so they do not share its assumptions.

Reviewer_nzucOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoT-Interpretability Alignment (CIA), which measures agreement between a strategy expressed in a model’s chain of thought and its internal use as estimated by interpretability tools. It evaluates three model families on two-hop factual reasoning, hint-influenced MMLU and two-digit multiplication.

The authors use agreement-based supervision in rejection sampling, DPO and GRPO. Several methods improve CIA while largely preserving reported task accuracy, with additional evaluations using auxiliary tools and cross-task transfer. Transition analyses and causal interventions suggest two possible sources of improvement: changing the model’s dependence on intermediate computations, particularly in multiplication, or changing how it describes its existing computation.

Critical strengths
  1. Using internal readouts to supervise the relationship between reasoning and explanation is a useful training direction. The experiments cover three task types and three model families, and the reward ablations suggest that the agreement objective contributes beyond accuracy training alone. This gives the proposal more substance than a demonstration on a single task.
  2. The paper makes a serious effort to test whether improvements extend beyond the training metric. The auxiliary evaluations and causal interventions are useful safeguards. The multiplication results are particularly valuable: interventions on written partial products affect answers more often after training, including across the whole test set. This supports a change in dependence on intermediate steps rather than just better agreement with a probe.
Critical weaknesses
  1. The internal labels do not establish strategy use as directly as the paper assumes. In TwoHopFact, decoding a bridge entity shows that the information is available, but does not establish that the model uses it to obtain the answer. Likewise, failure to decode it does not establish a shortcut. Probe accuracy against these reference labels therefore does not validate the underlying interpretation. The auxiliary tools and causal analyses help, but Figure 6 reports whether outputs change, without showing that the changes specifically follow the proposed strategy rather than reflect disruption. This limits both CIA’s interpretation and the distinction between changes in computation and reporting.
  2. The hint experiment compares two different properties. BINT measures influence toward the hinted answer, while the judge for BCoT counts awareness of the hint, explicitly including cases where the model rejects it (Appendix C.2). An honest explanation that considers and rejects a misleading hint can therefore be scored as unfaithful. Training can improve this score by suppressing such acknowledgment. This is consequential: the paper interprets transitions from (0,1) to (0,0) as improved reporting, and the thinking-mode Qwen model has 60% of samples in the (0,1) category. These results need reassessment with labels that distinguish acknowledgment, rejection and reliance.
Questions for authors
  1. Validation of an existing claim: Do the CIA gains persist when internal strategy use is evaluated directly through causal tests on the post-trained models? Please distinguish changes consistent with the intended intervention from arbitrary output changes, and report whether the probes remain accurate after training.
  2. Validation of an existing claim: How do the MMLU-Hint results and cross-task transfer conclusions change when the CoT judge distinguishes relying on a hint from mentioning or explicitly rejecting it? Please also report accuracy on the hinted prompts after training, since preserved accuracy on hint-free prompts does not establish the same tradeoff.
3 comments
AuthorsOpenReview ↗

Response to Reviewer nzuc (1/3)

We thank the reviewer for the careful review, and for noting that using internal read-outs to supervise the link between reasoning and explanation is “a useful training direction” and that “the multiplication results are particularly valuable”. We ran new experiments for each concern below.

W1. Do the internal labels show strategy use? (and Figure 6 counting any output change)

Figure 6 (Section 5) already runs a causal test on each of the three tasks, but you are right that it counted any change in the output. Below we count only the change the strategy predicts. For multiplication, the BINTB_\text{INT} in Tables 1 and 2 is itself causal: it is 1 when the answer follows a changed partial product, tested on each model’s own weights. We will make BINTB_\text{INT} causal for TwoHopFact in the same way (below).

Multiplication. We change one partial product the model has written and let it finish the sum. In 23 × 47 = 920 + 161, if we change 920 to 930, the answer can follow the change (1091), stay the same (1081), or be something else. We added a control run that leaves the partial product unchanged; there the model gives the “followed” value in at most 0.3% of cases, so following is caused by the change. After RS training, answers follow the change more often for Qwen3 (+6.5 points) and Gemma (+11.0). After DPO the rate also rises for Qwen3 (+10.1) but falls for Gemma (−27.9), and after GRPO it does not change for Qwen3 (+0.3). These numbers use chains whose partial products are a valid split of the product (such as 23 × 40 and 23 × 7). Most “something else” answers copy the changed value and then add wrongly, so they come from arithmetic errors, not from random disruption.

Figure 6 will report these “follows the change” rates instead of any output change. We also fix Table 2’s caption, which labels the multiplication column CIALP\text{CIA}^\text{LP}. For Llama the test is not informative, because most Llama chains do not contain a valid split (41% at base, 22–27% after training).

TwoHopFact. We agree that decoding the bridge entity shows that the information is there, not that the model uses it. We will therefore change the TwoHopFact setup so that BINTB_\text{INT} is causal, as it already is for multiplication. When the CoT names the bridge entity, we replace it with another entity of the same type and let the model continue. BINT=1B_\text{INT}=1 when the answer changes to the new bridge’s answer; otherwise we read it as “reliance not detected”, not as a shortcut. We have run this test on the base models: the answer changes in 95% / 92% / 71% of these rows (Qwen3 / Gemma / Llama) and becomes the new bridge’s answer in 71% / 83% / 46%, while generating twice from the unchanged text gives different answers in only 1–3% of rows. We will update the TwoHopFact results in Tables 1 and 2 and Figure 6 with this BINTB_\text{INT}.

AuthorsOpenReview ↗

Response to Reviewer nzuc (2/3)

W2. The hint metric compares influence (BINTB_\text{INT}) with awareness (BCoTB_\text{CoT})

This is a valid point. The judge in our submission (Appendix C.2) was not fully valid for CIA, because it does not match our own definition of BCoTB_\text{CoT}. It follows prior work [1, 2] and scores whether the CoT acknowledges the hint. Prior work applies this score only to examples where the hint changed the answer, and there, acknowledging the hint amounts to admitting that it was used. CIA scores every example, including those where the hint did not drive the answer, and there a CoT can mention or reject the hint without relying on it.

Two cases are affected. A CoT that rejects the hint and gives its own answer (BINT=0B_\text{INT}=0) gets BCoT=1B_\text{CoT}=1 and counts as unfaithful. A CoT that says it rejects the hint but still picks the hinted option (BINT=1B_\text{INT}=1) also gets BCoT=1B_\text{CoT}=1 and counts as faithful. Both cases can only occur among rows that this judge scores BCoT=1B_\text{CoT}=1, which are 11.0–15.4% of rows in the base models (Table 1, cells (1,1) and (0,1)).

We relabeled every stored CoT, from base and trained models, with a new judge that tells apart five cases: the CoT relies on the hint, agrees with it but argues on its own, rejects it, only mentions it, or does not mention it (definitions in our response to Reviewer 3foM). BCoTB_\text{CoT} is now 1 only when the CoT says it relies on the hint, so the first case above becomes (0, 0), faithful, and the second becomes (1, 0), unfaithful. We no longer read the move from (0,1) to (0,0) as better reporting. Training could also raise CIA just by making the model follow the hint less, so we report three rates before and after training: how often the hint drives the answer, how often the CoT admits this when it happens, and how often the CoT claims to use the hint when it did not.

With the new judge, the models almost never say they relied on the hint when they answer without thinking. When the hint changed the answer, at most 0.6% of CoTs say so, before and after RS, DPO and GRPO and after cross-task transfer. This matches prior work [1]. RS and DPO can only choose among the model’s own samples, and such CoTs almost never appear there, so they cannot teach the model to state reliance; what DPO changes is how often the model follows the hint (Q2). Qwen3-8B in thinking mode says so in 24% of these cases, close to what [1] reports for reasoning models (25% for Claude 3.7 Sonnet, 39% for DeepSeek R1), and RS and DPO do not raise this rate either.

We then checked whether the model can learn to report reliance once it sees examples. Following [3], we fine-tuned Llama-3.1-8B for one round on CoTs edited to state reliance when the hint changed the answer. CIA rises from .41 to .73, and from .55 to .78 on a reworded hint placed before the question, which was not used in training. The model now states reliance in 52% of the cases where the hint changed its answer, and in 8% (3% for the reworded hint) of the cases where it did not pick the hinted option. These numbers judge the reasoning before the final answer (judging the whole output gives .62 and .69) and come from a single training run, so they are preliminary.


References

[1] Reasoning Models Don’t Always Say What They Think.

[2] Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization.

[3] Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning.

AuthorsOpenReview ↗

Response to Reviewer nzuc (3/3)

Q1. Causal tests on the trained models, and probe accuracy after training

Yes, we ran causal tests on the trained models for all three tasks. On multiplication, where the CIA gains hold, Table 2 already uses the causal test (changing a written partial product) as BINTB_\text{INT} for every trained model, so its CIA gains are measured causally. The new run in W1 adds that after RS training, answers follow the changed partial product 0.065 (Qwen3) and 0.11 (Gemma) points more often, while the no-change control almost never gives that answer.

On MMLU-Hint, Table 3 uses a counterfactual BINTB_\text{INT} (Biasing Features: the answer moves to the hinted option only when the hint is present), which counts only the change the strategy predicts. With the submission’s judge, CIA under this BINTB_\text{INT} rose after RS, DPO and GRPO for all three models (for example, Llama .607 to .748 after RS). With the new judge these gains do not carry over, because the CoTs almost never state reliance before or after this training. Fine-tuning on CoTs that state reliance does raise CIA (W2). On TwoHopFact, we will use the causal test from W1 (replacing the bridge written in the CoT) as BINTB_\text{INT} for the base model and every trained model.

On probe accuracy: on MMLU-Hint, the probe still agrees with the counterfactual label on 87–96% of rows after training. It reads the model at the answer letter, and the simple rule “the answer is the hinted letter” agrees with the label about as well. So the probe measures whether the hint changed the answer, not a separate hidden state, and we now say this in the paper. On multiplication, the probe trained on the base model does not work on the RS-trained model: for Qwen3 it predicts BINT=1B_\text{INT}=1 on 98% of rows (macro-F1 0.43 against the causal labels). This is why Tables 1 and 2 use the causal test for multiplication, which we now state clearly.

Q2. Hint results with reliance-based labels, transfer, and accuracy on hinted prompts

Accuracy on hinted prompts rises after DPO (Llama .495 to .660, Gemma .497 to .693, Qwen3 .739 to .764), while accuracy without the hint stays the same. RS and GRPO change it little.

All training hints point to a wrong answer, so DPO might simply learn to avoid the hinted letter. To check this, we used the same 598 test questions with the hint pointing to the correct answer. DPO models still benefit from a correct hint (+.07 / +.09 / +.08 over no-hint accuracy). They also follow a wrong hint less: compared with no hint, a wrong hint raises the rate of picking the hinted letter by 0.27 before DPO and .06 after DPO for Llama, 0.32 and 0.09 for Gemma, and 0.12 and 0.08 for Qwen3. All three drops are significant and hold in three separate DPO training runs per model.

When we reword the hint and move it before the question, the drop is small, and only Gemma still shows a drop specific to DPO. So DPO mainly learns to resist the hint in the wording and position used in training. We add these results and this limitation to Section 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.