acceptodds
Under review as a conference paper at ICLR 2027

Two Runs, One Law: An Entropy Balance for Reasoning Bias

Abstract

We investigate how chain-of-thought reasoning handles biases by analysing its geometry. We show that reasoning bias is a two-run quantity, which can only be known as the difference between a cued run and its uncued twin. Taking as a mediator the vector a layer passes forward, its hidden state, we apply VanderWeele's four-way decomposition to split the bias into a natural indirect effect the state carries, a natural direct effect the later computation supplies, and their interaction. The depth at which the state becomes a near-complete mediator of the cue's effect we call the crossover, found in nine models: 7B to 70B, base and mixture-of-experts. We measure the second moment of the cued-minus-uncued differences across items and read two quantities from it, the number of directions the correction occupies and the share carried by one direction common to all items. Across nine bias datasets and six models these two quantities sort biases into those that a single fixed direction can correct and those that require a per-item correction; both published corpora of cognitive biases fall in the second class, on every model, and neither the amount of cue text nor the size of the state's response is what decides it. The crossover says where a correction belongs and why it must be per-item. Only the uncued twin supplies a per-item target, so we distill it into one pass. This counterfactual twin distillation on the uncued run's answers improves held-out balanced accuracy on cued prompts by 11 to 24 points across 30 Kahneman–Tversky families in five models, without prior knowledge of the target bias type. The two-run geometry holds for MMLU position bias, 11 BBQ categories, and six further published bias datasets.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought, bias mitigation, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nYwDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a geometric diagnostic of cue-induced bias in hidden states.

Using the paired difference between cued and uncued runs, it classifies biases into those that a single fixed direction can correct and those that require per-item correction; both cognitive-bias datasets fall in the second class.

For this second class, the authors fine-tune the model to answer each biased question the way it answers the unbiased version of the same question (“twin distillation”). This is reported to improve accuracy on bias types not seen during training.

Critical strengths

The paper shows a clear empirical finding about cognitive biases. Across models, the hidden-state shift caused by a cognitive-bias cue points in a different direction for each question, with no direction shared across questions. This offers a plausible explanation of why steering works for biases such as sycophancy, BBQ and refusal but not for cognitive biases, and the coherence measure could serve as a simple check before applying steering.

Critical weaknesses

The dataset design introduces systematic ambiguity in what the displacement measures. The paper treats the cued-minus-uncued difference as the bias, but what differs between the two arms varies widely across datasets: the authors’ corpus rewrites the whole scenario (no shared sentence), discrimination swaps a single word, BBQ removes context, and some datasets also change the options. This is compounded by several further mismatches: datasets contain very different numbers of bias types yet each appears as a single point; the 22 shared biases with Malberg et al. are matched by name only... The cue-swap controls (Exp 3/4) are run only within the authors’ own corpus and do not address cross-dataset differences. Most central analyses rely on the self-built, LLM-generated corpus, whose construction differs most from the others, and the paper does not explain why it was needed beyond the published Malberg et al. set.

The presentation of the work has serious shortcomings. The figures are poorly presented and hard to read, and a large amount of experimental detail is deferred to the appendix: roughly 20 experiments are referenced, but the main text lacks the details needed to assess them. This suggests the paper tries to cover too much for a single submission; perhaps splitting into more focused papers, the work would be a better fit for this venue.

Questions for authors

The title and abstract frame the work around chain-of-thought reasoning, but the experiments read hidden states at a forced single-letter answer position. Is any reasoning chain generated before the answer? If not, how do the findings relate to chain-of-thought?

Twin distillation seems to be related to bias-augmented consistency training (Chua et al., 2024), which also fine-tunes a model to reproduce its unbiased-prompt output on biased prompts and reports generalization to held-out biases. Could the authors discuss this relation, and if possible compare against it?

Reviewer_prByOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the bias of a language model as the difference between two runs: a prompt with a cue and the same one without one. The cue can be a word or phrase, or the way the question is structured. Using nine datasets, they construct pairs of questions (with and without a cue), and read the model’s internal state at the moment of responding, then subtract the states of each pair from one another. For each dataset, a second-moment matrix is calculated using these subtractions, and from it, two quantities are derived: the rank (how many distinct directions the subtractions occupy) and the coherence (what percentage points in a direction common to all pairs). Some datasets (sycophancy, BBQ, moral omission) have high coherence and low rank, and in those cases, a single direction would suffice to steer the response. Others, including the two cognitive bias corpora, have very low coherence (below 13%) and high rank (41 directions).

Critical strengths
  • There is a well-defined metric. Treating bias as the difference between two runs (cued and uncued) is a useful idea. The two measures (PR and κ) are well defined and easy to calculate.
  • A large number of experiments, datasets and models.
Critical weaknesses
  1. The use of the term "bias" is inadequate

    The paper treats behaviors of different natures as "biases". Refusal to answer is a learned behavior, not a bias. Option position and survey wording are examples of sensitivity to wording. Judge authority is a false citation attached to a response. Furthermore, the low coherence group (Figure 1) is not limited to the cognitive corpus, it includes other datasets. This undermines the distinction suggested in the abstract between "cognitive biases" and "the rest". The title and Section 5 ("across biases") promise a classification of biases, but what is actually classified are datasets. The authors acknowledge this in the "Limitations" section ("the datasets are not a sample of any population"), and that statement contradicts the abstract.

  2. Inconsistent and unjustified measurement across layers

    The summary by depth varies across experiments: PR peak (Experiment 2), average of seven blocks (Experiment 3), layer of maximum κ\kappa (Experiment 9), and three layers (Experiment 11). The classification in Figure 1 is based on the most favorable layer for each dataset, which for the cognitive corpus is likely one of the most superficial layers (κ\kappa decreases with depth), whereas in Section 7 the correction is set at the crossover, at a greater depth. It is not specified whether this rule was established prior to the measurement. This makes it difficult to compare the figures across experiments (PR 41, 47, 23, 14–20 for the same corpus).

  3. The rank appears to depend on how the dataset was constructed

    Low-dimensional datasets (sycophancy, judge authority) add a fixed template, and Experiment 4 itself shows that adding a fixed phrase to the cognitive corpus causes the participation ratio (PR), a measure of effective rank, to drop from 23–26 to 2.3–4.2. In other words, the same content changes class simply because of the way the cue is presented.

  4. Editorial

    The text is very dense and contains many cross-references to experiments and results that are mentioned before they are defined.

Questions for authors

Please respond to the weaknesses listed above (W1–W3).

Reviewer_R8bQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats a bias as the difference between two forward passes: a prompt with a cue, and the same question without it. At the forced-answer position it takes the cued minus uncued hidden state for each item and reads two numbers from their second moment: participation ratio (how many directions) and coherence (how much of it is one shared direction). On a new 741-pair cognitive-bias corpus (30 families) the displacement is high rank and low coherence (PR 41 on Llama-3.1-8B-Instruct, peak PR 39 to 57 on six models, Table 2). With thresholds fixed in advance (kappa > 0.30, PR < 10), sycophancy, BBQ and moral omission land in the "one direction is enough" region and both cognitive-bias corpora do not (Figure 1, Exp 9). The paper then scores SAE atoms and a fitted subspace against this, finds a "crossover" depth where the transplanted state carries most of the cue effect (Table 3), and proposes counterfactual twin distillation: a LoRA student sees the cued prompt and is trained to match the model's own answer on the uncued prompt. This gives +11.2 to +23.7 balanced accuracy on 10 held-out families across five models.

Critical strengths
  1. The (coherence, participation ratio) plane is a simple and useful check to run before picking a correction. Figure 1 puts nine datasets and refusal in one picture, and it matches what we already know: refusal (kappa 0.49 to 0.85, PR 1.3 to 3.7) is where one direction works, sycophancy and BBQ are close to it, and the cognitive-bias sets are far away. People often try steering vectors on every bias, and this gives a cheap way to predict when that will fail. The SAE scan points the same way: the best atom scores 4.44 against a fitted optimum of 73.47, and no atom is above 5 (Exp 11, Table 6).

  2. The reporting is honest. Thresholds were registered before measuring, sample size is matched across models, and failures are reported as failures (Exp 15, Exp 18, and the registered null of Exp 6). That makes the results that do hold easier to believe, and I can judge the claims from the paper itself. The practical result is also strong: twin distillation gives +11.2 to +23.7 balanced accuracy on five models with three seeds, and every interval excludes zero.

Critical weaknesses

The main classification does not have causal support yet, and the main corpus has a construction confound. The registered causal test (Exp 18) held on only 1 of 3 checkpoints, and the post hoc version rests on BBQ alone. Moral omission is "admitted" on 6/6 checkpoints even though its baseline flip rate is 0.000 to 0.004, so coherence can be high where the cue does not change behavior at all. Section 8 still says the two coordinates "prescribe the correction". Separately, in the cognitive-bias corpus no uncued prompt is a subsequence of its cued prompt (lines 136 to 137), and 287 of 300 twins contain sentences the cued prompt does not have. But Appendix J (line 1319) says the arms "differ only in the inserted cue". So the displacement may mix the cue with item-specific rewording, and rewording alone could produce high rank and low coherence. There is no paraphrase-only control. Malberg et al. is insertion-type and is still rejected, so the confound probably does not explain everything, but the PR 41 headline comes from this corpus. I think the diagnostic is useful as a predictor. The claims just need to be scoped to what Exp 18 shows, since the classification is the paper's main contribution.

Questions for authors
  1. How exactly are the cued and uncued arms built, given lines 136 to 137 vs Appendix J? Could you run a paraphrase control (uncued arm vs a cue-free paraphrase of it) and report PR and kappa?
  2. In Exp 17, is the cue span marked at test time? What happens with no marking, or with a span from a detector?
  3. Could you run the same transplant on a contrast that is not a bias, e.g. editing a fact so the correct answer changes? If the crossover shows up at the same depth, it may just mark where the last position finishes reading the context, and say little about bias.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.