- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a geometric diagnostic of cue-induced bias in hidden states.
Using the paired difference between cued and uncued runs, it classifies biases into those that a single fixed direction can correct and those that require per-item correction; both cognitive-bias datasets fall in the second class.
For this second class, the authors fine-tune the model to answer each biased question the way it answers the unbiased version of the same question (“twin distillation”). This is reported to improve accuracy on bias types not seen during training.
Critical strengths
The paper shows a clear empirical finding about cognitive biases. Across models, the hidden-state shift caused by a cognitive-bias cue points in a different direction for each question, with no direction shared across questions. This offers a plausible explanation of why steering works for biases such as sycophancy, BBQ and refusal but not for cognitive biases, and the coherence measure could serve as a simple check before applying steering.
Critical weaknesses
The dataset design introduces systematic ambiguity in what the displacement measures. The paper treats the cued-minus-uncued difference as the bias, but what differs between the two arms varies widely across datasets: the authors’ corpus rewrites the whole scenario (no shared sentence), discrimination swaps a single word, BBQ removes context, and some datasets also change the options. This is compounded by several further mismatches: datasets contain very different numbers of bias types yet each appears as a single point; the 22 shared biases with Malberg et al. are matched by name only... The cue-swap controls (Exp 3/4) are run only within the authors’ own corpus and do not address cross-dataset differences. Most central analyses rely on the self-built, LLM-generated corpus, whose construction differs most from the others, and the paper does not explain why it was needed beyond the published Malberg et al. set.
The presentation of the work has serious shortcomings. The figures are poorly presented and hard to read, and a large amount of experimental detail is deferred to the appendix: roughly 20 experiments are referenced, but the main text lacks the details needed to assess them. This suggests the paper tries to cover too much for a single submission; perhaps splitting into more focused papers, the work would be a better fit for this venue.
Questions for authors
The title and abstract frame the work around chain-of-thought reasoning, but the experiments read hidden states at a forced single-letter answer position. Is any reasoning chain generated before the answer? If not, how do the findings relate to chain-of-thought?
Twin distillation seems to be related to bias-augmented consistency training (Chua et al., 2024), which also fine-tunes a model to reproduce its unbiased-prompt output on biased prompts and reports generalization to held-out biases. Could the authors discuss this relation, and if possible compare against it?