- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper attempts to train multilingual (transfer) probes to detect hint-following in the context of chain-of-thought (CoT) reasoning. The authors take multiple-choice QA (MCQA) items and add a hint pointing towards a wrong answer; a model is deemed to be following the hint if it selects that wrong answer. The authors then train probes on English MCQA and use them on 1) unseen English items and 2) unseen non-English items. The probes are trained at various points in the CoT - the prompt, early CoT, and post-CoT. The probes only work well in the post-CoT setting, and also only really on Apertus and EuroLLM. They do transfer somewhat across languages, but the post-CoT probes aren't consistently better than an LLM judge, suggesting they just use info available in the text, rather than privileged internals-only info. They also don't work for steering. The intended conclusions are not clear to me.
Critical strengths
I think that this question - can we detect unfaithfulness in chain-of-thought reasoning - is valuable and interesting.
Critical weaknesses
Low impact due to low novelty, narrow focus, and inconclusive / negative results: This paper is, as it says, an audit. It digs a bit deeper than prior work into multilingual CoT monitoring probes, which is already a rather niche topic. It trains probe, but (Figure 1, Qwen) they only work well on 2/3 models. The authors then spend many pages just trying to determine if these CoT probes actually do transfer, and if they outperform a text-based LLM-judge baseline (they do not conclusively do the latter). The probes are also not useful for steering. In sum, this paper describes an inability to train probes for CoT hint-following.
To be clear, I don't want to claim that negative results are unimportant. But results like these, which amount essentially to "we failed to train a multilingual, causally effective CoT probe" are not very useful for the community at large. As discussed in the next point, they reflect issues with the authors' training setups more than they reflect some fundamental impossibility of training such probes. In the future, I would suggest that the authors focus more on finding setups that do work, than describing those that don't.
Experimental design issues: I think that at least some of the issues encountered by the authors are mainly due to experimental design. In particular, I don't think it makes much sense to probe at the prompt position (the model has probably not decided whether to hint-follow at that stage). Similarly, at the post-CoT position, it makes sense that the probes should do well but rely on information in the text; even if you cut off the model's final answer, surely it makes clear if it's going to output the final answer or not in its reasoning (either explicitly, with the answer choice, or not). I'm frankly only surprised that they don't do better
I think that what you probably want to detect is something like "the model is using the hint when generating this line of reasoning". That is, it's something that happens actively during reasoning, rather than at the end or necessarily in the very beginning. It might not even be easy to localize. But this is what I think is most likely to be findable, and not just be an artifact of answer presence.
This would also solve your steering issue: you probably can't steer using these probes because they're just picking up answer presence. But if you were to steer on the probes when your model was actively generating hint-biased reasoning, this would probably work much better. Perhaps it's worth investigating the raw reasoning traces in greater detail to see whether a) the intended answer is obviously visible (probably, since the LLM judges often pick it up) and b) you can somehow identify lines of biased reasoning (perhaps an LLM can find these too)
Questions for authors
Clarification:
- What are the baseline accuracies of these models in English / other languages on these MCQA questions? Do you filter to only questions that the models get correct in the first place?
- I find it difficult to understand the AUROC values that the authors report in the paper, and I struggle in particular to understand how they are framed. What constitutes, in the authors' opinion, a "good" AUROC on this task?
Suggestions:
- This paper is a bit light on content; it gets deep into the weeds of controls and experimental details for experiments with ultimately negative results. Some of these would be better left in an appendix, freeing up room for more
- More figures in the main text, fewer big tables
- Use AI less; this paper reads as quite AI-written. This is visible in the paper's peculiar phrasing, like "An English probe can detect hint-following in other languages. Sharing the same questions across languages can fake transfer (up to +0.16 AUROC)." This is quite confusing for humans - what it means to "fake transfer" is not immediately obvious, and this is an unclear way of expressing this concept in the intro. Many other cases of this exist in the paper.
- The language / LEACE experiments are not very informative, and can probably be cut