- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors investigate whether a belief probe continues to be reliable if a model engages in deception. They study this research question in the controlled setting of "One-Night Ultimate Werewolf". The probed "belief" is the role a LLM adopts within this game. The authors induce deception via steering and find that under strong steering the probed "belief representation" changes.
Critical strengths
The general research question of studying the conditions under which the "beliefs" of a LLM can be accurately measured, is timely and interesting. "One-Night Ultimate Werewolf" is a promising testbed for this research.
Critical weaknesses
Unclear Motivation for the work
I do not understand the motivation for this work. In the introduction, the authors state they would like to test whether the belief of the LLM can still be accurately measured when the LLM deceives. This is a reasonable research question. However, I do not understand why the authors choose to induce deception via steering. Steering is known to have strong side effects on the output quality and the activation distribution. I therefore do not find it surprising that strong steering can lead to a change in the model's activations that affect a probe. The much more natural research avenue would have been to study naturally occurring instances of deception, e.g. during role-play or when the model pursues a goal given in context and deception is instrumentally useful. Why do the authors focus on steering instead?
Conceptual and Methodological issues and Soundness
The authors suggest that deception steering changes the model's internal belief about its role. However, the evidence they provide seems perfectly compatible with a much more mundane explanation: The belief probe tracks the stated role of the model and not necessarily the internally believed role of the model. On the training dataset, the internally believed role of the model and the externally stated role of the model are identical. However, after steering with the deception direction, the model might state having a different role and this leads to a large KL divergence for a probe tracking the stated role of the model. More generally, strong steering leads to a large change in the activations. This might also lead to a measurable KL-divergence. For these experiments to be convincing, more evidence is needed to show that (i) the identified representation indeed corresponds to the models internal belief about its role and not just its stated belief and that (ii) steering along other directions with a similar strength leads to a much lower KL divergence, i.e. a control condition.
Presentation
I found it very hard to follow the paper. Formulations are often very confusing and essential components are not clearly explained. Just to name a few:
- The paper is centered around the game "One-Night Ultimate Werewolf" but this game is not clearly explained in Section 3.
- The authors never mention which layers they steer when activation steering.
- Example of a paragraph that is extremely confusingly written and I was not able to understand it: "The readout’s gain turns this overlap into a measurable shift, and the delta determines where the shift lands. It enters through the identity axes of the two roles whose win conditions reward lying. The lie grazes the belief representation; the readout amplifies the graze" (Line 61-63).
Questions for authors
Clarification
- Why is it relevant that sometimes the belief of the model and the ground truth diverge when its card is swapped during the night? You mention multiple times that this is important (e.g. "belief and truth are dissociated" in Line 102) but I do not understand why. If the model's internal state is not affected at all by the external ground truth, why does that matter? Accordingly, I do not understand "Control 1" (Line 246). If the LLM has no way of knowing its true role, how could the belief probe possibly track its true role?
- Please explain the data on which you train the deception probe in subsection 3.3 in more detail.