- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- This paper uses activation steering to make language models say that an event happened when the given text does not mention it, and analyzes the resulting hallucinations.
- The authors observe that false reports can come with fabricated supporting reasons, that a previous answer changes how much the model benefits from checking the record, and that the model's content choices can stay stable even when its answer changes.
Critical strengths
- The paper includes many control experiments and analyses, such as norm-matched random vectors and vectors from other events.
- It uses a controlled dataset with known labels.
Critical weaknesses
- The paper is very hard to read.
- Many terms are not explained. For example, "present-minus-absent residual direction" appears in the abstract without explanation and is only defined in Section 3.2.
- The setting is not natural, and the motivation is unclear. The paper does not explain how these findings can be used or how they would help future model development.
- Vectors from other events (Section 4.3, Figure 2) and from unrelated word pairs (Section 4.10, Figure 4) have almost the same effect. This suggests the intervention may not change the model's answer about the event itself, but simply make it more likely to say "Yes."
- The human evaluation is small (two raters, 16 events), and agreement between raters drops in the additional audit (κ = 0.667 → 0.494).
Questions for authors
- In what real-world situation would a model's internal state be shifted in this way? How can these findings be used?
- If many different directions have the same effect, what does the intervention actually change?
- Appendix A.1 shows that giving a "Yes" draft without steering gives the same two-pass result. What does steering add to this experiment?