- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The contribution of this paper is to study whether different types of dual-persona prompting affect information access between those personas.
Critical strengths
- The question of whether getting an LLM to roleplay multiple characters generates meaningful information separation between those characters is interesting, and the secret retrieval idea is also interesting.
Critical weaknesses
-
This paper is close to unreadable, and the 'tulpamancy framing' that the authors introduce as one of their non-LLM contributions adds little scientifically while making the paper hard to understand.
-
As far as I can tell, the scientific question being examined is whether dual-persona prompting (i.e. getting an LLM to play multiple roles by prompting) induces constraints on information routing via attention. However, the main causal experiment intervenes on residual stream activations after the final block, where no further attention can take place, and so by design cannot test the hypothesis. The authors in fact acknowledge this in section 4.5.
Even worse, the intervention is to zero the entire residual stream on the final layer. This makes the model's prediction uniform! Figure 3 is evidence of this: Llama 3 has a 128k vocab size, and so turning its next-token distribution into a uniform distribution implies a delta NLL of log(128256) ~= 11.8, which accounts for the vast majority of their results.
- Other non-causal evidence is too thin to substantiate the mechanistic claim "persona framing operates merely as a weak terminal-layer constraint on a single residual stream".
Questions for authors
Why use a distinction between tulpa/host dialogue and speaker a/b rather than single-persona vs dual persona prompting? Why isn't single-persona prompting the appropriate baseline?