- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies role spoofing in LLMs. In role spoofing, text in a tool output makes the model identify that text as a user message. The attack uses fake special tokens (fake structure) and user-like text (fake content). The authors show that this attack fools all tested frontier models in AgentDojo and in a simple tool-use testbed. They use a “Who Said What” task to show that models use both content cues and structure cues to find the role of a message. In a two-layer toy transformer, adversarial training on content-based injections moves the model to structure cues, but the model still accepts fake special tokens.
Critical strengths
- Clear analysis of role tracking. The paper divides role tracking into content-based and structure-based mechanisms. The “Who Said What” task tests each mechanism separately. The toy model and its attention patterns give a possible reason why adversarial training does not stop role spoofing.
- Simple and low-cost defense. The role probes do not need injection data or coherent conversations for training. On held-out domains and injections, they have a higher true positive rate at low false positive rates than the probes of Ye et al. (2026) and classification probes. The probes also detect injections that fool model self-attribution.
Critical weaknesses
- The defense has no external baselines. The authors compare their role probes only with other linear probes that they train. They do not compare with simple input sanitization, for example escaping or removing fake special token strings in tool outputs. The paper itself states that tools can use this type of filtering (Sec. 2.2), and the AgentDojo default harness already escapes tool outputs (Appendix B). The paper also does not compare with existing detectors or training-based defenses, for example PIShield, SecAlign, or StruQ. Also, the evaluation reports only detection ROC curves. It does not show the ASR and task utility of an agent that uses the probes. Because of this, the practical value of the probes is not clear.
- The “Who Said What” comparison is not balanced. For fake structure, the authors test a suite of fake special token styles and use the best one for each model (Sec. 3.2). For misleading content, they use only one construction. Thus, the structure results are a best-of-N value, but the content results are not. Appendix D also states that the results change with the wording of the query. In addition, the task measures the role that the model states when it is asked, not the role that controls its actions. Because of this, the claim that models use different “mixtures” of content and structure cues is not supported.
- The toy result follows from its design. Neither instruction tuning nor adversarial training in the toy setting contains fake special tokens. Thus, the model has no training signal to reject fake starts, and Appendix A.2 states this. The result that the model accepts fake structure after training is therefore expected. The Qwen3.5-27B experiment shows only that one type of training does not transfer to the other type, which is also expected. The paper does not show that frontier models have the same training conditions. Because of this, the explanation for frontier models is speculative.
Questions for authors
See weaknesses.
2 comments
Thank you for the response and the new experiments. W1 is largely resolved: the PIShield comparison, the benign-quotation test, and the ASR/utility results clarify the practical value of the probes. W2 and W3 are resolved mainly by narrowing the claims rather than by new evidence. I accept the weaker framing of the "mixtures" claim and of the toy setting as one plausible explanation. I am raising my score by one step, on the condition that the abstract and introduction are revised to match these narrower claims and the new results are included.
W1.
W2.
W3.