- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether the input format of LLM agents can affect security. The authors show that simple formatting choices such as XML vs. JSON or changes in field order and nesting can change attack success. To understand why, the paper examines how clearly the model distinguishes trusted instructions from external content. The authors then directly modify the model’s internal representation of external content to make it look more like trusted content. This makes the model more likely to follow the attacker, showing that source separation itself contributes to the security differences. The paper's main conclusion is that input formatting is an important part of an LLM agent’s security design.
Critical strengths
-
The paper identifies a practical security issue. The main finding is interesting, that is, changing the format of the same content can change attack success, even when other factors are fixed. This shows that formatting should be considered when designing and evaluating LLM agents.
-
The paper investigates the underlying reason why it happens. The authors show that attacks tend to be more successful when the model less clearly distinguishes trusted instructions from external content. They further intervene on the model’s internal representation and show that making external content appear more like trusted content increases attack success.
Critical weaknesses
-
The main explanation still needs more evidence. The paper shows two things: first, renderings with weaker source separation tend to have higher attack risk; second, directly changing a model’s internal representation of external content can increase attack success. But these two results do not complete the causal chain. The intervention does not change the rendering itself, and the paper does not show that a rendering becomes riskier because it weakens source separation. In other words, source separation may matter for attack behavior, while still not being the main reason different renderings behave differently. This is one of my major concerns because explaining the rendering effect is the key scientific contribution of the paper.
-
The strongest evidence comes from an artificial representation edit, but its relation to naturally occurring rendering differences is unclear. The causal experiment manually pushes hidden states in a chosen direction at a chosen layer and strength. The results show that the model is sensitive to this direction, but it does not show that natural modifications (e.g., changing XML to JSON) change the model in the same way or by a comparable amount. From my perspective, the experiment is convincing as evidence that source information can affect behavior, but it becomes much weaker as an explanation of the original rendering phenomenon.
Questions for authors
-
What evidence directly shows that the rendering effect is mediated by source separation?
-
What evidence supports the insight that naturally occurring rendering changes alter the model through the same source-related direction?
4 comments
Follow-up on our response
Dear Reviewer,
Thank you for your clear account of the causal chain and for finding the representation edit convincing evidence that source information can affect behavior. Section 4.4 projects the change from XML to nested JSON onto the edited direction, and Appendix D.3 aligns this shift across the two renderings. Section 3.5 applies the probe that defines the edit to fresh, position-matched item pairs. Section 6.1 relates source separation to the rendering effect.
If these analyses address your concerns, we would appreciate a re-evaluation of the submission. Thank you very much.
Thanks for the response
Thank you for the response. It has largely addressed my second concern but the first one remains. In Section 4, the authors show that switching from XML to JSON changes the internal representations, but when they align these changes between the two renderings, the difference in attacker margin changes very little (0.003 vs. 0.668). This seems to imply that changes in source separation does not reflect why the two renderings lead to different attack outcomes. It is good that the authors tone down the claims, but I still feel that the main question of the explanation remains.
Response to follow-up
Thank you for noting that the second concern is largely addressed.
At layer 16 of Qwen3-14B, the attacker-margin gap between XML and nested JSON is carried by the structural tokens that delimit the sources. The alignment in Section 4.4 moves one direction at the message tokens. Appendix D.5 copies the full layer-16 states instead. We use 46 position-matched XML and nested-JSON attack inputs whose message and question tokens are identical, and the margin gap on them is 0.363 [0.186, 0.544]. Copying the XML states into nested JSON at the structural tokens (field boundaries, markers, the warning, and padding) recovers the gap, with a recovered fraction of R = 1.25 [0.95, 1.73]. Copying them at the message tokens gives R = −0.17 [−0.35, −0.06].
Response to Reviewer uz8U
We thank the reviewer for the clear statement of the causal question. Revisions are highlighted in yellow, and the new analyses are collected in Appendix D.
The evidence links source separation to rendering-dependent risk at two levels. Across 23 renderings (Section 3), lower final-token PCSS accompanies higher memory-attack risk in Qwen3-14B (ρ = −0.633), Qwen3-8B (ρ = −0.521), and Mistral-7B (ρ = −0.715). With the rendering fixed, moving external-message states toward the trusted-source direction raises attack success by 4.0 pp [1.33, 7.33] (Section 4, Appendix B.6).
The revision also tests whether natural rendering changes act through the edited direction. Switching from XML to nested JSON moves external-message states along it, and aligning this shift leaves the attacker-margin gap essentially unchanged (Section 4.4). The revision therefore presents PCSS as an internal indicator of rendering-dependent risk (Section 6.1), and the Abstract and Introduction are revised accordingly.
Switching from XML to nested JSON moves external-message states along the same direction as the edit, by a smaller amount. The projection moves toward the trusted side by +0.135 [0.103, 0.169], in the same direction for 45 of the 50 held-out MA tasks (Section 4.4). On 256 fresh, position-matched item pairs, the frozen probe that defines the edit also detects the rendering change (d′ difference −1.52 [−1.58, −1.45], Section 3.5). The natural shift is two orders of magnitude smaller than the primary edit (α = 16), and aligning it changes the attacker-margin gap of 0.668 by 0.003 [−0.013, 0.019] (Appendix D.3). The edit therefore tests whether the direction can change adoption, and Section 4.4 describes it in these terms.
We hope these additional results address your questions. We will respond promptly to any further questions during the discussion period.