- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether untrusted examples or logs can alter how a model executes a task even when the literal content of the task instruction remains unchanged. The authors refer to this phenomenon as semantic reanchoring. By introducing conditions such as STANDARD, REVERSE, and INDUCTION, they distinguish unauthorized contextual adaptation from authorized adaptation and failures to adapt. The experiments cover eight models and primarily focus on arithmetic operators, device control, and deadline interpretation. The results demonstrate that this is a legitimate concern. The paper further investigates the stealthiness of such attacks, prompt-level prevention mechanisms, and detection methods.
Critical strengths
-
Distinguishing authorization from adaptation capability. The STANDARD condition requires models to resist reinterpretations suggested by contextual examples, whereas INDUCTION requires them to learn from designated examples. This design avoids mistakenly treating unconditional resistance to contextual information as a security capability.
-
The authors are transparent about several limitations. The paper explicitly acknowledges that visible reasoning traces do not establish the underlying causal mechanism, that role imitation in the controller setting has not been ruled out, that the auditors have not been evaluated against adaptive attacks, and that separate model calls do not demonstrate cross-turn memory changes. These clearly stated limitations are appreciated.
-
Broad model coverage and substantial experimental scale. The experiments evaluate eight LLMs from seven developers, covering both proprietary frontier models and open-weight models of varying sizes and capabilities.
Critical weaknesses
-
The experimental section suffers from poor clarity and organization. Section 4 introduces numerous experimental arms, task settings, and evaluation metrics without providing a clear overall framework. This makes it difficult for readers to understand how the experiments relate to one another and which research questions they address. Furthermore, quantitative results are scattered throughout the text, with limited visualization or systematic comparison, making the experimental findings even harder to interpret.
-
The mechanisms underlying the attacks are insufficiently explained. The authors mention several potential mechanisms or alternative explanations, such as self-anchoring, role imitation, and channel compensation. However, the main text does not clearly distinguish these explanations or explain which ones are supported or ruled out by the experiments. Although the appendix provides some additional discussion, the underlying causal mechanisms remain unverified.
-
The diversity of experimental tasks and samples is limited. Despite the substantial number of experimental runs, the core experiments primarily focus on three highly structured task families: arithmetic operators, device control, and deadline interpretation. Within each task family, different samples are mainly generated by varying numbers, device names, timestamps, or log entries, while the underlying semantic structures and attack patterns remain largely unchanged.
-
The appendix is overly extensive and disrupts the overall coherence of the paper. It contains numerous additional risk taxonomies, unexecuted experimental plans, withdrawn analyses, and methodological discussions, some of which are only loosely connected to the paper's central research questions. Meanwhile, important details necessary for understanding the main experiments are deferred to the appendix. This organization makes it unnecessarily difficult to identify the paper's main contributions and follow its supporting evidence.
Questions for authors
-
Regarding the attack mechanisms: Could the authors more clearly distinguish the proposed mechanisms and alternative explanations, and clarify what evidence the existing experiments provide for each? In particular, how can genuine semantic reinterpretation be distinguished from role imitation or other forms of contextual pattern matching?
-
Regarding the reliability and fairness of LLM-as-Judge evaluation: The paper uses different LLMs to evaluate outputs from different subject models. For example, GPT and Claude outputs are evaluated by different judges. Could this non-uniform evaluation setup introduce judge-specific biases that affect comparisons across models?