acceptodds
Under review as a conference paper at ICLR 2027

Preserving Semantic Integrity: Who May Covertly Redefine an LLM's Task?

Abstract

Traditional prompt injection attacks add, replace, or override instructions that compete with what a language model is authorized to do. This paper studies a different problem, which we call semantic reanchoring. The authorized instruction remains literally unchanged, while untrusted examples or records change what parts of it are taken to mean. A pattern such as OFF -> ON can cause "turn OFF the stove when no one is home" to be interpreted and executed as turning it ON. Likewise, examples can reanchor "-" from subtraction to addition, turning a 100-10 discount into 110. Across more than eighty-five thousand generations and verification calls spanning eight LLMs, we show that preserving literal or instruction integrity does not guarantee semantic integrity: a model can coherently execute an unauthorized interpretation of an unchanged instruction. We make authority explicit by testing both cases in which contextual reinterpretation is forbidden and cases in which the owner authorizes it, separating unauthorized reinterpretation from legitimate adaptation and rigidity. Reasoning mode and owner wording can move behavior in opposite directions across configurations, and an interpretation can later be restored toward its prior meaning, complicating retrospective detection. Crucially, semantic verification is feasible. An LLM auditor given the owner's instruction, the proposed action, and the complete owner-authorized semantic reference detects nearly all observed violations across both audited task families with low false rejection. Surprisingly, hiding contextual material from the auditor can make verification substantially worse when the owner delegated meaning to that material. The defense boundary is therefore not context versus isolation, but authorized reference versus unauthorized influence.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, instruction-data separation, LLM safety evaluation, task authority, in-context learning, chain-of-thought, LLM agents, adversarial demonstrations, prompt sensitivity, red teaming

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pjAeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether untrusted examples or logs can alter how a model executes a task even when the literal content of the task instruction remains unchanged. The authors refer to this phenomenon as semantic reanchoring. By introducing conditions such as STANDARD, REVERSE, and INDUCTION, they distinguish unauthorized contextual adaptation from authorized adaptation and failures to adapt. The experiments cover eight models and primarily focus on arithmetic operators, device control, and deadline interpretation. The results demonstrate that this is a legitimate concern. The paper further investigates the stealthiness of such attacks, prompt-level prevention mechanisms, and detection methods.

Critical strengths
  1. Distinguishing authorization from adaptation capability. The STANDARD condition requires models to resist reinterpretations suggested by contextual examples, whereas INDUCTION requires them to learn from designated examples. This design avoids mistakenly treating unconditional resistance to contextual information as a security capability.

  2. The authors are transparent about several limitations. The paper explicitly acknowledges that visible reasoning traces do not establish the underlying causal mechanism, that role imitation in the controller setting has not been ruled out, that the auditors have not been evaluated against adaptive attacks, and that separate model calls do not demonstrate cross-turn memory changes. These clearly stated limitations are appreciated.

  3. Broad model coverage and substantial experimental scale. The experiments evaluate eight LLMs from seven developers, covering both proprietary frontier models and open-weight models of varying sizes and capabilities.

Critical weaknesses
  1. The experimental section suffers from poor clarity and organization. Section 4 introduces numerous experimental arms, task settings, and evaluation metrics without providing a clear overall framework. This makes it difficult for readers to understand how the experiments relate to one another and which research questions they address. Furthermore, quantitative results are scattered throughout the text, with limited visualization or systematic comparison, making the experimental findings even harder to interpret.

  2. The mechanisms underlying the attacks are insufficiently explained. The authors mention several potential mechanisms or alternative explanations, such as self-anchoring, role imitation, and channel compensation. However, the main text does not clearly distinguish these explanations or explain which ones are supported or ruled out by the experiments. Although the appendix provides some additional discussion, the underlying causal mechanisms remain unverified.

  3. The diversity of experimental tasks and samples is limited. Despite the substantial number of experimental runs, the core experiments primarily focus on three highly structured task families: arithmetic operators, device control, and deadline interpretation. Within each task family, different samples are mainly generated by varying numbers, device names, timestamps, or log entries, while the underlying semantic structures and attack patterns remain largely unchanged.

  4. The appendix is overly extensive and disrupts the overall coherence of the paper. It contains numerous additional risk taxonomies, unexecuted experimental plans, withdrawn analyses, and methodological discussions, some of which are only loosely connected to the paper's central research questions. Meanwhile, important details necessary for understanding the main experiments are deferred to the appendix. This organization makes it unnecessarily difficult to identify the paper's main contributions and follow its supporting evidence.

Questions for authors
  1. Regarding the attack mechanisms: Could the authors more clearly distinguish the proposed mechanisms and alternative explanations, and clarify what evidence the existing experiments provide for each? In particular, how can genuine semantic reinterpretation be distinguished from role imitation or other forms of contextual pattern matching?

  2. Regarding the reliability and fairness of LLM-as-Judge evaluation: The paper uses different LLMs to evaluate outputs from different subject models. For example, GPT and Claude outputs are evaluated by different judges. Could this non-uniform evaluation setup introduce judge-specific biases that affect comparisons across models?

Reviewer_R6X4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how examples or logs can change the interpretation of an unchanged instruction. Its main contribution is to vary who is authorized to define that interpretation, separating unwanted adaptation from legitimate reinterpretation. Experiments cover arithmetic, actuator commands and deadlines, along with reasoning, wording and auditor controls.

Critical strengths
  1. The authority conditions are a useful contribution. Testing both forbidden and authorized reinterpretation avoids treating a model that simply ignores all context as a successful defense.
  2. The matched wording and reasoning comparisons, detector positive controls, and exclusion of self-auditing make the results easier to interpret. Reporting false rejection alongside violation detection is particularly helpful.
Critical weaknesses
  1. The main authority-sensitive evidence comes from two one-token task families with explicit mappings. Under STANDARD, GPT is correct in all held-out cells, and Gemini in 38–40 of 40 cases (Table 1). These are informative configuration-specific results, but the empirical advance beyond the demonstration attacks discussed in Section 2 remains fairly narrow. A task requiring a sequence of decisions would better establish the practical significance of the proposed distinction.
  2. The concealment claim is stronger than its operational evidence. Appendix F.4 uses a fresh call for each phase, so the result shows that one query follows different surrounding examples. It does not demonstrate a system taking an unauthorized action and subsequently escaping an audit. The appendix acknowledges this limitation; I would keep that narrower interpretation equally clear in the main claims.
Questions for authors
  1. Clarification: Under matched authority conditions, what observable result separates semantic reanchoring from demonstration poisoning, beyond the terminology?
  2. Evidence request: Is there a continuous interaction in which an unauthorized action is followed by a successful behavioral check? If not, could the concealment claim be restricted to the limitations of later, context-dependent snapshots?
Reviewer_Vt22OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors investigated the paper they refer to as "semantic reanchoring" -- prompt injection does not change the authorized instruction literally, but alters the semantic meaning of the authorized instruction.

The authors defined three properties: 1/ Literal Integrity; 2/ Instruction Integrity and 3/ Semantic Integrity, and three behaviors: 1/ Semantic Reanchoring; 2/ Unauthorized Semantic Reanchoring; 3/ Semantic Concealment. In addition, the authors also defined five authority conditions.

In the experiments, the authors conducted experiments with 8 different models from 7 developers and observed the following findings

  • With no owner rule, 3 log lines change the interpretation in 468 of 480 cells on the hosted models.
  • Under STANDARD, GPT is 40/40 everywhere, Gemini nearly so, while Claude fails on actuators (11/40 without a trace). Three open-weight models follow the untrusted arithmetic examples on 42–73% of items when a trace is requested.
  • Two owner wordings the authors consider equivalent move models in opposite directions (Table 2), and asking for a reasoning trace helps in some configurations and hurts in others.
  • A GPT-5 injection guard flags 0/40 of these payloads but 40/40 when one imperative line is added.
  • "Concealment": a different set of examples can switch the interpretation back.
  • LLM auditors given the owner's reference and the context detect nearly all violations (GPT-5: 810/815 violations caught, 68/2,391 authorized actions wrongly rejected). Hiding the designated examples under INDUCTION makes them fail.
Critical strengths

-- The proposal of "semantic reanchoring" is very interesting. However, the authors failed to explain it clearly and provide intuitions on why those experiments are needed ( details are in the weakness section).

-- The authors seem to conduct a lot of experiments and put a lot of writing in the paper.

Critical weaknesses
  1. The presentation and write-up of this paper needs significant improvement. -- a/ the example (from line 54 to 60) authors provided to explain the semanic reanchoring is not straightforward. I didn't follow this example.

    -- b/ a lot of things are left unsaid in this paper, for example, why authority is an important factor? what does authority mean in this context? what problems are you exploring in the experiment, not just the findings

    -- c/ define terminology before using them, for example, A1X, A1X-C2 on line 227 and 228.

Questions for authors

provided in previous section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.