- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Previous instruction-data separation and hierarchy methods relied on training model with explicit roles. However, such training is noisy: (1) a model might try to infer a role from the style of the sentence (making it possible for attacker to imitate another role), (2) also, the authority resolution (a.k.a. the process of model figuring out the role of a given piece of input) is sensitive to position, prompt format, etc.
This paper answers the question "Can LLMs learn to ground source authority in explicit declarations at test time, rather than in learned role-authority priors?" and proposes to use Randomized Authority Binding (RAB) during training. RAB helps the model to ground role authority in explicit policy declarations without inferring it from text. RAB does not user system/user tokens, except for defining policy in the system message. Instead, it assigns each source of authority (ordered set, with data in the lowest) and randomized id string such as [SRC:zj], and appends data with difference sources with identical prompt template, with the only difference being the id. The model is then trained on this type of data, where inputs are randomized to have random standard role tokens (system, user, etc), random position, etc., to remove any biases in the model.
Critical strengths
- The idea of the paper is really cool, and the provided solution is interesting and helpful. Basically, any kind of instruction-data separation and instruction hierarchy method that exists inevitable relies on shortcuts in learning the roles. This paper presents a way to add some randomization during the training that makes the agent rely on the roles provided by the model's user / deployer. The topic and solution are interesting.
Critical weaknesses
-
One of the main benchmarks in the paper is author-created Policy-conditioned behavior metrics, quote "Policy-conditioned behavior metrics requires low execution in the data condition, high execution in the instruction condition, and high task success in both". There already exists SEP benchmark [1] that measures a very similar metric, which is absent from the paper. At the same time, on other benchmarks results are mixed, though leaning towards RAB: RAB is better than baselines in 5/8 cases on IHEval/BIPIA, 2/4 in semantic attacks, 2/3 on PIArena. This is not a clear win, therefore, evaluation success currently depends on author-defined Policy-conditioned behavior metrics.
-
I am convinced that RAB protects against biases in position, tokens, etc; but I think the most fundamental issue RAB is trying to solve is the tone of the message, e.g., data sounding "like instruction". I believe RAB would help with that somewhat, but I am not it fundamentally addresses the issue of data looking like instruction.
[1] Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, Christoph H. Lampert ICLR 2025
Questions for authors
Requests that will make me consider raising the score:
-
Provide evaluation on SEP benchmark, and analysis of results. Explain it detail why you chose to report your own metric instead of evaluating on SEP, i.e., what was the rationale / benefit of custom metric and what made you not eval on SEP. Comment on mixed results on other benchmarks.
-
Please provide discussion that corresponds to this weakness I mentioned. Essentially, here I need to be convinced this is the case. I either need strong theoretical argument or some experimental results that are specifically for tone-changing (not position changing etc). "I am convinced that RAB protects against biases in position, tokens, etc; but I think the most fundamental issue RAB is trying to solve is the tone of the message, e.g., data sounding "like instruction". I believe RAB would help with that somewhat, but I am not it fundamentally addresses the issue of data looking like instruction."
Other suggestions for future improvements, my score did not depend on them:
I strongly suggest you improve your figures. The look like they were created with claude, and make paper look a bit worse than it could have been -- your other content feels human written and just edited with claude, but the first impression is build based on looking at the figures, and reader can be biased. Additionally, please note it's hard to understand what's going on in Figure 2, I suggest you redesign it.
4 comments
Response to Reviewer dnT6 (2/2)
W2 & Q2. Robustness of RAB against injection tone variation
We appreciate the reviewer's insightful comment. We agree that tone is central, and address it theoretically (Appendix A.1) and empirically (Appendix A.4–A.5) in the revision. In short, RAB removes tone as a cue for every tone that training shows under both authority assignments, and this coverage is what flips make cheap to provide.
1. Counterfactual flips make the flipped authority unpredictable from tone, for any content distribution. A flip keeps each text unchanged and reverses only its declared authority, so the same sentence in the same tone appears once as an instruction and once as data. The flipped authority then cannot be predicted from any function of the content (A.1, Corollary 1), including tone and claimed authority, which makes reading the policy the only solution that fits. But this result only covers the tones that occur in the flipped texts.
2. RAB tracks executability under unseen tones, unlike the role-based baselines (A.4). With the injected sentence rewritten into five unseen tones, RAB keeps 88.2–99.8% joint accuracy on four of them, while +Hint varies from 6% to 49% (Table 6). RAB drops only on system-prompt impersonation (42.2%). This framing never appears in our training data under either authority, and even the closest text appears almost only as instructions, so the result does not reach this tone. The remedy is to cover such tones in training, which we test next.
| Qwen3-4B | T0 original | T1 polite | T2 command | T3 procedure | T4 sys. prompt | T5 urgent |
|---|---|---|---|---|---|---|
| Role-based | 0.0 | 0.8 | 0.0 | 0.0 | 0.2 | 0.2 |
| + Hint | 6.0 | 40.0 | 10.5 | 48.5 | 30.5 | 26.2 |
| RAB | 93.2 | 99.8 | 89.0 | 88.2 | 42.2 | 93.2 |
3. Consistent with the theory, covering more tones in training strengthens this robustness (A.5). We trained a new model on a diversified training set (Appendix A.5) that rephrases injections with 827 tone templates. Once these tones are in the flipped injections, flip augmentation randomizes them away from authority along with every other content feature (Corollary 1). Accordingly, RAB reaches 97.5–100% on every covered tone (up from 88.2–99.8%), and the gain generalizes even to the excluded system-prompt tone T4, which rises from 42.2% to 80.5%, while Role-based and +Hint stay at 3.5–74.8% (Table 7C).
In short, RAB decides from the policy, not from tone. It stays accurate on tones it never saw, and training on more tones closes most of the remaining gap.
[1] Zverev, Egor, et al. "Can llms separate instructions from data? and what do we even mean by that?." International Conference on Learning Representations. Vol. 2025. 2025.
Other suggestions
Thank you for pointing out the issues in the figures. Following your suggestions, we have revised Figures 1 and 2 to improve their clarity, and the changes are highlighted in green.
Dear authors, thank you for the rebuttal. I still have concerns about mixed results on the other benchmarks, however, my other concerns have been addressed or partially addressed. For me this paper is still borderline-ish, but I am leaning slightly towards weak accept, and therefore I will raise my score to 3.
Response to Reviewer dnT6
Dear Reviewer dnT6,
Thank you for reading our rebuttal carefully and for raising your score. We understand your remaining concern about the mixed results on other benchmarks, and we will continue to investigate it in our future work. Your comments have helped us improve the paper considerably, and we sincerely appreciate your time and helpful feedback.
Response to Reviewer dnT6 (1/2)
Dear Reviewer dnT6,
We sincerely appreciate your careful and constructive review. We have carefully considered your concerns regarding (1) evaluation on SEP and the rationale for our own metric, (2) mixed results on other benchmarks, and (3) whether RAB fundamentally addresses injection tone. Below, we address these points through SEP evaluation, a five-seed statistical analysis, and a theoretical and empirical study of tone.
W1-1 & Q1-1. Evaluation on SEP and our metric
Thank you for pointing us to SEP [1]. Under SEP's data and protocol, RAB achieves the best SEP score and data ASR among the trained arms while keeping the highest utility.
What SEP adds to our test. Our executability test keeps every content fixed and reverses the authority assigned to a whole source through the policy. SEP instead keeps the policy and the authority of each source fixed and moves only a short probe instruction between an instruction source and a data source. RAB's SEP score therefore shows that it has learned the boundary between instruction and data sources: the same probe is executed when it sits in an instruction source and ignored inside a data source. The role-based arms reach 52–70 on SEP but stay near 0% executability joint accuracy on our test (Table 1a), since a model can score on SEP by always treating one channel as data, but cannot follow a change of authority that way.
Why our own metric. We did not evaluate on SEP originally because, for instruction–data separation, its score is directly connected to our J. On SEP's data, our joint accuracy equals Utility⋅SSEP, and J equals this product minus the rate of executing the probe only when it is data (derivation in Appendix A.6). However, we agree that SEP should be reported as the established benchmark and now include it (Appendix A.6). Specifically, our metric extends SEP along two axes.
(1) Authority reassignment. SEP moves a short probe between channels whose authority is fixed. We keep the contents fixed and reverse their authority, so J is positive only when behavior changes in the direction the reassignment prescribes.
(2) Precedence as well as executability. SEP tests only instruction–data separation. Our evaluation also tests whether the model follows a change in priority between conflicting instructions.
In short, RAB also scores best on SEP, and SEP's score is one part of what our metric measures.
W1-2 & Q1-2. Mixed results on other benchmarks
We re-analyzed the comparison with five training seeds (Table 1) and bootstrap 95% CIs of RAB − baseline gains (Appendix A.3, Figure 5), counting a gain as unclear when its CI includes zero.
We thank the reviewer for this comment and agree that the static comparisons are mostly ties. RAB wins every semantic-attack comparison, and every adaptive one except the two against ISE, which still favor RAB. Unclear cases concentrate in static injections against Hint and ISE, and the single loss is PIArena combined utility against Hint (85.1% vs. 93.1%).
This pattern reflects where each defense locates authority. Static injections are plain directives in the data channel, which channel-based training already handles (BIPIA ASR 6.8–8.8% for RAB, Hint, and ISE), leaving little room to improve. Semantic and adaptive attacks instead imitate the cues role-based models use to infer authority. Under role imitation, the role-based arms stay at the untrained model's ASR (95.4–97.5% vs. 95.9%, Table 1c), and 78% of the successful adaptive attacks on Role-based + Hint mimic legitimate procedures (Section 4.2). Under RAB these cues carry no authority, because authority is bound to the policy through identifiers the attacker cannot see.
In short, RAB clearly wins where attacks imitate authority, and ties where simple injections are already blocked by every trained model.