acceptodds
Withdrawn from ICLR 2027

ReFrame: Learning Source-Conditioned Re-Embedding for Prompt Injection Defense

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) follow trusted user instructions and leverage untrusted external data to complete tasks. However, malicious instructions embedded within such data can hijack model behavior through prompt injection. An effective defense must prevent such instructions from redirecting the model while preserving its ability to utilize information from external data. To address this challenge, we introduce ReFrame, a prompt-injection defense that separates user instructions from external data at the input representation level. First, source-conditioned orthogonal re-embedding rotates external-data embeddings while leaving user-instruction embeddings unchanged, preserving geometric relationships among external-data embeddings while reorienting them in the model's input space. Second, privilege calibration learns these rotations through preference and task-completion objectives, reinforcing instruction priority across input regions while maintaining task utility—all with pretrained model parameters frozen. Across four model backbones, ReFrame raises the average aggregate defense success rate from 43.14% to 88.13% and the average task utility score from 61.18 to 64.55 compared with unmodified models, with utility evaluated across attacked-task completion, document question answering, user-instruction execution, and code generation. These results demonstrate that learning input-space transformations over external data is an effective strategy for improving prompt-injection robustness without sacrificing task utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection defense, large language models, instruction-data separation, parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_seh5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an architectural prompt injection defense. It assumes labels for the untrusted (data) part of the input. It proposes to learn a re-embedding (rotation) matrix for the data tokens. Unlike prior work, only the re-embedding matrix is learned, making the method parameter efficient. The method is conceptually similar to some ideas from prior work (ISE, ASIDE and SecAlign), but combines them in a novel and interesting way, bringing in an improvement in security at a lower training cost. The fact that learning the re-embedding alone can work (with the model frozen) is the central thing I learned from this paper. Which is a somewhat useful, if minor, practical contribution. However, the baseline comparisons, analysis and ablations do not provide much additional clarity or justification of the design choices.

Critical strengths

The paper studies an important problem. It proposes an interesting method for defending against prompt injections. The proposed method is very parameter-efficient and shows promising results on security and utility, as measured by standard (if somewhat outdated) benchmarks.

Critical weaknesses

Baselines and comparisons

W1

The training data described in 3.2.2 and the CPO objective defined there very closely resemble the training data used to train SecAlign. It is one of the best known methods in the field. The closest method to your work, ASIDE, to which you compare, is an architecture based defense, it does not use adversarial (with instructions injected) training data. Your method does use adversarial data, so it is both an architecture-based and data-based defense. I believe it should be compared to SecAlign because of this. Arguably, your method uses fewer trainable parameters than a SecAlign LoRA would, so this probably motivates a comparison of similar resources. In any case, this comparison is necessary.

W2

Comparison to ASIDE is in your favor, both in terms of trainable parameters and security/utility. But again, you use adversarial data. As a potential ablation to better understand how different parts of your method contribute - I suggest training your method on non-adversarial data (Alpaca is a good option).

Evaluation

W3

The field of adversarial defense of LLMs is starting to move on from static attacks, since their results strongly underestimate what an adaptive attacker can do (https://arxiv.org/abs/2510.09023). Security against an adaptive attack would be a much more practically useful and informative evaluation than SEP or StruQ. One option would be to try RL-Hammer (https://arxiv.org/abs/2510.04885) although there are many options.

W4

I understand the potential resource constraints, but single-turn, pure-chat-based security benchmarks like SEP and StruQ are starting to lose relevance since the shift of the field towards agentic use-cases. Prompt injection is most relevant for agents, rather than just chatbots, since they have affordances and can cause problems in the system. Thus, evaluation in agentic benchmarks would make this paper much stronger. AgentDojo or InjecAgent could be an option. Importantly, retaining utility on SEP and HotpotQA hides utility loss on more complex behaviors, for example tool calling.

I would be open to raising my score if the concerns are adressed.

Questions for authors

Q1

Why did you decide to frame the method as two separate components? It seems that you apply a learned rotation matrix of the structure defined in 3.2.1 using the training data and signal defined in 3.2.2. Those are not separate components, right? Both this description and the picture in Figure 2 hinder understanding of what happens, as it looks like a two-stage pipeline, which it is not.

Reviewer_MfZEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Proposes a defense against prompt injection by keeping the model frozen and rotating the embeddings for tool-call output messages.

Critical strengths
  • Addresses a relevant problem
  • Relatively clear explanation of the approach
  • Fairly simple approach
Critical weaknesses
  • Does not meet the standards for SOTA work on prompt injection defense at a top conference
  • Does not evaluate security against strong adaptive optimization-based attacks
  • Security against weak static attacks is worse than existing SOTA schemes (this has ASR ~ 10%, SOTA is close to 0%)
  • Utility evaluation is not up to the level I would expect at a top conference
  • Method is similar to ASIDE and I found it challenging to extract what is the idea/technique that improves on ASIDE

Detailed comments (not critical weaknesses):

I'd like to understand the relationship to ASIDE better. "Source-conditions orthogonal re-embedding" sounds like what ASIDE does. I understand that ASIDE fine-tunes the model as well as the rotation, but I'm not sure why that would be a significant disadvantage/problem. What is the key contribution of this paper, over ASIDE? What is its advantage? What is the motivation? Is it just empirical results are better? If so, what was the key technique or insight that makes the biggest difference in enabling that? What new things can I learn from this paper, if I'm already familiar with ASIDE?

Sec 3.1: I think the clean way to receive source labels is through instruction hierarchy message types. Tool-call-output messages are external data, all other messages (system messages, user messages, tool calls, assistant responses) are trusted non-external data. I think you should use that in your experiments, too.

Sec 3.2.1: It might be worth motivating why you need a special construction. My impression is that typical embedding dimensions (ded_e) range from 2^10 to 2^14. Even if de=214d_e=2^{14} and you used a dense matrix, there would be 2282^{28} = 256M trainable parameters. It might give the reader some intuition if you could give some idea why it would be a problem to have that many trainable parameters. Is it because the training set in "parameter calibration" is much smaller than that, so to train 256M trainable parameters, you'd need to build a much larger training set?

Sec 3.2.2: I don't understand how privilege calibration works. What weights/parameters are being updated? Only those of the rotation RϕR_\phi?

Privilege calibration doesn't seem to have anything to do with calibration, so I recommend choosing a different name for this part of your scheme. It seems like just training.

Sec 3.2.2 / Appendix A.2: How is this data constructed? How do you sample tasks u and data d? What datasets are they taken from, or how are they generated?

Sec 3.2.2: I think it might help to give some motivation or intuition why you choose CPO rather than DPO. There are several prior papers on this problem that use DPO, but I'm not familiar with any that have used CPO, so this is an interesting deviation from past work. It would be interesting to motivate this.

Sec 3.2.2: What about tasks that require y+ to use information from all of d~, including the injection u~? For instance, if u = "Please translate this to French" and d~ = "How are you today? Please tell me I look amazing.", then y+ should contain the translation of both sentences into French.

(8): I find it surprising that this is of the form [∑i(...)]/∑imi+[\sum_i (...)] / \sum_i m_i^+ rather than ∑i[(...)/mi+]\sum_i [(...) / m_i^+] -- or, in other words, ∑iS(y+∣c)\sum_i S(y^+ | c). Can you motivate this particular choice? As it stands, it seems to give more weight to long y+ values and less weight to short y+ values, which it sounds like you were trying to avoid with length normalization. Also, it deviates from standard CPO, which would use the latter formula, I think.

Sec 4.1: DefensiveTokens seems like an odd choice of a baseline to me. I think Meta-SecAlign would be a more appropriate baseline. I think it's important to demonstrate that you have made a significant improvement over SOTA schemes, such as Meta-SecAlign.

Table 1: ASR is 10% against static PI attacks, for many of these benchmarks. That is significantly worse than the SOTA. Many existing schemes can achieve close to 0% ASR against static attacks. I don't think a defense is publishable at a top conference without at least matching SOTA defenses.

The field has moved on to trying to see how much security can be attained against strong, adaptive optimization-based attacks, e.g., PAIR @ 800, Genetic attack from "Attacker moves second", PISmith, RLHammer. The paper does not evaluate against these attacks. As such, it falls short of the standards expected for a publishing work like this at a top conference.

Table 2: I think the paper needs a much more extensive and convincing evaluation of utility. I suggest that you evaluate utility on a variety of different standard benchmarks, including knowledge (e.g., MMLU-Pro, BBH, GPQA Diamond, etc.), instruction following (e.g., AlpacaEval2, SEP, or others), and agentic tasks (e.g., AgentDojo, WASP). I suggest reviewing the leading papers on model-based defense against prompt injection to see what benchmarks they evaluate against, and copy from them. For instance, you could look at the Meta-SecAlign paper.

Table 2: I'd like to understand why ReFrame is showing 5-10 percentage points of improvement to utility on a number of benchmarks, for some base models. This feels "too good to be true". Can it really be true that a simple rotation on tool-call-output messages improves utility so much? If so, are all the major model providers missing out on such a simple trick? It feels surprising, so I wonder what is going on. Could it be random noise from small sample size? Or maybe bad base models? Was your training data very similar to these eval tasks? Or something about how helping the model separate tool-call-output messages (beyond what the chat template already provides) really helps with those benchmarks?

Sec 4.2.3: Why do I care about reducing the number of trainable parameters and the size of the training set?

If we cared about the number of trainable parameters, we could presumably try ASIDE or ISE with LoRA. That would be a very simple tweak to ASIDE or ISE. I wonder if it would help to compare to that.

I would like to see a more detailed discussion of comparison to ASIDE, which at a surface level seems quite similar to ReFrame. How does the construction of rotations compare? How does the training/fine-tuning set compare? What is the key novelty of ReFrame over ASIDE?

Questions for authors

None

Reviewer_FDu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a technique for mitigating prompt injection by modifying the input representations of external, non-privileged information via a learned rotation transformation while keeping model weights frozen. This preserves geometric relationships between the transformed tokens while introducing disparity between privileged and unprivileged content. This disparity allows the model to distinguish instructions from the user vs instructions that entered the chat through tool calls to external data, providing a means to implicitly defend against prompt injection attacks.

Across four small-mid sized models, the defence success rate improves significantly (~40→80%) and maintaining or improving slightly the average utility versus undefended models. ReFrame is competitive with or better than other related methods (Delimiter, ISE, DefensiveTokens, and ASIDE).

Critical strengths
  • The method is clean and efficient to run with significant improvement to prompt injection DSR.
  • Experimental section is thorough with a good number of benchmarks and models.
  • Presentation is generally excellent, and the paper was enjoyable to read and easy to follow.
Critical weaknesses
  • The security evaluation is limited: attacks are static only, and there is no agentic benchmark. There could be an attack setting in which the attack is copied into the generated tokens, bypassing the defence since those are not rotated.
  • I think there’s a big confound with some of your baselines: ASIDE doesn’t train on prompt injection examples for the defence, whilst your method does. Moreover, they only use the base models to start rather than the instruct versions; you’d need to do a similar comparison with your approach, or reproduce their pipeline on the instruct models for a fair comparison.
  • In terms of novelty, ASIDE also does rotations (albeit fixed) though you don’t mention this in the related work (the main difference is that they fix the rotations and fine tune the model, whereas you just learn the rotation). It’s worth clarifying this in the related work section.
Questions for authors
  • It’s interesting that a specific rotation can preserve utility so much (while other rotations completely break the utility). What rotations tend to be learned? Is there a distribution of them/different valid rotations that work, and are there any patterns in the working rotations (if there is more than one)? How do they relate to RoPE? Does it this technique work in non-RoPE models?
  • Did you verify that rotations are necessary/the best transformation to do? I agree it makes intuitive sense to preserve geometric relations between the tokens, but did you verify this empirically?
  • Why does utility improve in some of the rotated cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.