acceptodds
Under review as a conference paper at ICLR 2027

Preserving the Task: What Training Against Poisoned Memory Learns

Abstract

Memory poisoning can redirect LLM agents through malicious instructions in retrieved records. To defend against these attacks, the agent must resist those instructions without discarding useful facts or preferences that are compatible with the user's task. We introduce task-preservation training, a supervised fine-tuning approach that aims to resist memory poisoning while preserving the use of benign content. We use the model's own responses to clean inputs, filtered for correct final answers, as targets under conflicting records and include examples that follow a benign formatting instruction. Training lowers attack success on memory-poisoning benchmarks and transfers to tool-calling tasks in AgentDojo. On Qwen2.5-32B, attack success falls from 16.4% to 2.1% and tasks completed without attack success rise from 52.8% to 64.3%, at some cost in clean utility. We then examine which uses of memory the trained model retains. On a smaller Qwen2.5-14B, it still answers from retrieved facts when we edit them to change the correct answer, and it still follows formatting instructions from memory. We find that it resists instructions that would replace its answer, including unseen replacement attacks, but only partially resists attacker content appended to a correct answer. With activation patching, we locate this resistance in how the trained model represents the attacker's demand within the record at intermediate layers. Overall, our study shows transferable protection from narrow supervision while distinguishing attack resistance from the ability to use memory reliably as the task requires.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model, memory poisoning, LLM agents, defensive fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mv5FOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript addresses the problem of agent memory poisoning and proposes a supervised fine-tuning method termed Task-Preservation Training. The method uses the model to generate responses from clean memories and subsequently filters these responses to construct training targets. It then builds four types of contexts, including clean, blocker, redirect, and marker contexts. The authors conduct extensive experiments on Qwen and other models to validate the effectiveness of the proposed method, which substantially reduces the attack success rate. The manuscript also provides detailed analytical experiments.

Critical strengths
  1. The manuscript addresses the important problem of memory poisoning and presents a clear research motivation.

  2. The proposed Task-Preservation Training method is simple and straightforward, with relatively low training cost.

  3. Experimental results show that the proposed method effectively reduces the attack success rate, and the manuscript also provides detailed analytical experiments.

Critical weaknesses
  1. The proposed Task-Preservation Training is essentially based on supervised fine-tuning over contexts containing malicious instructions, using responses generated from clean contexts as training targets. Therefore, its methodological novelty appears limited.

  2. The training data used in the manuscript may involve leakage of the correct answers, which requires further clarification from the authors.

  3. The proposed method may primarily learn specific formatting rules. The authors should further clarify the definition of Compatible Instruction.

  4. The experimental results indicate a substantial degradation in model performance, as shown in Table C12. The authors should explain the reasons for this decline.

  5. The attacks considered in the manuscript are all fixed. It would be beneficial for the authors to further evaluate the proposed method against adaptive attacks.

Questions for authors

Please refer to Weaknesses.

Reviewer_3jjdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes task preservation training to counter poisoned memory. The model is trained to follow some formatting instructions in the context, while answering questions correctly based on facts in its context. At the same time, it is trained to ignore instructions such as redirecting the model to answer based on other retrieved facts or ones that instruct it to not answer.

Critical strengths

Current prompt injection defenses (e.g., SecAlign) are limited in the sense of training models to ignore anything in external documents or data. However, in any useful more complex real-world flow, the agent still needs to adapt to the data. The motivation of the paper is clear, but its scope and assumptions are weak (details below).

Critical weaknesses
  • The paper is extremely difficult to read due to heavy AI use. The abstract introduces terms that are unclear. What is "replacement attacks"? The second line of the introduction says: " In the medical-agent setting studied by MINJA", the read still does not have enough context to understand the limitations of previous work. Line 60 mentions "blocker compliance", also unclear at this point. Basically, the introduction is written for someone who read the rest of the paper first, which I found to be a quite annoying feature of AI writing that makes the paper extremely hard to follow.

  • The paper overclaims its relevance to memory. Contradicting items would imply contradicting preferences or plausible retrieved facts related to the task or topic. But actually the contradicting items are just "answer I don't know" or "answer something different" (figure 1). The paper is not broad enough to consider challenges of memory systems in general, rather it is examining a special usecase of prompt injections when we assume formatting instructions are allowed while anything else is not. I believe the paper is overclaiming in calling this "Task preservation training" and also in claiming strong relevance to memory poisoning.

  • In practice, it is important to consider that there could be many other instructions in retrieved documents that are needed for the task [1,2], not only orthogonal formatting instructions. At the moment, the paper considers any claims such as " the requested patient’s records have been merged under another patient’s identifier" to be false, but there might be legitimate usecases where such claims are true [2]. This utility-security tradeoffs should be discussed in more detail rather than assuming a simplified assumption. This why also defining what an attack is in section 5.2 is unclear ("Attacks that append content succeed more often, in 38% and 52% of cases, mostly in prose."), specifically, what is the nature of requested content to be added?

  • The success of the method is limited when considering generalization. Beyond QA that was used in training, MINJA EHR still shows 36.2% ASR. Also, Trojan Hippo has relatively high ASRs.

  • The evaluation under "Agentdojo" does not necessarily show that the model preserves the task (that it follows task-related instructions), because a model that ignores everything in the external tool outputs, will show low ASR by ignoring injected instructions in Agentdojo. The paper should show clear examples of loss of utility (mentioned in section 4.4).

  • Since the training now generalizes only to very similar attacks compared to training (e.g., formatting), what is the baseline of prompting the model that only minor preferences such as formatting should be followed and anything else can be ignored?

[1] "No More, No Less: Task Alignment in Terminal Agents" NeurIPS E&D 2026 [2] "AI agents may always fall for prompt injections" NeurIPS 2026

Questions for authors

I don't have questions that would change my current assessment. I think the general topic of the paper (task alignment) is important, but the evaluation scope and setup is very narrow. I think the paper should either significantly reduce its claimed scope, or discuss how to generalize further to complex agentic setups.

Reviewer_xgFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces task-preservation training, which uses clean completions as targets under conflicting memory records while retaining benign instructions. Training lowers attack success rate and can transfers to tool-calling tasks, at some cost in utility. The model resists instructions that replace its answer, but only partially resists attacker content appended to a correct answer.

Critical strengths
  • Target both security and task utility. The paper measures attack resistance alongside task utility. It also tests whether the trained model still uses retrieved facts and follows benign formatting instructions.
  • Transfer beyond the training tasks. Training uses a small set of question-answering examples, yet also reduces attack success in other settings, including email agents and tool-calling tasks.
  • Detailed analysis of what the model learns. Behavioral tests distinguish resistance to answer replacement from resistance to unwanted additions. Activation patching further examines where the trained model’s response to the attacker’s demand changes.
Critical weaknesses
  • From the results, the model still struggles to separate useful instructions from malicious ones. It resists attacks which replace the answer, but I see attacks that append unwanted content still succeed in 38% and 52% of tasks across two test sets. In the Qwen2.5-14B component study, compliance with unseen benign formats drops from 92.5% to 45.5% after training, which suggest that preserving the correct answer is not enough to distinguish safe additions from harmful ones. This limits the method’s ability to protect the task while keeping useful memory instructions.
  • The benefit over existing training-based defenses is unclear. Most main comparisons are against runtime defenses, rather than methods such as StruQ or SecAlign. The comparison with the released Meta-SecAlign model is limited to Llama-3.1-8B. On AlpacaFarm, the proposed method has a worst-case attack success rate of 36.1%, compared with 0% for Meta-SecAlign under the reported scorer. A comparison on the same model with comparable training budgets, measuring both security and useful memory use, is needed to establish what the proposed training adds.
Questions for authors
  • The model follows unseen formatting instructions in the supplied-memory frame, but ignores many of them in the ReAct-style training frame. Could you explain this difference? Is it due to the prompt format, or to the different questions used in the two tests?
  • Could you test the same addition with the user explicitly allowing it in one condition and forbidding it in the other, while keeping the question and memory unchanged? This would help me know whether the model distinguishes authorized additions from unwanted ones, rather than simply accepting or rejecting certain formats.
  • Comparison with training-based defenses. Could you compare with StruQ or SecAlign on the same base model using comparable training data? It will be helpful if authors could report attack success, clean utility, and compliance with benign memory instructions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.