- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper revisits SecAlign, a DPO-based defense for indirect prompt injection, and shows that at 70B scale it loses most of its agent utility (AgentDojo drops from 59.8% undefended to 6.2%; WASP from 62.2% to 45.9%). The authors trace this to two problems in SecAlign’s preference data. First, injections are always placed at the end of the data, so the model learns to ignore whatever instruction comes last. Second, the response labels come from an old text-davinci-003 dataset that is off-distribution for current models. Meta-SecAlign fixes both: it places injections at the start or end of the data at random, and it generates both preferred and dispreferred responses from the base model itself. On Llama-3.3-70B this keeps utility within 2.7 percentage points of the undefended model on eight benchmarks, brings ASR to single digits on six injection benchmarks, and holds up better than SecAlign under a GCG adaptive attack. The recipe is also applied to Llama-3.1-8B, Llama-4-Scout, Qwen3-4B, and Qwen3.6-27B, and the paper shows that LoRA α can be adjusted at test time to move along the utility–security trade-off.
Critical strengths
-
Showing that SecAlign breaks down at 70B scale (AgentDojo utility 6.2%) is a useful finding for the community. The paper identifies the failure mechanism—the model outputs nothing when the user prompt is the last instruction in the context (Figure 2)—and the fix is simple.
-
The experiments are broad: five model families up to 109B, eight utility benchmarks, six security benchmarks, and a GCG adaptive attack. The test-time LoRA-α interpolation (Figure 3) is a practical way to adjust the utility–security trade-off without retraining.
Critical weaknesses
-
The paper presents two separate takeaways (+69.0 pp from position randomization, +10.9 pp from self-generated labels), but combining Tables 1 and 4 shows that neither technique works on its own. Position randomization with davinci labels gives 15.5% AgentDojo utility, self-generated labels with end-only injection also give 15.5%, and only the combination reaches 84.5%. Presenting them as independent contributions in the introduction is not supported by the data, and it undercuts the mechanism in Section 3.1: if the end-position shortcut were the whole story, randomizing position should help regardless of the label source, and it does not. This matters because the paper’s methodological narrative rests on this attribution.
-
The headline result—Llama-3.3-70B utility rising from 59.8% to 84.5% after defense—should not be read as utility preservation. Footnote 1 links the increase to two non-security failure patterns in the undefended model (no tool call emitted, tool-call loops) and says the cause is unclear. The likely reading is that fine-tuning repaired the base model’s tool calling rather than preserved it, and every other family shows a small drop instead (e.g., Qwen3-4B from 43.3% to 42.3%). The clean evidence for the paper’s claim is those small drops, and they should be the headline, not the 70B jump.
Questions for authors
-
Does position randomization alone reduce the empty-output rate in Figure 2 at all when davinci labels are kept?
-
What fraction of undefended Llama-3.3-70B failures are empty tool calls or loops, and does a control SFT on clean Alpaca data (no DPO, no injections) also bring utility to about 84%?
-
Can you report SecAlign baseline numbers for the four other model families in Table 6?
-
Can you run at least one adaptive attack against the defended models on an agentic benchmark (AgentDojo or InjecAgent)?