acceptodds
Desk rejected at ICLR 2027

SecStep: Guarding Agents Against Prompt Injection via Action Backtracking

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) increasingly act as autonomous agents, interacting with tools and environments to execute multi-step tasks. Untrusted observations expose them to indirect prompt injection (IPI), where malicious instructions in tool outputs redirect execution away from the user’s goal. Existing defenses incur latency and utility costs through inference-time filtering or verification, or fine-tune on finite attack templates that may not generalize to adaptive injections. We introduce SecStep, a training-based defense that embeds action backtracking into the agent policy. SecStep enables agents to audit observations, detect conflicts with trusted user goals, isolate malicious instructions, and backtrack from contaminated paths to resume safe execution. To optimize these behaviors in long-horizon workflows, we further propose SecStep-RL, which combines implicit step-level rewards derived from trajectory-level preferences with global outcome rewards, providing dense supervision without costly process annotations. Our theoretical analysis shows that trajectory-level preference optimization can induce step-level rewards. Evaluations on AgentDojo and Agent Security Bench across open-weight and closed-source models demonstrate an improved security-utility trade-off. SecStep-RL achieves near-zero attack success rates under static attacks and 5.83% and 2.80% on Qwen3-4B and Llama-3.1-8B, respectively, under defense-aware adaptive attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Prompt Injection Attack, LLM Agent, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_kUDwOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new training time defense against indirect prompt injection. Specifically, they teach an agent "action backtracking", where before executing a tool call agents can audit their own intermediate observations, identify if there is a conflict between the execution path and the original user intent and recover from contaminated execution paths.

Authors implement this using a RL framework, specifically designed for sparse and delayed rewards in agent defenses using step level rewards.

They evaluate their method on 3 models (2 open source, one GPT mini), develop their method using data disjoint from the ones used for evaluation, and evaluates attack success and utility across benchmarks and (adaptive) attacks. Quite consistently across setups, the proposed method reduces ASR while maintaining great utility under attack, and better than previously proposed defenses.

Finally, authors also provide secondary results, including a comparison of defense overhead in training and inference, and ablations on the method.

Critical strengths
  • The paper is very well written and nice to follow.
  • Section 3.2 on the limitations of training based defenses is well argued, and intuitively lead to the proposed method relying on action backtracking.
  • The method is well explained and the proposed RL framework seems to nicely fit the goal of action backtracking.
  • Careful contamination between training and evaluation data/tasks.
  • The empirical results are extensive across models, benchmarks and baseline methods; and the results on ASR and UA for the proposed method are compelling across the board.
  • The inclusion of adaptive attack results is great, and also here the proposed method excels. I particularly like the inclusion of the decreasing ASR with steps in SecStep-RL, it shows a clear trend.
  • The results on comparing defense overhead in training and inference are a great addition, and only further emphasize the benefits of the method.
Critical weaknesses

Overall, I really like the paper. It's an intuitive method, well argued, and the extensive empirical results convincingly show the method's robustness and utility preservation.

So the weaknesses are limited, and below I add some thoughts and questions I have

  • I'm not an expert in RL post-training, but wonder how novel the individual components are in stage II of the method. It's not an issue, as the application of the methods to PI defenses and the compelling results speak for themselves, but as a reader with limited background in this i'd be interested to read this more explicitly.
  • Maybe it would be cool to further evaluate the method's data efficiency, showing how ASR and utility evolve with training samples: can you do with even less? Or can you further improve the method?
  • And would the authors have evidence that this method would generalize better across OOD data than other defenses?
  • I wonder whether the great results from the method really come from the RL framework + action backtracking, or might just come from a high quality dataset for post-training the defense. I believe appendix G partially answers this, but it would be great to also see the performance when the method is trained on the same data but without the explicit backtracking. Or is that not really possible?
  • Would it be possible to further push the adaptive attack success rate? I understand that the successful attacks explicitly try to skip backtracking, but I wonder if a greater attack budget or other insights would further improve it. Again not an issue, but I'm generally curious to push training time defenses to their limits.
Questions for authors

See weaknesses

2 comments

We appreciate your encouraging assessment and suggestions.

W1. Which parts of Stage II are new?

Stage II obtains action-level feedback for prompt-injection defense from an online trajectory preference model. Building on DPO, we train this model on complete rollouts ranked by their outcomes. Its log-ratio against the frozen behavior policy rϕ(hk,ak)=βlog⁡πϕ(ak∣hk)πθt(ak∣hk),r_\phi(h_k,a_k)=\beta\log\frac{\pi_\phi(a_k\mid h_k)}{\pi_{\theta_t}(a_k\mid h_k)}, provides an implicit score for each audit or tool action. Theorem 1 connects the sum of these scores to trajectory-level preference learning under its stated assumptions.

This formulation suits prompt-injection defense because the consequential trust decision occurs during the interaction, while safety and task completion are judged at the end. Outcome-only feedback is delayed and gives limited guidance on which intermediate decision caused a failure. Action-level scores provide feedback at these decision points without step annotations or additional rollouts. They also match the granularity of audits and tool calls, avoiding the need to assign separate credit to every generated token.

W2. How do security and utility change with training data?

Following your suggestion, we varied the Stage-I dataset between 500 and 2,000 demonstrations and evaluated each setting before RL and after 100 RL updates on Qwen3-4B. All metrics are percentages on AgentDojo.

Stage-I examplesRL updatesBU ↑Static ASR ↓Static UA ↑Mean adaptive ASR ↓
500063.572.8656.1915.48
2,000065.640.8659.2410.48
50010069.070.0063.665.83
2,00010070.100.0064.902.32

W3. Does the defense transfer to unseen environments?

Table 12 evaluates WASP web tasks after training on AgentDyn. On Qwen3-4B, SecStep-RL has no successful attacks among 84 injected cases, compared with 11.90% ASR for Meta-SecAlign. Benign utility is also higher. The same ordering holds on the 27B backbone. These results support better transfer to the evaluated web-agent environment.

W4. How much of the gain comes from the dataset itself?

To isolate the contribution of the backtracking trace, we trained Meta-SecAlign on our 2,000 tasks after removing the audit and branch trace, and compared it with SecStep trained with backtracking. This set is about one-tenth the size of the original Meta-SecAlign dataset. The results below are percentages on AgentDojo with Qwen3-4B. SecStep achieves better security and utility than Meta-SecAlign trained on the same examples, supporting the benefit of the training procedure beyond the choice of tasks alone.

Training methodBenign utility ↑Static ASR ↓Static UA ↑Mean adaptive ASR ↓
Meta-SecAlign, original data61.173.5258.5020.89
Meta-SecAlign, our data without backtracking60.8926.5257.2438.00
SecStep, our data with backtracking65.640.8659.2410.48

W5. Can stronger attackers push ASR higher?

We used GPT-6-Sol as the TAP attacker and increased the budgets of Genetic Search and RL-Hammer from 200 to 1,000 and from 80 to 400, respectively. Entries are ASR (%) on Qwen3-4B. The two SecStep-RL columns use 10 and 100 defense training updates.

Attack settingMeta-SecAlignSecStepSecStep-RL (10)SecStep-RL (100)
TAP, Llama-3.1-8B16.198.574.761.67
TAP, GPT-6-Sol52.8611.906.672.62
Genetic Search, budget 20031.9017.149.764.76
Genetic Search, budget 1,00069.0522.8613.336.90
RL-Hammer, budget 8014.296.192.860.00
RL-Hammer, budget 40086.1919.7615.001.43

With the larger budgets, RL-Hammer reaches the highest ASR against the 10-update policy (15.00%), while Genetic Search reaches the highest ASR against the 100-update policy (6.90%). The successful payloads in Appendix C include forged safety checks and instructions to skip backtracking, showing how adaptive attacks can target the defense itself.

Reviewer_kUDwOpenReview ↗

Answer to rebuttal

Many thanks for the detailed response to my review. I'm happy to keep my positive score and hope the paper gets accepted.

Reviewer_DkcHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper suggests training AI agents to "backtrack" to avoid prompt injection attacks. This basically means: when the agent suspects it would do something dangerous, it should notice, and then ignore recent instructions or those it thinks are responsible for the unsafe behavior. The training involves two stages: Stage 1 is supervised learning ("Backtrack Format Tuning") and Stage 2 is reinforcement learning.

The method seems to produce very strong empirical results as evaluated via benign utility (BU), attack success rate (ASR), and utility under attack (UA), as well as wall clock time and number of tokens.

Critical strengths

The idea seems interesting and promising. The results look very strong.

Critical weaknesses

The bigggest weakness is that crucial details of how the supervised learning dataset is constructed are missing; the best description I found was Figure 1.

There were other issues of clarity:

  • The meaning of eqn 1 wasn't clear enough... what does "the policy contribution to [a trajectory's] likelihood" mean more precisely/formally? This equation seems to neglect any influence of policy that routes through the dynamics of the environment, so I'm not convinced it is well described by this phrase.
  • The relevance of theorem 1 is not explained.
  • Some kind of AI slop-y text like: “Our scope is instruction-driven redirection through external content; backtracking does not provide transactional rollback for tool actions that have already changed the environment.”

I'm rating this as a "weak accept" on the assumption that these issues are fixable and will be fixed.

Questions for authors
  • Question: Eqn 5 seems to encourage backtracking, even when it is not necessary (no attack is present). Is that the case? Does it lead to any issues?
  • Suggestion: for Figure 3, it would be useful to know how this differs across settings where there is(n’t) an attack present, similar to BA/UA.
  • For the data used in training and in evaluation, is the payload always immediately after the prompt injection?
1 comment

Thank you for the careful reading and the specific suggestions on clarity.

W1. How were the supervised demonstrations constructed?

Appendix A.1 describes the AgentDyn seed tasks, GPT-4o-mini trajectory generation, quality filtering, and checks for overlap with evaluation data. Each retained demonstration shows the agent auditing an injected instruction, preserving useful context, and completing the legitimate task. We expanded these details in the revision and included the generation prompt and other details; we highlighted the additions in blue.

W2. What probability does Eq. (1) represent?

Let hkh_k contain the request, past actions, and observations through oko_k. For a trajectory ending at action aKa_K, the joint likelihood is

= p(o_1\mid x) \left[\prod_{k=1}^{K}\pi_\theta(a_k\mid h_k)\right] \left[\prod_{k=1}^{K-1}P(o_{k+1}\mid h_k,a_k)\right].$$ Eq. (1) is the first bracketed product, evaluated along the recorded history. Earlier actions affect subsequent observations through $P$ and therefore change which histories the policy reaches. When two policies score the same recorded trajectory in a shared environment, the initial-observation and transition factors cancel in their likelihood ratio, giving the ratio used in Eq. (7). We will state this factorization and define the action factor explicitly. **W3. What does Theorem 1 contribute to the method?** Theorem 1 explains the reward parameterization used in Stage II. Our supervision ranks complete rollouts by their outcomes, while the policy update uses feedback for individual actions. Under KL optimality, deterministic transitions, and zero terminal value, the action log-ratio in Eq. (6) equals the underlying reward plus a difference of state values. These value terms telescope, so the summed action scores differ from the original return only by an initial-state constant. For rollouts starting from the same history, they therefore induce the same Bradley–Terry preference probabilities. Their sum is also exactly the trajectory log-ratio optimized by DPO in Eq. (7). This gives a reward interpretation to learning action scores from trajectory preferences without step annotations. We use these scores as shaping signals alongside the outcome reward in Eq. (8), and test their practical benefit through the outcome-only ablation. With stochastic tool responses, the log-ratio factorization remains exact, but the equivalence to realized returns need not hold. We have clarified this connection and scope in the revision. **W4. Wording of the threat model** Thanks for your suggestion. We have rewritten the relevant passages to describe the threat model and the timing of backtracking more directly. **Q1. Does Eq. (5) encourage unnecessary audits?** Yes, Eq. (5) can reward an audit on a benign task if the trajectory ends safely and successfully. We allow such audits because checking a plan can be useful even without an attack. The Qwen3-4B benign-task breakdown below shows approximately 1%~3% more tokens and latency for SecStep-RL. Table 12 also reports modest utility gains on $\tau^2$-Bench and BFCL, consistent with preserving useful reasoning while learning the defense. As shown in Table 14, benign ORR falls from 15.5% after Stage I to 5.2% after Stage II. An audited trajectory that fails the task or safety check receives a negative bonus, and repeating audits earns no extra bonus. These incentives, together with step-level shaping, discourage unproductive audits. I think a judge with an LLM can provide more selective feedback, which is a possible direction for refining the reward. **Q2. How does inference cost differ between benign and attacked tasks?** Figure 3 reports aggregate costs on Llama-3.1-8B. We have a separate AgentDojo breakdown for Qwen3-4B. Each entry gives mean output tokens/latency in seconds. | Method | Benign tasks | Attacked tasks | | ------------- | ------------: | -------------: | | No defense | 531 / 5.41 | 552 / 5.67 | | MELON | 1,178 / 11.26 | 1,235 / 11.84 | | Meta-SecAlign | 536 / 5.47 | 559 / 5.74 | | SecStep | 549 / 5.63 | 585 / 6.06 | | SecStep-RL | 536 / 5.47 | 583 / 6.06 | **Q3. Where does the payload appear?** We understand “payload” here to refer to the `<|backtrack|>` / `<|branch|>` response generated after the agent reads an injected observation. In our demonstrations, this response usually follows the injected instruction immediately. The requirement is that the audit and plan revision occur before the agent executes the tool call influenced by the injection. Intervening reasoning is therefore allowed.
Reviewer_dU7POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SecStep, a training-based defense against indirect prompt injection in tool-using language agents. The policy learns to audit observations and revise contaminated plans before executing tool actions. Initial format tuning teaches this backtracking behavior, followed by SecStep-RL, which combines outcome rewards for safe task completion with implicit action-level rewards learned from trajectory preferences. The reward model is used only during training. Experiments evaluate security and task utility on AgentDojo and Agent Security Bench, with additional defense-aware adaptive attacks and transfer evaluations. The reported results show substantial improvements on the main static benchmarks and reduced adaptive attack success, accompanied by modest inference overhead.

Critical strengths
  1. The paper addresses an important aspect of prompt-injection defense that detection alone does not resolve. An agent may still need the factual content of an observation that also contains malicious instructions. Training the policy to identify the conflict, revise its planned action, and continue the original task directly addresses this difficulty.
  2. The training objective explicitly links security to successful task completion. The outcome reward requires both conditions to hold, while the implicit process model derives action-level feedback from trajectory preferences without requiring separately annotated intermediate decisions. This is a practical training design for multi-step agents, where terminal feedback provides limited information about which earlier decision led to failure.
  3. The main experiments provide substantial evidence of an improved security–utility trade-off. On ASB, Llama-3.1-8B with SecStep-RL reduces ASR from 6.75% with Meta-SecAlign to 0.50%, while increasing utility under attack from 58.65% to 81.00%. The defense-aware attacks and evaluations on additional environments broaden the evidence beyond static benchmark payloads. The explicit training-data filtering procedure also helps assess the transfer results.
  4. The method has a favorable deployment profile in the reported setting. The process reward model is used only during training, and inference requires a single trained policy. On Llama-3.1-8B, the default SecStep-RL configuration increases mean inference time from 8.15 to 8.58 seconds. This modest increase is a practical advantage given the substantial security improvements reported on the main benchmarks.
Critical weaknesses
  1. The contribution of action backtracking is not sufficiently separated from that of the defensive training data. The demonstrations already contain the desired behaviors: identifying an injected instruction, preserving relevant information, and completing the legitimate task. Comparisons with SecAlign and Meta-SecAlign change both the training procedure and the training examples. Appendix G provides useful evidence that audits frequently precede blocked attacks, but this does not establish whether demonstrating the backtracking process offers an advantage over learning secure actions from the same contaminated tasks.
  2. The evidence for the specific benefit of implicit step rewards is narrower than the main robustness results. Table 3 includes a useful outcome-only RL comparison and supports adding step rewards in the evaluated static setting. However, this analysis is restricted to Llama, while the adaptive experiments compare format tuning with the complete RL procedure. They therefore do not distinguish improvements due to step-level shaping from those obtained through further optimization of the outcome reward. This leaves an important part of the methodological contribution insufficiently tested in the setting where it is most consequential.
  3. The adaptive evaluation leaves the security–utility claim incomplete. Figure 2 reports whether the attacker achieves its objective, but not whether the agent completes the original task under those same attacks. An optimized injection can disrupt legitimate execution without satisfying the attacker’s success condition. The static experiments distinguish these outcomes carefully; corresponding task-completion results are needed to determine whether the utility advantage also holds under defense-aware attacks.
Questions for authors
  1. How would a policy trained on the same contaminated tasks and safe tool actions perform without the demonstrated backtracking process? A comparison with secure trajectory imitation would help identify what the proposed procedure contributes beyond the training examples.
  2. Does the outcome-only RL variant with α=0\alpha=0 retain a substantial performance gap from the full method under the adaptive attacks in Figure 2? This would clarify whether implicit step rewards explain the additional adaptive robustness.
  3. What is the task completion rate under the adaptive attacks, particularly Genetic Search? When an attack is blocked but the original task still fails, what typically prevents successful recovery?
2 comments

W1 / Q1. How much does the backtracking demonstration contribute on the same data?

To compare the methods on the same training data, we trained Meta-SecAlign on our 2,000 tasks after removing the audit and branch trace, and compared it with SecStep trained with backtracking. This set is about one tenth the size of the original Meta-SecAlign dataset. The results below are percentages on AgentDojo with Qwen3-4B. SecStep achieves better security and utility than Meta-SecAlign trained on the same examples, supporting the benefit of the training procedure beyond the choice of tasks alone.

Training methodBenign utility ↑Static ASR ↓Static UA ↑Mean adaptive ASR ↓
Meta-SecAlign, original data61.173.5258.5020.89
Meta-SecAlign, our data without backtracking60.8926.5257.2438.00
SecStep, our data with backtracking65.640.8659.2410.48

W2 / Q2. Does the step reward help under adaptive attacks on Qwen?

We evaluated outcome-only RL and the full method on Qwen3-4B using the same 420 held-out pairs. Each entry reports ASR ↓ / UA ↑ (%). The full method improves both metrics under every adaptive attack, whereas outcome-only RL leaves the TAP and Genetic Search ASRs at the Stage-I level.

MethodStaticTAPStrategyGenetic SearchRL-Hammer
SecStep (Stage I)0.86 / 59.248.57 / 57.6210.00 / 56.6717.14 / 54.526.19 / 57.14
SecStep-RL (α=0\alpha=0)0.71 / 62.148.57 / 60.718.10 / 60.0017.14 / 58.104.05 / 60.48
SecStep-RL (α=1\alpha=1)0.57 / 63.664.76 / 62.145.95 / 61.439.76 / 59.762.86 / 61.90

W3 / Q3. Do agents finish the user’s task under adaptive attack?

We measured UA (%) on the same Qwen3-4B runs. SecStep-RL preserves its utility advantage under every adaptive attack, including Genetic Search. Dashes indicate unreported values.

MethodStaticTAPStrategyGenetic SearchRL-Hammer
Meta-SecAlign58.5055.4853.5750.2454.76
SecStep59.2457.6256.6754.5257.14
SecStep-RL63.6662.1461.4359.7661.90

Thanks for taking the time to respond to my comments. I’ll keep my original score and continue to lean toward weak acceptance.

Reviewer_eu6qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SecStep and SecStep-RL, a two-stage training defense against indirect prompt injection (IPI) in tool-using agents. Stage I (SecStep) fine-tunes the agent on 2,000 synthetic AgentDyn trajectories to emit <|backtrack|> and <|branch|> meta-tokens, auditing planned actions and recovering benign facts before executing unsafe tool calls. Stage II (SecStep-RL) optimizes the policy over 900 on-policy rollouts by combining a programmatic trajectory outcome reward with an online DPO-based implicit process reward (PRM). Across AgentDojo, ASB, WASP, and DTap, the method achieves near-zero static ASR, lowers adaptive ASR to 2.80%–5.83%, and improves Utility Under Attack (UA) with minimal latency overhead.

Critical strengths
  • Resolving the Detection-Recovery Trade-off: Unlike external detectors (PIGuard, DataSentinel) that discard entire contaminated tool outputs along with useful task facts, SecStep isolates the injected span and resumes execution via <|branch|>, boosting AgentDojo UA on Qwen3-4B from 58.50% (Meta-SecAlign) to 63.66% (Table 1).
  • Strong Transfer and Defense-Aware Evaluation: Trained only on decontaminated AgentDyn tasks, the policy transfers zero-shot to AgentDojo, ASB, WASP, and DTap, and holds up against adaptive attackers that explicitly attempt to forge <|backtrack|> audit traces (Appendix C).
  • High Training and Inference Efficiency: SecStep-RL takes only 4.5 A100 GPU hours (vs. 25 hours for Meta-SecAlign, Table 2) and adds < 5% inference token/latency overhead (Figure 3).
Critical weaknesses
  • Unclear Necessity of the Implicit PRM Over Outcome-Only RL: While Section 4.2 heavily emphasizes the online DPO-based PRM, Table 3 (Llama-3.1-8B on AgentDojo) shows that Stage-I SFT already cuts static ASR from 13.49% to 0.43%, and outcome-only RL (alpha = 0, no PRM) reaches 0.29% ASR and 43.62% UA—leaving a very small marginal gain for the PRM (alpha = 1: 0.00% ASR, 45.00% UA). Crucially, alpha = 0 is never evaluated under adaptive attacks (Figure 2), leaving it unclear whether the PRM is actually needed for adaptive robustness.
  • Unexplained Drop in Benign Over-Reflection Under an Unconditional Backtracking Bonus: Equation (5) and Appendix A.2 award a +gamma bonus whenever <|backtrack|> is emitted in a successful trajectory (r_task = 1), even if no injection is present. Logically, this incentivizes spurious backtracking on benign tasks, yet Table 14 reports that Stage II reduces the benign Over-Reflection Rate (ORR) by 3x (15.5% to 5.2%). Moreover, the value of gamma is omitted from Table 4.
  • Attacker-Defender Capability Parity in Adaptive Evaluation: TAP and RL-Hammer use an undefended Llama-3.1-8B with a small query budget (B = 80, Table 5) as the attacker. The lower adaptive ASR on Llama-3.1-8B (2.80%) compared to Qwen3-4B (5.83%) may partly reflect capability ceilings when Llama-3.1-8B attacks a defended version of itself rather than facing a stronger frontier attacker.
Questions for authors
  • Clarification Requests:
    1. Adaptive ASR of Outcome-Only RL (alpha = 0): What is the ASR and UA of SecStep-RL (alpha = 0) (outcome reward only, without the implicit PRM) under the four adaptive attacks in Figure 2?
    2. Value of gamma and Mechanism Reducing Benign ORR: What is the exact value of gamma in Eq. (5), and does the Stage-II task pool include benign tasks? Why does Stage II reduce benign ORR (Table 14) when Eq. (5) rewards emitting <|backtrack|> on any successful task?
    3. Audit Miss vs. Recovery Failure on DTap Coding/Legal: In the failed Coding (24.5% ASR) and Legal (23.5% ASR) cases in Table 13, does the model fail to emit <|backtrack|> (audit miss), or does it emit <|backtrack|> but still follow the injection afterward (recovery failure)?
1 comment

Thank you for the detailed assessment.

W1 / Q1. What does the implicit PRM add beyond outcome-only RL?

To assess the PRM under adaptive attacks, we evaluated both RL variants on Qwen3-4B using the same 420 held-out AgentDojo task–injection pairs as in Figure 2. Each entry reports ASR ↓ / utility under attack (UA) ↑, in percent.

MethodStaticTAPStrategyGenetic SearchRL-Hammer
SecStep (Stage I)0.86 / 59.248.57 / 57.6210.00 / 56.6717.14 / 54.526.19 / 57.14
SecStep-RL (α=0\alpha=0)0.71 / 62.148.57 / 60.718.10 / 60.0017.14 / 58.104.05 / 60.48
SecStep-RL (α=1\alpha=1)0.57 / 63.664.76 / 62.145.95 / 61.439.76 / 59.762.86 / 61.90

Outcome-only RL improves task completion, but leaves TAP and Genetic Search ASR at the Stage-I level. Adding the implicit step reward reduces ASR and improves UA under all four adaptive attacks.

W2 / Q2. Why does benign over-reflection fall when backtracking earns a bonus?

We use γ=0.2\gamma=0.2 and will include it in Table 4. All 1,120 Stage-II tasks contain an injected observation, so benign ORR measures how the learned behavior transfers to clean tasks.

The bonus is paid once per trajectory, and its sign depends on safe task completion. Repeated audits earn no additional bonus, while an audited trajectory that fails receives −γ-\gamma. This discourages audits that discard needed information or derail the task. The implicit step scores also distinguish actions within a rollout, giving the policy feedback on its audit and tool-use decisions. Together, these incentives are consistent with the observed reduction in benign ORR from 15.5% after Stage I to 5.2% after Stage II.

W3. Could attacker strength explain the difference between backbones?

We agree that attacker strength can affect the cross-backbone comparison. We tested this on Qwen3-4B using GPT-6-Sol as the TAP attacker and larger budgets for Genetic Search and RL-Hammer. Entries are ASR (%), and 10 and 100 denote defense training updates.

Attack settingMeta-SecAlignSecStepSecStep-RL (10)SecStep-RL (100)
TAP, Llama-3.1-8B16.198.574.761.67
TAP, GPT-6-Sol52.8611.906.672.62
Genetic Search, budget 20031.9017.149.764.76
Genetic Search, budget 1,00069.0522.8613.336.90
RL-Hammer, budget 8014.296.192.860.00
RL-Hammer, budget 40086.1919.7615.001.43

Increasing the RL-Hammer budget raises ASR against the 10-update policy from 2.86% to 15.00%. SecStep-RL nevertheless retains lower ASR than Meta-SecAlign and Stage-I SecStep in every stronger setting. These results quantify sensitivity on Qwen. A corresponding stronger-attacker evaluation on Llama would be needed to resolve the cross-backbone comparison.

Q3. What goes wrong in the failed DTap Coding and Legal cases?

We inspected all successful indirect attacks against GPT-4.1-mini with Stage-I SecStep in these two domains. Among the 98 Coding cases, 71% had no <|backtrack|> before the injected call and 29% had an audit followed by the unsafe action. The split was 67% and 33% across the 94 Legal cases. Audit misses were more frequent in both domains, where injected instructions in code comments or contract clauses often resembled ordinary task content.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.