acceptodds
Under review as a conference paper at ICLR 2027

SABER: Toward Reliable Agent Red Teaming via Stage-Aware Bounded Repair

Abstract

Agent red teaming (ART) proactively uncovers vulnerabilities in LLM agents by constructing attack instances that induce malicious behaviors. Arguably, this process can be formulated as an optimization problem that searches for attack instances capable of steering target agents toward malicious goals. Among existing approaches, agentic ART leverages the capabilities of red teaming agents to construct and iteratively refine complex attacks, making it a promising paradigm for ART. However, we demonstrate that existing agentic ART methods struggle to make reliable optimization progress, as reflected in both forward evaluation and backward construction. We attribute these difficulties to a common weakness: existing feedback provides limited diagnosis of the progress underlying an execution failure and how that progress relates to the attack instance being optimized. Based on this insight, we propose Stage-Aware Bounded Evidence-guided Repair (SABER) for more reliable agentic ART. SABER identifies attack progress through five ordered stages and grounds the identified stage in execution evidence associated with concrete attack-instance components. It then uses this diagnosis to construct bounded repair objectives, guiding targeted modifications while preserving components that support achieved progress. Extensive experiments across diverse tasks, target agents, and threat models demonstrate the effectiveness of SABER.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Security, Agent Red Teaming, Prompt Injection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LZP3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SABER, a framework for improving the optimization progress of agentic Agent Red Teaming. It introduces five ordered attack progress stages and combines Feedback Approximation with Construction Approximation to diagnose execution failures and guide targeted, bounded repairs of attack instances. Experiments across direct and indirect prompt injection settings, multiple target agents, and transfer scenarios demonstrate substantial improvements in attack success rates over existing methods.

Critical strengths
  1. Well-motivated method design. SABER connects execution failures to attack-instance components through stage-aware diagnosis and uses explicit modification and preservation constraints to guide targeted repairs. This provides a meaningful approach to improving agentic ART optimization.

  2. Strong empirical results. SABER substantially outperforms existing baselines across multiple target agents under both IPI and DPI. The component ablations, human audit, transfer experiments, and optimization dynamics analyses further support its effectiveness.

Critical weaknesses
  1. Limited validation of strict repair-boundary enforcement. Although Appendix D.3 demonstrates improvements from enforcing bounded repair constraints, the incremental gains are relatively modest and evaluated on only one target agent. Further validation across different targets would strengthen the evidence for this design choice.

  2. Limited evaluation of execution robustness and generalizability. The evaluation uses 96 selected cases from DTap. Despite the promising results across multiple target agents, repeated execution success rates and statistical uncertainty are not reported, making it difficult to assess robustness under stochastic agent behavior and generalizability beyond the evaluated platform.

Questions for authors
  1. Can the authors further validate the contribution of bounded scope enforcement across different target agents and provide statistical evidence for its improvements?

  2. How stable are the discovered attack instances across repeated target-agent executions, given the stochastic nature of agent behavior?

  3. How would SABER compare with existing methods under matched attacker-side token budgets, considering the additional computational overhead reported in Appendix D.4?

5 comments
AuthorsOpenReview ↗

Author Response (Part I)

Thanks for your review and comments. We are encouraged by your positive comments on well-motivated design, strong empirical results. We hope following responses could clarify misunderstandings and alleviate your concerns.


Q1 & W1: The gains from bounded scope enforcement are modest and evaluated on only one target agent. Can additional evaluations across different target agents and statistical evidence establish its contribution more convincingly?

R1: Thank you for this insightful question!

  • We first respectfully note that bounded repair is one step within our construction approximation module. Its purpose is to prevent edits addressing the remaining failure from disrupting components that support achieved progress, thereby reducing destructive revisions during iterative optimization.
  • We fully understand your concern about the stability of bounded repair’s contribution across different target agents.
    • We repeat the complete red teaming procedure three times against OpenClaw with DeepSeek V4 Pro and Hermes with Qwen3.7 Max on the same 16 moderate cases, comparing full SABER with No bounded scope variant, which isolates the contribution of bounded repair. We report mean ASR, standard deviation, stage regression rates (the proportion of evaluated transitions in which the execution stage moves backward between consecutive iterations), and paired-test p-values. Considering both the reported standard deviations and p-values, Tables 1 and 2 support bounded repair’s contribution across both target agents. The results show a consistent pattern of higher mean ASR and lower stage regression, with the paired tests supporting the statistical significance of these improvements.
    • To further alleviate this concern, we expand the bounded-repair ablation by doubling the number of evaluation cases, including previously unevaluated cases from DTap. Table 3 continues to support our original conclusion regarding bounded repair’s contribution, with smaller standard deviations indicating more stable performance across independent runs.

We have added these experiments and the accompanying discussion to Appendix D.3 of the revised manuscript.

Table 1. Bounded-repair ablations across target agents.

TargetVariantASR: mean ± SD (%)Regression rate (%)
OpenClaw + DeepSeekNo bounded scope45.83 ± 7.229.84 (25/254)
OpenClaw + DeepSeekFull SABER60.42 ± 9.554.22 (10/237)
Hermes + QwenNo bounded scope72.92 ± 7.2210.81 (20/185)
Hermes + QwenFull SABER87.50 ± 0.006.32 (12/190)

Table 2. Paired permutation tests results, with Holm correction across variant comparisons within each target and metric.

TargetComparisonASR p-valueRegression p-value
OpenClaw + DeepSeekFull − No scope enforcement0.03130.00012
Hermes + QwenFull − No scope enforcement0.03120.00195

Table 3. ASR cross three independent runs on the expanded set of 32 cases.

VariantRun 1Run 2Run 3Mean ASR ± SD
No bounded scope43.7540.6340.6341.67 ± 1.80
Full SABER53.1362.5056.2557.29 ± 4.77
AuthorsOpenReview ↗

Author Response (Part II)

Q2 & W2: (1) Are SABER’s results stable across independent runs? (2) How stable are the discovered attack instances across independent runs? (3) Does SABER’s effectiveness generalize beyond the evaluated platform?

R2: Thank you for your insightful questions!

  • We first respectfully clarify that this setting is used simply to keep computational costs manageable. We deliberately chose DTap for its realistic simulated environments, but its complex, multi-step tasks require extensive agent–environment interactions, making evaluation more expensive than on simpler benchmarks. Our total experimental expenditure has already approached USD 20,000.
  • Arguably, our case selection was designed to be as comprehensive as possible within this budget: we balanced easy, moderate, and complex tasks under both IPI and DPI and distributed cases as evenly as possible across DTap’s application domains.
  • However, we fully understand your concern about the stability of our overall results across independent runs.
    • We repeat the complete red teaming procedure three times on the same 16 moderate cases against OpenClaw with DeepSeek V4 Pro. The standard deviations reported in Table 4 support the stability of SABER’s performance, with results consistent with those in the original manuscript. The 50% ASR reported in our main experiment is the lowest among these three runs, indicating that our reported result does not overstate SABER’s performance across the repeated evaluations.
    • To further alleviate this concern, We further double the number of evaluation cases, including previously unevaluated cases from DTap. Table 5 supports the same conclusion, with smaller standard deviations indicating more stable performance across independent runs.
    • To assess the stability of the discovered attack instances’ effectiveness, we replay 16 discovered attack instances against OpenClaw with DeepSeek V4 Pro, including eight originally successful and eight originally unsuccessful instances, with two replays per instance. Table 6 supports the repeatability of the discovered attacks. It shows that originally successful attacks remain effective in most replays, while originally unsuccessful attacks rarely become successful.
    • To further assess generalizability beyond the evaluated platform, we compare SABER with RedCodeAgent on the full RedCode-Gen benchmark, which was used in the original RedCodeAgent paper. Table 7 further supports SABER’s effectiveness beyond DTap, as it continues to substantially outperform RedCodeAgent on RedCode-Gen.

We have added these experiments and the accompanying discussion to Appendix D.9 and D.10 of the revised manuscript.

Table 4. SABER ASR across three independent runs.

MethodRun 1Run 2Run 3Mean ± SD
SABER50.0062.5068.7560.42 ± 9.55

Table 5. ASR cross three independent runs on the expanded set of 32 cases.

VariantRun 1Run 2Run 3Mean ASR ± SD
SABER53.1362.5056.2557.29 ± 4.77

Table 6. Replay success rates of discovered instances across two additional executions per instance.

Original OutcomeInstancesReplay Success Rate
Successful887.50
Unsuccesful86.25

Table 7. SABER's performance on RedCode-Gen.

ASR on RedCode-Gen
RedCodeAgent45.00
SABER72.50
AuthorsOpenReview ↗

Author Response (Part III)

Q3: How would SABER compare with existing methods under matched attacker-side token budgets?

R3: Thank you for this insightful question! We fully understand your concern about computational fairness and therefore compare SABER with the baselines under matched per-case token budgets, running each budget setting independently without a separate cap on target-agent rollouts.

  • Table 8 shows that SABER significantly outperforms both baselines from 0.3M tokens onward. RedCodeAgent performs better at smaller budgets mostly because its shorter context might allow more attack attempts within the same token budget.
  • More importantly, SABER continues to improve over a wider budget range, while the baselines saturate early. This gives SABER an additional advantage when more computation is available, enabling it to uncover vulnerabilities that remain undiscovered at smaller budgets.
  • Since our main experiments correspond to approximately 0.3M tokens, these results indicate that the reported performance still has room for improvement under larger budgets.

We have expanded the cost-accounting details and added this matched-budget evaluation to Appendix D.4 of the revised manuscript.

Table 8. ASR (%) under matched total token budgets. Each column specifies the same per-case token budget for all methods.

Method0.1M0.2M0.3M (Ours)0.5M0.75M1M1.5M2M3M
AutoRedTeamer12.5025.0025.0025.0025.0025.0025.0025.0025.00
RedCodeAgent43.7543.7543.7543.7543.7543.7543.7543.7543.75
SABER12.5025.0050.0062.5062.5075.0075.0075.0075.00
Reviewer_9bgiOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a framework to progress towards jailbreaks through five stages, each associated with evidence on the concrete components of an attack instance. The method is inspired by (not directly, perhaps, but I see many analogies) propose-then-repair methods that are very popular in recent planning with LLMs, as well as RSI. The key intuition is to preserve, at each stage and through the feedback of a judge, the components that positively contribute to a successful attack. The experiments are run on a set of easy, medium, moderate, and complex tasks and show remarkable ASR compared to the state of the art.

Critical strengths
  1. The abstract and introduction are well written, and the motivation is clear
  2. While the technique is a simple, guided optimization procedure, the results are strikingly strong, with many cases where the ASR doubles compared to the baseline (which is a state-of-the-art technique from 2025/26)
  3. The results are clearly presented, and an ablation study shows the contribution of each component of the method. Also, the paper tests the approach with different models, showing how it generalizes across different backbones
Critical weaknesses
  1. While the abstract and introduction are clearly written, the methodology is somewhat opaque: while the math may lead one to believe that the methodology is hard-coded, the technique is prompt- and judge-based; in this sense, the optimization is no more formal than any other prompting technique the authors compare it to
  2. The paper lacks a proper comparison against existing benchmarks: in this sense, without access to any code, it is impossible to understand whether the method the authors propose works because it is genuinely effective or because the adaptation of existing techniques is misconfigured. I strongly recommend that the authors share the code or report results on a benchmark used in the papers they compare against
  3. SABER consistently achieves a +100% performance improvement over other state-of-the-art methods. As the technique is not introduced very clearly and there are no real insights into why it should guarantee such an improvement, I am unsure how to judge the real contribution of the paper

Update: after rebuttal, I have decided to raise my score to Weak Rejection.

Questions for authors
  1. Can the authors clarify why they did not compare their method against any existing benchmarks and instead introduced a new dataset?
  2. Please provide the code to replicate the results. Given the striking ASR gains introduced by the technique, and without a clear comparison against existing methods on established benchmarks, this is a significant omission that makes it difficult to judge the contribution of the paper
4 comments
AuthorsOpenReview ↗

Author Response (Part I)

Thanks for your review and comments. We are encouraged by your positive comments on clear motivation, strong results, and good presentation. We hope following responses could clarify misunderstandings and alleviate your concerns.


W1: SABER is fundamentally prompting-based, but its mathematical presentation may overstate its methodological formality and obscure its actual contribution relative to other prompting-based methods.

R1: Thank you for this insightful comment.

  • We would first like to clarify that our mathematical presentation is necessary for our analysis, rather than merely serving as formal packaging.
    • Our mathematical formulation provided an analytical framework necessary for developing our argument. Throughout the paper, we used this framework to understand agent red teaming as an optimization problem, analyzed existing methods, and derived insights that guide our method design.
    • We did not claim that SABER is inherently superior to prompting-based methods. Instead, we used this analytical framework to identify potential limitations of existing methods (Section 3), which motivated us to develop a simple yet effective approach to address these limitations by diagnosing execution progress and preserving it during attack refinement.
  • We respectfully note that SABER is not a purely prompting-based method.
    • SABER is a mechanism-driven optimization method that goes beyond enriching prompts, employing stage-aware diagnosis, component association, and verifier-guided bounded repair as feasible approximate optimization approaches to support reliable agent red teaming. These core mechanisms connect execution failures to attack components and enforce explicit boundaries on subsequent modifications.
    • However, we fully understand your concern about whether the ablations fairly attribute SABER’s gains to its core mechanisms, rather than to more detailed prompts. To further alleviate this concern, We compare SABER with two variants that share its repair knowledge and common prompt instructions while isolating the effects of our core mechanisms: V1 removes all mechanisms, whereas V2 removes bounded repair. Table 1 supports the contribution of SABER’s core mechanisms beyond prompting and additional knowledge alone. Under shared repair knowledge and prompt instructions, stage-aware diagnosis and component association improve ASR in both settings, with boundary enforcement providing further gains.

We have expanded the corresponding clarifications in Appendix C.3 and incorporated these additional experiments in Appendix D.8.

Table 1. ASR (%) on moderate-difficulty cases. V1 removes all three core mechanisms; V2 removes only boundary enforcement.

VariantIPI ASRDPI ASR
V118.7550.00
V231.2568.75
Full SABER50.0087.50
AuthorsOpenReview ↗

Author Response (Part II)

Q1 & W2: Why did you introduce a new dataset? Are reported ASR gains reliable, or could they arise from the selected benchmark or misconfigured baseline? Comparisons on established benchmarks are needed to substantiate the method’s effectiveness.

R2: Thank you for this thoughtful question!

  • We first respectfully clarify that we deliberately chose DTap [1] because its realistic simulated environments and complex, multi-step workflows provide a demanding evaluation of whether attacks can induce malicious actions during agent execution. Its growing adoption across agent security studies also reflects increasing recognition among experts in the field. [2][3].
  • We further clarify that our experimental results are reliable: the baselines are correctly configured, and the reported data have not been altered. Although some settings differ slightly from those used in the baselines’ original benchmarks, all baselines exhibit substantial performance drops on DTap. This highlights the necessity of evaluating attack effectiveness on reliable benchmarks with realistic environments and complex workflows.
  • However, we fully understand your concern about whether SABER’s advantage holds in the baselines’ original evaluation settings. To address this, we compare SABER with RedCodeAgent on the full RedCode-Gen benchmark, which was used in the original RedCodeAgent paper.
    • Table 2 supports SABER’s effectiveness beyond the selected DTap cases. It shows that SABER continues to substantially outperform RedCodeAgent.
    • SABER also achieves substantially higher ASR on RedCode-Gen than in the compared DTap setting. This contrast further highlights the value of testing attacks on realistic tasks and complex workflows.
  • Moreover, different methods naturally incur different computational costs per attempt. To further control for this difference, we compare them under matched per-case token budgets, running each budget setting independently without a separate cap on target-agent rollouts.
    • Table 3 shows that SABER significantly outperforms both baselines from 0.3M tokens onward. RedCodeAgent performs better at smaller budgets mostly because its shorter context might allow more attack attempts within the same token budget.
    • More importantly, SABER continues to improve over a wider budget range, while the baselines saturate early. This gives SABER an additional advantage when more computation is available, enabling it to uncover vulnerabilities that remain undiscovered at smaller budgets.
    • Since our main experiments correspond to approximately 0.3M tokens, these results indicate that the reported performance still has room for improvement under larger budgets.

We have clarified the benchmark selection and baseline configurations in Appendix C.1 and C.2, added the full RedCode-Gen evaluation to Appendix D.10, and included the matched-token-budget comparison in Appendix D.4 of the revised manuscript.

Table 2. SABER's performance on RedCode-Gen.

ASR on RedCode-Gen
RedCodeAgent45.00
SABER72.50

Table 3. ASR (%) under matched total token budgets. Each column specifies the same per-case token budget for all methods.

Method0.1M0.2M0.3M (Ours)0.5M0.75M1M1.5M2M3M
AutoRedTeamer12.5025.0025.0025.0025.0025.0025.0025.0025.00
RedCodeAgent43.7543.7543.7543.7543.7543.7543.7543.7543.75
SABER12.5025.0050.0062.5062.5075.0075.0075.0075.00

[1] DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents.
[2] Twin Agent: Context Residual Compression for Privilege Separated Agents.
[3] EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents.
[4] Where You Inject Matters: The Trust Region Effect in LLM Agents.

AuthorsOpenReview ↗

Author Response (Part III)

Q2: Please provide the code to replicate the results.

R3: Thanks for your constructive suggestion.

  • We fully understand that your request for code is motivated by reproducibility concerns. However, to support responsible use and account for potential commercialization, we have not released the full attack-generation harness. Since SABER can generate attacks that could compromise real-world systems, we adopt controlled access for legitimate research and authorized security evaluation, as stated in our Ethics Statement.
  • We respectfully note that following DTap’s release approach, we have already released selected attack instances and execution trajectories to support independent inspection of our results and further security research.
  • To further alleviate your reproducibility concern while supporting responsible use and commercialization, we have deployed SABER as a publicly accessible service on an anonymous server at https://hats-appraisal-sussex-arrangements.trycloudflare.com/, enabling users to run and evaluate our method. Users can provide their own model API configurations and select an evaluation case and budget to run SABER without installing the full harness. We have added the service link and accompanying instructions to the Reproducibility Statement.

W3: SABER reports substantial ASR improvements, but the method and the insights explaining these gains are insufficiently clear, making its contribution difficult to assess.

R4: Thanks for your insightful comment.

  • We respectfully clarify that the original manuscript explained why SABER is effective in several places (Sections 3.3–4 and 5.3, Appendices D.3 and E). We summarize these explanations below.
    • SABER’s effectiveness stems from its targeted design: we first analyzed where existing methods struggle to make optimization progress, then developed a simple yet effective method whose mechanisms directly address the identified limitations.
    • We conducted extensive ablations to validate each module's contribution, and provided detailed analyses of the optimization process to further explain why SABER is effective.
  • To further alleviate your concern, we conduct a more stringent ablation to more fairly attribute SABER’s gains to its core mechanisms, rather than to more detailed prompts. We compare SABER with two variants that share its repair knowledge and common prompt instructions: V1 removes all three core mechanisms, whereas V2 removes only boundary enforcement. Table 4 supports the contribution of SABER’s core mechanisms beyond prompting and additional knowledge alone. Under these shared inputs, stage-aware diagnosis and component association improve ASR in both settings, with boundary enforcement providing further gains.

We have made these contributions more explicit in the revised manuscript, emphasizing how SABER’s targeted design addresses the identified limitations and why it leads to improved effectiveness.

Table 4. ASR (%) on moderate-difficulty cases. V1 removes all three core mechanisms; V2 removes only boundary enforcement.

VariantIPI ASRDPI ASR
V118.7550.00
V231.2568.75
Full SABER50.0087.50
AuthorsOpenReview ↗

Thanks to Reviewer 9bgi

Please allow us to thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of motivation, results and presentation.

Kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the post-rebuttal period. Your feedback will be greatly appreciated.

Reviewer_FJACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SABER, an automated agent red-teaming framework that addresses insufficiently informative failure feedback and destructive revisions during iterative attack optimization. SABER categorizes attack progress into five stages, associates execution evidence with specific injection carriers and payloads, and constructs explicit modification and preservation constraints for bounded repair. Experiments on direct and indirect prompt injection, as well as cross-agent transfer, show substantial improvements in attack success rates. The central contribution is a structured diagnosis-and-repair process that makes attack refinement more targeted.

Critical strengths
  1. Practical motivation and coherent design. The paper distinguishes between failures caused by unobserved payloads, observed but ineffective instructions, and incomplete malicious workflows. These distinctions provide actionable guidance for deciding whether to revise injection carriers or payloads. Explicit modification and preservation constraints offer a reasonable mechanism for reducing destructive revisions.

  2. Substantial empirical improvements with useful supplementary evaluations. On OpenClaw with DeepSeek, SABER improves indirect prompt injection ASR from the strongest comparison baseline’s 37.50% to 62.50%, and direct prompt injection ASR from 70.83% to 95.83%. Module ablations, repair-boundary ablations, transfer experiments, adaptive evaluations against defenses, and human audits provide initial evidence supporting the framework’s effectiveness.

Critical weaknesses
  1. The novelty and attribution of improvements need clearer justification. Prior approaches such as MUZZLE already use execution trajectories and failure feedback to iteratively refine attacks. SABER’s incremental contribution therefore lies primarily in stage-aware diagnosis, component association, and repair constraints. The current experiments do not fully distinguish the benefits of this organization from those of more detailed prompts, seven manually designed repair families, and additional verification. A stronger control should receive the same repair knowledge and computational budget without stage-conditioned organization.

  2. Component association lacks causal validation, and preserving components does not necessarily preserve progress. Associations are largely inferred from consistency between observed behavior and payload instructions, which cannot rule out effects arising from the benign task or interactions among multiple payloads. The 91.7% evidence-support rate in Table 13 establishes the plausibility of these associations rather than their causal validity. Likewise, retaining a carrier or payload does not ensure that the corresponding behavior will recur. Counterfactual component removal or replacement, repeated executions, and direct comparisons of regression rates would strengthen the proposed mechanism.

  3. Equal target interaction budgets do not eliminate computational budget confounds. SABER introduces additional diagnosis, constraint construction, and local repair. Table 10 reports 330.03K attacker-side tokens per successful indirect attack for SABER, compared with 118.05K for RedCodeAgent; the corresponding target rollout costs are 10.70 and 8.67. These results support improved effectiveness under a fixed target interaction cap, but do not establish an advantage under equal total cost. Comparisons under matched token or monetary budgets are needed.

  4. Limited sample sizes and missing statistical analyses weaken several conclusions. Evaluation relies on 96 selected DTap cases, with only 48 cases per injection setting and 16 per complexity group. The paper does not report repeated independent runs, confidence intervals, or paired significance tests. For example, enforcing repair boundaries improves indirect attack ASR from 58.33% to 62.50% in Table 9, corresponding to only two additional successful cases. This provides limited evidence for the independent contribution of boundary enforcement. Evaluation on a second independent benchmark would also strengthen the generalization claims.

  5. Important implementation details remain insufficient for independent reproduction. Although the appendix provides algorithms and condensed prompts, it does not adequately specify the concrete local repair budget BrB_r, complete repair rules and verifier implementations, or sampling configurations. The reproducibility statement primarily offers attack instances, execution trajectories, and controlled API access. These resources facilitate inspection of outcomes, but do not fully enable independent reconstruction of the optimization procedure or verification of its computational budget.

Questions for authors
  1. Clarification: Attribution of the improvements.
    Which repair knowledge, prompts, and verifier feedback are shared between SABER and its comparison methods? In particular, does the construction variant without bounded scope receive exactly the same repair objectives and local editing budget as full SABER? This distinction would help determine whether the gains arise from boundary enforcement or additional guidance and computation.

  2. Clarification: Computational budget and accounting.
    What concrete local repair budget BrB_r is used, and do the costs in Tables 10 and 11 include all diagnosis, judge, constraint generation, and repair calls? Do you have results under matched total token or monetary budgets? These details could materially change my assessment of the comparison’s fairness and practical efficiency.

  3. Evidence request: Reliability of bounded repair.
    For the matched variants in Table 9, can you report the paired case outcomes and stage regression rates, together with any available results across independent runs? The IPI improvement from 58.33% to 62.50% represents only two additional successful cases; evidence that boundary enforcement consistently reduces regression would strengthen my assessment of the central mechanism.

8 comments
AuthorsOpenReview ↗

Author Response (Part I)

Thanks for your review and comments. We are encouraged by your positive comments on practical motivation, coherent design, and substantial improvements. We hope following responses could clarify misunderstandings and alleviate your concerns.


W1 & Q1: Do SABER’s gains arise from stage-aware diagnosis and bounded repair, or from additional repair knowledge and detailed prompts, or computational resources? Which knowledge are shared across methods? Does the unbounded variant receive the same repair objectives and local editing budget as full SABER?

R1: Thank you for your thoughtful questions!

  • We agree that your summary captures SABER’s technical improvements: stage-aware diagnosis, component association, and bounded repair. More fundamentally, we would also highlight that these designs arise from our analysis of existing agentic ART methods, which identifies limited diagnosis of execution progress and its relation to attack instance as a common weakness underlying difficulties in forward evaluation and backward construction.
  • We fully understand your concern about whether the ablations fairly attribute SABER’s gains to its core mechanisms, rather than to more detailed prompts, additional repair knowledge, or greater computational resources. We hereby address these factors.
    • We respectfully clarify that the original ablation comparisons are fair: some prompts and repair knowledge are part of the modules being ablated, and some also help implement their functions. Therefore, removing a module entails removing its corresponding inputs, while keeping the other evaluation setting identical. Specifically, regarding the required additional prompt setting details, Table 1 summarizes the inputs shared across variants, and the unbounded variant receives the same repair objectives and local editing budget as full SABER. This design isolates the effect of boundary enforcement within construction approximation, allowing us to examine its contribution while retaining the other mechanisms.
    • To further alleviate this concern, we compare two variants sharing the same repair knowledge and prompt instructions as full SABER: V1 removes all core mechanisms (stage-aware diagnosis and bounded repair), while V2 removes only bounded repair. Table 2 supports the contributions of SABER’s core mechanisms, consistent with our original conclusion. Adding stage-aware diagnosis and component association improves ASR under both IPI and DPI, and enforcing repair boundaries yields further gains.
    • We further compare the methods under matched computational resources (token budget).
      • Table 3 shows that SABER outperforms both baselines from 0.3M tokens onward. RedCodeAgent performs better at smaller budgets because its shorter context might allow more attack attempts within the same token budget.
      • More importantly, SABER continues to improve over a wider budget range, while the baselines saturate early. This gives SABER an additional advantage when more computation is available, enabling it to uncover vulnerabilities that remain undiscovered at smaller budgets.

We have expanded the corresponding clarifications in Appendix C.3 and incorporated these additional experiments in Appendix. D.

Table 1. Inputs and boundary enforcement across different SABER variants.

VariantBase inputs, attack libraryMemoryStageComponent associationRepair knowledgeRepair objectiveLocal repair feedbackEditing toolsBoundary enforcement
Simple, without approximation modules✓✓✗✗✗✗✗✓✗
FA✓✓✓✓✗✗✗✓✗
FA + CA without bounded scope✓✓✓✓✓✓✓✓✗
Full SABER✓✓✓✓✓✓✓✓✓

Table 2. ASR (%) on moderate-difficulty cases. V1 removes all three core mechanisms; V2 removes only boundary enforcement.

VariantIPI ASRDPI ASR
V118.7550.00
V231.2568.75
Full SABER50.0087.50

Table 3. ASR (%) under matched total token budgets. Each column specifies the same per-case token budget for all methods.

Method0.1M0.2M0.3M0.5M0.75M1M1.5M2M3M
AutoRedTeamer12.5025.0025.0025.0025.0025.0025.0025.0025.00
RedCodeAgent43.7543.7543.7543.7543.7543.7543.7543.7543.75
SABER12.5025.0050.0062.5062.5075.0075.0075.0075.00
AuthorsOpenReview ↗

Author Response (Part II)

W2: Do the evidence–component associations reflect causal contributions? Does preserving the associated components reliably preserve achieved progress?

R2: Thank you for this insightful question.

  • We first respectfully clarify that our method neither assumes perfectly accurate evidence–component associations nor guarantees that the discovered attack instances are optimal.
  • Arguably, even when evidence–component associations are not fully accurate, SABER may still provide a more effective (though not necessarily optimal) optimization path. In contrast, existing methods may rely on less specific attribution of execution failures, which can limit the effectiveness of subsequent refinement.
    • Even simple guidance along this stage-aware path can improve existing methods, although their performance still falls short of SABER. We augment baselines with SABER’s progress-stage feedback while retaining its original refinement procedure. Table 4 shows that this simple augmentation yields modest overall improvements for the baselines, although they still underperform SABER.
  • To further examine the contribution of the identified components, we conduct your suggested counterfactual experiment by rerunning each repaired instance and a counterpart in which the payload identified as supporting progress is replaced with semantically unrelated text. Regression denotes reaching a lower stage than the pre-repair instance. Table 5 shows that preserving these components clearly improves repair effectiveness, reducing regression and increasing advancement. Note that these results describe single-step transitions in an iterative optimization process, where most steps leave the stage unchanged. It explains why advancement and regression rates sum to well below 100% and helps explain the relatively modest magnitude of the per-step improvements.

We have expanded the discussion of stage-aware diagnosis in Appendix D.6 and added these experiments to the revised manuscript.

Table 4. Effect of adding progress stage feedback to baselines.

IPI ASRDPI ASR
AutoRedteamer25.0075.00
Stage-augmented AutoRedTeamer31.2575.00
RedCodeAgent37.5056.25
Stage-augmented RedCodeAgent37.5062.50
SABER50.0087.50

Table 5. Stage transitions relative to the pre-repair instance (%).

instanceRegressionAdvance
Repaired instance7.1418.37
Counterfactual instance21.438.16

W3 & Q2: Equal target interaction budgets do not account for SABER’s additional computational overhead. Does its advantage persist under matched total token budgets? What local repair budget is used, and do the reported costs include all diagnosis, judge, constraint-generation, and repair calls?

R3: Thank you for this insightful comment.

  • The required additional information: the local repair budget is Br=10B_r=10, and the costs in Tables 10 and 11 (in the original manuscript) include all diagnosis, judge, constraint-generation, and repair calls. These details supplement the repair procedure and cost definition provided in Section 4.3 and Appendix D.4, and we have made them explicit in the revision.
  • We fully understand your concern about computational fairness and therefore compare SABER with the baselines under matched per-case token budgets, running each budget setting independently without a separate cap on target-agent rollouts.
    • Table 6 shows that SABER significantly outperforms both baselines from 0.3M tokens onward. RedCodeAgent performs better at smaller budgets mostly because its shorter context might allow more attack attempts within the same token budget.
    • More importantly, SABER continues to improve over a wider budget range, while the baselines saturate early. This gives SABER an additional advantage when more computation is available, enabling it to uncover vulnerabilities that remain undiscovered at smaller budgets.
    • Since our main experiments correspond to approximately 0.3M tokens, these results indicate that the reported performance still has room for improvement under larger budgets.

We have expanded the cost-accounting details and added this matched-budget evaluation to Appendix D.4 of the revised manuscript.

Table 6. ASR (%) under matched total token budgets. Each column specifies the same per-case token budget for all methods.

Method0.1M0.2M0.3M (Ours)0.5M0.75M1M1.5M2M3M
AutoRedTeamer12.5025.0025.0025.0025.0025.0025.0025.0025.00
RedCodeAgent43.7543.7543.7543.7543.7543.7543.7543.7543.75
SABER12.5025.0050.0062.5062.5075.0075.0075.0075.00
AuthorsOpenReview ↗

Author Response (Part III)

W4 & Q3: (1) The evaluation uses a small set of cases. Using a second independent benchmark would strengthen the evidence for SABER’s effectiveness. (2) The evaluation lacks repeated independent runs, confidence intervals, and paired significance tests. Can repeated runs, paired case outcomes, and stage regression rates provide stronger evidence for SABER’s reliability and the benefit of enforcing repair boundaries?

R4: Thanks for your insightful comment!

  • We first respectfully clarify that this setting is used simply to keep computational costs manageable. We deliberately chose DTap for its realistic simulated environments, but its complex, multi-step tasks require extensive agent–environment interactions, making evaluation more expensive than on simpler benchmarks. Our total experimental expenditure has already approached USD 20,000.
  • Arguably, our case selection was designed to be as comprehensive as possible within this budget: we balanced easy, moderate, and complex tasks under both IPI and DPI and distributed cases as evenly as possible across DTap’s application domains.
  • However, we fully understand your concern about the stability of our overall results and bounded repair’s contribution across independent runs.
    • We repeat the complete red teaming procedure three times on the same 16 moderate cases against OpenClaw with DeepSeek V4 Pro, comparing full SABER with No bounded scope variant, which isolates the contribution of bounded repair. We report mean ASR, standard deviation, stage regression rates, and paired-test p-values. Considering both standard deviations and paired-test p-values, Tables 7 and 8 support the repeatability of SABER’s performance advantage and the contribution of bounded repair across independent runs. The mean ± standard deviation range is 50.87–69.97% for full SABER versus 38.61–53.05% for No bounded scope, showing a higher performance range with only slight overlap. The paired tests further support statistically significant improvements in ASR and reductions in stage regression.
    • To further alleviate this concern, we expand the bounded-repair ablation by doubling the number of evaluation cases, including previously unevaluated cases from DTap. Table 9 continues to support our original conclusion regarding bounded repair’s contribution, with smaller standard deviations indicating more stable performance across independent runs.
    • To further evaluate generalizability, we compare SABER with RedCodeAgent on the full RedCode-Gen benchmark, which was used in the original RedCodeAgent paper. Table 10 further supports SABER’s effectiveness, providing evidence of its broader applicability across evaluation benchmarks. SABER continues to substantially outperform RedCodeAgent on this benchmark.

We have added the clarification to Appendix C.3 and incorporated these experiments and discussions to Appendix D of the revised manuscript.

Table 7. ASR across three independent runs and pooled stage regression rates.

VariantMean ASR ± SDRegression
No bounded scope45.83 ± 7.229.84 (25/254)
Full SABER60.42 ± 9.554.22 (10/237)

Table 8. Paired permutation tests results, with Holm correction across variant comparisons within each target and metric.

ComparisonsASR p-valueRegression p-value
Full SABER vs. No bounded scope0.03130.00012

Table 9. ASR cross three independent runs on the expanded set of 32 cases.

VariantRun 1Run 2Run 3Mean ASR ± SD
No bounded scope43.7540.6340.6341.67 ± 1.80
Full SABER53.1362.5056.2557.29 ± 4.77

Table 10. SABER's performance on RedCode-Gen.

ASR on RedCode-Gen
RedCodeAgent45.00
SABER72.50

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal and Revision Summary

Dear AC and Reviewers,

Thank you for your thoughtful reviews and constructive suggestions! Your insightful comments really help us improve our manuscript. We have carefully addressed all key concerns through controlled ablations, expanded evaluations, repeated experiments and clearer details. The main additions and findings are summarized below.

  • SABER’s core mechanisms contribute improvements beyond detailed prompts and repair knowledge. We compared variants with shared knowledge and prompt instructions to isolate the core mechanisms, confirming their contribution to SABER’s gains (Appendix D.8, Table 20).
  • Preserving identified progress-supporting components improves repair effectiveness. We conducted a counterfactual experiment showing lower regression and greater advancement when these components were preserved, and augmented existing baselines with stage-aware guidance, yielding modest overall improvements (Appendix D.6, Table 19).
  • Repeated and expanded evaluations support SABER’s repeatability and bounded repair’s contribution. We conducted three-run ablations across two target agents with paired significance tests and doubled the case set, obtaining consistent benefits from bounded repair. Fixed-instance replays further supports attack repeatability (Appendices D.3 and D.9, Tables 11–13 and 21).
  • SABER remains effective under matched token budgets and benefits from additional computation. We conducted independent comparisons at equal per-case budgets, showing that SABER outperforms both baselines from 0.3M tokens onward and continues to improve over a wider budget range (Appendix D.4, Table 16).
  • SABER’s effectiveness generalizes beyond DTap. We compared SABER with RedCodeAgent on the full RedCode-Gen, where SABER achieves 72.50% ASR compared with RedCodeAgent’s 45.00% (Appendix D.10, Table 22).

We have also clarified implementation details in Appendices A.3 and C.2–C.3, and provided an anonymous evaluation service (https://hats-appraisal-sussex-arrangements.trycloudflare.com/) to facilitate reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.