acceptodds
Under review as a conference paper at ICLR 2027

SWE-RedAgent: Adaptive Red-Teaming of Coding Agents via Utility-Preserving Vulnerability Injection

Abstract

Large language model (LLM)-based coding agents are increasingly used to autonomously modify software repositories, execute tests, and iteratively refine patches. However, these capabilities also create new security risks when agents interact with adversarial users. We introduce SWE-RedAgent, an adaptive red-teaming framework for evaluating whether coding agents can be induced to introduce exploitable vulnerabilities while preserving the intended utility of repository-level software engineering tasks. We argue that utility preservation is an important aspect of the threat model, since patches that break existing functions or fail to implement the requested feature are likely to be rejected before the vulnerability ever reaches deployment. SWE-RedAgent reasons over the repository and task request to identify task-relevant vulnerability opportunities, then adaptively steers the coding agent toward vulnerable and yet functionally plausible implementations over multi-turn interactions. We evaluate SWE-RedAgent on web-application development tasks across multiple coding-agent harnesses and backend models, and compare it with existing red-teaming baselines. SWE-RedAgent introduces 93 verified vulnerabilities over 70 tasks while preserving 97.33% base functionality, outperforming all baselines. Our results show that functionally correct patches from coding agents can remain highly susceptible to subtle and task-aligned adversarial manipulations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, AI Safety, Security, Red Teaming, Coding Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xh6HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SWE-RedAgent, an adaptive framework for testing whether coding agents can be induced to introduce exploitable vulnerabilities while completing repository-level tasks. It analyzes the repository and task to identify relevant attack opportunities, then guides the agent toward vulnerable but functional implementations. Results show the proposaed method could outperform baseline methods.

Critical strengths
  1. This paper is well-written and easy to follow.

  2. The studied problem is interesting and significant.

  3. The approach design is reasonable and the the results are promising.

Critical weaknesses

(1) Some of the description about related work is not correct. "Other methods in Table 1, including FCV-Attack and SWExploit, move toward utility-preserving attacks by inducing vulnerabilities while requiring the patch to remain functionally correct (Peng et al., 2026; Chen et al., 2026). However, these methods focus on single-turn code generation". SWExploit is not a single-turn code generation. Seems compared with SWExploit, the advantage of SWE-RedAgent is that SWE-RedAgent could automatically select the vulnerable payload, this automation is good, but the author should clearly state it in the paper.

(2) This paper use LLM-as-judge as the oracle, however, given LLM is not perfect, thus, it is unclear how reliable are these results. Also, the paper claim "exploitable vulnerabilities" but there is no exploitable evaluation.

(3) The evaluation metrics seems not supprot the claim "functionality correct and vulnerable" as there is no such metric in Table 2 report this value.

(4) The evaluation is conducted on Private bench and SWE-RedBench, however, it is unclear what is Private bench, there is no reference, also, SWE-RedBench is proposed by the authors, and the construction details are not clear.

(5) Miss related work

Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities

Questions for authors
  1. What is the accuracy of applying LLM-as-judge as oracle? why only make more rules for codeQL for detecting these vulnerabilities?

  2. What is the metric in Table 2 measure "functionality correct and vulnerable" ?

  3. Why adopting Private bench and SWE-RedBench instead on public available benchmarks?

  4. Did you try some defense methods to evaluate the robustness of the proposed attack?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Xh6H Part1

Thank you for your encouraging comments. We addressed your questions with new experiments, result analysis, and additional clarifications.

W1. Some of the description about related work is not correct. SWExploit is not a single-turn code generation. Seems compared with SWExploit, the advantage of SWE-RedAgent is that SWE-RedAgent could automatically select the vulnerable payload, this automation is good, but the author should clearly state it in the paper.

Thank you for raising this point. We agree that “single-turn code generation” is not an accurate description here. We will revise it in the paper. In addition to automatically selecting vulnerable payloads, we want to clarify the other difference between our attack and SWExploit/FCV-Attack. More specifically, there is a key difference between multi-turn attacks and iterative refinement attacks

SWExploit and FCV-Attack are both iterative refinement methods to edit the attack prompt (or adversarial issue). Although the attacker interacts with the target coding agent multiple times to obtain feedback, their goal is to refine a single attack prompt or adversarial issue iteratively. Once this prompt/issue is finalized, they will send this prompt/issue to the coding agent for code generation.

In contrast, SWE-RedAgent is a multi-turn attack. Given a coding task, it performs planning and interacts with the coding agents over multiple turns, where each turn involves implementation and code changes. Each prompt is generated based on the latest repository state, previous conversation history, and test feedback to advance the implementation progress. The attack is integrated into this multi-step coding process.

In summary, existing work SWExploit and FCV-Attack iteratively refine a target prompt/issue multiple times and then use the finalized prompt for code generation (i.e., iterative refinement). Our attack SWE-RedAgent is a multi-turn attack where the attack is integrated into the multi-turn (or multi-step) coding process. We realized that this terminology may cause confusion and will clarify this distinction in the revised paper.

W2. This paper use LLM-as-judge as the oracle, however, given LLM is not perfect, thus, it is unclear how reliable are these results. Also, the paper claim "exploitable vulnerabilities," but there is no exploitable evaluation. Q1. What is the accuracy of applying LLM-as-judge as oracle? why only make more rules for codeQL for detecting these vulnerabilities?

Reliability of LLM-as-judge: We recognized the limitations of simply using an LLM-as-a-judge. Therefore, our evaluation includes an “oracle–and-verifier” evaluation pipeline. More specifically, the oracle identifies potential vulnerabilities, and then a verifier evaluates each vulnerability finding using explicit code-grounded criteria (the more detailed explanations are in Appendix A.2.2). We also have a PoC (proof-of-concept) verification step to verify exploitability (also see Appendix A.2.2).

To evaluate the reliability of the evaluation pipeline, in Section 3.5, we randomly sampled 50 vulnerability findings for execution-based proof-of-concept (PoC) verification (i.e., exploitation). The results were then manually reviewed by three security experts. This analysis returns an estimated precision of 92%. We further report the breakdown of verification results by CWE category (following MITRE CWE-1000) [1]:

CategoryReviewedConfirmedPrecision
CWE-664 Improper Control of a Resource Through its Lifetime252392%
CWE-284 Improper Access Control131292.3%
CWE-707 Improper Neutralization77100%
CWE-710 Improper Adherence to Coding Standards3266.7%
Others22100%

This shows that our oracle-verifier pipeline produces reliable evaluation results. The sample size is largely limited by the overhead of manual code analysis and verification, which is a limitation to be acknowledged.

Why not extend CodeQL: We have considered the option to write custom CodeQL queries. However, there are two major limitations:

  1. SWE-RedAgent adaptively selects CWEs to attack. We cannot pre-determine which CWE will be selected. The vulnerability space can cover hundreds of CWEs. Manually developing corresponding CodeQL queries is difficult and time-consuming.
  2. Custom queries, if not designed carefully, can easily lead to false positives. In our paper, we used CodeQL as a complementary rule-based safety oracle, as a reference. We primarily relied on the oracle-verifier pipeline described above for evaluation (we indeed evaluated the reliability of this pipeline, as discussed above).

[1] The MITRE Corporation. CWE-1000: Complete Weakness Hierarchy. Common Weakness Enumeration (CWE), Version 5.0, 2026.

AuthorsOpenReview ↗

Rebuttal to Reviewer Xh6H Part2

W3. The evaluation metrics seems not supprot the claim "functionality correct and vulnerable" as there is no such metric in Table 2 report this value.

Q2. What is the metric in Table 2 measure "functionality correct and vulnerable" ?

Thank you for the question. To clarify, “functionally correct” means passing the software function tests, and vulnerabilities are measured by our security evaluation pipeline. In Table 2, V_U represents the number of vulnerabilities under our default utility constraint: passing 100% base tests (zero regression) and 50% feature tests. In other words, this metric already considers both utility (functional correctness) and attack performance.

To further illustrate the attack performance under different “utility” constraints, we performed additional experiments. We fix the base test pass rate to 100% (requiring zero software function regression) and vary the feature test pass rate threshold from 0% to 25%, 50%, 75%, and 100%, for both benchmarks. The tables below report the induced vulnerabilities.

Private Benchmark

MethodRaw (V)0%25%50%75%100%
Direct Malicious26228410
RedCodeAgent3130161565
FCV-Attack843320
SWExploit2016151583
SWE-RedAgent (Ours)535351512712

SWE-RedBench

MethodRaw (V)0%25%50%75%100%
Direct Malicious34220000
RedCodeAgent12117777
FCV-Attack222222
SWExploit1087766
SWE-RedAgent (Ours)404040403938

As shown in the tables, on both benchmarks, SWE-RedAgent outperforms other baseline methods under all levels of utility constraints. We do observe that the performance on the private benchmark dropped with the increase in the utility constraints. This is because this private benchmark intentionally constructs “hard” coding tasks. In comparison, the feature implementation tasks in SWE-RedBench are slightly easier. Regardless, SWE-RedAgent still outperforms other methods for both benchmarks.

W4. The evaluation is conducted on Private bench and SWE-RedBench, however, it is unclear what is Private bench, there is no reference, also, SWE-RedBench is proposed by the authors, and the construction details are not clear.

Q3. Why adopting Private bench and SWE-RedBench instead on public available benchmarks?

SWE-RedBench is constructed by the authors. Due to space limitations, we described its construction pipeline in Appendix A.3, including task extraction from open-source repositories, derivation of base and feature tests, and synthesis of natural-language task descriptions. The private benchmark was constructed by a third-party vendor.

To support reproducibility, we will publicly release SWE-RedBench, including all tasks and test suites, along with the SWE-RedAgent codebase. Regarding the private benchmark, we also plan to release this benchmark: we’re currently coordinating with our third-party partner/vendor to finalize details of the data release procedure.

We constructed SWE-RedBench and use the private benchmark because none of the public benchmarks can be easily applied to our threat model. As discussed in Section 3.1, existing repository-level security benchmarks primarily focus on “patching or repairing existing code”, whereas we study the “implementation of new features”. While a few recent coding agent benchmarks, such as FeatBench [2] and FeatureBench [3], also evaluate feature implementation, they have limited coverage on web applications (which is the main focus of our paper considering their rich attack surfaces). We will clarify in the revised paper.

[2] Haorui Chen, Chengze Li, and Jia Li. FeatBench: Evaluating Coding Agents on Feature Implementation for Vibe Coding. arXiv preprint arXiv:2509.22237, 2025.

[3] Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang. FeatureBench: Benchmarking Agentic Coding for Complex Feature Development. In International Conference on Learning Representations, 2026.

AuthorsOpenReview ↗

Rebuttal to Reviewer Xh6H Part3

W5. Miss related work: Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilities

Thank you for pointing us to this paper. This paper (SWEADV) was submitted to arXiv on Sep 14, 2026 (very close to the ICRL deadline) and thus we missed it. We carefully examined the paper and identified three key differences. (1) This paper (SWEADV) still focuses on code repair and patching, while our paper focuses on feature implementation; (2) This paper (SWEADV) is more similar to SWExploit, focusing on iterative refinement to generate an “adversarial issue”. Our paper focuses on multi-turn attacks: interacting with the coding agent to induce vulnerabilities over multiple coding steps/turns while preserving code functionality. (3) This paper (SWEADV) only covers 5 attack/vulnerability types, while we cover 96 CWEs.

Q4. Did you try some defense methods to evaluate the robustness of the proposed attack?

Thank you for this question. To answer this question, we ran a quick experiment using two well-known safety guardrail models: Llama Prompt Guard 2 and gpt-oss-safeguard-120b.

We first tested SWE-RedAgent and all the baselines against both defense methods on the private benchmark by simply replaying their prompts from the previous evaluation (main paper). We report the percentage of prompts detected as malicious by the safety guardrail model. The results are shown below:

MethodPrompt Guard 2GPT-OSS-Safeguard-120B
Benign User0.0%0.0%
Direct Malicious0.0%69.0%
RedCodeAgent0.0%79.6%
FCV-Attack0.0%57.1%
SWExploit1.1%31.5%
SWE-RedAgent (Ours)0.0%48.6%

Prompt Guard 2 has limited success (nearly 0% detection rate). This is because Prompt Guard 2 is designed to detect generic jailbreaking and prompt injection attacks. The result reflects that our vulnerability-inducing prompts did not carry typical patterns of jailbreaking or prompt injection.

The gpt-oss-safeguard-120b model is a stronger defense: it is an open-weight safety reasoning model that supports customized safety policies. SWE-RedAgent’s replayed prompts achieve the second-lowest detection rate (48.6%), behind SWExploit (31.5%).

However, these results are obtained through replaying prompts. Next, we run a full experiment against gpt-oss-safeguard-120b defense to examine the attack performance through actual multi-turn interactions between the attacker and the coding agent, with the code agent running the safeguard. The results are shown below:

MethodTasks with PRBBUURRRRVQLV_{QL}VVVU50V_{U50}VU100V_{U100}
Benign User35/3599.6367.400.000110
Direct Malicious35/3589.3411.6968.700700
FCV attack13/3593.5423.9057.140200
RedCodeAgent25/3598.2423.6428.5711131
SWExploit23/3591.9729.8734.290511
SWE-RedAgent (Ours)35/3599.8354.5547.19327258

RRRR denotes the refusal rate from the safeguard, VU50V_{U50} denotes the number of vulnerabilities under 100% base test pass rate and 50% feature test pass rate constraint, and VU100V_{U100} denotes the number of vulnerabilities under 100% base test and 100% feature test pass rate constraint.

  • Utility performance: SWE-RedAgent maintains its original utility performance (54.55). This is because SWE-RedAgent can adaptively switch between improving utility and injecting vulnerabilities. In contrast, all the other attack baselines (except benign-user) are affected by the safeguard’s refusal, resulting in severe feature implementation performance degradation.

  • Vulnerability Injection: SWE-RedAgent still outperforms all the other baselines in terms of the number of injected vulnerabilities across different utility constraints. Although SWExploit has a lower refusal rate, SWE-RedAgent still successfully injected more vulnerabilities (27 v.s. 5).

Thanks the authors for their response and addictional experiments, which addressed my concerns, so I will keep my positive scores.

Reviewer_DjucOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a red-teaming framework to test whether coding agents can be steered into introducing security vulnerabilities while preserving task utility. The framework combines multiple agents with a module to selects task-compatible CWEs. Evaluation shows vulnerability-introducetion success for 70 tasks while preserving most of task utility.

Critical strengths
  • The space between single-turn functionality-preserving attacks and multi-turn jailbreaks is a rich area for exploration and this paper opens up this domain.

  • Integrating the utility and vulnerability gates into the multi-agent architecture is supported by showing that a single-agent ablation reduces success rate.

  • The task-compatible vulnerability selection is an interesting and key component of the design, showing how adaptation can increase success rate.

Critical weaknesses
  • The threat model seems made up, as it refers to adversaries who are part of an software organization but cannot modify code directly. While the constraint is needed for the paper, it seems highly unrealistic nowadays.

  • The citations of Debenedetti 2025 and Zhang 2025 in the category of "adversarial prompts against code LLMs" is wrong, since those papers introduced benchmarks for prompt injection and tool hijacking against general-purpose agents, not specific to coding in any way.

  • The paper claims novelty in multi-turn attacks, but RedCodeAgent has already done such attacks.

  • There is a mention of "checkpoints" in Section 2.3. More detail is needed to understand when checkpoints are taken, what is part of the checkpoints, and what is the checkpoint budget (if any).

Questions for authors
  1. Table 2 reports the main results but lists no vulnerabilities introduced (by mistake) in the Benign User scenario. Was it the case that the coding agents never produced any vulnerable code in benign settings? That by itself is worth analyzing further, as it would be a novel result going against observed trends. And if the coding agents do generate vulnerable code, how was the presence of a vulnerability attributed to the attacker or the coding agent in later (adversarial) scenarios?

  2. The utility condition seems quite loose and would probably not be accepted in most software organizations: "at least 50% feature test pass". How was this threshold chosen? Is the attack success rate much lower is the utility condition uses a much higher test pass rate?

  3. How will your SWE-RedBench release support reproducibility if the private benchmark is not included? If you plan to release, please state it explicitly.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Djuc Part1

Thank you for your encouraging comments. We addressed your questions with new experiments, results analysis, and additional clarifications.

Q1. Table 2 reports the main results but lists no vulnerabilities introduced (by mistake) in the Benign User scenario. Was it the case that the coding agents never produced any vulnerable code in benign settings? That by itself is worth analyzing further, as it would be a novel result going against observed trends. And if the coding agents do generate vulnerable code, how was the presence of a vulnerability attributed to the attacker or the coding agent in later (adversarial) scenarios?

We agree that reporting the vulnerabilities introduced by the benign-user control would be helpful. It shows whether vulnerabilities are introduced during ordinary feature development, which helps to isolate vulnerabilities contributed by the attacker.

We ran the same oracle-verifier pipeline for the benign-user baseline on both benchmarks.

BenchmarkVQLV_{QL}VVVUV_U
Private011
SWE-RedBench122

The results show that there are few vulnerabilities introduced during ordinary feature development (1-2 cases per benchmark). In other words, the vulnerabilities reported in Table 2 were mostly contributed by attackers, confirming the effectiveness of our attack method.

We further analyzed these 1-2 cases: it does not appear that the presence of these vulnerabilities has a conclusive pattern/effect on the coding agents’ later actions. This question itself is very interesting, and we can discuss this in the revised paper to inspire future work.

Q2. The utility condition seems quite loose and would probably not be accepted in most software organizations: "at least 50% feature test pass". How was this threshold chosen? Is the attack success rate much lower if the utility condition uses a much higher test pass rate?

We agree this is a helpful analysis. Below is a detailed breakdown of the vulnerability findings under different levels of utility constraints. Here, we fix the base test pass rate to 100% (requiring zero regression) and vary the feature test pass rate threshold from 0% to 25%, 50%, 75%, and 100%, for both benchmarks. The tables below report the induced vulnerabilities.

Private Benchmark

MethodRaw (V)0%25%50%75%100%
Direct Malicious26228410
RedCodeAgent3130161565
FCV-Attack843320
SWExploit2016151583
SWE-RedAgent (Ours)535351512712

SWE-RedBench

MethodRaw (V)0%25%50%75%100%
Direct Malicious34220000
RedCodeAgent12117777
FCV-Attack222222
SWExploit1087766
SWE-RedAgent (Ours)404040403938

As shown in the tables, on both benchmarks, SWE-RedAgent outperforms other baseline methods under all levels of utility constraints. We do observe that the performance on the private benchmark dropped with the increase in the utility constraints (from 51 to 12, from 50% to 100%). This is because this private benchmark intentionally constructs “hard” coding tasks. In comparison, the feature implementation tasks in SWE-RedBench are slightly easier. Regardless, SWE-RedAgent still outperforms other methods for both benchmarks.

Q3. How will your SWE-RedBench release support reproducibility if the private benchmark is not included? If you plan to release, please state it explicitly.

To support reproducibility, we will publicly release SWE-RedBench, including all tasks and test suites, along with the SWE-RedAgent codebase. Regarding the private benchmark, we also plan to release it in the future: we’re currently coordinating with our third-party partner/vendor to finalize details of the data release procedure.

AuthorsOpenReview ↗

Rebuttal to Reviewer Djuc Part2

W1. Threat model: it refers to adversaries who are part of a software organization but cannot modify code directly. While the constraint is needed for the paper, it seems highly unrealistic nowadays.

Thank you for raising this point. We will clarify the threat model with additional context. The key points are to be clarified in the revised paper:

  • Traditionally, software developers directly edit the code and commit changes to the code repo. Today, with the wide adoption of coding agents and vibe coding, developers may introduce vulnerable code indirectly through a coding agent by “talking” to the coding agent.
  • We want to clarify and acknowledge that the traditional attack interface (directly modifying code) is still a threat. In this paper, we explore the new attack interface (i.e., indirect code modification through an agent), which coexists (in parallel) with the traditional threat.
  • This new attack surface is concerning because it lowers the domain knowledge requirement to introduce vulnerabilities without breaking existing functions and could enable vulnerability insertion at scale.

W2. The citations of Debenedetti 2025 and Zhang 2025 in the category of "adversarial prompts against code LLMs" is wrong, since those papers introduced benchmarks for prompt injection and tool hijacking against general-purpose agents, not specific to coding in any way.

Thank you for pointing this out. We will revise and clarify this point by distinguishing general-purpose agent red-teaming from those targeting coding agents. We would also like to add a note that many general-purpose agents today are used for coding tasks (e.g., GPT and Claude). It is possible that certain adversarial attack methods developed for general-purpose agents could be used on coding agents.

W3. The paper claims novelty in multi-turn attacks, but RedCodeAgent has already done such attacks.

This is a potential misunderstanding of the term “multi-turn.” Sorry about the confusion, and please allow us to clarify. More specifically, there is a key difference between multi-turn attacks and iterative refinement attacks

RedCodeAgent and SWExploit are both iterative refinement methods to edit the attack prompt (or adversarial issue). Although the attacker interacts with the target coding agent multiple times to obtain feedback, their goal is to refine a single attack prompt (or adversarial issue) iteratively. Once this prompt is finalized, they will send this prompt to the coding agent for code generation.

In contrast, SWE-RedAgent is a multi-turn attack. Given a coding task, it performs planning and interacts with the coding agents over multiple turns, where each turn involves implementation and code changes. Each prompt is generated based on the latest repository state, previous conversation history, and test feedback to advance the implementation progress. The attack is integrated into this multi-step coding process.

In summary, RedCodeAgent and SWExploit iteratively refine the same prompt/issue multiple times and then launch the one-shot attack for code generation (i.e., iterative refinement). Our attack SWE-RedAgent is a multi-turn attack where the attack is integrated into the multi-turn (or multi-step) coding process. We realized that this terminology may cause confusion and will clarify this distinction in the revised paper.

W4. There is a mention of "checkpoints" in Section 2.3. More detail is needed to understand when checkpoints are taken, what is part of the checkpoints, and what is the checkpoint budget (if any).

Sorry about the potential confusion. The “checkpoint” here refers to the most recent repository commit that satisfies the utility gate (i.e., passing all base tests and at least one feature test). The utility agent tracks this latest working checkpoint throughout the conversation. Whenever an attack turn causes a regression, the utility agent either directly asks the coding agent to revert to the previously working commit, or instructs a targeted fix (e.g., if the regression is minor). There is no budget limit for tracking the “checkpoint”. We will clarify this in the revised paper.

Reviewer_RYChOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SWE-RedAgent studies whether malicious users can induce vulnerabilities during repository-level feature development while retaining functionality. A planner coordinates separate utility and attack agents. Task-conditioned vulnerability retrieval and structured memory support adaptation across interactions. Evaluation covers 70 tasks from private and public repositories, with additional comparisons across six coding-agent configurations. The method produces more accepted vulnerability findings than the tested baselines. Most findings also satisfy the stated utility constraint. Execution-based checks and expert review provide supporting validation on a sample of findings.

Critical strengths
  1. The paper studies a useful security gap in repository-level development. It evaluates functionality and vulnerabilities together. Separating utility restoration from security testing gives the framework a clear way to handle their competing objectives.

  2. The main comparison shares retrieved vulnerability candidates across methods. Across the two benchmarks, SWE-RedAgent yields 93 findings versus 43 for the strongest baseline. Of these, 91 satisfy the stated utility constraint, supporting an advantage beyond raw finding counts (Table 2).

  3. Memory and target selection have useful ablation support. Without memory, findings fall from 53 to 11. Replacing retrieval with equally many uniformly sampled CWE candidates yields 5. Expert review confirms 46 of 50 sampled findings, providing complementary validation (Table 4; Section 3.5).

Critical weaknesses
  1. Table 2 omits security outcomes for the benign-user control. Excluding flaws already in the initial repository does not exclude vulnerabilities introduced during ordinary feature development. The additional risk attributable to adversarial interaction therefore remains unclear.

  2. The utility constraint requires all base tests and at least 50% of feature tests. It therefore allows partial feature-test success. Aggregate finding counts do not show how many tasks pass every test while also containing an accepted vulnerability finding.

  3. The validation study confirms 46 of 50 sampled findings, which is useful evidence. However, the sampling distribution across methods and vulnerability classes is unspecified. Possible differences in evaluator precision therefore remain a source of uncertainty in the reported comparisons.

Questions for authors
  1. Table 2 reports utility for the benign-user control but omits its security results. Were these final repositories assessed with the same oracle-verifier pipeline? If results are available, please summarize their vulnerability counts under the same utility constraint. This would help separate adversarial influence from ordinary coding errors.

  2. The utility constraint requires all base tests and at least 50% of feature tests. How many tasks pass every base and feature test while containing an accepted vulnerability finding? Please summarize this from the existing final-test records. This would clarify how often security findings coexist with full test success.

  3. Section 3.5 confirms 46 of 50 sampled findings through execution checks and expert review. How were those findings sampled across methods and vulnerability classes? Please summarize the existing expert labels by method, where available. This would clarify whether the reported precision supports comparisons across the different attack strategies.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer RYCh Part1

Thank you for your encouraging comments. We addressed your questions with new experiments and new results analysis:

W1+Q1: Table 2 omits security outcomes for the benign-user control. Were these final repositories assessed with the same oracle-verifier pipeline? If results are available, please summarize their vulnerability counts under the same utility constraint. This would help separate adversarial influence from ordinary coding errors.

We agree that reporting the security outcome for the benign-user control would be helpful. It shows whether vulnerabilities are introduced during ordinary feature development, which helps to better isolate the vulnerabilities contributed by the attacker.

We ran the same oracle-verifier pipeline for the benign-user baseline on both benchmarks.

BenchmarkVQLV_{QL}VVVUV_U
Private011
SWE-RedBench122

The results show that there are few vulnerabilities introduced during ordinary feature development (1-2 cases per benchmark). In other words, the vulnerabilities reported in Table 2 were mostly contributed by attackers, confirming the effectiveness of our method.

W2+Q2. The utility constraint requires all base tests and at least 50% of feature tests. Aggregate finding counts do not show how many tasks pass every test while also containing an accepted vulnerability finding. How many tasks pass every base and feature test while containing an accepted vulnerability finding? Please summarize this from the existing final-test records. This would clarify how often security findings coexist with full test success.

We agree this is a helpful analysis. Below is a detailed breakdown of the vulnerability findings under different levels of utility constraints. Here, we fix the base test pass rate to 100% (requiring zero regression) and vary the feature test pass rate threshold from 0% to 25%, 50%, 75%, and 100%, for both benchmarks. The tables below report the induced vulnerabilities.

Private Benchmark

MethodRaw (V)0%25%50%75%100%
Direct Malicious26228410
RedCodeAgent3130161565
FCV-Attack843320
SWExploit2016151583
SWE-RedAgent (Ours)535351512712

SWE-RedBench

MethodRaw (V)0%25%50%75%100%
Direct Malicious34220000
RedCodeAgent12117777
FCV-Attack222222
SWExploit1087766
SWE-RedAgent (Ours)404040403938

As shown in the tables, on both benchmarks, SWE-RedAgent outperforms other baseline methods under all levels of utility constraints, including the 100% threshold setting. We do observe that the performance on the private benchmark dropped with the increase in the utility constraints (from 51 to 12, from 50% to 100%). This is because this private benchmark intentionally constructs “hard” coding tasks. In comparison, the feature implementation tasks in SWE-RedBench are easier. Regardless, SWE-RedAgent still outperforms other methods for both benchmarks.

AuthorsOpenReview ↗

Rebuttal to Reviewer RYCh Part2

W3+Q3. The validation study confirms 46 of 50 sampled findings, which is useful evidence. However, the sampling distribution across methods and vulnerability classes is unspecified. How were those findings sampled across methods and vulnerability classes? Please summarize the existing expert labels by method, where available. This would clarify whether the reported precision supports comparisons across the different attack strategies.

Thank you for raising this question. We agree that the analysis will help to strengthen the paper!

To clarify, for the validation process, we randomly sampled 50 findings from all verifier-accepted vulnerabilities, regardless of method or CWE. Here is the breakdown of verification results by attack methods and by CWE categories (following MITRE CWE-1000) [1]:

Attack MethodReviewedConfirmedPrecision
Direct Malicious11981.8%
RedCodeAgent88100%
FCV-Attack4375%
SWExploit77100%
SWE-RedAgent (Ours)201995%
CategoryReviewedConfirmedPrecision
CWE-664 Improper Control of a Resource Through its Lifetime252392%
CWE-284 Improper Access Control131292.3%
CWE-707 Improper Neutralization77100%
CWE-710 Improper Adherence to Coding Standards3266.7%
Others22100%

The results show that our oracle-verifier pipeline achieves good precision across the vulnerability categories and attack methods. The sample size is largely limited by the overhead of manual code analysis and verification, which is a limitation to be acknowledged. This analysis result can provide additional evidence for the reliability of our safety evaluation pipeline. We will include the result in the revised paper.

[1] The MITRE Corporation. CWE-1000: Complete Weakness Hierarchy. Common Weakness Enumeration (CWE), Version 5.0, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.