acceptodds
Under review as a conference paper at ICLR 2027

PurpAgent: Integrating Security Reasoning into Coding Agents

Abstract

Large language model (LLM) agents increasingly automate software development, yet functionally correct implementations can still introduce security vulnerabilities. Integrating security review into these agents requires feedback that supports successful repair within a limited execution budget. In this paper, we introduce PurpAgent Harness, the first scaffold that seamlessly integrates secure coding and security review. Its key idea is to translate repository evidence into explicit security obligations that guide verification and repair in the original coding trajectory, without requiring a separate standalone security-analysis phase. We further develop PurpAgent 27B by applying mutual supervised fine-tuning (mutual SFT) on synthesized coding and security-review trajectories, equipping a single model to perform both agent roles natively. On SusVibes, PurpAgent Harness improves the joint functional and security pass rate (SecPass) by 2.15–14.49 percentage points over mini-SWE-agent across seven LLM backends, while maintaining or improving the functional pass rate (FuncPass) in general. Within the same harness, PurpAgent 27B achieves 36.60% FuncPass and 17.70% SecPass, improving over its base model by 11.33 and 6.95 percentage points, respectively, and leading open-source models of comparable size on both metrics. Third-party red-teaming further shows that PurpAgent Harness increases the proportion of Secure Apps from 35.0% to 55.0% with PurpAgent 27B, demonstrating improved robustness against adaptive exploitation attempts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Coding Agents, Secure Code Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y7zJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies how to make coding agents produce code that is both correct and secure. The authors propose PurpAgent Harness, which adds a first-step review and an independent final patch review to a coding agent. The review findings are sent back into the original coding trajectory to guide repair. The authors also train PurpAgent 27B on 16k synthesized coding and review trajectories, using one model for both roles. Experimental results on SusVibes show SecPass gains of 2.15 to 14.49 points across seven backends, and PurpAgent 27B improves SecPass from 10.75% to 17.70% over its base model.

Critical strengths
  • The paper proposes PurpAgent Harness to address the security issue in vibe coding. Different from the existing post-hoc methods, the harness provides a security review before and after the coding process, avoiding introducing vulnerabilities during feature implementation.
  • The authors also synthesize a large-scale training data for security, which includes 16k tasks across 192 CWE categories. This dataset can contribute to vibe coding training. They also verify the effectiveness of the synthesized data by training 27B PurpAgent.
  • Experimental results show that with PurpAgent harness, SecPass improves for all 7 backends, including both open-source and closed-source LLMs. It also beats DCR Harness in SecPass at a similar cost to DCR without PoC.
  • The results on red teaming show the proposed method can be generalized to Secure Apps.
Critical weaknesses
  • As claimed in the abstract, PurpAgent maintains FuncPass in general. However, FuncPass drops by 5 to 6 points for GPT-5.6 Sol and Grok 4.6, which is not an ignorable loss for pass rate.
  • The proposed PurpAgent harness contains a first-step review and a final-patch review. The first-step review is a structured instruction for the coding agent, which is similar to some simple modifications to the system prompt. The final-patch review is more like an independent post-hoc reviewer to check after the implementation. While the authors mention that they first propose the PurpAgent scaffold to address the challenges of post-hoc methods, the biggest gains come from adding a post-hoc reviewer (Table 3). The interaction between the coding agent and the post-hoc reviewer is straightforward.
  • There are no repeated runs or significance tests. Some numbers do not match 186 tasks, such as 9.70% in Table 1, 42.90% and 14.70% in Table 6.
  • In Table 3, more discussion should be provided for FuncPass. Removing first-step review and final-patch review alone lowers FuncPass by 5%, but having both recovers it to 46%.
  • In Table 5, each separate SFT model sees only half the data. The gain from mutual SFT may come from more data, not from transfer between roles.
  • Much of the PurpAgent 27B gain may come from better coding in general. Compared with Qwen 3.6 27B, PurpAgent 27B increases 11.33% on FuncPass but only 6.95% on SecPass. Btw, the authors don't mention where the training trajectories come from. The red teaming study is small. It uses one team and 20 tasks. The gain is 7 apps versus 11 apps. Only PurpAgent 27B is tested.
Questions for authors
  • The paper states that SusVibes has 186 tasks. However, several reported values, such as 9.70% in Table 1, 52.17% and 23.37% for Grok 4.6 in Table 1 and 42.90% and 14.70% in Table 6, cannot be obtained from 186 tasks.
  • Do you verify the contamination during training? Is it possible that training and test data have overlap, or based on the same repo?
  • How environments were built for the 16k tasks? The adversarial variants likely reuse the environments of their source tasks, but this should be described clearly. It is unclear how new features from feature grafting were tested, and the variants have no security tests.
  • What are SFT details such as learning rate, epochs, and hardware?
6 comments
AuthorsOpenReview ↗

Author Response to Reviewer y7zJ (Part 1/4)

We appreciate the reviewer’s detailed analysis and thoughtful questions. We address your concerns below.

W1: As claimed in the abstract, PurpAgent maintains FuncPass in general. However, FuncPass drops by 5 to 6 points for GPT-5.6 Sol and Grok 4.6, which is not an ignorable loss for pass rate.

We agree that the FuncPass declines for GPT-5.6 Sol and Grok 4.6 are meaningful and will revise the abstract to reflect this backend-dependent trade-off.

We want to gently remind that SecPass is the sole metric we use to determine end-to-end task success on SusVibes. As described in Section 4.1, it counts a task as successful only when the patch passes both functional and security tests. This reflects our goal of generating mergeable patches because, in practice, a patch with detected vulnerabilities will still be rejected from merging, even if it passes all functional tests. We report FuncPass to quantify the functional trade-off. Table 1 shows that PurpAgent Harness improves SecPass by up to 14.52 percentage points over mini-SWE-agent on all seven evaluated LLM backends, including GPT-5.6 Sol and Grok 4.6.

W2: The proposed PurpAgent harness contains a first-step review and a final-patch review. The first-step review is a structured instruction for the coding agent, which is similar to some simple modifications to the system prompt. The final-patch review is more like an independent post-hoc reviewer to check after the implementation. While the authors mention that they first propose the PurpAgent scaffold to address the challenges of post-hoc methods, the biggest gains come from adding a post-hoc reviewer (Table 3). The interaction between the coding agent and the post-hoc reviewer is straightforward.

Thank you for the question. We would like to clarify potential misunderstandings and highlight the importance of first-step review, the in-progress PR security review, and our fundamental differences with post-hoc reviewers. First, while we agree that final-patch review yields the larger gain when the two stages are enabled individually, first-step review also plays an important role. Specifically, with final-patch review enabled in both configurations, adding first-step review raises SecPass from 18.82% to 24.19% and FuncPass from 41.40% to 46.24%.

Furthermore, we want to kindly emphasize that there’s a fundamental difference between the post-hoc reviewer and PurpAgent. Post-hoc review begins when the completed PR is submitted. PurpAgent Harness is different, as it runs an independent security review whenever the coding agent submits a candidate patch to update the in-progress PR. The harness returns the findings as a new turn in the original coding trajectory, so the coding agent can use its implementation history to verify and address them before finalizing the PR. Consequently, while existing post-hoc reviewers may leave vulnerabilities introduced during implementation undetected until later stages, where the growing code complexity can make them much harder and more costly to identify/repair, PurpAgent Harness can successfully handle these vulnerabilities on-the-fly.

Lastly, we show that PurpAgent Harness clearly outperforms Anthropic DCR Harness, one of the most popular post-hoc reviewers. As shown in Table 4 in the paper, with Claude Opus 4.6 as the backend for all three configurations, PurpAgent Harness achieves 24.19% SecPass on SusVibes, compared with 20.97% for DCR w/o PoC and 21.51% for DCR w/ PoC. PurpAgent Harness takes 11.3 minutes and costs $3.06 per task, comparable to DCR w/o PoC (11.7 minutes and $3.14) and lower than DCR w/ PoC (82.9 minutes and $23.75). These results indicate that PurpAgent’s integrated review workflow can produce more patches that pass both functional and security tests with comparable or lower runtime and cost per task. We will clarify this distinction between post-hoc review and PurpAgent’s integrated approach more explicitly in the revision.

AuthorsOpenReview ↗

Author Response to Reviewer y7zJ (Part 2/4)

W3: There are no repeated runs or significance tests.

Q1: The paper states that SusVibes has 186 tasks. However, several reported values, such as 9.70% in Table 1, 52.17% and 23.37% for Grok 4.6 in Table 1 and 42.90% and 14.70% in Table 6, cannot be obtained from 186 tasks.

Thank you for pointing out these reporting inconsistencies. We rechecked the saved task-level outcomes and traced all discrepancies to denominator and rounding errors. In Table 1, 9.70% resulted from rounding 18/186 to one decimal place and then displaying a trailing zero; the correct value rounded directly to two decimals is 9.68%. Both the Grok 4.6 results in Table 1 (52.17%, 23.37%) and the internal-review results in Table 6 (42.90%, 14.70%) used a denominator of 184 rather than 186, as they were all carried over from an intermediate summary, corresponding to 96/184 and 43/184 for Grok and 79/184 and 27/184 for internal review. Among these, the internal-review ratios (79/184 and 27/184) were additionally rounded to one decimal place before being displayed with two, producing the reported 42.90% and 14.70%. Using the full 186-task denominator gives 51.61% (96/186) and 23.12% (43/186) for Grok, and 42.47% (79/186) and 14.52% (27/186) for internal review. None of these corrections change the relative ordering or conclusions of the paper. We will correct all affected tables and accompanying text in the revision.

To address the concern about repeated runs, we conducted three independent runs of both PurpAgent and mini-SWE-agent with Claude Opus 4.6 on the full 186-task SusVibes benchmark. As shown in the table, where values denote the mean ± sample standard deviation across three runs, PurpAgent consistently outperforms mini-SWE-agent in both FuncPass and SecPass, with improvements well outside the variance across runs. PurpAgent achieves 27.24 ± 2.65% SecPass, nearly tripling mini-SWE-agent’s 9.86 ± 0.31%, while also improving FuncPass from 42.47 ± 3.27% to 50.90 ± 4.18%. A Wilcoxon signed-rank test over task-level outcomes yields p = 1.06 × 10⁻¹², confirming that the differences are statistically significant.

Agent scaffoldFuncPass (%)SecPass (%)
mini-SWE-agent42.47±3.279.86±0.31
PurpAgent50.90±4.1827.24±2.65

W4: In Table 3, more discussion should be provided for FuncPass. Removing first-step review and final-patch review alone lowers FuncPass by 5%, but having both recovers it to 46%.

Thank you – this is a great observation. The major reason behind this phenomenon is that poorly grounded security findings can lead the coding agent to reject legitimate inputs, add overly restrictive checks, or disable requested behavior, resulting in functional regressions. In contrast, more accurate findings can help mitigate these regressions. Specifically, first-step review identifies the intended functionality and the security obligations the implementation should satisfy. Final-patch review examines the resulting code and provides repository-grounded feedback to the original coding agent. Together, the two stages can improve the quality of security findings and help preserve the intended functionality.

The FuncPass results in Table 3 are consistent with this explanation. First-step review alone achieves 41.00% FuncPass, and final-patch review alone achieves 41.40%. Using both stages restores FuncPass to 46.24%. The full harness also raises SecPass from the baseline’s 9.68% to 24.19%. We will expand this discussion in the revision.

W5: In Table 5, each separate SFT model sees only half the data. The gain from mutual SFT may come from more data, not from transfer between roles.

Thank you for raising this point. We first clarify that mutual SFT and separate SFT use the same total training budget: separate SFT trains two independent models on each half of the data, while mutual SFT trains one model on all of it.

The key question is whether mutual SFT’s gain comes from seeing both roles per task (role transfer) or simply from training on more data. To test this, we conduct a controlled experiment using a 5k-task subset. Separate SFT trains on all 5k tasks, using the 5k coding trajectories and 5k security-review trajectories independently. In contrast, mutual SFT trains on only 2.5k tasks from this subset but uses both their coding and security-review trajectories, giving it strictly less data while retaining the cross-role signal.

As shown in the table, mutual SFT matches separate SFT in FuncPass while achieving higher SecPass with only half the training tasks, confirming that the gain also comes from cross-role transfer. Specifically, both strategies achieve 28.49% FuncPass, but mutual SFT reaches 13.98% SecPass compared with 12.37% for separate SFT. We will include this controlled experiment in the revision.

Base ModelTraining StrategyFuncPassSecPass
Qwen 3.6 27BSeparate SFT28.49%12.37%
Mutual SFT28.49%13.98%
AuthorsOpenReview ↗

Author Response to Reviewer y7zJ (Part 3/4)

W6: Much of the PurpAgent 27B gain may come from better coding in general. Compared with Qwen 3.6 27B, PurpAgent 27B increases 11.33% on FuncPass but only 6.95% on SecPass. Btw, the authors don't mention where the training trajectories come from. The red teaming study is small. It uses one team and 20 tasks. The gain is 7 apps versus 11 apps. Only PurpAgent 27B is tested.

We agree that improved general coding ability can contribute to PurpAgent 27B’s SecPass gains because SecPass requires passing both functional and security tests. However, across the 35 tasks where both models pass the functional tests, PurpAgent 27B achieves a security test pass rate of 20/35 (57.14%), compared with 13/35 (40.00%) for Qwen 3.6 27B. This reflects PurpAgent’s SecPass gains on tasks where both models solve the functionality, indicating an improved ability to code securely rather than just a general coding improvement.

Regarding the source of training trajectories, we distilled the coding and security-review training trajectories from Claude Opus 4.8 and jointly fine-tuned Qwen 3.6 27B on those trajectories for both roles. We will identify the teacher model and describe how we generated the trajectories in the revision.

We also agree that the red-teaming evaluation reported in the paper is relatively small. To enrich our red-teaming evaluation, we further evaluated the base Qwen 3.6 27B model with both mini-SWE-agent and PurpAgent Harness on the same task set. Switching to PurpAgent Harness increases the Secure Apps rate from 15.0% to 20.0%, Task Pass from 0.0% to 15.0%, and Test Pass from 31.4% to 41.4%. Results show that both PurpAgent Harness and PurpAgent 27B fine-tuning yield significant improvements across all three red-teaming metrics, demonstrating complementary security benefits from the harness architecture and model specialization. We will include this expanded red-teaming evaluation in the revision.

LLM BackendAgent ScaffoldTask PassTest PassSecure Apps
Qwen 3.6 27Bmini-SWE-agent0.0%31.4%15.0%
PurpAgent Harness15.0%41.4%20.0%
PurpAgent 27Bmini-SWE-agent20.0%84.4%35.0%
PurpAgent Harness25.0%82.4%55.0%

Q2: Do you verify the contamination during training? Is it possible that training and test data have overlap, or based on the same repo?

To check for overlap between the synthetic SFT data and SusVibes, we traced the retained training trajectories to their source tasks and normalized the repository identifiers. For both synthesized task descriptions and their original benign source descriptions, we checked for exact matches after text normalization and screened for near duplicates using four-word-shingle Jaccard similarity with a threshold of 0.5. We also checked the source base and feature commit hashes against SusVibes task commits and compared normalized complete solution patches with the benchmark reference patches.

The analysis identified 11 shared repositories among the 101 repositories in SusVibes, covering 22 of its 186 tasks. Our checks found zero exact or near-duplicate task descriptions, matching task commits, or similar solution patches.

To test whether the gains persist on a repository-disjoint subset, we excluded the 22 tasks from shared repositories and recomputed both metrics on the remaining 164 tasks. To isolate the gain from fine-tuning, we evaluate both PurpAgent 27B and Qwen 3.6 27B within the same PurpAgent Harness and compare their performance differences. PurpAgent 27B achieves 34.76% FuncPass (57/164) and 15.85% SecPass (26/164), compared with 23.78% FuncPass (39/164) and 9.76% SecPass (16/164) for Qwen 3.6 27B. These are gains of 10.98 and 6.10 percentage points, respectively. These results eliminate the risk of data contamination, confirming that PurpAgent 27B’s gains in both FuncPass and SecPass do not come from repository overlaps between our synthetic SFT corpus and SusVibes. We will include the overlap analysis and this supplementary archived-run comparison in the revision.

Reviewer_e2XBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PurpAgent Harness, which translate repository evidence into explicit security obligations. This study also proposes PurpAengt 27B by applying mutual supervised fine-tuning oon synthesized coding and security-review trajectories. PurpAgent Harness improves the joint functioonal and security pass rate over mini-SWE-agent across 7 LLM backends.

Critical strengths
  • The investigated problem is meaningful and practical.
  • The joint-training results are promising. The authors have trained a small model on both coding and review trajectories which outperforms seperately trained model.
Critical weaknesses
  • The comparision is limited to a single harness, i.e., mini-SWE-agent. It is unclear whether the proposed harness is only better than mini-SWE-agent. Although Table 4 additionally compares against DCR, it remains unclear how PurpAgent performs against other widely used coding-agent harness. How about comparing with Claude Code and Codex harness?
  • Missing baselines on given the same agent the same total budget for multiple rounds of checking and repair. It is currently unclear on whether the benefits come from extra effort.
  • The task itself can be considered secure code generation. However, the current baseline set only includes general coding agents and DCR, while missing the specific secure coding generation approaches.
  • Missing clarification on whether the synthetic training data and SusVibes have overlaps, such as coming from the same repositories or duplicate task levels.
Questions for authors
  1. How to ensure the synthetic training data does not overlap with SusVibes?
  2. Under the same total inference budget, how does PurpAgent compare with giving the original coding agent additional rounds of checking and repair?
  3. How does PurpAgent compare with specialized secure code generation approaches, particularly those using security planning or iterative feedback?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer e2XB (Part 1/2)

We thank the reviewer for the thorough evaluation and constructive suggestions on broadening the comparison scope. Below, we address your specific concerns.

W1: The comparision is limited to a single harness, i.e., mini-SWE-agent. It is unclear whether the proposed harness is only better than mini-SWE-agent. Although Table 4 additionally compares against DCR, it remains unclear how PurpAgent performs against other widely used coding-agent harness. How about comparing with Claude Code and Codex harness?

Thank you for this excellent suggestion. To broaden the comparison, we evaluated additional harnesses/agent scaffolds, including Claude Code and Codex, on SusVibes with Claude Opus 4.6 as the backend for all four harnesses. Results show that PurpAgent Harness outperforms Codex, Claude Code, and mini-SWE-agent, nearly doubling or tripling their SecPass while matching or exceeding their FuncPass.

LLM BackendAgent ScaffoldFuncPassSecPass
mini-SWE-agent46.24%9.68%
Claude Opus 4.6Claude Code40.86%10.22%
Codex46.24%14.52%
PurpAgent Harness46.24%24.19%

PurpAgent Harness achieves 24.19% SecPass, compared with 10.22% for Claude Code and 14.52% for Codex. Its 46.24% FuncPass matches Codex and exceeds Claude Code’s 40.86%. We will report these results and the harness configurations alongside the DCR comparison in the revision.

W2: Missing baselines on given the same agent the same total budget for multiple rounds of checking and repair. It is currently unclear on whether the benefits come from extra effort.

Q2: Under the same total inference budget, how does PurpAgent compare with giving the original coding agent additional rounds of checking and repair?

We thank the reviewer for raising this point. To test whether the gain comes from additional inference, we gave mini-SWE-agent additional rounds to check and repair its own patch on SusVibes. We ran the extended baseline and PurpAgent Harness with the same total inference budget and the same Claude Opus 4.6 backend. Results show that the extra rounds of checking and repairing (i.e., same effort) only slightly increase FuncPass, but SecPass, our primary metric requiring both functional and security correctness, remains significantly lower than PurpAgent Harness.

LLM BackendAgent ScaffoldFuncPassSecPass
mini-SWE-agent46.24%9.68%
Claude Opus 4.6mini-SWE-agent w/ additional checking and repair47.85%10.22%
PurpAgent Harness46.24%24.19%

The additional rounds increase mini-SWE-agent’s SecPass from 9.68% to 10.22% and raise its FuncPass to 47.85%. At the same budget, PurpAgent Harness reaches 24.19% SecPass and 46.24% FuncPass. The extra rounds improve functional completion and recover only a small fraction of the observed SecPass gain. We will report this comparison and describe how we accounted for the inference budget in the revision.

W3: The task itself can be considered secure code generation. However, the current baseline set only includes general coding agents and DCR, while missing the specific secure coding generation approaches.

Q3: How does PurpAgent compare with specialized secure code generation approaches, particularly those using security planning or iterative feedback?

Thank you for the suggestion. We added AgenticSCR as a specialized security-feedback baseline. It reviews the candidate patch and sends its findings to the original coding agent for repair. All configurations use Claude Opus 4.6. Results show that PurpAgent Harness outperforms the AgenticSCR baseline in both SecPass and FuncPass.

LLM BackendFinal-patch ReviewFuncPassSecPass
N/A46.24%9.68%
Claude Opus 4.6AgenticSCR41.40%15.59%
PurpAgent Harness46.24%24.19%

AgenticSCR-guided repair increases SecPass from 9.68% to 15.59%. PurpAgent Harness achieves 24.19% SecPass and 46.24% FuncPass, improving on the AgenticSCR variant by 8.60 and 4.84 percentage points, respectively.

The ablation in Table 3 also quantifies the contribution of security planning within PurpAgent. Adding first-step review to the final-patch-only configuration increases SecPass by 5.37 percentage points. We will include the AgenticSCR results alongside this ablation in the revision.

AuthorsOpenReview ↗

Author Response to Reviewer e2XB (Part 2/2)

W4: Missing clarification on whether the synthetic training data and SusVibes have overlaps, such as coming from the same repositories or duplicate task levels.

Q1: How to ensure the synthetic training data does not overlap with SusVibes?

To check for overlap between the synthetic SFT data and SusVibes, we traced the retained training trajectories to their source tasks and normalized the repository identifiers. For both synthesized task descriptions and their original benign source descriptions, we checked for exact matches after text normalization and screened for near duplicates using four-word-shingle Jaccard similarity with a threshold of 0.5. We also checked the source base and feature commit hashes against SusVibes task commits and compared normalized complete solution patches with the benchmark reference patches.

The analysis identified 11 shared repositories among the 101 repositories in SusVibes, covering 22 of its 186 tasks. Our checks found zero exact or near-duplicate task descriptions, matching task commits, or similar solution patches.

To test whether the gains persist on a repository-disjoint subset, we excluded the 22 tasks from shared repositories and recomputed both metrics on the remaining 164 tasks. To isolate the gain from fine-tuning, we evaluate both PurpAgent 27B and Qwen 3.6 27B within the same PurpAgent Harness and compare their performance differences. PurpAgent 27B achieves 34.76% FuncPass (57/164) and 15.85% SecPass (26/164), compared with 23.78% FuncPass (39/164) and 9.76% SecPass (16/164) for Qwen 3.6 27B. These are gains of 10.98 and 6.10 percentage points, respectively. These results show that PurpAgent 27B’s gains in both FuncPass and SecPass do not come from repository overlaps between our synthetic SFT corpus and SusVibes. We will include the overlap analysis and this supplementary archived-run comparison in the revision.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer e2XB,

Thank you for your thoughtful review and constructive feedback. Your comments guided the additional experiments and analyses we included in our rebuttal.

If you have a chance before the discussion period ends, we would be grateful for any further thoughts on whether our response addresses your concerns. We would be happy to clarify any points or answer any remaining questions.

Thank you again for your time and consideration.

Thank you for the additional experiments and clarifications. Please include the budget accounting and baseline configurations in the revised paper so that the comparisons are clear. With these additions, I am more convinced by the results and will raise my rating to 3.

Reviewer_HkezOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PurpAgent, a framework that integrates security reasoning directly into LLM-based coding agents to ensure that generated code is both functionally correct and secure.

Critical strengths

1- Integrates security into the coding workflow rather than relying exclusively on post-hoc scanning. 2- Effective Fine-Tuning for Security-Aware Code Generation: The paper demonstrates that jointly fine-tuning an LLM on coding and security-review trajectories improves both functional correctness and security

Critical weaknesses

1- Success pass remain relatively low which is the highest reported is 29%. 2- Lack of evaluation on unseen vulnerabilities. 3- Lack of statistical significance analysis. The paper reports improvements across different models but does not provide confidence intervals, standard deviations, or statistical significance tests 4- Lack of vulnerability detection and repair analysis. although the paper reports improvements in SecPass, it does not explicitly quantify the number of security vulnerabilities detected or successfully repaired by the proposed framework

Questions for authors

1- Which types of security vulnerabilities are easier or more difficult for PurpAgent to detect and repair? 2- Have the authors considered comparing PurpAgent against traditional static code analysis tools (e.g., CodeQL or Semgrep) as additional baselines?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer Hkez (Part 1/2)

Thank you for your detailed evaluation and constructive suggestions. We address your concerns below.

W1: Success pass remain relatively low which is the highest reported is 29%

We agree that the best SecPass on SusVibes, 29.03% with Claude Opus 4.8, leaves substantial room for improvement. The low absolute SecPass primarily reflects the difficulty of the SusVibes benchmark and the ceiling of current LLM backends, rather than a limitation of PurpAgent Harness.

To validate this, we evaluated PurpAgent Harness on CWEval, a function-level secure code generation benchmark with 119 tasks covering 31 CWE categories across five programming languages. CWEval specifies the target function and its expected behavior, giving the agent a narrower implementation scope and making it less demanding than SusVibes. With Claude Opus 4.6, PurpAgent Harness passes both the functional and security tests on 109/119 tasks (91.60%), confirming strong performance when the task scope is narrower.

As additional context, SusVibes is substantially more challenging: each task requires the agent to implement a feature in an existing codebase and infer the relevant security requirements from the repository context, and SecPass requires the final patch to pass both functional and security tests. The same Claude Opus 4.6 backend with mini-SWE-agent achieves only 46.24% FuncPass and 9.68% SecPass on SusVibes; PurpAgent Harness improves SecPass by 14.52 percentage points. Across all seven LLM backends, PurpAgent Harness consistently improves SecPass, further confirming that the absolute values reflect the benchmark’s difficulty rather than PurpAgent’s capability. We will report the CWEval results alongside the SusVibes results in the revision.

W2: Lack of evaluation on unseen vulnerabilities

Thank you for raising this point. The central goal of secure code generation is to implement the requested feature while avoiding vulnerability patterns. Avoiding well-known vulnerabilities is the major part of this goal; reducing unknown (zero-day) vulnerabilities is a valuable benefit, but not the primary requirement. Consequently, we use SusVibes as our major benchmark, as it uses documented vulnerabilities to define the security requirements for feature-implementation tasks and evaluates generated patches with both functional and security tests.

That said, it is important to examine the generalizability of the trained model and ensure there is no leakage between training and testing. As such, we further checked whether tasks in SusVibes overlap with our synthetic SFT corpus by comparing repositories, task descriptions, commits, and solution patches. While the audit identified 11 shared repositories covering 22 of the 186 SusVibes tasks, there were zero exact or near-duplicate task descriptions, matching task commits, or similar solution patches. We then excluded the 22 tasks from shared repositories and recomputed SecPass on the remaining 164 tasks using the archived single-review replay evaluation for checkpoint 3381. On this repository-disjoint subset, PurpAgent 27B achieves 15.85% SecPass (26/164), compared with 9.76% (16/164) for Qwen 3.6 27B, a gain of 6.10 percentage points. These results show that PurpAgent 27B’s gains in both FuncPass and SecPass do not come from repository overlaps between our synthetic SFT corpus and SusVibes. We will include this supplementary analysis in the revision.

W3: Lack of statistical significance analysis. The paper reports improvements across different models but does not provide confidence intervals, standard deviations, or statistical significance tests

Thank you for highlighting the need for statistical analysis. To achieve this, we conducted three independent runs of both PurpAgent and mini-SWE-agent with Claude Opus 4.6 on the full 186-task SusVibes benchmark. As shown in the table, where values denote the mean ± sample standard deviation across three runs, PurpAgent consistently outperforms mini-SWE-agent in both FuncPass and SecPass, with improvements well outside the run-to-run variance. PurpAgent achieves 27.24 ± 2.65% SecPass, nearly tripling mini-SWE-agent’s 9.86 ± 0.31%, while also improving FuncPass from 42.47 ± 3.27% to 50.90 ± 4.18%.

Agent scaffoldFuncPass (%)SecPass (%)
mini-SWE-agent42.47 ± 3.279.86 ± 0.31
PurpAgent50.90 ± 4.1827.24 ± 2.65

Furthermore, we use a two-sided Wilcoxon signed-rank test to assess whether the SecPass improvement is statistically significant. For each task, we average each method’s binary SecPass outcomes across the three runs, obtaining 186 paired task-level scores. Using the exact conditional distribution with tied average ranks and omitting zero differences, we obtain p = 1.06 × 10⁻¹², which is substantially below the conventional significance level of 0.05 and indicates a statistically significant SecPass improvement for PurpAgent over mini-SWE-agent under this task-level comparison.

AuthorsOpenReview ↗

Author Response to Reviewer Hkez (Part 2/2)

W4: Lack of vulnerability detection and repair analysis. although the paper reports improvements in SecPass, it does not explicitly quantify the number of security vulnerabilities detected or successfully repaired by the proposed framework

Thank you for suggesting this analysis. Across all 186 SusVibes tasks evaluated with Claude Opus 4.6, PurpAgent’s independent reviewer reported 391 candidate security requirements. Of these, 249 were addressed through active code repair, and 142 were already satisfied by the original coding process without additional effort. In the end-to-end evaluation, PurpAgent Harness produced patches that passed both the functional and security tests on 45/186 tasks, compared with 18/186 for mini-SWE-agent, a net gain of 27 tasks. All 27 of these tasks involved security requirements that the coding agent actively addressed through code repair, confirming that the integrated review workflow contributes directly to the security improvements beyond simply writing more cautious code. We will include this analysis in the revision.

Q1: Which types of security vulnerabilities are easier or more difficult for PurpAgent to detect and repair?

We compared PurpAgent Harness with mini-SWE-agent using Claude Opus 4.6, grouping the results by SusVibes’ CWE annotations. Each category shown contains at least five tasks. A task with multiple CWE annotations contributes to each applicable row.

Vulnerability typeCWEPurpAgent FuncPassPurpAgent SecPassmini-SWE-agent SecPass
Regular-expression complexity / ReDoSCWE-13335/75/7 (71.43%)1/7 (14.29%)
Path traversalCWE-224/94/9 (44.44%)1/9 (11.11%)
Cross-site scriptingCWE-7913/266/26 (23.08%)1/26 (3.85%)
Information exposureCWE-2003/111/11 (9.09%)1/11 (9.09%)
SQL injectionCWE-890/60/60/6
Cross-site request forgeryCWE-3521/50/50/5

PurpAgent Harness yields the largest gains on categories with localized security patterns. Compared with mini-SWE-agent, SecPass improves from 1/7 to 5/7 for ReDoS (+57.14 percentage points), from 1/9 to 4/9 for path traversal (+33.33 points), and from 1/26 to 6/26 for cross-site scripting (+19.23 points). These categories involve security mechanisms that map to identifiable code locations, including regex anchoring, path canonicalization, and output sanitization, which align with PurpAgent’s repository-grounded review workflow. Both scaffolds achieve comparable results on information exposure (1/11 each) and neither succeeds on SQL injection (0/6) or cross-site request forgery (0/5). The zero SecPass on the latter two should be interpreted cautiously: all six SQL injection tasks fail the functional tests under both scaffolds, so their zero SecPass reflects functional difficulty rather than detection or repair limitations. We will include this per-CWE baseline comparison in the revision.

Q2: Have the authors considered comparing PurpAgent against traditional static code analysis tools (e.g., CodeQL or Semgrep) as additional baselines?

Thank you for suggesting these comparisons. We evaluated two variants that replace the independent final-patch reviewer in PurpAgent Harness with CodeQL or Semgrep. Both retain first-step review, the Claude Opus 4.6 coding agent, and the review-guided repair stage. PurpAgent Harness outperforms both CodeQL and Semgrep, nearly doubling their SecPass while maintaining full FuncPass.

CodeQL uses its Python security-extended query suite, and Semgrep uses 305 public Python security rules. Each tool analyzes an isolated checkout containing the cumulative candidate patch. We pass findings associated with modified files to the coding agent for validation and repair, including relevant CodeQL data-flow locations outside the edited lines.

LLM BackendFinal-patch ReviewFuncPassSecPass
N/A46.24%9.68%
Claude Opus 4.6CodeQL40.32%11.29%
Semgrep42.47%12.90%
PurpAgent Harness46.24%24.19%

The CodeQL and Semgrep variants achieve 11.29% and 12.90% SecPass, respectively, compared with 9.68% for mini-SWE-agent. PurpAgent Harness reaches 24.19% SecPass and retains mini-SWE-agent’s 46.24% FuncPass. Both scanner-guided variants have lower FuncPass. We will include these comparisons and the scanner configurations in the revision.

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer Hkez,

Thank you for your thoughtful review and constructive feedback. Your comments guided the additional experiments and analyses we included in our rebuttal.

If you have a chance before the discussion period ends, we would be grateful for any further thoughts on whether our response addresses your concerns. We would be happy to clarify any points or answer any remaining questions.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.