- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies how to make coding agents produce code that is both correct and secure. The authors propose PurpAgent Harness, which adds a first-step review and an independent final patch review to a coding agent. The review findings are sent back into the original coding trajectory to guide repair. The authors also train PurpAgent 27B on 16k synthesized coding and review trajectories, using one model for both roles. Experimental results on SusVibes show SecPass gains of 2.15 to 14.49 points across seven backends, and PurpAgent 27B improves SecPass from 10.75% to 17.70% over its base model.
Critical strengths
- The paper proposes PurpAgent Harness to address the security issue in vibe coding. Different from the existing post-hoc methods, the harness provides a security review before and after the coding process, avoiding introducing vulnerabilities during feature implementation.
- The authors also synthesize a large-scale training data for security, which includes 16k tasks across 192 CWE categories. This dataset can contribute to vibe coding training. They also verify the effectiveness of the synthesized data by training 27B PurpAgent.
- Experimental results show that with PurpAgent harness, SecPass improves for all 7 backends, including both open-source and closed-source LLMs. It also beats DCR Harness in SecPass at a similar cost to DCR without PoC.
- The results on red teaming show the proposed method can be generalized to Secure Apps.
Critical weaknesses
- As claimed in the abstract, PurpAgent maintains FuncPass in general. However, FuncPass drops by 5 to 6 points for GPT-5.6 Sol and Grok 4.6, which is not an ignorable loss for pass rate.
- The proposed PurpAgent harness contains a first-step review and a final-patch review. The first-step review is a structured instruction for the coding agent, which is similar to some simple modifications to the system prompt. The final-patch review is more like an independent post-hoc reviewer to check after the implementation. While the authors mention that they first propose the PurpAgent scaffold to address the challenges of post-hoc methods, the biggest gains come from adding a post-hoc reviewer (Table 3). The interaction between the coding agent and the post-hoc reviewer is straightforward.
- There are no repeated runs or significance tests. Some numbers do not match 186 tasks, such as 9.70% in Table 1, 42.90% and 14.70% in Table 6.
- In Table 3, more discussion should be provided for FuncPass. Removing first-step review and final-patch review alone lowers FuncPass by 5%, but having both recovers it to 46%.
- In Table 5, each separate SFT model sees only half the data. The gain from mutual SFT may come from more data, not from transfer between roles.
- Much of the PurpAgent 27B gain may come from better coding in general. Compared with Qwen 3.6 27B, PurpAgent 27B increases 11.33% on FuncPass but only 6.95% on SecPass. Btw, the authors don't mention where the training trajectories come from. The red teaming study is small. It uses one team and 20 tasks. The gain is 7 apps versus 11 apps. Only PurpAgent 27B is tested.
Questions for authors
- The paper states that SusVibes has 186 tasks. However, several reported values, such as 9.70% in Table 1, 52.17% and 23.37% for Grok 4.6 in Table 1 and 42.90% and 14.70% in Table 6, cannot be obtained from 186 tasks.
- Do you verify the contamination during training? Is it possible that training and test data have overlap, or based on the same repo?
- How environments were built for the 16k tasks? The adversarial variants likely reuse the environments of their source tasks, but this should be described clearly. It is unclear how new features from feature grafting were tested, and the variants have no security tests.
- What are SFT details such as learning rate, epochs, and hardware?
6 comments
Author Response to Reviewer y7zJ (Part 2/4)
W3: There are no repeated runs or significance tests.
Q1: The paper states that SusVibes has 186 tasks. However, several reported values, such as 9.70% in Table 1, 52.17% and 23.37% for Grok 4.6 in Table 1 and 42.90% and 14.70% in Table 6, cannot be obtained from 186 tasks.
Thank you for pointing out these reporting inconsistencies. We rechecked the saved task-level outcomes and traced all discrepancies to denominator and rounding errors. In Table 1, 9.70% resulted from rounding 18/186 to one decimal place and then displaying a trailing zero; the correct value rounded directly to two decimals is 9.68%. Both the Grok 4.6 results in Table 1 (52.17%, 23.37%) and the internal-review results in Table 6 (42.90%, 14.70%) used a denominator of 184 rather than 186, as they were all carried over from an intermediate summary, corresponding to 96/184 and 43/184 for Grok and 79/184 and 27/184 for internal review. Among these, the internal-review ratios (79/184 and 27/184) were additionally rounded to one decimal place before being displayed with two, producing the reported 42.90% and 14.70%. Using the full 186-task denominator gives 51.61% (96/186) and 23.12% (43/186) for Grok, and 42.47% (79/186) and 14.52% (27/186) for internal review. None of these corrections change the relative ordering or conclusions of the paper. We will correct all affected tables and accompanying text in the revision.
To address the concern about repeated runs, we conducted three independent runs of both PurpAgent and mini-SWE-agent with Claude Opus 4.6 on the full 186-task SusVibes benchmark. As shown in the table, where values denote the mean ± sample standard deviation across three runs, PurpAgent consistently outperforms mini-SWE-agent in both FuncPass and SecPass, with improvements well outside the variance across runs. PurpAgent achieves 27.24 ± 2.65% SecPass, nearly tripling mini-SWE-agent’s 9.86 ± 0.31%, while also improving FuncPass from 42.47 ± 3.27% to 50.90 ± 4.18%. A Wilcoxon signed-rank test over task-level outcomes yields p = 1.06 × 10⁻¹², confirming that the differences are statistically significant.
| Agent scaffold | FuncPass (%) | SecPass (%) |
|---|---|---|
| mini-SWE-agent | 42.47±3.27 | 9.86±0.31 |
| PurpAgent | 50.90±4.18 | 27.24±2.65 |
W4: In Table 3, more discussion should be provided for FuncPass. Removing first-step review and final-patch review alone lowers FuncPass by 5%, but having both recovers it to 46%.
Thank you – this is a great observation. The major reason behind this phenomenon is that poorly grounded security findings can lead the coding agent to reject legitimate inputs, add overly restrictive checks, or disable requested behavior, resulting in functional regressions. In contrast, more accurate findings can help mitigate these regressions. Specifically, first-step review identifies the intended functionality and the security obligations the implementation should satisfy. Final-patch review examines the resulting code and provides repository-grounded feedback to the original coding agent. Together, the two stages can improve the quality of security findings and help preserve the intended functionality.
The FuncPass results in Table 3 are consistent with this explanation. First-step review alone achieves 41.00% FuncPass, and final-patch review alone achieves 41.40%. Using both stages restores FuncPass to 46.24%. The full harness also raises SecPass from the baseline’s 9.68% to 24.19%. We will expand this discussion in the revision.
W5: In Table 5, each separate SFT model sees only half the data. The gain from mutual SFT may come from more data, not from transfer between roles.
Thank you for raising this point. We first clarify that mutual SFT and separate SFT use the same total training budget: separate SFT trains two independent models on each half of the data, while mutual SFT trains one model on all of it.
The key question is whether mutual SFT’s gain comes from seeing both roles per task (role transfer) or simply from training on more data. To test this, we conduct a controlled experiment using a 5k-task subset. Separate SFT trains on all 5k tasks, using the 5k coding trajectories and 5k security-review trajectories independently. In contrast, mutual SFT trains on only 2.5k tasks from this subset but uses both their coding and security-review trajectories, giving it strictly less data while retaining the cross-role signal.
As shown in the table, mutual SFT matches separate SFT in FuncPass while achieving higher SecPass with only half the training tasks, confirming that the gain also comes from cross-role transfer. Specifically, both strategies achieve 28.49% FuncPass, but mutual SFT reaches 13.98% SecPass compared with 12.37% for separate SFT. We will include this controlled experiment in the revision.
| Base Model | Training Strategy | FuncPass | SecPass |
|---|---|---|---|
| Qwen 3.6 27B | Separate SFT | 28.49% | 12.37% |
| Mutual SFT | 28.49% | 13.98% |
Author Response to Reviewer y7zJ (Part 3/4)
W6: Much of the PurpAgent 27B gain may come from better coding in general. Compared with Qwen 3.6 27B, PurpAgent 27B increases 11.33% on FuncPass but only 6.95% on SecPass. Btw, the authors don't mention where the training trajectories come from. The red teaming study is small. It uses one team and 20 tasks. The gain is 7 apps versus 11 apps. Only PurpAgent 27B is tested.
We agree that improved general coding ability can contribute to PurpAgent 27B’s SecPass gains because SecPass requires passing both functional and security tests. However, across the 35 tasks where both models pass the functional tests, PurpAgent 27B achieves a security test pass rate of 20/35 (57.14%), compared with 13/35 (40.00%) for Qwen 3.6 27B. This reflects PurpAgent’s SecPass gains on tasks where both models solve the functionality, indicating an improved ability to code securely rather than just a general coding improvement.
Regarding the source of training trajectories, we distilled the coding and security-review training trajectories from Claude Opus 4.8 and jointly fine-tuned Qwen 3.6 27B on those trajectories for both roles. We will identify the teacher model and describe how we generated the trajectories in the revision.
We also agree that the red-teaming evaluation reported in the paper is relatively small. To enrich our red-teaming evaluation, we further evaluated the base Qwen 3.6 27B model with both mini-SWE-agent and PurpAgent Harness on the same task set. Switching to PurpAgent Harness increases the Secure Apps rate from 15.0% to 20.0%, Task Pass from 0.0% to 15.0%, and Test Pass from 31.4% to 41.4%. Results show that both PurpAgent Harness and PurpAgent 27B fine-tuning yield significant improvements across all three red-teaming metrics, demonstrating complementary security benefits from the harness architecture and model specialization. We will include this expanded red-teaming evaluation in the revision.
| LLM Backend | Agent Scaffold | Task Pass | Test Pass | Secure Apps |
|---|---|---|---|---|
| Qwen 3.6 27B | mini-SWE-agent | 0.0% | 31.4% | 15.0% |
| PurpAgent Harness | 15.0% | 41.4% | 20.0% | |
| PurpAgent 27B | mini-SWE-agent | 20.0% | 84.4% | 35.0% |
| PurpAgent Harness | 25.0% | 82.4% | 55.0% |
Q2: Do you verify the contamination during training? Is it possible that training and test data have overlap, or based on the same repo?
To check for overlap between the synthetic SFT data and SusVibes, we traced the retained training trajectories to their source tasks and normalized the repository identifiers. For both synthesized task descriptions and their original benign source descriptions, we checked for exact matches after text normalization and screened for near duplicates using four-word-shingle Jaccard similarity with a threshold of 0.5. We also checked the source base and feature commit hashes against SusVibes task commits and compared normalized complete solution patches with the benchmark reference patches.
The analysis identified 11 shared repositories among the 101 repositories in SusVibes, covering 22 of its 186 tasks. Our checks found zero exact or near-duplicate task descriptions, matching task commits, or similar solution patches.
To test whether the gains persist on a repository-disjoint subset, we excluded the 22 tasks from shared repositories and recomputed both metrics on the remaining 164 tasks. To isolate the gain from fine-tuning, we evaluate both PurpAgent 27B and Qwen 3.6 27B within the same PurpAgent Harness and compare their performance differences. PurpAgent 27B achieves 34.76% FuncPass (57/164) and 15.85% SecPass (26/164), compared with 23.78% FuncPass (39/164) and 9.76% SecPass (16/164) for Qwen 3.6 27B. These are gains of 10.98 and 6.10 percentage points, respectively. These results eliminate the risk of data contamination, confirming that PurpAgent 27B’s gains in both FuncPass and SecPass do not come from repository overlaps between our synthetic SFT corpus and SusVibes. We will include the overlap analysis and this supplementary archived-run comparison in the revision.
Author Response to Reviewer y7zJ (Part 1/4)
We appreciate the reviewer’s detailed analysis and thoughtful questions. We address your concerns below.
We agree that the FuncPass declines for GPT-5.6 Sol and Grok 4.6 are meaningful and will revise the abstract to reflect this backend-dependent trade-off.
We want to gently remind that SecPass is the sole metric we use to determine end-to-end task success on SusVibes. As described in Section 4.1, it counts a task as successful only when the patch passes both functional and security tests. This reflects our goal of generating mergeable patches because, in practice, a patch with detected vulnerabilities will still be rejected from merging, even if it passes all functional tests. We report FuncPass to quantify the functional trade-off. Table 1 shows that PurpAgent Harness improves SecPass by up to 14.52 percentage points over mini-SWE-agent on all seven evaluated LLM backends, including GPT-5.6 Sol and Grok 4.6.
Thank you for the question. We would like to clarify potential misunderstandings and highlight the importance of first-step review, the in-progress PR security review, and our fundamental differences with post-hoc reviewers. First, while we agree that final-patch review yields the larger gain when the two stages are enabled individually, first-step review also plays an important role. Specifically, with final-patch review enabled in both configurations, adding first-step review raises SecPass from 18.82% to 24.19% and FuncPass from 41.40% to 46.24%.
Furthermore, we want to kindly emphasize that there’s a fundamental difference between the post-hoc reviewer and PurpAgent. Post-hoc review begins when the completed PR is submitted. PurpAgent Harness is different, as it runs an independent security review whenever the coding agent submits a candidate patch to update the in-progress PR. The harness returns the findings as a new turn in the original coding trajectory, so the coding agent can use its implementation history to verify and address them before finalizing the PR. Consequently, while existing post-hoc reviewers may leave vulnerabilities introduced during implementation undetected until later stages, where the growing code complexity can make them much harder and more costly to identify/repair, PurpAgent Harness can successfully handle these vulnerabilities on-the-fly.
Lastly, we show that PurpAgent Harness clearly outperforms Anthropic DCR Harness, one of the most popular post-hoc reviewers. As shown in Table 4 in the paper, with Claude Opus 4.6 as the backend for all three configurations, PurpAgent Harness achieves 24.19% SecPass on SusVibes, compared with 20.97% for DCR w/o PoC and 21.51% for DCR w/ PoC. PurpAgent Harness takes 11.3 minutes and costs $3.06 per task, comparable to DCR w/o PoC (11.7 minutes and $3.14) and lower than DCR w/ PoC (82.9 minutes and $23.75). These results indicate that PurpAgent’s integrated review workflow can produce more patches that pass both functional and security tests with comparable or lower runtime and cost per task. We will clarify this distinction between post-hoc review and PurpAgent’s integrated approach more explicitly in the revision.