- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper explores how to enable programming agents to proactively identify and avoid security vulnerabilities while completing their functions. The core approach integrates incorporating security planning, secure coding, and security testing into the training process, and then further enhancing it through reinforcement learning or prompt-guided self-distillation.
Critical strengths
- Users often only specify functional requirements and do not explicitly list security requirements. The paper focuses on repository-level programming tasks that are "functionally correct but have vulnerabilities," which is more aligned with the practical use of agents than the generation of secure code for individual functions.
- Separating security planning, coding, and testing into distinct training tasks, rather than merely increasing the number of security code samples, is beneficial. Compared to Simple Mixture, which has a similar sample size, Security Suite achieves a higher security pass rate, providing support for this training design.
- By combining the success trajectory ratio and vulnerability categories, explain why RL and prompt distillation perform differently. This provides valuable insights into the issue of sparse rewards in safety tasks.
Critical weaknesses
- SFT, GRPO, and online self-distillation all utilize existing techniques. The paper needs to more clearly specify whether the contributions primarily come from task organization, prompt design, or training objectives, and distinguish them through ablation studies.
- The conclusion that "the safety and functional gap has been mitigated" is insufficient. The increase in joint pass rates may partly result from enhanced functional capabilities, but it does not yet indicate that functionally correct code is less prone to vulnerabilities.
- The benefits of post-training are smaller and less stable.
- Insufficient ablation. The planning and testing tasks were introduced together, and there is a lack of experiments that introduce them separately. Prompt distillation actually also incorporates reward objectives, but there is no control group to examine the effect of distillation alone. There is also no general programming SFT control with an equivalent training budget.
- There are ambiguities in the data partitioning and training configurations that affect credibility. Table 1 refers to the data as a "held-out validation set," but Appendix C uses the same scale of data for post-training. Additionally, the descriptions of the number of samples, batch sizes, and sequence lengths in the main text and appendix are inconsistent and need clarification.
Questions for authors
see above weakness
3 comments
W4: Insufficient ablation. The planning and testing tasks were introduced together, and there is a lack of experiments that introduce them separately.
A: Distillation SFT. We added Functionality-Only SFT, trained on generic coding trajectories from the same teacher as SecureVibe_base. It improves FuncPass but lowers SecPass below the base model.
| SusVibes | BaxBench | |||
|---|---|---|---|---|
| FuncPass | SecPass | FuncPass | SecPass | |
| Qwen 3.5 35B | 28.14 | 10.57 | 32.91 | 19.71 |
| + Func-Only SFT | 39.78 | 6.45 | 40.31 | 20.41 |
| SECUREVIBE_base | 41.76 | 13.62 | 36.48 | 25.34 |
| SECUREVIBE_rl | 41.76 | 13.26 | 41.07 | 26.62 |
| SECUREVIBE_hg | 39.96 | 14.16 | 38.26 | 23.64 |
The existing distillation SFT baselines show the same pattern: distillation improves FuncPass on both datasets but lowers SecPass. Figure 4 shows that distillation is less effective in triggering security behaviors, especially in explicit security planning (55% fewer) and security testing (26% fewer).
Distillation Post-training. Moreover, SecureVibe_rl and SecureVibe_hg share the same task reward, so their difference isolates the hint-based teacher distillation term.
Planning vs. testing. We introduce them together because testing targets the risks identified during planning; without planning, security tests have no stated risk to check, and without testing, planned mitigations go unverified. We are collecting more planning and testing trajectories to run each ablation at full scale and will include the results in the final version.
W5: There are ambiguities in the data partitioning and training configurations that affect credibility.
A: No Table 1 instance is used in any training stage. We revised Section 4 and Appendix C to state both protocols:
- Table 1 (in-domain held-out). All training stages use 100 PatchEval-Gen and 140 AutoBaxBench instances; the remaining 100 and 420 instances are evaluation only, with no shared CVE IDs or backend scenarios. Appendix C's "remaining 100 and 420" describes the Table 2 models, not Table 1.
- Table 2 (cross-dataset). Starting from the Table 1 checkpoints, we continue post-training SecureVibe_rl and SecureVibe_hg on the remaining 100 PatchEval-Gen and 420 AutoBaxBench instances, so the Table 2 models have seen all 200 and 560 instances. Evaluation is on SusVibes and BaxBench, with no instance overlap.
Configuration, now consistent across sections: 128K is the inference context limit and 64K is the training sequence limit; the RL batch is 4 prompts × 8 samples = 32 (Appendix E's "8 × 4, batch 16" is corrected).
Thank you to the authors for addressing the responses, the functional-only SFT baseline, and the clarifications regarding data partitioning and hyperparameters, which resolved my initial concerns about potential data leakage in Table 1.
However, some issues remain. First, on the shared functional correct subset, SecureVibe_base underperforms the baseline model on SusVibes, while RL and HG only obtain one instance; on BaxBench, both post-training models show a decline relative to SecureVibe_base, which does not convincingly indicate that the gap between functionality and safety has been mitigated. Second, the clarified Table 2 shows that SecureVibe_rl and SecureVibe_hg underwent post-training on an additional 520 instances, yet despite this data advantage, their performance across benchmarks remains inconsistent. Finally, the ablation experiment decoupling safety planning from testing has been deferred to the final version. I will maintain my initial rating.
We are grateful for Reviewer JM4k's thoughtful review and valuable suggestions. It is encouraging that the reviewer appreciates our focus on code that works but remains vulnerable, the design of separate planning, coding, and testing tasks, and our analysis of RL versus hint-guided distillation under sparse rewards. The suggested controls have strengthened the paper, and we present the new results below.
W1: SFT, GRPO, and online self-distillation contribution on task performance. Specify whether the contributions primarily come from task organization, prompt design, or training objectives, and distinguish them through ablation studies.
A: The main research question in our paper is how to improve security awareness in coding agent. From observation in Section 3.1, we find that the main bottleneck is the agent is not aware of security during implementation. Therefore, we introduce our training recipe with SFT data mixture (trigger awareness) and Post-training methodology selection (enhance security and generalization) to improve security performance. We verify these agent security behaviors from a qualitative (Section 4) and quantitative perspective (Section 5).
In the paper, each factor is isolated by a specific comparison:
The results give three conclusions. The task mixture in SFT establishes basic security awareness and improves FuncPass and SecPass (Table 1, Figure 4). Post-training further improves security, especially on unseen vulnerability categories (Table 3). Hint-guided distillation gives denser supervision than RL, raising explicit security planning and testing, which suits hard tasks with sparse positive rewards.
W2: The conclusion that "the safety and functional gap has been mitigated" is insufficient. The increase in joint pass rates may partly result from enhanced functional capabilities, but it does not yet indicate that functionally correct code is less prone to vulnerabilities.
A: We thank the reviewer for this valuable suggestion. We compared SecPass on the tasks that every model solves functionally, so functional ability is held fixed:
On BaxBench, our three models improve the security level on the functionality correct solutions compared with the baseline. On SusVibes, the two post-trained models contribute more on making the solution more secure.
W3: The benefits of post-training are smaller and less stable.
A: The benefits of SFT seem to be larger than post-training because:
However, post-training shows better generalization capability. on unseen CWE, it increases 6.41 SecPass on SusVibes and 4.28 on BaxBench compared with the SFT checkpoint.
Our findings show that the post-training methods are effective in extending SFT capability, and provide guidance on how to select for different scenarios. These post-training models can benefit from larger training budgets (more powerful base model or longer training) to make these gains larger and more stable.