acceptodds
Under review as a conference paper at ICLR 2027

SecureVibe: Making Vibe Coding More Secure

Abstract

As vibe coding becomes increasingly capable and widespread, security vulnerabilities in even functionally correct solutions are a growing concern. When investigating functionally correct but insecure solutions, we find that the insecure agent is less than half as likely to conduct effective planning and testing for the hidden security risks behind the functional requirements. Motivated by this, we develop SECUREVIBE, a training recipe that explicitly targets planning and testing for code security. SECUREVIBE constructs training signals around these security behaviors. It includes supervised fine-tuning on the security suite with 4 security tasks, and post-training methods, SECUREVIBE_rl and SECUREVIBE_hg, to enhance security capabilities from verifiable execution feedback and hint-based self-supervision. Our SECUREVIBE outperforms the baseline on two types of security coding tasks across 4 benchmarks. Specifically, SECUREVIBE improves the security pass@1 by 6.9 points on BaxBench. The gains extend to unseen CWE categories, with improvements of 11.5 points on SusVibes. Meanwhile, it also improves functionality pass@1 by 13.6 points on the security coding task SusVibes and 4.1 points on the generic coding task SWE-bench Verified. Further analysis offers two practical insights: (i) diversifying supervision across security planning, coding, and testing strengthens security behaviors more effectively than adding coding trajectories alone, and (ii) hint-guided supervision is particularly valuable when the agent’s existing security capabilities are insufficient to learn effectively from outcome feedback.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cybersecurity, Coding Agent, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JM4kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores how to enable programming agents to proactively identify and avoid security vulnerabilities while completing their functions. The core approach integrates incorporating security planning, secure coding, and security testing into the training process, and then further enhancing it through reinforcement learning or prompt-guided self-distillation.

Critical strengths
  1. Users often only specify functional requirements and do not explicitly list security requirements. The paper focuses on repository-level programming tasks that are "functionally correct but have vulnerabilities," which is more aligned with the practical use of agents than the generation of secure code for individual functions.
  2. Separating security planning, coding, and testing into distinct training tasks, rather than merely increasing the number of security code samples, is beneficial. Compared to Simple Mixture, which has a similar sample size, Security Suite achieves a higher security pass rate, providing support for this training design.
  3. By combining the success trajectory ratio and vulnerability categories, explain why RL and prompt distillation perform differently. This provides valuable insights into the issue of sparse rewards in safety tasks.
Critical weaknesses
  1. SFT, GRPO, and online self-distillation all utilize existing techniques. The paper needs to more clearly specify whether the contributions primarily come from task organization, prompt design, or training objectives, and distinguish them through ablation studies.
  2. The conclusion that "the safety and functional gap has been mitigated" is insufficient. The increase in joint pass rates may partly result from enhanced functional capabilities, but it does not yet indicate that functionally correct code is less prone to vulnerabilities.
  3. The benefits of post-training are smaller and less stable.
  4. Insufficient ablation. The planning and testing tasks were introduced together, and there is a lack of experiments that introduce them separately. Prompt distillation actually also incorporates reward objectives, but there is no control group to examine the effect of distillation alone. There is also no general programming SFT control with an equivalent training budget.
  5. There are ambiguities in the data partitioning and training configurations that affect credibility. Table 1 refers to the data as a "held-out validation set," but Appendix C uses the same scale of data for post-training. Additionally, the descriptions of the number of samples, batch sizes, and sequence lengths in the main text and appendix are inconsistent and need clarification.
Questions for authors

see above weakness

3 comments

We are grateful for Reviewer JM4k's thoughtful review and valuable suggestions. It is encouraging that the reviewer appreciates our focus on code that works but remains vulnerable, the design of separate planning, coding, and testing tasks, and our analysis of RL versus hint-guided distillation under sparse rewards. The suggested controls have strengthened the paper, and we present the new results below.

W1: SFT, GRPO, and online self-distillation contribution on task performance. Specify whether the contributions primarily come from task organization, prompt design, or training objectives, and distinguish them through ablation studies.

A: The main research question in our paper is how to improve security awareness in coding agent. From observation in Section 3.1, we find that the main bottleneck is the agent is not aware of security during implementation. Therefore, we introduce our training recipe with SFT data mixture (trigger awareness) and Post-training methodology selection (enhance security and generalization) to improve security performance. We verify these agent security behaviors from a qualitative (Section 4) and quantitative perspective (Section 5).

In the paper, each factor is isolated by a specific comparison:

  • Prompt design. All models use the same inference prompt, so prompt design does not explain differences between them.
  • Task organization. The SFT baselines ablate task components with the same teacher: Security-Only SFT removes functionality-focused coding, Simple Mixture SFT removes planning and testing, and the new Functionality-Only SFT keeps only generic coding. SecureVibe_base outperforms all three (Table 1; Functionality-Only results in W4).
  • Training objectives. SecureVibe_base (SFT), SecureVibe_rl (outcome-based GRPO), and SecureVibe_hg (GRPO reward plus on-policy distillation) share data and differ only in objective.

The results give three conclusions. The task mixture in SFT establishes basic security awareness and improves FuncPass and SecPass (Table 1, Figure 4). Post-training further improves security, especially on unseen vulnerability categories (Table 3). Hint-guided distillation gives denser supervision than RL, raising explicit security planning and testing, which suits hard tasks with sparse positive rewards.


W2: The conclusion that "the safety and functional gap has been mitigated" is insufficient. The increase in joint pass rates may partly result from enhanced functional capabilities, but it does not yet indicate that functionally correct code is less prone to vulnerabilities.

A: We thank the reviewer for this valuable suggestion. We compared SecPass on the tasks that every model solves functionally, so functional ability is held fixed:

BenchmarkSusVibes (36)BaxBench (55)
Qwen 3.5 35B33.33%50.91%
SecureVibe_base30.56%65.45%
SecureVibe_rl36.11%63.64%
SecureVibe_hg36.11%60.00%

On BaxBench, our three models improve the security level on the functionality correct solutions compared with the baseline. On SusVibes, the two post-trained models contribute more on making the solution more secure.


W3: The benefits of post-training are smaller and less stable.

A: The benefits of SFT seem to be larger than post-training because:

  • SFT data is distilled from a stronger model 230B Minimax M2.7, and the model can quickly learn from distillation. For example, Security Only SFT can already improve 9.45 SecPass on AutoBaxBench held-out data.
  • Task Mixture in SFT triggers the security patterns from a low point. For example, SecureVibe_base has 2.25x security planning and 1.45x testing on PatchEval-Gen.

However, post-training shows better generalization capability. on unseen CWE, it increases 6.41 SecPass on SusVibes and 4.28 on BaxBench compared with the SFT checkpoint.

Our findings show that the post-training methods are effective in extending SFT capability, and provide guidance on how to select for different scenarios. These post-training models can benefit from larger training budgets (more powerful base model or longer training) to make these gains larger and more stable.

W4: Insufficient ablation. The planning and testing tasks were introduced together, and there is a lack of experiments that introduce them separately.

A: Distillation SFT. We added Functionality-Only SFT, trained on generic coding trajectories from the same teacher as SecureVibe_base. It improves FuncPass but lowers SecPass below the base model.

SusVibesBaxBench
FuncPassSecPassFuncPassSecPass
Qwen 3.5 35B28.1410.5732.9119.71
+ Func-Only SFT39.786.4540.3120.41
SECUREVIBE_base41.7613.6236.4825.34
SECUREVIBE_rl41.7613.2641.0726.62
SECUREVIBE_hg39.9614.1638.2623.64

The existing distillation SFT baselines show the same pattern: distillation improves FuncPass on both datasets but lowers SecPass. Figure 4 shows that distillation is less effective in triggering security behaviors, especially in explicit security planning (55% fewer) and security testing (26% fewer).

Distillation Post-training. Moreover, SecureVibe_rl and SecureVibe_hg share the same task reward, so their difference isolates the hint-based teacher distillation term.

Planning vs. testing. We introduce them together because testing targets the risks identified during planning; without planning, security tests have no stated risk to check, and without testing, planned mitigations go unverified. We are collecting more planning and testing trajectories to run each ablation at full scale and will include the results in the final version.


W5: There are ambiguities in the data partitioning and training configurations that affect credibility.

A: No Table 1 instance is used in any training stage. We revised Section 4 and Appendix C to state both protocols:

  • Table 1 (in-domain held-out). All training stages use 100 PatchEval-Gen and 140 AutoBaxBench instances; the remaining 100 and 420 instances are evaluation only, with no shared CVE IDs or backend scenarios. Appendix C's "remaining 100 and 420" describes the Table 2 models, not Table 1.
  • Table 2 (cross-dataset). Starting from the Table 1 checkpoints, we continue post-training SecureVibe_rl and SecureVibe_hg on the remaining 100 PatchEval-Gen and 420 AutoBaxBench instances, so the Table 2 models have seen all 200 and 560 instances. Evaluation is on SusVibes and BaxBench, with no instance overlap.

Configuration, now consistent across sections: 128K is the inference context limit and 64K is the training sequence limit; the RL batch is 4 prompts × 8 samples = 32 (Appendix E's "8 × 4, batch 16" is corrected).

Thank you to the authors for addressing the responses, the functional-only SFT baseline, and the clarifications regarding data partitioning and hyperparameters, which resolved my initial concerns about potential data leakage in Table 1.

However, some issues remain. First, on the shared functional correct subset, SecureVibe_base underperforms the baseline model on SusVibes, while RL and HG only obtain one instance; on BaxBench, both post-training models show a decline relative to SecureVibe_base, which does not convincingly indicate that the gap between functionality and safety has been mitigated. Second, the clarified Table 2 shows that SecureVibe_rl and SecureVibe_hg underwent post-training on an additional 520 instances, yet despite this data advantage, their performance across benchmarks remains inconsistent. Finally, the ablation experiment decoupling safety planning from testing has been deferred to the final version. I will maintain my initial rating.

Reviewer_GQTeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SecureVibe addresses a practical failure mode: a coding agent can implement a requested feature and still leave a vulnerability. The paper trains an agent to anticipate hidden security risks and test its own code, then applies outcome-based RL or hint-guided distillation. It reports higher joint functional-and-security pass rates on repository-level coding tasks.

Critical strengths
  1. The paper tackles a consequential failure mode of coding agents: a feature can work while its implementation remains vulnerable. The focus on recognizing implicit security requirements and testing them during repository-level work is a meaningful research direction, and the trajectory analysis gives that direction a concrete motivation.

  2. The external evaluations offer evidence beyond the disputed validation split. On BaxBench, the best trained model raises SecPass from 19.71% to 26.62%; on the unseen-CWE SusVibes subset, it rises from 7.69% to 19.23%. These are meaningful results for an important application.

Critical weaknesses
  1. The held-out status of Table 1 is unclear for the post-trained models. The caption calls the 100 PatchEval-Gen and 420 AutoBaxBench instances a held-out validation set, but Appendix C says the remaining 100 and 420 instances are used for SecureVibe_rl and SecureVibe_hg post-training. If these are the same instances, the RL/HG rows are not independent validation. This does not establish data leakage; the paper needs explicit instance IDs and stage-by-stage splits before Table 1 can support its held-out interpretation. The separate BaxBench and SusVibes results remain relevant.

  2. The post-training objective and configuration are inconsistent across sections. Section 3.3 presents SecureVibe_hg through a distillation loss, while Appendix E says it combines task-reward optimization with distillation. Under the appendix description, the comparison is RL versus RL plus distillation; a reward-matched control is needed to attribute the effect of hints. Section 4 specifies 4 prompts with 8 samples each, whereas Appendix E gives RL 8 prompts with 4 trajectories each and calls the resulting batch size 16, although the product is 32. These discrepancies impede reproduction and interpretation.

Questions for authors
  1. Which exact instance IDs were used for SFT, RL, HG, and Table 1 evaluation? Were any Table 1 instances seen during RL/HG post-training? If so, can the authors provide independent held-out results for those two benchmarks?

  2. What is the full SecureVibe_hg objective, and does it include the same task reward as SecureVibe_rl? Please provide a reward-matched comparison isolating the hint/distillation contribution and reconcile the prompt counts, group sizes, and batch sizes in Section 4 and Appendix E.

1 comment

Thank you for the careful reading and constructive suggestions. We are encouraged that the reviewer recognizes the practical importance of implicit security requirements in repository-level coding, and finds our BaxBench and unseen-CWE SusVibes results meaningful. Your questions on the data splits and the training objective helped us make these parts of the paper clearer, and we address them below.

W1 & Q1: The held-out status of Table 1 is unclear for the post-trained models.

A: The Table 1 instances were never seen during any training stage, including RL and HG post-training. Appendix C describes post-training for the cross-dataset models in Table 2, not Table 1; we revised Appendix C to make this explicit. We update the instance ID list in https://anonymous.4open.science/r/safevibe-anonymous-87FD/data/held-out-training.240.txt

  • Table 1 (in-domain held-out). All training stages (SFT trajectory sampling, GRPO, and OPSD) use 100 PatchEval-Gen and 140 AutoBaxBench instances. The remaining 100 PatchEval-Gen and 420 AutoBaxBench instances are evaluation only. The two splits share no CVE IDs and no web backend scenarios.
  • Table 2 (cross-dataset). Starting from the Table 1 checkpoints, we continue post-training SecureVibe_rl and SecureVibe_hg on the remaining 100 PatchEval-Gen and 420 AutoBaxBench instances, so the Table 2 models have seen all 200 and 560 instances. Evaluation is on SusVibes and BaxBench. AutoBaxBench's 40 backend scenarios do not overlap with BaxBench's 28, and PatchEval-Gen shares no CVE IDs or fix commits with SusVibes.

As noted, the separate unseen-CWE results also hold: on the 78 SusVibes instances with CWEs absent from training, SecPass rises from 7.69% to 19.23%.


W2 & Q2: Training objective between SecureVibe_hg and SecureVibe_rl. RL training hyperparameters.

A: SecureVibe_hg combines the on-policy distillation objective with the same task reward used by SecureVibe_rl. Distillation moves the student's distribution toward the hint-conditioned teacher on the student's own sampled trajectories, and the task reward prevents outcome performance from degrading. We updated Eqn. 2 to include the reward term, so Section 3.3 now matches Appendix E.

Because both models share the same task reward, SecureVibe_rl vs. SecureVibe_hg is the reward-matched comparison you asked for: the only difference is the hint-guided distillation term. The distillation term increases explicit security planning from 20.0% to 28.9% of trajectories (Figure 5). Configuration, now consistent across sections:

  • RL batch: 4 prompts × 8 samples = 32. Appendix E's "8 prompts × 4, batch size 16" was incorrect and has been fixed.
  • Context length: 128K is the inference limit (Section 4); 64K is the training sequence limit (Appendix E). Both sections now say so.
Reviewer_kQF8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper three failure modes that models make when genearting vulnerable code. Models do not plan, lack general code skill, and do not test properly. The paper finds, that targeted training of these capabilities rather than direct secure coding training, improves downstream performance and generalizes to other unseen datasets. The findings are interesting, as they support the notion that there is some transfer of procedure or that the models are memorizing a repeatable bug pattern from the traces.

Critical strengths

The authors identify axes of performance that influence secure code generation. This is interesting, since it identifies meta-structural capabilities for secure code generation that, in theory, will generalize better than just task-specific training. The paper shows a strong uplift in the performance of the Qwen model. Training on the proposed tasks improves performance significantly over the base on the relevant benchmarks. Furthermore, the rigor of the training evaluation is strong.

Critical weaknesses

The authors train on trajectories generated from the benchmarks they are testing on. This might cause data leakage due to structural features of the benchmark. The authors show transfer to other benchmarks, but the overlap of the evaluated datasets is large. It would be good to show performance on more varied benchmarks, and to also analyze performance on general purpose reasoning benchmarks to analyze loss of generalizability. The paper is well laid out and easy to understand. The core claims are not very surprising, and are lacking a bit in the rigor of evaluation. The paper would benefit from being able to defend and or isolate the main capabilities that it seeks to distill from the teacher model.

Questions for authors
  • Do the training and evaluation datasets overlap? If so, how much; if not, then please elaborate.
  • How much of the performance is from general distribution shift towards a stronger model and how much is from learning better procedures? One vanilla distillation baseline would be a strong control and strengthen the core claim.
1 comment

We sincerely appreciate Reviewer kQF8's positive feedback and insightful questions. We are glad the reviewer finds that the capabilities we target (security planning, coding, and testing) can generalize beyond task-specific training, and that our training evaluation is rigorous. We have revised the manuscript and address the remaining concerns as below

W1.1 & Q1: Do the training and evaluation datasets overlap?

No task instance overlaps between training and evaluation. We use two evaluation protocols, and neither evaluates on instances seen during training:

  • In-domain (Table 1). All training stages (SFT trajectory sampling, GRPO, and OPSD) use 100 PatchEval-Gen and 140 AutoBaxBench instances. The remaining 100 PatchEval-Gen and 420 AutoBaxBench instances are held out for evaluation. The two splits share no CVE IDs and no web backend scenarios.
  • Cross-dataset (Table 2). AutoBaxBench's 40 backend scenarios do not overlap with BaxBench's 28, and PatchEval-Gen shares no CVE IDs or fix commits with SusVibes.

There are some shared CWEs in training and evaluation. To test generalization beyond them, Section 5.3 (Table 6) evaluates on 78 SusVibes and 140 BaxBench instances whose CWEs never appear in training. There, our models gain up to 17.95 FuncPass and 11.54 SecPass points. We made both protocols explicit in Section 4.


W1.2 & Q2: How much of the performance is from general distribution shift towards a stronger model and how much is from learning better procedures? One vanilla distillation baseline would be a strong control and strengthen the core claim.

A2: Our baselines already include distillation controls, and we added the vanilla distillation baseline you suggested. All SFT variants are fine-tuned on trajectories from the same teacher (MiniMax M2.7) and differ only in task organization:

  • Functionality-Only SFT (new, the vanilla distillation baseline): generic coding trajectories only.
  • Security-Only SFT: security-focused coding, Simple Mixture SFT: functionality-focused and security-focused coding, SecureVibe_base: four task trajectories

Qwen 3.5 35B + Func-Only SFT distillation improves FuncPass but lowers SecPass on both SusVibes and BaxBench. SecureVibe_base also outperforms Security-Only and Simple Mixture SFT (Table 1). So imitating a stronger teacher alone does not produce the security gains; the planning and testing tasks do. Post-training adds further evidence. SecureVibe_rl and SecureVibe_hg improve over SecureVibe_base.

SusVibesBaxBench
FuncPassSecPassFuncPassSecPass
Qwen 3.5 35B28.1410.5732.9119.71
+ Func-Only SFT39.786.4540.3120.41
SECUREVIBE_base41.7613.6236.4825.34
SECUREVIBE_rl41.7613.2641.0726.62
SECUREVIBE_hg39.9614.1638.2623.64

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.