acceptodds
Under review as a conference paper at ICLR 2027

EvoSecCoder: Adversarial Co-Evolution for Secure Code Generation

Abstract

Large language models increasingly excel at code generation, but still generate vulnerable code. To generate more secure code, models need to learn to follow secure examples and avoid insecure patterns without losing functionality. Models can learn from secure and insecure examples drawn from a fixed dataset or generated by the model itself. However, fixed examples become less useful as the model learns, while the model's own functional insecure examples become rarer as its code becomes more secure. Both sources therefore provide less training signal as the model improves. We present EvoSecCoder, an adversarial co-evolution framework that jointly evolves secure and insecure models from the same base model while preserving functionality. The two models co-evolve by supplying adversarial samples to each other. In each evolution round, both models solve the same tasks, and executable tests verify their programs. Functional but insecure programs challenge the secure model, while secure programs guide the insecure model toward new insecure alternatives. Across five open models, with training rounds selected on held-out validation CWEs, EvoSecCoder improves the mean SecCodeBench correct-and-secure rate on the remaining test CWEs from 17.4% to 23.3%, while mean EvalPlus functional correctness changes from 65.6% to 64.9%. Removing or freezing the insecure model reduces the mean correct-and-secure rate to 21.1% or 21.8%, respectively; these ablations support the value of a dedicated, co-evolving insecure model beyond additional sampling alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Secure Code Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TJpaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a post-training method to improve the security of code generated by LLMs. The method co-trains two adapters, one for secure code generation, and one for insecure code generation. First, using sec-insec code data pairs from prior datasets, these two adapters are cold-started. Then, a two step loop is repeated: 1) sec-insec pairs are generated from the two adapters and preference learning is applied (sec preference for the sec adapter, and vice-versa); 2) GRPO RL is done on each adapter separately. The evaluation across 5 models shows some improvements over the base model and prior methods.

Critical strengths
  • Important topic as coding models and agents become ever more prevalent.
  • Interesting idea to maintain an opposite policy to aid the training of the target policy.
Critical weaknesses
  • I am not convinced about the effectiveness of the full method based on the results. Note that here I am referring to the, in my view, main metric: C&S. From the five models used in the evaluation, only in the case of Granite 4.1 is the full method clearly above any alternative. For Qwen 2.5 Coder 7B, an ablated version of the method is better + the improvement is minor compared to the best prior method; for Qwen 2.5 Coder 14B, even two ablated versions are better; for Phi-4-Mini, the full method does not improve over the base, but ablated versions do and prior work is better; and for CodeLlama, the improvement over SafeCoder is minor despite the large amount of additional data, complexity, and training.
  • The used models and resulting coding settings are somewhat outdated. Current coding is mostly focused on agentic coding driven by reasoning LLMs. While training frontier agentic models is admittedly not in scope for academic/research works; I do believe that there are already smaller scale capable reasoning code models (such as Qwen 3 or 3.5 series, for instance), that could be trained and evaluated in some simpler harnesses, making a more compelling case for any proposed method that its insights could extrapolate to frontier, or at least, practically relevant coding tasks and usecases.
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

[W1] Performance Gains With Limited Data

We thank the reviewer for their thoughtful feedback. We address the main concerns below.

A. Dataset

Our method uses 1,136 SafeCoder pairs and 885 SeCodePLT items for bootstrap, and 119 CWEval items for replay. After deduplicating the SeCodePLT items reused across bootstrap and replay, there are 2,140 original training items. In comparison, SafeCoder uses 46,719 general instructions and 1,141 security pairs, while DiSCo-LPO uses 9,489 pairs. TSP's actual training pairs contain 360 unique original instructions, and RESCUE requires no additional training. These counts refer to original items, excluding generated candidates and repeated training. Our method continually generates and refreshes training examples from this limited set of original tasks.

B. Performance

We agree that Full does not always achieve the highest score on each individual backbone. However, according to Table 1 in the main paper, Full's five-model mean test C&S is 23.3%, 3.3pp higher than the strongest external baseline, SafeCoder, at 20.0%. EvalPlus functionality is also higher for Full: 64.9% versus SafeCoder's 62.3%. DiSCo-LPO achieves 19.1% mean C&S and 60.5% EvalPlus. Thus, even these baselines with relatively close C&S results incur greater functionality losses than Full, and Full's advantage is clearer when both metrics are considered together.

To further substantiate our performance gains from limited original training data, we conducted additional experiments extending the training of both baselines and ablations. The results show that these alternatives did not reach Full's mean C&S even with additional training, supporting the value of continually generating and refreshing verified examples through co-evolution.

We extended the baselines whose measured training costs were at least 10% below Full's to bring their training time closer to Full's, retaining their data, learning rates, and loss settings. This covered DiSCo-LPO on all five base models and SafeCoder on Qwen-14B and Phi Mini. TSP already had similar training time and was not extended. The extensions used seed 42; Full reports the main paper's three-seed means. Cells report test C&S / EvalPlus functionality (%); N/A denotes a setting not extended.

ModelFullExtended DiSCo-LPOExtended SafeCoder
Granite 8B24.64 / 73.8011.59 / 73.06N/A
Qwen-7B27.54 / 70.8514.49 / 62.73N/A
Qwen-14B33.82 / 76.8815.94 / 69.0027.54 / 72.88
Phi Mini14.49 / 58.672.90 / 40.7710.14 / 58.49
CodeLlama 7B15.94 / 44.224.35 / 24.17N/A

Test C&S decreased relative to the original baseline results in all seven extended settings, with substantial functionality losses in some cases. Thus, simply increasing training time on their fixed data did not improve these baselines or close the gap to Full. In contrast, our method obtains gains by continually generating verified training examples from a limited set of original tasks.

We also extended No-B to R28 and Frozen-B to R15, approximately matching Full R10's total training budget based on estimated FLOPs. These experiments retained the original settings and seed-matched bootstrap, used three seeds (42/43/44), and followed the main paper's round-selection procedure. Cells report test C&S / EvalPlus functionality (%), with selected rounds in parentheses.

ModelFullNo-B (up to R28)Frozen-B (up to R15)
Granite 8B24.64 / 73.80 (R2)18.84 / 72.57 (R1)20.77 / 72.94 (R3)
Qwen-7B27.54 / 70.85 (R9)25.12 / 70.66 (R18)24.15 / 72.88 (R9)
Qwen-14B33.82 / 76.88 (R1)30.92 / 78.29 (R15)38.16 / 76.81 (R1)
Phi Mini14.49 / 58.67 (R3)14.49 / 58.30 (R3)14.98 / 57.63 (R5)
CodeLlama 7B15.94 / 44.22 (R7)13.53 / 46.31 (R7)11.11 / 43.85 (R2)
Model mean23.29 / 64.8820.58 / 65.2321.84 / 64.82

Even with these additional budgets, Full retained mean C&S advantages of 2.71pp over No-B and 1.45pp over Frozen-B, with similar EvalPlus functionality. Only No-B on Qwen-7B and Qwen-14B selected rounds beyond R10, and both remained below Full's C&S on those backbones. Although individual ablations still perform better on some backbones, longer training did not eliminate Full's average advantage. These results further support our central contribution: obtaining performance gains from limited original data through co-evolution, which the evaluated alternatives did not match on average by extending training alone.

AuthorsOpenReview ↗

[W2] Applicability To Reasoning Models

To examine applicability to reasoning models, we conducted additional experiments with Qwen3.5-4B in thinking-ON mode using three seeds (42/43/44).

To train reasoning, we added an auxiliary reasoning loss (weight 0.05) to both training stages and trained for up to five rounds. Due to training-time constraints, thinking and final code were each limited to 1,024 tokens. We followed the round-selection procedure in the main paper. Full reports a three-seed mean, while Base and RESCUE are existing single-evaluation results under the same thinking/code budgets. SafeCoder, DiSCo-LPO, and similar methods were excluded because their original SFT or preference-optimization training data do not include reasoning.

Model/methodTest C&SEvalPlus functionality
Qwen3.5-4B Base (thinking ON)14.49%57.20%
Qwen3.5-4B Full (thinking ON, R5)21.26%75.83%
Qwen3.5-4B RESCUE (thinking ON)13.04%54.61%

Compared with thinking-ON Base, Full improved mean test C&S from 14.49% to 21.26% and EvalPlus functionality from 57.20% to 75.83%. Our method improved both C&S and functionality in this setting, demonstrating its applicability to a reasoning model. In particular, compared with the average changes from Base across the original five backbones, this reasoning model showed more pronounced improvements in both C&S and functionality.

Future work will evaluate our method on a broader range of reasoning models using agentic coding benchmarks and suitable evaluation harnesses.

Thank you for the rebuttal. I believe the method is showing promise in timelier settings. However, these results are still preliminary. A paper focusing more on these results and settings would be a better fit for the conference. Therefore, I maintain my score.

Thank you for your response and thoughtful feedback! We’ll take your suggestions into account as we continue to improve the paper.

Reviewer_AEXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EvoSecCoder, a secure code generation framework that jointly trains a secure model and an insecure model using LoRA adapters. Both models generate solutions for the same programming tasks. Functional and security tests are used to construct preference pairs, which are optimized in opposite directions. The authors further apply localized GRPO to update both models. Experiments on five code models show an improvement in mean correct-and-secure rate from 17.4% to 23.3% on SecCodeBench, with a small average decrease in EvalPlus functional correctness.

Critical strengths
  1. Maintaining a separate evolving insecure model is interesting and addresses a reasonable limitation of fixed negative examples and single-policy self-play.

  2. The evaluation covers five backbone models and includes ablations of the insecure model, preference learning, and GRPO. The additional analysis of negative sample generation helps explain the training process.

Critical weaknesses
  1. Appendix A.2 confirms that all 98 SecCodeBench tasks were used for hyperparameter tuning before the subsequent 29/69 validation and test split. The average improvement over the strongest external baseline is only 3.3 percentage points, with marginal gains on several models. Comparisons with recent methods such as ProSec [1], PurpCode [2], and DeepGuard [3] are also missing.

  2. The insecure policy generates more valid negatives on previously seen tasks, but their difficulty for the secure policy remains unverified, and held-out negative generation deteriorates. Removing the insecure policy outperforms the full method on Qwen-7B, while freezing it performs better on Qwen-14B. The lack of a compute-matched single-policy baseline further limits the evidence for co-evolution.

  3. Both localized objectives operate on differing tokens rather than verified vulnerability locations. Independently generated implementations can differ substantially in structure, so the training signal may emphasize unrelated code changes rather than security-critical edits.

  4. The reward treats passing the fixed functional tests as sufficient evidence of correctness, leaving untested behavior unaccounted for. SecCodeBench functionality drops substantially on several models despite higher security scores, including a decline from 53.6% to 37.7% on Qwen-7B.

[1] Xu et al. ProSec: Fortifying Code LLMs with Proactive Security Alignment. ICML 2025.

[2] Liu et al. PurpCode: Reasoning for Safer Code Generation. NeurIPS 2025.

[3] Huang et al. DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation. ACL 2026.

Questions for authors
  1. How does the paper assess the reliability and coverage of these fixed tests, particularly the possibility of overfitting to test outcomes across training rounds?

  2. The localized preference loss operates on tokens that differ between two implementations. How are pairs with substantially different code structures handled? Is there any limit on the proportion of differing tokens, and how does this proportion affect training?

  3. Could the authors report the full benchmark performance of the secure model immediately after bootstrap and compare it with SafeCoder?

  4. Why were more recent secure code generation methods not included in the evaluation?

8 comments
AuthorsOpenReview ↗

[W1 and Q4] Evaluation Independence, Performance, And Baselines

We thank the reviewer for their thoughtful feedback. We address the main concerns below.

A. Hyperparameter And Checkpoint Selection

As disclosed in the Appendix, initial hyperparameter tuning used all 98 SecCodeBench tasks. To examine this concern, we compared hyperparameter settings again using only the 29 validation tasks, and the same configuration was selected for all five models.

B. Improvement Magnitude And Functionality

According to Table 1 in the main paper, Full's five-model mean test C&S is 23.3%, 3.3pp higher than the strongest external baseline, SafeCoder, at 20.0%. Its EvalPlus functionality is also higher: 64.9% versus SafeCoder's 62.3%. DiSCo-LPO is lower than Full on both metrics, with 19.1% C&S and 60.5% EvalPlus. In contrast, TSP and RESCUE, whose EvalPlus results are higher than Full's, achieve only 17.4% and 19.1% C&S, respectively. Thus, higher functionality did not translate into better correct-and-secure code generation.

These improvements also did not come from adding large amounts of external training data. Our method uses 2,140 deduplicated original training items, whereas SafeCoder uses 46,719 general instructions and 1,141 security pairs, and DiSCo-LPO uses 9,489 pairs. We verify generated code through execution and continually refresh training examples from a limited set of original tasks. Our contribution is better average security and functionality with limited original training data.

C. Additional Baseline Evaluation

We provide the requested additional comparisons with ProSec, DeepGuard, and PurpCode. Due to time constraints during the review period, additional baselines that we trained directly used only seed 42. For ProSec, we directly evaluated the authors' released checkpoints for Phi Mini and CodeLlama 7B, and trained Granite and the two Qwen models with original methods using the released data.

Training settings were based on configurations released in the authors' code or papers. DeepGuard included the security modules and token prior from the original implementation. Because PurpCode requires long training, we used a stratified subset of 4,096 questions from the public data and evaluated a reward-based RL configuration using CodeQL instead of AWS CodeGuru. Cells report SecCodeBench test C&S / EvalPlus functionality (%).

ModelBaseFullProSecDeepGuardPurpCode
Granite 8B15.94 / 74.5424.64 / 73.8013.04 / 72.8820.29 / 74.3513.04 / 74.35
Qwen-7B18.84 / 74.3527.54 / 70.8521.74 / 73.8023.19 / 66.9720.29 / 74.72
Qwen-14B23.19 / 78.6033.82 / 76.8826.09 / 77.1226.09 / 77.1223.19 / 78.04
Phi Mini14.49 / 59.7814.49 / 58.6713.04 / 61.4411.59 / 59.0411.59 / 61.07
CodeLlama 7B14.49 / 40.7715.94 / 44.222.90 / 34.8710.14 / 40.417.25 / 40.22

Full's five-model mean test C&S was 23.29%, exceeding ProSec (15.36%), DeepGuard (18.26%), and PurpCode (15.07%) by 7.92pp, 5.02pp, and 8.21pp, respectively. Full's C&S exceeded these additional baselines on all five models, retaining its advantage in correct-and-secure code generation in comparisons with recent methods.

AuthorsOpenReview ↗

[W2] Negative Difficulty And Compute-Matched Comparisons

A. Difficulty Of Valid Negatives

We considered that, as training progresses and secure policy A increasingly favors secure code, the log-probabilities it assigns to insecure samples generally decrease. To avoid repeatedly training on negatives that secure-policy A can already distinguish easily, we prioritized generated candidates to which secure-policy A assigns relatively high log-probabilities, aiming to obtain useful negatives for training.

B. Compute-Matched Single-Policy Baseline

Using the same model and seed-matched bootstrap, we retained the existing settings and extended No-B to R28 and Frozen-B to R15 to approximately match Full R10's training budget based on estimated FLOPs.

We used three seeds (42/43/44) and followed the round-selection procedure in the main paper. Cells report C&S on the 69 test tasks / functionality on all 542 EvalPlus tasks (%), with selected rounds in parentheses.

ModelFullNo-B (up to R28)Frozen-B (up to R15)
Granite 8B24.64 / 73.80 (R2)18.84 / 72.57 (R1)20.77 / 72.94 (R3)
Qwen-7B27.54 / 70.85 (R9)25.12 / 70.66 (R18)24.15 / 72.88 (R9)
Qwen-14B33.82 / 76.88 (R1)30.92 / 78.29 (R15)38.16 / 76.81 (R1)
Phi Mini14.49 / 58.67 (R3)14.49 / 58.30 (R3)14.98 / 57.63 (R5)
CodeLlama 7B15.94 / 44.22 (R7)13.53 / 46.31 (R7)11.11 / 43.85 (R2)
Model mean23.29 / 64.8820.58 / 65.2321.84 / 64.82

Full's mean test C&S was 23.29%, exceeding extended No-B (20.58%) by 2.71pp and Frozen-B (21.84%) by 1.45pp. EvalPlus results were similar at 64.88%, 65.23%, and 64.82%, respectively. Full's mean C&S advantage persisted even after additional training budgets were allocated. Rounds added after R10 were selected only for No-B on Qwen-7B (R18) and Qwen-14B (R15). Even in these cases, test C&S was 25.12% and 30.92%, respectively, below Full's results selected at earlier rounds: Qwen-7B R9 (27.54%) and Qwen-14B R1 (33.82%).

AuthorsOpenReview ↗

[W3] Security Relevance Of Localization

A. Security Relevance Of Training Code Pairs

To examine whether actual code differences relate to security fixes, we collected a sample from the code pairs actually used in Phase 1, stratified by model, seed, and training period, and then randomly selected 50 pairs from that sample. Two authors reviewed the differences against each problem and its target CWE and cross-checked their judgments. The results are as follows.

JudgmentPairsPercentage
Localized security fix1734.0%
Security mechanism and supporting changes2346.0%
Security fix mixed with substantial unrelated changes12.0%
Broad unrelated rewrite00.0%
No clear target-CWE difference identified918.0%
Total50100%

For 40 of the 50 pairs (80%), the differences concerned security fixes or supporting changes. One pair contained substantial unrelated changes, nine had no clear security difference, and none was classified as a broad unrelated rewrite. This shows that differences between independently generated programs can also be useful for security training.

B. Localization ON/OFF Experiments

To examine whether localization adversely affects model performance, we removed localization from both training stages while keeping the other settings unchanged. Each setting's round was selected using the procedure in the main paper. The table reports three-seed mean test C&S / EvalPlus functionality (%).

ModelLocalization ONLocalization OFF
Granite 8B24.64 / 73.80 (R2)18.84 / 72.08 (R9)
Qwen-7B27.54 / 70.85 (R9)25.12 / 70.42 (R9)
Qwen-14B33.82 / 76.88 (R1)36.23 / 76.94 (R1)
Phi Mini14.49 / 58.67 (R3)13.04 / 58.06 (R6)
CodeLlama 7B15.94 / 44.22 (R7)12.08 / 44.83 (R5)

Mean test C&S was 23.29% with localization ON and 21.06% with it OFF, an advantage of 2.22pp for ON. ON performed better on four of the five models, with Qwen-14B as the exception. In this comparison, localization therefore contributed to improved C&S rather than harming average performance.

Reviewer_r4x1OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This work proposes EvoSecCoder, a framework that co-evolves two code LLMs: one trained to generate more secure code and the other to generate more vulnerable code. EvoSecCoder first initializes both models using a combination of SFT and SimPO losses. It then co-evolves the models through multiple rounds of preference optimization and localized GRPO, using samples generated by the updated policies to guide subsequent training rounds. Experiments across five LLMs show that EvoSecCoder achieves competitive or better functional correctness and security rates on SecCodeBench. However, four of the five models exhibit reduced performance on EvalPlus after training.

Critical strengths
  1. The paper introduces an insecure policy to generate more useful insecure samples for training.
  2. The method is evaluated on five code LLMs, demonstrating its applicability across different models.
  3. The proposed method improves correctness and security rates on SecCodeBench (C&S).
Critical weaknesses
  1. While reading the paper, I got the impression that it relies heavily on auto-research, which is later validated in the AI use statement. Figures 1 and 2 appear to be AI-generated, especially Figure 1, whose quality is low.
  2. My main concern is the complexity of the proposed method. It requires an initial warm-up stage followed by multiple rounds of preference optimization and RL, combining SFT, preference optimization, and RL into a complicated training pipeline. According to Appendix A.3, the number of rounds can reach 7–9, which significantly limits the practicality of the method.
  3. Despite this complexity, the performance improvement is relatively marginal (+3.3 points in Figure 3), especially considering that most evaluated baselines require only a single round or even no training. The proposed method appears to incur substantially higher computational costs, such as GPU hours, than the baselines. This raises concerns about whether the modest gains justify the additional computational overhead.
  4. The ablation study in Table 1 also raises concerns about fairness in comparison. For example, the GRPO-only variant likely requires substantially fewer GPU hours than the full method. Therefore, comparisons with baselines and ablation variants under comparable computational budgets are necessary to better assess the effectiveness of the proposed method.
  5. The method also appears to significantly compromise model utility. According to Table 1, the functional correctness rate on SecCodeBench drops by up to 15.9 points. I suspect that the localized updates may contribute substantially to this degradation, which warrants further investigation.
  6. The paper is generally difficult to follow, particularly in the method and experimental sections. Several important implementation details are missing, such as the GRPO hyperparameter settings.
Questions for authors

See weaknesses, especially the computational cost of the proposed method.

7 comments
AuthorsOpenReview ↗

[W1] AI Use And Figure Preparation

We thank the reviewer for their thoughtful feedback. We address the main concerns below.

A. Auto-Research

First, we disagree with the reviewer's assessment that our work relies heavily on auto-research. We did not use auto-research in this work. The authors proposed and developed all core ideas and methodology. AI was used to assist with literature searches, automate experimental script execution, support writing and revision, and prepare initial figure drafts. The authors reviewed and verified all resulting outputs.

For example, we had planned to use GRPO at the beginning of the project. While using AI to survey recent RLVR research, we identified prior work on localized training and incorporated those insights into our methodology. This was led by the authors, with AI serving as an assistive tool, and should be distinguished from auto-research in which AI conducts research autonomously.

The basis for characterizing the overall research as heavily reliant on AI is unclear, given that the AI Use Statement specifies both its uses and our verification procedures. As this assessment concerns the authors' originality and intellectual contributions, we request specific supporting evidence.

B. Figure

We would also like to clarify how the figures were prepared. Our AI Use Statement discloses that we used AI to create initial figure drafts, which the authors subsequently revised and verified. Thus, the observation that Figure 1 appears AI-generated is consistent with our disclosed use of AI. We are, of course, willing to improve Figure 1 if its quality needs improvement. Figure 2 was created directly by the authors using Figma Design product. We also consider the use of AI for an initial draft and the quality of the final figure to be separate issues. We would welcome specific feedback on readability, visual composition, or communication of information so that we can address it.

We would therefore like to ask the reviewer the following two questions:

Considering the scope of AI use and verification procedures disclosed in the AI Use Statement, what specific evidence led you to conclude that our work relies heavily on auto-research? Could you specify which aspects of Figure 1 you consider low quality?

We believe it is important to disclose AI use and verification procedures transparently. We hope that such disclosure does not lead to misunderstandings about the originality of the work or the authors' contributions. We will address specific issues identified by the reviewer.

AuthorsOpenReview ↗

[W2] Method Complexity And Practicality

As Reviewer TJpa summarized, our procedure can be described as a cold-start stage followed by verifiable-reward learning based on executable tests. In the revised version, we will explain the roles and connections of the stages more concisely and clearly around this core flow.

A. Justification Of Each Component

Our motivation for using multiple training stages is to continually obtain useful training examples from limited data for which both functionality and security can be verified through execution. There are 2,140 deduplicated original training items. The cold-start stage gives the two policies distinct secure/insecure generation directions, and subsequent co-evolution continually generates and refreshes contrasting examples from the evolving policies. Generated code is verified with executable tests and used as training signal.

In the ablations evaluating this design, Full, which combines all components, achieved a five-model mean test C&S of 23.3%, compared with 21.1% for No-B, 21.8% for Frozen-B, 21.2% for preference-only, and 20.0% for GRPO-only. This supports the average effectiveness of the current training pipeline.

B. Practicality Of Training Time

Regarding the number of rounds, training does not necessarily need to continue to the maximum round. Performance may saturate early depending on the model, and the checkpoint with the best validation performance can be selected. The selected rounds indeed varied by model: Granite R2, Qwen-7B R9, Qwen-14B R1, Phi Mini R3, and CodeLlama R7. If needed, early stopping when validation improvement plateaus can reduce training cost, as in standard ML experiments.

Training time is also not impractical compared with other baselines. Moreover, by augmenting diverse data across rounds in a limited-data setting, our method offers performance advantages that other baselines do not reach even with longer training. Please refer to the following response (Weakness 3 & 4) for detailed training-time comparisons and experiments with matched training time supporting this point.

AuthorsOpenReview ↗

[W3 and W4] Computational Cost And Comparison Fairness

We agree that experiments matching training costs were missing. To address concerns about computational cost and comparison fairness, we therefore compared actual training times and allocated additional training budgets to baselines and ablations. In summary, Full(Ours)'s training time was comparable to some baselines, and its mean C&S advantage persisted in comparisons with extended budgets. Its mean C&S advantage also persisted in additional experiments that matched the ablations' training budgets to Full.

A. Original Training Time

We used AMD MI300A APUs (128 GB HBM3, ROCm 6.3). Most settings used one APU, while TSP used two or four depending on the model. APU-hours are the number of devices multiplied by training time.

The table below reports three-seed mean training times. Full includes bootstrap through R10. RESCUE adds security knowledge to the prompt and requires no separate training, so it is excluded from the training-time table.

ModelSafeCoderDiSCo-LPOTSPFull (through R10)
Granite 8B9.22.510.510.0
Qwen-7B6.51.77.16.9
Qwen-14B11.63.630.716.8
Phi Mini5.31.46.57.0
CodeLlama 7B17.02.08.48.6

SafeCoder and TSP had training costs comparable to or higher than Full depending on the LLM, while DiSCo-LPO was less expensive. Optimization of the public implementations and differences in hardware support may affect time, but even accounting for these factors, Full's training cost was not substantially higher than every baseline.

B. Extended Training Of Baselines

We allocated additional training to settings whose training costs were at least 10% lower than Full's, bringing their costs closer to Full's. We trained additional epochs from the existing seed-42 checkpoints while keeping the data, learning rates, and loss settings unchanged. For DiSCo-LPO, we retained the authors' public implementation's sequence of two SFT epochs followed by one LPO epoch. We repeated these two stages three additional times for Granite, Qwen-7B, and CodeLlama, and four additional times for Phi and Qwen-14B. For SafeCoder, we added 0.63 epoch for Phi and 0.90 epoch for Qwen-14B. TSP was excluded because its training time was similar to our method's.

Due to time constraints, the extended baseline experiments used seed 42. Each cell reports test C&S / EvalPlus functionality (%).

ModelFullExtended DiSCo-LPOExtended SafeCoder
Granite 8B24.64 / 73.8011.59 / 73.06N/A
Qwen-7B27.54 / 70.8514.49 / 62.73N/A
Qwen-14B33.82 / 76.8815.94 / 69.0027.54 / 72.88
Phi Mini14.49 / 58.672.90 / 40.7710.14 / 58.49
CodeLlama 7B15.94 / 44.224.35 / 24.17N/A

Compared with the original baseline results in the main paper, test C&S decreased in all seven extended-training settings, and functionality also deteriorated substantially in some cases. These baselines repeatedly use fixed data without generating or augmenting new examples during training, suggesting that simply extending training can lead to overfitting and performance collapse. Even with extended training, these baselines did not match our method, which continually generates diverse training samples.

C. Extended Training Of Ablations

Using the same model and seed-matched bootstrap, we retained the existing settings and extended No-B to R28 and Frozen-B to R15 to approximately match Full R10's training budget based on estimated FLOPs.

We used three seeds (42/43/44) and followed the round-selection procedure in the main paper. Cells report C&S on the 69 test tasks / functionality on all 542 EvalPlus tasks (%), with selected rounds in parentheses.

ModelFullNo-B (up to R28)Frozen-B (up to R15)
Granite 8B24.64 / 73.80 (R2)18.84 / 72.57 (R1)20.77 / 72.94 (R3)
Qwen-7B27.54 / 70.85 (R9)25.12 / 70.66 (R18)24.15 / 72.88 (R9)
Qwen-14B33.82 / 76.88 (R1)30.92 / 78.29 (R15)38.16 / 76.81 (R1)
Phi Mini14.49 / 58.67 (R3)14.49 / 58.30 (R3)14.98 / 57.63 (R5)
CodeLlama 7B15.94 / 44.22 (R7)13.53 / 46.31 (R7)11.11 / 43.85 (R2)
Model mean23.29 / 64.8820.58 / 65.2321.84 / 64.82

Full's mean test C&S was 23.29%, exceeding extended No-B (20.58%) by 2.71pp and Frozen-B (21.84%) by 1.45pp. EvalPlus results were similar at 64.88%, 65.23%, and 64.82%, respectively. Full's mean C&S advantage persisted even after additional training budgets were allocated. Rounds added after R10 were selected only for No-B on Qwen-7B (R18) and Qwen-14B (R15). Even in these cases, test C&S was 25.12% and 30.92%, respectively, below Full's results selected at earlier rounds: Qwen-7B R9 (27.54%) and Qwen-14B R1 (33.82%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.