acceptodds
Under review as a conference paper at ICLR 2027

SRCode: Learning to Generate Secure Code via Token-Level Rewards

Abstract

Large language models (LLMs) have demonstrated strong capabilities in code generation, yet they remain prone to producing security vulnerabilities. Existing approaches commonly suffer from two key limitations: the scarcity of high-quality security data and coarse-grained reinforcement learning reward signals. To address these challenges, we propose Vul2Safe, a new secure code generation framework that leverages LLM self-reflection to construct high-confidence repair pairs from real-world vulnerabilities, and further generates diverse implicit prompts to build the PrimeVul+ dataset. Meanwhile, we introduce SRCode, a novel training framework that pioneers the use of token-level rewards in reinforcement learning for code security, which enables the model to continuously attend to and reinforce the critical fine-grained security patterns during training. Compared with traditional instance-level reward schemes, our approach allows for more precise optimization of local security implementations. Extensive experiments show that PrimeVul+ and SRCode substantially reduce security vulnerabilities in generated code while improving overall code quality across multiple benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model post-training, reinforcement learning for language models, token-level rewards, code language models, secure code generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ncGvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an SFT + RL based method to improve LLMs in secure code generation. First, in the SFT stage, the method trains on examples of vulnerability detection and fixing. Then, in the RL stage, secure code generation is trained with PPO, where the reward consists of token-level positive reward for each "secure" token, and sample-level negative reward for each sample that is detected as vulnerable. For calculating these rewards, a larger "teacher" model is employed to detect secure spans and judge vulnerability. Results on secure code generation benchmarks show clear improvements.

Critical strengths
  • Important topic.
  • Interesting and effective approach to construct an SFT + RL curriculum.
  • Promising results on the tested benchmarks and models.
Critical weaknesses
  • I have some concerns about the method. It relies on a "teacher" model that is able to detect secure spans in the code and also to judge if the final code snippet is secure or not. I do believe that this works well in the examined settings, which is relatively outdated (more on this in the next point), but I have some doubts over how well this method extrapolates towards the frontier as coding tasks become more complex and harder for any model to judge.
  • The finetuned models, used benchmarks, and overall setting is somewhat outdated. Current coding workloads use mostly agentic setups with often larger, or at least more capable, reasoning models as backbones. This research however targets still single-turn and pure LLM-based, tightly scoped coding tasks. It is therefore unclear how relevant this method is to the current coding frontier, and if any components of the method would be useful to improve current models. While I understand that in part (or largely) this is also due to the fact that working on larger/more capable models is prohibitively expensive for academics. However, there are still smaller-scale LLMs that are excellent at agentic coding, which could come at comparable training budgets as the models used in the paper (e.g., smaller models in the latest Qwen reasoning series). Also, even the specific field of secure code generation is already moving further. For instance Wang et al. (2026), uses the more advanced benchmarks of BaxBench for secure coding and LiveCodeBench for functionality testing.
  • I also have some concerns about the comparison to SafeCoder. SafeCoder is highly sensitive to the specific set of CWEs that are contained in the training data. As the training data of the presented method and that of SafeCoder differ, I am wondering how much of the resulting differences due to the training data being different. Could you report the CWEval results of both your method and that of SafeCoder reduced only to the CWEs that are contained in the SafeCoder dataset? Or could you examine constructing an extension to SafeCoder's dataset using your dataset and then training with SafeCoder's training method?
  • Further, I am a bit surprised, and somewhat sceptical about the strong functionality preservation (even improvement) results. Could you discuss in more detail how this is possible? In my experience, most recent open-weight models are so well trained that it is incredibly hard to not hurt their base capabilities without doing some delicate additional training aimed at preserving these capabilities. Perhaps to this point it would also be important to evaluate the models on some harder tasks, e.g., LiveCodeBench.
  • Finally, I have some points about presentation. The paper is in parts hard to follow, because the "teacher" is never well defined or presented.

References

Wang et al. SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization. arXiv 2026.

Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ncGv (Part 1 / 5)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are encouraged by your positive recognition of the importance of our research topic, our SFT–RL methodological design, and the promising empirical results.

We hope these clarifications and new results will address your remaining concerns and we respectfully ask if you're comfortable to reconsider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


W1 & W2: Generalization to More Challenging Coding Tasks and Newer Models.

Concern: The reliance on teacher-generated security judgments may limit scalability to increasingly complex coding tasks. Moreover, the original evaluation focuses on relatively constrained, single-turn tasks and older model architectures, leaving its relevance to contemporary coding workloads unclear.

A1–A2: We appreciate the reviewer for highlighting the importance of evaluating SRCode beyond its original experimental setting. To address these concerns, we conducted two additional sets of experiments: (i) evaluations on more challenging security and coding benchmarks, and (ii) experiments with a newer-generation Qwen model.

(1) Evaluation on more challenging security and coding benchmarks.

We additionally evaluated SRCode on BaxBench and A.S.E. to assess security-oriented coding performance, alongside LiveCodeBench-v6-new to evaluate general code generation capabilities.

R3-Table 1: Performance on additional security-oriented and general coding benchmarks.

ModelsBaxBenchA.S.EA.S.EA.S.ELiveCodeBench-v6-new
Correct & SecureSecurityQualityOverallPass@1
Base9.915.353.225.318.9
SafeCoder13.223.437.132.714.9
SecCoderX16.821.855.637.320.6
SRCode23.430.657.345.621.1

As shown in R3-Table 1, SRCode consistently achieves the best performance across all five metrics. On BaxBench, SRCode obtains a Correct & Secure score of 23.4, outperforming SecCoderX by 6.6 percentage points. On A.S.E., SRCode achieves 30.6 in Security and 45.6 in Overall, exceeding the strongest baselines by 7.2 and 8.3 points, respectively. Meanwhile, its LiveCodeBench-v6-new Pass@1 reaches 21.1, compared with 18.9 for the base model and 20.6 for SecCoderX.

These results are encouraging because more challenging coding tasks require the model to balance functional correctness with security constraints across multiple implementation decisions. Although TLR assigns localized rewards during training, its benefits extend beyond the original benchmarks, suggesting that fine-grained security reinforcement can improve coding behavior in more demanding evaluation settings.

AuthorsOpenReview ↗

Response to Reviewer ncGv (Part 2 / 5)

W1 & W2 (Continued from Part 1):

A1-A2: We further present two representative case studies.

Case Study 1: Balancing Functionality and Path Security in BaxBench.

We further examine the FileSearch scenario in BaxBench, which requires implementing a file-search API within /data while supporting legitimate subdirectory requests and preventing directory traversal. SRCode successfully balances functional correctness and security by normalizing requested paths and validating their containment within the permitted directory. In contrast, SafeCoder sacrifices functionality for security by rejecting legitimate requests such as search_dir="/songs", while SecCoderX preserves functionality but introduces a security vulnerability by incorrectly using string-prefix matching to validate directory boundaries. The Base model similarly preserves normal search behavior but lacks sufficient path-boundary validation, allowing directory traversal. These results demonstrate that SRCode's advantage lies not merely in imposing stricter security restrictions, but in precisely enforcing security constraints while preserving legitimate functionality.

Case Study 2: Context-Aware Authorization in A.S.E.

We additionally examine the tsinghua_lunary_cve_2024_4146 instance in A.S.E, a repository-level TypeScript code-completion task associated with CWE-639. The task requires completing the masked checkProjectAccess function using repository context that describes an existing organization-level authorization check and indicates the need for a direct account_project lookup. SRCode correctly enforces project-level authorization by verifying the explicit user–project relationship. In contrast, SafeCoder and SecCoderX preserve insecure organization-level checks, allowing users within the same organization to access projects without explicit authorization. The Base model additionally mishandles the SQL query result by converting the returned object directly to a boolean, causing incorrect access decisions. Unlike these implementations, SRCode identifies the security-critical authorization relationship and preserves the intended functionality. This case demonstrates SRCode's ability to incorporate repository-level security semantics rather than relying solely on local input validation or existing implementation patterns.

Together, these case studies provide concrete evidence complementing the quantitative results in R3-Table 1. They demonstrate that SRCode can preserve legitimate functionality while enforcing security constraints, ranging from filesystem boundary validation to repository-level authorization. These findings further support SRCode's generalizability to more challenging and realistic coding tasks.

(2) Evaluation on a newer-generation model.

To further examine the applicability of SRCode to more recent model architectures, we conducted additional experiments using Qwen3.5-9B, a newer-generation model with a parameter scale suitable for controlled training comparisons. We followed the same evaluation protocols as in the original experiments.

R3-Table 2: Performance comparison on Qwen3.5-9B. | Method | CWEval | CWEval | CodeLMSec | CyberSecEval | LiveCodeBench-v6-new | |:---|:---:|:---:|:---:|:---:|:---:| | | FS@1 (C/C++) | FS@1 (Python) | Sec.Rate | Pass.Rate | Pass@1 | | Base | 26.9 | 40.4 | 68.2 | 56.1 | 39.4 | | SafeCoder | 15.4 | 33.8 | 76.7 | 66.4 | 28.6 | | SecCoderX | 28.9 | 42.3 | 72.2 | 62.4 | 34.9 | | SRCode | 32.7 | 44.2 | 79.9 | 67.8 | 36.6 |

SRCode achieves the best performance across all four security-oriented metrics, outperforming SecCoderX by 3.8, 1.9, 7.7, and 5.4 percentage points, respectively. Importantly, SRCode also achieves the highest LiveCodeBench Pass@1 among the security-tuned methods (36.6), narrowing the functionality gap relative to the base model.

However, we observe that security-oriented fine-tuning on general-purpose models leads to a decline in general coding capabilities, unlike the improvements observed with code-specialized models. We discuss this distinction and its implications in greater detail in Part 4.

These findings demonstrate that SRCode remains effective on a newer-generation model rather than relying exclusively on the original Qwen2.5-Coder initialization.

We will incorporate R3-Tables 1–2 into Section 4.2 and Section 4.5, respectively, and provide the corresponding benchmark configurations and model-specific training settings in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer ncGv (Part 3 / 5)

W3: Fair Comparison with SafeCoder under Matched Training Data.

Concern: SafeCoder is sensitive to the CWE distribution of its training data. Differences between SRCode and SafeCoder may therefore reflect differences in training data rather than the effectiveness of the proposed method.

A3: We thank the reviewer for this important observation. To isolate the influence of training data, we conducted an additional data-matched SafeCoder experiment. Following your suggestion, we incorporated PrimeVul+ into SafeCoder's original training data, following its data construction and training procedure, trained SafeCoder using the same base model as in our original experiments, and evaluated the resulting model on CWEval and LiveCodeBench-v6-new.

R3-Table 3: Comparison with SafeCoder augmented with PrimeVul+. | Model | CWEval | CWEval | LiveCodeBench-v6-new | |:---|:---:|:---:|:---:| | | FS@1 (C/C++) | FS@1 (Python) | Pass@1 | | Base | 28.9 | 45.0 | 18.9 | | SafeCoder | 19.2 | 43.5 | 14.9 | | SafeCoder (origin & PrimeVul+) | 26.9 | 43.5 | 13.7 | | SRCode | 38.5 | 54.2 | 21.1 |

As shown in R3-Table 3, augmenting SafeCoder's original training data with PrimeVul+ improves its CWEval FS@1 (C/C++) from 19.2 to 26.9, confirming that the training data distribution can meaningfully influence its security performance. However, SRCode still substantially outperforms SafeCoder (origin & PrimeVul+), achieving 38.5 versus 26.9 on CWEval (C/C++) and 54.2 versus 43.5 on CWEval (Python), corresponding to improvements of 11.6 and 10.7 percentage points, respectively.

Moreover, SafeCoder (origin & PrimeVul+) achieves 13.7 Pass@1 on LiveCodeBench-v6-new, compared with 21.1 for SRCode. This indicates that augmenting SafeCoder's original training data with PrimeVul+ does not reproduce SRCode's improvements in either secure code generation or general coding functionality.

These results directly address the concern about training-data differences. While data composition affects baseline performance, the remaining gap under matched training data provides stronger evidence that SRCode's SFT–RL curriculum and token-level reward optimization contribute beyond the choice of training dataset.

We will incorporate this data-matched comparison into Section 4.2 and Appendix I, explicitly documenting the SafeCoder data construction procedure, matched base model, and training configurations to improve the fairness and reproducibility of the comparison.

Reviewer_kiB7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrimeVul+, a dataset constructed from vulnerable C/C++ functions, and SRCode, an SFT-to-RL framework for secure code generation. SFT uses vulnerability detection and repair tasks. During RL, a teacher identifies secure spans in generated code, and a fixed positive reward budget is assigned to the corresponding policy tokens, alongside global reward terms. Experiments compare security performance, general code generation, reward granularity, teacher scale, and training stability. The external evaluations, controlled reward comparison, and manual audits make the empirical contribution promising. I lean toward acceptance, however, I also highlight that its effectiveness in agentic coding remains untested.

Critical strengths
  • Consistent empirical improvements: SRCode improves security metrics across multiple backbones and external benchmarks.

  • Novel Method: Token-level RL reward in SCG is indeed novel and interesting.

  • Useful controlled ablation: Table 4 compares token-level and uniformly distributed rewards using the same teacher judgments, positive reward budget, global penalties, and PPO settings. This is a much more informative comparison than simply comparing the complete framework with the untuned model.

  • Substantial quality checks: The authors separately audit 500 dataset candidates and 500 teacher outputs, report annotator agreement for the dataset audit, check training–evaluation overlap, and repeat training with three seeds. These checks provide meaningful support for data quality and result stability.

Critical weaknesses
  • The implicit-prompt claim conflicts with the provided example: Section 3.1 says that function-requirement prompts do not reveal the target CWE or repair strategy. However, Listing 7 explicitly includes “CWE ID: CWE-918” and specifies concrete validation steps. It is unclear which prompt format was actually used for RL and whether the claimed realism of the training tasks holds.

  • A central reward component is not specified: Equations (3)–(4) include an auxiliary reward model that produces the terminal reward, but the manuscript does not identify this model, explain its training, or specify its reward scale. Since this signal is combined with the proposed token rewards, it is necessary for reproducing the method and understanding what the policy optimizes.

  • The reward ablation does not fully isolate the value of correct span localization: The matched control distributes positive rewards uniformly across all tokens. Concentrating them on a smaller set also changes reward sparsity and timing under GAE. A control using equally many random or shuffled span positions, or the same total reward at the terminal step, would clarify how much benefit comes specifically from identifying the right security-relevant tokens .

  • No evaluation on agentic coding benchmarks: Security results are limited to CWEval, CodeLMSec, and CyberSecEval's secure-code generation tests; HumanEval Pro and MBPP Pro assess general coding performance. The paper does not evaluate agentic SCG benchmarks such as BaxBench, SusVibes, or A.S.E.. It remains unclear whether the gains from token-level security rewards transfer to repository-level or multi-turn coding workflows, where vulnerabilities can depend on interactions across functions and files. However, given this time of the era, lacking agentic benchmark evaluation in SCG makes the contribution of this paper smaller as it is not certain whether it will transfer to a more realtistic settings in coding. I would raise my score if the authors could provide such evaluation during the rebuttal period

Questions for authors

See weaknesses

6 comments
AuthorsOpenReview ↗

Response to Reviewer kiB7 (Part 1 / 4)

We sincerely appreciate your insightful and constructive feedback. We are delighted by your recognition of the novelty of our method, the consistent empirical improvements, the rigorous data quality checks, and the value of our controlled ablations. Your recognition of these aspects means a lot to us, serving as a beacon of encouragement that motivates us to continue refining and advancing this line of research.

In addition to your acknowledgement of our paper’s main contributions, we have carefully considered your constructive feedback with more experiments. Specifically, we have clarified key implementation details and conducted additional experiments on localization robustness and challenging coding benchmarks.

These findings strongly validate SRCode's robustness to imperfect security supervision and its generalizability to more challenging secure code generation tasks. To fully address your concerns, we provide a more detailed discussion and experiments below.


W1: Clarification of Implicit Prompts Used for RL Training.

Concern: Listing 7 explicitly includes the CWE ID and repair instructions, which appears inconsistent with the claim that implicit prompts do not reveal the target vulnerability category or repair strategy.

A1: We sincerely apologize for the confusion caused by Listing 7 and thank you for bringing this to our attention. The listing inadvertently presents dataset metadata alongside the task description without clearly distinguishing it from the actual model input.

In our implementation, the Task Type and CWE ID fields are used only for dataset organization and sample retrieval and are excluded from the RL training prompts. We additionally filter the task instructions to remove explicit security-related labels, including terms such as "CWE" and "secure," ensuring that the policy does not receive the target vulnerability category or explicit repair guidance through these fields.

To clarify the actual training setting, we provide a representative RL prompt below.

System:

You are a secure code assistant that performs code vulnerability analysis, repair, and secure generation.

Instruction:

Task description as follows:

The function cma_translate_ib processes InfiniBand address information to configure an RDMA device address structure. It takes two input parameters: a pointer to an InfiniBand socket address structure (sib) and a pointer to an RDMA device address structure (dev_addr).

Intended logic:

  1. Verify that both input pointers are valid and non-null. If either pointer is invalid, exit immediately.
  2. Set the destination device type to indicate an InfiniBand device.
  3. Populate the source global identifier (SGID) using the address information from sib.
  4. Configure the partition key (P_Key) by converting its value from network byte order to host byte order.

The function directly modifies the destination structure and does not return a value. Additionally, ensure that the generated code includes the necessary headers, compiles successfully, follows proper C/C++ formatting, and does not contain Markdown code fences.

Importantly, the task instruction describes the intended functionality without explicitly identifying the target CWE or providing a vulnerability-specific repair patch. For example, although it requests valid, non-null pointers, a check such as if (sib == NULL || dev_addr == NULL) return; only verifies nullness. It does not establish whether the referenced memory is accessible or whether a generic socket-address buffer is sufficiently large for an InfiniBand-specific structure. An unchecked cast followed by field access may therefore introduce type-confusion or out-of-bounds access risks, depending on the calling context.

This distinction is important, as functional requirements do not necessarily specify all security constraints. The policy must still learn appropriate defensive implementation patterns rather than reproduce an explicitly provided CWE-specific fix.

We will correct Listing 7 in Appendix H to clearly separate dataset metadata from the actual training input and explicitly document the prompt-filtering procedure.

AuthorsOpenReview ↗

Response to Reviewer kiB7 (Part 2 / 4)

W2: Specification of the Reward Model.

Concern: The reward model used in Equations (3)–(4) is insufficiently specified, particularly its initialization, training procedure, and reward scale.

A2: We thank the reviewer for highlighting this missing implementation detail. The reward model is initialized from Qwen2.5-Coder-7B-Instruct and trained on PrimeVul+ through a two-stage procedure.

First, we perform supervised fine-tuning using Task 1 (vulnerability detection) and Task 2 (vulnerability repair), enabling the model to acquire vulnerability recognition and remediation capabilities. Starting from this SFT checkpoint, we then construct preference pairs using Task 3 (secure code generation) and perform preference-based reward-model training. The resulting model assigns a scalar reward to each generated code sequence, providing sequence-level feedback.

To bound its contribution to the overall RL objective, the reward score is clamped to [-1, 1] before being incorporated into the reward formulation. As specified in Equations (3)–(4), this scalar reward is assigned to the terminal generation step and combined with the global vulnerability penalty and localized token-level rewards.

We will expand Section 3.3 and Appendix C to specify the reward model initialization, two-stage training procedure, preference construction, score clipping, and integration into the PPO objective.


W3: Controlled Evaluation of Secure-Span Localization.

Concern: The matched-budget ablation does not fully distinguish the benefits of correct secure-span localization from those of reward sparsity and timing. A random or shuffled-span control would provide stronger evidence.

A3: Following your suggestion, we conducted an additional controlled perturbation experiment using Qwen2.5-Coder-7B-Instruct under the original training and evaluation settings.

Specifically, we randomly replaced 25%, 50%, and 100% of teacher-identified reward positions with positions sampled outside the original secure spans. We preserved the number of rewarded tokens and the total positive reward budget, while keeping the remaining reward components and PPO settings unchanged.

R2-Table 1: Robustness of TLR under controlled secure-span localization perturbations. | Setting | CWEval | CWEval | CodeLMSec | CyberSecEval | |:---|:---:|:---:|:---:|:---:| | | FS@1 (C/C++) | FS@1 (Python) | Sec.Rate | Pass.Rate | | Original teacher localization | 38.5 | 54.2 | 88.7 | 74.8 | | Uniform reward allocation | 30.8 (-7.7) | 51.2 (-3.0) | 83.9 (-4.8) | 66.8 (-8.0) | | 25% random replacement | 36.5 (-2.0) | 50.1 (-4.1) | 85.5 (-3.2) | 71.2 (-3.6) | | 50% random replacement | 30.8 (-7.7) | 43.8 (-10.4) | 80.2 (-8.5) | 59.0 (-15.8) | | 100% random replacement | 26.9 (-11.6) | 42.4 (-11.8) | 67.2 (-21.5) | 57.9 (-16.9) |

As shown in R2-Table 1, TLR maintains substantial performance under moderate localization noise. With 25% of reward positions corrupted, SRCode still outperforms uniform reward allocation on three of the four metrics, retaining 36.5 CWEval FS@1, 85.5% CodeLMSec security rate, and 71.2% CyberSecEval pass rate. This suggests that imperfect teacher localization does not eliminate the benefits of token-level security rewards.

Performance progressively decreases as the corruption ratio increases. Notably, 100% random replacement performs worse than uniform reward allocation across all four metrics, despite preserving the same reward sparsity and total positive reward budget as the original TLR setting. This demonstrates that the semantic relevance of rewarded positions, rather than reward sparsity alone, is critical to effective security optimization.

These findings complement the matched-budget ablation in Section 4.4, which establishes the advantage of localized rewards over uniform allocation under identical teacher judgments and training settings. The teacher-scale analysis in the same section further shows that reducing secure-span identification accuracy from 90.62% to 81.52% changes downstream performance by at most 1.4 percentage points, providing additional evidence of performance stability under variations in teacher supervision quality.

Together, these experiments support both the effectiveness of accurate security-aware localization and TLR's tolerance to moderate localization errors.

We will incorporate R2-Table 1 and the controlled perturbation analysis into Section 4.4, with additional experimental settings and perturbation details provided in Appendix C.

AuthorsOpenReview ↗

Response to Reviewer kiB7 (Part 3 / 4)

W4: Additional Evaluation on Agentic Security and Challenging Coding Benchmarks.

Concern: The current evaluation focuses on bounded, single-turn secure code generation tasks, leaving the effectiveness of SRCode on more realistic agentic coding benchmarks and challenging functionality evaluations unclear.

A4: Following your suggestion, we conducted additional experiments on BaxBench [1], A.S.E [2], and LiveCodeBench-v6-new [3] to evaluate SRCode in more challenging coding settings. Recent benchmarks, including SusVibes [4], further highlight the growing importance of evaluating security risks in realistic agent-generated code [1-4]. Our additional experiments provide complementary evidence of SRCode's generalizability beyond the original security benchmarks.

R2-Table 2: Additional evaluation on agentic security and challenging coding benchmarks. | Models | BaxBench | A.S.E | A.S.E | A.S.E | LiveCodeBench-v6-new | |:---|:---:|:---:|:---:|:---:|:---:| | | Correct & Secure | Security | Quality | Overall | Pass@1 | | Base | 9.9 | 15.3 | 53.2 | 25.3 | 18.9 | | SafeCoder | 13.2 | 23.4 | 37.1 | 32.7 | 14.9 | | SecCoderX | 16.8 | 21.8 | 55.6 | 37.3 | 20.6 | | SRCode | 23.4 | 30.6 | 57.3 | 45.6 | 21.1 |

(1) Generalization to more realistic security-oriented coding tasks.

SRCode achieves the best performance on both BaxBench and A.S.E. On BaxBench, its Correct & Secure score reaches 23.4, outperforming SecCoderX by 6.6 percentage points. On A.S.E., SRCode achieves 30.6 in Security and 45.6 in Overall, exceeding the strongest baseline by 7.2 and 8.3 points, respectively.

These improvements are particularly relevant because vulnerabilities in more complex coding settings may arise from interactions among multiple implementation decisions rather than isolated statements. Although TLR provides localized security feedback during training, the resulting improvements extend beyond the original training tasks. This suggests that reinforcing fine-grained defensive coding behaviors can benefit more challenging security-oriented coding workflows.

Case Study 1: Balancing Functionality and Path Security in BaxBench.

We further examine the FileSearch scenario in BaxBench, which requires implementing a file-search API within /data while supporting legitimate subdirectory requests and preventing directory traversal. SRCode successfully balances functional correctness and security by normalizing requested paths and validating their containment within the permitted directory. In contrast, SafeCoder sacrifices functionality for security by rejecting legitimate requests such as search_dir="/songs", while SecCoderX preserves functionality but introduces a security vulnerability by incorrectly using string-prefix matching to validate directory boundaries. The Base model similarly preserves normal search behavior but lacks sufficient path-boundary validation, allowing directory traversal. These results demonstrate that SRCode's advantage lies not merely in imposing stricter security restrictions, but in precisely enforcing security constraints while preserving legitimate functionality.

Case Study 2: Context-Aware Authorization in A.S.E.

We additionally examine the tsinghua_lunary_cve_2024_4146 instance in A.S.E, a repository-level TypeScript code-completion task associated with CWE-639. The task requires completing the masked checkProjectAccess function using repository context that describes an existing organization-level authorization check and indicates the need for a direct account_project lookup. SRCode correctly enforces project-level authorization by verifying the explicit user–project relationship. In contrast, SafeCoder and SecCoderX preserve insecure organization-level checks, allowing users within the same organization to access projects without explicit authorization. The Base model additionally mishandles the SQL query result by converting the returned object directly to a boolean, causing incorrect access decisions. Unlike these implementations, SRCode identifies the security-critical authorization relationship and preserves the intended functionality. This case demonstrates SRCode's ability to incorporate repository-level security semantics rather than relying solely on local input validation or existing implementation patterns.

Together, these case studies provide concrete evidence complementing the quantitative results in R2-Table 2. They demonstrate that SRCode can preserve legitimate functionality while enforcing security constraints, ranging from filesystem boundary validation to repository-level authorization. These findings further support SRCode's generalizability to more challenging and realistic coding tasks.

Reviewer_ZDBEOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SRCode, a two-stage framework for secure code generation. It constructs PrimeVul+ from real-world vulnerable C/C++ functions, uses vulnerability detection and repair tasks for supervised fine-tuning, and then applies reinforcement learning to code generation. The token-level reward (TLR) concentrates positive feedback on code spans that a teacher model identifies as secure. The paper reports improvements on several security benchmarks and evaluates general code-generation performance to check for capability loss.

Critical strengths
  1. The matched-budget ablation in Section 4.4 is a useful part of the evaluation. It holds teacher judgments and training settings fixed while changing where positive rewards are assigned, providing evidence for the benefit of reward localization.
  2. The experiments cover several security benchmarks and include general code-generation tests. The teacher-supervision audit and teacher-scale comparison also provide evidence about the quality of the training signals.
Critical weaknesses
  1. The framework mainly combines SFT, PPO, and teacher-guided token-level rewards. While the data and teacher judgments are security-specific, it remains unclear what methodological advance the framework introduces for code security beyond adapting existing localized optimization ideas. The matched-budget ablation demonstrates the benefit of localization, but does not resolve this novelty concern.
  2. The policy-model experiments use Qwen2.5-Coder, DeepSeek-Coder, and CodeLlama. These results show gains across the tested backbones, but leave open whether TLR offers similar benefits for a stronger code model with better initial secure-coding performance.
  3. Training starts from a limited pool of C/C++ functions covering 14 CWE categories. Table 1 provides initial Python results, but transfer to held-out vulnerability categories and project-level code is not established. The teacher audit and scale comparison address some supervision concerns; a controlled test of span-label errors would clarify how robust TLR is to incorrect localization.
Questions for authors
  1. What specific methodological advance does SRCode introduce over prior localized security optimization, such as DiSCo/LPO, beyond combining SFT, PPO, and teacher-guided rewards? Which results substantiate this distinction?
  2. Does TLR still help a stronger policy model that already performs well on secure code generation?
  3. Beyond the teacher-scale comparison, how does TLR respond to controlled errors in secure-span labels? Are results available with CWE categories held out from training?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ZDBE (Part 1 / 4)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We are particularly encouraged by your recognition of our controlled ablation design, comprehensive benchmark evaluations, and teacher-supervision analyses.

We hope these clarifications and new results will address your remaining concerns and we respectfully ask if you're comfortable to reconsider the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


W1 & Q1: Methodological Contribution Beyond Existing Localized Optimization.

Concern: The methodological advance over existing localized security optimization, such as DiSCo/LPO, is unclear beyond combining SFT, PPO, and teacher-guided rewards.

A1: We appreciate the opportunity for clarification. SRCode's methodological contribution lies in its security-aware token-level reward formulation for on-policy reinforcement learning, distinguishing it from DiSCo, which is a dataset, and LPO, an off-policy preference learning method independent of reinforcement learning that relies heavily on static preference data. SRCode differs from existing localized preference optimization in three key aspects.

(1) Explicit token-level rewards for precise security optimization.

LPO builds on SimPO and localizes a pairwise preference objective using token masks. In contrast, SRCode assigns explicit rewards to security-relevant tokens in policy-generated code, providing localized feedback through PPO's return and advantage estimation. This directly reinforces fine-grained implementation decisions, such as boundary checks and validation conditions, which are critical to preventing localized vulnerabilities.

(2) Contextual localization and direct reinforcement of secure implementations.

LPO identifies security-relevant tokens through differences between paired secure and insecure responses, where incidental edits may also affect localization. SRCode instead leverages teacher model judgments to identify secure implementations directly within policy-generated code and allocates a fixed positive reward budget to the corresponding tokens. This enables targeted supervision of security-critical tokens without relying on pairwise response differences.

(3) On-policy adaptation to evolving security risks.

Unlike off-policy methods such as LPO and SafeCoder, which rely on fixed training datasets, SRCode continuously optimizes against code generated by the current policy, allowing reward signals to adapt to its evolving vulnerability patterns. This is particularly valuable for secure code generation, where subtle vulnerabilities may emerge as the model's coding behavior changes. On-policy optimization reduces dependence on fixed training distributions and enables targeted reinforcement of security-critical decisions throughout training.

The matched-budget ablation in Section 4.4 demonstrates that TLR outperforms uniform reward allocation under identical teacher judgments, reward budgets, global penalties, and PPO settings. Furthermore, our new controlled perturbation experiment (detailed in W3 & Q3) shows that progressively replacing correctly localized reward positions with random non-target positions degrades security performance, even when reward sparsity and budgets are preserved. These complementary results demonstrate the importance of accurate security-aware localization beyond reward concentration alone.

We will clarify the methodological distinctions from DiSCo/LPO in Section 2, incorporate the new controlled perturbation evidence (detailed in W3 & Q3) into Section 4.4 and Appendix D. These revisions will more clearly establish SRCode's contribution as contextual secure-span localization and fixed-budget token-level reward allocation for on-policy secure code generation, rather than simply combining existing training techniques.

AuthorsOpenReview ↗

Response to Reviewer ZDBE (Part 2 / 4)

W2 & Q2: Effectiveness on Stronger Security-Specialized Models

Concern: Whether TLR remains effective on a stronger policy model that already has substantial secure code generation capabilities.

A2: We thank the reviewer for this valuable suggestion. To evaluate SRCode under a stronger security-specialized initialization, following your suggestion, we conducted additional experiments using SecureCode-7B (2026), a security-specialized model built upon Qwen2.5-Coder-7B. This setting examines whether SRCode can further improve a policy that has already undergone security-oriented fine-tuning. We followed the original experimental settings and evaluation protocols.

R1-Table 1: Performance comparison using SecureCode-7B as the security-specialized initialization. | Method | CWEval | CWEval | CodeLMSec | CyberSecEval | |:---|:---:|:---:|:---:|:---:| | | FS@1 (C/C++) | FS@1 (Python) | Sec.Rate | Pass.Rate | | Base: SecureCode-7B | 36.5 | 47.2 | 83.5 | 70.8 | | SafeCoder | 26.9 | 37.3 | 86.7 | 76.5 | | SecCoderX | 42.3 | 52.5 | 87.2 | 79.1 | | SRCode | 44.2 | 54.8 | 91.0 | 82.3 |

As shown in R1-Table 1, SRCode achieves the best performance across all four metrics, substantially improving upon the SecureCode-7B (2026) initialization and outperforming SecCoderX by 1.9, 2.3, 3.8, and 3.2 pp, respectively. These results demonstrate that SRCode remains effective even when initialized from a model with stronger security capabilities. Together with the matched-budget ablation in Section 4.4, they provide complementary evidence for the effectiveness of SRCode and the value of localized reward allocation.

We will incorporate the SecureCode-7B results into Section 4.2, with detailed experimental configurations and comparisons provided in Appendix C. This will explicitly demonstrate the applicability of SRCode to security-specialized models beyond the original initialization.

AuthorsOpenReview ↗

Response to Reviewer ZDBE (Part 3 / 4)

W3 & Q3: Robustness to Span-Label Errors and Generalization Beyond Training Categories.

Concern: Limited evidence of generalization to held-out CWE categories and project-level code, and unclear robustness to controlled errors in secure-span localization.

A3: We appreciate the reviewer for highlighting these important aspects. We address robustness to localization errors and generalization beyond the training distribution separately.

(1) Robustness to secure-span labeling errors.

To directly evaluate TLR's noise tolerance, we conducted an additional controlled perturbation experiment using Qwen2.5-Coder-7B-Instruct under the original training and evaluation settings. Specifically, we randomly replaced 25%, 50%, and 100% of teacher-identified reward positions with positions sampled outside the original secure spans, while preserving the number of rewarded tokens and the total positive reward budget.

R1-Table 2: Robustness of TLR under controlled secure-span localization perturbations. | Setting | CWEval | CWEval | CodeLMSec | CyberSecEval | |:---|:---:|:---:|:---:|:---:| | | FS@1 (C/C++) | FS@1 (Python) | Sec.Rate | Pass.Rate | | Original teacher localization | 38.5 | 54.2 | 88.7 | 74.8 | | Uniform reward allocation | 30.8 (-7.7) | 51.2 (-3.0) | 83.9 (-4.8) | 66.8 (-8.0) | | 25% random replacement | 36.5 (-2.0) | 50.1 (-4.1) | 85.5 (-3.2) | 71.2 (-3.6) | | 50% random replacement | 30.8 (-7.7) | 43.8 (-10.4) | 80.2 (-8.5) | 59.0 (-15.8) | | 100% random replacement | 26.9 (-11.6) | 42.4 (-11.8) | 67.2 (-21.5) | 57.9 (-16.9) |

The results demonstrate that TLR exhibits strong tolerance to moderate localization noise. Even with 25% of reward positions corrupted, SRCode retains 36.5% CWEval FS@1 (C/C++), 85.5% CodeLMSec security rate, and 71.2% CyberSecEval pass rate, outperforming uniform reward allocation on three of the four metrics. This indicates that TLR can preserve substantial security benefits despite imperfect teacher-generated span annotations.

To further explore extreme cases, we deliberately increased the perturbation ratio to 50% and 100%. Performance declined as localization errors increased, with 100% random replacement performing worse than uniform rewards. Nevertheless, SRCode maintains substantial performance under partial perturbations. These findings highlight TLR's strong robustness to localization noise, even when teacher-generated annotations are imperfect.

Moreover, in Section 4.4, we have shown that the teacher-scale comparison shows that secure-span identification accuracy decreases from 90.62% to 81.52% with a smaller teacher, while downstream performance differs by at most 1.4 percentage points. Although teacher scale also affects other supervision characteristics, this provides complementary evidence of performance stability under variations in teacher quality.

Together, these results demonstrate that TLR combines effective fine-grained security optimization with meaningful tolerance to imperfect localization signals.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

For clarity and simplicity, we will refer to Reviewers ZDBE, kiB7, and ncGv as R1, R2, and R3, respectively.

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the key contributions and strengths of our work.

We have carefully addressed each individual comment provided by the reviewers and believe we have successfully responded to most of their concerns. In our revised manuscript, we have incorporated the suggested experiments, additional discussions, and relevant updates to further strengthen our work. Below, we summarize the core contributions of our study, the updates to our experiments, and the in-depth discussions included in our revision.


Core Contributions of Our Work

  1. Realistic Security Supervision: We propose Vul2Safe to construct high-confidence vulnerability-repair pairs and implicit prompts from real-world vulnerabilities. Building upon Vul2Safe, we develop PrimeVul+, a curriculum-based dataset that progressively organizes vulnerability detection, repair, and secure code generation tasks.
  2. Token-Level Reinforcement Learning: We introduce SRCode, a two-stage training framework that incorporates security-aware token-level rewards into on-policy reinforcement learning. By assigning localized rewards to security-relevant implementations, SRCode reinforces critical defensive coding patterns beyond conventional sequence-level supervision.
  3. Extensive Evaluation and Discussion: We conduct extensive experiments across multiple model families and security benchmarks, demonstrating consistent and substantial security improvements while preserving general code generation capabilities. Additional evaluations further validate the effectiveness, robustness, and generalizability of SRCode across diverse settings.

We hope these contributions can provide useful insights into security-aware code generation and inspire further exploration of fine-grained supervision for improving the security of LLM-generated code.


Updates of experimental results during Rebuttal

  • Main Text: Added reward-localization robustness experiments (R1, R2) and security–functionality trade-off evaluations on general-purpose models (R3).
  • Appendix F.1: Added evaluations on BaxBench, A.S.E., and LiveCodeBench, together with representative security case studies (R2, R3).
  • Appendix F.2: Added comparisons with SafeCoder trained using augmented PrimeVul+ data (R3).
  • Appendix F.3: Added experiments on the security-specialized SecureCode-7B model (R1).

Updates of in-depth discussions during Rebuttal

  • Main Text: Clarified the token-level reward mechanism and teacher supervision (R2), and discussed the security–functionality trade-off across different model types (R3).
  • Appendix C: Provided additional details on reward model training, optimization settings, inference protocols, and training stability, together with training–evaluation overlap and CWE-category coverage analyses (R1, R2).
  • Appendix D: Clarified the definition and roles of the teacher model (R3).
  • Appendices H and I: Expanded the dataset quality audit and prompt-construction details to improve transparency and reproducibility (R2).
  • References: Added relevant references discussed during the rebuttal period to strengthen the related work and discussions.

We believe these additions and clarifications comprehensively address the reviewers' concerns and enhance the overall quality of our manuscript. All revisions are highlighted in blue-colored text for ease of reference. Our manuscript is updated on Nov 18, AOE time.

We look forward to the reviewers' favourable consideration and remain grateful for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.