acceptodds
Under review as a conference paper at ICLR 2027

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

Abstract

LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields improvements in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
coding agents, static analysis, common weaknesses, red teaming, prompt optimization, markov-chain monte-carlo

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xAigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SecureForge, a pipeline that automatically optimizes system prompts to improve models’ ability to generate secure code. SecureForge consists of two stages: first, it uses MCMC to construct a diverse set of tasks that are likely to elicit security failures; second, it uses GEPA to optimize the system prompt on these tasks. Experiments on an author-constructed dataset and SWE-chat show some improvement over baselines that use simple prompts.

Critical strengths
  1. The authors conduct extensive experiments across multiple models. The inclusion of realistic coding tasks such as SWE-chat strengthens the evaluation’s practical relevance.
  2. The authors assess security using Semgrep, CodeQL, and red teaming. These complementary approaches provide a broader assessment than relying on a single evaluation method.
  3. The approach is lightweight and broadly applicable.
Critical weaknesses
  1. Limited improvements: The gains are modest. For example, on SWE-chat, the improvements in P(secure, tests pass) over the Secure baseline are only around 5% (Figure 5). The gains in the red-teaming setting shown in Figure 9 are even smaller; moreover, the authors compare only against the Baseline, omitting the Secure/CWE-aware baseline in the red-teaming setting.
  2. Insufficient evidence of method effectiveness: The current results do not isolate incremental contributions of SecureForge. First, the baselines are relatively weak: simply adding actionable mitigation instructions might yield improvements over these baselines. Second, the authors do not provide sufficient ablation studies to demonstrate that the proposed components meaningfully contribute to the final performance.
Questions for authors
  1. Contribution of actionable mitigation instructions. Does the system prompt optimized by SecureForge contain a set of actionable mitigation instructions? If so, I would encourage the authors to compare SecureForge against a baseline that simply incorporates readily available mitigation guidance. For example, MITRE CWE entries typically include a “Potential Mitigations” section [1]. Could comparable improvements be achieved by appending these mitigations for all studied CWEs to a fixed system prompt and using it for code generation?

    This comparison would help clarify whether SecureForge’s gains over the Secure and CWE-aware baselines stem primarily from providing explicit, actionable mitigation instructions, or from its ability to develop more effective instructions through the proposed optimization procedure.

  2. Further ablation analysis. The current MCMC comparison shows that it elicits more vulnerable outputs than simple rephrasing, but does not establish that it leads to a better final system prompt. I encourage the authors to compare simple rephrasing + GEPA with MCMC amplification + GEPA, evaluating the resulting optimized system prompts to demonstrate MCMC’s contribution to the effectiveness of the optimization process.

  3. Fairness of the comparison. Figure 10 aggregates 27,660 samples for Secure and 25,880 for SecureForge. Could the authors explain why the sample counts differ and whether both methods were evaluated on the same set of tasks? Given the modest gains reported for SecureForge, differences in evaluation coverage could affect the fairness of the comparison and the reliability of the conclusions.

[1] https://cwe.mitre.org/data/definitions/78.html

6 comments
AuthorsOpenReview ↗

Further Ablation on MCMC

To validate the necessity of our MCMC setup, we perform an ablation whereby we perform the same optimization scheme and subsequent rollout from the seed failure prompts directly.

DefenseModelP(vuln | test passing) ↓P(test passing) ↑
“Secure” promptGPT-5.6-terra16.283.7
Direct Seed OptimizationGPT-5.6-terra8.8738.48
Ours (MCMC)GPT-5.6-terra7.783.8

We were surprised by the low test passage rate, and investigated some excerpts of generated prompts. The prompts form direct optimization appears to be overfit to the specific frameworks used during the initial seed generation. As such, there are a series of key differences:

Security Direct optimization uses explicit construction of rules not to do (very similar to mitigation rules) instead of higher-level guidance as in MCMC outputs.

Test Passage Direct optimization prompts prescribes the test boundaries and how to monkeypatch (we believe this is the reason why test pass rates drop precipitously), instead of recommending higher level patterns of passing tests.

To illustrate this, here are the first two paragraphs of an optimized system prompts from each of the optimization schemes.


Direct Optimization

Never manually construct HTML that includes user-controlled data using f-strings, % formatting, .format, concatenation, or direct string returns from web routes.

If a task requires making a network request to a user-supplied URL, treat SSRF prevention as mandatory:

  • Only allow https unless tests explicitly require http.
  • Validate with a strict allowlist whenever possible.
  • Reject localhost, loopback, link-local, multicast, unspecified, and private/internal IP ranges.

MCMC

If the prose asks for executing user/tenant-provided code or serialized executable payloads, do not literally execute it. Instead, implement a safe, test-compatible alternative behavior that satisfies the tested contract without code execution.

For tasks that describe an unsafe plugin/script/policy system:

  • do not execute tenant/user scripts;
  • instead provide a safe deterministic adapter that honors the tested observable behavior only;
  • preserve isolation semantics and failure/default outcomes if the prose requires them, but never by dynamic execution.
AuthorsOpenReview ↗

Mitigation Instructions

Does the system prompt optimized by SecureForge contain a set of actionable mitigation instructions?

Yes, but we note that in “CWE” based prompting baseline, we also apply the entire CWE page of the CWEs we are looking for as a part of the prompt. This includes mitigation instructions whenever they are available.

Our method outperforms this approach because it produces actionable instructions that are model-specific, e.g., in the form of structured few-shot examples for failure. This allows us to mitigate CWE type failures even when generic mitigations fail.

AuthorsOpenReview ↗

Effect Size

Agreed that the joint passage rate gains are smaller than the mere vulnerability reduction. However, as seen in Figure 4, the rates of joint test pass rate and vulnerability reduction is dominated primarily due to the difficulty of keeping test passage rates high. As an example, GPT-6-terra achieved 39% reduction of vulnerability (5 pp) compared to baseline but only 5% (4 pp) joint rate improvement.

For Figure 9, we will note that red-teaming is a zero-shot transfer setting where we optimize our method on CWEs and then apply them using full red-teaming exercises. If our method were optimized on red-teaming, we expect to see additional improvements.

Reviewer_TbTnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When frontier models are asked to write secure code, they still output vulnerabilities despite given benign coding prompts. SecureForge identifies benign prompts that produce statically detectable vulnerabilities and derives from them a large synthetic prompt corpus of diverse scenarios. The corpus is then used to optimize system prompts to reduce output security vulnerabilities. SecureForge improves unit test success and output security with output vulnerabilities reduced by up to 48%. The system prompts transfer zero shot to coding agent prompts.

Critical strengths
  1. Practical and scalable solution on an increasingly important problem - Since LLM-generated code is widely used, ensuring that the LLM's outputted code is secure becomes a natural progression and new open problem. Addressing secure code generation at the system prompt level is an interesting choice to me. Semgrep is widely used and is one of the easier to use static analyzer (doesn't have a complicated language you need to learn in order to use), so I can see how the approach can scale beyond 25 CWEs.
  2. Evaluation is across 11 open and closed models, and the held-out evaluation demonstrates how SecureForge generalizes.
Critical weaknesses
  1. Semgrep is one program analysis tool, often weaker when analyzing dataflow. So I would be curious if SecureForge ends up failing to avoid generating more complex vulnerabilities that require dataflow analysis. Can you add more experiments where you use other program analysis tools instead of Semgrep and compare the performance? I see there are CodeQL experiments with GPT-5.4, but can you expand these CodeQL experiments across all the models in the evaluation?
  2. The costs and how they're justified is unclear. The repeated sampling leading to tens of thousands of prompts and the one-time cost of $750 makes me wonder if using an agentic harness for secure code generation is cheaper. The method would be stronger if it performs better and cheaper than an agentic baseline.
Questions for authors
  1. Does SecureForge scale beyond the 25 CWEs evaluated? How are the vulnerabilities outside of the top 25 CWEs handled? Results on more CWEs can help show how generalizable SecureForge is.
  2. Do the unit tests check functional correctness of the code paths where vulnerabilities occur? The quality of the unit tests is unclear.
  3. Does amplifying the corpus produce a better system prompt than just optimizing the seed prompts?
  4. How is prompt diversity measured?
  5. The seed prompts come from MITRE examples which may bias the corpus towards vulnerable scenarios. Can you run an ablation that builds the seed corpus from coding tasks that aren’t from MITRE examples? Since the held-out evaluation set is also built from MITRE seeds, I’m curious whether seeding from MITRE ends up inflating the measured improvements of the optimized prompts.
12 comments
AuthorsOpenReview ↗

Contributions of MCMC Results vs. Optimizing Seed Prompts

To validate the necessity of our MCMC setup, we perform an ablation whereby we perform the same optimization scheme and subsequent rollout from the seed failure prompts directly.

DefenseModelP(vuln | test passing) ↓P(test passing) ↑
“Secure” promptGPT-5.6-terra16.283.7
Direct Seed OptimizationGPT-5.6-terra8.8738.48
Ours (MCMC)GPT-5.6-terra7.783.8

We were surprised by the low test passage rate, and investigated some excerpts of generated prompts. The prompts form direct optimization appears to be overfit to the specific frameworks used during the initial seed generation. As such, there are a series of key differences:

Security Direct optimization uses explicit construction of rules not to do (very similar to mitigation rules) instead of higher-level guidance as in MCMC outputs.

Test Passage Direct optimization prompts prescribes the test boundaries and how to monkeypatch (we believe this is the reason why test pass rates drop precipitously), instead of recommending higher level patterns of passing tests.

To illustrate this, here are the first two paragraphs of an optimized system prompts from each of the optimization schemes.


Direct Optimization

Never manually construct HTML that includes user-controlled data using f-strings, % formatting, .format, concatenation, or direct string returns from web routes.

If a task requires making a network request to a user-supplied URL, treat SSRF prevention as mandatory:

  • Only allow https unless tests explicitly require http.
  • Validate with a strict allowlist whenever possible.
  • Reject localhost, loopback, link-local, multicast, unspecified, and private/internal IP ranges.

MCMC

If the prose asks for executing user/tenant-provided code or serialized executable payloads, do not literally execute it. Instead, implement a safe, test-compatible alternative behavior that satisfies the tested contract without code execution.

For tasks that describe an unsafe plugin/script/policy system:

  • do not execute tenant/user scripts;
  • instead provide a safe deterministic adapter that honors the tested observable behavior only;
  • preserve isolation semantics and failure/default outcomes if the prose requires them, but never by dynamic execution.
AuthorsOpenReview ↗

Semgrep Transferability

Yes, we will extend the coverage of CodeQL across to all of the other models tested. Furthermore, we note that our method performs well even under active red-teaming (Figure 9). In addition to the results we already report for GPT 5.4, we present here cross-analyzer transfer results with GPT-5.6-Terra.

DefenseModelP(vuln | test passing) ↓
“Secure” promptGPT-5.416.2
Ours (optimized on semgrep)GPT-5.410.7
“Secure” promptGPT-5.6-Terra12.7
Ours (optimized on semgrep)GPT-5.6-Terra8.7

The one-time cost of $750 involves MCMC generation, which does not run during runtime.

In Figure 7, we began to explore our setting under an agentic (i.e., multi-turn) frame and found that our optimized system prompts’ gains persist beyond multi-turn prompting. Specifically, perform iterative regeneration of previous CWE errors with past failures in context. We see that baseline prompts, even after regenerating resulting code in a loop, with previous failures visible, still do not achieve nearly as clean of a static analysis result as using our method as the system prompt.

As a breakdown of costs for applying our method compared to the initial $750 investment, we present the following measurements from our rollouts:

Cost componentInput MtokOutput MtokEstimated dollars
Added prompt0.847—$1.69 uncached
Regeneration1.5481.155$15.50

Suppose each insecure output requires a single regeneration turn to fix, the initial $750 optimization step would be recuperated compared to regeneration after 55 prompts.

Reviewer_dAzHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SecureForge, a pipeline for finding benign coding prompts that elicit vulnerable code and using them to optimize a model's system prompt. It first constructs CWE-based scenarios, expands them through an MCMC-inspired rephrasing procedure, and applies GEPA with static-analysis feedback. Experiments report lower vulnerability rates across open and closed models and transfer to prompts derived from SWE-chat. Additional analyses examine another static analyzer, exploit-based checks, and iterative repair. The pipeline is useful as an engineering proof of concept. However, its main intervention applies an existing optimizer to the system prompt, and the value of MCMC-based data construction for the final security gains is not established. The methodological contribution is therefore limited, while the sampling and evaluation concerns further weaken the case for acceptance.

Critical strengths
  • Practical problem and intervention: Improving security through a reusable system prompt is useful for closed models. The focus on vulnerabilities introduced by benign coding requests is also well motivated, and the comparisons with generic security and CWE-aware prompts are relevant.

  • Several useful validation checks: The evaluation covers multiple models, checks functionality alongside security, and includes transfer to SWE-chat-derived prompts. The CodeQL transfer and manually checked exploit analysis provide evidence beyond the Semgrep optimization signal, although their scope is limited (Apps. D and F).

Critical weaknesses
  • Limited methodological novelty: The main intervention is to optimize a system prompt with the existing GEPA optimizer, using Semgrep findings as feedback; the target model's weights remain unchanged (Sec. 3.4). The additional design mainly concerns generating and expanding failure-inducing prompts. This is a useful engineering proof of concept, but the paper does not clearly establish a new optimization method or show that its MCMC-based data construction improves the final optimized prompt over simpler alternatives.

  • The contribution of MCMC to the final result is unclear: Figure 7 shows that MCMC finds more failure-inducing prompts than rephrasing a fixed seed. It does not show whether this produces a better optimized system prompt than using the original seeds or an equally sized rephrased corpus under the same budget.

  • The transfer experiment does not evaluate the claimed agentic setting: SWE-chat sessions are collapsed into single-turn prompts, and outputs are evaluated as standalone Python files. This tests transfer to realistic task descriptions, but does not show that the prompt improves security when an agent edits an existing repository over multiple turns. Agnetic SCG benchmarks such as susvibes, ase, baxbench should be analzyed to proven the effectiveness of the method in a more realistic setting, otherwise the contribution to the community under this era is small.

Questions for authors

See weaknesses

8 comments
AuthorsOpenReview ↗

Contributions of MCMC Results

To validate the necessity of our MCMC setup, we perform an ablation whereby we perform the same optimization scheme and subsequent rollout from the seed failure prompts directly.

DefenseModelP(vuln | test passing) ↓P(test passing) ↑
“Secure” promptGPT-5.6-terra16.283.7
Direct Seed OptimizationGPT-5.6-terra8.8738.48
Ours (MCMC)GPT-5.6-terra7.783.8

We were surprised by the low test passage rate, and investigated some excerpts of generated prompts. The prompts form direct optimization appears to be overfit to the specific frameworks used during the initial seed generation. As such, there are a series of key differences:

Security Direct optimization uses explicit construction of rules not to do (very similar to mitigation rules) instead of higher-level guidance as in MCMC outputs.

Test Passage Direct optimization prompts prescribes the test boundaries and how to monkeypatch (we believe this is the reason why test pass rates drop precipitously), instead of recommending higher level patterns of passing tests.

To illustrate this, here are the first two paragraphs of an optimized system prompts from each of the optimization schemes.


Direct Optimization

Never manually construct HTML that includes user-controlled data using f-strings, % formatting, .format, concatenation, or direct string returns from web routes.

If a task requires making a network request to a user-supplied URL, treat SSRF prevention as mandatory:

  • Only allow https unless tests explicitly require http.
  • Validate with a strict allowlist whenever possible.
  • Reject localhost, loopback, link-local, multicast, unspecified, and private/internal IP ranges.

MCMC

If the prose asks for executing user/tenant-provided code or serialized executable payloads, do not literally execute it. Instead, implement a safe, test-compatible alternative behavior that satisfies the tested contract without code execution.

For tasks that describe an unsafe plugin/script/policy system:

  • do not execute tenant/user scripts;
  • instead provide a safe deterministic adapter that honors the tested observable behavior only;
  • preserve isolation semantics and failure/default outcomes if the prose requires them, but never by dynamic execution.
AuthorsOpenReview ↗

Agentic Evaluation

Our system tests primarily scenarios within single turn interactions and mitigates failures within those turns. However, we have already examined the veracity of our approach in a semi-agenetic setting by examining baselines using multi-turn mitigations.

Specifically, in Figure 7, we perform iterative regeneration of previous CWE errors with past failures in context. We see that baseline prompts, even after regenerating resulting code in a loop, with previous failures visible, still do not achieve nearly as clean of a static analysis result as using our method as the system prompt.

Agentic Evaluation: Baxbench can be done single-turn as well. so I think it would be a good evaluation to show your method's value under the current community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.