acceptodds
Under review as a conference paper at ICLR 2027

FORGE: FINDING AND ELIMINATING VULNERABILITIES WITH SOUND PROGRAM ANALYSIS

Abstract

LLM agents can discover and repair software vulnerabilities, but searching large codebases is costly and can leave exploitable bugs undiscovered. Static analysis can guide this search, yet practical analyzers often sacrifice coverage to limit false positives that burden human reviewers. We present FORGE, a system that revisits this tradeoff with an LLM agent performing triage. FORGE combines a conservative JavaScript/TypeScript analyzer with an agent that investigates candidate vulnerabilities, constructs executable reproductions, and repairs vulnerable code using analysis feedback. Across 15 large open-source repositories, FORGE finds 50% more vulnerabilities using just 18% of the model tokens of the next-best configuration. In a broader campaign, FORGE reproduced 176 vulnerabilities across 164 repositories, finding approximately 60% more non-race vulnerabilities than CodeQL-guided investigation and an additional 79 race and asynchronous-interleaving bugs. These findings included an unauthenticated remote code execution vulnerability in production code that was patched a day later. On six known real-world vulnerabilities, FORGE-guided repairs blocked every exploit in all five runs while consuming 23% fewer tokens than the unaided agent. These results demonstrate that conservative analysis can improve both the coverage and inference efficiency of agenticvulnerability discovery and repair.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Vulnerability Discovery, Large Language Models, Static Analysis, Abstract Interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tWEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents can find and fix bugs but searching large codebases can cost a lot of tokens and leave bugs undiscovered. Forge is a system that uses a sound JavaScript/TypeScript static analyzer with an LLM agent that investigates, reproduces, and patches vulnerable code using feedback from the analyzer. Forge finds 50% more vulnerabilities while using 18% of the model tokens compared to using plain Codex (daybreak-blue-latest) on 15 large repositories. Over 164 repositories, FORGE reproduced 176 vulnerabilities. FORGE-guided Claude prevented 6 exploits vs 3.6/6 exploits using Claude alone.

Critical strengths
  1. The paper addresses a current open problem of token efficiency when using LLM agents to find and repair bugs. I think the problem is realistic and very relevant now that using LLMs to for program analysis and repair is widespread and effective.
  2. Rigorous validation of discovered bugs. The process for validating a bug in your evaluation is clearly explained.
Critical weaknesses
  1. One of the main components of Forge, the static program analyzer Tandem (Section 3.2), does not have clear implementation details. You don't specify the abstract domains, or how the JavaScript semantics and heap are modeled. It's also unclear why building your own analyzer is justified instead of using existing program analysis tools. You also don't compare Tandem to program analysis tools besides CodeQL, such as Joern or TAJS.
  2. Without measurements such as Forge's latency per experiment run or precision, token usage can only tell the reader so much about whether Forge truly has performance gains over using unaided coding agents. I would like to know if token usage efficiency persists at scale beyond the 15 large repositories and the fixed number of bug classes that Forge supports. In the broad vulnerability discovery evaluation, there is no comparison on token usage.
Questions for authors
  1. What are the precision and recall numbers for Tandem and its baselines? Is Tandem's false positive rate higher than that of CodeQL's?
  2. For the 176 findings, how many have been confirmed, patched, or assigned CVEs by maintainers, and were they checked against existing advisories? Are these new vulnerabilities? By saying that Forge reproduced 176 vulnerabilities, does that mean Forge found existing bugs?
  3. How did you implement your abstract interpretation? What is your abstract domain and concrete domain? What is the transfer function?
  4. Can you explain why you resorted to implementing your own analyzer instead of using existing tools? Why did you choose only CodeQL as a baseline? Can you add more static analysis tools as baselines, such as semgrep?
  5. How did you decide on the vulnerability classes in Table 1? Was there an initial study? How were they derived? How do you address vulnerability classes outside of Table 1? Can Forge generalize beyond these classes?
  6. Can you include an example in the Appendix of Tandem's input and output provided to LLMs?
Reviewer_rU5jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FORGE, a vulnerability discovery and repair system that combines conservative static analysis with an LLM agent. Its core idea is to let the analyzer prioritize coverage rather than minimizing false positives, leaving the agent to investigate candidate findings, reproduce vulnerabilities, and validate repairs through iterative analysis.

The evaluation covers 2,604 repositories and 4,990 packages, reporting 176 reproduced vulnerabilities. After excluding race and asynchronous-interleaving bugs outside the CodeQL configuration's coverage, FORGE finds 97 vulnerabilities compared with 61 for CodeQL-guided investigation. The paper also reports improved repair results on six known vulnerabilities and higher security-test pass rates for LLM-generated applications.

The paper addresses a practical problem and presents promising results. The combination of static analysis, executable reproduction, and repair feedback is a sensible direction. However, the strength of the conclusions depends on the fairness of the baselines, the generality of the evaluation, and the distinction between reduced model-token usage and reduced overall cost.

Critical strengths

A useful design choice. The paper makes a convincing case for using static analysis to generate a broad set of candidate vulnerabilities and letting the LLM handle triage. This is a reasonable alternative to relying on either high-precision static analysis or open-ended agent search alone.

Evaluation on real-world code. The large-scale study covers thousands of repositories and packages rather than relying exclusively on synthetic benchmarks. Requiring executable evidence before counting a vulnerability also makes the discovery results more credible than evaluations based only on model-generated claims.

Attention to repair correctness. FORGE re-runs analysis after code changes, and the repair evaluation checks whether known exploits still work. This is stronger than treating a plausible patch or a disappearing warning as evidence of a successful repair.

Useful findings about existing tools. The comparison with CodeQL suggests that analyzer design can substantially affect what an LLM agent discovers. The results also show that providing static-analysis reports does not automatically improve an agent's performance; the quality and coverage of those reports matter.

Critical weaknesses

The comparison with CodeQL is not fully conclusive. The paper reports 97 non-race vulnerabilities for FORGE and 61 for CodeQL + Claude, but CodeQL's coverage differs across vulnerability classes. The paper also acknowledges that prior CodeQL use may have removed some vulnerabilities before the evaluation. These factors make it difficult to attribute the entire difference to FORGE's analysis strategy.

The repair evaluation is too small to support broad claims. The existing-code experiment covers only six vulnerabilities, with five runs per condition. The greenfield evaluation is also limited to a particular application-generation setup and security-test suite. These results are encouraging, but they do not establish that the approach generalizes to diverse projects, vulnerability types, or repair settings.

Token savings do not establish lower total cost. Conservative analysis can be expensive in both time and memory, and the paper acknowledges timeouts on repositories that CodeQL successfully analyzed. The evaluation emphasizes model tokens without providing a sufficiently comprehensive comparison of end-to-end runtime, compute resources, and total operational cost. This limits the claim that FORGE is more efficient in practice.

The coverage guarantee needs careful qualification. The underlying analysis may be sound under its modeled semantics, but end-to-end vulnerability coverage also depends on entry-point discovery, source and sink models, query correctness, and language-feature support. The paper explicitly states that its race-condition queries are unsound. The distinction between analysis soundness and vulnerability-query soundness is important, and the broad claims about coverage should be interpreted accordingly.

Reproducibility remains limited. The paper withholds experimental artifacts and repository identities because some vulnerabilities remain unpatched. This is understandable from a responsible-disclosure perspective, but it prevents independent verification of the reported findings and makes it difficult to reproduce the main comparisons. The appendix documents the protocols, but this is not a complete substitute for runnable artifacts.

The evidence for the main design choice could be stronger. The paper argues that broader candidate coverage is beneficial because the agent can filter false positives. However, the evaluation does not fully characterize the relationship between candidate volume, false-positive burden, triage accuracy, and discovery yield across different projects. Without this analysis, it is difficult to determine when conservative analysis is most beneficial and when the additional reports may overwhelm the agent.

Questions for authors

Can the authors provide a more controlled comparison with CodeQL using matched vulnerability classes, equivalent entry-point coverage, and comparable analysis configurations? How much of the observed improvement remains under these conditions?

Can the authors report end-to-end cost, including static-analysis runtime, peak memory, model inference, and the number of unresolved candidates? Does FORGE remain more cost-effective when analyzer overhead is included?

How sensitive are the results to the number of candidate reports? In particular, what happens to triage accuracy, token consumption, and the number of reproduced vulnerabilities as the report volume increases?

Can the authors expand the repair evaluation to a larger and more diverse set of vulnerabilities and repositories? It would also be useful to report regression-test results and whether patches preserve the intended functionality, rather than only blocking the supplied exploits.

Given that some queries are unsound and entry-point and source/sink coverage are incomplete, what evidence supports the claim that FORGE offers broader vulnerability coverage in practice? Can the authors quantify the main sources of missed vulnerabilities?

Reviewer_mpM1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FORGE, a framework combining conservative static program analysis with LLM-based vulnerability investigation and repair. Its central motivation is that LLM-based triage changes the traditional tradeoff between false positives and false negatives in static analysis: broader candidate coverage may become preferable even when it produces additional spurious reports.

FORGE uses Tandem, an abstract-interpretation-based analyzer for JavaScript and TypeScript, to generate potential vulnerability reports with supporting program-analysis evidence. An LLM agent then investigates these reports, attempts executable reproductions, and repairs vulnerable code with additional analysis feedback.

The evaluation includes (1) vulnerability discovery across 15 large repositories, (2) a broader campaign covering 2,604 repositories and approximately 94 million lines of code, and (3) vulnerability repair in existing and LLM-generated applications. The authors report 176 reproduced vulnerabilities across 164 repositories, approximately 60% more non-race vulnerabilities than a CodeQL-guided baseline, and improved repair success with reduced LLM token consumption.

Overall, the paper presents an interesting systems approach and substantial empirical evidence for the practical value of analysis-guided security agents.

Critical strengths
  • Substantial real-world evaluation with meaningful security outcomes. The most compelling contribution is the scale and practical relevance of the vulnerability-discovery evaluation. Notably, the authors investigate thousands of real-world packages and require executable reproductions through public interfaces before counting a vulnerability. The discovery of 176 reproducible vulnerabilities, including concurrency-related bugs that are difficult to identify through conventional approaches, provides meaningful evidence of practical utility.

  • A well-motivated reconsideration of the static-analysis coverage trade-off. The paper presents an interesting argument that the availability of LLM-based triage may change how static analyzers should be designed. The proposed approach prioritizes broader candidate coverage and delegates investigation to an automated agent.

  • The paper is well-written and easy to follow.

Critical weaknesses
  • The central claim about conservative analysis is not adequately validated. The experiments compare different analyzers rather than isolate the benefit of conservative analysis. In the 15-repository evaluation, Tandem detects all six target vulnerabilities while CodeQL detects none (Appendix A.1). The gains may therefore stem from differences in query coverage rather than soundness. Moreover, the paper lacks a systematic evaluation of false-positive rates and LLM triage reliability, despite these being central to its motivation.

  • Efficiency claims lack end-to-end cost evaluation. The efficiency gains are primarily measured in LLM tokens, overlooking the runtime and memory overhead of Tandem, which can even time out on programs successfully analyzed by CodeQL (Section 5). Without end-to-end measurements, it remains unclear whether FORGE actually reduces overall computational cost.

  • The repair evaluation is limited and potentially confounded. The repair experiments cover only six vulnerabilities across two repositories. More importantly, the Claude-only and FORGE conditions are not prompt-identical (Appendix A.3), making it difficult to attribute improvements solely to analysis feedback. The greenfield evaluation also relies on generated security tests, with limited evidence of broader functional correctness and repair generalization.

  • The threat model and adversarial robustness are underexplored. FORGE relies on LLM agents to inspect potentially untrusted code and make security-critical decisions, yet the trust boundaries and robustness against adversarial repository content (e.g., prompt injection through source comments) are not clearly established. Such attacks could potentially manipulate vulnerability triage or induce unsafe patches. Without adversarial evaluation or explicit threat-model assumptions, the reliability of FORGE in security-sensitive deployments remains uncertain.

Questions for authors

Q1. Can the authors isolate the benefits of conservative analysis from differences in vulnerability-query coverage?

Q2. Does FORGE remain more efficient when accounting for analyzer runtime, memory, and LLM costs?

Q3. Can the authors provide a prompt-matched repair ablation and report whether patches preserve functional correctness?

Reviewer_kUFXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

FORGE pairs a JavaScript/TypeScript analyzer, Tandem, with coding agents that investigate reports, build executable reproductions, and repair code using reanalysis. A controlled study evaluates six fixed vulnerabilities, and a larger campaign reproduces 176 vulnerabilities. Excluding race-related classes not covered by the CodeQL configuration, the campaign reports 97 findings versus 61 for CodeQL-guided Claude. Repair is evaluated using held-out exploits on the six targets and generated functional/security tests on 32 applications. The paper distinguishes soundness of the modeled analysis from query coverage and the reliability of agent decisions.

Critical strengths
  1. The real-world findings have practical substance. The campaign counts executable witnesses against actual program interfaces, and the non-race comparison shows that the entire difference is not explained by adding classes absent from the CodeQL configuration. This is stronger evidence than warning counts or an agent’s own success report.

  2. The results distinguish analyzer coverage from agent judgment. Tandem reports all six controlled targets, but Claude and Codex confirm different numbers. Reanalysis and held-out exploit checks also avoid treating a disappearing warning as sufficient evidence of repair. These choices reveal important limits of the combined workflow.

Critical weaknesses
  1. The discovery results do not establish that conservative analysis is the reason for the gain. Tandem and CodeQL differ in supported models, queries, and report evidence; CodeQL reports none of the six fixed targets. That establishes that Tandem gives better leads for this cohort, not that a general false-positive/false-negative tradeoff explains the improvement. The paper also gives insufficient technical detail to determine which query guarantees hold, while entry-point gaps, unsupported features, timeouts, and unsound race queries limit coverage. The broad campaign demonstrates useful output, but cannot by itself establish the methodological advantage suggested by the sound-analysis framing.

  2. The repair and efficiency comparisons have material limitations. Appendix A.3 changes prompts and client versions between conditions and adds a random-output prelude only to the baseline. This is not evidence that the prelude necessarily harms performance, but it is an unnecessary asymmetric intervention. The six targets are also concentrated in two repositories, and blocking a known exploit does not establish preservation of normal behavior. In the generated-application study, FORGE passes security tests on 32/32 applications, versus 31/32 for both controls; the stronger joint endpoint is 29/32 versus 26/32 and 27/32. These are useful small-sample results, not evidence of comprehensive secure repair. Finally, token savings exclude analyzer resource use, so they do not establish lower total cost.

Questions for authors
  1. What causes the discovery gain? Appendix A.1 says Tandem reports all six fixed targets while CodeQL reports none. How were those six targets originally established, and which analysis/query differences account for each discrepancy? Please also provide an anonymized breakdown of the five CodeQL-confirmed vulnerabilities Tandem misses in the larger campaign, separating modeling gaps, query gaps, and timeouts. These cases would let the reader assess the claimed coverage advantage without exposing exploit details.

  2. Repair control. With the same client version and prompt in both conditions and without the asymmetric 4,096-digit prelude, does replacing only the analyzer report preserve the 6.0/6 versus 3.6/6 gap? Please report normal functional-test results for the resulting patches, since defensive exceptions count as blocked exploits. This would address both attribution to analysis feedback and whether repair preserves intended behavior.

  3. Total effort. For the broad campaign, how many reports did each analyzer emit, how many were investigated, and how many remained unresolved or timed out? Please pair these counts with analyzer runtime/peak memory and agent-token use under the respective stopping rules. Without these denominators, 97 versus 61 findings cannot establish which workflow gives better coverage per unit of total effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.