- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLM agents can find and fix bugs but searching large codebases can cost a lot of tokens and leave bugs undiscovered. Forge is a system that uses a sound JavaScript/TypeScript static analyzer with an LLM agent that investigates, reproduces, and patches vulnerable code using feedback from the analyzer. Forge finds 50% more vulnerabilities while using 18% of the model tokens compared to using plain Codex (daybreak-blue-latest) on 15 large repositories. Over 164 repositories, FORGE reproduced 176 vulnerabilities. FORGE-guided Claude prevented 6 exploits vs 3.6/6 exploits using Claude alone.
Critical strengths
- The paper addresses a current open problem of token efficiency when using LLM agents to find and repair bugs. I think the problem is realistic and very relevant now that using LLMs to for program analysis and repair is widespread and effective.
- Rigorous validation of discovered bugs. The process for validating a bug in your evaluation is clearly explained.
Critical weaknesses
- One of the main components of Forge, the static program analyzer Tandem (Section 3.2), does not have clear implementation details. You don't specify the abstract domains, or how the JavaScript semantics and heap are modeled. It's also unclear why building your own analyzer is justified instead of using existing program analysis tools. You also don't compare Tandem to program analysis tools besides CodeQL, such as Joern or TAJS.
- Without measurements such as Forge's latency per experiment run or precision, token usage can only tell the reader so much about whether Forge truly has performance gains over using unaided coding agents. I would like to know if token usage efficiency persists at scale beyond the 15 large repositories and the fixed number of bug classes that Forge supports. In the broad vulnerability discovery evaluation, there is no comparison on token usage.
Questions for authors
- What are the precision and recall numbers for Tandem and its baselines? Is Tandem's false positive rate higher than that of CodeQL's?
- For the 176 findings, how many have been confirmed, patched, or assigned CVEs by maintainers, and were they checked against existing advisories? Are these new vulnerabilities? By saying that Forge reproduced 176 vulnerabilities, does that mean Forge found existing bugs?
- How did you implement your abstract interpretation? What is your abstract domain and concrete domain? What is the transfer function?
- Can you explain why you resorted to implementing your own analyzer instead of using existing tools? Why did you choose only CodeQL as a baseline? Can you add more static analysis tools as baselines, such as semgrep?
- How did you decide on the vulnerability classes in Table 1? Was there an initial study? How were they derived? How do you address vulnerability classes outside of Table 1? Can Forge generalize beyond these classes?
- Can you include an example in the Appendix of Tandem's input and output provided to LLMs?