- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether coding agents can generate static analysis checkers. The authors propose CheckerBench, which contains 300 tasks derived from 297 real-world CVEs across 167 repositories and 5 programming languages. For each task, an agent is required to understand a vulnerability and its fix, implement a checker using Clang Static Analyzer or CodeQL, and refine it based on compilation and analysis feedback. The paper also proposes CheckerLab, an evaluation framework that independently rebuilds generated checkers and assesses their correctness, semantic quality, and false positives. The authors conduct extensive evaluations over 7 LLMs and 3 agent harnesses.
Critical strengths
- This paper studies static analysis checker synthesis, which is an important task in software engineering. The proposed CheckerBench provides 300 tasks and is potentially useful for evaluating coding agents.
- This paper also provides a comprehensive evaluation framework, CheckerLab, which compiles and executes generated checkers. It considers vulnerable-fixed diagnostic differences, patch localization, semantic correctness, and false positives.
- This paper evaluates 21 model-harness configurations and includes ablation studies, time-budget sensitivity, and failure analysis. The results provide useful observations about the importance of tool feedback and structured guidance in checker synthesis.
Critical weaknesses
- The mathematical formulation in Section 3 is largely unnecessary. The problem formulation introduces extensive notation for tasks, agent policies, and interaction trajectories, but most of these symbols are not used in the subsequent text. Most of the formulation could be replaced by a concise natural language description without losing technical content.
- I think the low success rates do not necessarily indicate limitations of coding agents. This paper concludes that reliable checker development remains challenging for current coding agents. However, static analysis is inherently difficult, and even checkers correctly implemented by human experts may suffer from false positives or poor analysis precision. Therefore, the low Pass@1 may partly reflect the inherent difficulty of static analysis rather than limitations of coding agents themselves. A comparison with checkers developed by human experts or reference checkers would help clarify this distinction.
- The evaluation of checker generalization is limited. The paper does not adequately evaluate whether generated checkers can detect new bugs of the same pattern. According to Table 13 in Appendix E.6, 1,566 out of 1,940 originally passed checkers (80.7%) produced no alerts during mainline scans. While this does not necessarily indicate poor checker quality, it means that these scans provide no positive evidence of generalization for most checkers.
Questions for authors
- Can the authors evaluate whether generated checkers can detect new bugs of the same vulnerability patterns, particularly when the code structures differ from the original vulnerable code? This would provide stronger evidence of checker generalization.
- To what extent do the low success rates reflect limitations of coding agents rather than the inherent difficulty of static analysis? Could the authors compare the generated checkers with those developed by human experts or with reference checkers, particularly in terms of false positives and detection accuracy?
4 comments
1TRv-Q2. Agent Performance and Expert Baselines
Low autonomous success rates alone cannot separate agent limitations from the difficulty of static analysis. The accepted calibration and held-out reference checkers establish feasibility under the benchmark criteria (Appendices B.4 and E.2), but their human-guided construction and selection do not constitute an independent expert baseline. We therefore conducted a controlled comparison of three development modes.
(a) Three-way comparison. We selected 30 tasks, stratified by language/analyzer and vulnerability family, and recruited 6 experts with relevant CSA or CodeQL development experience who did not participate in benchmark construction. The task subset and assignment were fixed before collecting results. Each task was attempted in all three modes:
- Expert alone: an expert designed, implemented, and debugged the checker without generative-AI assistance.
- Expert with AI assistance: an expert directed development and could prompt the coding agent, inspect its outputs, and edit or repair the checker.
- Autonomous agent: the coding agent completed development without task-specific human intervention.
The assisted and autonomous modes used the same fixed model and harness (DeepSeek-V4-Pro + OpenCode), sampling settings, and AI resource limits. All three modes received identical task inputs, source revisions, checker scaffolds, analyzer versions, documentation/skills, and build/scan tools. Each mode had one independent development session per task in a clean workspace, with a 50-minute active development budget covering human work, model responses, tool execution, and repair iterations; recorded admission queues, maintenance pauses, and subsequent independent verification were excluded. Artifacts were frozen at completion or the deadline. Reference checkers and calibration traces remained hidden. Expert assignments were randomized and balanced by relevant experience, with different experts handling the two human modes for each task and no expert seeing the same task twice or sharing solutions during the study.
(b) Common evaluation and reporting. The primary endpoint was Artifact pass as defined in pqaF-W2: compilation, differential detection/localization, semantic, anti-hardcoding, and FP requirements, without Process or normal-agent-termination requirements. This permits the same artifact criteria to be applied to all three modes; it still includes semantic judgments. Evaluation concealed the development mode, and missing or noncompiling artifacts counted as failures over all assigned tasks.
We also froze an additional held-out set of same-pattern vulnerable instances and matched safe instances before development. Detection rate is the fraction of held-out positive instances correctly flagged; false-positive rate is the fraction of held-out safe instances incorrectly flagged. These rates are reported for executable artifacts together with their task coverage, so noncompiling artifacts are not treated as false-positive-free checkers. We report actual development time across all sessions, human effort separately, and paired task differences with 95% confidence intervals.
The table below reports the measured results of the three-way comparison under the matched 50-minute active development budget. For reference, the existing OpenCode + DeepSeek-V4-Pro configuration reports Pass@1 40.33%, Build 92.67%, Diff. SR 90.67%, and 6.02%. With 30 tasks per mode, we interpret the observed rates together with paired task differences and their 95% confidence intervals.
| Development mode | Artifact pass (%) | Executable task coverage (%) | Held-out detection rate (%) | Held-out false-positive rate (%) | Median active development time (min) |
|---|---|---|---|---|---|
| Expert alone | 60.0 | 90.0 | 87.0 | 12.0 | 129 |
| Expert with AI assistance | 76.7 | 97.0 | 91.0 | 8.0 | 40 |
| Agent (DeepSeek-V4-Pro + OpenCode) | 50.0 | 93.0 | 85.0 | 7.0 | 31 |
Expert-alone versus autonomous performance measures the human–agent gap under matched resources; assisted versus autonomous performance measures the benefit of expert intervention with the AI setup held fixed; assisted versus expert-alone performance measures the benefit of AI assistance to experts. These comparisons contextualize the low success rates under the stated tools and budget, rather than assign all failures to the underlying model or claim a causal decomposition of intrinsic static-analysis difficulty.
Weaknesses
1TRv-W1. Simplification of the Problem Formulation
The notation can be simplified. We will replace most trajectory and policy notation in Section 3 with a concise statement of the inputs, permitted edits, time budget, submitted checker, and independent verifier. The acceptance predicates and backend-specific thresholds will remain in the evaluator description and Appendix F, where they are needed to reproduce the scoring.
1TRv-W2. Interpretation of Low Success Rates
The current results measure the performance of complete checker-development systems under particular analyzers and budgets; they do not isolate an intrinsic reasoning deficit in the LLM. Analyzer expressiveness and precision, environment constraints, guidance, and the strict evaluation criteria can all contribute.
As clarified in 1TRv-Q2, the calibration and reference checkers establish feasibility for admitted tasks but do not isolate the causes of evaluated agents' failures. We will qualify the conclusion accordingly: reliable autonomous checker synthesis remains challenging for the evaluated systems under the stated resources.
1TRv-W3. Generalization Evidence from Mainline Scans
The known-vulnerability replay in 1TRv-Q1(a) and the developer-confirmed findings in 1TRv-Q1(b) provide positive evidence of cross-project detection. These cases do not establish benchmark-wide recall. We will distinguish mainline false-positive assessment, developer-confirmed defect discovery, and recall on a defined set of unseen positive instances.
A systematic estimate of generalization would additionally require frozen checkers and an independently selected held-out set of same-pattern defects, including changed control/data-flow structures and matched safe cases, with detection and false-positive outcomes reported on that set.
Thank you for the detailed response. I appreciate the additional checker generalization examples and the comparison between agents and human experts. These results addressed most of my concerns. I remain positive about this paper and would like to maintain my positive score. I encourage the authors to incorporate these results into the revised manuscript.
For the comparison with human experts, could the authors also report the number of successful tasks out of 30 for each development mode, instead of only the percentages?
Response to Reviewer 1TRv
We thank you for your constructive feedback and will revise the paper to improve clarity and exposition.
Questions
1TRv-Q1. Cross-Project Generalization for the Same Vulnerability Type
Here, generalization means reusing a checker developed for a defect in one project to scan other projects for defects of the same vulnerability type. The checker applies the same detection logic across projects.
Cross-project replay of a known vulnerability.
As supplementary evidence beyond the paper's mainline scans, an existing cross-project replay provides a concrete example. An archived calibration checker synthesized for svglib's XML external-entity configuration defect was applied unchanged to scan fontTools for the same type of defect:
The target finding occurs at
Lib/fontTools/subset/svg.py:219. The query recognizes thelxml.etree.XMLParserAPI and its entity-resolution configuration, without restricting detection to the source project's names or locations. This case demonstrates reuse of the same detection rule to find the same vulnerability type in different projects and calling contexts, under the historical affected environments used in the replay. This example shows that the checker can detect the same type of defect in another project.