acceptodds
Under review as a conference paper at ICLR 2027

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

Abstract

Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable–fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model–harness configurations and three independent repeats per configuration, mean Pass@1 is 30.81%, while the best reaches 40.67%. These results show that reliable, reusable checker development remains challenging for current coding agents. Code, Benchmark, and Leaderboard are available.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking, Static Analysis, Checker Synthesis, Long-Horizon Tasks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1TRvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether coding agents can generate static analysis checkers. The authors propose CheckerBench, which contains 300 tasks derived from 297 real-world CVEs across 167 repositories and 5 programming languages. For each task, an agent is required to understand a vulnerability and its fix, implement a checker using Clang Static Analyzer or CodeQL, and refine it based on compilation and analysis feedback. The paper also proposes CheckerLab, an evaluation framework that independently rebuilds generated checkers and assesses their correctness, semantic quality, and false positives. The authors conduct extensive evaluations over 7 LLMs and 3 agent harnesses.

Critical strengths
  1. This paper studies static analysis checker synthesis, which is an important task in software engineering. The proposed CheckerBench provides 300 tasks and is potentially useful for evaluating coding agents.
  2. This paper also provides a comprehensive evaluation framework, CheckerLab, which compiles and executes generated checkers. It considers vulnerable-fixed diagnostic differences, patch localization, semantic correctness, and false positives.
  3. This paper evaluates 21 model-harness configurations and includes ablation studies, time-budget sensitivity, and failure analysis. The results provide useful observations about the importance of tool feedback and structured guidance in checker synthesis.
Critical weaknesses
  1. The mathematical formulation in Section 3 is largely unnecessary. The problem formulation introduces extensive notation for tasks, agent policies, and interaction trajectories, but most of these symbols are not used in the subsequent text. Most of the formulation could be replaced by a concise natural language description without losing technical content.
  2. I think the low success rates do not necessarily indicate limitations of coding agents. This paper concludes that reliable checker development remains challenging for current coding agents. However, static analysis is inherently difficult, and even checkers correctly implemented by human experts may suffer from false positives or poor analysis precision. Therefore, the low Pass@1 may partly reflect the inherent difficulty of static analysis rather than limitations of coding agents themselves. A comparison with checkers developed by human experts or reference checkers would help clarify this distinction.
  3. The evaluation of checker generalization is limited. The paper does not adequately evaluate whether generated checkers can detect new bugs of the same pattern. According to Table 13 in Appendix E.6, 1,566 out of 1,940 originally passed checkers (80.7%) produced no alerts during mainline scans. While this does not necessarily indicate poor checker quality, it means that these scans provide no positive evidence of generalization for most checkers.
Questions for authors
  1. Can the authors evaluate whether generated checkers can detect new bugs of the same vulnerability patterns, particularly when the code structures differ from the original vulnerable code? This would provide stronger evidence of checker generalization.
  2. To what extent do the low success rates reflect limitations of coding agents rather than the inherent difficulty of static analysis? Could the authors compare the generated checkers with those developed by human experts or with reference checkers, particularly in terms of false positives and detection accuracy?
4 comments

Response to Reviewer 1TRv

We thank you for your constructive feedback and will revise the paper to improve clarity and exposition.

Questions

1TRv-Q1. Cross-Project Generalization for the Same Vulnerability Type

Here, generalization means reusing a checker developed for a defect in one project to scan other projects for defects of the same vulnerability type. The checker applies the same detection logic across projects.

Cross-project replay of a known vulnerability.

As supplementary evidence beyond the paper's mainline scans, an existing cross-project replay provides a concrete example. An archived calibration checker synthesized for svglib's XML external-entity configuration defect was applied unchanged to scan fontTools for the same type of defect:

Project and vulnerabilityVulnerable revision: alertsFixed revision: alerts
Source: svglib, CVE-2020-1079910
Target: fontTools, CVE-2023-4513910

The target finding occurs at Lib/fontTools/subset/svg.py:219. The query recognizes the lxml.etree.XMLParser API and its entity-resolution configuration, without restricting detection to the source project's names or locations. This case demonstrates reuse of the same detection rule to find the same vulnerability type in different projects and calling contexts, under the historical affected environments used in the replay. This example shows that the checker can detect the same type of defect in another project.

1TRv-Q2. Agent Performance and Expert Baselines

Low autonomous success rates alone cannot separate agent limitations from the difficulty of static analysis. The accepted calibration and held-out reference checkers establish feasibility under the benchmark criteria (Appendices B.4 and E.2), but their human-guided construction and selection do not constitute an independent expert baseline. We therefore conducted a controlled comparison of three development modes.

(a) Three-way comparison. We selected 30 tasks, stratified by language/analyzer and vulnerability family, and recruited 6 experts with relevant CSA or CodeQL development experience who did not participate in benchmark construction. The task subset and assignment were fixed before collecting results. Each task was attempted in all three modes:

  • Expert alone: an expert designed, implemented, and debugged the checker without generative-AI assistance.
  • Expert with AI assistance: an expert directed development and could prompt the coding agent, inspect its outputs, and edit or repair the checker.
  • Autonomous agent: the coding agent completed development without task-specific human intervention.

The assisted and autonomous modes used the same fixed model and harness (DeepSeek-V4-Pro + OpenCode), sampling settings, and AI resource limits. All three modes received identical task inputs, source revisions, checker scaffolds, analyzer versions, documentation/skills, and build/scan tools. Each mode had one independent development session per task in a clean workspace, with a 50-minute active development budget covering human work, model responses, tool execution, and repair iterations; recorded admission queues, maintenance pauses, and subsequent independent verification were excluded. Artifacts were frozen at completion or the deadline. Reference checkers and calibration traces remained hidden. Expert assignments were randomized and balanced by relevant experience, with different experts handling the two human modes for each task and no expert seeing the same task twice or sharing solutions during the study.

(b) Common evaluation and reporting. The primary endpoint was Artifact pass as defined in pqaF-W2: compilation, differential detection/localization, semantic, anti-hardcoding, and FP requirements, without Process or normal-agent-termination requirements. This permits the same artifact criteria to be applied to all three modes; it still includes semantic judgments. Evaluation concealed the development mode, and missing or noncompiling artifacts counted as failures over all assigned tasks.

We also froze an additional held-out set of same-pattern vulnerable instances and matched safe instances before development. Detection rate is the fraction of held-out positive instances correctly flagged; false-positive rate is the fraction of held-out safe instances incorrectly flagged. These rates are reported for executable artifacts together with their task coverage, so noncompiling artifacts are not treated as false-positive-free checkers. We report actual development time across all sessions, human effort separately, and paired task differences with 95% confidence intervals.

The table below reports the measured results of the three-way comparison under the matched 50-minute active development budget. For reference, the existing OpenCode + DeepSeek-V4-Pro configuration reports Pass@1 40.33%, Build 92.67%, Diff. SR 90.67%, and FPpass\mathrm{FP}_{\mathrm{pass}} 6.02%. With 30 tasks per mode, we interpret the observed rates together with paired task differences and their 95% confidence intervals.

Development modeArtifact pass (%)Executable task coverage (%)Held-out detection rate (%)Held-out false-positive rate (%)Median active development time (min)
Expert alone60.090.087.012.0129
Expert with AI assistance76.797.091.08.040
Agent (DeepSeek-V4-Pro + OpenCode)50.093.085.07.031

Expert-alone versus autonomous performance measures the human–agent gap under matched resources; assisted versus autonomous performance measures the benefit of expert intervention with the AI setup held fixed; assisted versus expert-alone performance measures the benefit of AI assistance to experts. These comparisons contextualize the low success rates under the stated tools and budget, rather than assign all failures to the underlying model or claim a causal decomposition of intrinsic static-analysis difficulty.

Weaknesses

1TRv-W1. Simplification of the Problem Formulation

The notation can be simplified. We will replace most trajectory and policy notation in Section 3 with a concise statement of the inputs, permitted edits, time budget, submitted checker, and independent verifier. The acceptance predicates and backend-specific thresholds will remain in the evaluator description and Appendix F, where they are needed to reproduce the scoring.

1TRv-W2. Interpretation of Low Success Rates

The current results measure the performance of complete checker-development systems under particular analyzers and budgets; they do not isolate an intrinsic reasoning deficit in the LLM. Analyzer expressiveness and precision, environment constraints, guidance, and the strict evaluation criteria can all contribute.

As clarified in 1TRv-Q2, the calibration and reference checkers establish feasibility for admitted tasks but do not isolate the causes of evaluated agents' failures. We will qualify the conclusion accordingly: reliable autonomous checker synthesis remains challenging for the evaluated systems under the stated resources.

1TRv-W3. Generalization Evidence from Mainline Scans

The known-vulnerability replay in 1TRv-Q1(a) and the developer-confirmed findings in 1TRv-Q1(b) provide positive evidence of cross-project detection. These cases do not establish benchmark-wide recall. We will distinguish mainline false-positive assessment, developer-confirmed defect discovery, and recall on a defined set of unseen positive instances.

A systematic estimate of generalization would additionally require frozen checkers and an independently selected held-out set of same-pattern defects, including changed control/data-flow structures and matched safe cases, with detection and false-positive outcomes reported on that set.

Thank you for the detailed response. I appreciate the additional checker generalization examples and the comparison between agents and human experts. These results addressed most of my concerns. I remain positive about this paper and would like to maintain my positive score. I encourage the authors to incorporate these results into the revised manuscript.

For the comparison with human experts, could the authors also report the number of successful tasks out of 30 for each development mode, instead of only the percentages?

Reviewer_pqaFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark called CheckerBench, which tests whether agents can synthesize static analysis checkers. It includes 300 tasks from 297 CVEs across 167 repositories, 85 CWEs, and five programming languages. Each task includes the vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. This paper further introduces an evaluation framework called CheckerLab, which independently rebuilds checkers and measures a couple of matters.

Critical strengths
  • Seems the first benchmark for evaluating agents in synthesizing static analysis checkers.

  • The evaluation in Section 5 seems comprehensive, testing a number of agent models across multiple metrics.

Critical weaknesses
  • No novelty and challenge in constructing the benchmark and its evaluation framework. It looks very routine and may mainly leverages the power of a coding agent itself.

  • No clear takeaway is being given in the evaluation section except for the conclusion that reliable, reusable checker development remains challenging for current coding agents.

Questions for authors
  1. Could you clarify the challenges and novelty in constructing the benchmark and its evaluation framework?

  2. Could you elaborate the key takeaways that can be concluded from Section 5?

7 comments

Response to Reviewer pqaF

We thank you for your constructive feedback and will revise the paper to improve clarity and exposition.

Questions

pqaF-Q1. Evaluation Target and Vulnerable–Fixed Contrast

CheckerBench measures specification-conditioned checker synthesis. The agent receives a known vulnerability's patch and code context and must turn that information into an executable analyzer rule. Discovering the original, unknown vulnerability is not a prerequisite of this task. The vulnerable and fixed revisions provide a paired behavioral test: the checker should report the defect and recognize the safety condition introduced by the fix. This contrast tests the synthesized rule's behavior; it does not prove that the patch is correct in every respect.

The two revisions provide positive and negative cases for the same intended defect rule. Evaluating them together tests whether the synthesized checker distinguishes the defect from the safety condition introduced by the fix. We will clarify this objective in the task definition.

pqaF-Q2. Skill Availability and Tooling Improvements

(a) Actual skills and prompts.

The guidance has two forms. CSA uses named skills for patch interpretation, checker architecture, implementation, scoping, validation, and compilation recovery, together with analyzer references and reusable examples. CodeQL tasks provide language-specific API guidance, AST/control-flow and data-flow patterns, query metadata instructions, and a convergence protocol under references/codeql/. The task prompts specify when these resources should be consulted. These shared materials supply analyzer knowledge and development procedures; the defect-specific checker must still be synthesized.

(b) Anonymous release and the evaluated Full configuration.

We have added the original guidance as a directly browsable skills/ directory in the anonymous release, and updated the anonymous snapshot. It contains the original 24 named CSA skills and their supporting resources, plus the CodeQL guidance for all four languages: 127 files across five backend/language bundles. The files preserve the archived contents byte for byte. skills/manifest.json records hashes and sizes, and skills/task-map.csv maps every one of the 300 tasks to its guidance bundle and existing tasks/<dataset>/<task>/task_prompt.md. skills/README.md explains the contents and harness-loading paths. These materials are also available through the anonymous repository's ZIP download.

The Full configuration already includes this guidance. Its remaining failures therefore occur with the experimental guidance enabled.

(c) Better skills or richer MCP tooling.

Richer feedback, better context access, and improved skills are plausible routes to higher performance. We have not measured the gain from such an upgraded tool stack and will not treat the current best score as a capability ceiling. We will describe the results as performance under the specified resources and expose those resources so that subsequent improvements can be evaluated under a fixed verifier.

The existing component comparisons test removing guidance or feedback. They do not isolate the effect of higher-quality skills or an upgraded MCP tool stack; pqaF-W2 gives the results and their comparability limits.

pqaF-Q3. Reference Model in Table 2

GPT-5.6 serves only as the table's numerical reference. An increase relative to that row does not imply above-average or best performance, and for metrics such as false positives an increase is undesirable. The subscripts add avoidable interpretive burden, so we will remove the GPT-relative deltas, retain the absolute values and within-harness best/second-best markings, and keep the harness averages visible. This presentation makes the comparison direct without privileging an arbitrary reference model.

Weaknesses

pqaF-W1. Scope of Checker Reuse

The archived calibration checker synthesized for svglib's XML external-entity configuration defect (CVE-2020-10799) also detected the same defect type in fontTools (CVE-2023-45139). Applied without modification, it produced one alert on the vulnerable fontTools revision and none on the fixed revision. The query recognizes lxml.etree.XMLParser and its entity-resolution configuration across the two projects and calling contexts.

This case was selected after inspecting the existing checker and the target fix, and covers one known vulnerability in one target project. It demonstrates transfer for this checker and defect mechanism in the replayed environments.

Additional developer-confirmed defects found through cross-project scans are reported in 1TRv-Q1(b).

pqaF-W2. Failure Attribution and Component Ablations

The benchmark evaluates the combined model, harness, analyzer, and guidance system, so its failure rate cannot be assigned entirely to the underlying LLM.

The selected cohort of 4,360 failed runs is categorized as follows (Section 5.6):

Recorded failure categoryShare of selected failures (%)
Budget exhaustion25.32
Inadequate synthesis process22.96
Inadequate semantic modeling21.15
Excess false positives15.96
Differential detection or localization8.17
Patch-specific overfitting or anchoring3.56
Missing or invalid artifacts1.83
Compilation or registration1.05

These categories describe observed failure outcomes, not a causal decomposition into reasoning, environment, scaffold, and skill quality. For example, an inadequate semantic implementation could reflect either reasoning difficulty or insufficient analyzer guidance.

The component comparisons in the paper provide evidence for the value of the supplied support. Appendix H.2 reports the following results for DeepSeek-V4-Pro with OpenCode:

ConfigurationFull Pass@1 (%)Artifact pass (%)
Full40.3348.67
Without skills and strategy references8.3318.67
Without execution feedback13.3320.33
One-shot generation without tools0.331.33

Artifact pass removes both Process and normal completion while retaining semantic, anti-hardcoding, executable, and FP checks. It differs from the gate ablations in a4cA-Q2, which retain normal completion. The substantial gaps persist under this artifact metric. As Appendix H.2 explains, historical Full differs from the new runs in execution date, context limits, and workspace/tool isolation. These comparisons therefore characterize the tested configurations, without separately identifying the causal effects of environment, scaffolding, or skill quality.

The actual guidance and release status are detailed in pqaF-Q2. We will state the performance conclusions in terms of the complete evaluated system and distinguish observed failure categories from causal attribution.

Response: Evaluating Cross-Project Reuse of 30 Archived Checkers

Thank you for highlighting that the original reuse example involved only one selected checker and one target vulnerability. We extended the evaluation to 30 archived, formally generated checkers across five languages, with six checkers each for C/C++, Python, Java, JavaScript, and Go. The reported evaluation separates transfer to known vulnerabilities in external projects from prospective defect discovery in current project snapshots. Across 60 historical external targets, the checkers achieved 36 paired successes (60.0%); 22 of the 30 checkers (73.3%) transferred successfully to at least one target. The prospective evaluation yielded seven locally reproduced defects, including four potentially new defects.

Table 1. Reported cross-project transfer results for the 30 archived checkers.

LanguageCheckersVulnerable targets detectedFixed sites flaggedPaired successesCheckers with ≥1 successCheckers with 2 successes
C/C++673532
Python6101954
Java692853
JavaScript682642
Go681853
Total30429362214

The vulnerable-target detection rate is 42/60 = 70.0%, the fixed-site alert rate is 9/60 = 15.0%, and the paired transfer success rate is 36/60 = 60.0%. Since each checker has two valid targets, the macro-averaged per-checker paired success rate is also 60.0%. Fourteen checkers succeeded on both targets, eight succeeded on one, and eight succeeded on neither; therefore, 22/30 = 73.3% succeeded at least once.

Table 2. Reported prospective-discovery results on 30 current projects.

LanguageProjectsAssignmentsRaw alertsDistinct candidatesLocally reproduced defectsPreviously reportedPotentially newFalse positivesUnresolved
C/C++6185411021
Python6187531220
Java6184310111
JavaScript6184311011
Go6184310120
Total3090241873483

Across 90 checker-project assignments, the 24 raw alerts were deduplicated into 18 candidates. The reported adjudication identified seven locally reproduced defects, eight false positives, and three defect candidates. Of the seven reproduced defects, three had previously been reported and four were potentially new.

The four potentially new defects occurred in three projects and were detected by three distinct checkers: two Python defects in one project, one Java defect, and one Go defect.

Reviewer_a4cAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CheckerBench is a benchmark of 300 tasks (297 CVEs, 167 repos, 85 CWEs, 5 languages) in which agents must write a static-analysis checker (Clang Static Analyzer or CodeQL) that flags the vulnerable revision but not the fixed one. CheckerLab rebuilds each submitted checker in a clean environment and scores diagnostic contrast, patch localization, mainline false positives, and LLM-judged process and semantic quality. Across 21 model–harness pairs with three repeats each, mean Pass@1 is 30.81% and the best is 40.67%.

Critical strengths
  • It is the first benchmark for end-to-end checker synthesis, verified by an independent held-out rebuild that checks vulnerable–fixed contrast and mainline false positives, not just patch-level success.
  • The evaluation is broad and repeated (21 configurations × 3 runs on all 300 tasks), which is stronger than most agent benchmarks.
Critical weaknesses
  • Tasks were kept only if a human-guided calibration agent passed them within three attempts with at least 40 steps, which biases the benchmark toward tasks solvable by that setup and may favor its model and harness.

  • Passing depends partly on LLM-judged process and semantic scores, so Pass@1 mixes checker correctness with subjective trajectory quality.

Questions for authors
  • Which model and harness were used for calibration, and how do results change if that configuration is excluded?
  • How much does Pass@1 change if the process-quality gate is removed and only executable and false-positive criteria are used?
  • With 107 of 300 tasks from Linux and analyzer tied to language, can you report results that separate language from analyzer and reweight by repository?
5 comments

Response to Reviewer a4cA

We thank you for your constructive feedback and will revise the paper to improve clarity and exposition.

Questions

a4cA-Q1. Calibration Configuration and Exclusion Analysis

The checker-generating agent used DeepSeek-V4-Flash-0424 with a Claude Code agent. We chose Flash to keep the cost and turnaround time of repeated calibration attempts manageable, and Claude Code to reuse our existing workflow for repository inspection, checker editing, compilation, and analyzer execution. Each candidate may require multiple inspect–implement–compile–scan–repair cycles, making cost, speed, and workflow reuse practical considerations during benchmark construction.

The role of calibration is to establish feasibility in the pinned environment and collect accepted checker–trajectory pairs. We fix the model and harness versions, tools, and per-attempt budgets so that candidates undergo a consistent construction procedure. Each submitted checker must pass the frozen, independent verifier before admission. Each candidate permits at most three attempts in total: one initial rollout and up to two retries following human review of verification failures, stopping at the first pass.

DeepSeek-V4-Flash is not one of the seven models in Table 2, so the exact calibration model–harness pair is already absent from the reported comparison.

The calibration setup does not rely on a flagship model or the strongest evaluated harness. DeepSeek-V4-Flash is the smaller, efficiency-oriented V4 variant, and Claude Code's mean Pass@1 is 32.62%, below OpenCode's 34.10%. Even this practical configuration yields independently verified solutions for every retained task through human-guided calibration, while the best autonomous configuration reaches only 40.67% Pass@1. This supports CheckerBench's effectiveness in measuring the gap between demonstrated task feasibility and reliable autonomous checker development.

We checked whether the average Pass@1 remains low when a model or harness is excluded. Starting from Table 2, we separately removed all configurations using DeepSeek-V4-Pro, all configurations using Claude Code. For each exclusion, we summed the remaining configurations' reported Pass@1 rates and divided by the number of configurations retained:

Configurations retainedNumber of configurationsMean Pass@1 (%)
All reported configurations2130.81
Excluding DeepSeek-V4-Pro across all harnesses1829.52
Excluding Claude Code across all models1429.90

a4cA-Q2. Pass@1 Sensitivity to Evaluation Gates

(a) Removing Process and semantic gates.

We conducted a systematic supplementary evaluation covering all 21 model–harness configurations and 6,300 task runs. Using the saved evidence for each run, we evaluated the full criteria and three ablations: removing Process only, removing Semantic only, and removing both. The submitted artifacts, budgets, and normal-completion rules were held fixed.

Acceptance criteriaAccepted / 6,300Success rate (%)Change (percentage points)
Original full criteria1,94030.79—
Remove Process only2,301–2,30336.52–36.56+5.73–5.76
Remove Semantic only2,844–2,86245.14–45.43+14.35–14.63
Remove both Process and Semantic3,595–3,61957.06–57.44+26.27–26.65

All three ablations retain independent artifact, compilation, differential/localization, FP, and anti-overfitting checks, including hardcoding, project-specific scope, and anchoring checks. Semantic denotes the backend-specific semantic-quality requirements, including mechanism adequacy and semantic prerequisites, rather than the numerical score alone.

In this archived cohort, removing Semantic alone produces a larger increase than removing Process alone. Consequently, the individual gains cannot simply be added: the gates reject overlapping sets of submissions. These results support interpreting full Pass@1 as success under combined development and artifact requirements, with the component criteria reported separately.

For the broader executable-plus-FP endpoint requested in the question, additionally removing judge-based hardcoding, scope, and anchoring checks yields 63.38%–63.78% (3,993–4,018/6,300). This is a separate, more permissive endpoint than removing Process and Semantic alone.

(b) FP-threshold sensitivity.

Separately, we varied an additional all-recorded-alert FP cutoff while holding all other original gates fixed. This uses the 1,940 original passes and their 1,335 audited alerts; all original failures remain failures. Zero-alert checkers retain their original eligibility.

FP cutoff (%)102025304050
Success rate (%)29.3529.4429.5129.5429.7029.90

The 25% cutoff retains 1,859 passes. Relaxing this cutoff from 10% to 50% adds 35 passes, or 0.56 percentage points.

a4cA-Q3. Language–Analyzer Coupling and Repository Reweighting

Both requests can be met, though to different degrees. We address repository weighting directly by reporting results under multiple weighting schemes. Language and analyzer cannot be separated in the task-equal aggregate, because the backend is determined by language: all 159 C/C++ tasks run on CSA, and the remaining 141 tasks, spanning four other languages, run on CodeQL. We therefore separate the two factors wherever the design permits, and add a bounded same-task comparison for the stratum where they co-vary.

Two consequences follow from this nesting. The analyzer effect is not identifiable from the aggregate, because a same-task estimate would require implementing each task under both backends, which no submitted checker can supply. The expanded Appendix H.4 reports Pass@1 per language with CodeQL held fixed: for DeepSeek-V4-Pro on OpenCode, the difference between the highest and lowest Pass@1 across the four CodeQL languages is 15.49 percentage points. The observed ordering of the 21 configurations varies across language strata. The 159/141 split therefore acts mainly as a difference in difficulty distribution between the two backend strata, rather than as evidence that one language is intrinsically harder at a fixed backend.

To keep the analyzer effect from remaining purely qualitative, we conducted a bounded same-task comparison. A subset of 40 C/C++ tasks, stratified by CWE family and repository size, was re-implemented under CodeQL from the same task prompt, guidance bundle, and acceptance criteria, with one fixed configuration (DeepSeek-V4-Pro on OpenCode). Because the tasks, generator, and verifier were held fixed and only the backend changed, the paired difference is a same-task backend effect for one language pair; we report it as such and do not extrapolate it to the four CodeQL languages.

For repository weighting, Linux is a single repository that supplies 107 of the 300 tasks. Task-equal weighting therefore assigns 35.7% of total weight to one repository, while repository-equal weighting counts it like a repository that contributes a single task. Neither scheme is neutral, so we report the family and read the differences as follows:

Weighting schemeQwen3.8-Max (%)DeepSeek-V4-Pro (%)Qwen − DeepSeek (pp)
Equal weight per task (300 tasks)34.3338.56−4.23
Equal weight per repository (167 repositories)50.5747.29+3.28
Excluding Linux (193 tasks in 166 repositories)46.8044.04+2.76

Both corrections raise every reported rate and reverse the sign of the Qwen–DeepSeek difference. DeepSeek's task-weighted lead of 4.23 pp is carried by the Linux repository; away from it the two configurations lie within 2.8–3.3 pp of each other. We will therefore present these two configurations without an ordering claim, describe them as indistinguishable, and keep the task-weighted rates as the primary measure with the repository-weighted and Linux-excluded rates reported alongside. Table 2 will be labelled as task-weighted, with the 107-task Linux share stated in its caption. The paper's conclusions concern the level of performance under the stated resources rather than a fine-grained ranking, so this restatement leaves them unchanged; what changes is that we no longer present a pairwise ordering that depends on one repository.

These reweightings also change the contribution of the language, analyzer, and CWE strata to each reported rate. The revision will report that decomposition stratum by stratum in Appendix H.4, and will state in Section 5 that analyzer–language separation and repository weighting are identification limits, giving the measured consequence of each rather than the caveat alone.

Weaknesses

a4cA-W1. Selection Bias in Benchmark Construction

The construction procedure conditions the task distribution on one calibration setup, and the direction of the resulting bias is definite: candidates that DeepSeek-V4-Flash with Claude Code could not resolve within three attempts were dropped, so the retained 300 tasks are those a comparatively inexpensive configuration could be guided to solve. Reported Pass@1 is therefore an upper bound on performance over the broader population of checker-synthesis tasks, not an estimate of that population, and we will state this where the benchmark is introduced.

Section 3.1 and Appendix B.4 give the procedure behind that bound: one initial attempt, at most two retries after a human review of the verification evidence, and admission only with a passing verifier and an accepted trajectory of at least 40 interaction steps. The procedure establishes feasibility in the pinned environment and selects sustained calibration interaction. It does not establish that every solver needs 40 steps, nor that the retained set spans the difficulty range of the excluded tasks; the 40-step threshold is a curation condition on accepted trajectories, not a property of the task.

The exclusion analysis in a4cA-Q1 shows that the low mean persists across the remaining seven systems, which bounds how much of the result can come from the calibration configuration alone, but does not remove construction-time bias. The revision will separate the generator, runner, and feedback-planner roles, distinguish human-guided construction from autonomous evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.