- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies crash-guided synthesis of standalone C reproducers for Linux kernel bugs, using sanitizer reports, execution logs, and revision-matched source code without providing an existing reproducer or security patch as generation input. Crash2PoC combines evidence-checked crash localization, static candidate-trace recovery with LLM filtering, and agent-based PoC synthesis followed by execution in a matched QEMU environment. The authors report 264 successful reproductions among 337 KASAN cases (78.3%). They also report 8/20 successes for mini-SWE-agent and lower success rates after removing localization or static analysis, although the baseline results do not constitute a controlled common-case comparison.
Critical strengths
-
A practically meaningful task and input setting. Generating standalone kernel reproducers from crash artifacts without requiring a fix patch is useful for diagnosis and regression testing. The reported evaluation on 337 real syzbot cases gives the work practical relevance beyond a small collection of illustrative examples.
-
A sensible separation of reasoning and external checks. Revision locking, literal-evidence checks, and source-location validation provide concrete safeguards against unsupported intermediate outputs. The design is a useful way to constrain an agent's context before synthesis, and compilation plus execution is an appropriate basis for evaluating generated programs.
Critical weaknesses
-
The central trace-recovery mechanism is not specified sufficiently to assess its technical contribution or reproduce it. Section 3.3 explicitly notes that runtime addresses cannot directly identify static source locations, yet Section 3.4 treats these addresses as static taint sources without explaining the mapping to source/IR entities or heap-object abstractions. The analyzer, propagation rules, treatment of aliases and indirect calls, and bounds on trace enumeration are unspecified. The submission also does not independently assess the LLM filter's root-cause decisions. Since related work already combines LLMs, program-analysis context, and execution, the missing kernel-specific mechanism prevents a convincing assessment of the contribution beyond system integration.
-
The evaluation does not establish the incremental benefit or make the reported success criterion auditable. Table 1 compares 337 Crash2PoC cases with only 20 baseline cases and provides no paired outcomes; the authors appropriately acknowledge this limitation. Table 2 omits sample sizes and absolute counts, reports 78.4% where the text gives 78.3%, and does not explain how NoLocalization supplies the inputs required by static analysis. Moreover, although success must reproduce the target vulnerability, no operational matching procedure is provided to distinguish the same root cause from another crash. These omissions prevent assessment of both the reported success rate and the component-level attribution; stating that budgets are equal without giving their values further limits reproducibility.
Questions for authors
-
What static analyzer is used, and how is a concrete KASAN fault address or memory-state row base mapped to a source/IR-level taint source? Please provide an example tracing this mapping through candidate-path extraction, including the analysis rules and enumeration bounds.
-
What exact procedure determines that a generated PoC triggers the target vulnerability? Please explain the evidence used beyond crash-category agreement, any manual or differential checks, and how reference reproducers and fixes are kept inaccessible to generation. Representative original/generated crash logs would make this criterion assessable.
-
Are the 20 baseline cases included in the 337-case dataset, how were they selected, and what are the paired Crash2PoC outcomes on those same cases? What numerical execution, interaction, token, and retry budgets were applied to both systems?
-
What are the case sets, absolute success counts, and aggregation protocol for every row of Table 2? Please reconcile 78.4% with 78.3% and explain what supplies the bug function and taint inputs when localization is removed. If all rows use one binary outcome for each of 337 cases, none of the three printed percentages follows from integer counts rounded to one decimal.
Thank you for pressing us on what Crash2PoC contributes and how much of its performance can be attributed to that contribution. The design is visible in the evidence required for each localization decision. Measuring its incremental benefit requires a controlled comparison.
Q1. Technical novelty (996H)
We respectfully disagree that orchestration alone captures the contribution. K-Repro and SyzHarness start from a fix patch (Section 2.3). Our task starts from a crash report, log and kernel revision. Without a patch as a designated input, what justifies accepting a proposed location? Section 3.3 requires literal support in the report or log and agreement with the source at that revision. Even a function named in the report fails the check if the cited line falls outside its definition. This acceptance rule makes the agent’s interpretation independently inspectable. We will foreground it in the contribution statement.
Q2. Static analysis and trace filtering (FWBC, 996H)
The design distinguishes a crash site we can check against source from a candidate causal path the LLM judges consistent with the report. We agree that Section 3.4 does not adequately explain how the analyzer obtains its inputs. We will use an annotated example to document the actual correspondence, the analyzer, its propagation rules, its treatment of aliases and indirect calls, and its path limits.
Q3. What counted as a successful reproduction (BeNT, FWBC)
We agree that the paper should state the rule. The program had to compile and run without manual edits in the specified environment. An LLM judge compared the original report with the QEMU serial log. Acceptance required a genuine kernel error, a match on both sanitizer and bug category, and a corresponding crashing function or source location. Inlining, wrapper or callee differences were accepted only when the reports supported the same fault.
We have no separate non-target crash count or independent manual or fixed-kernel validation rate to report. We will distinguish this judgment from deterministic localization checks and illustrate it with paired log excerpts.
Q4. Baseline comparability and budgets (996H, BeNT, FWBC)
We agree that Table 1 cannot establish an advantage over mini-SWE-agent. All 20 baseline cases belong to the 337-case cohort. Table 3 lists one per KASAN category. Eight reproduced, ten did not reproduce and two could not execute, giving 8/20 (40.0%). We omitted the selection rule within each category, the rationale for stopping at 20 and Crash2PoC’s outcomes on those cases. Section 4.2’s equal-budget statement also leaves the baseline’s “one-round” feedback access unclear. We will check the records for the selection rationale, paired outcomes, feedback access, and execution, interaction, token and retry limits. A matched comparison on all 337 cases remains planned.
Q5. Ablation statistics and reporting corrections (996H, FWBC)
The reviewers are correct. The full-system count of 264/337 rounds to 78.3%. The 78.4% in Table 2 is a transcription error. For each verifiable row, we will report case membership, successes, denominator, number of runs and aggregation rule. We will withdraw unsupported effect sizes, align the table formats and use distinct symbols in Eq. (2) for the complete output and the generated program.
Q6. What the ablations establish (996H, BeNT, FWBC)
We agree that NoLocalization removes a whole stage. Section 4.4 leaves unspecified which inputs the downstream analysis received in that run. We will clarify their provenance. Table 2 also lacks comparisons that isolate the three localization checks or measure filtering on a fixed set of candidate traces. The current rows support only a comparison of configurations.
Q7. Direct evaluation of root cause analysis (BeNT, FWBC)
Root cause analysis is part of the method because the system judges whether a candidate explanation fits a reported failure. We agree that end-to-end success cannot tell us how often this judgment is correct. A verified crash site can be downstream of the defect. Comparing the localized function with functions touched by the fixing commit, as BeNT suggests, would provide an independent check. A fix can change a caller or several functions, so disagreements would require review.
Q8. Reference access and training contamination (BeNT)
We agree that both risks affect the headline result. We clarify that, during PoC generation, the agent was not provided with any web search tools; it only had access to basic local file access and code execution tools. Consequently, the agent could not actively retrieve publicly available reference reproducers or fixes through a web search tool. We currently have no success-rate data for bugs fixed after a verified GLM-5.3 training cutoff date. We will add a corresponding evaluation.