- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents OptFuzz, a fuzzer for language processors such as compilers and interpreters. The main contribution is that when coverage stops growing, OptFuzz calls one of two LLM components:
- The Breadth Optimizer scores each region in the codebase by how much code is still uncovered, how large the region is, and how past attempts went, then prompt the LLM to generate programs that triggers those region. The LLM is also used to repair failed programs over several rounds using compiler error messages.
- The Depth Optimizer picks which hard-to-cover "deep" branch to target. It frames this choice as a LinUCB bandit with 6 features to schedule. OptFuzz source code around the branch, symbol lookups, and execution feedback to con, and repeats. For five 24-hour experiments, OptFuzz better coverage and bug detection than other fuzzers. The authors run OptFuzz for a month and report 74 previously unknown bugs,57 are confirmed by the maintainers.
Critical strengths
-
A lot real-world zero-day bug findings. A month-long fuzzing experiments on 8 widely used language processors found 74 previously unknown bugs, 57 of them confirmed by maintainers, is a solid contribution to the security community.
-
Well-motivated context constructions tp use the LLM for different exploration tasks (breathe and depth). Breadth prompts describe a poorly covered region of the source tree, and depth prompts carry the source code around one bottleneck branch, symbol lookups, and execution feedback. I find this design motivating, since reaching a new region and flipping a single branch towards a deeper state need different information. The ablations show the benefit of this design clearly.
Critical weaknesses
- The paper present poorly on the design details. The main contribution is how the two optimizers work with the fuzzer, but the paper never describe the following in an understandable way: how SelectMode decide when stagnated fuzzing loop is a breathe or depth
- The paper says the LLM runs alongside the fuzzer and communicate async, which keeps fuzzing over 99% of the time, but Algorithm 1 calls the LLM inside the fuzzing loop. As a result, the method cannot be reproduced, and the gains cannot be traced to specific design choices.
- Several symbols in section 4 and 5 lack proper definition, making the methodology hard to understand and hence cannot be mentally evaluated clearly.
- The evaluation is far from complete.
- There is no justificate that the improve comes from the design instead of AFL++ with a grammar-based mutator (i.e., ATLR4 used in the paper).
- Only one LLM-based baseline is too thin to support the claim. There are quite a few well-known published LLM-based fuzzers with public code cover the same kinds of targets, for example Fuzz4All (ICSE'24), WhiteFox (OOPSLA'24), and CovRL (ISSTA'24). The authors should at least provide justification why HLPFUZZ is the only one selected.
- The paper does not say how it checked that the 74 bugs were previously unknown. It fuzzed release versions (Clang 22.1.1, Lua 5.5.0, and others). A month of fuzzing on release versions usually finds many bugs that are already reported or already fixed on the development branch. It seems like either the authors only discovered zero-day bugs, or did not report the other bugs.
Questions for authors
- Is the LLM layer a separate process running alongside AFL++, or does it run inline as in Algorithm 1? If it is separate, what explains the drop from 177.8 to ~45 exec/s in Table 5, and how was "> 99% of the time" measured?
- Please define
SelectModeexactly (inputs, rule, thresholds,N_stag) andExtractBottlenecks(what counts as a bottleneck, and the candidate-set size). Have you compared the switching rule against fixed alternation or random mode choice? - What is the coverage of AFL++ with the ANTLR4 mutator and no LLM?
- For the month-long campaign: what was the setup (LLM backend, number of instances, LLM budget)? Please give the full breakdown: unique crashes → known or duplicate → reported → confirmed / rejected / pending → fixed / CVE. How was zero-day status checked (re-run on the latest development version, or search of the issue trackers)?
- How is each target split into regions (depth, count, and how files outside the main source tree are handled)? How many regions do Lua and njs have? Do LLM seeds increase coverage in the chosen region more than elsewhere?
- How much past feedback does Listing 1's
FEEDBACK HISTORYhold, and is old feedback truncated? In the Depth Optimizer loop, what is the sliding-window size, and what is the maximum number of rounds and tokens per attempt?
3 comments
Follow-up on Our Response to Your Review
Dear Reviewer ZgaP,
Thank you again for your time and valuable feedback. As the discussion period ends soon, we would like to kindly check whether our response has addressed your concerns.
If any points remain unclear, we would be happy to provide further clarification or additional results before the deadline. We hope our response helps address your concerns, and we would be grateful if you could take it into account in your assessment.
We understand that you may have a busy schedule, and we would sincerely appreciate it if you could find the time to review and respond to our reply.
Best regards,
Authors of Submission #33339
Thank you for the response and revision. The notation and hyperparameter tables in Appendix D resolve my concern about undefined symbols.
- Asynchrony and throughput. Algorithm 1 still calls the LLM inside a single sequential loop. The abstract, section 3, and conclusion still say the method "preserves high throughput." The authors give no measurements to support their explanation for the 4× throughput drop.
- Baselines. Fuzz4All and CovRL are relevant baselines here because they were evaluated on Clang/Clang++ and JavaScript engines, respectively. The authors' generic harness cannot substitute for an independent baseline. The revision mentions the AFL++ with ANTLR4 control without showing results.
- Bug accounting. The revision omits the requested breakdown and leaves open how the bugs were checked against existing reports. The new package contains 4 PoCs built against release versions. Table 1 reports bug counts as "per-run means," while section 7.2 and Table 2 call the same 15 bugs "unique."
My questions about the campaign setup, the region partition for Lua and njs, and the feedback-history size remain unanswered. Every new mean and standard deviation in Tables 2 and 5 ends in 0 or 5. Please provide the per-run values too. I will keep my original score.
Response to Reviewer ZgaP
We thank the reviewer for the careful reading. Revisions will be highlighted in the revised paper.
W1/Q2: SelectMode and ExtractBottlenecks.
Action: We have defined both components in Secs. 3 and 5.1, moved Algorithm 1 to Appendix A, and added the ablation to Sec. 7.4.
W2/Q1, W4.1/Q3: Asynchrony, throughput, ANTLR4 control. Algorithm 1 was misleading, and the ">99%" claim was inconsistent with our throughput numbers.
Action: We have revised Sec. 3 (async architecture description), moved Algorithm 1 to Appendix A with an asynchronous architecture note, rewritten the throughput discussion in Sec. 7.5, and stated the shared-mutator control in Sec. 7.1.
W3: Undefined symbols. Zij is the CFG adjacency matrix (Zij=1 if there is an edge i→j, else 0). The symbol α was overloaded (Eq. 1 weight, Katz attenuation, LinUCB exploration), so each now has its own symbol, and we separate xt from xc. We add a notation table and a default hyperparameter table: w1=w2=1, v1=v2=0.5, γ=3.0, Bbase=1.0, K=3, Katz attenuation 0.1, LinUCB exploration 1.0.
Action: We have revised Secs. 4–5 and added both tables in Appendix D.
W4.2: Choice of LLM-based baseline. We chose HLPFuzz because it is the only published LLM-based method targeting the same objects and domain as OptFuzz: language processors, with a hybrid design that uses the LLM for bottleneck-branch solving. Fuzz4All, WhiteFox, and CovRL were designed for different targets and settings, so a comparison on our targets would not be like-for-like. To avoid relying on one baseline, we also built a generic LLM agent harness (LLM on stagnation plus symbol lookup, no scheduling) with the same model and seeds. On Clang++ it reaches 82,315 ± 2,840 edges, versus 94,782 ± 3,025 for HLPFuzz and 104,537 ± 3,382 for OptFuzz, at similar cost ($0.55–0.59/h). The gains therefore come from scheduling and repair, not from more queries.
Action: We have added the rationale and harness comparison to Secs. 7.1–7.2.
W4.3/Q4: Novelty of the bugs. We replace "zero-day" with "previously unknown bugs." If a target has a development trunk, we reproduce each bug on the trunk. Otherwise we verify on the latest release and submit the report to the public community for discussion with the developers. Representative bugs, configuration instructions, and reproduction steps are uploaded to the anonymous OSF repository.
Action: We have revised the terminology throughout, added the procedure and separate counts to Sec. 7.3, and added bug descriptions to the anonymous OSF repository.
Q5/Q6: Regions and context. As stated in Sec. 4.1, we partition each target by second-level directories (e.g.,
clang/lib,llvm/IR). Coarser regions lower the LLM success rate, while finer ones no longer reflect breadth. We use no context-compression tricks and set the token limit to each model's upper limit.Action: We have made both choices explicit in Sec. 4.1 and Appendix D.