acceptodds
Withdrawn from ICLR 2027

OptFuzz: Rethinking LLM-Assisted Fuzzing from Stateless Generation to Closed-Loop Semantic Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) can generate syntactically valid test cases, but existing LLM-assisted fuzzers mostly treat them as stateless, single-shot generators. For critical infrastructure systems with deep semantic constraints—such as compilers, runtimes, and bytecode interpreters—this design lacks global search memory, architectural state tracking, and iterative error recovery, leaving exploration perpetually stuck on coverage plateaus. In this paper, we propose OptFuzz, a novel hybrid fuzzing framework that reframes LLM-assisted fuzzing into hierarchical closed-loop semantic optimization. OptFuzz preserves the high throughput of coverage-guided fuzzing (CGF) while selectively invoking the LLM when coverage stagnation is detected. We devise two collaborative optimization engines: the Breadth Optimizer estimates dynamic region values combining static potential metrics and historical solving feedback to steer global state-space exploration toward under-covered logic; the Depth Optimizer models hard bottleneck branch penetration as a contextual multi-armed bandit (LinUCB) sequential decision process over multimodal structural features, iteratively repairing and approximating branch constraints via compilation diagnostics and execution traces. Evaluated across 8 industrial-grade language processors (including Clang, Flang, Hermes, ChakraCore, and Lua), OptFuzz improves edge coverage by up to 115.9% over state-of-the-art fuzzers and discovers 74 real-world vulnerabilities (57 confirmed by upstream maintainers), proving the immense security value of closed-loop semantic optimization. Our code and artifacts are publicly available at https://osf.io/5g2ez/overview?view_only=9a341081ce104a60ad8b3102a7ebb430.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fuzzing, Coverage-Guided Fuzzing (CGF), Large Language Models (LLMs), Compiler Testing, Closed-loop Semantic Optimization, Contextual Multi-Armed Bandit (LinUCB), Vulnerability Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZgaPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents OptFuzz, a fuzzer for language processors such as compilers and interpreters. The main contribution is that when coverage stops growing, OptFuzz calls one of two LLM components:

  • The Breadth Optimizer scores each region in the codebase by how much code is still uncovered, how large the region is, and how past attempts went, then prompt the LLM to generate programs that triggers those region. The LLM is also used to repair failed programs over several rounds using compiler error messages.
  • The Depth Optimizer picks which hard-to-cover "deep" branch to target. It frames this choice as a LinUCB bandit with 6 features to schedule. OptFuzz source code around the branch, symbol lookups, and execution feedback to con, and repeats. For five 24-hour experiments, OptFuzz better coverage and bug detection than other fuzzers. The authors run OptFuzz for a month and report 74 previously unknown bugs,57 are confirmed by the maintainers.
Critical strengths
  1. A lot real-world zero-day bug findings. A month-long fuzzing experiments on 8 widely used language processors found 74 previously unknown bugs, 57 of them confirmed by maintainers, is a solid contribution to the security community.

  2. Well-motivated context constructions tp use the LLM for different exploration tasks (breathe and depth). Breadth prompts describe a poorly covered region of the source tree, and depth prompts carry the source code around one bottleneck branch, symbol lookups, and execution feedback. I find this design motivating, since reaching a new region and flipping a single branch towards a deeper state need different information. The ablations show the benefit of this design clearly.

Critical weaknesses
  1. The paper present poorly on the design details. The main contribution is how the two optimizers work with the fuzzer, but the paper never describe the following in an understandable way: how SelectMode decide when stagnated fuzzing loop is a breathe or depth
  2. The paper says the LLM runs alongside the fuzzer and communicate async, which keeps fuzzing over 99% of the time, but Algorithm 1 calls the LLM inside the fuzzing loop. As a result, the method cannot be reproduced, and the gains cannot be traced to specific design choices.
  3. Several symbols in section 4 and 5 lack proper definition, making the methodology hard to understand and hence cannot be mentally evaluated clearly.
  4. The evaluation is far from complete.
    1. There is no justificate that the improve comes from the design instead of AFL++ with a grammar-based mutator (i.e., ATLR4 used in the paper).
    2. Only one LLM-based baseline is too thin to support the claim. There are quite a few well-known published LLM-based fuzzers with public code cover the same kinds of targets, for example Fuzz4All (ICSE'24), WhiteFox (OOPSLA'24), and CovRL (ISSTA'24). The authors should at least provide justification why HLPFUZZ is the only one selected.
    3. The paper does not say how it checked that the 74 bugs were previously unknown. It fuzzed release versions (Clang 22.1.1, Lua 5.5.0, and others). A month of fuzzing on release versions usually finds many bugs that are already reported or already fixed on the development branch. It seems like either the authors only discovered zero-day bugs, or did not report the other bugs.
Questions for authors
  1. Is the LLM layer a separate process running alongside AFL++, or does it run inline as in Algorithm 1? If it is separate, what explains the drop from 177.8 to ~45 exec/s in Table 5, and how was "> 99% of the time" measured?
  2. Please define SelectMode exactly (inputs, rule, thresholds, N_stag) and ExtractBottlenecks (what counts as a bottleneck, and the candidate-set size). Have you compared the switching rule against fixed alternation or random mode choice?
  3. What is the coverage of AFL++ with the ANTLR4 mutator and no LLM?
  4. For the month-long campaign: what was the setup (LLM backend, number of instances, LLM budget)? Please give the full breakdown: unique crashes → known or duplicate → reported → confirmed / rejected / pending → fixed / CVE. How was zero-day status checked (re-run on the latest development version, or search of the issue trackers)?
  5. How is each target split into regions (depth, count, and how files outside the main source tree are handled)? How many regions do Lua and njs have? Do LLM seeds increase coverage in the chosen region more than elsewhere?
  6. How much past feedback does Listing 1's FEEDBACK HISTORY hold, and is old feedback truncated? In the Depth Optimizer loop, what is the sliding-window size, and what is the maximum number of rounds and tokens per attempt?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZgaP

We thank the reviewer for the careful reading. Revisions will be highlighted in the revised paper.

W1/Q2: SelectMode and ExtractBottlenecks.

  • Trigger: the LLM layer is invoked when new-edge growth over a 90 s window falls below (total edges × 10⁻⁴). This replaces the iteration-based NstagN_{stag} in Algorithm 1.
  • SelectMode: it mirrors how analysts study programs, broad understanding first and depth second. Breadth is the default. The system switches to Depth after 3 consecutive breadth failures in a region, and returns to Breadth after 5 consecutive failures on one constraint, which is then deprioritized.
  • ExtractBottlenecks: AFL++ exports only the coverage bitmap and the edge-to-block map. Uncovered direct successors of covered blocks are ranked by Katz centrality on the reversed uncovered subgraph and aggregated to function/line level. The top-5 form the bandit's arm set.
  • Ablation (Clang++, 24h, 5 runs, same setup), edges / bugs: OptFuzz 104,537 ± 3,382 / 8; Fixed Alternation 93,274 ± 2,953 / 6; Random Mode 94,946 ± 3,482 / 6. Both alternatives reach about 9–11% less coverage.

Action: We have defined both components in Secs. 3 and 5.1, moved Algorithm 1 to Appendix A, and added the ablation to Sec. 7.4.

W2/Q1, W4.1/Q3: Asynchrony, throughput, ANTLR4 control. Algorithm 1 was misleading, and the ">99%" claim was inconsistent with our throughput numbers.

  • Architecture: the LLM layer and AFL++ run as two separate processes, and generated tests are enqueued asynchronously. Algorithm 1 was a logical view; it now shows both processes and the queue.
  • Throughput: the drop from 177.8 to ~45 exec/s occurs because the targets must compile LLM-generated test cases, which often crash or abnormally terminate the target and reduce AFL++'s executions. We withdraw the ">99%" and "preserves throughput" statements. The 45.6 exec/s is an end-to-end 24h figure on Clang++, compared only with HLPFuzz, which shares the same pipeline.
  • ANTLR4 control: all reported experiments, including AFL++ and HLPFuzz, use the same ANTLR4 syntax-aware mutator, a standard practice in compiler fuzzing. The AFL++ rows are therefore already "AFL++ + ANTLR4, no LLM" (Clang++ 56,029; Flang 69,843; Hermes 13,947; Lua 3,395). The only difference from OptFuzz is the LLM layer, so the gains cannot come from the grammar mutator. The OSF quick-start commands reproduce this setup.

Action: We have revised Sec. 3 (async architecture description), moved Algorithm 1 to Appendix A with an asynchronous architecture note, rewritten the throughput discussion in Sec. 7.5, and stated the shared-mutator control in Sec. 7.1.

W3: Undefined symbols. ZijZ_{ij} is the CFG adjacency matrix (Zij=1Z_{ij}=1 if there is an edge i→ji\to j, else 0). The symbol α\alpha was overloaded (Eq. 1 weight, Katz attenuation, LinUCB exploration), so each now has its own symbol, and we separate xtx_t from xcx_c. We add a notation table and a default hyperparameter table: w1=w2=1w_1=w_2=1, v1=v2=0.5v_1=v_2=0.5, γ=3.0\gamma=3.0, Bbase=1.0B_{base}=1.0, K=3K=3, Katz attenuation 0.1, LinUCB exploration 1.0.

Action: We have revised Secs. 4–5 and added both tables in Appendix D.

W4.2: Choice of LLM-based baseline. We chose HLPFuzz because it is the only published LLM-based method targeting the same objects and domain as OptFuzz: language processors, with a hybrid design that uses the LLM for bottleneck-branch solving. Fuzz4All, WhiteFox, and CovRL were designed for different targets and settings, so a comparison on our targets would not be like-for-like. To avoid relying on one baseline, we also built a generic LLM agent harness (LLM on stagnation plus symbol lookup, no scheduling) with the same model and seeds. On Clang++ it reaches 82,315 ± 2,840 edges, versus 94,782 ± 3,025 for HLPFuzz and 104,537 ± 3,382 for OptFuzz, at similar cost ($0.55–0.59/h). The gains therefore come from scheduling and repair, not from more queries.

Action: We have added the rationale and harness comparison to Secs. 7.1–7.2.

W4.3/Q4: Novelty of the bugs. We replace "zero-day" with "previously unknown bugs." If a target has a development trunk, we reproduce each bug on the trunk. Otherwise we verify on the latest release and submit the report to the public community for discussion with the developers. Representative bugs, configuration instructions, and reproduction steps are uploaded to the anonymous OSF repository.

Action: We have revised the terminology throughout, added the procedure and separate counts to Sec. 7.3, and added bug descriptions to the anonymous OSF repository.

Q5/Q6: Regions and context. As stated in Sec. 4.1, we partition each target by second-level directories (e.g., clang/lib, llvm/IR). Coarser regions lower the LLM success rate, while finer ones no longer reflect breadth. We use no context-compression tricks and set the token limit to each model's upper limit.

Action: We have made both choices explicit in Sec. 4.1 and Appendix D.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer ZgaP,

Thank you again for your time and valuable feedback. As the discussion period ends soon, we would like to kindly check whether our response has addressed your concerns.

If any points remain unclear, we would be happy to provide further clarification or additional results before the deadline. We hope our response helps address your concerns, and we would be grateful if you could take it into account in your assessment.

We understand that you may have a busy schedule, and we would sincerely appreciate it if you could find the time to review and respond to our reply.

Best regards,

Authors of Submission #33339

Thank you for the response and revision. The notation and hyperparameter tables in Appendix D resolve my concern about undefined symbols.

  1. Asynchrony and throughput. Algorithm 1 still calls the LLM inside a single sequential loop. The abstract, section 3, and conclusion still say the method "preserves high throughput." The authors give no measurements to support their explanation for the 4× throughput drop.
  2. Baselines. Fuzz4All and CovRL are relevant baselines here because they were evaluated on Clang/Clang++ and JavaScript engines, respectively. The authors' generic harness cannot substitute for an independent baseline. The revision mentions the AFL++ with ANTLR4 control without showing results.
  3. Bug accounting. The revision omits the requested breakdown and leaves open how the bugs were checked against existing reports. The new package contains 4 PoCs built against release versions. Table 1 reports bug counts as "per-run means," while section 7.2 and Table 2 call the same 15 bugs "unique."

My questions about the campaign setup, the region partition for Lua and njs, and the feedback-history size remain unanswered. Every new mean and standard deviation in Tables 2 and 5 ends in 0 or 5. Please provide the per-run values too. I will keep my original score.

Reviewer_TbjnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OptFuzz combines AFL++, syntax-aware mutation, LLM generation, and execution feedback. Upon coverage stagnation, a Breadth Optimizer selects regions using coverage gaps, code complexity, and historical outcomes. A Depth Optimizer uses six-dimensional features and LinUCB to select bottleneck branches, then iterates with code slices, symbol retrieval, compilation diagnostics, and execution feedback.

The controlled evaluation covers Clang++, Flang, Hermes, and Lua, with five 24-hour trials per configuration, reporting approximately 10.2%–42.4% coverage gains over HLPFuzz. A separate month-long campaign across eight targets reportedly discovers 74 previously unknown vulnerabilities, 57 confirmed by maintainers. Component ablations, alternative schedulers, and model cost/throughput comparisons supplement these results.

Critical strengths

S1. A practical system design addressing genuine semantic bottlenecks. Selective model intervention combines inexpensive local mutation with semantic input generation. Region valuation, branch-level scheduling, and diagnostic repair form a connected search process rather than a collection of isolated generation examples. Figure 4's reduced triggers and case analyses help explain how complex inputs pass frontend checks and reach defects in constant evaluation or optimization. This is a useful engineering direction for language-processor testing.

S2. Potentially meaningful empirical gains, with component-level evidence. OptFuzz has the highest reported coverage on all four controlled targets in Table 1, including approximately 10.2%–42.4% gains over the closer HLPFuzz baseline. Table 3's breadth-only/depth-only and component-removal results support contributions from several parts of the system. Table 4 compares LinUCB with other schedulers, while Table 5 covers three model backends. If issue identities, deduplication, and confirmation status are verified, 57 maintainer-confirmed issues would constitute substantial practical impact.

Critical weaknesses

W1. The characterization of prior work is too broad, and the methodological increment over existing feedback-driven fuzzing is insufficiently isolated. The abstract and Sections 1–2 repeatedly characterize existing methods as stateless, single-shot generators without historical feedback. However, the direct baseline HLPFuzz explicitly introduces hybrid centrality prioritization and iterative context construction for LLM-based language-processor constraint solving. WhiteFox also explicitly feeds optimization-triggering tests back into subsequent generation. These primary sources establish prior feedback-based designs, so the transition from stateless generation to a closed loop is not, by itself, a sufficiently precise novelty boundary. See the official HLPFuzz page (https://www.usenix.org/conference/usenixsecurity25/presentation/yang-yupeng) and WhiteFox paper abstract (https://arxiv.org/abs/2310.15991).

The more persuasive contribution may be the particular combination of dynamic region valuation and contextual LinUCB scheduling. Yet the evaluation does not fully align context construction, feedback, syntax-aware mutation, and resources with the closest baseline. The random-selection ablations in Table 3 and scheduler alternatives in Table 4 provide relevant evidence, but their held-fixed components and query budgets need clearer documentation. Breadth-only/depth-only comparisons change available components, while removing interaction also reduces feedback opportunities. The paper should establish whether gains arise from scheduling, additional source information, mutation infrastructure, prompting, or more generation/repair attempts. This affects the contribution assessment, not merely its wording.

W2. Statistical and vulnerability reporting contain checkable inconsistencies and lack essential audit details.

Statistical intervals: Appendix Table 6 labels its values “Mean ± SD” from five runs. For Clang++, OptFuzz is 104,537 ± 3,382, with a reported 95% CI of [101,523,107,551], whose half-width is 3,014. If these are the sample mean and SD of five independent runs, a conventional Student-t interval has half-width approximately 2.776 × 3382 / √5 = 4,199, giving approximately [100,338,108,736]. For Flang, 128,714 ± 4,215 similarly yields a conventional half-width of approximately 5,234 rather than the reported 3,714. Other rows do not exhibit a consistent SD-to-interval conversion either. A bootstrap or paired procedure could target a different quantity, but the estimand, procedure, and five raw observations must then be supplied. The blanket p < 0.001 claim is not currently verifiable from the table. Wider intervals would not automatically eliminate the performance advantage, but the statistical reporting needs reconstruction.

Vulnerability accounting: Table 1 is labeled “24h Mean,” but its integer Bug entries are not defined as means, best-run counts, or cross-run unions. Section 7.2 sums OptFuzz's 8+5+2=15 against HLPFuzz's 4+3+1=8 without explaining root-cause deduplication across trials or tools. Table 2 lists zero assertion failures for Clang, while Appendix Table 6 includes a Clang assertion failure. Table 2 lists zero Hermes null dereferences, while Figure 4(c) and Appendix Table 6 describe one. Different classifications or populations might explain these differences, but none is specified. Clang and Clang++ also share substantial implementation, making root-cause deduplication important beyond sanitizer address matching. Calling all 74 findings zero-day vulnerabilities, when 57 are reported confirmed, requires distinct evidence for identities, discovery dates, and status.

The manuscript lacks an issue-by-issue table of IDs, fixes, and confirmation status; the appendix provides category examples instead. An anonymized or embargoed audit trail could reasonably replace public links, but an auditable trail is still needed. This directly affects the paper's strongest impact claim.

Questions for authors
  1. Increment over prior methods and existing controls (Sections 1–2, pp. 1–2; Section 7.4 and Tables 3–4, p. 8): What is the precise increment over HLPFuzz's prioritization/context construction and WhiteFox's feedback generation? Table 3 already includes random-selection ablations, and Table 4 already compares alternative schedulers. For these controls, which components and query/repair budgets were held fixed? Please distinguish the evidence for the selection policy from effects of additional context, syntax-aware mutation, or more interaction opportunities. The prior-work comparison is grounded in the original sources cited in Critical Weaknesses.

  2. Confidence intervals and repeated runs (Section 7.1, p. 6; Appendix B and Table 6, p. 14): Section 7.1 already states that controlled quantitative experiments were independently repeated five times, and Table 6 explicitly labels its spread as SD. Please provide the five coverage values and the CI calculation that reconciles those SDs with the displayed intervals, together with the test statistic, degrees of freedom, and pairing protocol for the reported two-tailed Student's t-tests. Please confirm whether the five-run protocol also applies to Tables 3–4 and supply their variability rather than only point estimates.

  3. Vulnerability accounting (Table 1, p. 6; Table 2 and Case C, p. 7; Section 8, p. 9; Table 6, p. 14): Table 2 lists zero Clang assertion failures and zero Hermes null dereferences, while Table 6 includes both categories and Case C describes the Hermes null dereference. Please explain the category/population mapping. Section 8 already mentions ASan/addr2line deduplication; how does this procedure distinguish root causes across runs, tools, and Clang/Clang++? Please define whether Table 1's Bug entries are run means, rounded means, or pooled unique counts, and provide an anonymized issue-level inventory connecting the 74 findings to their confirmation status and reproducer evidence.

  4. Comparison configuration (Sections 6–7.1, pp. 5–6): The paper already reports dedicated-core containers, a common GPT-5 backend, and initial corpora of identical origin and comparable scale. Please supply the remaining per-tool allocation details: core/process counts, actual model/query/repair settings, and token or call totals. Were all corpora replayed against a common instrumented binary for the reported coverage? Please document target-specific adaptations and command lines for the language-aware baselines, so the coverage comparisons can be reproduced.

  5. Missing settings and throughput definitions (Algorithm 1, p. 3; Sections 3–5, pp. 3–5; Table 5 and Section 7.5, p. 9): The paper describes a breadth-first/depth-assisted policy and defines the stagnation counter, but does not give the full SelectMode rule or a numerical Nstag. Please provide those, the sliding-window length, w1/w2 and v1/v2, and the LinUCB exploration coefficient. Section 5.1 normalizes code depth but gives x4 as Lnest + Nvar; is x4 further scaled? Finally, what denominator is used for the claim that native CGF runs more than 99% of the time, and is Table 5's 45.6 exec/s measured end-to-end or within a generation-specific benchmark? Please clarify how it should be compared with AFL++'s 177.8 exec/s.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Tbjn

We thank the reviewer. Due to space limits, replies here are brief; detailed revisions and additions will be highlighted in the revised paper.

Part 1/4 — W1 & Q1: Increment over HLPFuzz/WhiteFox

We thank the reviewer. We agree that "stateless, single-shot" over-generalizes: HLPFuzz and WhiteFox both use feedback. Our claim is now hierarchical, bandit-scheduled integration.

HLPFuzz uses static Katz ranking in a single-level repair loop (our λ builds on its solving-history calibration, now credited); WhiteFox lacks runtime region/branch scheduling and diagnostic-driven repair. OptFuzz adds (i) two-level allocation (Breadth: where; Depth: which constraint); (ii) online LinUCB over six features with a gated reward; (iii) multi-round diagnostic repair with failure-fatigue back-off.

Controls. AFL++, HLPFuzz and OptFuzz share the same ANTLR4 mutator; LLM-based tools share GPT-5 and the 24h budget. Ablations change only the named module; w/o IO variants still retry at comparable cost.

Evidence (Clang++ edges, mean±SD, 5 runs):

MethodEdgesCost/h
Generic agent harness (LLM on stagnation + symbol lookup, no scheduling)82,315±2,840$0.59
HLPFuzz94,782±3,025$0.58
OptFuzz104,537±3,382$0.55

Spend is similar but coverage differs, so gains come from scheduling and repair, not more queries. The w/o IO variants cost $0.57/$0.53 per hour yet reach only 70,918/73,517 edges. On four targets, LinUCB beats the best alternative (Thompson sampling) by 4.8–8.0%.

Action. Revise abstract and Secs. 1–2 (add mechanism table in Appendix A); add harness comparison (Sec. 7.2), cost table (Appendix A, referenced in Sec. 7.4), and scheduler table (Sec. 7.4).

Part 2/4 — W2 & Q2: Intervals and significance

We thank the reviewer. Your calculation is correct; our original intervals were wrong. With n=5, df=4, the half-width is 2.776·SD/√5 = 1.2417·SD. Corrected 95% CIs (edges):

TargetAFL++HLPFuzzOptFuzz
Clang++[53,379, 58,679][91,026, 98,538][100,338, 108,736]
Flang[66,726, 72,960][112,010, 121,676][123,480, 133,948]
Hermes[13,224, 14,670][23,782, 26,052][34,091, 36,873]
Lua[3,241, 3,549][4,028, 4,488][5,387, 5,903]

OptFuzz's interval is disjoint from the strongest baseline on every target.

Significance. We replace the blanket "p<0.001" with unpaired two-tailed Welch's t-tests. OptFuzz vs. HLPFuzz: Clang++ p=1.4×10⁻³; Flang p=1.7×10⁻³; Hermes p=3.0×10⁻⁷; Lua p=4.2×10⁻⁶. We now claim p<0.01 for Clang++/Flang, p<0.001 elsewhere. The same 5-run protocol applies to Tables 3–4; we add SDs to both.

Action. Rewrite Appendix C/Table 6 (CIs, test, statistics, df); revise Sec. 7.1; add SDs to scheduler Table 4 (Sec. 7.4).

Part 3/4 — Q3: Bug accounting

We thank the reviewer for the careful cross-checking; Table 2 contained a classification error, now fixed. The Clang assertion failure had been counted as HBOF; it is now counted as AF and the HBOF column reduced accordingly. The Hermes null dereference is now ND, not SV. We re-verified all row and column sums; the grand total of 74 is unchanged. Table 1's Bug entries are per-run means rounded to the nearest integer (now in the caption).

Oracle and deduplication. Bugs are detected via crashes, sanitizer reports (ASan/UBSan) and target-internal assertions, deduplicated via ASan stack traces and addr2line, then manually triaged.

Terminology. "Zero-day bugs" becomes "previously unknown bugs". Representative bug descriptions with reproduction steps will be on the anonymous OSF page.

Action. Correct Table 2 and Table 1 caption (Secs. 7.2–7.3); add oracle and dedup rules (Sec. 7.3).

Part 4/4 — Q4 & Q5: Settings, cost, throughput

We thank the reviewer; all settings are now in a new Appendix D.

  • Stagnation trigger: LLM invoked when new-edge growth over a 90 s window falls below (total edges × 10⁻⁴).
  • Switching: Breadth→Depth after 3 consecutive breadth failures in a region; Depth→Breadth after 5 failures on one constraint, which is then deprioritized.
  • Budget: 3 LLM calls per optimization.
  • Weights: w₁=w₂=1, v₁=v₂=0.5, γ=3.0, B_base=1.0, K=3; LinUCB exploration 1.0; Katz α=0.1; x₄ uses standard feature scaling.

Bottleneck identification. AFL++ exports only the coverage bitmap and edge-to-block map. Uncovered direct successors of covered blocks are ranked by Katz centrality on the reversed uncovered subgraph, aggregated to function/line level, and the top-N selected.

Cost. OptFuzz $0.55/h, HLPFuzz $0.58/h, generic harness $0.59/h.

Throughput. The 45.6 exec/s is end-to-end over 24h Clang++ runs, so not comparable to AFL++'s 177.8. We remove the "preserves throughput" and ">99%" wording and compare throughput only against HLPFuzz, which shares the pipeline.

Action. Add Appendix D; revise Sec. 7.5 and Sec. 3.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer Tbjn,

Thank you again for your time and valuable feedback. As the discussion period ends soon, we would like to kindly check whether our response has addressed your concerns.

If any points remain unclear, we would be happy to provide further clarification or additional results before the deadline. We hope our response helps address your concerns, and we would be grateful if you could take it into account in your assessment.

We understand that you may have a busy schedule, and we would sincerely appreciate it if you could find the time to review and respond to our reply.

Best regards,

Authors of Submission #33339

Reviewer_WCN2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents OptFuzz, a hybrid fuzzing framework for language processors that combines coverage-guided fuzzing with selective LLM assistance when coverage stagnates. Its Breadth Optimizer prioritizes underexplored code regions using coverage gaps, code complexity, and historical feedback, while its Depth Optimizer uses a LinUCB contextual bandit to select bottleneck branches for targeted exploration. Compilation diagnostics, execution traces, and on-demand source-code retrieval support iterative generation and repair of test inputs. The evaluation includes repeated 24-hour comparisons on four targets and extended testing across eight language processors. The authors report improved edge coverage over existing baselines and 74 discovered vulnerabilities, including 57 confirmed by maintainers.

Critical strengths
  1. The paper presents an interesting application of LLMs to program testing, using execution feedback to guide test generation toward difficult-to-reach program logic.

  2. The experimental results appear promising: the reported coverage improvements and discoveries of maintainer-confirmed bugs suggest that the approach could be useful in practice.

Critical weaknesses
  1. Unclear positioning and significance of the framework. The paper does not clearly explain what fuzzing-specific challenges make a straightforward adaptation of an existing LLM agent harness insufficient, or why the proposed breadth/depth architecture is necessary to address them. Its relationship to closely related work, particularly HLPFuzz, is also insufficiently clarified, given that HLPFuzz already incorporates history-aware prioritization and iterative context construction with execution feedback. These gaps make it difficult to assess the framework’s distinct role and the significance of its contribution beyond integrating established components.

  2. Insufficient isolation of the sources of improvement. Although the paper includes component ablations, it is unclear whether LLM call or token budgets are matched across variants. Removing iterative feedback also removes opportunities for generation and repair within each interaction, so the reported gains may partly reflect additional attempts rather than the effectiveness of the feedback strategy itself. A budget-matched comparison against repeated generation without feedback would help distinguish these effects. Similarly, isolating the scheduling contribution requires keeping the model, context-construction procedure, and repair budget fixed while changing only the target-selection policy.

  3. Limited evidence for the advantage of LinUCB. Table 4 compares LinUCB with alternative schedulers, but reports results only on Clang++ without variability estimates or confidence intervals. Its coverage improvement over Thompson Sampling is approximately 5.1% (104,537 versus 99,504 edges), leaving the stability and cross-target generality of this advantage unclear. The paper also provides limited justification for the linear reward model and little analysis of the contributions of individual features. Evaluating the schedulers across additional targets with uncertainty estimates and feature ablations would better substantiate this design choice.

Questions for authors
  1. What fuzzing-specific challenges make a straightforward adaptation of an existing LLM agent harness insufficient?

  2. Were LLM call or token budgets matched across the ablation variants? Could you provide a budget-matched comparison between iterative feedback and repeated generation without feedback to distinguish the benefit of feedback from additional generation attempts?

  3. Does LinUCB consistently outperform the alternative schedulers across targets and independent runs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer WCN2

We thank the reviewer for the constructive feedback and address each point below, with new experiments. All LLM-based methods use GPT-5 (HLPFuzz re-run from its open-source release). Due to space limits, replies here are brief; detailed revisions and additions will be highlighted in the revised paper.

W1/Q1: Positioning and relation to HLPFuzz.

Fuzzing-specific challenges. We agree the positioning needs to be sharper. A 24h+ campaign is dominated by cheap native CGF, so LLM calls are scarce and must be spent carefully. The system must decide where to spend them (which region), on what (which bottleneck branch), and when to back off (after repeated failures). A generic harness that calls the LLM on stagnation lacks these controls.

Harness comparison. We built a generic agent harness that invokes the LLM on stagnation and supports symbol lookup (reusing HLPFuzz's LSP), with no region/branch scheduling or divergence analysis. All methods share the model, seeds and duration. Coverage is 24h edge coverage (mean ± SD over 5 runs); Bugs are unique bugs over the four targets.

MethodClang++FlangHermesLuaBugsCost/hCompile Pass
AFL++ (no LLM)56,029 ± 2,13469,843 ± 2,51013,947 ± 5823,395 ± 1242$0.00–
Generic Agent Harness82,315 ± 2,840101,420 ± 3,45019,840 ± 7604,085 ± 1505$0.5924.6% ± 1.8%
HLPFuzz94,782 ± 3,025116,843 ± 3,89224,917 ± 9144,258 ± 1858$0.5846.2% ± 1.5%
OptFuzz104,537 ± 3,382128,714 ± 4,21535,482 ± 1,1205,645 ± 20815$0.5561.8% ± 1.4%

The three LLM-based methods cost about the same ($0.55–$0.59/h) but differ substantially in results, which reflects how calls are scheduled and repaired.

Relation to HLPFuzz. HLPFuzz prioritizes branches with Katz centrality calibrated by solving history, and refines context via divergence analysis. Our calibration factor λ builds on this idea, and we now credit it in the paper. OptFuzz differs in three ways: (i) two-level decisions, with region-level Breadth plus branch-level Depth; (ii) an online LinUCB scheduler with exploration over six features; (iii) closed-loop feedback from compiler diagnostics and execution traces.

We have added a positioning paragraph in Section 1 and Related Work, a mechanism comparison table in Appendix A, and the harness comparison in Section 7.2.

W2/Q2: Isolating sources of improvement.

Ambiguity in the original wording. We thank the reviewer for raising this concern; the original wording was ambiguous. "Stateless, single-shot generation without feedback retries" meant that the w/o IO variants lack our feedback strategy (compiler diagnostics, execution traces, on-demand symbol retrieval). They still retry, but independently and at comparable cost. We have clarified this in Section 7.4.

Scheduling variants. The policies (w/o VA, w/o DS) change only which target is chosen, not the prompt or interaction structure.

Cost. The table reports Clang++ cost per hour and per thousand edges gained over AFL++ (24h cost / edge gain).

VariantCost/h ($)Cost per k-edges gained ($)
OptFuzz (Full)0.550.27
Breadth-only0.480.32
├ w/o VA (random region)0.570.61
└ w/o IO (plain independent retries)0.570.92
Depth-only0.590.36
├ w/o DS (random branch)0.680.65
└ w/o IO (plain independent retries)0.530.73

The w/o IO variants cost $0.57 and $0.53/h, close to the full system's $0.55, yet cover far fewer edges (70,918 and 73,517 vs. 104,537). Random scheduling costs more ($0.57 and $0.68/h), likely due to more failed attempts. Among all variants, the full system has the lowest cost per coverage gain.

We have added the cost table to Appendix A (referenced in Section 7.4).

W3/Q3: Evidence for LinUCB.

We have extended the scheduler comparison to Flang, Hermes and Lua (Clang++ was in the original Table 4), with 5 independent 24h runs each (mean ± SD).

SchedulerClang++FlangHermesLuaAvg. Rank
LinUCB104,537 ± 3,382128,714 ± 4,21535,482 ± 1,1205,645 ± 2081.00
Thompson99,504 ± 3,140122,810 ± 3,95032,840 ± 1,0505,280 ± 1952.00
ε-greedy94,419 ± 2,980115,920 ± 3,76028,150 ± 9804,820 ± 1753.00
Failure-Only90,102 ± 2,850109,840 ± 3,51023,450 ± 8604,510 ± 1604.25
Katz-Only89,436 ± 2,910110,250 ± 3,62021,980 ± 8204,420 ± 1554.75

LinUCB has the highest mean coverage on all four targets, with a 4.8%–8.0% margin over Thompson Sampling, the strongest alternative. We chose a linear model because the context has only six features, the update is O(d²), and it stays stable when feedback is sparse.

We have replaced Table 4 with this version in Section 7.4.

AuthorsOpenReview ↗

Follow-up on Our Response to Your Review

Dear Reviewer WCN2,

Thank you again for your time and valuable feedback. As the discussion period ends soon, we would like to kindly check whether our response has addressed your concerns.

If any points remain unclear, we would be happy to provide further clarification or additional results before the deadline. We hope our response helps address your concerns, and we would be grateful if you could take it into account in your assessment.

We understand that you may have a busy schedule, and we would sincerely appreciate it if you could find the time to review and respond to our reply.

Best regards, Authors of Submission #33339

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and AC

We thank the reviewers and the AC for their time and detailed feedback. Reviewers noted (i) the design targeting semantic bottlenecks in language processors (WCN2, Tbjn, ZgaP), (ii) the separate contexts for Breadth and Depth, supported by ablations (ZgaP, Tbjn), and (iii) the empirical results and maintainer-confirmed bugs. All changes are in blue in the revised PDF. Individual replies are under each review; below are the shared concerns.

1. Positioning relative to HLPFuzz/WhiteFox (WCN2 W1, Tbjn W1, ZgaP W4.2)

We agree that "stateless, single-shot" over-generalized, since HLPFuzz and WhiteFox use feedback. We now credit HLPFuzz (λ builds on its solving-history idea) and restate our contribution: (i) two-level allocation (Breadth: where; Depth: which constraint), (ii) online LinUCB over six features with a gated reward, (iii) multi-round repair from compiler diagnostics and execution traces with failure-fatigue back-off (Sec. 1-2, App. A mechanism table). HLPFuzz is the primary baseline as the only published LLM-based method we found for language processors with hybrid bottleneck solving; Fuzz4All, WhiteFox and CovRL target other settings.

2. Sources of improvement (WCN2 W2, ZgaP W4.1, Tbjn W1)

  • Mutator control: All tools, including AFL++ and HLPFuzz, use the same ANTLR4 mutator, so the AFL++ rows are "AFL++ + ANTLR4, no LLM" (Sec. 7.1).
  • Generic harness (new, Sec. 7.2): LLM on stagnation with symbol lookup, no scheduling; same GPT-5, seeds and 24h budget. Clang++ edges (mean±SD, 5 runs): harness 82,315±2,840 / HLPFuzz 94,782±3,025 / OptFuzz 104,537±3,382, at $0.59 / $0.58 / $0.55 per hour.
  • Ablation cost (new, App. A): w/o IO variants retry independently at similar cost ($0.57/$0.53 vs. $0.55 per hour) and reach 70,918/73,517 edges. The full system has the lowest cost per k-edges ($0.27). w/o VA and w/o DS change only target selection.
  • LinUCB (new, Sec. 7.4): On all four targets (5 runs), LinUCB has the highest mean, 4.8-8.0% above Thompson Sampling.
  • SelectMode (new, Sec. 7.4): On Clang++, fixed alternation reaches 93,274 edges and random mode 94,946, versus 104,537 for our rule.

3. Statistics (Tbjn W2, WCN2 W3)

Tbjn's calculation is correct; our original intervals were wrong. We recomputed them with t(0.975, df=4)=2.776 (half-width 1.2417·SD), e.g. Clang++ OptFuzz [100,338, 108,736]; its interval does not overlap the strongest baseline's on any target. Welch's t-tests vs. HLPFuzz replace the blanket "p<0.001": Clang++ 1.4e-3, Flang 1.7e-3, Hermes 3.0e-7, Lua 4.2e-6. Tables report SDs over 5 runs (App. C).

4. Design details and reproducibility (ZgaP W1-W3, Tbjn Q5)

  • Architecture: AFL++ and the LLM layer run as separate processes with asynchronous enqueuing; Algorithm 1 is now in App. A with a clarifying note.
  • Throughput: We withdrew the ">99%" and "preserves throughput" statements. The 45.6 exec/s is an end-to-end 24h figure on Clang++, not comparable to AFL++'s 177.8; the drop comes from compiling LLM-generated tests that often crash the target. We compare only with HLPFuzz, which shares the pipeline.
  • Definitions: Stagnation is new-edge growth below |E|×10^-4 over 90 s. SelectMode moves Breadth→Depth after 3 consecutive breadth failures in a region, and Depth→Breadth after 5 failures on one constraint. ExtractBottlenecks ranks uncovered successors of covered blocks by Katz centrality; the top 5 form the arm set. Regions are second-level directories; no context compression is used.
  • Symbols: All symbols are defined, α is split into separate symbols, and x_t is distinguished from x_c; notation and hyperparameter tables are in App. D.

5. Bug accounting (ZgaP W4.3, Tbjn W2)

"Zero-day" is replaced by "previously unknown bugs". Bugs are detected by crashes, ASan/UBSan and assertions, deduplicated by ASan stack traces and addr2line, and triaged manually by root cause, including across Clang/Clang++. Bugs are reproduced on the development trunk where one exists, otherwise on the latest release. Two category errors in Table 2 (a Clang assertion failure, a Hermes null dereference) are corrected; sums are rechecked and the total remains 74. Table 1 bug counts are per-run means rounded to integers.

Revisions: Abstract, Sec. 1-2 (positioning); Sec. 3, 5.1 (definitions); Sec. 7.1-7.5 (controls, statistics, harness, bug accounting, schedulers, throughput); App. A, C, D (algorithm, tables, CIs, hyperparameters).

We hope these changes address the concerns and are glad to discuss any remaining questions. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.