- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents DF-Fuzz, a dependency-guided context retrieval framework for LLM-assisted language processor fuzzing. The method constructs a dual-flow dependency graph that combines local control-flow constraints with cross-file data, type, macro, and inheritance dependencies. It then serializes a compact context for single-round mutation, using semantic pruning, API co-occurrence exemplars, and a compiler-feedback-driven self-healing loop to recover omitted definitions. The evaluation compares DF-Fuzz with conventional, symbolic, grammar-based, and LLM-guided fuzzers across several open-source language processors. The paper reports improvements in coverage, bug discovery, execution throughput, and API efficiency, together with an extended campaign that found previously unknown issues.
Critical strengths
The paper identifies a concrete failure mode in LLM-assisted compiler fuzzing: isolated function slices omit the declarations and relationships needed to produce valid inputs. Unifying control reachability with cross-file dependency resolution in an offline graph is a sensible and technically motivated response, and the single-round design has clear practical appeal for high-throughput fuzzing.
The pruning, exemplar retrieval, and diagnostic-driven repair components address different sources of failure and are connected through an understandable end-to-end workflow. The modular design and walkthrough make the system easy to follow and extend, and distinguish it from a monolithic prompt-engineering contribution.
Critical weaknesses
The causal claim is not isolated. The paper argues that missing cross-file context causes invalid LLM-generated inputs, but it only measures end-to-end coverage and bugs. It does not report input validity directly. That baseline's configuration is also unclear, so the benefit of dependency reasoning cannot be separated from simply adding more context or engineering differences.
Questions for authors
Can the authors report frontend pass rates with and without retrieval? Can they also compare against the LLM baseline under the same model and prompt budget?
4 comments
Response to Reviewer KkPj
Dear Reviewer,
Thank you for the follow-up. This suggestion helps us separate which dependencies are selected from how much context is provided, which makes the evaluation clearer. We have compiled the prompt-size statistics below and completed the matched-budget experiments.
1. Average prompt tokens per strategy. All numbers are average input tokens per call, taken from our existing experiment logs (Clang++, 24h runs, averaged over 5 runs). The three DF-HDG variants use the same LSP lookup, few-shot exemplars, and polymorphic recall as the "Cross-file symbol lookup" baseline. The difference lies only in the retrieval module: DF-HDG selects dependencies anchored on the uncovered branch and its dominating predicates, whereas the baseline uses HLPFuzz's native cross-file symbol lookup.
| Retrieval strategy | Avg. input tokens / call | Pass rate | Edge coverage |
|---|---|---|---|
| No retrieval (enclosing function only) | 1.2k | 18% | 85,426 |
| Cross-file symbol lookup (HLPFuzz-style) | 4.3k | 39% | 94,810 |
| DF-HDG, control edges only | 3.5k | 51% | 98,214 |
| DF-HDG, data edges only | 3.8k | 48% | 97,953 |
| Full DF-HDG | 3.7k | 59% | 100,153 |
Full DF-HDG uses fewer input tokens per call than the HLPFuzz-style baseline (3.7k vs. 4.3k) while achieving a higher pass rate and higher coverage.
2. Why the HLPFuzz-style baseline uses more input tokens. HLPFuzz builds context iteratively (Sections 5.1–5.3 of its paper). It starts from the enclosing function and, after each failed attempt, expands the context via divergence analysis and LSP symbol lookup, with a maximum of 5 iterations. Its own evaluation reports that a successful solution needs 2.64 iterations on average, and that only 5%–9.1% of solutions succeed in a single iteration (Section 7.3 and Table 6, under their GPT-4o-mini setting). One possible explanation is that, in the baseline, failed attempts trigger further rounds of context expansion, which increases the per-call input size. We note that the HLPFuzz paper does not state this link itself; it is our reading of its design together with our measured 39% pass rate.
3. Matched-budget comparison. An equal-token comparison is the most direct way to separate selection from size, so we ran it on Clang++ under the same protocol (24h, 5 runs averaged, same seeds and hardware). In both settings, the prompt is hard-truncated at the stated limit, and this is applied identically to both systems.
| Budget (input tokens / call) | Strategy | Pass rate | Edge coverage |
|---|---|---|---|
| 3.7k cap | Cross-file symbol lookup (HLPFuzz-style) | 30% | 90,457 |
| 3.7k cap | Full DF-HDG | 51% | 97,186 |
| 4.3k cap | Cross-file symbol lookup (HLPFuzz-style) | 36% | 92,235 |
| 4.3k cap | Full DF-HDG | 55% | 98,804 |
Under both budgets, Full DF-HDG achieves a higher pass rate (+21 and +19 percentage points) and higher edge coverage (+7.4% and +7.1%) than the baseline. This suggests that the improvement comes largely from which dependencies are selected rather than from how much context is provided.
Action. We will add the per-call token statistics, the matched-budget results, and the explanation of the baseline's token overhead to Section 5.4.2, and reference them in Section 5.1.
Thank you again for the constructive feedback, which has made the evaluation more rigorous.
Best regards,
The Authors
Follow-up on Our Response to Your Feedback
Dear Reviewer,
We have added the relevant experiments, including the prompt-size statistics and a matched-budget comparison, and we hope they address your concern. If you have any further questions, we would be happy to discuss and resolve them.
Best regards,
The Authors
Thank you for the follow-up. The token statistics and matched-budget results address my remaining concern. I will maintain my rating of weak accept.
Thank you for the detailed rebuttal. The added pass-rate results address my main concern about input validity. It would also help to report the average prompt tokens for each strategy, ideally with a matched-budget comparison, to further separate dependency selection from context size. Overall, I will keep my score of weak accept.