- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, they proposed RedHerring, an approach to hinder LLM based vulnerability discovery by diverting agents verification budgets toward decoy code. Those decoy code guide LLM to an unsatisfiable guard that keeps the dangerous sink unreachable. They evaluated their approach on 70 instances from OSS-Fuzz projects; and found that the crashes were decreased by 60.4%.
Critical strengths
-
Interesting idea on hindering LLM based vulnerability discovery.
-
Performance analysis specially trajectories shows more consumption of completion tokens (around 60.4% increase) and runtime (49.9% increase).
Critical weaknesses
-
Limited instances to evaluate. The manuscript only tested on 70 instances.
-
They haven't discussed their approach in details. For example, how does vulnerability chains make the agent an attractive sink and how that impacts on vulnerability discovery. The same goes for false bridge. It is very difficult to conclude that the proposed approach is generalized or only work on a few samples.
-
The same confusion hold valid for repository adaptation as well.
-
Did they evaluated their attack in the presence of some existing defense?
-
Attack motivation and threat model is not clear. Why do we need to limit the objective bounded by B?
-
The manuscript has not investigated with existing static analysis tool. Or even PoCE (https://conf.researchr.org/details/issta-2026/issta-2026-research-papers/43/PoCE-Automated-Proof-of-Concept-Synthesis-using-Large-Language-Models-for-Robust-Val) CyberGym (https://www.cybergym.io) or ExploitGYM (https://www.cybergym.io/exploitgym/) would be beneficial.
Questions for authors
-
Can you extend your analysis on a bigger dataset?
-
Clarify the threat models (how much realistic it is) and motivation of the work.
-
Robustness of the work is missing.
6 comments
Response to Reviewer HGTR (2/3)
Q2a: Clarify the threat models (how much realistic it is)
W5: Attack motivation and threat model is not clear. Why do we need to limit the objective bounded by B?
Each agent run costs money for API calls and compute, and an attacker has limited time to search a given target.
Why a budget bound. Because the defender does not know where the vulnerabilities are, they remain in the released repository, and an attacker with an unlimited budget could eventually find them. A practical defense therefore raises the attacker's cost, as obfuscation does [1]. The budget B makes this cost measurable: RedHerring reduces the real vulnerabilities an agent finds within the same budget.
Realistic. Evaluations of security agents bound each run. CyberGym limits each run to 100 agent steps [2], CyberGym-E2E sets USD 10 and 90 minutes per task [3], and ExploitGym imposes a two-hour timeout per task [4]. Industry evaluations also set explicit limits. OpenAI evaluates exploit development under a standard 300-turn setting [5], and each finalist team in DARPA's AI Cyber Challenge received USD 85,000 in Azure compute and USD 50,000 in LLM API credits [6]. Our budget of 3 hours and 300 turns per run matches or exceeds the per-run limits above.
Motivation. Frontier LLM agents have lowered the cost and expertise required for vulnerability discovery, which makes attack and defense increasingly asymmetric. An attacker can search entire repositories at low cost, whereas a defender cannot patch vulnerabilities it has not found. Existing defenses either require knowing which code to protect or act only after a vulnerability is found. RedHerring offers a new defense direction against vulnerability-discovery agents by treating their verification effort as a defense surface. It diverts this effort to decoys and requires no knowledge of whether or where the repository contains vulnerabilities. Each decoy shifts the asymmetry toward the defender: it is cheap to confirm as safe but costly for the agent to verify.
W4: Did they evaluated their attack in the presence of some existing defense?
We compared RedHerring with two existing techniques, adversarial comments [7] and opaque predicates from classic obfuscation [1]. Our results show that RedHerring reduces real vulnerability discovery substantially more than both techniques.
Adversarial comments. This defense keeps the code logic unchanged and inserts comments that mislead an LLM’s judgment of the code. We built a Comments-only version of each instance.
Opaque predicates. An opaque predicate is a condition whose value is fixed but hard to determine by static analysis. Tigress-Opaque keeps the same vulnerability chains and insertion sites as RedHerring and replaces only the false bridge with an always-false opaque predicate that Tigress [8] builds from data-structure invariants.
| Setting | Confirmed real vulnerabilities | Reduction |
|---|---|---|
| Baseline | 264 | – |
| Comments-only | 222 | 16.0% |
| Tigress-Opaque | 195 | 26.2% |
| RedHerring | 146 | 44.6% |
The table gives results for Qwen3.8-Flash on all 140 instances. With the same chains and insertion sites, RedHerring confirms 25.1% fewer real vulnerabilities than Tigress-Opaque. We will add these results to the revision.
References
[1] Collberg et al. A taxonomy of obfuscating transformations. Technical Report 148, The University of Auckland, 1997.
[2] Wang et al. CyberGym: Evaluating AI agents' real-world cybersecurity capabilities at scale. ICLR, 2026.
[3] Shi et al. CyberGym-E2E: Scalable real-world benchmark for AI agents' end-to-end cybersecurity capabilities. ICML, 2026.
[4] Wang et al. ExploitGym: Can AI agents turn security vulnerabilities into real attacks? arXiv:2605.11086, 2026.
[5] OpenAI. A thousand fewer paths to root, 2026.
[6] Zhang et al. SoK: DARPA's AI Cyber Challenge (AIxCC). arXiv:2602.07666, 2026.
[7] Thornton. Can adversarial code comments fool AI security reviewers? arXiv:2602.16741, 2026.
[8] Collberg. Tigress: A diversifying virtualizer/obfuscator for the C language. https://tigress.wtf.
Response to Reviewer HGTR (3/3)
W6: The manuscript has not investigated with existing static analysis tool. Or even PoCE (…) CyberGym (…) or ExploitGYM (…) would be beneficial.
We added an evaluation with static analysis tools and clarify how RedHerring relates to CyberGym, ExploitGym, and PoCE.
Static analysis does not identify the decoys as dead code. We randomly selected 50 instances and scanned them with Cppcheck and CodeQL, both before and after inserting five decoys per instance, which covers 250 decoys.
| Tool | Alerts before insertion | Alerts after insertion | Alerts in decoy code | Decoys with alerts | Gates reported as always false |
|---|---|---|---|---|---|
| Cppcheck | 22,388 | 22,426 | 38 | 13 / 250 | 0 |
| CodeQL | 3,002 | 3,002 | 0 | 0 / 250 | 0 |
Neither tool reports any gate as always false. The decoys also blend into the static analysis output. Cppcheck reports 38 alerts in decoy code, 0.17% of its 22,388 alerts on the original code, and CodeQL reports none. The 13 decoys with Cppcheck alerts appear as ordinary vulnerability candidates rather than dead code. We will add these results to the revision.
CyberGym [1] is a large-scale benchmark that evaluates AI agents on real-world vulnerabilities, and its tasks are built from OSS-Fuzz projects. Our instances are drawn from the same source.
ExploitGym [2] evaluates a later stage of an attack. Given an input that already triggers a known vulnerability, the agent must turn it into a working exploit. RedHerring targets the earlier stage, in which the agent discovers vulnerabilities.
PoCE. [3] We thank the reviewer for the suggestion to consider PoCE, which became available on October 1, 2026, during the review period. PoCE starts from a reported vulnerability and an initial successful PoC, and expands this PoC into multiple validated variants. It thus works after a vulnerability has been identified, whereas RedHerring acts earlier, while the agent is still searching the repository and choosing which candidates to verify. As the authors have not released the implementation for security reasons, we could not include it in our experiments. We will cite and discuss PoCE in the revision.
Q3: Robustness of the work is missing.
We address robustness in two dimensions. RedHerring remains effective across settings, including different models, programs, and runs. It also remains effective against stronger attackers.
Across settings. RedHerring reduces discovery for all five models, by 38.7% to 60.4% (RQ1). Across programs and runs, it reduces discovery by 44.6% on average over 140 instances with five runs each, and the reduction holds in each of the five runs and on the 70 newly added instances.
Against stronger attackers. When the agent is informed that decoys may be present (RQ3), it still finds 37.2% fewer vulnerabilities. We further added three experiments. When we remove the restriction on fuzzers and brute-force search from the audit prompt, RedHerring still reduces discovery by 35.4%. When the agent is told to skip any gate that never passes on its test inputs, it confirms 31.9% fewer vulnerabilities than the Baseline without this instruction. An attacker also cannot filter out the decoys with static analysis tools, since neither Cppcheck nor CodeQL reports any gate as always false.
If our understanding differs from the reviewer's concern, we would be grateful if the reviewer could point us to the specific aspect. We are happy to provide additional analysis during the discussion period and include it in the revision.
We hope these responses address the reviewer's concerns, and we would appreciate it if the reviewer could re-evaluate the paper in light of the new results.
References
[1] Wang et al. CyberGym: Evaluating AI agents' real-world cybersecurity capabilities at scale. ICLR, 2026.
[2] Wang et al. ExploitGym: Can AI agents turn security vulnerabilities into real attacks? arXiv:2605.11086, 2026.
[3] Tithy et al. PoCE: Automated proof-of-concept synthesis using large language models for robust validation. Proc. ACM Softw. Eng., 3(ISSTA), 2026.
Response to Reviewer HGTR (1/3)
We thank the reviewer for the constructive comments. Following the reviewer's suggestions, we doubled the evaluation set, compared RedHerring with existing defenses, evaluated whether static analysis tools can filter out the decoys, and tested RedHerring against stronger attackers. We address each point below.
We doubled the evaluation to 140 instances and ran each instance five times. RedHerring reduces confirmed real vulnerabilities by 44.6%.
The 70 new instances were randomly selected from OSS-Fuzz and extend the evaluation from 33 to 69 projects. Each run audits an entire repository for up to 3 hours and 300 turns, so the expanded evaluation contains 1,400 runs, about 3,200 agent hours. Given the substantial API cost at this scale, we ran it on Qwen3.8-Flash, the model used in our informed-agent and ablation experiments.
The reduction also holds on the 70 newly added instances alone. The result is stable across runs: RedHerring confirms fewer real vulnerabilities than the Baseline in each of the five runs. The table gives confirmed real vulnerabilities for Qwen3.8-Flash, averaged over the five runs. We will add these results to the revision.
Vulnerability chain. The vulnerability chain reproduces patterns from real CVEs to match agents’ behavioral prior: their tendency to prioritize familiar vulnerability cues when deciding what to verify. Our empirical observations of agent trajectories show that, when performing vulnerability discovery, agents often begin by searching for typical vulnerability patterns or dangerous sinks, then assess and rank the matches for deeper investigation. We design vulnerability chains around this prior to draw verification effort toward decoys. Our ablation study shows that removing the chain lowers the reduction in real vulnerability discovery from 60.4% to 25.5% (RQ4).
False bridge. The false bridge makes the decoy’s apparent reachability depend on a condition built from a computationally hard problem, directing verification effort toward determining whether that condition can hold. Our empirical observation is that agents spend much of their vulnerability-verification effort analyzing reachability and constructing PoCs to confirm or rule out suspected bugs. The false bridge targets this workflow by placing the hard condition upstream of the vulnerability chain. Every path to the dangerous sink passes through an unsatisfiable guard. The defender can efficiently certify its unsatisfiability using private construction information, while the released code exposes only public parameters. Our ablation study shows that replacing the false bridge with a simple guard lowers the reduction in real vulnerability discovery from 60.4% to 32.9% (RQ4).
Repository adaptation. Repository adaptation aligns each decoy with the target repository’s semantics. The decoy should look like part of the repository’s own logic. It must also compile correctly and preserve the program’s intended behavior. We connect the chain to an existing input-processing function and use the repository’s types, calls, and naming. We place the bridge at a site that matches local operations and keep its condition and public parameters unchanged.
Generalization. Decoy insertion is independent of the repository’s structure, and RedHerring applies to all 140 instances from 69 OSS-Fuzz projects. To insert a decoy, RedHerring only needs a suitable attachment site in the project’s code and an existing variable there as the gate input. Insertion adds a single branch, guarded by the false bridge, that places the decoy on the program’s execution path. The original code stays unchanged. An LLM agent adapts the decoy to each repository, and a deterministic verifier checks every insertion, so no repository-specific rules are needed. The 69 projects range from network protocol analysis to compression (c-blosc2), and the decoys use 75 chain templates from different CVEs and four bridge constructions based on different hard problems.