acceptodds
Under review as a conference paper at ICLR 2027

RedHerring: Verification Effort as a Defense Surface Against LLM Agents in Vulnerability Discovery

Abstract

Autonomous LLM agents turn vulnerability discovery into a repository-scale search: they generate many vulnerability hypotheses but can verify only a subset under a finite budget. We show that autonomous vulnerability discovery exhibits a hypothesis–verification asymmetry, where verifying a candidate hypothesis through reachability analysis, execution, and proof-of-concept construction is substantially more expensive than forming it. Under a finite resource budget, this makes au- tonomous discovery a resource-bounded selective-verification process, further exposing verification effort as a unique defense surface. We present RedHerring, which inserts certifiably safe decoys that divert verification effort from real vul- nerabilities. Each decoy combines a CVE-derived vulnerability chain that attracts verification with a false bridge that keeps its dangerous sink unreachable. A private certificate lets the defender verify this property efficiently, while establishing the same fact from the released repository requires solving a computationally hard problem. RedHerring further adapts each decoy to the target repository so that it reads as ordinary program logic. Across 33 OSS-Fuzz projects, 70 evaluation instances, and five models under matched budgets, RedHerring reduces real vul- nerabilities discovered by 38.7–60.4%. Trajectory analysis shows that agents spend 30.6–51.5% of completion tokens and an estimated 32.5–49.9% of runtime verifying decoys, showing that RedHerring redirects a substantial fraction of the fixed search budget toward decoys. When explicitly informed that decoys may be present, the agent adapts its search strategy, yet RedHerring still reduces vulner- abilities discovered by 37.2% relative to an informed Baseline, showing that its effectiveness does not depend on decoy secrecy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, cybersecurity, cyber deception, budget allocation, vulnerability discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HGTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, they proposed RedHerring, an approach to hinder LLM based vulnerability discovery by diverting agents verification budgets toward decoy code. Those decoy code guide LLM to an unsatisfiable guard that keeps the dangerous sink unreachable. They evaluated their approach on 70 instances from OSS-Fuzz projects; and found that the crashes were decreased by 60.4%.

Critical strengths
  1. Interesting idea on hindering LLM based vulnerability discovery.

  2. Performance analysis specially trajectories shows more consumption of completion tokens (around 60.4% increase) and runtime (49.9% increase).

Critical weaknesses
  1. Limited instances to evaluate. The manuscript only tested on 70 instances.

  2. They haven't discussed their approach in details. For example, how does vulnerability chains make the agent an attractive sink and how that impacts on vulnerability discovery. The same goes for false bridge. It is very difficult to conclude that the proposed approach is generalized or only work on a few samples.

  3. The same confusion hold valid for repository adaptation as well.

  4. Did they evaluated their attack in the presence of some existing defense?

  5. Attack motivation and threat model is not clear. Why do we need to limit the objective bounded by B?

  6. The manuscript has not investigated with existing static analysis tool. Or even PoCE (https://conf.researchr.org/details/issta-2026/issta-2026-research-papers/43/PoCE-Automated-Proof-of-Concept-Synthesis-using-Large-Language-Models-for-Robust-Val) CyberGym (https://www.cybergym.io) or ExploitGYM (https://www.cybergym.io/exploitgym/) would be beneficial.

Questions for authors
  1. Can you extend your analysis on a bigger dataset?

  2. Clarify the threat models (how much realistic it is) and motivation of the work.

  3. Robustness of the work is missing.

6 comments
AuthorsOpenReview ↗

Response to Reviewer HGTR (1/3)

We thank the reviewer for the constructive comments. Following the reviewer's suggestions, we doubled the evaluation set, compared RedHerring with existing defenses, evaluated whether static analysis tools can filter out the decoys, and tested RedHerring against stronger attackers. We address each point below.

W1: Limited instances to evaluate. The manuscript only tested on 70 instances.

Q1: Can you extend your analysis on a bigger dataset?

We doubled the evaluation to 140 instances and ran each instance five times. RedHerring reduces confirmed real vulnerabilities by 44.6%.

The 70 new instances were randomly selected from OSS-Fuzz and extend the evaluation from 33 to 69 projects. Each run audits an entire repository for up to 3 hours and 300 turns, so the expanded evaluation contains 1,400 runs, about 3,200 agent hours. Given the substantial API cost at this scale, we ran it on Qwen3.8-Flash, the model used in our informed-agent and ablation experiments.

The reduction also holds on the 70 newly added instances alone. The result is stable across runs: RedHerring confirms fewer real vulnerabilities than the Baseline in each of the five runs. The table gives confirmed real vulnerabilities for Qwen3.8-Flash, averaged over the five runs. We will add these results to the revision.

Evaluation setBaselineRedHerringReduction (95% CI)Paired pDecoy time shareDecoy token share
All 140 instances264.2146.444.6% (35.0–52.3%)< 0.00138.4%43.8%
70 newly added instances137.881.440.9% (28.1–49.8%)< 0.00134.6%43.9%

W2: They haven't discussed their approach in details. For example, how does vulnerability chains make the agent an attractive sink and how that impacts on vulnerability discovery. The same goes for false bridge. It is very difficult to conclude that the proposed approach is generalized or only work on a few samples.

W3: The same confusion hold valid for repository adaptation as well.

Vulnerability chain. The vulnerability chain reproduces patterns from real CVEs to match agents’ behavioral prior: their tendency to prioritize familiar vulnerability cues when deciding what to verify. Our empirical observations of agent trajectories show that, when performing vulnerability discovery, agents often begin by searching for typical vulnerability patterns or dangerous sinks, then assess and rank the matches for deeper investigation. We design vulnerability chains around this prior to draw verification effort toward decoys. Our ablation study shows that removing the chain lowers the reduction in real vulnerability discovery from 60.4% to 25.5% (RQ4).

False bridge. The false bridge makes the decoy’s apparent reachability depend on a condition built from a computationally hard problem, directing verification effort toward determining whether that condition can hold. Our empirical observation is that agents spend much of their vulnerability-verification effort analyzing reachability and constructing PoCs to confirm or rule out suspected bugs. The false bridge targets this workflow by placing the hard condition upstream of the vulnerability chain. Every path to the dangerous sink passes through an unsatisfiable guard. The defender can efficiently certify its unsatisfiability using private construction information, while the released code exposes only public parameters. Our ablation study shows that replacing the false bridge with a simple guard lowers the reduction in real vulnerability discovery from 60.4% to 32.9% (RQ4).

Repository adaptation. Repository adaptation aligns each decoy with the target repository’s semantics. The decoy should look like part of the repository’s own logic. It must also compile correctly and preserve the program’s intended behavior. We connect the chain to an existing input-processing function and use the repository’s types, calls, and naming. We place the bridge at a site that matches local operations and keep its condition and public parameters unchanged.

Generalization. Decoy insertion is independent of the repository’s structure, and RedHerring applies to all 140 instances from 69 OSS-Fuzz projects. To insert a decoy, RedHerring only needs a suitable attachment site in the project’s code and an existing variable there as the gate input. Insertion adds a single branch, guarded by the false bridge, that places the decoy on the program’s execution path. The original code stays unchanged. An LLM agent adapts the decoy to each repository, and a deterministic verifier checks every insertion, so no repository-specific rules are needed. The 69 projects range from network protocol analysis to compression (c-blosc2), and the decoys use 75 chain templates from different CVEs and four bridge constructions based on different hard problems.

AuthorsOpenReview ↗

Response to Reviewer HGTR (2/3)

Q2a: Clarify the threat models (how much realistic it is)

W5: Attack motivation and threat model is not clear. Why do we need to limit the objective bounded by B?

Each agent run costs money for API calls and compute, and an attacker has limited time to search a given target.

Why a budget bound. Because the defender does not know where the vulnerabilities are, they remain in the released repository, and an attacker with an unlimited budget could eventually find them. A practical defense therefore raises the attacker's cost, as obfuscation does [1]. The budget B makes this cost measurable: RedHerring reduces the real vulnerabilities an agent finds within the same budget.

Realistic. Evaluations of security agents bound each run. CyberGym limits each run to 100 agent steps [2], CyberGym-E2E sets USD 10 and 90 minutes per task [3], and ExploitGym imposes a two-hour timeout per task [4]. Industry evaluations also set explicit limits. OpenAI evaluates exploit development under a standard 300-turn setting [5], and each finalist team in DARPA's AI Cyber Challenge received USD 85,000 in Azure compute and USD 50,000 in LLM API credits [6]. Our budget of 3 hours and 300 turns per run matches or exceeds the per-run limits above.

Motivation. Frontier LLM agents have lowered the cost and expertise required for vulnerability discovery, which makes attack and defense increasingly asymmetric. An attacker can search entire repositories at low cost, whereas a defender cannot patch vulnerabilities it has not found. Existing defenses either require knowing which code to protect or act only after a vulnerability is found. RedHerring offers a new defense direction against vulnerability-discovery agents by treating their verification effort as a defense surface. It diverts this effort to decoys and requires no knowledge of whether or where the repository contains vulnerabilities. Each decoy shifts the asymmetry toward the defender: it is cheap to confirm as safe but costly for the agent to verify.

W4: Did they evaluated their attack in the presence of some existing defense?

We compared RedHerring with two existing techniques, adversarial comments [7] and opaque predicates from classic obfuscation [1]. Our results show that RedHerring reduces real vulnerability discovery substantially more than both techniques.

Adversarial comments. This defense keeps the code logic unchanged and inserts comments that mislead an LLM’s judgment of the code. We built a Comments-only version of each instance.

Opaque predicates. An opaque predicate is a condition whose value is fixed but hard to determine by static analysis. Tigress-Opaque keeps the same vulnerability chains and insertion sites as RedHerring and replaces only the false bridge with an always-false opaque predicate that Tigress [8] builds from data-structure invariants.

SettingConfirmed real vulnerabilitiesReduction
Baseline264–
Comments-only22216.0%
Tigress-Opaque19526.2%
RedHerring14644.6%

The table gives results for Qwen3.8-Flash on all 140 instances. With the same chains and insertion sites, RedHerring confirms 25.1% fewer real vulnerabilities than Tigress-Opaque. We will add these results to the revision.

References

[1] Collberg et al. A taxonomy of obfuscating transformations. Technical Report 148, The University of Auckland, 1997.

[2] Wang et al. CyberGym: Evaluating AI agents' real-world cybersecurity capabilities at scale. ICLR, 2026.

[3] Shi et al. CyberGym-E2E: Scalable real-world benchmark for AI agents' end-to-end cybersecurity capabilities. ICML, 2026.

[4] Wang et al. ExploitGym: Can AI agents turn security vulnerabilities into real attacks? arXiv:2605.11086, 2026.

[5] OpenAI. A thousand fewer paths to root, 2026.

[6] Zhang et al. SoK: DARPA's AI Cyber Challenge (AIxCC). arXiv:2602.07666, 2026.

[7] Thornton. Can adversarial code comments fool AI security reviewers? arXiv:2602.16741, 2026.

[8] Collberg. Tigress: A diversifying virtualizer/obfuscator for the C language. https://tigress.wtf.

AuthorsOpenReview ↗

Response to Reviewer HGTR (3/3)

W6: The manuscript has not investigated with existing static analysis tool. Or even PoCE (…) CyberGym (…) or ExploitGYM (…) would be beneficial.

We added an evaluation with static analysis tools and clarify how RedHerring relates to CyberGym, ExploitGym, and PoCE.

Static analysis does not identify the decoys as dead code. We randomly selected 50 instances and scanned them with Cppcheck and CodeQL, both before and after inserting five decoys per instance, which covers 250 decoys.

ToolAlerts before insertionAlerts after insertionAlerts in decoy codeDecoys with alertsGates reported as always false
Cppcheck22,38822,4263813 / 2500
CodeQL3,0023,00200 / 2500

Neither tool reports any gate as always false. The decoys also blend into the static analysis output. Cppcheck reports 38 alerts in decoy code, 0.17% of its 22,388 alerts on the original code, and CodeQL reports none. The 13 decoys with Cppcheck alerts appear as ordinary vulnerability candidates rather than dead code. We will add these results to the revision.

CyberGym [1] is a large-scale benchmark that evaluates AI agents on real-world vulnerabilities, and its tasks are built from OSS-Fuzz projects. Our instances are drawn from the same source.

ExploitGym [2] evaluates a later stage of an attack. Given an input that already triggers a known vulnerability, the agent must turn it into a working exploit. RedHerring targets the earlier stage, in which the agent discovers vulnerabilities.

PoCE. [3] We thank the reviewer for the suggestion to consider PoCE, which became available on October 1, 2026, during the review period. PoCE starts from a reported vulnerability and an initial successful PoC, and expands this PoC into multiple validated variants. It thus works after a vulnerability has been identified, whereas RedHerring acts earlier, while the agent is still searching the repository and choosing which candidates to verify. As the authors have not released the implementation for security reasons, we could not include it in our experiments. We will cite and discuss PoCE in the revision.

Q3: Robustness of the work is missing.

We address robustness in two dimensions. RedHerring remains effective across settings, including different models, programs, and runs. It also remains effective against stronger attackers.

Across settings. RedHerring reduces discovery for all five models, by 38.7% to 60.4% (RQ1). Across programs and runs, it reduces discovery by 44.6% on average over 140 instances with five runs each, and the reduction holds in each of the five runs and on the 70 newly added instances.

Against stronger attackers. When the agent is informed that decoys may be present (RQ3), it still finds 37.2% fewer vulnerabilities. We further added three experiments. When we remove the restriction on fuzzers and brute-force search from the audit prompt, RedHerring still reduces discovery by 35.4%. When the agent is told to skip any gate that never passes on its test inputs, it confirms 31.9% fewer vulnerabilities than the Baseline without this instruction. An attacker also cannot filter out the decoys with static analysis tools, since neither Cppcheck nor CodeQL reports any gate as always false.

If our understanding differs from the reviewer's concern, we would be grateful if the reviewer could point us to the specific aspect. We are happy to provide additional analysis during the discussion period and include it in the revision.

We hope these responses address the reviewer's concerns, and we would appreciate it if the reviewer could re-evaluate the paper in light of the new results.

References

[1] Wang et al. CyberGym: Evaluating AI agents' real-world cybersecurity capabilities at scale. ICLR, 2026.

[2] Wang et al. ExploitGym: Can AI agents turn security vulnerabilities into real attacks? arXiv:2605.11086, 2026.

[3] Tithy et al. PoCE: Automated proof-of-concept synthesis using large language models for robust validation. Proc. ACM Softw. Eng., 3(ISSTA), 2026.

Reviewer_azC6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RedHerring, a defense against LLM agents that search code repositories for vulnerabilities. The authors observe that agents spend most of their budget checking whether a suspected bug is real, not finding suspected bugs, so they can only check some of them. RedHerring adds decoys to the code to use up this checking budget. Each decoy is a fake bug path based on a real CVE, placed behind a gate that no input can pass. The gate is built on a hard math problem, so the defender can prove it is safe, but the attacker cannot easily tell. On 33 OSS-Fuzz projects, 70 instances, and five models, RedHerring reduces the number of real bugs found by 38.7–60.4%.

Critical strengths

The main strength is the new view of the problem. The paper treats the agent's checking effort as something a defender can attack. This means the defender does not need to know where the real bugs are. Earlier defenses either need the defender to know which code to protect, or only work after a bug is already found. Using a hard math problem lets the defender cheaply prove that each decoy is safe, while making the decoy costly for the attacker to rule out.

Critical weaknesses
  1. My main concern is that the defense is not shown to work against a smart attacker. Section 4.3 argues that if an attacker skips code behind such gates, it will also miss real bugs behind similar checks. This is only true if many real bugs can be reached only through a complex check, but the paper does not measure how common this is. If it is rare, an attacker can skip every cryptographic gate and lose almost nothing. Also, the attacker does not need to solve the math problem to spot a decoy. Because of the safety and behavior rules, every decoy gate never passes on real inputs, changes nothing when it fails, and uses hard-coded constants that are not used anywhere else. Real cryptographic checks do not look like this, so an attacker can spot decoys with simple program analysis or by running test inputs. In addition, the hardness proof in Appendix C assumes the gate input can be any value in [0, N). But in Figure 9b and Figure 5, the gate input is built by repeating one 32-bit value, so there are only about 2³² possible inputs. Trying all of them takes a few hours on one machine. The experiments do not show this because the audit prompt (Appendix F.1) forbids brute-force search. RQ3 only tells the agent that decoys may exist, and does not test any of these simple strategies. As a result, the claim that the defense does not depend on keeping decoys secret is not well supported.
  2. My second concern is that the evidence is weak. Each setting is run only once, with no variance, confidence intervals, or paired tests per instance. The correlation in Figure 4c (r = 0.99) uses only five points from the same runs. The ablation also does not fully match the claimed mechanism: Harmless-Control uses more tokens on inserted code (29.0%) than No-Chain (26.1%) and Simple-Gate (24.0%), but gives the smallest drop (14.1%). Finally, the key numbers depend on labels from Qwen3.8-Flash, which is also one of the tested models, and no human check is reported. Appendix B also counts unclear steps as verification, which may make the 73.8% and 11.1× numbers that motivate the paper too high.
Questions for authors
  1. How well does RedHerring work against an attacker that is told to skip gates that never pass on test inputs, have no effect when they fail, and use constants not used elsewhere?
  2. After adaptation, how many possible inputs can reach each type of gate? How well do the decoys work if the agent may write search or filtering scripts?
  3. Can you report results from repeated runs, with variance and paired tests?
  4. In RQ3, how much of the 37.2% drop comes from the decoys, and how much from the agent becoming more careful after reading the notice? The notice alone lowers the Baseline from 149 to 121.
6 comments
AuthorsOpenReview ↗

Response to Reviewer azC6 (1/4)

Thank you for the detailed analysis and constructive comments.

W1: My main concern is that the defense is not shown to work against a smart attacker. Section 4.3 argues that if an attacker skips code behind such gates, it will also miss real bugs behind similar checks. This is only true if many real bugs can be reached only through a complex check, but the paper does not measure how common this is. … Also, the attacker does not need to solve the math problem to spot a decoy. Because of the safety and behavior rules, every decoy gate never passes on real inputs, changes nothing when it fails, and uses hard-coded constants that are not used anywhere else. Real cryptographic checks do not look like this, so an attacker can spot decoys with simple program analysis or by running test inputs. … RQ3 only tells the agent that decoys may exist, and does not test any of these simple strategies. As a result, the claim that the defense does not depend on keeping decoys secret is not well supported.

We agree that our claim that RedHerring's "effectiveness does not depend on decoy secrecy" was not stated precisely enough. Our claim applies to attackers who know that decoys may be present, but do not know their specific implementations in advance. We will revise this sentence in the abstract and introduction accordingly and state this assumption in the threat model. We test the gate-skipping attacker directly in Q1, where RedHerring still reduces discovery by 31.9%.

More broadly, our core contribution is verification effort as a defense surface against vulnerability discovery agents. RedHerring makes vulnerability discovery costly by requiring agents to verify whether suspicious vulnerability paths are feasible. The cryptographic false bridge is one way to provide this cost, and it gives the largest reduction among the gates we tested (60.4%, RQ4). Importantly, RedHerring also retains effectiveness with a simpler gate. Simple-Gate uses an always-false condition whose falsity can be established through code inspection, yet it still reduces discovery by 32.9% (RQ4). These results show that RedHerring's effectiveness is not tied to a single gate construction, and that defenders can use different constructions across deployments.

We also compare the false bridge with a conventional opaque predicate [1] to examine whether its verification mechanism provides an advantage over traditional obfuscation. This variant keeps the same vulnerability chains and insertion sites as RedHerring and replaces only the false bridge with a classic opaque predicate generated by Tigress [2]. The table reports results for Qwen3.8-Flash on the expanded set of 140 instances (the original 70 and 70 new OSS-Fuzz instances).

SettingConfirmed real vulnerabilitiesReduction
Baseline264–
Tigress-Opaque19526.2%
RedHerring14644.6%

Beyond varying the gate constructions, repository adaptation makes it difficult for detectors based on known decoy patterns to generalize. Each decoy is distributed across multiple files, requiring a detector to locate its fragments and reconstruct the relevant code structure. Repository adaptation also tailors each decoy to the semantics of its attachment site, introducing structural and semantic variation across decoys.

Our evaluation shows that detectors trained on known gates generalize poorly to repository-adapted gates. A source-level decision tree and a CodeBERTScore-based [3] similarity detector recognize only 0% and 18.7% of the 150 adapted gates, respectively. In addition, we evaluated Cppcheck and CodeQL on 250 decoys across 50 instances. Neither tool identifies any gate as always false, although Cppcheck reports 38 alerts in decoy code, covering 13 of the 250 decoys. These results suggest that repository adaptation makes decoys difficult to identify using either learned detection rules or conventional static analysis.

References

[1] Collberg et al. A taxonomy of obfuscating transformations. Technical Report 148, The University of Auckland, 1997.

[2] Collberg. Tigress: A diversifying virtualizer/obfuscator for the C language. https://tigress.wtf.

[3] Zhou et al. CodeBERTScore: Evaluating code generation with pretrained models of code. EMNLP, 2023.

AuthorsOpenReview ↗

Response to Reviewer azC6 (2/4)

Q1: How well does RedHerring work against an attacker that is told to skip gates that never pass on test inputs, have no effect when they fail, and use constants not used elsewhere?

We added an experiment on the expanded set of 140 instances to evaluate this gate-skipping strategy. Even with this additional instruction, RedHerring reduces the number of confirmed real vulnerabilities by 31.9% relative to the standard Baseline.

To evaluate this strategy, we add the following instruction to the RedHerring audit prompt: "If a gate never passes on your test inputs, has no effect when it fails, and uses constants not used elsewhere, consider skipping it." The table reports results for Qwen3.8-Flash. All reductions are calculated relative to the standard Baseline.

SettingConfirmed real vulnerabilitiesReductionDecoy token shareDecoy time share
Baseline264.2–––
RedHerring146.444.6%43.8%38.4%
Skip-gate variant180.031.9%43.1%36.2%

The decoy token and time shares remain close to those without the instruction. Thus, even when explicitly instructed to skip suspicious gates, the agent continues to spend substantial effort analyzing the inserted decoys. We will include this experiment and the corresponding trajectory measurements in the revision.

W1: In addition, the hardness proof in Appendix C assumes the gate input can be any value in [0, N). But in Figure 9b and Figure 5, the gate input is built by repeating one 32-bit value, so there are only about 2³² possible inputs. Trying all of them takes a few hours on one machine. The experiments do not show this because the audit prompt (Appendix F.1) forbids brute-force search.

Q2: After adaptation, how many possible inputs can reach each type of gate? How well do the decoys work if the agent may write search or filtering scripts?

Gate input space. For all four gate types, RedHerring selects an attachment site and uses a native value from that site as the gate input through an adapter (Appendix C.1). The number of possible inputs is therefore the number of values this field can take. As the reviewer notes, the sites in Figures 5 and 9 supply a 32-bit length or size field, yielding at most 2³² possible inputs. By contrast, a site supplying a native 128-bit value provides up to 2¹²⁸ possible inputs, making exhaustive search computationally infeasible. Our source-level survey finds at least one native input source of 128 bits or more in all 140 instances across 69 projects. In the revision, we will add an example whose attachment site natively supplies a 128-bit input, and we will state input width as a criterion for selecting attachment sites.

Allowing search and fuzzing. To evaluate whether search-based strategies can bypass the gates, we conducted an additional experiment on the expanded set of 140 instances. We removed the audit-prompt restrictions on fuzzing and brute-force search for both Baseline and RedHerring, allowing the agent to use these strategies freely. The table reports results for Qwen3.8-Flash. Each reduction is calculated against the Baseline under the corresponding audit condition.

Audit settingReduction (95% CI)Paired p value
Standard audit prompt44.6% (35.0–52.3%)< 0.001
Fuzzing and brute-force search enabled35.4% (25.9–43.3%)< 0.001

With fuzzing and brute-force search enabled, RedHerring still reduces confirmed real vulnerabilities by 35.4% (p < 0.001). We will include this experiment in the revision.

AuthorsOpenReview ↗

Response to Reviewer azC6 (3/4)

W2: My second concern is that the evidence is weak. Each setting is run only once, with no variance, confidence intervals, or paired tests per instance. The correlation in Figure 4c (r = 0.99) uses only five points from the same runs. The ablation also does not fully match the claimed mechanism: Harmless-Control uses more tokens on inserted code (29.0%) than No-Chain (26.1%) and Simple-Gate (24.0%), but gives the smallest drop (14.1%). Finally, the key numbers depend on labels from Qwen3.8-Flash, which is also one of the tested models, and no human check is reported. Appendix B also counts unclear steps as verification, which may make the 73.8% and 11.1× numbers that motivate the paper too high.

Q3: Can you report results from repeated runs, with variance and paired tests?

Repeated runs. To assess the reliability of our results, we expanded the evaluation from 70 to 140 instances and performed five independent runs for each instance under both Baseline and RedHerring. RedHerring reduces the number of confirmed real vulnerabilities by 44.6% (95% CI: 35.0–52.3%, paired p < 0.001).

Given the substantial API cost of the repeated agent runs, we used only Qwen3.8-Flash for this expanded study. We evaluated both the original 70 instances and 70 newly sampled OSS-Fuzz instances under both conditions. The table reports the mean, standard deviation (SD), and variance of the total number of confirmed real vulnerabilities across the five runs over all 140 instances.

SettingRunsConfirmed real vulnerabilities (mean ± SD)VarianceReduction (95% CI)Paired p value
Baseline5264.20 ± 39.191535.70––
RedHerring5146.40 ± 19.58383.3044.6% (35.0–52.3%)< 0.001

Across the five runs, RedHerring consistently confirms fewer real vulnerabilities than Baseline, and the reduction also holds on the newly added instances. We will include these repeated-run results and the statistical analysis in the revision.

Figure 4c. We agree that five model-level points are insufficient to establish a robust correlation. We will remove the fitted line and correlation coefficient and retain the figure as a descriptive cross-model comparison. Across the five models, the decoy completion-token share and the reduction in vulnerability discovery exhibit the same ranking: models with higher decoy-token shares also show larger discovery reductions. We will revise the accompanying discussion to emphasize this observed empirical trend.

Ablation. Figure 7b measures the share of agent effort spent on inserted code, and this share counts both reading and verification. Harmless-Control inserts code without a dangerous sink, so the agent reads this code and has no vulnerability-related path to verify. Reading takes little budget from real candidates, which gives the smallest reduction. The full RedHerring combines both components and has both the largest share (51.5%) and the largest reduction (60.4%). The expanded 140-instance ablation preserves this order: RedHerring 44.6%, Simple-Gate 22.8%, No-Chain 17.4%, and Harmless-Control 13.4%.

Annotation labels. Our primary metric, the number of confirmed real vulnerabilities, does not rely on model annotation. A vulnerability counts only when its PoC reproduces a sanitizer-detected crash on replay, and duplicate findings are counted only once (Appendix E.3). For metrics that rely on model annotations, we manually labeled trajectories from 20 instances and compared our labels with the model-generated labels. The model-generated labels achieve 89.11% agreement with our manual labels (Cohen's κ = 0.624).

Ambiguous turns. We now report the results under both extreme assignments of ambiguous turns. Assigning every ambiguous turn to verification gives a verification share of 73.8% and a cost ratio of 11.1×. Assigning every ambiguous turn to hypothesis formation gives a cost ratio of 7.7×. Even under the conservative assignment, verification remains substantially more costly than hypothesis formation, with a cost ratio of 7.7×.

We will incorporate these results into the revision.

Reviewer_kGzbOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an obfuscation-based approach to defend software against vulnerability-discovery agents. Based on the observation that a large part of vuln-discovery costs is the verification, the paper's framework inserts decoys backed into code, thus diverting 30–50% of costs to decoy investigation. If vuln-discovery budget is fixed, this reduces confirmed real vulnerability discovery by 30–64%.

Critical strengths
  • The problem statement of vulnerability discovery as a resource-bounded process is interesting and seems novel.

  • Empirical validation appears thorough.

Critical weaknesses
  • Vuln-discovery agents may adapt by ranking vulnerability candidates based on their likelihood to contain a false bridge. This is tested in RQ3, but only minimally. I'd expect that the set of false-bridge techniques is finite in practice and as such a fast classifier can identify them and de-prioritize verification of any vuln candidates that contain them.

  • False bridges seems quite similar to Collberg's opaque predicates. Is there a reason they are distinct terms?

  • It is unclear whether the vuln-discovery rates and costs after RedHerring are normalized by the increase in code size. Since the amount of code is increased by RedHerring by 13% (RQ5), the agent has 13% more code the analyze within the same budget as before.

  • The measurement of 11x higher cost of verification vs discovery seems biased, as Appendix B states that "reported verification share is an upper bound and the formation share is a lower bound".

  • The evaluation numbers seems to be based on a single run for each data sample. Unless the runs are completely deterministic (how?), multiple seeds should be used and confidence intervals reported.

Questions for authors
  • What is the difference between "trajectory step", "agent round", and "trajectory turn"?

  • How do you distinguish delays introduced by the RQ3 prompt ("there may be decoys") and delays introduced by verification?

6 comments
AuthorsOpenReview ↗

Response to Reviewer kGzb (1/4)

We thank the reviewer for raising the questions on adaptive agents, opaque predicates, code size, and repeated runs.

Q1: What is the difference between "trajectory step", "agent round", and "trajectory turn"?

The three terms refer to the same unit, which we will consistently call a turn in the revision. A turn is one model response in the agent loop, including its thinking, text, and tool calls. Each turn is identified by its message ID (Appendix E.4). The 300-round limit (Section 5.1), the phase labels (Appendix B), and the decoy labels and both effort shares (Appendix E.4) are all counted in turns.

Q2: How do you distinguish delays introduced by the RQ3 prompt ("there may be decoys") and delays introduced by verification?

The 37.2% reduction is measured against a Baseline that receives the same decoy notice. We complement this matched comparison with a trajectory analysis that distinguishes suspicion of native code from effort spent on the inserted decoys.

Matched comparison. Both RQ3 conditions receive the notice that decoys may be present. The table gives confirmed real vulnerabilities for Qwen3.8-Flash on the original 70 instances (Figure 6a).

BaselineRedHerring
Standard prompt14959
Informed prompt12176

Thus, the 37.2% reduction is calculated by comparing RedHerring (76) against the informed Baseline (121), with the decoy notice held constant.

Trajectory analysis. We manually examined 20 trajectories from the informed runs to distinguish effort spent suspecting native code from effort spent analyzing inserted decoys. Turns in which the agent suspects native code to be a decoy account for only around 3% of agent effort in the informed Baseline and 4% in informed RedHerring. In contrast, analysis of inserted decoys accounts for 29.8% of completion tokens and 32.5%(Figure 6b) of runtime in informed RedHerring. This suggests that the additional overhead primarily comes from analyzing and verifying inserted decoys, rather than from the decoy notice itself. We will add these trajectory analysis results to the Appendix.

AuthorsOpenReview ↗

Response to Reviewer kGzb (2/4)

W1: Vuln-discovery agents may adapt by ranking vulnerability candidates based on their likelihood to contain a false bridge. This is tested in RQ3, but only minimally. I'd expect that the set of false-bridge techniques is finite in practice and as such a fast classifier can identify them and de-prioritize verification of any vuln candidates that contain them.

First, although the set of gate constructions is finite, repository adaptation makes the resulting decoy code diverse. The false bridge is one mechanism for creating asymmetric verification costs, and the four constructions presented in the paper instantiate this mechanism. Our core contribution is treating verification effort as a defense surface, and new constructions can extend it. Within each construction, the code of a decoy varies with its target repository. Repository adaptation rewrites the vulnerability chain using the target repository's own types, call structure, and naming conventions. It also places the bridge in a matching program context and distributes the decoy across multiple files.

Second, a fast classifier is hard to build. Each decoy is distributed across multiple files, requiring a classifier to locate its fragments and reconstruct the relevant code structure. Moreover, repository adaptation makes each decoy follow the semantics of its attachment site, resulting in substantial variation across decoys. Related evidence from code clone detection supports this. Semantics-preserving transformations challenge ML-based clone detectors [1, 2], and Type-4 clones, which share functionality but differ in syntax, remain hard for GPT-4 to detect [3]. Since the defender selects the constructions and adapts each decoy to its target repository, the resulting decoys can vary substantially across deployments, making it difficult for a classifier trained on known examples to generalize.

Third, we tested two fast classifiers, and they fail to recognize most adapted gates deployed by RedHerring. We evaluated a source-level decision tree and a CodeBERTScore-based similarity detector [4] on 150 gates from 30 instances across 15 projects. We evaluated each detector on two versions of the same 150 gates: the original, unadapted versions and versions adapted to their target repositories using RedHerring's adaptation procedure. To account for a defender who can query publicly available detectors before deployment, we also allowed the adaptation to use feedback from the CodeBERTScore detector. We measured false positives on 24,068 native functions.

DetectorRecall, unadapted gatesRecall, adapted gatesFalse-positive rate
Decision tree (two variants)100%0%≤ 0.0125%
CodeBERTScore100%18.7%0.162%

After adaptation, the decision trees recognize none of the gates, while CodeBERTScore recognizes only 28 of the 150. In 21 of the 30 instances, CodeBERTScore misses all five gates. Consequently, an agent relying on these detectors to filter or deprioritize candidates would still encounter most decoys as unflagged candidates.

References

[1] Zhang et al. Challenging machine learning-based clone detectors via semantic-preserving code transformations. IEEE TSE, 2023.

[2] Hooda et al. Auto-SPT: Automating semantic preserving transformations for code. arXiv:2512.06042, 2025.

[3] Zhang and Saber. Assessing the code clone detection capability of large language models. ICCQ, 2024.

[4] Zhou et al. CodeBERTScore: Evaluating code generation with pretrained models of code. EMNLP, 2023.

AuthorsOpenReview ↗

Response to Reviewer kGzb (3/4)

W2: False bridges seems quite similar to Collberg's opaque predicates. Is there a reason they are distinct terms?

Thank you for pointing out this connection. A false bridge is a special case of an opaque predicate that differs in the source of its verification hardness. Traditional opaque predicates rely on obscuring program properties to make their evaluation difficult for automated analysis [5]. Once an agent reconstructs the relevant data flow and predicate invariant, it can establish that the predicate is always false. In contrast, a false bridge grounds reachability verification in a computationally hard problem. Understanding its implementation therefore does not eliminate the computational effort required to establish that the vulnerability sink is unreachable.

To isolate the effect of the verification mechanism, we construct a Tigress-Opaque variant that preserves the vulnerability chains and insertion sites but replaces the false bridge with an always-false predicate generated by Tigress 4.2.1 [6]. The table reports the number of confirmed real vulnerabilities discovered by Qwen3.8-Flash on the expanded evaluation set of 140 instances (the original 70 instances and 70 new instances from OSS-Fuzz).

SettingConfirmed real vulnerabilitiesReduction
Baseline264–
Tigress-Opaque19526.2%
RedHerring14644.6%

The results show that the false bridge imposes a stronger barrier to vulnerability discovery than a conventional opaque predicate (44.6% vs. 26.2%). We will clarify this distinction in Section 2.2 and add the comparison to Section 5.5.

References

[5] Collberg et al. A taxonomy of obfuscating transformations. Technical Report 148, The University of Auckland, 1997.

[6] Collberg. Tigress: A diversifying virtualizer/obfuscator for the C language. https://tigress.wtf.

W3: It is unclear whether the vuln-discovery rates and costs after RedHerring are normalized by the increase in code size. Since the amount of code is increased by RedHerring by 13% (RQ5), the agent has 13% more code the analyze within the same budget as before.

Our reported results are not normalized by code size. In the expanded evaluation over 140 instances, the mean code growth is 13.3%, consistent with the 13.8% reported in RQ5 on the original 70 instances. Our expanded ablation results show that code growth alone cannot explain the reduction in vulnerability discovery.

We compare code growth and discovery results across the four variants in the expanded 140-instance evaluation, under the same audit budget.

VariantMean code growthDiscovery reduction
RedHerring13.3%44.6%
No-Chain12.3%17.4%
Simple-Gate1.5%22.8%
Harmless-Control0.5%13.4%

Code-size measurements cover all 140 instances. Each discovery reduction compares a variant with Baseline on the same set of instances. Baseline and RedHerring reference counts are summed per-instance averages over five runs.

No-Chain provides the closest comparison in added code volume. It retains the false bridges but removes the vulnerability chains, adding nearly as much code as RedHerring. With similar code growth, No-Chain reduces discovery by 17.4%, compared with 44.6% for RedHerring. This indicates that the vulnerability chains contribute to RedHerring’s effectiveness beyond simply increasing the code the agent must analyze.

Simple-Gate and Harmless-Control provide complementary comparisons with substantially smaller code additions. We will include these per-variant code-size measurements and matched discovery comparisons in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for the constructive comments and for recognizing the strengths of the paper. The reviewers find the idea interesting (HGTR) and the formulation of vulnerability discovery as a resource-bounded process novel (kGzb). Treating verification effort as a defense surface gives a new view of the problem and protects vulnerabilities whose locations are unknown (azC6).

New experiments on an expanded set of 140 instances show that RedHerring's effect holds across repeated runs and against stronger attackers, and that it exceeds the effect of existing techniques. We summarize the shared concerns here and give the details under each review.

1. Reliability of the evidence (kGzb W4/W5, azC6 W2/Q3, HGTR W1/Q1)

  • Expanded evaluation and statistical analysis : we expanded the evaluation from 70 to 140 instances, covering 69 OSS-Fuzz projects, and conducted five independent runs per instance for both Baseline and RedHerring. RedHerring reduces confirmed real vulnerabilities by 44.6% (95% CI 35.0–52.3%, paired p < 0.001). The reduction holds in each of the five runs and on the 70 newly added instances (40.9%).
  • Metric reliability : our primary metric counts distinct crash signatures confirmed through PoC replay with sanitizers and does not rely on model annotation. For the metrics that rely on annotation, the model-generated labels achieve 89.11% agreement with our manual labels (Cohen’s κ = 0.624).
  • Verification cost asymmetry : under the two assignments of ambiguous turns, verification costs 7.7× as much as hypothesis formation at the lower bound and 11.1× at the upper bound. The cost asymmetry thus persists even under the conservative assignment.

2. Stronger attackers (kGzb W1, azC6 W1/Q1/Q2, HGTR W6/Q3)

RedHerring remains effective under stronger attack strategies, while repository adaptation helps decoys evade detection by the evaluated methods.

  • Gate skipping : An agent instructed to skip suspicious gates still confirms 31.9% fewer real vulnerabilities with RedHerring than in the standard Baseline. Its reported decoy completion-token share remains 43.1%.
  • Fuzzing and Brute-force : With fuzzing and brute-force search enabled in both conditions, RedHerring still reduces confirmed real vulnerabilities by 35.4% relative to the Baseline (p < 0.001).
  • Decoy detection : The decision tree variants recognize none of the 150 adapted gates, while the CodeBERTScore detector recognizes 18.7%. In a separate evaluation of 250 decoys, neither Cppcheck nor CodeQL reports any gate as always false.

3. Comparison with existing techniques (kGzb W2, HGTR W4)

RedHerring reduces discovery by 44.6%, compared with 26.2% for Tigress opaque predicates using the same vulnerability chains and attachment sites, and 16.0% for adversarial comments alone.

The main changes made in the revision are summarized below:

  • New results (Section 5 and Appendix H). Section 5 reports the expanded-set results under the corresponding RQs, and Appendix H gives the full details.
  • Claim, design, and threat model (Abstract and Sections 1, 3.2, and 4). The abstract and introduction scope the informed-attacker claim to attackers who know that decoys may be present. Section 3.2 states the attacker’s knowledge and explains the budget bound. Section 4 motivates each component with the agent behavior it targets.
  • Measurement (Figure 4c and Appendices B, C.1, D.5, and E.4). We report both assignments of ambiguous turns, the agreement with manual labels, and the gate input space with a 128-bit example. Figure 4c now shows the cross-model comparison only, with the fitted line and correlation coefficient removed.
  • Terminology and related work (Section 2). We use “turn” throughout and discuss PoCE, ExploitGym, adversarial comments, and Tigress opaque predicates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.