acceptodds
Under review as a conference paper at ICLR 2027

MobileCybench: Evaluating Agent Vulnerability Discovery via Executable Probes

Abstract

AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application, and require considerable human labor to process. To mitigate this, we introduce a framework for evaluating vulnerability reports via probes, executable checks of security properties. A reported exploit is evaluated by replaying it against the application and running the probes: a triggered probe indicates both that the exploit succeeded and which security property it violated. As a probe encodes a security property rather than a known vulnerability, it can detect vulnerabilities that were not known when the probe was written. We instantiate the framework as MobileCybench, a benchmark for vulnerability discovery by AI agents in 13 Android applications, with 495 probes written and reviewed by the authors. We evaluate 5 coding agents (OpenCode with GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code with Opus 4.8 and Opus 5) under 4 settings: as a malicious app on the victim's device or as a remote attacker with a low-privilege account, each with either only an obfuscated APK or access to the application's source code. Given only the obfuscated APK, the top agent, OpenCode with GPT-5.6-Sol, triggers probes in 53.8% of applications in the malicious-app setting and 16.7% in the remote-attacker setting. With source code, the trigger rate across all agents and both attack settings increases from 28.8% to 32.8%. Building and running the benchmark surfaced 23 previously unreported vulnerabilities, the majority of which have been confirmed by maintainers.

Primary area:
datasets and benchmarks
Keywords:
AI Agents, Benchmark, Cybersecurity, Offense, Android, Risk, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2DoAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript introduces an approach to evaluate the vulnerability reports. They proceed with probing the security properties. Their introduced benchmark includes 13 Android Applications, with 495 probes. They evaluated their approach using five different agents. Evaluation result indicates a success rate with improvement of 5% (approx.) compared to the no-source code setting.

Critical strengths
  1. Identified 23 zero-day vulnerabilities.

  2. Introduced 495 probes which encodes both application and generic security properties.

Critical weaknesses
  1. Threat model is not easy to follow. Also it is not realistic. What is the attack assumption that the attacker will have access to the victim's device? And latest Android OS prevents app from communicating with each other. They work as a sandbox manner. How the attack will survive in those situation? The manuscript has not tested the attack on the real device.

  2. For the remote attacker, how they will get access to the credential? In figure 2, the manuscript mention that the victim will receive the forged intent from the attacker. How much realistic that scenario is? Can the research work provide more details?

  3. How many security probes are investigated by the manuscript? What is the comprehensiveness of the study?

  4. Comparison with the baseline is missing. There are many pen testing tool available for mobile devices. Why those will not work in this case? a. https://github.com/roomkangali/droid-llm-hunter b. https://github.com/mobsf/mobile-security-framework-mobsf c. https://github.com/linkedin/qark

  5. Contribution of the work is not clear. Most of the work in finding the trigger/exploit is done by LLM. Also I find it difficult the manuscript contribution in guiding the LLM as well.

  6. I know the manuscript mentioned about 30 author-hours per application. But what is the total author-hours effort? is it 495*30 hours?

  7. The robustness of those probes are not studied.

Questions for authors
  1. Can you illustrate the motivation and threat model of the study?

  2. Comparing against the baseline will provide reader more confidence about your approach.

  3. Have you analyzed the robustness of those probes?

2 comments

Weaknesses and questions

  1. Threat model is not easy to follow. Also it is not realistic. What is the attack assumption that the attacker will have access to the victim's device?

Thank you for the feedback. We have worked to improve our explanation of the threat model.

The basic setting is that we assume an unprivileged attacker who has network access and an unprivileged account on the server, and the obfuscated APK of the app. From there, it needs to attack the victim, without access to the victim’s account, phone, or any other information.

From there, we test how well attackers perform with greater access while maintaining realism. For instance, we evaluate how well the attacker can perform when they have their own android device with the app installed for testing (distinct from and without access to the victim’s device). Additionally, we evaluate the case where the victim accidentally installs a malicious attacker app (i.e. an accidental or mistaken install from the app store) where the target app still is isolated through the android operating system, but can be accessed through intents and other cross-application mechanisms.

For each of these methods, we evaluate the effect of providing source code, with and without a standard non-obfuscated APK and search capabilities.

And latest Android OS prevents app from communicating with each other. They work as a sandbox manner. How the attack will survive in those situation? The manuscript has not tested the attack on the real device.

Thanks for the suggestion. We tested all 12 malicious-app vulnerabilities on real devices, and all 12 worked as expected across a Pixel 9 running Android 16 (Appendix B.4).

The Android OS prevents apps from communicating with each other except through specified channels such as intents. Certain vulnerabilities here leverage those specified channels in inappropriate ways, e.g. sending an intent to control a user's home, even when it should not have access to it. Other vulnerabilities act through other channels such as the server-side and do not depend on the Android OS.

  1. For the remote attacker, how they will get access to the credential? In figure 2, the manuscript mention that the victim will receive the forged intent from the attacker. How much realistic that scenario is? Can the research work provide more details?

We assume that the remote attacker can sign up for an account like any other user to get an unprivileged non-admin account. This is fairly typical for attackers e.g. signing up for an ordinary user account to explore flows and to get a basic level of access. They do not have access to the victim's credentials.

In the malicious-app setting as in Figure 2, we assume that the victim installs the attacker's app. This is also typical e.g. from accidentally clicking on a download or downloading a malicious or compromised app for supposed functionality. Then the malicious app can send a broadcast using ordinary Android inter-app interfaces. In this case, Home Assistant’s location receiver accepted that broadcast without a permission or sender check, and this missing authorization is what allowed the unauthorized location update. The maintainers fixed it as CVE-2026-54318; Appendix D.6 gives the exploit and probe outcomes on vulnerable and patched builds.

Other weaknesses and questions

  1. How many security probes are investigated by the manuscript? What is the comprehensiveness of the study?

We evaluated 495 probes, comprising 414 application-specific probe entries and 81 enabled instances of 9 generic probe types (Section 3). They cover selected confidentiality, integrity, availability and access-control properties rather than an exhaustive specification of application security. On the selected independent set, the probes trigger on 12 of 20 public CVEs (60% recall; Appendix A.4).

  1. Comparison with the baseline is missing. There are many pen testing tool available for mobile devices. Why those will not work in this case? (a) droid-llm-hunter, (b) MobSF, (c) QARK.

Thanks for the suggestion. The key challenge with the pentesting tools is that they have lower precision i.e. they surface a lot of false positives. You can imagine this as a step after the pentesting tool, i.e. the pentesting tool can surface a potential vulnerability, which the validator can then assess on semantic validity.

We did run the pentesting baselines as a reference, though it's not entirely apples to apples since our probes will actually be passed in the reports (and the pentesting tools need to find them), and the pentesting tools surface a lot of noisy issues i.e. so the table is intended more for rough intuition.

AnalyzerFindings/app median (range)Total findings
MobSF97 (20–509)2,136
QARK988 (9–5,261)20,221
droid-llm-hunter7 (2–14)92

That is, we see that pentesting tools raise many issues (and it's hard to say which is correct) and only catch a fraction of the issues raised in the reports (which are likely humans + pentesting tools + AI and other tools).

Additionally, we added a baseline comparison for the agentic report validation layer, where this tool is more comparable. These are the results (Appendix C.7):

Existing set (selected benchmark reports)

MethodPrecisionRecall
Probes100%95%
Opus 5.586.7%65%
GPT-6.1-Sol85.7%60%
A2-style80%40%

Independent set (public CVEs)

MethodPrecisionRecall
Probes100%60%
Opus 5.5100%25%
GPT-6.1-Sol100%30%
A2-style100%15%
  1. Contribution of the work is not clear. Most of the work in finding the trigger/exploit is done by LLM. Also I find it difficult the manuscript contribution in guiding the LLM as well.

Thanks for the feedback, our core contributions are the validation framework and its Android instantiation.

  1. A validation framework. We introduce probes that validate LLM-generated vulnerability reports against security properties, including vulnerabilities unknown when the probes were written.

  2. An Android instantiation. We implemented the protocol in 13 runnable Android applications and used it to evaluate LLM agents’ vulnerability discovery.

The key value proposition is that this can be used for automatic validation of vulnerability reports. For instance, application maintainers are currently being flooded with vulnerability reports and the number of vulnerability reports is only increasing. By adopting probes, they can validate/deduplicate/categorize the vulnerability reports with high precision and recall, greater than classical pentesting tools (MobSF, QARK, droid-llm-hunter) and agentic tools (Opus 5.5, GPT-6.1-Sol, and A2 agentic judges).

The specific vulnerabilities are illustrative case studies to demonstrate that our system can be useful in the real world, rather than simply being theoretical.

  1. I know the manuscript mentioned about 30 author-hours per application. But what is the total author-hours effort? is it 495*30 hours?

The total is about 400 author-hours across all 13 applications, averaging about 30 hours per application.

  1. The robustness of those probes are not studied.

We tested the existing probes on 20 independent invalid reports that maintainers had closed as non-vulnerabilities or bugs without security impact. The probes trigger on none of these reports (0/20) (Appendix A.5). They also trigger on none of 25 no-op tests (0/25). On 20 independent valid public CVEs, the probes recall 12 (60%; Appendix A.4). Together, these experiments test the probes on valid, invalid, benign and legitimate activity (Appendix A.5).

Reviewer_r89JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MobileCybench, a benchmark comprising 13 Android applications and 495 executable probes. Agents submit either a malicious Android app or a remote-attacker script, which is replayed against a seeded environment. Hidden probes check for violations of application-specific or generic security properties. A separate patch-differential procedure attributes successful exploits to reference vulnerabilities.

The evaluation covers five agent systems, two attacker settings, and two access levels, with two attempts per configuration and 500 runs in total. There are 77 triggered configurations out of 250, corresponding to 124 triggered runs and 19 attributed vulnerabilities. Benchmark construction and evaluation together surfaced 23 previously unreported findings, of which 12 have maintainer validation. The paper clearly distinguishes these disclosure counts from the benchmark scores.

Critical strengths
  1. Runnable mobile apps, backend services, victim-state preparation, and malicious-APK replay require considerable engineering.

  2. Scoring demonstrated effects against security properties is a sensible approach. Keeping scoring separate from vulnerability attribution also avoids requiring a known vulnerability identifier before an exploit can receive credit.

  3. The evaluation includes real findings rather than only planted bugs. The reported maintainer acknowledgements, patches, and public CVEs give the work practical credibility.

Critical weaknesses
  1. The probe validation does not yet establish the reliability of the scoring oracle.

The paper argues that a triggered probe demonstrates a security violation. This requires both a correct security specification and a check that distinguishes a violation from other ways of reaching the observed state. Appendix A.6 reports no triggers in 25 no-op replays, which is useful for detecting spontaneous failures. However, a no-op does not test whether ordinary authorized actions, benign background activity, or attacker actions within the granted permissions are mistakenly scored as violations. The 120/124 patch-differential result in Appendix D.5 provides valuable evidence about the successful exploits. As the authors correctly note, it is attribution coverage rather than an active false-positive estimate. Most patches were written by the authors, and basic health checks do not establish that they preserve the relevant legitimate functionality.

  1. The evidence for scoring previously unseen vulnerabilities needs a cleaner separation from benchmark construction.

The properties were fixed before the scored grid, and the authors report that subsequent probe edits did not broaden acceptance and had zero net effect on the headline count. I appreciate these safeguards. Still, the properties were derived partly from pilot agent runs, and the reference bank contains findings surfaced during development or evaluation. The current aggregate results do not show how many distinct vulnerabilities were first discovered only after the relevant probes had been frozen, without any finding-driven modification to those probes or their seeded state.

Appendix A.5 also shows that only 19 of 24 authored reference exploits trigger the standard suite. Four misses are coverage gaps and one requires a live victim interaction. This is an informative limitation, but the reference set itself comes from the authors' findings. It does not establish how well the suite handles independently discovered violations.

A per-finding timeline would help: property definition, probe freeze, first discovery, and any subsequent edits. A prospective evaluation using a frozen suite and findings from an independent search would be more convincing. I am not asking for complete vulnerability recall; I am asking for direct evidence of the claimed advantage over checks designed around already observed findings.

  1. The source-access comparison is confounded.

Sections B.3 and C.3 show that APK-only and source-visible runs differ in source availability, APK obfuscation, and network policy. Source-visible runs can retrieve public material beyond the mounted snapshot. Section C.4 further acknowledges differences involving batches, reruns, and replay re-grades. These are meaningful changes in the agent's information and execution conditions.

The increase from 28.8% to 32.8% therefore cannot isolate how much source access helps, despite the motivation given in Section 4.1. The paper acknowledges this in the appendix, but the experimental framing should reflect it in the main text as well. A matched comparison should hold network access, APK behavior, scaffold version, and replay settings fixed while changing the source mount. Otherwise, the two conditions should be presented as bundled deployment settings, with no causal interpretation of their difference.

  1. The remote-attacker protocol needs a stricter boundary and a clearer account of mobile impact.

The remote replay container has ADB access, and agents also receive ADB during development. The audit of all 44 successful remote runs and the no-ADB replay of the four UI-driving exploits are useful checks. They show that those scored backend effects do not require device-side actions at replay. They do not establish that the exploit could have been developed using only observations available to the stated off-device attacker.

There is a second issue: the protocol clears the victim app's local state and prepares the victim after the remote script executes. All successful remote scores rest on backend state. This makes it difficult to tell how much of this track measures Android-client exploitation and how much measures ordinary backend authorization failures. It may also miss effects on existing sessions or local state that are erased during preparation; the live-victim miss in Appendix A.5 illustrates a related coverage limit.

I would prefer ADB to be disabled throughout remote-attacker generation and replay, or development-time device access to be explicitly included in the threat model. Please also separate backend-only findings from those whose security impact depends on Android-client execution.

  1. The released evidence supports only part of the reported evaluation.

The paper promises the harness, environments, and probes, but withholds submitted exploits and run logs. Only five public-CVE attribution tasks are released, compared with the 24-entry attribution bank used for the results. I understand the disclosure rationale. The consequence is that readers cannot currently reproduce most of the reported vulnerability attribution, refusal audit, or lookup audit from the public release.

Technical and Presentation Issues

  1. Bring the uncertainty in Appendix C.1 into the main results. No agent pair separates on the APK-only grid, and GPT-5.6-Sol and GPT-5.5 differ by only one triggered configuration when both access levels are pooled. "Highest observed score" would be a better description than language suggesting an established capability ranking.

  2. Explain what the 495 probes count. They comprise 414 application-specific checks and 81 enabled instances of nine generic check types. Readers should not interpret this as 495 distinct security concepts or independent evaluation tasks.

  3. The fact that generic probes never trigger is useful, but the generic layer does not run in the remote setting. Please scope this comparison accordingly and explain which generic checks could reasonably detect the observed effects. Zero generic triggers alone does not establish superiority over a matched alternative evaluator.

Questions for authors
  1. What active negative-control tests and independent annotations establish that probe triggers correspond to unauthorized behavior, rather than legitimate state changes? Can the relevant attribution patches pass feature-level regression tests as well as health checks?

  2. How many distinct vulnerabilities were first found after the corresponding property, probe implementation, and seeded state were frozen? Can these be separated from findings that informed construction or pilot runs?

  3. Can the source-access comparison be repeated with identical network policy, APK behavior, scaffold versions, and replay settings? If not, which conclusions about source availability remain supported?

  4. Can the remote track enforce an off-device boundary during both generation and replay, and report Android-dependent impact separately from backend-only violations?

3 comments

Weaknesses and questions

  1. The probe validation does not yet establish the reliability of the scoring oracle.

The paper argues that a triggered probe demonstrates a security violation. This requires both a correct security specification and a check that distinguishes a violation from other ways of reaching the observed state. Appendix A.6 reports no triggers in 25 no-op replays, which is useful for detecting spontaneous failures. However, a no-op does not test whether ordinary authorized actions, benign background activity, or attacker actions within the granted permissions are mistakenly scored as violations.

What active negative-control tests and independent annotations establish that probe triggers correspond to unauthorized behavior, rather than legitimate state changes?

Thanks for the feedback. As an additional negative control, we evaluated the probes on 20 exploits derived from independent reports that maintainers judged to be non-vulnerabilities or bugs without security impact. None of these reports triggered the probes. These demonstrate that the probes can distinguish valid vulnerability reports from false positives from a sample of real independent reports.

Most patches were written by the authors, and basic health checks do not establish that they preserve the relevant legitimate functionality.

Can the relevant attribution patches pass feature-level regression tests as well as health checks?

All relevant attribution patches pass the applications' health checks, and none of the patches with a unit-test suite introduces a behavioral regression. The only edit required from the patch is from adding new functionality which updated the device_id and required updating tests to match (Appendix D.9).

  1. The evidence for scoring previously unseen vulnerabilities needs a cleaner separation from benchmark construction.

The properties were fixed before the scored grid, and the authors report that subsequent probe edits did not broaden acceptance and had zero net effect on the headline count. I appreciate these safeguards. Still, the properties were derived partly from pilot agent runs, and the reference bank contains findings surfaced during development or evaluation. The current aggregate results do not show how many distinct vulnerabilities were first discovered only after the relevant probes had been frozen, without any finding-driven modification to those probes or their seeded state.

Appendix A.5 also shows that only 19 of 24 authored reference exploits trigger the standard suite. Four misses are coverage gaps and one requires a live victim interaction. This is an informative limitation, but the reference set itself comes from the authors' findings. It does not establish how well the suite handles independently discovered violations.

A per-finding timeline would help: property definition, probe freeze, first discovery, and any subsequent edits. A prospective evaluation using a frozen suite and findings from an independent search would be more convincing. I am not asking for complete vulnerability recall; I am asking for direct evidence of the claimed advantage over checks designed around already observed findings.

Thank you for the suggestion. To test whether the probes detect independently discovered violations, we ran our original, frozen probes on 20 independent public CVEs that did not inform their design. The probes triggered on 12 of 20 CVEs (60%), providing direct evidence that they detect vulnerabilities beyond already observed findings (Appendix A.4) and performs better than other methods we compared with (Opus 5.5 Judge at 25%, GPT-6.1-Sol at 30%, and A2-style at 15%).

Weaknesses and questions (continued)

  1. The source-access comparison is confounded.

Sections B.3 and C.3 show that APK-only and source-visible runs differ in source availability, APK obfuscation, and network policy. Source-visible runs can retrieve public material beyond the mounted snapshot...The increase from 28.8% to 32.8% therefore cannot isolate how much source access helps, despite the motivation given in Section 4.1. The paper acknowledges this in the appendix, but the experimental framing should reflect it in the main text as well. A matched comparison should hold network access, APK behavior, scaffold version, and replay settings fixed while changing the source mount. Otherwise, the two conditions should be presented as bundled deployment settings, with no causal interpretation of their difference.

Thank you for the suggestion. We isolated the effect of source access with a matched comparison: for all 5 agents, we added only source code to the APK-only condition and held network access, APK behavior, scaffold version, reasoning effort and budget fixed. The trigger rate rises from 29% to 34% (Appendix C.3).

Additionally, we’ve updated our presentation of the original APK-only and source-visible runs as bundled deployment settings, with no causal interpretation of their difference.

  1. The remote-attacker protocol needs a stricter boundary and a clearer account of mobile impact...I would prefer ADB to be disabled throughout remote-attacker generation and replay, or development-time device access to be explicitly included in the threat model. Please also separate backend-only findings from those whose security impact depends on Android-client execution.

Thank you for the feedback. We added a new baseline, a no-device remote-attacker setting, with no mobile or ADB access during development or replay. We re-ran all 5 agents on the 12 remote-attacker applications in this setting.

Even in the no-device setting, the agents still find a client-dependent vulnerability, an Audiobookshelf stored-XSS that executes when the victim's app renders the payload; the other vulnerabilities are backend-only.

In the own-device setting, the attacker's own Android device is used only for development (Section 2.1), and we remove the attacker's ADB during replay. Separating backend-only findings from those whose impact depends on Android-client execution, half of the remote-attacker vulnerabilities are client-dependent and half are backend-only (Appendix B.6).

There is a second issue: the protocol clears the victim app's local state and prepares the victim after the remote script executes. All successful remote scores rest on backend state. This makes it difficult to tell how much of this track measures Android-client exploitation and how much measures ordinary backend authorization failures. It may also miss effects on existing sessions or local state that are erased during preparation; the live-victim miss in Appendix A.5 illustrates a related coverage limit.

Thanks for pointing this out. We re-scored all remote-attacker runs without clearing the victim app's local state, and the scores were unchanged, so they do not rest only on the cleared state. See the discussion above for the backend-only versus client-dependent split (Appendix B.6).

  1. The released evidence supports only part of the reported evaluation.

The paper promises the harness, environments, and probes, but withholds submitted exploits and run logs. Only five public-CVE attribution tasks are released, compared with the 24-entry attribution bank used for the results. I understand the disclosure rationale. The consequence is that readers cannot currently reproduce most of the reported vulnerability attribution, refusal audit, or lookup audit from the public release.

Thanks, the challenge of cybersecurity research is that we need to weigh the value of transparency for defenders and the research community with the cost of malicious use by malicious attackers, especially now that zero days and AI-driven cyber attacks are increasingly relevant.

To help improve transparency for defenders and the research community while mitigating risks, we have decided to release 20 anonymized agent-run logs, added to the supplementary zip. These logs are our attempt to balance transparency with risk, i.e. selecting illustrative logs to cover all five agents across the threat model, and recording the agents' reasoning and actions across successful, unsuccessful and incomplete attempts, including a refusal and changes of approach. These logs are for vulnerabilities that have already been fixed; we release the rest as their fixes land, rather than publish a working exploit for an issue still being patched.

Technical and Presentation Issues

Bring the uncertainty in Appendix C.1 into the main results. No agent pair separates on the APK-only grid, and GPT-5.6-Sol and GPT-5.5 differ by only one triggered configuration when both access levels are pooled. "Highest observed score" would be a better description than language suggesting an established capability ranking.

Section 4 now describes the result as the "highest observed trigger rate" and states that no agent pair differs significantly on the APK-only grid. Appendix C.1 reports the uncertainty and pairwise tests.

Explain what the 495 probes count. They comprise 414 application-specific checks and 81 enabled instances of nine generic check types. Readers should not interpret this as 495 distinct security concepts or independent evaluation tasks.

Section 3 now states that the 495 probes comprise 414 application-specific checks and 81 enabled instances of 9 generic check types rather than 495 distinct security concepts or evaluation tasks.

The fact that generic probes never trigger is useful, but the generic layer does not run in the remote setting. Please scope this comparison accordingly and explain which generic checks could reasonably detect the observed effects. Zero generic triggers alone does not establish superiority over a matched alternative evaluator.

Thanks, we've softened the claim to state that empirically, application-specific checks are triggered substantially more often than generic checks. One perspective is that generic checks can often be covered by standard security tools in the development process vs application-specific security semantics which generic tools cannot cover by default. Generic checks can be valuable for more universal security properties e.g. server unavailability, memory leaks, etc.

Reviewer_y4crOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a probe-based framework for evaluating exploits by checking whether they violate application security properties. The framework can score exploits of previously uncatalogued vulnerabilities, including application-specific authorization and business logic flaws. The authors wrote and reviewed 495 probes and constructed MobileCybench, a benchmark comprising 13 Android applications. They evaluated five coding agents’ vulnerability discovery and exploitation capabilities under two attack settings: malicious apps and remote attackers.

Critical strengths

I believe this paper addresses an important topic: determining whether bug reports are valid. Reliable validation is a prerequisite for using these reports to assess agents’ actual offensive capabilities. This direction has considerable potential.

Another strength of the paper is the substantial engineering and validation effort, including setting up the applications and their frontend and backend environments, as well as constructing the probes.

Critical weaknesses

However, I have several concerns regarding this work, primarily concerning the experimental evaluation.

Although the reference-exploit coverage, no-op checks, and patch-differential verification provide useful evidence, they do not establish precision and recall on an independent evaluation set. Such an evaluation would establish how reliably the method distinguishes valid from invalid exploits and provide evidence of its coverage beyond the authors’ own reference findings.

A second gap in the experimental evaluation is the lack of a systematic comparison with alternative validation methods (particularly agentic validation methods) on the same set of reports. It remains unclear whether existing approaches could achieve comparable or better accuracy at lower cost or with shorter validation times. The comparison should account for both probe execution costs and the upfront effort required to construct them.

Third, the limitations of manually authored probes warrant deeper analysis. Although the paper reports reference-exploit coverage and initial authoring effort, it does not sufficiently characterize coverage on independent findings, maintenance costs as applications evolve, or the false negatives caused by missing probes. These omissions also make it difficult to assess how probe coverage affects the comparison of agents’ offensive capabilities.

Questions for authors
  1. Could you evaluate the probe-based method’s precision and recall on an independent set of valid and invalid vulnerability reports with executable PoCs, to distinguish agent failures from probe false negatives?

  2. How does your method compare with alternative validation approaches on the same set of reports in terms of accuracy, validation time, and cost, including the upfront effort required to construct the probes?

2 comments

Weaknesses and questions

  1. Although the reference-exploit coverage, no-op checks, and patch-differential verification provide useful evidence, they do not establish precision and recall on an independent evaluation set. Such an evaluation would establish how reliably the method distinguishes valid from invalid exploits and provide evidence of its coverage beyond the authors' own reference findings.

Thank you for the thoughtful suggestion. We evaluated the probes on an independent set of 20 valid (public CVEs) and 20 invalid (maintainer-rejected) reports and precision is 100% and recall is 60% (Appendices A.4 and A.5).

We expect probe quality and coverage to improve with the system over time, much as traditional CI/CD and testing coevolve with the software they protect.

  1. A second gap in the experimental evaluation is the lack of a systematic comparison with alternative validation methods (particularly agentic validation methods) on the same set of reports. It remains unclear whether existing approaches could achieve comparable or better accuracy at lower cost or with shorter validation times. The comparison should account for both probe execution costs and the upfront effort required to construct them.

We compared our method with agentic LLM judges (Opus 5.5 on Claude Code and GPT-6.1-Sol on Codex) and an A2-style agentic validator on the same reports, on both our existing and independent evaluation sets (Appendix C.7).

Existing set (selected benchmark reports)

MethodPrecisionRecall
Probes100%95%
Opus 5.586.7%65%
GPT-6.1-Sol85.7%60%
A2-style80%40%

Independent set (public CVEs)

MethodPrecisionRecall
Probes100%60%
Opus 5.5100%25%
GPT-6.1-Sol100%30%
A2-style100%15%

Our method outperforms the other methods on both precision and recall. The reason why alternative methods have lower precision on our existing set is because our existing set has more reports that rely on specific application-semantics that alternative methods mark as false positives but probes are able to differentiate.

In terms of validation time, each probe verdict took a median of 3.8 minutes compared to 1.8 minutes for an Opus 5.5 judge verdict, 0.7 minutes for a GPT-6.1-Sol judge verdict, and 4.5 minutes for an A2-style verdict, which needs the same replay as the probes plus a median 0.7 minutes of validation. We did not invest in efficiency and probes can be optimized to be faster.

Our method also is more cost efficient starting from about 410 reports per application. Prior to that, probes have a higher upfront cost of about 30 author-hours per application compared to 5 hours for the judge and 10 hours for A2-style agentic validation across all application. However, the significant cheaper costs after initial probe creation makes probes more efficient for larger real-world codebases.

This is similar to testing, where the cost of investing in good testing is initially more expensive than code review, but the cost is amortized over the lifetime of an application.

  1. Third, the limitations of manually authored probes warrant deeper analysis. Although the paper reports reference-exploit coverage and initial authoring effort, it does not sufficiently characterize coverage on independent findings, maintenance costs as applications evolve, or the false negatives caused by missing probes. These omissions also make it difficult to assess how probe coverage affects the comparison of agents' offensive capabilities.

Thanks for the thoughtful comment; we agree that a key challenge of software comes from maintenance as applications evolve and concerns about missing probes. The analogy to testing and CI/CD is relatively apt here as well; you need to continue to update your tests as your applications evolve, and need to be concerned about application failures due to missing test coverage.

One promising point is that probes can be relatively robust to code implementation insofar as security and semantic behavior remains invariant. For instance, all the security logic of the probes remained invariant across 18 code versions across 13 applications and we only needed to change a single API URL in the seeding logic and updating version pins as needed.

As false negatives are caught and raised, application developers can convert those cases into new probes and coverage should increase as developers increasingly invest in probes.

Additionally, as security properties become increasingly specified and AI agents become more powerful, more probes can be automated and the challenge will move to defining application-specific security semantics. One can imagine a workflow where automated report triage includes a combination of probes and other tools such as agentic-validators. Then, if certain high-priority probes are triggered, they can be routed and triaged with high urgency. If there are false negatives that probes miss but are covered via other methods, agents can work with the developers to specify the missing property and define new probes to ensure such classes of reports are covered in the future.

Comments on the paper

1 comment

We really appreciate all the thoughtful feedback and suggestions from the reviewers. We have made the following improvements:

  1. We have demonstrated our method is effective on an independent set of reports. We constructed an independent set of 40 reports from real-world vulnerabilities consisting of 20 valid and 20 invalid reports. Our method has 100% precision and 60% recall on these reports, and can likely improve recall with improved probe quality and coverage.

  2. We have compared against other methods and demonstrated higher precision and recall. We compared our method against agentic LLM judges (Opus 5.5 on Claude Code and GPT-6.1-Sol on Codex) and an A2-style agentic validator and found that our method scored at least as well as all other methods.

  3. We have added additional ablations exploring the effect of various variables more deeply. In particular, we have added ablations to better explore the effect of source access, ADB, mobile access, etc.

  4. We have run on real Android devices to demonstrate that our method works on real devices.

  5. We have released a representative sample of 20 logs to help provide more transparency and clarity over how agents ran, while balancing the risks of malicious use.

  6. We have made general improvements regarding clarity, transparency, and other quality-of-life improvements.

We truly appreciate the thoughtfulness and care that each reviewer put into helping us improve our work, and are really grateful for the opportunity to improve our work during this period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.