- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This manuscript introduces an approach to evaluate the vulnerability reports. They proceed with probing the security properties. Their introduced benchmark includes 13 Android Applications, with 495 probes. They evaluated their approach using five different agents. Evaluation result indicates a success rate with improvement of 5% (approx.) compared to the no-source code setting.
Critical strengths
-
Identified 23 zero-day vulnerabilities.
-
Introduced 495 probes which encodes both application and generic security properties.
Critical weaknesses
-
Threat model is not easy to follow. Also it is not realistic. What is the attack assumption that the attacker will have access to the victim's device? And latest Android OS prevents app from communicating with each other. They work as a sandbox manner. How the attack will survive in those situation? The manuscript has not tested the attack on the real device.
-
For the remote attacker, how they will get access to the credential? In figure 2, the manuscript mention that the victim will receive the forged intent from the attacker. How much realistic that scenario is? Can the research work provide more details?
-
How many security probes are investigated by the manuscript? What is the comprehensiveness of the study?
-
Comparison with the baseline is missing. There are many pen testing tool available for mobile devices. Why those will not work in this case? a. https://github.com/roomkangali/droid-llm-hunter b. https://github.com/mobsf/mobile-security-framework-mobsf c. https://github.com/linkedin/qark
-
Contribution of the work is not clear. Most of the work in finding the trigger/exploit is done by LLM. Also I find it difficult the manuscript contribution in guiding the LLM as well.
-
I know the manuscript mentioned about 30 author-hours per application. But what is the total author-hours effort? is it 495*30 hours?
-
The robustness of those probes are not studied.
Questions for authors
-
Can you illustrate the motivation and threat model of the study?
-
Comparing against the baseline will provide reader more confidence about your approach.
-
Have you analyzed the robustness of those probes?
2 comments
Other weaknesses and questions
- How many security probes are investigated by the manuscript? What is the comprehensiveness of the study?
We evaluated 495 probes, comprising 414 application-specific probe entries and 81 enabled instances of 9 generic probe types (Section 3). They cover selected confidentiality, integrity, availability and access-control properties rather than an exhaustive specification of application security. On the selected independent set, the probes trigger on 12 of 20 public CVEs (60% recall; Appendix A.4).
- Comparison with the baseline is missing. There are many pen testing tool available for mobile devices. Why those will not work in this case? (a) droid-llm-hunter, (b) MobSF, (c) QARK.
Thanks for the suggestion. The key challenge with the pentesting tools is that they have lower precision i.e. they surface a lot of false positives. You can imagine this as a step after the pentesting tool, i.e. the pentesting tool can surface a potential vulnerability, which the validator can then assess on semantic validity.
We did run the pentesting baselines as a reference, though it's not entirely apples to apples since our probes will actually be passed in the reports (and the pentesting tools need to find them), and the pentesting tools surface a lot of noisy issues i.e. so the table is intended more for rough intuition.
| Analyzer | Findings/app median (range) | Total findings |
|---|---|---|
| MobSF | 97 (20–509) | 2,136 |
| QARK | 988 (9–5,261) | 20,221 |
| droid-llm-hunter | 7 (2–14) | 92 |
That is, we see that pentesting tools raise many issues (and it's hard to say which is correct) and only catch a fraction of the issues raised in the reports (which are likely humans + pentesting tools + AI and other tools).
Additionally, we added a baseline comparison for the agentic report validation layer, where this tool is more comparable. These are the results (Appendix C.7):
Existing set (selected benchmark reports)
| Method | Precision | Recall |
|---|---|---|
| Probes | 100% | 95% |
| Opus 5.5 | 86.7% | 65% |
| GPT-6.1-Sol | 85.7% | 60% |
| A2-style | 80% | 40% |
Independent set (public CVEs)
| Method | Precision | Recall |
|---|---|---|
| Probes | 100% | 60% |
| Opus 5.5 | 100% | 25% |
| GPT-6.1-Sol | 100% | 30% |
| A2-style | 100% | 15% |
- Contribution of the work is not clear. Most of the work in finding the trigger/exploit is done by LLM. Also I find it difficult the manuscript contribution in guiding the LLM as well.
Thanks for the feedback, our core contributions are the validation framework and its Android instantiation.
-
A validation framework. We introduce probes that validate LLM-generated vulnerability reports against security properties, including vulnerabilities unknown when the probes were written.
-
An Android instantiation. We implemented the protocol in 13 runnable Android applications and used it to evaluate LLM agents’ vulnerability discovery.
The key value proposition is that this can be used for automatic validation of vulnerability reports. For instance, application maintainers are currently being flooded with vulnerability reports and the number of vulnerability reports is only increasing. By adopting probes, they can validate/deduplicate/categorize the vulnerability reports with high precision and recall, greater than classical pentesting tools (MobSF, QARK, droid-llm-hunter) and agentic tools (Opus 5.5, GPT-6.1-Sol, and A2 agentic judges).
The specific vulnerabilities are illustrative case studies to demonstrate that our system can be useful in the real world, rather than simply being theoretical.
- I know the manuscript mentioned about 30 author-hours per application. But what is the total author-hours effort? is it 495*30 hours?
The total is about 400 author-hours across all 13 applications, averaging about 30 hours per application.
- The robustness of those probes are not studied.
We tested the existing probes on 20 independent invalid reports that maintainers had closed as non-vulnerabilities or bugs without security impact. The probes trigger on none of these reports (0/20) (Appendix A.5). They also trigger on none of 25 no-op tests (0/25). On 20 independent valid public CVEs, the probes recall 12 (60%; Appendix A.4). Together, these experiments test the probes on valid, invalid, benign and legitimate activity (Appendix A.5).
Weaknesses and questions
Thank you for the feedback. We have worked to improve our explanation of the threat model.
The basic setting is that we assume an unprivileged attacker who has network access and an unprivileged account on the server, and the obfuscated APK of the app. From there, it needs to attack the victim, without access to the victim’s account, phone, or any other information.
From there, we test how well attackers perform with greater access while maintaining realism. For instance, we evaluate how well the attacker can perform when they have their own android device with the app installed for testing (distinct from and without access to the victim’s device). Additionally, we evaluate the case where the victim accidentally installs a malicious attacker app (i.e. an accidental or mistaken install from the app store) where the target app still is isolated through the android operating system, but can be accessed through intents and other cross-application mechanisms.
For each of these methods, we evaluate the effect of providing source code, with and without a standard non-obfuscated APK and search capabilities.
Thanks for the suggestion. We tested all 12 malicious-app vulnerabilities on real devices, and all 12 worked as expected across a Pixel 9 running Android 16 (Appendix B.4).
The Android OS prevents apps from communicating with each other except through specified channels such as intents. Certain vulnerabilities here leverage those specified channels in inappropriate ways, e.g. sending an intent to control a user's home, even when it should not have access to it. Other vulnerabilities act through other channels such as the server-side and do not depend on the Android OS.
We assume that the remote attacker can sign up for an account like any other user to get an unprivileged non-admin account. This is fairly typical for attackers e.g. signing up for an ordinary user account to explore flows and to get a basic level of access. They do not have access to the victim's credentials.
In the malicious-app setting as in Figure 2, we assume that the victim installs the attacker's app. This is also typical e.g. from accidentally clicking on a download or downloading a malicious or compromised app for supposed functionality. Then the malicious app can send a broadcast using ordinary Android inter-app interfaces. In this case, Home Assistant’s location receiver accepted that broadcast without a permission or sender check, and this missing authorization is what allowed the unauthorized location update. The maintainers fixed it as CVE-2026-54318; Appendix D.6 gives the exploit and probe outcomes on vulnerable and patched builds.