- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents MobileJailBench, an evolving jailbreak benchmark for MLLM-based mobile agents built on a browser-based, Android-like simulator covering 28 apps, with 520 tasks across 14 risk categories organized into six sub-benchmarks (Benign, Explicit, Implicit, Benign-First and Harm-First Composite, and Expansion), each equipped with a programmatic oracle that compares initial and final device states. It introduces Dual-Dynamic Evolution, in which FDJE routes failed attacks by failure type (task-flow camouflage for refusals, witness-guided navigation for execution failures, and implicit-track evolution that moves harmful content into environmental carriers and iteratively refines prefixes), while WDBE converts observed weaknesses into new tasks that pass four reviews, automated code and judge-test generation, and GUI validation before entering the next frozen release. It also proposes SAF, a stratified adjudication framework combining rule-based anchor matching, two isolated judgments from the same model with verbatim evidence citations, and evidence auditing with conservative arbitration, yielding metrics such as TSR, ASR, SVR, RHR, and SRR. Across 20 agent–model pairings (GenericV2 and M3A × 10 MLLMs), implicit tasks reduce mean SRR from 26.8% to 6.6% and raise SVR from 68.9% to 86.6% while ASR slightly decreases (59.8% to 56.4%); FDJE raises cross-model coverage from 69.6% to 85.9%, reaching 100% on Qwen3.5-122B; the 50 WDBE tasks yield 36% to 82% first-run ASR on nine other models; and SAF reaches 97.26% agreement with the reference, versus 91.14% for a single LLM judge.
Critical strengths
-
The engineering design of an "evolving benchmark" is novel. The frozen-release boundary explicitly ensures that new tasks enter only the next release and never change scores in the current one. WDBE distinguishes generated from certified task counts and requires version-matched GUI execution evidence. This offers a practical scheme for updating a benchmark as models improve while preserving comparability.
-
Multi-dimensional metrics reveal phenomena that ASR misses. On implicit tasks, ASR drops slightly, yet the refusal rate falls sharply and the violation rate rises, showing that ASR alone underestimates risk. SAF separately records expressed adoption, timely refusal, risky actions, and final consequences, providing substantially more information than binary judgments.
-
SAF's evidence audit is a valuable design. Requiring the judge to quote the trajectory verbatim and verifying each quote by exact substring matching directly rules out judgments based on fabricated evidence. The appendix further identifies a "refusal anchoring" problem in single judges, where later refusal-like wording overshadows earlier objective adoption, and shows that voting and averaging cannot correct this shared bias.
Critical weaknesses
-
Headline numbers come from non-comparable protocols, and the main text overstates them. The "100% ASR on Qwen3.5-122B" is task-level coverage under multiple adaptive search attempts with first-success stopping (pass@budget), not single-episode ASR, and the additional cost ranges from 116 to 2,278 episodes per model. For the comparison between FDJE and baselines such as Tree, BoN, and PAIR, the appendix acknowledges differing permissions and budgets that preclude an equal-budget superiority claim, yet the abstract and main text still present 100% as a headline result. The strategy ablation in Figure 3(b) is further "projected to a 70-task scale with 60 assumed successes" from 10 residual tasks, rather than measured.
-
Judge validation relies on a self-constructed reference, raising circularity concerns. SAF's 97.26% agreement is computed against a "constructed reference" that combines cross-arm agreement, endpoint checks, only five manually inspected boundary tasks, and re-adjudication, rather than independent human annotation. The appendix itself states that it reports "reference agreement rather than independent annotation accuracy." The two "independent" L1 judgments come from the same model, which the authors acknowledge are not independent annotators. Without an independent human gold standard, the claim that SAF outperforms a single judge is not very convincing.
-
The explicit-versus-implicit comparison is confounded by task content. Explicit (T0001–T0070) and Implicit (T0071–T0140) are two different task sets, not paired rewrites of the same harmful objectives. The finding that implicit tasks yield lower refusal and higher violation rates may therefore partly reflect differences in task content, difficulty, or risk distribution rather than presentation alone. This is one of the paper's central findings, yet it lacks support from a paired design.
-
The 520-task scale is partly inflated. The two Composite sub-benchmarks contribute 280 tasks, derived from only 28 attack sources × 5 benign workflows × 2 orders, and share objectives with the core tasks; 266 of the 322 cross-app tasks come from these concatenated tasks. The number of independent harmful objectives is roughly 140 core tasks plus 50 expansion tasks. In addition, each task in the main experiments is run only once (pass@2 is used only in some experiments), and the paper does not report whether the number of repetitions behind its confidence intervals is sufficient.
-
Simulator fidelity is not validated. The environment is a browser-simulated "Android-like" interface, and the authors acknowledge that it abstracts away real Android process isolation and live service behavior. Although the paper repeatedly describes it as "high-fidelity," it provides no comparison against real apps or physical devices. Whether violations observed in the simulator transfer to real apps therefore remains unknown.
-
Attack components conflate jailbreaking with capability assistance. Witness-guided navigation supplies same-app operation examples and hints to help agents that failed during execution complete the task. Part of the resulting ASR gain thus compensates for limited GUI capability rather than bypassing safety alignment. The main text also describes this component as improving "less capable agents' ability to recognize, resist, and refuse malicious instructions," which contradicts its actual role (described in the appendix as providing execution support).
-
WDBE's independence and certification are incomplete. Expansion tasks are driven by weaknesses exposed during development, with a deliberately uneven distribution (Mail appears in 23 of 50 tasks, and 39 are L1). The appendix records pass@2 of 48/50 and notes that the earlier 50/50 result cannot serve as certification after specification or oracle revisions. The main text does not clarify which model's failures drive WDBE or whether selection bias arises when transferring to other models.
-
Poor readability, with much key information deferred to the appendix. The main text introduces many acronyms (FDJE, WDBE, SAF, CPSD, SVR, RHR, SRR, SU), and Figure 2 is so dense that it requires zooming in to read. Key details such as budgets, reference construction, and task pairing can only be understood by consulting the appendix. Nine pages of main text can hardly support this many components on their own, making it difficult for readers to tell which component drives which conclusion.
Questions for authors
-
Could you compare FDJE with baselines such as Tree, BoN, and PAIR under the same episode budget and the same permissions? What is FDJE's ASR when it is also limited to a single episode?
-
Could you have at least two independent annotators label a randomly sampled (non-enriched) set of trajectories without access to any judge outputs, and report the agreement rate and κ between SAF and the human labels, as well as inter-annotator agreement?
-
Could you construct explicit and implicit versions of the same harmful objectives and recompare SRR, SVR, and ASR on these paired tasks, to rule out confounding from differences in task content?
-
Among the breaches enabled by witness-guided navigation, how many were originally capability failures rather than refusals? Could you separately report the breach rate on tasks that were originally refused, as a measure of the pure jailbreak effect?
-
Could you reproduce a subset of tasks on real Android devices or emulators and report the outcome agreement rate? How might differences in interface and interaction logic between the browser simulator and real apps affect agents' refusal behavior?
-
Each task in the main table is run only once. Could you repeat runs for some models and report the variance? At what unit are the bootstrap confidence intervals mentioned in the paper computed?
-
Which development model(s) drove the failures used to generate the expansion tasks? On models that played no role in development, how does ASR on the expansion tasks compare with that on the core tasks? Could you report expansion-task results with the development model(s) excluded?