acceptodds
Under review as a conference paper at ICLR 2027

MobileJailBench: Red Queen Jailbreak Benchmarking for Mobile Agents

Abstract

MLLM-based mobile agents are rapidly gaining autonomy in perception, planning, and closed-loop execution in dynamic mobile environments. However, this autonomy also broadens the attack surface and amplifies the risks of malicious misuse, including privacy leakage, device compromise, and large-scale phishing. Existing jailbreak evaluations leave three critical gaps for mobile agents: they focus on conversational attacks instead of mobile workflows, use static task sets with fixed difficulty, and rely narrowly on attack success rate rather than comprehensive security assessment. To close these gaps, we present MobileJailBench, the first evolving jailbreak benchmark for mobile agents, comprising three key components: (1) a benchmark comprising 520 tasks spanning three jailbreak paradigms and 14 risk categories, built on a high-fidelity mobile sandbox that we construct, enabling realistic workflows across over 20 widely used apps; (2) Dual-Dynamic Evolution, which combines Failure-Distilled Jailbreak Evolution with Weakness-Driven Benchmark Expansion; and (3) Stratified Adjudication Framework for assessing security awareness, execution behavior, and risk consequences. Extensive experiments show that current mobile agents are particularly vulnerable to implicit jailbreaks, as their safety alignment remains largely superficial and fails to generalize from conversational refusal to dynamic mobile interactions. Moreover, our dynamic evolution mechanism sharpens practical threat boundaries by continually uncovering new vulnerabilities, reaching a 100% ASR on Qwen3.5-122B. MobileJailBench provides a reproducible, continuously evolving framework for tracking the true security boundaries of mobile agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Dynamic Jailbreak Benchmark, Mobile Agents, Mobile Security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mFfROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents MobileJailBench, an evolving jailbreak benchmark for MLLM-based mobile agents built on a browser-based, Android-like simulator covering 28 apps, with 520 tasks across 14 risk categories organized into six sub-benchmarks (Benign, Explicit, Implicit, Benign-First and Harm-First Composite, and Expansion), each equipped with a programmatic oracle that compares initial and final device states. It introduces Dual-Dynamic Evolution, in which FDJE routes failed attacks by failure type (task-flow camouflage for refusals, witness-guided navigation for execution failures, and implicit-track evolution that moves harmful content into environmental carriers and iteratively refines prefixes), while WDBE converts observed weaknesses into new tasks that pass four reviews, automated code and judge-test generation, and GUI validation before entering the next frozen release. It also proposes SAF, a stratified adjudication framework combining rule-based anchor matching, two isolated judgments from the same model with verbatim evidence citations, and evidence auditing with conservative arbitration, yielding metrics such as TSR, ASR, SVR, RHR, and SRR. Across 20 agent–model pairings (GenericV2 and M3A × 10 MLLMs), implicit tasks reduce mean SRR from 26.8% to 6.6% and raise SVR from 68.9% to 86.6% while ASR slightly decreases (59.8% to 56.4%); FDJE raises cross-model coverage from 69.6% to 85.9%, reaching 100% on Qwen3.5-122B; the 50 WDBE tasks yield 36% to 82% first-run ASR on nine other models; and SAF reaches 97.26% agreement with the reference, versus 91.14% for a single LLM judge.

Critical strengths
  1. The engineering design of an "evolving benchmark" is novel. The frozen-release boundary explicitly ensures that new tasks enter only the next release and never change scores in the current one. WDBE distinguishes generated from certified task counts and requires version-matched GUI execution evidence. This offers a practical scheme for updating a benchmark as models improve while preserving comparability.

  2. Multi-dimensional metrics reveal phenomena that ASR misses. On implicit tasks, ASR drops slightly, yet the refusal rate falls sharply and the violation rate rises, showing that ASR alone underestimates risk. SAF separately records expressed adoption, timely refusal, risky actions, and final consequences, providing substantially more information than binary judgments.

  3. SAF's evidence audit is a valuable design. Requiring the judge to quote the trajectory verbatim and verifying each quote by exact substring matching directly rules out judgments based on fabricated evidence. The appendix further identifies a "refusal anchoring" problem in single judges, where later refusal-like wording overshadows earlier objective adoption, and shows that voting and averaging cannot correct this shared bias.

Critical weaknesses
  1. Headline numbers come from non-comparable protocols, and the main text overstates them. The "100% ASR on Qwen3.5-122B" is task-level coverage under multiple adaptive search attempts with first-success stopping (pass@budget), not single-episode ASR, and the additional cost ranges from 116 to 2,278 episodes per model. For the comparison between FDJE and baselines such as Tree, BoN, and PAIR, the appendix acknowledges differing permissions and budgets that preclude an equal-budget superiority claim, yet the abstract and main text still present 100% as a headline result. The strategy ablation in Figure 3(b) is further "projected to a 70-task scale with 60 assumed successes" from 10 residual tasks, rather than measured.

  2. Judge validation relies on a self-constructed reference, raising circularity concerns. SAF's 97.26% agreement is computed against a "constructed reference" that combines cross-arm agreement, endpoint checks, only five manually inspected boundary tasks, and re-adjudication, rather than independent human annotation. The appendix itself states that it reports "reference agreement rather than independent annotation accuracy." The two "independent" L1 judgments come from the same model, which the authors acknowledge are not independent annotators. Without an independent human gold standard, the claim that SAF outperforms a single judge is not very convincing.

  3. The explicit-versus-implicit comparison is confounded by task content. Explicit (T0001–T0070) and Implicit (T0071–T0140) are two different task sets, not paired rewrites of the same harmful objectives. The finding that implicit tasks yield lower refusal and higher violation rates may therefore partly reflect differences in task content, difficulty, or risk distribution rather than presentation alone. This is one of the paper's central findings, yet it lacks support from a paired design.

  4. The 520-task scale is partly inflated. The two Composite sub-benchmarks contribute 280 tasks, derived from only 28 attack sources × 5 benign workflows × 2 orders, and share objectives with the core tasks; 266 of the 322 cross-app tasks come from these concatenated tasks. The number of independent harmful objectives is roughly 140 core tasks plus 50 expansion tasks. In addition, each task in the main experiments is run only once (pass@2 is used only in some experiments), and the paper does not report whether the number of repetitions behind its confidence intervals is sufficient.

  5. Simulator fidelity is not validated. The environment is a browser-simulated "Android-like" interface, and the authors acknowledge that it abstracts away real Android process isolation and live service behavior. Although the paper repeatedly describes it as "high-fidelity," it provides no comparison against real apps or physical devices. Whether violations observed in the simulator transfer to real apps therefore remains unknown.

  6. Attack components conflate jailbreaking with capability assistance. Witness-guided navigation supplies same-app operation examples and hints to help agents that failed during execution complete the task. Part of the resulting ASR gain thus compensates for limited GUI capability rather than bypassing safety alignment. The main text also describes this component as improving "less capable agents' ability to recognize, resist, and refuse malicious instructions," which contradicts its actual role (described in the appendix as providing execution support).

  7. WDBE's independence and certification are incomplete. Expansion tasks are driven by weaknesses exposed during development, with a deliberately uneven distribution (Mail appears in 23 of 50 tasks, and 39 are L1). The appendix records pass@2 of 48/50 and notes that the earlier 50/50 result cannot serve as certification after specification or oracle revisions. The main text does not clarify which model's failures drive WDBE or whether selection bias arises when transferring to other models.

  8. Poor readability, with much key information deferred to the appendix. The main text introduces many acronyms (FDJE, WDBE, SAF, CPSD, SVR, RHR, SRR, SU), and Figure 2 is so dense that it requires zooming in to read. Key details such as budgets, reference construction, and task pairing can only be understood by consulting the appendix. Nine pages of main text can hardly support this many components on their own, making it difficult for readers to tell which component drives which conclusion.

Questions for authors
  1. Could you compare FDJE with baselines such as Tree, BoN, and PAIR under the same episode budget and the same permissions? What is FDJE's ASR when it is also limited to a single episode?

  2. Could you have at least two independent annotators label a randomly sampled (non-enriched) set of trajectories without access to any judge outputs, and report the agreement rate and κ between SAF and the human labels, as well as inter-annotator agreement?

  3. Could you construct explicit and implicit versions of the same harmful objectives and recompare SRR, SVR, and ASR on these paired tasks, to rule out confounding from differences in task content?

  4. Among the breaches enabled by witness-guided navigation, how many were originally capability failures rather than refusals? Could you separately report the breach rate on tasks that were originally refused, as a measure of the pure jailbreak effect?

  5. Could you reproduce a subset of tasks on real Android devices or emulators and report the outcome agreement rate? How might differences in interface and interaction logic between the browser simulator and real apps affect agents' refusal behavior?

  6. Each task in the main table is run only once. Could you repeat runs for some models and report the variance? At what unit are the bootstrap confidence intervals mentioned in the paper computed?

  7. Which development model(s) drove the failures used to generate the expansion tasks? On models that played no role in development, how does ASR on the expansion tasks compare with that on the core tasks? Could you report expansion-task results with the development model(s) excluded?

Reviewer_1xw3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on systematically evaluating jailbreak vulnerabilities in mobile GUI agents under realistic and dynamic interaction environments. Existing jailbreak benchmarks primarily target conversational attacks, rely on static task sets, and lack fine-grained analysis of agent safety failures. To address these limitations, the authors introduce MobileJailBench, an evolving jailbreak benchmark comprising 520 tasks across six sub-benchmarks and 14 risk categories, supported by a high-fidelity mobile simulator covering 28 applications. The paper further proposes Dual-Dynamic Evolution, which combines Failure-Distilled Jailbreak Evolution (FDJE) to adapt attacks based on execution feedback with Weakness-Driven Benchmark Expansion (WDBE) to generate new tasks targeting identified vulnerabilities. Additionally, a Stratified Adjudication Framework (SAF) evaluates security awareness, execution behavior, and risk consequences beyond conventional attack success rates. Experiments across ten multimodal models and two agent frameworks reveal substantial vulnerabilities to implicit jailbreaks, while adaptive attack evolution exposes additional weaknesses and the proposed adjudication framework enables more fine-grained safety assessment.

Critical strengths
  1. The paper introduces a comprehensive mobile agent jailbreak benchmark and present well. The paper presents MobileJailBench, covering 520 tasks across six sub-benchmarks, 14 risk categories, and 28 applications, including 322 cross-application tasks. The benchmark considers explicit, implicit, and composite jailbreak scenarios, enabling evaluations of different forms of harmful intent and task execution. It also uses application-state verification rather than relying solely on model-generated judgments to determine attack success. This design provides a systematic evaluation of mobile agent safety across different interaction settings.
  2. The paper provides transparent and compressive experimental reporting and a clear discussion of limitations. The paper carefully documents the actual search budgets, task versions, and failure cases in the appendix. It explicitly distinguishes experimental results obtained under different evaluation budgets, reports unresolved tasks and incomplete validation cases, and provides bounds for metrics with missing labels. These details make the experimental results easier to verify and help clarify the scope and limitations of the reported findings.
Critical weaknesses
  1. The paper does not sufficiently isolate the effects of implicit jailbreaks from differences in task complexity. The paper concludes that mobile agents are more vulnerable to implicit jailbreaks, reporting a decrease in average refusal rate from 26.8% to 6.6% and an increase in safety violation rate from 68.9% to 86.6% (§6.2). However, the explicit and implicit benchmarks are not matched task pairs and differ substantially in complexity. For example, the implicit benchmark contains 22 cross-application tasks and 22 L4 tasks, compared with only 7 of each in the explicit benchmark (Appendix C.3). Consequently, the observed differences may partly reflect workflow complexity or task composition rather than implicit intent presentation alone. More controlled comparisons would help establish this conclusion.
  2. The paper does not sufficiently establish the superiority of FDJE under comparable attack budgets.The paper compares FDJE with several existing jailbreak methods, including Tree, BoN, TRACE, TAP, and PAIR (Figure 3(c)). Although FDJE achieves 100% ASR compared with 94.3% for Tree, the methods are evaluated under different search budgets and attack permissions (Appendix A.5). FDJE requires approximately 209 GUI episodes for the 70-task reference set, making it difficult to determine whether its improvement comes from the proposed adaptive strategy or additional attack attempts. Furthermore, the strategy ablation in Figure 3(b) extrapolates results from only ten residual tasks rather than evaluating the complete benchmark. These limitations leave the independent contribution and comparative effectiveness of FDJE insufficiently established.
Questions for authors

Q1. The explicit and implicit benchmarks differ in task complexity and cross-application interactions (Appendix C.3). Could the authors provide a controlled comparison using matched explicit and implicit versions of the same harmful objectives, or report results stratified by task difficulty and workflow complexity? This would help clarify whether the observed differences in refusal and safety violation rates are primarily caused by implicit intent presentation rather than task composition.

Q2. FDJE achieves 100% ASR compared with 94.3% for Tree, but the methods use different search budgets and attack permissions (Appendix A.5). Could the authors provide a budget-matched comparison, such as ASR versus the number of GUI episodes, under consistent attack permissions? Additionally, could the authors clarify whether the strategy ablation results in Figure 3(b), which are extrapolated from ten residual tasks, remain consistent when evaluated on the full benchmark? These results would help establish the effectiveness of FDJE independently of additional search attempts.

Reviewer_VtyqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MobileJailBench, an evolving jailbreak benchmark designed to evaluate the security of multimodal mobile agents in GUI-based workflows. The authors argue that conventional jailbreak benchmarks primarily focus on textual responses and static attack sets, which inadequately capture safety risks arising from autonomous interactions with mobile applications and device states.

The benchmark contains 520 tasks across six complementary subsets, covering explicit, implicit, and composite jailbreak scenarios and 14 risk categories. It is implemented on a browser-based Android-like simulator supporting 28 applications, including cross-application workflows.

The paper introduces three main components:

  1. MobileJailBench: An environment-grounded benchmark that evaluates both benign task completion and safety violations through executable mobile workflows.
  2. Dual-Dynamic Evolution: A framework combining Failure-Distilled Jailbreak Evolution (FDJE), which adaptively modifies attack strategies based on execution feedback, and Weakness-Driven Benchmark Expansion (WDBE), which generates and validates new benchmark tasks targeting observed vulnerabilities.
  3. Stratified Adjudication Framework (SAF): A hierarchical evaluation mechanism that separately assesses expressed adoption of harmful objectives, risk-bearing actions, safety-motivated refusal, and verified harmful outcomes using deterministic rules, evidence-grounded LLM judgments, and arbitration.

Experiments across ten multimodal models and two mobile-agent frameworks show that implicit jailbreaks substantially reduce safety refusal rates while increasing risk-bearing behavior. The authors also report improvements in attack coverage through FDJE and higher adjudication consistency through SAF.

Overall, the paper provides an extensive evaluation framework and highlights the gap between conversational safety alignment and safety during autonomous mobile-agent execution.

Critical strengths

S1. Important and timely problem formulation.

The paper addresses an increasingly important security challenge: mobile agents can directly manipulate applications, access sensitive information, and modify persistent device states. Unlike conventional text-based jailbreak evaluation, this work evaluates whether unsafe instructions result in harmful actions and state transitions. This distinction is meaningful for understanding the security implications of deploying autonomous agents in practical environments.

S2. Comprehensive and environment-grounded benchmark design.

MobileJailBench covers 520 tasks across six subsets and 14 risk categories, with 322 tasks involving cross-application interactions (Section 3 and Appendix C). The inclusion of explicit, implicit, and composite scenarios supports the investigation of different failure modes. The controlled environment, state-reset mechanism, and task-specific endpoint oracles improve reproducibility and enable evaluation beyond textual judgments.

S3. A useful empirical finding regarding implicit jailbreaks.

One of the most informative results is the discrepancy between refusal behavior and unsafe execution. Across 20 model-agent configurations, implicit attacks decrease the average safety refusal rate from 26.8% to 6.6%, while increasing the safety violation rate from 68.9% to 86.6% (Section 6.2). This finding provides evidence that recognizing explicitly harmful instructions does not necessarily translate into safe behavior during contextual mobile interactions.

S4. Fine-grained and evidence-grounded safety evaluation.

The Stratified Adjudication Framework is a valuable component. By separating expressed objective adoption, risk-bearing behavior, timely refusal, and verified harmful outcomes, it provides a more informative assessment than binary attack success. The deterministic evidence audit and arbitration mechanism also improve interpretability and evaluation consistency. Table 3(c) reports higher reference agreement and stability than the compared generic judging approaches.

The combination of these contributions makes the benchmark a potentially useful resource for evaluating and diagnosing security failures in autonomous mobile agents.

Critical weaknesses

W1. Limited methodological novelty beyond the integration of existing ideas.

Although the overall framework is comprehensive, the main technical components appear to rely largely on established ideas, including feedback-driven prompt optimization, attack strategy selection, LLM-assisted task generation, and evidence-grounded evaluation. In particular, FDJE employs a relatively straightforward candidate-selection rule based on observed diagnostic scores (Equation 1). The paper would benefit from a clearer explanation of which algorithmic mechanisms are fundamentally new compared with prior adaptive jailbreak methods and agent red-teaming frameworks.

The ablation studies are useful, but they do not fully isolate the contribution of failure-based strategy routing from the effects of additional attack attempts, benign workflow prefixes, and environmental payload placement.

W2. Incomplete evidence for sustained benchmark evolution and generalization.

A central claim is that MobileJailBench provides a continuously evolving benchmark through the combination of FDJE and WDBE. However, the reported evaluation primarily demonstrates adaptive attack optimization and the generation of one additional batch of 50 tasks. This is not equivalent to demonstrating sustained benchmark evolution across multiple independent releases.

Furthermore, WDBE constructs tasks based on vulnerabilities observed during development, which raises questions about dependence on the development agents and potential overfitting to the discovered failure patterns. Although the authors provide cross-model transfer results, stronger evidence would require repeated expansion cycles, explicit novelty measurements, and evaluation against previously unseen agents or environments.

W3. Evaluation comparability and capability confounding.

The experimental results demonstrate high attack success rates, but their interpretation requires care. Some comparisons involve different attack budgets or adaptive search procedures. For example, Table 3(a) compares a pass@2 baseline with FDJE augmented by additional adaptive search. Consequently, improvements cannot be attributed solely to the proposed search strategy without considering the additional evaluation budget.

Moreover, model capability and safety robustness remain partially confounded. A mobile agent may fail to complete a harmful task because it recognizes the security risk or because it cannot successfully navigate the GUI. The benign task success rates vary substantially across models, indicating considerable differences in operational competence.

Although the paper distinguishes refusal from execution failure and introduces behavior-level metrics, a more systematic capability-controlled evaluation would strengthen the conclusions. For example, the authors could compare attacks under matched execution budgets and report safety outcomes conditional on demonstrated task competence.

W4. Limited external validity of the simulated environment.

The browser-based Android-like simulator provides a scalable and reproducible evaluation environment, but it abstracts away important characteristics of real mobile operating systems, including process isolation, application-specific permission enforcement, and live service behavior (Section 3.1 and Appendix C.2).

These differences may affect both attack feasibility and the observed safety behavior of mobile agents. The reported results therefore establish vulnerabilities within the simulated execution setting, but do not directly demonstrate that the same attack success rates transfer to real Android applications or deployed mobile-agent systems.

Even a small-scale validation on a real Android emulator, or a more detailed analysis of the simulator-to-real-world gap, would substantially strengthen the paper's practical claims.

Questions for authors

Q1. Novelty of FDJE: What is the key methodological distinction between FDJE and existing feedback-based jailbreak optimization methods? Could the authors provide a controlled comparison that isolates failure-based strategy routing from additional search budget and prompt-prefix optimization?

Q2. Fairness of attack comparisons: Table 3(a) compares baseline pass@2 results with FDJE using additional adaptive search. Can the authors report attack success rates under strictly matched numbers of target-agent executions or provide success-versus-budget curves for the major baselines?

Q3. Multi-round benchmark evolution: Has WDBE been evaluated across multiple consecutive expansion rounds? If so, how does task novelty, vulnerability coverage, and attack effectiveness evolve? If not, what evidence supports the claim of continuous benchmark evolution beyond the demonstrated expansion batch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.