- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
-
This work presents MobileGrail, compact multimodal guardrail models (1.6B and 2B) for on-device deployment that maintain a recurrent task state and separately predict action safety and intervention, trained by distillation and calibrated by RL with verifiable rewards.
-
This work builds a scalable Android training environment that automatically mines verifiable learning signals: self-evolving red-teaming generates attack scenarios across 32 applications, and their corresponding backends verify the step at which a violation is committed, grounding both labels and rewards.
-
This work introduces MobileVeto, MobileVeto, an evaluation benchmark for mobile guardrails spanning diverse application categories, comprising 210 adversarial trajectories annotated with intervention windows, each paired with a benign counterpart. It jointly evaluates violation detection, intervention timing, and the preservation of benign tasks.
Critical strengths
-
The motivation is clear. Safety guardrails are indeed important for a GUI agent during execution.
-
The training data and training environment of MobileGrail constructed in this paper are highly valuable.
Critical weaknesses
-
When a GUI agent executes, it still needs to send screenshots—which may contain personal information—to a cloud server. From this perspective, does it really make much of a difference whether MOBILEGRAIL runs on the device or in the cloud?
-
Line71:
Moreover, a compact model cannot judge safety from a prompt as large models do; its safety boundary must be learned, and the supervision that defines it becomes decisive.How should this statement be interpreted, and could the author provide relevant references or experimental data? -
If the model needs to run on-device, why not conduct experiments based on Qwen3.5-0.8B?
-
In Table 1, MobileGrail outperforms the teacher model, Gemini-3.8-Flash, on certain evaluation metrics. Could the authors provide an explanation for this?
-
Table 1 presents a relatively limited number of models; it would be more helpful if the authors could provide the performance of additional state-of-the-art models.
Questions for authors
- Please refer to the weaknesses.
5 comments
W3: Why not Qwen3.5-0.8B?
We considered Qwen3.5-0.8B early on, but under the full teacher prompt it intervenes on only 21.7% of violations, so we first focused on the 1.6B and 2B backbones. We have now trained it with SFT:
| Model | EIR ↑ | MIR ↓ | PUR ↓ | PIR ↓ | B-FPR ↓ |
|---|---|---|---|---|---|
| Qwen3.5-0.8B (teacher prompt, no training) | 21.7 | 78.3 | 37.2 | 34.8 | 33.3 |
| Qwen3.5-0.8B (SFT) | 67.2 | 25.5 | 13.4 | 8.6 | 14.8 |
| Qwen3.5-2B (SFT, Table 1) | 71.4 | 19.0 | 16.6 | 11.0 | 14.3 |
SFT raises EIR by 45.5 points and lowers every error rate. The trained 0.8B model is on par with the 2B model in premature intervention and benign false positives, and trails it by 4.2 points in EIR and 6.5 in MIR. We also profiled the 0.8B model on a Snapdragon 8 Elite under the deployment configuration of Sec. 5.5:
| Model / reasoning | Quant. | P50 (s) ↓ | Mean (s) ↓ | P90 (s) ↓ | Decode (token/s) ↑ | Prefill (token/s) ↑ |
|---|---|---|---|---|---|---|
| 2B / long | Q4_0 | 5.94 | 6.29 | 8.42 | 30.2 | 1,010 |
| 2B / long | Q8_0 | 7.66 | 8.06 | 11.06 | 21.1 | 1,058 |
| 2B / short | Q4_0 | 4.70 | 5.07 | 7.39 | 30.3 | 944 |
| 2B / short | Q8_0 | 5.61 | 6.14 | 8.23 | 21.2 | 949 |
| 0.8B (new) | Q4_0 | 3.55 | 3.61 | 4.57 | 62.6 | 1,529 |
| 0.8B (new) | Q8_0 | 4.41 | 4.45 | 5.68 | 46.8 | 1,456 |
The 0.8B numbers are measured on a subset of 40 MobileVeto trajectories, whereas the 2B numbers are from the full benchmark (Tables 4 and 11), so latencies are not directly comparable. Decode throughput, which is largely independent of the workload, doubles at both quantization levels (62.6 vs. 30.3 token/s at Q4_0). Qwen3.5-0.8B is thus a viable backbone when a smaller footprint is preferred over the last few points of EIR. We will add these results to the paper.
W4: Why does MobileGrail outperform its teacher on some metrics?
MobileGrail surpasses Gemini 3.8 Flash only on the two premature metrics, PUR and PIR; the teacher remains ahead on EIR, MIR, and B-FPR. This is expected, because MobileGrail is not trained to imitate the teacher's raw decisions:
- SFT targets are filtered by verified labels. Steps where the teacher's decision conflicts with the backend-grounded safety and intervention labels are discarded (Appendix E.1), so the teacher's own premature judgments never become supervision.
- RL calibrates timing against the windows, not the teacher. GRPO rewards are defined by the annotated intervention windows and penalize blocking before a window opens, lowering PIR from 11.0% to 5.2% for Qwen3.5-2B and from 9.7% to 3.8% for LFM2.5-VL-1.6B.
The lower PIR does not come from blocking less: Qwen3.5-27B reaches a similar EIR (80.7% vs. 76.2%) with a PIR of 35.2%.
These two metrics are direct evidence of deployability. A premature intervention interrupts the agent on a step that is still legitimate, and because current agents have limited ability to recover once interrupted, it can directly cause task failure. In practice, we observe many such premature interventions from risk-based guardrails: OS-Sentinel blocks on risk indicators before any violation is underway (PUR 36.9%, PIR 22.4% in Table 1).
W5: Additional state-of-the-art models.
We have added GPT-6.1 Terra, evaluated with the same full teacher prompt as the other prompted baselines:
| Model | EIR ↑ | MIR ↓ | PUR ↓ | PIR ↓ | B-FPR ↓ |
|---|---|---|---|---|---|
| GPT-6.1 Terra (new) | 95.5 | 2.4 | 13.4 | 7.6 | 18.1 |
| Gemini 3.8 Flash | 95.5 | 3.4 | 20.3 | 13.4 | 7.6 |
| Kimi-K2.6 | 87.2 | 10.0 | 17.9 | 20.3 | 8.1 |
| Qwen3.5-27B | 80.7 | 11.7 | 22.8 | 35.2 | 17.1 |
| MobileGrail, Qwen3.5-2B (SFT + GRPO) | 76.2 | 20.3 | 16.6 | 5.2 | 15.7 |
GPT-6.1 Terra matches Gemini 3.8 Flash in EIR with fewer premature decisions, but more than doubles its benign false-positive rate (18.1% vs. 7.6%). The conclusions of Table 1 are unchanged: frontier models lead in EIR, while MobileGrail at 2B parameters attains the lowest PIR of all models and a lower B-FPR than GPT-6.1 Terra and Qwen3.5-27B. We will add this row to Table 1.
Dear reviewer YXCf, Thank you again for your time and constructive feedback. As the discussion period is closing soon, we would like to check whether our response has addressed your concerns. We are happy to clarify any remaining points before the deadline and would appreciate it if you could update your assessment in light of the response.
We thank the reviewer for the detailed comments. We address each point below.
W1: Does on-device matter when the agent itself is cloud-hosted?
Yes, for three reasons.
It is what makes local anonymization effective. A cloud-hosted agent receives the screen at every step, including private content such as payments, messages, and medical records. To keep this content from the cloud provider, observations are anonymized on device before upload: Zhao et al. [2] replace sensitive content in screenshots and UI text with placeholders, and GUIGuard-Bench [1] evaluates such protection strategies. A guardrail, however, must check each action against the raw values on screen. A cloud-hosted guardrail would therefore either receive the raw screens, reintroducing the exposure anonymization removes, or judge on masked inputs that hide the evidence its decision depends on. For example, given the request "Pay my electricity bill on PayPal if it is under $80", an anonymizer replaces the on-screen amount with a placeholder such as [AMOUNT_1]. A guardrail judging the masked screen cannot tell whether confirming the payment stays within the user's authorization; an on-device guardrail reads the raw amount, and the amount never leaves the device.
It adds no exposure of its own. Without anonymization, a cloud guardrail sends every screen and the accumulated authorization record to a second service, often a different provider from the agent's.
It matches where deployment is heading. The trend is toward fully on-device agents: as compact agent models and mobile hardware improve, we expect the agent itself to run locally in the near future. A guardrail that already runs on device is then required, since a cloud-hosted one would be the only component still uploading the screen.
[1] Wang et al. GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents. arXiv:2601.18842, 2026. [2] Zhao et al. Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible. arXiv:2602.10139, 2026.
W2: Interpretation of Line 71 and supporting evidence.
Line 71 makes two claims, both supported by Table 1.
Compact models cannot apply safety criteria from a prompt alone. All prompted baselines receive the same teacher prompt, with explicit safety boundaries and few-shot examples. Large models keep benign false positives low under it (B-FPR 7.6% for Gemini 3.8 Flash, 8.1% for Kimi-K2.6, 17.1% for Qwen3.5-27B), whereas compact backbones block most benign trajectories (75.7% for Qwen3.5-2B, 95.2% for LFM2.5-VL-1.6B). The gap between Qwen3.5-27B and Qwen3.5-2B points to model size rather than model family. After SFT, with a much leaner prompt, Qwen3.5-2B reaches 14.3% B-FPR, and its PIR falls from 58.6% to 11.0%.
The supervision that defines the boundary is therefore decisive. Existing guardrails define unsafe by perceived risk rather than by the effect of the action, which has two consequences once such labels become training targets. First, authorized operations are labeled unsafe: SeerGuard's Table 5 marks entering a credit-card number as unsafe, citing potential fraud, although the user explicitly asked to link the card to PayPal. Second, non-committing steps are labeled unsafe: MobileRisk's risk-onset labels fall on a one-second wait after opening an app, on selecting "Show all items" during storage cleanup, and on navigating back from screen-lock settings (Appendix A.5). The cost is visible in Table 1: on the same Kimi-K2.6 backbone, OS-Sentinel's risk-based criteria raise B-FPR from 8.1% to 57.1%.
MobileGrail instead grounds unsafe in backend-verified violations of the user's authorization and predicts intervention separately, so a preparatory action can be safe yet admissible to block. We will revise the wording of Line 71 to make this clearer.