- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HalluClear tries to separate hallucinations in GUI agents from mismatches between an action and a reference answer. Under a POMDP, the authors define a hallucination as a case in which the policy prefers an action whose value gap from the best attainable action, given the current observable information state, exceeds δ. The actual evaluation does not compute this value. It instead groups failure cases into perception and reasoning, with eight subtypes plus NonH. Evaluation has three stages: experts annotate JQ-Bench, that set is used to qualify VLM judges and estimate their credibility, and the qualified judges then estimate credibility-weighted hallucination rates on HR-Pool. JQ-Bench and HR-Pool share 3,410 difficult queries. A mitigation probe then applies OODA-structured SFT and GRPO on 9K samples. Across three open-source agents, hallucination rates decrease, while ScreenSpot grounding and action success on AndroidControl and GUI-Odyssey increase.
Critical strengths
- The evaluation target is well chosen. Action matching for GUI agents does not separate an evidence-supported alternative action from an unsupported commitment. On 200 outputs from each of three agents, labeled independently by three experts, the eight-subtype taxonomy has an average Fleiss’ κ of 0.745. Repeated labeling by the judges themselves has an average κ of 0.855. Table 5 and Figure 4 give an informative breakdown: closed-source models are relatively balanced between perception and reasoning, GUI-specialized models are dominated by instruction and context reasoning hallucinations, and Qwen2.5-VL-7B has stronger perception hallucinations. Grounding and action-fidelity metrics do not provide this diagnosis.
- The mitigation study at least includes a generic post-training control, rather than comparing only with the original model. Appendix Table 8 separates SFT, RL, and OODA, and action metrics also improve on out-of-distribution web and desktop settings. The results therefore cannot all be explained as format overfitting on AndroidControl.
Critical weaknesses
- The measurement pipeline can inflate agreement between judges and humans, while the main text still treats that agreement as evidence of reliability. Human–Judge Alignment in Appendix D allows annotators to revise a JQ-Bench gold label when judges consistently predict a subtype and the annotators unanimously consider it reasonable. Those same judges then receive credibility scores on this JQ-Bench and are used on HR-Pool. Judge disagreement in Table 5 is already large: GUI-Owl’s overall hallucination rate is 54.46, 39.02, and 41.77 under Gemini-3, Seed-1.8, and Qwen3-VL, respectively. The hallucination rates in Table 6 match the Gemini column of Table 5 exactly, rather than the credibility-weighted ensemble promised in the text. The formal δ-hallucination definition is also never computed.
- The expert labels are a classification protocol, not a validated proxy for Q values. The causal claim that the diagnosis guided the intervention is too strong. For Qwen2.5-VL-7B, the hallucination rate falls from 57.54 to 39.17 with SFT alone, to 35.81 with generic SFT+RL, and to 24.98 with the full OODA probe. Most of the reduction comes from ordinary post-training on additional GUI trajectories. OODA is a generic observe–orient–decide–act template; it does not use subtype-specific rewards or data for the eight categories. The paper also does not establish that the 9K samples, HR-Pool, and the AndroidControl and GUI-Odyssey test splits are disjoint at the task-instance level.
I would like to raise my score if the authors can solve my concerns.
Questions for authors
- Please recompute Cohen’s κ and credibility on a frozen annotation subset that the judges never influenced, and report the weighted hallucination rate of the three qualified judges rather than the Gemini column alone. If agreement drops substantially on the frozen subset, the current reliability claim does not hold.
- Please confirm that the 9K mitigation samples are disjoint from HR-Pool and from the official AndroidControl and GUI-Odyssey test splits. If they overlap, the hallucination-rate reduction and the action-success gains in Table 6 cannot be interpreted as diagnosis-driven generalization.
Thank you for the detailed rebuttal. My concerns have been addressed, and I confirm my original score.