acceptodds
Under review as a conference paper at ICLR 2027

HalluClear: A Diagnostic and Evaluation Suite with Mitigation Probe for Hallucinations in GUI Agents

Abstract

As GUI agents increasingly benefit from industrial-scale training, ungrounded hallucinations can still lead to cascading failures during sequential GUI interaction. Unlike hallucinations in general vision-language tasks, GUI-agent hallucinations span both visual perception and task reasoning and are further complicated by partial observability. Existing benchmarks primarily assess task success, visual grounding, or action fidelity, but lack fine-grained hallucination diagnosis and reliable evaluation. To bridge this gap, we introduce **HalluClear**, a suite for diagnosing and evaluating hallucinations in GUI agents, with a mitigation probe. HalluClear consists of three components: (1) a diagnostic framework that formalizes GUI-agent hallucination as unsupported belief-action commitment under partial observability and derives a fine-grained taxonomy from empirical failure analysis; (2) a three-stage evaluation workflow that combines an expert-annotated benchmark with qualified VLM judges for credibility-calibrated hallucination measurement; and (3) a lightweight mitigation probe that instantiates a structured post-training intervention motivated by patterns identified through our diagnostic analysis. Experiments with representative GUI agents on public benchmarks reveal systematic hallucination patterns across agent families and failure subtypes, providing diagnostic information complementary to conventional visual grounding and action-fidelity metrics. Across three open-source GUI agents, a post-training intervention using 9K samples consistently reduces hallucination rates while improving visual grounding and action fidelity.

Primary area:
datasets and benchmarks
Keywords:
GUI Agents, Hallucination, Diagnostic and Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_rjaEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HalluClear tries to separate hallucinations in GUI agents from mismatches between an action and a reference answer. Under a POMDP, the authors define a hallucination as a case in which the policy prefers an action whose value gap from the best attainable action, given the current observable information state, exceeds δ. The actual evaluation does not compute this value. It instead groups failure cases into perception and reasoning, with eight subtypes plus NonH. Evaluation has three stages: experts annotate JQ-Bench, that set is used to qualify VLM judges and estimate their credibility, and the qualified judges then estimate credibility-weighted hallucination rates on HR-Pool. JQ-Bench and HR-Pool share 3,410 difficult queries. A mitigation probe then applies OODA-structured SFT and GRPO on 9K samples. Across three open-source agents, hallucination rates decrease, while ScreenSpot grounding and action success on AndroidControl and GUI-Odyssey increase.

Critical strengths
  1. The evaluation target is well chosen. Action matching for GUI agents does not separate an evidence-supported alternative action from an unsupported commitment. On 200 outputs from each of three agents, labeled independently by three experts, the eight-subtype taxonomy has an average Fleiss’ κ of 0.745. Repeated labeling by the judges themselves has an average κ of 0.855. Table 5 and Figure 4 give an informative breakdown: closed-source models are relatively balanced between perception and reasoning, GUI-specialized models are dominated by instruction and context reasoning hallucinations, and Qwen2.5-VL-7B has stronger perception hallucinations. Grounding and action-fidelity metrics do not provide this diagnosis.
  2. The mitigation study at least includes a generic post-training control, rather than comparing only with the original model. Appendix Table 8 separates SFT, RL, and OODA, and action metrics also improve on out-of-distribution web and desktop settings. The results therefore cannot all be explained as format overfitting on AndroidControl.
Critical weaknesses
  1. The measurement pipeline can inflate agreement between judges and humans, while the main text still treats that agreement as evidence of reliability. Human–Judge Alignment in Appendix D allows annotators to revise a JQ-Bench gold label when judges consistently predict a subtype and the annotators unanimously consider it reasonable. Those same judges then receive credibility scores on this JQ-Bench and are used on HR-Pool. Judge disagreement in Table 5 is already large: GUI-Owl’s overall hallucination rate is 54.46, 39.02, and 41.77 under Gemini-3, Seed-1.8, and Qwen3-VL, respectively. The hallucination rates in Table 6 match the Gemini column of Table 5 exactly, rather than the credibility-weighted ensemble promised in the text. The formal δ-hallucination definition is also never computed.
  2. The expert labels are a classification protocol, not a validated proxy for Q values. The causal claim that the diagnosis guided the intervention is too strong. For Qwen2.5-VL-7B, the hallucination rate falls from 57.54 to 39.17 with SFT alone, to 35.81 with generic SFT+RL, and to 24.98 with the full OODA probe. Most of the reduction comes from ordinary post-training on additional GUI trajectories. OODA is a generic observe–orient–decide–act template; it does not use subtype-specific rewards or data for the eight categories. The paper also does not establish that the 9K samples, HR-Pool, and the AndroidControl and GUI-Odyssey test splits are disjoint at the task-instance level.

I would like to raise my score if the authors can solve my concerns.

Questions for authors
  1. Please recompute Cohen’s κ and credibility on a frozen annotation subset that the judges never influenced, and report the weighted hallucination rate of the three qualified judges rather than the Gemini column alone. If agreement drops substantially on the frozen subset, the current reliability claim does not hold.
  2. Please confirm that the 9K mitigation samples are disjoint from HR-Pool and from the official AndroidControl and GUI-Odyssey test splits. If they overlap, the hallucination-rate reduction and the action-success gains in Table 6 cannot be interpreted as diagnosis-driven generalization.
Reviewer_kdE8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HalluClear is a diagnostic and evaluation framework for hallucinations in GUI agents. It introduces a taxonomy of eight hallucination types and a three-stage evaluation pipeline that combines expert annotations with qualified VLM judges. The paper also proposes an OODA-style mitigation probe trained through SFT and GRPO. Experiments reveal distinct hallucination patterns across GUI agents and demonstrate improvements in hallucination rates, visual grounding, and action performance.

Critical strengths
  1. The paper provides a taxonomy and annotation protocol for diagnosing GUI-agent hallucinations beyond conventional action correctness. By distinguishing different failure types and recognizing valid alternatives to reference actions, it offers valuable insights for evaluating and improving GUI agents.
  2. The paper provides extensive experiments across multiple GUI agents and benchmarks, offering useful evidence for its diagnostic framework and demonstrating promising improvements in hallucination rates and downstream GUI performance.
Critical weaknesses
  1. Although the paper defines hallucination as an unsupported belief-action commitment under partial observability, several categories substantially overlap with conventional GUI-agent errors, such as grounding mistakes and instruction-following failures. It remains unclear whether these errors necessarily reflect unsupported beliefs or simply suboptimal decisions, raising concerns about whether the proposed evaluation framework consistently measures hallucination as a distinct phenomenon.
Questions for authors
  1. The mitigation experiments demonstrate the effectiveness of OODA reasoning, but how can the authors distinguish these improvements from the specific benefits of HalluClear's hallucination diagnosis? What evidence shows that the proposed taxonomy contributes to the performance gains beyond generic structured reasoning?
  2. How does the proposed framework distinguish hallucinations from general GUI-agent errors, such as grounding mistakes or instruction-following failures?
  3. How do the authors ensure reliable and consistent hallucination annotations when multiple actions may be valid and different hallucination types may overlap?
1 comment

Thank you for the detailed rebuttal. My concerns have been addressed, and I confirm my original score.

Reviewer_ZhwhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HalluClear, a framework for diagnosing, evaluating, and mitigating hallucinations in GUI agents. The proposed framework formalizes GUI-agent hallucinations as unsupported belief-action commitments under partial observability and develops a fine-grained taxonomy of hallucination types. It further introduces an evaluation pipeline that combines expert annotations with calibrated VLM judges to measure hallucinations, along with a post-training approach to mitigate hallucination. The framework is evaluated across multiple GUI-agent models and benchmarks to assess its effectiveness in hallucination detection and mitigation.

Critical strengths
  • This paper investigates why GUI agents fail, aiming to distinguish fine-grained perception and reasoning hallucinations.
  • By distinguishing different types of hallucinations, the proposed framework enables a more detailed analysis of why GUI agents hallucinate, potentially facilitating future research on identifying the underlying causes of hallucinations and developing targeted mitigation strategies.
  • The authors assess inter-annotator agreement using the Kappa statistic, providing quantitative evidence of annotation consistency. This strengthens the reliability and credibility of JQ-Bench as a benchmark for evaluating GUI-agent hallucinations.
Critical weaknesses
  • Since VLM judges are selected based on their agreement with the annotated JQ-Bench, the process may favor judges that align with the annotation preferences rather than those that objectively identify hallucinations.

  • HR-Pool primarily consists of error-prone queries, which may represent difficult cases. Consequently, the reported hallucination rates may not accurately reflect the prevalence of hallucinations in real-world GUI tasks.

  • The paper does not provide a systematic comparison between the proposed benchmark and existing hallucination benchmarks. Such a comparison would help clarify its unique contributions and advantages

Questions for authors
  • What is the hallucination reduction on other benchmarks using mitigation probe.
  • Could the authors clarify the scale and composition of JQ-Bench and HR-Pool, like, the number of samples.
  • How does the theoretical definition of hallucination relate to the practical annotation and evaluation criteria?
1 comment

Thank you for the rebuttal. The response has addressed most of my concerns, so I am raising my score. I encourage the authors to include the clarifications and additional results in the final version.

Reviewer_eReROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HalluClear introduces a taxonomy and evaluation pipeline for hallucinations in GUI agents. It uses expert annotations to qualify VLM judges and reports hallucination profiles across five agents.

An OODA-style post-training intervention reduces measured hallucinations and improves grounding and action accuracy.

Critical strengths

The paper makes a useful distinction between hallucinations and valid actions that differ from the reference. The examples in Appendix E explain this well. The human agreement results also support the annotation effort.

The annotation protocol handles cases that reference matching often gets wrong.

The mitigation results are promising across three agents. The comparison with generic post-training and the ablations in Table 8 help show the benefit of structured reasoning.

Critical weaknesses

The judge validation would benefit from a more independent check. Appendix D.3 describes revising human labels after reviewing judge disagreements. This may improve annotation quality, but it also makes the resulting agreement harder to interpret. A held-out human audit would help.

It would also be useful to clarify why Table 6’s baseline HR values match the Gemini column in Table 5 rather than the stated weighted ensemble.

The connection between the formal definition and the annotation rules could be clearer. Equation (1) appears to cover some suboptimal decisions that do not involve unsupported beliefs.

I was a little unclear about what Act++ actually verifies. Does it see a new screenshot after the action, or just check whether the proposed action matches the reasoning?

Questions for authors
  1. Did you keep a separate set of human annotations that was not used to revise labels or develop the judge prompts? If so, how well do the judges perform on it?

  2. Which judge or ensemble is used for the HR results in Tables 6–8? Do the other judges show similar improvements?

  3. Under Equation (1), how would you distinguish a hallucination from a planning mistake based on correctly understood evidence?

  4. Does Act++ get a new screenshot after executing the action, or does it only check whether the action matches the reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.