acceptodds
Withdrawn from ICLR 2027

GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advancements in image generation models enable the prediction of future Graphical User Interface (GUI) states based on user instructions. However, existing benchmarks primarily focus on general-domain visual fidelity, leaving evaluation of state transitions and temporal coherence in GUI-specific contexts underexplored. To address this gap, we introduce GUI-GenBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation. GUI-GenBench comprises 700 carefully curated samples spanning five task suites, covering both single-step interactions and multi-step trajectories across real-world and fictional scenarios, as well as grounding point localization. To support systematic evaluation, we propose GUI-Score, a five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Extensive evaluation indicates that current models perform well on single-step transitions but struggle with temporal coherence and spatial grounding over longer interaction sequences. Moreover, our findings identify icon interpretation, text rendering, and localization precision as key bottlenecks, and suggest promising directions for future research toward high-fidelity generative GUI environments.

Primary area:
datasets and benchmarks
Keywords:
GUI generation, image generation, benchmark, multimodal evaluation, GUI agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BngCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents GUI-GenBench, a benchmark designed to evaluate image generation models as interactive GUI environments. It includes 700 curated samples across five task suites: single-step transitions, multi-step planning, fictional-app generation, real-app trajectories, and coordinate grounding, evaluated on Chinese and English subsets. The authors introduce GUI-Score, a rubric-based metric scored by VLMs across five dimensions: Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality. Evaluating 12 generative models, the paper shows that high visual fidelity and single-step performance do not guarantee multi-step coherence or grounding accuracy.

Critical strengths
  1. Novel and Timely Research Topic: The reviewer thinks the proposed topic of "Evaluating Image Generation Models as Interactive GUI Environments" is pretty novel, as it reframes visual generative models from passive content creators into dynamic, action-conditioned world simulators. While prior GUI benchmarks predominantly focus on agent decision-making on static environments, this work uniquely explores whether generative models can faithfully reproduce discrete state-transition dynamics, which is timely and valuable for building scalable simulation environments for GUI agents.

  2. Thoughtful Diagnostic Taxonomy: The proposed GUI-Score provides a well-structured evaluation rubric. By decoupling functional execution (GOAL, LOGIC) from interface plausibility and rendering quality (CONS, UI, QUAL), the benchmark successfully exposes "hallucinatory correctness"—cases where models generate crisp, visually authentic UI components that completely fail to execute the requested interaction logic.

  3. Extensive Model Coverage and Multi-Judge Setup: The empirical evaluation covers a diverse set of 12 representative systems, spanning both proprietary APIs and open-source generative architectures. Furthermore, reporting individual evaluation results across three frontier VLM judges (Gemini-3-Pro-Native, GPT-4o, and Qwen3-VL-235B) adds transparency and helps contextualize potential judge bias.

  4. Actionable Insights into Generative Failure Modes: The paper provides meaningful diagnostic takeaways beyond raw numbers. The clear categorization of failure modes - especially highlighting the stark disconnect between high visual fidelity and poor spatial coordinate grounding, semantic icon misinterpretation, and text rendering flaws—provides valuable directions for the community to improve GUI-oriented world models.

Critical weaknesses
  1. One-Pass Generation vs. Interactive Environment: Multi-step rollouts are generated as 5 frames in a single forward pass without intermediate action inputs. This tests sequential image generation rather than an interactive environment transition dynamic (St+at−>St+1)(S_t + a_t -> S_{t+1}).

  2. Small Sample Size per Sub-Category: The 700 samples are split across five tasks and two independently sampled languages. As a result, critical tasks like Grounding and Fictional Apps contain only ~50 samples per language, which introduces high variance and limits statistical power.

  3. VLM Reliability on Spatial Grounding: Current VLMs struggle with fine-grained coordinate precision. While the authors report high aggregate Pearson correlation (r > 0.95), sample-average correlation masks instance-level false positives and false negatives on coordinate evaluation.

  4. Averaged Metric Masks Functional Failure: Equal weighting across all five dimensions allows models to achieve decent aggregate scores (50-60) simply by rendering crisp, plausible UI elements, even when the requested interaction completely fails.

Questions for authors
  1. Interactive Rollout: Why was one-pass generation chosen over an autoregressive, frame-by-frame rollout conditioned on preceding outputs and actions?
  2. Instance-Level Agreement: What is the instance-level agreement (e.g., Cohen's kappa) between human raters and VLM judges specifically on the Grounding GOAL and LOGIC dimensions?
  3. Coordinate Prompting: Did VLM judges receive visual prompts (e.g., tap markers) during grounding evaluation, or did they rely entirely on raw numerical coordinates in the text prompt?
  4. Data Contamination: For the Real-App suite sourced from public datasets, how did you control for potential memorization by closed-source foundation models?
Reviewer_x7ieOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is proposing GUI-GenBench, which is a 700 sample benchmark that tests whether image generation models can generate GUI environments (i.e. producing the correct  next screens with respect to the actions). It defines 5 tasks, and outputs are scored by the GUI-Score which is 5 dimensional metric applied by the VLM judges across various defined dimensions by the paper. Then authors evaluate many commercial and open-source models to show their performance on this GUI generation tasks, where they perform poorly on multi-step trajectories than single step ones. They also support it by human correlation study.

Critical strengths
  • It defines an interesting problem that whether image generators can be used as GUI environments. It also separates the functional evaluation protocols from the visual ones, suggesting that visually plausible results could be wrong.
  • The coverage of the models is very broad and limitations are stated.
Critical weaknesses
  • Some metrics such as CONS seems to be trying to reward doing nothing/keeping identity?
  • The conclusions are not informative at all. The rankings and absolute values of the metrics are highly dependent on the VLM judge that we are using. Also table 1's judge is not shown.
  • How are the human validations reported? Because there is a disagreement among the judges themselves, and then how is this human metric correlation becomes this high?
  • Novelty is very limited as using the VLM's as judge is very common practice.

I believe the paper is well below the standards of this venue. The whole idea is based on using VLM as a judge on GUI task.

Questions for authors
  • Can you report the metrics on the real videos? Do metrics achieve high scores on real world/true GUI interaction videos?
  • Which judge was used for Table 1? How well the rankings are judge dependent?
  • Which judge was correlated with the human study?
  • What is the variance across the repeated runs of each judge?
  • How is it different from previous works, and what is the exact positioning against previous works (e.g. MobileWorldBench , ViMO)?
Reviewer_rB9oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for using image generation models as generative GUI environments i.e., models that render the next screen in response to an action. GUI-GenBench contains 700 samples in five task suites. Chinese and English subsets are sampled independently. Outputs are rated by VLM judges on five dimensions including goal achievement, interaction logic, content consistency, UI plausibility and visual quality. Three judges are used and twelve models are evaluated. The paper reports that Nano Banana Pro leads the Chinese subset and GPT-Image-1.5 the English subset. It also reports that strong single-step performance does not consistently extend to multi-step trajectories, and that grounding-based goal achievement is very low.

Critical strengths
  1. The problem is well motivated. whether image generators can serve as GUI world models matters for scalable agent training. separating functional dimensions from appearance is the right instinct.

  2. The proposed benchmark tasks move from fine-grained single-step instruction following, through goal-directed trajectories, to coordinate-conditioned interaction. This structure can expose different failure modes.

Critical weaknesses
  1. I think, the objective of the benchmark is to study is an environment that maps actions to states, but the proposed tasks do not align. I mean, three suites depart from this framing (if I understood it correctly). For instance, the fictional-app suite has no reference screen and no action. It is T2I interface generation and says nothing about environment dynamics. Similarly, in real-app suite, the instructions shown in Figure 2 describe the target screens themselves. The task therefore reduces to caption-to-image generation with the answer provided in the prompt. Moreover, the multi-step suite gives only a high-level goal ("Order a coffee"). Many trajectories are valid, and I do not find in the paper whether the employed judge compares against a reference trajectory or scores plausibility alone.

  2. Content Consistency, UI Plausibility and Visual Quality all reward preserving the input screen, while only GOAL and LOGIC reward performing the action. Under this criterian, I think a model that returns its input unchanged can thus score high on three of five equally weighted dimensions while doing nothing. I seems like a design flaw in the metric to me.

  3. I have confusion regarding the equation 1. It averages over samples, which would weight the 200 sample suites twice as heavily as the others. Table 1 instead reports the unweighted mean of the five suite scores for eleven of twelve models. For GPT-Image-1.5, neither rule reproduces the table. Also, the English ranking happens to survive, but the aggregation rule must be stated and applied uniformly.

  4. Each output is judged three times, yet no run-to-run variance or significance test is reported. Many neighbouring differences in Table 1 are plausibly within judge noise. I think it would nice to see some uncertainty metrics to have a clear picture.

  5. ViMo and MobileWorldBench are cited but not compared. I think, these are considered among the seminal works in this direction, so should be discussed as a brief comparison to highlight the contribution of the proposed benchmark.

  6. VLMs as a judge is a quite growing practice in our community, however, it also involves risk of hallucination and related factors, as stated in 'GenColorBench [CVPR'26]'. However, the authors do not discuss this aspect in the paper.

Questions for authors
  1. For the multi-step and real-app suites, what is the ground truth? Does the judge compare against a reference trajectory, and do the real-app instructions describe the target screens?
  2. What scores do an identity baseline and a repeated-frame baseline obtain on each suite and dimension? Would a GOAL-gated aggregate change the rankings?
  3. How do single-image models produce five-frame trajectories? How do results change under a sequential protocol?
  4. How does grounding performance change when the click location is drawn on the screen?
  5. Which aggregation rule produced Table 1, and why does GPT-Image-1.5's aggregate differ from the mean of its suite scores?
  6. Which judge produced Table 1 and which was compared in human preference study? Given that the multi-step gap reverses under GPT-4o and Qwen3-VL, which judge should readers trust?
  7. What are the within-task correlations, the absolute agreement and the inter-annotator agreement in the human study? What is the variance across the three judge runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.