acceptodds
Under review as a conference paper at ICLR 2027

MUTUI-BENCH: BENCHMARKING MULTI-TURN IMAGE GENERATION AND EDITING FOR VISUAL INTERFACES

Abstract

A visual interface editor must change the intended element, preserve relevant context, and interpret user intent across turns. We introduce , a compact image-output benchmark with three tracks: Design for stylistic and structural editing, Assist for sequential mouse guidance, and Simulate for rendering browsing-action consequences. The corpus contains 76 tasks and 274 turns: 255 design edits across seven domains and twelve edit types, and 19 interaction turns across six websites. An audit of two archived image-generation systems separates image quality from instruction correctness, removes repeated judgments, and reports matched comparisons with task-level uncertainty. On Design, mean image quality exceeds 8.3/10 while instruction correctness remains below 6.1/10; icon replacement is particularly weak despite polished outputs. The interaction tracks expose complementary demands on pointing and state rendering, although their small size and reference-conditioned evaluation limit generalization. provides an inspectable testbed for studying what should change, what should persist, and what should happen next in visual UI editing.

Primary area:
datasets and benchmarks
Keywords:
Multi-turn image editing, User interface generation and editing, Visual UI editing benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tRa3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

MuTUI-Bench evaluates multi-turn raster image generation and editing for visual interfaces. It contains 76 tasks and 274 turns organized into Design, Assist, and Simulate: interface editing, goal-conditioned pointing, and rendering the visual consequences of interface actions. Design is the dominant component, with 70 tasks and 255 turns across seven domains and 12 edit types.

The paper analyzes an archive of outputs from two image-generation systems using a language-model judge that scores image quality and instruction-related key points. It also reports pairwise preferences, edit-type analyses, and trajectory endpoints. A substantial part of the contribution is a careful reconstruction of the evaluation archive, including score-independent deduplication, matched comparisons, task-level bootstrap intervals, and sensitivity to missing evaluations. The resulting analysis distinguishes visual polish from instruction compliance and highlights difficult interface-editing categories.

Critical strengths
  1. The benchmark targets useful interface-specific editing requirements. Exact text, icon identity, layout relationships, and preservation of surrounding elements matter in interface imagery and can be poorly represented by general image-quality measures. The Design taxonomy and the separation of Design, Assist, and Simulate provide a useful structure for examining these requirements. The corpus offers concrete material for studying why visually plausible outputs may still fail a requested interface edit.

  2. The archive audit and statistical treatment are unusually careful. The paper deduplicates records without selecting by score, uses matched comparisons, accounts for dependence within tasks through task-level bootstrapping, and examines the impact of missing scores. It also avoids overstating the observed model difference in Design key-point scores: the paired difference of 0.26 has a confidence interval of [−0.15, 0.65], whereas the image-quality difference is more clearly supported. These choices make the descriptive analysis more credible and prevent misleading conclusions from raw row averages or incomplete evaluations.

Critical weaknesses
  1. The principal Design evaluation does not adequately represent the state and requirements of the multi-turn generation process. Section 5.2 states that generation is chained through each model’s previous output, while the evaluator uses the initial image at every turn. Section 6.4 further acknowledges that the judge receives the current instruction without the full edit history. Consequently, the judge is not evaluating the current edit against the actual image supplied to the generator and cannot reliably distinguish earlier requested changes from unintended changes. Correctly retained prior edits may appear unexplained relative to the initial image, while violations of earlier requirements may be missed when only the current instruction is available. The authors disclose this limitation, but it affects the central construct being measured, particularly because Design accounts for approximately 93% of turns. A benchmark for multi-turn editing needs to distinguish current-turn execution from preservation of accumulated requirements.

  2. The validity of the automated measurements is not independently established. The reported conclusions depend heavily on one language-model judge, without a human-calibration study. This matters because Section 6.3 identifies an example where the judge’s icon-related rationale contradicts the displayed substitution, making it unclear how much of the low score reflects model failure versus evaluation error. Pairwise comparisons also use a fixed presentation order, leaving possible order effects unresolved. Bootstrap intervals quantify sampling uncertainty under the chosen evaluator, but do not address systematic judgment errors. Since the benchmark’s value depends on distinguishing subtle text, icon, and preservation failures, the absence of a targeted validity check is a material limitation.

Questions for authors
  1. [Assessment-relevant validation] Can the existing Design outputs be evaluated using the actual preceding model output for current-turn execution, together with an explicit record of accumulated requirements for preservation? Do the principal model comparisons and edit-category findings remain stable under this state- and history-aware evaluation?

  2. [Assessment-relevant validation] What agreement does the automated judge achieve with human assessment on a representative sample covering text edits, icon substitutions, layout changes, and preservation? Please also assess pairwise preferences with counterbalanced presentation order. The resulting error rates and ranking stability would materially affect my confidence in the benchmark.

  3. [Clarification] What checks ensure that later instructions remain semantically valid after the intended earlier edits, particularly when they refer to element positions or identities? Given that sequence construction uses the initial image and preceding text, a targeted audit of sequence validity would help distinguish failures of the evaluated models from ambiguity in the benchmark tasks.

Reviewer_MK5EOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MUTUI-Bench, an image-in/image-out benchmark for visual UI editing with three tracks: Design (255 turns of stylistic/structural edits over 70 synthetic interfaces in seven domains and twelve edit types), Assist (10 turns in which the model draws a mouse pointer indicating the next click toward a goal), and Simulate (9 turns in which the model renders the screen after a browsing action). Rather than running new evaluations, the paper re-analyzes archived outputs of two systems (gpt-image-2 and gemini-3.1-flash-lite-image) together with their archived judge scores (gpt-5-nano for absolute scoring) on image quality (IQ) and key-point instruction correctness (KP), after deduplicating repeated log entries. The main finding is that on Design, IQ is high (>8.3) while KP is lower (<6.1), with icon replacement scoring lowest (KP 3.84/4.32 despite IQ 8.96/8.52).

Critical strengths
  • Separating visual quality from instruction correctness is a sensible framing
  • Results are reported per track rather than pooled
Critical weaknesses
  1. The evaluation protocol does not measure multi-turn behavior. Design inference chains previous outputs, but the KP judge always uses the initial image as its reference (Sec. 5.2). Later turns are therefore judged against the original interface rather than the model's actual input, so legitimate earlier edits may be penalized as drift, and cumulative preservation, which is the core multi-turn claim, is not measured; the authors acknowledge this mismatch (Sec. 5.2, 7). All 19 interaction turns receive recorded rather than generated before-states (Sec. 4.2), so no interaction turn is genuinely chained. Moreover, Sec. 6.4 finds no first-to-last degradation over these 3–4-turn tasks, so the paper provides no evidence that the multi-turn setting poses a distinct challenge.

  2. Motivation for pixel-only outputs. The paper evaluates UI editing purely as raster-to-raster generation but does not justify why this output modality is needed. Practical UI modification is typically code-based, and multi-turn code-level editing is already covered (MT-Web2Code, EvoGenUI-Bench). Assist reduces to coordinate grounding (cf. ScreenSpot); rendering the answer as an image entangles localization with image preservation and makes evaluation less precise. Image outputs are well-motivated for GUI world models (GEBench, GUI-CC), where the generated image is the agent's observation. However, Simulate (9 turns on recorded before-states, no closed loop) is too small to serve that role, while the dominant Design track (93% of turns) has a natural code-based alternative.

  3. Judge reliability is unestablished and the headline metric is confounded. The absolute judge is gpt-5-nano, the pairwise judge is not identified, and no judge–human calibration is reported (Sec. 5.4). The paper itself reports a case where the archived judge rationale contradicts the output (Sec. 6.3), and absolute and pairwise judgments disagree on Simulate (Sec. 6.2: M has the higher absolute KP, yet G wins 8 of 9 pairwise KP comparisons). KP includes a 30% visual-quality component (Sec. 5.2), so the headline "high IQ, low KP" gap is not a clean separation of quality and correctness. Recorded targets are not provided to the absolute judge even for Assist/Simulate (Sec. 3.3, 5.2); pointing correctness could be verified objectively (e.g., pointer-in-box) if target-element boxes were annotated.

  4. Baselines. Only two proprietary pixel-generation systems are evaluated, using archived outputs with no new model calls (Sec. 5.1). There are no open-weight editors (e.g., Qwen-Image-Edit, FLUX.1 Kontext), no code-based or agentic/tool-using baselines, and no UI-finetuned model. Without these, it is unclear whether the low icon-replacement scores reflect a capability limit, the pixel-only setting, or judge error, which the authors acknowledge can coexist (Sec. 6.3). Overall, the paper reads more as an audit of archived logs than as a benchmark evaluation.

  5. Most importantly, Unclear contribution. The paper claims three contributions (Sec. 1): a task formulation linking the three tracks, a 76-task corpus, and an audit of archived outputs. Each is thin on its own. (i) The formulation (Eq. 1) is a generic image-conditioned generator with an optional before-state, and the three tracks are analyzed separately throughout; the paper never shows what evaluating them jointly reveals that separate evaluation would not. (ii) The paper itself concedes that neither local nor multi-turn image editing is unique to it (MagicBrush, ImgEdit; Sec. 2), that Assist is related to grounding (ScreenSpot; Sec. 2), and that prior work "already cover[s] important parts of Simulate" (GEBench, GUI-CC; Sec. 2). The remaining distinction, interface-specific constraints such as icon identity, exact label wording, and control alignment (Sec. 2), is not measured by any dedicated metric and is folded into a single LLM-judged KP score; Sec. 7 states that element-level preservation constraints are left to a "stronger successor." (iii) The audit makes no new model calls (Sec. 5.1) and mainly selects and deduplicates existing log records, with findings framed as diagnostic rather than validated (Sec. 1, 6.3). As a result, it remains unclear what new finding or resource the benchmark provides beyond existing editing, grounding, and GUI world-model benchmarks.

Questions for authors

Please refer to the Weaknesses and address them

Reviewer_vesVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MUTUI-Bench, an image-output benchmark for visual interface editing with three tracks: Design, Assist, and Simulate. The corpus contains 76 tasks and 274 turns, with most examples belonging to Design. The paper evaluates archived outputs from two image-generation systems using automated absolute and pairwise judgments. It reports differences between visual quality and instruction correctness, analyzes performance across editing categories, and discusses limitations involving reference conditioning, judge reliability, and interaction coverage.

Critical strengths

1.The paper addresses a practical problem: visual interface editing requires satisfying requested changes while preserving existing content and respecting interaction context. The distinction between visual quality and instruction fulfillment is useful. 2.The Design track provides an interpretable taxonomy spanning seven domains and twelve editing categories, which could support targeted analysis of interface-editing capabilities.

Critical weaknesses

1.The evaluation protocol does not adequately measure the intended multi-turn capabilities. Design generation is conditioned on previous model outputs, while the evaluator always receives the initial screenshot as its reference. Furthermore, the evaluator does not receive the complete editing history. This mismatch limits the interpretability of later-turn KP scores and the benchmark’s ability to assess cumulative preservation. Although acknowledged, its impact is not quantified through corrected evaluation. For a benchmark centered on multi-turn editing, this is a fundamental concern about measurement validity. 2.IQ and KP scores are produced by a single automated judge without human calibration. Therefore, the difference between average IQ and KP scores provides limited evidence about the extent of instruction-following failures. Pairwise evaluation uses a fixed presentation order, leaving potential position bias unmeasured. These issues make it difficult to determine whether reported category differences and system preferences reflect model behavior or evaluator artifacts. 3.The paper does not quantify the prevalence of ambiguous or inconsistent sequences, and the interaction targets lack established independent human validation. File-integrity checks and record deduplication do not resolve these concerns. 4.The manuscript distinguishes several generation-input policies, evaluator-reference policies, and absolute versus pairwise evaluation subsets. Their relationship requires substantial effort to track. Moreover, multi-turn image editing, GUI grounding, and GUI state generation have existing benchmarks. The paper does not convincingly demonstrate what additional insight is enabled by grouping these capabilities into three tracks.

Questions for authors

Please refer to the Weaknesses and address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.