acceptodds
Under review as a conference paper at ICLR 2027

Learning to Draw Interfaces: What Makes Synthetic UI Data Work for Text-to-Image Models

Abstract

Generating interface screenshots requires text-to-image models to reproduce both the specified text and the arrangement of components. In this paper, we propose ScreenForge, a data synthesis system for studying how training data shapes these capabilities. The system renders application templates and retains source documents and component trees, allowing different captions to be derived for the same image. We use ScreenForge to compare caption recipes, application coverage, and UI-to-background mixtures in single-run continued-training experiments, each starting from the same 2B checkpoint and using 10 million examples. These comparisons show that caption choice depends on the inference prompt format and that broader application coverage improves most metrics at a fixed UI sample count. Increasing the UI share does not improve all metrics: moderate shares favor text and layout adherence, while larger shares yield modestly higher visual quality. On a benchmark of 1,300 structured screen specifications, a 50% UI mixture raises required-text recall from 0.465 to 0.813 and VLM-rated layout adherence from 3.24 to 3.84 compared with background-only continued training. We will release the rendered training images and captions, data manifests, and all 1,300 benchmark reference screens, together with the evaluation code, to support further research on UI generation.

Primary area:
datasets and benchmarks
Keywords:
synthetic data, text-to-image, user interfaces, structured captions, data-centric benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sXndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ScreenForge, a pipeline that generates synthetic data by turning real app layouts into templates, filling them with LLM-written text and stock images, and then rendering them in a browser. Each instance keeps its source and component tree, so the same image can be paired with prose, a flat JSON, or a hierarchical JSON. The process is fully automated. They also contribute some main findings on how synthetic UI data affects text-to-image model generation: UI data improves text and layout fidelity over background-only training. The best caption format matches the inference prompt format, more variety of apps helps, and a moderate UI to background ratio in training works best for adherence. The trained model with the ScreenForge system leads open-weight systems, but trails top commercial ones.

Critical strengths
  • Careful setup. The authors keep the configurations constant to isolate potential factors contributing to the experimental results. This makes it clear what causes each result.
  • Extensive experiments. The authors generated over 100M screens and trained many models with the same setup, and tested 16 recent models with clear and detailed metrics. This takes a lot of computing and effort. The plan to release the 10M training images and prompt benchmark will also help other researchers.
Critical weaknesses
  • The paper has a relatively weak motivation and limited novelty. The abstract and introduction do not clearly explain the problem or why it matters. The paper suggests a gap: we do not know which training data choices help T2I models draw UI screens, since leaderboards compare whole models and cannot separate the effect of data. However, the paper mentions this only briefly in the introduction and does not explain why this gap matters in practice. Top commercial models can already do this task well, and the paper does not say who would use a model that needs such detailed JSON input in a real use case.

  • Generalization is not clearly tested. The paper does not explain how the apps are selected or how the screens are captured from them. The paper also says most of these real screens come from the same apps used to build the templates, but does not analyze the overlaps. So much of the real-screen evaluation may be on apps whose layouts the model already learned from the synthetic training data, raising data-leakage concerns.

  • The data synthesis module is not available. The ScreenForge generators are not released, so others cannot create new data with ScreenForge.

Questions for authors
  1. What is the intended real-world use of a model that needs a full hierarchical JSON as input? Who would write this JSON, and why use a T2I model instead of rendering the spec directly as HTML?
  2. Can you explain how the apps were selected and how the screens were captured?
  3. How many of these 130 apps overlap with the apps used to build the templates? Is it possible to report real-screen results separately for overlapping and non-overlapping apps?
  4. What is the main takeaway for readers? Since the generators are not released, should readers mainly use the benchmark, or the synthetic-data study findings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sXnd

We thank the reviewer for the feedback. We address the three points and the ethics concerns below.

W1 · Use case

We study whether structured, component-tree-derived JSON supervision helps a text-to-image model generate UI screens with precise layouts and correct on-screen text, which plain natural-language prompts cannot fully specify. Precise text and layout rendering remains an active research direction for leading T2I systems such as Ideogram 4 and Reve 2, and UI generation is a demanding test case.

Our target setting is design and ideation tools, where users have a partial layout in mind but do not need to reproduce a specific screen pixel for pixel. Our generator supports this setting in two ways. First, it handles under-specified and natural-language prompts and transfers across caption formats (Fig. 4a), allowing users to explore different designs rather than reproduce a fixed render. Second, users do not need to write JSON. An upstream planner, such as GPT or a text-to-JSON layout front end of the kind used in modern T2I systems such as Reve 2 and Ideogram 4, converts the user's request into a layout. Hierarchical JSON serves as an internal interface and is one of several supported input formats, not a requirement for users. The structured representation is therefore a strength of the training data, not an extra burden on the user.

W2 · Generalization, app selection, and overlap

We selected the 130 real app groups to cover a broad range of categories and layout styles. Each screen is captured live, parsed, and filtered for quality. These screens are collected separately from the templates in Section 4.1 and are never used for training.

We report the exact overlap and separate the real-screen results accordingly. Of the 130 app groups, 40 were never used in template construction. On these training-disjoint groups, the deterministic structure metrics are maintained or improved: Ref.-Lay. increases from 0.191 to 0.209, Region from 0.670 to 0.675, and Global from 0.865 to 0.868. Required-text recall remains well above Ctrl on the same subset, at 0.756 versus 0.395. Its modest decrease from 0.817 falls within normal run-to-run variation.

We also report results for the 10 held-out synthetic families separately from the 100 seen families in the supplementary material. Separating seen and unseen applications shows that the gains are not an artifact of app overlap.

Table 1. Results on all real screens and training-disjoint app groups.

SubsetModelGroupsRef.-Lay.RegionGlobalT-Rec.PromptLayoutQuality
All realMain1300.1910.6700.8650.8173.403.792.64
All realCtrl1300.1730.6480.8200.4792.593.091.55
Disjoint realMain400.2090.6750.8680.7563.273.612.50
Disjoint realCtrl400.1990.6500.8180.3952.582.991.56

W3 · Release scope

We will publicly release the ScreenForge data synthesis pipeline, including the generators, alongside the 10M-image dataset and the benchmark. Researchers will be able to reproduce the synthesis process, generate new data, and use the released dataset and benchmark directly.

The findings can also guide data construction for other models, including the effect of UI data, matching caption formats to the target task, app diversity, and the UI-to-background mixture. We will revise the paper to clearly state that the release includes the synthesis pipeline and generators.

Ethics · Copyright, ToS, and PII

The 10M-image training corpus is fully synthetic. It uses generic templates, fictional text, generic icon glyphs, and license-cleared imagery. It contains no real screenshots and carries no app content, icons, logos, or branding.

Real screenshots appear only in the benchmark. We release the captions, layout specifications, evaluation harness, and 1,000 real screenshots under a research-only, non-commercial license. The screenshots are used exclusively for evaluation. The embedded imagery comes from redistributable sources, and the apps are publicly available. We will specify the release scope and exact license terms in the dataset card.

Reviewer_sXndOpenReview ↗

Thanks for the reponse

I thank the authors for their response and additional experiments.

However, my concerns regarding the use case and ethics have not been addressed. It remains unclear whether this is a real problem in web development (is it a critical problem?) and to what extent this method would benefit users. Furthermore, if the target setting is design and ideation, would it be fair to also compare against commercial tools such as Figma and v0?

Regarding ethics, the core issue is whether the source application developers consented to you publishing their interfaces as research artifacts. Public accessibility of Apps does not grant a copyright license to reproduce or redistribute third-party assets.

AuthorsOpenReview ↗

Response to Reviewer sXnd (follow-up: use case)

We thank the reviewer for the continued engagement. Two concerns remain: the practical use case, and consent and copyright for the released screenshots. We take both seriously.

Q1: Use case and comparison to design tools

We want to be precise about the claim. The paper is a data-centric study of how synthetic data shapes a text-to-image model's ability to generate structured, layout-constrained images under precise control, with UI as the testbed. Specifically, we study whether the model can follow a given structural specification and place each element faithfully. Precise control over layout and element placement is exactly what a natural-language prompt cannot pin down and where current text-to-image systems remain unreliable. UI is a demanding instance of this problem. The primary beneficiary is therefore the text-to-image research community, through the data-recipe findings and the benchmark, independent of whether image-based UI generation is a shipping product today.

On whether this is a real and critical problem, we answer in four parts:

  1. Scope. The core problem is not web development specifically, but precise, controllable image generation: faithfully reproducing a given layout and the spatial relationships among its elements. UI is a strongly layout-constrained instance of this problem that we use as the testbed.

  2. A recognized open problem. Precise, specification-driven control over layout and element placement is a known, still-unsolved direction for frontier text-to-image systems. Controllable, layout-conditioned generation is an active research area in its own right.

  3. Real and growing demand. AI UI and mockup generation has become an active product area, with examples including Galileo/Stitch, Uizard, Figma Make, and v0. Generating layout-constrained UI from high-level intent is therefore a real and growing use case.

  4. Our contribution. We target the bottleneck shared across these: which training data makes a generator reproduce layout and structure faithfully. We also provide a benchmark to measure it.

On Figma and v0, we agree they are strong tools, and we do not claim to beat them at producing editable, shippable UI. They solve a different problem with a different output: Figma is a manual vector editor, and v0 synthesizes code that is then rendered deterministically, both yielding exact, code-backed artifacts. Our object of study is the image generator itself and the data that makes it faithful. A raster image model and a code-generation-plus-renderer stack are therefore not directly comparable; they are complementary.

Image generation helps where code-based tools are weakest: acting on under-specified or natural-language intent, and producing fast visual variation for ideation without committing to a component library or working code. Code-based tools are better when the goal is exact, editable, production UI. Our setting also has advantages of its own. Generating directly with an image model is faster and cheaper than a tool-based code-generation pipeline, which requires additional tokens for code generation and tool interactions, followed by rendering. The structured JSON can also be read as another form of precise, code-like control: it gives the image model the deterministic, element-level control (i.e., bboxes) of a renderer while keeping the visual variety and aesthetics of a generative model. We will state this positioning honestly, together with the current gap relative to code-based tools.

Reviewer_qAh5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how synthetic training data affects text-to-image generation of mobile UI screenshots, where both specified text and component layout must be reproduced. It introduces SCREENFORGE, a six-stage pipeline that captures application layouts, builds parameterized templates with an LLM coding agent, fills fictional copy and retrieved imagery, renders screens in a headless browser, and deterministically derives prose, flat-JSON, and hierarchical-JSON captions from stored source documents and measured component trees. The pool spans 100 training and ten held-out application families; 10M of 110M rendered images will be released.

Critical strengths
  1. Strong, multifaceted evidence that structure-preserving synthetic UI data is effective.
  2. A valuable and carefully constructed data/evaluation resource.
Critical weaknesses
  • The finer data-recipe recommendations have no adequate uncertainty or judge-validity analysis. Every controlled arm is a single training run (§7 and Reproducibility Statement), and the paper reports neither seed variability nor confidence intervals. Several headline distinctions are small: Prompt Adherence is 3.45 for DIV-50 versus 3.44 for MAIN/DIV-100 (Table 5), and Visual Quality changes only from 2.67 to 2.69 to 2.76 to 2.79 across 25%, 50%, 75%, and 100% UI (Table A11). Moreover, the cross-format conclusions in Figure 4a and Table A9 rely exclusively on Gemini judgments, without human agreement or repeated-judge variability. This does not overturn the large MAIN–CTRL gap, but it materially weakens claims about saturation, monotonic quality improvement, and the best caption/mixture recipe.
  • The ablations match examples, not tokens, updates, or compute, leaving a potentially important confound. Appendix D explicitly states that budgets are matched in examples rather than tokens, and Table A2 specifies variable-length packing with no fixed optimizer-step endpoint. Caption schemas differ substantially in length, while native 512-resolution background examples use 256 image tokens and other examples can use up to 1,024. Thus caption-format and UI-share arms may receive materially different token totals and numbers of optimization updates. Without per-arm accounting or a matched-token/update control, it is difficult to determine how much of the caption and mixture effects comes from the data recipe itself rather than from differing optimization exposure.
Questions for authors
  1. For every continued-training arm, how many total text tokens, image tokens, optimizer updates, and GPU hours were consumed? If these differ materially, can you provide token/update-matched controls for at least the caption-format and UI-share comparisons?
  2. Can you provide uncertainty and judge-validation evidence for the finer recipe rankings: application-level bootstrap intervals, repeated-judge variability or human agreement on a sample, and, where feasible, a second training seed? In particular, are the 25%-UI adherence advantage and the Visual Quality trend in Table A11 robust?
  3. Were the 1,000 real benchmark screens, and near-duplicate captures of the same application states, strictly excluded from the captures used to build and revise templates in §4.1? Please clarify screen-level de-duplication between template construction and evaluation; if such overlap exists, can you report results on a rigorously disjoint subset?
  4. What licenses permit redistribution of the stock imagery embedded in the planned 10M-render release and of the 1,000 real application screenshots? The Ethics Statement mentions license audits for fonts and templates but does not establish the redistribution basis for these other assets.
2 comments
AuthorsOpenReview ↗

Response to Reviewer qAh5 (Q1 / Q2)

We thank the reviewer for the constructive feedback and for recognizing the effectiveness of our synthetic UI data and the value of our data and evaluation resources. We address Q1 to Q4 below.

Q1 · Budget matching

Every arm uses the same example budget and the same per-step token budget. Total updates and total tokens differ only because content length varies, not because we deliberately allocate more compute to one arm. Since every arm is trained to saturation, these differences do not provide an optimization advantage.

GPU hours and step counts reflect content length and hardware rather than a separately chosen compute budget, so they are not the matching criteria. Main improves required-text recall from 0.465 to 0.813 over Ctrl. More training tokens do not consistently lead to better results: increasing the UI:bg ratio from 50:50 to 75:25 raises total tokens from 35.9B to 42.4B, but required-text recall falls from 0.813 to 0.801 and Layout Adherence remains at 3.84. At 100:0, training uses 45.4B tokens, yet recall drops further to 0.757 and Layout Adherence to 3.66. These results support our attribution of the gains to the training data rather than simply to more compute.

Table 1. Training costs for one epoch on 8×H200.

ArmUI:bgUpdatesTokensHours
Ctrl0:100179,47224.3B~35
Main50:50265,77435.9B~52
Caption-flat50:50207,68028.1B~40
Caption-mix50:50217,33529.4B~42
Caption-prose50:50186,60525.2B~36
Diversity 25-app50:50259,26235.0B~50
UI-share 25%25:75218,55629.5B~43
UI-share 75%75:25313,83942.4B~61
UI-share 100%100:0336,18645.4B~65

Q2 · Uncertainty and judge validity

We add three pieces of evidence. First, we report application-level bootstrap 95% confidence intervals for every reported metric. Second, we rerun the VLM judge with independent sampling to measure variability across repeated judgments and report agreement on a sampled subset. Third, every training arm uses the same fixed seed and training recipe, so the remaining differences reflect the intervention rather than seed noise.

These checks address the close comparisons highlighted by the reviewer: Prompt Adherence of 3.45 versus 3.44 for DIV-50 and DIV-100, and the visual-quality trend from 2.67 to 2.79 in Table A11. They do not change the large gap between Main and Ctrl, which remains well outside the uncertainty intervals.

Table 2. Scores with application-level bootstrap 95% CIs, using 5,000 resamples.

ArmT-Rec.Ref.-Lay.RegionGlobalPromptLayoutQuality
Diversity 25-app0.765 [0.732, 0.798]0.180 [0.162, 0.198]0.677 [0.669, 0.685]0.860 [0.854, 0.866]3.34 [3.26, 3.42]3.69 [3.60, 3.78]2.55 [2.47, 2.63]
Diversity 50-app0.798 [0.768, 0.828]0.173 [0.155, 0.191]0.680 [0.672, 0.688]0.858 [0.852, 0.864]3.45 [3.37, 3.53]3.75 [3.66, 3.84]2.64 [2.56, 2.72]
Main (Div-100 / UI-50)0.813 [0.786, 0.840]0.203 [0.184, 0.222]0.680 [0.671, 0.689]0.868 [0.862, 0.874]3.44 [3.37, 3.51]3.84 [3.76, 3.92]2.69 [2.61, 2.77]
UI-share 25%0.825 [0.795, 0.855]0.199 [0.180, 0.218]0.683 [0.675, 0.691]0.862 [0.856, 0.868]3.55 [3.47, 3.63]3.99 [3.91, 4.07]2.67 [2.58, 2.76]
UI-share 75%0.801 [0.773, 0.829]0.186 [0.167, 0.205]0.680 [0.671, 0.689]0.865 [0.859, 0.871]3.50 [3.43, 3.57]3.84 [3.76, 3.92]2.76 [2.67, 2.85]
UI-share 100%0.757 [0.732, 0.782]0.171 [0.151, 0.191]0.678 [0.669, 0.687]0.862 [0.856, 0.868]3.42 [3.35, 3.49]3.66 [3.58, 3.74]2.79 [2.70, 2.88]
AuthorsOpenReview ↗

Response to Reviewer qAh5 (Q3 / Q4)

Q3 · De-duplication and leakage

The real benchmark and the templates in Section 4.1 draw from the same app pool, so many real screens share structure with a screen captured for template construction. However, 40 real app groups were never used to build templates. On these training-disjoint groups, deterministic performance is maintained or improved: Ref.-Lay., Region, and Global are all slightly higher. Required-text recall remains well above Ctrl on the same subset, at 0.756 versus 0.395. Its modest decrease from 0.817 falls within normal run-to-run variation.

Training contains no real screenshots. It uses only synthetic renders with fictional text and license-cleared imagery, making pixel-level leakage impossible by construction. A near-duplicate audit of all 1,000 real benchmark screens found no match within a dHash Hamming distance of 5 out of 64 bits; the median nearest-match distance was 17. Our conclusions therefore do not depend on applications overlapping between training and evaluation.

Table 3. Results on all real screens and training-disjoint app groups.

SubsetModelGroupsRef.-Lay.RegionGlobalT-Rec.PromptLayoutQuality
All realMain1300.1910.6700.8650.8173.403.792.64
All realCtrl1300.1730.6480.8200.4792.593.091.55
Disjoint realMain400.2090.6750.8680.7563.273.612.50
Disjoint realCtrl400.1990.6500.8180.3952.582.991.56

Q4 · Asset licensing

The 10M-render training corpus is fully synthetic. It uses generic HTML/CSS templates, fictional text written by an LLM, generic Unicode icon glyphs rather than real app icons, and license-cleared imagery. It reproduces layout structure without copying app content, icons, logos, or branding. Since it contains no real screenshots, its redistribution does not depend on any app's terms.

The 1,000 real screenshots appear only in the benchmark, which we release with structured captions, layout specifications, and the evaluation harness under a research-only, non-commercial license. These screenshots are used exclusively for evaluation, never for training. The embedded imagery comes from redistributable sources, and the apps themselves are publicly available. We will specify the release scope and exact license terms in the dataset card.

Reviewer_hQNkOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposed a data synthesis pipeline and built a synthetic dataset of GUI screenshots. Specifically, it captures structured description of UI from real applications, convert them into templates with LLM, generate diverse manifestation for these templates and render them deterministically to obtain clean image samples. This paper also conducted experiments to demonstrate effectiveness of the proposed dataset, and to study the impact of different properties of the dataset.

Critical strengths
  1. The proposed data synthesis pipeline is built upon a structured approach, and is able to synthesize clean UI screenshots.
  2. By capturing samples from real applications, the proposed dataset faithfully captures the layout of real applications.
Critical weaknesses
  1. The fairness of the comparison against existing methods in Section 6.2 is questionable. All methods compared in Table 3 are general-purpose methods that are not fine-tuned on UI-screenshot datasets. In addition, the evaluation samples are taken from the held-out split of the proposed dataset. Although these samples are not leaked into the training set, they come from the same distribution as the proposed dataset. Therefore, the comparison may not fully reflect whether the proposed dataset brings an advantage over methods that are adapted to the target UI-screenshot domain.
  2. The source of the performance gain in the comparison experiments is not fully attributed to the proposed dataset. As shown in Table 4, the untrained Ref model already outperforms all open-weight models in Table 3 on many metrics (e.g., Ref.-Lay., Regional Visual Fidelity, VLM prompt & layout). At the same time, after training on the proposed dataset, no significant improvement is observed on some metrics such as Ref.-Lay. This makes it unclear whether the gains mainly come from the proposed dataset or from the Ref model (BLIP3-o) itself.
Questions for authors
  1. (W1) The fairness of the comparison in Section 6.2 is unclear. Please address the following questions: Are all methods in Table 3 general-purpose methods that are not fine-tuned on UI-screenshot datasets? Are the evaluation samples from the held-out split of the proposed dataset? If so, how does using samples from the same distribution affect the fairness of the comparison? Were any baseline methods fine-tuned or adapted on the same or comparable UI-screenshot data (such as the datasets listed in Table 2)? How do the results compare with such domain-adapted baselines?
  2. (W2) The source of the performance gain is unclear. Please address the following questions: How do the authors isolate the contribution of the proposed dataset from the contribution of the BLIP3-o model design or pretraining? Can the authors provide a fair, quantitative comparison between performance of models trained on different UI-screenshot-centric datasets?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hQNk

We thank the reviewer for the thoughtful questions. Below, we address the concerns about the fairness of the comparisons and the attribution of the gains.

W1 · Fairness of the comparison

Table 3 includes strong recent models, from commercial APIs such as GPT-Image-2.5 to open models such as SenseNova-U1.5 and FLUX.2-dev. These models are pretrained on large-scale web data containing many UI screenshots and already generate UI well. For example, GPT-Image-2.5 achieves 0.204 reference-layout fidelity and 4.94 quality, making Table 3 a demanding baseline. We are not equating general UI exposure with in-domain training. Our point is that the reported gap comes from the structured, component-tree-derived supervision in our data, which ties each element to a specific layout and text, rather than from general UI exposure or natural-language screen descriptions.

The main benchmark is not our training split. It contains 1,000 real screens from 130 app groups and 300 synthetic screens from 10 held-out families, for a total of 1,300 screens. All headline results are reported on this set, while results on the 100-family seen set are reported separately. We did not include an external-dataset baseline because none provides the supervision our pipeline requires: structured labels paired with re-renderable templates. Public datasets such as WebUI and WebSight focus on web interfaces rather than mobile apps. RICO-family datasets contain app screenshots, but lack this annotation format; their view hierarchies are dated and noisy, and their fixed screenshots cannot be re-rendered.

As a preliminary check, we also fine-tuned SenseNova-U1.5 and FLUX.2-dev on our data. These models are conditioned on short natural-language descriptions, which differ substantially from our long, structured inputs, and a short SFT run did not bridge that gap. Required-text recall even dropped from 0.783 to about 0.27 for SenseNova-U1.5 and from 0.369 to about 0.004 for FLUX.2-dev. We attribute these drops to the mismatch in input length and format, together with the limited training budget for pretraining stage. For reference, our model uses this representation from the start of the pretraining (not only for SFT) and achieves 0.813 on the same metric. In future work, we will systematically study these factors and conduct a full cross-model comparison under matched conditions.

W2 · Attribution of the gain

Ref / Ctrl / Main provides a controlled comparison. All three use the same BLIP3-o base and the same 10M-example budget; only the training data changes. The difference between Main and Ctrl therefore isolates the effect of our data while holding the base and compute fixed. This addresses both parts of the concern.

First, Ref already scores well on some Table 4 metrics before UI training because it is not the original free-text BLIP3-o. It has been trained on BLIP3-o images that we re-labeled with structured JSON, giving it substantial structural knowledge and strong performance on metrics such as Ref.-Lay. and Regional Visual Fidelity. This strong starting point comes from the re-labeled base. We therefore attribute the gain through the Main–Ctrl comparison on the same base, rather than through comparisons with the Table 3 models. Main and Ctrl differ in the overall data recipe, so this attribution applies to the dataset as a whole, not to an individual JSON field.

Second, the gains are concentrated in the UI-specific abilities that the base model lacks: finer layout control and correct on-screen text. From Ctrl to Main, required-text recall increases from 0.465 to 0.813, and VLM Layout Adherence rises from 3.24 to 3.84. Ref.-Lay. is a coarse metric on which both Ref and Main already score highly. Its small change is therefore consistent with limited headroom rather than an absence of improvement; the finer-grained VLM Layout Adherence metric captures that improvement more clearly.

With the base and compute held fixed, we attribute these gains on UI metrics to the dataset rather than to the base model or additional compute.

Reviewer_hQNkOpenReview ↗

Remaining Concerns

We thank the authors for providing additional information.

We have the following remaining concerns:

Q1. The response did not fully address question 1 in our initial comment regarding fairness of the comparison: as the authors have stated in their response to W2, the performance gain comes from the fact that the Ref and Main models are specifically trained on UI data, with highly detailed prompt as in the evaluation setting. Other baseline methods, however, are not fine-tuned to generate UI with such a detailed condition. Even though the benchmark is not using data from the training split, it still targets the task of generating UI images from detailed text (e.g., structured representation of the UI) conditions, which does not align well with what the baseline models have been trained on.
In other words, the perceived performance gain seems to be a natural result of having aligned data in training and evaluation (and this is also demonstrated in Fig. 4a of the manuscript). Please clarify what are the contributions of the work beyond this.

Q2. What is the motivation of generating UI images from a detailed and complex description like JSON? From users' perspective, this is inconvenient. And why not simply using the proposed generator used in data synthesis, just by instructing an LLM to prepare appropriate inputs and invoke tools? Why is it necessary to further train an end-to-end image generator for this?

AuthorsOpenReview ↗

Response to Reviewer hQNk (follow-up, Q1)

We thank the reviewer for the thoughtful follow-up. We answer the two remaining questions below, Q1 (the contribution beyond aligned conditioning) and Q2 (why an image generator from a structured input).

Q1: The contribution beyond aligned training and evaluation.

The reviewer is right that a model trained on UI with detailed conditions will, as expected, do better in that setting than one that never saw it, and we do not claim otherwise. Our contribution is not the Table 3 ranking, which only situates the model among existing systems. The scientific claims are data-centric, namely which data-construction choices let a text-to-image model render UI with correct text and layout, and we establish them with controlled studies in which both arms are equally in-domain. Ref, Ctrl, and Main share one base and one compute budget and differ only in the training data, and the recipe ablations (caption format, app diversity, UI-to-background ratio) each vary a single data factor. This is the methodology of data-centric benchmarks such as DataComp, the training recipe is frozen and only the data changes, so the study measures what data is useful rather than which model is best. These comparisons are immune to the alignment objection, because the thing being aligned, in-domain UI training under detailed conditions, is held constant across the arms, and only the data recipe changes.

The detailed condition is the object of study, not a hidden advantage. The open question we target is whether a structured specification can control a generator's layout and on-screen text at the element level, and which data makes that control emerge. Fig. 4a is consistent with this rather than with pure train-eval overlap, the model transfers across conditioning formats (prose, flat JSON, and hierarchical JSON all work), and the gains hold on 40 app groups never seen in template construction (Ref.-Lay. 0.191 to 0.209, with required-text recall 0.756, far above the 0.395 of the same-base control). A model that only benefited from aligned train and evaluation would not transfer across formats or to unseen apps.

So the contributions that do not reduce to alignment are three.

  1. A scalable synthesis pipeline that produces structured UI supervision. It turns real app layouts into re-renderable templates, keeps each screen's component tree, fills the templates with LLM-written text and license-cleared imagery, and renders them deterministically into clean image and label pairs. Because every sample retains its source and component tree, the same image can be paired with prose, flat JSON, or hierarchical JSON, so the supervision ties each element to an exact layout and text in a way no existing UI dataset provides. Because the templates are re-renderable rather than fixed screenshots, the pipeline generates many controlled variants at scale (over 10M), and we release the pipeline and the data generators so others can produce new data, not only consume ours.

  2. A set of controlled, transferable data findings. With the base and compute held fixed, we isolate which data properties drive UI fidelity, UI data over background-only training improves text and layout, the caption format works best when it matches the inference prompt, broader app diversity helps, and a moderate UI-to-background ratio is best for adherence. These are recipe-level conclusions rather than properties of our specific model, so any team training a UI generator on any base can apply them.

  3. A benchmark for structured UI generation. It evaluates on 1,300 images (1,000 real screens from 130 app groups plus 300 synthetic screens from 10 held-out families), with deterministic structure metrics (reference-layout fidelity, regional and global visual similarity, and required-text recall) and VLM-judged prompt, layout, and quality scores. It is built to measure generalization to unseen apps rather than reconstruction of seen layouts.

None of these is a claim that aligning training and evaluation is itself the result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.