- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes ScreenForge, a pipeline that generates synthetic data by turning real app layouts into templates, filling them with LLM-written text and stock images, and then rendering them in a browser. Each instance keeps its source and component tree, so the same image can be paired with prose, a flat JSON, or a hierarchical JSON. The process is fully automated. They also contribute some main findings on how synthetic UI data affects text-to-image model generation: UI data improves text and layout fidelity over background-only training. The best caption format matches the inference prompt format, more variety of apps helps, and a moderate UI to background ratio in training works best for adherence. The trained model with the ScreenForge system leads open-weight systems, but trails top commercial ones.
Critical strengths
- Careful setup. The authors keep the configurations constant to isolate potential factors contributing to the experimental results. This makes it clear what causes each result.
- Extensive experiments. The authors generated over 100M screens and trained many models with the same setup, and tested 16 recent models with clear and detailed metrics. This takes a lot of computing and effort. The plan to release the 10M training images and prompt benchmark will also help other researchers.
Critical weaknesses
-
The paper has a relatively weak motivation and limited novelty. The abstract and introduction do not clearly explain the problem or why it matters. The paper suggests a gap: we do not know which training data choices help T2I models draw UI screens, since leaderboards compare whole models and cannot separate the effect of data. However, the paper mentions this only briefly in the introduction and does not explain why this gap matters in practice. Top commercial models can already do this task well, and the paper does not say who would use a model that needs such detailed JSON input in a real use case.
-
Generalization is not clearly tested. The paper does not explain how the apps are selected or how the screens are captured from them. The paper also says most of these real screens come from the same apps used to build the templates, but does not analyze the overlaps. So much of the real-screen evaluation may be on apps whose layouts the model already learned from the synthetic training data, raising data-leakage concerns.
-
The data synthesis module is not available. The ScreenForge generators are not released, so others cannot create new data with ScreenForge.
Questions for authors
- What is the intended real-world use of a model that needs a full hierarchical JSON as input? Who would write this JSON, and why use a T2I model instead of rendering the spec directly as HTML?
- Can you explain how the apps were selected and how the screens were captured?
- How many of these 130 apps overlap with the apps used to build the templates? Is it possible to report real-screen results separately for overlapping and non-overlapping apps?
- What is the main takeaway for readers? Since the generators are not released, should readers mainly use the benchmark, or the synthetic-data study findings?
5 comments
Thanks for the reponse
I thank the authors for their response and additional experiments.
However, my concerns regarding the use case and ethics have not been addressed. It remains unclear whether this is a real problem in web development (is it a critical problem?) and to what extent this method would benefit users. Furthermore, if the target setting is design and ideation, would it be fair to also compare against commercial tools such as Figma and v0?
Regarding ethics, the core issue is whether the source application developers consented to you publishing their interfaces as research artifacts. Public accessibility of Apps does not grant a copyright license to reproduce or redistribute third-party assets.
Response to Reviewer sXnd (follow-up: use case)
We thank the reviewer for the continued engagement. Two concerns remain: the practical use case, and consent and copyright for the released screenshots. We take both seriously.
Q1: Use case and comparison to design tools
We want to be precise about the claim. The paper is a data-centric study of how synthetic data shapes a text-to-image model's ability to generate structured, layout-constrained images under precise control, with UI as the testbed. Specifically, we study whether the model can follow a given structural specification and place each element faithfully. Precise control over layout and element placement is exactly what a natural-language prompt cannot pin down and where current text-to-image systems remain unreliable. UI is a demanding instance of this problem. The primary beneficiary is therefore the text-to-image research community, through the data-recipe findings and the benchmark, independent of whether image-based UI generation is a shipping product today.
On whether this is a real and critical problem, we answer in four parts:
-
Scope. The core problem is not web development specifically, but precise, controllable image generation: faithfully reproducing a given layout and the spatial relationships among its elements. UI is a strongly layout-constrained instance of this problem that we use as the testbed.
-
A recognized open problem. Precise, specification-driven control over layout and element placement is a known, still-unsolved direction for frontier text-to-image systems. Controllable, layout-conditioned generation is an active research area in its own right.
-
Real and growing demand. AI UI and mockup generation has become an active product area, with examples including Galileo/Stitch, Uizard, Figma Make, and v0. Generating layout-constrained UI from high-level intent is therefore a real and growing use case.
-
Our contribution. We target the bottleneck shared across these: which training data makes a generator reproduce layout and structure faithfully. We also provide a benchmark to measure it.
On Figma and v0, we agree they are strong tools, and we do not claim to beat them at producing editable, shippable UI. They solve a different problem with a different output: Figma is a manual vector editor, and v0 synthesizes code that is then rendered deterministically, both yielding exact, code-backed artifacts. Our object of study is the image generator itself and the data that makes it faithful. A raster image model and a code-generation-plus-renderer stack are therefore not directly comparable; they are complementary.
Image generation helps where code-based tools are weakest: acting on under-specified or natural-language intent, and producing fast visual variation for ideation without committing to a component library or working code. Code-based tools are better when the goal is exact, editable, production UI. Our setting also has advantages of its own. Generating directly with an image model is faster and cheaper than a tool-based code-generation pipeline, which requires additional tokens for code generation and tool interactions, followed by rendering. The structured JSON can also be read as another form of precise, code-like control: it gives the image model the deterministic, element-level control (i.e., bboxes) of a renderer while keeping the visual variety and aesthetics of a generative model. We will state this positioning honestly, together with the current gap relative to code-based tools.
Response to Reviewer sXnd
We thank the reviewer for the feedback. We address the three points and the ethics concerns below.
W1 · Use case
We study whether structured, component-tree-derived JSON supervision helps a text-to-image model generate UI screens with precise layouts and correct on-screen text, which plain natural-language prompts cannot fully specify. Precise text and layout rendering remains an active research direction for leading T2I systems such as Ideogram 4 and Reve 2, and UI generation is a demanding test case.
Our target setting is design and ideation tools, where users have a partial layout in mind but do not need to reproduce a specific screen pixel for pixel. Our generator supports this setting in two ways. First, it handles under-specified and natural-language prompts and transfers across caption formats (Fig. 4a), allowing users to explore different designs rather than reproduce a fixed render. Second, users do not need to write JSON. An upstream planner, such as GPT or a text-to-JSON layout front end of the kind used in modern T2I systems such as Reve 2 and Ideogram 4, converts the user's request into a layout. Hierarchical JSON serves as an internal interface and is one of several supported input formats, not a requirement for users. The structured representation is therefore a strength of the training data, not an extra burden on the user.
W2 · Generalization, app selection, and overlap
We selected the 130 real app groups to cover a broad range of categories and layout styles. Each screen is captured live, parsed, and filtered for quality. These screens are collected separately from the templates in Section 4.1 and are never used for training.
We report the exact overlap and separate the real-screen results accordingly. Of the 130 app groups, 40 were never used in template construction. On these training-disjoint groups, the deterministic structure metrics are maintained or improved: Ref.-Lay. increases from 0.191 to 0.209, Region from 0.670 to 0.675, and Global from 0.865 to 0.868. Required-text recall remains well above Ctrl on the same subset, at 0.756 versus 0.395. Its modest decrease from 0.817 falls within normal run-to-run variation.
We also report results for the 10 held-out synthetic families separately from the 100 seen families in the supplementary material. Separating seen and unseen applications shows that the gains are not an artifact of app overlap.
Table 1. Results on all real screens and training-disjoint app groups.
W3 · Release scope
We will publicly release the ScreenForge data synthesis pipeline, including the generators, alongside the 10M-image dataset and the benchmark. Researchers will be able to reproduce the synthesis process, generate new data, and use the released dataset and benchmark directly.
The findings can also guide data construction for other models, including the effect of UI data, matching caption formats to the target task, app diversity, and the UI-to-background mixture. We will revise the paper to clearly state that the release includes the synthesis pipeline and generators.
Ethics · Copyright, ToS, and PII
The 10M-image training corpus is fully synthetic. It uses generic templates, fictional text, generic icon glyphs, and license-cleared imagery. It contains no real screenshots and carries no app content, icons, logos, or branding.
Real screenshots appear only in the benchmark. We release the captions, layout specifications, evaluation harness, and 1,000 real screenshots under a research-only, non-commercial license. The screenshots are used exclusively for evaluation. The embedded imagery comes from redistributable sources, and the apps are publicly available. We will specify the release scope and exact license terms in the dataset card.