- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UI2App benchmarks the generation of executable, interactive web applications from sets of screenshots without explicit behavioral specifications. It contains 600 screenshots from 95 applications, curated from permissively licensed GitHub repositories and organized into coherent sets covering multiple routes or visual states. Models generate applications within a fixed React–TypeScript scaffold using a planning and per-file generation protocol, with error-feedback repair for build failures.
The evaluation separates three dimensions: executability, visual fidelity, and Interaction Inference Score. IIS evaluates seven interaction categories against a screenshot-derived Reference Interaction Inventory, combining implementation outcomes with state-complexity weights. An automated evaluator examines generated source code and rendered screenshots; its assessments are validated against human runtime evaluation on a 45-application subset.
Experiments cover six frontier VLMs and a Qwen2.5-VL scaling study. The main finding is that visual fidelity and interaction quality produce substantially different model rankings: Gemini leads VFS, whereas Claude leads IIS.
Critical strengths
- A useful benchmark that evaluates application behavior beyond visual reconstruction.
The multi-route screenshot sets and separate EXEC/VFS/IIS axes address a meaningful gap in screenshot-to-code evaluation. The finding that the VFS leader substantially trails the IIS leader is consequential: it shows that a strong visual score can conceal largely nonfunctional output. The reference-based, implementation-agnostic rubric is also preferable to requiring an exact source implementation or a single interaction trace.
- Substantial validation of the central empirical finding.
The authors validate automated IIS against human execution-based assessments on 45 applications and report agreement by annotation axis, score error, and ranking preservation. They additionally examine three independent generations and provide a hierarchical bootstrap interval for the Claude–Gemini difference. These analyses make the main ranking mismatch substantially more credible than a single-generation, unvalidated-judge comparison.
Critical weaknesses
- Static automated evaluation is insufficiently validated for the most consequential runtime claims.
Auto IIS inspects source code and screenshots without executing actions, while route screenshots are obtained by direct URL loading. This can establish the presence of plausible implementation logic but not necessarily correct event execution or state propagation across a navigation sequence. Aggregate agreement and correlation across six model-level scores may conceal category-specific errors, and outcome/scope agreement is reported only on mutually applicable items. Given that cross-route persistence is the paper’s principal bottleneck, runtime agreement, false-positive rates, and uncertainty specifically for S3 interactions are important to establish the reliability of that conclusion.
Questions for authors
- How reliably can the reference interactions be inferred from screenshots alone?
Were the two experts’ Reference Interaction Inventory labels produced independently, and what is their agreement on category applicability and reference scope? Please provide examples of ambiguous cases, particularly form validation, notifications, and cross-route persistence.
4 comments
[Q1] Independent RII annotation, agreement, and ambiguity handling
Two front-end engineers independently completed the RII for all 95 apps from the input screenshots using the same annotation rubric, without seeing each other’s labels. Before adjudication by a third expert, they agreed on applicability for 650 of the 665 app–category items (97.74%, Cohen’s κ = 0.945). Where both marked the category applicable, exact agreement on reference scope was 98.92%. The third expert then adjudicated disagreements using the same rubric and screenshot evidence.
Beyond reporting overall agreement, we use the three types of cases you highlighted to illustrate how the annotation rubric handles details that require interpretation from screenshots:
-
Form validation (C05,
vue-admin-arco). The ambiguity in a multi-step form is whether each step validates independently (S1), or later steps depend on earlier validation state (S2). The visible required-field, name-length, and URL-format rules concern the current step and do not show cross-step validation dependencies. The RII therefore marks C05 applicable with reference scope S1. Screenshot evidence: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-arco -
Notification (C06,
react-duolingo). The pre-submission screenshot does not show feedback.CHECKcould mean submit and continue or verify the answer. The separateSKIPcontrol supports the latter interpretation, so the RII includes feedback on the submitted answer under C06 (S1): learners should be able to tell whether their answer is correct, without prescribing the feedback format. Screenshot evidence: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-lesson -
Cross-route state (C07,
sanity-stripe-store). A cart drawer on a single product page could represent either page-local state or a cart shared across product pages. The input screenshots also provide a cart entry on another product page, supporting a shared-cart interpretation. The RII therefore marks C07 applicable with reference scope S3: added products should be retained after navigating between product pages within the app. Input and runtime screenshots: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-cart
Thank you for your response. Your clarifications have addressed most of my concerns. After considering the response, I will maintain my score.
Thank you for taking the time to consider our responses and for your thoughtful feedback. We appreciate your acknowledgment that our clarifications have addressed most of your concerns. We are grateful for your careful review and constructive suggestions throughout the process.
Thank you for recognizing UI2App’s multi-route benchmark and separate EXEC/VFS/IIS evaluation, its implementation-agnostic rubric, and the visual–interaction ranking mismatch supported by human validation and repeated-generation analyses. We address your concerns below.
[W1] Runtime validation and uncertainty for S3
To address whether static assessments reflect actual interaction execution and cross-route state propagation, we provide runtime checks and quantify S3 agreement, false-positive rates, and score uncertainty.
Auto IIS cross-route judgments and runtime verification. Auto IIS identifies interaction controls from screenshots, then checks the source code to assess whether handlers are bound, state is correctly updated and shared, and destination pages read that state to display the expected results. For example, it identified local state that was not shared across routes in
snapgram-social, and a cart page that did not read shared data inshopco-ecommerce. We verified these judgments by running the apps, performing the interactions, and navigating in-app. The observed outcomes matched Auto’s judgments. Original explanations and runtime evidence: Snapgram: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#snapgram-cross-route Shopco: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#shopco-cross-routeItem-level runtime agreement and false-positive rates for S3. To quantify Auto IIS reliability for S3, we examine reference S3 interactions in the 45-app validation subset. To validate state propagation across routes captured by direct URL loading, human annotators triggered the interaction, navigated via in-app controls within the same browser session, and labeled the outcome observed on the destination page. Under the annotation rules, result and scope are left blank for items marked not applicable, but the corresponding reference S3 requirements remain in the scoring denominator and receive zero credit (Fig. 3, §3.5, Eq. 1). To measure agreement between the evaluators’ judgments of interaction outcomes, we calculate W/P/F agreement on items with outcome labels from both evaluators.
The score comparison covers all 119 evaluable reference S3 items, including those marked not applicable by at least one evaluator, and compares Auto and human zero/nonzero credit and scores item by item to capture scoring errors arising from applicability disagreements.
Across these items, Cohen’s κ for nonzero versus zero credit is 0.83, and the mean absolute difference in per-item IIS scores is 3.6 points on a 100-point scale.
Auto IIS S3 scores and uncertainty. To quantify differences between Auto IIS and human runtime S3 scores, we compare both on the same 45-app validation subset. Brackets give 95% confidence intervals to quantify uncertainty in the S3 score estimates, obtained by resampling apps with reference S3 interactions within this subset. 0.0 denotes an observed zero on this subset.
The Auto and human S3 point estimates differ by at most 3 points. Comparing scopes within the same subset, all six models also have lower human S3 point estimates than their S1 and S2 estimates. Thus, weak cross-route performance is also observed under human runtime evaluation on this subset.
Effect of evaluator error on full-benchmark S3 results. We perform a two-level bootstrap with 2,000 replicates. In each replicate, we resample from the 45 apps with human runtime annotations and from all 95 apps, and use the human-score distributions for the same interaction category and Auto label to estimate the effect of evaluator error on S3 scores. Claude’s estimated S3 score is 14.5 (95% interval: 6.4–25.4, original Auto score: 15.8). The upper endpoint of each of the other five models’ intervals is at most 9.3. Together with the human runtime scores, this analysis indicates that cross-route state remains a weak point for models on this benchmark.