acceptodds
Under review as a conference paper at ICLR 2027

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Abstract

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, existing visual-reconstruction benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to infer application behavior from screenshots alone, without added behavioral guidance. UI2App comprises 600 screenshots grouped into 95 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along three dimensions: executability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 8.1 on IIS, ranking fourth and trailing the IIS leader by . High-complexity interactions such as cross-page state remain a pervasive bottleneck, with five of the six evaluated models scoring at most 3.5 on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.

Primary area:
datasets and benchmarks
Keywords:
Website Generation, VLM, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bDbzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

UI2App benchmarks the generation of executable, interactive web applications from sets of screenshots without explicit behavioral specifications. It contains 600 screenshots from 95 applications, curated from permissively licensed GitHub repositories and organized into coherent sets covering multiple routes or visual states. Models generate applications within a fixed React–TypeScript scaffold using a planning and per-file generation protocol, with error-feedback repair for build failures.

The evaluation separates three dimensions: executability, visual fidelity, and Interaction Inference Score. IIS evaluates seven interaction categories against a screenshot-derived Reference Interaction Inventory, combining implementation outcomes with state-complexity weights. An automated evaluator examines generated source code and rendered screenshots; its assessments are validated against human runtime evaluation on a 45-application subset.

Experiments cover six frontier VLMs and a Qwen2.5-VL scaling study. The main finding is that visual fidelity and interaction quality produce substantially different model rankings: Gemini leads VFS, whereas Claude leads IIS.

Critical strengths
  1. A useful benchmark that evaluates application behavior beyond visual reconstruction.

The multi-route screenshot sets and separate EXEC/VFS/IIS axes address a meaningful gap in screenshot-to-code evaluation. The finding that the VFS leader substantially trails the IIS leader is consequential: it shows that a strong visual score can conceal largely nonfunctional output. The reference-based, implementation-agnostic rubric is also preferable to requiring an exact source implementation or a single interaction trace.

  1. Substantial validation of the central empirical finding.

The authors validate automated IIS against human execution-based assessments on 45 applications and report agreement by annotation axis, score error, and ranking preservation. They additionally examine three independent generations and provide a hierarchical bootstrap interval for the Claude–Gemini difference. These analyses make the main ranking mismatch substantially more credible than a single-generation, unvalidated-judge comparison.

Critical weaknesses
  1. Static automated evaluation is insufficiently validated for the most consequential runtime claims.

Auto IIS inspects source code and screenshots without executing actions, while route screenshots are obtained by direct URL loading. This can establish the presence of plausible implementation logic but not necessarily correct event execution or state propagation across a navigation sequence. Aggregate agreement and correlation across six model-level scores may conceal category-specific errors, and outcome/scope agreement is reported only on mutually applicable items. Given that cross-route persistence is the paper’s principal bottleneck, runtime agreement, false-positive rates, and uncertainty specifically for S3 interactions are important to establish the reliability of that conclusion.

Questions for authors
  1. How reliably can the reference interactions be inferred from screenshots alone?

Were the two experts’ Reference Interaction Inventory labels produced independently, and what is their agreement on category applicability and reference scope? Please provide examples of ambiguous cases, particularly form validation, notifications, and cross-route persistence.

4 comments

Thank you for recognizing UI2App’s multi-route benchmark and separate EXEC/VFS/IIS evaluation, its implementation-agnostic rubric, and the visual–interaction ranking mismatch supported by human validation and repeated-generation analyses. We address your concerns below.

[W1] Runtime validation and uncertainty for S3

To address whether static assessments reflect actual interaction execution and cross-route state propagation, we provide runtime checks and quantify S3 agreement, false-positive rates, and score uncertainty.

Auto IIS cross-route judgments and runtime verification. Auto IIS identifies interaction controls from screenshots, then checks the source code to assess whether handlers are bound, state is correctly updated and shared, and destination pages read that state to display the expected results. For example, it identified local state that was not shared across routes in snapgram-social, and a cart page that did not read shared data in shopco-ecommerce. We verified these judgments by running the apps, performing the interactions, and navigating in-app. The observed outcomes matched Auto’s judgments. Original explanations and runtime evidence: Snapgram: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#snapgram-cross-route Shopco: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#shopco-cross-route

Item-level runtime agreement and false-positive rates for S3. To quantify Auto IIS reliability for S3, we examine reference S3 interactions in the 45-app validation subset. To validate state propagation across routes captured by direct URL loading, human annotators triggered the interaction, navigated via in-app controls within the same browser session, and labeled the outcome observed on the destination page. Under the annotation rules, result and scope are left blank for items marked not applicable, but the corresponding reference S3 requirements remain in the scoring denominator and receive zero credit (Fig. 3, §3.5, Eq. 1). To measure agreement between the evaluators’ judgments of interaction outcomes, we calculate W/P/F agreement on items with outcome labels from both evaluators.

The score comparison covers all 119 evaluable reference S3 items, including those marked not applicable by at least one evaluator, and compares Auto and human zero/nonzero credit and scores item by item to capture scoring errors arising from applicability disagreements.

MeasureProportion95% Wilson CI
W/P/F outcome agreement93.2%85.9%–96.8%
False-positive rate for nonzero credit (human zero → Auto nonzero)1.8%0.5%–6.4%

Across these items, Cohen’s κ for nonzero versus zero credit is 0.83, and the mean absolute difference in per-item IIS scores is 3.6 points on a 100-point scale.

Auto IIS S3 scores and uncertainty. To quantify differences between Auto IIS and human runtime S3 scores, we compare both on the same 45-app validation subset. Brackets give 95% confidence intervals to quantify uncertainty in the S3 score estimates, obtained by resampling apps with reference S3 interactions within this subset. 0.0 denotes an observed zero on this subset.

ModelAuto IIS S3Human runtime S3
Claude Sonnet 4.616.0 [4, 30]19.0 [4, 35]
Kimi K2.58.0 [0, 18]5.0 [0, 14]
Qwen3.5-397B-A17B0.00.0
Gemini 3.1 Pro Preview0.00.0
GPT-5.44.0 [0, 10]5.0 [0, 14]
GLM-4.6V0.00.0

The Auto and human S3 point estimates differ by at most 3 points. Comparing scopes within the same subset, all six models also have lower human S3 point estimates than their S1 and S2 estimates. Thus, weak cross-route performance is also observed under human runtime evaluation on this subset.

Effect of evaluator error on full-benchmark S3 results. We perform a two-level bootstrap with 2,000 replicates. In each replicate, we resample from the 45 apps with human runtime annotations and from all 95 apps, and use the human-score distributions for the same interaction category and Auto label to estimate the effect of evaluator error on S3 scores. Claude’s estimated S3 score is 14.5 (95% interval: 6.4–25.4, original Auto score: 15.8). The upper endpoint of each of the other five models’ intervals is at most 9.3. Together with the human runtime scores, this analysis indicates that cross-route state remains a weak point for models on this benchmark.

[Q1] Independent RII annotation, agreement, and ambiguity handling

Two front-end engineers independently completed the RII for all 95 apps from the input screenshots using the same annotation rubric, without seeing each other’s labels. Before adjudication by a third expert, they agreed on applicability for 650 of the 665 app–category items (97.74%, Cohen’s κ = 0.945). Where both marked the category applicable, exact agreement on reference scope was 98.92%. The third expert then adjudicated disagreements using the same rubric and screenshot evidence.

Beyond reporting overall agreement, we use the three types of cases you highlighted to illustrate how the annotation rubric handles details that require interpretation from screenshots:

  • Form validation (C05, vue-admin-arco). The ambiguity in a multi-step form is whether each step validates independently (S1), or later steps depend on earlier validation state (S2). The visible required-field, name-length, and URL-format rules concern the current step and do not show cross-step validation dependencies. The RII therefore marks C05 applicable with reference scope S1. Screenshot evidence: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-arco

  • Notification (C06, react-duolingo). The pre-submission screenshot does not show feedback. CHECK could mean submit and continue or verify the answer. The separate SKIP control supports the latter interpretation, so the RII includes feedback on the submitted answer under C06 (S1): learners should be able to tell whether their answer is correct, without prescribing the feedback format. Screenshot evidence: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-lesson

  • Cross-route state (C07, sanity-stripe-store). A cart drawer on a single product page could represent either page-local state or a cart shared across product pages. The input screenshots also provide a cart entry on another product page, supporting a shared-cart interpretation. The RII therefore marks C07 applicable with reference scope S3: added products should be retained after navigating between product pages within the app. Input and runtime screenshots: https://anonymous.4open.science/w/artifact-B55C/reviewer-bdbz.html#ambiguous-cart

Thank you for your response. Your clarifications have addressed most of my concerns. After considering the response, I will maintain my score.

Thank you for taking the time to consider our responses and for your thoughtful feedback. We appreciate your acknowledgment that our clarifications have addressed most of your concerns. We are grateful for your careful review and constructive suggestions throughout the process.

Reviewer_LFVEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UI2App asks whether a model can look at static screenshots, figure out what features an application should have without being explicitly told, and build a runnable app with multiple pages. The dataset contains 95 applications and 600 screenshots.

The evaluation checks whether the app runs and how closely it matches the screenshots. It also uses IIS to check whether the model implements the interactions implied by the screenshots. The experiments show that making an interface look right does not mean its features work well. The authors also use human evaluation to check how reliable the automated scores are.

Critical strengths

I think this is a useful problem to study. The benchmark does not just check whether a model can build a single page. It also checks whether features work across different pages of the same app, which is closer to what is needed in actual development. The large gap between visual similarity and working interactions also shows why looking at appearance alone is not enough.

The dataset and evaluation pipeline provide a useful starting point for further research. The human validation gives me more confidence in the automated scores. I also agree with accepting implementations that work differently but provide the same functionality. With only a few screenshots, there is often no single clear answer to how a feature should work, so the evaluation should not accept only one implementation.

Critical weaknesses

The paper provides RII and the scoring rules, but I would still like to see a complete scoring example. For example, what features can be inferred from the screenshots, which ones did the model actually implement, and how was the final score assigned? This is especially useful when there are several reasonable ways to implement a feature. An example would make it easier to understand what counts as a complete implementation and how partial implementations are scored.

The human validation in Appendix I is helpful. However, automated IIS uses code and static screenshots, so I would also like to know where its judgments differ from those of someone actually using the app. The authors could give a few examples of disagreements to show where the automated evaluation might be less reliable.

Questions for authors
  1. Could you provide a complete scoring example? Please show what features are implied by the screenshots, what the model implements, and how the result is judged and scored. I am particularly interested in how you handle a different but reasonable implementation, and how you score cases where some functionality is missing.
  2. When automated and human scores disagree, what are the common reasons? I am especially interested in features such as keeping state correctly when switching pages. The reported agreement is helpful, but a few concrete examples would make it easier to understand what the automated evaluation can and cannot reliably judge.
  3. How much do these categories and levels tell us about interaction difficulty? For example, when a feature requires managing state across pages, is it harder to implement, or harder to infer from the screenshots? It would also help to briefly explain whether the interaction categories overlap.
6 comments

Thank you for recognizing the practical value of cross-page evaluation, the gap between visual similarity and working functionality, and the dataset and evaluation pipeline's usefulness for future research. We also appreciate your positive assessment of the human validation and of accepting functionally equivalent implementations.

[W1, Q1] From screenshot requirements and observed behavior to the final score

We use Claude Sonnet 4.6's generated shopco-ecommerce as a complete example, connecting screenshot-implied requirements, observed behavior, category labels, and the final score.

Requirements implied by the screenshots. The model received five screenshots, with no app-specific behavior description. To make the evaluation basis explicit, we annotated the visual cues on the webpage and linked them to the behaviors assessed below. For cross-route state, the product page's Add to Cart button, the header cart icons, and the cart page jointly imply that an added item should remain in the cart after navigation. These cues specify the required behavior without prescribing a state management implementation. Annotated inputs and behavior checks: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#example

How completeness is judged. Under Appendix B.3, W means all checked behaviors work fully, P means only some work or at least one works partly, and F means none work. Their scoring factors are 1, 0.5, and 0, respectively.

Category (reference scope)Screenshot cues and corresponding requirementsObserved behaviorVerified labelExample credit
C01 Toggle (S2)Color, size, quantity, thumbnails, and filter swatches should respond to selection or adjustment.All five behaviors work.W2
C02 Expand/collapse (S1)The banner's close control, filter groups, sort menu, and detail tabs should change the corresponding displayed content.All four behaviors work.W1
C03 List operations (S2)Search, category, price, color, and sorting controls should change the product list.Search, category and price filtering, and sorting work; color filtering does not change the list.P1
C04 Data CRUD (S2)Add to Cart should create a record containing the selected configuration.A cart record is created with the selected size and quantity.W2
C05 Form validation (S1)The newsletter email input and submission control call for checking invalid addresses.Invalid addresses are not rejected.F0
C06 Notification (S1)Newsletter submission should produce feedback.Submitting a valid address produces no feedback either.F0
C07 Cross-route state (S3)The addition should persist after navigation and appear in the header and cart page.The count persists after navigation, and the cart lists the added item.W3

How the score is calculated. Under Equation 1, S1, S2, and S3 have weights of 1, 2, and 3 in this example. Four of the five C03 operations work, so C03 receives P and contributes 0.5 × 2 = 1. The reference weights sum to 12:

QS = (2 + 1 + 1 + 2 + 0 + 0 + 3) / 12 = 0.75.

Using the verified labels in the table, the example scores 75.0/100. Model-level IIS is the mean QS across all applications multiplied by 100.

How missing functionality affects credit. The failed color filter gives C03 partial credit. Validation and notification do not work, so C05 and C06 receive zero. An entirely missing reference category also receives zero, while its reference weight remains in the denominator. W/P/F assigns credit according to functional completeness: full, partial, or none. Retaining the reference weight ensures that missing requirements remain reflected in the overall score.

Cross-route behavior on the same task illustrates complete, partial, and failed implementations: Claude updates both the header count and cart contents (C07: W); GPT-5.4 updates the count but leaves the cart empty (P); Gemini updates neither (F). Behavior comparison (also Appendix G.6, Fig. 9): https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#shopco-cross-route

How different but reasonable implementations receive credit. In maciekt07-todoapp, the screenshot requires a task name but does not prescribe how to block an empty submission. Claude disables Create Task when the name is empty, whereas Kimi K2.5 uses the browser's required check to block submission. Both prevent unnamed tasks, and both Auto and human evaluators assign W for C05. In fashion-ecommerce, Auto likewise credits both Claude's search on submission and GPT-5.4's search while typing as satisfying the search requirement. Credit therefore follows the required behavior while allowing different implementations. Implementation comparisons: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#reasonable

We hope these examples clarify how screenshot-implied requirements are judged and scored.

[W2, Q2] Where Auto IIS and runtime judgments differ

Thank you for recognizing the human validation in Appendix I. To further explain typical reasons for disagreement between automated and human scores, we present two concrete examples using the original scoring rationales, input screenshots, and runtime observations.

Adding a target-page display requirement (under-crediting). In Claude's react-duolingo, both Auto and human assessment credit working cross-route state. Auto additionally requires saved Name and Username fields to appear on Profile and assigns P because they do not; the final human assessment is W. The input screenshots provide Name and Username editing on Account, while Profile shows no corresponding fields. Auto treats this additional target-page display requirement as a condition for full credit, leading to a lower completion score for the implemented cross-route functionality. Assessment records and input screenshots: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#cross-route-completeness

Mistaking selection feedback for working functionality (over-crediting). In Claude's shadcn-admin, both Auto and human assessment credit working state toggles. Auto assigns W to C01 based on component state updates and UI feedback, while the final human assessment is P. Runtime interaction shows that selecting Dark and applying the setting changes the selection outline, but the interface remains light. This example shows that detecting a change in selection does not establish that the corresponding theme-switching functionality takes effect. Runtime recording and assessment records: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#theme-application

These cases mark the boundary of static assessment. Auto IIS reliably judges whether handlers are bound and whether state is shared and read across routes: on the 45-app validation subset, its C07 outcome agreement with human runtime assessment is 93.2%. The residual disagreements arise in narrow situations: confirming that a selected setting visibly takes effect (shadcn-admin), or reading an extra display requirement into the screenshots (react-duolingo). In the latter case, the error makes Auto stricter rather than more lenient. At the model level, these disagreements do not change the ranking: Auto and human IIS order all six models identically (Pearson 0.991).

[Q3] Category and scope, inference and implementation, and category overlap

Category and scope help identify different functional weaknesses. Category distinguishes the functionality to implement, while scope describes its state management range: S1 covers UI state, S2 data state, and S3 cross-route state. Together, they help locate the specific weaknesses behind overall IIS.

In Table 3, all six models score lower on S3 than on their own S1 and S2. Claude, for example, scores 39.9, 37.8, and 15.8, respectively, identifying cross-route functionality as a shared weakness in this end-to-end task. Category analysis further shows that functions with similar state scopes can differ in performance: expand/collapse (C02) and notification (C06) are both typically S1, yet Claude scores 59.5 and 11.0 on them, respectively (Fig. 4). Scope thus helps locate differences in state management range, while category adds differences in functionality.

Cross-route state requires a continuous process from requirement inference to functional implementation. UI2App evaluates whether a model can infer functional requirements from static screenshots and implement them in a runnable application. For cross-route state, the model must combine cues from different pages to identify data dependencies, organize shared state, and make an update on one page appear correctly on another. To examine the specific difficulties in this process, we analyzed the generated code of 282 runnable artifacts whose reference screenshots imply cross-route state, together with their C07 scores.

The analysis identifies two failure patterns. First, the required cross-route state was not realized as working shared state: 247 artifacts (87.6%) had no detected common mechanism for shared state and received no C07 credit. Second, detecting a sharing mechanism did not guarantee C07 credit: of the 33 artifacts with a detected mechanism, 15 (45.5%) still received no credit.

These results identify a major failure pattern: the cross-route requirements implied by the screenshots do not translate into working shared state. A sharing mechanism also does not guarantee correct final behavior. Completing these tasks requires the model to infer from cues across pages which data should be linked, and to make those links take effect during navigation. UI2App evaluates requirement inference and implementation as a continuous ability through whether the final application performs the screenshot-implied interactions.

Categories can involve the same action while checking different functional outcomes. In shopco-ecommerce, C04 checks whether clicking “Add to Cart” creates or updates a cart entry for the selected item. C07 checks whether the cart page displays the added item after in-app navigation. In Claude's generated app, the cart page displays the added item. GPT-5.4's generated app writes the item to shared cart state, and the cart badge updates to 1. After navigation, its cart page still shows an empty cart because it does not read the record (Fig. 9). Side-by-side runtime comparison: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#shopco-cross-route

One action may involve several functional requirements, and each category is judged by its corresponding outcome. Consequently, the scores reflect both implemented and missing functionality, so a successful local update does not mask missing cross-route functionality.

Reviewer_eLWUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UI2App, a benchmark for generating runnable multi-page web applications from screenshots without explicit behavioral specifications. It contains 600 screenshots across 95 applications and evaluates generated artifacts along three dimensions: executability (EXEC), visual fidelity (VFS), and interaction inference (IIS). Experiments on six frontier vision-language models reveal a substantial gap between visual reconstruction and functional interaction, particularly in cross-route state management.

Critical strengths
  1. Unlike existing screenshot-to-code benchmarks that primarily assess visual similarity, UI2App explicitly evaluates whether models can infer and implement interactions without behavioral guidance. This addresses an important limitation in evaluating real-world application generation.
  2. The benchmark combines executability, visual fidelity, and seven interaction categories across different state-management scopes. Evaluations of six frontier models, supported by human validation and repeated-generation experiments, reveal a substantial disconnect between visual and functional performance, providing useful insights into current model limitations.
Critical weaknesses
  1. The core IIS metric relies on an LLM evaluator inspecting generated code and screenshots rather than executing interactions. Although human validation is provided, the reported 81.8% outcome agreement indicates non-negligible evaluation errors. This raises concerns about the reliability of fine-grained interaction scores, especially for complex cross-route behaviors.
  2. Interactions are inherently underdetermined by static screenshots, making the Reference Interaction Inventory dependent on subjective assumptions about expected functionality. Furthermore, aggregating multiple behaviors into seven category-level outcomes may obscure important differences in implementation completeness. These issues potentially affect the interpretability and generalizability of IIS.
Questions for authors
  1. How do the authors ensure consistency when constructing the Reference Interaction Inventory from ambiguous screenshots? Could they report inter-annotator agreement for the reference annotations and analyze how alternative valid interaction interpretations affect IIS?
  2. Could the authors provide additional runtime-based validation of Auto IIS, particularly for cross-route state interactions, and quantify how evaluator errors affect fine-grained scores and model rankings?
4 comments

Thank you for your careful review and for recognizing the value of evaluating interaction inference without explicit behavioral specifications.

[W1, Q2] Runtime validation of Auto IIS and the effect of evaluator errors on scores and rankings

Human runtime validation and scoring errors. To quantify how evaluator errors affect fine-grained scores and model rankings, we recomputed IIS using labels obtained through human execution of interactions on the paper’s existing validation subset of 45 apps and 217 runnable artifacts, and compared these scores with the paper’s Auto IIS scores.

Across the 42 model–interaction-category scores, the mean absolute difference between Auto IIS and human rescoring was 3.2 points on a 0–100 scale. For cross-route state (C07), it was 1.2 points across the six models. For overall IIS, the mean absolute difference was 1.57 points, and the six models ranked in the same order (Kendall’s τ = 1.00).

Additional runtime-based validation. To further validate Auto IIS interaction scores and model rankings against runtime results, we added a browser-executed dynamic evaluation on the same 217 runnable artifacts. Both static and dynamic evaluation used GPT-6 Luna with medium reasoning effort, the same RII and IIS formula, and the same human reference labels. A browser agent clicked controls, entered text, and submitted forms in production builds of the generated apps. For C07, it performed an action, navigated to the destination page using in-app controls within the same browser session, and checked whether the state change was visible there.

Comparison with human runtime evaluationStatic GPT-6 Luna (paper)Dynamic GPT-6 Luna
Applicability agreement86.4%86.1%
Outcome agreement81.8%79.7%
Scope agreement91.1%91.3%
Model × category score MAE (42 scores)3.25.7
Mean absolute difference in C07 scores1.22.2
Overall IIS ranking Kendall's τ1.001.00
Mean duration (seconds/artifact)61.0486.8

On this validation subset, static and dynamic evaluation achieved outcome agreement with human judgments of 81.8% and 79.7%, respectively. Both reproduced the human ranking of the six models by overall IIS (Kendall’s τ = 1.00 in each case), and S3 remained the lowest-scoring scope for every model.

In this comparison, dynamic evaluation did not reduce model–category or C07 score errors relative to human evaluation, while taking approximately eight times as long on average (486.8 versus 61.0 seconds). Considering agreement with human judgments, fine-grained score errors, and evaluation cost together, we therefore adopt static Auto IIS as the main evaluation method and verify its reliability through human runtime evaluation.

[W2, Q1] RII annotation agreement, alternative interpretations, and category-level scoring

RII construction and annotation agreement. Two front-end experts independently constructed a Reference Interaction Inventory (RII) for each of the 95 apps. Following predefined annotation guidelines and using the input screenshots, each expert judged whether each interaction category was applicable and assigned a reference scope to each applicable category. A third expert adjudicated disagreements using the same guidelines.

Before adjudication, the two experts agreed on category applicability for 97.74% of the 665 app–category items (Cohen’s κ = 0.945). Agreement on reference scope was 98.92%.

Effect of alternative valid reference interpretations in the RII on IIS. The RII records category applicability and reference scope supported by the input screenshots; it does not prescribe a unique workflow or code implementation. Generated artifacts are evaluated by assigning result and scope based on functional effects, and different implementations satisfying the same interaction requirement can receive credit. For example, when the screenshots do not specify when a search should run, both live search and search on submission can receive credit.

If alternative valid reference interpretations change category applicability or reference scope, the categories included in scoring or their weights can change even for the same generated artifact. To isolate the effect of reference interpretation differences on IIS, we reused the existing functional evaluation results for the same generated artifacts and recomputed IIS using each expert’s independently annotated RII as the reference. Switching between the experts’ reference annotations changed overall IIS by at most 0.33 points across the six models on a 0–100 scale, and the ranking was unchanged.

Interpretability and cross-app applicability of category-level evaluation. IIS uses interaction categories as the units of scoring so that apps with different functionality and page organization can be compared along common functional dimensions. Apps share the same category definitions, functional-outcome criteria, and scope rules; the categories included in scoring and their reference scopes are determined by each app’s RII based on its input screenshots.

Category outcomes describe the state of functional implementation, while scope incorporates different state-management requirements, such as local state updates and cross-page state propagation, into scoring. Within each category, IIS evaluates the sub-features present in the generated interface: Working when all work completely, Partial when some functionality is present but at least one sub-feature fails or works incompletely, and Failed when none work.

Implementations labeled Partial can differ in completeness. To show how observed behavior is summarized into category outcomes and converted into a score, we provide a visual worked example on an anonymous webpage (https://anonymous.4open.science/w/artifact-B55C/reviewer-elwu.html#subinteractions). The example shows which sub-interactions work, which do not, and how these results are aggregated into category outcomes and incorporated into the score.

Effect of within-category completeness differences on model comparisons. To test whether completeness differences not distinguished by category labels could change model comparisons, we held the RII, category applicability, scope, and Working/Failed labels fixed across all 95 apps. We used failure and full implementation as the lower and upper scoring bounds for Partial, respectively, to calculate each model’s possible IIS range.

Claude’s lower bound (28.4) exceeded every other model’s upper bound. The lower bounds for Kimi (14.2) and Qwen3.5 (11.6) both exceeded the upper bounds for Gemini (10.4) and GPT-5.4 (7.9). Thus, even when each Partial receives its own score within these bounds according to its completeness, Claude remains first, and Kimi and Qwen3.5 each remain ahead of both Gemini and GPT-5.4.

We hope this addresses your questions and would be happy to clarify any remaining points.

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Submission

Dear Reviewer,

Thank you again for your thoughtful review. In our responses, we have provided additional analyses on RII annotation consistency, the impact of alternative reference interpretations, and the reliability of Auto IIS, including human and browser-based runtime validation. We would appreciate your assessment of whether these additional results address your main concerns, and whether any substantive concerns remain. We would be happy to provide further clarification or discuss any remaining questions.

Best regards,

The Authors

Thank you again for your review. We have uploaded a revised PDF that incorporates the additional analyses discussed in our responses. All changes are in the appendix and marked in blue.

  • Appendix B.5: RII inter-annotator agreement, and the effect of using either expert's RII as the reference (at most 0.33 IIS points; ranking unchanged).
  • Appendix B.6: lower and upper bounds for Partial credit; the model ordering holds within these bounds.
  • Appendix I.1: category-level differences between Auto and human runtime IIS, as reported in our response.
  • Appendix I.2: runtime validation of cross-route state, which you specifically asked about, with Auto and human runtime S3 scores for all six models.

We hope these additions address your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

We thank all reviewers for their constructive engagement. Below we summarize the discussion and our responses.

1. Strengths noted by reviewers

Evaluating interaction inference without behavioral guidance "addresses an important limitation in evaluating real-world application generation" (eLWU). Reviewer bDbz called the visual–interaction ranking gap "consequential": "a strong visual score can conceal largely nonfunctional output." Cross-page evaluation is "closer to what is needed in actual development", the dataset and pipeline are "a useful starting point for further research" (LFVE), the implementation-agnostic rubric was endorsed (LFVE, bDbz), and bDbz noted the "substantial validation of the central empirical finding".

2. Shared concern: is IIS reliable enough for the headline findings?

Our headline findings are F1: the visual-fidelity leader ranks only 4th on IIS, trailing the IIS leader by 4.5×; and F2: cross-route state is the weakest scope for every model. The main methodological concerns bear on whether IIS measures these reliably:

  1. Is static Auto IIS accurate, especially for cross-route state? (eLWU, LFVE, bDbz) Against human runtime judgments (45 apps, 217 artifacts), overall IIS differs by 1.57 points on average, model×category MAE is 3.2 (C07: 1.2), and the six-model ranking is identical (Kendall τ = 1.00). On the 119 evaluable cross-route (S3) items, verified by navigating in-app within one session, the false-positive rate for nonzero credit is 1.8%, and outcome agreement on items labeled by both evaluators is 93.2% (95% CI 85.9–96.8). Propagating evaluator error to the full benchmark gives Claude an S3 of 14.5 [6.4, 25.4] and every other model an upper bound ≤ 9.3. A browser-executed variant of the evaluator did not improve agreement and was ~8× slower, so we keep static evaluation.
  2. Is the reference inventory (RII) subjective? (eLWU, bDbz) Two experts annotated all 95 RIIs independently: applicability agreement 97.74% (κ = 0.945), scope 98.92%. Recomputing IIS with either expert's RII changes overall IIS by ≤ 0.33 points; the ranking is unchanged.
  3. Does category-level aggregation hide completeness differences? (eLWU) Scoring every Partial at 0 and at full credit, Claude's lower bound (28.4) exceeds every other model's upper bound, and the lower bounds of Kimi (14.2) and Qwen3.5 (11.6) exceed the upper bounds of Gemini (10.4) and GPT-5.4 (7.9).

Net effect. The F1 ranking mismatch persists under human runtime scoring on the validation subset, under either expert's RII, and for any credit assigned to Partial outcomes with all other labels fixed. For F2, human runtime scores place S3 below S1 and S2 for every model on the validation subset, and the evaluator-error analysis supports weak S3 performance on the full benchmark.

After the rebuttal, Reviewer LFVE wrote "I have no more questions and will keep my positive score," and Reviewer bDbz wrote "Your clarifications have addressed most of my concerns."

3. Further clarifications (LFVE)

  • End-to-end scoring example: screenshot cues → observed behavior → W/P/F labels → score (75.0/100). Alternative valid implementations receive the same credit (e.g., disabled button vs. browser required check; live search vs. search on submit). Visual example: https://anonymous.4open.science/w/artifact-B55C/reviewer-lfve.html#example
  • Where Auto and humans disagree: two boundary cases, over-crediting a selected but unapplied setting and under-crediting by reading an extra display requirement into screenshots (i.e., stricter, not lenient).
  • What scopes reveal: of 282 artifacts whose screenshots imply cross-route state, 87.6% had no detected shared-state mechanism, and 45.5% of those with one still failed. The bottleneck spans both inferring from cues across pages which data should be shared and making that sharing take effect during navigation.

4. Paper revision

We have uploaded a revised PDF (changes in blue). It adds RII agreement and annotator-swap sensitivity (App. B.5), Partial-credit bounds (App. B.6), the category-level error of Auto IIS (App. I.1), and runtime verification of S3 judgments (App. I.2).

We thank the reviewers again for their feedback, which has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.