- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VISTA, a benchmark for coding agents to build Web and Android apps from Figma designs and text requirements. There are human annotations for individual components, and it is evaluate whether generated apps have components in the correct location and have the intended functionalities.
Critical strengths
-
Evaluating whether generated apps match the intended design and work correctly is important and needed as more apps become LLM-generated.
-
The human component annotations are a good resource for evaluation.
-
The benchmark covers both Web and Android.
Critical weaknesses
-
The main concern is whether the benchmark can accurately evaluate functionality. The component evaluation checks whether individual components accept input, change state, or navigate. As mentioned in the strengths section, these are useful component tests. However, what is perhaps equally as important are user journey tests for judging whether an app supports the functionality that it was designed for. These could span many components and many different screens to see that all states and data is correctly passed to and from each other. It is unclear whether the current evaluation can accurately checks whether user journeys are correctly implemented.
-
Similarly, the human annotation and validation (which again are useful) also does not cover these important user journeys and whether the generated apps correctly support these journies.
Questions for authors
-
Can we safely assume that if components work as they should, multi-component & multi-screen user journeys would also work as expected?
-
How are backend updates, persistence, and consistency across pages verified?
4 comments
I thank the authors for adding the user journey tests. These are good initial analysis that makes the benchmark more complete. I do strongly believe that these more complete sequences of tests are needed in the age of LLM agents. I encourage the authors to continue the effort to cover both web and mobile apps come completely.
However, I think it would be useful to compare the component and journey results on the same apps. For example, how often did all the components in a journey pass their individual tests while the journey itself failed? These sort of analyses make the benchmark much more useful. Also since the new tests only cover web apps, I am unsure whether the same checks can be done for mobile.
Thank you for this suggestion. We have added a paired component--journey analysis on the same generated applications and extended the journey tests to Android. The tests compose existing annotated interactions into workflows and check their resulting cross-screen and business state. Overall, 42/72 Web journeys and 5/36 Android journeys passed.
Android scope and results
The Android suite covers four systems and all eight task categories, with 32 generated applications, nine journey definitions, and 36 execution instances. Five journeys passed and 31 failed.
| Category | Workflows |
|---|---|
| Medical | Edit profile; save and reopen |
| Fitness | Edit profile; save and read across views |
| Furniture | Increase cart quantity; navigate to profile and back |
| Food delivery | Increase cart quantity; visit checkout and return |
| Recipes | Edit profile; save and reopen |
| Finance | Create expense; read transaction list |
| Chat | Send message; leave and reopen conversation |
| Ecommerce | Increase cart quantity; remove item; cross-page readback |
| System | Instances | Pass | Fail |
|---|---|---|---|
| GPT-6-astra | 9 | 5 | 4 |
| Fable-5 | 9 | 0 | 9 |
| Grok-4.5 | 9 | 0 | 9 |
| Opus-4.8 | 9 | 0 | 9 |
| Total | 36 | 5 | 31 |
For each journey, we collect the component scores for its bound annotated controls and ask whether the journey succeeds when every recorded component receives full behavior credit (). On Web, three of the six such journeys failed; on Android, two of six failed. The complete cross-tabulation is reported below.
| Platform | Journey instances | All bound components : journey passes | All bound components : journey fails |
|---|---|---|---|
| Web | 72 | 3 | 3 |
| Android | 36 | 4 | 2 |
For example, in an Android cart-removal workflow, the delete and navigation components each received full credit in isolation. Independent SQL confirmed that deletion reduced the stored cart from two items to one, but navigating to the profile and back displayed two cart rows. In a Web profile-editing workflow, all bound components received full credit, yet the saved name disappeared when settings were reopened. These cases show why checking individual responses does not establish cross-screen state consistency, and identify failures that the added journey evaluation captures.
We will add the paired tables and representative failure traces to the appendix and include the Android workflow protocol and evidence with the benchmark materials. We will continue expanding the journey suite to cover additional workflows on both Web and Android.
We thank the reviewer for these constructive suggestions, which have helped make the paper clearer and the evaluation more rigorous.
We have further expanded the journey evaluation with 16 additional workflow definitions and 64 additional executions across Web and Android. The updated app.J reports all 172 journey executions, including the paired component–journey analysis and representative failure cases. We appreciate the reviewer’s thoughtful comments and constructive discussion, which have helped strengthen the paper.
We thank the reviewer for recognizing the importance of evaluating design fidelity and functionality, the value of the component annotations, and the coverage of both Web and Android. Your suggestion to evaluate complete user journeys motivated additional workflow tests, which we describe below.
1. Can we safely assume that working components imply successful multi-component and multi-screen user journeys?
User journeys build on sequences of component interactions, making the component annotations a foundation for workflow-level evaluation. To evaluate how these interactions compose across screens, we added journey tests that execute annotated components in sequence and verify the resulting state and data consistency.
Specifically, we added 18 user-journey definitions across nine task categories, evaluated on four systems using 36 original generated Web applications and 72 workflow executions. These tests check whether components work together to complete the selected tasks.
The overall passing fraction was 58.3%, with an application-clustered 95% confidence interval of [44.4%, 72.2%]. These tests directly assess workflow completion alongside the component-level measurements. We will include the journey definitions, verification protocol, and results in the updated appendix.
2. How are backend updates, persistence, and consistency across pages verified?
The added journey tests verify outcomes at multiple levels:
These journey results are reported separately from the component-level score S. The component annotations support both levels of evaluation: individual probes assess each interaction, while the journey tests assess their composition across screens and verify the associated backend state and persistence.