- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies dynamically generated interfaces for database-backed service tasks. GenUI-Harness separates information retrieval, interface generation, and action execution. Dynamic UX supports parallel rendering, while Reward Auditor revises training rewards using task-execution outcomes. The authors introduce UI-TAU Bench with ten domains and evaluate on its Lite and Full splits. On Lite, the system improves macro-averaged Pass@3 over smolagents by 4.48 percentage points. Role-specific training and reinforcement learning bring the 4B model to 58.00% Pass@3.
Critical strengths
- The paper connects generated interfaces to complete service workflows and checks task completion through the final database state, going beyond interface appearance or executable code. The problem has practical value, and the evaluation target matches the intended use.
- Table 3 fixes upstream retrieval outputs and the downstream model, replacing only interface generation, and improves task performance for all five models. This controlled intervention supports the trained GUI-Coder's cross-model value and is an important reason to support acceptance.
Critical weaknesses
- The independent advantage of generated interfaces over text remains unclear. The main comparison changes both the communication channel and agent organization, while the comparison between the trained 4B system and zero-shot models also includes training differences. This limits attribution of system-level gains to the interface as a communication medium.
- The main simulator can inspect source code, and the human study selects tasks with consistently usable interfaces from all four source models. The evidence therefore more directly supports interaction efficiency on usable interfaces than benefits for real users over the full task distribution.
Questions for authors
- With retrieval and execution fixed and comparable training budgets, how does task performance change when switching only between text and UI interaction?
- With user goals unchanged, does the advantage over text baselines persist when the simulator cannot access interface source code?
7 comments
Author Reply 1/3
W1: The independent advantage of generated interfaces over text remains unclear. The main comparison changes both the communication channel and agent organization, while the comparison between the trained 4B system and zero-shot models also includes training differences. This limits attribution of system-level gains to the interface as a communication medium.
A: Thank you for raising this question. We are happy to have this opportunity to further clarify the motivation of our work.
In the tasks we study, the user has to compare several retrieved options and state several preferences (Figure 1), which is laborious in plain text. A visual layout can group the information that is used together, which reduces the search and inference the user has to do [1], and direct manipulation lets the user act on visible objects instead of describing the action in words [2, 3]. User studies support this. In an experimental study, a chatbot interface led to higher cognitive load and lower user satisfaction than a menu-based one [4]. With a direct-manipulation interface to an LLM, participants were 50% faster and needed 50% fewer prompts than with ChatGPT [5]. Human raters also prefer generated interfaces to text responses [6, 7]. Our own survey, which was part of the submission, compares the two channels with real people (Section 6.3, Appendix F). Six participants completed 100 tasks with consistently usable interfaces, with the same task information presented either as a generated UI or as text, and the UI reduced the average number of rounds from 3.4 to 1.2.
However, this advantage has rarely been evaluated at scale. The existing evidence comes from user studies and preference ratings, which are costly to repeat and cannot be used as a training signal. Few evaluations check objectively whether a generated interface lets the user complete the task. This is why we brought the problem into an agent setting and formulated it as a verifiable problem: in UI-Tau Bench, the final database state of each task is checked by exact match, on 300 Lite and 1,000 Full tasks from 10 domains. A verifiable outcome also allows us to train the interface generator, which is what we do for the 4B model with Dynamic UX and Reward Auditor.
As noted in Section 6.2, Table 2 compares complete systems. To separate the channel from agent organization and training, we ran two additional comparisons (see our answer to Q1). The UI is higher by 3.41 Pass@3 points on average without training and by 4.33 points with the same training on both sides, and it stays ahead when the user simulator cannot read the interface code.
[1] Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65-100.
[2] Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. Computer, 16(8), 57-69.
[3] Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct manipulation interfaces. Human-Computer Interaction, 1(4), 311-338.
[4] Nguyen, Q. N., Sidorova, A., & Torres, R. (2022). User interactions with chatbot interfaces vs. menu-based interfaces: An empirical study. Computers in Human Behavior, 128, Article 107093.
[5] Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2024). DirectGPT: A direct manipulation interface to interact with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery.
[6] Chen, J., Zhang, Y., Zhang, Y., Shao, Y., & Yang, D. (2026). Generative interfaces for language models. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 1499-1519). Association for Computational Linguistics.
[7] Leviathan, Y., Valevski, D., Kalman, M., Lumen, D., Segalis, E., Molad, E., Pasternak, S., Natchu, V., Nygaard, V., Venkatachary, S., Manyika, J., & Matias, Y. (2026). Generative UI: LLMs are effective UI generators. arXiv:2604.09577.
Author Reply 2/3
Q1: With retrieval and execution fixed and comparable training budgets, how does task performance change when switching only between text and UI interaction?
A: We ran two comparisons in which only the interaction channel changes, one without training and one with the same training on both sides.
In the first, retrieval and execution are fixed and no model is trained. Inside GenUI-Harness, each text run reuses the stored pre-exploration output of the corresponding UI run, and the same model performs post-execution. The only difference is the middle step: in place of an interface, the model sends the user a single text message, and the user simulator, the same one that the text baselines in Table 2 interact with, answers once in natural language. In the second, the training budget is matched. We trained a text agent from the same base model as the GUI Coder (Qwen3.5-4B), with the same SFT recipe and on the same tasks, after rewriting each reference interface into a chat message with the same content. Both agents are paired with the same fixed SFT Tool Agent, which performs retrieval and execution.
Each cell gives Pass@3 / Avg@3 (%) on the Lite split.
| Setting | Model | UI | Text | Difference |
|---|---|---|---|---|
| No training | DeepSeek V4 Flash | 49.67 / 30.22 | 47.00 / 27.67 | +2.67 / +2.55 |
| No training | Qwen3.6-35B-A3B | 47.33 / 26.89 | 43.67 / 24.22 | +3.66 / +2.67 |
| No training | GPT-5.4 | 40.00 / 25.22 | 36.00 / 22.11 | +4.00 / +3.11 |
| No training | Claude Sonnet 4.6 | 51.33 / 35.67 | 48.00 / 32.33 | +3.33 / +3.34 |
| No training | Average | 47.08 / 29.50 | 43.67 / 26.58 | +3.41 / +2.92 |
| Same SFT on both sides | Qwen3.5-4B | 47.33 / 27.67 | 43.00 / 24.33 | +4.33 / +3.34 |
In both comparisons the UI is higher. Without training it is higher for all four models, by 3.41 Pass@3 points on average, and with matched training it is higher by 4.33 Pass@3 points. The agent organization is the same on both sides in each comparison, and in the second the training is the same as well, so these differences are the part that comes with the interface as a communication medium. For the same four models, smolagents reaches 44.17 Pass@3 on average in Table 2 and our text version reaches 43.67, so changing the agent organization alone leaves the result about the same. We make the trained comparison at the SFT stage, where both sides receive the same training. RL then trains only the GUI Coder, with the Tool Agent fixed, and raises Pass@3 from 47.33 to 58.00. This gain comes from optimizing the interface generator and is not part of the channel comparison. When the user simulator cannot read the interface code (see our answer to Q2), the margins are smaller and the UI stays ahead in both comparisons: it is higher for each of the four untrained models (45.25 against 43.67 Pass@3 on average), and the SFT GUI Coder scores 44.67 / 25.00 against 43.00 / 24.33 for the text agent. The UI also needs less input: in the first comparison the user simulator writes 48 words per task on average with text, while with the UI it enters 8 words and makes 3 selections. We have added both comparisons to the appendix of the revised paper (Appendix I.1 and I.3, Tables 17 and 20).
General Response
We sincerely thank you for the precise review and for appreciating the end-to-end evaluation through the final database state, the practical value of the problem, and the controlled GUI Coder intervention in Table 3. We address each of your comments and questions point by point below.