acceptodds
Under review as a conference paper at ICLR 2027

When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction

Abstract

Most human-agent interaction today remains text-based. However, natural language is often not a perfect medium for complex tasks, as it can lead to cognitive overload, ambiguity, information chaos, and a slow input process; ephemeral generative UIs can present structured information and guide users toward task completion. In this work, we propose **GenUI-Harness**, a multi-agent harness that pairs a Tool Agent for information retrieval and task execution with a GUI Coder Agent that identifies ambiguities and generates front-end code for structured interfaces, helping users complete their tasks. Training such a versatile coder with reinforcement learning is challenging: verifiable rewards for interactive UI generation require costly execution, while LLM-as-a-Judge rewards are prone to reward hacking. We address the first challenge with **Dynamic UX**, a lightweight package that supports dynamic interaction and reward collection within a single sandbox, and the second with **Reward Auditor**, a meta-reward mechanism that monitors reward distributions and automatically distills diagnostic patterns into a shared rubric and scoring specification. To evaluate this task, we introduce **UI-TAU Bench**, a benchmark for active human-agent interaction through generated UI code, built on **10** real-world domain databases constructed from public data sources and based on Tau-Bench tool use settings, with Lite (**300 tasks**) and Full (**1,000 tasks**) splits. Experiments show that GenUI-Harness achieves an average Pass@3 gain of **4.48 percentage points** over smolagents on Lite. Training with GenUI-Harness improves a 4B backbone from a **9.33%** baseline to **58.00%** Pass@3, even outperforming larger frontier models such as Claude Opus 5 (**46.67%**). We further show that GenUI-Harness remains robust not only on ambiguous queries, but also on non-ambiguous ones. In a reviewer survey comparing communication channels, generated UIs reduce the average number of dialogue rounds from **3.4** to **1.2**. These results show that data-aware generative interfaces can support effective task completion and reduce dialogue rounds in the evaluated database-backed workflows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agents, Generative User Interfaces

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uyRuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamically generated interfaces for database-backed service tasks. GenUI-Harness separates information retrieval, interface generation, and action execution. Dynamic UX supports parallel rendering, while Reward Auditor revises training rewards using task-execution outcomes. The authors introduce UI-TAU Bench with ten domains and evaluate on its Lite and Full splits. On Lite, the system improves macro-averaged Pass@3 over smolagents by 4.48 percentage points. Role-specific training and reinforcement learning bring the 4B model to 58.00% Pass@3.

Critical strengths
  1. The paper connects generated interfaces to complete service workflows and checks task completion through the final database state, going beyond interface appearance or executable code. The problem has practical value, and the evaluation target matches the intended use.
  2. Table 3 fixes upstream retrieval outputs and the downstream model, replacing only interface generation, and improves task performance for all five models. This controlled intervention supports the trained GUI-Coder's cross-model value and is an important reason to support acceptance.
Critical weaknesses
  1. The independent advantage of generated interfaces over text remains unclear. The main comparison changes both the communication channel and agent organization, while the comparison between the trained 4B system and zero-shot models also includes training differences. This limits attribution of system-level gains to the interface as a communication medium.
  2. The main simulator can inspect source code, and the human study selects tasks with consistently usable interfaces from all four source models. The evidence therefore more directly supports interaction efficiency on usable interfaces than benefits for real users over the full task distribution.
Questions for authors
  1. With retrieval and execution fixed and comparable training budgets, how does task performance change when switching only between text and UI interaction?
  2. With user goals unchanged, does the advantage over text baselines persist when the simulator cannot access interface source code?
7 comments
AuthorsOpenReview ↗

General Response

We sincerely thank you for the precise review and for appreciating the end-to-end evaluation through the final database state, the practical value of the problem, and the controlled GUI Coder intervention in Table 3. We address each of your comments and questions point by point below.

AuthorsOpenReview ↗

Author Reply 1/3

W1: The independent advantage of generated interfaces over text remains unclear. The main comparison changes both the communication channel and agent organization, while the comparison between the trained 4B system and zero-shot models also includes training differences. This limits attribution of system-level gains to the interface as a communication medium.

A: Thank you for raising this question. We are happy to have this opportunity to further clarify the motivation of our work.

In the tasks we study, the user has to compare several retrieved options and state several preferences (Figure 1), which is laborious in plain text. A visual layout can group the information that is used together, which reduces the search and inference the user has to do [1], and direct manipulation lets the user act on visible objects instead of describing the action in words [2, 3]. User studies support this. In an experimental study, a chatbot interface led to higher cognitive load and lower user satisfaction than a menu-based one [4]. With a direct-manipulation interface to an LLM, participants were 50% faster and needed 50% fewer prompts than with ChatGPT [5]. Human raters also prefer generated interfaces to text responses [6, 7]. Our own survey, which was part of the submission, compares the two channels with real people (Section 6.3, Appendix F). Six participants completed 100 tasks with consistently usable interfaces, with the same task information presented either as a generated UI or as text, and the UI reduced the average number of rounds from 3.4 to 1.2.

However, this advantage has rarely been evaluated at scale. The existing evidence comes from user studies and preference ratings, which are costly to repeat and cannot be used as a training signal. Few evaluations check objectively whether a generated interface lets the user complete the task. This is why we brought the problem into an agent setting and formulated it as a verifiable problem: in UI-Tau Bench, the final database state of each task is checked by exact match, on 300 Lite and 1,000 Full tasks from 10 domains. A verifiable outcome also allows us to train the interface generator, which is what we do for the 4B model with Dynamic UX and Reward Auditor.

As noted in Section 6.2, Table 2 compares complete systems. To separate the channel from agent organization and training, we ran two additional comparisons (see our answer to Q1). The UI is higher by 3.41 Pass@3 points on average without training and by 4.33 points with the same training on both sides, and it stays ahead when the user simulator cannot read the interface code.

[1] Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65-100.

[2] Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. Computer, 16(8), 57-69.

[3] Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct manipulation interfaces. Human-Computer Interaction, 1(4), 311-338.

[4] Nguyen, Q. N., Sidorova, A., & Torres, R. (2022). User interactions with chatbot interfaces vs. menu-based interfaces: An empirical study. Computers in Human Behavior, 128, Article 107093.

[5] Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2024). DirectGPT: A direct manipulation interface to interact with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (pp. 1-16). Association for Computing Machinery.

[6] Chen, J., Zhang, Y., Zhang, Y., Shao, Y., & Yang, D. (2026). Generative interfaces for language models. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 1499-1519). Association for Computational Linguistics.

[7] Leviathan, Y., Valevski, D., Kalman, M., Lumen, D., Segalis, E., Molad, E., Pasternak, S., Natchu, V., Nygaard, V., Venkatachary, S., Manyika, J., & Matias, Y. (2026). Generative UI: LLMs are effective UI generators. arXiv:2604.09577.

AuthorsOpenReview ↗

Author Reply 2/3

Q1: With retrieval and execution fixed and comparable training budgets, how does task performance change when switching only between text and UI interaction?

A: We ran two comparisons in which only the interaction channel changes, one without training and one with the same training on both sides.

In the first, retrieval and execution are fixed and no model is trained. Inside GenUI-Harness, each text run reuses the stored pre-exploration output of the corresponding UI run, and the same model performs post-execution. The only difference is the middle step: in place of an interface, the model sends the user a single text message, and the user simulator, the same one that the text baselines in Table 2 interact with, answers once in natural language. In the second, the training budget is matched. We trained a text agent from the same base model as the GUI Coder (Qwen3.5-4B), with the same SFT recipe and on the same tasks, after rewriting each reference interface into a chat message with the same content. Both agents are paired with the same fixed SFT Tool Agent, which performs retrieval and execution.

Each cell gives Pass@3 / Avg@3 (%) on the Lite split.

SettingModelUITextDifference
No trainingDeepSeek V4 Flash49.67 / 30.2247.00 / 27.67+2.67 / +2.55
No trainingQwen3.6-35B-A3B47.33 / 26.8943.67 / 24.22+3.66 / +2.67
No trainingGPT-5.440.00 / 25.2236.00 / 22.11+4.00 / +3.11
No trainingClaude Sonnet 4.651.33 / 35.6748.00 / 32.33+3.33 / +3.34
No trainingAverage47.08 / 29.5043.67 / 26.58+3.41 / +2.92
Same SFT on both sidesQwen3.5-4B47.33 / 27.6743.00 / 24.33+4.33 / +3.34

In both comparisons the UI is higher. Without training it is higher for all four models, by 3.41 Pass@3 points on average, and with matched training it is higher by 4.33 Pass@3 points. The agent organization is the same on both sides in each comparison, and in the second the training is the same as well, so these differences are the part that comes with the interface as a communication medium. For the same four models, smolagents reaches 44.17 Pass@3 on average in Table 2 and our text version reaches 43.67, so changing the agent organization alone leaves the result about the same. We make the trained comparison at the SFT stage, where both sides receive the same training. RL then trains only the GUI Coder, with the Tool Agent fixed, and raises Pass@3 from 47.33 to 58.00. This gain comes from optimizing the interface generator and is not part of the channel comparison. When the user simulator cannot read the interface code (see our answer to Q2), the margins are smaller and the UI stays ahead in both comparisons: it is higher for each of the four untrained models (45.25 against 43.67 Pass@3 on average), and the SFT GUI Coder scores 44.67 / 25.00 against 43.00 / 24.33 for the text agent. The UI also needs less input: in the first comparison the user simulator writes 48 words per task on average with text, while with the UI it enters 8 words and makes 3 selections. We have added both comparisons to the appendix of the revised paper (Appendix I.1 and I.3, Tables 17 and 20).

Reviewer_w3w7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

this paper is a very in-time submission regarding openai's https://openai.com/index/gpt-6-for-everyone/ also https://majiaju.io/ambigchat.html and https://www.youtube.com/watch?v=UkH_ZOLf-IY

the authors contributed GenUI-Harness w/ Tool agent and GUI coder agent; to train the GUI Coder w/ GRPO, they built dynamic UX and reward auditor

UI-TaoBench was designed as a 10-domain stateful benchmark modeled after tao-bench

overall, the direction is great, with well-motivated problem and modular split, strong table 3

using actual task execution outcomes to diagnose where an online llm judge is getting tricked by fancy but non-functional ui is useful

it is hard to believe that over half of failures still come from broken 2d ui interfactions like unclickable controls...

Critical strengths

new benchmark on genui!

reward auditing & the overall bench is useful

code-stage intervention improves performance.

Critical weaknesses
  1. raw tsx code was leaked into the simulator prompt, allwing the simulator to potentially trick by feeding variable names into the ui form
  2. text baselines would better be fine-tuned on the text trajectories of the 40 databases.
Questions for authors
  1. can we double check if there are any leakage of ground-truth to user simulator? giving the simulator raw tsx_code may let it bypass visual or label defects by reading the variable names from the source...

can you run an ablation on UI-Tao bench lite where the user simulator is given only the user persona and goal and only the rendered dom without raw tsx code?

how much do pass@3 change?

  1. can we double check how much gain was from UI modality or from tool-calling trajectories on in-distribution schemas? the GenUI-harness gain was lower than all three text-only baselines where non of them were fine-tuned on text trajectories from the 40 databases as baseline

  2. minor anonymity issue in the code --- i won't call it out as it was unintentional.

  3. there are EvoGenUI-Bench and Generative Interfaces for Language Models that should be mentioned in related work instead of appendix ;)

5 comments
AuthorsOpenReview ↗

General Response

Thanks for your thoughtful review and for highlighting the timely direction, the well-motivated problem and modular split, the usefulness of outcome-based reward auditing and of the benchmark, and the strong code-stage results in Table 3. Below, we respond to each of your concerns and questions in turn.

AuthorsOpenReview ↗

Author Reply 1/2

W1: raw tsx code was leaked into the simulator prompt, allwing the simulator to potentially trick by feeding variable names into the ui form

Q1: can we double check if there are any leakage of ground-truth to user simulator? giving the simulator raw tsx_code may let it bypass visual or label defects by reading the variable names from the source... can you run an ablation on UI-Tao bench lite where the user simulator is given only the user persona and goal and only the rendered dom without raw tsx code? how much do pass@3 change?

A: Thank you for this concrete suggestion. Before presenting the ablation, we would like to clarify what the user simulator has access to.

The user simulator holds the user's goal by design, as in existing interactive benchmarks. In Tau-Bench [1], whose tool-use setting our benchmark follows, the user simulator receives a hidden instruction with the user's identity, intent, and preferences, written so that only one database outcome is possible. ColBench [2] gives its human simulator the ground-truth artifact, and BIRD-Interact [3] gives its user simulator the reference SQL. This lets the user simulator play a real user, who has a clear goal and expects a definite outcome. If the language model had to make up the goal, it would bring in its own preferences and they would differ between runs, so the result could not be checked against a fixed target. In our benchmark the goal is the reference action, and the evaluation checks whether the system reaches it through the interface it generates. It is a reference for what the user wants and not a hint to the system under evaluation. It is never given to the Tool Agent or the GUI Coder Agent, so the interface code is written without knowledge of it, and the user simulator holds it in the same way when it converses with the text baselines. The user simulator cannot call tools or modify the database, so it can act only through the controls of the rendered page (Appendix A.2). When these controls cannot express the required value, or when a control's accessible name or label binding does not support the interaction, the run fails although the user simulator knows the goal (Appendix E).

The source code can still expose internal details such as variable names, as you point out. To measure this effect, we ran the ablation as you describe on UI-Tau Bench Lite. We removed the UI code block from the user simulator prompt and left everything else in place, so the user simulator works only from the rendered page, given as its accessibility (ARIA) snapshot, and from the same goal as before (the reference action). The snapshot contains the accessible roles, names, and structure of the rendered interface (Appendix D.1) and no variable names. To isolate the effect of the code, we reused the stored pre-exploration outputs and generated interfaces from the runs in the paper and reran only the user simulator, the post-agent, and the evaluator.

ModelWith code (Pass@3 / Avg@3)Without code (Pass@3 / Avg@3)Change
DeepSeek V4 Flash49.67 / 30.2247.67 / 28.11−2.00 / −2.11
Qwen3.6-35B-A3B47.33 / 26.8945.00 / 24.44−2.33 / −2.45
GPT-5.440.00 / 25.2238.67 / 23.89−1.33 / −1.33
Claude Sonnet 4.651.33 / 35.6749.67 / 33.33−1.66 / −2.34
GenUI-4B58.00 / 36.0055.33 / 33.78−2.67 / −2.22
Average49.27 / 30.8047.27 / 28.71−2.00 / −2.09

Removing the code lowers Pass@3 by 1.33 to 2.67 points (2.00 on average) and Avg@3 by 2.09 on average. The decrease is similar for all five systems, and their order is the same in both settings on both metrics, with GenUI-4B remaining the highest. This indicates that access to the source code changed the outcome in only a small share of cases, and that the ranking of the systems does not depend on it.

We have added these results to the revised paper (Appendix I.2, Table 18).

[1] Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2025). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. In The Thirteenth International Conference on Learning Representations.

[2] Zhou, Y., Jiang, S., Tian, Y., Weston, J., Levine, S., Sukhbaatar, S., & Li, X. (2025). SWEET-RL: Training multi-turn LLM agents on collaborative reasoning tasks. arXiv:2503.15478.

[3] Huo, N., Xu, X., Li, J., Jacobsson, P., Lin, S., Qin, B., Hui, B., Li, X., Qu, G., Si, S., Han, L., Alexander, E., Zhu, X., Qin, R., Yu, R., Jin, Y., Zhou, F., Zhong, W., Chen, Y., . . . Cheng, R. (2026). BIRD-INTERACT: Re-imagining text-to-SQL evaluation via lens of dynamic interactions. In The Fourteenth International Conference on Learning Representations.

AuthorsOpenReview ↗

Author Reply 2/2

W2: text baselines would better be fine-tuned on the text trajectories of the 40 databases.

Q2: can we double check how much gain was from UI modality or from tool-calling trajectories on in-distribution schemas? the GenUI-harness gain was lower than all three text-only baselines where non of them were fine-tuned on text trajectories from the 40 databases as baseline

A: Separating these two sources of gain is important, and we trained a text counterpart of the GUI Coder inside our harness to do so, which keeps the training data and the Tool Agent identical on both sides. The text agent starts from the same base model and is trained with the same SFT recipe on the same tasks, which come from the 25 SFT databases (the other 15 are used only for RL of the GUI Coder). Each reference interface in the training data is rewritten into a chat message with the same content (retrieved items, options, and proposed values), so the two agents differ only in their output channel. Both are paired with the same fixed SFT Tool Agent, and the text agent interacts with the same user simulator as the text baselines in Table 2.

Qwen3.5-4B systemWhat is trainedPass@3Avg@3
Untrained, in GenUI-Harnessnothing9.333.22
SFT Tool Agent + SFT text agentboth agents, SFT43.0024.33
SFT Tool Agent + SFT GUI Coderboth agents, SFT47.3327.67
SFT Tool Agent + RL GUI Coder (final)GUI Coder only, RL58.0036.00

Read in Pass@3, the path from 9.33 to 58.00 has three parts. Untrained, the 4B model is weak in every harness, with 9.33 in GenUI-Harness and 10.67 to 13.67 in the three text harnesses. SFT with text interaction brings it to 43.00, which is the part that comes with training on the tool-calling trajectories. The training databases are disjoint from the 10 evaluation domains, with no exact overlap in task text, reference actions, or tool-name sequences (Appendix D.1). What training shares with the benchmark is the harness and the task format, and the text agent receives the same benefit from this, since it is trained on the same teacher trajectories generated by DeepSeek V4 Flash. Switching to the UI with the same training adds 4.33 points (47.33), which is the part that comes with the UI modality. RL then adds the last 10.67 points (58.00). This step trains only the GUI Coder while the Tool Agent stays fixed, so tool calling is identical across these runs and the gain cannot come from tool-calling trajectories. Without code in the user simulator prompt (the setting of Q1), the same path reads 43.00, 44.67, and 55.33: the UI adds 1.67 points at the SFT stage and RL adds 10.66. Table 3 points the same way with tool calling that was never trained on our data: five other models keep their own pre-exploration and post-execution and only use our GUI Coder, and Avg@3 rises by 2.44 to 8.77 points.

Q3: minor anonymity issue in the code --- i won't call it out as it was unintentional.

A: Thank you for the kind reminder and for your understanding. We have updated the supplementary material accordingly.

Q4: there are EvoGenUI-Bench and Generative Interfaces for Language Models that should be mentioned in related work instead of appendix ;)

A: We appreciate the pointer to these two works. Because of the page limit at submission, our detailed discussion of related work, including EvoGenUI-Bench and Generative Interfaces for Language Models, was placed in Appendix B. With the additional page allowed during the discussion period, we have moved the discussion of both works into the Related Work section of the main text (marked in red in the revised paper).

Reviewer_M8tGOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The current paper introduces:

1. A multi-agent harness (GENUI) that builds dynamic adaptive ephemeral interfaces meant to provide an alternative to text-based communication between agents and human users in complex database-backed tasks. The goal of this is to make the interaction between agents and humans in preparation for task solving more efficient. The harness executes database exploration via the Tool Agent, then interacts via dynamic React/TSX UIs generated by the GUI Coder Agent, collecting data, to finally solve the task via databaes operations performed by the Tool Agent. 2. a benchmark (UI-Tau) to evaluate an agent's end-to-end ability to generate data-aware dynamic UIs tailored to given tasks and complete those tasks via tool use based on the information collected from that UI. It's a state-evauluation benchmark build on 10 domains, coming in both a Lite (300 tasks) and a Full (1000 tasks) split.

The following contributions unlock successful fine-tuning of the GUI coder via Reinforcement Learning.

  • Dynamic UX (an engineering contribution) that parallelises rendering and offers browser context isolation in a shared sandbox;
  • Reward Auditor: an offline meta-reward diagnostic system that analyzes and refines the judge and the scoring specifications;

After role-specialized SFT on both agents, optimizing the GUI Coder with GRPO using the audited reward enables a 4B model to achieve 58.00% Pass@3 on UI-TAU Bench Lite, outperforming frontier models like Claude Opus 5 (46.67%). In a 100-task human reviewer survey, generated UIs reduced average dialogue rounds from 3.4 to 1.2 compared to text interaction.

Critical strengths

The problem tackled here is interesting: replacing text communication with ad-hoc UIs for suitable tasks. The harness has a sensible design (roles and wiring between agents executing those roles).

The paper brings together a benchmark, a harness, and trained agents (with well-designed targeted rl fine-tuning on top of standard SFT). The trained agents are compared against competitive off-the-shelf models (Claude Opus). This is a well-rounded contribution.

Critical weaknesses

While the authors acknowledge that Table 2 compares complete agent systems rather than isolating the communication medium, the ~4.5 pp Pass@3 gain over smolagents remains difficult to attribute. Although Table 3 isolates GUI-Coder quality and Appendix F isolates the channel for human dialogue rounds, there is no automated benchmark ablation comparing GENUI-Harness's two-stage architecture using text-only clarification against UI generation. Without this, it is unclear how much of the Pass@3 improvement stems from the structured pre-exploration/post-execution split versus the UI medium itself.

The reward revisions in Audit Rounds 1 and 2 simultaneously introduce LLM-judge rubric caps, deterministic failure-stage reward bands, icon gates, and source-based accessibility penalties (Appendix D.9). Because these mechanisms are ablated only as composite rounds in Figure 6, it is unclear whether the RL gains are primarily driven by dense deterministic shaping (preventing zero-reward collapse in GRPO) or improved LLM judging criteria.

Questions for authors

How general is the Reward Auditor beyond this specific front-end setup? Right now, the rules distilled during the audit rounds—such as Material UI icon checks, ARIA length cutoffs, and accessibility penalties—feel heavily customized to this particular benchmark.

Did the meta-auditor discover these failure modes entirely on its own from execution traces, or did it require human prompting and seed heuristics to look for them? More broadly, could this diagnostic setup be applied to other agentic RL problems (like general tool use or API execution) without having to manually re-engineer the audit space for each new environment?

6 comments
AuthorsOpenReview ↗

General Response

Thank you for your careful review and for recognizing the interesting problem, the sensible harness design, and the well-rounded combination of benchmark, harness, and targeted RL training. We will address each of your concerns and questions below.

AuthorsOpenReview ↗

Author Reply 1/3

W1: While the authors acknowledge that Table 2 compares complete agent systems rather than isolating the communication medium, the ~4.5 pp Pass@3 gain over smolagents remains difficult to attribute. Although Table 3 isolates GUI-Coder quality and Appendix F isolates the channel for human dialogue rounds, there is no automated benchmark ablation comparing GENUI-Harness's two-stage architecture using text-only clarification against UI generation. Without this, it is unclear how much of the Pass@3 improvement stems from the structured pre-exploration/post-execution split versus the UI medium itself.

A: Thank you for suggesting this ablation. We ran this experiment inside GenUI-Harness. Pre-exploration, post-execution, the models, and the evaluator are unchanged, and each text run reuses the stored pre-exploration output of the corresponding UI run. Only the middle step differs: the model that would write the interface writes one text message instead, the user simulator replies once in natural language, and this exchange is passed to the post-agent. The user simulator is the same one that the text baselines in Table 2 interact with. The protocol follows Table 2 (300 Lite tasks, three runs).

Each cell gives Pass@3 / Avg@3 (%) on the Lite split. The smolagents and UI columns are the Table 2 results, and the text column is the new experiment.

ModelsmolagentsGenUI-Harness, textGenUI-Harness, UIUI − Text
DeepSeek V4 Flash49.33 / 29.7047.00 / 27.6749.67 / 30.22+2.67 / +2.55
Qwen3.6-35B-A3B43.67 / 25.9043.67 / 24.2247.33 / 26.89+3.66 / +2.67
GPT-5.435.67 / 18.9036.00 / 22.1140.00 / 25.22+4.00 / +3.11
Claude Sonnet 4.648.00 / 31.3048.00 / 32.3351.33 / 35.67+3.33 / +3.34
Average44.17 / 26.4543.67 / 26.5847.08 / 29.50+3.41 / +2.92

With the structure held fixed, the UI version is higher for all four models, by 3.41 Pass@3 and 2.92 Avg@3 points on average. The smolagents column lets us separate the two effects in your question. Moving from smolagents (44.17 Pass@3 on average) to the text version of our harness (43.67) keeps text interaction and changes the agent structure, and the result stays about the same. Moving from the text version to the UI version (47.08) changes only the medium, and this is where the gain appears. We also evaluated the UI version with a user simulator that cannot read the interface source code, by removing the code block from its prompt. In that setting the UI version remains above the text version for all four models (45.25 against 43.67 Pass@3 on average).

Besides the performance comparison, we also analyzed the input that the user simulator provides in the two versions. With the UI it enters 8 words and makes 3 selections per task on average, and with text it writes 48 words. The human study that you mention (Section 6.3, Appendix F) points the same way: on 100 tasks with consistently usable interfaces, the same task information was presented to people either as a generated UI or as text, and the UI reduced the average number of rounds from 3.4 to 1.2. In both cases the interface saves interaction effort, which the task success measured in the ablation does not capture. We expect people to benefit more from the interface than the user simulator does, as humans are visual creatures: vision tends to dominate the other senses [1], people grasp the meaning of a visual scene within a fraction of a second [2, 3], and they can find information and draw inferences from it more easily in a spatial layout than in sentences [4].

We have added this ablation experiment and its analysis to the revised paper (Appendix I.1, Table 17).

[1] Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance. Psychological Review, 83(2), 157-171.

[2] Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system. Nature, 381(6582), 520-522.

[3] Potter, M. C., Wyble, B., Hagmann, C. E., & McCourt, E. S. (2014). Detecting meaning in RSVP at 13 ms per picture. Attention, Perception, & Psychophysics, 76(2), 270-279.

[4] Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65-100.

AuthorsOpenReview ↗

Author Reply 2/3

W2: The reward revisions in Audit Rounds 1 and 2 simultaneously introduce LLM-judge rubric caps, deterministic failure-stage reward bands, icon gates, and source-based accessibility penalties (Appendix D.9). Because these mechanisms are ablated only as composite rounds in Figure 6, it is unclear whether the RL gains are primarily driven by dense deterministic shaping (preventing zero-reward collapse in GRPO) or improved LLM judging criteria.

A: Thank you for your careful reading and for this precise question. We appreciate the chance to explain further how the two rounds are composed.

The two rounds change different parts of the reward, and both revisions were produced automatically by Reward Auditor. Audit Round 1 keeps the initial judge and revises only the deterministic scoring, introducing the failure-stage scores, the icon gate, and the accessibility multiplier together as one revision. Audit Round 2 keeps that scoring and revises only the judge, adding the caps listed in Table 14. We compare at the level of rounds because each round is one complete revision produced by this automatic process. The three policies are trained independently from the same SFT checkpoint with the GRPO configuration in Table 9, so Figure 6 already compares the two components step by step: revising the scoring raises Avg@3 from 30.67 to 32.67, and revising the judge on top of it raises it to 36.00.

To look at the mechanism you describe more directly, we also replayed the three reward versions on a common set of 2,400 rollouts. These rollouts are sampled from the SFT GUI Coder checkpoint, which is the starting point of all three RL runs, on 300 tasks sampled at random from the RL databases, with eight samples per task. None of these tasks were used in SFT. We then compared each reward with the actual task outcome, computing within-group AUROC with 95% bootstrap intervals. Note that the AUROC covers only the 123 mixed groups, which contain both successful and failed rollouts, and excludes the 177 groups in which all eight rollouts either succeed or fail.

Reward versionGroups with identical rewards ↓Distinct values per group ↑Within-group AUROC ↑Avg@3 after training (Figure 6)
Direct RL6.7%2.930.460 [0.422, 0.494]30.67
Round 1 (scoring only)0.0%6.950.615 [0.571, 0.660]32.67 (+2.00)
Round 2 (+ judge, final)0.0%7.310.720 [0.681, 0.758]36.00 (+3.33)

On these rollouts, the table shows what each revision changes in the reward signal. Under the Direct RL reward, 6.7% of the groups have identical rewards for all eight rollouts, and a group has 2.93 distinct reward values on average. After the scoring revision, no group has identical rewards and the number of distinct values rises to 6.95, since the failure stages that all received zero before now receive different scores. The same revision also improves how well the reward ranks rollouts within a group (AUROC from 0.460 to 0.615). The judge revision adds almost no density (6.95 to 7.31), but it improves the ranking further (0.615 to 0.720), and the three intervals do not overlap.

Based on these results, we believe the RL gains are driven not only by dense shaping but also by the improved judging criteria. The scoring revision, which makes the reward denser, accounts for 2.00 of the 5.33 points gained over Direct RL. The judge revision leaves the density almost unchanged and accounts for the other 3.33 points. We have described this composition explicitly in Appendix D.9 and updated the caption of Table 14 in the revised paper (marked in red).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.