acceptodds
Under review as a conference paper at ICLR 2027

LiveUIBench: Evaluating Agent as App Workflows

Abstract

Applications provide convenient features for users to complete daily tasks. However, different users may have different or even conflicting requirements for the same application, while the requirements of the same user may change over time. Personalizing applications produced by agent-to-app or UI-to-code approaches can require changes to the generated implementation. We propose Agent as App, a task in which users express their requirements in natural language and an agent creates a personalized application while continuing to supply its behavior during use. We implement LiveUIAgent, which generates a task-specific orchestration connecting the user interface, saved information, and service actions. A shared agent runtime renders and updates the interface while carrying out the requested operations. Revising the orchestration and interface while reusing that runtime provides a lightweight basis for personalization as requirements evolve. We introduce LiveUIBench, a benchmark with diverse requirements and ordered interactions across personal services. The empirical findings show that the evaluated models struggle to complete these workflows reliably. Applications can satisfy many local checks while leaving the overall task unfinished, and greater partial coverage can coincide with fewer completed workflows.

Primary area:
datasets and benchmarks
Keywords:
GUI agents, agent-generated applications, workflow automation, behavioral evaluation, grounded interfaces, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SMzrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces the Agent‑as‑App task: agents build personalized apps from natural‑language user requests and keep behaving during usage. The authors implement LiveUIAgent, which leverages a shared runtime for UI rendering and service execution, allowing lightweight updates of workflows and interfaces when user needs shift. They also release LiveUIBench, a benchmark covering diverse personal‑service workflows. Evaluations show existing models struggle to finish full workflows reliably. Models can pass many local checks yet fail end‑to‑end tasks, and higher partial performance does not guarantee better full‑task completion. Topic: datasets and benchmarks.

Critical strengths
  1. Proposes the fresh Agent‑as‑App task, combining GUI agent and personalized application generation. This opens up a new research angle.
  2. Builds LiveUIBench and identifies the disconnect between local checks and end‑to‑end workflow success. This finding offers practical hints for GUI‑agent evaluation.
Critical weaknesses
  1. Weak benchmark validation. Task curation rules, difficulty calibration and human validation for LiveUIBench are insufficient. It is hard to confirm whether this benchmark reliably measures real‑world model capability.
  2. Limited system and experimental evidence. LiveUIAgent brings limited technical novelty. Missing direct ablations and fair comparisons against existing GUI‑agent systems, so its practical advantages remain unclear.
Questions for authors
  1. How were LiveUIBench tasks curated? Was sufficient human validation done to verify task feasibility and difficulty spread?
  2. What is LiveUIAgent’s key advantage over existing UI‑agent or agent‑to‑code baselines? Can you add head‑to‑head comparisons?
  3. What main error patterns lead to high local‑check scores but failed full workflows in your results?
Reviewer_2vt6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Agent as App, in which an agent continues to interpret user input and perform service operations after generating a personalized application. LiveUIAgent uses task orchestration to connect the interface, persistent state, and service actions, updating the application as subsequent requirements arise. Built on AppWorld, LiveUIBench contains 500 tasks covering creation, submission, revision, and return visits, testing sequences of operations and their dependencies on existing records. The results show that models can relatively easily generate applications that open successfully, but complete-workflow success rates remain low. Passing individual checks does not necessarily mean that all requirements have been met.

Critical strengths
  1. Subsequent requests must modify or reuse previously created records, and the application must continue to support their use when the user returns. Shared-world tasks further require new applications to use service state written by earlier applications. These dependencies are well suited to exposing problems that one-shot page-generation evaluations can miss.
  2. Separately analyzing interface feedback, state persistence, and service outcomes is useful. In the priority-update example, the interface confirms the update and the record remains available after reopening, but an independent read reveals that the saved value was not updated correctly. Such examples concretely illustrate failures that can arise during continued application use.
Critical weaknesses
  1. The main experiments do not follow the actual interface interaction path, so they cannot establish whether users can complete tasks through the generated applications. Directly invoking predefined actions can test backend logic, but input components, event bindings, and parameter passing may still fail. The connection between the interface and backend services is precisely what distinguishes Agent as App from ordinary multi-turn tool use. Only a small subset of cases is tested through the actual interface, which is insufficient to support the benchmark’s intended evaluation scope.
  2. The coverage of checks on key business fields in the main metrics is unclear, so cases labeled as fully successful may still fail to meet user requirements. In the priority-update example, the interface reports success and the record persists, but the stored value is incorrect. This problem is identified through an additional service-data read, yet the paper does not clarify whether the main evaluation systematically verifies the target records and requested values. For tasks involving repeated modification and reuse of existing data, these checks directly affect the credibility of the success labels.
Questions for authors
  1. In the main evaluation, what selects the declared actions and supplies their input parameters? Do these calls pass through the generated interface’s input bindings?
  2. Could you provide a task judged fully successful by the main evaluation and show how the user’s requirements, target records, and requested field values map to the actual assertions?
Reviewer_yNV8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces “Agent as App,” an approach where AI agents create personalized applications from natural-language requests and adapt them as users interact with them. The authors develop LiveUIAgent and LiveUIBench to evaluate this approach across 500 tasks involving multiple services.

The results show that even the best model completes only 19.6% of full workflows. Agents handle initial interactions better than later steps, such as submitting and revising requests. The study also finds that different evaluation metrics can produce different model rankings, and 339 tasks remain unsolved by every tested model.

The authors conclude that creating functional applications is not enough; agents must also reliably handle user interactions, revisions, and ongoing tasks.

Critical strengths
  1. The task framing and evaluation unit are well motivated. Scoring a retained application over an ordered sequence (open, submit, revise, return) with a conjunctive success criterion is a more demanding target than single-turn UI generation or one-shot tool use. Grounding checks in AppWorld service state lets the benchmark separate an acknowledgment from a saved effect. The wetland case (Figure 6) shows this concretely: the interface says “Normal” and the preservation check passes, but independent reads find “low” instead of the requested “medium”.

  2. The accounting is careful and mostly transparent. Rejected generations and unavailable responses stay in the denominators. All numbers are recounted from archived per-task and per-episode records. The paper gives a first-failure partition (Figure 5) and a joint service/visible-check breakdown. I checked that the per-family FWRs in Table 2 reconcile with the overall FWRs, and the 7,925 failures in Figure 5 match the sum of the 17 runs’ non-successes. The authors also repeatedly state what their analyses do not show (the union is not an ensemble, stage labels are not causes).

  3. The results have headroom and show useful structure. The best model completes under 20% of workflows, and 339 of 500 tasks are unsolved by every run. Success sets are not nested in the aggregate ranking, so aggregate scores hide complementary strengths. The finding that passing service checks while failing visible checks is more common than the reverse (2,305 vs. 1,351 episodes) is a useful diagnostic for this kind of benchmark.

Critical weaknesses
  1. The central claims about Agent as App and LiveUIAgent are not tested. The abstract calls revising the orchestration on a shared runtime “a lightweight basis for personalization”, but no experiment measures this. There is no comparison with agent-to-app/UI-to-code generation or a plain tool-using chat agent, no ablation of the orchestration/runtime design, and no revision cost. Section 3 gives no orchestration schema, prompts or runtime details. The aggregate survey also “invokes declared actions” without the UI (Section 3.3), so Table 2 appears to measure stateful tool use with a form front-end, much like AppWorld or τ-bench. Per-user variation (r_{u,t} ≠ r_{v,t}) is not evaluated at all.

  2. Benchmark construction and validity are not substantiated, and the materials are private. The main text does not say how tasks and assertions were generated, even though the AI-use statement mentions “benchmark construction”. The annotation in Appendix D is qualitative only, with no counts, verdict distribution or agreement. No human judgments of application success are collected, so FWR and APR are never validated, and the visible checks are not specified. 68% of tasks are unsolved by every run, with no audit to separate “hard” from “over-constrained”. Table 2 also mixes in development tasks and is not split by partition. Appendix A states that no runnable release is provided.

  3. The model comparison is single-run, heterogeneous and over-interpreted. Each model is run once, with no confidence intervals or paired tests, and the “recorded generation procedures” differ per model. Several highlighted reversals rest on a few tasks: Kimi vs. sol is 6 vs. 5 in multi-tool and 4 vs. 3 in shared, DeepSeek’s multi-tool lead is 7 vs. 6, and the Grok opening gap is 496 vs. 491. Generation errors make up 53.7% of failures, so the ordering of weaker models likely reflects format or API robustness. The APR/FWR reversal is largely a consequence of a conjunctive metric. The failure analysis is descriptive and gives little guidance on what to improve.

Questions for authors
  1. How does LiveUIAgent compare, on the same tasks and with the same underlying model, with (a) direct agent-to-app code generation, (b) a tool-using agent behind a chat interface, and (c) a variant that regenerates the whole application instead of revising the orchestration? Please also say whether the same model generates the orchestration and executes it at runtime, whether any revision episode actually requires orchestration changes, and what the revision cost or latency is.

  2. Can you describe how tasks and assertions were constructed and validated? Specifically: how tasks relate to AppWorld tasks, how many tasks the annotators reviewed, the verdict distribution and inter-annotator agreement, how visible checks are computed, whether FWR/APR correlate with human judgments of application success on a sample, and what a manual audit of a random sample of the 339 unsolved tasks shows. Please also report results separately for the development, public and exploratory partitions.

  3. Can you add repeated runs with task-level bootstrap confidence intervals and paired tests for the headline comparisons (Kimi vs. sol, Grok 4.6 vs. 4.5, DeepSeek in multi-tool)? Can you also report FWR conditional on valid generation, and under a standardized generation pipeline with matched decoding and retries? Without these, I cannot tell how much of the model ordering and the “gap” reflects the benchmark versus format and pipeline effects.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.