- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces the Agent‑as‑App task: agents build personalized apps from natural‑language user requests and keep behaving during usage. The authors implement LiveUIAgent, which leverages a shared runtime for UI rendering and service execution, allowing lightweight updates of workflows and interfaces when user needs shift. They also release LiveUIBench, a benchmark covering diverse personal‑service workflows. Evaluations show existing models struggle to finish full workflows reliably. Models can pass many local checks yet fail end‑to‑end tasks, and higher partial performance does not guarantee better full‑task completion. Topic: datasets and benchmarks.
Critical strengths
- Proposes the fresh Agent‑as‑App task, combining GUI agent and personalized application generation. This opens up a new research angle.
- Builds LiveUIBench and identifies the disconnect between local checks and end‑to‑end workflow success. This finding offers practical hints for GUI‑agent evaluation.
Critical weaknesses
- Weak benchmark validation. Task curation rules, difficulty calibration and human validation for LiveUIBench are insufficient. It is hard to confirm whether this benchmark reliably measures real‑world model capability.
- Limited system and experimental evidence. LiveUIAgent brings limited technical novelty. Missing direct ablations and fair comparisons against existing GUI‑agent systems, so its practical advantages remain unclear.
Questions for authors
- How were LiveUIBench tasks curated? Was sufficient human validation done to verify task feasibility and difficulty spread?
- What is LiveUIAgent’s key advantage over existing UI‑agent or agent‑to‑code baselines? Can you add head‑to‑head comparisons?
- What main error patterns lead to high local‑check scores but failed full workflows in your results?