- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper evaluates whether LLMs can build working applications from underspecified requests. The authors built MiniProgramBench, 200 tasks that each pair a developer's initial request with a hidden specification. The hidden specification is reconstructed from the developer's later change requests and reviewed by human developers. For evaluation, a simulated developer tests each generated app and gives feedback for up to two revisions. An LLM judge then scores the app against a fixed set of verification points, which are mined from failures of other models' apps. Across 11 models, the best pass rate is 41% before feedback and 57% after two revisions. Revisions often fail to fix the reported problem and sometimes break features that worked. Giving models the hidden specification helps far more than letting them infer one. The authors conclude that recovering developer intent is a key bottleneck.
Critical strengths
- The paper targets a timely task and constructs a useful dataset.
- The tasks span diverse application domains and include backend and multi-user workflows.
- Beyond the overall score, the paper checks whether each revision fixes the reported problem and whether it breaks features that already worked.
- The evaluation is broad, covering 11 recent models and two agentic harnesses.
Critical weaknesses
- The authors kept only the checks that earlier models failed, which is a reasonable way to build a hard benchmark. However, the low scores may then overstate how much models struggle.
- The simulator may act as an expert rather than a novice. It knows the full hidden specification, so its revision feedback may not resemble that of a real novice.
- Even with the full specification, models still fail 35–72% of the checks. This is potentially caused by unfamiliarity with the WeChat platform, so intent recovery may not be the only main bottleneck the paper claims.
- The scope is narrow, since all tasks are WeChat Mini Programs. These apps rely on WeChat-specific markup, styles, and APIs, which are far less common in LLM training data than mainstream languages such as Python or Java. Performance on more common types of programs is therefore not evaluated.
Questions for authors
- The hidden specifications are extracted by an LLM and reviewed by ten developers. Was each requirement reviewed by more than one developer? If so, please report their agreement, since this shows how reliable the specifications are.
- How was the simulated developer evaluated? How similar is its feedback to the real developers' change requests that the authors collected?
3 comments
Response to Reviewer mYtW (2/2)
W4: The scope is narrow, since all tasks are WeChat Mini Programs. These apps rely on WeChat-specific markup, styles, and APIs, which are far less common in LLM training data than mainstream languages such as Python or Java. Performance on more common types of programs is therefore not evaluated.
MiniProgramBench targets personalized, on-demand application generation for novice developers, for which the WeChat Mini Program ecosystem provides a practical setting for creating, using, and sharing applications. As discussed in the Introduction, WeChat Mini Programs operate within an ecosystem of over one billion monthly active users, provide a consistent runtime, and allow access without installing separate native applications. Meta’s Muse Spaces [1] and WeChat’s experimental Xiaowei tools [2] illustrate efforts to bring personalized application creation to everyday users, highlighting the role of host platforms in connecting creation with continued use. We therefore consider mini-programs a promising research setting for novice developers to create and share applications tailored to their needs through natural-language specifications.
In practice, our benchmark draws on over 76,000 generation trajectories collected through our deployed service on the WeChat Mini Program platform (Section 3.1), which grounds the benchmark in real development histories and defines the platform scope of our current evaluation. We have released the dataset in our anonymous repository and provide a public hosted evaluation platform demo covering all 200 benchmark tasks. Researchers can access it with a browser and their model’s API endpoint, while application execution and evaluation are handled server-side. Our future work will investigate how generation systems can better recover novice developers’ requirements and incorporate limited feedback to help them build and share applications on the WeChat Mini Program platform.
Q1: The hidden specifications are extracted by an LLM and reviewed by ten developers. Was each requirement reviewed by more than one developer? If so, please report their agreement, since this shows how reliable the specifications are.
As described in Appendix A.6, ten experienced mini-program developers reviewed the extracted requirements. The initial review flagged 5.75% of requirements as problematic and 6.75% as uncertain. Although not every requirement was independently double-annotated, each flagged requirement was subsequently reviewed by two to three other senior developers from the ten-person team. They revisited its supporting evidence and derivation, discussed the identified concerns, and revised the requirement if necessary.
We have also released a standalone version of our annotation tool under tools/human-annotation/ in the anonymous project repository. We have added these statistics and details of the additional review procedure to Appendix A.6.
Thank you for the detailed response and the additional analyses. I will keep my score.
Response to Reviewer mYtW (1/2)
We are grateful to the reviewer for the careful examination of our evaluation methodology, simulated feedback, human annotation procedure, and platform scope.
Our evaluation focuses on end-to-end user experience through observable checks of intended functionality and interaction behavior. These checks are grounded in task requirements and allow alternative valid implementations (Appendix D.1), with semantic screening during search and expert review of flagged cases (Appendices D.3–D.4). The newly added examples in Appendix D.4 illustrate the practical consequences of failing these checks, such as misleading save confirmations and loss of unfinished recordings. Although failure mining prioritizes challenging checks, failures on these checks reveal concrete gaps in the functionality and interaction quality users need.
The simulator grounds its feedback in user requirements reconstructed from real development histories and reviewed by experienced mini-program developers. As specified in Appendix C, each message describes unmet behavior from the requesting user’s perspective in at most two sentences, without exposing requirement identifiers or evaluator terminology.
Two assessors independently reviewed 50 simulated feedback messages. Both judged 43/50 (86%) to describe genuine problems observable through ordinary application use, with disagreements on the remaining seven. The same assessors also evaluated resemblance to novice-user feedback, yielding a mean score of 2.98/3, with 3 indicating a plausible user request that is not overly technical. We have added the assessment protocol and results to Appendix C.2.
We agree that implementation ability remains a bottleneck even when the full specification is provided. Nevertheless, our ablation reveals a substantial performance drop without the reference specification: Kimi K3’s Product score falls from 64.70 with the reference specification to 43.20 with a self-inferred specification, while the generating model, platform, and evaluation protocol remain fixed (Table 5). This 21.50-point gap highlights the substantial impact of inadequate requirement recovery.
To examine the role of platform unfamiliarity, we further analyzed 20 randomly sampled Direct-generation cases per model for Hy3, Claude Sonnet 5, and GPT-5.6 Sol (newly added Appendix E.7). Using Claude Opus 5.5 and our analysis prompt, we classified each failed VP in the initial generations. Platform-specific errors accounted for 10.0% of failed VPs in Hy3, 3.4% in Claude Sonnet 5, and 2.8% in GPT-5.6 Sol. These results suggest that platform unfamiliarity accounts for only a small share of the analyzed failures. Together with the substantial specification gap in Table 5, they highlight requirement recovery as a major challenge even for strong coding models.