acceptodds
Under review as a conference paper at ICLR 2027

The Specifications not Said: Can LLMs Turn Novice Developers' Wishes into Working Products?

Abstract

LLMs increasingly enable novice developers to build applications from scratch with natural language specifications, yet these developers often leave critical requirements unstated. Consequently, turning their immature specifications into working products not only requires implementing the application as instructed but recovering latent specifications as well. This gap is common in practice: in a deployed service where novice developers build mini-programs (a popular type of web application used by over one billion users) from natural-language specifications, only 3.94% of applications satisfy the developer after a single conversation. To evaluate this specification-recovery challenge with real-world developer data, we propose MiniProgramBench, a benchmark of 200 tasks across 10 application domains, built from developers' actual application creation and modification experiences and covering both frontend and backend functionality. MiniProgramBench pairs feedback from simulated developers with a frozen verification-point rubric, enabling evaluation of both specification recovery and product completion under a constrained feedback budget. Across 11 LLMs, verification-point pass rates reach at most 41.14% initially and 56.97% after up to two feedback-driven revisions. These revisions reveal that models struggle to incorporate feedback while preserving existing functionality. Controlled experiments further show that models perform substantially better with reference specifications than with self-inferred ones, underscoring specification recovery as a key bottleneck.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Application Generation, Benchmarking, Underspecified Requirements, Intent Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mYtWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates whether LLMs can build working applications from underspecified requests. The authors built MiniProgramBench, 200 tasks that each pair a developer's initial request with a hidden specification. The hidden specification is reconstructed from the developer's later change requests and reviewed by human developers. For evaluation, a simulated developer tests each generated app and gives feedback for up to two revisions. An LLM judge then scores the app against a fixed set of verification points, which are mined from failures of other models' apps. Across 11 models, the best pass rate is 41% before feedback and 57% after two revisions. Revisions often fail to fix the reported problem and sometimes break features that worked. Giving models the hidden specification helps far more than letting them infer one. The authors conclude that recovering developer intent is a key bottleneck.

Critical strengths
  • The paper targets a timely task and constructs a useful dataset.
  • The tasks span diverse application domains and include backend and multi-user workflows.
  • Beyond the overall score, the paper checks whether each revision fixes the reported problem and whether it breaks features that already worked.
  • The evaluation is broad, covering 11 recent models and two agentic harnesses.
Critical weaknesses
  • The authors kept only the checks that earlier models failed, which is a reasonable way to build a hard benchmark. However, the low scores may then overstate how much models struggle.
  • The simulator may act as an expert rather than a novice. It knows the full hidden specification, so its revision feedback may not resemble that of a real novice.
  • Even with the full specification, models still fail 35–72% of the checks. This is potentially caused by unfamiliarity with the WeChat platform, so intent recovery may not be the only main bottleneck the paper claims.
  • The scope is narrow, since all tasks are WeChat Mini Programs. These apps rely on WeChat-specific markup, styles, and APIs, which are far less common in LLM training data than mainstream languages such as Python or Java. Performance on more common types of programs is therefore not evaluated.
Questions for authors
  • The hidden specifications are extracted by an LLM and reviewed by ten developers. Was each requirement reviewed by more than one developer? If so, please report their agreement, since this shows how reliable the specifications are.
  • How was the simulated developer evaluated? How similar is its feedback to the real developers' change requests that the authors collected?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mYtW (1/2)

We are grateful to the reviewer for the careful examination of our evaluation methodology, simulated feedback, human annotation procedure, and platform scope.

W1: The authors kept only the checks that earlier models failed, which is a reasonable way to build a hard benchmark. However, the low scores may then overstate how much models struggle.

Our evaluation focuses on end-to-end user experience through observable checks of intended functionality and interaction behavior. These checks are grounded in task requirements and allow alternative valid implementations (Appendix D.1), with semantic screening during search and expert review of flagged cases (Appendices D.3–D.4). The newly added examples in Appendix D.4 illustrate the practical consequences of failing these checks, such as misleading save confirmations and loss of unfinished recordings. Although failure mining prioritizes challenging checks, failures on these checks reveal concrete gaps in the functionality and interaction quality users need.

W2: The simulator may act as an expert rather than a novice. It knows the full hidden specification, so its revision feedback may not resemble that of a real novice.

Q2: How was the simulated developer evaluated? How similar is its feedback to the real developers' change requests that the authors collected?

The simulator grounds its feedback in user requirements reconstructed from real development histories and reviewed by experienced mini-program developers. As specified in Appendix C, each message describes unmet behavior from the requesting user’s perspective in at most two sentences, without exposing requirement identifiers or evaluator terminology.

Two assessors independently reviewed 50 simulated feedback messages. Both judged 43/50 (86%) to describe genuine problems observable through ordinary application use, with disagreements on the remaining seven. The same assessors also evaluated resemblance to novice-user feedback, yielding a mean score of 2.98/3, with 3 indicating a plausible user request that is not overly technical. We have added the assessment protocol and results to Appendix C.2.

W3: Even with the full specification, models still fail 35–72% of the checks. This is potentially caused by unfamiliarity with the WeChat platform, so intent recovery may not be the only main bottleneck the paper claims.

We agree that implementation ability remains a bottleneck even when the full specification is provided. Nevertheless, our ablation reveals a substantial performance drop without the reference specification: Kimi K3’s Product score falls from 64.70 with the reference specification to 43.20 with a self-inferred specification, while the generating model, platform, and evaluation protocol remain fixed (Table 5). This 21.50-point gap highlights the substantial impact of inadequate requirement recovery.

To examine the role of platform unfamiliarity, we further analyzed 20 randomly sampled Direct-generation cases per model for Hy3, Claude Sonnet 5, and GPT-5.6 Sol (newly added Appendix E.7). Using Claude Opus 5.5 and our analysis prompt, we classified each failed VP in the initial generations. Platform-specific errors accounted for 10.0% of failed VPs in Hy3, 3.4% in Claude Sonnet 5, and 2.8% in GPT-5.6 Sol. These results suggest that platform unfamiliarity accounts for only a small share of the analyzed failures. Together with the substantial specification gap in Table 5, they highlight requirement recovery as a major challenge even for strong coding models.

AuthorsOpenReview ↗

Response to Reviewer mYtW (2/2)

W4: The scope is narrow, since all tasks are WeChat Mini Programs. These apps rely on WeChat-specific markup, styles, and APIs, which are far less common in LLM training data than mainstream languages such as Python or Java. Performance on more common types of programs is therefore not evaluated.

MiniProgramBench targets personalized, on-demand application generation for novice developers, for which the WeChat Mini Program ecosystem provides a practical setting for creating, using, and sharing applications. As discussed in the Introduction, WeChat Mini Programs operate within an ecosystem of over one billion monthly active users, provide a consistent runtime, and allow access without installing separate native applications. Meta’s Muse Spaces [1] and WeChat’s experimental Xiaowei tools [2] illustrate efforts to bring personalized application creation to everyday users, highlighting the role of host platforms in connecting creation with continued use. We therefore consider mini-programs a promising research setting for novice developers to create and share applications tailored to their needs through natural-language specifications.

In practice, our benchmark draws on over 76,000 generation trajectories collected through our deployed service on the WeChat Mini Program platform (Section 3.1), which grounds the benchmark in real development histories and defines the platform scope of our current evaluation. We have released the dataset in our anonymous repository and provide a public hosted evaluation platform demo covering all 200 benchmark tasks. Researchers can access it with a browser and their model’s API endpoint, while application execution and evaluation are handled server-side. Our future work will investigate how generation systems can better recover novice developers’ requirements and incorporate limited feedback to help them build and share applications on the WeChat Mini Program platform.

Q1: The hidden specifications are extracted by an LLM and reviewed by ten developers. Was each requirement reviewed by more than one developer? If so, please report their agreement, since this shows how reliable the specifications are.

As described in Appendix A.6, ten experienced mini-program developers reviewed the extracted requirements. The initial review flagged 5.75% of requirements as problematic and 6.75% as uncertain. Although not every requirement was independently double-annotated, each flagged requirement was subsequently reviewed by two to three other senior developers from the ten-person team. They revisited its supporting evidence and derivation, discussed the identified concerns, and revised the requirement if necessary.

We have also released a standalone version of our annotation tool under tools/human-annotation/ in the anonymous project repository. We have added these statistics and details of the additional review procedure to Appendix A.6.

[1] Meta: Design the Muse App, Just for You

[2] Yicai Global: Chinese Super App WeChat Starts Testing AI Assistant for Ordering Takeout, Buying Tickets, Other Tasks

Thank you for the detailed response and the additional analyses. I will keep my score.

Reviewer_ZoMpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MiniProgramBench evaluates application generation from underspecified requests using 200 tasks reconstructed from real developers’ creation and modification histories. Each task includes a public request, a private latent specification, and verification points. A simulated developer provides feedback over up to two revisions, while a code-based judge evaluates generated applications using source code and GUI traces. The benchmark covers frontend, backend, persistent state, and multi-user workflows. Experiments evaluate 11 models and two agentic harnesses, including analyses of feedback-driven repair and generation with reference specifications.

Critical strengths
  1. The tasks are grounded in actual development histories. Using initial requests and subsequent modifications provides a useful source of realistic requirements. Retaining provenance and involving experienced developers in reviewing the specifications strengthens the connection between benchmark tasks and practical application needs.

  2. The evaluation covers more than initial code generation. Backend execution, persistent state, multi-user workflows, and feedback-driven revisions provide a richer setting than checking whether an application compiles or renders. The evaluation across 11 models and two harnesses, together with judge validation, represents a substantial experimental effort.

Critical weaknesses
  1. The important community gap is not clearly established. Underspecified requests and iterative refinement are already studied in work such as Asuka-Bench, while the paper also discusses benchmarks for evolving requirements and full-stack generation. MiniProgramBench combines real trajectories, mini-program execution, and a mined verification rubric, but the paper does not sufficiently explain which important research question existing benchmarks cannot answer and this benchmark uniquely enables. The related-work discussion lists differences without establishing their significance. A comparison table covering requirement provenance, underspecification, interaction protocol, backend and multi-user support, evaluation method, and release accessibility is needed. More importantly, the paper should explain how these distinctions change what researchers can learn. As presented, it risks being another benchmark with a different platform and task source, rather than filling a clearly demonstrated gap. Asuka-Bench

  2. The case for broad adoption and lasting research value is weak. The evaluation combines a mini-program-specific runtime, cloud-function execution, multi-actor GUI exploration, and several model-based construction or judging stages. Although an anonymous release is provided, the paper offers limited evidence about the effort and cost required for an external group to reproduce and extend the benchmark. It is also unclear which concrete research directions the benchmark would support beyond comparing model scores. Findings such as better performance with reference specifications and incomplete repair after feedback are plausible, but the paper needs to show what additional insight this benchmark provides and how it can guide improved systems. Low scores alone establish difficulty, not the need for another evaluation suite.

Questions for authors
  1. Can you provide a structured comparison with Asuka-Bench, SR-Eval, interactive SWE benchmarks, and full-stack generation benchmarks? Which consequential capability or research question is missing from these resources?

  2. What specific experiment or system-development direction does MiniProgramBench enable that would be difficult to study using existing benchmarks? Can you demonstrate an insight that depends on its distinctive design?

  3. What does an external researcher need to run and extend the benchmark? Please report setup requirements, evaluation cost and runtime, model/API dependencies, and which components can be reproduced without access to the original service or private trajectories.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ZoMp (1/2)

We sincerely thank the reviewer for the thoughtful comments and concrete suggestions on clarifying MiniProgramBench’s research contribution, practical value, and reproducibility.

W1: The important community gap is not clearly established.

Q1: ... Which consequential capability or research question is missing from these resources?

MiniProgramBench targets personalized, on-demand application generation for novice developers, evaluating whether systems can turn their rough ideas and limited feedback into applications that meet their practical needs and can be used and shared. Existing benchmarks for underspecified requests emphasize intent recovery and iterative refinement, but offer limited support for studying this complete application-generation scenario. To study this setting in practice, we collect real user development histories, build an evaluation pipeline covering practical end-to-end application workflows, and choose WeChat Mini Programs as a platform suited to application creation, use, and sharing. These distinctions from existing benchmarks are driven by the demands of real use, and each has concrete implications for what researchers can evaluate and learn, as detailed below.

1. Requirements grounded in real development histories. Existing benchmarks such as Asuka-Bench, SR-Eval, and SWE-Interact construct underspecified inputs or staged requirement disclosure from predefined task specifications. The correspondence between initial requests and target requirements is therefore constructed through synthesis or simulation, rather than reconstructed from the corresponding users’ development histories. MiniProgramBench preserves the underspecification of actual initial requests and reconstructs target requirements from subsequent development evidence, with traceable provenance and expert review.

2. Comprehensive evaluation of generated applications. Our evaluation covers frontend interactions, backend execution, persistent state, and workflows involving multiple user roles. This broader coverage helps determine whether users’ requirements are implemented in practical application workflows, supporting the intended use and sharing of personalized applications.

3. Mini-programs as a platform for personalized, on-demand application generation. Meta’s Muse Spaces [1] and WeChat’s experimental Xiaowei assistant [2] illustrate recent interest in personal applications created through natural-language interaction and the role of host platforms in connecting creation with everyday use. WeChat’s large user base and ease of distribution make mini-programs a promising setting for this direction, whose research implications we discuss in response to W2.

The table below compares how the relevant benchmarks support this application-generation setting. We have added this comparison as Table 1 and clarified the research gap and research directions in the Introduction and Related Work.

BenchmarkRequirements grounded in real revision historiesCoordinated multi-identity GUI evaluationEvaluation platformExplicitly underspecified / partial inputsImplementation-dependent user feedbackBackend / database execution
Asuka-Bench××Web (frontend)✓✓×
SR-Eval××Python / Java✓×✓
SWE-Interact××Dockerized code repositories✓✓✓
Full-stack generation benchmarks (e.g., FullStack-Bench)××Web (full-stack)××✓
MiniProgramBench✓✓WeChat Mini Programs✓✓✓

W2: ... Low scores alone establish difficulty, not the need for another evaluation suite.

Q2: What specific experiment or system-development direction does MiniProgramBench enable that would be difficult to study using existing benchmarks? Can you demonstrate an insight that depends on its distinctive design?

MiniProgramBench aims to support community research on personalized, on-demand application generation for novice developers, an emerging direction closely tied to practical user needs. WeChat’s large user base and ease of distribution make mini-programs a practical platform for studying how users create, use, and share such applications. This platform is already attracting industry efforts, with WorkBuddy and Coze [3,4] supporting the creation of WeChat Mini Programs through natural language. Within this ecosystem, generation agents could enable hundreds of millions or even billions of non-programmers to create and run customized applications through natural language and share them with others who have similar needs.

To encourage more researchers to study this problem, we provide an easy-to-use hosted evaluation platform demo covering all 200 benchmark tasks. Its use and replication costs are discussed in our response to Q3.

AuthorsOpenReview ↗

Response to Reviewer ZoMp (2/2)

W2 + Q2 (continued)

Our paper translates this vision into concrete research questions. Our real-world usage study shows that novice developers provide little revision feedback, motivating research on requirement recovery and effective revision under limited interaction. As discussed in W1, we construct tasks from actual initial requests and reconstruct target requirements from subsequent development histories, rather than synthesizing underspecified requests from complete PRDs. This grounds our evaluation in requirements revealed during real application development. Together, our usage study and model evaluations provide the following three insights, which are also discussed in the Introduction of our original submission.

Insight 1. Real users provide sparse feedback. Of over 76,000 real generation trajectories, approximately 7.1% contain developer-requested modifications, and 84.1% of those involve only one or two revisions (Sec. 4.1). This finding identifies limited feedback as a practical constraint and motivates systems that recover requirements early and use each revision effectively.

Insight 2. Models’ implementation capabilities remain underutilized when inputs are underspecified. With the generating model and evaluation protocol held fixed, Kimi K3 achieves a Product score of 64.70 with the reference specification versus 43.20 with a self-inferred specification (Sec. 4.6). This reveals a requirement-recovery bottleneck and supports improving how systems identify and express users’ requirements.

Insight 3. Feedback-driven revision may leave other behavioral conditions of the same requirement unmet. The highest observed repair rate for previously failed verification points associated with feedback-targeted requirements is 57.1%, and previously passed points can also fail after revision (Sec. 4.4). This motivates revision strategies that address the full intended behavior while preserving existing functionality.

In practice, we have used the benchmark and these insights to guide targeted model training. In subsequent model development, evaluation revealed revisions that updated frontend state without correctly updating backend data, preventing information from being shared correctly across users. We introduced targeted curriculum learning to address this failure mode and observed that it was alleviated after training.

Q3: What does an external researcher need to run and extend the benchmark? ...

To enable the community to study this application-generation setting, we provide a public hosted evaluation platform demo covering all 200 benchmark tasks. Researchers can access it with a browser and their model’s API endpoint, while application execution and evaluation are handled server-side.

Researchers pay only for their generation model’s inference, while the hosted service handles application execution and evaluation without additional evaluation fees or local runtime setup. In our experiments, the mean cumulative generation cost per completed task, including initial generation and two revisions, was approximately USD 0.02 for Hy3 and USD 0.66 for Claude Sonnet 5. Under the Hy3-based configuration, the complete workflow, including generation and evaluation over the initial round and two revision rounds, averaged 55.45 minutes per task, or 18.48 minutes per round (newly added Appendix B.1).

For custom experiments, we plan to release the simulated-user and GUI-agent code so researchers can use the official WeChat Mini Program DevTools to independently run and extend evaluation, without access to the original service or private developer trajectories.

[1] Meta: Design the Muse App, Just for You

[2] Yicai Global: Chinese Super App WeChat Starts Testing AI Assistant for Ordering Takeout, Buying Tickets, Other Tasks

[3] CloudBase Team (2026). Generate a WeChat Mini Program with One Sentence in WorkBuddy — Backend and Database Handled by CloudBase.

[4] Coze. Develop Mini Programs (official documentation).

AuthorsOpenReview ↗

follow-up to Reviewer ZoMp

Dear Reviewer ZoMp,

Thank you again for your thoughtful comments! As the discussion deadline approaches, we would appreciate your feedback on our two-part response, including the benchmark comparison, the research directions supported by MiniProgramBench, and the evaluation platform and reproduction costs. Please let us know if any of your main concerns remain unresolved so that we can address them during the discussion.

Reviewer_R8NMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MiniProgramBench evaluates application generation using 200 tasks drawn from real users’ requests and modification histories. Models receive an incomplete request and up to two rounds of feedback. The benchmark checks the apps against a fixed checklist using code and interaction records. Feedback helps, but many requirements remain unmet; providing complete requirements leads to larger gains.

Critical strengths

The problem is important and realistic. Users often know what they need but cannot explain every detail upfront. Real development histories give these tasks a useful practical basis. The evaluation checks more than appearance. It covers saved data, backend functions, multiple users, and time-dependent behavior. The same checklist is used before and after each revision. Concrete examples help explain the findings.

Critical weaknesses

Finding more problems does not necessarily mean measuring what matters to users. Results show that these checks find more failures, not that the scores match users’ views of product quality. Some independent human assessments would help.

Some missing requirements may not be reasonable to guess. A user may request something later that was never clear in the original request. Giving the model complete requirements naturally helps but does not show whether it misunderstood the user or simply lacked information. Allowing models to ask questions would help separate these explanations.

Questions for authors

none

3 comments
AuthorsOpenReview ↗

Response to Reviewer R8NM

We sincerely thank the reviewer for the careful assessment and constructive suggestions on alignment with human judgments and the role of requirement clarification.

W1: Finding more problems does not necessarily mean measuring what matters to users. Results show that these checks find more failures, not that the scores match users’ views of product quality. Some independent human assessments would help.

We agree that detecting more failures alone does not establish alignment with users’ judgments of product quality. Appendices D.4 and E.5 examine how our checks relate to user requirements. Appendix D.4 documents a user-value audit in which three experts independently reviewed flagged VPs without seeing model identities or benchmark outcomes. Appendix E.5 maps all 2,804 VPs to reviewed user requirements and traces feedback to implementation changes, providing evidence that VP outcomes reflect concrete successes and failures in satisfying user needs.

Following your suggestion, we conducted an Arena-style blind human evaluation of six models on six selected tasks, yielding 73 eligible pairwise comparisons with decisive human preference and unequal Product scores (Appendix E.8). The preferred implementation received the higher Product score in 66 cases (90.4%), providing quantitative evidence of alignment.

W2: Some missing requirements may not be reasonable to guess. A user may request something later that was never clear in the original request. Giving the model complete requirements naturally helps but does not show whether it misunderstood the user or simply lacked information. Allowing models to ask questions would help separate these explanations.

We agree that some requirements need clarification and that the reference-spec gain can partly reflect missing information. Our default protocol follows the data collection process, in which users try generated applications and provide feedback that helps reconstruct requirements left unstated in their initial requests.

Following your suggestion, we added a “With clarification” condition on all 200 tasks, inspired by the interactive setting of SWE-INTERACT [1]. Before generation, each model was allowed to ask the simulated user clarification questions, with the questioning model’s cumulative output capped at 8,192 tokens during clarification (newly added Appendix E.6). As shown below, clarification yields limited gains over self-inferred specifications, while Product scores remain below those obtained with the reference specification.

ModelInput conditionProductUsefulUsableDesirable
Hy3Direct11.8414.3513.497.12
Hy3Self-inferred spec18.8523.9820.2512.31
Hy3With clarification20.7625.3521.7014.87
Hy3Reference spec44.0854.2347.1431.53
DeepSeek-V4-FlashDirect12.5013.7114.519.22
DeepSeek-V4-FlashSelf-inferred spec14.5214.9116.1712.20
DeepSeek-V4-FlashWith clarification16.9717.9718.2714.15
DeepSeek-V4-FlashReference spec28.3532.8931.3921.58
Kimi K3Direct38.7140.5243.1832.32
Kimi K3Self-inferred spec43.2048.3248.4933.46
Kimi K3With clarification47.0950.5552.0739.03
Kimi K3Reference spec64.7072.7070.1951.09

[1] Raghavendra et al. (2026). SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions. arXiv:2606.30573.

AuthorsOpenReview ↗

follow-up to Reviewer R8NM

Dear Reviewer R8NM,

Thank you again for your helpful feedback. As the discussion deadline approaches, we would appreciate your thoughts on our responses, particularly the added blind human evaluation and pre-generation clarification experiment. Do these additions address your concerns about alignment with user judgments and the role of missing information? We would be happy to clarify any remaining questions.

Dear Reviewer R8NM,

With the discussion period nearing its close, we would appreciate any further feedback on our responses, particularly whether any of your main concerns remain unresolved. We remain available to provide additional clarification. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.