- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a new method for automatically constructing software engineering benchmarks, scoped to the Web Development domain. The authors build a mine–craft–patch pipeline for automatic task and evaluation generation: mine discovers replayable behaviors, craft masks the corresponding code, and patch evaluates coding agents. Across 26 apps, 1,975 behaviors are mined and 4,063 tasks are constructed without human intervention. Atomic tasks are composed into cumulative tasks to test long-term ability, and the success rate drops dramatically as tasks accumulate.
Critical strengths
The paper's concept is easy to follow.
The work introduces a new methodology for automatically generating web development benchmarks.
The experiments cover a wide range of setups.
Critical weaknesses
4,063 tasks make evaluation prohibitively expensive.
The tasks appear to encourage copying the behavior of other web applications (the authors acknowledge this in the ethics statement).
The reference-guided tasks seem limited in scope.
Questions for authors
-
How is code leakage prevented? The selected apps appear to be open source.
-
The app behavior is monitored — but what about the UI (colors, fonts)? How complex are the computer-use operations required (e.g., holding a button to send a voice message)?
-
What are your thoughts on adapting this method to the mobile app development domain?
3 comments
Dear Reviewer BTvt,
As the discussion period closes soon, we would appreciate knowing whether our responses have addressed your questions.
- Evaluation cost: The 4,063 tasks are not meant to be run in full every time. They form a pool that can be drawn on for different purposes. ProgramDistill-300 supports standard comparison at about 7% of full cost, and rankings among statistically distinguishable models hold on a fresh application-balanced sample (Appendix M). Targeted evaluations by application or depth, as well as training data, can be built from the same pool. This flexibility is a direct benefit of scalable automatic generation.
- Code leakage: Both source access and network access are blocked, and submissions match the original source on only 0.3% of lines (Appendix I.7, Appendix K).
- UI fidelity: We deliberately score behavioral fidelity with deterministic checks for reproducible and meaningful grading of web application functionality. Visual similarity would require relying on VLM judges, so we excluded it, but it can be added as a separate axis if needed (Limitations).
- Interaction complexity: The browser helper supports a broad range of interactions, including drag-and-drop, hover, and file uploads (Appendix C).
ProgramDistill builds both tasks and verifiers from any working application, so it extends naturally across web applications and to platforms such as mobile that expose analogous interfaces like accessibility trees. We would be happy to address any remaining questions.
Reminder from AC
Dear Reviewer BTvt ,
As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.
Thank you for your time, your thoughtful reviews, and your service to the ICLR community.
Best Regards, AC
We thank the reviewer for recognizing the clarity of our approach, the novel automated benchmark-generation methodology, and the breadth of our experiments, and for the questions on cost, scope, leakage, UI fidelity, and mobile extension addressed below. All revisions are marked in blue in the revised manuscript.
[W1] Evaluation cost of 4,063 tasks
The full pool is not intended to be evaluated for every model, and ProgramDistill-300 serves as our default evaluation suite. Its depth-stratified selection across all 26 applications enables comparisons at approximately 7% of the full cost. More importantly, we view the ability to generate a large pool of verifiable SWE tasks at scale as a key strength of ProgramDistill, especially as scalable task and environment generation becomes increasingly important for advancing coding agents. The larger pool adds behavioral coverage that researchers can draw on as their needs and budgets allow, rather than a cost that every evaluation must incur.
[W2] Copying the behavior of other applications
Our goal is not to encourage unauthorized copying but to evaluate reference-guided software engineering, where agents discover and implement functionality by interacting with a working application rather than relying solely on written specifications. This emerging direction is also reflected in ProgramBench [1], and ProgramDistill extends it to interactive, stateful web applications. As noted in our ethics statement, such capabilities have legitimate uses in maintenance, migration, testing, and authorized reconstruction, and our experiments use self-contained benchmark applications.
[1] Yang, John, et al. "Programbench: Can language models rebuild programs from scratch?." arXiv preprint arXiv:2605.03546 (2026).
[W3] Scope of reference-guided tasks
We view the reference-guided setting as a strength rather than a limitation. In practice, developers often infer expected functionality from existing products, prototypes, or earlier versions rather than from explicit written instructions. ProgramDistill captures this workflow by requiring agents to discover, implement, and validate behavior through interaction with a working reference, complementing issue-driven SWE benchmarks that state the desired behavior explicitly.
[Q1] How is code leakage prevented?
Agents cannot access the reference implementation, and their submissions show almost no verbatim overlap with the original source.
Appendix I.7 shows that all attempts to access these channels were blocked. We also compared all 36 full-application reconstruction submissions with their original source code. Only 0.3% of substantive generated lines matched the original verbatim, close to the 0.1% matched against unrelated applications, and the longest identical block was three lines. The rate was similarly low for long-standing open-source projects (0.4%). Matches were mostly generic CSS or visible interface text, whereas an original source file matched all 112 of its lines under the same procedure. We have added this analysis, including the matching procedure, to the new Appendix K.
[Q2] UI fidelity and complexity of computer-use operations
ProgramDistill evaluates behavioral fidelity rather than visual similarity, and its tasks involve a broad range of browser interactions. Colors, fonts, and layout are not currently scored, and visual fidelity could be added through screenshot- or layout-based checks, as noted in our Limitations paragraph.
Our Playwright-based browser helper supports keyboard shortcuts, scrolling, hover, drag-and-drop, and file uploads beyond basic clicking and typing (Appendix C). ProgramDistill-300 contains 552 replay traces with 5,939 actions (median 6 per trace, maximum 101), and 32% of traces include interactions beyond click, type, and key press. Press-and-hold gestures and device inputs such as microphone recording are not yet supported, but they extend the interaction interface rather than limit the task-generation approach.
[Q3] Extension to mobile app development
We agree that mobile development is a natural extension. The core mine–craft–patch approach carries over, since mobile platforms provide analogous mechanisms such as accessibility trees and emulator snapshots. Adapting it raises challenges, including longer build cycles, complex gestures, device-dependent inputs, and nondeterminism from animations, which we leave as future work.
We hope this response addresses the reviewer’s concerns. We would be happy to answer any further questions.