acceptodds
Under review as a conference paper at ICLR 2027

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

Abstract

Coding agents are usually evaluated on tasks whose desired behavior is stated in an issue, instruction, or test. In web development, however, the specification often exists only as working software, such as a prototype or a comparable product. We study reference-guided software engineering, in which an agent discovers missing behavior by interacting with a live reference whose source is hidden and implements it in an incomplete codebase. We introduce ProgramDistill, a benchmark in which a recorded browser workflow of a running application serves as both a task and its verifier. Its automated task generation pipeline, mine–craft–patch, deletes the code behind each workflow and replays the workflow to check the repair, so no issues or human-written tests are needed. Masking several dependent behaviors controls restoration depth, yielding 4,063 tasks from 26 applications. Across nine frontier agents, performance drops sharply with depth. GPT-6 Astra falls from 100% at depth 1 to 64.0% at depth 8, and every other agent retains less than half of its depth-1 performance. When rebuilding entire applications, the best agent passes only 49.2% of end-to-end workflows. Most unrecovered behaviors (59.2%) were never observed in the reference, suggesting that exploration and validation, not only code generation, are key bottlenecks.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Software Engineering Benchmarks, LLM Agents, Behavioral Reconstruction, Interactive Web Applications

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BTvtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a new method for automatically constructing software engineering benchmarks, scoped to the Web Development domain. The authors build a mine–craft–patch pipeline for automatic task and evaluation generation: mine discovers replayable behaviors, craft masks the corresponding code, and patch evaluates coding agents. Across 26 apps, 1,975 behaviors are mined and 4,063 tasks are constructed without human intervention. Atomic tasks are composed into cumulative tasks to test long-term ability, and the success rate drops dramatically as tasks accumulate.

Critical strengths

The paper's concept is easy to follow.

The work introduces a new methodology for automatically generating web development benchmarks.

The experiments cover a wide range of setups.

Critical weaknesses

4,063 tasks make evaluation prohibitively expensive.

The tasks appear to encourage copying the behavior of other web applications (the authors acknowledge this in the ethics statement).

The reference-guided tasks seem limited in scope.

Questions for authors
  1. How is code leakage prevented? The selected apps appear to be open source.

  2. The app behavior is monitored — but what about the UI (colors, fonts)? How complex are the computer-use operations required (e.g., holding a button to send a voice message)?

  3. What are your thoughts on adapting this method to the mobile app development domain?

3 comments

We thank the reviewer for recognizing the clarity of our approach, the novel automated benchmark-generation methodology, and the breadth of our experiments, and for the questions on cost, scope, leakage, UI fidelity, and mobile extension addressed below. All revisions are marked in blue in the revised manuscript.

 

[W1] Evaluation cost of 4,063 tasks

The full pool is not intended to be evaluated for every model, and ProgramDistill-300 serves as our default evaluation suite. Its depth-stratified selection across all 26 applications enables comparisons at approximately 7% of the full cost. More importantly, we view the ability to generate a large pool of verifiable SWE tasks at scale as a key strength of ProgramDistill, especially as scalable task and environment generation becomes increasingly important for advancing coding agents. The larger pool adds behavioral coverage that researchers can draw on as their needs and budgets allow, rather than a cost that every evaluation must incur.

 

[W2] Copying the behavior of other applications

Our goal is not to encourage unauthorized copying but to evaluate reference-guided software engineering, where agents discover and implement functionality by interacting with a working application rather than relying solely on written specifications. This emerging direction is also reflected in ProgramBench [1], and ProgramDistill extends it to interactive, stateful web applications. As noted in our ethics statement, such capabilities have legitimate uses in maintenance, migration, testing, and authorized reconstruction, and our experiments use self-contained benchmark applications.

[1] Yang, John, et al. "Programbench: Can language models rebuild programs from scratch?." arXiv preprint arXiv:2605.03546 (2026).

 

[W3] Scope of reference-guided tasks

We view the reference-guided setting as a strength rather than a limitation. In practice, developers often infer expected functionality from existing products, prototypes, or earlier versions rather than from explicit written instructions. ProgramDistill captures this workflow by requiring agents to discover, implement, and validate behavior through interaction with a working reference, complementing issue-driven SWE benchmarks that state the desired behavior explicitly.

 

[Q1] How is code leakage prevented?

Agents cannot access the reference implementation, and their submissions show almost no verbatim overlap with the original source.

  • No reference-source access. Pre-mask Git history is removed and derived artifacts are regenerated.
  • No internet access. Network egress is disabled, so upstream repositories cannot be downloaded.
  • Protected grading. Grading artifacts are inaccessible, and the reference is reachable only through the browser helper.

Appendix I.7 shows that all attempts to access these channels were blocked. We also compared all 36 full-application reconstruction submissions with their original source code. Only 0.3% of substantive generated lines matched the original verbatim, close to the 0.1% matched against unrelated applications, and the longest identical block was three lines. The rate was similarly low for long-standing open-source projects (0.4%). Matches were mostly generic CSS or visible interface text, whereas an original source file matched all 112 of its lines under the same procedure. We have added this analysis, including the matching procedure, to the new Appendix K.

 

[Q2] UI fidelity and complexity of computer-use operations

ProgramDistill evaluates behavioral fidelity rather than visual similarity, and its tasks involve a broad range of browser interactions. Colors, fonts, and layout are not currently scored, and visual fidelity could be added through screenshot- or layout-based checks, as noted in our Limitations paragraph.

Our Playwright-based browser helper supports keyboard shortcuts, scrolling, hover, drag-and-drop, and file uploads beyond basic clicking and typing (Appendix C). ProgramDistill-300 contains 552 replay traces with 5,939 actions (median 6 per trace, maximum 101), and 32% of traces include interactions beyond click, type, and key press. Press-and-hold gestures and device inputs such as microphone recording are not yet supported, but they extend the interaction interface rather than limit the task-generation approach.

 

[Q3] Extension to mobile app development

We agree that mobile development is a natural extension. The core mine–craft–patch approach carries over, since mobile platforms provide analogous mechanisms such as accessibility trees and emulator snapshots. Adapting it raises challenges, including longer build cycles, complex gestures, device-dependent inputs, and nondeterminism from animations, which we leave as future work.

 

We hope this response addresses the reviewer’s concerns. We would be happy to answer any further questions.

Dear Reviewer BTvt,

As the discussion period closes soon, we would appreciate knowing whether our responses have addressed your questions.

  • Evaluation cost: The 4,063 tasks are not meant to be run in full every time. They form a pool that can be drawn on for different purposes. ProgramDistill-300 supports standard comparison at about 7% of full cost, and rankings among statistically distinguishable models hold on a fresh application-balanced sample (Appendix M). Targeted evaluations by application or depth, as well as training data, can be built from the same pool. This flexibility is a direct benefit of scalable automatic generation.
  • Code leakage: Both source access and network access are blocked, and submissions match the original source on only 0.3% of lines (Appendix I.7, Appendix K).
  • UI fidelity: We deliberately score behavioral fidelity with deterministic checks for reproducible and meaningful grading of web application functionality. Visual similarity would require relying on VLM judges, so we excluded it, but it can be added as a separate axis if needed (Limitations).
  • Interaction complexity: The browser helper supports a broad range of interactions, including drag-and-drop, hover, and file uploads (Appendix C).

ProgramDistill builds both tasks and verifiers from any working application, so it extends naturally across web applications and to platforms such as mobile that expose analogous interfaces like accessibility trees. We would be happy to address any remaining questions.

Area_Chair_YGhVOpenReview ↗

Reminder from AC

Dear Reviewer BTvt ,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Reviewer_A5o8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProgramDistill proposes a pipeline for converting interactive web applications into verifiable, reference-guided software engineering tasks. It mines reproducible interaction traces, removes the functionality needed to reproduce selected behaviors, and constructs repair tasks with executable verification. It also combines related tasks to vary restoration depth.

The resulting dataset contains 4,063 tasks across 26 applications. The paper evaluates nine models on a 300-task partial reconstruction subset and three models on full application reconstruction across 12 applications. The experiments examine performance across restoration depths and analyze agents’ observation and implementation failures.

Critical strengths

A useful task formulation grounded in executable applications. Asking agents to recover functionality by interacting with a running reference application provides an interesting setting for studying how agents connect observed behavior with implementation. The prerequisite structure also allows evaluation of tasks involving multiple dependent functionalities.

A systematic construction pipeline with executable checks. The pipeline checks that the reference behavior is reproducible, that masking breaks the intended functionality while preserving tested prerequisites, and that a reference patch restores it. These checks provide a useful foundation for constructing tasks at scale.

Critical weaknesses

Independent validation of task and verifier quality is insufficient. Task construction relies heavily on LLMs, yet the paper does not report an independent human audit of the resulting tasks. The automated checks establish internal consistency, but they do not fully establish whether task descriptions are adequate, removed functionality corresponds to meaningful development work, or the verifiers distinguish correct implementations from patches that merely satisfy the recorded interaction trace. This leaves uncertainty about what the reported success rates measure.

The usefulness of the task generation approach is not sufficiently established beyond benchmarking. I view the pipeline as an approach to scaling software engineering tasks and environments. From this perspective, the central contribution should include evidence about the quality, diversity, and usefulness of the generated tasks. The current experiments primarily report model performance and failure patterns. Training on generated tasks and evaluating transfer to independently constructed tasks would provide stronger evidence of their broader value. Such training experiments are not mandatory for a benchmark paper, but their absence makes independent validation of the benchmark’s quality and measurement value particularly important.

Questions for authors
  1. Clarification on independent validation: Was any independent human audit conducted? If so, what proportion of tasks was examined, what criteria were used, and what issues were found regarding task descriptions, meaningfulness, and verifier correctness?

  2. Clarification on verifier validity: Beyond replaying the recorded interactions, have the verifiers been tested against plausible but incorrect implementations, alternative inputs, or alternative interaction paths? This would help establish whether passing reflects functional correctness beyond the recorded trace.

  3. Clarification on contribution and scope: Is the intended contribution primarily an evaluation benchmark, or a general approach to generating tasks for both evaluation and learning? If the latter, is there evidence of downstream learning benefits? I regard a training-and-transfer experiment as a useful extension rather than a prerequisite for a contribution explicitly limited to benchmarking.

3 comments

We thank the reviewer for recognizing our task formulation and systematic construction pipeline, and for the suggestions on independent validation and the scope. All revisions are marked in blue in the revised manuscript.

 

[Q1] Was independent human audit conducted?

Following the reviewer’s suggestion, we independently validated task descriptions, meaningfulness, and verifier correctness.

(1) Task descriptions. Five evaluators, four of them non-author PhD students, reviewed 200 tasks (4.9% of the pool) stratified across all 26 applications and both task types. They judged whether the target behavior was identifiable from the instruction and reference, and whether it matched the code removed by the mask. Of the 200 tasks, 196 (98.0%) were rated consistent, 4 (2.0%) partially consistent, and none inconsistent. In the four partial cases, the instruction was ambiguous or omitted seeded content removed by the mask, but the missing information was observable in the reference, and GPT-6 Astra solved all four. Because the reference app serves as the specification, such gaps can be resolved through interaction.

(2) Meaningfulness. The same audit confirmed that each behavior matched the code removed by the mask, so each task asks the agent to rebuild a coherent, user-facing behavior rather than an arbitrary code fragment. This is substantial development work. The median atomic task requires restoring 69.5 lines across three files, and the median cumulative task 435 lines across seven files, while the largest 10% of cumulative tasks span 21 or more files.

(3) Verifier correctness. GPT-6 Astra and human reviewers jointly examined all 977 failed behaviors, as detailed in our response to Reviewer 93ZM [Q3], and found no false rejection. [Q2] below covers false acceptance.

We have added the audit to the new Appendix O and the task-size analysis to Appendix J.

 

[Q2] Have the verifiers been tested against alternative inputs or incorrect implementations?

We tested the verifier against both alternative inputs and incorrect implementations. No model submission was falsely accepted under altered inputs or initial states, and the verifier rejected 76.0% of deliberately injected faults.

(1) Alternative inputs and initial states. To test whether passing model solutions depended on the specific values in the recorded traces, for 100 masked behaviors we changed the typed text or the initial application state, keeping only variants that pass on the reference. The gold code passed all 116 resulting traces, and re-running each trace on every model solution that had passed its original version gave 304 evaluations, all of which passed.

(2) Incorrect implementations. Beyond model solutions, we stress-tested the verifier by injecting five types of deliberately designed faults into the gold code, including faults designed to evade detection. The verifier rejected 218 of 287 incorrect variants (76.0%). Because it checks persisted state and side effects, it caught 87–90% of faults that skip a save or modify the wrong record.

ModificationRejected
Remove the data-saving call54/62 (87.1%)
Update the wrong record43/48 (89.6%)
Omit a notification, redirect, or download57/68 (83.8%)
Save an incorrect value28/44 (63.6%)
Display expected text without performing the operation36/65 (55.4%)

All 69 misses lie outside what the recorded traces check. They appear only after a reload or a later step (35) or in a value the trace does not check (34). Adding checks derived from the reference can catch them without introducing false rejection. We have added both audits to the new Appendix N and discussed this limitation and trace extension as future work in the Limitations paragraph.

 

[Q3] The scope of our contribution

Our primary contribution is an evaluation benchmark and a scalable pipeline for generating verifiable SWE tasks, and Appendix J now analyzes the diversity and usefulness of the generated pool.

  • Task diversity. The 4,063 tasks span 26 applications in five domains, built with five frontend and five backend frameworks and totaling 8,445 source files. They target 1,519 distinct application-level behaviors, whose replay traces use 11 browser action types.

  • Evaluation usefulness. On ProgramDistill-300, 238 of 300 tasks distinguish among the nine evaluated models, and many separate even the strongest models, such as the 37 tasks solved only by GPT-6 Astra. All 24 tasks that no model solves are cumulative, so restoration depth provides a controllable difficulty axis.

  • Potential for training. With executable environments, automatic rewards, and controllable difficulty, the tasks are a promising source of training curricula, and training on them and evaluating transfer is a natural next step.

 

We hope this response addresses the reviewer’s concerns. We would be happy to answer any further questions.

Dear Reviewer A5o8,

As the discussion period closes soon, we would appreciate knowing whether our responses have addressed your concerns.In summary, ProgramDistill is a reference-guided SWE benchmark generated at scale across a wide range of difficulty levels, and we additionally confirmed its task quality and verifier accuracy through independent audits.

  • Task quality: An independent audit of 200 tasks rated 98.0% consistent (2.0% partially consistent, none inconsistent). Even the partially consistent cases were resolvable through interaction with the reference, since the executable reference itself serves as the specification (Appendix O).
  • Verifier accuracy: There was no false acceptance on held-out traces with altered inputs and initial states, and the verifier rejected 76.0% of deliberately injected faults, with all misses lying outside what the recorded traces check. Because the verifier checks only properties the agent can observe in the reference, a review of all failed behaviors found no false rejection either (Appendix N).
  • Diversity and discriminative power as a benchmark: The tasks span 5 domains, 5 frontend and 5 backend frameworks, and 1,519 distinct behaviors. 238 of 300 tasks distinguish among models, and some remain unsolved by any model (Appendix J).
  • Potential for training: Executable environments, deterministic rewards, and controllable difficulty make the tasks directly usable as training curricula.

The same pipeline can keep producing such tasks without human-written issues or tests, providing a scalable foundation for both evaluation and training. We would be happy to address any remaining questions.

Area_Chair_YGhVOpenReview ↗

Reminder from AC

Dear Reviewer A5o8,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Reviewer_93ZMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ProgramDistill studies reference-guided software engineering: an agent receives an incomplete web application and must recover behavior by interacting with a working reference whose source is hidden. The mine–craft–patch pipeline records replayable behavior traces, removes implementation portions corresponding to validated behaviors, and creates atomic and cumulative repair tasks with gold-patch positive controls.

The resulting collection contains 4,063 tasks from 26 applications. Nine models are evaluated on a 300-task, depth-stratified subset; a second study asks three models to reconstruct applications from minimal scaffolds. The paper reports declining task success as more dependent behaviors are masked and attributes a large fraction of full-reconstruction failures to behaviors never inspected in the reference.

Critical strengths

Careful positive and negative construction controls. Atomic masks must preserve prerequisite behaviors while breaking the target, and reversing a cumulative mask must restore complete lineage replay (Secs. 3.2–3.3). These controls reduce the incidence of malformed, unsolvable tasks.

A meaningful workload beyond conventional issue repair. Reference interaction, source edits, and iterative verification are coupled in one task. The 26-application, 4,063-task corpus and the two reconstruction settings support several forms of analysis (Secs. 4.1–4.3).

Good operational detail. Deterministic clock/state reset, restricted reference access, protected test artifacts, trace replay, and supplied mining/crafting prompts improve reproducibility (Appendices B–G).

Critical weaknesses

The central depth result is partly mechanical. Binary success for a depth-r task requires every masked target in its dependent lineage to pass (Eq. 1). Even if per-target success remained unchanged, the probability of passing all r targets would decline as r grows. Moreover, code lines to restore and trace actions increase by 9.3× and 10.7× from depths 1 to 8 (Fig. 8). The context-length diagnostic (Appendix I.3) does not isolate these factors. The paper needs a depth-normalized, per-target analysis or a matched comparison to establish an additional compositional difficulty rather than a conjunction/size effect.

Fixed replay is a narrow behavioral oracle. Tests replay specific selectors, labels, signals, inputs, and seeded state. The full-reconstruction instructions explicitly require matching exposed selectors (Appendix G.2). Correct alternative implementations can be penalized for different but functionally equivalent interaction structure, while implementations tailored to fixed examples may pass without generalizing. The paper acknowledges held-out input variation and adversarial oracle audits as future work (Sec. 5).

The main evaluation uses a small and deliberately depth-stratified fraction of the generated task pool. ProgramDistill-300 includes 300 of 4,063 tasks, with only 50 atomic tasks and 250 cumulative tasks selected to stratify depth (Sec. 4.1). Tasks share applications, prerequisite traces, and source fragments, so treating all 300 as independent would overstate the effective sample size. Multiple runs, application-level uncertainty, and broader evaluation would strengthen model comparisons.

The failure taxonomy is not independently validated. The interpretation that 59.2% of failures arise from unobserved behaviors (Fig. 9) is potentially valuable, but the AI-use statement says GPT-6 Astra was used for categorization. No human-adjudicated subset or agreement statistics are reported. The conclusion that exploration is the predominant bottleneck therefore depends on an unaudited classification process.

The “full-application” experiment has a narrower implementation substrate than its name suggests. It starts from a constrained scaffold with a prescribed app.js request-handler contract and context.state (Appendix G.2). The reported 49.15% cumulative score reflects replayed workflows on twelve selected applications, not equivalence of their complete production architectures, persistence systems, or untested behavior.

Questions for authors

Can the authors report success per masked target, conditioned on its prerequisites, and compare observed depth-r success against a null model that composes independent per-target success probabilities? Can mask size and workflow length be controlled?

What is the false-rejection rate on independent, behaviorally correct repairs with different selectors or DOM structure, and the false-acceptance rate under held-out inputs or mutated application state?

How were the 977 failed behaviors categorized? Please provide instructions, a blinded human-labeled sample, and inter-rater or human–model agreement for the four categories.

Are the nine-model rankings stable over multiple trajectories and over a broader, application-balanced sample of the 4,063 tasks? Please separate atomic and cumulative outcomes in the primary analysis.

4 comments

We thank the reviewer for recognizing our careful task-construction controls, the meaningful reference-guided SWE setting, and the reproducibility of our evaluation, and for the detailed questions that guided the additional experiments below. All revisions are marked in blue in the revised manuscript.

 

[Q1 / W1] Is the depth effect just the product of per-target failures?

The depth effect goes beyond the product of per-target failures. GPT-6 Astra restores nearly every target in isolation, yet as depth grows, it stops early, before restoring all of them.

As the reviewer suggested, we measured per-target success conditioned on intact prerequisites and compared observed depth-r success against an independence null. For every masked target in the 250 cumulative tasks of ProgramDistill-300, we evaluated the corresponding atomic task, yielding 612 atomic controls. Because running them for all nine models would cost roughly $21k, we used the strongest model, GPT-6 Astra (high). Astra solved 99.3% of the controls, so the independence baseline stays at or above 88% at every depth. Observed success falls well below it, with the gap reaching 23–30 pp at depths 6–8.

Table R1. Independence baseline vs. observed success (GPT-6 Astra).

DepthProduct baselineObserved binary success
2100.0%95.6%
3100.0%91.1%
497.5%80.0%
5100.0%88.6%
693.3%70.0%
793.3%63.3%
888.0%64.0%

Each atomic control already includes the full work of restoring its target, so Astra could pass most combined tasks simply by giving each target the effort it receives in isolation. Instead, per-target observation and editing effort decrease as depth grows, and more target files and mask-added stubs are left untouched (Fig. 8). Regarding mask size and workflow length, the atomic controls hold the per-target workload fixed, and Appendix I.3 shows that the effect persists among runs with similar context lengths. We have added the independence product baseline to Fig. 6(b) and the full experiment to the new Appendix I.4.

 

[Q2 / W2] Can the fixed-replay verifier falsely reject or falsely accept?

Our verifier avoids false rejection by grounding every check in agent-observable attributes, and no model submission was falsely accepted under held-out inputs or mutated state.

(1) False rejection. Every property the verifier checks or uses to locate elements appears in the agent’s browser observations of the reference, such as visible text, roles, accessible names, URLs, and persisted state. Replay selectors are built only from these attributes (e.g., role=button[name="Save"]), so the verifier does not depend on code organization, DOM nesting, CSS classes, or internal APIs. This avoids a common source of false rejection in SWE benchmarks, where tests check implementation details absent from the task specification [1]. Recent benchmarks address this by adding the expected names or interfaces to the task input [2], whereas in ProgramDistill the reference application abstracts away such details. By design, implementations that differ in code structure pass as long as they reproduce the agent-observable content or behavior.

Changing the agent-observable content or behavior itself, however, deviates from the specification rather than offering a functionally equivalent alternative, just as a dropdown in place of specified radio buttons does not follow a designer’s specification. The review of all 977 failed behaviors, detailed in our response to [Q3 / W4] below, found no correct implementation rejected because of a non-observable property.

 

(2) False acceptance. To test whether passing model solutions depended on the specific values in the recorded traces, we sampled 100 masked behaviors and created 116 held-out traces that change the typed input or the initial application state, keeping only variants that pass on the reference. The gold code passed all 116, and re-running each trace on every model solution that had passed its original version gave 304 evaluations, all of which passed. No model submission was therefore falsely accepted.

We further stress-tested the verifier with deliberately designed faults injected into the gold code rather than model solutions, as detailed in our response to Reviewer A5o8 [Q2]. The verifier rejected 76.0% of these faulty variants. The remaining 24.0% passed because their faults appear only in steps the recorded traces do not include, such as after a reload, or in values the traces do not check. Extending the traces with such follow-up checks would catch them. We have added these audits to the new Appendix N, and this limitation, with trace extension as future work, to the Limitations paragraph.

 

[1] OpenAI, "Why SWE-bench Verified no longer measures frontier coding capabilities," 2026.

[2] Deng, Xiang, et al. "Swe-bench pro" arXiv:2509.16941.

[Q3 / W4] How were the 977 failures categorized?

Our failure categorization combines repeated classification by GPT-6 Astra with human review, and a blinded human audit supports its reliability.

We ran three independent categorization runs with Astra, each required to cite evidence from submitted code, verifier reports, and recorded observations. We kept labels only when all three runs agreed with sufficient evidence, which held for 854 of the 977 behaviors (87.4%), and the authors manually reviewed the other 123. In the blinded audit, three non-author PhD students in CS, EE, and AI independently classified 90 cases without seeing Astra’s labels, and 83.3% agreed with Astra. For all 15 disagreements, re-examining the cited evidence supported Astra's label. We have added the decision rules, the full categorization prompt, and the human audit to Appendix L of the revised manuscript.

 

[Q4 / W3] Are the rankings stable, and are atomic and cumulative tasks separated?

The rankings were stable in all three requested analyses.

(1) Repeated runs and an application-balanced sample. Given the cost and the limited rebuttal period, we ran two additional evaluations on the top five models rather than all nine. We re-ran ProgramDistill-300 with a new seed, and since ProgramDistill-300 is balanced by depth, we also evaluated an application-balanced sample of 100 new tasks drawn four per application. On the re-run, scores changed by at most 4 pp. Order changed only between pairs that were already statistically indistinguishable. GPT-5.6 Sol and Grok 4.6 swapped on atomic tasks by one to two tasks, and Claude Sonnet 5 and Grok 4.6, which were 0.4 points apart on cumulative tasks, swapped in both new evaluations.

ModelAtomic (original)Atomic (re-run)Atomic (app-balanced)Cumulative (original)Cumulative (re-run)Cumulative (app-balanced)
GPT-6 Astra10010098.181.282.478.3
Claude Opus 596.098.094.463.262.465.2
GPT-5.6 Sol92.088.087.054.453.656.5
Grok 4.688.090.085.240.441.641.3
Claude Sonnet 580.080.083.340.839.234.8

 

(2) Application-level uncertainty. Even after accounting for application-level uncertainty, the ranks of the top three and bottom two models remain fixed. Using a cluster bootstrap over the 26 applications (10,000 resamples), we obtained the following cumulative rank intervals. The four models ranked 4–7, whose scores lie within 3.2 pp of one another, share a rank interval. The median Kendall τ between the original and bootstrap rankings is 0.89.

Rank intervalModels (cumulative score, 95% CI)
1GPT-6 Astra (81.2, 73.9–88.7)
2Claude Opus 5 (63.2, 51.9–75.3)
3GPT-5.6 Sol (54.4, 44.3–65.8)
4–7Claude Sonnet 5 (40.8, 31.1–51.6), Grok 4.6 (40.4, 30.5–51.8), GPT-5.3 Codex (38.8, 29.0–50.5), Gemini 3.7 Flash (37.6, 27.7–49.2)
8Gemini 3.6 Flash (23.2, 15.8–32.1)
9Gemini 3.1 Pro Preview (12.8, 7.5–19.6)

 

(3) Separating atomic and cumulative tasks. The revised manuscript reports atomic and cumulative scores separately with their confidence intervals. We have added all three analyses to the new Appendix M and refer to them from Sec. 4.2.

 

[W5] Scope of the full-application experiment

The setting reconstructs the full functionality of each application. Agents receive no application-specific source code and build every route, UI component, data model, and piece of business logic from scratch through reference interaction. The scaffold in Appendix G.2 standardizes only the execution environment that deterministic replay requires. The app.js contract lets the runtime manage ports, and context.state lets the verifier reset application state between replays, while none of the functionality being reconstructed is provided.

As discussed in [Q2 / W2], the internal architecture and persistence mechanisms the reviewer mentions are not observable through the reference, so they are not part of the specification and cannot fairly be required of the agent. We agree that the score reflects only the behaviors exercised by the recorded workflow tests, and we have added this limitation to the Limitations paragraph. We have also clarified the scaffold in Sec. 4.3.

 

We hope this response addresses the reviewer’s concerns. We would be happy to answer any further questions.

Dear Reviewer 93ZM,

As the discussion period closes soon, we would appreciate knowing whether our responses have addressed your concerns. We addressed each point with additional experiments and reflected them in the revised manuscript.

  • Depth effect: Observed success falls 23–30 pp below the independence baseline at depths 6–8, indicating compositional difficulty beyond a simple product of per-target failures (Fig. 6(b), Appendix I.4).
  • Verifier validity: There was no false acceptance on held-out traces with altered inputs and initial states. Because the verifier checks only properties the agent can observe in the reference, no false rejection was found either (Appendix N).
  • Failure taxonomy: We kept only labels on which three independent Astra runs agreed (87.4%), and the authors manually reviewed the remaining cases. A blinded human audit showed 83.3% agreement (Appendix L).
  • Ranking stability: Rankings were stable across a re-run, an application-balanced sample, and a cluster bootstrap. Atomic and cumulative scores are now reported separately (Appendix M, Sec. 4.2).
  • Full-app scope: Agents receive no functional source code and implement every route, UI component, data model, and business logic through interaction with the reference alone. The scaffold only standardizes the runtime required for deterministic replay, as clarified in Sec. 4.3.

These results show that ProgramDistill’s depth axis exposes a genuine capability limit in compositional restoration: agents that restore nearly every behavior in isolation fail when dependent behaviors must be restored together, and this gap is reproducible across runs and applications. We would be happy to address any remaining questions.

Area_Chair_YGhVOpenReview ↗

Reminder from AC

Dear Reviewer 93ZM,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their careful evaluations and many helpful suggestions.

 

Key Contributions

We would first like to clarify our main contributions:

  • ProgramDistill is the first benchmark that requires coding agents to reconstruct parts of or entire real-world web applications by interacting with a live reference app.
  • We propose a record-and-replay verifier that grades submissions against recorded observations of the live app, making the app itself the specification.
  • Using this verifier, we build a pipeline that automatically generates many tasks of varying difficulty.
  • We release 4,063 diverse tasks with analyses for the research and benchmarking communities.

 

Key Additional Experiments

  • Our core finding, that agents stop early or fail to scale their effort as task difficulty grows, is not simply the compounded effect of per-target failures. (Reviewer 93ZM)
  • Agents implement what is observable in the reference app, and grading uses the same observations, so false rejections are prevented by design. A separate audit also found none. (Reviewer 93ZM, A5o8)
  • We found no false acceptances (e.g., hard-coding) in any agent submission, and the verifier caught 76% of deliberately defective submissions. The rest can be caught by recording more behavior, which we leave for future work. (93ZM, A5o8)
  • An independent human audit of task quality and diversity found that 98% of task descriptions fully describe the missing state. For the remaining 2%, observing the app fills the gaps, since the app is the primary specification. Tasks stay well-defined even when the description is incomplete or missing, which is a key strength of our approach. (Reviewer A5o8)
  • The large task pool (4,063) lets users build subsets for their needs, evaluate agents from multiple angles, and reuse the tasks for future training. (Reviewer A5o8, BTvt)

Our individual responses cover the other questions, including ranking stability, code leakage, and extension to mobile. We have incorporated all new results into the revised paper. Due to the 9-page limit, most details are in the appendix, and we plan to move them into the main text in a future revision.

 

Together, these additional results further show that our reference-guided, record-and-replay setting reliably evaluates agent capability and behavior from a new angle that more closely reflects how human app developers work. It is also robust to incomplete specifications and false rejections, and useful in many ways.

We thank the reviewers again and are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.