acceptodds
Withdrawn from ICLR 2027

UntilItPasses: Benchmarking Repository Generation Against a Verifiable Goal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic coding capabilities have expanded from resolving individual issues to generating entire repositories. Existing repository benchmarks often evaluate agents in a single run and identify premature exit and unhandled edge cases as the leading failure patterns. However, real-world agentic coding involves feedback and multiple rounds of revision. To bridge this gap, we introduce UntilItPasses, an evaluation of 46 challenging repository-construction tasks that replaces the single submission with a minimal feedback loop. Each agent builds incrementally against a natural-language specification, commits its changes throughout development, and may make a feedback call at any point. Feedback reports only the fraction of hidden tests that pass, providing a milestone signal without revealing individual test cases. The agent continues working until it receives a perfect result or reaches a calibrated budget equivalent to six hours. Across twelve agent systems, 2.5% of runs exit on a perfect score. Our analysis of commit histories and feedback-call records traces agents’ progress through initial implementation and later repair, showing rapid early gains and persistent implementation gaps. Selected trajectories show agents using scalar feedback to broaden self-tests and revise mistaken assumptions shared by their code and tests. Model efficiency rankings also change with the required completion level. These findings highlight feedback-guided self-correction as a key capability to develop and evaluate in autonomous coding agents.

Primary area:
datasets and benchmarks
Keywords:
Repository Generation, Coding Agents, Agentic Coding, Long horizon Coding benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VBATOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper takes 46 Python repository-generation tasks from DeNovoSWE (specification, hidden tests, images) and changes the protocol: the agent may call repo-feedback at any time to get the scalar hidden-suite pass rate, is forced to continue after voluntary exits, and runs until pass rate 1.0 or a token budget equal to six hours at the model's calibrated generation speed. Commits are replayed in an isolated evaluator to give a best-pass-rate trajectory over budget. The main metric measures how many tasks reach each pass-rate level, with higher levels weighted more, and how this grows over the budget. Twelve agent systems are each run once. The authors argue that early gains come from implementing the spec and later gains from feedback-guided repair, and that a single-pass run scores lower than the feedback loop. Model rankings also change with the required completion level.

Critical strengths
  1. The paper studies whether a coding agent keeps improving after its first self-declared completion, instead of stopping early. Forcing continuation and replaying every commit gives a progress curve over the whole run, and makes premature exit visible as a failure mode.
  2. The experiments cover twelve agent systems with isolated evaluators and network blocked, so upstream source cannot be retrieved. The analysis of gain composition, and of how rankings change with the target pass-rate threshold, is informative.
Critical weaknesses
  1. The protocol tunes on the test set. Offline scoring uses the same fixed test suite as online feedback, so there is no held-out split. Feedback calls are unlimited, which is like submitting again and again to fit a Kaggle public leaderboard, and the final score is optimistically biased. This setup also does not match real development. A developer, human or agent, does not hold a complete and certainly correct test suite as an oracle before writing the program, and does not develop with that suite as the optimization target.
  2. The budget analysis uses time as the unit, so the budget depends on serving speed. DSV4.1 Flash at 87 tok/s gets 1,879,200 tokens, Kimi K3 at 26.71 tok/s gets only 576,936. The same model on faster hardware would get more budget. The current comparison and rankings are therefore based on an unfair unit.
Questions for authors
  1. Beyond using the oracle test pass rate as a verified signal to guide repository generation, which is an unrealistic setup, what is the benchmark's difference and contribution compared to previous work such as ProgramBench?
  2. Section 3.1 states that the documentation-writing agent's prompt "explicitly requires test-relevant signatures, exceptions, return fields, constants, and defaults to be stated." If the NL specification already carries this information, why is the oracle test pass rate needed as the feedback signal? Why not use self-generated tests to steer the search?
Reviewer_HKPoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces UntilItPasses, an evaluation protocol for repository generation built on 46 Python tasks reused from DeNovoSWE. Instead of a single submission, agents develop incrementally, commit throughout, and may query repo-feedback, which returns only the hidden-test pass rate. Each run continues, with forced continuation on early exit, until the agent reaches a perfect score or exhausts a 6-hour budget calibrated to each model's generation speed. Every commit is replayed in isolation to build best-pass-rate trajectories, and runs are summarized by a weighted reach-rate AUC.

Across 12 agent systems (552 trials), only 2.5% of runs reach a perfect score. The authors report that leadership shifts with budget stage and completion threshold. Based on LLM annotations, they attribute early gains mainly to specification-driven function additions and later gains mainly to feedback-guided revisions. The results are supported by sensitivity analyses, case studies, and a single-model single-pass comparison.

Critical strengths
  1. Trajectory-level evaluation reveals differences that a final score hides. Commit replay on a common budget axis shows that models excel at different stages and completion targets.
  2. The evaluation pipeline is carefully executed. The pipeline includes:
  • isolated replay scoring;
  • speed-calibrated budgets with a conservative error bound (64 of 66 pairwise orders preserved);
  • seed repetitions that report task-level variance;
  • case studies that explicitly separate online feedback, local tests, independent scoring, and author diagnostics.
Critical weaknesses
  1. The central claim lacks causal support. The paper emphasize feedback-guided self-correction. However, scalar feedback is always confounded with forced continuation and a larger budget. The only control uses a single model and simultaneously changes feedback, continuation, budget, and score selection. As a result, it is impossible to tell whether the gains come from the feedback signal or simply from preventing premature exit. The attribution in Section 5.1 does not close this gap:

    • it relies on DSV4.1 Flash annotations based on temporal links between feedback readings and subsequent edits;
    • the annotator can see the scores;
    • the annotations are not validated against human labels.

    The resulting category shares therefore cannot support the claim that feedback drives late-stage progress.

  2. The setting's positioning and real-world relevance are unclear. The feedback is perfectly reliable, fully aligned with the goal, and covers the entire requirement set. This is far more idealized than the partial, noisy, or misaligned feedback typical in practice. In Case E.4, the agent disables functionality to observe score changes, which is effectively probing the hidden test suite. The paper presents this as a positive example but neither discusses nor quantifies such oracle exploitation. The paper also claims relevance to real development without specifying which scenario it models, for example porting projects with complete acceptance tests or verifiable-reward RL environments. The scope of its conclusions is therefore unclear.

Questions for authors
  1. Can you add "continuation without feedback" and "feedback without continuation" ablations on at least two or three models? If continuation without feedback performs close to the full setting, would the paper's core narrative need to change?
  2. How many feedback calls does each model make on average, and how are they distributed over the budget? What fraction are probing actions, such as disabling features to measure the score change? How do you distinguish using the signal to correct understanding from exploiting the oracle to infer test coverage?
  3. Scaffolds, reasoning tiers, and speed-calibration sources differ across systems, and two models use proxy speeds from related models. To what extent does the leaderboard reflect the models themselves rather than these configuration differences?
  4. Task difficulty is calibrated against DeepSeek-V4-Pro's pass rates. Have you checked whether this affects the ranking of DeepSeek-family or other models? Did you run any contamination check against the upstream implementations, such as similarity between generated code and the original repositories?
Reviewer_jRs6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a benchmark of iterative repo generation using the fraction of hidden tests passed as scalar feedback. Across 12 agents, 2.5% of runs pass everything.

Critical strengths

The setup is useful for testing long coding work. Agents have a clear goal and can check their progress. This is a useful setting for studying whether agents can keep improving an incomplete repo. Intermediate progress can also be measured by scoring the intermediate commits.

Critical weaknesses
  1. The value of feedback isn't really shown. The single-pass baseline removes the feedback tool but also removes forced continuation. So it shortens the run. And the final workspace is scored instead of the best commit. The increase from 0.4 to 0.485 could come mainly from extra time and refusing to the let the agent quit. The experiment that would isolate feedback is same six hours, same forced continuation, and no repo feedback.
  2. The accuracy of the behavioral labels is questionable. DSV4.1 Flash labels all lifts, including whether feedback prompted each change. The paper doesn't measure agreement with human labels. The instructions say that simply following a feedback call isn't enough to count as feedback-driven but we don't know how reliable the model follows that rule, especially given that DSV4.1 Flash is not even the top performing model in this ranking. So the claim that feedback accounts for most later gains cannot be supported.
Questions for authors
  1. Could you compare runs with and without repo feedback, but keeping the same calibrated 6-hour budget, forced continuation, and scoring rule? This would be a valuable ablation to isolate the effects of feedback.
  2. What did the target annotation checks involve? How many labels were checked, who checked them, and what did they find? Did humans independently check whether feedback actually prompted the changes labeled as feedback-driven?
  3. How do the scores and rankings change if you score the final workspace instead of the best commit? How often do agents lose earlier gains by the end of a run?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.