acceptodds
Withdrawn from ICLR 2027

From Dead Code and Static Page to Working Engines: Software Revival with Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents bring two interesting questions into software engineering: can they bring scientific and commercial software that no longer runs back to a working state without changing its method, and can they build the core engine of commercial or industrial software from an open specification alone? We introduce ReviveBench, a benchmark that tests whether coding agents can restore nonfunctional scientific and commercial software while preserving its underlying methods, and reconstruct industrial software engines from open specifications alone. ReviveBench combines two complementary task families with hidden verifiers calibrated against native execution environments or industrial-grade reference systems. The revival family comprises ten tasks spanning dependency incompatibilities with Keras 3 and NumPy 2, as well as a GPU-based foundation model; every starting workspace fails verification. The strongest evaluated model restores all ten. In contamination-control experiments, line similarity to the original implementations drops from 0.51–0.96 to 0.03–0.44 without reducing any evaluated model’s pass rate. On repositories created after the cutoff, the strongest model succeeds in eight of nine runs, while weaker models achieve fewer successes. The reconstruction family comprises thirteen clean-room tasks covering statistics, SPICE circuit simulation, programmable logic controllers, finite elements, 2D and 3D CAD, logic synthesis, chip place-and-route, computational fluid dynamics, and enterprise and manufacturing systems. These engines are evaluated using numerical tolerances, formal proofs, or zero-tolerance transactional invariants; two frontier models satisfy the verification criteria for all thirteen. Crucially, constructing and validating the benchmark uncovers 28 verifier defects, including 24 false negatives and two false positives—more failures attributable to verification than to the agents themselves. Our contribution is both a benchmark for software revival and industrial engine reconstruction, and empirical evidence that verifier reliability can become the dominant evaluation bottleneck as agent capabilities improve. These findings demonstrate substantial capabilities under the tested specifications while establishing verifier validation as a central requirement for trustworthy agent evaluation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
coding agents, software engineering, software design

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oU92OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces ReviveBench, a benchmark for coding agents graded by hidden executable verifiers. It contains ten software revival tasks, which restore nonfunctional scientific software affected by dependency rot, deleted modules, legacy builds, or post-cutoff changes. It also contains thirteen clean-room reconstruction tasks, which implement bounded functionality of engineering engines from public specifications. The revival tasks are evaluated on three Anthropic models, and the reconstruction tasks on eight models from three vendors. The strongest model passes all ten revival tasks in at least one run, and two models pass all thirteen reconstruction tasks. The paper also audits its own verifiers, reports 28 defects (24 false negatives and 2 false positives), and proposes three checks for verifiers. Most model–task cells contain one or two runs, and the best run is reported.

Critical strengths

S1. Task domains with checkable semantics. The benchmark targets domains whose correctness is externally defined and checkable: SAT-based equivalence for synthesized netlists, NIST StRD certified values for statistics, and integer-cent ledger invariants for transactional engines. It also frames revival so that it must reproduce the original software's outputs (e.g., PyMOL structure metrics, QuPath cell detection), not merely make the code run.

Critical weaknesses

W1. The benchmark is neither documented nor designed as a measurement instrument.

The paper describes the tasks but not how they were built. It does not explain how the packages and domains were selected, who wrote and reviewed the specifications, how checks and pass thresholds were set, how the transactional reference implementations were validated, or how the post-cutoff repositories were chosen. Section 9 says only that thresholds were "chosen by us and anchored to reference measurements." This matters because the paper's own audit shows that errors arise during construction: all 28 defects were found while the suite was being built and audited, and one false positive invalidated a claim in an earlier version. Without a description of the construction process, readers cannot judge whether the remaining tasks are trustworthy, and others cannot extend the benchmark.

The benchmark also lacks a defined construct. Each of the ten revival tasks exercises a different skill, such as framework porting, C++ build repair, Java build migration, CUDA kernel compatibility, or reimplementation from a paper. Each skill has only one to three instances, so none is measured reliably. The two families also use different model sets (three vs. eight models) and different scoring rules (success in any run vs. the best run's score against task thresholds), and the paper provides no aggregate score.

W2. The evidence is too thin to support the headline claims.

The suite has 23 nominal tasks, with one task per domain in the reconstruction family. This is far fewer than agentic benchmarks with similarly large tasks, such as Commit0 (54 libraries), MLE-bench (75 competitions), and CORE-Bench (270 tasks). In reconstruction, every model passes MES and PLM, and CFD does not measure numerical-solver capability, leaving ten informative tasks. In revival, all three models pass four of the ten tasks. The benchmark is also saturated at the frontier on release: two models pass all thirteen reconstruction tasks, and one passes all ten revival tasks. The authors themselves note that "saturated tasks have limited power to reveal improvements in accuracy" (Appendix B).

Most cells contain one or two runs, only the best run is reported, and the number of attempts differs across models. In revival, Fable 5.1 has 34 runs, compared with 11 for Sonnet 5 and 10 for Haiku 4.5. The 10/10 vs. 8/10 gap between Fable 5.1 and Sonnet 5 rests on two tasks. On those tasks, Sonnet 5 had one run each, while Fable 5.1 succeeded in 2 of 6 runs on one task and 2 of 3 on the other. The claim that the contamination controls "provide evidence of capability beyond verbatim recall" rests on one obfuscated task and three post-cutoff repositories, evaluated on three models from one vendor. The authors acknowledge that the results "do not establish stable model rankings, pass probabilities, or robustness to alternative thresholds" (L430–431). Several reported numbers also do not reconcile. Table 3 lists 9 valid runs for Opus 5 but reports 13/13 engines passed, and at least 17 of the 215 runs are unaccounted for.

Less critical weaknesses

W3. Unclear framing. The abstract and introduction mix the benchmark, the capability results, and the verifier audit without a clear thesis, and terms such as "protocol condition" and "valid run" are never defined.

W4. Unsystematic verifier audit. Section 8 illustrates selected defects but provides no complete table of the 28 cases, no total number of checks, and no analysis of how each defect affected scores or rankings.

W5. Confounded cross-vendor comparison. Non-Anthropic models were run in the Claude Code harness through a protocol adapter, which also changed the tool set available to GPT-5.6. A fairer comparison would run each model in its vendor's native harness (e.g., Codex for GPT-5.6) or in a vendor-neutral harness such as OpenHands or mini-SWE-agent.

Questions for authors
  1. How was the benchmark constructed? Please describe how the packages and domains were selected, who wrote and reviewed the specifications, how checks and pass thresholds were set, and how the reference implementations for the transactional engines were validated.
  2. Please provide the complete table of the 28 defects, including the type of each defect, when it was found, and whether fixing it changed any pass/fail outcome in Tables 1 and 2. The paper classifies 24 defects as false negatives and 2 as false positives; what are the other two? Is the released version frozen, and were any defects found after the last round of fixes?
  3. Table 3 lists 9 valid runs for Opus 5, yet Table 2 reports 13/13 engines passed. Which runs underlie the Opus 5 column? Section 4 also reports 215 runs, of which 155 are valid, 37 ended in gateway or quota errors, and 6 exhausted the turn budget. These numbers do not sum to 215, so please give a full breakdown. Were runs superseded after verifier fixes excluded?
Reviewer_GcuyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ReviveBench, which is a benchmark testing the abilities of coding agents in "reviving codebases". The benchmark has two kinds of problems, the "software revival" tasks, where the goal for the agent is to fix the issues in nonfunctional codebases and the "clean-room reconstruction" tasks, where the goal of the agent is to come up with an implementation for a software given a spec. The former includes 10 tasks, while the latter has 13 tasks. Authors test the abilities of frontier proprietary models from Anthropic and OpenAI on the benchmark, along with GLM 5.3 flash, an open source model by Zhipu. They find that these models perform well on both of the task subsets, albeit some inter model differences.

Critical strengths

None

Critical weaknesses

The idea of a "ReviveBench", where the goal is to re-implement/fix issues with old codebases, could be quite compelling! However, the paper has several shortcomings that are hard to fix. The most critical one is that given that the models tested seem to be already doing well on the benchmark, it's utility is unclear. Moreover, the presentation of the paper is not at the level of what would be expected from an ICLR paper: it's extremely hard to follow and reads more as an experiment logbook as opposed to a fully fleshed out paper.

The paper also includes several writing artifacts that are unexpected in a fully fleshed out research paper (i.e., line 079, "A verifier-defect taxonomy with 28 documented cases, including two false positives, one of which led us to withdraw a claim from an earlier version of this work (Section 8)".

Questions for authors

What is the utility of this benchmark given the strong scores obtained by frontier LLMs?

Reviewer_GiDCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReviveBench introduces a dual-family benchmark designed to evaluate coding agents on software revival and clean-room engine reconstruction across 23 distinct tasks. The revival track challenges agents to restore nonfunctional legacy software suffering from dependency rot, deleted core modules, or outdated build systems. The reconstruction track requires agents to build industrial software engines from open specifications without access to the source code. To measure success, the authors employ hidden, executable verifiers calibrated against native environments and reference tools. Beyond the agent evaluation, the paper conducts a meta-analysis on the reliability of execution-based grading, documenting how automated verifiers can introduce substantial measurement errors.

Critical strengths

High-Value Task Construction: The authors invested immense manual effort into crafting the benchmark, and every single problem in the suite represents a valuable, high-quality contribution. Integrating native environment calibration, external reference tools (like SAT solvers and fuzzers), and bespoke reference implementations for each task provides a highly rigorous testing ground.

Rigorous Contamination Controls: The methodology robustly addresses the risk of benchmark memorization by utilizing identifier obfuscation and testing against post-cutoff repositories, effectively isolating genuine problem-solving capabilities from verbatim recall.

Crucial Verifier Audit: The empirical audit of the grading systems is a major contribution to the field. By documenting 28 verifier defects and establishing three reusable validation methodologies—the achievability probe, candidate consensus checking, and the rule against grading self-reported metrics—the authors provide a blueprint for improving future execution-based benchmarks.

Critical weaknesses

Scalability and Sample Size: Because the creation of hidden verifiers and reference environments requires intense manual labor, the benchmark is limited to just 23 tasks. This small scale restricts statistical reliability, as most evaluation cells contain only one or two seeds, making it impossible to establish stable pass probabilities or robust, long-term model rankings.

Differentiability and Headroom: The benchmark is already saturated by current frontier models, limiting its ability to differentiate top-tier performance. Fable 5.1 achieved a perfect score across all tasks using relatively modest compute budgets, and increasing reasoning effort yielded no score improvements due to this ceiling effect. Additionally, the MES and PLM reconstruction tasks were passed by every evaluated model, including a low-cost difficulty probe, rendering them entirely non-discriminative.

Verifier Vulnerabilities (the author mentioned, should be more like a limitation instead of a real weakness): Despite the bespoke construction, the executable verifiers still suffered from severe false positives. Most notably, the CFD task awarded full marks to a 134-line script that performed no numerical solving and simply returned hard-coded outputs, highlighting vulnerabilities in the automated grading logic.

Questions for authors

The manual effort invested in constructing these highly rigorous tasks and native verifiers is commendable, and each problem provides immense value. However, given that top-tier models currently saturate the benchmark using modest compute budgets, what specific dimensions of complexity could be introduced in future iterations to provide the difficulty headroom needed to effectively differentiate frontier models?

Considering the intense labor required to calibrate native environments and build bespoke reference implementations, what strategies or automated generation pipelines might be viable to scale up the number of tasks and improve the benchmark's overall statistical power?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.