- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces ReviveBench, a benchmark for coding agents graded by hidden executable verifiers. It contains ten software revival tasks, which restore nonfunctional scientific software affected by dependency rot, deleted modules, legacy builds, or post-cutoff changes. It also contains thirteen clean-room reconstruction tasks, which implement bounded functionality of engineering engines from public specifications. The revival tasks are evaluated on three Anthropic models, and the reconstruction tasks on eight models from three vendors. The strongest model passes all ten revival tasks in at least one run, and two models pass all thirteen reconstruction tasks. The paper also audits its own verifiers, reports 28 defects (24 false negatives and 2 false positives), and proposes three checks for verifiers. Most model–task cells contain one or two runs, and the best run is reported.
Critical strengths
S1. Task domains with checkable semantics. The benchmark targets domains whose correctness is externally defined and checkable: SAT-based equivalence for synthesized netlists, NIST StRD certified values for statistics, and integer-cent ledger invariants for transactional engines. It also frames revival so that it must reproduce the original software's outputs (e.g., PyMOL structure metrics, QuPath cell detection), not merely make the code run.
Critical weaknesses
W1. The benchmark is neither documented nor designed as a measurement instrument.
The paper describes the tasks but not how they were built. It does not explain how the packages and domains were selected, who wrote and reviewed the specifications, how checks and pass thresholds were set, how the transactional reference implementations were validated, or how the post-cutoff repositories were chosen. Section 9 says only that thresholds were "chosen by us and anchored to reference measurements." This matters because the paper's own audit shows that errors arise during construction: all 28 defects were found while the suite was being built and audited, and one false positive invalidated a claim in an earlier version. Without a description of the construction process, readers cannot judge whether the remaining tasks are trustworthy, and others cannot extend the benchmark.
The benchmark also lacks a defined construct. Each of the ten revival tasks exercises a different skill, such as framework porting, C++ build repair, Java build migration, CUDA kernel compatibility, or reimplementation from a paper. Each skill has only one to three instances, so none is measured reliably. The two families also use different model sets (three vs. eight models) and different scoring rules (success in any run vs. the best run's score against task thresholds), and the paper provides no aggregate score.
W2. The evidence is too thin to support the headline claims.
The suite has 23 nominal tasks, with one task per domain in the reconstruction family. This is far fewer than agentic benchmarks with similarly large tasks, such as Commit0 (54 libraries), MLE-bench (75 competitions), and CORE-Bench (270 tasks). In reconstruction, every model passes MES and PLM, and CFD does not measure numerical-solver capability, leaving ten informative tasks. In revival, all three models pass four of the ten tasks. The benchmark is also saturated at the frontier on release: two models pass all thirteen reconstruction tasks, and one passes all ten revival tasks. The authors themselves note that "saturated tasks have limited power to reveal improvements in accuracy" (Appendix B).
Most cells contain one or two runs, only the best run is reported, and the number of attempts differs across models. In revival, Fable 5.1 has 34 runs, compared with 11 for Sonnet 5 and 10 for Haiku 4.5. The 10/10 vs. 8/10 gap between Fable 5.1 and Sonnet 5 rests on two tasks. On those tasks, Sonnet 5 had one run each, while Fable 5.1 succeeded in 2 of 6 runs on one task and 2 of 3 on the other. The claim that the contamination controls "provide evidence of capability beyond verbatim recall" rests on one obfuscated task and three post-cutoff repositories, evaluated on three models from one vendor. The authors acknowledge that the results "do not establish stable model rankings, pass probabilities, or robustness to alternative thresholds" (L430–431). Several reported numbers also do not reconcile. Table 3 lists 9 valid runs for Opus 5 but reports 13/13 engines passed, and at least 17 of the 215 runs are unaccounted for.
Less critical weaknesses
W3. Unclear framing. The abstract and introduction mix the benchmark, the capability results, and the verifier audit without a clear thesis, and terms such as "protocol condition" and "valid run" are never defined.
W4. Unsystematic verifier audit. Section 8 illustrates selected defects but provides no complete table of the 28 cases, no total number of checks, and no analysis of how each defect affected scores or rankings.
W5. Confounded cross-vendor comparison. Non-Anthropic models were run in the Claude Code harness through a protocol adapter, which also changed the tool set available to GPT-5.6. A fairer comparison would run each model in its vendor's native harness (e.g., Codex for GPT-5.6) or in a vendor-neutral harness such as OpenHands or mini-SWE-agent.
Questions for authors
- How was the benchmark constructed? Please describe how the packages and domains were selected, who wrote and reviewed the specifications, how checks and pass thresholds were set, and how the reference implementations for the transactional engines were validated.
- Please provide the complete table of the 28 defects, including the type of each defect, when it was found, and whether fixing it changed any pass/fail outcome in Tables 1 and 2. The paper classifies 24 defects as false negatives and 2 as false positives; what are the other two? Is the released version frozen, and were any defects found after the last round of fixes?
- Table 3 lists 9 valid runs for Opus 5, yet Table 2 reports 13/13 engines passed. Which runs underlie the Opus 5 column? Section 4 also reports 215 runs, of which 155 are valid, 37 ended in gateway or quota errors, and 6 exhausted the turn budget. These numbers do not sum to 215, so please give a full breakdown. Were runs superseded after verifier fixes excluded?