- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SWE-bench Science, a benchmark with 119 scientific software engineering tasks from 98 repositories across 20 scientific fields. The tasks fall into three groups: Issue-driven, Expert-exploratory, and Engineering-integration. The evaluation focuses on scientific requirements, separates public and private tests, and checks whether a repair works beyond the visible example. The authors evaluate nine model and agent harness configurations. The best configuration reaches 50.42% Pass@1 while scoring 100% on the public tests. The paper also groups failures into four categories and compares runs with and without extra scientific information on 91 tasks. The effect of this information differs across configurations and changes direction between two reasoning settings of the same model.
Critical strengths
-
The benchmark focuses on scientific correctness. Its private tests cover scientific invariants, boundary cases, and requirements across modules. The 119 tasks from 98 repositories provide broad coverage of scientific software.
-
The results reveal a useful evaluation gap. The best configuration scores 100% on public tests but solves only 50.42% of tasks under the private tests (Table 2). This shows why visible test success is not enough to judge a complete repair.
Critical weaknesses
-
The scientific tests need stronger independent validation. The paper does not clearly report how many tasks were checked by independent experts or how often alternative valid patches and deliberately wrong patches passed. These checks are needed to establish that the tests reliably measure scientific correctness.
-
The knowledge experiment mixes several effects. The added material includes scientific principles, diagnosis, and repair hints. It also changes context length substantially. With only three configurations and no reported repeated runs per condition, the results do not isolate why the effect changes with reasoning depth.
Questions for authors
-
Who checked the scientific oracles, how independent were they from the test authors, and how many tasks did they cover? Please report pass rates for reference patches, alternative valid patches, and deliberately wrong patches. Was a subset checked by independent domain experts? I am asking for audit results and error types, not for the hidden answers to be made public.
-
Could you show a few complete pairs of inputs with and without the extra information? Please mark general scientific principles, localization hints, upstream repair information, and added tokens separately. Who checked the boundary between required scientific context and information that could be removed?
-
Could you report uncertainty for the paired task results in Figure 5 and repeat runs on a predefined subset? It would also help to separate general scientific principles from diagnosis or repair hints and check whether the change in effect remains. This would not require rerunning all nine configurations.
5 comments
Response to Reviewer rQtD (2/5) - Q1, Q2
Q1 / W1. Scientific-oracle review and patch validation (continued)
All four negative candidates for Tasks 041 and 094 pass the public tests. For Task 049, the candidate omitting the Milstein repair also passes the public reproduction, while the candidate retaining the old NumPy primitive scaling fails it. Five of the six negative candidates across these examples therefore pass the public tests. The private tests further distinguish omissions in coordinate frames, symmetry transformations, and stochastic scaling across solvers.
Independent numerical or chemical checks also establish concrete scientific errors. Covalent links between mutually exclusive conformations violate conformer compatibility. Incorrect continuation of a valid continuous function violates the required boundary behavior. Loss of the unknown stereochemical state of a newly formed imine bond affects the downstream chemical representation. Appendix F presents the submitted behavior and its scientific basis. Further checks also identified implementation-dependent acceptance conditions. The evidence from candidate acceptance rates is therefore scoped to the examined implementations and error mechanisms.
In the existing full-benchmark results, reference repairs pass the complete private suite on all 119 tasks, and empty patches achieve complete success on none of the 119 tasks. The additional 60-task reference controls check the runtime environments used in this supplementary validation. Appendix A.1 reports candidate construction, results by paradigm, and specific scientific error types. Individual results are retained in the existing experimental records.
Manuscript revisions
Section 4 describes the acceptance requirement for equivalent implementations and construction-time trial checks. Appendix A.1 reports the 60-task selection, candidate preparation, same-environment tests, Table 5 results, and examples from Tasks 041, 094, and 049, together with the scope of validation. Section 6.1 and Appendix F add scientific-error analysis of submitted repairs.
Q2 / W2. Paired inputs and material composition
Appendix C.1 provides paired initial inputs and auxiliary-file inventories with and without information for Tasks 002, 033, and 073, covering Engineering-integration, Expert-exploratory, and Issue-driven tasks. Both conditions use the same initial user instruction. The difference lies in the auxiliary files the agent can read on demand. Appendix C.1 provides the complete initial instructions, shared task statements, public execution instructions, file inventories for both conditions, and auxiliary-material annotations. It points to the corresponding task directories in the anonymous repository for the complete paper.md files and other auxiliary materials.
We annotate the materials for scientific principles, software-use or localization cues, and upstream repair information. The materials for Task 002 primarily provide the range-separation method and computational settings. Task 033 provides equivalence between periodic representations, density-fitting literature, and software-use examples. Task 073 provides information on PDBx/mmCIF atom records, alternate locations, occupancy, struct_conn, and public interfaces. Some passages supply both scientific background and workflow-localization cues, and the examples retain these mixed functions. Across these three examples, the count of upstream repair commits, diffs, or issue/pull-request repair descriptions in the inspected auxiliary files is 0.
Using a fixed version of the official DeepSeek-V4-Flash-0731 tokenizer, the number of additional initial-instruction tokens is 0 for all three examples. The table reports paper.md text lengths and the types and quantities of the supplied auxiliary materials. Link counts refer to the reference entries listed within the note.
| Task | Added initial-instruction tokens | paper.md text tokens | Supplied auxiliary materials and quantities | Standalone image files |
|---|---|---|---|---|
| 002 | 0 | 16,355 | 1 paper text containing methodological background and results; 17 paper images | 17 |
| 033 | 0 | 567 | 1 index, 3 offline HTML articles, 3 Python usage examples, and 1 README | 0 |
| 073 | 0 | 839 | 1 structural-biology note containing 6 documentation/data-model links and 3 project-source links | 0 |
Response to Reviewer rQtD (3/5) - Q2, Q3
Q2 / W2. Paired inputs and material composition (continued)
The text bundle for Task 002 consists of paper.md, accompanied by 17 standalone paper images. The bundle for Task 033 contains a paper.md index, three offline HTML articles, three Python usage examples, and a README, totaling 84,196 reference tokens of readable text. This total comprises 567 index tokens, 81,625 readable-article tokens, 1,762 Python-example tokens, and 242 README tokens. Readable HTML text is counted after removing script/style/noscript elements and extracting the text. Markdown and Python files are counted as complete file text. Encoding counts for the raw HTML files are retained in the per-file records. These reference counts describe the text available in the supplied materials. Cumulative trajectory input is measured from actual model calls. The full structural-biology note for Task 073 is included in its 839 tokens. Its six documentation/data-model links and three project-source links are embedded references and are listed as links in the table.
The with-information and without-information conditions preserve the same core problem, repository, execution environment, public entry points, and private tests. Domain-informed researchers review the scientific objectives and task-statement requirements. Technical contributors check material separation, executable dependencies, and public reproduction. These checks establish the material boundary, retaining the engineering context needed to start the task while checking the sufficiency of public evidence and the isolation of hidden evaluation information. This ablation measures the incremental effect of the complete auxiliary-material bundle. Scientific principles, methodological explanations, and workflow cues jointly constitute the material condition, and the interpretation is scoped to the bundle actually supplied. The remaining 28 tasks lack clearly separable auxiliary materials or have relevant background intertwined with repository content such as code comments. They remain in the 119-task main evaluation and are excluded from this paired comparison.
Manuscript revisions
We have added these three input examples and material annotations in Appendix C.1 (Examples of Paired Inputs and Material Content), with material types, quantities, and text lengths in Table 6. Section 4.2 (Scientific Auxiliary Information Separation) points to these examples. Appendix C.1 describes the types, quantities, and text lengths of available materials. Section 6.2, Figure 4, and Appendix C.2 report actual cumulative trajectory token usage.
Q3 / W2. Repeated ablations and paired results
We completed three runs of each of the four DeepSeek-V4-flash conditions, high/max effort with/without information, on the full predefined 91-task subset. The direction of the material effect remained consistent across all three repetitions. The paired differences were +4, +4, and +5 tasks at high effort, and -3, -2, and -3 tasks at max effort. The between-run sample standard deviation of the paired material effect was 0.63 percentage points at each effort level, quantifying repeated-run variation on the fixed 91-task set. For material composition, we annotated scientific principles, diagnostic or localization information, and upstream repair information. The repeated runs used the complete material bundle. The independent effects of its components remain to be tested through component ablations.
The original DeepSeek evaluation reused an existing shared deployment to control running costs, first for the 91-task ablation and then for the with-information evaluation extended to all 119 tasks. A deployment review confirmed that both evaluations used a quantized model. The revised evaluation uses non-quantized DeepSeek-V4-flash-0731, with three runs in each of the four high/max and with/without-information conditions on the original 91 tasks, plus one complete 119-task with-information evaluation at each effort level.
| Reasoning effort | Scientific materials | Run 1 | Run 2 | Run 3 | Mean Pass@1 | Sample SD, percentage points |
|---|---|---|---|---|---|---|
| high | With | 33/91 | 32/91 | 33/91 | 35.90% | 0.63 |
| high | Without | 29/91 | 28/91 | 28/91 | 31.14% | 0.63 |
| max | With | 33/91 | 34/91 | 34/91 | 37.00% | 0.63 |
| max | Without | 36/91 | 36/91 | 37/91 | 39.93% | 0.63 |
Response to Reviewer rQtD (1/5) - Q1
Q1 / W1. Scientific-oracle review and patch validation
Post-construction scientific review covered all 119 tasks, and the domain-informed reviewers were different people from the evaluator implementers. Researchers screened scientific objectives and value at the proposal stage, then reviewed the completed task statements, scientific objectives, and expected solution requirements. The same researcher participated in screening and final review of a given task. Evaluator implementers developed the runtime environments, reference repairs, and tests. Scientific reviewers and test implementers were therefore separate people, with scientific review embedded in task construction. The construction team performed the 60-task validation of alternative and incorrect implementations, while full-benchmark reference and empty-patch validation covered all 119 tasks.
During construction, evaluators were required to accept equivalent implementations and representations satisfying the same scientific requirements. We conducted repeated trial repairs with GPT-5.6-sol at xhigh effort under the information visibility and runtime conditions of the formal evaluation, with private-test source code held by the evaluator. During task design, we reviewed candidate solutions against the scientific requirements using trial patches, trajectories, and evaluation outcomes. We checked whether private tests correctly accepted equivalent implementations and representations satisfying those requirements and whether scientifically incorrect or incomplete repairs could nevertheless pass. We also examined implementation-specific acceptance conditions and plausible shortcuts.
Thank you for asking about alternative correct implementations and incorrect repairs. We conducted a supplementary validation experiment covering 60 tasks, comprising 26 Issue-driven, 25 Expert-exploratory, and 9 Engineering-integration tasks. We used stratified random sampling in approximately the proportions of the three paradigms in the benchmark. Each task contributed a reference repair, an alternative implementation prepared to satisfy the same scientific requirements, and two incorrect or incomplete repair candidates based on the task’s original error mechanisms.
We used GPT-6.1-sol at high effort to assist with candidate preparation. Preserving the original scientific definition and repair objective, we drew on upstream code, reference repairs, and recoverable historical implementations to construct alternative implementations and omissions that could arise in real repairs. Errors included omitted coordinate-transformation steps, inconsistent state updates, incorrect physical-quantity weighting, and incomplete cross-module repairs. Candidate preparation could use reference repairs and private-test source code. All four implementations for a task used the same runtime image and the same public and private tests, with the reference repair serving as a same-environment control.
All 60 reference candidates and all 60 alternative candidates passed the complete private suite, and all 120 specified negative candidates were rejected. Among the negative candidates, 93 passed the public tests, or 77.5%. These results provide concrete evidence of acceptance of the selected alternatives and detection of the specified error mechanisms.
Three concrete cases illustrate the three task paradigms.