acceptodds
Under review as a conference paper at ICLR 2027

SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

Abstract

Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce SWE-bench Science, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. The best-performing configuration, GPT-6 Astra with Codex at max reasoning depth, achieves 50.42% Pass@1, leaving 59 of the 119 tasks unresolved and highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Across paired conditions, the value of additional information reverses with the configuration's standalone capability, turning negative once deeper reasoning already recovers the scientific contract without guidance. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.

Primary area:
datasets and benchmarks
Keywords:
SWE-bench, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_rQtDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SWE-bench Science, a benchmark with 119 scientific software engineering tasks from 98 repositories across 20 scientific fields. The tasks fall into three groups: Issue-driven, Expert-exploratory, and Engineering-integration. The evaluation focuses on scientific requirements, separates public and private tests, and checks whether a repair works beyond the visible example. The authors evaluate nine model and agent harness configurations. The best configuration reaches 50.42% Pass@1 while scoring 100% on the public tests. The paper also groups failures into four categories and compares runs with and without extra scientific information on 91 tasks. The effect of this information differs across configurations and changes direction between two reasoning settings of the same model.

Critical strengths
  1. The benchmark focuses on scientific correctness. Its private tests cover scientific invariants, boundary cases, and requirements across modules. The 119 tasks from 98 repositories provide broad coverage of scientific software.

  2. The results reveal a useful evaluation gap. The best configuration scores 100% on public tests but solves only 50.42% of tasks under the private tests (Table 2). This shows why visible test success is not enough to judge a complete repair.

Critical weaknesses
  1. The scientific tests need stronger independent validation. The paper does not clearly report how many tasks were checked by independent experts or how often alternative valid patches and deliberately wrong patches passed. These checks are needed to establish that the tests reliably measure scientific correctness.

  2. The knowledge experiment mixes several effects. The added material includes scientific principles, diagnosis, and repair hints. It also changes context length substantially. With only three configurations and no reported repeated runs per condition, the results do not isolate why the effect changes with reasoning depth.

Questions for authors
  1. Who checked the scientific oracles, how independent were they from the test authors, and how many tasks did they cover? Please report pass rates for reference patches, alternative valid patches, and deliberately wrong patches. Was a subset checked by independent domain experts? I am asking for audit results and error types, not for the hidden answers to be made public.

  2. Could you show a few complete pairs of inputs with and without the extra information? Please mark general scientific principles, localization hints, upstream repair information, and added tokens separately. Who checked the boundary between required scientific context and information that could be removed?

  3. Could you report uncertainty for the paired task results in Figure 5 and repeat runs on a predefined subset? It would also help to separate general scientific principles from diagnosis or repair hints and check whether the change in effect remains. This would not require rerunning all nine configurations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer rQtD (1/5) - Q1

Q1 / W1. Scientific-oracle review and patch validation

Post-construction scientific review covered all 119 tasks, and the domain-informed reviewers were different people from the evaluator implementers. Researchers screened scientific objectives and value at the proposal stage, then reviewed the completed task statements, scientific objectives, and expected solution requirements. The same researcher participated in screening and final review of a given task. Evaluator implementers developed the runtime environments, reference repairs, and tests. Scientific reviewers and test implementers were therefore separate people, with scientific review embedded in task construction. The construction team performed the 60-task validation of alternative and incorrect implementations, while full-benchmark reference and empty-patch validation covered all 119 tasks.

During construction, evaluators were required to accept equivalent implementations and representations satisfying the same scientific requirements. We conducted repeated trial repairs with GPT-5.6-sol at xhigh effort under the information visibility and runtime conditions of the formal evaluation, with private-test source code held by the evaluator. During task design, we reviewed candidate solutions against the scientific requirements using trial patches, trajectories, and evaluation outcomes. We checked whether private tests correctly accepted equivalent implementations and representations satisfying those requirements and whether scientifically incorrect or incomplete repairs could nevertheless pass. We also examined implementation-specific acceptance conditions and plausible shortcuts.

Thank you for asking about alternative correct implementations and incorrect repairs. We conducted a supplementary validation experiment covering 60 tasks, comprising 26 Issue-driven, 25 Expert-exploratory, and 9 Engineering-integration tasks. We used stratified random sampling in approximately the proportions of the three paradigms in the benchmark. Each task contributed a reference repair, an alternative implementation prepared to satisfy the same scientific requirements, and two incorrect or incomplete repair candidates based on the task’s original error mechanisms.

We used GPT-6.1-sol at high effort to assist with candidate preparation. Preserving the original scientific definition and repair objective, we drew on upstream code, reference repairs, and recoverable historical implementations to construct alternative implementations and omissions that could arise in real repairs. Errors included omitted coordinate-transformation steps, inconsistent state updates, incorrect physical-quantity weighting, and incomplete cross-module repairs. Candidate preparation could use reference repairs and private-test source code. All four implementations for a task used the same runtime image and the same public and private tests, with the reference repair serving as a same-environment control.

Task paradigmTasksReference candidates accepted by the complete suiteAlternative candidates accepted by the complete suiteSpecified negative candidates accepted by the complete suite
Issue-driven2626/2626/260/52
Expert-exploratory2525/2525/250/50
Engineering-integration99/99/90/18
Total6060/6060/600/120

All 60 reference candidates and all 60 alternative candidates passed the complete private suite, and all 120 specified negative candidates were rejected. Among the negative candidates, 93 passed the public tests, or 77.5%. These results provide concrete evidence of acceptance of the selected alternatives and detection of the specified error mechanisms.

Three concrete cases illustrate the three task paradigms.

CaseComputation in the reference and alternative implementationsSpecified negative candidates and private-test results
041, geocentric coordinate transformationsThe reference repair follows the required intermediate frames, and the alternative explicitly composes rotation matrices. Both pass 24/24 checks.A repair retaining an incorrect equinox passes 21/24. A repair using the forward rotation in the inverse transformation passes 22/24.
094, magnetic-space-group setting transformationsThe reference repair and the alternative based on solving linear equations preserve the same basis and origin transformation. Both pass 5/5 checks.Omitting the setting transformation yields 3/5. Omitting the origin-shift term yields 4/5.
049, stochastic updates on spatial gridsThe reference applies cell-measure scaling in the solvers. The alternative divides by cell measure in the Euler, implicit, and Milstein paths. Both pass 13/13 checks.Retaining the old NumPy primitive scaling yields 7/13. Omitting the Milstein repair yields 11/13.
AuthorsOpenReview ↗

Response to Reviewer rQtD (2/5) - Q1, Q2

Q1 / W1. Scientific-oracle review and patch validation (continued)

All four negative candidates for Tasks 041 and 094 pass the public tests. For Task 049, the candidate omitting the Milstein repair also passes the public reproduction, while the candidate retaining the old NumPy primitive scaling fails it. Five of the six negative candidates across these examples therefore pass the public tests. The private tests further distinguish omissions in coordinate frames, symmetry transformations, and stochastic scaling across solvers.

Independent numerical or chemical checks also establish concrete scientific errors. Covalent links between mutually exclusive conformations violate conformer compatibility. Incorrect continuation of a valid continuous function violates the required boundary behavior. Loss of the unknown stereochemical state of a newly formed imine bond affects the downstream chemical representation. Appendix F presents the submitted behavior and its scientific basis. Further checks also identified implementation-dependent acceptance conditions. The evidence from candidate acceptance rates is therefore scoped to the examined implementations and error mechanisms.

In the existing full-benchmark results, reference repairs pass the complete private suite on all 119 tasks, and empty patches achieve complete success on none of the 119 tasks. The additional 60-task reference controls check the runtime environments used in this supplementary validation. Appendix A.1 reports candidate construction, results by paradigm, and specific scientific error types. Individual results are retained in the existing experimental records.

Manuscript revisions

Section 4 describes the acceptance requirement for equivalent implementations and construction-time trial checks. Appendix A.1 reports the 60-task selection, candidate preparation, same-environment tests, Table 5 results, and examples from Tasks 041, 094, and 049, together with the scope of validation. Section 6.1 and Appendix F add scientific-error analysis of submitted repairs.

Q2 / W2. Paired inputs and material composition

Appendix C.1 provides paired initial inputs and auxiliary-file inventories with and without information for Tasks 002, 033, and 073, covering Engineering-integration, Expert-exploratory, and Issue-driven tasks. Both conditions use the same initial user instruction. The difference lies in the auxiliary files the agent can read on demand. Appendix C.1 provides the complete initial instructions, shared task statements, public execution instructions, file inventories for both conditions, and auxiliary-material annotations. It points to the corresponding task directories in the anonymous repository for the complete paper.md files and other auxiliary materials.

We annotate the materials for scientific principles, software-use or localization cues, and upstream repair information. The materials for Task 002 primarily provide the range-separation method and computational settings. Task 033 provides equivalence between periodic representations, density-fitting literature, and software-use examples. Task 073 provides information on PDBx/mmCIF atom records, alternate locations, occupancy, struct_conn, and public interfaces. Some passages supply both scientific background and workflow-localization cues, and the examples retain these mixed functions. Across these three examples, the count of upstream repair commits, diffs, or issue/pull-request repair descriptions in the inspected auxiliary files is 0.

Using a fixed version of the official DeepSeek-V4-Flash-0731 tokenizer, the number of additional initial-instruction tokens is 0 for all three examples. The table reports paper.md text lengths and the types and quantities of the supplied auxiliary materials. Link counts refer to the reference entries listed within the note.

TaskAdded initial-instruction tokenspaper.md text tokensSupplied auxiliary materials and quantitiesStandalone image files
002016,3551 paper text containing methodological background and results; 17 paper images17
03305671 index, 3 offline HTML articles, 3 Python usage examples, and 1 README0
07308391 structural-biology note containing 6 documentation/data-model links and 3 project-source links0
AuthorsOpenReview ↗

Response to Reviewer rQtD (3/5) - Q2, Q3

Q2 / W2. Paired inputs and material composition (continued)

The text bundle for Task 002 consists of paper.md, accompanied by 17 standalone paper images. The bundle for Task 033 contains a paper.md index, three offline HTML articles, three Python usage examples, and a README, totaling 84,196 reference tokens of readable text. This total comprises 567 index tokens, 81,625 readable-article tokens, 1,762 Python-example tokens, and 242 README tokens. Readable HTML text is counted after removing script/style/noscript elements and extracting the text. Markdown and Python files are counted as complete file text. Encoding counts for the raw HTML files are retained in the per-file records. These reference counts describe the text available in the supplied materials. Cumulative trajectory input is measured from actual model calls. The full structural-biology note for Task 073 is included in its 839 tokens. Its six documentation/data-model links and three project-source links are embedded references and are listed as links in the table.

The with-information and without-information conditions preserve the same core problem, repository, execution environment, public entry points, and private tests. Domain-informed researchers review the scientific objectives and task-statement requirements. Technical contributors check material separation, executable dependencies, and public reproduction. These checks establish the material boundary, retaining the engineering context needed to start the task while checking the sufficiency of public evidence and the isolation of hidden evaluation information. This ablation measures the incremental effect of the complete auxiliary-material bundle. Scientific principles, methodological explanations, and workflow cues jointly constitute the material condition, and the interpretation is scoped to the bundle actually supplied. The remaining 28 tasks lack clearly separable auxiliary materials or have relevant background intertwined with repository content such as code comments. They remain in the 119-task main evaluation and are excluded from this paired comparison.

Manuscript revisions

We have added these three input examples and material annotations in Appendix C.1 (Examples of Paired Inputs and Material Content), with material types, quantities, and text lengths in Table 6. Section 4.2 (Scientific Auxiliary Information Separation) points to these examples. Appendix C.1 describes the types, quantities, and text lengths of available materials. Section 6.2, Figure 4, and Appendix C.2 report actual cumulative trajectory token usage.

Q3 / W2. Repeated ablations and paired results

We completed three runs of each of the four DeepSeek-V4-flash conditions, high/max effort with/without information, on the full predefined 91-task subset. The direction of the material effect remained consistent across all three repetitions. The paired differences were +4, +4, and +5 tasks at high effort, and -3, -2, and -3 tasks at max effort. The between-run sample standard deviation of the paired material effect was 0.63 percentage points at each effort level, quantifying repeated-run variation on the fixed 91-task set. For material composition, we annotated scientific principles, diagnostic or localization information, and upstream repair information. The repeated runs used the complete material bundle. The independent effects of its components remain to be tested through component ablations.

The original DeepSeek evaluation reused an existing shared deployment to control running costs, first for the 91-task ablation and then for the with-information evaluation extended to all 119 tasks. A deployment review confirmed that both evaluations used a quantized model. The revised evaluation uses non-quantized DeepSeek-V4-flash-0731, with three runs in each of the four high/max and with/without-information conditions on the original 91 tasks, plus one complete 119-task with-information evaluation at each effort level.

Reasoning effortScientific materialsRun 1Run 2Run 3Mean Pass@1Sample SD, percentage points
highWith33/9132/9133/9135.90%0.63
highWithout29/9128/9128/9131.14%0.63
maxWith33/9134/9134/9137.00%0.63
maxWithout36/9136/9137/9139.93%0.63
Reviewer_DmwnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-bench Science introduces 119 scientific software-engineering tasks from 98 repositories across 20 scientific domains. It evaluates an extensive spectrum of frontier model configurations (including GPT, claude, KIMI, GLM, Qwen, Deepseek) using hidden tests, public tests and studies how auxiliary scientific information affects repair performance. Authors also provide some initial insights in four recurring failure modes in scientific SWE, and pair with ablations that remove scientific guidance, while preserving repo engineering context, to highlight the science part.

Critical strengths

S1: Timely and important topic. As LLM agents become increasingly capable, realistic, long horizon benchmarks for scientific SWE are needed to evaluate their capabilities and reliability. This work could contribute to that effort and provide lots of value to the broader community.

S2: The paper provides extensive model coverage, useful failure-mode analysis, and ablations that distinguish software engineering capabilities from the contribution of scientific knowledge. In particular, it finds that scientific guidance is not uniformly beneficial. These are valuable initial insights for evaluating agents on scientific SWE tasks.

Critical weaknesses

W1 (minor): Limited comparison with concurrent work. This is a rapidly developing and competitive area, with several concurrent benchmarks pursuing similar goals. The paper does not adequately discuss related efforts, such as Terminal-Bench Science, or clearly explain its relative strengths and limitations. A more explicit comparison would help establish the benchmark’s distinct contribution. I list it as minor because I still recognize the novelty and importance of this bench, which is not diminished by the concurrent ones.

W2 (major): Insufficient domain-expert involvement and quality control. The paper does not clearly describe whether or how scientific domain experts participate and how they review. For comparison, Terminal-Bench Science incorporates domain-expert review at both the proposal and final evaluation stages, as the case owner. Relying on publicly available repository issues without explicit expertise in science may not be sufficient to establish a very clean verified evaluation set in science (science is hard).

My eyeballing of several examples also raised questions about the science validation of the underlying repositories. For instance, Task 002 in the appendix draws on the chemistry repository of zero star: https://github.com/aditisingh4812/simplified_tuned_range_separated, which appears to be maintained by a single contributor. This alone does not imply poor quality, but it makes independent validation particularly important.

Questions for authors

Q1: What quality-control procedures ensure that the tasks and tests are scientifically valid? Could the authors clarify whether domain experts review task proposals and final evaluations, or explain what alternative validation process provides comparable high-quality quality control?

Q2: What are the benchmark’s main strengths and limitations relative to concurrent work, such as Terminal-Bench Science? How do differences in task construction, scientific depth, and evaluation design affect what each benchmark measures?

Q3: Could the authors provide more detailed cross-model failure analysis? For example, what distinguishes Claude’s performance from GPT’s, and which capabilities separate the leading models from lower-performing models? Terminal-Bench Science appears to show a larger performance gap between these frontier and second tier models than SWE-Bench Science. What differences in task difficulty, scientific reasoning requirements, or evaluation design might explain this pattern or any intuition is appreciated.

Q4: is there any license of data you need to explicitly state in the paper?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Dmwn (1/4) - Q1

Q1 / W2. Scientific review and task quality

Domain-informed researchers reviewed both task proposals and the completed tasks, with post-construction scientific review covering all 119 tasks. Scientific reviewers and evaluator implementers were different people. The former assessed scientific objectives, task statements, and expected solution requirements, while the latter developed runtime environments, reference repairs, and evaluators. We checked scientific validity through this human review, repeated construction-time trial repairs, and validation of reference, alternative, and incorrect implementations.

We contacted researchers with relevant domain knowledge from approximately 15 academic departments and research groups at leading universities and research institutions, asking them to recommend code they routinely use in research, problems they encounter, and recent papers in their fields. We identified the corresponding open-source repositories from these recommendations, examined their scientific basis and research use cases, and then developed tasks from relevant issues, pull requests, and practical research problems.

Construction combines scientific-objective review with technical implementation. LLMs assist with proposing candidate tasks, and researchers with relevant domain knowledge assess their scientific objectives and value. Technical contributors experienced in model-training data synthesis work with LLMs to develop the tasks, runtime environments, reference repairs, and evaluators, while checking for deviations from the intended objectives and plausible shortcuts. After construction, the same domain-informed researcher who screened a task reviews its statement, scientific objectives, and expected solution requirements, assessing whether it substantively tests the relevant scientific knowledge and reasoning. Candidates with insufficient scientific content are removed. The three paradigms emphasize the following requirements.

Task paradigmSources and scientific objectivesFocus of construction and final task review
Issue-drivenExisting defects, issues/pull requests, and their scientific use cases in research repositoriesReproducibility of the defect, restoration of correct scientific behavior, and preservation of existing functionality
Expert-exploratoryMethodological literature, scientific phenomena, and anomalies under equivalent conditions or representationsAn observable phenomenon that supports investigation, substantive scientific reasoning in its explanation and repair, and validity across changed conditions
Engineering-integrationGaps in cross-module capability chains within scientific workflowsPropagation of scientific quantities, state, and data flow through the complete workflow, including necessary steps that a local repair may omit

Reference repairs are developed primarily from correct upstream code, repair records, and final task review. Evaluator construction requires acceptance of equivalent implementations and representations that satisfy the same scientific requirements. We also conducted repeated trial repairs with GPT-5.6-sol at xhigh reasoning effort. The solving agent operated under the formal evaluation conditions, with private-test source code accessible only to the evaluator. During task design, we reviewed candidate solutions against the task's scientific requirements using the trial patches, trajectories, and evaluation outcomes. We checked whether the private tests correctly accepted equivalent implementations and representations satisfying the same scientific requirements and whether scientifically incorrect or incomplete repairs could nevertheless pass. We also examined implementation-specific acceptance conditions and plausible shortcuts.

For this review, we selected 60 tasks for supplementary validation using stratified random sampling in approximately the proportions of the three task paradigms, comprising 26 Issue-driven, 25 Expert-exploratory, and 9 Engineering-integration tasks. For each task, we evaluated a reference repair, an alternative candidate prepared to satisfy the same scientific requirements, and two negative candidates based on the task’s error mechanisms. All 60 reference and 60 alternative candidates passed the complete private suite, and all 120 specified negative candidates were rejected, including 93 that passed the public tests. GPT-6.1-sol at high reasoning effort assisted with preparing the candidates, drawing on the original scientific requirements, defect mechanisms, and recoverable historical implementations. Candidates for the same task used the same environment and frozen tests. Appendix A.1 provides the detailed methodology and scientific examples for the three paradigms.

AuthorsOpenReview ↗

Response to Reviewer Dmwn (2/4) - Q1, Q2

Q1 / W2. Scientific review and task quality (continued)

Task 002 was developed from a recent paper and its associated code recommended by a researcher in the relevant field. Task selection was based on the scientific problem, methodological provenance, and verifiability. The research area is specialized, and the upstream repository has a small maintenance team. We examined its provenance and requirements through the published method, upstream code, and validation of scientific behavior within the task. We retain the paper's provenance and the code snapshot for inspection.

We also examined the complete trajectories and used independent numerical or chemical checks to assess the scientific behavior implemented by submitted repairs. The cases concern species identity, mutually exclusive conformer bonds, boundary continuation of continuous functions, and stereochemical uncertainty. These analyses supplement the repair outcomes measured by test-pass rates.

Manuscript revisions

Following the four-stage overview in Section 4 (Benchmark Construction), we have added the process for obtaining research sources, proposing candidates with LLM assistance, conducting domain-informed review, implementing tasks, and reviewing their scientific content after construction. We explicitly describe the acceptance requirement for equivalent scientific implementations, repeated construction-time trial repairs with GPT-5.6-sol at xhigh effort, and checks of whether private tests correctly accept equivalent solutions and detect incorrect repairs during task design. Appendix B (Class-Specific Task Re-design Details) now states the review focus for each paradigm. Section 4 summarizes the 60-task implementation-validation results and points to the methodology, paradigm-specific results in Table 5, and scientific examples in Appendix A.1.

Q2 / W1. Relation to Terminal-Bench Science

Thank you for pointing out this parallel project. Terminal-Bench Science organizes tasks around researchers' real scientific workflows and evaluates them through executable checks. SWE-bench Science organizes tasks around the maintenance and extension of scientific-software repositories, requiring agents to preserve scientific behavior, interfaces, and regression properties after modifying the code. We construct tasks from code repositories used in research, relevant issues and pull requests, and practical problems supplied by researchers. Our three task paradigms cover repairing known scientific defects, autonomously investigating scientific phenomena, and integrating cross-module capability chains end to end.

The scientific requirements are expressed through concrete software behavior. For example, Task 041 requires coordinate transformations to preserve reference-frame semantics and inverse-transformation consistency. Task 094 requires magnetic symmetry operations to transform correctly with changes in basis and origin setting. Task 049 requires random primitives to be independent of grid geometry and cell-measure scaling to be applied consistently in the Euler, implicit, and Milstein solvers. Our supplementary validation uses alternative implementations and incomplete repairs to demonstrate how the evaluators distinguish these requirements.

A strength of our benchmark is that it directly checks whether scientific-software repairs remain correct across boundary conditions, equivalent representations, and cross-module paths, while separately reporting repair progress, complete success, and regression preservation. Its scope is limited to maintenance and extension within the repositories represented by the 119 tasks. Terminal-Bench Science evaluates execution of research workflows and their artifacts, including analyses, simulations, proofs, code, and data products. The two benchmarks therefore emphasize repair and preservation of scientific-software behavior and completion of research-workflow outputs, respectively. Terminal-Bench Science 0.1 announcement

AuthorsOpenReview ↗

Response to Reviewer Dmwn (3/4) - Q2, Q3

Q2 / W1. Relation to Terminal-Bench Science (continued)

The two benchmarks differ in their task sources, task distributions, and evaluation objectives. We have added a related-work comparison in Section 2.2. Our main benchmark construction and task design predate the public release of Terminal-Bench Science version 0.1 on August 27, 2026. The final main results table also includes model evaluations added subsequently. We explain this chronology here and have added a project comparison and citations to public project materials in Section 2.2 and Table 1.

Manuscript revisions

We have added a comparison with Terminal-Bench Science in Section 2.2 (LLMs for Scientific Workflows), explaining its focus on scientific research workflows and our focus on repository maintenance and extension, the three task paradigms, and scientific-behavior validation. Table 1 now includes an entry for the corresponding version, and the related-work discussion cites public project materials. The chronology is explained in this response to clarify the timing of the original related-work coverage.

Q3. Cross-model failures and performance gaps

We further compared GPT-6 Astra and Claude-Opus-5 on the same tasks. They share 44 successes, with 16 GPT-only successes, 13 Claude-only successes, and 46 shared failures. The overall results of 60/119 for GPT and 57/119 for Claude include different success/failure outcomes on 29 tasks. The results by paradigm are shown below.

ParadigmGPT-6 AstraClaude-Opus-5
Issue-driven24/5220/52
Expert-exploratory28/4932/49
Engineering-integration8/185/18

Across the three paradigms, GPT solves four more Issue-driven tasks and three more Engineering-integration tasks, while Claude solves four more Expert-exploratory tasks. Their similar aggregate scores therefore include complementary performance on different task requirements.

The differences involve both scientific constraints and software behavior. On Task 019, GPT passes 13/14 private checks, with the single failure involving compatibility with a legacy-object entry point, while Claude passes 14/14. On Task 024, GPT passes 7/8 checks, with the failed check requiring zero probability for inputs that violate the model's constraints, while Claude passes 8/8. On Task 035, GPT passes 14/14 checks and Claude passes 8/14, with the failed checks involving continuous-state consistency, trajectory continuity, and event localization. On Task 094, which concerns magnetic-space-group setting transformations, GPT passes 4/5 checks and Claude passes 5/5. The difference involves the transformation of symmetry operations under a nonstandard setting.

To compare leading and lower-scoring configurations, we report specific differences between Qwen3.5-397B and GPT-6 Astra from the existing 119-task failure audit. Qwen3.5-397B has 15 exploration or surface-repair failures, 33 repair-coverage or system-integration failures, and 28 failures of scientific-knowledge generalization, compared with 1, 19, and 16, respectively, for GPT-6. These counts describe differences in extending a local repair across complete code paths and generalizing it to changed conditions. Each failure is assigned one primary category under the existing audit rules. The comparison concerns the evaluated configurations on the same 119 tasks.

Our concrete hypothesis for the smaller model gap is that existing repository code and public reproduction entry points provide useful engineering support to lower-scoring models, while complete correctness across code paths, regressions, and changed conditions can still turn a leading model's partial repair into a task failure. This combination can let lower-scoring models solve some tasks with clear engineering entry points while limiting how much a leading model's scientific-reasoning advantage translates into additional complete successes. For example, Qwen3.5-397B scores 96.64% on public tests and 14.29% in complete success, showing that this configuration can handle many public reproduction scenarios while complete scientific repair remains difficult.

The scientific requirements and scoring mechanism behind this explanation are visible in the task-level results. On Task 019 above, GPT satisfies 13/14 private checks but loses complete-task credit because of one compatibility path. On Task 094, it satisfies 4/5 checks but misses a requirement under a nonstandard scientific representation. Pass@1 requires every private check to pass. On tasks where both models miss at least one required behavior, stronger local reasoning or more passed checks therefore yields no additional complete-task credit. This provides a concrete mechanism through which some capability advantages can remain absent from the task-level score gap.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer WBAX (1/2) - Q1, Q2

Q1 / W1. Domain coverage and evaluation value

These tasks provide executable, inspectable capability evaluations for the scientific scenarios they represent. Our main model comparisons use all 119 tasks and the three groups of 52 Issue-driven, 49 Expert-exploratory, and 18 Engineering-integration tasks. The paradigms examine repair of known scientific defects, autonomous investigation of unfamiliar scientific discrepancies, and complete integration of cross-module scientific workflows. The 20 domain labels describe the scientific objects and computational workflows represented by these tasks.

For domains represented by only one or two tasks, the available sample is insufficient for an unbiased and stable estimate of domain-wide capability. Section 3 and Appendix D clarify this scope while retaining the tasks' value for evaluating specific research problems. Each task has an executable environment, explicit scientific requirements, and public and private tests. The individual results allow inspection of whether a model satisfies the scenario's scientific invariants, boundary conditions, representation consistency, and regression requirements.

The strongest configuration in Table 2 scores 100.00% on public tests and 50.42% in complete private-test success, exposing the gap between passing visible examples and satisfying the full repair requirements. Paradigm-specific results and the four failure mechanisms in Table 3 further identify scientific abstraction, exploration, repair coverage, and generalization across changed conditions as sources of difficulty. Tasks from 98 repositories make these issues observable and inspectable across diverse real scientific software.

Manuscript revisions

In the task-composition discussion in Section 3 (SWE-bench Science), we have clarified the priority of covering scientific problems and engineering requirements across domains. We explain that the 20 domains are fine-grained labels for scientific task content and that a single contributing source can span multiple labels. The overall and paradigm-specific results concern the 119 tasks and the three groups of 52/49/18 tasks. The caption of Figure 1(a) and Appendix D (Scientific Domain Coverage) explain that the domain distribution describes task-set composition. The main text and appendix state the sample-size limitations on capability assessment in sparsely represented domains.

Q2 / W2. Evidence for the scientific-information effect

We retain the empirical conclusion that the effect of scientific materials varies across the evaluated configurations and can be negative. The supplementary experiment repeats the high and max reasoning settings on the same 91 tasks with the same model and harness. The material effect is positive in all three high-effort runs and negative in all three max-effort runs. The revised conclusion therefore has repeated experimental support. General laws across a broader range of models and mechanistic explanations involving weaker-performing configurations and deeper reasoning are presented as preliminary interpretations, with their scope specified in the Abstract and Conclusion.

The original DeepSeek evaluation reused an existing shared deployment to control running costs, first for the 91-task ablation and then for the with-information evaluation extended to all 119 tasks. A deployment review confirmed that both evaluations used a quantized model. The revised evaluation uses non-quantized DeepSeek-V4-flash-0731, with three runs in each of the four high/max and with/without-information conditions on the original 91 tasks, plus one complete 119-task with-information evaluation at each effort level.

At high effort, mean Pass@1 is 35.90% with information and 31.14% without it, giving a gain of 4.76 percentage points. At max effort, the corresponding values are 37.00% and 39.93%, giving a material effect of -2.93 percentage points. The direction is consistent across the three runs at each effort level. Individual run success counts for the four conditions are shown below.

Reasoning effortScientific materialsRun 1Run 2Run 3Mean Pass@1Sample SD, percentage points
highWith33/9132/9133/9135.90%0.63
highWithout29/9128/9128/9131.14%0.63
maxWith33/9134/9134/9137.00%0.63
maxWithout36/9136/9137/9139.93%0.63
AuthorsOpenReview ↗

Response to Reviewer WBAX (2/2) - Q2

Q2 / W2. Evidence for the scientific-information effect (continued)

The sample standard deviation of Pass@1 is 0.63 percentage points in each of the four conditions, calculated across the three runs with an n−1 denominator. Pairing runs as recorded in the task-level matrices gives material effects of +4, +4, and +5 tasks at high effort, and −3, −2, and −3 tasks at max effort. The sample standard deviation of the paired effect is also 0.63 percentage points at each effort level. These quantities describe between-run variation on the same 91-task set. The three-run results support a repeated observation of the material-effect direction within the evaluated configurations, with conclusions scoped to these configurations, tasks, and material bundles.

The experiment's value lies in examining material benefits alongside a model's standalone repair capability on the same task set. Opposite material effects at two reasoning depths of the same model and harness make the usefulness of scientific materials an empirical question evaluated through complete repair outcomes. This provides concrete evidence for understanding when external materials can supplement repair capability and when their incremental value declines. The paper uses may and suggests to discuss this relationship and explicitly specifies the evaluated configurations, tasks, and materials.

These results further characterize the relationship between weaker-performing configurations and deeper reasoning. Without information, increasing effort from high to max raises mean Pass@1 by 8.79 percentage points. With information, the corresponding increase is 1.10 percentage points. The high-effort configuration with the lower standalone baseline benefits from the materials, while their incremental value declines in the max-effort configuration. We therefore discuss the possibility that weaker-performing configurations may depend more on external materials and that deeper reasoning may reduce their incremental value, with the scope restricted to the evaluated configurations, tasks, and materials. In the original single-run GPT-5.6-sol comparison, 29/91 tasks are solved with information and 33/91 without it. Pass@1 decreases from 36.26% to 31.87%, a reduction of 4.40 percentage points.

We also completed non-quantized reruns on all 119 tasks with information. High effort solves 40/119 tasks (33.61%), and max effort solves 42/119 (35.29%). These full-benchmark scores provide the updated DeepSeek-V4-flash main-evaluation results. The incremental material effects are assessed through the repeated paired with/without-information comparisons on the same 91 tasks.

We also analyzed failure mechanisms in one pair of max-effort runs with and without scientific information, examining the trajectories and patches of all 55 unsuccessful attempts without information and all 57 with information. Under the four definitions in Table 3, the without/with-information counts are 21/11 for knowledge or abstraction, 3/5 for exploration or surface repair, 21/21 for repair coverage or system integration, and 10/20 for failure of scientific-knowledge generalization. Among the 49 tasks that fail in both conditions, 14 change primary category, including seven with the same private-test pass count and failed tests. Section 6.2 reports this failure-mechanism analysis, and Appendix C.2 identifies the paired runs used. Table 4 in the main text reports the three-run means and standard deviations, Figure 4 reports mean scores and token usage, and Figure 5 illustrates task-level success overlap for one pair of runs per configuration.

Manuscript revisions

We have updated the material-effect conclusions in the Abstract, the summary in the Introduction, the third contribution, the final paragraph of Section 6.2, and the Conclusion. Section 6.2 and Table 4 in the main text report the numerical results, three-run means, and standard deviations. Figure 4 presents mean scores and token usage, Figure 5 illustrates task-level success overlap, and Appendix C.2 describes the experimental settings. Section 6.2 discusses baseline performance and reasoning depth using the high-to-max gains of 8.79 percentage points without information and 1.10 percentage points with information. The interpretation involving weaker-performing configurations and deeper reasoning is scoped to the evaluated model, tasks, and materials.

AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their constructive feedback. We have substantially strengthened the manuscript through additional experiments, systematic analyses, and clarifications. Our key revisions are summarized as follows:

  1. Strengthened scientific validity and quality control. We clarified the research-driven task sourcing, domain-informed review procedures, and separation between scientific reviewers and evaluator implementers. All 119 tasks underwent scientific review.

  2. Provided direct evidence of evaluator reliability. In a stratified validation of 60 tasks, all 60 reference repairs and 60 alternative valid implementations passed, while all 120 targeted incorrect implementations were rejected, supporting the evaluators' ability to recognize scientifically valid alternatives and detect critical errors.

  3. Expanded benchmark comparisons and cross-model failure analyses. We clarified our distinction from Terminal-Bench Science and provided task-level analyses across GPT, Claude, Qwen, and DeepSeek, identifying differences in scientific reasoning, repair coverage, system integration, and generalization.

  4. Strengthened findings through repeated ablation experiments. We conducted three runs for each of four conditions across 91 tasks. Scientific materials improved Pass@1 by 4.76 percentage points at high effort but reduced it by 2.93 points at max effort, with consistent directions across repetitions. We also investigated failure mechanisms and carefully bounded the conclusions.

  5. Improved transparency and clarified limitations. We added paired input examples, detailed auxiliary-material inventories, token statistics, provenance and licensing information, and explicit discussions of domain sparsity and generalizability.

Together, these revisions directly address the reviewers' central concerns and provide substantially stronger empirical support for SWE-bench Science as a scientifically grounded, reliable, and challenging benchmark for scientific software engineering agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.