acceptodds
Under review as a conference paper at ICLR 2027

SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

Abstract

SciCode evaluates whether language models can turn advanced scientific reasoning into working numerical code. Yet recent frontier models cluster near 60% subproblem accuracy, making progress difficult to distinguish. We show that this compression is partly caused by defects in the benchmark. We conduct a problem-by-problem, domain-expert audit of all 65 test problems and identify 264 defects. Of these, 192 reject correct instruction-following solutions and affect 91% of the main problems, through non-reproducible or incorrect gold values, over-tight tolerances, hidden numerical conventions, or contradictions between prompts and graders. Most score-suppressing defects require specialized physics or mathematics knowledge to detect. We correct every repairable defect, remove the one problem whose specification admits no verifiable target, and release the resulting SciCode-Verified benchmark with an itemized audit trail and reproducible grading harness. The corrections add the constraints required for well-posed tasks, repair grading, and strengthen tests that were too lenient; they do not lower the scientific difficulty of the problems. Re-evaluating twelve frontier models under a matched pass@1 protocol raises subproblem accuracy from 45–60% to 84–98% and main-problem accuracy from 9–27% to 69–92%. These results show that benchmark defects can compress both absolute scores and model rankings, and that domain-expert verification is necessary when evaluation targets scientific reasoning and code jointly. SciCode-Verified provides a more faithful and discriminating instrument for measuring scientific coding.

Primary area:
datasets and benchmarks
Keywords:
scientific coding, benchmark verification, benchmark evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_6JXsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits the 65 test problems of SciCode and reports 263 defects. Most of them cause correct solutions to be rejected, through wrong gold answers, tolerances that are too tight, tests tied to a particular random number sequence, or conventions that the prompt never specifies. The authors argue that most of these defects can only be spotted with domain expertise. They fix the defects, drop one problem that cannot be made verifiable, and release SciCode-Verified together with a log of every change. Twelve frontier models are re-evaluated on both versions: subproblem accuracy rises from 45–60% to 84–98%, main-problem accuracy rises sharply as well, and the ranking of the models changes.

Critical strengths
  1. This is a valuable contribution for a widely used benchmark. The per-problem log makes every correction checkable, and many fixes are clearly right. Examples include missing physical constants, a sign error that made a transfer matrix non-unitary, and tolerances set below double precision. Weak tests are also tightened, so the corrections do not only go in one direction.

  2. The before/after comparison uses a matched harness for ten of the twelve models, which isolates the effect of the benchmark itself. Replacing tests that depend on an exact random sequence with physically meaningful statistical checks is good practice. The analysis of what the background text provides also gives some insight into what models still lack.

Critical weaknesses
  1. My main concern is that the audit lacks external validation. It is not stated whether the auditors could see model failures while looking for defects, and more than half of the defects were recovered from the diff after the changes had been made and only then confirmed. All reviews were done by the authors and no agreement measure is reported, so "independently re-checked" in the abstract should be qualified.

  2. The claim that the corrections preserve difficulty is never measured. Many fixes resolve an unspecified convention by writing the gold solution's choice into the prompt, which makes the problems more fully specified.

Minor: most results are single pass@1 runs, and some highlighted ranking changes, such as Seed 2.1 Pro dropping from first to sixth, rest on one-problem gaps.

Questions for authors
  1. If some models were consulted, do the gains hold for models that were not?
  2. Did you check that the relaxed tests still reject typical wrong solutions?
  3. How many fixes add method-level information to the problem statement rather than notation?
5 comments
AuthorsOpenReview ↗

1. Audit chronology and catalogue compilation

My main concern is that the audit lacks external validation. It is not stated whether the auditors could see model failures while looking for defects, and more than half of the defects were recovered from the diff after the changes had been made and only then confirmed. All reviews were done by the authors and no agreement measure is reported, so "independently re-checked" in the abstract should be qualified.

The audit and corrections were carried out by two authors with relevant domain expertise. In each round, we reviewed the problem statements, reference answers, and tests for defects. We also generated and examined model responses to understand how those defects affected the generated solutions and evaluation results.

The second round rechecked the first-round findings and proposed corrections, and also examined problems introduced during the initial repairs. For example, we revised prompt additions that explained derivations in excessive detail and made the tasks easier than intended.

The 153 diff-based entries refer to second-round checks of the first-round work. This count described how those changes were reviewed, not when the underlying defects were first discovered. Our earlier wording did not make this distinction clear, and we have removed the ambiguous provenance statement from the caption of Table 3.

Here, “independently re-checked by a second domain expert” refers to the other author re-examining the initial findings and proposed corrections.

We did not calculate inter-annotator agreement because the two authors did not produce separate, independent defect lists. Instead, the second round checked and revised the first-round findings and corrections.

AuthorsOpenReview ↗

2. Single-sample scores and ranking changes

Minor: most results are single pass@1 runs, and some highlighted ranking changes, such as Seed 2.1 Pro dropping from first to sixth, rest on one-problem gaps.

We have performed additional with-background evaluations and now report the three-run mean and sample standard deviation for six models. The scores below incorporate the test updates for 62.5 and 55.4 raised during review.

ModelRunsSubproblem accuracy (%)Main-problem accuracy (%)
DeepSeek V4 Pro Preview390.2 ± 1.578.1 ± 1.6
DeepSeek V4 Flash Preview383.7 ± 4.468.8 ± 5.6
GLM-5.2387.0 ± 3.275.5 ± 5.5
Kimi K3391.1 ± 2.679.7 ± 5.4
DeepSeek V4 Flash (0731)391.6 ± 0.381.8 ± 2.4
Seed 2.1 Pro388.5 ± 1.677.6 ± 3.6

Seed 2.1 Pro passes 51, 51, and 47 of the 64 main problems across its three runs, a spread of four problems. These fluctuations reinforce our caution against interpreting one-problem gaps as reliable differences in model rank.

AuthorsOpenReview ↗

3. Use of models during validation

We used Fable 5 during the repair process to help analyze how the identified defects affected model responses. In the second review round, the primary basis for checking the corrections was the problems themselves, the first-round changes, and the relevant scientific literature and textbooks. The corrections were not tailored to a particular model's responses.

The observed recovery is broad: all twelve evaluated models improve substantially after correction (Table 1). The corrected benchmark also reveals concrete differences in implementation reliability and scientific understanding. For example, in the saved with-background runs:

  • 75.2 — Graphene hopping parameters. Kimi K3 treated the one-dimensional displacement-index arrays as a two-dimensional array and attempted to access dn[:, 0], raising an IndexError.
  • 52.2 — Radial Schrödinger integration. DeepSeek V4 Pro Preview correctly chose to integrate from large to small radius but passed ascending evaluation points to solve_ivp, whose sampling order must follow the integration direction. This raised a ValueError before integration.
  • 21.2 — Semiconductor absorption. Claude Opus 4.8 evaluated the square root of a negative photon-energy-minus-bandgap value, producing NaN instead of zero absorption below the bandgap in the specified model.

GPT-5.5, GPT-5.6 Sol, and Gemini 3.5 Flash passed all three steps in their corresponding saved runs. The failing outputs terminated normally, and their preceding steps passed; these examples are not caused by output truncation or inherited upstream failures. They illustrate genuine remaining implementation and scientific-understanding errors, without implying that every small score gap establishes a stable model ranking.

Reviewer_3GfVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper attributes the recent plateau of SciCode (a widely-used benchmark even by many frontier model teams) scores to defects in the benchmark -- this matches many practitioner's general impression.

A per-problem domain-expert audit of all 65 test problems finds 263 defects. Among these, 193 cause correct, instruction-following solutions are rejected. This work reports that 78 percent of the score-suppressing defects need specialized physics or mathematics expert knowledge to detect. The authors correct every confirmable defect, record each change with its justification, and release SciCode-Verified. On twelve frontier model snapshots, subproblem accuracy rises from 45–60% to 84–98%, and main-problem accuracy rises from 9–26% to 69–92%. Validation consists of a ledger of all changes, a second-expert confirmation of each, regeneration of releases from source records, gold-solution self-tests, and an analysis of two execution environments.

Critical strengths

The defect audit is concrete and inspectable. The paper gives a taxonomy, the direction of each defect, and a per-problem ledger. Examples such as gauge-dependent exact comparisons, matching of random-number trajectories, malformed contracts, and impossible tolerances show real evaluation pathologies. A community relying on this benchmark gains from them. The revision process is traceable. Changes are ledgered and checked mechanically in both directions, and gold solutions are re-run. The two-environment analysis diagnoses false negatives caused by infrastructure. Both are rare in benchmark repair work.

Critical weaknesses

The oracle is validated only by the authors while the original paper has their artifacts produced by domain experts. The paper states that all reviews were conducted by the authors. (Please also feel free to correct me if my impression was wrong).

The paper calls no-background evaluation its most realistic setting, and corrected main-problem accuracy falls to 40.6–65.6% there. The title's claim should separate assisted from independent scientific coding. Nearly all scores are single samples under default sampling, so smaller gaps and rankings are not very well supported.

Questions for authors

q1: Will the authors obtain blinded external adjudication of a stratified sample of high-impact corrections, without exposing model outcomes? further, does the score recovery survive using only unanimously validated changes? q2: can you provide the paired original and verified results in the no-background condition for reference? q3: Do the ranking changes survive repeated sampling, with paired intervals on the shared 64-problem set?

3 comments
AuthorsOpenReview ↗

1. External validation and non-author feedback

Will the authors obtain blinded external adjudication of a stratified sample of high-impact corrections, without exposing model outcomes? further, does the score recovery survive using only unanimously validated changes?

Thank you for the constructive review. We respond to your questions below.

Colleagues in our research group with a physics background, who are not coauthors, reviewed some of our audit reports and agreed with the identified issues and corresponding corrections.

AuthorsOpenReview ↗

2. Original versus verified without background

can you provide the paired original and verified results in the no-background condition for reference?

Thank you for this suggestion. Our main before/after evaluation focused on the with-background setting used in Artificial Analysis's SciCode evaluation (methodology). We have now additionally evaluated DeepSeek V4 Flash Preview on the original benchmark without background and compare it with the existing no-background runs on SciCode-Verified.

For this comparison, both rows use the shared 64 problems and 287 scored subproblems, with the same two-environment pass-if-either grading rule:

Benchmark, without backgroundGeneration runsSubproblem accuracyMain-problem accuracy
Original SciCode136.6% (105/287)4.7% (3/64)
SciCode-Verified3, mean62.7%40.6%

On the full original test split, counting the excluded, unverifiable problem as failed, the original scores are 36.5% (105/288) and 4.6% (3/65), respectively. The common-set comparison above avoids attributing a denominator change to performance recovery.

The verified scores incorporate the strengthened test for subproblem 55.4 introduced during rebuttal. This changes the three-run subproblem mean from the submitted 62.8% to 62.7%, with no change to main-problem accuracy.

There is a serving-backend limitation: the new original run uses the April 23 Flash Preview model through OpenRouter/DeepInfra FP8, whereas the earlier verified runs used the official DeepSeek API. We therefore report this as a one-model reference comparison, not a fully matched-backend experiment or a result established for all twelve models. Original no-background results for the other models are not yet available.

The verified no-background scores also remain far from saturation; the near-saturation results with expert background should not be interpreted as near-perfect independent scientific coding.

AuthorsOpenReview ↗

3. Repeated sampling and ranking uncertainty

Nearly all scores are single samples under default sampling, so smaller gaps and rankings are not very well supported.

Do the ranking changes survive repeated sampling, with paired intervals on the shared 64-problem set?

We agree that single-run results do not establish stable fine-grained model rankings. For DeepSeek V4 Pro Preview and Flash Preview, we have three with-background runs on SciCode-Verified. We now report their mean and sample standard deviation in the text:

  • DeepSeek V4 Pro Preview: subproblem accuracy 90.2% ± 1.5 percentage points; main-problem accuracy 78.1% ± 1.6 percentage points.
  • DeepSeek V4 Flash Preview: subproblem accuracy 83.7% ± 4.4 percentage points; main-problem accuracy 68.8% ± 5.6 percentage points.

These repeats use the earlier harness revision disclosed in Table 1.

We have additionally completed two full with-background runs each for GLM-5.2, Kimi K3, DeepSeek V4 Flash (0731), and Seed 2.1 Pro on SciCode-Verified, incorporating the two test updates introduced during review (62.5 and 55.4). Each run covers all 287 scored subproblems and 64 main problems under the same two-environment grading rule. Combining the originally reported run with these two additional runs gives the following three-run mean and sample standard deviation:

ModelRunsSubproblem accuracy (%)Main-problem accuracy (%)
GLM-5.2387.0 ± 3.275.5 ± 5.5
Kimi K3391.1 ± 2.679.7 ± 5.4
DeepSeek V4 Flash (0731)391.6 ± 0.381.8 ± 2.4
Seed 2.1 Pro388.5 ± 1.677.6 ± 3.6

For the leading closed-source models, performance on SciCode-Verified is close to saturation, so small score differences should not be interpreted as a stable ordering of capabilities. We have made this qualification explicit in the text. Our main conclusion is the large and consistent recovery across models after benchmark correction, rather than the exact ordering of models near the ceiling.

Reviewer_TiJGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits the 65-problem SciCode test split and reports 263 defects, including 193 that can reject correct solutions. It distinguishes scientific from surface defects, repairs the verifiable problems, excludes one underdetermined problem, and records the corrections in a per-problem audit trail. Under a matched evaluation harness, twelve model snapshots improve from approximately 45–60 percent to 84–98 percent subproblem accuracy and from 9–26 percent to 69–92 percent main-problem accuracy. Additional evaluations distinguish performance with and without expert background and quantify the effect of grading across two software environments.

Critical strengths
  1. The substantive increment is a scientific audit and corrected resource, rather than a new general benchmark-repair methodology. Earlier work such as ELT-Bench-Verified already establishes the repair-and-reevaluate argument, and SciCode itself used expert validation. The contribution here is the test-split-wide re-examination, numerical-science-specific diagnoses, and inspectable correction ledger. Section 4's Householder, Lanczos, and spatial-filter examples illustrate why generic proofreading is insufficient. Appendix A's second adversarial check, reference self-tests, and bidirectional ledger/release checks make the proposed resource useful beyond a claim that another benchmark contains errors.

  2. The evaluation demonstrates a large and consistent practical consequence across twelve model snapshots. The authors rerun the original data under the comparison harness, disclose the excluded problem and denominator conventions, and avoid treating the public leaderboard as an exactly matched baseline. Table 2 also shows that removing expert background still reduces accuracy substantially, so the paper does not simply equate high with-background scores with unrestricted scientific competence. The size and consistency of the recovery make benchmark validity a consequential issue even without relying on small ranking differences.

Critical weaknesses
  1. Score recovery is not fully separated from added solution guidance. Despite Section 4's unchanged-difficulty claim, Appendix C adds BVP settings (43.3) and update formulas/Fourier propagation (55.1), potentially changing what models must infer. Mapping 164 new passes to corrections establishes provenance, but does not distinguish repaired grading from assistance introduced by revised prompts. The full gain therefore cannot yet be attributed solely to previously underestimated ability.

  2. The rejection power of some replacement tests remains insufficiently established. In 62.5, H's eigenvalues and the separate singular values of conn_Sz and conn_Sp do not alone ensure a consistent shared basis transformation. In 55.4, the described replacement checks field/spectrum shapes, with value checks confined to stripe observables. Useful wrong-variant checks elsewhere do not establish these comparators' adequacy. This is a validation gap, not demonstrated score inflation.

Questions for authors
  1. In the 164-new-pass attribution analysis, which cases pass when the original saved program is regraded with corrected tests, and which require generation from revised prompts? Where the interfaces remain compatible, is this distinction available separately for gold/test repairs and additions of algorithmic guidance? This would materially clarify how much of the recovery demonstrates false rejection of an already-correct output.

  2. For problem 62.5, what additional joint-operator or downstream checks rule out operators that preserve their separate spectra or singular values but do not share one consistent basis transformation? For problem 55.4, what checks exclude a numerically incorrect field or spectrum with the right shape and matching stripe summaries?

  3. Which corrections were already documented in upstream reports or proposed fixes at the evaluated source revision, and which scientific defects were newly established by this audit? In particular, can the ledger distinguish dataset/header corrections from the previously reported class-reference extraction failures? This would clarify the resource's concrete increment over existing maintenance work.

5 comments
AuthorsOpenReview ↗

1. Prompt corrections and solution guidance

Score recovery is not fully separated from added solution guidance. Despite Section 4's unchanged-difficulty claim, Appendix C adds BVP settings (43.3) and update formulas/Fourier propagation (55.1), potentially changing what models must infer. Mapping 164 new passes to corrections establishes provenance, but does not distinguish repaired grading from assistance introduced by revised prompts. The full gain therefore cannot yet be attributed solely to previously underestimated ability.

Thank you for the constructive review and for identifying two concrete gaps in the replacement tests. We address the concerns point by point below.

We explain the numerical requirements addressed by the two examples below, and report the grading-only comparison in our separate reply on regrading original programs.

55.1: specifying the discretization already selected by the gold target. The original background already split the Swift–Hohenberg equation into a local term and a spatial-derivative term, prescribed their order, and required the second update in Fourier space using FFT/IFFT. It did not specify the integration choices for these substeps: explicit Euler for the local term and exact exponential propagation for the linear Fourier term. The correction states these choices, their update formulas, and the unit grid spacing. The continuous equation, splitting order, tests, and stored targets are unchanged.

This omission is consequential because different discretizations of the same equation need not give identical finite-step arrays [1]. In the original second test, a spatially uniform initial field of one, ϵ=0.7\epsilon=0.7, Δt=0.01\Delta t=0.01, and T=0.05T=0.05 reduce the equation to du/dt=−0.3u−u3du/dt=-0.3u-u^3:

Solution at T=0.05T=0.05ValueAccepted by the original test?
Five explicit-Euler steps0.9390147346Yes; matches the gold
Analytical solution of the same equation0.9399020766No

Thus even the analytical solution fails: the grader selects a particular discretization, rather than a method-independent solution of the stated equation. Making that choice explicit removes a hidden requirement. The exponential factor follows from the already-given linear operator once exact integration of that substep is selected; it is not a new physical equation.

43.3: specifying the numerical problem and returned profile. The original task already required a BVP solution using the rate equations and boundary conditions from preceding steps, and listed solve_bvp as a dependency. However, it requested an inversion profile without specifying sampling positions, while grading against a 100-element array. Two implementations can solve the same continuous problem but return values at different positions or on different adaptive meshes. Specifying 100 equally spaced output positions defines the return contract, not the answer values. The requested output grid and the solver's adaptive internal mesh are distinct [2].

The other additions specify accuracy/resource settings, a 30 W signal initial guess, and saturation-power definitions used in the existing equations. The initial guess is not the computed output power. These additions are not all mere notation changes: initialization and explicit formulas can reduce numerical trial-and-error or inference. Their purpose is to make the numerical task and grading requirements explicit. The audit also regenerated the affected targets after identifying inaccuracies in the stored profiles.

Our repair logic is to resolve missing numerical definitions and output conventions that otherwise leave solvers guessing the grader's requirements, not to simplify the underlying scientific task. Our separate fixed-program comparison establishes recovery that requires no revised prompt or new generation.

References

  1. A.-K. Kassam and L. N. Trefethen. Fourth-Order Time-Stepping for Stiff PDEs. SIAM Journal on Scientific Computing, 26(4):1214–1233, 2005. DOI.
  2. SciPy 1.13.1 documentation, scipy.integrate.solve_bvp.
AuthorsOpenReview ↗

2. Regrading saved original programs

In the 164-new-pass attribution analysis, which cases pass when the original saved program is regraded with corrected tests, and which require generation from revised prompts? Where the interfaces remain compatible, is this distinction available separately for gold/test repairs and additions of algorithmic guidance? This would materially clarify how much of the recovery demonstrates false rejection of an already-correct output.

We group the 287 retained scored subproblems by direct changes to their prompts and/or grading. Here, prompt changes include descriptions, background, function documentation, and return requirements; grading changes mean test code and/or gold targets. The table reports passed subproblems in the separately generated original and verified runs for the four models in the attribution analysis, with background.

Direct changesSubproblemsGPT-5.5Gemini 3.5 FlashClaude Opus 4.8GLM-5.2
Prompt only10761 → 10562 → 10556 → 10051 → 89
Test code/gold only258 → 2211 → 2511 → 2410 → 23
Both7215 → 6715 → 6616 → 6116 → 55
Neither directly8376 → 7977 → 8369 → 8067 → 75
Total287160 → 273165 → 279152 → 265144 → 242

Of the 25 subproblems with changes only to test code and/or gold targets, eight remain after excluding those with prompt changes in preceding subproblems, since SciCode conditions each subproblem on earlier steps. On these eight subproblems, all four models improve from three passes to eight.

We also regraded the saved original programs on this subset, keeping all generated code—including preceding steps—unchanged. Under the same two-environment grading rule, each model again improves from three passes with the original tests to eight with the corrected tests. The five recovered subproblems are 5.1, 61.1, 64.2, 68.3, and 73.1; 14.2, 48.1, and 79.1 remain passing.

AuthorsOpenReview ↗

3. Consistent basis for the operators in 62.5

The rejection power of some replacement tests remains insufficiently established. In 62.5, H's eigenvalues and the separate singular values of conn_Sz and conn_Sp do not alone ensure a consistent shared basis transformation.

For problem 62.5, what additional joint-operator or downstream checks rule out operators that preserve their separate spectra or singular values but do not share one consistent basis transformation?

Thank you for identifying this gap. We agree that matching the operators' spectra separately does not guarantee a consistent shared basis. We have strengthened the 62.5 test to require a single unitary transformation that simultaneously maps the reference Hamiltonian and both connection operators to the returned operators, while retaining the existing checks. This preserves the freedom to choose a different basis while checking consistency across all three operators directly, without relying on a downstream test.

We validated the strengthened check with 50 common unitary transformations of a correct operator tuple, all accepted, and 50 inconsistent-basis constructions that preserve the separate spectra, all rejected despite passing the previous checks. This validation concerns the test's fixed four-dimensional, untruncated operator tuple.

We then re-tested 37 saved programs under both grading environments, without generating new answers. All old/new verdicts agree: previously passing programs remain passing, and previously failing programs remain failing. No existing subproblem or main-problem score changes.

Reviewer_nVnNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper identify SCICODE benchmark has defects, and after fixing the defects, the models can achieve at least 70% accuracy. The authors also release SCICODE-VERIFIED as a fixed version of SCICODE for future benchmarking usage.

Critical strengths
  1. The paper clearly identifiers the defects of SCICODE, clearly states that the lower performance of frontier models is majorly caused by defects instead of challenging tasks.
  2. Comprehensive reevaluation of models, and clear documentations of code and defects in the original benchmark (both code base and appendix)
Critical weaknesses
  1. The paper addresses the issue of SCICODE benchmark. However, after fixing the defects and introducing SCICODE-VERIFIED, I believe SCICODE-VERIFIED is not hard for a lot of "big" large language models, including opensource ones. SCICODE-VERIFIED lose the value of being a hard benchmark to test the LLM's capability on scientific coding questions. The authors does not propose a solution, or a potential solution to it.
Questions for authors

Q1. Related to the weakness, have you tried creating maybe like 10-20 harder questions in the scientific coding area for the models to solve? If there are hard enough questions, potentially we can use synthetic data generation and scale it, since SCICODE-VERIFIED already has a valid audit pipeline.

2 comments
AuthorsOpenReview ↗

Benchmark difficulty and future agentic evaluation

The paper addresses the issue of SCICODE benchmark. However, after fixing the defects and introducing SCICODE-VERIFIED, I believe SCICODE-VERIFIED is not hard for a lot of "big" large language models, including opensource ones. SCICODE-VERIFIED lose the value of being a hard benchmark to test the LLM's capability on scientific coding questions. The authors does not propose a solution, or a potential solution to it.

Q1. Related to the weakness, have you tried creating maybe like 10-20 harder questions in the scientific coding area for the models to solve? If there are hard enough questions, potentially we can use synthetic data generation and scale it, since SCICODE-VERIFIED already has a valid audit pipeline.

Thank you for this suggestion. SciCode is an established benchmark used in major model evaluations, including the Artificial Analysis Intelligence Index. Its defects therefore affect existing capability assessments and model comparisons. A systematic audit, correction, and re-evaluation of this widely used benchmark merits a dedicated study, independently of developing a harder benchmark. Finding that the corrected benchmark approaches saturation also provides a more accurate picture of its remaining difficulty.

Since completing the audit, we have begun developing a more challenging scientific-coding benchmark. We believe scientific-coding ability is better assessed through agentic tasks than through isolated QA-style code generation. We are therefore developing a standalone agent-based benchmark and plan to report it separately. We would be delighted to collaborate with interested researchers in the community to develop challenging scientific tasks and build this agent-based benchmark together.

Thank you for the reply. The reply addressed the concern partially. Changed to weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.