- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits the 65 test problems of SciCode and reports 263 defects. Most of them cause correct solutions to be rejected, through wrong gold answers, tolerances that are too tight, tests tied to a particular random number sequence, or conventions that the prompt never specifies. The authors argue that most of these defects can only be spotted with domain expertise. They fix the defects, drop one problem that cannot be made verifiable, and release SciCode-Verified together with a log of every change. Twelve frontier models are re-evaluated on both versions: subproblem accuracy rises from 45–60% to 84–98%, main-problem accuracy rises sharply as well, and the ranking of the models changes.
Critical strengths
-
This is a valuable contribution for a widely used benchmark. The per-problem log makes every correction checkable, and many fixes are clearly right. Examples include missing physical constants, a sign error that made a transfer matrix non-unitary, and tolerances set below double precision. Weak tests are also tightened, so the corrections do not only go in one direction.
-
The before/after comparison uses a matched harness for ten of the twelve models, which isolates the effect of the benchmark itself. Replacing tests that depend on an exact random sequence with physically meaningful statistical checks is good practice. The analysis of what the background text provides also gives some insight into what models still lack.
Critical weaknesses
-
My main concern is that the audit lacks external validation. It is not stated whether the auditors could see model failures while looking for defects, and more than half of the defects were recovered from the diff after the changes had been made and only then confirmed. All reviews were done by the authors and no agreement measure is reported, so "independently re-checked" in the abstract should be qualified.
-
The claim that the corrections preserve difficulty is never measured. Many fixes resolve an unspecified convention by writing the gold solution's choice into the prompt, which makes the problems more fully specified.
Minor: most results are single pass@1 runs, and some highlighted ranking changes, such as Seed 2.1 Pro dropping from first to sixth, rest on one-problem gaps.
Questions for authors
- If some models were consulted, do the gains hold for models that were not?
- Did you check that the relaxed tests still reject typical wrong solutions?
- How many fixes add method-level information to the problem statement rather than notation?
5 comments
2. Single-sample scores and ranking changes
Minor: most results are single pass@1 runs, and some highlighted ranking changes, such as Seed 2.1 Pro dropping from first to sixth, rest on one-problem gaps.
We have performed additional with-background evaluations and now report the three-run mean and sample standard deviation for six models. The scores below incorporate the test updates for 62.5 and 55.4 raised during review.
| Model | Runs | Subproblem accuracy (%) | Main-problem accuracy (%) |
|---|---|---|---|
| DeepSeek V4 Pro Preview | 3 | 90.2 ± 1.5 | 78.1 ± 1.6 |
| DeepSeek V4 Flash Preview | 3 | 83.7 ± 4.4 | 68.8 ± 5.6 |
| GLM-5.2 | 3 | 87.0 ± 3.2 | 75.5 ± 5.5 |
| Kimi K3 | 3 | 91.1 ± 2.6 | 79.7 ± 5.4 |
| DeepSeek V4 Flash (0731) | 3 | 91.6 ± 0.3 | 81.8 ± 2.4 |
| Seed 2.1 Pro | 3 | 88.5 ± 1.6 | 77.6 ± 3.6 |
Seed 2.1 Pro passes 51, 51, and 47 of the 64 main problems across its three runs, a spread of four problems. These fluctuations reinforce our caution against interpreting one-problem gaps as reliable differences in model rank.
3. Use of models during validation
We used Fable 5 during the repair process to help analyze how the identified defects affected model responses. In the second review round, the primary basis for checking the corrections was the problems themselves, the first-round changes, and the relevant scientific literature and textbooks. The corrections were not tailored to a particular model's responses.
The observed recovery is broad: all twelve evaluated models improve substantially after correction (Table 1). The corrected benchmark also reveals concrete differences in implementation reliability and scientific understanding. For example, in the saved with-background runs:
- 75.2 — Graphene hopping parameters. Kimi K3 treated the one-dimensional displacement-index arrays as a two-dimensional array and attempted to access
dn[:, 0], raising anIndexError. - 52.2 — Radial Schrödinger integration. DeepSeek V4 Pro Preview correctly chose to integrate from large to small radius but passed ascending evaluation points to
solve_ivp, whose sampling order must follow the integration direction. This raised aValueErrorbefore integration. - 21.2 — Semiconductor absorption. Claude Opus 4.8 evaluated the square root of a negative photon-energy-minus-bandgap value, producing
NaNinstead of zero absorption below the bandgap in the specified model.
GPT-5.5, GPT-5.6 Sol, and Gemini 3.5 Flash passed all three steps in their corresponding saved runs. The failing outputs terminated normally, and their preceding steps passed; these examples are not caused by output truncation or inherited upstream failures. They illustrate genuine remaining implementation and scientific-understanding errors, without implying that every small score gap establishes a stable model ranking.
1. Audit chronology and catalogue compilation
The audit and corrections were carried out by two authors with relevant domain expertise. In each round, we reviewed the problem statements, reference answers, and tests for defects. We also generated and examined model responses to understand how those defects affected the generated solutions and evaluation results.
The second round rechecked the first-round findings and proposed corrections, and also examined problems introduced during the initial repairs. For example, we revised prompt additions that explained derivations in excessive detail and made the tasks easier than intended.
The 153 diff-based entries refer to second-round checks of the first-round work. This count described how those changes were reviewed, not when the underlying defects were first discovered. Our earlier wording did not make this distinction clear, and we have removed the ambiguous provenance statement from the caption of Table 3.
Here, “independently re-checked by a second domain expert” refers to the other author re-examining the initial findings and proposed corrections.
We did not calculate inter-annotator agreement because the two authors did not produce separate, independent defect lists. Instead, the second round checked and revised the first-round findings and corrections.