- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Scirigor is a benchmark of 100 data-analysis cases drawn from 27 empirical articles across several domains. It evaluates whether the claims produced by a scientific coding agent are correct and, at the same time, whether they are supported by the evidence the agent itself produced. The agent submits artefacts such as code, results, and claims, and the evaluator links each claim to supporting evidence such as computations, plotted marks, and text. Each piece of evidence is assessed for reference fidelity and local support. The checks are then combined by taking the minimum over required checks and the maximum over approved alternatives, yielding a claim-level Path F1 and an all-or-nothing Strict metric. Cases are built from the original authors’ code and quality-controlled through two clean executions and a human check. Across eight model–runner configurations, 64 of 357 submissions with a perfect claim score have incomplete support, and no configuration exceeds 40.8% Strict. The authors validate the evaluator with an expert check of 78 submissions.
Critical strengths
-
A principled, interpretable evaluation design. Separating reference fidelity from local support, and combining checks with min-over-required and max-over-alternatives rules, yields a scoring scheme that is easy to reason about and shows where support breaks down. The aggregation ablation (Table 3) shows that the minimum rule accounts for most of the drop from flat averaging, which supports this design. Figure 1 clearly motivates why final-answer scoring alone is insufficient.
-
Careful, reproducible construction and candid reporting. The use of original author code, two clean runs, file-access traces, input hashes, and strict separation of agent-visible and evaluator-only files makes the benchmark more rigorous than many comparable ones. The authors also openly report their evaluator’s limitations, including its low failure-detection recall, which helps readers interpret the results correctly.
Critical weaknesses
-
The evaluator’s accuracy, and therefore the size of the reported gap, is uncertain. On the random audit sample, failure-detection recall is 30% (3 of 10), and the 100% precision rests on only three detected failures. The audit also covers only local support, not Path F1 or Strict, so the headline Strict metric has no human validation. Errors could go in either direction: missed failures suggest the 64-of-357 gap is an undercount, while overly narrow references (see Weakness 2) could produce false failures. As a result, the main empirical claims are not yet on solid footing.
-
Strict failures may partly reflect the design of the reference requirements rather than agent errors. The paper states that methods outside the approved alternatives “require expert review before acceptance and rescoring,” but it is unclear whether this review was applied to the evaluated submissions, or how often valid but unlisted methods were penalized. The large domain differences (e.g., 8.7% vs. 76.2% Strict for GPT-5.5 xhigh), together with the authors’ note that Strict depends on the number of mandatory requirements, suggest that case-level requirement design strongly influences the scores.
Questions for authors
-
(Clarification; could affect my rating.) In the reported experiments, how many submissions used valid methods outside the approved alternatives, and were they expert-reviewed and rescored?
-
(Clarification; could affect my rating.) Can you validate Strict and Path F1 against human judgment on a sample, and report failure-detection recall over a larger set of positive cases?
-
(Clarification.) GPT-5.6 assisted with task drafting and translations during curation. Is there evidence that this does not advantage GPT-family models? Did agents have network access during evaluation, given that the source articles and code are public?
-
(Clarification.) How did two cases with invalid reference findings pass the human audit, and does this suggest other undetected reference errors?
-
(Suggestion.) Please add FrontierChallenge and NatureBench to Table 1, since they are the closest related benchmarks discussed in the text. Please also state in the Table 3 caption that it covers executed submissions only.
5 comments
Response (2/3)
W2 / Q1. Valid methods outside the approved alternatives
Strict failures reflect genuine shortfalls under the benchmark requirements: only 2 of 52 reviewed failures involved a valid method outside the approved alternatives, and approving both leaves the 64-of-357 gap and the highest Strict of 40.8% unchanged.
How we reviewed. For this response, experts reviewed and rescored 52 Strict failures, following the paper's procedure for methods outside the approved alternatives (expert acceptance, then rescoring). The evaluator records where each failure first occurs along the evidence chain, from data and results to figures and claims [Paper Appendix Q]. Data and result checks are where a different valid method most directly shows up, so the experts reviewed:
- all 32 failures at data or result checks (4 at data checks, 28 at result checks), and
- 20 drawn at random from failures at visualization or claim checks.
What we found.
- In Study 1 above, experts confirmed all 43 Strict failures the evaluator reported.
- 2 of the 52 reviewed failures used a valid method outside the approved alternatives. Experts accepted both, and we rescored all 800 submissions with them added.
- The 64-of-357 gap and the highest Strict of 40.8% are both unchanged.
Why valid methods pass. Requirements target scientific content rather than code:
- Results are identified by scientific meaning rather than variable names or file locations [Paper Section 3.2].
- Numbers are compared within specified tolerances, and figures by plotted quantities, groups, units, and uncertainty [Paper Section 4].
- Each claim is scored by its best approved alternative (the max rule) [Paper Section 3.4].
Number of requirements and domain differences. Both observations are consistent with task content rather than requirement design:
- Requirement count. Requirement count is one factor in Strict, but Strict has no monotonic trend with it. For GPT-5.5 (xhigh), Strict across three groups of cases, from fewest to most requirements, is 48.8%, 26.7%, and 41.7% by required results, and 56.8%, 21.6%, and 33.3% by visual requirements [Paper Appendix P].
- Requirement narrowness. Only 2 of 52 reviewed failures involved a valid unlisted method, and approving both leaves the scores unchanged.
The domain contrast (8.7% in computing/information sciences vs. 76.2% in social/behavioral sciences) is therefore consistent with differences in the analyses each domain's cases require [Paper Section 5.1, Appendix A].
Response (3/3)
Q3. GPT-5.6-assisted curation and network access
Every reference comes from the original authors' workflow, scoring compares scientific content rather than GPT-5.6's wording, and agents ran offline.
Role of GPT-5.6. GPT-5.6 assisted curators with indexing, locating analysis operations, drafting task text and input documentation, and proposing Python translations of the author workflows. Each proposal entered a case only after verification against the sources:
- Curators checked each proposal against the archived author workflow. Each case's code, data, figures, and claims come from the original article and author materials [Paper Section 4, Appendix G].
- Each case passed two clean runs that match the source results exactly or within specified tolerances [Paper Section 4, Appendix H].
- Curators checked the visible tasks for ambiguity and answer leakage [Paper Appendix H].
Scoring compares each submission with the authors' results by scientific meaning, not with GPT-5.6's wording or code [Paper Section 3.2], so familiarity with that wording earns no credit.
Network access. Agents ran offline while producing and running their submissions. They received a neutral question, the declared input files, and the scientific context needed to interpret the measurements. The source paper, author code, reference outputs, and scoring rules stayed hidden [Paper Section 4]. Each scored claim must be linked to operations, results, and plotted marks from the same run [Paper Section 3.2], so credit rests on the agent's own computation.
Q4. The two invalid reference findings
Each error is a single finding that does not match its task wording, and a new finding-level check of all 100 cases found no new errors.
Why they passed review. The final human review checks each case's question, figure panel, source passage, author code, inputs, reference outputs, and claims together [Paper Section 4, Appendix H]. In each case, these components describe the same analysis, and the registered finding (the reference conclusion a submission must reach) is a correct sentence from the matching source passage. The error lies only in how that sentence relates to the task wording:
- Case 1: the task asks about human preferences in Experiment 3 (source Figure 4B), and the finding is the source's cross-reference "S9 for results from Exp. 2", which states no result.
- Case 2: the task rules out conclusions about estimator performance, but the finding requires one (maximum-likelihood phase-recovery performance).
Other reference errors. We have since compared every required finding in all 100 cases with its visible task, and the check found no new errors. The experts in the new studies flagged no further issues. We will make this two-reviewer, finding-level check part of the release process.
Q5. Table 1 and Table 3
- We will add FrontierChallenge and NatureBench to Table 1.
- Table 3 covers executed submissions only, and its caption will state this.
Response (1/3)
We thank the reviewer for recognizing the principled scoring design and the careful, reproducible construction. Our answers in brief:
W1 / Q2. Human validation of Strict and Path F1
Expert judgment validates both metrics: the evaluator matches experts on Strict for 58 of 60 submissions and on the Path F1 gap in both directions, and it flags 116 of 120 injected faults while passing all 120 harmless rewrites.
Study 1: expert judgment on 60 submissions. Three experts, blind to evaluator scores, judged 60 submissions that all have a perfect final-claim score, so final-answer scoring would accept every one of them. To test both error directions, we sampled two groups:
The experts reached a consensus label for each submission.
Strict (complete support plus every mandatory check, so it fails more often than Path F1: 177 vs. 64 of the 357 [Paper Section 5.2]):
Path F1 gap (whether evidence support is complete):
In both groups, the evaluator's support labels match expert judgment, supporting the 64-of-357 gap from both error directions.
Study 2: faulty and harmless variants. From 30 random submissions that pass Strict, so each starts with no detected errors, we built two kinds of variants:
How this relates to the paper's 3/10. In the paper's audit of individual claim–evidence links, the evaluator flagged 3 links, all of them true failures [Paper Section 5.3]. Strict and Path F1 score whole submissions, and a submission fails when any of its required checks fails. At that level, the evaluator detects 95.6% of failures.
We will add both studies to Section 5.3 and Appendix K.