acceptodds
Under review as a conference paper at ICLR 2027

SCIRIGOR: EVALUATING OPEN-ENDED SCIENTIFIC ANALYSIS BEYOND FINAL SCORES

Abstract

Scientific coding agents produce code, results, figures, and written findings within a single run. A correct final claim can nevertheless accompany contradictory anal- ysis or incomplete evidence. We introduce SCIRIGOR, a benchmark that evalu- ates both claim correctness and support from the run’s computations, figures, and analysis. It contains 100 cases from 27 articles across six domains and 17 sub- fields, with source code, inputs, results, and figures for verification. An evidence graph connects each claim to numerical results, plotted comparisons, and written interpretations. The evaluator checks these records against reference requirements and tests whether they support one another. Each claim’s support is limited by its weakest required check, while approved alternatives accommodate valid methods and visual encodings. Across eight model–runner configurations, the evaluator identifies incomplete support in 64 of 357 submissions with perfect final-claim scores. The highest Path F1, which measures supported claims and finding cover- age, is 84.4%; no configuration exceeds 40.8% Strict success, which requires all mandatory checks to pass.

Primary area:
datasets and benchmarks
Keywords:
Scientific Data Analysis, Agent Evaluation, Cross-Artifact Consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_b2pjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Scirigor is a benchmark of 100 data-analysis cases drawn from 27 empirical articles across several domains. It evaluates whether the claims produced by a scientific coding agent are correct and, at the same time, whether they are supported by the evidence the agent itself produced. The agent submits artefacts such as code, results, and claims, and the evaluator links each claim to supporting evidence such as computations, plotted marks, and text. Each piece of evidence is assessed for reference fidelity and local support. The checks are then combined by taking the minimum over required checks and the maximum over approved alternatives, yielding a claim-level Path F1 and an all-or-nothing Strict metric. Cases are built from the original authors’ code and quality-controlled through two clean executions and a human check. Across eight model–runner configurations, 64 of 357 submissions with a perfect claim score have incomplete support, and no configuration exceeds 40.8% Strict. The authors validate the evaluator with an expert check of 78 submissions.

Critical strengths
  1. A principled, interpretable evaluation design. Separating reference fidelity from local support, and combining checks with min-over-required and max-over-alternatives rules, yields a scoring scheme that is easy to reason about and shows where support breaks down. The aggregation ablation (Table 3) shows that the minimum rule accounts for most of the drop from flat averaging, which supports this design. Figure 1 clearly motivates why final-answer scoring alone is insufficient.

  2. Careful, reproducible construction and candid reporting. The use of original author code, two clean runs, file-access traces, input hashes, and strict separation of agent-visible and evaluator-only files makes the benchmark more rigorous than many comparable ones. The authors also openly report their evaluator’s limitations, including its low failure-detection recall, which helps readers interpret the results correctly.

Critical weaknesses
  1. The evaluator’s accuracy, and therefore the size of the reported gap, is uncertain. On the random audit sample, failure-detection recall is 30% (3 of 10), and the 100% precision rests on only three detected failures. The audit also covers only local support, not Path F1 or Strict, so the headline Strict metric has no human validation. Errors could go in either direction: missed failures suggest the 64-of-357 gap is an undercount, while overly narrow references (see Weakness 2) could produce false failures. As a result, the main empirical claims are not yet on solid footing.

  2. Strict failures may partly reflect the design of the reference requirements rather than agent errors. The paper states that methods outside the approved alternatives “require expert review before acceptance and rescoring,” but it is unclear whether this review was applied to the evaluated submissions, or how often valid but unlisted methods were penalized. The large domain differences (e.g., 8.7% vs. 76.2% Strict for GPT-5.5 xhigh), together with the authors’ note that Strict depends on the number of mandatory requirements, suggest that case-level requirement design strongly influences the scores.

Questions for authors
  1. (Clarification; could affect my rating.) In the reported experiments, how many submissions used valid methods outside the approved alternatives, and were they expert-reviewed and rescored?

  2. (Clarification; could affect my rating.) Can you validate Strict and Path F1 against human judgment on a sample, and report failure-detection recall over a larger set of positive cases?

  3. (Clarification.) GPT-5.6 assisted with task drafting and translations during curation. Is there evidence that this does not advantage GPT-family models? Did agents have network access during evaluation, given that the source articles and code are public?

  4. (Clarification.) How did two cases with invalid reference findings pass the human audit, and does this suggest other undetected reference errors?

  5. (Suggestion.) Please add FrontierChallenge and NatureBench to Table 1, since they are the closest related benchmarks discussed in the text. Please also state in the Table 3 caption that it covers executed submissions only.

5 comments
AuthorsOpenReview ↗

Response (1/3)

We thank the reviewer for recognizing the principled scoring design and the careful, reproducible construction. Our answers in brief:

  • W1 / Q2. Experts judged 60 submissions directly against Strict. The evaluator matched them on 58 (κ = 0.92), with 100% precision and 95.6% recall over 45 failures. It also flagged 116 of 120 injected faults and passed all 120 harmless rewrites.
  • W2 / Q1. Experts reviewed 52 Strict failures and found 2 that involved valid methods outside the approved alternatives. Approving both leaves the 64-of-357 gap unchanged.
  • Q3. Every reference comes from the original authors' workflow, scoring compares scientific content rather than GPT-5.6's wording, and agents ran offline.
  • Q4. Each error is a single finding that does not match its task wording, and a new finding-level check of all 100 cases found no new errors.
  • Q5. We will update Table 1 and the Table 3 caption.

W1 / Q2. Human validation of Strict and Path F1

Expert judgment validates both metrics: the evaluator matches experts on Strict for 58 of 60 submissions and on the Path F1 gap in both directions, and it flags 116 of 120 injected faults while passing all 120 harmless rewrites.

Study 1: expert judgment on 60 submissions. Three experts, blind to evaluator scores, judged 60 submissions that all have a perfect final-claim score, so final-answer scoring would accept every one of them. To test both error directions, we sampled two groups:

  • Flagged group: 30 at random from the 64 the evaluator flagged for incomplete support (Path F1 below one), to check for false failures.
  • Other group: 30 at random from the other 327 submissions (all except the first group), to check for missed failures.

The experts reached a consensus label for each submission.

Strict (complete support plus every mandatory check, so it fails more often than Path F1: 177 vs. 64 of the 357 [Paper Section 5.2]):

ComparisonResult
Strict pass/fail, all 60 submissionsEvaluator and experts agree on 58 (Cohen's κ = 0.92)
Failures the evaluator reported43, all confirmed by experts (precision 100%)
Failures the experts identified45, of which the evaluator detected 43 (recall 95.6%)

Path F1 gap (whether evidence support is complete):

GroupExpert judgment
Flagged groupIncomplete support confirmed in 29 of 30
Other groupAgrees with the evaluator on all 30 (29 complete, 1 incomplete)

In both groups, the evaluator's support labels match expert judgment, supporting the 64-of-357 gap from both error directions.

Study 2: faulty and harmless variants. From 30 random submissions that pass Strict, so each starts with no detected errors, we built two kinds of variants:

  • 120 faulty variants, each with one injected error. The evaluator should flag them.
  • 120 harmless rewrites, which change only how the analysis is written, so the science stays correct. The evaluator should pass them.
VariantEvaluator result
Faulty: numerical error30 of 30 flagged
Faulty: plot-mapping error (a plotted mark differs from its computed result)29 of 30 flagged
Faulty: report count that contradicts the results (as in Figure 1)28 of 30 flagged
Faulty: confidence-interval error (wrong uncertainty bounds)29 of 30 flagged
Harmless rewrites120 of 120 passed

How this relates to the paper's 3/10. In the paper's audit of individual claim–evidence links, the evaluator flagged 3 links, all of them true failures [Paper Section 5.3]. Strict and Path F1 score whole submissions, and a submission fails when any of its required checks fails. At that level, the evaluator detects 95.6% of failures.

We will add both studies to Section 5.3 and Appendix K.

AuthorsOpenReview ↗

Response (2/3)

W2 / Q1. Valid methods outside the approved alternatives

Strict failures reflect genuine shortfalls under the benchmark requirements: only 2 of 52 reviewed failures involved a valid method outside the approved alternatives, and approving both leaves the 64-of-357 gap and the highest Strict of 40.8% unchanged.

How we reviewed. For this response, experts reviewed and rescored 52 Strict failures, following the paper's procedure for methods outside the approved alternatives (expert acceptance, then rescoring). The evaluator records where each failure first occurs along the evidence chain, from data and results to figures and claims [Paper Appendix Q]. Data and result checks are where a different valid method most directly shows up, so the experts reviewed:

  • all 32 failures at data or result checks (4 at data checks, 28 at result checks), and
  • 20 drawn at random from failures at visualization or claim checks.

What we found.

  • In Study 1 above, experts confirmed all 43 Strict failures the evaluator reported.
  • 2 of the 52 reviewed failures used a valid method outside the approved alternatives. Experts accepted both, and we rescored all 800 submissions with them added.
  • The 64-of-357 gap and the highest Strict of 40.8% are both unchanged.

Why valid methods pass. Requirements target scientific content rather than code:

  • Results are identified by scientific meaning rather than variable names or file locations [Paper Section 3.2].
  • Numbers are compared within specified tolerances, and figures by plotted quantities, groups, units, and uncertainty [Paper Section 4].
  • Each claim is scored by its best approved alternative (the max rule) [Paper Section 3.4].

Number of requirements and domain differences. Both observations are consistent with task content rather than requirement design:

  • Requirement count. Requirement count is one factor in Strict, but Strict has no monotonic trend with it. For GPT-5.5 (xhigh), Strict across three groups of cases, from fewest to most requirements, is 48.8%, 26.7%, and 41.7% by required results, and 56.8%, 21.6%, and 33.3% by visual requirements [Paper Appendix P].
  • Requirement narrowness. Only 2 of 52 reviewed failures involved a valid unlisted method, and approving both leaves the scores unchanged.

The domain contrast (8.7% in computing/information sciences vs. 76.2% in social/behavioral sciences) is therefore consistent with differences in the analyses each domain's cases require [Paper Section 5.1, Appendix A].

AuthorsOpenReview ↗

Response (3/3)

Q3. GPT-5.6-assisted curation and network access

Every reference comes from the original authors' workflow, scoring compares scientific content rather than GPT-5.6's wording, and agents ran offline.

Role of GPT-5.6. GPT-5.6 assisted curators with indexing, locating analysis operations, drafting task text and input documentation, and proposing Python translations of the author workflows. Each proposal entered a case only after verification against the sources:

  • Curators checked each proposal against the archived author workflow. Each case's code, data, figures, and claims come from the original article and author materials [Paper Section 4, Appendix G].
  • Each case passed two clean runs that match the source results exactly or within specified tolerances [Paper Section 4, Appendix H].
  • Curators checked the visible tasks for ambiguity and answer leakage [Paper Appendix H].

Scoring compares each submission with the authors' results by scientific meaning, not with GPT-5.6's wording or code [Paper Section 3.2], so familiarity with that wording earns no credit.

Network access. Agents ran offline while producing and running their submissions. They received a neutral question, the declared input files, and the scientific context needed to interpret the measurements. The source paper, author code, reference outputs, and scoring rules stayed hidden [Paper Section 4]. Each scored claim must be linked to operations, results, and plotted marks from the same run [Paper Section 3.2], so credit rests on the agent's own computation.

Q4. The two invalid reference findings

Each error is a single finding that does not match its task wording, and a new finding-level check of all 100 cases found no new errors.

Why they passed review. The final human review checks each case's question, figure panel, source passage, author code, inputs, reference outputs, and claims together [Paper Section 4, Appendix H]. In each case, these components describe the same analysis, and the registered finding (the reference conclusion a submission must reach) is a correct sentence from the matching source passage. The error lies only in how that sentence relates to the task wording:

  • Case 1: the task asks about human preferences in Experiment 3 (source Figure 4B), and the finding is the source's cross-reference "S9 for results from Exp. 2", which states no result.
  • Case 2: the task rules out conclusions about estimator performance, but the finding requires one (maximum-likelihood phase-recovery performance).

Other reference errors. We have since compared every required finding in all 100 cases with its visible task, and the check found no new errors. The experts in the new studies flagged no further issues. We will make this two-reviewer, finding-level check part of the release process.

Q5. Table 1 and Table 3

  • We will add FrontierChallenge and NatureBench to Table 1.
  • Table 3 covers executed submissions only, and its caption will state this.
Reviewer_Wu6rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

AI coding agents were given 100 data-analysis tasks rebuilt from 27 published studies, and each had to return its code, numbers, a plot and a written report. The authors' grading program checks not only the final claims but whether the numbers, the plot and the report agree with each other and with the original study. Eight model-and-tool combinations each tried every task once. Among the 357 attempts whose final claims were all correct, the grader flagged 64 as not fully backed by their own work. Human experts re-checked only some kinds of check, and on those the grader caught 3 of the 10 problems they found.

Critical strengths

The paper targets a real failure mode: an agent can state the right result while its own report or figure says otherwise, and final-answer grading counts that as success. Grading each claim against the numbers, plot data and report sentences that should support it is a sensible idea ("mark the working, not only the answer"), and the cases are built with care: they are rebuilt from 27 published studies with the original code re-run twice and the references hidden from the agents. The authors also report their evaluator's low recall openly. The numbers agree with each other almost everywhere.

Critical weaknesses

The opening example is impenetrable. I could make neither head nor tail of the example given in the opening paragraph of the Introduction. “Nature science news”? Names “predicted as East Asian”? Qwen3.8-Max? The same example is given in Figure 1 but leaves me none the wiser. Predicting ethnic origin from names is a contentious topic and an odd choice of example here (Lockhart, King and Munsch, “Name-based demographic inference and the unequal distribution of misrecognition”, Nature Human Behaviour 7(7):1084–1095, 2023). A neutral example would make the same point. (Perhaps also flippant remark, but one would hope that “scientific data analysis turns data into research findings” is (a) uncontroversial and (b) established earlier than the 2025 and 2026 sources cited for it, which are agent benchmarks.)

The submission format works against best practice. Agents must submit code, “results” (apparently separate from figures), a figure, a written analysis and a list of claims as separate artifacts. This is at odds with best research practice, which might promote literate programming instead (Knuth, 1984; Sweave, knitr, Jupyter, Quarto). where the numbers in the text are computed by the code that produces them. That would obviate such failures as in Figure 1.

The headline metric is not defined. Path F1 is defined in Eq. 4 but "Claim F1" apparently not. Raw accuracy for agreement score is perhaps not that helpful a statistic when most relations are fine. The LLM judge is also not identified.

Questions for authors

How are the Claim F1 and F_C(c) computed?

Why are percentages given to 1 decimal place?

Did the agents have web access?

6 comments
AuthorsOpenReview ↗

Response(1/2)

We thank the reviewer for the detailed review and for recognizing the failure mode SciRIGOR targets ("mark the working, not only the answer") and the care in case construction. In short: the opening example is a real submission on data from a published study, and we explain it below; SciRIGOR shares the goal of literate programming and also checks the step it leaves open; we define Claim F1 and F_C(c) and name the judge (GPT-5.4); and the agents had no web access.

W1. The opening example is hard to follow, and a neutral example would make the same point.

The example is a real evaluated submission whose final answer is correct while its own report contradicts it; we explain each element below. The case comes from a published study of science journalism, which compares people mentioned in Nature news articles with authors of Nature research papers. The agent receives the study's annual shares of names with a predicted East Asian origin for 2005–2020 and determines in how many of the 16 years the news share is lower. The predictions are input data from the source study; neither the agent nor our evaluator infers anyone's origin. Qwen3.8-Max is one of the eight evaluated agent configurations.

We chose it because the conflict is easy to verify: the results and plot show a lower news share in all 16 years, and the final answer says so, but the written report says zero years, while final-answer scoring marks the submission as correct. We agree with Lockhart et al. (2023) that such predictions are not verified ethnicity; Figure 1 and Appendix I.8 already state this. SciRIGOR checks whether the agent's own artifacts agree with each other and with the reference, independent of how the input shares were produced. Figure 2 illustrates the same problem, a correct answer with incomplete support, using a neutral heat-workload case whose confidence-interval evidence is missing.

We will cite Lockhart et al. in the Figure 1 caption, and rewrite the first sentence of the Introduction to state the task directly and cite agent benchmarks only where we discuss agent evaluation.

W2. The submission format works against literate programming.

SciRIGOR's six files are its evaluation interface, not a prescribed research workflow, and they are compatible with literate programming: a notebook or Quarto document can be evaluated once its code writes the six files. The submitted code must regenerate the results and figure from the inputs (Section 3.1), so every number stays tied to the computation that produced it. results.json holds those values; the figure plots them, and the report and claims cite them, so each can be checked individually. The Figure 1 failure shows why this extra check matters: the report sentence was itself generated by the agent's code, which inserted the number of years with a higher news share, so the report said 0/16 while the results said 16/16 (Appendix I.8). Literate programming ties each number to its code; SciRIGOR additionally checks that the sentence around it states the number correctly.

AuthorsOpenReview ↗

Response (2/2)

W3a / Q1. How are Claim F1 and F_C(c) computed?

Claim F1 is the final-answer score, and F_C(c) is its per-claim term. F_C(c) rates whether a generated claim c is correct relative to the reference findings, using the graded text scores of Section 3.4 and Appendix R: 1 if correct and appropriately scoped; 0.5 if partly correct or missing an important qualification; 0 if false, contradicted by the reference, or an unsupported extrapolation. Within each submission:

  • Claim P = mean of F_C(c) over the generated claims;
  • Claim R = mean coverage of the required reference findings by the generated claims taken together, rated on the same 1/0.5/0 scale;
  • Claim F1 = 2·P·R / (P + R), and zero when both are zero.

Main-table entries (Table 2) average these per-submission scores over cases, with execution failures scored zero (Section 5).

Path F1 (Eq. 4) has the same form but uses the support score S = min{S_comp, S_vis, F_C} (Eq. 3) for each claim and each required finding, where S_comp and S_vis take the best approved path for the claim's computation and plotted marks from the same run, and are 1 only if every required check on that path passes. A correct claim therefore still loses credit if its own computation or figure does not support it. This is the gap behind our headline result: 64 of 357 submissions with perfect Claim F1 have Path F1 below one (Section 5.2). We will add these definitions to Section 3.4.

W3b. The LLM judge is not identified.

The judge is GPT-5.4, which is not among the evaluated agents. It applies a fixed rubric to produce F_C(c), finding coverage, the score for the written report, and analysis–claim consistency. Numerical values, data lineage, execution, and plot bindings are checked by programs, and, as stated in Section 3.3, a text judgment cannot override a numerical, structural, or provenance failure.

W3c. Raw accuracy is not informative when most relations are fine.

The paper already reports chance-corrected statistics alongside raw agreement: Cohen's κ = 0.798 between the two experts, and the evaluator's failure-detection precision, recall, and F1 (Section 5.3), with full confusion counts in Table 11.

Q2. Why are percentages given to one decimal place?

One decimal is a display format, not a claim of precision; every main-table score can be traced to per-case scores in the supplementary release (Reproducibility Statement).

Q3. Did the agents have web access?

No. Network access was disabled, and, as described in Section 4, agents saw only the question, the study context, and the declared input files; the source papers, author code, reference outputs, and scoring rules were hidden.

References.

  • Lockhart, King & Munsch 2023, Name-based demographic inference and the unequal distribution of misrecognition, Nature Human Behaviour 7(7):1084–1095.

I still think that it is quite a convoluted example to lead with. I am not saying it is not a correct example of the phenomenon, but that for pedagogical value you might be better off with a toy example and then referring to this as a real one.

Isn't this post-hoc literate programming though, rather than literate programming? Analogous to interpretable models versus post-hoc explanations?

Reviewer_kPJpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark that evaluates both claim correctness and support from the run’s computations, figures, and analysis.

Critical strengths
  • This paper introduces an important resource in a timely problem.
Critical weaknesses
  • The method is underspecified. What harnesses were used? What is its effect?
  • It is not clear having the claim path is ideal. Is the path unique?
Questions for authors

Other related work includes: Veritas: Towards a General-Purpose Replication Tool for Scientific Research by Liu et al. ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers? by Ye et al.

7 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the review and for the pointers to related work. In short: in our setting, the harness barely changes SciRIGOR scores; the claim path exposes correct claims that lack support and accepts alternative implementations, methods, and evidence; and we will add VERITAS and ReplicationBench.

W1. Method details: what harnesses were used, and what is their effect?

SciRIGOR evaluates agent systems, and its evaluator works with any harness. A model, its harness, and its reasoning setting together produce the six artifacts (code, results, figure, analysis, claims, provenance), and the evaluator scores only those artifacts. The main table (Table 2) therefore compares eight model–harness configurations.

Both harnesses can execute code, and all agents received identical inputs. As in Section 5, five configurations use the Codex CLI harness: GPT-5.5 at low, high, and xhigh reasoning, and GPT-5.6 Sol at low and high. These agents run shell commands and edit files. Three use an API tool loop: Kimi-K3, Qwen3.8-Max, and Qwen3-Coder-30B-A3B. Its tools list input files, inspect data files, read table rows, and execute Python code. All agents had network access disabled, and, as described in Section 4, the paper, author code, references, and scoring rules were hidden.

Switching harnesses for the same model barely changes the scores. We re-ran Kimi-K3 and Qwen3.8-Max under the Codex harness on the same 100 cases with identical task inputs; the API-loop rows below are the main-table results (Table 2), and the Codex rows are new. Exec. is the share of cases whose program runs successfully. Claim F1 scores final claims only. Path F1 also requires each claim's computational and visual support from the same run. Strict requires every mandatory check to pass.

ModelHarnessExec.Claim F1Path F1Strict
Kimi-K3API loop77.066.656.114.3
Kimi-K3Codex79.067.254.516.3
Qwen3.8-MaxAPI loop77.058.736.511.2
Qwen3.8-MaxCodex78.061.235.113.3

The central finding, correct claims without complete support, holds under both harnesses: Claim F1 exceeds Path F1 by 10.5 and 12.7 points for Kimi-K3, and by 22.2 and 26.1 for Qwen3.8-Max. Across the two models, switching harnesses changes Exec. by one or two cases, Path F1 by less than two points, and Strict by two cases.

In our setting, SciRIGOR scores track the agent's capability far more than its harness. Under Codex, Path F1 is 35.1 for Qwen3.8-Max and 54.5 for Kimi-K3, rises from 73.1 to 79.0 for GPT-5.6 Sol (low to high), and rises from 55.2 to 71.1 to 84.4 for GPT-5.5 (low, high, xhigh; Table 2). By contrast, the harness switch above stays within two Path F1 points and keeps Kimi-K3 ahead of Qwen3.8-Max. Execution also depends on the model: every GPT configuration executes all 100 cases, versus 79 for Kimi-K3 and 78 for Qwen3.8-Max. The gap between Claim F1 and Path F1 appears for all seven Codex configurations (the five GPT ones and the two re-runs), from 5.5 points (GPT-5.5, xhigh) to 26.1 (Qwen3.8-Max). We will add these harness details and comparisons to Section 5.

AuthorsOpenReview ↗

Response (2/2)

W2. Is having the claim path ideal? Is the path unique?

The claim path catches failures that two simpler alternatives miss. Final-answer scoring cannot detect correct claims that the agent's own artifacts do not support: across the 800 main-table submissions, 64 of 357 with fully correct final claims lack complete support (Section 5.2; Figure 1 shows one). Requiring every check on the path keeps these gaps visible in the score. If instead each claim were scored by the fraction of its checks that pass, the same checks would give a pooled F1 of 94.7% instead of 68.4% Path F1 (Table 3), hiding most of the incomplete support.

The claim path is not unique by design: SciRIGOR fixes which relations must hold, not how an agent establishes them. Each primary claim needs computational and visual support from the same run, and the report must not contradict the claim. The question names the scientific quantities to compute, and, as specified in Sections 3.2–3.4 and 4, alternatives are accepted at three levels.

  • Implementation. Checks compare scientific quantities within a tolerance and verify links from execution traces, result values, and plot bindings, not variable names or file locations. Plot checks test which quantities, groups, units, and uncertainty are plotted, not pixels.
  • Method. When several estimators are scientifically equivalent, the relevant check accepts each of them. For example, a case fitting an asteroid's post-impact fading curve (Appendix I.6) accepts either the authors' curve_fit weighting or algebraically equivalent weighted least squares, and either an analytic crossing time or a numerical root.
  • Path. When valid evidence differs in structure, the score takes the best approved path, for example a line or a reference line for an impact marker.

Q. VERITAS and ReplicationBench

Thank you; both papers are complementary to SciRIGOR. We will add them to Section 2. ReplicationBench (Ye et al., 2025) evaluates paper-scale replication in astrophysics. VERITAS (Liu et al., 2026) is a general-purpose replication framework that judges a paper's claims against replication evidence. Both start from published work and ask whether its results can be reproduced. SciRIGOR instead hides the paper and author code, and evaluates whether the artifacts generated in the same run support the agent's own claims.

References.

  • Ye et al. 2025, ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?, arXiv:2510.24591.
  • Liu, Tjiaranata & Tan 2026, VERITAS: Towards a General-Purpose Replication Tool for Scientific Research, arXiv:2607.02931.

Thanks for the reply! It would still be useful to explore the harness part a bit more. I am going to keep my score as I think 2,3,3 is probably a reasonable overall assessment of this paper in my opinion.

Comments on the paper

1 comment
Area_Chair_9qUoOpenReview ↗

Please take a look at the response!

Dear reviewers,

As the rebuttal period is about to end, please take a look at the authors' responses.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.