acceptodds
Withdrawn from ICLR 2027

What Does a Passing Check Protect? Contract-Based Auditing of Research Claims

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A research workflow can pass its checks and still break the requirements behind a paper's claims. We introduce GateCheck, a contract-based framework for auditing this gap in research compendia, the repositories released with papers. It describes each check by what it protects, its reference, what it observes, when, and for which artifact. A witness that never reads the verdict labels each accepted fault, per artifact, as restored, escaped or unresolved. Five compendia by other authors give three findings. First, a pass and a protected claim are different things: the reference can move with the edit, the check may not observe the violation, or a build fixes the output but not the commit. Two checks that reject equally many faults left versus violations in the outputs. Second, complementary requirements need complementary checks. Re-deriving labels from their evidence and matching records against an approved list caught disjoint faults on one compendium; together they rejected every fault we made there and, given declared approvals, allowed every permitted change, but also an incomplete removal. Third, a correct diagnosis need not add protection. On a compendium with a different verifier, the contract reading predicted the native check's verdicts, but the reused repairs caught nothing it missed. Verification should be judged by the requirements and artifacts it protects, not by how many mutations it rejects.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mutation testing, defect injection, computational reproducibility, research compendia, scientific software verification, cross-artifact consistency.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8TpXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that "the check passed" and "the claim is protected" are different propositions. It introduces GateCheck, which describes any check by a five-part verification contract, namely: 1) the requirement it protects, 2) the reference it compares against, 3) what it observes, 4) at which stage, and 5) for which artefact. It then classifies each injected fault per artefact as restored, escaped, or unresolved. The paper states that a witness judges this without reading the check's verdict. The main findings of the paper are that 1) a pass need not mean a protected claim, 2) complementary requirements need complementary checks, and 3) a correct diagnosis need not add protection.

Critical strengths
  • I find that the paper identifies an under-examined problem. The distinction it draws between "the check passed" and "the claim is protected" is intuitive and useful.

  • The framework is clearly motivated.

  • The experiments are run primarily on repositories released by other authors, and the paper is careful to describe what each case does and does not support.

  • The limitations are clearly stated, and the artifacts are released.

Critical weaknesses
  • Although ML provides the setting, the research questions, methodology, and evaluation are primarily grounded in empirical software engineering and meta-science. The framework is domain-agnostic, which is a strength, but the paper provides limited insight that appears specific to machine learning. This is also reflected in the related work, which draws substantially from the FSE/ISSTA community. As written, the paper seems more naturally aligned with venues such as ICSE, FSE, ISSTA, or EMSE.

  • The related work section appears too limited. It consists of roughly one paragraph and four references (citation contexts), and does not sufficiently position the contribution relative to the broader relevant literature.

  • Parts of the paper are difficult to follow. In particular, the figures do not sufficiently clarify the relationships among the verification contract, injected faults, witnesses, artefacts, and outcome labels.

  • The paper does not establish to what extent each component of the five-part verification contract is necessary or independently contributes to the framework's conclusions.

  • The claimed independence of the witness is weaker than the presentation suggests. The paper guarantees verdict independence ("a witness that never reads the verdict"), but this should not be conflated with broader forms of independence. For example, the paper states that "the same team wrote both implementations." This does not violate verdict independence, but it permits shared assumptions or correlated implementation errors between the system being checked and the witness. The paper should more clearly delimit what its independence guarantee does and does not establish.

  • Some claims appear broader than what the counterexamples and case studies directly support. A constructed counterexample is sufficient to refute a universal implication such as "a passing check implies a protected claim." Still, it does not by itself establish how prevalent or practically consequential the failure mode is. The paper should distinguish more clearly between logical counterexamples and empirical findings from the studied repositories.

Questions for authors

The abstract says "five compendia by other authors give three findings," but Section 6 says the findings rest on three, with two used for transfer, one of which shares an author. Please align.

Reviewer_jQJ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Research code often ships with a check, such as "rebuild the results and confirm nothing changed", and a passing check is read as "the numbers are right". The authors deliberately introduced errors into public research repositories and used an independent re-computation to see whether each error survived in the committed files or in the rebuilt outputs. A check can pass while an error "survives": it may compare against a reference that the same edit changed, look at something other than the claim, or let the rebuild fix the output while the committed file stays wrong. Two checks designed together caught every error the authors made in one repository; in another, reusing the fixes added nothing.

Critical strengths

The question is real and practical: a check can pass while the claim it seems to guard is wrong, and the paper's opening example shows this in a published repository, where make output && git diff exits 0 after a one-line change moves a reported accuracy. Asking of any check what it compares against, what it reads, and whether its verdict covers the committed files or the rebuilt output is a simple and useful lens for artifact evaluation. The studies seem to be carefully written: every synthetic error has an unedited control, protocols were committed before the runs, the body states its scope and it reports its own null result. The released records are detailed enough to check the tables.

Critical weaknesses

The paper is a weak fit for ICLR. Machine learning is a use case rather than a component of the contribution. Many things in the paper are asserted without supporting evidence, including fairly strong statements such as "When the check passes, a reader tends to trust the claims". The paper is hard to read and the abstract does not name the field or the question and relies on prior knowledge and several terms that are not self-explanatory (although the introduction is clearer). The submission - or at least its supplement - is laid out like a lab notebook. The introduction assumes rather too much of the reader and the motivating example is extremely specific without clearly explaining what make output && git diff actually means and its relevance to research compendia in general - a few types of which should be named before diving into such technical detail. The evidence consists of counterexamples judged against a reference that is not independent.

Questions for authors

What evidence supports the premise that readers, reviewers or artifact-evaluation committees treat a passing check as validation of a paper's claims, and how common are such checks among research compendia?

What is the ML-specific contribution and why is this paper appropriate for ICLR?

How independent is the witness of the checks it grades, given that for claim drift it compares against a frozen reference (p. 3) and for the label rule it re-implements the check (App. P)?

Reviewer_5p7fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GateCheck, a contract-based framework for auditing whether checks in scientific research repositories actually protect the claims they are intended to verify. Its verification contract consists of requirement, reference, observation, stage and artifact. An independent witness evaluates whether an accepted mutation leaves the protected requirement restored, violated or unresolved. Using research compendia associated with published work, including data-analysis, graph-learning and agent-evaluation code, controlled mutations show how checks pass despite incorrect claims, how complementary mechanisms protect different requirements, and why a detailed diagnosis need not add protection. The paper argues for evaluating checks by protected requirements and artifacts rather than raw rejection counts.

Critical strengths

S1. The paper presents useful verification-contract abstraction. A check’s implemented condition is separated from the scientific requirement behind it. This is important when generated output and committed artifacts differ.

S2. Interesting empirical study and interpretation. The analysis separates rejected, escaped, restored and unresolved faults, plus permitted changes. Complementary verification and selected counterexamples are helpful to verify the findings.

Critical weaknesses

W1. Generality remains limited by selected case studies.

The study uses five research compendia, often with faults chosen after inspecting code (Section 3, p. 4). Some findings follow directly from deliberately selected faults and checks. While the paper acknowledges these are counterexamples, not representative rates, proper broader characterization organized in terms of contract type, fault families, targeted goal for auditing and potential guarantees would be necessary.

I found the reference Shulepov, “Mutation Testing for Reproducibility Safeguards in Machine Learning Research Software: An Empirical Study” (2026), is discussed in the paper. Prospective testing on independent repositories, with predefined contracts and fault families, may better establish discovery of previously unknown gaps.

W2. The real practical benefit is not made clear and remain uncertain beyond simpler verification.

Section 4.4 finds richer typed claim representation detected nothing beyond a numeric comparison at the same reading point; Section 4.5 finds reused repairs on another verifier caught nothing extra. These are informative negative results. But these alone does not necessarily show that diagnosis lead to or translate into protection that is the main goal with practical benefit.

Constructing scientific requirements and independent witnesses (Section 2) also demands domain expertise. Practitioner effort and added benefit over simpler mutation-testing and reference-checking workflows are not evaluated.

Questions for authors

Q1 (W1, Section 3, p. 4; Section 6). How does the method generalize to unseen repositories and requirements? Could contracts and fault families be fixed before a prospective evaluation?

Q2 (W2, Sections 4.4–4.5). What practical advantage does contract-based diagnosis offer relative to simpler independent reference comparisons and mutation testing, given cases where richer diagnoses or repairs add no detection?

Q3 (W2, Section 2). What manual effort is needed to specify contracts and witnesses, and how sensitive are conclusions to errors in those specifications?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.