- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that "the check passed" and "the claim is protected" are different propositions. It introduces GateCheck, which describes any check by a five-part verification contract, namely: 1) the requirement it protects, 2) the reference it compares against, 3) what it observes, 4) at which stage, and 5) for which artefact. It then classifies each injected fault per artefact as restored, escaped, or unresolved. The paper states that a witness judges this without reading the check's verdict. The main findings of the paper are that 1) a pass need not mean a protected claim, 2) complementary requirements need complementary checks, and 3) a correct diagnosis need not add protection.
Critical strengths
-
I find that the paper identifies an under-examined problem. The distinction it draws between "the check passed" and "the claim is protected" is intuitive and useful.
-
The framework is clearly motivated.
-
The experiments are run primarily on repositories released by other authors, and the paper is careful to describe what each case does and does not support.
-
The limitations are clearly stated, and the artifacts are released.
Critical weaknesses
-
Although ML provides the setting, the research questions, methodology, and evaluation are primarily grounded in empirical software engineering and meta-science. The framework is domain-agnostic, which is a strength, but the paper provides limited insight that appears specific to machine learning. This is also reflected in the related work, which draws substantially from the FSE/ISSTA community. As written, the paper seems more naturally aligned with venues such as ICSE, FSE, ISSTA, or EMSE.
-
The related work section appears too limited. It consists of roughly one paragraph and four references (citation contexts), and does not sufficiently position the contribution relative to the broader relevant literature.
-
Parts of the paper are difficult to follow. In particular, the figures do not sufficiently clarify the relationships among the verification contract, injected faults, witnesses, artefacts, and outcome labels.
-
The paper does not establish to what extent each component of the five-part verification contract is necessary or independently contributes to the framework's conclusions.
-
The claimed independence of the witness is weaker than the presentation suggests. The paper guarantees verdict independence ("a witness that never reads the verdict"), but this should not be conflated with broader forms of independence. For example, the paper states that "the same team wrote both implementations." This does not violate verdict independence, but it permits shared assumptions or correlated implementation errors between the system being checked and the witness. The paper should more clearly delimit what its independence guarantee does and does not establish.
-
Some claims appear broader than what the counterexamples and case studies directly support. A constructed counterexample is sufficient to refute a universal implication such as "a passing check implies a protected claim." Still, it does not by itself establish how prevalent or practically consequential the failure mode is. The paper should distinguish more clearly between logical counterexamples and empirical findings from the studied repositories.
Questions for authors
The abstract says "five compendia by other authors give three findings," but Section 6 says the findings rest on three, with two used for transfer, one of which shares an author. Please align.