- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
ECP proposes a ten-stage preregistered governance workflow for AI evaluation, separating execution validity, evidence integrity, mechanical outcomes, and subsequent scientific interpretation. The paper integrates deterministic and model-based campaigns, seeded-fault tests, reclassification exercises, and third-party structured-output and prompt-injection audits. It emphasizes immutable records, explicit denominator rules, preserved failures, and bounded claims. Reported demonstrations include 60/60 schema-valid outputs in A17 and 141 unaccepted adversarial submissions in A16, whose final registered verdict remains inconclusive. The paper presents these as operational demonstrations rather than validation of a reasoning-measurement instrument.
Critical strengths
- The separation of execution status, evidence integrity, and task outcome is useful and consistently illustrated by adverse cases. Transport failures, defective controls, repaired campaigns, and inconclusive outcomes remain visible instead of being silently converted into favorable scores. This makes the submission informative about practical evaluation failure modes.
- The manuscript is unusually explicit about the limits of its observations. It discloses auditor-independence limitations, channel and snapshot mismatches, quote-in-refusal effects, and the failed A16 localization gate. These qualifications make its bounded operational claims more credible than unqualified claims of capability or security robustness.
Critical weaknesses
- The evidence does not establish the added scientific or practical value of the integrated protocol over a competent existing audit workflow. M2 uses only six non-divergent cases, and both arms share the same answer parser and equality test; Appendix C states that answer-level disagreement is structurally impossible. Seeded fixtures validate implementations against designed rules, but do not compare independently detected faults, audit effort, or decision quality with an appropriate competing process. Since the controls themselves are not claimed to be new, this leaves the central integrated contribution insufficiently validated.
- The strongest commercial-claim examples do not test equivalent conditions, limiting their evidentiary relevance. A17 observes a different model snapshot from the one named in the source claim. A18 uses simulated browser context, single-message turn markers, and a gateway-added persona; some mechanical successes are quotations within refusals. The qualifications are disclosed, but an opposite direction in this altered environment is not a controlled test of the original claim. Much of the underlying evidence is referenced as repository paths and closure records rather than reproduced in the PDF, so its inspectability depends on the accompanying package.
Questions for authors
- What comparative evidence shows that ECP detects consequential evaluation faults or improves audit decisions beyond an existing workflow with preregistration, preserved logs, and separate transport/outcome accounting? Please identify an existing experiment that can discriminate the approaches, rather than only exercise the frozen classifier.
- Where can reviewers inspect the complete registrations, raw outputs, independent verifier, claim ledger, and exact source of the A18 vendor statement? How is registration-before-execution anchored independently of mutable repository state?
- Which direct-provider, model-matched or native-browser controls are available for A17/A18? Without such controls, how should these examples support practical claim auditing rather than characterize the gateway-conditioned systems alone?