- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CertiLoop audits frozen learned agent artifacts using paired executions, simultaneous confidence intervals, and nested benefit–risk contracts with positive, negative, or unresolved outcomes. Experiments on 30 artifacts from three producers across AppWorld and τ-Retail show that risk constraints refute five of 25 development-selected artifacts, while all eight tested judge configurations recommend at least one reference-negative artifact.
Critical strengths
-
The manuscript is well-organized and readable.
-
The experimental setup is described in a detailed and thorough manner.
Critical weaknesses
-
Limited methodological advance. The framework combines established risk-controlled selection, simultaneous intervals and prediction-powered inference, and Lemma 1 and Proposition 1 follow directly from adding constraints and from joint coverage. There is no matched comparison showing a gain over Prompt Risk Control or a standard paired multi-risk test.
-
Undefined protected-risk labels. All risk conclusions rest on the binary violation label, but Table 4 defines only stratum membership, not the violation rubric. The label-generation procedure is not specified or validated.
-
Appendix B.11 states that the task-ID rosters, stratum counts, Retail simulator ID and allocation manifest are missing. Per-artifact estimates, intervals and failing constraints are also not reported.
-
Certification is never checked against the reference under the same contract. Prediction assistance costs more in the measured profiles, and the human-adjudication savings rely on assumed inputs.
Questions for authors
refer to weaknesses
We thank the reviewer for the careful assessment and for recognizing the organization of the paper and the detail of the experimental setup.
W1. Methodological Contribution
CertiLoop contributes an artifact-reuse audit and decision analysis, rather than a new concentration bound. It fixes the complete artifact–comparator intervention, separates development from independent reference evidence, and compares nested contracts using unchanged intervals. Lemma 1 and Proposition 1 establish the attribution and validity of these decisions (Secs. 3.3–3.5). Table 1 provides the controlled comparison: among the same 25 selected artifacts, benefit-only evaluation refutes none, overall-risk evaluation refutes two, and the guarded contract refutes five. Exactly three refutations therefore require conditional-risk evidence. The contribution is identifying and quantifying these consequential differences in reuse decisions, not claiming tighter bounds or better sample efficiency than PRC or a standard paired multi-risk test.
W2. Protected-Risk Labels
Table 4 defines task populations, whereas the binary outcome records whether an execution violates the registered protection requirement. Appendix B.2 explicitly separates them: an authorized refund or state change need not be a violation, and a temporal mistake can reduce utility without creating a protected violation. Equation (1) measures the treatment–comparator difference in these execution labels, not group membership or task failure. Section 3.2 requires the outcome rubric and execution configuration to match the frozen audit specification before a record enters inference. These provisions define how labels are used consistently; the complete trace-to-label implementation and independent label-validation results are not documented in the submission.
W3. Reproducibility and Artifact-Level Evidence
The reference procedure is specified in Sec. 4.1 and Appendices A.5 and B.3: 8,192 fresh paired draws per artifact, four target intervals, stratum-specific sample counts and variances, and the empirical-Bernstein construction in Equation (18). The analysis package reproduces the reported tables and figures from aggregate inputs. Appendix B.11 distinguishes this from execution-level reproduction and identifies the unavailable task-ID rosters, stratum counts, Retail simulator identifier, and run-level allocation manifest. Reconstructing individual reference labels additionally requires the artifact-specific estimates, interval endpoints, and failing constraints. The released summaries support reproduction of the aggregate analyses, but do not provide those underlying records; we do not equate the two levels of reproducibility.
W4. Backend Validation and Prediction-Assistance Costs
The main findings in Tables 1–2 use fixed-endpoint reference labels, not an assumption that DIRECT or PA certificates agree with them. Equation (20) defines the required backend–reference comparison, but the operational profiles do not report its artifact-level intersections or establish a matched operational contract (Appendix B.6). Their cost result is nevertheless clear: at 2,048 blocks, DIRECT, PA-Single, and PA-Ising report 12, 11, and 12 certificates at normalized costs of 0.9736, 1.0768, and 1.2470, respectively (Table 7). Neither PA backend is cheaper at this endpoint. Appendix B.10 separately evaluates a fixed-input accounting scenario: PA becomes cheaper only when the adjudication-cost multiplier exceeds 0.5620 or 0.5839, respectively, with other inputs held fixed. These are conditional break-even results, not measured human-labor savings or evidence of certification accuracy.