acceptodds
Under review as a conference paper at ICLR 2027

CertiLoop: Auditing Learned Agent Artifacts under Benefit–Risk Contracts

Abstract

An average benchmark gain does not establish that a learned agent artifact should be reused under explicit risk constraints. We introduce CertiLoop, an audit protocol that compares each complete frozen artifact against its comparator, fixes risk strata from initial task information before execution, nests benefit–risk contracts (benefit-only, whole-scope, and stratum-guarded) on shared intervals so that each refutation is attributable to a specific requirement, and reports every decision as positive, negative, or unresolved. We audit 30 artifacts from ACE, GEPA, and ReasoningBank on fixed AppWorld and -Retail rosters, using 8,192 independent paired executions per artifact as reference evidence. Development selection agrees with every resolved benefit label, yet the reference refutes five of the 25 selected artifacts (20%) on risk grounds, three of them only once the stratum conditions are added. All eight judge configurations we evaluate recommend at least one reference-negative artifact, including six that retain every reference-positive artifact, and task-level prediction accuracy does not order the reliability of the resulting recommendation sets.

Primary area:
learning theory
Keywords:
LLM agents, Skill certification, Partial identification:

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FGBpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CertiLoop audits frozen learned agent artifacts using paired executions, simultaneous confidence intervals, and nested benefit–risk contracts with positive, negative, or unresolved outcomes. Experiments on 30 artifacts from three producers across AppWorld and τ-Retail show that risk constraints refute five of 25 development-selected artifacts, while all eight tested judge configurations recommend at least one reference-negative artifact.

Critical strengths
  1. The manuscript is well-organized and readable.

  2. The experimental setup is described in a detailed and thorough manner.

Critical weaknesses
  1. Limited methodological advance. The framework combines established risk-controlled selection, simultaneous intervals and prediction-powered inference, and Lemma 1 and Proposition 1 follow directly from adding constraints and from joint coverage. There is no matched comparison showing a gain over Prompt Risk Control or a standard paired multi-risk test.

  2. Undefined protected-risk labels. All risk conclusions rest on the binary violation label, but Table 4 defines only stratum membership, not the violation rubric. The label-generation procedure is not specified or validated.

  3. Appendix B.11 states that the task-ID rosters, stratum counts, Retail simulator ID and allocation manifest are missing. Per-artifact estimates, intervals and failing constraints are also not reported.

  4. Certification is never checked against the reference under the same contract. Prediction assistance costs more in the measured profiles, and the human-adjudication savings rely on assumed inputs.

Questions for authors

refer to weaknesses

1 comment

We thank the reviewer for the careful assessment and for recognizing the organization of the paper and the detail of the experimental setup.

W1. Methodological Contribution

CertiLoop contributes an artifact-reuse audit and decision analysis, rather than a new concentration bound. It fixes the complete artifact–comparator intervention, separates development from independent reference evidence, and compares nested contracts using unchanged intervals. Lemma 1 and Proposition 1 establish the attribution and validity of these decisions (Secs. 3.3–3.5). Table 1 provides the controlled comparison: among the same 25 selected artifacts, benefit-only evaluation refutes none, overall-risk evaluation refutes two, and the guarded contract refutes five. Exactly three refutations therefore require conditional-risk evidence. The contribution is identifying and quantifying these consequential differences in reuse decisions, not claiming tighter bounds or better sample efficiency than PRC or a standard paired multi-risk test.

W2. Protected-Risk Labels

Table 4 defines task populations, whereas the binary outcome records whether an execution violates the registered protection requirement. Appendix B.2 explicitly separates them: an authorized refund or state change need not be a violation, and a temporal mistake can reduce utility without creating a protected violation. Equation (1) measures the treatment–comparator difference in these execution labels, not group membership or task failure. Section 3.2 requires the outcome rubric and execution configuration to match the frozen audit specification before a record enters inference. These provisions define how labels are used consistently; the complete trace-to-label implementation and independent label-validation results are not documented in the submission.

W3. Reproducibility and Artifact-Level Evidence

The reference procedure is specified in Sec. 4.1 and Appendices A.5 and B.3: 8,192 fresh paired draws per artifact, four target intervals, stratum-specific sample counts and variances, and the empirical-Bernstein construction in Equation (18). The analysis package reproduces the reported tables and figures from aggregate inputs. Appendix B.11 distinguishes this from execution-level reproduction and identifies the unavailable task-ID rosters, stratum counts, Retail simulator identifier, and run-level allocation manifest. Reconstructing individual reference labels additionally requires the artifact-specific estimates, interval endpoints, and failing constraints. The released summaries support reproduction of the aggregate analyses, but do not provide those underlying records; we do not equate the two levels of reproducibility.

W4. Backend Validation and Prediction-Assistance Costs

The main findings in Tables 1–2 use fixed-endpoint reference labels, not an assumption that DIRECT or PA certificates agree with them. Equation (20) defines the required backend–reference comparison, but the operational profiles do not report its artifact-level intersections or establish a matched operational contract (Appendix B.6). Their cost result is nevertheless clear: at 2,048 blocks, DIRECT, PA-Single, and PA-Ising report 12, 11, and 12 certificates at normalized costs of 0.9736, 1.0768, and 1.2470, respectively (Table 7). Neither PA backend is cheaper at this endpoint. Appendix B.10 separately evaluates a fixed-input accounting scenario: PA becomes cheaper only when the adjudication-cost multiplier exceeds 0.5620 or 0.5839, respectively, with other inputs held fixed. These are conditional break-even results, not measured human-labor savings or evidence of certification accuracy.

Reviewer_TYyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CertiLoop formulates the evaluation of learned agent artifacts as a joint benefit–risk audit. An artifact must demonstrate sufficient average benefit while satisfying constraints on additional risk within predefined strata. The framework uses paired executions and simultaneous confidence bounds to distinguish certified, refuted, and unresolved artifacts. The main evaluation examines 30 frozen artifacts produced by three learning methods across two environments, using 8,192 paired draws per artifact. The reference audit classifies 13 artifacts as certified, nine as refuted, and eight as unresolved. Comparisons with development selection and benefit-only evaluation show that favorable utility assessments can coexist with violations of the specified conditional-risk contract. The central contribution is the audit formulation and its fixed-endpoint empirical application; sequential auditing and cost projections remain supplementary.

Critical strengths

1.The joint contract addresses an important limitation of utility-centered artifact selection. Requiring both benefit and bounded additional risk makes explicit why improved average performance is insufficient for deployment. The empirical comparison is consequential: among 25 development-selected artifacts, five are refuted and seven remain unresolved under the reference contract.

2.The audit preserves uncertainty rather than forcing every artifact into a binary decision. The three-way outcome and simultaneous bounds provide a principled way to distinguish evidence of acceptability, evidence of violation, and insufficient information. The paper also appropriately conditions its empirical conclusions on the fixed task rosters and execution setup.

Critical weaknesses

1.The central empirical classifications are not sufficiently auditable from the reported materials. The appendices acknowledge missing task rosters, stratum counts, simulator details, and a complete run-allocation manifest. The paper also does not provide enough artifact-level information about the observed benefits, risk differences, uncertainty bounds, and violated strata to reconstruct the substantive reference decisions. Aggregate classification counts and scripts that reproduce those counts do not resolve this gap. Because the empirical contribution depends on these reference labels, the missing detail materially weakens confidence in the findings.

2.The practical significance of the observed contract failures remains undercharacterized. The headline results use one executor, two task rosters, and fixed benefit and risk thresholds. Budget sensitivity is examined, but the paper provides limited visibility into the margins by which selected artifacts violate the risk constraints or how the conclusion changes under nearby contract thresholds. Without this information or concrete failure examples, it is difficult to distinguish substantial operational risk from sensitivity to the chosen contract boundary.

Questions for authors

1.Clarification: Can you provide, for each artifact and risk stratum, the sample count, estimated paired difference, confidence interval, and exact outcome predicate used to assign the reference label?

2.Clarification: How robust is the finding that five development-selected artifacts are refuted to nearby risk thresholds? Which strata drive those decisions, and what behaviors constitute the corresponding violations?

1 comment

We thank the reviewer for recognizing the importance of the joint benefit–risk contract and the distinction between refuted and unresolved artifacts.

W1 / Q1. Auditability of the Reference Classifications

Each reference label is determined from four artifact-specific intervals, not inferred from aggregate counts. The reference uses 8,192 fresh paired executions per artifact; utility and overall risk use the full stream, while each stratum uses its own membership-filtered observations (Sec. 4.1; Appendix B.3). Equation (18) constructs each empirical-Bernstein interval from that stream’s sample count, mean, and variance, with error allocation 0.05/318. Equation (4) then supports reuse only when the utility lower bound reaches 0.02 and every risk upper bound is at most 0.05; it refutes reuse when the utility upper bound falls below 0.02 or any risk lower bound exceeds 0.05. This specifies the calculation behind the 13/9/8 reference classification in Table 1. The requested artifact-by-stratum records are the appropriate evidence for independently checking those calculations. The released aggregate summaries do not supply those numerical records, the complete execution-level scoring specification, or the missing execution metadata identified in Appendix B.11; reproducing the reported counts is therefore distinct from reconstructing individual reference decisions.

W2 / Q2. Practical Significance and Threshold Sensitivity

The five selected refutations are not cases that merely failed to obtain certification: each requires a risk lower bound above 0.05, while seven selected artifacts with insufficient evidence remain unresolved. Table 1 identifies two refutations from overall risk and three additional refutations requiring G1 or G2, with the same intervals throughout. Thus, the finding concerns interval-supported increases exceeding five percentage points relative to the comparator, not point estimates crossing a boundary. Holding the evidence and selected set fixed, all five remain refuted under any uniformly tighter risk tolerance, including 0.04; this follows directly from Equation (4). Persistence at a looser tolerance such as 0.06 depends on the individual lower bounds and cannot be determined from aggregate counts. Table 4 defines the relevant populations, including temporal and persistent-state tasks in AppWorld and order-state and monetary-entitlement tasks in Retail. The unauthorized-refund example in the Introduction illustrates a protected violation, rather than documenting a trajectory from these five artifacts. The reported results establish the contribution of conditional-risk requirements on the fixed rosters; they do not quantify each failure’s excess-risk margin or identify its particular execution behavior.

Reviewer_QR7XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper evaluates whether an already learned prompt configuration, playbook, or memory with its retrieval rule is suitable for reuse under specified performance and risk requirements. It proposes an audit protocol and decision analysis based on expected task-utility gains and violation-rate differences against a baseline, assessing risk both overall and within two predefined task groups in each environment. Jointly valid confidence intervals and preset thresholds determine whether the reuse contract is supported, refuted, or unresolved. The authors evaluate 30 frozen artifacts from ACE, GEPA, and ReasoningBank on AppWorld and text-based τ-Retail. They assess development-based selection and eight judge-based recommendation rules against an independent reference audit, and report separate evidence-cost analyses.

Critical strengths

The decision analysis goes beyond directly mapping confidence intervals and preset thresholds to supported, refuted, or unresolved decisions. It examines which added requirements change the decisions and whether judges recommend contract-failing artifacts or exclude potentially qualifying ones. Adding the task-group risk requirements refutes three additional development-selected artifacts beyond the whole-scope risk condition, showing why the added requirements matter to the reuse decision.

Critical weaknesses
  • The presentation needs improvement, especially the logic and flow. The paper defines the artifact and comparator, but it is difficult to follow how these connect to the measured outcomes, statistical machinery, and final reuse decision. A complete worked example connecting these parts would help explain the workflow and the role of each component.
  • The paper evaluates violation-rate increases against a baseline, but needs a more concrete explanation of how the protected-violation outcome is measured. It specifies task-group membership, but does not clearly specify the scoring rules and procedure for assigning execution-level violation labels in each environment.
  • The distinction from existing risk-control formulations needs to be clearer. Prompt Risk Control (Zollo et al., 2024) already supports simultaneous risk bounds, utility-based selection, multiple thresholds, and group-disparity measures. CertiLoop studies paired effects of complete artifacts and reports supported, refuted, or unresolved decisions, but the paper does not clearly establish the incremental contribution beyond adapting existing risk-control tools to this setting.
  • The paper gives 0.02 and 0.05 as practical effect thresholds, but does not explain their specific selection or report a threshold-sensitivity study.
  • The experiments report how many artifacts DIRECT and the prediction-assisted auditing methods certify, and their costs, but do not compare these decisions for individual artifacts with the independent reference audit. This makes their practical auditing performance difficult to assess.
Questions for authors
  • How were the utility and risk-difference thresholds chosen, and do the main conclusions hold under other reasonable choices?
  • Which events count as protected violations in each environment, and how are they scored from execution records? What procedure and validation were used to assign these execution-level labels?
  • How does the formulation differ from applying Prompt Risk Control's multiple-risk bounds to the same paired artifact outcomes and predefined task groups? What additional contribution comes from the audit protocol and decision analysis?
2 comments

We thank the reviewer for the careful reading and for recognizing the value of identifying which risk requirements change reuse decisions.

W1. Connecting the Artifact, Measurements, and Reuse Decision

For ReasoningBank, we compare the frozen memory store and retrieval rule against the same agent without learned memory, starting both arms from isolated copies of the same initial state (Sec. 3.1). The independent reference collects 8,192 paired executions per artifact to construct intervals for utility gain, overall risk difference, and two group risk differences. Equation (4) supports reuse when every requirement is established, refutes it when any requirement is contradicted, and otherwise returns unresolved. Table 1 (Sec. 4.2) demonstrates this progression for the same 25 selected artifacts: supported/refuted/unresolved counts are 24/0/1 under benefit-only, 20/2/3 with overall risk, and 13/5/7 with group risk. The intervals remain unchanged, so all five refutations require risk evidence, and exactly three require the group constraints.

W2 / Q2. Protected-Violation Outcomes and Scoring

Protected violations are distinct from task failures and group membership. The Retail example concerns unauthorized refund-amount changes; an authorized refund need not violate a protected requirement. Similarly, an AppWorld temporal mistake can reduce utility without causing a protected violation (Sec. 1; Appendix B.2). Equation (1) subtracts the comparator’s binary violation label from the treatment’s: a treatment-only violation contributes +1, a comparator-only violation contributes −1, and matching labels contribute 0. These differences are averaged overall and within groups fixed before execution. Section 3.2 requires the outcome rubric and execution configuration to match the registered audit before a record updates an interval. These checks enforce scoring consistency; independent label-validation results are not reported.

W3 / Q3. Contribution Beyond Prompt Risk Control

CertiLoop contributes an artifact-reuse audit and decision analysis, rather than a new multiple-risk bound. It fixes the complete artifact–comparator intervention, separates development from independent reference evidence, and compares nested requirements using unchanged intervals. This allows requirement-specific attribution while preserving unresolved cases in recommendation and missed-opportunity analyses (Secs. 3.1–3.5). These protocol choices complement existing risk-control tools. Their empirical value is concrete: Table 1 identifies three refutations requiring group constraints, while Figure 3 shows that Majority improves utility-difference MAE from 0.135 to 0.117 relative to Single but still recommends the same number of reference-negative artifacts: six. The audit therefore measures a decision-level failure that improved average prediction error does not resolve.

W4 / Q1. Threshold Selection and Sensitivity

The thresholds define a common contract fixed on the development side before reference evaluation: a minimum normalized utility gain of 0.02 and a maximum violation-rate increase of five percentage points, overall and within each group (Sec. 3.3; Appendix A.1). The risk threshold is relative to the comparator, not an absolute 5% violation-rate limit. These are study requirements rather than application-calibrated safety limits. With intervals and the selected set fixed, Equation (4) establishes that all five selected risk refutations persist under tighter risk tolerances. All 13 supported artifacts likewise remain supported when the utility floor is at most 0.02 and every risk tolerance is at least 0.05. These are exact directional conclusions; unchanged counts or judge rankings across alternative thresholds would require a threshold sweep. Table 9 separately evaluates reference-budget sensitivity.

W5. DIRECT and Prediction-Assisted Decisions Against the Reference

Table 7 (Appendix B.6) measures count–cost behavior: at 2,048 processed blocks, DIRECT, PA-Single, and PA-Ising report 12, 11, and 12 certificates at normalized costs of 0.9736, 1.0768, and 1.2470. Certificate totals are not correct-decision counts. Assessing backend agreement requires the artifact-matched three-state comparison defined in Equation (20), which these profiles do not report. The main recommendation results use matched reference labels directly: Table 2 shows that Dev-Mean retains 13 positive, five negative, and seven unresolved artifacts, whereas Agent-as-a-Judge retains 13, one, and two. Thus, it preserves every reference-positive artifact while reducing included reference negatives from five to one. These findings establish recommendation quality independently of backend accuracy; the two evaluations are not interchangeable.

Thanks to the authors for the response. The clarifications address some of my comments and questions, but concerns remain about the empirical evidence and the significance of the contribution. I would maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.