- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Earned Guardrails (EG), a framework for validating self-generated guardrail skills before admitting them into an evolving agent skill library. It addresses a key limitation of failure-driven agent improvement: agents may generate guardrails from fabricated or incorrectly attributed failures, while conventional admission mechanisms rely on plausibility or later task success rather than evidence that a guardrail actually prevents its claimed failure. EG has three components: (1) typed failure compilation, which converts execution traces into structured failure records with predefined classes, locations, and matching predicates; (2) counterfactual replay admission, which tests each candidate through a necessity arm (does the failure recur without the guardrail?) and a sufficiency arm (does the guardrail eliminate it?), combining repeated necessity evidence with a sufficiency check in an e-value-based statistical test under a prespecified error budget; and (3) a post-admission ledger, which estimates each guardrail's marginal utility via randomized exposure and promotes or retracts guards based on statistical evidence. On AppWorld, Terminal-Bench, and a determinism-filtered SWE-bench-lite subset, EG achieves a pooled false guardrail rate of 0.08 while retaining 0.92 of genuine guardrails, outperforming suppression-only and LLM-judge-based admission mechanisms. It also admits none of the 300 injected fabricated candidates across 30 randomized streams.
Critical strengths
- Important and underexplored problem. The paper identifies a critical reliability issue in self-evolving agents: fabricated failure records, incorrect causal attribution, and accumulated admission errors can degrade self-generated guardrail libraries. This problem is particularly relevant to autonomous agents that continuously update their behavior without human supervision.
- Intuitive and well-motivated core idea. Treating self-generated guardrails as falsifiable causal claims is a compelling approach. The necessity–sufficiency replay design directly tests whether a claimed failure recurs without the guardrail and is prevented by its presence, rather than relying on plausibility or subsequent task success.
- Principled statistical guarantees. The e-value-based admission mechanism provides explicit error control over a prespecified proposal horizon, addressing the accumulation of false admissions. The paper also clearly states the assumptions underlying its theoretical guarantees.
- Adaptive guardrail lifecycle management. The post-admission ledger extends validation beyond initial admission by estimating marginal utility through randomized exposure. Promoting beneficial guardrails and retracting harmful ones provides a principled mechanism for maintaining an evolving skill library.
- Comprehensive experimental evaluation. Experiments across three agent benchmarks demonstrate improvements over existing admission strategies. The ablation studies, horizon analysis, matcher evaluation, and fabricated-record injection tests provide useful evidence supporting the effectiveness of the proposed components.
Critical weaknesses
- The theoretical guarantee relies on a strong and difficult-to-verify assumption. The e-value-based error control depends critically on Assumption A1, which bounds the conditional reproduction probability of phantom failures by . However, the paper provides limited justification for the chosen value () or how such a bound can be established in realistic agent environments. Violations of this assumption could invalidate the claimed guarantee.
- The formal definition of phantom guardrails does not fully capture causal correctness. The paper defines phantom guardrails primarily through failure reproducibility rather than whether the proposed intervention addresses the underlying cause. Consequently, ineffective or overly restrictive guardrails associated with reproducible failures may fall outside the formal definition. The theoretical FGR guarantee therefore does not necessarily establish the causal effectiveness of admitted guardrails.
- Limited evidence of downstream agent performance improvements. The evaluation primarily focuses on false guardrail rates and genuine-guard retention, rather than end-to-end task performance. Although the post-admission ledger improves retained utility, the paper provides limited evidence that the resulting guardrail library improves task success rates, prevents recurring failures, or reduces unnecessary restrictions compared with existing skill-learning approaches.
- Strong dependence on deterministic replay and accurate failure matching. EG requires faithful episode reconstruction, controlled guardrail insertion, and predefined failure matchers, which may be difficult to achieve in dynamic or nondeterministic environments. Approximately 15% of candidates are excluded due to replay ineligibility, while the matcher achieves only 0.84 macro-F1. The impact of replay failures and matching errors on the framework's reliability remains insufficiently explored.
- Experimental limitations and insufficient statistical reporting. The evaluation is restricted to a proposal horizon of 200 candidates, two backbone families, and a determinism-filtered SWE-bench-lite subset, with some ablations relying on a single seed. Additionally, the phantom-injection experiment reports rate-implied rather than directly observed admission counts for baselines. Larger-scale evaluations, multiple seeds, and uncertainty estimates would strengthen the conclusions.
- Insufficient analysis of computational overhead. EG requires repeated counterfactual replays for candidate admission and additional randomized evaluations for post-admission utility estimation. These procedures may incur substantial computational costs, particularly in long-horizon or tool-intensive agent environments. The paper lacks a detailed comparison of runtime, inference costs, and replay overhead against simpler admission mechanisms, making its practical cost-effectiveness difficult to assess.
Questions for authors
- How is q_0=0.15 justified, and how sensitive is the claimed error-control guarantee to violations of Assumption A1?
- How does EG distinguish genuinely effective guardrails from ineffective or overly restrictive interventions associated with reproducible failures, given that the formal FGR guarantee primarily concerns failure reproducibility?
- Do the admitted guardrails improve end-to-end agent task success rates and reduce recurring failures compared with existing skill-learning approaches?
- How does EG perform under nondeterministic execution, incomplete replay, or failure-matcher errors, and how do these factors affect its statistical guarantees?
- What are the runtime and computational costs of counterfactual replay and post-admission monitoring compared with simpler admission strategies?
5 comments
(continued)
Summarized W4 & Q4 (2): Do nondeterministic execution, incomplete replay, or failure-matcher errors break guarantees?
A6: We respectfully clarify that nondeterministic execution and incomplete replay do not break the statistical guarantees, which hold under Assumption A1 for the candidates that are tested. Both are removed by quarantine, which costs coverage but not error control. Failure-matcher errors are an empirical matter rather than part of the guarantee: they are kept small by deciding each replay round by the typed failure class at the recorded location, within a window of two steps, which clears the 0.80 F1 floor on every failure class in the 150-trace audit. When this matcher is replaced by a weaker LLM-judged location with an F1 of 0.68, FGR rises from 0.08 to 0.14, so on the tested benchmarks the current matcher is accurate enough to keep FGR within the budget.
Following your suggestions, we have updated the matcher agreement in Section 5.3 (Lines 443–447) accordingly.
Summarized W5 (1): Multiple seeds for the ablations.
A7: Following your suggestion, we additionally report the mean and standard deviation of every removal of Table 2 (Lines 368–377) over three seeds. Table R3 shows that every ablation conclusion holds across seeds, as the standard deviation of FGR stays at or below 0.011 and the ranking of all components by mean FGR is unchanged (with arm N and the composed e-value highest, both at 0.25). The additive ladder of Table 3 keeps a single seed. On scale, the evaluation covers a 200-candidate horizon on three benchmarks with two backbone families, eighteen confirmatory runs over three seeds, and the removals of Table 2 over the same three seeds; Appendix A.13 (Lines 1984–1994) sets out longer horizons and further backbones as the next step.
Table R3: Three-seed results of Table 2 (Lines 368–377). ‡: the value is the admitted library's retained utility. ¶: read on the unscreened stream.
| Removed | FGR ↓ | Retention ↑ |
|---|---|---|
| composed e-value¶ | 0.25 ± 0.003 | 0.99 ± 0.007 |
| quarantine | 0.18 ± 0.005 | 0.94 ± 0.004 |
| typed matcher | 0.14 ± 0.011 | 0.90 ± 0.004 |
| arm N¶ | 0.25 ± 0.004 | 0.91 ± 0.007 |
| arm S‡ | 0.08 ± 0.007 | 0.83 ± 0.021 |
| randomized exposure | 0.08 ± 0.007 | 0.91 ± 0.007 |
Following your suggestions, we have revised the paper and replaced Table 2 in Section 5.3 with these three-seed results (Lines 368–394), and updated the related text in Section 5.1 (Lines 341–342), Section 5.2 (Lines 354–355), Appendix A.9 (Lines 1657–1675 and 1758–1815) and Appendix A.10 (Lines 1863–1868 and 1900–1906).
Summarized W5 (2): Add directly observed admission counts for baselines.
A8: Following your suggestion, we additionally report the stream-level admission counts of the baselines in the phantom-injection experiment, the numbers of injected phantoms each baseline admits over the 30 streams (column Stream-level), alongside the original rate-implied counts. Table R4 shows that the stream-level counts match the rate-implied counts within three phantoms.
Table R4: Stream-level admission counts in Table 6 (Lines 432–441).
| Method | Rate-implied | Stream-level |
|---|---|---|
| Suppression-only | 209.1 | 208 |
| LLM-judge | 123.9 | 126 |
| Naive threshold | 288.3 | 291 |
| EG (Ours) | 0 | 0 |
Following your suggestions, we have revised the paper and added these stream-level counts to Table 6 in Section 5.3 (Lines 432–441), and updated the stress-test text in Section 5.3 (Line 424) and Appendix A.9 (Lines 1686–1697).
Summarized W6 & Q5: Runtime and computational cost versus simpler admission strategies.
A9: Following your suggestion, we additionally report the runtime and computational cost of EG against the simpler admission strategy Suppression-only, in terms of replay executions, tokens, and wall-clock time per candidate, on 50 randomly sampled candidates from AppWorld. Table R5 shows that counterfactual replay costs about 1.4 minutes per candidate as a one-time cost at admission, while post-admission monitoring adds no extra replay because it reuses the episodes the agent runs anyway. Table R5: Runtime and computational cost per candidate.
| Method | Replay executions | Tokens | Wall-clock |
|---|---|---|---|
| Suppression-only | 1 | 0.041M | 0.8 min |
| EG (Ours) | 21 | 0.72M | 1.4 min |
Following your suggestions, we have added these results to Appendix A.8 (Lines 1633–1634 and 1646–1651), and updated the Limitations in Appendix A.0 (Lines 903–907).
Best Wishes,
The Authors of Paper 52845
Thank you for the detailed responses and additional experiments, which have addressed many of my concerns. My remaining concern is that the choice of (q_0 = 0.15) appears to be empirically motivated. It remains unclear whether this bound generalizes across different models, tasks, and agent environments. If the actual reproduction probability substantially exceeds this bound, the theoretical guarantees may no longer hold.
Response to Reviewer Mhwh
Dear Reviewer Mhwh:
Thank you for recognizing the importance of the problem, the well-motivated core idea, the principled guarantees, and the comprehensive evaluation of our work. We have responded point by point to your concerns regarding the theoretical assumption, causal effectiveness, end-to-end performance, robustness, statistical reporting, and computational cost. In the marked copy appended to the revised PDF (pages 38–75), the revisions made in response to your comments are tagged with @Reviewer Mhwh. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1 & Q1 (1): Is the chosen value q0=0.15 justified?
A1: We respectfully clarify that the chosen value q0=0.15 is justified by the replay behavior of the two guardrail populations, as, apart from the locus-displaced phantoms, phantom guardrails reproduce their named failure mostly below 0.15 whereas genuine guardrails reproduce it at 0.6 to 0.85, so q0=0.15 sits between the two. Moreover, any phantom bound q0 between 0.13 and 0.16 yields exactly the same admission decisions, so our results do not hinge on the precise choice.
Summarized W1 & Q1 (2): How sensitive is the error-control guarantee to violating Assumption A1?
A2: We respectfully clarify that the error-control guarantee is only mildly sensitive to violating Assumption A1, since it loosens gradually rather than breaking. Table R1 shows that when the true reproduction probability q of a phantom exceeds the phantom bound q0, its admission probability per test rises smoothly and stays within the nominal level of 5×10−4 until q approaches 0.20, absorbing a misspecification of about one third of q0. In realistic agent environments, the bound is further protected by restarting every replay from an exact versioned state and quarantining sources that cannot be faithfully replayed.
Table R1: Admission probability of a phantom under violated Assumption A1. The binomial necessity tail of Appendix A.8 (Lines 1595–1601) for a per-round reproduction probability of at most q conditional on the history, an upper bound on the admission probability.
Following your suggestions, we have revised Appendix A.8 (Lines 1566–1576 and 1584–1601) to make this clearer.
Summarized W2 & Q2: How does EG distinguish ineffective or overly restrictive guardrails?
A3: We respectfully clarify that EG distinguishes ineffective or overly restrictive guardrails through two components beyond the FGR guarantee. First, an ineffective guardrail fails the sufficiency check at admission when replay shows that its named failure still occurs with the guardrail in place. Second, a guardrail that hurts the task, including an overly restrictive one, is retracted after admission once the ledger certifies that it lowers task utility, and in the reported results this retracts 11 of 161 admitted guardrails and raises task utility from 0.86 to 0.92.
Summarized W3 & Q3: End-to-end task success and recurring failures versus existing approaches.
A4: Following your suggestion, we additionally compare EG with existing skill-learning approaches, including the ExpeL class, the Reflexion class, LLM-judge, and No-guard under the protocol of Appendix A.6 (Lines 1437–1497), in terms of task utility relative to the oracle library, which is the retained utility of Table 3 (Lines 405–412) and not the raw task success rate, and recurring failures. Table R2 shows that EG achieves the highest task utility while reducing recurring failures on par with the strongest approach.
Table R2: Comparison with existing skill-learning approaches.
Following your suggestions, we have added these results to Appendix A.9 (Lines 1817–1843).
Summarized W4 & Q4 (1): How does EG perform in nondeterministic environments?
A5: We respectfully clarify that on the replay-eligible subset of the tested benchmarks, EG remains reliable in the presence of nondeterminism, because it quarantines any candidate that cannot be faithfully replayed instead of testing it on weaker evidence. Across the three benchmarks, this excludes 12% to 18% of candidates while FGR stays at 0.07 to 0.09 and genuine-guard retention stays at 0.91 to 0.93.