acceptodds
Under review as a conference paper at ICLR 2027

CertAgent: Conformal Risk Control for Agent Runtime Fragility Certification

Abstract

LLM agents extend language models from text generation to interactive task solving, but their deployment can fail through fragile runtime behaviors arising from reasoning or action errors. Existing mitigation methods improve empirical robustness, but they do not provide formal risk guarantees over the entire Agent Runtime. In this paper, we propose CertAgent, a framework for Agent Runtime Fragility Certification. We first formulate Agent Runtime Fragility as system level risk over the full agent runtime trajectory, while using stage specific failure modes to guide runtime control. CertAgent instantiates a Verifier Subagent to detect unreliable runtime proposals, a Gate Module to control runtime conservativeness through a configuration parameter , and a reflection mechanism to revise rejected proposals. To provide deployment time guarantees, we apply Conformal Risk Control to certify Agent Runtime Fragility, enabling users to either obtain a certified runtime risk bound for a fixed configuration or identify certified runtime configurations under a target risk level. We further introduce policy optimization to reduce system level runtime risk and improve the resulting certification. Experiments on representative agent benchmarks show that CertAgent improves runtime reliability, supports practical risk utility tradeoffs, and yields more effective certification. Our code is available at https://anonymous.4open.science/r/CertAgent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Conformal Risk Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7VLPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CertAgent, a ReAct runtime that verifies reasoning and action proposals, gates them using a shared acceptance threshold, and reflects on rejected proposals. Its certification target is a trajectory loss defined by average verifier exposure for completed executions and unit loss for executions that exhaust the reflection budget. To accommodate nonmonotone loss trajectories, the method uses a nonincreasing majorant and adds a zero-utility abstention configuration. It applies conformal risk control to select configurations at a specified expected-loss target and gives a rank-based guarantee for the loss of a fresh deployment. Policy optimization with LoRA is performed before calibration. Experiments on SWE-Bench Lite compare three model backbones and report task success, certification diagnostics, and runtime costs.

Critical strengths
  1. The paper offers a useful trajectory-level formulation covering both reasoning and action control, with explicit accounting for budget exhaustion and nondeployment. Reporting abort rates and majorant gaps makes the certification conditions auditable.

  2. Experiments under a shared harness show initial gains across three backbones, with further gains from policy optimization. The configuration and cost analyses help assess the proxy-risk/utility tradeoff and deployment overhead.

Critical weaknesses
  1. The methodological advance is not sufficiently clear. The majorant and quantile-control tools already appear in CRC, while CCPO, Conformal Risk Training and Conformal Policy Control study related calibration, policy control or training. Their settings differ, but the paper needs to establish what its dual-stage runtime adds. The methodological contribution currently appears incremental.

  2. The experimental coverage is too narrow to support the deployment claims. The open-model study uses only Llama-3.1-8B and Gemma-3-4B, and policy optimization is evaluated exclusively on these small backbones. Including GPT-5.5 is useful, but one closed model and 60 test tasks on a single benchmark provide limited evidence of transfer across model scales and agent settings. The verifier proxy and held-out certificate behavior also lack independent validation, while the fixed-configuration risk interpretation requires clarification.

Questions for authors
  1. What independent evidence supports the verifier-proxy assumption? In particular, can you report how verifier scores relate to observable execution errors or independently assessed reasoning/action failures on the evaluated trajectories?

  2. For a configuration fixed before calibration, what precisely is guaranteed by the reported CRC level in Table 2(a)? Please specify the source of randomness in the guarantee and distinguish a fixed-target deployment rule, marginal coverage for a fresh trajectory, and an upper confidence bound on population expected loss.

  3. What is the additional methodological or empirical contribution over existing CRC and the published work on conformal policy control and training discussed above? Please clarify which aspects are specific to certifying the dual-stage runtime (e.g., the incremental benefit of jointly verifying reasoning and action stages over either stage alone), and how they differ from orchestration-policy coverage, policy interpolation and training through calibration.

  4. What motivates the choice of two small open models and a single closed model? On the same benchmark, can you provide evidence from a stronger or additional representative backbone, reporting realized certificate loss or exceedance rates alongside utility and cost? A comparison with fixed or heuristic threshold selection on the same controller would also help establish whether calibrated configuration selection remains useful beyond the current model choices.

6 comments

W1 & Q3: The methodological contribution currently appears incremental. Please clarify which aspects are specific to certifying the dual-stage runtime.

To W1 & Q3: We thank the reviewer for this question. Beyond a direct application of conformal risk control, CertAgent certifies the system-level loss of a dual-stage runtime under a single shared gate, where monotonicity and feasibility do not hold by default and are established through a non-increasing majorant and an abstention option (Section 3.3). To examine the empirical gains, we evaluate on SWE-Bench Lite and GAIA with Llama-3.1-8B over three seeds, including a no-gate baseline that runs the same runtime at λ=0\lambda=0.

MethodSWE-Bench LiteGAIA
AutoAgent21.7±1.79.1±1.2
CertAgent (no gate)21.1±2.58.5±1.2
CertAgent23.3±1.712.1±1.8

CertAgent performs best on both benchmarks, while removing the gate brings the runtime back to the level of AutoAgent, showing that the gains come from the certified gate rather than the runtime alone. These results have been added to the appendix of the revised PDF.

W2 & Q4: The experimental coverage is too narrow. Can you provide evidence from a stronger backbone and a comparison with fixed or heuristic threshold selection?

To W2 & Q4: We thank the reviewer for this comment. Policy optimization requires open weights and is therefore evaluated on open models, while GPT-5.5 covers the inference-only setting. We further add Qwen3-32B on SWE-Bench Lite and, at α=0.1\alpha=0.1, compare the CRC selection with a fixed threshold λ=0.5\lambda=0.5 and the gate-implied rule λ=1−α\lambda=1-\alpha on the test split.

Selection ruleSelected λTest riskUtilityTokens (k)
Fixed threshold0.50.270.3538.6
Gate-implied (1−α)0.90.170.2542.1
CertAgent (CRC)abstain0.000.000.0

Both heuristic rules exceed the target on the test split, and the gate-implied rule fails even at its most conservative configuration because aborted executions push the realized risk above 1−λ1-\lambda, a violation neither rule can detect before deployment. CertAgent instead recognizes from the calibration data that no configuration attains α=0.1\alpha=0.1 and abstains, while a user who relaxes α\alpha obtains a certified configuration with positive utility. CertAgent is thus the only rule that respects the target risk level. These results have been added to the appendix of the revised PDF.

Q1: What independent evidence supports the verifier-proxy assumption?

To Q1: We appreciate this question. We report the AUROC of 1−q1-q for two failure signals that the verifier does not produce, evaluated on no-gate trajectories (λ=0\lambda=0) where every proposal is scored but none is blocked, so the evaluation is free of selection by the gate.

Failure signalAUROC of 1−q
Next-step execution error (harness)0.842
Runtime execution error (agent)0.896

The verifier scores separate failing proposals from successful ones under both signals, including harness errors recorded by the external evaluation environment independently of the agent and the verifier. Since 1−q1-q is exactly the per-proposal exposure in the system loss of Definition 1, this result directly supports the verifier-proxy assumption and grounds the certified loss in observable runtime failures. These results have been added to the appendix of the revised PDF.

Q2: For a configuration fixed before calibration, what precisely is guaranteed by the CRC level in Table 2(a)?

To Q2: We thank the reviewer for this question. For a configuration fixed before calibration, a CRC level α\alpha means that, if the configuration is certified, its expected loss on a fresh task is at most α\alpha, averaging over both calibration and the fresh execution. This is an average guarantee; it does not mean every individual trajectory has loss at most α\alpha, nor is it an upper confidence bound on R(λ)R(\lambda).

We sincerely thank the reviewer for the constructive comments. We are confident that our responses have fully resolved all of the concerns raised. In light of this, we would be grateful if the reviewer could kindly consider raising the score.

As the discussion period is approaching its end, we would very much appreciate hearing your thoughts on our rebuttal, so please let us know if you have any remaining questions, and we would be happy to address them promptly.

Thank you for the clarifications and additional experiments. The Qwen3-32B and GAIA evaluations, results over three seeds, and verifier AUROC analysis meaningfully alleviate my concerns about experimental coverage and the proxy’s predictive value.

However, the methodological contribution remains incremental relative to existing conformal control and training methods. The no-gate ablation supports the usefulness of gating, but does not isolate the contribution of conformal configuration selection. The CRC comparison demonstrates appropriate abstention under a strict risk target, while providing limited evidence of useful certified deployment with positive utility.

Overall, the response strengthens the empirical support, but does not sufficiently change my assessment of the contribution’s significance and practical value. I therefore maintain my rating.

Reviewer_RD2sOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents CertAgent which uses a ReAct agent with a verifier that scores each step, a gate that accepts a step only if its score is at least λ\lambda and a reflection loop that retries rejected steps. It then uses conformal risk control to certify a risk built from those verifier scores. The experiments are on 60 SWE-bench Lite test instances with 60 more used for calibration.

Critical strengths

The paper is careful about the conditions conformal risk control needs and it reports diagnostics so readers can check the calibration. The proofs look correct.

Critical weaknesses
  • The experiments in the paper are extremely small. one benchmark with 60 test examples, no seeds or error bars, and the gains over AutoAgent are 1-2 examples. There are no ablations and no no-gate baseline.

  • What is being certified seems almost guaranteed by design. Since the gate only accepts scores of at least λ\lambda, the risk cannot really go above 1 - λ\lambda, and the certified numbers in Table 2a are basically 1 - λ\lambda. It also is not linked to whether the agent actually solves the task.

  • I could not tell if λ\lambda does anything useful. The verifier scores go up a lot across λ\lambda, but the number of solved tasks stays around 11-14 out of 60.

  • The paper does not report the average number of tool calls or reflection rounds, so the cost claims are hard to judge.

  • λ\lambda is hard to understand from the main text. Its simple meaning (the minimum verifier score needed to accept a step) only shows up in the appendix


I would recommend the authors to include illustrative running examples to clear the motivation of this work. Additionally, please include more experiments to justify the research contributions of the work.

To the best of our knowledge, formal certification for controlling system level Agent Runtime fragility remains largely unexplored. 


This is not true. I have come across some papers. Kamath et al., "Enforcing Temporal Constraints for LLM Agents" and Bhardwaj et al. “Formal specification and runtime enforcement for reliable autonomous ai agents”

Questions for authors

What exactly is the verifier, and do its scores actually predict failures?

What are the average tool calls and reflection rounds for each λ\lambda and each baseline? What does the certificate tell us beyond 1 - λ\lambda?

How does it do with no gate with seeds and on a second benchmark?

4 comments

W1 & Q3: The experiments are extremely small. How does it do with no gate, with seeds, and on a second benchmark?

To W1 & Q3: We thank the reviewer for this question. We conduct additional experiments with Llama-3.1-8B over three random seeds, including a no-gate baseline that runs the same runtime at λ=0\lambda=0 and benchmark GAIA.

MethodSWE-Bench LiteGAIA
AutoAgent21.7±1.79.1±1.2
CertAgent (no gate)21.1±2.58.5±1.2
CertAgent23.3±1.712.1±1.8

CertAgent achieves the best performance on both benchmarks across seeds, including GAIA, whose tasks involve longer and more open-ended tool-use trajectories. Removing the gate brings the runtime back to the level of AutoAgent, indicating that the gains come from the certified gate, which intercepts risky reasoning and action steps before they compound along the trajectory. These results have been added to the appendix of the revised PDF, and we thank the reviewer for this suggestion, which strengthens our empirical evaluation.

W2 & Q2.2: What is being certified seems almost guaranteed by design. What does the certificate tell us beyond 1−λ1-\lambda?

To W2 & Q2.2: We appreciate this valuable question. Since aborted executions are charged the maximal loss, the gate only implies R^n(λ)≤1−λ+an(λ)λ\hat R_n(\lambda)\le 1-\lambda+a_n(\lambda)\lambda, where the abort rate ana_n is unknown before calibration (Appendix C). At λ=0.9\lambda=0.9 the empirical risk is 0.120 and thus exceeds 1−λ=0.101-\lambda=0.10. A user relying on 1−λ1-\lambda would deploy λ=0.9\lambda=0.9 for α=0.10\alpha=0.10 and violate the target, whereas CertAgent certifies 0.13 and abstains (Table 2b). Moreover, policy optimization lowers the certificate at every configuration (Figure 2a), showing that the certificate reflects the quality of the runtime rather than being fixed by the gate.

W3: The verifier scores go up a lot across λ\lambda, but the number of solved tasks stays around 11–14 out of 60.

To W3: We thank the reviewer for this observation. This behavior is expected, since λ\lambda controls runtime-attributable risk rather than backbone capability, and raw task failure also includes capability failures that no runtime control can correct (Section 2.3). The decrease in certified risk is not a mere selection effect of the gate, and the table in our response to W4 shows that raising λ\lambda also reduces observable execution errors, which are measured independently of the verifier. Meanwhile, utility does not degrade as the certified risk falls from 0.85 to 0.48 between λ=0.1\lambda=0.1 and λ=0.5\lambda=0.5, so tighter runtime control comes at no cost in task success.

W4 & Q2.1: The paper does not report the average number of tool calls or reflection rounds.

To W4 & Q2.1: We thank the reviewer for this question. Beyond the token usage and inference time reported in Figure 2(b), we report the per-task averages of tool calls, reflection rounds, and execution errors on SWE-Bench Lite with Llama-3.1-8B, comparing AutoAgent with CertAgent under different values of λ\lambda.

MethodTool callsReflection roundsExecution errors
AutoAgent4.800.001.65
CertAgent, λ=0.14.200.601.25
CertAgent, λ=0.53.102.700.70
CertAgent, λ=0.91.705.200.30

As λ\lambda increases, the gate steadily redirects low-confidence actions into reflection before they reach the environment, and execution errors drop accordingly. This confirms that CertAgent effectively suppresses runtime failures in observable execution. The results have been added to the appendix of the revised PDF. We sincerely thank the reviewer for this valuable suggestion, which improves the comprehensiveness of our empirical evaluation.

W5: λ\lambda is hard to understand from the main text.

To W5: We thank the reviewer for pointing this out. In Section 2.1, the runtime configuration λ\lambda is converted by a non-decreasing map τ\tau into the acceptance threshold of the gate shared by the reasoning and action stages. Our implementation uses τ(λ)=λ\tau(\lambda)=\lambda, so λ\lambda is exactly the minimum verifier score that a proposal must reach to be deployed. A proposal below this threshold is blocked and revised through reflection, and the execution aborts if no proposal is accepted within the reflection budget. A larger λ\lambda therefore imposes a stricter acceptance criterion at both stages. We have stated this interpretation explicitly in Section 2.1 of the revised PDF.

W6: Please include illustrative running examples.

To W6: We thank the reviewer for this suggestion. We use issue sympy__sympy-19007 as an example, in which SymPy returns an element from the wrong block when a user indexes into a block matrix. The model first proposed one long action that combined several code searches with a file read. The code was truncated, so the verifier flagged an Argument Error with q=0.37<λ=0.5q=0.37<\lambda=0.5 and rejected the action. Reflection then reduced it to a single search for BlockMatrix, which was accepted with q=0.78q=0.78. From there, the agent widened its search and located the indexing logic. It found that elements were always taken from the first block, and it produced a patch that fixed the issue.

W7: The claim that formal certification remains largely unexplored is not true; see Kamath et al. and Bhardwaj et al.

To W7: We thank the reviewer for this helpful comment. Our original statement described formal certification of system-level Agent Runtime fragility as largely unexplored, and it was not intended to suggest that no prior work exists in this direction. We agree that both works are relevant. Bhardwaj et al. (2026) is already cited in the introduction, and Kamath et al. is also cited in the revised PDF. We have further narrowed the statement to: “Formal certification for controlling system level Agent Runtime fragility still calls for further investigation.”

Q1: What exactly is the verifier, and do its scores actually predict failures?

To Q1: We thank the reviewer for this question. The verifier is a lightweight subagent. It scores each proposal from local runtime evidence against the two failure modes of its stage and returns textual feedback on the dominant one. To test its predictive value, we report the AUROC of 1−q1-q for two independent failure signals on no-gate trajectories (λ=0\lambda=0), where every proposal is scored but none is blocked.

Failure signalAUROC of 1−q
Next-step execution error (harness)0.842
Runtime execution error (agent)0.896

The verifier scores consistently separate failing proposals from successful ones under both signals. The harness errors are recorded by the external evaluation environment, entirely independent of the agent and the verifier, and 1−q1-q remains highly predictive of them. The quantity 1−q1-q is precisely the per-proposal exposure accumulated in the system loss of Definition 1, so this result provides direct empirical support for the verifier-proxy assumption in Section 2.3 and shows that the certified loss is grounded in observable runtime failures. The results have been added to the appendix of the revised PDF. We sincerely thank the reviewer for this constructive suggestion.

We sincerely thank the reviewer for the constructive comments. We are confident that our responses have fully resolved all of the concerns raised. In light of this, we would be grateful if the reviewer could kindly consider raising the score.

Thanks for answering my questions.

I do not think the authors have sufficiently addresed my concerns and questions that are mainly due to lack of readability of the paper. There have been new additions in the Appendix that are useful but the main content of the updated paper has not changed much.

  1. Where was the sympy__sympy-19007 example added? I do not see it in the paper.

The paper does not report the average number of tool calls or reflection rounds.

why is this only added in Appendix. Shouldn't Figure 2 be updated accordingly including cost from reflection rounds?

  1. What about the error bars?

We sincerely thank the reviewer for the follow-up questions and for the continued engagement with our work.

To Q1: We have updated the PDF and included a step-by-step case study of this example in Appendix D.6, together with a new figure illustrating the key steps of the process.

To Q2/Q3: We now run all relevant experiments with 3 random seeds and report the mean ± std. We have also added the corresponding bar plots with error bars as a new Figure 3 in Section 4.4 of the main text, together with a detailed discussion of how gate strictness affects agent behavior.

Table: Per-task runtime diagnostics on SWE-Bench Lite with Llama-3.1-8B (mean ± std over three seeds).

MethodTool callsReflection roundsExecution errors
CertAgent, λ=0.1\lambda=0.14.28 ± 0.080.59 ± 0.021.19 ± 0.14
CertAgent, λ=0.5\lambda=0.53.06 ± 0.102.78 ± 0.240.68 ± 0.03
CertAgent, λ=0.9\lambda=0.91.71 ± 0.045.28 ± 0.080.27 ± 0.03

Overall, a stricter gate converts more external tool calls into internal reflection rounds and leads to fewer execution errors.

Table: CertAgent token breakdown.

ModelReasoning/ActionReflectTotal
Llama-3.1-8B47,294 (93.864%)3,092 (6.136%)50,386 (100%)
Gemma-3-4B104,822 (89.690%)12,050 (10.310%)116,872 (100%)

We further analyze the additional overhead introduced by reflection, which accounts for only about 6% and 10% of the total cost on Llama-3.1-8B and Gemma-3-4B, respectively.

Readability: We agree that discussing these results in the main text improves the clarity of the paper, and we have moved the key findings on tool calls, reflection rounds, and their cost from the Appendix into Section 4.4 accordingly.

We are grateful for these valuable suggestions, which have helped us improve the paper. If the reviewer has any further questions or concerns, we would be happy to address them promptly.

Reviewer_6kh5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CertAgent, a framework for Agent Runtime Fragility Certification. It combines stage-specific Verifier Subagents, a shared Gate Module, and reflection to control ReAct agent execution. CertAgent applies Conformal Risk Control to a non-increasing majorant of a verifier-induced trajectory loss, identifying deployable configurations under a target risk level. Policy optimization further improves the risk–utility tradeoff. Experiments on SWE-Bench Lite evaluate task performance, certification, and runtime efficiency.

Critical strengths
  1. The framework recognizes that stricter gating can increase reflection and budget exhaustion, so trajectory loss need not decrease monotonically. It addresses this through a non-increasing majorant, assigns maximal loss to aborted executions, and reports calibration diagnostics.

  2. CertAgent returns a certified configuration set, allowing selection based on task utility, and explicitly abstains when no deployable configuration meets the target. Separating policy optimization from calibration also preserves the independence required by the certification procedure.

Critical weaknesses

My primary concern is that the framework appears somewhat ad hoc: the stage-specific verifiers, shared gating threshold, reflection mechanism, and trajectory-level loss involve several design choices that are insufficiently justified. The certification guarantees largely follow from general CRC constructions, leaving their connection to the specific runtime design underdeveloped. Moreover, the lack of targeted ablations makes it difficult to establish which components are necessary and whether their combination offers meaningful benefits over simpler alternatives.

Questions for authors

See Weaknesses.

2 comments

W1.1: The framework appears somewhat ad hoc, and its design choices are insufficiently justified.

To W1.1: We thank the reviewer for raising this point. Each choice is required by the certificate. (1) Reasoning and action steps fail in different ways, so each stage has its own verifier, and its feedback lets reflection revise a rejected proposal in a targeted way. (2) A single threshold gives a totally ordered grid, which the monotone majorant and Theorem 2 rely on; two thresholds would give a two-dimensional grid that needs a multiple-testing correction over K2K^2 candidates. (3) Without reflection, every rejection aborts the execution at maximal loss, so a stricter gate would increase risk; with reflection, the deployed score is monotone in λ\lambda (Lemma 1), and at most 2 of 60 executions abort even at λ=0.9\lambda=0.9 (Table 4). (4) The trajectory loss charges aborts the maximal loss, so the risk cannot be lowered simply by aborting more.

W1.2: The guarantees largely follow from general CRC, leaving their connection to the runtime design underdeveloped.

To W1.2: We appreciate this comment. We do not claim a new conformal inequality. Our contribution is to make CRC valid for an agent runtime, where its preconditions fail by default because a stricter gate triggers more reflection and may cause aborts. The runtime design is what establishes these conditions: coupling the randomness across configurations makes loss paths exchangeable, the gate–reflection loop makes the deployed score monotone in λ\lambda (Lemma 1), and the abort-charged loss with the majorant enforces monotonicity at an auditable cost, with 56 of 60 paths already monotone and a gap of at most 0.018 (Table 4).

W1.3: The lack of targeted ablations makes it difficult to establish which components are necessary.

To W1.3: We thank the reviewer for this suggestion. We add a component ablation on SWE-Bench Lite with Llama-3.1-8B at λ=0.5\lambda=0.5, reporting the test success rate and the CRC level on the calibration split.

Verified stageSuccess (%)CRC level
Reasoning only20.00.580
Action only21.70.640
Both (CertAgent)23.30.480

Verifying both stages yields the highest success rate and the lowest CRC level, reducing the certified risk from 0.580 and 0.640 under single-stage verification to 0.480. Reasoning and action failures are distinct yet sequentially coupled, so verifying only one stage leaves the other failure path uncontrolled and allows its errors to propagate through the trajectory. Joint verification closes both paths and thereby tightens the risk bound over the full trajectory. These results indicate that each verification stage contributes to the overall performance and that the dual-stage design is necessary. This supplementary experiment has been added to the appendix of the revised PDF.

We sincerely thank the reviewer for the constructive comments. If any questions remain, we would be glad to address them promptly.

Reviewer_6kh5OpenReview ↗

Response

Thanks for the response. I'll keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response and Summary of Manuscript Revisions

Dear Area Chair and Reviewers,

We sincerely thank Reviewers 6kh5, RD2s, and 7VLP for their thoughtful feedback. We have responded to each reviewer and incorporated the revisions below into the updated manuscript. All manuscript revisions are highlighted in red.

1. Recognized Strengths

  • Trajectory-level formulation: The paper certifies runtime risk over the full trajectory, covering both reasoning and action control with explicit accounting for budget exhaustion and non-deployment.
  • Careful treatment of CRC conditions: The non-increasing majorant and maximal loss for aborted executions handle non-monotone trajectory loss, with correct proofs.
  • Auditable and actionable certification: Reported abort rates and majorant gaps make the calibration auditable, and CertAgent returns a certified configuration set or explicitly abstains when no configuration meets the target.
  • Consistent empirical gains: CertAgent improves over baselines across three backbones, with further gains from policy optimization.

2. Summary of Manuscript Revisions

  • Appendix D.1 (6kh5-W1.1, W1.3): Added a component ablation in which verifying both stages attains the highest success rate of 23.3% and the lowest CRC level of 0.480, compared with 0.580 and 0.640 under single-stage verification.
  • Appendices D.2 and D.5 (RD2s-W1, Q3; 7VLP-W2, Q4): Over three seeds, CertAgent reaches 23.3±1.7% on SWE-Bench Lite and 12.1±1.8% on GAIA, while removing the gate returns performance to the AutoAgent level. On Qwen3-32B, heuristic threshold rules exceed the target risk whereas CRC correctly abstains.
  • Appendices C and D.4 (RD2s-W2, Q1; 7VLP-Q1, Q2): Clarified that aborts make the certificate depend on more than the gate threshold, and showed that verifier exposure predicts independently recorded failures with AUROC of 0.842 and 0.896.
  • Section 4.4; Figure 3 (RD2s-W4, Q2, Q3): Ran all relevant experiments with three seeds and added per-task runtime diagnostics with error bars to the main text. A stricter gate converts tool calls (4.28 → 1.71) into reflection rounds (0.59 → 5.28) and reduces execution errors from 1.19 to 0.27, while reflection accounts for only 6.1% and 10.3% of total tokens on Llama-3.1-8B and Gemma-3-4B.
  • Sections 1 and 2.1; Appendix D.6 (RD2s-W5, W6, W7): Stated the meaning of λ\lambda in the main text, added a step-by-step case study on sympy__sympy-19007, and cited Kamath et al. with a narrowed novelty statement.

We welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.