- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CertAgent, a ReAct runtime that verifies reasoning and action proposals, gates them using a shared acceptance threshold, and reflects on rejected proposals. Its certification target is a trajectory loss defined by average verifier exposure for completed executions and unit loss for executions that exhaust the reflection budget. To accommodate nonmonotone loss trajectories, the method uses a nonincreasing majorant and adds a zero-utility abstention configuration. It applies conformal risk control to select configurations at a specified expected-loss target and gives a rank-based guarantee for the loss of a fresh deployment. Policy optimization with LoRA is performed before calibration. Experiments on SWE-Bench Lite compare three model backbones and report task success, certification diagnostics, and runtime costs.
Critical strengths
-
The paper offers a useful trajectory-level formulation covering both reasoning and action control, with explicit accounting for budget exhaustion and nondeployment. Reporting abort rates and majorant gaps makes the certification conditions auditable.
-
Experiments under a shared harness show initial gains across three backbones, with further gains from policy optimization. The configuration and cost analyses help assess the proxy-risk/utility tradeoff and deployment overhead.
Critical weaknesses
-
The methodological advance is not sufficiently clear. The majorant and quantile-control tools already appear in CRC, while CCPO, Conformal Risk Training and Conformal Policy Control study related calibration, policy control or training. Their settings differ, but the paper needs to establish what its dual-stage runtime adds. The methodological contribution currently appears incremental.
-
The experimental coverage is too narrow to support the deployment claims. The open-model study uses only Llama-3.1-8B and Gemma-3-4B, and policy optimization is evaluated exclusively on these small backbones. Including GPT-5.5 is useful, but one closed model and 60 test tasks on a single benchmark provide limited evidence of transfer across model scales and agent settings. The verifier proxy and held-out certificate behavior also lack independent validation, while the fixed-configuration risk interpretation requires clarification.
Questions for authors
-
What independent evidence supports the verifier-proxy assumption? In particular, can you report how verifier scores relate to observable execution errors or independently assessed reasoning/action failures on the evaluated trajectories?
-
For a configuration fixed before calibration, what precisely is guaranteed by the reported CRC level in Table 2(a)? Please specify the source of randomness in the guarantee and distinguish a fixed-target deployment rule, marginal coverage for a fresh trajectory, and an upper confidence bound on population expected loss.
-
What is the additional methodological or empirical contribution over existing CRC and the published work on conformal policy control and training discussed above? Please clarify which aspects are specific to certifying the dual-stage runtime (e.g., the incremental benefit of jointly verifying reasoning and action stages over either stage alone), and how they differ from orchestration-policy coverage, policy interpolation and training through calibration.
-
What motivates the choice of two small open models and a single closed model? On the same benchmark, can you provide evidence from a stronger or additional representative backbone, reporting realized certificate loss or exceedance rates alongside utility and cost? A comparison with fixed or heuristic threshold selection on the same controller would also help establish whether calibrated configuration selection remains useful beyond the current model choices.
6 comments
As the discussion period is approaching its end, we would very much appreciate hearing your thoughts on our rebuttal, so please let us know if you have any remaining questions, and we would be happy to address them promptly.
Thank you for the clarifications and additional experiments. The Qwen3-32B and GAIA evaluations, results over three seeds, and verifier AUROC analysis meaningfully alleviate my concerns about experimental coverage and the proxy’s predictive value.
However, the methodological contribution remains incremental relative to existing conformal control and training methods. The no-gate ablation supports the usefulness of gating, but does not isolate the contribution of conformal configuration selection. The CRC comparison demonstrates appropriate abstention under a strict risk target, while providing limited evidence of useful certified deployment with positive utility.
Overall, the response strengthens the empirical support, but does not sufficiently change my assessment of the contribution’s significance and practical value. I therefore maintain my rating.
To W1 & Q3: We thank the reviewer for this question. Beyond a direct application of conformal risk control, CertAgent certifies the system-level loss of a dual-stage runtime under a single shared gate, where monotonicity and feasibility do not hold by default and are established through a non-increasing majorant and an abstention option (Section 3.3). To examine the empirical gains, we evaluate on SWE-Bench Lite and GAIA with Llama-3.1-8B over three seeds, including a no-gate baseline that runs the same runtime at λ=0.
CertAgent performs best on both benchmarks, while removing the gate brings the runtime back to the level of AutoAgent, showing that the gains come from the certified gate rather than the runtime alone. These results have been added to the appendix of the revised PDF.
To W2 & Q4: We thank the reviewer for this comment. Policy optimization requires open weights and is therefore evaluated on open models, while GPT-5.5 covers the inference-only setting. We further add Qwen3-32B on SWE-Bench Lite and, at α=0.1, compare the CRC selection with a fixed threshold λ=0.5 and the gate-implied rule λ=1−α on the test split.
Both heuristic rules exceed the target on the test split, and the gate-implied rule fails even at its most conservative configuration because aborted executions push the realized risk above 1−λ, a violation neither rule can detect before deployment. CertAgent instead recognizes from the calibration data that no configuration attains α=0.1 and abstains, while a user who relaxes α obtains a certified configuration with positive utility. CertAgent is thus the only rule that respects the target risk level. These results have been added to the appendix of the revised PDF.
To Q1: We appreciate this question. We report the AUROC of 1−q for two failure signals that the verifier does not produce, evaluated on no-gate trajectories (λ=0) where every proposal is scored but none is blocked, so the evaluation is free of selection by the gate.
The verifier scores separate failing proposals from successful ones under both signals, including harness errors recorded by the external evaluation environment independently of the agent and the verifier. Since 1−q is exactly the per-proposal exposure in the system loss of Definition 1, this result directly supports the verifier-proxy assumption and grounds the certified loss in observable runtime failures. These results have been added to the appendix of the revised PDF.
To Q2: We thank the reviewer for this question. For a configuration fixed before calibration, a CRC level α means that, if the configuration is certified, its expected loss on a fresh task is at most α, averaging over both calibration and the fresh execution. This is an average guarantee; it does not mean every individual trajectory has loss at most α, nor is it an upper confidence bound on R(λ).
We sincerely thank the reviewer for the constructive comments. We are confident that our responses have fully resolved all of the concerns raised. In light of this, we would be grateful if the reviewer could kindly consider raising the score.