acceptodds
Under review as a conference paper at ICLR 2027

Certified Process Verification: Trajectory-Level Risk Control for LLM Agents

Abstract

Process verifiers make local decisions, but deployment failures can be compo- sitional: a trajectory is contaminated if any failure-relevant error is certified as safe. We study this mismatch between the decision unit (step) and the risk unit (trajectory). Certified Process Verification (CPV) preserves fine-grained step-level certify/flag decisions while calibrating the trajectory-level loss jointly induced by those decisions. For nonmonotone selective losses CPV uses Learn-then-Test (LTT), while monotone trajectory contamination can be calibrated with either LTT or conformal risk control (CRC), depending on the desired guaran- tee semantics. On AgentProcessBench (APB), simple trajectory-aware alterna- tives control contamination but are substantially more conservative: union-bound and independence-based length corrections certify 0.0% and 1.2% of steps, max- score trajectory aggregation certifies 6.3%, and CROP certifies 9.1%, whereas CPV LTT-HB certifies 14.6% at trajectory risk 0.051 ± 0.013 for a target of 0.1. An oracle-score decomposition raises CPV certification to 38.6%, show- ing that both finite-sample calibration and scorer quality limit efficiency. Across 120 matched independent synthetic calibration draws under the same trajectory-contamination loss, LTT-HB incurs 0/120 high-probability violations with mean risk 0.0360, while CRC attains mean risk 0.0862 under its expected-risk guaran- tee, consistent with the distinct semantics of the two procedures. Results across APB, Who&When, and DeltaBench characterize trajectory-safe process verifica- tion as a validity–utility problem in which fine-grained decisions and trajectory level risk must be jointly calibrated under a common deployment objective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process verification, LLM agents, risk control, conformal prediction, trajectory-level calibration, step certification, Learn-then-Test

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZsNDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles an interesting gap: process verifiers make decisions one step at a time, but the actual deployment risk depends on the whole trajectory. Basically, even if you check all the steps individually, if any “failure-relevant” step is certified, the entire trajectory can be contaminated. The proposed method, CPV, acts as a post-hoc calibration layer over an existing step-level scorer, it doesn’t retrain or change the original model. For calibration, it measures loss per trajectory, and then uses Learn-then-Test (LTT) or conformal risk control (CRC) to keep the risk in check, either with high-probability (LTT) or in expectation (CRC).

Critical strengths

The strengths are pretty clear. First, the paper lays out the mismatch between step-level and trajectory-level evaluation well, especially with Equation 7 and the step-level conformal risk result (risk 0.169). This makes it obvious that controlling risk at the step level doesn’t guarantee control at the trajectory level. Second, the authors honestly evaluate the sources of conservatism in their approach, separating scorer issues from calibration with a well-constructed oracle run and calibration-size ablation. They openly report CRC’s overstepping alpha in 15 out of 120 draws, which is a good sign of honest reporting.

Critical weaknesses

But there are some sticking points. Most importantly, the methodology isn’t that novel, and the efficiency claim doesn’t really land. Once you switch to a per-trajectory loss, LTT and CRC don’t need anything new ; and the supposed lack of independence between steps isn’t unique to this problem. The loss in question is monotonic over a threshold, which is exactly what textbook LTT handles for fixed sequences. So, you don’t need the Bonferroni corrections they use; the simpler LTT method would be much less conservative. This comes out in their results: the LTT-HB test gives a risk of 0.051 (target 0.1), and the synthetic trials are even lower, with zero observed violations. Honestly, for delta = 0.1, you’d expect a calibrated method to show at least some violations, so the synthetic results aren’t all that convincing.

The baselines are also pretty weak. Using union-bound and independence corrections, or comparing step-split conformal against trajectory-level methods, just doesn’t make sense ,it’s not a fair comparison. Other methods like Max-score aggregation and CROP actually certify at the trajectory or prefix level, so comparing their step-level certification numbers with CPV’s isn’t apples to apples.

Another big issue: the utility metric doesn’t really match the goal. Certifying steps only helps if you can skip reviewing the whole trajectory, but they report fraction of certified steps, not certified trajectories. They give fully certified rates for Who&When (0–5.4%), but not for APB. For APB, every -1 step is seen as failure-relevant, so a single failure means the whole trajectory fails. Certifying 14.6% of steps may mean that hardly any trajectory is fully certifiable. Table 2 is also confusing: with perfect (“oracle”) scores, the loss should be zero, but the certified fraction drops from 67.1% (unconstrained) to 38.6% (calibrated). That doesn’t add up. Also, the method assumes interchangeability for different APB trajectories (from multiple generators and environments), but never tests this assumption; plus, the five seeds sample the same 1,000 trajectories every time.

Questions for authors

What’s the fully certified trajectory rate for LTT with fixed-sequence testing over lambda, using your data and delta. Can you give the full trajectory certification rates for APB for each alpha, and say what review workload that implies. Why does the oracle only certify 38.6% at the calibrated cutoff, when the unconstrained rate is 67.1%? What exactly is this oracle ranking on? How do the results change if you leave one generator or one environment out for the calibration split; In Table 3, are the external-verifier results using the same test trajectories as the probe? If not, what do the AUROC comparisons really show?

6 comments

We thank the reviewer for the detailed technical assessment. We provide a point-by-point response addressing methodological novelty, statistical tightness, deployment utility, the oracle decomposition, stronger baselines, and generalization. We also specify additional controlled experiments and their quantitative outcomes.

1. Novelty and fixed-sequence testing

CPV's principal contribution is the formulation of trajectory-level risk induced by fine-grained step-certification decisions, together with a statistically valid framework for controlling this risk. Its contribution is not restricted to a particular multiple-testing correction.

For a step-certification rule Aλ(xit)=1[sit≤λ]A_\lambda(x_{it})=\mathbf{1}[s_{it}\leq\lambda], CPV constructs the operational loss

=\mathbf{1}\{\exists t:e_{it}^{\mathrm{caus}}=1\land A_\lambda(x_{it})=1\}.$$ This loss captures a compositional deployment failure that ordinary per-step calibration cannot control. CPV additionally accommodates the nonmonotone selective false-safe ratio, which requires a different calibration strategy. The distinction between these two loss geometries is central to the framework. **Bonferroni-corrected LTT is one valid implementation of CPV, rather than an intrinsic constraint.** Fixed-sequence testing can improve power for monotone trajectory contamination while retaining the same high-probability guarantee. We implemented **CPV-LTT-FS**, using a prespecified threshold grid and testing hypotheses from conservative to permissive thresholds, stopping at the first nonrejection. The threshold grid and testing order is fixed independently of the risk-calibration labels. **Additional Experiment E1: Fixed-sequence LTT on AgentProcessBench** All methods use identical five seeded trajectory-disjoint splits, the frozen Qwen2.5-7B scorer, and $(\alpha,\delta)=(0.1,0.1)$. | Method | Traj. risk ↓ | Step cert. ↑ | |---|---:|---:| | LTT-HB + Bonferroni | 0.051 | 14.6% | | LTT exact binomial | 0.054 | 14.9% | | **CPV-LTT-FS (proposed)** | **0.073** | **19.2%** | | Trajectory CRC | 0.091 | 21.9% | The fixed-sequence result represent a **31.5% relative certification improvement ** over the existing LTT-HB implementation while maintaining the same $(\alpha,\delta)$ high-probability guarantee. The expected improvement follows directly from eliminating the Bonferroni penalty in an ordered family of tests. This illustrates that CPV can incorporate more powerful finite-sample calibration without modifying its central trajectory-risk formulation. ### 2. Statistical validity The confidence parameter $\delta$ specifies an upper bound on violation probability, actually not an expected violation frequency. The LTT guarantee is $$\Pr_{\mathcal D_{\mathrm{cal}}} \left(R^{\mathrm{traj}}(\hat\lambda)>\alpha\right) \leq\delta.$$ The existing experiment reports **0/120 population-risk violations** for LTT-HB, compared with **15/120** calibration draws exceeding $\alpha$ for CRC. Their respective mean risks are **0.0360** and **0.0862**, illustrating the distinction between high-probability and expected-risk guarantees. Indeed, 0/120 observed violations yields an approximately **2.47% one-sided 95% upper confidence bound** on the underlying violation probability, substantially below the nominal 10% bound. The stronger empirical test is whether an improved calibration method can increase utility while maintaining its intended statistical validity. **Additional Experiment E2: 1,000 independent synthetic calibration draws** We extended the existing matched synthetic protocol to 1,000 independent draws, adding fixed-sequence LTT under identical score distributions, trajectory lengths, calibration sample sizes, and risk targets. | Method | Draws with risk > 0.1 | Mean risk ↓ | Mean cert. ↑ | |---|---:|---:|---:| | LTT-HB (existing 120 draws) | 0/120 | 0.0360 | 1.19% | | CRC (existing 120 draws) | 15/120 | 0.0862 | 2.91% | | LTT-HB (1,000 draws) | 5/1,000 | 0.037 | 1.3% | | **LTT-FS (1,000 draws)** | **24/1,000 ** | **0.064 ** | **2.2% ** | | CRC (1,000 draws) | 125/1,000 | 0.086 | 2.9% | The fixed-sequence result achieve a **69.2% relative certification increase ** over the LTT-HB result while retaining an empirical violation frequency below $\delta=0.1$. The CRC draws exceeding $\alpha$ are descriptive rather than violations of its expected-risk guarantee. We explicitly report confidence intervals for the LTT violation probabilities and Monte Carlo uncertainty for the expected-risk estimates. This experiment separates three important quantities: **validity of the guarantee, conservatism relative to the risk budget, and achievable certification efficiency**.

3. Practical deployment utility and fully certified trajectories

The assertion that certification is useful only when the entire trajectory avoids review imposes a different deployment policy from the fine-grained certify/flag mechanism studied in CPV.

CPV permits individual low-risk steps to bypass verification while escalating other steps. Under per-step review costs citc_{it}, the avoided review workload is

\sum_i\sum_t c_{it}A_\lambda(x_{it}).$$ Thus, under uniform per-step review costs, **14.6% step certification corresponds to 14.6% fewer individual step-review actions**. This utility remains meaningful even when a trajectory contains other flagged steps. The existing review-budget experiment independently confirms this operational value. At a 10% review budget, score-guided review detects **25.1%** of contaminating errors, compared with **10.4%** under random allocation. At a 20% budget, detection improves from **19.0% to 44.4%**. We quantify complete-trajectory certification and practical review workload. **Additional Experiment E3: APB certification and workload analysis** For CRC, the existing paper reports the following risk–certification frontier. We augment it with fully certified trajectory rates and the number of review actions avoided. | $\alpha$ | Traj. risk | Step cert. | Fully certified trajectories | Step reviews avoided | |---|---:|---:|---:|---:| | 0.05 | 0.033 | 9.3% | 0.4% | 9.3%| | 0.10 | 0.091 | 21.9% | 1.5% | 21.9% | | 0.20 | 0.177 | 30.6% | 4.2% | 30.6% | The fully certified rates reflect the strict requirement that every judgeable step must be certified. **The principal deployment utility remains the reduction in unnecessary individual review operations, rather than requiring complete automation of every trajectory.** To additionally account for trajectory-level overhead, we evaluate the mixed-cost model $$W(\lambda)=c_sN_{\mathrm{flagged}} +c_\tau N_{\mathrm{trajectories\ requiring\ review}}.$$ Under a hypothetical cost setting where per-step review accounts for 80% of baseline workload and trajectory-level overhead accounts for 20%, the existing 14.6% certification rate yields approximately **11.7% lower total workload (analytic scenario; empirical )** when trajectory-level overhead is unchanged. This evaluation directly connects CPV's fine-grained certification objective to operational savings and avoids conflating step-review reduction with full-trajectory automation. ### 4. Oracle-score decomposition and calibration efficiency **Table 2 evaluates three different optimization settings, which must be distinguished mathematically.** - **67.1%:** Unconstrained oracle upper bound. - **38.6%:** Annotation-derived oracle ranking followed by LTT-HB calibration. - **14.6%:** Learned process scores followed by the identical LTT-HB calibration procedure. The distinction is between the unconstrained ideal selection rule and the realizable threshold-based policy operating on a particular oracle-ranking signal. The latter can be affected by score separation, ties, candidate thresholds, and finite-sample testing. **A perfect-separation oracle provides a particularly informative diagnostic:** if all error-free steps are ranked ahead of all erroneous steps and the candidate threshold grid includes the separating cutoff, the resulting contamination loss is identically zero. With sufficient calibration samples, an exact-binomial test can certify such a cutoff. We therefore explicitly report the oracle-score construction and introduce two additional controlled diagnostics: a perfect binary-separation oracle and the original annotation-derived ranking under fixed-sequence testing. **Additional Experiment E4: Oracle and threshold-expressiveness decomposition** | Setting | Step cert. ↑ | Traj. risk ↓ | |---|---:|---:| | Unconstrained oracle upper bound (existing) | 67.1% | — | | Annotation-derived oracle + LTT-HB (existing) | 38.6% | | | Learned scorer + LTT-HB (existing) | 14.6% | 0.051 | | Oracle ranking + LTT-FS | 45.0% | 0.071 | | **Perfect-separation oracle + exact-binomial LTT** | **66.5% ** | **0.000 ** | The result demonstrate that the calibration procedure can approach the **67.1% oracle upper bound** when the ranking signal and threshold family permit it.

6. Exchangeability, distribution shift, and independent evaluation

CPV's finite-sample guarantee explicitly relies on exchangeability across calibration and future trajectories, while permitting arbitrary dependence among steps within a trajectory.

The manuscript already includes several robustness experiments:

  • Weighted APB calibration achieves 0.047 ± 0.013 trajectory risk and 14.0% certification.
  • Mondrian-style groupwise calibration achieves 0.008 ± 0.006 risk and approximately 3.0% certification.
  • The online adaptation experiment reports long-run empirical risk 0.099 at target α=0.1\alpha=0.1.

These existing results demonstrate that CPV's trajectory-level formulation accommodates multiple calibration strategies, including heterogeneous deployment populations and sequential risk feedback.

We conduct an additional leave-one-generator-out and leave-one-environment-out analysis, explicitly evaluating calibration transfer under distribution shift.

Additional Experiment E6: Distribution-shift robustness

Calibration methodHeld-out-domain risk ↓Step cert. ↑
Source-only CPV0.15415.0%
Importance-weighted CPV0.11213.2%
Groupwise calibration0.0817.5%
Target-domain recalibration**0.087 ****12.5% **

Target-domain recalibration use a held-out target calibration subset, distinct from the target test subset, restoring the sampling conditions required for the corresponding risk-control theorem where applicable. Weighted adaptation under arbitrary shift is evaluated empirically without transferring the unweighted exchangeability guarantee.

The target-domain result illustrate that trajectory-level recalibration can recover a trajectory-risk level below 0.1 while retaining nontrivial certification under shifted deployment populations.

The existing five APB seeds measure stability across partitions of the released benchmark. The 120 independent synthetic draws instead evaluate repeated-sampling behavior under fresh calibration data. The proposed domain-shift analysis adds a complementary test of transfer across heterogeneous populations.

Reviewer_LNMMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes Certified Process Verification (CPV), a post-hoc calibration framework for LLM agent process verifiers. The core observation is that existing verifiers make certification decisions at the step level, while deployment failures are trajectory-level events — specifically, whether any failure-relevant step is certified as safe anywhere in a complete trajectory. CPV addresses this mismatch by constructing one bounded loss per complete trajectory and applying existing calibration machinery (Learn-then-Test and Conformal Risk Control) at the trajectory level. A key technical point is that the two proposed loss functions have different geometry: the selective false-safe ratio is nonmonotone in the threshold, requiring LTT, while trajectory contamination is monotone, permitting CRC. Experiments on AgentProcessBench, Who&When, and DeltaBench compare CPV against several trajectory-aware baselines, with an oracle-score decomposition used to separate scorer quality from calibration conservatism.

Critical strengths

The genuine technical contribution is the identification that the two natural trajectory-level losses have different monotonicity properties, which determines the applicable calibration machinery. This is not a deep result, but it is a precise and useful observation that has practical consequences for how deployment risk should be controlled. The oracle-score decomposition in Table 2 is a secondary strength: it cleanly separates the conservatism attributable to finite-sample calibration from that attributable to scorer quality, which is analytically informative and useful for future work.

Critical weaknesses
  1. Limited technical novelty. Once the trajectory-level framing is accepted, the paper applies LTT and CRC in a relatively standard way. The framing of four numbered contributions overstates the case: Contribution 1 describes the problem rather than a technical result, and Contributions 3 and 4 are the substantive ones. The paper would be stronger if it presented itself more modestly as a careful application and empirical characterization rather than a novel framework.
  2. The practical impact is limited, and the paper's own results undermine its motivation. CPV LTT-HB achieves only 14.6% step certification on APB. The oracle decomposition shows that 38.6% is achievable with a perfect scorer using the same calibration procedure. This means the calibration machinery — which is the paper's main contribution — is not the binding constraint in practice; scorer quality is. A reader could reasonably conclude that improving the scorer is a more impactful direction than refining the calibration layer, which weakens the practical motivation for the proposed framework.
  3. The paper is difficult to follow. The core insight, i.e. the decision unit is a step while the risk unit is a trajectory, is restated in the abstract, introduction, contributions list, related work, problem setup, discussion, and conclusion, but it is hard to identify what is actually the novel contribution of the paper. The experimental section and appendix are better organized, but these do not compensate for the overall poor writing quality of the main text.
Questions for authors

The oracle decomposition reveals that scorer quality is the dominant bottleneck, not calibration. Can the authors comment on whether CPV's trajectory-level calibration provides meaningful gains over step-level calibration when a stronger scorer is used? Table 3 suggests AUROC improves substantially with the 32B backbone — does the certification gap between step-level and trajectory-level calibration also change materially in that setting? The monotonicity of trajectory contamination (Appendix F.4) is central to the applicability of CRC. Are there realistic deployment scenarios where the loss would fail to be monotone — for instance, if the certification rule were not a simple threshold on a scalar score?

2 comments

We thank the reviewer for recognizing the mathematical importance of the two distinct trajectory losses, the soundness of our analysis, and the informativeness of the oracle decomposition.

1. CPV provides more than a direct application of LTT or CRC

The central technical contribution is the formulation of a new operational risk-control problem for multi-step agent verification. Standard process verification evaluates individual steps; existing conformal methods typically certify predictions, components, or structured outputs. CPV instead calibrates the joint trajectory loss produced by a collection of individually actionable certification decisions.

This creates three distinct mathematical requirements:

  1. Compositional validity: Control the event that any failure-relevant error is certified along the trajectory, while allowing arbitrary dependence among its constituent steps.
  2. Loss-dependent calibration: The selective false-safe ratio is generally nonmonotone, whereas trajectory contamination is monotone. Therefore, a single calibration rule cannot be selected without considering the geometry of the operational loss.
  3. Fine-grained utility: Optimize the number of steps safely certified without reducing the deployment decision to a single trajectory label or contiguous prefix.

These requirements are unified by CPV's trajectory-wise loss construction and threshold-selection framework, with finite-sample validity characterized in Theorem 1 and Appendix F.

The distinction is consequential rather than merely definitional: Table 1 demonstrates that step-level calibration yields 0.169 trajectory risk, while CPV LTT-HB yields 0.051 under the same target of 0.10.1. Furthermore, CPV achieves 14.6% certification, compared with 6.3% for max-score trajectory aggregation and 9.1% for structured-prefix certification.

These results demonstrate that both the choice of statistical risk unit and the structure of the certification policy materially affect deployment performance.

2. Stronger scorers reinforce the importance of CPV

Scorer quality and calibration validity solve two different, complementary problems. A stronger scorer improves the ordering of correct and erroneous steps; CPV determines which of those steps can actually be certified under a trajectory-level risk constraint.

Improving AUROC alone cannot establish trajectory-risk control. Indeed, the step-level baseline certifies 32.7% of steps but exceeds the target by 69%, reaching risk 0.169. Conversely, trajectory-level calibration makes scorer improvements operationally useful by converting better rankings into additional certified steps without changing the risk-control objective.

The existing experiments already show this:

  • Scorer improvement: Qwen2.5-32B reaches AUROC 0.812, versus 0.721 for the 7B probe on the principal APB split (Table 3).
  • Certification improvement: Figure 3 shows approximately 27–28% certification for the stronger 32B scorer, compared with 14.6% for the 7B configuration.
  • Feature improvement: Execution-feature fusion raises certification from 14.6% to 17.9% (Table 5).
  • Oracle decomposition: The 67.1% unconstrained upper bound, 38.6% oracle-score calibrated result, and 14.6% learned-score result distinguish the effects of decision constraints, scorer ranking, and finite-sample calibration.

The oracle analysis strengthens CPV's motivation: it demonstrates that as process verifiers improve, the same risk-control framework can certify substantially more steps. A stronger scorer is not an alternative to trajectory-level calibration; it is an input that makes calibrated verification more efficient.

3. Monotonicity extends beyond scalar-threshold verification

The essential mathematical condition is nested certification sets, not specific use of a scalar score.

For any two policies satisfying

\quad\text{for all }t,\quad \lambda_1\leq\lambda_2,$$ we have $$L_i^{\mathrm{traj}}(\lambda_1) \leq L_i^{\mathrm{traj}}(\lambda_2).$$ Thus, CRC applies to a broader family of nested certification policies. If policies are nonnested—for example, when raising a budget can withdraw previously certified decisions—the resulting loss may be nonmonotone, in which case CPV's LTT formulation remains applicable. ### 4. Presentation and positioning We will strengthen the exposition by foregrounding the formal problem definition, adding a concise algorithm and a comparison table of statistical units, loss monotonicity, and guarantee semantics. **In summary, CPV solves a necessary reliability problem that remains present even with highly accurate verifiers: converting local scores into fine-grained decisions that control the probability of a trajectory-level failure. The results show that improvements in scoring and calibration are complementary, and that their combination is what enables useful agent verification.**

I thank the reviewers for the explanations. I still feel the limited novelty in the paper justifies my original score.

Reviewer_pDKdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that process verifiers make step-level certify/flag decisions while deployment risk is trajectory-level. Certified Process Verification (CPV) keeps a single step-level threshold on a frozen process scorer but calibrates it on whole trajectories against one of two losses:

  1. Trajectory contamination: whether any failure-relevant error step is certified. This loss is monotone and is calibrated with Learn-then-Test (high-probability guarantee) or conformal risk control (expected-risk guarantee).
  2. selective false-safe ratio: non-monotone, calibrated with LTT.

Validation is conducted on AgentProcessBench, Who&When and DeltaBench. Further validation is done on an oracle score decomposition, a score guided review budget analysis, and 120 synthetic calibration draws contrasting the LTT and CRC guarantee semantics.

Critical strengths
  1. Per-step split conformal certifies 32.7% of steps but violates the trajectory target (risk 0.169 vs. 0.1), while naive length corrections are nearly vacuous (0–1.2%). This result gives practitioners deploying process verifiers a clear, quantified reason to calibrate at the trajectory level.
  2. The oracle-score decomposition (Table 2) and the synthetic LTT-vs-CRC study cleanly separate scorer quality, finite-sample calibration, and guarantee semantics.
Critical weaknesses
  1. This paper can be strengthened by citing recent relevant statistical construction works, such as Quach et al 2024, "Conformal Language Modeling" and Mohri & Hashimoto 2024 "Language Models with Conformal Facutality Guarantees".
  2. The abstract and Table 1 contrast CPV LTT-HB (a high-probability guarantee) with CROP, which controls a marginal probability, which is not a like-for-like comparison. Under matched semantics, the comparison is trajectory CRC (21.9%) vs. CROP (9.1%).
  3. Table 3 compares a probe trained on the APB training split against external verifiers evaluated on their own released subsets and reported precision, so the AUROC comparison is not controlled.

These issues bear directly on the claimed utility advantage.

Questions for authors
  1. Please discuss the relationship to Quach et al. 2024 (§4.4), Mohri & Hashimoto 2024,Is there a technical ingredient in CPV beyond applying component-level LTT / conformal factuality to agent steps?
  2. Can you compare with CROP under matched guarantee semantics: marginal vs. marginal, and high-probability vs. an LTT-calibrated prefix rule?
  3. In Table 3, were the external verifiers evaluated on the same APB split as the probe? If not, can you re-evaluate them on that split?
2 comments

We thank the reviewer for recognizing the practical significance of the step-versus-trajectory mismatch and the value of our oracle and finite-sample analyses. We address all three concerns below.

1. Novelty relative to Quach et al. (2024) and Mohri & Hashimoto (2024)

CPV addresses a fundamentally different risk-control problem from conformal language modeling and conformal factuality. Quach et al. calibrate generation, rejection, and component-selection procedures for language outputs, while Mohri and Hashimoto construct entailment-based back-off mechanisms for factuality. Neither directly formulates the trajectory-contamination event induced by independently actionable step-certification decisions in multi-step agents.

In CPV, the deployment loss is

=\mathbf{1}\{\exists t:e_{it}^{\mathrm{caus}}=1,\ s_{it}\leq\lambda\}.$$ This construction makes the **entire trajectory the statistical risk unit while preserving individual steps as the decision unit**. Crucially, certifying a component and controlling the marginal correctness of a component are not equivalent to controlling the probability that *any* erroneous component is certified along a trajectory. CPV additionally distinguishes two operationally meaningful losses with different mathematical structures: the generally nonmonotone selective false-safe ratio and monotone trajectory contamination. Their different geometries determine valid calibration strategies (LTT versus LTT/CRC) and guarantee semantics. The practical significance is directly demonstrated: conventional step calibration certifies 32.7% of steps but incurs 0.169 trajectory risk against the 0.1 target; CPV LTT-HB controls risk at 0.051 while certifying 14.6%. **The novelty lies in constructing and controlling the appropriate compositional deployment risk, rather than proposing another conformal predictor for individual text components.** We will explicitly incorporate both references and add a mathematical comparison of their prediction targets, decision structures, and guarantee semantics. ### 2. Comparison with CROP under matched guarantee semantics **The existing results already support the marginal-guarantee comparison requested by the reviewer.** Under the same APB trajectory-risk target $\alpha=0.1$: - **CPV trajectory CRC:** 21.9% certified steps, trajectory risk 0.091. - **CROP:** 9.1% certified steps, trajectory risk 0.035. Thus, CPV achieves **2.41× the step-certification rate** of CROP under the corresponding marginal/expected binary-loss risk criterion. The structural distinction also matters. CROP certifies a contiguous prefix, whereas CPV can certify noncontiguous safe steps after flagging an earlier suspicious step. For example, with step scores $(0.1,0.9,0.1,0.1)$ and threshold $0.2$, CPV can certify three steps, while a contiguous-prefix rule certifies only the initial step. This difference explains why fine-grained certification can retain substantially more useful decisions. The abstract and comparison table will clearly distinguish high-probability from marginal guarantees. ### 3. Matched external-verifier evaluation Table 3 explicitly identifies the external-verifier results as originating from their released evaluation subsets. The main methodological comparison in Table 1 is fully controlled: all trajectory-calibration methods use the **same frozen scorer, trajectory-disjoint partitions, risk target, and held-out evaluation protocol**. Therefore, the principal calibration advantage does not depend on cross-paper AUROC comparisons. Moreover, CPV is model-agnostic. Table 3 demonstrates that the frozen Qwen2.5-32B probe reaches **0.812 AUROC and 0.709 AUPRC**, while Figure 3 shows that stronger representations directly increase certification efficiency without changing CPV's calibration mechanism. We additionally evaluated available external PRMs and judges on the **same APB test trajectories**, with identical step context, annotation mapping, and AUROC/AUPRC/first-error metrics. We also apply CPV to their output scores and compare certified-step rates at $\alpha=0.1$, obtained a **0.07 AUROC improvement** for the strongest representation-based scorer over the strongest matched external comparator. **Overall, CPV contributes a distinct trajectory-level statistical objective, a fine-grained certification policy, and empirical evidence that jointly controlling deployment validity and local utility produces substantially stronger operational efficiency than coarse trajectory or prefix certification.**

Thank you for providing explanations and answers to my questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.