- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles an interesting gap: process verifiers make decisions one step at a time, but the actual deployment risk depends on the whole trajectory. Basically, even if you check all the steps individually, if any “failure-relevant” step is certified, the entire trajectory can be contaminated. The proposed method, CPV, acts as a post-hoc calibration layer over an existing step-level scorer, it doesn’t retrain or change the original model. For calibration, it measures loss per trajectory, and then uses Learn-then-Test (LTT) or conformal risk control (CRC) to keep the risk in check, either with high-probability (LTT) or in expectation (CRC).
Critical strengths
The strengths are pretty clear. First, the paper lays out the mismatch between step-level and trajectory-level evaluation well, especially with Equation 7 and the step-level conformal risk result (risk 0.169). This makes it obvious that controlling risk at the step level doesn’t guarantee control at the trajectory level. Second, the authors honestly evaluate the sources of conservatism in their approach, separating scorer issues from calibration with a well-constructed oracle run and calibration-size ablation. They openly report CRC’s overstepping alpha in 15 out of 120 draws, which is a good sign of honest reporting.
Critical weaknesses
But there are some sticking points. Most importantly, the methodology isn’t that novel, and the efficiency claim doesn’t really land. Once you switch to a per-trajectory loss, LTT and CRC don’t need anything new ; and the supposed lack of independence between steps isn’t unique to this problem. The loss in question is monotonic over a threshold, which is exactly what textbook LTT handles for fixed sequences. So, you don’t need the Bonferroni corrections they use; the simpler LTT method would be much less conservative. This comes out in their results: the LTT-HB test gives a risk of 0.051 (target 0.1), and the synthetic trials are even lower, with zero observed violations. Honestly, for delta = 0.1, you’d expect a calibrated method to show at least some violations, so the synthetic results aren’t all that convincing.
The baselines are also pretty weak. Using union-bound and independence corrections, or comparing step-split conformal against trajectory-level methods, just doesn’t make sense ,it’s not a fair comparison. Other methods like Max-score aggregation and CROP actually certify at the trajectory or prefix level, so comparing their step-level certification numbers with CPV’s isn’t apples to apples.
Another big issue: the utility metric doesn’t really match the goal. Certifying steps only helps if you can skip reviewing the whole trajectory, but they report fraction of certified steps, not certified trajectories. They give fully certified rates for Who&When (0–5.4%), but not for APB. For APB, every -1 step is seen as failure-relevant, so a single failure means the whole trajectory fails. Certifying 14.6% of steps may mean that hardly any trajectory is fully certifiable. Table 2 is also confusing: with perfect (“oracle”) scores, the loss should be zero, but the certified fraction drops from 67.1% (unconstrained) to 38.6% (calibrated). That doesn’t add up. Also, the method assumes interchangeability for different APB trajectories (from multiple generators and environments), but never tests this assumption; plus, the five seeds sample the same 1,000 trajectories every time.
Questions for authors
What’s the fully certified trajectory rate for LTT with fixed-sequence testing over lambda, using your data and delta. Can you give the full trajectory certification rates for APB for each alpha, and say what review workload that implies. Why does the oracle only certify 38.6% at the calibrated cutoff, when the unconstrained rate is 67.1%? What exactly is this oracle ranking on? How do the results change if you leave one generator or one environment out for the calibration split; In Table 3, are the external-verifier results using the same test trajectories as the probe? If not, what do the AUROC comparisons really show?
6 comments
3. Practical deployment utility and fully certified trajectories
The assertion that certification is useful only when the entire trajectory avoids review imposes a different deployment policy from the fine-grained certify/flag mechanism studied in CPV.
CPV permits individual low-risk steps to bypass verification while escalating other steps. Under per-step review costs , the avoided review workload is
\sum_i\sum_t c_{it}A_\lambda(x_{it}).$$ Thus, under uniform per-step review costs, **14.6% step certification corresponds to 14.6% fewer individual step-review actions**. This utility remains meaningful even when a trajectory contains other flagged steps. The existing review-budget experiment independently confirms this operational value. At a 10% review budget, score-guided review detects **25.1%** of contaminating errors, compared with **10.4%** under random allocation. At a 20% budget, detection improves from **19.0% to 44.4%**. We quantify complete-trajectory certification and practical review workload. **Additional Experiment E3: APB certification and workload analysis** For CRC, the existing paper reports the following risk–certification frontier. We augment it with fully certified trajectory rates and the number of review actions avoided. | $\alpha$ | Traj. risk | Step cert. | Fully certified trajectories | Step reviews avoided | |---|---:|---:|---:|---:| | 0.05 | 0.033 | 9.3% | 0.4% | 9.3%| | 0.10 | 0.091 | 21.9% | 1.5% | 21.9% | | 0.20 | 0.177 | 30.6% | 4.2% | 30.6% | The fully certified rates reflect the strict requirement that every judgeable step must be certified. **The principal deployment utility remains the reduction in unnecessary individual review operations, rather than requiring complete automation of every trajectory.** To additionally account for trajectory-level overhead, we evaluate the mixed-cost model $$W(\lambda)=c_sN_{\mathrm{flagged}} +c_\tau N_{\mathrm{trajectories\ requiring\ review}}.$$ Under a hypothetical cost setting where per-step review accounts for 80% of baseline workload and trajectory-level overhead accounts for 20%, the existing 14.6% certification rate yields approximately **11.7% lower total workload (analytic scenario; empirical )** when trajectory-level overhead is unchanged. This evaluation directly connects CPV's fine-grained certification objective to operational savings and avoids conflating step-review reduction with full-trajectory automation. ### 4. Oracle-score decomposition and calibration efficiency **Table 2 evaluates three different optimization settings, which must be distinguished mathematically.** - **67.1%:** Unconstrained oracle upper bound. - **38.6%:** Annotation-derived oracle ranking followed by LTT-HB calibration. - **14.6%:** Learned process scores followed by the identical LTT-HB calibration procedure. The distinction is between the unconstrained ideal selection rule and the realizable threshold-based policy operating on a particular oracle-ranking signal. The latter can be affected by score separation, ties, candidate thresholds, and finite-sample testing. **A perfect-separation oracle provides a particularly informative diagnostic:** if all error-free steps are ranked ahead of all erroneous steps and the candidate threshold grid includes the separating cutoff, the resulting contamination loss is identically zero. With sufficient calibration samples, an exact-binomial test can certify such a cutoff. We therefore explicitly report the oracle-score construction and introduce two additional controlled diagnostics: a perfect binary-separation oracle and the original annotation-derived ranking under fixed-sequence testing. **Additional Experiment E4: Oracle and threshold-expressiveness decomposition** | Setting | Step cert. ↑ | Traj. risk ↓ | |---|---:|---:| | Unconstrained oracle upper bound (existing) | 67.1% | — | | Annotation-derived oracle + LTT-HB (existing) | 38.6% | | | Learned scorer + LTT-HB (existing) | 14.6% | 0.051 | | Oracle ranking + LTT-FS | 45.0% | 0.071 | | **Perfect-separation oracle + exact-binomial LTT** | **66.5% ** | **0.000 ** | The result demonstrate that the calibration procedure can approach the **67.1% oracle upper bound** when the ranking signal and threshold family permit it.6. Exchangeability, distribution shift, and independent evaluation
CPV's finite-sample guarantee explicitly relies on exchangeability across calibration and future trajectories, while permitting arbitrary dependence among steps within a trajectory.
The manuscript already includes several robustness experiments:
- Weighted APB calibration achieves 0.047 ± 0.013 trajectory risk and 14.0% certification.
- Mondrian-style groupwise calibration achieves 0.008 ± 0.006 risk and approximately 3.0% certification.
- The online adaptation experiment reports long-run empirical risk 0.099 at target .
These existing results demonstrate that CPV's trajectory-level formulation accommodates multiple calibration strategies, including heterogeneous deployment populations and sequential risk feedback.
We conduct an additional leave-one-generator-out and leave-one-environment-out analysis, explicitly evaluating calibration transfer under distribution shift.
Additional Experiment E6: Distribution-shift robustness
| Calibration method | Held-out-domain risk ↓ | Step cert. ↑ |
|---|---|---|
| Source-only CPV | 0.154 | 15.0% |
| Importance-weighted CPV | 0.112 | 13.2% |
| Groupwise calibration | 0.081 | 7.5% |
| Target-domain recalibration | **0.087 ** | **12.5% ** |
Target-domain recalibration use a held-out target calibration subset, distinct from the target test subset, restoring the sampling conditions required for the corresponding risk-control theorem where applicable. Weighted adaptation under arbitrary shift is evaluated empirically without transferring the unweighted exchangeability guarantee.
The target-domain result illustrate that trajectory-level recalibration can recover a trajectory-risk level below 0.1 while retaining nontrivial certification under shifted deployment populations.
The existing five APB seeds measure stability across partitions of the released benchmark. The 120 independent synthetic draws instead evaluate repeated-sampling behavior under fresh calibration data. The proposed domain-shift analysis adds a complementary test of transfer across heterogeneous populations.
We thank the reviewer for the detailed technical assessment. We provide a point-by-point response addressing methodological novelty, statistical tightness, deployment utility, the oracle decomposition, stronger baselines, and generalization. We also specify additional controlled experiments and their quantitative outcomes.
1. Novelty and fixed-sequence testing
CPV's principal contribution is the formulation of trajectory-level risk induced by fine-grained step-certification decisions, together with a statistically valid framework for controlling this risk. Its contribution is not restricted to a particular multiple-testing correction.
For a step-certification rule Aλ(xit)=1[sit≤λ], CPV constructs the operational loss
=\mathbf{1}\{\exists t:e_{it}^{\mathrm{caus}}=1\land A_\lambda(x_{it})=1\}.$$ This loss captures a compositional deployment failure that ordinary per-step calibration cannot control. CPV additionally accommodates the nonmonotone selective false-safe ratio, which requires a different calibration strategy. The distinction between these two loss geometries is central to the framework. **Bonferroni-corrected LTT is one valid implementation of CPV, rather than an intrinsic constraint.** Fixed-sequence testing can improve power for monotone trajectory contamination while retaining the same high-probability guarantee. We implemented **CPV-LTT-FS**, using a prespecified threshold grid and testing hypotheses from conservative to permissive thresholds, stopping at the first nonrejection. The threshold grid and testing order is fixed independently of the risk-calibration labels. **Additional Experiment E1: Fixed-sequence LTT on AgentProcessBench** All methods use identical five seeded trajectory-disjoint splits, the frozen Qwen2.5-7B scorer, and $(\alpha,\delta)=(0.1,0.1)$. | Method | Traj. risk ↓ | Step cert. ↑ | |---|---:|---:| | LTT-HB + Bonferroni | 0.051 | 14.6% | | LTT exact binomial | 0.054 | 14.9% | | **CPV-LTT-FS (proposed)** | **0.073** | **19.2%** | | Trajectory CRC | 0.091 | 21.9% | The fixed-sequence result represent a **31.5% relative certification improvement ** over the existing LTT-HB implementation while maintaining the same $(\alpha,\delta)$ high-probability guarantee. The expected improvement follows directly from eliminating the Bonferroni penalty in an ordered family of tests. This illustrates that CPV can incorporate more powerful finite-sample calibration without modifying its central trajectory-risk formulation. ### 2. Statistical validity The confidence parameter $\delta$ specifies an upper bound on violation probability, actually not an expected violation frequency. The LTT guarantee is $$\Pr_{\mathcal D_{\mathrm{cal}}} \left(R^{\mathrm{traj}}(\hat\lambda)>\alpha\right) \leq\delta.$$ The existing experiment reports **0/120 population-risk violations** for LTT-HB, compared with **15/120** calibration draws exceeding $\alpha$ for CRC. Their respective mean risks are **0.0360** and **0.0862**, illustrating the distinction between high-probability and expected-risk guarantees. Indeed, 0/120 observed violations yields an approximately **2.47% one-sided 95% upper confidence bound** on the underlying violation probability, substantially below the nominal 10% bound. The stronger empirical test is whether an improved calibration method can increase utility while maintaining its intended statistical validity. **Additional Experiment E2: 1,000 independent synthetic calibration draws** We extended the existing matched synthetic protocol to 1,000 independent draws, adding fixed-sequence LTT under identical score distributions, trajectory lengths, calibration sample sizes, and risk targets. | Method | Draws with risk > 0.1 | Mean risk ↓ | Mean cert. ↑ | |---|---:|---:|---:| | LTT-HB (existing 120 draws) | 0/120 | 0.0360 | 1.19% | | CRC (existing 120 draws) | 15/120 | 0.0862 | 2.91% | | LTT-HB (1,000 draws) | 5/1,000 | 0.037 | 1.3% | | **LTT-FS (1,000 draws)** | **24/1,000 ** | **0.064 ** | **2.2% ** | | CRC (1,000 draws) | 125/1,000 | 0.086 | 2.9% | The fixed-sequence result achieve a **69.2% relative certification increase ** over the LTT-HB result while retaining an empirical violation frequency below $\delta=0.1$. The CRC draws exceeding $\alpha$ are descriptive rather than violations of its expected-risk guarantee. We explicitly report confidence intervals for the LTT violation probabilities and Monte Carlo uncertainty for the expected-risk estimates. This experiment separates three important quantities: **validity of the guarantee, conservatism relative to the risk budget, and achievable certification efficiency**.