- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how the trajectory score used in conformal risk control allocates wrong commits across trajectory-length groups. It introduces trajectory spatial contrast (TSC), which interpolates step uncertainty onto a fixed grid and contrasts spatial-rank norms relative to correct and incorrect reference trajectories, and a length-local variant, L-TSC. A separate calibration split supplies the standard marginal wrong-commit guarantee. Experiments compare twelve scoring rules across nine benchmark–model populations, finding that accumulating scores often concentrate wrong commits on short trajectories, while the TSC family generally reduces length-group concentration at some cost in acceptance rate.
Critical strengths
-
The distinction between marginal risk and its allocation across lengths is useful and supported by concrete, consequential diagnostics. For example, Table 2 (p. 8) shows pseudo-Noisy-OR concentrating 94–100% of wrong commits in one length group in several populations, whereas L-TSC substantially reduces that concentration. Reporting both the largest error share H and the largest within-group wrong-commit rate W avoids conflating equal error mass with equal group risk. The mixture-shift analysis in Appendix B (pp. 21–22) makes the deployment implication precise under explicitly invariant within-group laws.
-
The calibration scope and experimental controls are carefully specified. Section 2.3 and Theorems A1–A2 (pp. 5, 18–19) separate reference fitting from calibration, handle strict thresholds and ties, and state an expectation guarantee for marginal wrong commits rather than claiming conditional accuracy or groupwise control. The shared-representation experiment, extra reference draws, label-budget control, and signal/geometry sensitivity analyses help assess whether the observed trade-off is stable. These are substantial strengths even though the CRC guarantee itself is inherited.
Critical weaknesses
-
The evidence is less conclusive about the specific value of spatial-rank contrast than about the general diagnosis. In the common-representation study, TSC uses correctness-labeled reference clouds, while all five competing aggregations of the same U fit nothing and consume no reference labels (Appendix C, p. 22). The equal-label-budget control instead gives those methods more calibration examples (pp. 27, 30), so it does not distinguish spatial-rank geometry from the benefit of learning a nonlinear correctness score on the four coordinates. A single conventional nonlinear score fitted only on the same reference split, with the same CRC calibration, would directly address that attribution. This limits the methodological contribution established by Figure 1/Table 9; it does not invalidate the length-concentration diagnosis or the stated guarantee.
-
The strongest acceptance-rate results for L-TSC occur in two WebShop populations with problematic agent–environment interaction. Appendix D.1 (p. 26) states that Gemma fails the pilot parsing/length checks and that 95% of Mistral test steps are unparsed; both populations are retained in the main comparisons and nine-population summaries. Table 2 (p. 8) gives L-TSC its acceptance-rate leads on these two populations. Transparent reporting is welcome, but it remains unclear how much those gains reflect handling malformed-action loops rather than uncertainty in functioning agents. The other populations still support the concentration finding, so this primarily limits the breadth of the practical utility evidence.
Questions for authors
-
Can the authors compare TSC with one reference-fitted nonlinear correctness score on exactly the same U (and, for the L-TSC comparison, the same available length information), preserving the reference/calibration/test separation? This would clarify whether the favorable acceptance/concentration trade-off depends specifically on spatial-rank contrast.
-
For the two WebShop populations discussed on p. 26, how many episodes contain parsing failures, and how do the acceptance and wrong-commit differences break down between trajectories with and without those failures? This would clarify the source and practical scope of the L-TSC acceptance-rate gains.
I have read the rebuttal and decided to maintain my score.