- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks an operational question: what fraction of agent-trajectory evaluation can be handed to an LLM judge with a certificate that the error rate among auto-decided trajectories stays below α? It shows that i.i.d. selective-judging certificates are anti-conservative in the multi-rollout regime agent evaluation actually produces, that finite-sample cluster-valid alternatives certify nothing at realistic task counts, and proposes TASKBOOT, a task-level bootstrap certificate occupying the working regime between invalid and vacuous. Around it: a 4B logprob judge (SFT + reject-weighted GRPO) that is the only judge certifying on both headline corpora at ~100× lower cost than the frontier configuration that beats it on one; a two-parameter ex-ante predictability result; and the certificate reused as a self-training filter with contamination bounded by α.
Critical strengths
- Moving judge evaluation from agreement to a certified automation fraction is the language deployment teams need.
- Experimental discipline. Task-level splits, CAL-only selection, a single TEST read, failed corpora kept in the main text, and all audits reported.
Critical weaknesses
- TASKBOOT's validity is asymptotic: the paper calls it an approximation, not a finite-sample theorem. The small-G violation rate is therefore simulation-backed — twelve task-effect cells plus three adversarial designs, all within the authors' parametric family. The only real-data check at G=32 is a single CAL→TEST audit, and the 17.5% motivating audit has no matched control on TASKBOOT's own threshold; neither can estimate a rate. For cluster structures outside that family, nothing bounds the G≈20–32 rate, so "uniformly safe" is an empirical claim, not a guarantee.
Questions for authors
- Can the bridge between population-validity in simulation and audit failure on WEB-M be made quantitative (e.g., relating audit violation rate to DEFF or resampling variance) rather than narrative?
- Were frontier judges given any prompt optimization (few-shot, trajectory-aware) before concluding they do not transfer? And unparsed responses are excluded rather than counted against the judge — how sensitive is Table 3 to this choice?
3 comments
Thanks authors for the rebuttal. The revision addresses most of my concerns, and I maintain my weak accept rating.
Dear Reviewer bN5P,
Thank you for reading the revision and for your continued support. Your questions led directly to several of its main additions: the real-data validity study with naive CP as the matched control, the design-effect model of naive CP’s failures (Table 8), and the prompt-optimised frontier baseline with every response parsed. Thank you again for your time and your constructive review.
Gentle follow-up before the discussion closes
Dear Reviewer bN5P,
Thank you again for your positive and constructive review. Before the discussion closes on November 18, we wanted to check whether our response answers your questions. In brief: (W1) TaskBoot's violation rate is now measured on 11 real cluster structures with naive CP as the matched control (TaskBoot worst 3.5% from 50 tasks; naive CP up to 19%), a new lower bound explains why no procedure can be both uniformly valid and useful below about 29 tasks, and a sweep of 104 adversarial settings built on that bound, run this week, keeps the revised certificate within δ in every setting (worst 4.1%; naive CP up to 29.7%; Appendix E); (Q1) the design-effect model predicts naive CP's failure quantitatively (Web-M: observed 0.191, predicted 0.189; Table 8); (Q2) a few-shot, rubric-guided frontier judge with every response parsed still certifies 0 on Tool, so Table 3's conclusion does not depend on how unparsed rows are treated.
If these strengthen your assessment, we would be grateful if you would consider reflecting that in your rating. We are happy to answer any further questions.