- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper decomposes selective-generation risk into infeasible candidate pools and selection errors within mixed pools, using this distinction for diagnosis while certifying total risk with an exact binomial bound. CulturalFrames and LiveCodeBench illustrate different failure sources; three simpler benchmarks pass certification, and synthetic experiments examine nondegenerate pools and risk–coverage tradeoffs.
Critical strengths
- Acceptance uses total selected failures, while component bounds explain rejection, avoiding invalid combinations of uncertain component estimates.
- The paper provides useful identification results. Sharp bounds show why pairwise accuracy cannot reliably establish top-one success and why selected-only labels cannot establish improvement over random selection.
- In cultural failure diagnosis, SigLIP makes 21 avoidable selection errors among 94 emitted requests, with an adjusted lower bound of 14.4% exceeding the entire risk budget under the stated independence assumption.
Critical weaknesses
- In Table 2, there are multiple 'tbc's, which I suspected is some unfinished results?
- The usefulness of attribution for choosing repairs is insufficiently demonstrated. All positive benchmark rows are K=2, all-mixed cases; comparing generator and verifier interventions on a common cohort at matched coverage would better establish whether the diagnosis predicts effective improvements.
- The paper sweeps 60 thresholds without specifying simultaneous correction or independent confirmation of the selected threshold, so pointwise 95% bounds alone do not justify the claimed maximum certifiable coverage.
Questions for authors
- Can you provide Qwen’s cohort sizes and evaluation stages? The development interval matches mixed-pool successes, which appears inconsistent with a nominal 20% development split of 203 pools; please explain the cohort and distinguish the reported evaluation from the unscored sealed test.
- How are candidate-level cultural labels aggregated, and why was 0.75 selected?