- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates whether a fixed, cheap (in terms of computational resources) evaluation protocol can provide sufficient statistical evidence for making decisions about the performance of an underlying reasoning model. Protocols are compared on validity, decisiveness and cost on two benchmark datasets: CommonsenseQA and OpenBookQA. The paper introduces a planning framework for finding good protocols based on pre-existing logs. Using this planner, it finds that partial evaluations (e.g., 50% of data) are effective for the CommonsenseQA, but not necessarily for OpenBookQA. They identify that, at their chosen threshold of 10% accepted errors, it is harder to detect cases where a model is passing, than to detect when a model is obviously failing.
Critical strengths
The idea of studying how cheaper evaluation protocols can be a stand in for more expensive ones on particular evaluation decisions is useful, especially if monitoring cost becomes substantial as agentic workflows become increasingly useful.
Critical weaknesses
- The paper suffers from heavy readability issues due to obscure terms and definitions and extremely dense prose. It introduces labels such as “decision service”, “frozen forecast worlds” and “source ownership” instead of using plain English terms, or re-using earlier terminology. Beta is the budget, which is a key parameter they vary, but this is never actually defined directly and has to be inferred. As is, the paper buries its fairly reasonable but standard statistical design questions under dense prose and nonstandard terminology.
- Experimental validation involves only two models and two Q&A benchmarks, which seems quite narrow if the purpose is to show that their planning framework is generally able to identify strong low-cost audit protocols. As far as I can tell, their planner also mostly finds random uniform sampling for the main 50% budget setting, so the implicit claim that the “selected audit” helps seems somewhat vacuous and mostly a negative result for their planning framework (in fact, it plausibly increases total cost relative to just selecting uniform sampling a priori).
Questions for authors
Given the fundamental issues with the paper, I have no questions that could materially affect my assessment.