acceptodds
Under review as a conference paper at ICLR 2027

When Is a Cheap LLM Audit Decisive? Prospective Forecasts and Measured Costs

Abstract

The value of efficient evaluation lies in the decisions its evidence supports. We introduce a prospective audit protocol that separates false-assertion control, the probability of a correct definitive decision, and the cost of acquiring model outputs. Before observing new target outcomes, a fixed planner chooses among whole-question sampling, a paid target pilot, partial trajectories, and complete execution. On two new question pools shared by Qwen3 and Ministral3, the selected CommonsenseQA audits detect excessive risk with probabilities 97.51% and 98.80% at 47.5% and 40.1% of full-evaluation cost when source logs already exist. OpenBookQA audits cost 51.5% and 58.1% but confirm the acceptable pools with probabilities 63.97% and 26.83%. Constructing the source logs used by the planner raises its four evaluation-cost ratios to 1.25–1.76. All four prospective uniform-audit forecasts underestimate decision probability. Exact sample-size curves, same-observation inference controls, and an algorithmic CollabEval comparison distinguish the roles of sampling, inference, and information cost. These measurements connect evaluation efficiency to decision readiness under fixed finite-pool designs.

Primary area:
datasets and benchmarks
Keywords:
language model evaluation, finite-population inference, statistical auditing, decision power, evaluation cost

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bwdkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether a fixed, cheap (in terms of computational resources) evaluation protocol can provide sufficient statistical evidence for making decisions about the performance of an underlying reasoning model. Protocols are compared on validity, decisiveness and cost on two benchmark datasets: CommonsenseQA and OpenBookQA. The paper introduces a planning framework for finding good protocols based on pre-existing logs. Using this planner, it finds that partial evaluations (e.g., 50% of data) are effective for the CommonsenseQA, but not necessarily for OpenBookQA. They identify that, at their chosen threshold of 10% accepted errors, it is harder to detect cases where a model is passing, than to detect when a model is obviously failing.

Critical strengths

The idea of studying how cheaper evaluation protocols can be a stand in for more expensive ones on particular evaluation decisions is useful, especially if monitoring cost becomes substantial as agentic workflows become increasingly useful.

Critical weaknesses
  1. The paper suffers from heavy readability issues due to obscure terms and definitions and extremely dense prose. It introduces labels such as “decision service”, “frozen forecast worlds” and “source ownership” instead of using plain English terms, or re-using earlier terminology. Beta is the budget, which is a key parameter they vary, but this is never actually defined directly and has to be inferred. As is, the paper buries its fairly reasonable but standard statistical design questions under dense prose and nonstandard terminology.
  2. Experimental validation involves only two models and two Q&A benchmarks, which seems quite narrow if the purpose is to show that their planning framework is generally able to identify strong low-cost audit protocols. As far as I can tell, their planner also mostly finds random uniform sampling for the main 50% budget setting, so the implicit claim that the “selected audit” helps seems somewhat vacuous and mostly a negative result for their planning framework (in fact, it plausibly increases total cost relative to just selecting uniform sampling a priori).
Questions for authors

Given the fundamental issues with the paper, I have no questions that could materially affect my assessment.

Reviewer_REy5OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies when a reduced-cost LLM evaluation can support a definitive decision about accepted-answer risk and coverage. It combines prospective acquisition planning, finite-population confidence bounds, and measured execution costs, finding that cheap audits can be decisive on some fixed question pools but remain inconclusive on others.

Critical strengths

Clear separation of validity, decisiveness, and cost.

Transparent reporting of unsuccessful forecasts.

Critical weaknesses

The decision-oriented perspective is useful, and I appreciate the explicit accounting for source-information costs. However, I still have several major concerns that make me recommend rejection.

First, the methodological contribution appears limited. Section 3 uses classical hypergeometric tail inversion, a union bound, and enumeration of possible sample counts to compute fixed-test power. The distinction between error control, power, and acquisition cost is useful, but the paper does not clearly establish what new statistical or algorithmic capability follows from combining them. The first and sustained crossings also characterize the chosen discrete test rather than improve it. What does your protocol enable beyond a conventional finite-population power analysis with an explicit cost model?

Second, the prospective planner does not reliably answer the paper’s central operational question. Table 1 forecasts uniform-audit decisiveness of 48.75% and 33.28% on CommonsenseQA, whereas the actual values are 97.51% and 99.81%. Both actual error counts exceed the maximum of all 200 forecast worlds. These are substantial failures in predicting whether an audit will meet the 90% service target, even though their conservative direction is reported honestly. Since the exact power curves require the hidden census and are therefore retrospective, how can an evaluator use your method to identify a decision-ready budget before purchasing the target outputs? The paper needs stronger prospective validation of this link.

Third, much of the source-cost result concerns information that the selected audit does not need. Three of the four primary selections are uniform sampling. As Section 5.4 acknowledges, these audits can omit source construction and planning altogether. Consequently, the reported 1.25 to 1.76 cost ratios establish the expense of this particular procurement pipeline, rather than a necessary cost of obtaining decisive evidence. The Ministral CommonsenseQA result provides a more substantive tradeoff: S4 takes 91.5 minutes versus 112.0 minutes for source-free uniform sampling, with slightly lower decisiveness.

Questions for authors

How can you reliably identify a budget meeting the 90% decisiveness target before observing the target census?

What methodological advance does your protocol provide beyond classical finite-population hypothesis testing, power analysis, and explicit acquisition-cost accounting?

Reviewer_VJ1sOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims at proposing an approach to audit LLMs in a way to be able to assess when reducing audit computation still permits statistical validity of this audit conclusions. Related work is poorly (to not at all) integrated in the paper. It contains formal modeling, experiments, and a brief discussion.

Critical strengths

The research question is important and practical: how to have valid audits under fixed budget constraints? (While I am not able to really assess novelty due to the lack of proper comparisons)

Critical weaknesses

This paper is poorly written in my opinion, for the current writing is the mere exposure of the approach (statistically qualified decision about a fixed policy's accepted-answer risk and coverage) alone, without a real placement of its core positioning regarding the related work. This translates into a purely technical piling of elements all in the sense of the proposal, with zero criticism of (supposedly failing) other competing audit approaches. Having a clear description of why these fail, and even some implemented as baselines in the experiment, would help the reader no to only face a "block"-like presentation of the paper's claim. In other words, I am not confident in the current approach, as I do not see any counterfactual-based presentation of problems with non "validity-aware" forms of audits. This can be observed in the abstract and introduction, where after just several lines we dive deep into the chosen technique with no proper scientific balance in its positioning. The idea might be a good one, but the current writing makes it impossible for me to assess properly. The choice of the 2 LLMs and the 2 datasets experimented with are arbitrary picks with no justification, so no real possibility of generalizing the claims to other LLMs and datasets.

Questions for authors

In the same experimental framework, how would you include the closest possible audit forms/competitors, to showcase the current method's superiority?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.