- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how much evaluation is sufficient for LLM systems in high-risk financial applications. It formulates LLM evaluation as a sequential stopping problem over a finite, stratified evaluation pool and proposes Risk-Aware Adaptive Evaluation (RAAE), a method that combines uncertainty-driven allocation with pre-declared minimum coverage requirements for critical risk strata. The method uses without-replacement confidence sequences and a union-bound construction to provide an anytime-valid guarantee for the population-weighted mean score estimate at stopping time. Empirically, the authors construct a 267-item, 8-stratum financial evaluation suite, evaluate three OpenAI mini-class candidate models with two OpenAI mini-class judges, and compare RAAE against exhaustive, uniform, stratified, and uncertainty-only baselines through resampling from one exhaustive judged pass. The empirical results are nuanced: model ranking fidelity is already saturated near the adaptive methods' natural stopping point; RAAE stops only after evaluating about 90% of the pool, yielding roughly 10% savings; its early-budget critical-failure recall advantage over naive sampling is small and sensitive to the judge-defined labels; and its most robust contribution is enforcing a pre-declared critical-stratum coverage floor rather than consistently improving recall over uncertainty-only sampling.
Critical strengths
-
The paper addresses a practical limitation of aggregate LLM evaluation: a precise estimate of average performance does not ensure adequate coverage of high-risk categories. Incorporating pre-declared critical-stratum coverage requirements into the stopping rule provides a useful formulation for settings with asymmetric failure costs.
-
The statistical construction accounts for adaptive allocation across finite strata sampled without replacement. The paper clearly distinguishes its anytime-valid guarantee on population-weighted mean scores from the deterministic coverage requirement and the empirically measured critical-failure recall.
-
The empirical analysis includes matched-budget comparisons, coverage-threshold ablations, and judge-specific sensitivity analyses. These experiments help separate the benefit of enforcing coverage from improvements in sampling efficiency. The authors report unfavorable results and support the judge-disagreement analysis with a blinded human check, while acknowledging its limited scope.
Critical weaknesses
-
The practical advantage of RAAE over simpler baselines is limited. At its natural stopping point, RAAE evaluates about 90% of the pool, so the realized cost saving is modest. At the matched budget of 240/267, RAAE does not improve critical-failure recall over uniform, stratified, or uncertainty-only sampling. Its budget-90 advantage over uniform under the primary OR-combined labels is statistically significant but small, corresponding to roughly one additional critical item out of 52 on average, and uncertainty-only sampling performs comparably. Under the gpt-5-mini-only sensitivity analysis, this early-budget advantage largely disappears. Thus the evidence supports RAAE more as an enforceable coverage mechanism than as a generally superior adaptive sampling strategy.
-
The empirical conclusions depend heavily on judge-defined labels and a limited evaluation setting. All candidate and judge models are OpenAI mini-class models; two judges also appear as candidates; self-judgments are included in the OR-combined critical-failure ground truth; and gpt-4o-mini alone flags the same 52 critical-failure cases as the OR-combined pool while gpt-5-mini alone flags only 23, even though the single-rater human disagreement check mostly sides with gpt-5-mini. The experiment is also an offline resampling study from one exhaustive judged pass on a 267-item author-validated dataset. These design choices do not invalidate the paper, but they make the recall and judge-reliability conclusions difficult to generalize without independent judges, stronger human adjudication, and larger or externally sourced evaluation pools.
Questions for authors
-
How do the main recall, stopping-budget, and ranking results change if self-judgments are excluded from the OR-combined critical-failure ground truth?
-
How sensitive are the conclusions to the label aggregation rule? For example, what happens under consensus, majority, or human-adjudicated labels rather than OR-combination?
-
Can the authors add a non-OpenAI or otherwise more independent judge and report whether RAAE's early-budget advantage survives under that judge or judge ensemble?
-
Can the judge-reliability analysis be strengthened with multiple independent human raters and a sample that is not concentrated on judge-disagreement cases?
-
In practical deployment, how should users choose the risk taxonomy and coverage thresholds τ_s, and how should RAAE handle important risk categories that are unknown or emerge after the taxonomy is fixed?