What a Reported Pooled AUC Identifies When Scores Are Unavailable
Abstract
A pooled AUC over a stratified evaluation is often interpreted as evidence that a score discriminates well within strata. In general, however, the pooled AUC does not determine within-stratum discrimination. When per-example scores are unavailable, a reader may observe only the reported AUC, the outcome labels, and the stratum memberships. We study exactly what this information identifies, separating two distinct questions. First, what reference value should the reported AUC be compared against? We define its conditional permutation mean: the expected pooled AUC when labels are permuted within strata while scores remain fixed. This is not a new metric, but a reference value induced by the unreleased scores themselves, and it need not equal 0.5. From annotations alone, we derive sharp extrema for this quantity, attained by rankings that place strata in contiguous blocks. Conditioning additionally on an exact reported AUC yields tighter sharp bounds together with an explicit attaining construction. Second, what does the reported pooled AUC imply about within-stratum discrimination? We characterise exactly when it is compatible with within-stratum AUCs of 0 or 1. On PRM800K, the reported AUC substantially narrows the annotation-only range and can exclude 0.5 as the conditional permutation mean. Yet none of the seven AUCs we recompute from the released data rules out a within-problem AUC below 0.5. On ProcessBench, interpreting the published figure as a standard AUC, scores matching 0.765 at the reported precision attain within-solution AUCs of both 0 and 1. These results separate what a pooled AUC identifies from what it leaves undetermined and specify the per-stratum statistics sufficient to identify each quantity.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.