- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies, in two-class soft classification, the relationship between the AUC for a complete test set (called pooled AUC) and the AUC for a subset of it (called within-stratum AUC). By AUC, one should understand the area under the ROC (Receiver Operating Chatracteristic) curve obtained by thresholding the score predicted by the classifier. The pooled AUC is reported; the within-stratum AUCs are unknown. The relationship between them is studied through two questions. (1) What reference value should the pooled AUC be compared against? (2) What does the pooled AUC imply about within-stratum AUCs? The dataset PRM800K (« 800,000 step-level correctness labels on LLM solutions to MATH problems ») is taken as usecase.
Critical strengths
It is often necessary to obtain aggregate values for performance scores when classifiers have been evaluated on different sets of samples. The pooling discussed in this paper is one possible approach. It is therefore important to understand its implications. Aside from stratified datasets, I can think of a few other situations where this occurs: (1) in cross-validation; (2) in computer vision, where classifications can be performed at the pixel level, grouped into images, which are themselves grouped into videos, which are themselves grouped into datasets; (3) in domain adaptation, when we are interested in the performance of classifiers evaluated across multiple domains.
- Forman, G., & Scholz, M. (2010). Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM Sigkdd Explorations Newsletter, 12(1), 49-57.
- Piérard, S., & Van Droogenbroeck, M. (2020). Summarizing the performances of a background subtraction algorithm measured on several videos. In ICIP, pp. 3234-3238. IEEE.
Critical weaknesses
Presentation
-
[W1] Reading this paper requires the reader to make assumptions.
- (1) The word « classification » does not appear anywhere, even though the entire paper focuses specifically on soft classification in the very particular case of two classes. This should be stated explicitly from the very beginning.
- (2) The authors must specify what the acronym AUC stands for. Readers in our community will naturally assume it refers to « Area Under Curve ». But the area under which curve? In two-class classification, there are several (ROC curve, PR curve, precision-recall-gain curve, etc.). By analyzing the introductory example, one can conclude that the authors are referring to the area under the ROC curve, but this requires some effort from the reader.
-
[W2] The notations are not consistent. For exemple, the global AUC is sometimes denoted by (as in l. 120) and sometimes as (as in l. 131).
-
[W3] Missing references to the works in which the 7 classifiers (scoring rule) listed in table 1 have been introduced.
Informativeness
-
[W4] The introductory example (l. 42-48) illustrates in fact a well-known statistical paradox described by K. Pearson in 1899, G. U. Yule in 1903 and E. Simpson in 1951, here in the specific case of the AUC. It is therefore necessary to cite the literature on the Yule-Simpson effect. The authors are familiar with this work, as they cite Simpson in suppl. mat. on page 18. However, the authors chose to completely ignore these works in the paper.
-
[W5] From my perspective, in its current form, this paper offers little new insight to readers who are classification professionals. It is clear that a central trend in performance (whether that trend is obtained through averaging or pooling) cannot under any circumstance serve as a bound (i.e., as the worst- or best-case scenario). In classification, if we look at the smallest possible scale (i.e., that of the sample), we nearly always have a non-informative classification. The random variable representing the ground truth becomes nearly always deterministic, as does the one representing the prediction. Therefore, in the information theory perspective there is null mutual information between them: they are independent at that scale. However, anyone who has ever worked with classifiers knows that on datasets with multiple samples, it is possible to obtain non-zero mutual information between the ground truth and the prediction! It is therefore not surprising at all that a classifier can be non-informative on a partition but informative on the overall dataset. This implies that it is not surprising at all that a classifier can have an AUC = 0.5 on all subsets and an AUC ≠ 0.5 on the overall dataset.
-
[W6] Does that mean we need a point of comparison other than 0.5 for the overall AUC? No, unless the classifier has information about the per-sample stratum. If the classifier knows the stratum and the positive prior for that stratum, it can adapt to it and, overall, perform better than a non-informative (random) classifier. To the best of my knowledge, such situations are exceptional. Is it the case when one evaluates mathematical proofs by steps (with PRM800K)?
-
[W7] The practical value of the reported results is unclear. It is not specified if some code will be publicly available to compute the answers to Q1 and Q2. In the absence of publicly relesed code, the paper does not present any practical algorithm for these computations. Also, the computational complexity is not specified. The practical limits are not clear: is it possible to compute the two answers in a reasonable amount of time in all situations? Is it much less costly to do the proposed computations instead of performing a new evaluation of the classifier to obtain the missing per-stratum AUCs?
Soundness
-
[W8] Knowing how to calculate is one thing; knowing how to use it is another. On lines 178 and 181, it is mentioned that this value is simply subtracted from the reported overall AUC. Such a correction for chance is clearly inappropriate as the result is not interpretable. If the overall AUC is 1, then we have a perfect classifier and want to retain the maximum score. Furthermore, if the overall AUC is 0, then we have a perfect classifier if we reverse the decisions, and we want to retain that information. There is a need to study a correction for chance applied to the overall AUC such that 0 → 0, nu → ½, and 1 → 1.
-
[W9] The paper fixates on the value 0.5. But what does it mean? The paper doesn’t say. Let’s assume the authors have in mind the case of non-informative classifiers (sometimes called random or no-skill classifiers). In the very specific case where the ROC curve is convex, an AUC of 0.5 implies that the curve is the ROC diagonal, which corresponds to a non-informative classifier (zero mutual information between the ground truth and the prediction), which is what can be obtained by chance when the axis of ROC (FPR and TPR) are probabilities. This highlights two fundamental problems. First, there is no assumption of convex curves anywhere in the paper. An AUC value of 0.5 therefore does not indicate that we have a non-informative classifier. Second, when there are few samples, the FPR and TPR values are proportions, not probabilities. Any point on the ROC space can be obtained with a classifier that behaves purely randomly. For these reasons, the value of 0.5 is meaningless as a point of comparison for the AUC.
Questions for authors
Questions
[Q1] Can you explain why, if one is interested by some given strata only, she or he would not evaluate the classifier on these strata but rather only consider an AUC computed on all strata ? Please give a concrete example.
[Q2] Can you please provide curves showing how the computation time evolves depending on the dataset size, for different numbers of strata? Can you also explain why the factorial (see line 238) is not a problem in practice ?
[Q3] Figure 1 shows a zonotope. A zonotope is typically a linear projection of some hyper-cube. The linearity of both axes comes from the linearity of and with respect to (see eq. 3). But the possible vectors for do not cover a hypercube. So can you explain why we have a zonotope in Figure 1?
[Q4] Can you explain in simple words why the minimum value is not achievable when and why the maximum value is not achievable when ?
Suggestion
Is the introductory example (l42-48) well chosen? It is a very specific case. If we swap the labels in the two strata, nu remains constant. This example therefore does not help the reader to understand why can fall within a wide range.
2 comments
I keep my opinion that the paper offers little new insight to readers who are classification professionals. Moreover, it remains unclear what one can do with the value of once it has been computed, and how the interpretation we give to the AUC should be corrected (for chance) based on this value. Also, the paper still requires substantial reading efforts in places. Therefore, I maintain my rating.
Thank you for the detailed review. Changes are in the revised PDF.
W1-W3. The first sentence now states the setting (two-class soft classification) and that AUC is the area under the ROC curve. Sec. 5.1 cites the source of each of the seven rules (Lightman et al., 2024; Wang et al., 2024; the last-step, mean and first-step rules are our aggregations). The pooled AUC is now A_pooled throughout.
W4. Agreed. The introduction now names the example as the Yule-Simpson effect (Pearson et al., 1899; Yule, 1903; Simpson, 1951), and Related Work cites the two suggested papers.
W5. We agree this is not surprising, and do not claim it. We also do not use a pooled figure as a bound. We ask how far the printed figure and per-stratum counts constrain ν and the within-stratum AUC, and the answer differs by design: on PRM800K, 0.941 confines ν to [0.562, 0.908]; on ProcessBench, 0.765 leaves ν in [0.434, 0.901] and allows within-solution AUC 0 and 1.
W6. Yes, and it is typical: an LLM verifier reads the problem, so it has the stratum information by construction. Measured ν is 0.607 to 0.797 on PRM800K and 0.636 to 0.735 on Math-Shepherd. Within-problem discrimination is also real on PRM800K (within-problem AUC 0.884 for the ORM); the point is that the printed 0.941 alone does not establish it.
W7, Q2. The algorithm (Sec. 4, Appendix A) evaluates a dual by one O(n log n) sort over a fixed number of search iterations, so the computation is O(n log n). New Table 4 gives run times (as a table, not curves): one endpoint takes 0.1 to 0.3 s at n = 10^5 and 1.2 to 2.6 s at n = 8 x 10^5, about twice as long with 10^4 strata as with 10. The factorial appears only in exhaustive checks on at most eight items. The code is in the supplementary archive and will be released publicly with the paper. A new evaluation needs the scores, which a reader of a published figure does not have.
W8. We do not propose A_pooled minus ν, or the suggested rescaling, as a corrected measure; Sec. 3 now says so. Neither recovers within-stratum discrimination: in the eight-item design (Sec. 4), A_pooled = 0.5625 and ν = 0.5, so any such rescaling gives more than 1/2, yet A_within = 1/3.
W9. Sec. 3 now states what 0.5 means here: the permutation mean of an AUC inside one stratum, and the value a pooled AUC is commonly read against. We assume no convex ROC curve and never equate 0.5 with a non-informative classifier. ν is an exact finite-sample permutation expectation, so nothing treats FPR and TPR as probabilities.
Q1. Lee et al. report 0.765 on ProcessBench MATH. Their step scorer is used to find the first wrong step inside a solution, so a reader needs the within-solution AUC. But their code pools all steps into one AUC, and the scores are not released, so the reader cannot evaluate per solution.
Q3. The hull of the points in Figure 1 is a zonotope because the permutohedron, whose vertices are the rank vectors, is itself a zonotope (the Minkowski sum of the segments [e_i, e_j]), and a linear image of a zonotope is a zonotope.
Q4. The largest ν needs strata in contiguous blocks ordered by share of positives (Proposition 1). A_pooled = 1 needs every positive above every negative. With mixed strata the two conflict; by symmetry the same holds for the minimum at 0. In the eight-item design, ν reaches its maximum 0.75 at every pooled AUC from 9/16 to 15/16, but at A_pooled = 1 its maximum is 0.71875 (checked over all 8! orderings).
Suggestion. Agreed. The introduction now adds the mirror case: with the same labels, scoring the hard problem above the easy one gives ν = 0.10 instead of 0.90.
Ethics concern 1 (doubtful results). The sentence that no check certifies how labels and strata were read discloses a limitation; it does not indicate an error. We have now checked our parsing against published counts. ProcessBench MATH: 594 erroneous and 406 correct solutions, matching the benchmark paper's Table 2. PRM800K: 500 problems with at most 1,860 samples each, as stated in the release's README. The README gives no total, so our total of 815,632 solutions is checked only by rebuilding it from the release, which reproduces it exactly. The scripts are in the revised supplementary archive.
Ethics concern 2 (AI assistance). Our wording caused a misunderstanding. The AI Use Statement already disclosed that the scripts, including verify_artifact.py, were written with AI assistance; "independent of that assistance" was meant to say that the exhaustive enumerations were implemented separately from the solver, not that a human wrote them. The proofs were checked by one of the authors; the scripts were checked by running them and comparing their outputs with the released data. The revised AI Use Statement says this explicitly.