- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper discusses that when only bag-level (aggregate) label means are released, the AUC of a fixed scorer is only partially identified. The paper defines the sharp AUC width: the range of AUC values consistent with exact bag means. It treats the bag partition as a design choice made before feedback is released. The experimental results show that contiguous bags give width 2.4×10⁻⁴ versus 0.992 for random bags on Criteo. On small blocks, the search reduces the AUC range by 60–88% as shown in Table 3.
Critical strengths
- Considering bag design for evaluating a fixed model is interesting as it matters how to rank the performance of a model, as if simply change the evaluation method could flip model rank, that could confuse the partitioners.
Critical weaknesses
- Aggregate labels are ususally released for privacy protection such that the individal information won't be revealed; but there is no clear discussion how the different bag method could affect the privacy leakage. e.g., label-inference risk is not evaluated.
- To design the bag, the curator needs to know the evaluated model's scores; however, for a fair evaluation, isn't these two parts should be independent?
Questions for authors
-
How does the width design trade off against label leakage? Can you measure it along with the widith shrinkage together on Criteo and Adult? This is the main concern I had, as naturally, by reducing the number of samples in a group, by probing multiple times, the true label information could be leaked.
-
In table 3, why the adult with 24 has way higher upper 95% confident interval?
Response to Reviewer PAEh
Thank you for your careful review. We have added matched comparisons on Criteo and Adult showing how bag capacity controls the tradeoff between AUC width and pure-bag disclosure. With one capacity-256 release, adjacent-rank widths are 2.4 × 10⁻⁴ and 7.1 × 10⁻³, with fewer than 1% of rows in pure bags in both datasets. A two-release test quantifies the effect of repeated probing. We also explain why score-aware grouping preserves interval validity and why the Adult-24 confidence limit is high.
W1/Q1: How does narrower AUC uncertainty trade off against label disclosure?
On Criteo and Adult, adjacent-rank grouping trades narrower AUC intervals for greater pure-bag disclosure. We compare adjacent ranks, round-robin and ten balanced-random partitions using the same rows and count budget at capacities 32, 64, 128 and 256. Each design receives one count vector, allowing evaluation information and disclosure to be measured jointly.
The disclosure measure follows from the released information: a row's binary label is fixed by counts and membership if and only if its bag is pure. In a mixed bag, swapping a positive and a negative preserves all counts. The matched results at capacity 256 are:
As the table shows, the large AUC-width gains coincide with greater pure-bag disclosure. Increasing adjacent-rank capacity from 32 to 256 reduces exposure from 41.48% to 0.93% on Criteo and from 14.48% to 0.99% on Adult, at the cost of wider intervals. Bag capacity therefore provides an explicit evaluation–disclosure tradeoff.
A complementary guessing measure shows why determined labels matter. At capacity 256, adjacent ranks give a 4.99 percentage-point advantage over global-majority guessing on Adult. On Criteo, every design stays at the 96.75% global baseline despite differences in determined labels. We also measure the base-2 log-count of compatible joint labelings per row across all capacities.
We also tested repeated probing: a capacity-256 adjacent partition and a second partition shifted by 128 score ranks, with wraparound. Both rules are fixed independently of labels. From their joint counts and memberships, 6.59% of Criteo labels and 2.97% of Adult labels are determined, versus 0.93% and 0.99% after the first release. The sharp AUC widths become 1.19 × 10⁻⁴ and 3.54 × 10⁻³. Both widths and disclosure rates match a single capacity-128 release at the same total count budget. We compute joint extrema exactly from the intersection-cell count constraints. This comparison quantifies how overlapping releases combine information.
W2: Does using model scores compromise evaluation fairness?
Score-aware grouping changes evaluation precision while preserving the target and validity for a fixed scorer. We fix the scorer and design rule before feedback and match rows and count budgets. Scores organize known ranks, while released counts constrain unknown labels. Thus every resulting sharp interval contains the same realised-label AUC.
For the population target, score-aware design also retains coverage under fixed scores/states and conditionally independent Bernoulli labels. The concentration event holds simultaneously across partitions of those fixed rows. To test a practical pre-feedback protocol, we additionally freeze prediction-based and training-state designs before reading evaluation-label-derived probabilities.
Q2: Why is the Adult-24 upper confidence limit in Table 3 unusually high?
The high upper bootstrap limit comes from two of ten blocks attaining 100% width reduction. The median reduction is 88.35%, with a 95% percentile interval of [84.2%, 94.9%]. Specifically, we first compute (risk width − search width)/risk width within each paired block, then take the median. This interval describes variation in width reduction across blocks, whereas the AUC identification interval describes compatible AUC values.