- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to aggregate predictions on a fixed label hierarchy when up to f sources can be arbitrarily corrupted. It defines safety relative to the majority subtree of the uncorrupted sources and characterizes the finest safe output for label reports. It derives exact specificity losses and worst-case error events, extends the approach to probability reports, and establishes a safety threshold for hierarchical selective classification. Experiments on image classification, cell annotation, and human labels examine the trade-off between adversarial error and prediction specificity, together with the conditions under which statistical risk certificates transfer.
Critical strengths
- S1: The paper gives a precise answer to a useful aggregation question: how much specificity can be retained while respecting the majority information in every possible uncorrupted subset. The contribution extends beyond a supermajority threshold to a characterization of the finest safe output, exact corruption costs, and a probability-report safety boundary. It carefully distinguishes this guarantee from correctness against ground truth.
- S2: The empirical study examines the error-specificity trade-off with substantial controls. The probability-rule comparison uses the same reports and matches clean validation error. Under these matched conditions, the probability rule achieves lower worst-case error than HSC across the evaluated machine-source datasets. Safe-threshold baselines, specificity curves, persistent failures, and negative results help delimit the benefit rather than suggesting universal superiority.
Critical weaknesses
- W1: The operational value of the retained coarse answers remains less established than the formal guarantee. The paper identifies an operating regime and measures specificity, but does not connect that specificity to a downstream decision or task-specific cost of abstention. On Aircraft, the probability rule at f=2 has clean normalized depth 0.17, and 94.2% of examples admit an attack forcing complete abstention. This is an attack-existence statistic, not an observed deployment abstention rate. In PBMC, the ten sources are feature-panel models, so losing one entire modality corrupts five sources and falls outside the regime that preserves unanimous answers. These disclosed limitations make the practical benefit depend strongly on a justified failure budget and the value of a coarse prediction.
- W2: The empirical certification protocol only partly supports direct deployment of the reported risk bounds. The main validation split also selects model hyperparameters or temperatures, while the statistical guarantee assumes a fixed rule and independent certification data. The disjoint-selection check is reassuring for the label rule on the image tasks, but an equivalent check is not reported for the probability rule. Separately, the PBMC label-rule top-level bound is 1.3% while the test worst-case error is 3.6% under donor shift. The paper acknowledges this failure and explains that its warning requires uncorrupted reports and does not repair the certificate. This limits deployment assurance and empirical coverage claims, without refuting the conditional theorem.
Questions for authors
- Q1: Could you work through one of the evaluated applications with a justified corruption budget and a concrete downstream decision or cost model? Showing when a coarse output is useful, when abstention is acceptable, and how the resulting utility compares with HSC would clarify the practical significance of the existing error-specificity results.
- Q2: Could you report the probability rule's certificates using data disjoint from all model and temperature selection, and distinguish these results from certificates affected by validation reuse? This would clarify how much empirical support remains for the deployable guarantee once its sampling conditions are enforced.
2 comments
Thank you for the helpful response. The certificates computed on data disjoint from model and temperature selection address my concern about validation reuse. The worked PBMC cost analysis also clarifies when coarse predictions are useful. These additions strengthen my positive assessment, and I would like to keep my current score, which is already positive.
New: certificates on data disjoint from all model and temperature selection (every all-error bound still holds), and a worked decision-cost analysis on PBMC with a justified corruption budget
Thank you for two precise deployment questions. We ran both analyses; the results are in the revised PDF (changes in blue).
Q2. Certificates on data disjoint from every selection step. For the image datasets we retrained the ten probes with weight decay and temperature chosen on one half of the validation split (stratified) and certified on the other half (2,500 CIFAR-100 images, 1,700 aircraft). The PBMC networks never see the validation donor, whose cells were used only to fit temperatures; we refitted the temperatures on one half and certified on the other (5,017 cells). One-sided 95% Clopper-Pearson bounds at f=2, all errors (top-level errors in parentheses), in %:
Every all-error bound covers the exact test worst case for both rules on all three datasets, and also at f=3. The top-level bounds hold on the image datasets and fail on PBMC exactly as before, because its test cells come from other donors. Reusing the split for selection had moved the bounds by at most 1.0 point. The remaining failure is therefore the i.i.d. assumption under donor shift, not validation reuse; the label-free warning flags it but does not repair it, as the paper says. (New Table 27; Section 5.4.)
Q1. A worked application with a cost model. PBMC cell typing at f=2. Each source reads one gene or antibody panel, so two independent panel failures (for example a failed antibody lot and a mis-pooled gene panel) corrupt two sources. Losing a whole modality corrupts five sources, beyond any safe rule (M≤3f), as Section 6 states and as you note. Cost per cell: a wrong label costs λw, an abstention at the root (the cell goes to manual gating) costs 1, and a correct but coarser label at depth d costs 0.5(1−d/L). Under attack, the adversary picks for every cell the corruption that maximizes its cost (a wrong label if it can, else the root, else the coarsest correct label); this is exact for Q2 and Q2p and a lower bound for HSC, which favours HSC. HSC at θc+ climbs at the safe threshold, just above θc=21+2Mf. Expected cost per cell (Table 28 rounds to two decimals):
When a wrong label is cheap, the more specific calibrated rule is best without an adversary. Under attack, the two safe rules are best as soon as a wrong label costs two abstentions; they differ by at most 0.06 in every column. The attack column is the worst case, with every cell attacked: at λw=10 abstaining on every cell is then cheaper than any rule, which is when abstention is the right call. If a random share of cells is attacked, the safe rules beat abstaining up to a 72% share, against 50% for HSC calibrated to Q2p and 18% for HSC calibrated to Q2. The image datasets show the same pattern (new Tables 28 and 29).
On the 94.2% for FGVC-Aircraft. It is the share of images for which some corruption of two sources can force Q2p to the root, not an observed abstention rate. Without any attack, Q2p already abstains on 76% of the images (the vote rule Q2 on 51%), because the weak aircraft sources disagree at the top level. Safety is expensive there, and the paper says so; on PBMC and CIFAR-100 the clean abstention rates of Q2p are 6% and 23%.
Thank you; these two analyses make the deployment conditions of the guarantee explicit.