- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an interesting task on how can two labs produce identical split-conformal prediction sets, using a fixed prediction model but independent calibration samples, while requring each to retain marginal coverage conditional on the shared information. The authors develop a upward-rounding method in coverage units and two public-pilot sample constructions. The theoretical analysis shows how a shared public sample can help independent labs produce identical prediction sets while preserving coverage guarantees. Experiments on ImageNet and CIFAR-100 show that the methods make independently calibrated predictions more consistent, at the cost of larger prediction sets. Language-model and answer-cache experiments are also conducted for additional demonstrations.
Critical strengths
-
The proposed methods are simple and address an interesting problem. The method and theory guarantee that each side retains its coverage guarantee even if the shared public sample is unrepresentative, provided it is independent of the local data. This makes the method solid when consistent predictions are required but a common calibration dataset cannot be adopted.
-
The paper demonstrates a clear theoretical explanation of how greater agreement comes at the cost of larger prediction sets, and how a shared public sample can reduce such cost. The analysis provides practical guidance for choosing parameters. Experiments on various types of data source have verified the theoretical part.
Critical weaknesses
The practical importance of a precise agreement is not significantly motivated in this paper. Independently calibrated models may already produce similar predictions, so it remains unclear when making them exactly identical is worth the cost of larger prediction sets. The paper would benefit from stronger motivation of how this objective addresses a meaningful need in its related applications.
Questions for authors
- What practical settings makes an exact agreement necessary compared to similar predictions?
- What evidence may show that the system benefits outweigh the cost by larger prediction sets?
2 comments
Thank you for the comprehensive response. All of my concerns have been satisfactorily addressed, and I have no further questions.
Response to Reviewer vhc6
Thank you for the constructive feedback. We agree that the paper should state the need for exact agreement more concretely. The revised PDF does so, with changes in blue. As Reviewer Zf2R asked, we now write agent, threshold, agreement and set-size inflation for analyst, cutoff, identity and premium.
Q1. Which settings need exact agreement? Exact agreement is needed when a second party must reproduce the set served for an individual input. Similar thresholds do not reproduce that set. On ImageNet with APS, two independent standard calibrations give different sets on 31.9% of test inputs. Our settings also require each party to keep its coverage from its own calibration data (Section 1). The introduction now states two such settings.
(1) An auditor must confirm the set that a vendor served for a given input and reruns the calibration on its own sample. Under our protocol, two agents share one fixed score and an independent pilot (a public labeled sample) from one population, and each draws its own calibration sample. They deploy the same threshold with probability at least 1 − ρ (Proposition 1, Theorem 3). Equality with the auditor's threshold is therefore a reproducibility check of the vendor-reported threshold, and a compliant vendor fails this check with probability at most ρ, on average over the pilot and both calibration samples.
(2) Two sites serve the same input. When the service must reproduce the second site's prescribed answer set, reuse of a stored set requires the two sets to coincide (Table S17).
Are independent calibrations already similar? Table S24 is new. It gives the share of test inputs on which two agents deploy different sets, with the set-size inflation in parentheses (ImageNet, n = 15,000, n_p = 2,000, the 1,000 agent pairs of Table 2). The grid and the floor are the two seedless rules.
With APS, both rules lower the share from 31.9% to at most 1.6% at ρ = .1, and the observed share at ρ = .01 is 0.0% over 20 pilot redraws. With RAPS the sets are small, and standard calibrations give the same set on 92.5% of inputs. There the pilot floor lowers the share of differing sets from 7.5% to 0.8% at ρ = .1, and both rules lower it to at most 0.1% at ρ = .01.
Q2. Do the benefits outweigh the larger sets? The answer depends on the deployment, and the paper reports both sides of the trade.
(a) The paper bounds the cost. With a pilot, the cap on over-coverage is at most of order √log(1/ρ) (n^(−1/2) + n_p^(−1/2)) (Corollary 2). With a seed only, no threshold rule avoids a cap of order 1/ρ in the worst case (Theorem 4, Corollary 1).
(b) On ImageNet, the gains in the table cost 5% to 24% in set size. The centred grid at ρ = .01 raises the mean set from 5.5 to about 6.1 labels with RAPS and from 34.6 to about 43 with APS (Tables 2, S3).
(c) The pilot floor costs the same as the pilot-only threshold (+12%, APS, ρ = .1). Under a shifted pilot the pilot floor keeps coverage above .901, where the pilot-only threshold falls to .855 (Table S6).
(d) On CIFAR-100, with six times less calibration data (n = 2,460), the two seedless rules cost +20% to +102% for ρ from .1 to .01 (Table S4).
(e) In the answer cache (Table S17, in-pool, one site partition, α = .48), seeded rounding lowers the share of differing answer sets from an observed 39% to an offset-averaged bound of 5.2%. Stored answers per question rise from 4.4 to 8.4.
Changes to the paper. The introduction states settings (1) and (2). Section 7 reports the shares above, the costs on CIFAR-100 and the label cost of the pilot.
If further discussion would be helpful, we will be happy to provide additional evidence during the discussion period.