acceptodds
Under review as a conference paper at ICLR 2027

Replicable Split-Conformal Calibration in Coverage Units

Abstract

Two laboratories that calibrate the same prediction model on their own data deploy different prediction sets. The reason is that the calibrated cutoff is computed from continuous scores, so two calibrations never produce the same cutoff, even when both are correct on average. As a result, predictions stored by one laboratory cannot be reused by another, and an auditor cannot reproduce a vendor's prediction sets. We introduce replicable split-conformal calibration. It rounds each calibrated cutoff up to the nearest of a few values that both laboratories agree on in advance, so that both deploy identical prediction sets with a probability they choose, while each keeps its own coverage guarantee. The cost is larger prediction sets. How large depends on what the laboratories share to choose these values, and we prove three results about it. First, without shared information, identical prediction sets are impossible. Second, a shared random seed makes them possible, but at a cost that no seed-based threshold rule can avoid in the worst case. Third, a small public sample from the same population removes the need for a seed, lowers the cost, and gives guarantees that hold for every continuous score distribution. We implement our method on ImageNet, CIFAR-100 and four language models, GPT-2, Pythia, Qwen and Llama, and confirm the theory. On ImageNet, the chance of identical prediction sets rises from zero to above ninety percent, for prediction sets about a fifth larger. On CIFAR-100, with six times less calibration data, the same pattern holds at a higher cost. On the language models, a shared seed reaches the chosen probability of identical prediction sets in all 42 settings, and its cost falls as the calibration data grow. In a question-answering service built on Llama, two laboratories whose stored predictions differ on more than a third of the questions under standard calibration end up with identical stored predictions ninety-five percent of the time, at the cost of storing larger prediction sets.

Primary area:
learning theory
Keywords:
conformal prediction, replicability, calibration, uncertainty quantification, reproducibility, distribution-free inference, learning theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vhc6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an interesting task on how can two labs produce identical split-conformal prediction sets, using a fixed prediction model but independent calibration samples, while requring each to retain marginal coverage conditional on the shared information. The authors develop a upward-rounding method in coverage units and two public-pilot sample constructions. The theoretical analysis shows how a shared public sample can help independent labs produce identical prediction sets while preserving coverage guarantees. Experiments on ImageNet and CIFAR-100 show that the methods make independently calibrated predictions more consistent, at the cost of larger prediction sets. Language-model and answer-cache experiments are also conducted for additional demonstrations.

Critical strengths
  1. The proposed methods are simple and address an interesting problem. The method and theory guarantee that each side retains its coverage guarantee even if the shared public sample is unrepresentative, provided it is independent of the local data. This makes the method solid when consistent predictions are required but a common calibration dataset cannot be adopted.

  2. The paper demonstrates a clear theoretical explanation of how greater agreement comes at the cost of larger prediction sets, and how a shared public sample can reduce such cost. The analysis provides practical guidance for choosing parameters. Experiments on various types of data source have verified the theoretical part.

Critical weaknesses

The practical importance of a precise agreement is not significantly motivated in this paper. Independently calibrated models may already produce similar predictions, so it remains unclear when making them exactly identical is worth the cost of larger prediction sets. The paper would benefit from stronger motivation of how this objective addresses a meaningful need in its related applications.

Questions for authors
  1. What practical settings makes an exact agreement necessary compared to similar predictions?
  2. What evidence may show that the system benefits outweigh the cost by larger prediction sets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vhc6

Thank you for the constructive feedback. We agree that the paper should state the need for exact agreement more concretely. The revised PDF does so, with changes in blue. As Reviewer Zf2R asked, we now write agent, threshold, agreement and set-size inflation for analyst, cutoff, identity and premium.

Q1. Which settings need exact agreement? Exact agreement is needed when a second party must reproduce the set served for an individual input. Similar thresholds do not reproduce that set. On ImageNet with APS, two independent standard calibrations give different sets on 31.9% of test inputs. Our settings also require each party to keep its coverage from its own calibration data (Section 1). The introduction now states two such settings.

(1) An auditor must confirm the set that a vendor served for a given input and reruns the calibration on its own sample. Under our protocol, two agents share one fixed score and an independent pilot (a public labeled sample) from one population, and each draws its own calibration sample. They deploy the same threshold with probability at least 1 − ρ (Proposition 1, Theorem 3). Equality with the auditor's threshold is therefore a reproducibility check of the vendor-reported threshold, and a compliant vendor fails this check with probability at most ρ, on average over the pilot and both calibration samples.

(2) Two sites serve the same input. When the service must reproduce the second site's prescribed answer set, reuse of a stored set requires the two sets to coincide (Table S17).

Are independent calibrations already similar? Table S24 is new. It gives the share of test inputs on which two agents deploy different sets, with the set-size inflation in parentheses (ImageNet, n = 15,000, n_p = 2,000, the 1,000 agent pairs of Table 2). The grid and the floor are the two seedless rules.

Score (size)StandardGrid, ρ = .1Floor, ρ = .1Grid, ρ = .01Floor, ρ = .01
APS (34.6 labels)31.9% ± 1.51.3% (+15%)1.6% (+12%)0.0% (+24%)0.0% (+21%)
RAPS (5.5 labels)7.5% ± 1.16.0% (+6%)0.8% (+5%)0.0% (+10%)0.1% (+8%)

With APS, both rules lower the share from 31.9% to at most 1.6% at ρ = .1, and the observed share at ρ = .01 is 0.0% over 20 pilot redraws. With RAPS the sets are small, and standard calibrations give the same set on 92.5% of inputs. There the pilot floor lowers the share of differing sets from 7.5% to 0.8% at ρ = .1, and both rules lower it to at most 0.1% at ρ = .01.

Q2. Do the benefits outweigh the larger sets? The answer depends on the deployment, and the paper reports both sides of the trade.

(a) The paper bounds the cost. With a pilot, the cap on over-coverage is at most of order √log(1/ρ) (n^(−1/2) + n_p^(−1/2)) (Corollary 2). With a seed only, no threshold rule avoids a cap of order 1/ρ in the worst case (Theorem 4, Corollary 1).

(b) On ImageNet, the gains in the table cost 5% to 24% in set size. The centred grid at ρ = .01 raises the mean set from 5.5 to about 6.1 labels with RAPS and from 34.6 to about 43 with APS (Tables 2, S3).

(c) The pilot floor costs the same as the pilot-only threshold (+12%, APS, ρ = .1). Under a shifted pilot the pilot floor keeps coverage above .901, where the pilot-only threshold falls to .855 (Table S6).

(d) On CIFAR-100, with six times less calibration data (n = 2,460), the two seedless rules cost +20% to +102% for ρ from .1 to .01 (Table S4).

(e) In the answer cache (Table S17, in-pool, one site partition, α = .48), seeded rounding lowers the share of differing answer sets from an observed 39% to an offset-averaged bound of 5.2%. Stored answers per question rise from 4.4 to 8.4.

Changes to the paper. The introduction states settings (1) and (2). Section 7 reports the shares above, the costs on CIFAR-100 and the label cost of the pilot.

If further discussion would be helpful, we will be happy to provide additional evidence during the discussion period.

Thank you for the comprehensive response. All of my concerns have been satisfactorily addressed, and I have no further questions.

Reviewer_isAEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes replicable split-conformal calibration. The authors observe that two sites calibrating the same model on separate datasets usually get different conformal thresholds, so they produce different prediction sets. The proposed method rounds the threshold upward onto a shared grid, and thus preserves coverage. When two thresholds land in the same grid cell, the sites deploy the same prediction-set map. This paper demonstrates that replication is impossible under the paper’s two-sided validity requirement when the sites share nothing, and seed-based rounding can be costly in the worst case. In contrast, a representative public pilot can provide distribution-free identity guarantees with lower over-coverage. The experiments are conducted on ImageNet, CIFAR-100, language models, and an answer cache. The empirical results show the effectiveness of the proposed method.

Critical strengths
  1. This paper is well-motivated. Different calibration datasets can produce different prediction sets for the same model.
  2. The coverage unit is interesting. The distribution of the conformal cutoff’s coverage does not depend on the score distribution, and thus avoids the need for a density assumption.
Critical weaknesses
  1. The public-pilot assumption may be hard to satisfy. It requires being representative of the analysts’ population and independent of their calibration samples. It seems to be hard to achieve.
  2. The shifted-pilot experiment shows that coverage can remain protected while identity. For example, the centered grid’s identity drops to around 0.73 in some tested shifts.
  3. The paper should quantify the pilot’s acquisition and governance costs alongside the prediction-set-size premium.
  4. The paper limits the lower bounds to scalar-threshold rules with distribution-independent shared randomness, while does not provide the public-pilot lower bound. It may be unclear in practice.
  5. The results on language models are obtained using the uncertified score-unit rule. The distribution-free seedless guarantees are not demonstrated on those model settings.
Questions for authors
  1. How sensitive are the identity guarantees to distribution shift from pilot to deployment?
  2. Is it possible that the public pilot is no longer representative in your settings?
  3. Why are the certified seedless methods not evaluated under the language-model settings?
  4. Can the lower bound be extended to rules that use a public pilot, or to broader classes of conformal prediction maps?
1 comment
AuthorsOpenReview ↗

Response to Reviewer isAE

Thank you for the constructive feedback. Changes are blue in the revised PDF. As Reviewer Zf2R asked, we now write agent, threshold, agreement and set-size inflation for analyst, cutoff, identity and premium. New values come from reruns that reproduce the agreement values of Tables S6 and S11.

W1, Q2 (pilot assumption). The two guarantees need different things from the pilot. Coverage holds for every pilot that is independent of the agent's data, representative or not (Theorems 2(i), 3(i), Proposition 1(i)). Even without independence, τ ≥ τ̃ gives unconditional coverage of at least 1 − α, where τ̃ is the agent's own split-conformal threshold. Agreement needs a pilot from the agents' population, and such a pilot can stop being representative. Tables S6 and S16 test that case. Under shifted pilots every local rule we ran keeps coverage above .901 (ImageNet) and at least .912 (census), while the pilot-only threshold falls to .855 and .893. The conclusion now states the pilot assumption as a limitation.

Section 5 and Appendix F.7 add a pilot diagnostic that each agent runs on its own. The agent counts the pilot scores N at or below τ̃. Under a representative, independent pilot, N has one Beta-binomial law for every continuous score distribution. On representative pilots the 5% test rejected in .006 of pairs on ImageNet and in .025 to .056 on the language models.

W2, Q1 (shifted pilot). We agree and qualified the statement in the introduction. Coverage is protected under a shifted pilot, and agreement is not certified there. Over the eleven pilot compositions of Table S6, the pilot floor has agreement 1.000 at the five negative gaps and .90/.39/.03/0 at gaps of .03/.41/.79 cells and above. The centred grid ranges from .73 to 1.00, and seeded coverage-unit rounding from .869 to .933. Agent A's pilot diagnostic rejects in every pair at the two compositions where the grid has agreement .73 and .75. It rejects in .75 to 1.00 of pairs at the tested gaps of at least .69 cells in absolute value, and in .23 and .27 of pairs at gaps of −.32 and .41 cells. Threshold agreement is fixed at calibration, so a shift at test time changes coverage and leaves threshold agreement unchanged (Appendix F.7).

W3 (pilot cost). We quantify the pilot in labels and in communication. On ImageNet (APS, ρ = .1, Table S5), pilots of 250 to 4,000 points are 0.8% to 11.8% of the 2n + n_p labels. The set-size inflation falls from +155% to +12% for the centred grid and from +40% to +9% for the pilot floor. The agents receive 38 grid thresholds at b = .0265, or one number for the floor, and the pilot diagnostic uses the n_p pilot scores. We did not measure acquisition and governance costs, which depend on the application, and the conclusion says so.

W4, Q4 (lower bounds). Theorem 1 covers arbitrary set-valued procedures that are uniformly two-sided ε-valid. Theorem 4 and Corollary 1 cover the scalar index of a fixed sublevel family, which includes split-conformal methods with a fixed score. Proposition 8 analyzes one full-conformal construction, and refitted or sample-dependent families are outside this scope (Appendix D). Extending the lower bound to rules that use a pilot is open, because the proof of Theorem 4 conditions on a seed whose law does not depend on the distribution. Section 6 and Appendix F.7 now state the scope of each bound.

W5, Q3 (language models). The logits were not archived at submission. We re-extracted them and ran both seedless rules on the four models, with the split of Table S11 and 100 pilot redraws of 20 pairs (Tables 3 and S23). With APS at ρ = .01, the centred grid has agreement .990 to 1.000 (certified ≥ .990) at a set-size inflation of +41% to +64%. The pilot floor has .989 to 1.000 (exact .991) at +35% to +52%. The seeded score-unit rule costs +12,172% to +133,531%. At ρ = .1 the seedless set-size inflation is +20% to +41%. Table S23 prints eight of the 24 settings of model, score and ρ, and the supplement holds the rest. In all 24 settings the centred grid is at or above its certificate or within .3 clustered standard errors of it (at or above in 20). The pilot floor is within 1.9 clustered standard errors of its exact value in 18 settings and above it in six. Five of the six are at ρ = .01, where at most 5 of 2,000 pairs mismatched. Seeded set sizes in Tables 3 and S11 are now integrated over the offset on all 1,000 pairs, which replaces the four-offset estimates of the submission. The largest change at ρ = .1 is Qwen with APS, from +183% to +275% ± 111.

If further discussion would be helpful, we will be happy to provide additional evidence during the discussion period.

Reviewer_Zf2ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The problem is as follows: there are two or more agents in the same environment, each with their own sample of the same distribution, and they're doing conformal prediction on it. What the authors want is to have common information passed between the agents before the calibration, and they want them to reach the same alpha quantile specifically. This means that the prediction sets would be identical over the same input if passed to either of them. The way they do this is by quantizing the space into various levels and rounding up the threshold to that level. They have three steps for this:

  1. Map it in the coverage probability space through an estimator of the CDF of the coverage.
  2. Divide this into bins, map it to the upper bin, and return back to the threshold space. They study a consensus probability over conformal prediction among all these agents. What probability? With what probability do they reach the same threshold, and how can they design the threshold in case you get the least possible set size, basically the smallest threshold that could be a high-probability consensus?
Critical strengths

The first strength I found in the paper is in theory. The authors have designed a specific setup, studied it in detail with the impossibility results and the design choices, and I could not find flaws in the algorithms. I think, up to this point, the theory is quite solid.

I think also the authors are posing an important and interesting problem, not in the case of two agents, but in the case of more than two agents. Even a swarm of agents could be more relevant.
I can see that the hospitals could not share their data, and even in case of calibrating in different timesteps even the scores are not possible to be exchanged. Additionally if the agents are more than some number, even setting aside the privacy issues, the exchange of conformity scores could not potentially be done. I can see that for explainability results and for audits, the fact that the conformal sets are mostly identical is partially important.

I would appreciate it if the authors could extend their framework to more than two agents and, instead of a consensus over all experts, increase the probability of a majority vote or a uniform consensus of k out of n agents.

Critical weaknesses

Specifically for the case of two agents, the amount of information that would be necessary to be exchanged in order to get identical sets or identical quantiles is trivially a single scalar. In that case, specifically, I cannot see any particular use case because the predefined protocol is at a similar cost to exchanging thresholds. Also, combining scores together could result in a more statistically concentrated coverage distribution.

My most major concern about the paper is about the readability. The story is not clearly told.
Specifically, the problem that the authors are pursuing to answer is an optimization problem: to minimize the commonly deployed threshold, subject to the identical maps being above some probability, and even including the stochasticity and everything else. This is an optimization problem that could be stated explicitly. I couldn't find it in the paper, and this took me a while while reading the paper to understand that this is the main object of interest.
Also, there are some vocabulary choices that I didn't find easy to read. For example:
- The cutoff is a new terminology proposed for the term conformal threshold, which is commonly used in the literature.
- Analysts could be replaced with agents.
- Identity is the probability of equal thresholds.
- Premium is another term for set size inflation, which is easier to read.
Each of these minor choices makes the paper really difficult to read.
Also, in many cases, like around line 172 and paragraph 4, there are notations that are not introduced before and are directly being used, like the reverse function from the probability space to the threshold space or the function from the threshold space to the probability space. Both are not clearly described before use.

Questions for authors

Question 1 Can authors give a concrete deployment scenario in which two parties can fix a protocol but not communicate their scalar threshold after calibrating? I can't understand this for more than two agents, specifically if the number of agents is too large or they come at different times. For a specific case of two agents, which is mainly evaluated in the main paper, I cannot find a use case in which such a thing is necessary. Can the authors elaborate a bit more?

Question 2 Can the authors elaborate a bit more from the beginning, specifically discussing the use case of hospitals or laboratories? What I was thinking was that, in these setups, specifically when the draws are evaluated on different sites, there could be minor potential distribution shifts (which could, in real-life use cases, be due to measurement errors or other environmental variables).

How much these rounding setups could help reduce the effect of covariate shift? Specifically, I assume that because you're always rounding up, this added coverage should also be able to tolerate some small amount of distribution shift (which could also be connected to another paper called "Conformal Prediction Beyond Exchangeability").

Question 3 How does the guarantee change while increasing the number of agents? I can't see a table in the appendix reporting the measured identity for different n's (2, 5, and 10), but I cannot find what happens to the set size and overcoverage in those cases. Can the authors report those values as well?

Can you please add the solid definition of functions like GG and Qg+Q^+_g? As I understood, these are the greedy and the reverse greedy function, which is, I guess, called anchoring the paper. Can you use the terminologies in a more clear way?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Zf2R

Thank you for the careful reading. We agree that many agents deserve more emphasis, and we revised the presentation as you suggested. The revised PDF adds the requested analyses, with changes in blue. The text and the figures now use your terms: agents, agreement (identity), threshold (cutoff), set-size inflation (premium).

Q1 and W1 (two agents, one scalar). You are right that two agents can exchange one scalar. Broadcasting one threshold gives agreement 1 with coverage that rests on the sender's data (Table S9). Pooling the scores also gives agreement 1, with more concentrated coverage that rests on the pooled data. Our setting requires that each agent's coverage hold by its own calibration data, conditionally on everything shared (Section 1). One deployment with this requirement is an audit in which the auditor recalibrates on its own sample. If the auditor deployed the vendor's threshold, its coverage statement would rest on the vendor's data. Under the public protocol the auditor calibrates alone and compares one number. With a common score and one population, a compliant vendor fails this check with probability at most ρ, on average over both calibration samples and an independent pilot (a public labeled sample) from that population. Local calibration also protects coverage under a shifted pilot. On ImageNet, a threshold computed from that pilot alone covers as little as .855, while every rule that rounds or floors a local threshold covers above .901 (Table S6). The protocol needs no exchange after calibration, which helps with many agents and with agents that calibrate at different times, as you note.

Q3 (more agents, m of M). Table S25 re-certifies each rule so that all M agents agree with probability at least .9 (n = 15,000 per agent, n_p = 2,000 pilot points, α = .1). The pilot floor deploys the larger of the agent's threshold and one pilot order statistic. The centred grid rounds the agent's threshold up to a grid with a cell centred at 1 − α. Coverage is from fresh draws, inflation from ImageNet (APS).

MFloor: cov.infl.Grid: cov.infl.Seeded, cov. units: cov.infl.
2.9106+12%.9132+15%.9167+22%
5.9118+13%.9142+17%.9307+53%
10.9126+14%.9151+18%.9399+87%
100.9145+17%.9175+21%.9607+751%

From 2 to 100 agents, the two seedless rules add 5 to 6 points of set-size inflation. The seeded width b_M = (E R_M + 1/n_p)/ρ grows with the range R_M of the M coverage levels (Remark 3). With rank and width held fixed, each agent's over-coverage and set size do not depend on M. For the pilot floor and 2 ≤ m ≤ M, Remark 3 gives P(at least m of M deploy the same threshold) = E[P(Bin(M, H_n(V_j)) ≥ m)]. At the two-agent rank (ρ = .1), a majority agrees with probability .944, .940 and .946 for M = 5, 10 and 100, while all M agree with .878, .851 and .760. With 5 and 10 disjoint agents from the ImageNet pool (n = 6,000 and 3,000 each), all agreed in .947 ± .033 and .906 ± .043 of draws for the floor (exact .907, .908). For the grid they agreed in .981 ± .011 and .930 ± .041 (certified ≥ .904, .901).

Q2 (hospitals, laboratories and shift). The introduction now opens with two laboratories that calibrate one approved model. Between sites, each agent calibrates on its own data, so coverage holds on its own population (.917 to .919 on five census states, Table S15). Agreement is certified for one population. For seeded score-unit rounding, agreement stayed at .91 to .93 for state pairs whose quantiles differ by at most .05 grid cells. It fell to .585 to .75 at gaps of .24 to .42 cells (Table S14, n = 20,000, ρ = .1). At deployment, your intuition holds on average. For test scores from F_test and calibration scores from F, mean coverage falls by at most sup|F − F_test|. The average over-coverage is .011 to .017 (floor) and .013 to .020 (grid) for ρ from .1 to .01, so a smaller shift leaves mean coverage at or above 1 − α. With scores N(0,1), test scores N(.05,1) and ρ = .1, standard split conformal covers .891, the floor .902 and the grid .905 (Appendix F.7). We now cite Barber et al. (2023).

Presentation. (i) The introduction states the two questions, and Section 2 states Problem 1. Problem 1 minimizes the cap e on over-coverage subject to agreement, P(τ_A ≠ τ_B) ≤ ρ, validity conditional on what is shared, and P(F(τ) > 1 − α + e) ≤ δ. (ii) Section 4 defines both maps. G is the public map from a score to a coverage level, the pilot's interpolated empirical distribution function. Q_G^+(v) = inf{t : G(t) > v} maps a level back to a threshold. Each agent rounds the level G(t) of its threshold t up to a grid point v, and Q_G^+(v) ≥ t keeps coverage. The anchor is the pilot order statistic of the pilot floor. (iii) Section 4 adds a worked example with two agents in one grid cell.

If further discussion would be helpful, we will be happy to provide additional evidence during the discussion period.

I thanks the authors for the edits they made to the paper, and addressing my concerns. On Q1/W1 I am still unable to imagine the setup sharply, but yes I am convinced at least that such setup would be applicable in case of more than 2 agents. I am very positive about the added experiment on more agents.

Thanks again, I am happy to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.