acceptodds
Under review as a conference paper at ICLR 2027

Statistical Rank Transport for Efficient Conformal Recalibration After Model Updates

11 views

Abstract

Model updates invalidate cached conformal scores, while full recalibration reevaluates every calibration example with the updated model. We study arbitrary fixed updates through statistical rank transport. A small sample fits a monotone map from cached reference scores to updated scores; an independent residual sample supplies a maximum-residual radius; and a contamination-aware order-statistic argument localizes the updated threshold to a contiguous block that is rescored exactly. The maximum residual induces a distribution-free beta-binomial upper reference for the number of calibration violations, including discrete residual laws, so the contamination allowance is chosen directly from rather than through an intermediate tolerance parameter. The same target- construction also yields a training-conditional guarantee after adding the calibration-order margin required by . On Llama-3-8B backbone LoRA and ViT-B/16 quantization and pruning, the marginal-efficient setting uses roughly 6-18% of full-calibration forwards while stale thresholds materially undercover. Against a matched-budget PAC mini split, the PAC-tuned transport setting reduces the delivered target by roughly 0.3-0.6 percentage points and mean set growth by about 3 percentage points in the evaluated moderate-cache regime. The advantage is not monotone in cache size: at fixed it peaks before boundary-block cost overtakes the statistical benefit, while scaling with the cache preserves the favorable regime over a wider range of .

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, recalibration, model updates, prediction sets, model compression, pruning, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ENqVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at recalibrating a split conformal predictor after a model update (backbone LoRA, quantization, pruning) without rerunning the new model on the whole calibration set. Instead of bounding the change deterministically, the authors fit an isotonic map from cached old scores to new ones on a small sample, and use the max residual ε^\hat{\varepsilon} on a second, independent sample of size rEr_E as an envelope radius. Since this is a sample max, the number of the nn calibration points falling outside the envelope is stochastically dominated by BetaBin⁡(n,1,rE)\operatorname{BetaBin}\left(n, 1, r_E\right), whose (1−δV)\left(1-\delta_V\right)-quantile gives a contamination allowance mm. The new kk-th order statistic is then pinned between the transported scores at ranks k−mk-m and k+mk+m, give or take ε^\hat{\varepsilon}, so only a contiguous block of the sorted cache gets rescored. With probability 1−δV1-\delta_V the threshold is no smaller than the full-recalibration one, giving 1−α−δV1-\alpha-\delta_V marginal coverage, and a training-conditional version just enlarges mm. Experiments on Llama-3-8B and ViT-B/16 show large savings in forward passes over full recalibration with slightly larger sets, though a same-budget mini split remains a competitive baseline.

Critical strengths

the paper's interesting claim is that an arbitrary fixed update, like LoRA inside the transformer blocks, INT4, 30% magnitude pruning, can be turned into a conservative conformal threshold without any deterministic network envelope, for rT+rEr_T+r_E forwards on fresh samples plus a boundary block

Critical weaknesses
  1. The procedure that is implemented is not the procedure that is proved, and in one place this could be a correctness failure, i.e., the fallback breaks the PAC guarantee. Appendix H. 3 says that when rT+rE+∣B∣≥nr_T+r_E+|B| \geq n the system "performs full recalibration" and that this "changes computation but not ... the statistical statement". Full recalibration was defined in Section 2 as returning b(k)b_{(k)}. For the marginal statement that is harmless. For the training-conditional statement it is not: Corollary 5 with the eq (27) allowance needs q~≥b(k+cQ)\tilde{q} \geq b_{\left(k+c_Q\right)} on the good event, and b(k)b_{(k)} discards exactly those cQc_Q ranks. A counterexample inside the paper's own assumptions: S0≡0,S1∼Unif⁡(0,1)S_0 \equiv 0, S_1 \sim \operatorname{Unif}(0,1), every role iid and independent. Isotonic regression on a constant input returns h≡ch \equiv c, so z1=⋯=znz_1=\cdots=z_n, all transported intervals coincide, B={1,…,n}B=\{1, \ldots, n\}, and the fallback fires with probability one. With the MMLU parameters (n=10,000,α=0.1,k=9001)(n=10,000, \alpha=0.1, k=9001),

\begin{equation*} \mathbb{P}\{F\left(b_{(9001)}\right)<0.9\}=\mathbb{P}\{\operatorname{Bin}(10,000,0.9) \geq 9001\} \approx 0.495 \end{equation*}

versus the advertised δV+δQ=0.005\delta_V+\delta_Q=0.005. No hypothesis of Corollary 5 is violated (ties among centres are allowed; the density condition belongs to Corollary 6, not Corollary 5). So the 0.5% training-conditional claim does not hold for the algorithm as written, and the optimized 9.01% / 4.30% bounds for the marginal-efficient route don't survive rank- kk fallback either.

  1. Where the method is supposed to win, the measured gain is small and narrow. For the marginal objective the authors concede mini split is better

  2. the closest existing framework is absent. Semi-supervised risk control via prediction-powered inference (Einbinder, Ringel & Romano) handles exactly this large-proxy / small-gold structure with finite-sample PAC risk control and a binary-loss specialisation; with proxy h(ai)h\left(a_i\right), gold bjb_j and loss 1{B>q}\mathbf{1}\{B>q\} it yields a valid PAC threshold with no block rescoring, and plain PPI already has nonasymptotic quantile inference. Relatedly, the Beta⁡(1,rE)/\operatorname{Beta}\left(1, r_E\right) / beta-binomial facts in Propositions 1-2 are standard calibration-conditional order-statistic results (Hulsman 2022 gives both, ties included), and the betting sequence is Waudby-Smith-Ramdas off the shelf. The only novelty is the specific combination, and the value of that combination is precisely what the missing comparisons would establish

Questions for authors
  1. Which isotonic implementation is used (step function or linear interpolation, out-of-range handling), and how much calibration mass near rank kk lands on plateaus? equivalently, can you give plateau / boundary-mass diagnostics so one can see whether Corollary 6 is describing the actual runs?

  2. When the PAC objective is requested and the fallback fires, what threshold is returned? If it is b(k)b_{(k)}, do you agree with the ≈49.5%\approx 49.5 \% failure rate in the constant-reference example, and which repair do you adopt? did any row in Tables 2 or 6 or any checkpoint in the 64-stream, actually invoke the fallback?

  3. could you confirm Proposition 4 is a lower bound rather than an identity, and report per redraw the actual full-calibration rank of q~\tilde{q} and its slack above k+m−Vk+m-V

2 comments
AuthorsOpenReview ↗

Response to reviewer ENqV – Part 1

Thank you for the careful theoretical review. We have corrected the algorithm and proof, and we separately address each of your methodological and empirical requests below.

Where each concern is addressed

Concern raisedResolution in manuscriptPrecise location
Rank-kk fallback invalidates the PAC guaranteeCorrected to a protected, prespecified rank; proof now covers adaptive fallback§§3.3–3.4; Corollary 5; Appendix E
Actual trigger versus stated triggerOne rule: bracket unbounded or rT+rE+∣B∣≥k+c0r_T+r_E+\lvert B\rvert\ge k+c_0§3.3; Appendix C.4; Appendix H.3
Did the reported runs fall back?Per-redraw branch count: zero in principal cells and checkpoint stream; stress test does fall back§6.1, §6.4; Appendix C.4
Proposition 4 presented as identity; realized ranks/slackLower-bound correction, complementary upper bound, per-redraw rangesAppendix A; Appendix E; Table 16
Isotonic details, ties/plateaus, and Corollary 6 relevanceImplementation specified; exact central/plateau/boundary decompositionAppendix C.1; Appendix A.1; Table 15
Semi-supervised risk control/PPI and other comparisonsDiscussion plus matched-budget comparison§2; Appendix H.9, Table 13
Credit and limited effect sizeExplicit citations and qualified claims§§1, 3.1, 6.2; Tables 2, 6–7

1. Corrected PAC fallback and its effect on the experiments

We agree that returning b(k)b_{(k)} after a data-dependent fallback preserves marginal coverage but does not preserve the claimed training-conditional bound. With a constant cache at n=10,000n=10{,}000, the original fallback misses the PAC target with probability 0.4951, not 0.005. The revised fallback returns b(k+c0)b_{(k+c_0)}, with c0c_0 set before observing calibration scores. The single trigger is an unbounded bracket or

rT+rE+∣B∣≥k+c0.r_T+r_E+\lvert B\rvert\ge k+c_0.

For MMLU, k+c0=9,084k+c_0=9{,}084 in PAC mode and 9,0539{,}053 in marginal mode. The stress update has ∣B∣=8,941\lvert B\rvert=8{,}941 and prospective cost 64+600+8,941=9,60564+600+8{,}941=9{,}605, so it does fall back under the corrected rule; the originally stated trigger against n=10,000n=10{,}000 would not have activated. Theorem 3 and Corollary 5 now cover the same implemented procedure. In particular, when V≤m−cV\le m-c for c≤c0c\le c_0, both the efficient and fallback branches return a threshold at least b(k+c)b_{(k+c)}, so the proof does not require an independent fallback decision (§§3.3–3.4, Appendix E).

Fallback audit (Tables 1–3, 6; §§6.1, 6.4):

ExperimentModeMedian blockMedian total forwardsFallbacks
MMLU LoRAmarginal3661,0300/20
ARC LoRAmarginal2028660/20
ImageNet INT8marginal8421,5060/20
ImageNet INT4marginal1,2921,9560/20
ImageNet pruningmarginal1,7122,3760/20
MMLU LoRA, rE=600r_E=600PAC5301,1940/20
MMLU LoRA, rE=2,000r_E=2{,}000PAC4102,4740/20
ImageNet INT8PAC1,1061,7700/20
64 LoRA checkpointsmarginal stream—1,048 median per checkpoint0/64

Thus, apart from the deliberately stressful update, the fallback repair does not alter the reported main results. The revised paper also corrects the checkpoint summary: 1,048 is the median per-checkpoint cost; multiplying it by 64 is not an observed cumulative total (§6.4).

2. Proposition 4, realized ranks and the slack

We changed the statement from an “identity” to a lower bound. Writing RR for the realized rank and VV for the number of residual-band violations, Appendix A and the proof in Appendix E give, off fallback under the stated conditions,

k+m−V≤R≤k+m+V.k+m-V\le R\le k+m+V.

This also bounds the slack above the guaranteed floor by 2V2V. Table 16 provides the bounds for every one of the 20 redraws, rather than only averages. For example, on marginal MMLU k+m=9,103k+m=9{,}103: the redraw with V=0V=0 has R=9,103R=9{,}103, while the redraw with V=37V=37 has R∈[9,066,9,140]R\in[9{,}066,9{,}140]. These are guaranteed rank ranges, not directly measured ranks. We leave the full 20-row diagnostic in Table 16 instead of repeating it here.

AuthorsOpenReview ↗

Response to reviewer ENqV – Part 2

3. Implementation and block-composition evidence

Appendix C.1 now specifies weighted PAVA on cached-score midranks, pooled ties and multiplicity weights, linear interpolation between fitted knots, and clipping outside the fitted range. The map is fitted independently of the residual and calibration samples. Appendix A.1 gives the exact decomposition

∣B∣=(2m+1)+(T−+T+)+(O−+O+),\lvert B\rvert=(2m+1)+(T_-+T_+)+(O_-+O_+),

with central ranks, plateau ties and residual-boundary points, respectively.

Block diagnostics (Table 15; median-block redraw in each cell):

UpdateCentral ranksPlateauBoundaryMedian ∣B∣\lvert B\rvert
MMLU LoRA20527134366
ARC LoRA1051582202
ImageNet INT850358281842
ImageNet INT45031066831,292
ImageNet pruning5031491,0601,712

These counts explain the actual work without assuming a continuous transported-score density. Corollary 6's density bound does not apply at the isotonic plateaus, which contribute 7–9% of each observed block. We have made this distinction explicit rather than claiming the measured results verify the density-based bound (§3.6; Appendix A.1; Table 15).

4. PPI, SPI, attribution and scope

Section 2 distinguishes transport from semi-supervised risk control/prediction-powered inference and SPI: those methods can obtain a threshold from proxy and gold samples without exact block rescoring, but do not in general guarantee pointwise containment of the full-recalibration set. The latter is the extra protected-event property of transport.

Matched proxy-assisted comparison (Table 13, MMLU; 1,194 forwards for each budget-limited method):

MethodGuaranteeCoverageSet growth vs full
PAC transportPAC, δ=0.005\delta=0.00591.66%7.7%
PAC mini splitPAC, δ=0.005\delta=0.00592.16%10.8%
Semi-supervised risk controlPAC, δ=0.005\delta=0.00592.35%11.3%
SPI (expected, not measured)marginal≈90%≈0–1%

The observed advantage over the PAC alternatives is modest and specific to the tested regime. The PAC transport–mini-split improvement is 2.8–3.1 percentage points of set growth across three executed comparisons (Tables 2 and 6); Table 7 shows how it can reverse at fixed rEr_E for much larger caches. We also credit Hulsman (2022) for the standard maximum-residual/order-statistic facts and Waudby-Smith and Ramdas for the betting confidence sequence (§§1, 3.1; Appendix B). Our claim is the combined score-transport, independent residual certification, indexed rescoring and protected-rank procedure, not novelty of those underlying facts.

Reviewer_3gp8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to recalibrate a split conformal predictor after the underlying model is updated, without rescoring the whole calibration set. The key observation is that only calibration points whose scores lie near the conformal threshold can change which order statistic is selected. The method fits an isotonic regression of post-update scores on cached (pre-update) scores, which the paper calls rank transport. It then places a high-probability error band around each predicted score. Only the points whose band could cross the threshold are rescored with the updated model. The resulting sets keep distribution-free marginal coverage. Section 3.6 gives a bound on the number of points that must be rescored. Experiments report savings in model evaluations relative to full recalibration on calibration sets of 10k to 25k points.

Critical strengths

The problem is clearly posed and the setting is well chosen. Rescoring only the calibration points that can move the threshold is a simple and intuitive idea, and the coverage guarantee is distribution-free. From a theoretical point of view, this is an interesting contribution.

Critical weaknesses

Critical Weaknesses

  1. The motivation and practical value are not established.

(a) Application need. The paper gives examples of models being updated. It does not identify applications where the conformal sets must be recomputed after each update, rather than once for the final model. Without such a setting, it is unclear when the saving matters.

(b) Baseline. Tables 1 and 3 report savings against full recalibration on calibration sets of 10k to 25k points. Standard split conformal prediction never requires using every available calibration point. The natural baseline is ordinary split conformal with the updated model on a random subset of the calibration set, at the same evaluation budget as the proposed method.

Table 2 includes this baseline (“marginal mini split”). However, it uses 1,194 evaluations against 1,030 for the proposed method, so the comparison is still not budget matched. At the reported budgets, mini split has a set-size inflation of 1.3%, against 4.4% for the proposed method, and it also keeps marginal validity. The proposed method also has overhead that mini split does not have, from fitting the transport map and computing the bounds.

The usual worry with a small calibration subset is that the estimated quantile becomes inflated. A subset of 1,000 points is already a substantial calibration budget, though. I would not expect ordinary calibration on 1,030 points to give larger sets than the proposed method. A more informative comparison would do three things. It would vary the budget from very small to large, match the two methods at each budget, and count all computational overhead.

(c) Reliance on the cached scores. The method assumes that the cached scores remain predictive of the updated scores. This is in some tension with the reason for updating a model, which is usually that the old model was inadequate.

Consider an extreme case. Suppose the old model predicts every example incorrectly with full confidence, so every cached nonconformity score equals 1. The updated model may correct these predictions with varying confidence, so the new scores differ across examples. The transport map receives the same input for every example. It therefore returns the same predicted score and the same band for all of them. The algorithm then rescores the entire calibration set and saves nothing. An ablation that varies how well the fitted map predicts the new scores would show how quickly the computational advantage disappears.

(d) Marginal coverage only. The evaluation reports only marginal coverage and set size. Marginal coverage is a relatively weak target. Against a budget-matched baseline, the method also does not appear to win on either coverage or set size. Conditional coverage would say more about whether the sets are reliable when the data are heterogeneous, for example coverage by class or by question characteristics. The paper does not discuss it.

  1. The presentation needs a systematic revision.

The paper is difficult to follow. It introduces a large private vocabulary (envelope, refresh, rank transport, contamination allowance) for objects that have standard names (interval, recalibration, isotonic map, violation tolerance). The prose is dense and formulaic, which adds to the reading burden. There are also inconsistencies. The proposed method is labelled “statistical” in Tables 1 and 3 but “transport” in Table 2. The notations rEr_E, δV\delta_V and δQ\delta_Q appear in the abstract and introduction before they are defined. A rewrite in standard conformal terminology would make the contribution much easier to assess.

Questions for authors

(Clarification) In which applications must the conformal sets be recomputed after each model update, rather than once for the final model? A concrete setting would strengthen the motivation.

(Requested experiment) Please compare against ordinary split conformal with the updated model on a random calibration subset of the same size as your rescoring budget. Please vary the budget from small to large. Report coverage, set size and wall-clock time, including the cost of fitting the transport map and computing the bounds. The current Table 2 comparison (1,194 against 1,030 evaluations) is not budget matched.

(Clarification and requested experiment) The gain comes from similarity between the cached model and the updated model. How is this similarity reflected in the theoretical results? How does performance change as the fitted map becomes less predictive of the new scores, for example in the degenerate case described in Weakness 1(c)?

(Clarification) How does the computation bound in Section 3.6 relate to the evaluation counts observed in the experiments?

(Requested experiment) Please report conditional coverage for both your method and the budget-matched baseline, for example class-conditional coverage or coverage across question types.

3 comments
AuthorsOpenReview ↗

Response to reviewer 3gp8 – Part 1

Thank you for the detailed feedback on motivation, baseline fairness, cache dependence, conditional coverage and presentation. We have made the purpose of the method narrower and more explicit, and added the requested comparisons and ablations.

Where each concern is addressed

Concern raisedResolution in manuscriptPrecise location
Why recalibration after each update?Repeated deployed LoRA/compressed checkpoints; each release needs validity§1; §6.4
Fair mini-split baseline, small budgets and overheadMatched marginal sweep and end-to-end timing; PAC comparison at matched failure§3.5; Tables 2, 6, 12; Appendix H.3
Why PAC ranking differs; dependency on cached modelFull-cache order-statistic margin; limits made explicit§§3.3–3.6; Tables 2, 6–7
Constant/poor cacheGuaranteed validity but no savings on fallback; graded ablation§3.3; Appendix C.4; Table 17
Compare predicted cost with measured blockExact block decomposition, plateau caveat§3.6; Appendix A.1; Table 15
Conditional/group coverageContainment-based group inequality and matched group audit§3.3; Appendix E; Table 14
Unclear terminology/labelsStandard terms; Figure 1; revised captionsAbstract; §§1–3; Tables 1–3

1. Application and the two different baseline objectives

The setting is a stream of fixed model updates—for example, deployed LoRA adapters or INT8/INT4/pruned versions of a shared backbone—where each released predictor needs a fresh conformal threshold. Section 6.4 evaluates 64 predetermined LoRA checkpoints (four runs × 16 checkpoints) sharing a 10,000-example cached MMLU calibration set. The median checkpoint uses 1,048 updated-model forwards versus 10,000 under full recalibration; none falls back. A per-release PAC bound does not automatically imply a simultaneous guarantee for all releases without allocating failure probability across them (§1, §6.4).

You are correct that a random updated-model calibration subset is the natural baseline. We now explicitly concede that mini split is better for marginal validity alone. Table 12 uses the same updated-forward budget and includes end-to-end overhead, whereas the marginal mini-split reference rows in Table 2 deliberately reuse the higher PAC budget and are not forward matched to marginal transport.

Marginal, forward-matched comparison (Table 12; MMLU; 20 paired redraws):

Updated forwardsMini-split set growthTransport set growthTransport time / mini-split time
1,0001.3%4.5%1.08×
2,0000.8%2.6%1.05×
4,0000.4%1.4%1.03×

The overhead includes updated-model evaluations, transfer, score processing, isotonic fitting, block discovery, order statistics, and fallback checking (Appendix H.8). The minimum possible marginal transport budget with n=10,000n=10{,}000, rT=64r_T=64 and δV=0.0025\delta_V=0.0025 is 759 forwards over choices of rEr_E; with the default rE=600r_E=600, its floor is 869. Mini split can operate below those budgets.

For the PAC objective, however, the comparison reverses within the evaluated moderate-cache regime. Both methods use the same failure level δ=0.005\delta=0.005, but mini split derives its conservative rank from its own small sample, whereas transport exploits the cached ranking of all 10,000 calibration examples:

TaskMatched forwardsPAC transport set growthPAC mini-split set growthPAC transport test coveragePAC mini-split test coverage
MMLU, rE=600r_E=6001,1947.7%10.8%91.66%92.16%
MMLU, rE=2,000r_E=2{,}0002,4746.7%9.5%91.09%91.49%
ImageNet INT81,7707.5%10.4%91.34%91.78%

These are measured comparisons (Tables 2, 6), not a claim of dominance for all nn. The fixed-rEr_E design analysis in Table 7 predicts a crossover near n≈100,000n\approx100{,}000; scaling rEr_E approximately as n\sqrt n can extend the favorable region at higher sampling cost. Those Table 7 numbers are design calculations, not additional experiments.

AuthorsOpenReview ↗

Response to reviewer 3gp8 – Part 2

2. Exactly how cache quality changes the result

The guarantee requires a fixed updated model and independent fitting/residual/calibration roles, not that the original and updated scores are close. When the cached order is informative, isotonic transport removes common monotone score shifts; on MMLU the median raw change is 0.071, while the median transported residual radius is 0.0211 (§6.2). A poor cache instead increases the band and the rescored block, until the protected fallback eliminates the computational advantage (§3.3).

Graded degradation (Table 17; MMLU marginal mode; 20 redraws):

Cached scoresRank correlationResidual radiusMedian blockMedian forwardsFallbacks/20Coverage
Original isotonic baseline0.9320.02113661,030090.78%
Added noise, σ=0.01\sigma=0.010.9010.02524661,130090.75%
Added noise, σ=0.02\sigma=0.020.8350.03316721,336090.71%
Added noise, σ=0.04\sigma=0.040.6620.04881,1901,854090.69%
Added noise, σ=0.08\sigma=0.080.3920.07922,3803,044090.60%
Constant cache—≥0.410,00010,6642090.52%

For a constant cache, all transported centers coincide, the block becomes the entire cache, and all 20 redraws rescore every point, using 64+600+10,000=10,66464+600+10{,}000=10{,}664 forwards. This is worse than full recalibration in cost, and we make no efficiency claim in that case. The protected rank, however, still provides the appropriate guarantee. These Table 17 experiments are marginal mode, not the PAC mode with 0.005 failure budget. Appendix C.4 additionally shows that replacing the isotonic map with the identity enlarges MMLU's block from 366 to 1,806, and reusing fit points for residual certification makes 7.4% of nominal brackets miss the exact threshold.

3. Work bound and actual rescoring

For the principal marginal rows, total forwards equal rT+rE+∣B∣=664+∣B∣r_T+r_E+\lvert B\rvert=664+\lvert B\rvert. Appendix A.1 and Table 15 now report the exact composition of the median-sized block:

TaskCentral ranks 2mV+12m_V+1Plateau pointsBoundary pointsMedian blockTotal forwards
MMLU LoRA205271343661,030
ARC LoRA1051582202866
ImageNet INT8503582818421,506
ImageNet INT45031066831,2921,956
ImageNet pruning5031491,0601,7122,376

This accounting does not presume the density condition of Corollary 6. Isotonic plateaus create atoms and therefore violate that assumption for approximately 7–9% of each block. We now explicitly distinguish the exact decomposition (Table 15) from the theoretical density-bound approximation (§3.6, Appendix A.1).

4. Conditional coverage, group audit and presentation

Theorem 3 gives containment of the full-recalibration set on a protected calibration event. Thus for any positive-probability group GG,

Pr⁡Y∈Cstat(X)∣X∈G≥Pr⁡Y∈Cfull(X)∣X∈G−δV.\Pr\\{Y\in C_{\mathrm{stat}}(X)\mid X\in G\\}\ge \Pr\\{Y\in C_{\mathrm{full}}(X)\mid X\in G\\}-\delta_V.

This inherits whatever group coverage full recalibration attains; it does not establish 90% coverage for every group. Table 14 gives the following one-redraw empirical audit, alongside PAC mini split at its matched budget:

MMLU subject groupTest examplesFresh recalibration coveredPAC transport coveredPAC mini split covered
Humanities674609 (90.4%)617 (91.5%)621 (92.1%)
STEM445395 (88.8%)402 (90.3%)405 (91.0%)
Social sciences444404 (91.0%)410 (92.3%)413 (93.0%)
Other455412 (90.5%)420 (92.3%)421 (92.5%)
All2,0181,820 (90.19%)1,849 (91.63%)1,860 (92.17%)

These are descriptive counts, not groupwise PAC certificates, and the mini-split sets are larger on average. Finally, the revision standardizes residual band, violation allowance, block recalibration, and Transport, defines symbols at first use, revises table captions and adds Figure 1 (§§1–3). We believe these additions answer the methodological and readability concerns without overstating the scope.

Thank you for the detailed response and the additional experiments. They address several of my questions, and I appreciate the candid discussion of the method’s scope. Two points remain on my side:

  1. Under the standard marginal guarantee, the new forward-matched comparison suggests that the mini-split baseline performs somewhat better, as the authors also note. The advantage then lies mainly in the PAC setting, which is a narrower contribution than the original framing. A clearer link between the application and the need for a PAC guarantee at each model update would strengthen the motivation.
  2. The cache-quality ablation does not yet include the baseline. A comparison against PAC mini split at the same budget would be informative, in particular the level of perturbation at which the baseline starts to outperform the method.

These points concern the scope and motivation of the work, and I find the technical content sound. I am keeping my current rating at this stage.

Reviewer_FTACOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers a problem of calibration after model updating and proposes an approach that makes use of conformity scores calculated before and after the model update. The main idea is to build a transporter between two sets of scores using probabilistic arguments (Beta-Binomial-based calculations) and derive a conservative rule for constructing prediction sets after the model update. This is practically useful, as demonstrated in the experiments.

Critical strengths
  1. The analysis based on ordered ranks is used to construct a statistical transformation of scores; the conservative coverage guarantees are appealing.

  2. Diverse experiments demonstrate the effectiveness of the proposed approach.

Critical weaknesses
  1. The motivation for the construction (for instance, section 3.3) is not described well. Also, the paper introduces many new terms without defining/reference (see my Q1).

  2. It would be nice to add a visualization of how the process works and how the statistical transporter is constructed. It will improve the readability of the paper.

  3. The technical novelty is not clear to me. For instance, the ideas of the statistical transporter follow usual calculations of order rank statistics, two sets of scores (see, for instance, SPI of Bashari, Meshi, et al.(2026)). I believe the proposed transporter idea is much closer to SPI, reframed for the particular application to the model updating issue. Could the authors clarify how their approach is distinct from SPI at the theoretical level?

References:

  1. Bashari, Meshi, et al. "Synthetic-powered predictive inference." Advances in Neural Information Processing Systems 38 (2026): 117704-117764.
Questions for authors
  1. What is the directional rank-contamination inequality on line 203? What is the block in line 210?

  2. Could the authors clarify how their approach is distinct from SPI at the theoretical level?

  3. How do we interpret Tables 1, 2, and 3? Do the experiments also compare with full recalibration with size equal to what is being used for transporter training+rE+∣B∣r_E+|B|? Is there any particular reasoning behind the choices of several set sizes rE,rT,δr_E, r_T, \delta?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FTAC – Part 1

Thank you for the precise requests for clarity, a visualization and a stronger baseline comparison. The revised text now explains the rank bracket and the block directly, includes a new Figure 1, and adds the requested comparisons and sensitivity results.

Where each concern is addressed

Concern raisedWhat was changedPrecise location
Old line 203: unexplained directional rank inequalityDefined as directional rank-violation inequality and cited where used§3.3, Eq. (13); Lemma 8, Appendix E
Old line 210: undefined blockSet BB defined as intervals meeting rank bracket§3.3, Eq. (15); Figure 1b
Motivation for selective rescoringExplanation that only threshold-adjacent bands can alter the indexed quantile§3.3, Figure 1
How the transporter is builtTwo-panel construction diagram and step-by-step text§§3.1–3.3; Figure 1a–b
Undefined/custom terminologyStandardized terms and symbol definitionsAbstract; §§1–3
Novelty relative to SPIRelated-work and proxy-assisted comparison, containment distinction§2; §3.3; Appendix H.9, Table 13
Interpreting Tables 1–3; mini-split baseline; rT,rEr_T,r_E choiceRevised captions; matched-budget sweep; sample-size sensitivitiesTables 1–3, 9–10, 12–13; §3.5

1. What the inequality and block mean — with the new Figure 1

The new Figure 1a illustrates fitting a monotone isotonic map hh from paired old and updated scores on the rTr_T transport examples, then setting a radius ε^\widehat\varepsilon to the maximum residual on an independent sample of size rEr_E. Figure 1b shows the transported cached order, the bracket [L,U][L,U], and the contiguous block to be rescored.

Specifically, if at most mm final-calibration scores violate their residual bands, Lemma 8 bounds the updated rank-kk threshold by

z[k−m]−ε^≤b(k)≤z[k+m]+ε^,z_{[k-m]}-\widehat\varepsilon\le b_{(k)}\le z_{[k+m]}+\widehat\varepsilon,

where zi=h(ai)z_i=h(a_i). With these endpoints L,UL,U, the block is

B=i:zi+ε^≥L,zi−ε^≤U.B=\\{i:z_i+\widehat\varepsilon\ge L,\\ z_i-\widehat\varepsilon\le U\\}.

Thus, BB contains precisely the cached-order positions whose bands intersect the relevant rank bracket. Points entirely below are counted and points entirely above are ignored; only BB requires exact updated scores. Since hh is nondecreasing, two binary searches locate the contiguous block. If it would be too expensive (or the bracket is unbounded), the protected fallback returns b(k+c0)b_{(k+c_0)}. This is the step-by-step motivation added to §3.3 and visualized in Figure 1; the technical proof is in Appendix E.

2. Distinction from SPI and what the added comparison actually shows

We now distinguish our setting from SPI explicitly (§2; Appendix H.9). SPI aligns unpaired synthetic and real score samples and returns a threshold constrained by order statistics of the small real sample; its guarantee is marginal. Transport fits on paired old/new scores, independently calibrates a residual-violation allowance, and rescores the exact indexed block. On an event of probability at least 1−δV1-\delta_V, its prediction set contains the full-recalibration set for every input. The same construction can be PAC-tuned with a prespecified rank margin; the two methods do not offer identical guarantees.

Table 13's MMLU comparison:

MethodUpdated forwardsType of guaranteeCoverageSet growth vs full
Fresh recalibration10,000marginal reference90.20%0.0%
PAC transport1,194PAC, δ=0.005\delta=0.00591.66%7.7%
PAC mini split1,194PAC, δ=0.005\delta=0.00592.16%10.8%
Semi-supervised risk control1,194PAC, δ=0.005\delta=0.00592.35%11.3%
SPI (expected, not measured)1,194marginal≈90%≈0–1%

The SPI estimates are not an additional experiment, and its marginal row should not be presented as a PAC comparison. The numerical worst-case SPI bound discussed in Appendix H.9 is explicitly stated there for continuous synthetic scores; the transported isotonic scores can have ties, so we do not treat that continuous-case bound as directly established for the plateau setting. The robust point is the different guarantee: PAC validity and full-set containment for transport versus a marginal-only SPI threshold.

I thank authors for their response. However, I don't see the revised version yet. Please let know once the revision is uploaded, so that I can review it and provide my reassessment.

AuthorsOpenReview ↗

Response to Reviewer FTAC – Part 2

3. How to read Tables 1–3 and the correct budget comparison

Tables 1 and 3 compare full recalibration, a stale threshold, and Transport on one fixed update. ∣B∣\lvert B\rvert is the median rescored block; Forwards includes all rT+rE+∣B∣r_T+r_E+\lvert B\rvert updated-model evaluations; Reduction compares these evaluations with full recalibration; coverage and set-size change use the same held-out data. Table 2 distinguishes marginal-efficient from PAC-tuned transport; only its PAC transport and PAC mini-split rows are matched at both forwards and failure probability. The marginal mini-split reference rows use the larger PAC budgets, so they are not matched to marginal transport. The revised captions state this explicitly.

The new Table 12 supplies the missing marginal forward-matched benchmark, and is unambiguous:

Matched forwardsMini-split set growthTransport set growthTransport relative time
1,0001.3%4.5%1.08×
2,0000.8%2.6%1.05×
4,0000.4%1.4%1.03×

Thus, mini split is preferred for marginal coverage alone. The reason to use transport is its protected containment property and, when a PAC guarantee is required, smaller sets in the executed matched-budget MMLU and ImageNet INT8 comparisons (Tables 2, 6). These objectives should not be conflated.

4. Why rT=64r_T=64, rE=600r_E=600?

These parameters balance the price of fitting the monotone map, the number of residual evaluations, the beta-binomial violation allowance and the block width. At fixed n=10,000n=10{,}000 and rT=64r_T=64, increasing rEr_E reduces both mVm_V and set growth while increasing total forwards:

Residual-sample sensitivity (Table 9, MMLU marginal mode):

rEr_EmVm_VMedian blockTotal forwardsSet growth
3002004408047.6%
600 (default)1023661,0304.4%
1,000622561,3202.8%

At rE=600r_E=600, the measured forward counts for rT=32,64,128,256r_T=32,64,128,256 are 1,048, 1,030, 1,078, 1,197, respectively (Table 10), so rT=64r_T=64 is the lowest-cost setting among these tested fit sizes. The choices are supported by sensitivity experiments, not claimed to be universally optimal. We hope the new Figure 1, explicit definitions and matched comparisons make the approach and its limits easier to assess.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers for their thorough feedback. We have uploaded a substantially revised manuscript. The revision corrects a genuine issue in the PAC fallback, adds budget-matched and proxy-assisted comparisons, analyzes cache quality and the actual size of the rescored block, and rewrites the exposition with a new two-panel Figure 1. The table below provides a direct guide to the changes.

Revision / concernReviewer(s)Where in revised PDF
PAC fallback repaired: return prespecified protected rank b(k+c0)b_{(k+c_0)}; one cost trigger, with proof covering either branchENqV§§3.3–3.4; Appendices A, E, C.4
Rank accounting: Proposition 4 is a lower bound, not an identity; complementary upper bound and per-redraw rank rangesENqVAppendix A; Appendix E; Table 16
Isotonic details: weighted PAVA, ties, interpolation, clipping; central/plateau/boundary decomposition of BBENqV, 3gp8Appendix C.1; Appendix A.1; Table 15
Comparable baselines: forward-matched marginal mini split and PAC mini split at the same failure budget3gp8, FTAC§3.5; Tables 2, 6, 12
Proxy-assisted baselines and SPI distinctionENqV, FTAC§2; Appendix H.9, Table 13
Cache dependence: constant-cache fallback and graded perturbation3gp8§3.3; Appendix C.4; Table 17
Expected work vs observed blocks: role of isotonic plateaus and density assumptionENqV, 3gp8§3.6; Appendix A.1; Table 15
Inherited group coverage and empirical subject audit3gp8§3.3; Appendix E; Table 14
Deployment motivation, repeated releases, checkpoint stream3gp8§1; §6.4; Appendix H.7
Clearer method exposition and visualization: motivation, Lemma 8, definition of BB, consistent terminology and table captionsFTAC, 3gp8§§1–3; Figure 1; Tables 1–3; Appendix E
Crediting existing calibration/anytime-valid resultsENqV§§1, 3.1; Appendix B

Main empirical result (marginal mode; Tables 1 and 3). Each row summarizes 20 redraws, with rT=64r_T=64, rE=600r_E=600, and δV=0.0025\delta_V=0.0025. Forwards include both small samples and exact rescoring of the boundary block.

Updated model / taskFull forwardsTransport forwardsTransport coverageStale-threshold coverageTransport set growth vs full
Llama-3-8B LoRA / MMLU10,0001,03090.78%87.60%4.4%
Llama-3-8B LoRA / ARC5,00086690.80%87.10%4.4%
ViT-B/16 INT8 / ImageNet25,0001,50690.84%88.45%4.5%
ViT-B/16 INT4 / ImageNet25,0001,95690.85%86.92%4.9%
ViT-B/16 magnitude pruning / ImageNet25,0002,37690.87%87.61%5.1%

The central qualification is now explicit. For marginal coverage alone, mini split produces smaller sets at matched forwards, and the new Table 12 shows that transport is also 3–8% slower end-to-end than mini split. We do not claim transport dominates that simpler baseline. Its benefits are (i) a high-probability pointwise containment relation to full recalibration, and (ii) more economical sets for the training-conditional (PAC) objective in the tested moderate-cache regime. For the latter, transport and PAC mini split use identical forward budgets and total failure budget δ=0.005\delta=0.005:

TaskMatched updated forwardsPAC transport coverage / set growthPAC mini-split coverage / set growthDifference in set growth
MMLU (rE=600r_E=600)1,19491.66% / 7.7%92.16% / 10.8%3.1 percentage points
MMLU (rE=2,000r_E=2{,}000)2,47491.09% / 6.7%91.49% / 9.5%2.8 percentage points
ImageNet INT81,77091.34% / 7.5%91.78% / 10.4%2.9 percentage points

These are empirical, not universal, gains (Tables 2 and 6). Table 7 identifies their limits: a fixed residual sample eventually loses its PAC advantage for sufficiently large caches; the rE∝nr_E\propto\sqrt n design extends the favorable region at additional sampling cost. Under the corrected fallback, no principal-table redraw and no stream checkpoint fell back, so the repair does not change the principal experimental values (§6.1, §6.4).

Additional evidence. Table 13 reports semi-supervised risk control and discusses SPI; the SPI row contains expected values, not executed measurements. Table 14 is a group audit for one MMLU calibration redraw, not a groupwise PAC theorem. Tables 15–17 give the exact block decomposition, per-redraw rank bounds, and cache-degradation experiment. Finally, we withdraw the earlier CIFAR-100/PAMAP2 comparison and group-audit claims made in our November 16 comments, which are not results in this revision. We hope these revisions make both the contribution and its limitations easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.