- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at recalibrating a split conformal predictor after a model update (backbone LoRA, quantization, pruning) without rerunning the new model on the whole calibration set. Instead of bounding the change deterministically, the authors fit an isotonic map from cached old scores to new ones on a small sample, and use the max residual on a second, independent sample of size as an envelope radius. Since this is a sample max, the number of the calibration points falling outside the envelope is stochastically dominated by , whose -quantile gives a contamination allowance . The new -th order statistic is then pinned between the transported scores at ranks and , give or take , so only a contiguous block of the sorted cache gets rescored. With probability the threshold is no smaller than the full-recalibration one, giving marginal coverage, and a training-conditional version just enlarges . Experiments on Llama-3-8B and ViT-B/16 show large savings in forward passes over full recalibration with slightly larger sets, though a same-budget mini split remains a competitive baseline.
Critical strengths
the paper's interesting claim is that an arbitrary fixed update, like LoRA inside the transformer blocks, INT4, 30% magnitude pruning, can be turned into a conservative conformal threshold without any deterministic network envelope, for forwards on fresh samples plus a boundary block
Critical weaknesses
- The procedure that is implemented is not the procedure that is proved, and in one place this could be a correctness failure, i.e., the fallback breaks the PAC guarantee. Appendix H. 3 says that when the system "performs full recalibration" and that this "changes computation but not ... the statistical statement". Full recalibration was defined in Section 2 as returning . For the marginal statement that is harmless. For the training-conditional statement it is not: Corollary 5 with the eq (27) allowance needs on the good event, and discards exactly those ranks. A counterexample inside the paper's own assumptions: , every role iid and independent. Isotonic regression on a constant input returns , so , all transported intervals coincide, , and the fallback fires with probability one. With the MMLU parameters ,
\begin{equation*} \mathbb{P}\{F\left(b_{(9001)}\right)<0.9\}=\mathbb{P}\{\operatorname{Bin}(10,000,0.9) \geq 9001\} \approx 0.495 \end{equation*}
versus the advertised . No hypothesis of Corollary 5 is violated (ties among centres are allowed; the density condition belongs to Corollary 6, not Corollary 5). So the 0.5% training-conditional claim does not hold for the algorithm as written, and the optimized 9.01% / 4.30% bounds for the marginal-efficient route don't survive rank- fallback either.
-
Where the method is supposed to win, the measured gain is small and narrow. For the marginal objective the authors concede mini split is better
-
the closest existing framework is absent. Semi-supervised risk control via prediction-powered inference (Einbinder, Ringel & Romano) handles exactly this large-proxy / small-gold structure with finite-sample PAC risk control and a binary-loss specialisation; with proxy , gold and loss it yields a valid PAC threshold with no block rescoring, and plain PPI already has nonasymptotic quantile inference. Relatedly, the beta-binomial facts in Propositions 1-2 are standard calibration-conditional order-statistic results (Hulsman 2022 gives both, ties included), and the betting sequence is Waudby-Smith-Ramdas off the shelf. The only novelty is the specific combination, and the value of that combination is precisely what the missing comparisons would establish
Questions for authors
-
Which isotonic implementation is used (step function or linear interpolation, out-of-range handling), and how much calibration mass near rank lands on plateaus? equivalently, can you give plateau / boundary-mass diagnostics so one can see whether Corollary 6 is describing the actual runs?
-
When the PAC objective is requested and the fallback fires, what threshold is returned? If it is , do you agree with the failure rate in the constant-reference example, and which repair do you adopt? did any row in Tables 2 or 6 or any checkpoint in the 64-stream, actually invoke the fallback?
-
could you confirm Proposition 4 is a lower bound rather than an identity, and report per redraw the actual full-calibration rank of and its slack above
2 comments
Response to reviewer ENqV – Part 2
3. Implementation and block-composition evidence
Appendix C.1 now specifies weighted PAVA on cached-score midranks, pooled ties and multiplicity weights, linear interpolation between fitted knots, and clipping outside the fitted range. The map is fitted independently of the residual and calibration samples. Appendix A.1 gives the exact decomposition
with central ranks, plateau ties and residual-boundary points, respectively.
Block diagnostics (Table 15; median-block redraw in each cell):
| Update | Central ranks | Plateau | Boundary | Median |
|---|---|---|---|---|
| MMLU LoRA | 205 | 27 | 134 | 366 |
| ARC LoRA | 105 | 15 | 82 | 202 |
| ImageNet INT8 | 503 | 58 | 281 | 842 |
| ImageNet INT4 | 503 | 106 | 683 | 1,292 |
| ImageNet pruning | 503 | 149 | 1,060 | 1,712 |
These counts explain the actual work without assuming a continuous transported-score density. Corollary 6's density bound does not apply at the isotonic plateaus, which contribute 7–9% of each observed block. We have made this distinction explicit rather than claiming the measured results verify the density-based bound (§3.6; Appendix A.1; Table 15).
4. PPI, SPI, attribution and scope
Section 2 distinguishes transport from semi-supervised risk control/prediction-powered inference and SPI: those methods can obtain a threshold from proxy and gold samples without exact block rescoring, but do not in general guarantee pointwise containment of the full-recalibration set. The latter is the extra protected-event property of transport.
Matched proxy-assisted comparison (Table 13, MMLU; 1,194 forwards for each budget-limited method):
| Method | Guarantee | Coverage | Set growth vs full |
|---|---|---|---|
| PAC transport | PAC, | 91.66% | 7.7% |
| PAC mini split | PAC, | 92.16% | 10.8% |
| Semi-supervised risk control | PAC, | 92.35% | 11.3% |
| SPI (expected, not measured) | marginal | ≈90% | ≈0–1% |
The observed advantage over the PAC alternatives is modest and specific to the tested regime. The PAC transport–mini-split improvement is 2.8–3.1 percentage points of set growth across three executed comparisons (Tables 2 and 6); Table 7 shows how it can reverse at fixed for much larger caches. We also credit Hulsman (2022) for the standard maximum-residual/order-statistic facts and Waudby-Smith and Ramdas for the betting confidence sequence (§§1, 3.1; Appendix B). Our claim is the combined score-transport, independent residual certification, indexed rescoring and protected-rank procedure, not novelty of those underlying facts.
Response to reviewer ENqV – Part 1
Thank you for the careful theoretical review. We have corrected the algorithm and proof, and we separately address each of your methodological and empirical requests below.
Where each concern is addressed
1. Corrected PAC fallback and its effect on the experiments
We agree that returning b(k) after a data-dependent fallback preserves marginal coverage but does not preserve the claimed training-conditional bound. With a constant cache at n=10,000, the original fallback misses the PAC target with probability 0.4951, not 0.005. The revised fallback returns b(k+c0), with c0 set before observing calibration scores. The single trigger is an unbounded bracket or
rT+rE+∣B∣≥k+c0.
For MMLU, k+c0=9,084 in PAC mode and 9,053 in marginal mode. The stress update has ∣B∣=8,941 and prospective cost 64+600+8,941=9,605, so it does fall back under the corrected rule; the originally stated trigger against n=10,000 would not have activated. Theorem 3 and Corollary 5 now cover the same implemented procedure. In particular, when V≤m−c for c≤c0, both the efficient and fallback branches return a threshold at least b(k+c), so the proof does not require an independent fallback decision (§§3.3–3.4, Appendix E).
Fallback audit (Tables 1–3, 6; §§6.1, 6.4):
Thus, apart from the deliberately stressful update, the fallback repair does not alter the reported main results. The revised paper also corrects the checkpoint summary: 1,048 is the median per-checkpoint cost; multiplying it by 64 is not an observed cumulative total (§6.4).
2. Proposition 4, realized ranks and the slack
We changed the statement from an “identity” to a lower bound. Writing R for the realized rank and V for the number of residual-band violations, Appendix A and the proof in Appendix E give, off fallback under the stated conditions,
k+m−V≤R≤k+m+V.
This also bounds the slack above the guaranteed floor by 2V. Table 16 provides the bounds for every one of the 20 redraws, rather than only averages. For example, on marginal MMLU k+m=9,103: the redraw with V=0 has R=9,103, while the redraw with V=37 has R∈[9,066,9,140]. These are guaranteed rank ranges, not directly measured ranks. We leave the full 20-row diagnostic in Table 16 instead of repeating it here.