acceptodds
Under review as a conference paper at ICLR 2027

Tail Dependence, Not Intraclass Correlation: Design Effects and a Cluster Floor for Conformal Calibration

Abstract

Conformal prediction is often calibrated on clustered data, such as patients within hospitals or cells within donors. Clustering has little effect on marginal coverage, but it can substantially change training-conditional coverage, the coverage induced by the calibration set at hand. We give an exact finite-sample characterization of this effect. The calibration counting process satisfies a design-effect identity governed by the within-cluster correlation of threshold indicators rather than by the intraclass correlation of the scores. At extreme thresholds, this indicator correlation converges to the upper tail-dependence coefficient of the within-cluster copula. Thus two problems with the same intraclass correlation can require very different corrections: the design-effect penalty vanishes for Gaussian dependence as the miscoverage level shrinks, whereas it persists for a tail-dependent copula. The identity connects Vovk’s i.i.d. Beta law to the comonotone regime, where the effective sample size is the number of clusters. We also prove a cluster floor: no rule can guarantee PAC coverage, and no procedure can guarantee conformal risk control, from fewer than clusters; for , this is 22 clusters. Finally, we construct dependence for which a single-number summary of within-cluster correlation understates the design effect by a factor approaching the cluster size. On multi-site neuroimaging, threshold-level calibration is the only implementable method among those compared that meets the empirical PAC target.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, clustered data, tail dependence, design effect, training-conditional coverage, PAC coverage

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HmWCOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Disclaimer: I lack the expertise in conformal prediction to assess the theoretical results, and I have indicated this to the ACs. This summary reflects my understanding of the paper’s aims and empirical findings.

The paper addresses conformal calibration when data are not IID but clustered by source, as in life-science settings where data come from different hospitals or donors. The authors argue that clustering has little effect on marginal coverage but can substantially affect conditional coverage, because score distributions differ between clusters. They attribute this to tail dependency from within-cluster thresholding rather than intraclass correlation. They show theoretically that conformal prediction and conformal risk control cannot guarantee PAC coverage when there are too few clusters, which motivates a cluster floor. The experiments use two multi-site/multi-donor datasets, one neuroimaging and one single-cell. The proposed threshold-level correction is the only PAC-compliant method on the neuroimaging data and the best-suited method on the single-cell data.

Critical strengths

The problem is highly relevant: clustered, non-IID data are common in medical and life-science applications, where calibration matters.

The work could guide both experimental design and the evaluation of conformal prediction methods, in medical tasks and beyond.

It contributes to trust in predictions and to the reliability of ML methods for such tasks.

Critical weaknesses

I lack the expertise in conformal prediction to assess the correctness of the propositions, theorems and proofs, and I have indicated this to the ACs. My assessment therefore covers only the relevance of the problem.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HmWC

We thank Reviewer HmWC for the supportive evaluation and for stressing the practical importance of trustworthy calibration in multi-site medical and life-science studies.

Protection for individual hospitals and donors (new §5, Proposition 3). Average coverage can hide poor performance at specific centers, so we added an exact finite-sample site-level calibration rule that certifies coverage for at least a 1−γ1-\gamma fraction of new sites. Across the neuroimaging, single-cell and census benchmarks, it reduces the share of under-covered sites or donors from 27–46% under standard i.i.d. calibration to 0–7.5% (Figure 3, Tables 2–3, 11).

Comparison with published hierarchical methods (§7, Figure 3). We added five baselines from Dunn et al. (2023) and Lee et al. (2025) and a third benchmark, 51-state U.S. Census data, and we report both high-probability pooled validity and per-site coverage. Our two procedures are the only methods that meet the PAC target with finite prediction intervals in all six real-data settings, including when coverage is computed exactly against the real site populations (Appendix K).

Study-design guidance and assumption checks (§3.1, §5, Appendix G). Following your point on experimental design, the paper gives sample-size rules: about b≥log⁡δ/log⁡(1−α)b \ge \log\delta/\log(1-\alpha) clusters (2222 hospitals or donors at 90% coverage and 90% confidence) suffice for a pooled PAC guarantee without assumptions on within-site dependence, and about b≥log⁡δ/log⁡(1−γ)b \ge \log\delta/\log(1-\gamma) sites suffice for site-level guarantees. Cross-fitted residual diagnostics (Appendix G) let practitioners check between-site heterogeneity and cohort range before deployment.

Readability. We rewrote the abstract in plain language, added a workflow diagram (Figure 1), and included a concrete neuroimaging running example after the main results.

Reviewer_HmWCOpenReview ↗

Thank you for the response.

Thanks for the clarifications and additional results. As stated previously I have a hard time assessing the validity of the main claims. But the relevance of the problems and the additional experiments supporting results makes me inclined to keep the weak accept score (but very low confidence of 1).

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for reviewing our additional experiments and for highlighting the practical value of reliable multi-site calibration in biomedical studies.

Reviewer_BYzLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper focusses on the problem of conformal prediction. Specifically, the authors study the how clustering of calibration data impact the reliability of conformal prediction. They study this problem theoretically and show that traditional approaches may achieve target coverage on an average but actual coverage achieved for a particular calibration dataset used in deployment can vary greatly due to correlations within clusters. They obtain a finite sample characterization in the presence of this effect. Based on this analysis, they proposed a threshold-dependent calibration correction to fix these issues. They evaluated their approach and compared it to baselines using simulations, multi-site neuroimaging data, and single-cell data. The results are mixed.

Critical strengths
  • I liked the fact that authors analyzed this problem mathematically and identified/explained the gaps using their analysis. The exact design-effect identity (Theorem 1) and its connection to tail dependence (Theorem 2) and their consequences provide useful insights into why intraclass correlation can be misleading for conformal calibration.

  • Quantification in Theorem 3 of the discrepancy due to single/scalar dependecy estimate can underestimate the design effect, motivating the proposed threshold-level correction (Section 5) is a great way to bridge theory with practice.

Critical weaknesses

I have not identified any critical weaknesses. However, I have added some questions and suggestions for the author in the next section.

Questions for authors
  • The paper is theory-heavy and difficult to follow for non-expert readers. I am wondering if authors could include a running example (or a simple workflow diagram) with intuitive explanations of different theorems and how they connect with algorithm in Section 5 for practitioners?

  • I found experimental results to be weak. This raises concern whether the theoretical insights are translating into a practical calibration method or not. a) can authors better explain the impact/consequence of having width (yr) on Table 3 larger than baselines? b) why does their method fail to meet the PAC target on the single-cell dataset?

  • Could authors test their approach on some recent ML conformal prediction benchmarks wrt state of the art mentioned in Section 7 on them? This will help in better highlighting practical utility of the approach.

  • Abstract should be made more accessible. I found the arguments to be convoluted and confusing, until I read the full paper.

  • Is there any discrepancy (in assumptions) between Theorem 1 and method/algorithm used in Section 5?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BYzL

We thank Reviewer BYzL for the encouraging assessment and the practical questions, which led us to isolate the small-cluster approximation error in §6 and to expand the benchmarks.

Q1 and Q4 (abstract, workflow diagram, running example). We rewrote the abstract in plain language. It now states the failure mode, the threshold-indicator mechanism, the cluster floor, our two procedures and the empirical findings. The new Figure 1 shows the two routes: estimating ρind(t)\rho_{\mathrm{ind}}(t) for the pooled design-effect rule (§6, top) and aggregating per-site binomial quantile bounds for the exact site-level rule (§5, Proposition 3, bottom). Both routes are bounded by the cluster floor. A concrete ABIDE example runs through §1, §3 and §5, with a plain-language reading after Theorem 1, Remark 1 and Propositions 2–3. After Theorem 1, n=244n=244 clustered subjects carry the information of about 1616 independent ones. After Remark 1, the score ICC (0.200.20) and the median indicator correlation (0.050.05) differ from the value at the 90% threshold (0.160.16), so a median-based summary understates the design effect 2.72.7-fold.

Q2b and Q5 (Theorem 1 versus §6, and the single-cell result). Your question identified why our original plug-in rule exceeded δ=0.10\delta=0.10 on TMS. Theorem 1 adds no assumption, but §6 makes three approximations, which Appendix D (Table 6) now isolates. The normal approximation of N(1−α)N(1-\alpha) is conservative at small bb when the true ρind\rho_{\mathrm{ind}} is known (violation 0.0140.014 at b=6b=6); a Bernstein–Cantelli variant is valid in finite samples given ρind\rho_{\mathrm{ind}} but returns the whole line for b≤20b\le 20. The cluster-bootstrap upper bound under-covered ρind(0.9)\rho_{\mathrm{ind}}(0.9) when bb was small, and a deterministic leave-one-cluster-out jackknife bound (bb evaluations instead of 200200 resamples) doubles that coverage at small bb and brings end-to-end violations onto target for all b∈[6,320]b\in[6, 320] (0.0760.076–0.1050.105 in Table 6; 0.0870.087–0.0980.098 in Table 1). The third approximation, evaluation at the empirical threshold τ^\widehat{\tau}, has a negligible effect as nn grows.

With the jackknife bound, the pooled correction meets the PAC target on TMS within Monte Carlo error with b=11b=11 donors (0.1150.115, one-sided binomial p=0.27p=0.27, previously 0.1550.155; Table 3) and with b=28b=28 donors (0.1100.110). The exact site-level rule (§5, Proposition 3) avoids all three approximations and achieves 0.0550.055 and 0.0050.005. Coverage realized on a few test donors is itself noisy, so Appendix K also computes it exactly against all 4444 donors: the pooled correction then violates the target with probability 0.0980.098 and 0.0830.083, and the site-level rule with 0.0250.025 and 0.0000.000 (Table 14).

Q2a (interval width). Our pooled correction gives the shortest valid intervals wherever the baselines' finite-sample slack has disappeared. On ACS (Table 11), pooled intervals are within 5% of the population quantile and at least 12% narrower than every other valid method; at b=13b=13 the width is 1.201.20 against 1.881.88–1.891.89 for HCP and M3, 36% narrower. On TMS with b=28b=28 (Table 3), the pooled rule is 12% narrower (21.4121.41 against 24.2924.29 months) than M2, the only baseline still valid there. With 1111–1212 clusters, M3 and HCP are valid and 3–10% narrower than our correction, because their 1/(K+1)1/(K+1) mass at infinity forces a quantile near 0.9750.975 at that bb. This slack is not a guarantee: the same rules return the whole line with 66 sites and fail on 25–26% of splits with 2828 donors. With b=6b=6 ABIDE sites (Table 2), the wider intervals (28.2528.25 years) reflect the information available from six clusters with strong site effects (DEFF=15.3\mathrm{DEFF}=15.3): narrower rules fail on 32–41% of splits, M3, HCP and HCP2 return the whole line, and even the test-site oracle under-covers 44% of sites. The calibrated level tightens quickly as bb grows, from 0.9850.985 at b=6b=6 to 0.9180.918 at b=160b=160 (Table 1).

Q3 (benchmarks and state of the art). We added all five applicable methods from §8, M1–M3 of Dunn et al. (2023) and HCP and HCP2 of Lee et al. (2025), and a tabular ML benchmark, ACSIncome across all 51 U.S. states (folktables, Ding et al., 2021; §7.4, Table 11). Across the six real-data settings (Figure 3), our two procedures are the only methods that meet the PAC target with finite prediction sets in every setting, on held-out splits and under exact evaluation against the real site populations (Tables 11, 13–14).

Thank you for your response. I remain positive in my assessment of the paper.

AuthorsOpenReview ↗

Thank you for your support

Thank you for your continued support, and for the questions on the finite-cluster approximation and benchmarks that helped strengthen the revised paper.

Reviewer_Noy8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors tackle the problem of conformal prediction when the calibration set contains clustered data (e.g. patients from different hospitals), and the test points come from a new cluster, which can violate conditional coverage. They characterize this problem theoretically, expressing the probability of a coverage violation exactly in terms of the number of calibration points below the threshold, deriving an exact identity for this count’s variance, and proving that no non-trivial procedure can guarantee PAC coverage with fewer than a certain number of clusters. They then derive a corrected threshold, and evaluate it on synthetic and real-world data.

Critical strengths
  • The authors tackle an important problem in conformal prediction.

  • The authors evaluate on real-world imaging and cellular datasets with multiple clusters to verify their proposed correction.

Critical weaknesses
  1. The theory assumes that the test point comes from a new cluster that is absent from the calibration data, and independent of it. It also assumes that the new site has the same marginal distribution as the calibration scores. These assumptions are quite restrictive, and would break under simple covariate shifts, e.g. a different scanner or patient population that is not represented in the calibration sets. The authors should also verify whether these assumptions are satisfied in the empirical datasets.

  2. The coverage guarantee from the paper is for a random new site, but does not cover the worst-case site. So a particular new site can be very under-covered, and another over-covered. This limits the significance of the theoretical result. Having per-site guarantees would greatly strengthen the theory, and would address some of the motivations of the paper, e.g. the example in the intro that 40% of individual sites violate coverage.

  3. Similarly, the authors should report per-site coverage for each method in the empirical evaluations.

  4. The paper is generally hard to follow. There is a lot of notation that is used before it is defined, especially in the introduction. There is little intuition provided about the propositions and theorems. Overall, the theory is very dense and hard to parse.

  5. The authors should evaluate a few other baselines, such as the two other approaches in Dunn et al. (2023), and the one proposed in Lee et al. (2025).

Questions for authors
  1. Proposition 2 says that no distribution-free procedure can guarantee PAC coverage at α=β=0.1\alpha = \beta = 0.1 with fewer than 22 calibration clusters, but both of the real-world datasets use far fewer (6 and 11 respectively). How should the empirical results be interpreted in light of this? Is this a flaw with the assumptions in Proposition 2?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Noy8

We thank Reviewer Noy8 for the detailed critique; the points on per-site guarantees, assumption checks and hierarchical baselines each led to new results.

W1 (assumptions, verification and shifts). §2 now states the hierarchical model behind (A1)–(A4), the two-level model of Dunn et al. (2023) and Lee et al. (2025): each site jj draws its score distribution Pj∼ΠP_j \sim \Pi from a population Π\Pi of sites, and its observations are drawn from PjP_j. Individual sites need not share a distribution: scanners, protocols or cohorts may differ arbitrarily, F=∫P dΠ(P)F = \int P\ d\Pi(P) is the induced mixture, and between-site differences in PjP_j create the indicator correlation ρind(t)\rho_{\mathrm{ind}}(t) and the design effect. (A4) excludes only a new site whose PnewP_{\mathrm{new}} is not drawn from Π\Pi, such as a modality absent from calibration; without labels or likelihood ratios from that site, no distribution-free method, including those of Dunn et al. and Lee et al., can correct such a shift.

Appendix G (Table 8) checks the assumptions on cluster-cross-fitted residuals. Between-site heterogeneity is strong (Kruskal–Wallis p=1.8×10−14p = 1.8\times 10^{-14} and <10−300< 10^{-300}), as the model allows; there is no acquisition-order drift within ABIDE sites (p=0.07p=0.07); and cluster size is not significantly related to the site's 90% quantile (p=0.06p=0.06 and 0.430.43). Cohort age predicts site quantiles (Spearman 0.690.69), so new sites should fall within the calibrated age range. Appendix H (Table 9) draws the new site from a shifted population: the pooled PAC rules remain valid up to a location shift of half a between-site standard deviation, and the site-level rule remains valid at every shift tested, because its per-site target leaves a margin at the pooled level.

W2 and W3 (site-level guarantee and per-site coverage). Guaranteeing every possible site forces infinite intervals (Remark 2), so §5 targets a tolerance over the site population: with probability ≥1−δ\ge 1-\delta, at least a 1−γ1-\gamma fraction of new sites receive site-conditional coverage ≥1−α\ge 1-\alpha (Proposition 3, Corollary 5). This addresses the 40% of under-covered sites in §1. The rule takes a one-sided binomial upper bound Q^j\widehat{Q}_j on each calibration site's (1−α)(1-\alpha) quantile and applies Vovk's PAC order statistic to Q^1,…,Q^b\widehat{Q}_1,\dots,\widehat{Q}_b. It is exact in finite samples under arbitrary heterogeneity, and its requirement b≥log⁡δ/log⁡(1−γ)b \ge \log\delta/\log(1-\gamma) is necessary, so with few sites we report its smallest certifiable γ\gamma (0.360.36 at b=6b=6, 0.170.17–0.200.20 at b=11b=11–1313, at most 0.100.10 at b=28b=28 and 3131).

Every method now reports the share of under-covered sites (Figure 3 right; Tables 2–3, 7, 10–11, 13–14), with the 1010th percentile and the worst site in Table 10. In simulation, pooled methods leave 22–40% of sites under-covered, while the site-level rule never violates its target (Table 7). On data it reduces under-covered sites from 40% to 7% (b=6b=6) and 2% (b=12b=12) on ABIDE, from 27–31% to 7.5% and 3% on TMS, and from 43–46% to below 1% on ACS. Scored exactly against all 2424 ABIDE sites and 4444 TMS donors (Appendix K), the site-level rule leaves more than its certified fraction of sites under-covered with probability 0.0120.012–0.0790.079, below δ=0.1\delta=0.1, whereas for every finite pooled rule more than 10% of sites are under-covered with probability at least 0.360.36.

W4 (presentation). The notation of §1 is now defined at first use, the new Figure 1 maps both calibration routes, and an ABIDE running example follows the main results.

W5 (baselines). We now benchmark against M1–M3 of Dunn et al. and HCP and HCP2 of Lee et al. on ABIDE, TMS and 51-state ACS (Figure 3; Tables 2–3 and 10–14). Our two procedures are the only ones that are PAC-valid with finite sets in all six settings, on held-out splits and under exact evaluation. M1 fails everywhere. M3, HCP and HCP2 return the whole line at b=6b=6, and M3 and HCP fail at b=28b=28 (25–26% of splits, 18% under exact evaluation). On ACS our pooled rule is at least 12% narrower than any other valid method.

Q1 (the 22-cluster floor of Proposition 2 versus b=6b=6 and 1111). There is no contradiction, as §3.1 now states. Proposition 2 is a worst case over all within-cluster dependence structures, attained by comonotone dependence (ρind≡1\rho_{\mathrm{ind}}\equiv 1, where the nn points collapse to bb values). Below 2222 clusters no dependence-agnostic rule can be valid for every structure: M2 fails at b=6,11,12,13b=6, 11, 12, 13 and succeeds only at b=28b=28 and 3131 (Tables 2–3, 11, 13–14). When dependence is weaker than comonotone (ρ^ind≈0.15\widehat{\rho}_{\mathrm{ind}} \approx 0.15 on ABIDE and 0.400.40 on TMS), a rule that estimates ρind\rho_{\mathrm{ind}} uses the within-cluster sample (neff≈16n_{\mathrm{eff}} \approx 16 and 3030) and achieves PAC validity below the worst-case floor.

Thank you for the additional experiments and clarifications. Some of my concerns have been addressed, though I would like to keep my original score for the following reasons:

  1. Despite the improvements, I think there are still clarity issues with the paper for non-experts, especially in the introduction and exposition to the theory and methods.

  2. The revision introduces substantial new theory, especially the site-level guarantees. I think the revised paper would benefit from another full review cycle to assess the new results and their implications, especially with new reviewers who are able to check the math in detail.

  3. The empirical results are mixed with the addition of the new baselines. Specifically, HCP outperforms the proposed methods in ABIDE with 12 calibration sites and Tabula Muris Senis with 11 donors (i.e. lower violation rate, fewer under-covered sites, or narrower intervals).

AuthorsOpenReview ↗

Factual clarifications on the three points

Thank you for the follow-up. We note three factual points regarding this assessment.

1. Accessibility for non-experts. The revision puts the practical content before the proofs: Figure 1 lays out both calibration procedures and their cluster floors on page 2, §2 grounds (A1)–(A4) in the two-level model of multi-site sampling used by Dunn et al. (2023) and Lee et al. (2025), the notation of §1 is defined at first use, and concrete ABIDE calculations follow Theorem 1, Remark 1 and Propositions 2–3, translating each bound into effective sample sizes and site counts.

2. Scope and verifiability of §5. The results, assumptions and proofs of the submission (Propositions 1–2, Theorems 1–3, Corollaries 1–4) are retained, with Proposition 2 now noting that its bound is attained by comonotone dependence (Q1). The new theory is confined to §5 and was added in direct response to Weakness 2, which requested per-site guarantees. Its main result, Proposition 3, needs no new technical machinery: the complete proof in Appendix E is under half a page and reduces in three elementary steps to a within-site binomial quantile bound, independence across sites, and the classical Beta law of an order statistic (Wilks, 1941; Vovk, 2013). Its necessity counterpart, Corollary 5, is Proposition 2 applied to site quantiles. Its finite-sample validity is also confirmed numerically: the rule never exceeds its target in simulation (Table 7), and against the exact ABIDE and TMS site populations it exceeds its certified fraction with probability 0.0120.012–0.0790.079, below δ=0.1\delta = 0.1 (Appendix K).

3. HCP across cluster counts. HCP targets marginal rather than PAC coverage; whether it meets a PAC target is an artifact of the cluster count KK, not a calibrated property of the procedure. Its mass 1/(K+1)1/(K+1) at +∞+\infty inflates its effective quantile level on the finite scores to (1−α)(K+1)/K(1-\alpha)(K+1)/K. With K=6K=6 ABIDE sites this level is 1.051.05, so HCP returns (−∞,∞)(-\infty, \infty) on every split. With K=11K=11 and 1212 it equals 0.9820.982 and 0.9750.975, an uncalibrated cushion at those two counts. Even there HCP has no per-site control: it leaves 12% of ABIDE test sites under-covered against 2% for our site-level rule (Table 2), matches our site-level rule's width on TMS (19.6319.63 against 19.7519.75 months, Table 3) without certifying donor-level coverage, and under exact evaluation leaves more than 10% of sites under-covered with probability at least 0.360.36 (Appendix K). As KK grows the cushion vanishes: HCP violates the PAC target on 26% of splits with K=28K=28 TMS donors (18% under exact evaluation; Tables 3 and 14), and in simulation its violation rate climbs from 0.020.02 at b=24b=24 to 0.150.15 at b=80b=80 and 0.260.26 at b=160b=160 (Table 7). Under mild clustering on 51-state ACS, our pooled rule is 36% narrower than HCP with 13 states and 12% narrower with 31 (Table 11).

A marginal procedure whose PAC validity cannot be known in advance gives a practitioner no assurance; our two rules are the only methods that maintain PAC validity with finite intervals in all six real-data settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

We thank the three reviewers for their careful reading. Revisions in the updated PDF are marked in blue, and the main text remains within the 10-page limit.

Site-level guarantee (new §5, Proposition 3, Corollary 5; Reviewer Noy8). No procedure that returns finite prediction sets can guarantee coverage at every conceivable site (Remark 2). We therefore add an exact finite-sample rule that certifies site-conditional coverage over a prescribed fraction of the site population: with probability at least 1−δ1-\delta, at least a 1−γ1-\gamma fraction of new sites receive site-conditional coverage ≥1−α\ge 1-\alpha. Each calibration site contributes one binomial upper confidence bound on its own (1−α)(1-\alpha) quantile, and Vovk's PAC index (with Poisson-binomial tails for unequal site sizes) is applied across the bb bounds. The guarantee holds in finite samples under arbitrary between-site heterogeneity, and its requirement b≥log⁡δ/log⁡(1−γ)b \ge \log\delta / \log(1-\gamma) is necessary (Corollary 5).

Jackknife bound for the pooled rule (§6, Appendix D, Table 6; Reviewer BYzL). Following Reviewer BYzL's question on the link between Theorem 1 and the practical rule, Table 6 isolates each of the three approximations in §6. The cluster-bootstrap upper bound on ρind\rho_{\mathrm{ind}} was optimistic at small bb. A leave-one-cluster-out jackknife bound, which is deterministic and needs bb evaluations instead of hundreds of resamples, removes this optimism at almost the same interval width. The pooled correction now meets the PAC target δ=0.1\delta=0.1 at every cluster count from b=6b=6 upward in simulation (0.0870.087–0.0980.098 in Table 1, against 0.1120.112–0.1180.118 before for b≤20b\le 20) and brings Tabula Muris Senis (TMS) within Monte Carlo tolerance (0.1150.115, binomial p=0.27p=0.27).

Baselines, a census benchmark and per-site metrics (§7, Figure 3, Tables 2–3 and 7–14; Reviewers Noy8 and BYzL). We implemented the three methods of Dunn et al. (2023) (M1 CDF pooling, M2 one draw per cluster, M3 repeated subsampling) and HCP and HCP2 of Lee et al. (2025), and we report pooled and per-site validity in six real-data settings: ABIDE with b=6b=6 and 1212 calibration sites (Table 2), TMS with b=11b=11 and 2828 donors (Table 3), and ACSIncome over all 51 U.S. states with b=13b=13 and 3131 calibration states (§7.4, Tables 11–12). Appendix K also scores every rule exactly against the real site populations (24 ABIDE sites, 44 TMS donors, 51 states), free of test-site noise. Tables 7 and 9 add heavy-tailed site heterogeneity and shifted test sites in simulation.

Our two procedures are the only methods that meet the PAC target with finite intervals in all six settings (Figure 3). The pooled correction has violation rates 0.055,0.102,0.115,0.110,0.053,0.0730.055, 0.102, 0.115, 0.110, 0.053, 0.073 and the site-level rule 0.015,0.000,0.055,0.005,0.000,0.0000.015, 0.000, 0.055, 0.005, 0.000, 0.000; under exact evaluation on ABIDE and TMS these are 0.0830.083–0.1010.101 and at most 0.0250.025. At b=6b=6, M3, HCP and HCP2 return the whole line, because one-sided marginal sets need at least 99 clusters (9999 for HCP2). At b=28b=28, the 1/(K+1)1/(K+1) finite-sample slack that makes M3 and HCP conservative at b=11b=11–1212 vanishes, and they violate the target on 25–26% of splits. On ACS, where clustering is mild, the pooled correction is within 5% of the population quantile and at least 12% narrower than every other valid method. The site-level rule reduces the share of under-covered sites from 27–46% under nominal i.i.d. calibration to 0–7.5% on every dataset, and on the real site populations it leaves more than its certified fraction of sites under-covered with probability 0.0120.012–0.0790.079, below δ\delta.

Model, diagnostics and readability (§1–§2, Figure 1, Appendices G–H; all reviewers). We rewrote the abstract in plain language, define the notation of §1 at first use, added a workflow diagram (Figure 1) and an ABIDE running example, state the two-level generative model explicitly in §2, and check the assumptions on cross-fitted residuals in Appendix G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.