acceptodds
Under review as a conference paper at ICLR 2027

λ-Cert: A Linear Calibration Budget Rules Out a Pipeline's Certificate Before Any Data Is Collected

Abstract

Conformal calibration certifies one stage of an LLM pipeline, but a deployer needs the guarantee for the whole composition, and reusing calibration data across stages inflates the family-wise error unless it is accounted for. λ-Cert is a type-and-effect discipline whose typing judgment carries per-stage certificates: a linear calibration budget makes such reuse unrepresentable, the effect sums the risk-bearing leaves, and a well-typed feed-forward pipeline satisfies with no end-to-end recalibration. No prior compositional risk-control result we know of has a mechanized composition step: Rocq checks with no axioms that typing and the union bound give at natural-number levels , with the per-node bounds and per-stage certificates as hypotheses; the budget step is proved on paper. An impossibility result marks the scope: for closed loops whose deployment law moves with the selected threshold, no static calibrator meeting a utility floor holds the target, and a pair of such dynamics, each certifiable alone, exists. From the budget arithmetic alone, before any calibration data is collected, the discipline rules out 20 of 64 pipeline configurations. On Qwen2.5 and Qwen2.5-VL over four benchmarks, certificates hold where the confidence signal separates and are declined where it does not, and two further generations reproduce fourteen of the sixteen verdicts. Typing certifies a dependent chain from its two leaf certificates without an end-to-end pass, at a looser bound than an exact end-to-end bound computed from the labels on a fixed-threshold version of the same pipeline, with the typed leaf levels fixed in advance, and only where the recalibrated downstream leaf certifies. The correction costs coverage and certification power rather than validity: in a 3,000-split replication over the same configurations, the uncorrected family-wise error on the test-fold proxy is significantly above its target in one cell, which the corrected condition violates as well.

Primary area:
learning theory
Keywords:
conformal prediction, LLM pipelines, type systems, calibration budget, compositional guarantees, certified risk

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JRoXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the composition of per-stage risk certificates in LLM pipelines, motivated by the observation that repeatedly searching thresholds over the same calibration data inflates the family-wise error. The authors propose λ-Cert, a type-and-effect discipline in which the calibration budget is a linear, non-duplicable resource in the typing context, the risk effect is the sum of the levels of the risk-bearing leaves, and three data-flow restrictions ensure that every output loss traces to exactly one calibrated leaf. Experiments on Qwen2.5/Qwen2.5-VL across four benchmarks report per-stage and composed certificates, and measure the cost of the budget correction in coverage and certification power.

While reviewing the paper, I found that nearly every positive claim carries two to three trailing qualifying clauses, so the text reads like an audit report rather than an argument.

Critical strengths
  1. The calibration budget is a linear resource. The mechanism is the absence of contraction on the linear context Δ\Delta, which makes the static ledger equal the runtime test count. That equality is what licenses the Bonferroni composition step.

  2. The evaluation covers six text models plus two vision--language models across four benchmarks.

Critical weaknesses
  1. The central motivation lacks empirical support. The budget correction was never shown to be necessary for validity on these tasks.

  2. The threat model does not hold as a model of the deployment problem. §1 motivates the work with "a generator with retrieval, tool calls, verifiers and aggregators" , but the syntax has only oracle, cgen and agg over the types Unit\mathsf{Unit}, OO, →\to and ×\times . There is no retrieval or tool-call node, and no evidence that any real agent satisfies this fragment. In my assessment the threat model fails as an account of why deployed LLM systems cannot be statically certified.

  3. The framework assumes calibration and deployment data are i.i.d. from one law. That is the standard conformal assumption, yet the common threats in deployment are distribution shift, prompt-injection contamination of the calibration sample, and calibration sets drawn from a different source than live traffic. The paper models none of these.

  4. No experiment deliberately violates the i.i.d. assumption or the loss marker LL to see whether the certificate fails or declines as the theory predicts. A certification method that works should show an observable failure signal when calibration and deployment distributions diverge, or at least discriminating power in a thin-margin design.

  5. The authors titled their own fifth contribution'' a worst- case guarantee whose price is measured and whose bite is not observed'' and state outright that without the correction, none of the 48 configurations puts the family- wise error significantly above its target (§1 C5; §6 RQ4; §7). A motivation refuted by the paper's own experiments sits in the contributions list.

  6. RQ3 produced a cost accounting that favors the baseline, and the paper reports it honestly. The result refutes the method's practicality claim.

  7. The composition experiments use one task ( GSM8K ) and one gate type ( $m= 8 $ selective self-consistency ). The retrieval, tool calls and verifiers of §1 never appear.

  8. Table1 still lists a linear δ\delta-ledger on the machine-decidable side, which conflicts with §5 placing the budget step among the paper-only arguments. The aggregation rule is never run on real models (Table 3), so the DAG-composition core of the theory has no empirical support.

  9. No equation in the paper is numbered, so no derivation step can be located at an addressable position.

  10. Two citations do not match the claims they support. "A threshold tuned rather than calibrated states no such bound" cites (Xiao et al., 2025), a time-series anomaly detection paper for the electricity industry. "every merge, whatever fusion it implements, routed through a certifying aggregator" cites (Xu et al., 2026), a survey of knowledge fusion in medical vision-language models. Neither citation supports the claim it is attached to.

Questions for authors
  • What real deployment scenario would make the budget correction genuinely necessary for validity? Can you construct an experiment in which calibration-data reuse inflates the family-wise error so that the uncorrected condition significantly exceeds the δ target while the corrected condition holds it?

  • What does the certificate do under moderate drift between the calibration and deployment distributions? Does the gate decline to certify, or does it issue an invalid certificate? Have you tested any shifted condition?

  • In the dependent-chain experiments, the downstream leaf is re-certified per split on the upstream-accepted subset (Table 3). Please state precisely the exchangeability or i.i.d. condition that licenses this re-calibrated certificate. Does it require the upstream gate's selection event to be independent of the downstream calibration sample, or is same-distribution enough?

1 comment
AuthorsOpenReview ↗

Response to Reviewer JRoX

Thank you for the review.

Why the correction matters. The correction is a proof obligation of Theorem 3. Each stage searches a threshold grid on a calibration sample, and each site execution adds one family of tests on it. The union bound over these families needs a leaf certificate per site and Σδ_i ≤ δ_tot. For a well-formed program, the site executions of every finite run number at most its syntactic sites (Lemma 5). The abstract states the claim: the correction costs coverage and certification power rather than validity. RQ4 measures that cost. C5 is now headed "The measured cost of the budget correction" and states the 24 of 64 count.

Question 1: when is the correction needed? It is a proof obligation of Theorem 3: the union bound over the site executions on one calibration sample, under unknown dependence between stages, needs a leaf certificate per site and Σδ_i ≤ δ_tot. The well-formedness check rejects a program with a run that executes more sites than it has syntactic sites; for a program that passes, Theorem 3 gives 1 − δ_tot under its premises. We built the experiment after the review (Appendix F, Table 14): 200 cells, 5,000 splits each. The paper's uncorrected condition (δ/K per test) was significantly above δ = 0.05 in 0 cells (largest family-wise error 0.0010); the corrected condition likewise (0.0006). With no accounting at all (δ per test) it was above δ in 2 cells, 0.0756 and 0.0762 at n_cal = 100 and α = 0.03, where the corrected condition issued no certificate in 5,000 splits.

Threat model and retrieval or tool-call nodes. The syntax is a small core calculus. A retrieval or tool-call stage can be an oracle term, with effect 1 plus its argument's (Lemma 1). A gate or aggregator over it carries its calibrated level.

The i.i.d. premise, and Question 2 on drift. Theorem 3 states its premise: each stage is calibrated on an i.i.d. in-context sample of the inputs reaching it. The certificate is a statement about the law that the calibration sample is drawn from. For a different deployment law, a leaf calibrated on that law composes in the same way, because the typing accepts any calibrated leaf. The type system fixes the accounting; the deployer chooses the calibration law. We tested shifted conditions after the review (Appendix F, Table 15): calibrate on one stored benchmark and deploy on another, 28 pairs, 200 splits each, δ = 0.05. The gate reads only the calibration sample, so it cannot see the shift. At α = 0.05 the gate declined on 3,540 of 5,600 splits and certified on 2,060; under a full shift 1,452 of those certificates (70.5%) were invalid on the target law, against 16 with no shift. Table 15 gives partial shifts and α = 0.08 and 0.10.

Checking the premise and the loss marker. Table 2 shows the discriminating behavior the method is built for. Certificates are issued for GSM8K at four model scales and for ScienceQA at two scales, and they are refused where the vote share does not separate correct from wrong answers.

Practicality (RQ3). RQ3 concerns one fixed pipeline. The claim of the method is different. A leaf certificate is reused wherever the leaf sees the same inputs; a leaf behind a gate is re-certified on the subset that gate accepts, from labels its pool already holds. The composed bound needs no end-to-end labels for the new pipeline.

One task and one gate type. Table 2 reports per-stage certificates on four benchmarks, for six text models and two vision-language models. Theorem 3 holds for any losses in [0,1] and any calibrated gate. The composition runs use GSM8K leaves, where all four Qwen2.5 scales certify.

Table 1, aggregation and equations. Fixed in the revision: Table 1 no longer lists the δ-ledger as machine-decidable. It lists budget conservation, and the caption states that the budget step, Equation (2), is proved on paper only. Fixed in the revision: the four central equations are numbered. The aggregator is part of Theorem 3 and of the mechanized composition step; Table 3's pair calibrates no merge loss.

Citations. Fixed in the revision: the two citations that did not support their sentences are removed.

Question 3: the condition for the downstream certificate. The upstream threshold is fixed by the upstream calibration fold alone. Both downstream samples are split out of the accepted items of the upstream test fold. Given the threshold, acceptance is a function of each item. Theorem 3 assumes independent in-context samples for each stage; in the experiments the folds are drawn without replacement from a fixed pool. The downstream certificate is conditional on the upstream threshold. In a post-review check on the stored records (Appendix F, Table 18), a replay of 400 splits (200 for each of the two chains) finds the upstream calibration fold disjoint from every other fold, and the two downstream folds, both inside the upstream test fold, disjoint from each other.

Reviewer_bFCzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the risk of multiple testing (or pp-hacking) when certifying multi-component AI systems. It proposes an interesting approach that treats reliability as a budget and introduces a call-by-value λ\lambda-calculus with a type-and-effect system. Applying programming-language type systems to statistical guarantees, with calibration budgets treated as linear resources, is a clever idea. The paper establishes theoretical guarantees and conducts empirical evaluations across multiple language and vision-language models on four benchmarks.

Critical strengths

The main strength is the novel formulation of statistical certification as a programming-language type system. The linear calibration budget provides a principled way to track calibration resources and enforce compositional guarantees.

Critical weaknesses
  1. Restricted applicability. The framework is limited to restricted feed-forward DAGs and does not support feedback loops, which are common in iterative LLM agents and many hierarchical orchestrator–subagent systems.

  2. Conservative budget composition. The additive risk bounds and Bonferroni-style confidence accounting are conservative, potentially limiting certification power and practical scalability as pipeline complexity increases.

  3. Limited theoretical novelty. Theorem 3 appears to follow largely from standard union-bound arguments once the structural and per-stage calibration assumptions are established. Theorem 4 is interesting, but its practical implications are less clear.

  4. Limited empirical evidence of practical advantage. The experiments demonstrate compositional certification, but the resulting bounds are often loose and some pipelines rarely receive certificates. The evaluation focuses primarily on end-to-end calibration and budget corrections, with limited comparison to alternative compositional certification methods.

Questions for authors

1. Budget allocation and tightness.

The additive risk bounds and Bonferroni correction can be rather conservative, particularly for pipelines with many components or uneven calibration difficulty. The paper establishes how to enforce a given budget but does not sufficiently explore how that budget should be allocated.

Could the authors investigate optimized allocation of the risk budget αi\alpha_i and confidence budget δi\delta_i across components, rather than relying on fixed allocations? For example, could the allocation account for component-specific calibration sample sizes, certification difficulty, or the probability that a component is reached? Would such optimization meaningfully improve certification power, answer coverage, or bound tightness while preserving the system-level guarantee?

2. Comparison with existing compositional certification methods.

How does λ\lambda-Cert compare with existing approaches to compositional uncertainty quantification and risk control, such as:

  • Ramalingam et al. (2024), Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction, which propagates conformal prediction sets through program composition using abstract interpretation.

  • Kotte (2026), PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines, which avoids stage-wise Bonferroni correction by calibrating a joint nonconformity score.

  • Rahman et al. (2026), Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls, which studies separate risk budgets for different semantic roles in agent tool calls.

Although these methods certify different quantities and may require different assumptions, they address related challenges in composing or allocating statistical guarantees across structured AI systems.

Could the authors clarify the relative advantages of λ\lambda-Cert in terms of guarantee tightness, certification power, calibration cost, and applicability to realistic pipelines? In particular, does the type-system formulation provide benefits beyond automating the accounting of existing statistical guarantees?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bFCz

Thank you for the review.

Restricted applicability. Theorem 4 covers closed-loop dynamics whose law moves with the selected threshold and places the loss adversarially. For these, no static calibrator that meets a utility floor holds the target, once two such dynamics agree on everything a pre-deployment sample sees. The appendix exhibits a certifiable pair of such dynamics. So for such dynamics the feed-forward restriction is a boundary of the problem rather than of the technique. Inside the boundary, the fragment covers pipelines of generators, gates and aggregators with a single consumer per gate output. The paper names on-policy or iterative recalibration under a contractivity assumption as the route for restricted loops.

Conservative budget composition. The additive risk level and the Bonferroni confidence split make the guarantee hold under arbitrary dependence between stages. Both are declared in the type, so the cost of a stage is visible before any calibration. The budget arithmetic rules out 24 of 64 swept configurations without data. Table 3 shows certification on a real chain and on a parallel pair: the pair certifies on 200 of 200 splits at a median composed risk of 0.0067 under the typed bound of 0.10, and the 3B-to-7B chain certifies on 41 of 200 splits with the realized risk under the bound on every one (median 0.0152).

Theoretical novelty. The probability bound in Theorem 3 is a union bound. The contribution is the discipline that makes the union bound applicable to a program. A linear budget plus a feed-forward restriction that uses every bound variable at most once bounds the number of calibration-site executions in each finite run by the number of sites (Lemma 5); Rocq checks this for the core calculus. We know of no usage grading that expresses this resource, and none of the works in Section 8 types the calibration budget. Rocq checks the composition step with no axioms, and we know of no prior compositional risk-control result whose composition step is mechanized. Theorem 4 marks the scope: no static calibrator that always accepts mass ≥ ρ > 0 holds the target on two closed-loop dynamics that agree on all a pre-deployment sample sees and of which, at every such threshold, at least one has risk above the target.

Practical evidence and comparison. Table 2 reports per-stage certificates on four benchmarks, and Table 3 reports the composed results above. Typing composes per-stage certificates without an end-to-end pass: a leaf certificate is reused wherever the leaf sees the same inputs, a leaf behind a gate is re-certified on the subset that gate accepts from labels its pool already holds, and the composed bound needs no end-to-end recalibration.

Question 1: allocation of α_i and δ_i. Theorem 3 holds for any split with Σδ_i ≤ δ_tot and levels α_i that are fixed before the calibration fold is read. A deployer can set them from component fold sizes, from an estimate of the probability that a component is reached made on data other than the calibration fold, or from earlier pilot data. The guarantee then holds for the chosen allocation. An allocation chosen on the calibration outcomes themselves is a further search on that fold, and the linear budget requires it to be counted. After the review we compared allocations on the cached GSM8K chains (Appendix F, Table 16; α_tot = δ_tot = 0.10, 2,000 splits). Of six chains, three certify (3B→7B, 3B→14B, 7B→14B). Certified splits for the fixed 0.5/0.5 split on all 700 items (F700): 399, 414 and 255. On the 490 items a valid adaptive rule leaves for certification, the fixed split (F490) certified on 245, 263 and 329; allocation by calibration size (N-V) on 205, 219 and 244; allocation by estimated reach probability (P-V) on 274, 269 and 299. N-V never beat F490 and P-V beat it on two chains of three. Under these four rules the median realized composed risk stayed between 0.0177 and 0.0276, against the declared 0.10. The discipline checks the accounting; it does not prescribe an allocation.

Question 2: comparison with compositional methods. The three methods certify different objects.

  • Ramalingam et al. propagate prediction sets through a program by abstract interpretation. λ-Cert certifies a risk level for an abstaining pipeline. A set-valued output has no accept event, so the two certify different objects.
  • Kotte (PASC) gives joint coverage across stages for set-valued outputs.
  • Rahman et al. split one risk budget across tool-call roles.

None of them types the calibration budget. λ-Cert adds four things beyond automating the accounting. First, for a well-formed program the number of site executions in each finite run is at most the number of sites (Rocq, core calculus). Second, the composed bound Pr[R(e) ≤ α_top] ≥ 1 − δ_tot needs no end-to-end recalibration. Third, the composition step is mechanized. Fourth, Theorem 4 gives the scope.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' clarifications, particularly regarding the theoretical contributions and budget allocation. However, my concerns about conservative bounds and limited practical advantages remain. I will maintain my original score of 2 (Weak Rejection).

Reviewer_iMmcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

While existing conformal calibration techniques can provide reliability guarantees for an individual model or pipeline stage, those guarantees do not automatically extend to a multi-stage system, especially when calibration evidence is reused across several decisions. This paper introduces λ\lambda-Cert, a type-and-effect system for composing risk certificates across multi-stage LLM pipelines without requiring end-to-end recalibration. They achieve this by treating calibration confidence as a linear resource that cannot be duplicated across different systems. Each pipeline stage carries a certified risk level, and the type system accumulates these into a global risk bound. They only support feed-forward pipelines represented as directed acyclic graphs (DAGs), meaning information can flow forward through a sequence of certified components but cannot form feedback loops. Each component must be individually calibrated and associated with its own confidence budget δ\delta. When two certified branches merge, λ\lambda-Cert requires the merge operation itself to be certified. The confidence budget associated with that certification is added to the accumulated budget of the incoming branches; the risk is not assigned to one branch, but rather combined according to the composition rules of the type system. The overall certificate is then obtained by composing the local certificates, with the global failure probability bounded by the sum of the individual confidence budgets. The authors prove that any well-typed feed-forward pipeline satisfies a composed guarantee of the form Pr⁡[R(e) ≤\le α\alphatop] ≥\ge 1- δ\delta where the overall risk budget is the sum of the individual stage risks and confidence budgets. The authors instantiate λ\lambda-Cert on multi-stage LLM and VLM workflows and empirically measure the certified error thresholds obtained through composition. Their results demonstrate that the framework preserves statistical guarantees across pipeline stages and correctly declines certification in settings where reliable risk control cannot be established. The paper's contribution is a formal type system framework for safely composing existing per-stage certificates into a certificate for an entire LLM pipeline, together with theoretical guarantees and mechanized verification.

Critical strengths
  1. The paper identifies a subtle failure mode where repeated calibration searches can invalidate guarantees and introduces a linear treatment of calibration confidence for tracking confidence budgets through composition.
  2. They develop formal composition rules and provide a mechanized proof in Rocq, substantially increasing confidence in the correctness of the certification theorem.
  3. The paper provides a general composition framework that can leverage existing certification and risk-control methods, making the approach broadly applicable rather than tied to a specific calibration technique.
Critical weaknesses
  1. While the paper develops composition and aggregation rules for general feed-forward DAGs, the empirical evaluation focuses primarily on relatively simple cascaded pipelines. Consequently, the experiments do not convincingly demonstrate the practical benefits of the framework's more sophisticated branching and aggregation operators. Evaluating richer architectures, such as multi-agent debate systems, agentic use-cases with tool use, or retrieval-and-fusion pipelines where multiple reasoning branches are combined, would provide stronger evidence of the utility of these composition rules. Furthermore, the authors claim that the aggregation rule is not run on real models.
  2. The composition procedure may become conservative for larger pipelines, as confidence budgets accumulate across multiple certified components. It would be useful to better understand how quickly this conservativeness grows in realistic deployments involving many stages.
Questions for authors

The framework assumes that the risk and confidence budgets associated with different components can be specified a priori. In practice, choosing and allocating these budgets across a complex pipeline may itself be a challenging design problem, especially when calibration data is sparse.

1 comment
AuthorsOpenReview ↗

Response to Reviewer iMmc

Thank you for the review.

Richer architectures and aggregation. The theory covers feed-forward programs with gates and aggregators. Theorem 3 and the Rocq composition step include both node kinds: the per-node bounds in the capstone cover oracle, gate and aggregator nodes (Mechanization section). On real models, Table 3 runs a parallel pair and dependent chains on GSM8K. The pair answers only when both gates accept. It certifies on 200 of 200 splits at a median composed risk of 0.0067, under the typed bound of 0.10. The 3B-to-7B chain certifies on 41 of 200 splits, and on every one of them the realized risk respects the typed bound (median 0.0152). The pair is a conjunction of two gates; a calibrated merge loss is not run on real models.

A retrieval or tool-call stage can be written as an oracle term, the calculus's unknown-distribution black box, with effect 1 plus its argument's effect (Lemma 1); a gate or aggregator over it carries its calibrated level. For any well-formed program built this way, Theorem 3 gives the composed bound under its premises. A fusion stage follows the same pattern: each branch is a certified leaf, and the merge is a certified aggregator.

Conservativeness for larger pipelines. The bound has two parts, and the type declares both before any data is collected: the risk level is the sum of the leaf levels, and the confidence budget is the sum of the leaf allowances. The confidence part holds under arbitrary dependence between stages, with no independence assumption. Its price in labels is the minimum fold size below. A stage certifies only if the fold has at least ln(δ_i/K)/ln(1−α) items (Analysis section). With δ_i = δ_tot/k, this floor grows with the logarithm of the number of stages k. The same arithmetic rules out 24 of the 64 swept configurations before any data is collected. We recomputed k parallel GSM8K leaves, 700 labels each, on one shared fold per split (post-review; Appendix F, Table 17): all certified on 2,000 of 2,000 splits for k = 1 to 3 and on 1,863, 1,634 and 1,631 for k = 4 to 6 (the 14B leaf, then 3 splits from the sixth leaf); the summed leaf test-fold risks had a median of 0.267 to 0.350 of α_top. A deployer chooses the leaf levels so that their sum meets the target risk.

Choosing budgets a priori. In λ-Cert the budgets are part of the program's type, and the judgment checks them. Theorem 3 holds for any split with Σδ_i ≤ δ_tot that is fixed before the calibration fold is read. A deployer can therefore allocate by fold size, by an estimate of the probability that a stage is reached taken on data other than the calibration fold, or by any other quantity fixed before the fold is read. The minimum-fold-size arithmetic above tells in advance which allocations cannot certify at a given fold size, so sparse calibration data shows up at design time. The discipline fixes the accounting. The allocation is the deployer's choice.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Three points from our replies, as the revised PDF states them

We traced every number in the paper to its artifact and uploaded a revision. The numbers quoted in our earlier replies are unchanged. The revised PDF states three of those points more precisely, and we give them here in the same words.

Dependent chain, 41 of 200 splits, median 0.0152 (replies to iMmc and bFCz). For a chain, Table 3 divides the downstream accepted errors by the size of the downstream test fold. The median of this risk over the 41 certified splits is 0.0152, and it is at or below the typed bound of 0.10 on every one of them.

Allocation rules, composed risk 0.0177 to 0.0276 (reply to bFCz, Question 1). In Table 16 the risk adds the upstream accepted-error risk to the downstream accepted-error risk weighted by upstream reach. The range is the upper median of that sum over certified splits, under the four rules that carry a guarantee.

What refuses a certificate (reply to JRoX). Section 7 gives the cause. Under the archived scoring, four of the five Qwen2.5 and Qwen2.5-VL declines have one cause: unanimous votes are wrong on 42 of 664, 38 of 761, 48 of 884 and 36 of 915 items. As a share of the pool, which is what the certificate bounds, each is below α. The finite-sample correction at δ/K on a calibration fold of 700 refuses these certificates, not the point estimate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.