- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the composition of per-stage risk certificates in LLM pipelines, motivated by the observation that repeatedly searching thresholds over the same calibration data inflates the family-wise error. The authors propose λ-Cert, a type-and-effect discipline in which the calibration budget is a linear, non-duplicable resource in the typing context, the risk effect is the sum of the levels of the risk-bearing leaves, and three data-flow restrictions ensure that every output loss traces to exactly one calibrated leaf. Experiments on Qwen2.5/Qwen2.5-VL across four benchmarks report per-stage and composed certificates, and measure the cost of the budget correction in coverage and certification power.
While reviewing the paper, I found that nearly every positive claim carries two to three trailing qualifying clauses, so the text reads like an audit report rather than an argument.
Critical strengths
-
The calibration budget is a linear resource. The mechanism is the absence of contraction on the linear context , which makes the static ledger equal the runtime test count. That equality is what licenses the Bonferroni composition step.
-
The evaluation covers six text models plus two vision--language models across four benchmarks.
Critical weaknesses
-
The central motivation lacks empirical support. The budget correction was never shown to be necessary for validity on these tasks.
-
The threat model does not hold as a model of the deployment problem. §1 motivates the work with "a generator with retrieval, tool calls, verifiers and aggregators" , but the syntax has only oracle, cgen and agg over the types , , and . There is no retrieval or tool-call node, and no evidence that any real agent satisfies this fragment. In my assessment the threat model fails as an account of why deployed LLM systems cannot be statically certified.
-
The framework assumes calibration and deployment data are i.i.d. from one law. That is the standard conformal assumption, yet the common threats in deployment are distribution shift, prompt-injection contamination of the calibration sample, and calibration sets drawn from a different source than live traffic. The paper models none of these.
-
No experiment deliberately violates the i.i.d. assumption or the loss marker to see whether the certificate fails or declines as the theory predicts. A certification method that works should show an observable failure signal when calibration and deployment distributions diverge, or at least discriminating power in a thin-margin design.
-
The authors titled their own fifth contribution'' a worst- case guarantee whose price is measured and whose bite is not observed'' and state outright that without the correction, none of the 48 configurations puts the family- wise error significantly above its target (§1 C5; §6 RQ4; §7). A motivation refuted by the paper's own experiments sits in the contributions list.
-
RQ3 produced a cost accounting that favors the baseline, and the paper reports it honestly. The result refutes the method's practicality claim.
-
The composition experiments use one task ( GSM8K ) and one gate type ( $m= 8 $ selective self-consistency ). The retrieval, tool calls and verifiers of §1 never appear.
-
Table1 still lists a linear -ledger on the machine-decidable side, which conflicts with §5 placing the budget step among the paper-only arguments. The aggregation rule is never run on real models (Table 3), so the DAG-composition core of the theory has no empirical support.
-
No equation in the paper is numbered, so no derivation step can be located at an addressable position.
-
Two citations do not match the claims they support. "A threshold tuned rather than calibrated states no such bound" cites (Xiao et al., 2025), a time-series anomaly detection paper for the electricity industry. "every merge, whatever fusion it implements, routed through a certifying aggregator" cites (Xu et al., 2026), a survey of knowledge fusion in medical vision-language models. Neither citation supports the claim it is attached to.
Questions for authors
-
What real deployment scenario would make the budget correction genuinely necessary for validity? Can you construct an experiment in which calibration-data reuse inflates the family-wise error so that the uncorrected condition significantly exceeds the δ target while the corrected condition holds it?
-
What does the certificate do under moderate drift between the calibration and deployment distributions? Does the gate decline to certify, or does it issue an invalid certificate? Have you tested any shifted condition?
-
In the dependent-chain experiments, the downstream leaf is re-certified per split on the upstream-accepted subset (Table 3). Please state precisely the exchangeability or i.i.d. condition that licenses this re-calibrated certificate. Does it require the upstream gate's selection event to be independent of the downstream calibration sample, or is same-distribution enough?
Response to Reviewer JRoX
Thank you for the review.
Why the correction matters. The correction is a proof obligation of Theorem 3. Each stage searches a threshold grid on a calibration sample, and each site execution adds one family of tests on it. The union bound over these families needs a leaf certificate per site and Σδ_i ≤ δ_tot. For a well-formed program, the site executions of every finite run number at most its syntactic sites (Lemma 5). The abstract states the claim: the correction costs coverage and certification power rather than validity. RQ4 measures that cost. C5 is now headed "The measured cost of the budget correction" and states the 24 of 64 count.
Question 1: when is the correction needed? It is a proof obligation of Theorem 3: the union bound over the site executions on one calibration sample, under unknown dependence between stages, needs a leaf certificate per site and Σδ_i ≤ δ_tot. The well-formedness check rejects a program with a run that executes more sites than it has syntactic sites; for a program that passes, Theorem 3 gives 1 − δ_tot under its premises. We built the experiment after the review (Appendix F, Table 14): 200 cells, 5,000 splits each. The paper's uncorrected condition (δ/K per test) was significantly above δ = 0.05 in 0 cells (largest family-wise error 0.0010); the corrected condition likewise (0.0006). With no accounting at all (δ per test) it was above δ in 2 cells, 0.0756 and 0.0762 at n_cal = 100 and α = 0.03, where the corrected condition issued no certificate in 5,000 splits.
Threat model and retrieval or tool-call nodes. The syntax is a small core calculus. A retrieval or tool-call stage can be an oracle term, with effect 1 plus its argument's (Lemma 1). A gate or aggregator over it carries its calibrated level.
The i.i.d. premise, and Question 2 on drift. Theorem 3 states its premise: each stage is calibrated on an i.i.d. in-context sample of the inputs reaching it. The certificate is a statement about the law that the calibration sample is drawn from. For a different deployment law, a leaf calibrated on that law composes in the same way, because the typing accepts any calibrated leaf. The type system fixes the accounting; the deployer chooses the calibration law. We tested shifted conditions after the review (Appendix F, Table 15): calibrate on one stored benchmark and deploy on another, 28 pairs, 200 splits each, δ = 0.05. The gate reads only the calibration sample, so it cannot see the shift. At α = 0.05 the gate declined on 3,540 of 5,600 splits and certified on 2,060; under a full shift 1,452 of those certificates (70.5%) were invalid on the target law, against 16 with no shift. Table 15 gives partial shifts and α = 0.08 and 0.10.
Checking the premise and the loss marker. Table 2 shows the discriminating behavior the method is built for. Certificates are issued for GSM8K at four model scales and for ScienceQA at two scales, and they are refused where the vote share does not separate correct from wrong answers.
Practicality (RQ3). RQ3 concerns one fixed pipeline. The claim of the method is different. A leaf certificate is reused wherever the leaf sees the same inputs; a leaf behind a gate is re-certified on the subset that gate accepts, from labels its pool already holds. The composed bound needs no end-to-end labels for the new pipeline.
One task and one gate type. Table 2 reports per-stage certificates on four benchmarks, for six text models and two vision-language models. Theorem 3 holds for any losses in [0,1] and any calibrated gate. The composition runs use GSM8K leaves, where all four Qwen2.5 scales certify.
Table 1, aggregation and equations. Fixed in the revision: Table 1 no longer lists the δ-ledger as machine-decidable. It lists budget conservation, and the caption states that the budget step, Equation (2), is proved on paper only. Fixed in the revision: the four central equations are numbered. The aggregator is part of Theorem 3 and of the mechanized composition step; Table 3's pair calibrates no merge loss.
Citations. Fixed in the revision: the two citations that did not support their sentences are removed.
Question 3: the condition for the downstream certificate. The upstream threshold is fixed by the upstream calibration fold alone. Both downstream samples are split out of the accepted items of the upstream test fold. Given the threshold, acceptance is a function of each item. Theorem 3 assumes independent in-context samples for each stage; in the experiments the folds are drawn without replacement from a fixed pool. The downstream certificate is conditional on the upstream threshold. In a post-review check on the stored records (Appendix F, Table 18), a replay of 400 splits (200 for each of the two chains) finds the upstream calibration fold disjoint from every other fold, and the two downstream folds, both inside the upstream test fold, disjoint from each other.