- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the recall ceiling of sampling-based conformal prediction for language models: a prediction set restricted to generated answers cannot recover an ungenerated ground truth. It introduces a hybrid nonconformity score combining empirical generation frequency with encoder-space discrepancies, yielding an implicit prediction set with split-conformal marginal coverage under exchangeability. A finite vocabulary makes additional answers enumerable, with coverage then depending on vocabulary omissions. The paper also learns an overlap-aware ordering of discrepancy cells to reduce explicit set size. Evaluation covers five short-answer QA datasets, two language models, sampling-budget and coordinate ablations, and full versus recoverable populations.
Critical strengths
-
The problem formulation and guarantee boundaries are clear. Sections 3–4 distinguish implicit membership coverage from explicit vocabulary-restricted enumeration. Independent cell construction and recalibration preserve the conformal argument, while Appendix A separately accounts for vocabulary omissions. This is a meaningful extension beyond the generation-recall ceiling rather than an unsupported claim that arbitrary text can be enumerated.
-
The empirical analysis makes important failure modes inspectable. The experiments cover ten dataset–model combinations, and Appendices D and F–I report computational costs, vocabulary pruning, component ablations, full-population results across target levels, and an explicit limitations discussion. Reporting the large set sizes and vocabulary failures is particularly valuable for evaluating the practical consequences of the guarantee.
Critical weaknesses
-
Useful explicit prediction sets are not yet established on the original task distributions. At a nominal 90% target, Table 5 reports 53.22% full-population coverage for the cell-based explicit sets; Table 6 reports an average of 10,376.87 answers, with only 1.27% size reduction from cell ordering. The stronger 90.01% coverage and 20.02% size reduction concern the recoverable subset, selected according to whether the ground truth was generated or belongs to the vocabulary. This disclosed conditional analysis is legitimate, but does not establish comparable utility on the full population. The paper acknowledges this limitation in its limitations discussion (Appendix I), and the shortfall reflects vocabulary omissions rather than a calibration failure; the hybrid set's headline 90.25% coverage (abstract) is likewise reported without any size or informativeness measure, since that set is not enumerated. The concern is therefore about the sufficiency of the demonstrated contribution, not its correctness. Moreover, the principal external baselines are restricted to sampled answers, whereas the proposed method accesses an additional vocabulary. The norm-semantic control is useful, but does not isolate sequential overlap-aware ordering from static cell ranking or simpler calibrated candidate-ranking alternatives. Comparisons using the same vocabulary, representation and data/compute budget would establish the incremental contribution more clearly. These gaps limit the demonstrated significance of the method; they do not refute the implicit coverage theorem.
-
The radial-coordinate result needs a reproducible implementation explanation. Equation 5 depends on differences in embedding norms, and Table 4 attributes a substantial benefit to combining radial and angular coordinates. However, the paper names all-mpnet-base-v2 as the encoder (p. 8), a model distributed as a SentenceTransformer whose configuration ends with a Normalize module, so its default outputs have unit norm and the radial coordinate would be essentially zero. Yet the radial-only variants in Tables 3–4 are far from degenerate (e.g., 6,317 of 30,161 vocabulary entries for Gemma/TriviaQA at about 90% coverage), so the embeddings actually used must have non-trivial norm variation. Pre-normalization extraction or a modified encoder could explain the result, but the extraction point, pooling, normalization and norm distribution are not specified clearly enough to assess this component.
Questions for authors
- Which encoder outputs were used, with what pooling and normalization? What are their norm statistics, and how do they explain the radial-coordinate ablation in Table 4?
- Which existing control most directly isolates the sequential overlap-aware ordering from added vocabulary access and norm-based scalarization? A comparison with static ordering or calibrated ranking over the same candidates would strengthen the acceptance case.
- Which full-population result supports a concrete downstream use case at the reported coverage and set sizes?
8 comments
Response to Weakness 1 (continued)
The stronger 90.01% coverage and 20.02% size reduction concern the recoverable subset, selected according to whether the ground truth was generated or belongs to the vocabulary. This disclosed conditional analysis is legitimate, but does not establish comparable utility on the full population with only 1.27% size reduction from cell ordering.
Our preceding response discusses full-population coverage and the reduction in absolute set size. Here, we focus on the smaller additional size reduction from cell ordering relative to norm scalarization.
The original cell procedure constructs centers from ungenerated ground-truth answers within the vocabulary and rewards their recovery while penalizing incorrect vocabulary answers. This matches the answer availability in the recoverable versions. Full-version recalibration, however, also evaluates ungenerated ground-truth answers outside the vocabulary. An ordering optimized for recovering ground-truth answers within the vocabulary may therefore be less effective at reducing set size when calibrated on the full population.
To test this explanation, we removed the vocabulary restrictions in Eqs. (17) and (20), allowing all ungenerated ground-truth answers to contribute to cell construction and ordering. On the full datasets, the mean set-size reduction relative to norm scalarization reached 6.51%, compared with the originally reported 1.27%. These results support including all ungenerated ground-truth answers in cell construction and ordering when calibrating on the full population, although the improvement remains smaller than the 20.02% reduction reported on the recoverable versions.
The remaining gap may reflect differences in the ground-truth answer distributions between the full and recoverable versions of the same datasets. In particular, the full versions may contain a larger proportion of hard-to-cover ground-truth answers whose discrepancy vectors share cells with many incorrect candidates. Calibrating to cover these answers may therefore admit more candidates at test time, leaving less room for set-size reduction.
the hybrid set's headline 90.25% coverage (abstract) is likewise reported without any size or informativeness measure, since that set is not enumerated.
Although the implicit set does not enumerate all answers it covers, its informativeness can be assessed by how much it narrows a supplied candidate pool. Each candidate can be screened using the implicit set’s inclusion rule without knowing the ground-truth answer. We use the complete vocabulary as a fixed candidate pool to quantify this screening effect. For example, on full SciQ with Llama, norm scalarization with the original statement embeddings excludes an estimated 82.5% of vocabulary answers on average. Direct answer embeddings increase this proportion to 99.1% (Appendix L). At the nominal 90% target, its coverage guarantee limits the marginal probability of rejecting the ground-truth answer to 10% under exchangeability.
Moreover, the principal external baselines are restricted to sampled answers, whereas the proposed method accesses an additional vocabulary.
The comparison with sampling-based baselines measures the benefit of extending prediction sets beyond sampled answers. To isolate the contribution of our scoring method under the same vocabulary access, we additionally evaluate two simpler controls: Euclidean-distance scoring and LM sequence-probability scoring. Both retain cumulative-frequency scoring for sampled answers and use the same external vocabulary for extension.
(1) Euclidean-distance control.
We evaluate Euclidean distance between question–answer statement embeddings as a simpler scoring rule. Each ungenerated answer receives one plus its minimum Euclidean distance to a distinct generated answer. We also evaluate a cell variant that constructs and orders one-dimensional cells from these pairwise distances. Both Euclidean variants use the same generated answers, vocabulary, embeddings, and data splits as their angular–radial counterparts, with independent recalibration. The table reports equal-weight averages across the five datasets for Llama-3.2-3B-Instruct, at =10 and a nominal 90% target. Size increases are relative to angular–radial cell scalarization within each population. At comparable aggregate coverage, angular–radial cell scalarization produces smaller mean sets on both full and recoverable versions. Appendix K provides the complete results, including the norm comparisons.
| Population | Method | Coverage (%) | Size | Size increase |
|---|---|---|---|---|
| Full | Angular–radial: cell | 56.27 | 8,670.40 | — |
| Full | Euclidean: cell | 55.94 | 9,818.37 | +13.24% |
| Recoverable | Angular–radial: cell | 89.67 | 1,996.66 | — |
| Recoverable | Euclidean: cell | 89.91 | 2,864.82 | +43.48% |
Response to Weakness 1 (continued)
(2) LM sequence-probability control.
Another intuitive control ranks vocabulary answers by the probability assigned by the LM to each complete answer. This probability is computed from the product of token-level probabilities conditioned on the question and preceding answer tokens, including answer termination. Across the ten recoverable test configurations, Table 1 reports 36,514 test instances, while Table 2 gives vocabulary sizes of 20,457–30,193 answers. Scoring every vocabulary answer for each question therefore requires approximately 0.75–1.10 billion question–answer likelihood evaluations, excluding calibration costs.
Given this cost, we evaluated Gemma-3-4B-it and Llama-3.2-3B-Instruct on WebQuestions and SciQ using the same vocabulary as angular–radial scoring, =10, and a 90% target. All methods use a single split; parentheses show mean-size increases over angular–radial cell scalarization. LM sequence-probability scoring produces substantially larger sets in all four combinations on both populations.
| Dataset | Model | Method | Full coverage (%) | Full mean size | Recoverable coverage (%) | Recoverable mean size |
|---|---|---|---|---|---|---|
| WebQuestions | Gemma | Angular–radial: cell | 48.09 | 13,743.04 | 89.64 | 4,396.23 |
| WebQuestions | Gemma | LM sequence-probability | 46.28 | 24,538.37 (+78.55%) | 89.38 | 24,232.74 (+451.22%) |
| WebQuestions | Llama | Angular–radial: cell | 53.51 | 9,757.90 | 88.70 | 3,098.30 |
| WebQuestions | Llama | LM sequence-probability | 52.07 | 24,079.07 (+146.76%) | 90.83 | 21,935.69 (+607.99%) |
| SciQ | Gemma | Angular–radial: cell | 66.25 | 6,233.38 | 90.43 | 1,062.65 |
| SciQ | Gemma | LM sequence-probability | 65.20 | 24,294.70 (+289.75%) | 89.23 | 19,497.61 (+1,734.81%) |
| SciQ | Llama | Angular–radial: cell | 68.00 | 4,442.22 | 90.13 | 463.64 |
| SciQ | Llama | LM sequence-probability | 67.43 | 23,348.94 (+425.61%) | 89.82 | 15,085.51 (+3,153.71%) |
(3) Existing coordinate ablations.
Appendix G, Tables 3–4 additionally compare angular-only, radial-only, and joint angular–radial scoring using the same vocabulary on the recoverable versions. Angular-only scoring performs similarly to the joint norm, whereas radial-only scoring produces substantially larger sets. Under cell scalarization, combining both coordinates produces smaller sets than either coordinate alone.
Together, these comparisons support an efficiency benefit from our scoring beyond access to additional vocabulary answers.
The norm-semantic control is useful, but does not isolate sequential overlap-aware ordering from static cell ranking or simpler calibrated candidate-ranking alternatives.
To isolate sequential overlap-aware ordering, we apply the static rule of Tawachi and Laufer-Goldshtein (2025) to the same cell partition, ranking cells once by their initial false-to-true answer ratios. We retain the generated answers, vocabulary, statement embeddings, construction data, and recalibration/test splits, with independent recalibration.
The table reports equal-weight averages across ten dataset–model combinations at =10 and a 90% target. Sequential ordering yields smaller aggregate sets at comparable coverage; complete results appear in Appendix E, Tables E.1–E.2.
| Population | Ordering | Coverage (%) | Mean size | Size increase |
|---|---|---|---|---|
| Full | Sequential (ours) | 53.85 | 10,110.48 | — |
| Full | Static | 53.77 | 10,757.11 | +6.40% |
| Recoverable | Sequential (ours) | 89.52 | 2,603.03 | — |
| Recoverable | Static | 89.64 | 2,903.58 | +11.55% |
Comparing an answer with multiple generated answers can place its discrepancy vectors in different cells. Once any such cell admits the answer, selecting another yields neither additional recovery for a true answer nor additional set-size cost for an incorrect answer. Static counts can therefore overestimate recovery or cost. Our sequential procedure evaluates each candidate cell together with the selected cells, counting each input–answer pair only once.
Thank you for your constructive feedback. Our updated PDF includes additional experiments, referenced in the responses below.
Response to Weakness 1
At the nominal 90% target, explicit cell-based sets achieve 53.22% coverage on the full datasets, compared with 39.61% for the strongest sampling baseline, using the same sampling budget of M=10. This represents an increase of 13.61 percentage points, or 34.36% relative improvement. The mean size of 10,376.87 answers nevertheless motivates improving set efficiency.
(1) Reducing prediction-set size.
Our controlled experiment in Appendix L replaces question–answer statement embeddings with direct answer embeddings. We retain the same sampled outputs, vocabulary, and encoder, and independently recalibrate each variant. Empirical coverage remains comparable (Table 14), while cell-based mean set sizes decrease by 55.77–94.00% on the full versions and 75.81–97.78% on the recoverable versions. Table 12 reports the norm results.
Shared question and template content may obscure differences between answers. Direct answer embeddings remove this shared content, potentially improving the distinction between missed ground-truth answers and incorrect vocabulary answers, consistent with the smaller sets at comparable coverage.
(2) Assessing and improving explicit coverage.
The remaining coverage risk can be assessed before deployment using an independent labeled validation set. We estimate δW, defined in Eq. (15), by checking whether each ground-truth answer is absent from both the model’s generated answers and the vocabulary. Under i.i.d. sampling from the deployment population, its upper confidence bound UW, combined with Eq. (16), gives a conservative marginal-coverage lower bound. Appendix J illustrates this procedure through a retrospective subsampling simulation based on the recorded full-dataset counts.
Reaching the nominal target requires improving answer availability. Appendix F, Table 2 reports availability ceilings below 90% across all evaluated configurations, so scoring alone cannot eliminate the shortfall. Simply enlarging the vocabulary may be inefficient: Table 2 shows that removing 79.4–86.1% of entries reduces the availability ceiling by only 2.35–4.73 percentage points. We therefore evaluated whether a vocabulary better matched to SQ-Wiki could improve coverage without enlarging prediction sets.
We construct candidate categories from WordNet, Wikidata5M, GeoNames, and wordfreq. Using separate development data, we greedily select complete categories according to their additional recovery of missed ground-truth answers relative to added vocabulary size. The resulting 19,996-entry vocabulary is shared by Gemma and Llama and fixed before final fitting, calibration, and testing.
The experiment compares the pruned WordNet vocabulary used in the paper with the newly selected vocabulary. We use direct answer embeddings throughout, keeping the LM, encoder, and M=10 sampled outputs unchanged. The table reports single-split, full-version results for cell scalarization at a nominal 90% target. Arrows indicate pruned WordNet → selected vocabulary.
Dataset-specific vocabulary selection substantially improves explicit coverage with slightly smaller vocabularies and prediction sets, while keeping the LM and sampling budget fixed. By comparison, sampling-based CP relies on improving sampling recall, for example by using a more capable LM or increasing the sampling budget, which can incur additional inference costs.