acceptodds
Under review as a conference paper at ICLR 2027

Beyond Sampled Answers: Conformal Prediction for Language Models via Latent-Space Expansion

Abstract

Conformal prediction (CP) provides prediction sets with finite-sample marginal coverage at level and has recently been applied to language models (LMs). For open-ended generation, sampling-based CP scores calibration examples using the empirical frequency of their ground-truth answers across repeated generations. However, if repeated generation misses the ground-truth answer for more than an fraction of calibration inputs, these methods may fail to attain the target coverage or return the uninformative entire answer space. To recover ground-truth answers missed during generation, we introduce a hybrid nonconformity score that combines empirical generation frequencies with discrepancies measured in an encoder-induced latent space. The hybrid prediction set therefore extends beyond sampled answers through a latent region that permits checking whether an answer is included but does not enumerate the answers it covers. We enumerate the additional answers covered by this region within a finite vocabulary to obtain an explicit semantic prediction set. To reduce set size, we partition the discrepancy space into cells and prioritize those that recover more missed ground-truth answers while admitting fewer incorrect vocabulary answers. Under exchangeability, the hybrid prediction set guarantees marginal coverage of at least . At a target coverage of , its empirical coverage averages 90.25% across five datasets versus 39.59% for sampling-based CP. The semantic prediction set retains this guarantee when the vocabulary contains all missed ground-truth answers.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ce1uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the recall ceiling of sampling-based conformal prediction for language models: a prediction set restricted to generated answers cannot recover an ungenerated ground truth. It introduces a hybrid nonconformity score combining empirical generation frequency with encoder-space discrepancies, yielding an implicit prediction set with split-conformal marginal coverage under exchangeability. A finite vocabulary makes additional answers enumerable, with coverage then depending on vocabulary omissions. The paper also learns an overlap-aware ordering of discrepancy cells to reduce explicit set size. Evaluation covers five short-answer QA datasets, two language models, sampling-budget and coordinate ablations, and full versus recoverable populations.

Critical strengths
  1. The problem formulation and guarantee boundaries are clear. Sections 3–4 distinguish implicit membership coverage from explicit vocabulary-restricted enumeration. Independent cell construction and recalibration preserve the conformal argument, while Appendix A separately accounts for vocabulary omissions. This is a meaningful extension beyond the generation-recall ceiling rather than an unsupported claim that arbitrary text can be enumerated.

  2. The empirical analysis makes important failure modes inspectable. The experiments cover ten dataset–model combinations, and Appendices D and F–I report computational costs, vocabulary pruning, component ablations, full-population results across target levels, and an explicit limitations discussion. Reporting the large set sizes and vocabulary failures is particularly valuable for evaluating the practical consequences of the guarantee.

Critical weaknesses
  1. Useful explicit prediction sets are not yet established on the original task distributions. At a nominal 90% target, Table 5 reports 53.22% full-population coverage for the cell-based explicit sets; Table 6 reports an average of 10,376.87 answers, with only 1.27% size reduction from cell ordering. The stronger 90.01% coverage and 20.02% size reduction concern the recoverable subset, selected according to whether the ground truth was generated or belongs to the vocabulary. This disclosed conditional analysis is legitimate, but does not establish comparable utility on the full population. The paper acknowledges this limitation in its limitations discussion (Appendix I), and the shortfall reflects vocabulary omissions rather than a calibration failure; the hybrid set's headline 90.25% coverage (abstract) is likewise reported without any size or informativeness measure, since that set is not enumerated. The concern is therefore about the sufficiency of the demonstrated contribution, not its correctness. Moreover, the principal external baselines are restricted to sampled answers, whereas the proposed method accesses an additional vocabulary. The norm-semantic control is useful, but does not isolate sequential overlap-aware ordering from static cell ranking or simpler calibrated candidate-ranking alternatives. Comparisons using the same vocabulary, representation and data/compute budget would establish the incremental contribution more clearly. These gaps limit the demonstrated significance of the method; they do not refute the implicit coverage theorem.

  2. The radial-coordinate result needs a reproducible implementation explanation. Equation 5 depends on differences in embedding norms, and Table 4 attributes a substantial benefit to combining radial and angular coordinates. However, the paper names all-mpnet-base-v2 as the encoder (p. 8), a model distributed as a SentenceTransformer whose configuration ends with a Normalize module, so its default outputs have unit norm and the radial coordinate would be essentially zero. Yet the radial-only variants in Tables 3–4 are far from degenerate (e.g., 6,317 of 30,161 vocabulary entries for Gemma/TriviaQA at about 90% coverage), so the embeddings actually used must have non-trivial norm variation. Pre-normalization extraction or a modified encoder could explain the result, but the extraction point, pooling, normalization and norm distribution are not specified clearly enough to assess this component.

Questions for authors
  1. Which encoder outputs were used, with what pooling and normalization? What are their norm statistics, and how do they explain the radial-coordinate ablation in Table 4?
  2. Which existing control most directly isolates the sequential overlap-aware ordering from added vocabulary access and norm-based scalarization? A comparison with static ordering or calibrated ranking over the same candidates would strengthen the acceptance case.
  3. Which full-population result supports a concrete downstream use case at the reported coverage and set sizes?
8 comments

Thank you for your constructive feedback. Our updated PDF includes additional experiments, referenced in the responses below.

 

Response to Weakness 1

At a nominal 90% target, Table 5 reports 53.22% full-population coverage for the cell-based explicit sets; Table 6 reports an average of 10,376.87 answers.

At the nominal 90% target, explicit cell-based sets achieve 53.22% coverage on the full datasets, compared with 39.61% for the strongest sampling baseline, using the same sampling budget of MM=10. This represents an increase of 13.61 percentage points, or 34.36% relative improvement. The mean size of 10,376.87 answers nevertheless motivates improving set efficiency.

(1) Reducing prediction-set size.

Our controlled experiment in Appendix L replaces question–answer statement embeddings with direct answer embeddings. We retain the same sampled outputs, vocabulary, and encoder, and independently recalibrate each variant. Empirical coverage remains comparable (Table 14), while cell-based mean set sizes decrease by 55.77–94.00% on the full versions and 75.81–97.78% on the recoverable versions. Table 12 reports the norm results.

ModelDatasetFull: mean sizeReductionRecoverable: mean sizeReduction
GemmaTriviaQA11,797.03 → 2,944.6375.04%2,208.69 → 53.8797.56%
GemmaSciQ6,233.38 → 678.9489.11%1,062.65 → 26.6097.50%
GemmaNQ-Open11,803.93 → 3,272.4072.28%2,271.31 → 167.4392.63%
GemmaSQ-Wiki14,175.40 → 6,269.3155.77%6,108.10 → 675.0588.95%
GemmaWebQuestions13,743.04 → 5,268.6461.66%4,396.23 → 1,063.4975.81%
LlamaTriviaQA7,086.73 → 706.1190.04%297.77 → 7.1397.61%
LlamaSciQ4,442.22 → 266.6094.00%463.64 → 10.2897.78%
LlamaNQ-Open8,156.80 → 1,647.6579.80%532.26 → 27.0094.93%
LlamaSQ-Wiki13,908.34 → 5,680.6159.16%5,591.34 → 514.9290.79%
LlamaWebQuestions9,757.90 → 3,142.4867.80%3,098.30 → 519.1683.24%

Shared question and template content may obscure differences between answers. Direct answer embeddings remove this shared content, potentially improving the distinction between missed ground-truth answers and incorrect vocabulary answers, consistent with the smaller sets at comparable coverage.

(2) Assessing and improving explicit coverage.

The remaining coverage risk can be assessed before deployment using an independent labeled validation set. We estimate δW\delta_{\mathcal W}, defined in Eq. (15), by checking whether each ground-truth answer is absent from both the model’s generated answers and the vocabulary. Under i.i.d. sampling from the deployment population, its upper confidence bound UWU_{\mathcal W}, combined with Eq. (16), gives a conservative marginal-coverage lower bound. Appendix J illustrates this procedure through a retrospective subsampling simulation based on the recorded full-dataset counts.

Reaching the nominal target requires improving answer availability. Appendix F, Table 2 reports availability ceilings below 90% across all evaluated configurations, so scoring alone cannot eliminate the shortfall. Simply enlarging the vocabulary may be inefficient: Table 2 shows that removing 79.4–86.1% of entries reduces the availability ceiling by only 2.35–4.73 percentage points. We therefore evaluated whether a vocabulary better matched to SQ-Wiki could improve coverage without enlarging prediction sets.

We construct candidate categories from WordNet, Wikidata5M, GeoNames, and wordfreq. Using separate development data, we greedily select complete categories according to their additional recovery of missed ground-truth answers relative to added vocabulary size. The resulting 19,996-entry vocabulary is shared by Gemma and Llama and fixed before final fitting, calibration, and testing.

The experiment compares the pruned WordNet vocabulary used in the paper with the newly selected vocabulary. We use direct answer embeddings throughout, keeping the LM, encoder, and MM=10 sampled outputs unchanged. The table reports single-split, full-version results for cell scalarization at a nominal 90% target. Arrows indicate pruned WordNet → selected vocabulary.

ModelVocabulary entriesCoverage (%)Mean set size
Gemma20,457 → 19,99644.84 → 60.336,269.31 → 6,123.5
Llama21,149 → 19,99646.83 → 61.725,680.61 → 5,654.5

Dataset-specific vocabulary selection substantially improves explicit coverage with slightly smaller vocabularies and prediction sets, while keeping the LM and sampling budget fixed. By comparison, sampling-based CP relies on improving sampling recall, for example by using a more capable LM or increasing the sampling budget, which can incur additional inference costs.

Response to Weakness 1 (continued)

The stronger 90.01% coverage and 20.02% size reduction concern the recoverable subset, selected according to whether the ground truth was generated or belongs to the vocabulary. This disclosed conditional analysis is legitimate, but does not establish comparable utility on the full population with only 1.27% size reduction from cell ordering.

Our preceding response discusses full-population coverage and the reduction in absolute set size. Here, we focus on the smaller additional size reduction from cell ordering relative to norm scalarization.

The original cell procedure constructs centers from ungenerated ground-truth answers within the vocabulary and rewards their recovery while penalizing incorrect vocabulary answers. This matches the answer availability in the recoverable versions. Full-version recalibration, however, also evaluates ungenerated ground-truth answers outside the vocabulary. An ordering optimized for recovering ground-truth answers within the vocabulary may therefore be less effective at reducing set size when calibrated on the full population.

To test this explanation, we removed the vocabulary restrictions in Eqs. (17) and (20), allowing all ungenerated ground-truth answers to contribute to cell construction and ordering. On the full datasets, the mean set-size reduction relative to norm scalarization reached 6.51%, compared with the originally reported 1.27%. These results support including all ungenerated ground-truth answers in cell construction and ordering when calibrating on the full population, although the improvement remains smaller than the 20.02% reduction reported on the recoverable versions.

The remaining gap may reflect differences in the ground-truth answer distributions between the full and recoverable versions of the same datasets. In particular, the full versions may contain a larger proportion of hard-to-cover ground-truth answers whose discrepancy vectors share cells with many incorrect candidates. Calibrating to cover these answers may therefore admit more candidates at test time, leaving less room for set-size reduction.

 

the hybrid set's headline 90.25% coverage (abstract) is likewise reported without any size or informativeness measure, since that set is not enumerated.

Although the implicit set does not enumerate all answers it covers, its informativeness can be assessed by how much it narrows a supplied candidate pool. Each candidate can be screened using the implicit set’s inclusion rule without knowing the ground-truth answer. We use the complete vocabulary as a fixed candidate pool to quantify this screening effect. For example, on full SciQ with Llama, norm scalarization with the original statement embeddings excludes an estimated 82.5% of vocabulary answers on average. Direct answer embeddings increase this proportion to 99.1% (Appendix L). At the nominal 90% target, its coverage guarantee limits the marginal probability of rejecting the ground-truth answer to 10% under exchangeability.

 

Moreover, the principal external baselines are restricted to sampled answers, whereas the proposed method accesses an additional vocabulary.

The comparison with sampling-based baselines measures the benefit of extending prediction sets beyond sampled answers. To isolate the contribution of our scoring method under the same vocabulary access, we additionally evaluate two simpler controls: Euclidean-distance scoring and LM sequence-probability scoring. Both retain cumulative-frequency scoring for sampled answers and use the same external vocabulary for extension.

(1) Euclidean-distance control.

We evaluate Euclidean distance between question–answer statement embeddings as a simpler scoring rule. Each ungenerated answer receives one plus its minimum Euclidean distance to a distinct generated answer. We also evaluate a cell variant that constructs and orders one-dimensional cells from these pairwise distances. Both Euclidean variants use the same generated answers, vocabulary, embeddings, and data splits as their angular–radial counterparts, with independent recalibration. The table reports equal-weight averages across the five datasets for Llama-3.2-3B-Instruct, at MM=10 and a nominal 90% target. Size increases are relative to angular–radial cell scalarization within each population. At comparable aggregate coverage, angular–radial cell scalarization produces smaller mean sets on both full and recoverable versions. Appendix K provides the complete results, including the norm comparisons.

PopulationMethodCoverage (%)SizeSize increase
FullAngular–radial: cell56.278,670.40—
FullEuclidean: cell55.949,818.37+13.24%
RecoverableAngular–radial: cell89.671,996.66—
RecoverableEuclidean: cell89.912,864.82+43.48%

Response to Weakness 1 (continued)

(2) LM sequence-probability control.

Another intuitive control ranks vocabulary answers by the probability assigned by the LM to each complete answer. This probability is computed from the product of token-level probabilities conditioned on the question and preceding answer tokens, including answer termination. Across the ten recoverable test configurations, Table 1 reports 36,514 test instances, while Table 2 gives vocabulary sizes of 20,457–30,193 answers. Scoring every vocabulary answer for each question therefore requires approximately 0.75–1.10 billion question–answer likelihood evaluations, excluding calibration costs.

Given this cost, we evaluated Gemma-3-4B-it and Llama-3.2-3B-Instruct on WebQuestions and SciQ using the same vocabulary as angular–radial scoring, MM=10, and a 90% target. All methods use a single split; parentheses show mean-size increases over angular–radial cell scalarization. LM sequence-probability scoring produces substantially larger sets in all four combinations on both populations.

DatasetModelMethodFull coverage (%)Full mean sizeRecoverable coverage (%)Recoverable mean size
WebQuestionsGemmaAngular–radial: cell48.0913,743.0489.644,396.23
WebQuestionsGemmaLM sequence-probability46.2824,538.37 (+78.55%)89.3824,232.74 (+451.22%)
WebQuestionsLlamaAngular–radial: cell53.519,757.9088.703,098.30
WebQuestionsLlamaLM sequence-probability52.0724,079.07 (+146.76%)90.8321,935.69 (+607.99%)
SciQGemmaAngular–radial: cell66.256,233.3890.431,062.65
SciQGemmaLM sequence-probability65.2024,294.70 (+289.75%)89.2319,497.61 (+1,734.81%)
SciQLlamaAngular–radial: cell68.004,442.2290.13463.64
SciQLlamaLM sequence-probability67.4323,348.94 (+425.61%)89.8215,085.51 (+3,153.71%)

(3) Existing coordinate ablations.

Appendix G, Tables 3–4 additionally compare angular-only, radial-only, and joint angular–radial scoring using the same vocabulary on the recoverable versions. Angular-only scoring performs similarly to the joint norm, whereas radial-only scoring produces substantially larger sets. Under cell scalarization, combining both coordinates produces smaller sets than either coordinate alone.

Together, these comparisons support an efficiency benefit from our scoring beyond access to additional vocabulary answers.

 

The norm-semantic control is useful, but does not isolate sequential overlap-aware ordering from static cell ranking or simpler calibrated candidate-ranking alternatives.

To isolate sequential overlap-aware ordering, we apply the static rule of Tawachi and Laufer-Goldshtein (2025) to the same cell partition, ranking cells once by their initial false-to-true answer ratios. We retain the generated answers, vocabulary, statement embeddings, construction data, and recalibration/test splits, with independent recalibration.

The table reports equal-weight averages across ten dataset–model combinations at MM=10 and a 90% target. Sequential ordering yields smaller aggregate sets at comparable coverage; complete results appear in Appendix E, Tables E.1–E.2.

PopulationOrderingCoverage (%)Mean sizeSize increase
FullSequential (ours)53.8510,110.48—
FullStatic53.7710,757.11+6.40%
RecoverableSequential (ours)89.522,603.03—
RecoverableStatic89.642,903.58+11.55%

Comparing an answer with multiple generated answers can place its discrepancy vectors in different cells. Once any such cell admits the answer, selecting another yields neither additional recovery for a true answer nor additional set-size cost for an incorrect answer. Static counts can therefore overestimate recovery or cost. Our sequential procedure evaluates each candidate cell together with the selected cells, counting each input–answer pair only once.

Reviewer_HPzfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper combines generation frequencies with latent-space discrepancies to expand prediction sets beyond sampled answers.
  • Existing sampling-based conformal prediction (CP) methods cannot include ground-truth answers that the LLM fails to generate in the first place.
Critical strengths
  • The paper addresses an important limitation of existing sampling-based conformal prediction methods, particularly when LLMs fail to generate correct answers.

  • The proposed approach is intuitive and provides clear theoretical coverage guarantees under exchangeability.

  • The method substantially improves empirical coverage over existing baselines.

Critical weaknesses
  • The practical utility of the prediction sets is highly limited. At the 90% target, the average semantic prediction set contains approximately 10,377 answers on the full datasets, while empirical coverage reaches only 53.22%. Such large sets are difficult to use for actual question answering.

  • There is a substantial gap between the theoretical guarantees and the practical output. The 90% coverage guarantee applies to the implicit hybrid set. The explicit semantic set depends on vocabulary coverage and achieves approximately 90% coverage only on the recoverable subset, which excludes examples whose answers are absent from both generated outputs and the vocabulary.

  • The applicability of the method is limited to short-form question answering. All experiments focus on tasks with short, well-defined answers, and the method relies on a predefined WordNet-based vocabulary. Its applicability to free-form text generation therefore remains unclear.

Questions for authors
  • In what practical applications would prediction sets containing thousands of answers be useful? Could the authors provide ways to achieve smaller sets?

  • How the method provide guaranteed coverage for explicit semantic prediction sets without relying on the assumption that the ground-truth answer is contained in a predefined vocabulary?

  • Although it could be an out-of-scope suggestion, could the proposed latent-space expansion be extended to free-form text generation, where a finite answer vocabulary is generally unavailable?

10 comments

Thank you for your constructive feedback. We have uploaded an updated PDF with additional experiments, referenced below. For ease of reading, we place our response to each question immediately after the response to its corresponding weakness.

 

Response to Weakness 1

The practical utility of the prediction sets is highly limited. At the 90% target, the average semantic prediction set contains approximately 10,377 answers on the full datasets, while empirical coverage reaches only 53.22%. Such large sets are difficult to use for actual question answering.

We agree that prediction sets containing approximately 10,377 answers are difficult to use directly as final QA outputs. We address this concern through experiments on set size and vocabulary selection, together with an analysis of the coverage limits imposed by answer availability.

(1) Prediction-set size reduction by direct answer embeddings.

We replace query-conditioned statement embeddings with embeddings of the answers themselves. The detailed experimental setting is provided in Appendix L. At MM=10, we keep the generated answers, vocabulary, encoder, and data splits fixed and independently recalibrate each variant. The table compares mean cell-based set sizes using statement → direct answer embeddings and reports the corresponding percentage reductions. Empirical coverage remains comparable as shown in Table 14. Corresponding results for norm scalarization are provided in Table 12.

ModelDatasetFull: mean sizeReductionRecoverable: mean sizeReduction
GemmaTriviaQA11,797.03 → 2,944.6375.04%2,208.69 → 53.8797.56%
GemmaSciQ6,233.38 → 678.9489.11%1,062.65 → 26.6097.50%
GemmaNQ-Open11,803.93 → 3,272.4072.28%2,271.31 → 167.4392.63%
GemmaSQ-Wiki14,175.40 → 6,269.3155.77%6,108.10 → 675.0588.95%
GemmaWebQuestions13,743.04 → 5,268.6461.66%4,396.23 → 1,063.4975.81%
LlamaTriviaQA7,086.73 → 706.1190.04%297.77 → 7.1397.61%
LlamaSciQ4,442.22 → 266.6094.00%463.64 → 10.2897.78%
LlamaNQ-Open8,156.80 → 1,647.6579.80%532.26 → 27.0094.93%
LlamaSQ-Wiki13,908.34 → 5,680.6159.16%5,591.34 → 514.9290.79%
LlamaWebQuestions9,757.90 → 3,142.4867.80%3,098.30 → 519.1683.24%

Cell-based set sizes decrease by 55.77–94.00% on the full versions and 75.81–97.78% on the recoverable versions. Shared question and template content may obscure differences between answers; direct answer embeddings remove this shared content. The observed reductions are consistent with this explanation, although some configurations still yield large sets.

(2) Coverage assessment and improvement.

In the original full-dataset evaluation, explicit cell-based sets achieve 53.22% coverage, compared with 39.57–39.61% for the sampling baselines. This is a gain of approximately 13.6 percentage points, or a 34.4% relative improvement, with the same LM sampling budget. This demonstrates substantial recovery beyond the generated answers.

The potential coverage shortfall can be assessed before deployment through δW\delta_{\mathcal W}, defined in Eq. (15) as the probability that the ground-truth answer is neither generated nor contained in the vocabulary. Eq. (16) guarantees explicit coverage of at least 1−α−δW1-\alpha-\delta_{\mathcal W}. An independent labeled validation set can be used to estimate δW\delta_{\mathcal W} and obtain its upper confidence bound UWU_{\mathcal W}. Substituting UWU_{\mathcal W} into Eq. (16) gives a conservative coverage lower bound. Appendix J illustrates this procedure through a retrospective subsampling simulation based on the recorded full-dataset counts.

We can also assess δW\delta_{\mathcal W} without LM inference by checking whether the ground-truth answers in labeled validation data belong to W\mathcal W. The fraction outside the vocabulary estimates Pr⁡(Y∉W)\Pr(Y\notin\mathcal W), which upper-bounds δW\delta_{\mathcal W}. By comparison, directly estimating sampling recall for sampling-based CP on new validation inputs requires additional LM inference.

The absolute coverage must also be interpreted relative to answer availability. Appendix F, Table 2 reports availability ceilings below 90% across all evaluated dataset–model combinations. Even returning every generated answer and vocabulary entry would fall short of the target, so improving the scoring rule alone cannot eliminate this shortfall.

Simply enlarging the vocabulary may be inefficient: Table 2 shows that removing 79.4–86.1% of entries reduces the availability ceiling by only 2.35–4.73 percentage points. We therefore evaluated whether a vocabulary better matched to SQ-Wiki could improve coverage without enlarging prediction sets.

Response to Weakness 1 (continued)

We construct candidate categories from WordNet, Wikidata5M, GeoNames, and wordfreq. On separate development data, we greedily select complete categories by additional missed-answer recovery per added vocabulary entry. The resulting 19,996-entry vocabulary is shared by Gemma and Llama and fixed before final fitting, calibration, and testing.

We compare the paper’s pruned WordNet vocabulary with the selected vocabulary using direct answer embeddings, with the same LM, encoder, and MM=10 outputs. The table reports single-split full-version cell results at a 90% target; arrows indicate pruned WordNet → selected vocabulary.

ModelVocabulary entriesCoverage (%)Mean set size
Gemma20,457 → 19,99644.84 → 60.336,269.31 → 6,123.5
Llama21,149 → 19,99646.83 → 61.725,680.61 → 5,654.5

These results show that dataset-specific vocabulary selection substantially improves explicit coverage with slightly smaller vocabularies and prediction sets, while keeping the LM and sampling budget fixed. Methods restricted to sampled answers instead depend on improving sampling recall.

 

Response to Question 1

In what practical applications would prediction sets containing thousands of answers be useful?

Large prediction sets could serve as intermediate candidate pools for automated evidence checking. In document-grounded QA, candidate answers could be matched against retrieved passages before further verification; in knowledge-base QA, they could be filtered using entity-type and relation constraints.

Could the authors provide ways to achieve smaller sets?

Our response to Weakness 1 demonstrates smaller sets using direct answer embeddings. For sets that remain large, retrieving a question-specific vocabulary subset could exclude irrelevant answers, reducing enumeration cost and set size. Another direction is to improve the encoder or identify informative discrepancy coordinates using independent development data, helping cell scalarization distinguish missed ground-truth answers from incorrect vocabulary answers and admit fewer incorrect answers. Both directions require evaluation at comparable coverage.

 

Response to Weakness 2

There is a substantial gap between the theoretical guarantees and the practical output. The 90% coverage guarantee applies to the implicit hybrid set. The explicit semantic set depends on vocabulary coverage and achieves approximately 90% coverage only on the recoverable subset, which excludes examples whose answers are absent from both generated outputs and the vocabulary.

The explicit semantic set has a population-level guarantee accounting for unavailable answers. Under exchangeability, Eq. (16) and the availability ceiling in Section 5.1 give:

1−α−δW≤Pr⁡(Y∈CρY→W)≤1−δW.1-\alpha-\delta_{\mathcal W} \leq \Pr(Y\in\mathcal C_{\rho}^{\mathcal Y\to\mathcal W}) \leq 1-\delta_{\mathcal W}.

Thus, at the nominal 90% target, the explicit set’s population coverage is guaranteed to be within 10 percentage points of the maximum attainable coverage using the generated answers and vocabulary. This guarantee applies to the full population, including unrecoverable examples.

Recoverable experiments examine δW=0\delta_{\mathcal W}=0, measuring coverage and size when answers are available. Separately calibrated full-version experiments assess performance with omissions. These settings evaluate different regimes of the explicit-set guarantee.

 

Response to Question 2

How the method provide guaranteed coverage for explicit semantic prediction sets without relying on the assumption that the ground-truth answer is contained in a predefined vocabulary?

Explicit semantic sets can retain the nominal target coverage 1−α1-\alpha even when δW>0\delta_{\mathcal W}>0, provided vocabulary restriction introduces no additional coverage failures. Appendix A (lines 711–712 and 725–737) characterizes a sufficient condition under which coverage is preserved despite vocabulary omissions.

Let Escore\mathcal E_{\mathrm{score}} denote the implicit set’s coverage-failure event and EW\mathcal E_{\mathcal W} the event that the ground-truth answer is neither generated nor contained in the vocabulary. Explicit coverage fails on Escore∪EW\mathcal E_{\mathrm{score}}\cup \mathcal E_{\mathcal W}, so vocabulary restriction introduces additional failures only on EW∖Escore\mathcal E_{\mathcal W}\setminus \mathcal E_{\mathrm{score}}. If EW⊆Escore\mathcal E_{\mathcal W}\subseteq \mathcal E_{\mathrm{score}}, these additional failures disappear, and $\Pr(\mathcal E_{\mathrm{score}}\cup \mathcal E_{\mathcal W}) = \Pr(\mathcal E_{\mathrm{score}}) \leq \alpha. $ Thus, explicit coverage remains at least 1−α1-\alpha, even when δW>0\delta_{\mathcal W}>0. This overlap condition is not automatically ensured by calibration.

Without such control of additional failures, the general guarantee in Eq. (16) applies (see our response to Weakness 2).

Response to Weakness 3

The applicability of the method is limited to short-form question answering. All experiments focus on tasks with short, well-defined answers, and the method relies on a predefined WordNet-based vocabulary. Its applicability to free-form text generation therefore remains unclear.

Our experiments focus on open-ended short-form QA, where recovering correct answers absent from the initial generations is already a substantive challenge. The vocabulary experiment in our response to Weakness 1 also uses entity names and geographic terms from other public resources, demonstrating that explicit answer enumeration is not limited to WordNet. The hybrid score and implicit set do not require a predefined answer vocabulary.

Extending explicit prediction sets to long-form responses requires further work, because a vocabulary of words and phrases cannot adequately represent complete responses. Appendix I.3 discusses this limitation, and our response to Question 3 outlines concrete approaches to constructing explicit sets of long-form responses and evaluating their coverage.

 

Response to Question 3

Although it could be an out-of-scope suggestion, could the proposed latent-space expansion be extended to free-form text generation, where a finite answer vocabulary is generally unavailable?

We outline two possible extensions for constructing explicit sets of long-form responses without enumerating a predefined answer vocabulary.

(1) Embedding inversion.

As discussed in Appendix I.3, an inversion model reconstructs text from a target embedding. For example, Morris et al. (2023) iteratively generate, re-encode, and correct text to approximate that embedding. We could select target embeddings within the calibrated latent region, decode them into complete responses, and re-encode each response to check its inclusion. This approach requires a compatible inversion model, and its reconstruction accuracy for long responses would need evaluation.

(2) Autoregressive construction with latent-space expansion.

A different direction is to construct responses token by token. Although complete long-form answers have no tractable predefined vocabulary, the tokenizer provides a finite vocabulary of possible next tokens. At each prefix, we could adapt our angular–radial scoring to contextual representations of candidate token extensions, allowing tokens not sampled at that prefix to enter the candidate set. Retaining multiple continuations creates a tree whose completed paths represent candidate responses, including responses absent from the original complete-answer samples.

Deutschmann et al. (2024) provide a related foundation by incorporating conformal prediction into autoregressive decoding. Our proposed extension would introduce latent-space expansion into candidate construction. It requires a new local scoring rule and calibration on complete responses. For example, the maximum local score along each reference response, including termination, could serve as the sequence-level score. A complete response would be retained if all its local scores were at most the calibrated threshold. Calibrating their maximum therefore targets coverage of the entire response. The main computational challenge is the growth of the search tree, since pruning may remove paths admitted by the calibrated rule.

Coverage assessment.

Both finite embedding inversion and pruned autoregressive search may miss ground-truth responses covered by their underlying calibrated sets. Their final explicit outputs therefore require direct coverage assessment.

With scoring and calibration fixed, we could evaluate prespecified decoding or search budgets on an independent labeled validation set. Following Learn then Test (Angelopoulos et al., 2025), simultaneous one-sided binomial confidence bounds could support selecting the smallest tested budget whose upper bound on miscoverage does not exceed α\alpha. This would certify the selected procedure at the prespecified confidence level under i.i.d. sampling. If no budget passes, the target coverage cannot be certified within the tested range.

 

[1] Morris, John, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander Rush. 2023. Text Embeddings Reveal (Almost) As Much As Text. Proceedings of EMNLP 2023, pp. 12448–12460.

[2] Deutschmann, Nicolas, Marvin Alberts, and María Rodríguez Martínez. 2024. Conformal Autoregressive Generation: Beam Search with Coverage Guarantees. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10):11775–11783.

[3] Angelopoulos, Anastasios N., Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. 2025. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2):1641–1662.

Reviewer_rX2eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is about conformal prediction, which returns a set of possible answers. The focus is on calibration with a specified probability, e.g., 90%, under exchangeability. Th eproblem the paper addresses is that if the correct answer never appears among the model's sampled responses, the model cannot recover it from those responses. The authors propose to address this by expanding the set of responses using distances between query-aware answer embeddings. However, the practical value remains unclear. The explicit answer sets are very large, and the strongest coverage results rely on filtering the evaluation data. The novelty also needs clearer positioning.

Critical strengths

The paper addresses a real problem: selecting among generated answers cannot help if the model has never generated the correct answer. This is a sensible method and uses separate data for learning and calibration. The experiments also show how the different parts of the method contribute to the results.

Critical weaknesses

It is great to see more correct answers, but 53% coverage is not so strong.

The strongest explicit-set results use a filtered population, and here the method achieves 90% coverage. This shows how well the method works when candidates are availabile. However, at deployment, the correct answer is unknown, and thus this experiment is not applicable.

The headline 90% coverage is for an implicit set that supports membership checks. The explicit answer sets achieve only about 53% coverage on the full datasets, forroughly 10,000 answers. The baselines have no access to the same vocabulary, and thus the experiments do not isolate the benefit of the proposed scoring method.

Detailed comments:

  • 028–030, 099–105: The headline coverage results concern different outputs and evaluation settings. Please make the distinction between implicit hybrid sets, explicit answer sets, and recoverable examples clearer when presenting these numbers.

  • 254–264: The implicit set allows checking whether a given answer belongs, but does not list the answers it covers. What is the use when the correct answer is unknown?

  • 283–292, 346–348, Appendix E: The cell-based method is based on prior work. Make it clear what is old and what is new.

  • 384–386 versus 1022–1025: The main text specifies Qwen2.5-7B for statement generation, while the computation section reports template-based generation. Clarify which implementation gives the main results and if the reported costs cover it.

  • 393–398: The recoverable version has examples based on whether the correct answer was generated or appears in the vocabulary. How should we interpret these results for deployment, where recoverability is unknown?

  • 400–404: The baselines select from generated answers, while your method accesses a large additional vocabulary. Compare against simpler scoring methods using the same vocabulary to isolate the benefit of your method.

  • 1273–1312, Tables 5–6: At the 90% target, explicit cell-based sets achieve about 53% coverage on the full datasets with roughly 10,000 answers. These results are central to assessing practical usefulness and should be discussed clearly in the main text.

  • 1299–1312, Table 6: Cell ordering reduces mean set size by 20.02% on recoverable examples, but only 1.27% on the full datasets. Explain this gap and qualify the headline improvement accordingly.

Questions for authors

Questions:

  1. What can we conclude if we only have 53% coverage?

  2. Does your score outperform simpler methods using the same vocabulary?

  3. What is new beyond standard conformal prediction and prior cell-based methods?

  4. Why does cell ordering reduce set size by 20% on filtered, data but only 1.27% on the full data?

6 comments

Thank you for your constructive feedback. We have uploaded an updated PDF with additional experiments, which we reference in our responses below.

 

Response to Comment 1:

The headline coverage results concern different outputs and evaluation settings. Please make the distinction between implicit hybrid sets, explicit answer sets, and recoverable examples clearer when presenting these numbers.

Thank you for this suggestion. The abstract already attributes the 90.25% coverage to the hybrid prediction set and states that the semantic set retains the guarantee when the vocabulary contains all missed ground-truth answers. The introduction explicitly identifies and defines the recoverable setting before reporting semantic coverage; the 20.02% size reduction follows within that context.

We have clarified that the 90.25% coverage refers to the implicit hybrid set on the full datasets, explicitly identified the recoverable setting alongside the 20.02% size reduction, and added the 53.22% full-dataset coverage of explicit cell-based sets to the introduction.

 

Response to Comment 2:

The implicit set allows checking whether a given answer belongs, but does not list the answers it covers. What is the use when the correct answer is unknown?

We would like to clarify that checking whether an answer belongs to the implicit set does not require knowing the ground-truth answer.

The implicit set provides the inclusion criterion for our explicit semantic set and can also screen candidates supplied by retrieval or another generator. In our direct-answer embedding experiment (Appendix L), norm scalarization excludes approximately 99.1% of vocabulary answers on average on full SciQ with Llama. At the nominal 90% target, the implicit set’s guarantee bounds the marginal probability of rejecting the ground-truth answer by 10% under exchangeability, regardless of whether that answer was generated or belongs to the vocabulary.

 

Response to Comment 3:

Appendix E: The cell-based method is based on prior work. Make it clear what is old and what is new.

Our cell procedure builds on the prior static ranking rule proposed by Tawachi and Laufer-Goldshtein (2025), as acknowledged in Section 4.2 and discussed in Appendix E. We inherit Voronoi partitioning, cell evaluation using correct and incorrect labels, and independent recalibration. Our adaptations concern two aspects.

(1) Cell construction.

In the prior method, each cell-construction example provides one multidimensional score vector for its ground-truth label. In our setting, comparing the ground-truth answer with each distinct generated answer produces multiple angular-and-radial discrepancy vectors. We retain these vectors as candidate centers only when the ground-truth answer was not generated but belongs to the vocabulary. This focuses cell construction on answers that vocabulary conversion can newly recover.

(2) Cell ranking.

The prior method ranks cells once using their individual false-to-true label ratios. In our setting, the same input–answer pair can occur in multiple cells, making these counts redundant across selected cells. Our sequential procedure evaluates each candidate cell together with the already selected cells, computing the cumulative false-to-true answer ratio while counting each input–answer pair only once.

Empirical comparison.

We replace only the ordering rule, keeping the embeddings, cell partition, generated answers, vocabulary, and data splits fixed, with independent recalibration. The table reports equal-weight averages across ten dataset–model combinations at MM=10 and a 90% target. Sequential ordering yields smaller aggregate sets at comparable coverage; complete results appear in Appendix E, Tables E.1–E.2.

PopulationOrderingCoverage (%)Mean sizeSize increase
FullSequential (ours)53.8510,110.48—
FullStatic53.7710,757.11+6.40%
RecoverableSequential (ours)89.522,603.03—
RecoverableStatic89.642,903.58+11.55%

 

Response to Comment 4:

The main text specifies Qwen2.5-7B for statement generation, while the computation section reports template-based generation. Clarify which implementation gives the main results and if the reported costs cover it.

The main results use the template-based implementation in Appendix B.2: Qwen2.5-7B generates one statement template per input, and each answer is inserted by string replacement. No separate Qwen call is required for each answer. The reported costs include statement generation and encoding (Appendix D.3; Table 1).

Response to Comment 5

How should we interpret these results for deployment, where recoverability is unknown?

Recoverable results describe performance when the ground-truth answer is available for explicit inclusion; they do not establish deployment coverage.

Deployment coverage can be assessed through δW\delta_{\mathcal W} in Eq.(15), the probability that the ground-truth answer is neither generated nor contained in the vocabulary. We can estimate it on independent labeled validation data by checking whether each ground-truth answer is absent from both sources. Under i.i.d. sampling from the deployment population, a one-sided 95% binomial upper confidence bound UWU_{\mathcal W}, together with Eq. (16), gives a conservative marginal-coverage lower bound of max⁡0,1−α−UW\max\\{0,1-\alpha-U_{\mathcal W}\\}.

Appendix J illustrates this procedure through a retrospective simulation of validation counts for 1,000-example subsets, using the recorded full-dataset counts. Across the ten dataset–model combinations, the mean absolute difference from the full-dataset reference rates was 0.94 percentage points, with a maximum of 1.94 points.

A conservative assessment of δW\delta_{\mathcal W} is also possible without LM inference: checking whether the ground-truth answers in labeled validation data belong to the vocabulary estimates Pr⁡(Y∉W)\Pr(Y\notin\mathcal W), which upper-bounds δW\delta_{\mathcal W}. By comparison, directly estimating sampling recall for sampling-based CP on new validation inputs requires additional LM inference.

Moreover, explicit coverage can remain close to the target even when δW>0\delta_{\mathcal W}>0, because some unrecoverable answers may already be excluded by the implicit set and therefore cause no additional coverage loss during vocabulary conversion (Appendix A).

 

Response to Comment 6

Compare against simpler scoring methods using the same vocabulary to isolate the benefit of your method.

(1) Additional Euclidean-distance control.

We replace angular–radial discrepancies with Euclidean distances between question–answer statement embeddings and evaluate both norm and cell scalarizations. Norm scalarization uses the distance directly, while cell scalarization constructs one-dimensional cells and learns their ordering. We retain the generated answers, vocabulary, statement embeddings, and data splits, with independent recalibration. The table reports equal-weight averages across five datasets for Llama-3.2-3B-Instruct with cell scalarization, at M=10M=10 and a 90% target. Euclidean cell scalarization produces larger sets at comparable aggregate coverage. Appendix K, Tables 8–11 report the complete comparisons.

PopulationMethodCoverage (%)SizeSize increase
FullAngular–radial: cell56.278,670.40—
FullEuclidean: cell55.949,818.37+13.24%
RecoverableAngular–radial: cell89.671,996.66—
RecoverableEuclidean: cell89.912,864.82+43.48%

(2) LM sequence-probability scoring.

Another control scores each complete vocabulary answer by its LM probability conditioned on the question. Generated answers retain their cumulative-frequency scores; LM sequence probabilities score ungenerated vocabulary answers. Based on the test-set counts and vocabulary sizes in Tables 1–2, scoring these vocabularies across the ten recoverable test configurations requires 0.75–1.10 billion question–answer likelihood evaluations, excluding calibration costs.

Given this computational cost, we evaluated the LM sequence-probability control on WebQuestions and SciQ with both Gemma-3-4B-it and Llama-3.2-3B-Instruct, using the same vocabulary as the corresponding angular–radial method, MM=10, and a nominal 90% target. All results use a single split; parentheses show mean-size increases relative to angular–radial cell scalarization. Our cell-based method produces substantially smaller sets on both versions across all four dataset–model combinations.

DatasetModelMethodFull coverage (%)Full mean sizeRecoverable coverage (%)Recoverable mean size
WebQuestionsGemmaAngular–radial: cell48.0913,743.0489.644,396.23
WebQuestionsGemmaLM sequence-probability46.2824,538.37 (+78.55%)89.3824,232.74 (+451.22%)
WebQuestionsLlamaAngular–radial: cell53.519,757.9088.703,098.30
WebQuestionsLlamaLM sequence-probability52.0724,079.07 (+146.76%)90.8321,935.69 (+607.99%)
SciQGemmaAngular–radial: cell66.256,233.3890.431,062.65
SciQGemmaLM sequence-probability65.2024,294.70 (+289.75%)89.2319,497.61 (+1,734.81%)
SciQLlamaAngular–radial: cell68.004,442.2290.13463.64
SciQLlamaLM sequence-probability67.4323,348.94 (+425.61%)89.8215,085.51 (+3,153.71%)

Response to Comment 6 (continued)

(3) Existing comparisons.

Appendix G provides angular-only and radial-only controls using the same vocabulary. In particular, Table 4 shows that removing either coordinate from our cell method increases mean prediction-set size across all ten dataset–model combinations. These comparisons demonstrate the benefit of using angular and radial information jointly.

 

Response to Comment 7

Tables 5–6: At the 90% target, explicit cell-based sets achieve about 53% coverage on the full datasets with roughly 10,000 answers. These results are central to assessing practical usefulness and should be discussed clearly in the main text.

(1) Set size.

We tested whether the statement embeddings contribute to these large sets. Specifically, we directly encode answer strings instead of query-conditioned statements, while retaining the same generated answers and vocabulary and recalibrating each variant. The detailed experimental setting is provided in Appendix L. The table reports mean cell-based set sizes using statement → direct answer embeddings on the full and recoverable versions at MM=10, with percentage reductions relative to statement embeddings.

ModelDatasetFull: mean sizeReductionRecoverable: mean sizeReduction
GemmaTriviaQA11,797.03 → 2,944.6375.04%2,208.69 → 53.8797.56%
GemmaSciQ6,233.38 → 678.9489.11%1,062.65 → 26.6097.50%
GemmaNQ-Open11,803.93 → 3,272.4072.28%2,271.31 → 167.4392.63%
GemmaSQ-Wiki14,175.40 → 6,269.3155.77%6,108.10 → 675.0588.95%
GemmaWebQuestions13,743.04 → 5,268.6461.66%4,396.23 → 1,063.4975.81%
LlamaTriviaQA7,086.73 → 706.1190.04%297.77 → 7.1397.61%
LlamaSciQ4,442.22 → 266.6094.00%463.64 → 10.2897.78%
LlamaNQ-Open8,156.80 → 1,647.6579.80%532.26 → 27.0094.93%
LlamaSQ-Wiki13,908.34 → 5,680.6159.16%5,591.34 → 514.9290.79%
LlamaWebQuestions9,757.90 → 3,142.4867.80%3,098.30 → 519.1683.24%

This change reduces cell-based mean set sizes by 55.77–94.00% on the full versions and 75.81–97.78% on the recoverable versions. Empirical coverage remains comparable as shown in Table 14. Corresponding results for norm scalarization are provided in Table 12.

Statement construction was intended to contextualize short answers. However, this controlled comparison shows that it substantially increased set size. Shared question content may weaken the distinction between answer embeddings. We have acknowledged the adverse effect of this design choice and included the comparison.

(2) Coverage.

On the full versions, explicit cell-based sets achieve 53.22% coverage, compared with 39.57–39.61% for the sampling baselines. This represents a gain of approximately 13.6 percentage points, or a 34.4% relative improvement over the strongest sampling baseline, demonstrating substantial recovery beyond the LM’s generated outputs.

The absolute coverage must be interpreted relative to answer availability. Appendix F, Table 2 reports availability ceilings below 90% across all evaluated dataset–model combinations, meaning that even returning every generated answer and vocabulary entry cannot reach the target.

Simply enlarging the vocabulary may be inefficient: Table 2 shows that removing 79.4–86.1% of entries reduces the availability ceiling by only 2.35–4.73 percentage points. We therefore evaluated whether a vocabulary better matched to SQ-Wiki could improve coverage without enlarging prediction sets.

We construct candidate categories from WordNet, Wikidata5M, GeoNames, and wordfreq. Using separate development data, we greedily select complete categories according to their additional recovery of missed ground-truth answers relative to added vocabulary size. The resulting 19,996-entry vocabulary is fixed before calibration, and testing.

The experiment compares the pruned WordNet vocabulary used in the paper with the newly selected vocabulary. We use direct answer embeddings throughout, keeping the LM, encoder, and MM=10 sampled outputs unchanged. The table reports single-split, full-version results for cell scalarization at a nominal 90% target. Arrows indicate pruned WordNet → selected vocabulary.

ModelVocabulary entriesCoverage (%)Mean set size
Gemma20,457 → 19,99644.84 → 60.336,269.31 → 6,123.5
Llama21,149 → 19,99646.83 → 61.725,680.61 → 5,654.5

These results show that dataset-specific vocabulary selection substantially improves explicit coverage with slightly smaller vocabularies and prediction sets, while keeping the LM and sampling budget fixed. Methods restricted to sampled answers instead depend on improving sampling recall.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.