acceptodds
Under review as a conference paper at ICLR 2027

Infrasampling: A framework for language modeling without hallucinations

Abstract

In this paper, we investigate efficient, general, and scalable methods for selective prediction in open-domain language modeling. We reconceptualize hallucinations as *data points whose probability under the model is too high*, and distinguish *epistemic* and *aleatoric* hallucinations by analogy to uncertainty quantification. From these definitions follows a general framework for trading off between epistemic hallucinations and abstentions in language models, which we call *infrasampling*. The key idea is that we sample from a *lower bound* on the distribution—rather than an average density estimate—and treat the slack as abstention. We instantiate this framework for autoregressive language models in the soft distillation setting, using new output heads that capture the model's epistemic uncertainty about the teacher distribution. The resulting hallucination-abstention frontiers dominate strong heuristic baselines and suggest compounding gains with scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, hallucinations, uncertainty

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KhytOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

Authors propose a method of tuning a student model for abstention based on a teacher model's probabilities. They introduce a new head for the student model that effectively predicts when the student model is providing much larger probabilities than the teacher. They use both an absolute and a relative cap. When the second head flags a specific token as "risky" (deviating too far from the teacher), they move the probability mass of that token to an <abstain> token instead. If the abstain token is "sampled", the generation ends. This is what I understand of the method.

The paper is hard to decipher, and I don't believe the authors' framing of aleatoric vs. epistemic hallucinations are a novel contribution to the literature, enough to differentiate the paper from existing works. Authors provide no tables, only plots, making it difficult to understand where their method actually improves against the baselines they provide.

Critical strengths

None

Critical weaknesses

Paper was extremely hard to understand due to dense, jargon-filled writing, and vague descriptions.

There are many existing works that add another token to the vocabulary to represent abstention. I don't believe the authors do a successful job at convincing me that their method has any edge against the existing methods. In a footnote, they claim for example that existing methods don't provide a trade-off frontier. However, if the model has a new vocabulary token "<abstain>", then this token has an associated probability, and you can sample it only when it is above a certain threshold, providing you a tunable frontier of sensitivity. I do see that IDK-tuning paper didnt report results over thresholding the IDK token, however the method plausibly could do this, so I need to see IDK-tuning as a baseline here for sure.

I need to see tables please. Charts are not enough. The edge the method has is very unclear to me. Everything is left far too vague.

Related works:

Cohen, R., Dobler, K., Biran, E., & de Melo, G. (2024). I Don’t Know: Explicit Modeling of Uncertainty with an [IDK] Token. NeurIPS 37, 10935–10958. arXiv:2412.06676. Huang, L., Feng, X., Ma, W., Fan, Y., Feng, X., Gu, Y., Ye, Y., Zhao, L., Zhong, W., Wang, B., Wu, D., Hu, G., Kong, L., Xiao, T., Liu, T., & Qin, B. (2025). Alleviating Hallucinations from Knowledge Misalignment in Large Language Models via Selective Abstention Learning. ACL 2025, 24564–24579. doi:10.18653/v1/2025.acl-long.1199. Jain, N., Shrivastava, A., Zhu, C., Liu, D., Samuel, A., Panda, A., Kumar, A., Goldblum, M., & Goldstein, T. (2025). Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models. COLM 2025. arXiv:2412.06748. † Chuang, Y.-N., et al. (2025). Learning to Route LLMs with Confidence Tokens (Self-REF). ICML 2025. arXiv:2410.13284.

Questions for authors

Please explain clearly and concisely how you work differs from/has an advantage compared to the numerous existing works that add an abstention token to the vocabulary of a language model and train it. Is it just the tunable runtime sensitivity?

4 comments

Thank you for your review. We’ve added a substantial number of new experiments and improved the clarity of the manuscript (changes are highlighted in blue). Please let us know if there are any additional reservations standing in the way of a higher score.

We address your points in order:

I don't believe the authors' framing of aleatoric vs. epistemic hallucinations are a novel contribution to the literature, enough to differentiate the paper from existing works

To clarify our novelty claim: the distinction between epistemic and aleatoric uncertainty is classical, and we do not claim otherwise. One of our contributions is to introduce a distinction between epistemic and aleatoric hallucinations: errors introduced by a model’s failure to capture its training distribution, versus undesirable behavior already present in a concrete training distribution. As we argue in Section 2, we believe this decomposition as well as our purely statistical framework, in which hallucinations are simply excess probability, fills a significant gap in the prior literature, where there is still significant disagreement about what constitutes a hallucination (see our Section 2). A handful of works graze similar concepts (e.g. Kalai et al. implicitly minimize aleatoric hallucinations by assuming that training data is perfect) but to our knowledge none grok them fully. If the reviewer has specific prior work in mind, we would be happy to incorporate it and adjust our claim accordingly. This distinction leads to the central idea of the paper: instead of estimating a distribution and inevitably overestimating some outcomes, we aim to model a lower bound and treat the missing mass as abstention. This directly targets epistemic hallucinations while preserving valid variation in the training distribution. Concretely, the ordinary language-model head predicts next-token probabilities, while the uncertainty head estimates uncertainty about the teacher’s probability for each candidate token. At inference, the chosen tolerance determines how much probability mass to retain for each candidate. Uncertain mass is removed and assigned to abstention. The method therefore operates on individual candidates within a next-token distribution, allowing it to preserve plausible alternatives while suppressing modeling errors.

There are many existing works that add another token to the vocabulary to represent abstention… so I need to see IDK-tuning as a baseline here for sure.

We don’t exactly add another token to represent abstention; one can think of our method as something like adding a distinct abstention token for each original token in the vocabulary, so the model can remove each independently. Plus, we make it possible to adjust the abstention threshold at inference time.

In the most recent revision, we’ve added [IDK] results on both language and chess (see e.g. Figures 2 and 4), including thresholded variants thereof. Thresholding this method is actually somewhat nontrivial; see the new Appendix I for details about how we did it. Unlike our method, [IDK] has the conceptual shortcoming that it makes no distinction between aleatoric and epistemic uncertainty (all student model “mistakes” during one-hot training are funnelled into the [IDK] token, regardless of whether it was actually wrong or simply picked a different option from a larger set of valid choices). Perhaps for this reason, it performs very poorly on evals.

I need to see tables please. Charts are not enough. The edge the method has is very unclear to me. Everything is left far too vague.

This was an oversight on our part. We’ve added tables corresponding to every main-paper figure to the appendix.

The paper is hard to decipher

Our paper introduces a number of new concepts which need to be named, and we try hard to do that with minimal jargon, in part by relating them to existing, well known concepts in the literature like aleatoric and epistemic uncertainty. We’re able to state our definition of hallucinations (“probability that is too high”) with no new jargon at all. Please note that both other reviewers thought that the paper was written well.

Though we disagree overall, we’d still very much like to work with the reviewer to make sure the paper is as clear as possible. We’ve already rewritten several sections in the revised version of the manuscript (all changes highlighted in blue) across the paper, including most of Section 3, where we present our method. We also hope that the addition of tables will make it easier to digest our plots. Please let us know if there’s any specific terminology or paragraphs that need additional work.

Please explain clearly and concisely how you work differs from/has an advantage compared to the numerous existing works that add an abstention token to the vocabulary of a language model and train it. Is it just the tunable runtime sensitivity?

Not on its own. You’re right that any abstention token can be thresholded: Refusal Tokens (Jain et al.) already do this, and we trace a full tradeoff curve for our [IDK] baseline the same way (Appendix I.1). What’s new is per-candidate abstention that you can tune at inference time. A vocabulary token can’t offer that combination.

The underlying problem is that the student is usually unsure which of its candidate continuations are hallucinations. Sometimes it is confident. For example, it may know it doesn’t know the answer to a factual question, and then any abstention mechanism, including an [IDK] token, can remove all of that mass. More often it is only partly sure: a move it thinks is probably legal, or a name that is plausible but may be made up. For these candidates, keeping or dropping is a tradeoff between risking an error and giving up a likely valid continuation, and the right choice depends on the application’s error budget. Our uncertainty head represents this per-candidate uncertainty, and the tolerance α picks the cutoff at inference. Each tolerance removes a different set of candidates while keeping the valid alternatives.

In principle, an abstention token can absorb the mass of specific candidates, but only for one tradeoff fixed during training. After that, the model has one [IDK] probability per position and no record of which candidates that mass came from. So any threshold or bias scales every candidate at the position equally. The frontier you describe changes which positions get answered, but never which continuations survive. That works when each question has one right answer. It fails when small errors are spread across positions with many valid options. Jain et al. and Chuang et al. have the same limitation even more strongly, since they decide once per response. Huang et al. train their [REJ] token with Cohen et al.’s objective and use it only as a beam-search penalty, never as an abstention.

The training signal differs too. Cohen et al. move target mass to [IDK] whenever the model prefers a token other than the gold one. Where many answers are valid, that teaches [IDK] to take mass from valid alternatives rather than from errors. Our head learns from the teacher’s full distribution which candidates are overestimated. An [IDK] token trained against the teacher in this way could match infrasampling at one tolerance, but it would then be infrasampling with the tolerance fixed during training.

Chess shows the practical effect (Table 6, at 20% response, as a fraction of the original invalid-move rate). Per-position methods give no improvement at all: max softmax probability 1.29, entropy 1.24. Thresholded [IDK] reaches 0.71, the training-free ε-threshold 0.30, and infrasampling 0.009. On single-answer benchmarks like TriviaQA the methods perform about the same, as Section 5.1 predicts.

Thank you again for your time and valuable feedback. Do let us know if there’s anything else we can do to earn a higher score.

I appreciate the authors' detailed responses. I am convinced this paper is topical and relevant to the community at large, that the two types of hallucinations are a useful distinction, and that the authors' method succeeds in doing what they set out to do. I have adjusted my score accordingly. Thank you for your hard work!

Reviewer_mNfaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines a hallucination as an output whose probability under the model exceeds its probability under the training distribution, and separates this epistemic case from aleatoric hallucinations that the training distribution itself contains. From this definition it proposes infrasampling: train the model to under-estimate the distribution everywhere, treat the missing probability mass as abstention, and let the user choose the confidence level of the lower bound at inference time. The framework is instantiated in soft distillation, where the teacher stands in for the training distribution. A second output head predicts, for each candidate token, a distribution over the teacher's probability for that token, and inference truncates the student's distribution at a chosen quantile. Experiments on chess with a rule-based teacher and on language modeling with a 32B teacher show that, relative to the teacher, the method lowers the hallucination rate by orders of magnitude at high abstention and improves more with scale than a probability-threshold baseline. On downstream benchmarks scored against ground truth it matches the baseline, and its tolerance transfers across a mixture of tasks where the baseline degrades.

Critical strengths
  1. The core idea is new. The model learns a lower bound on the distribution and abstention is the slack. The head works per candidate token, so it can remove part of a next-token distribution and keep the rest, which no entropy or max-probability rule can do.

  2. The chess testbed is the right way to show the mechanism. The verifier and the teacher are exact, so the gains there are unambiguous.

Critical weaknesses
  1. Freedom from hallucination is measured as agreement with a teacher, and in the language experiments the teacher is a 32B base model that hallucinates itself. On the tasks scored against ground truth in Figure 5, the method only matches the baseline. What the paper shows is a better way to distill abstention from a teacher, which is a smaller claim than the title.

  2. The method cannot be easily adopted as described. It needs a teacher streaming logits during training, the head has to be trained from scratch with the trunk since Section 7.2 reports that finetuning it into a frozen model is substantially worse, and a sequence is abstained whenever any token is, so only short continuations survive. The headline gains come at 70 to 88 percent abstention.

  3. The comparisons are thin. The closest prior method, a trained abstention token, is cited in footnote 4 and never compared. The scaling claim rests on two language model sizes, and the 7B result in Figure 12 uses the frozen-trunk variant only.

Questions for authors
  1. What is the hallucination rate of the 32B teacher itself on TriviaQA at the response rates in Figure 5? Without this, the reader cannot tell how much of the remaining student error is inherited.

  2. Footnote 4 mentions a trained abstention token as the closest prior method. Why is it not among the baselines? It is the one method that also learns abstention during training, so it is the comparison that would show what the per-candidate head adds.

  3. At what continuation length does infrasampling stop being usable without rejection sampling, and what is the compute cost of recovering a 50 percent response rate at 64 tokens?

4 comments

We thank the reviewer for recognizing the novelty of our method and lower-bound formulation. We’ve added a substantial number of new experiments and improved the clarity of the manuscript; changes are highlighted in blue. Please let us know if there are any additional reservations standing in the way of a higher score.

We address the points in order:

Freedom from hallucination is measured as agreement with a teacher… which is a smaller claim than the title.

We distinguish epistemic from aleatoric hallucinations for precisely this reason. In any realistic language distillation experiment, the teacher will have imperfections: we call these aleatoric hallucinations, and we don’t think it’s the job of a student model to correct them (if we had a way to correct them in the student, surely we could just apply that method to the teacher and eliminate the hallucination in the first place!). Errors introduced by the student’s failure to model that distribution are epistemic hallucinations, and are much easier to measure and target. The least we can ask of our student models is not to introduce new errors not present in the training data. Infrasampling is designed to target the latter, and succeeds. This is one reason the chess experiments are particularly instructive. There, the teacher never produces invalid moves, so the distinction is unambiguous: the student’s invalid moves are epistemic hallucinations, and infrasampling reduces them by orders of magnitude. Chess also lets us study the scaling behavior across a wider range of model sizes. The language results remain useful in their own right: a 1B student that generates substantially fewer errors relative to its 32B teacher than ordinary distillation is a meaningful result. Our choice of teacher was also constrained by reproducibility and compute. We chose OLMo because its openly available training data supports reproducible training experiments, and 32B was the largest OLMo teacher we could reasonably distill from with our available compute. However, our framework is not inherently limited to a particular teacher. Stronger teachers can reduce inherited errors, while increasing student capacity improves its ability to model the teacher, consistent with the scaling trends we observe (our experiments already span 250M–7B parameters). We therefore expect larger students distilled from stronger teachers to yield more useful reliability-coverage tradeoffs using our framework.

On the tasks scored against ground truth in Figure 5, the method only matches the baseline.

As we note in the paper, this has less to do with the choice of teacher model and more to do with the fact that these tasks feature no or little aleatoric uncertainty (also acknowledged by prior work), and so matching the answer ranking implied by the model’s main predicted distribution is the best you could hope to do here. See Section 5.1 in the paper for more discussion. But when aleatoric uncertainty is present, it should be possible to avoid unnecessary abstention. See e.g. Table 1 for a qualitative analysis that demonstrates the head performing well in such a setting and also Figure 12 for our pooled baseline evaluation, where we clearly outperform the baseline on a mixture of downstream tasks.

The method cannot be easily adopted as described... the headline gains come at 70 to 88 percent abstention.

Soft distillation already requires teacher logits, and is also extremely common in both pretraining and posttraining (on-policy distillation, e.g.), so we do not consider this a disadvantage of our method. We reuse this supervision to train our uncertainty heads; the teacher is not needed at inference.

Though we’d like to note that frozen-trunk finetuning not doing well wouldn’t necessarily say much about our method, since the initial submission, we have also improved our finetuning setup with a better cooldown procedure. In the new pooled-benchmark results, the finetuned multibin relative head actually performs comparably to the jointly pretrained head, and both substantially outperform ε-threshold. Pretraining still performs better for the binary head.

The headline examples at 70–88% abstention emphasize the high-reliability regime, where the improvements are most dramatic. However, the gains are not confined to those operating points. Our models lead the strongest baseline almost everywhere, almost always by several times. Plus, the tradeoff curve improves with scale, so you get greater relative improvements at lower abstention rates. Most of our plots are on a log scale, which make the size of improvements elsewhere less immediately apparent. We have added corresponding numerical tables to make the comparisons easier to inspect.

The comparisons are thin... frozen-trunk variant only.

Our scaling evidence also extends beyond the two jointly pretrained language model sizes: the chess experiments cover five scales, from 1M to 250M parameters (we added 250M in the most recent revision, up from 50M), and show an increasingly favorable tradeoff relative to the baseline. The language experiments, while limited by compute (pretraining 1B-scale models from scratch on hundreds of tokens per parameter is expensive!) show the same direction of improvement.

In the new revision, we’ve also added experiments with the [IDK] abstention token from Cohen et al. We had initially skipped them because the method is heuristic and makes no distinction between epistemic and aleatoric uncertainty, and so we expected it to perform very poorly in our evaluations (not to mention in the limit). Indeed it does: on both chess and language, it trails almost all of our baselines.

What is the hallucination rate of the 32B teacher itself on TriviaQA at the response rates in Figure 5? Without this, the reader cannot tell how much of the remaining student error is inherited.

On the same TriviaQA prompts and grader, the 32B teacher’s error among attempted answers is 26% (greedy) and 40% (temperature 1, the student’s decoding) at full response, falling to 5% and 7% at 20% response when it abstains on its own token probabilities. Note that, as mentioned in the paper, TriviaQA has quite a bit of label noise—the best human score is something like 80%. Inheritance accounts for little of the student’s error: at every response rate in Figure 5, at most 26% of the student’s errors fall on questions the teacher also gets wrong, and the student reproduces the teacher’s own answer in under 8 percentage points of them. The remaining error is the 1B student’s own, on questions the teacher answers correctly. We’ve added these numbers to the appendix. We will add the 32B teacher’s own TriviaQA hallucination rates at the response rates considered in Figure 5.

Footnote 4 mentions a trained abstention token as the closest prior method. Why is it not among the baselines? It is the one method that also learns abstention during training, so it is the comparison that would show what the per-candidate head adds.

We have added experiments with the [IDK] abstention token from Cohen et al. (see above).

At what continuation length does infrasampling stop being usable without rejection sampling, and what is the compute cost of recovering a 50 percent response rate at 64 tokens?

There is no single continuation length at which infrasampling becomes unusable: this depends on the desired reliability, student capacity, and the gap between student and teacher. Asking a 1B student to remain completely faithful to a 32B teacher at low error tolerances is inherently a big ask, and the fact that we’re able to get several tokens out at a time is highly nontrivial. Our scaling trends show that increasing student capacity and using a larger, less noisy teacher permits longer continuations. Scaling also helps for the separate reason that the student's baseline error decreases with scale (i.e. the chance of sampling in a probability region that is uncertain decreases with scale, even with a constant relative gap in scale between the student and teacher). Note that the head’s rejections could ultimately be distilled back into the student as verbal abstentions, so “abstention” here does not necessarily need to result in stopping generation; what’s important is that the head can detect deviations from the teacher.

We thank the reviewer for their careful reading and substantive feedback. Let us know if you have any remaining reservations.

The discussion period is coming to a close, so please do let us know if we've resolved your remaining concerns. We've substantially expanded and improved the manuscript, including with the [IDK] baseline you mentioned. Overall, we find that per-position methods lag far behind our new method. A datapoint: on chess at 20% response (see Table 6), max softmax probability reaches 1.29, entropy 1.24, thresholded [IDK] 0.71, the training-free ε-threshold 0.30, whereas infrasampling gets to 0.009.

Reviewer_mNfaOpenReview ↗

Response

Thanks very much for the considered response and for the work you've put into the new experiments. I do appreciate the addition of the abstention token baseline as that does help with respect to the comparisons. I do like the idea and remain a positive about the paper so I'm sticking with weak acceptance.

Reviewer_DWqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new method for hallucination detection for open source LLMs such as Olmo. The key idea of the paper is infrasampling where sampling is performed from a lower bound on the distribution rather than an average density estimate. This is instantiated using a soft distillation setting, where a new uncertainty head is learnt. The evaluation is performed on both synthetic chess and real-world text datasets.

Critical strengths
  • The paper is well written and provides a holistic view on hallucinations: the paper considers both epistemic hallucinations and aleatoric hallucinations.

  • The proposed method is simple and easy to implement.

  • The evaluation covers both synthetic chess and real-world text datasets.

Critical weaknesses
  • The evaluation is somewhat limited and hard to understand. The paper considers just a single Olmo-3-1125-32B as the baseline. The paper should consider a wider array of baselines such as Qwen3.5/3.6 model families. Although the training datasets are not available, many of the hallucination detection methods are applicable to such baselines as well: e.g., ϵ\epsilon - threshold.

  • The paper does not compare to prior works such as "Enhancing hallucination detection through noise injection, ICLR 2026" which also takes a Bayesian approach to hallucination detection, taking into account both epistemic and aleatoric uncertainty.

  • The proposed approach is not training free: for every new LLM the proposed approach requires retraining the uncertainty head. This is not practical.

  • Currently, the paper directly evaluates only hallucination vs abstention. The paper should evaluate on additional text datasets such as TriviaQA where both accuracy and hallucination rates can be studied.

Questions for authors
  • The paper should perform a more extensive survey and compare to related work.
  • The paper should better justify the choice of baseline LLMs.
  • The evaluation should be extended to cover both accuracy and hallucination rates.
3 comments

We thank the reviewer for their time and careful reading. We’ve added a substantial number of new experiments and improved the clarity of the manuscript. Changes are highlighted in blue. Please let us know if there are any additional reservations standing in the way of a higher score.

We address the points in order:

The paper considers just a single Olmo-3-1125-32B as the baseline.

First, we note that we also include extensive experiments in chess which feature a different teacher baseline. Though our language experiments still show very promising results, they are also inherently limited by computational constraints (we can only pretrain so many models at the 1B scale, make the teacher so big, etc.). In chess, we’re able to use a near-perfect engine as the teacher model (i.e. one that doesn’t aleatorically hallucinate), and so we’re able to demonstrate the performance of our method in the limit. Furthermore, we chose OLMo because we’re able to guarantee that our experiments are run on fresh data, without which our experiments would be significantly confounded. If a teacher model has (partially) memorized a specific example from its training data, it might start to output overly sharp distributions and misrepresent the amount of aleatoric uncertainty to the student. This could influence downstream evaluations where correctly modeling aleatoric uncertainty is necessary. There are only so many dense model families with open training data and a reasonable scale ladder. Note that, though we fix the teacher, we do run experiments across multiple student scales: 250M, 1B, and 7B. Nevertheless, we’re currently running a couple of fine-tuning experiments on Qwen models using Olmo data. We’ll add them to the manuscript as soon as they finish.

The paper does not compare to prior works such as "Enhancing hallucination detection through noise injection, ICLR 2026" which also takes a Bayesian approach to hallucination detection, taking into account both epistemic and aleatoric uncertainty.

Our approach is not Bayesian, and we do not claim to be the first to distinguish aleatoric from epistemic uncertainty. We instead introduce the concepts of aleatoric vs epistemic hallucinations. Following Ahdritz et al. (2025), we give these concepts a precise interpretation in terms of variation within the training distribution versus the model’s failure to capture that distribution. We then provide an accurate and scalable method for suppressing the resulting epistemic hallucinations. The chess and language overestimation results demonstrate the efficacy of our approach. Its inference cost is on the order of one additional output head, and it requires neither an ensemble nor repeated generations to estimate uncertainty. Our scaling experiments further show improvements in both absolute hallucination rates and the relative reductions achieved through abstention. The cited noise-injection method, like related sample-based approaches, estimates uncertainty from multiple generated responses (10 in its main experiments) which, unlike our method, entails substantial inference-time computation. There is also a separate question of what those estimates represent. Sensitivity to a chosen perturbation of model parameters or activations does not by itself establish an accurate decomposition into aleatoric and epistemic modeling error. In our setting, teacher probabilities provide direct supervision for the quantity the head is intended to estimate, thus making epistemic hallucinations explicit.

The proposed approach is not training free: for every new LLM the proposed approach requires retraining the uncertainty head. This is not practical.

We respectfully disagree with this framing: a trained method that outperforms training-free ones (as ours does) is valuable. Practitioners train new models all the time (i.e., every new LLM requires training an LLM), and we show in our experiments that the uncertainty head can easily be co-trained with an original model with no penalty to language modeling loss. Even fine-tuned heads are effective. If anything, given that most training-free baselines require extensive re-sampling at inference time, whereas our method requires training once and is essentially free thereafter, we’d argue that it is significantly more practical. A large part of our contribution, beyond any specific method (be it our heads or the training-free epsilon-threshold baseline we surface), is the overarching framework: epistemic vs aleatoric hallucinations, the right quantities to measure and target, etc. It admits a wide variety of possible approaches, including training-free ones.

Currently, the paper directly evaluates only hallucination vs abstention. The paper should evaluate on additional text datasets such as TriviaQA where both accuracy and hallucination rates can be studied.

The requested accuracy evaluations are already included in the submission. Figure 5 evaluates TriviaQA accuracy against ground truth at different response rates, and the appendix includes additional QA evaluations. The TriviaQA curve therefore shows both how accuracy improves with abstention and how frequently attempted answers remain incorrect. The pooled-benchmark results address a further practical question: whether a single tolerance setting works consistently across different tasks. Here, infrasampling shows a clear advantage over the strongest baseline.

The paper should perform a more extensive survey and compare to related work.

We’d welcome specific additional references the reviewer believes are missing.

The paper should better justify the choice of baseline LLMs.

We have updated the paper to include the justifications we mention above.

The evaluation should be extended to cover both accuracy and hallucination rates.

Our paper already covers both accuracy and hallucination rates (see above).

It's the last day of the discussion period, so please let us know whether we've resolved your concerns. We've added a lot of new stuff (a new (trained) baseline in the [IDK] token from Cohen et al., a bigger chess ladder, rewritten sections, new appendix sections), and we think the manuscript is substantially improved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.