- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
Authors propose a method of tuning a student model for abstention based on a teacher model's probabilities. They introduce a new head for the student model that effectively predicts when the student model is providing much larger probabilities than the teacher. They use both an absolute and a relative cap. When the second head flags a specific token as "risky" (deviating too far from the teacher), they move the probability mass of that token to an <abstain> token instead. If the abstain token is "sampled", the generation ends. This is what I understand of the method.
The paper is hard to decipher, and I don't believe the authors' framing of aleatoric vs. epistemic hallucinations are a novel contribution to the literature, enough to differentiate the paper from existing works. Authors provide no tables, only plots, making it difficult to understand where their method actually improves against the baselines they provide.
Critical strengths
None
Critical weaknesses
Paper was extremely hard to understand due to dense, jargon-filled writing, and vague descriptions.
There are many existing works that add another token to the vocabulary to represent abstention. I don't believe the authors do a successful job at convincing me that their method has any edge against the existing methods. In a footnote, they claim for example that existing methods don't provide a trade-off frontier. However, if the model has a new vocabulary token "<abstain>", then this token has an associated probability, and you can sample it only when it is above a certain threshold, providing you a tunable frontier of sensitivity. I do see that IDK-tuning paper didnt report results over thresholding the IDK token, however the method plausibly could do this, so I need to see IDK-tuning as a baseline here for sure.
I need to see tables please. Charts are not enough. The edge the method has is very unclear to me. Everything is left far too vague.
Related works:
Cohen, R., Dobler, K., Biran, E., & de Melo, G. (2024). I Don’t Know: Explicit Modeling of Uncertainty with an [IDK] Token. NeurIPS 37, 10935–10958. arXiv:2412.06676. Huang, L., Feng, X., Ma, W., Fan, Y., Feng, X., Gu, Y., Ye, Y., Zhao, L., Zhong, W., Wang, B., Wu, D., Hu, G., Kong, L., Xiao, T., Liu, T., & Qin, B. (2025). Alleviating Hallucinations from Knowledge Misalignment in Large Language Models via Selective Abstention Learning. ACL 2025, 24564–24579. doi:10.18653/v1/2025.acl-long.1199. Jain, N., Shrivastava, A., Zhu, C., Liu, D., Samuel, A., Panda, A., Kumar, A., Goldblum, M., & Goldstein, T. (2025). Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models. COLM 2025. arXiv:2412.06748. † Chuang, Y.-N., et al. (2025). Learning to Route LLMs with Confidence Tokens (Self-REF). ICML 2025. arXiv:2410.13284.
Questions for authors
Please explain clearly and concisely how you work differs from/has an advantage compared to the numerous existing works that add an abstention token to the vocabulary of a language model and train it. Is it just the tunable runtime sensitivity?
4 comments
Please explain clearly and concisely how you work differs from/has an advantage compared to the numerous existing works that add an abstention token to the vocabulary of a language model and train it. Is it just the tunable runtime sensitivity?
Not on its own. You’re right that any abstention token can be thresholded: Refusal Tokens (Jain et al.) already do this, and we trace a full tradeoff curve for our [IDK] baseline the same way (Appendix I.1). What’s new is per-candidate abstention that you can tune at inference time. A vocabulary token can’t offer that combination.
The underlying problem is that the student is usually unsure which of its candidate continuations are hallucinations. Sometimes it is confident. For example, it may know it doesn’t know the answer to a factual question, and then any abstention mechanism, including an [IDK] token, can remove all of that mass. More often it is only partly sure: a move it thinks is probably legal, or a name that is plausible but may be made up. For these candidates, keeping or dropping is a tradeoff between risking an error and giving up a likely valid continuation, and the right choice depends on the application’s error budget. Our uncertainty head represents this per-candidate uncertainty, and the tolerance α picks the cutoff at inference. Each tolerance removes a different set of candidates while keeping the valid alternatives.
In principle, an abstention token can absorb the mass of specific candidates, but only for one tradeoff fixed during training. After that, the model has one [IDK] probability per position and no record of which candidates that mass came from. So any threshold or bias scales every candidate at the position equally. The frontier you describe changes which positions get answered, but never which continuations survive. That works when each question has one right answer. It fails when small errors are spread across positions with many valid options. Jain et al. and Chuang et al. have the same limitation even more strongly, since they decide once per response. Huang et al. train their [REJ] token with Cohen et al.’s objective and use it only as a beam-search penalty, never as an abstention.
The training signal differs too. Cohen et al. move target mass to [IDK] whenever the model prefers a token other than the gold one. Where many answers are valid, that teaches [IDK] to take mass from valid alternatives rather than from errors. Our head learns from the teacher’s full distribution which candidates are overestimated. An [IDK] token trained against the teacher in this way could match infrasampling at one tolerance, but it would then be infrasampling with the tolerance fixed during training.
Chess shows the practical effect (Table 6, at 20% response, as a fraction of the original invalid-move rate). Per-position methods give no improvement at all: max softmax probability 1.29, entropy 1.24. Thresholded [IDK] reaches 0.71, the training-free ε-threshold 0.30, and infrasampling 0.009. On single-answer benchmarks like TriviaQA the methods perform about the same, as Section 5.1 predicts.
Thank you again for your time and valuable feedback. Do let us know if there’s anything else we can do to earn a higher score.
I appreciate the authors' detailed responses. I am convinced this paper is topical and relevant to the community at large, that the two types of hallucinations are a useful distinction, and that the authors' method succeeds in doing what they set out to do. I have adjusted my score accordingly. Thank you for your hard work!
Thanks!
Thank you for your review. We’ve added a substantial number of new experiments and improved the clarity of the manuscript (changes are highlighted in blue). Please let us know if there are any additional reservations standing in the way of a higher score.
We address your points in order:
To clarify our novelty claim: the distinction between epistemic and aleatoric uncertainty is classical, and we do not claim otherwise. One of our contributions is to introduce a distinction between epistemic and aleatoric hallucinations: errors introduced by a model’s failure to capture its training distribution, versus undesirable behavior already present in a concrete training distribution. As we argue in Section 2, we believe this decomposition as well as our purely statistical framework, in which hallucinations are simply excess probability, fills a significant gap in the prior literature, where there is still significant disagreement about what constitutes a hallucination (see our Section 2). A handful of works graze similar concepts (e.g. Kalai et al. implicitly minimize aleatoric hallucinations by assuming that training data is perfect) but to our knowledge none grok them fully. If the reviewer has specific prior work in mind, we would be happy to incorporate it and adjust our claim accordingly. This distinction leads to the central idea of the paper: instead of estimating a distribution and inevitably overestimating some outcomes, we aim to model a lower bound and treat the missing mass as abstention. This directly targets epistemic hallucinations while preserving valid variation in the training distribution. Concretely, the ordinary language-model head predicts next-token probabilities, while the uncertainty head estimates uncertainty about the teacher’s probability for each candidate token. At inference, the chosen tolerance determines how much probability mass to retain for each candidate. Uncertain mass is removed and assigned to abstention. The method therefore operates on individual candidates within a next-token distribution, allowing it to preserve plausible alternatives while suppressing modeling errors.
We don’t exactly add another token to represent abstention; one can think of our method as something like adding a distinct abstention token for each original token in the vocabulary, so the model can remove each independently. Plus, we make it possible to adjust the abstention threshold at inference time.
In the most recent revision, we’ve added [IDK] results on both language and chess (see e.g. Figures 2 and 4), including thresholded variants thereof. Thresholding this method is actually somewhat nontrivial; see the new Appendix I for details about how we did it. Unlike our method, [IDK] has the conceptual shortcoming that it makes no distinction between aleatoric and epistemic uncertainty (all student model “mistakes” during one-hot training are funnelled into the [IDK] token, regardless of whether it was actually wrong or simply picked a different option from a larger set of valid choices). Perhaps for this reason, it performs very poorly on evals.
This was an oversight on our part. We’ve added tables corresponding to every main-paper figure to the appendix.
Our paper introduces a number of new concepts which need to be named, and we try hard to do that with minimal jargon, in part by relating them to existing, well known concepts in the literature like aleatoric and epistemic uncertainty. We’re able to state our definition of hallucinations (“probability that is too high”) with no new jargon at all. Please note that both other reviewers thought that the paper was written well.
Though we disagree overall, we’d still very much like to work with the reviewer to make sure the paper is as clear as possible. We’ve already rewritten several sections in the revised version of the manuscript (all changes highlighted in blue) across the paper, including most of Section 3, where we present our method. We also hope that the addition of tables will make it easier to digest our plots. Please let us know if there’s any specific terminology or paragraphs that need additional work.