Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation

Quantifying uncertainty in Large Language Models (LLMs) is essential for mitigating hallucinations and enabling risk-aware deployment in safety-critical tasks. However, estimating Epistemic Uncertainty (EU) via Deep Ensembles is computationally prohibitive at the scale of modern models. We propose a framework that leverages the small draft models to efficiently estimate token-level EU, bypassing the need for full-scale ensembling. Theoretically grounded in a Bias-Variance Decomposition, our approach approximates EU via Jensen-Shannon divergence among drafts (variance proxy) and KL divergence between the draft mixture and the target (bias proxy). We additionally bound finite-reference EU transfer error by teacher-specific predictive mismatch, distinguishing disagreement preservation from predictive-mean approximation. To support accurate estimation, we introduce Online Stochastic Distillation (OSD) to efficiently approximate target aggregation and the Data-Diverse Drafts (DDD) strategy to enhance draft diversity for better target approximation. Crucially, our approach achieves Hallucination Detection performance competitive with heavy perturbation-based methods like TokUR while replacing repeated target-model uncertainty passes with draft and proxy scoring, offering a practical solution for uncertainty-aware LLM deployment.