- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper constructs UncertainBench, a token‑level benchmark to separate epistemic(EU) and aleatoric(AU) uncertainty. It evaluates 67 LLMs and 35 uncertainty estimators, discovers strong EU signals in low‑probability token tails, proposes a trained estimator M_{train}, and tests EU estimators on five downstream tasks with mixed results.
Critical strengths
-
Builds the fine‑grained token‑level UncertainBench benchmark with automatic annotation pipeline and decent quality, filling the gap of direct EU evaluation.
-
Proposes EU‑purity metric, large‑scale benchmark of existing uncertainty methods; finds tail tokens carry purer epistemic‑uncertainty signals.
-
Proposed M_{train} outperforms baselines and generalizes across held‑out domains; fairly discusses inconsistent downstream performance.
Critical weaknesses
-
EU/AU are proxy labels (not ground‑truth oracle values); limitations are buried in appendices and deserve more main‑text emphasis.
-
The tail‑suppression mechanistic hypothesis is only correlational, lacking causal training‑based verification.
-
No experiments on zero‑shot transfer of M_{train} to completely unseen LLMs of new architectures.
-
Benchmark annotations heavily rely on annotator LLMs; human evaluation sample size is small. Downstream task absolute performance remains moderate.
Questions for authors
-
What controlled training experiment can verify your tail‑suppression hypothesis?
-
Why do EU estimators fail on reasoning‑pivot branching positions? Is this a fundamental limitation?
-
How do you mitigate annotation bias brought by annotator LLMs for niche domain inputs?