Semantic Entropy Is Estimable Below the Recovery Threshold: Minimax Rates Under a Noisy Judge

Semantic entropy scores a language model's uncertainty by the entropy of the distribution from which its $n$ sampled answers draw their meanings, and those meanings are never observed: an entailment judge is called on pairs and the answers are grouped by its verdicts. That grouping can fail when the judge is too weak for the available sample size. We model the judge as a channel that, given the meanings, declares each pair equivalent independently with probability $q+\Delta$ when the two answers share a meaning and $q$ otherwise. For at most three meanings, a known channel with $q\in [q_0,1-q_0]$ and a weakly separated judge, $n{\Delta }^2\le c_1$, we determine the minimax squared risk of the entropy: $R_n^{\ast }\asymp \min \{1,(n^3{\Delta }^4{)}^{-1/2}\}$, with both constants depending only on $q_0$. The rate is attained by an estimator that never groups the answers. It reads the collision probability off the agreement density and the triple-collision probability off a centred statistic in the squared degrees, projects the pair onto the set of valid moments, and solves a cubic. The matching lower bound is proved along a family that holds the agreement density fixed while the entropy changes, so every linear statistic of the verdicts has the same mean at its endpoints. With $\Delta =n^{-\gamma }$, the entropy is uniformly consistently estimable exactly for $\gamma <3/4$, and on $\gamma \in (1/2,3/4)$ the minimax risk is of order $n^{(4\gamma -3)/2}$. For every $\gamma >1/2$, below the recovery threshold $n{\Delta }^2\asymp 1$, no estimator of the meanings improves asymptotically on assigning each answer to the most likely one, even when given all other answers' meanings. On $\gamma \in (1/2,3/4)$ the entropy is therefore estimable while the grouping is unrecoverable.