Minimax Semantic Entropy for Four Meanings from Weak Pairwise Judgments

Semantic entropy scores a language model's uncertainty by the entropy of meanings among its sampled answers, seen only through a pairwise judge. When the judge is weak, the answers cannot be sorted by meaning, yet the verdicts may still determine the entropy. We model the judge as a known channel on the complete graph: given the meanings, it declares each pair of answers equivalent independently, with probability $q+\Delta$ within a meaning and $q$ across meanings. With at most three meanings, the edge and wedge averages of the verdict graph estimate the pair and triple collision probabilities, which fix the entropy; with four they do not, and the three-star average supplies the missing moment. For $n$ answers, at most four meanings and $q$ bounded away from $0$ and $1$, the minimax squared risk has order $\min \{1,(n^4{\Delta }^6{)}^{-1/3}\}$ on every fixed polynomial window $n{\Delta }^2\le n^{-\kappa }$, $\kappa >0$, and the entropy is uniformly consistently estimable exactly when $n^4{\Delta }^6\to \infty$. With $\Delta =n^{-\gamma }$ the critical exponent is $2/3$, against $3/4$ for three meanings in concurrent work, so a judge with $2/3\le \gamma <3/4$ suffices for three meanings and not for four. The rate is attained by an estimator that projects the three averages onto the valid moment set and solves a quartic. For the lower bound, a third-order inclusion–exclusion sweep of the likelihood score, whose coefficients obey an exact motif identity, bounds the Fisher information. If only pairs fixed in advance are judged, the risk bound depends on the number of three-stars they form, which hub designs maximize up to a constant factor.