acceptodds
Under review as a conference paper at ICLR 2027

Minimax Semantic Entropy for Four Meanings from Weak Pairwise Judgments

Abstract

Semantic entropy scores a language model's uncertainty by the entropy of meanings among its sampled answers, seen only through a pairwise judge. When the judge is weak, the answers cannot be sorted by meaning, yet the verdicts may still determine the entropy. We model the judge as a known channel on the complete graph: given the meanings, it declares each pair of answers equivalent independently, with probability within a meaning and across meanings. With at most three meanings, the edge and wedge averages of the verdict graph estimate the pair and triple collision probabilities, which fix the entropy; with four they do not, and the three-star average supplies the missing moment. For answers, at most four meanings and bounded away from and , the minimax squared risk has order on every fixed polynomial window , , and the entropy is uniformly consistently estimable exactly when . With the critical exponent is , against for three meanings in concurrent work, so a judge with suffices for three meanings and not for four. The rate is attained by an estimator that projects the three averages onto the valid moment set and solves a quartic. For the lower bound, a third-order inclusion–exclusion sweep of the likelihood score, whose coefficients obey an exact motif identity, bounds the Fisher information. If only pairs fixed in advance are judged, the risk bound depends on the number of three-stars they form, which hub designs maximize up to a constant factor.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
semantic entropy, uncertainty quantification, minimax rates, stochastic block model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3YVXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper is trying to evaluate the uncertainty of language model even when the judge is weak to decide what meaning each answer belongs to. Under the setting of at most 4 meanings for nn answers, this work conducted the lower bound and upper bound of the semantic entropy for these answers by fully utilizing structures on judge graph.

Critical strengths

S1: The design and derivation of the lower bound and upper bound for semantic entropy is ingenious and rigorous. And this work pushes the bound from at most three meanings setting to at most four meanings setting, which can bring some insights to the community.

Critical weaknesses

W1: The work is really difficult to follow. The presentation of this paper does not provide enough background explanation to the motivation. It is hard to know why the authors need to find the minimax rate of semantic entropy for at most four meanings, which is not a practical setting in real life.

W2: The anonymous concurrent submission that give the proof under at most 3 meanings is not properly cited. And the reliability of such work is doubtful.

Questions for authors

Q1: Is it reasonable to use the same q and delta for all responses? Why do this?

Q2: Why assume that there are at most four meanings? Would such a restrictive assumption be still limiting even it has made contribution to existing work?

Q3: How can we know the distinguish ability of judger? It looks like Δ\Delta is importance for the estimation, but knowing DeltaDelta in real life is not very simple.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3YVX (1/2): W1

Thank you for the review. We take the presentation score seriously. This reply first explains the motivation in plain terms, and we will rewrite the opening of the introduction along the same lines.

One clarification, since our presentation may have caused it: the upper and lower bounds in the paper are bounds on the error with which the entropy can be estimated, not bounds on the entropy itself. The upper bound limits the worst-case expected squared error of our estimator, and the lower bound is an error level that no estimator can beat on some meaning distribution.

W1. Presentation and motivation; why the minimax rate for at most four meanings.

In plain words. "Paris" and "the French capital" give one answer in different words. Semantic entropy measures how varied the meanings of a model's sampled answers are, and a judge decides which pairs of answers mean the same thing. The usual estimator groups the answers using these verdicts, so judge mistakes can distort its entropy estimate. We ask how accurately the entropy can be measured from such a judge, by any method.

The observation. With at most three meanings, the chances that two and that three independent answers share a meaning determine the entropy. Under model (1), these chances can be estimated by averaging adjusted verdicts over pairs of answers, and products of two adjusted verdicts that share one answer over triples, even when a single verdict is only slightly better than chance. Four meanings is the first case where two distributions can agree on both chances and still have different entropy. Averaging products of three adjusted verdicts from one answer to three others then estimates the missing fourth chance, that four answers share a meaning.

What the theorems say. The upper bound limits our estimator's worst-case expected squared error, and the lower bound shows that every estimator incurs an error of that order on some meaning distribution, in the stated regime. In symbols: write Δ\Delta for the judge's advantage, so that it says "equivalent" with probability q+Δq+\Delta for a same-meaning pair and qq otherwise. For qq bounded away from 00 and 11, uniformly consistent estimation is possible if and only if n4Δ6→∞n^4\Delta^6\to\infty (Corollary 3), and the best achievable worst-case squared error is of order min⁡{1,(n4Δ6)−1/3}\min\{1,(n^4\Delta^6)^{-1/3}\} on the windows nΔ2≤n−κn\Delta^2\le n^{-\kappa} (Theorems 1 and 2).

Why four. The distributions (12,14,14,0)(\tfrac12,\tfrac14,\tfrac14,0) and (0.45,0.4,0.1,0.05)(0.45,0.4,0.1,0.05) both have P2=0.375P_2=0.375 and P3=0.15625P_3=0.15625, the probabilities that two and that three answers share a meaning, yet their entropies differ by about 0.0660.066 nats (Section 1). Four is also where the answer changes: for Δ=n−γ\Delta=n^{-\gamma} the critical exponent drops from 3/43/4 with three meanings, as reported in the concurrent work, to 2/32/3 with four, where uniformly consistent estimation requires γ<2/3\gamma<2/3. So the strength required of the judge depends on the number of meanings. "At most four" covers prompts with one, two, three or four meanings.

A real example, from the experiments we ran for this response (top-level comments "Experiments with real LLM samples and real judges"). Asked "With 47 attacks in the year 2010 alone, what body of water is known as Pirate Alley?", Llama-3.1-8B-Instruct gave 30 short answers that the reference grouped into four meaning groups: 18 say the Gulf of Mexico, 10 the Gulf of Aden (the correct answer), one the Gulf of Guinea and one the Cape of Campeche. The entropy of these proportions is 0.90 nats, and the model's greedy answer, "Gulf of Mexico", is wrong: the uncertainty that semantic entropy is meant to reveal. Among the 250 new short-answer questions, 19 have exactly four groups and 153 have at most four.

Besides the new opening of Section 1, we will add a notation table to the appendix and a worked example with concrete numbers of answers and judge calls to Section 2.

AuthorsOpenReview ↗

Response to Reviewer 3YVX (2/2): W2 and Q1 to Q3

W2. The concurrent anonymous submission.

Citation. The citation is anonymous because that manuscript is under double-blind review. The bibliography entry carries no title, however, which makes it less informative than it should be. We will replace it with an anonymous entry that carries the title.

Reliability. All results used in our proofs are established within this submission. The tools that the two papers share are restated here with their proofs, and the concurrent manuscript is cited for comparison and context: its three-meaning rate and threshold, the statement that the meanings cannot be recovered once nΔ2→0n\Delta^2\to0, and two results mentioned in Appendix A that we do not treat. None of these is a premise of a proof in this paper, so Theorems 1, 2 and 7 and Corollaries 3 and 8 can be checked without it. The first paragraph of Appendix A separates the shared ingredients, their extensions here, and the new arguments.

Q1. Is it reasonable to use the same qq and Δ\Delta for all responses? Why do this?

It is a simplification, and it is exact in one situation beyond its literal reading. Suppose each pair {i,j}\{i,j\} has its own unobserved rates (qij,Δij)(q_{ij},\Delta_{ij}) that give valid probabilities, are drawn independently across pairs and independently of all meanings, and have common means (qˉ,Δˉ)(\bar q,\bar\Delta), and the verdicts are independent given the meanings and the rates. After the rates are integrated out, the verdict graph follows model (1) with the average rates, because the probability mass of a Bernoulli variable is affine in its parameter, and our theorems apply under their own hypotheses with (qˉ,Δˉ)(\bar q,\bar\Delta). This is a statement about risk averaged over the rate draws, and it does not cover arbitrary fixed pair-specific rates. We will add it after the model.

Why this model: two rates are the smallest description of a judge that separates same-meaning pairs from different-meaning pairs, and they make the answer a formula in nn and Δ\Delta. It is the standard model of a noisy pairwise oracle in the clustering literature (for example Mazumdar and Saha, 2017, with a symmetric error rate) and the observation model of the earlier analysis of semantic entropy by Liu (2026).

What the model does not cover: rates that depend on the meaning, and verdicts that are dependent because they share an answer; both are stated in the Limitations. In our experiments with real judges, the estimated rates vary between questions and between reference groups within a question; these differences include sampling noise, and the diagnostics do not separate rate heterogeneity from dependence.

Q2. Why at most four meanings? Is such an assumption still limiting?

W1 gives the reason for four. It is a limit of the paper, and Section 6 says so: prompts with more than four meanings are outside the theorems. In our experiments, the reference assigns at most four groups to the 30 answers of 56% to 61% of the evaluation questions, and most greedy answers that fail the gold-alias match lie in questions with more than four groups among their 30 answers; these are counts in a sample, not the support of the meaning distribution. For KK meanings we conjecture the threshold γc(K)=K/(2K−2)\gamma_c(K)=K/(2K-2), which gives 3/43/4 and 2/32/3 for K=3,4K=3,4 and decreases towards 1/21/2, so more meanings would require a stronger judge. Proving it needs a least-favourable path and an entropy transfer for general KK, which we do not have.

Q3. How can the distinguishing ability of the judge be known?

By measuring it on pairs whose relation is known. Take mm calibration verdicts on pairs of answers known to share a meaning and mm on pairs known to differ, independent of each other and of the estimation data. The two agreement frequencies p^\hat p and q^\hat q estimate q+Δq+\Delta and qq. Run Algorithm 1 with (q^,p^−q^)(\hat q,\hat p-\hat q) when p^>q^\hat p>\hat q, and return log⁡2\log2 otherwise. Corollary 31(ii) bounds the resulting risk for every finite mm: provided n4Δ6→∞n^4\Delta^6\to\infty, consistency is kept once mΔ2→∞m\Delta^2\to\infty, and for nΔ2≤1n\Delta^2\le1 the rate of the upper bound is kept once m≥n4Δ4m\ge n^4\Delta^4. Corollary 31(i) gives the tolerance under its conditions: the rate survives an error in the two rates of order Δ (n4Δ6)−1/2\Delta\,(n^4\Delta^6)^{-1/2}.

Some side information of this kind is needed: the concurrent analysis reports that with an unknown background rate qq the verdict graph alone does not permit uniform consistency in its weak regime. Labelled pairs are one sufficient source. The model describes one prompt, so reusing one calibration set across prompts requires that the same two rates apply to their answer populations, which is an additional assumption.

We hope these explanations make the motivation and the scope of the paper clearer, and we are glad to answer further questions before the discussion closes.

Reviewer_pHMROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies semantic-entropy estimation from noisy pairwise equivalence judgments, when answers have at most four latent meanings. It introduces a three-star moment estimator, proves the minimax rate min⁡1,(n4Δ6)−1/3\min\\{1,(n^4\Delta^6)^{-1/3}\\} and the consistency threshold n4Δ6→∞n^4\Delta^6\to\infty, and analyzes sparse query graphs. Validation is limited to simulations under the proposed synthetic channel model.

Critical strengths
  1. The paper identifies a nontrivial difference between three and four meanings and shows why three-star statistics recover the missing fourth collision moment.
  2. It provides a fairly complete minimax treatment, including an estimator, upper and lower bounds, and query-design results.
Critical weaknesses
  1. The paper has no experiments with real LLM outputs or judges, and no downstream evaluation, so its practical machine-learning relevance is unclear.
  2. The assumptions are restrictive: known and homogeneous q,Δq,\Delta, conditional independence, and at most four meanings. Robustness to unknown or misspecified judge behavior is not established.
Questions for authors
  1. Are q and ∆ known in practice, or estimated? What guarantees hold under finite calibration data?
  2. How does the method behave with real LLM samples, heterogeneous judge accuracy, or correlated pairwise judgments?
2 comments
AuthorsOpenReview ↗

Response to Reviewer pHMR (1/2): W1 and W2

Thank you for the review and for the accurate summary. Our replies follow your numbering.

W1. No experiments with real LLM outputs or judges; no downstream evaluation.

The submitted paper has no such experiments, so we ran two pre-registered TriviaQA studies for this discussion: 800 distinct questions, 36,000 sampled answers from Llama-3.1-8B-Instruct, five judges, and reference meanings from two strong LLMs. On 139 held-out sentence questions with at most four reference groups among their 30 answers, with identical answers treated as having the same meaning by both methods, Algorithm 1 with the NLI judge of Kuhn et al. (2023) has entropy RMSE 0.45 and 0.43 nats at 10 and 30 answers, against 0.58 and 0.74 for greedy clustering. In a post hoc check that caps clustering at log 4, the largest value Algorithm 1 can return, clustering has RMSE 0.51 and 0.56, and the paired differences in mean squared error, Algorithm 1 minus capped clustering, are -0.063 [-0.093, -0.033] and -0.132 [-0.184, -0.087] (95% bootstrap intervals). A constant fitted on calibration questions has 0.29 and remains more accurate on these questions, 66 of which have a single reference group. On short answers, merging identical answers makes clustering competitive with Algorithm 1 or better (seven of 16 paired comparisons among the four original judges favour clustering, one favours Algorithm 1). For downstream use, ranking greedy answers that fail the gold-alias match across all evaluation questions, 12 of 24 paired AUROC intervals for these four judges favour identity-merged clustering and none favour Algorithm 1; most such answers lie in questions with more than four reference groups. The top-level comments "Experiments with real LLM samples and real judges" give the results for every judge and sample size.

What the theory offers machine-learning practice is a limit that holds for every method within the model, and an estimator that reaches it. With at most four meanings and qq bounded away from 00 and 11, no estimator is uniformly consistent unless n4Δ6→∞n^4\Delta^6\to\infty (Corollary 3(ii)). On the windows nΔ2≤n−κn\Delta^2\le n^{-\kappa}, the minimax rate depends, up to constants, only on n4Δ6n^4\Delta^6, and Algorithm 1 attains it (Theorems 1 and 2). Its three statistics cost O(n2)O(n^2) arithmetic operations. Keeping n4Δ6n^4\Delta^6 unchanged when the separation Δ\Delta is halved multiplies the number of sampled answers by 23/2≈2.82^{3/2}\approx2.8, a scaling comparison rather than a minimum sample size.

Published judge accuracies vary with the setting: Kuhn et al. (2023, Appendix A.3) report over 92% accuracy of their equivalence judge against manual labels, and 61% for answers sampled at temperature 1.5, and Balamurali and Cheng (2025, Table 1) report accuracy 0.56 and Matthews correlation 0.004 for an 8B-parameter LLM judge that evaluates verbose answers against reference answers, a different task from pairwise equivalence. These aggregate metrics neither identify our separation Δ\Delta nor validate our channel assumptions. We will mention them in Section 5 with this caveat, and add a worked example with concrete numbers of answers and judge calls to Section 2.

W2. Restrictive assumptions; robustness to unknown or misspecified judge behaviour.

Part of this robustness is established in the paper, in the Calibration paragraph of Section 2 and in Corollary 31, and we will make it more visible.

  • Misspecified rates. Run Algorithm 1 with (q′,Δ′)(q',\Delta'), Δ′>0\Delta'>0, in place of (q,Δ)(q,\Delta), and put s=Δ/Δ′s=\Delta/\Delta', β=(q′−q)/Δ′\beta=(q'-q)/\Delta'. For ∣s−1∣≤1/2|s-1|\le1/2 and ∣β∣≤1|\beta|\le1 the risk is at most 904 (V4+(∣s−1∣+∣β∣)2)1/3904\,(V_4+(|s-1|+|\beta|)^2)^{1/3} (Corollary 31(i)); for nΔ2≤1n\Delta^2\le1 the rate is kept when the error in the two rates is of order Δ (n4Δ6)−1/2\Delta\,(n^4\Delta^6)^{-1/2}.
  • Unknown rates. They can be estimated from labelled pairs with a finite-sample guarantee (Q1).
  • Pair-to-pair variation. If each pair has unobserved rates that give valid probabilities, are drawn independently across pairs and of all meanings, and have common means (qˉ,Δˉ)(\bar q,\bar\Delta), and the verdicts are independent given the meanings and the rates, then integrating the rates out gives model (1) with the average rates, because a Bernoulli mass is affine in its parameter. Theorems 1, 2 and 7 then apply under their respective hypotheses with the average rates. This concerns risk averaged over the rate draws, not arbitrary fixed pair-specific rates; we will add it after the model.

Not established, and stated in the Limitations: rates that depend on the meaning, dependence between verdicts that share an answer (Q2), and more than four meanings. "At most four" includes one, two or three meanings, since zero coordinates of θ\theta are allowed.

AuthorsOpenReview ↗

Response to Reviewer pHMR (2/2): Q1 and Q2

Q1. Are qq and Δ\Delta known in practice, or estimated? What guarantees hold under finite calibration data?

They can be estimated, and Corollary 31(ii) gives the guarantee for a finite calibration set. Take mm calibration verdicts on pairs of answers known to share a meaning and mm on pairs known to differ, independent of each other and of the estimation data, with agreement frequencies p^\hat p and q^\hat q. Run Algorithm 1 with (q^,p^−q^)(\hat q,\hat p-\hat q) when p^>q^\hat p>\hat q, and return log⁡2\log2 otherwise. For every θ\theta the risk is at most

904 (V4+4mΔ2)1/3+4(log⁡4)2e−mΔ2/32.904\,\Big(V_4+\frac{4}{m\Delta^2}\Big)^{1/3}+4(\log 4)^2e^{-m\Delta^2/32}.

Provided n4Δ6→∞n^4\Delta^6\to\infty, consistency is kept once mΔ2→∞m\Delta^2\to\infty. If nΔ2≤1n\Delta^2\le1 and m≥n4Δ4m\ge n^4\Delta^4, Corollary 31(ii) also bounds the risk by 1600min⁡{1,(n4Δ6)−1/3}1600\min\{1,(n^4\Delta^6)^{-1/3}\}, so the rate of Theorem 1 is kept. In the weak regime nΔ2→0n\Delta^2\to0, the choice m=⌈n4Δ4⌉m=\lceil n^4\Delta^4\rceil keeps this rate while the 2m2m calibration calls are a vanishing fraction of the (n2)\binom n2 calls on the estimation data.

Two caveats apply. The corollary needs the true relations of the calibration pairs. And the model describes one prompt: a calibration set can be reused across prompts only if the same two rates apply to their answer populations, which is an additional stationarity assumption. If the rates differ, Corollary 31(i) bounds the effect of the mismatch under its conditions.

Q2. Behaviour with real LLM samples, heterogeneous judge accuracy, or correlated pairwise judgments.

  • Real LLM samples. See W1 and the top-level comments "Experiments with real LLM samples and real judges".
  • Heterogeneous judge accuracy. Covered in the restricted form of W2, not when the accuracy depends on the meaning. In the experiments, nominal tests flag unequal rates across reference groups in 21% to 62% of the eligible questions for the four original judges; these tests are descriptive.
  • Correlated judgments. Not covered by our guarantees, and the estimator is sensitive to them. Let answer ii shift the agreement probability of all its pairs by bib_i, where the bib_i are i.i.d. with mean zero, variance s2s^2 and third moment μ3\mu_3, independent of the meanings. Assume the shifted probabilities are valid and the verdicts are independent given the meanings and the shifts. A direct calculation gives: the edge average stays unbiased for P2P_2, the wedge average has mean P3+s2/Δ2P_3+s^2/\Delta^2, and the three-star average has mean P4+3P2 s2/Δ2+μ3/Δ3P_4+3P_2\,s^2/\Delta^2+\mu_3/\Delta^3. When ss is comparable to Δ\Delta, the wedge shift alone is of order one. The shift is small only if both s2/Δ2s^2/\Delta^2 and μ3/Δ3\mu_3/\Delta^3 are small. We have not bounded the variance of the statistics under this dependence, so this is a sensitivity calculation and not a theorem. In the experiments, verdicts that share an answer have positive mean residual products, which can also reflect meaning-dependent rates. We will add the calculation, with a simulation of the size of the effect, to the appendix, and sharpen the corresponding sentence of the Limitations.

We hope these answers settle your questions about unknown and misspecified judge rates, and we are glad to answer further questions before the discussion closes.

Reviewer_PbvqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how well semantic entropy can be estimated when a weak judge decides, pair by pair, whether two answers from a language model mean the same thing. It assumes the answers fall into at most four meanings and that the judge is only slightly more likely to agree on answers that share a meaning. In this regime the meanings themselves cannot be recovered, yet the paper shows the entropy can still be estimated. The key observation is that with four meanings, simple pair and triple statistics are no longer enough, and a statistic based on three-stars in the verdict graph supplies the missing information. The authors give an estimator built on these statistics, prove matching upper and lower bounds on its error, and show that the judge's advantage needed for consistent estimation is smaller than in the three-meaning case. Synthetic simulations support the theory.

Critical strengths
  1. I think the central result is clean and sharp. Showing exactly which extra statistic is needed for four meanings, and how it changes the threshold for consistent estimation, explains the problem well, and the small example in the introduction makes the need for it easy to see.

  2. The matching lower bound is, in my view, the main technical contribution, and the proof technique may be useful beyond this setting.

  3. The simulations, the limitations section and the appendix that separates this work from the concurrent three-meaning paper are all clearly written.

Critical weaknesses
  1. The analysis assumes the judge behaves the same way on every pair and that its verdicts are independent. I feel real NLI or LLM judges are unlikely to behave this way, so it is not yet clear to me how much of the theory carries over to practice.

  2. All experiments are synthetic. The theory also needs many samples per prompt and a judge call for a large number of pairs, whereas semantic entropy is usually computed from about ten samples. I think even a small experiment with a real judge could show whether this regime arises in practice.

  3. Much of the setup, several tools and the three-meaning comparison come from a concurrent anonymous submission. Since reviewers cannot see that work, I found it harder to judge how large the new contribution is, which in my reading lies mainly in the lower bound.

  4. The lower bound holds only in part of the weakly separated regime, and the extension beyond four meanings remains a conjecture. The authors state both limits openly, but I feel they narrow how general the results are.

Questions for authors
  1. How can the judge's parameters be obtained in an enterprise pipeline, given that the concurrent work suggests they cannot be estimated without labelled pairs?

  2. How sensitive is the estimator to verdicts that are correlated because they share an answer?

  3. Is the bias at the boundary distributions in the simulations mainly due to the projection step?

Suggestion:

  1. A short worked example with a realistic number of samples and a plausible judge, showing how many judge calls would be needed, might help readers see when the results apply.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Pbvq (1/3): W1 and W2

Thank you for the careful reading. We agree that the lower bound is the main technical contribution. One clarification of the summary: the judge's advantage needed for consistent estimation is larger with four meanings than with three, not smaller. For Δ=n−γ\Delta=n^{-\gamma}, consistency requires γ<2/3\gamma<2/3 with four meanings against γ<3/4\gamma<3/4 with three, so as nn grows four meanings require a larger separation.

W1. A judge that behaves the same on every pair, with independent verdicts.

  • Wrong values of the two rates. Run Algorithm 1 with (q′,Δ′)(q',\Delta'), Δ′>0\Delta'>0, in place of (q,Δ)(q,\Delta), and put s=Δ/Δ′s=\Delta/\Delta', β=(q′−q)/Δ′\beta=(q'-q)/\Delta'. For ∣s−1∣≤1/2|s-1|\le1/2 and ∣β∣≤1|\beta|\le1 the risk is at most 904 (V4+(∣s−1∣+∣β∣)2)1/3904\,(V_4+(|s-1|+|\beta|)^2)^{1/3} (Corollary 31(i)); for nΔ2≤1n\Delta^2\le1 the rate is kept when the error in the two rates is of order Δ (n4Δ6)−1/2\Delta\,(n^4\Delta^6)^{-1/2}.
  • A restricted form of pair-to-pair variation. If each pair {i,j}\{i,j\} has unobserved rates (qij,Δij)(q_{ij},\Delta_{ij}) that give valid probabilities, are drawn independently across pairs and of all meanings, and have common means (qˉ,Δˉ)(\bar q,\bar\Delta), and the verdicts are independent given the meanings and the rates, then integrating the rates out gives model (1) with the average rates, because a Bernoulli mass is affine in its parameter. Theorems 1, 2 and 7 then apply under their own hypotheses with (qˉ,Δˉ)(\bar q,\bar\Delta). This concerns risk averaged over the rate draws, not arbitrary fixed pair-specific rates. We will add this remark after the model.
  • Not covered, as the Limitations state: rates that depend on the meaning, and dependence between verdicts that share an answer (Q2). Our experiments with real judges show patterns consistent with meaning-dependent rates, dependence through a shared answer, or both.

W2. Synthetic experiments; many samples and many judge calls.

Experiments. For this response we ran two pre-registered experiments with real samples and real judges, reported in the top-level comments "Experiments with real LLM samples and real judges"; the submitted paper has none, and Section 6 states that the model has not been validated on real judges. On whether the weak regime arises: at pooled rates the useful judges are strongly separated on short answers (0.69 to 0.91). For the 0.5B judge on sentences, low estimated separation is common question by question: of 73 evaluation questions with at most four reference groups and at least ten pairs of each kind, 26 have a positive estimate with 10Δ^2<110\hat\Delta^2<1 and 17 with 30Δ^2<130\hat\Delta^2<1 (a post hoc count). These estimates use the 30 answers of each question, include sampling noise and do not establish the asymptotic regime. At ten answers, on the 153 new short-answer and 139 sentence questions with at most four reference groups and against the 30-answer reference entropy, Algorithm 1 with the identity rule has root mean squared error 0.42, 0.24, 0.29 and 0.29 on short answers for the 0.5B, 3B, 8B and NLI judges, against 0.28 for a constant, and 0.33, 0.33, 0.30 and 0.45 on sentences, against 0.29; clustering with the same rule has 0.31, 0.22, 0.28 and 0.26, and 0.31, 0.30, 0.31 and 0.58.

Sample size. The requirement is a property of the problem: for qq bounded away from 00 and 11, no estimator is uniformly consistent unless n4Δ6→∞n^4\Delta^6\to\infty (Corollary 3(ii)), and on the windows where the rate is matched the minimax risk depends, up to constants, only on n4Δ6n^4\Delta^6 (Theorems 1 and 2). Halving Δ\Delta at fixed n4Δ6n^4\Delta^6 multiplies nn by 23/2≈2.82^{3/2}\approx2.8, and the scale n=Δ−3/2n=\Delta^{-3/2} is approximately 6, 11, 32 and 89 for Δ=0.3\Delta=0.3, 0.2, 0.1 and 0.05. These are scaling comparisons, not minimum sample sizes.

Published judge accuracies. Kuhn et al. (2023, Appendix A.3) report over 92% accuracy of their equivalence judge against manual labels, and 61% for answers sampled at temperature 1.5. Balamurali and Cheng (2025, Table 1) report accuracy 0.56 and Matthews correlation 0.004 for an 8B-parameter LLM judge that evaluates verbose answers against reference answers, a different task from pairwise equivalence. These do not identify Δ\Delta, and we will mention them in Section 5 with this caveat.

Judge calls. Ten answers give 45 pairs, or 90 directional NLI classifications; skipping pairs of identical answers would leave on average 19, 20 and 36 in the three settings. Under the assumptions of Theorem 7 and Corollary 8, for a query graph fixed before labels and verdicts are seen and a budget n−1≤Q≤(n2)n-1\le Q\le\binom n2, hub designs make the risk bound vanish when n2QΔ6→∞n^2Q\Delta^6\to\infty; this compares upper bounds.

AuthorsOpenReview ↗

Response to Reviewer Pbvq (2/3): W3, W4 and Q1

W3. Reliance on a concurrent anonymous submission.

You are right that several tools are shared and that reviewers cannot inspect that manuscript. All results used in our proofs are established within this submission. The shared tools are restated with their proofs, some in extended form: the general lower-bound tools (Lemmas 9 to 13), the variances of the edge and wedge averages (Lemma 28(ii), (iii)), the entropy transfer (Lemma 29), calibration (Corollary 31) and the approximate projection (Remark 32). The concurrent manuscript is cited for comparison and context: its three-meaning rate and threshold (Table 1, Figure 1(a), and the remark that 2/3≤γ<3/42/3\le\gamma<3/4 suffices for three meanings and not for four), the statement that the meanings cannot be recovered once nΔ2→0n\Delta^2\to0 (Sections 1 and 5), and two results mentioned in Appendix A that we do not treat. None of these is a premise of a proof in this paper, and the first paragraph of Appendix A separates the shared ingredients, their extensions here, and the new arguments.

New in this paper: the three-star average and its variance (Lemma 28(iv)); the least-favourable path inside the fibre of fixed (P2,P3)(P_2,P_3) (Lemma 4); sweeps of order above one, with the motif identity and the factor ε\varepsilon per majority vertex (Proposition 5); the pairing of overlapping motifs with the Taylor remainder, which controls the full likelihood score (Lemma 6); and query design through the number of three-stars (Theorem 7, Corollary 8). Within our construction, sweeps of order one and two leave coefficient second moments of orders ε4\varepsilon^4 and ε5\varepsilon^5 where order ε6\varepsilon^6 is needed (Remark 23, Figure 1(b)), which is why the single-sweep argument for three meanings does not extend.

W4. Lower bound on part of the regime; more than four meanings.

The consistency threshold has no window condition: for qq bounded away from 00 and 11, estimators with vanishing worst-case risk exist if and only if n4Δ6→∞n^4\Delta^6\to\infty (Corollary 3(ii)). The window restricts only where the rate is matched from below, and the matched windows cover every fixed power law Δ=n−γ\Delta=n^{-\gamma} in the weak regime γ>1/2\gamma>1/2: the rate is settled for all γ∈(1/2,2/3]\gamma\in(1/2,2/3] (Corollary 3(i)), and for γ>2/3\gamma>2/3 the risk is of constant order. Open are the range in which nΔ2n\Delta^2 tends to zero more slowly than every power of nn, for example nΔ2=1/log⁡nn\Delta^2=1/\log n, and the range in which it stays of constant order; there Theorem 1 holds and the matching lower bound is not proved. For more than four meanings the moment statistics extend, since the product of centred verdicts over any tree on kk vertices has mean PkP_k (Section 2); the least-favourable path and the entropy transfer do not yet, and we state γc(K)=K/(2K−2)\gamma_c(K)=K/(2K-2) as a conjecture only.

Q1. Obtaining the judge's parameters in a pipeline.

Some side information is needed: the concurrent analysis reports that with an unknown background rate qq the verdict graph alone does not permit uniform consistency in its weak regime. Labelled pairs are one sufficient source. Take mm calibration verdicts on pairs known to share a meaning and mm on pairs known to differ, independent of each other and of the estimation data, with agreement frequencies p^\hat p and q^\hat q, and run Algorithm 1 with (q^,p^−q^)(\hat q,\hat p-\hat q) when p^>q^\hat p>\hat q, returning log⁡2\log2 otherwise. For every θ\theta the risk is at most

904 (V4+4mΔ2)1/3+4(log⁡4)2e−mΔ2/32904\,\Big(V_4+\frac{4}{m\Delta^2}\Big)^{1/3}+4(\log 4)^2e^{-m\Delta^2/32}

(Corollary 31(ii)). Provided n4Δ6→∞n^4\Delta^6\to\infty, consistency is kept once mΔ2→∞m\Delta^2\to\infty; for nΔ2≤1n\Delta^2\le1 the rate of Theorem 1 is kept once m≥n4Δ4m\ge n^4\Delta^4. In a pipeline this is an audit of the judge on answer pairs whose relation is known. The corollary needs the true relations, so labels produced by another judge add an error source that it does not cover, and reusing one calibration set across prompts assumes that the same two rates apply to their answer populations. If the rates differ, Corollary 31(i) bounds the effect of the mismatch under its conditions. In our experiments the per-question separations vary widely, which bears on this assumption; the dispersion includes sampling noise.

AuthorsOpenReview ↗

Response to Reviewer Pbvq (3/3): Q2, Q3 and the suggestion

Q2. Verdicts correlated through a shared answer.

Our guarantees do not cover this case, and the estimator is sensitive to it. Let answer ii shift the agreement probability of all its pairs by bib_i:

Aij∣Z,b ∼ Bern(q+Δ 1{Zi=Zj}+bi+bj),A_{ij}\mid Z,b\ \sim\ \mathrm{Bern}\big(q+\Delta\,\mathbf 1\{Z_i=Z_j\}+b_i+b_j\big),

where the bib_i are i.i.d. with mean zero, variance s2s^2 and third moment μ3\mu_3, independent of the meanings, the displayed probabilities are valid, and the verdicts are independent given (Z,b)(Z,b). Then the edge average is still unbiased for P2P_2, the wedge average has mean P3+s2/Δ2P_3+s^2/\Delta^2, and the three-star average has mean P4+3P2 s2/Δ2+μ3/Δ3P_4+3P_2\,s^2/\Delta^2+\mu_3/\Delta^3. The moment vector is shifted by B=∥(0, s2/Δ2, 3P2s2/Δ2+μ3/Δ3)∥B=\big\|\big(0,\ s^2/\Delta^2,\ 3P_2s^2/\Delta^2+\mu_3/\Delta^3\big)\big\|; when ss is comparable to Δ\Delta, the wedge shift alone is of order one. The projection and transfer steps behind Corollary 31(i) do not use independence and give a bound C(Vdep+B2)1/3C(V_{\mathrm{dep}}+B^2)^{1/3}, with VdepV_{\mathrm{dep}} the sum of the variances of the three statistics under this dependence. We have not bounded VdepV_{\mathrm{dep}}, so this is a sensitivity calculation and not a theorem.

A synthetic check made for this response, with the six test distributions and the grid projection of Table 6 and the true rates supplied: q=0.3q=0.3, Δ=0.1\Delta=0.1, n=400n=400, bi=±sb_i=\pm s with equal probability, 10,000 replicates per cell. The observed means of the three statistics agree with the formulas (all 72 comparisons in this setting within two Monte Carlo standard errors; over all 288 run, the largest deviation is 3.0 standard errors). The largest mean squared error over the six distributions is

s/Δs/\Delta00.10.250.51
largest MSE4.4×10−34.4\times10^{-3}1.0×10−21.0\times10^{-2}2.0×10−22.0\times10^{-2}0.130.131.061.06
root MSE (nats)0.070.100.140.361.03

Answer effects with standard deviation Δ/10\Delta/10 thus more than double the largest mean squared error. For the skewed law b=2sb=2s with probability 1/51/5 and b=−s/2b=-s/2 with probability 4/54/5 (μ3=1.5 s3\mu_3=1.5\,s^3), the last four largest MSEs become 1.0×10−21.0\times10^{-2}, 2.5×10−22.5\times10^{-2}, 0.140.14 and 1.631.63. In our real-judge experiments, verdicts that share an answer have positive mean residual products; for the four original judges, their square root over the separation is 0.08 to 0.13 on short answers and 0.14 to 0.31 on sentences, and since this can also reflect meaning-dependent rates it does not identify ss. We will add the calculation and the simulation to the appendix, and sharpen the corresponding sentence of the Limitations.

Q3. Bias at the boundary distributions.

We cannot isolate the projection as the main source of the bias from these experiments. The three moment averages are exactly unbiased (Lemma 28(i)), so the bias arises in the nonlinear map from moments to entropy, of which the projection is the first step. The projection is often active: the raw moment estimate falls outside the moment set in every replicate at the vertex and in 69% to 100% of replicates at the point with a zero coordinate (per-cell values are in the outputs included in the supplementary material; Appendix I gives 61% to 100% over all cells and distributions). At the vertex the true entropy is zero, so every valid estimate has nonnegative error there; this explains the sign of the bias at the vertex but not at the point with a zero coordinate. Refining the grid point with a local solver changes a cell's mean squared error by at most 2.6% (Appendix I). The bias is included in Theorem 1, which bounds the full mean squared error.

Suggestion. A worked example.

We will add one to Section 2. From Table 6, with q=0.3q=0.3 and Δ=0.1\Delta=0.1 (the judge says "equivalent" for 40% of same-meaning pairs and 30% of different-meaning pairs): n=100n=100 answers need 4,950 judge calls and give largest mean squared error 3.4×10−23.4\times10^{-2} over the six test distributions (root 0.19 nats), and n=400n=400 need 79,800 calls and give 4.6×10−34.6\times10^{-3} (root 0.07). The calls exclude calibration; for scale, log⁡4≈1.39\log4\approx1.39 nats. With Δ=0.2\Delta=0.2 and n=400n=400 the largest error is 2.2×10−32.2\times10^{-3}. Table 6 uses 500 replicates per cell, and the check in Q2 uses 10,000. The example illustrates costs and errors under the model; the results at ten answers with real judges are in W2.

Thank you again for the detailed questions. We are glad to clarify anything further before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.