acceptodds
Under review as a conference paper at ICLR 2027

Repetition Is Not Accuracy in Large Language Model Evaluation

Abstract

Repeated queries to a large language model often produce differently worded answers with the same meaning, and evaluators use this semantic agreement as a black-box reliability signal. We ask whether agreement can support an accuracy claim when labeled source prompts and unlabeled deployment prompts come from different populations. A model can repeatedly return the same wrong answer, so agreement measures concentration rather than truth. We capture this distinction with a promptwise agreement bias: the amount by which agreement overstates or understates accuracy. Under an oracle law for labeled source evaluation, the worst-case ambiguity is exactly the deployment mass outside source coverage. When source information is reduced to mean agreement bias and a justified promptwise range, we derive the full compatible risk interval in the corresponding summary model. Conditional shift bounds and finite-sample intervals then separate the effects of sampling new prompts from sampling additional responses to a fixed prompt. Matching constructions show when more responses help, when broader prompt coverage matters, and when target labels or external assumptions remain unavoidable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Uncertainty Quantification, Accuracy Estimation, Distribution Shift, Self Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ea2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors study LLM uncertainty estimation and whether/when self-agreement of repeated model responses can support valid accuracy claims, in the setting where training and deployment inputs/prompts come from different populations.

They present and prove a number of theoretical results. The paper contains no experimental evaluation.

Critical strengths
  • The studied problem (whether self-agreement can support a valid deployment accuracy claim under population/prompt shift) is interesting and important.
Critical weaknesses
  • At least to me personally, I found the paper extremely difficult to follow and understand (I found Section 2 and the start of Section 3 OK, but the rest of the paper very difficult).

  • This is an entirely theoretical work (without any empirical case study or experimental results whatsoever) and its actual practical relevance/usefulness is highly unclear (does this give any concrete guidance or tool beyond just generic common sense advice of "self-consistency should not be assumed to guarantee accuracy" or similar in practice?).

Questions for authors

Questions/suggestions:

  • Could you clarify what the main concrete contributions of the paper are (I struggled to understand point 1 to 4 in the Introduction)? As someone interested in estimation and evaluation of LLM uncertainty, what are the main things I should take away from this paper? Why is your work and these theoretical results important for me? How could/should I use this?

  • Is there really no type of experimental evaluation you could add to the paper? Or at least some kind of simple illustrative example?

Minor things:

  • "Definition 1 (Accuracy agreement and agreement bias)" --> "Definition 1 (Accuracy, agreement and agreement bias)"?
  • You use Definition 1, Lemma 2, Theorem 3, Theorem 4, Corollary 5, etc. I.e. one global numbering for all categories. Could it not be better to instead do Definition 1, Lemma 1, Theorem 1, Theorem 2, Corollary 1, etc?


Rating justification:

This seems like solid work to me and the authors seem very knowledgeable, but I just found the paper very difficult to follow/understand, and its practical relevance/usefulness is highly unclear to me. "I am familiar with this topic and its relevant literature" is correct for me, but I'm not overly confident in the review itself.

Reviewer_MypFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether agreement among repeated LLM responses can support accuracy claims when labeled source prompts and unlabeled target prompts come from different populations. The authors theoretically analyze the conditions under which target error rates can be identified and derive bounds on these rates under distribution shift.

Critical strengths

None

Critical weaknesses

1. The presentation lacks clarity, particularly in explaining key concepts.

The paper is written in an abstract manner, making it difficult to understand the problem setup and the implications of the theoretical results. In Section 2, intuitive meanings and roles of concepts (such as "accuracy, agreement, and agreement bias") in LLM evaluation are not sufficiently explained.

In particular, the meaning of "one fixed task score" in line 104 and the conditions under which the same score can measure both response agreement and correctness remain unclear. Its implications for practical applicability are not sufficiently discussed.

The presentation also contains repetitive explanations about what the results do not establish, which sometimes obscure the main contributions. Furthermore, Appendix B.5 reports 37 verification checks, but their purposes and connections to the main theoretical results are not sufficiently explained.

2. The practical relevance of the results to real LLM evaluation is insufficiently demonstrated.

Although the paper is motivated by real-world LLM evaluation, the analysis relies entirely on theoretical assumptions, without empirical validation using actual LLM responses. Moreover the validity of several key assumptions remains unclear. It is also unclear whether the proposed risk intervals would be sufficiently narrow to be practically useful.

While extensive experiments are not necessarily required for a theoretical paper, some evidence supporting the validity of the assumptions and the practical usefulness of the results would strengthen the contribution.

Questions for authors

1. Empirical validation with real LLMs

Given the motivation of practical LLM evaluation, could the authors explain why no empirical validation using real LLM responses was included? Even a small-scale experiment examining the main assumptions and the resulting risk intervals would be informative.

2. Novelty relative to prior work

The distinction between self-agreement and accuracy has been discussed in prior work. Target risk estimation under distribution shift has also been studied. Could the authors clarify what new theoretical insights each main theorem provides beyond these existing results?

3. Practical interpretation of the three prompt types

How do the easy, ambiguous, and confidently wrong prompt types introduced in Section 2 manifest in real LLM responses? How could these types be distinguished without access to target labels?

4. Validity of the task score and theoretical assumptions

Under what conditions can the same task score be used to measure both response agreement and correctness? How could this score and the main theoretical assumptions be validated in realistic LLM evaluation settings?

5. Purpose and scope of the reproducibility protocol

Could the authors clarify what the 37 verification checks in Appendix B.5 establish and how they relate to the main theorems? How do they support the paper's claims about LLM evaluation?

Reviewer_vqrxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper discusses whether self-agreement can support a valid deployment accuracy claim under population shift. The authors note that “More generations can reveal the response distribution of the model ever more precisely, but they cannot reveal a missing answer key.” (page 1)

Critical strengths
  1. The theory precisely characterizes the worst-case ambiguity under full labeled-source information. Theorem 3 establishes that “Thus β is the exact worst-case identification diameter.” (page 4), giving an attainable boundary rather than only a conservative upper bound.

  2. The sampling analysis carefully accounts for dependence among response pairs. The observation that “Pair scores share responses and are not independent.” (Section 6, page 8) supports the distinction between prompt-sampling and response-sampling uncertainty.

Critical weaknesses
  1. The analysis does not establish how imperfect semantic scoring affects the usefulness of its accuracy guarantees. Although the authors state that “Validating that score is a separate empirical problem.” (Section 2, page 2), a biased similarity score could make an interval valid for the chosen score yet misleading about human-judged correctness. I recommend testing controlled scoring errors and alternative semantic scorers against human-annotated correctness, to clarify this mismatch and strengthen the framework’s practical relevance.

  2. The resource-allocation analysis does not establish how to choose the number of responses per prompt in practice. The paper carefully limits its conclusion to “only the deterministic concentration surcharge is minimized at K = 2.” (Section 6, page 8), but minimizing this bound alone does not establish the best trade-off between interval width and coverage in realistic settings.

  3. The experimental validation is essentially absent. The paper states that "The figure contains no simulated or empirical results".

  4. The writing is dense and hard to follow, and the motivation and presentation need substantial improvement.

Questions for authors

Please see the Weaknesses section.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of Revisions in Response to the Reviews

Dear Area Chair and Reviewers,

Thank you for your careful reading and constructive feedback. We appreciate the recognition of the importance of assessing deployment accuracy under prompt shift, the sharp characterization of worst-case ambiguity under partial source coverage, and the analysis of dependence among response pairs that share a generation.

In response to your feedback, we have made the following revisions.

  • Problem setup and theoretical contributions. We have reorganized the introduction around a worked example and a roadmap that connects each result to the information available to an evaluator. In the example, carrying over the mean source agreement bias gives a risk estimate of 0.511 for a true target risk of 0.640, even with full source coverage. We have also clarified the relationship to prior work and adopted separate numbering for each statement type.
  • Scoring validity. We have clarified how a fixed score measures response agreement and reference accuracy. We have added human-annotation audits and controlled scoring perturbations, and explained how a justified bound on scoring error can extend a score-risk interval to human-judged risk.
  • Interval utility and call allocation. Section 6.1 now reports an equal-budget ARC comparison and a free-form evaluation with an independent target audit. In ARC, both methods covered the reference risk in all 1,000 trials per shift. At a risk tolerance of 0.40, direct weighting certified all trials in both shifts, while agreement transfer certified 68 trials in one shift and zero in the other. The allocation study also shows how prompt costs can affect the choice between narrower intervals and lower point-estimation error.
  • Reproducibility. Appendix C now includes the experimental protocols, full results, and six supplemental checks with runnable code and expected outputs.

Please refer to the latest revised PDF for the full set of changes, highlighted in blue, and for the experimental protocols and results.

Thank you for the revisions. However, as all reviewers have raised concerns about the presentation (rating it 1), I believe substantial improvements in readability are still necessary, particularly in clearly communicating the significance and practical relevance of the theoretical contributions.

Thank you for raising this again. We have made another substantial revision to the manuscript since our last response. We rewrote and more than doubled the opening of the Introduction, following one evaluation team from its first model queries to the decision of whether the model meets a deployment error tolerance. That decision now motivates the theory before we state the contributions.

  1. What the team can measure (Introduction, paragraph 1, p. 1). We start with the evidence the team has on hand. It has reference answers from an earlier evaluation and can query the same model on unlabeled deployment prompts. “Paris” and “The capital of France is Paris” make semantic agreement concrete and show why repeated queries are a useful first check before the team collects deployment labels.

  2. Why the source correction can mislead (Introduction, paragraph 2, p. 1). We now walk through all three prompt types in Table 1. The type on which the model tends to repeat the same wrong answer rises from 20% of source prompts to 50% of deployment prompts. All three types are covered at source, yet the average source correction gives an estimated risk of 0.511 when the true risk is 0.640. At a tolerance of 0.60, relying on the estimate would lead to the wrong deployment decision. We introduce agreement bias here so readers can see why Section 4 asks which prompt types carry the bias when the source record retains only an average.

  3. What repeated queries can establish (Introduction, paragraph 3, p. 1). We have restored the example of a model that gives the same answer every time on prompts outside source coverage. The same observed prompts, responses, and source labels are compatible with deployment risks of zero and one, depending on which answers are correct. This gives Section 3 a concrete purpose. It shows how much uncertainty can remain even when the team knows the complete response distribution.

  4. What evidence would move the decision forward (Introduction, paragraph 4, pp. 1–2). We now explain when direct importance weighting is available and what the team needs to check when source coverage is incomplete, calibration has been pooled, or transfer depends on assumptions about agreement bias. Section 6 returns to the same example. The population interval of [0.376, 0.646] leaves the decision open at 0.60 even with exact population averages. In this case, stronger evidence about how bias changes across domains could narrow the interval. In finite samples, the analysis also shows when more independent prompts or more responses per prompt would help.

The rewritten Introduction is highlighted in blue on pages 1–2 of the latest PDF. The four numbered contributions immediately below these paragraphs link directly to Sections 3–6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.