acceptodds
Under review as a conference paper at ICLR 2027

Certificate Semantics Determines Query Geometry in LLM Evaluation

Abstract

Large language model (LLM) evaluations commonly aggregate responses across prompt formats, option orders, and other representation variants. Such aggregation can obscure whether behavior is consistent within every registered representation group. We formulate representation-conditioned certification as a group-level ternary decision: for a fixed item, determine whether all group-conditioned response probabilities lie above a threshold, all lie below it, or groups occur on both sides. Certificate semantics induces distinct first-order information geometry. Robustness is universal, requiring dense resolution of potential blockers; heterogeneity is existential, requiring only a positive and a negative witness. By specializing established fixed-confidence pure-exploration geometry, we derive closed-form characteristic times and introduce ActiveOrbit, a sequential allocator that tracks the weakest blocker or balances opposite witnesses. We establish item-level anytime correctness and selector-specific first-order behavior. A mechanism intervention matches the predicted regime asymmetry: witness-selective allocation yields exactly query reduction in all six robust scenarios and positive reductions in all six heterogeneous scenarios (mean ). Across six LLM model–task cells spanning BoolQ, PIQA, and RTE and three open-weight model families, ActiveOrbit reduces query cost by – relative to round robin, with every paired 95% confidence interval excluding zero. These results establish a practical principle: query allocation should be designed around the logical structure of the claim being certified.

Primary area:
learning theory
Keywords:
LLM evaluation, statistical certification, pure-exploration bandits, sequential inference, adaptive sampling, representation robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9F1COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates representation-conditioned LLM evaluation as a ternary certification problem and proposes ACTIVEORBIT, a sequential query allocation method based on certificate structure. It derives characteristic times from pure-exploration geometry and reports query savings over round robin.

Critical strengths

1.The universal-versus-existential certificate distinction provides a useful interpretation of sampling requirements. 

2.The common stopping rule supports controlled allocation comparisons.

Critical weaknesses
  1. Limited technical novelty. The characteristic-time derivations specialize existing pure-exploration and convex-hull membership results. Section 7 acknowledges this connection, but the paper does not clearly establish what fundamentally new capability ACTIVEORBIT provides over existing threshold-aware sampling methods. The empirical advantage over stronger adaptive baselines is also less pronounced than the headline round-robin comparison.

  2. The efficiency claims do not fully capture practical evaluation cost. Section 6.1 measures accepted Bernoulli updates, while Theorem 1 excludes raw-query cost and unresolved probability. The reported 12.03%–36.61% savings therefore do not establish equivalent reductions in total model invocations, particularly under parsing failures or abstentions.

Questions for authors

1.What theoretical or algorithmic result distinguishes ACTIVEORBIT from existing CHM sampling approaches?

2.How much improvement remains over the strongest adaptive baseline when measured using total raw model calls?

3.Are the savings preserved at matched certification coverage?

Reviewer_eN4jOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies LLM evaluation sensitivity to query variants, arguing that aggregate certification across prompt formats or option orders obscures whether behavior is consistent within every registered representation group. Motivated by this distinction between average-case and group-wise claims, it formulates representation-conditioned certification as a group-level ternary decision problem: robust positive, robust negative, or heterogeneous. The authors propose ACTIVE ORBIT, a sequential allocator that queries the weakest blocker when robustness is plausible and balances opposite witnesses under apparent heterogeneity, with closed-form characteristic times derived from pure-exploration geometry showing dense blocker support for robust claims versus sparse opposite-witness support for heterogeneous claims. Experiments show query reductions relative to round robin, while a controlled intervention confirms 0% reduction in all robust scenarios and positive reductions in all heterogeneous scenarios.

Critical strengths
  • [S1] The paper addresses a critical practical issue in LLMs evaluations by demonstrating that traditional aggregation of responses (across prompt formats, option orders, and other variants) can obscure inconsistencies in model behavior across different representation groups. By formulating representation-conditioned certification as a group-level ternary decision problem, the authors provide a more robust framework for identifying whether a model is truly consistent or merely relying on pooled response rates.
Critical weaknesses
  • [W1] The paper suffers from several clarity issues that make it difficult to follow the methodology and results. For instance, in the Introduction, the authors mention 1,000 synthetic items but fail to describe what these samples actually look like or the process used to synthesize them. Furthermore, the claims regarding robustness and heterogeneity in the Introduction are confusing: it is not clearly explained why the authors assert that for robustness "every group can remain a blocker" while for heterogeneity "one reliably positive and one negative witness suffice." Additionally, Figures 1 and 2 appear to be low quality; Figure 1 has significant text overflow, and Figure 2 requires much more elaboration to help the reader interpret the geometric characteristic times.

  • [W2] The experimental evaluation primarily compares the proposed ACTIVE ORBIT mechanism against a round robin baseline but lacks a comprehensive discussion of existing works related to the specific problem of representation-conditioned LLM evaluation. A more robust review and comparison against established literature would better consolidate the specific contribution of this work to the field [1].


[1] Kang, Cheongwoong, et al. "When Format Changes Meaning: Investigating Semantic Inconsistency of Large Language Models." EMNLP 2025.

Questions for authors

Refer to the weaknesses section.

Reviewer_JHXaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

From my understanding, the paper studies LLM evaluation when the question is in different formats. The authors argue that aggregating model responses across different ""representations" hides inconsistencies. So they formulate a problem where they determine if the model's response distribution is consistent across all representation groups or if some groups lead to different responses. The propose an adaptive query allocation algorithm to decide which representaiton groups to sample in order to reach a conclusion using fewer queries.

I found the motivation, formulation, and theoretical analysis difficult to follow, and the technical contributions are not communicated well.

Critical strengths

None

Critical weaknesses

very hard to follow and understand, not suited for academic publication.

The abstract and introduction rely heavily on technical terms that don't mean anything in context and not sufficiently motivated or explained or defined. Examples are "certificate semantics", "joint robustness", "information geometry, blockers and witnesses before providing a clear explanation of the problem being studied.

The introduction in second paragraph presents substantial empirical claims without enough context to understand what is being evaluated. for example, the second paragraph reports aggregate certification rate, pairwise robustness, joint robustness but these quantities are not defined or explained at all by this point.

the paper does not communicate the motivation and contributions clearly enough for me to reliably assess their technical significance.

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.