acceptodds
Under review as a conference paper at ICLR 2027

Predictable Labels, Fragile Rankings: An Empirical Study of LLM Evaluator Procedures

Abstract

Automated hypothesis generation creates a selection problem: which candidates should proceed to expert review, simulation, or experiment? Benchmarks assess evaluators through checkable labels, but a label’s predictability and an evaluator’s ability to recover it are different properties. We examine both on ResearchBench, whose fixed candidate pools link one hypothesis to a designated source study. Across 1,313 eligible items, a supervised cross-fitted candidate-only classifier achieves 98.40% sixteen-way Hit@1 without the background or research question. On the same pools, six zero-shot LLM evaluators average 55.79% source–alternative pairwise recovery when scoring candidates separately (Isolated), but 35.99% when generating their scores together (Vector), below the 50% pairwise reference. These endpoints diagnose label signal and procedure behavior under different training conditions; they are not a common ability ranking. Paired behavioral analysis shows that similar strict-win rates can conceal substantial exchanges among wins, ties, and losses. Controlled comparisons further show that replacing surrounding material can reduce recovery while preserving the target text and scalar output action, with effects that depend on the model and setting. With visible materials matched, it remains unresolved whether joint rather than separate scoring increases or reduces this loss. The evidence supports evaluating label meaning, candidate discrimination, and recovery in the intended context separately before treating benchmark performance as evidence of reliable scientific screening.

Primary area:
datasets and benchmarks
Keywords:
LLM evaluation, LLM-as-a-judge, benchmark evaluation, scientific hypothesis ranking, ranking robustness, context sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7dhYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses whether scientific evaluation benchmarks can measure meaningful judgements when labels are predicted from a set candidate word distribution and rankings depend on the scoring procedure. The authors audit ResearchBench and finds that candidate text alone identifies the source hypothesis with 98% accuracy, while tested LLMs lose in pairwise source recovery. Further experiments establish context sensitivity in specific settings.

Critical strengths
  1. The paper convincingly shows that ResearchBench’s source labels can be recovered without scientific context: the candidate-only classifier reaches 98.40% accuracy, and simply choosing the shortest candidate achieves 79.51%.
  2. Its paired procedure comparisons are also carefully analyzed, separating changes in rankings from ties, parsing failures, and repeated observations.
Critical weaknesses
  1. The strongest context experiment leaves a plausible alternative unresolved. Notably, after presenting fifteen additional text blocks, the prompt asks the model to score “this candidate,” which could create uncertainty about the intended target.
  2. More broadly, the paper establishes that construction cues predict labels and that scoring procedures change rankings, but does not establish whether those cues explain the ranking failures.
Questions for authors
  1. Does the scalar context loss persist when the final instruction explicitly names a clearly delimited target, with the same items, donors, order, and output schema?
  2. Does the Isolated–Vector gap persist on length-matched source–alternative comparisons? Given the strong shortest-candidate baseline, this would help assess whether length differences account for much of the effect.
  3. For the high-performing contemporary judges, what happens when the background and research question are removed or shuffled? This would help distinguish successful source identification through candidate-only cues from effective use of the scientific context.
2 comments

Thank you for noting that the paired procedure comparisons are "carefully analyzed, separating changes in rankings from ties, parsing failures, and repeated observations." Shared points are in the General Response (S1–S4).

Explicit delimited target (Q1). Done; the context loss persists in the tested setting. We reran C and D for GPT-5.6 Sol on a hash-selected subset of 40 items (39 complete dependency groups; selection uses no model outputs). Explicitly delimiting and naming the target leaves a C−D gap of 6.42 pp (95% CI [3.42, 9.92]), compared with 7.00 pp under the original wording on the same subset; items, donors, order, and score schema are unchanged apart from target-reference wording (S2; App. I.1, Table 41, p. 54). The change between wordings, −0.58 [−2.60, 1.42] pp, is unresolved. For reference, the primary gap on all 237 items is 11.53 [9.36, 13.85] pp (§5.1, p. 7). The dual-focus study (S4) is a distinct control, not a repair of the original target wording: it identifies both focal hypotheses but differs in sample, templates, orders, focal organization, and output action (§5.2, pp. 8–9; App. F.6, pp. 37–39).

Length-matched gap (Q2). Done; the gap persists, but this subset does not support below-chance Vector recovery. Restricting source–alternative length differences to at most 10% leaves an Isolated–Vector gap of 19.32 [16.30, 22.38] pp. Vector recovery is 49.66 [47.77, 51.58]%, an interval that includes the 50% random reference (S1; App. H.1, Table 39, p. 53). This restricts saved-output comparisons: the original Vector inputs are unchanged, and other construction cues remain.

Background/question for contemporary judges (Q3). Done for one judge. For GPT-5.6 Sol, pairwise source recovery is 99.83% with intact context, 88.51% with the background and question removed, and 50.90% with them jointly shuffled; ties rise from 0.1% to 22.5% and 92.0% (S3; App. I.2, Table 42, p. 55). This supports sensitivity to the supplied context, while the shuffled result largely reflects ties. The ablation covers one contemporary judge, does not separate background from question effects, and does not establish scientific screening quality.

Remaining limitation. These analyses do not establish that construction cues cause the ranking failures. The classifier-margin strata (App. H.2, Table 40, p. 53) are observational, and the two upper strata contain only 8 and 4 relations; we report them descriptively, not as a causal explanation.

The analyses for all three of your questions are now in the revised paper (Q3 for one contemporary judge); if they resolve your concerns, we would be grateful if you would consider raising your score.

Thank you to the authors for their detailed response. While I appreciate the follow-up note and your added analyses, they still leave my core concerns unresolved. For example, in the length-matched subset, below-chance vector recovery is no longer supported (49.66% spanning the 50% baseline) while strong text-only cues persist. The causal mechanism behind the Isolated-Vector gap also remains open. Moreover, the background/question ablation also tests only a single contemporary judge and largely reflects ties rising to 92% under shuffling - the study still doesn't validate whether higher source recovery translates to better scientific screening. Given these remaining limitations, I will be keeping my score.

Reviewer_CfPWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper deals with evaluating LLMs in the task of identifying the source background and research question for a scientific hypothesis. It uses a previously devised evaluation dataset, the researchBench, which has a true hypothesis and 15 alternatives. The paper then asks how the specific formulation of the question affects the results (e.g. sending the 16 hypothesis one by one and scoring them, or asking the llm to score them all together, etc.). The main finding, I think, is that scoring one by one achieves much better results, whereas scoring them all together is below random recovery. There are other more specific findings.

Critical strengths

None

Critical weaknesses
  • The scope is very limited. If it deals with the general problem of how you should run evals, why fixate on this single benchmark, or why not dive deeper on why it works, or give other papers that discuss this problem. If you are interested with understanding the benchmark itself, than why dive so deeply into presentation orders, some of which obviously do not work. (also, the benchmark itself is not new).

  • The papers writing makes it very hard to parse. This is clear AI slop.

Questions for authors
  • I didn't understand what the main messages of the paper where.
2 comments

Thank you for the review.

Main messages. Our three main messages are:

  1. On ResearchBench, the source hypothesis can be identified from candidate text alone: the shortest-text rule reaches 79.51% and a supervised text-only classifier 98.40%, against 6.25% chance (§3.2, pp. 5–6). A high recovery score on this benchmark therefore needs checking before it is read as judgment quality.
  2. On the full census, changing only the scoring procedure moves mean pairwise recovery from 55.79% (Isolated) to 35.99% (Vector), across the 50% random reference, and changes model rankings (§3.1, pp. 4–5). The 19.80 pp difference comes from tie credit (+20.19 pp), not from more strict wins (−0.40 pp) (§4, p. 6).
  3. Which candidate is selected depends on presentation order and surrounding context, and on other evaluators and label sets the size and direction of procedure effects vary (§6, p. 9; §7, pp. 9–10). The paper is an audit case with a reusable method, not a universal law.

To make these messages easier to find, the revision adds a scope statement to the introduction (§1, pp. 2–3), states the exact tie-credit decomposition in §4 (p. 6), and separates practical implications and limitations in §7 (pp. 9–10). The results are developed in §3.1 (pp. 4–5; recovery and rankings), §3.2 (pp. 5–6; text-only prediction), and §4 (pp. 5–7) with Apps. C.4–C.5 (pp. 20–22) for tie accounting and App. H.1 (p. 53) for length checks; Figures 3–5 (pp. 5–7) show the evidence, and §2.2 (pp. 3–4) defines the endpoints and estimation.

Scope and what is new. We agree that the benchmark is not new; we do not claim a new benchmark. The contribution is to separate three properties that a single benchmark score mixes together, label predictability, scoring behavior, and selection stability, and to measure each on the same fixed candidate pools. On ResearchBench, the labels are predictable from text alone, two procedures differ by about 20 pp in recovery mainly through tie credit rather than strict wins, and Vector winner sets agree across presentation orders in only 14.02% of cases. The task is to identify the hypothesis associated with a supplied background and research question (§2.1, p. 3); the source label is not independently verified truth. §6 tests transfer on two external label sets (App. E.2–E.4, pp. 31–34), three contemporary models on ResearchBench (App. E.1, p. 30), and across selection budgets and candidate-pool sizes (App. D, pp. 22–30). These comparisons delimit transfer; they do not estimate a shared cross-task reversal.

Depth and related work. The tie decomposition accounts for the observed recovery gap, but does not identify its internal cause. Scalar-context controls show sensitivity with the output action fixed; the matched-visibility interaction remains unresolved (§5.1, pp. 7–8; §5.2, pp. 8–9). The introduction relates the study to prompt formatting, choice sets, order, score–comparison inconsistency, and hypothesis-only label cues (§1, pp. 1–2), citing Sclar et al. (2024), Bansal et al. (2024), Tripathi et al. (2025), Tang et al. (2024), Wang et al. (2026), Gururangan et al. (2018), and Poliak et al. (2018).

Presentation orders. Position bias in LLM judges is well documented: changing candidate order alone can alter the judged ranking (Zheng et al., 2023; Wang et al., 2024), and the revision now cites this work in §1. Our presentation-order analyses ask a further question: whether the same source remains eligible for selection when only order changes. Census Vector winner-set agreement is 14.02% (§6, p. 9; App. C.1, Table 6, p. 17); a fixed midrank aggregation lowers recovery from 35.99% to 31.39% (§4, p. 6; Apps. C.5 and D.3, pp. 21–22, 25–27). Average recovery and stable selection thus answer different questions, and the tested aggregation rule does not repair this procedure.

For practice, audit source-label predictability and report how the scoring procedure handles ties (§7, p. 9); scientific screening requires independent expert or experimental validation.

We hope these answers help the reassessment.

Thank you for the clarification.

After reviewing the main messages, I still think that the scope of the paper is very narrow as I explain in my review. I therefore do not change my score.

Reviewer_qcctOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the reliability of LLM-based hypothesis evaluation on ResearchBench. It separates three questions: what makes source-hypothesis labels predictable from candidate text, whether LLM evaluators recover those labels, and how candidate context and scoring format affect their judgments. By comparing separate and joint scoring and controlling surrounding material, it investigates what benchmark performance can establish about an evaluator’s suitability for scientific screening.

Critical strengths
  • The study is focused and has a sensible framework: It examines whether routine scoring choices change which hypotheses survive selection. This addresses a concrete reliability concern for LLM-based scientific screening and could inform how these pipelines are validated.
  • Clear results: (a) The 1313-item 6 model comparison measures procedure effects on the same candidate pools, with paired uncertainty estimates; (b) the candidate-only baseline separates label predictability from evaluator recovery, while the scalar-context experiment shows that recovery losses can occur without requesting vector scores.
Critical weaknesses
  • The benchmark has strong construction cues. Choosing the shortest candidate already gives 79.51% Hit@1, and a candidate-only classifier reaches 98.40%. The rewriting pilot did not produce qualified interventions, so the paper never tests whether the main reversal survives after these cues are reduced. It is unclear how much the result depends on the construction of these candidate pools.

  • The controls do not explain the main 19.80-point Isolated–Vector gap. The original comparison changes candidate visibility, output format, and call count together. The follow-up studies use different templates and focal arrangements. With visibility matched, the separate-versus-joint interaction remains statistically unresolved. The paper establishes procedure sensitivity, but leaves the main reversal unexplained.

  • (Minor): The title is broader than the study. The core of the study is about scientific discovery tasks and hypothesis testing. This is not a major concern, but having a narrower (and more informative) title would help the readers.

Questions for authors
  • Does better source-hypothesis recovery lead to better scientific screening? Can the authors validate this using assessments of truth, novelty, or experimental value?
  • Does the Isolated-Vector reversal persist after reducing candidate-only label cues, for example through length matching or meaning-preserving rewrites?
  • See the minor weakness above.
1 comment

Thank you for noting that the study "has a sensible framework" and "addresses a concrete reliability concern for LLM-based scientific screening." Shared points are in the General Response (S1, S4).

Does the reversal persist after reducing construction cues? Partly. Under length matching (≤10% relative length difference), the Isolated–Vector gap persists (I−V 19.32 [16.30, 22.38] pp), but Vector recovery is 49.66 [47.77, 51.58]%, so this subset does not support below-chance Vector recovery (S1; App. H.1, Table 39, p. 53). Strong text-only cues remain within the matched subset: choosing the shorter text still earns 64.63 [59.13, 70.07]% pairwise credit, and the frozen out-of-fold classifier selects the source in 234/237 matched subpools. As you note, the candidate-rewriting pilot, which sought to change presentation while preserving content, did not yield a validated meaning-preserving intervention, so we report no Isolated–Vector comparison on rewritten candidates. Validating such rewrites is difficult for these texts: most contain numbers, units, or symbols, and the corpus has no validated proposition spans or expert semantic-equivalence labels, so mechanical checks cannot certify that meaning is preserved (App. F.5, p. 37). Length matching and classifier-margin strata (App. H.2, Table 40, p. 53) are the closest controls; both are observational. Thus the gap survives a length restriction, while dependence on other construction cues remains unresolved; the classifier is not an ability comparison with zero-shot judges.

What the accounting explains. The accounting gives a behavioral explanation of the 19.80 pp Isolated–Vector gap, not a causal mechanism. At the level of scored outputs, recovery is 55.79% versus 35.99%; strict wins contribute −0.40 pp and tie credit +20.19 pp, summing before rounding to 19.80 pp (§4, pp. 5–7; Fig. 5, p. 7; transition-level contributions in App. C.4, Table 12, p. 21). The gap therefore arises mostly from tie credit, not from more strict wins. The accounting does not show why the two procedures produce different ties, and it does not separate visibility, output format, and call count. Marked scalar scoring falls between Isolated and Vector, closer to Vector, on the 237-item diagnostic sample (App. D.1, Table 15, p. 23). The control studies use different templates and focal arrangements, and the matched-visibility interaction is unresolved (S4). They establish procedure sensitivity, not a causal decomposition or repair of the original reversal.

Does better recovery imply better scientific screening? Our experiments do not establish this. The reference label identifies the source hypothesis; it is not an independent assessment of truth, novelty, or experimental value (§1, p. 1; §7, pp. 9–10). What we audit is how this proxy should be read and how reliable it is: candidate-only cues strongly predict the label, while scoring procedures change recovery, tie behavior, and model rankings on the same candidate pools. These findings call for validating the intended scoring procedure against independent expert judgments or experimental outcomes; they do not show that Isolated scoring is a better scientific screener.

Title. We keep the title because its object is the evaluator procedure. §1 (pp. 2–3) and §7 (p. 10) state that strong label predictability is a ResearchBench finding and that below-chance recovery concerns the original six-model panel; on other evaluators and label sets, procedure effects vary in size and direction (§6, p. 9). We do not claim a general failure of scientific judgment or LLM evaluation.

If these clarifications and the added analyses address your concerns, we would be grateful if you would consider updating your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers. All page, table, and appendix numbers refer to the revised PDF (55 pages; the main text still ends on p. 10). New analyses are in App. H–I (Tables 39–42, pp. 53–55). All figures were redrawn from the same data for readability (presentation only; captions clarified). Revised passages are marked in blue.

Main messages. (1) On ResearchBench, the source hypothesis can be predicted from candidate text alone (§3.2, pp. 5–6). (2) Changing only the scoring procedure changes source recovery and model rankings on the same pools, and the full-census difference is dominated by tie credit (§3.1, pp. 4–5; §4, pp. 5–7). (3) On other evaluators and label sets, procedure choice still matters, but its size and direction vary (§6, p. 9; §7, pp. 9–10).

S1. Length-matched recovery (App. H.1–H.2, Tables 39–40, p. 53). Restricting relative Unicode-character length difference to ≤10% retains 237 items, 331 relations, and 222 source clusters. This subset differs from the 237-item diagnostic population; only 33 items overlap. The Isolated–Vector gap persists: I−V is 19.32 [16.30, 22.38] pp, with Isolated at 68.98 [66.39, 71.63]% and Vector at 49.66 [47.77, 51.58]%, so this subset does not support below-chance Vector recovery. The gap is 20.22 [16.29, 24.29] and 18.45 [16.17, 20.71] pp at 5% and 20% tolerances. Strong text-only cues remain: choosing the shorter text still earns 64.63 [59.13, 70.07]% pairwise credit, and the frozen out-of-fold classifier selects the source in 234/237 matched subpools and 225/237 original sixteen-text pools (the selection endpoints differ in pool size). Matching is observational and leaves construction cues; original full-pool validity and Vector inputs are unchanged.

S2. Explicit-target control (§5.1, pp. 7–8; App. I.1, Table 41, p. 54). C−D is source recovery with scientific text from other questions appended (C) minus recovery with the other fifteen candidates appended (D) (§5.1, p. 7). On all 237 scalar-context items, the primary GPT-5.6 Sol gap is 11.53 [9.36, 13.85] pp. The explicit-target control reruns C and D for Sol on a hash-selected subset of 40 items in 39 complete dependency groups; selection uses no model outputs. On this subset the original-wording gap is 7.00 pp, and delimiting the focal target changes it to 6.42 [3.42, 9.92] pp. The focal−original change is −0.58 [−2.60, 1.42] pp, and D recovery changes by +0.67 [−1.28, 2.65] pp. Inputs, donors, order, and score schema stay fixed apart from target-reference wording. Original C recovery on this subset is near ceiling (99.83%). The gap persists under explicit wording, while its change is unresolved.

S3. Background/question ablation (App. I.2, Table 42, p. 55). GPT-5.6 Sol scores each hypothesis separately, with no other text appended, on 237 items in 224 dependency groups. Pairwise source recovery is 99.83 [99.70, 99.94]% with intact context, 88.51 [85.74, 91.31]% with context removed, and 50.90 [49.47, 52.43]% with context shuffled (intact−removed +11.32 [8.53, 14.10] pp; intact−shuffled +48.93 [47.39, 50.37] pp); ties are 0.1%, 22.5%, and 92.0%. Candidate texts, labels, order, instructions, and score schema stay fixed; background/question pairs are shuffled jointly within discipline and across groups. Removal also shortens input, and intervals condition on the fixed shuffle.

Intervals in S1–S3 are marginal 95% intervals from 10,000 synchronized bootstrap draws with equal item weights, resampling whole source clusters (App. H.1) or dependency groups (App. I.1–I.2). The length-matched estimates also weight the fixed models equally and retain half-credit ties and zero-credit invalid relations.

S4. Matched visibility (§5.2, pp. 8–9; Fig. 18, p. 39; App. F.6, pp. 37–39). On 199 common-complete items, context losses are 3.02 pp for separate scoring and 2.18 pp for joint scoring; the interaction is 0.84 pp (95% CI [−0.29, 2.14]), leaving the direction of moderation unresolved.

Scope. Source identity does not establish truth, novelty, or experimental value. The full-census below-chance finding concerns the original model panel; contemporary models and the different labels in JudgeBench and TruthHypo do not show a uniform reversal, and TruthHypo's direction depends on readout (§6, p. 9; §7, pp. 9–10).

We hope these answers help the reassessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.