acceptodds
Under review as a conference paper at ICLR 2027

What Does LLM-Judged Novelty Reward? Controlled Experiments on Context, Workflow, and Measurement Validity

Abstract

Large language model (LLM) research-ideation systems are increasingly evaluated with LLM judges of novelty, yet what those judgments reward is unclear. We vary paper contexts, generation instructions, and revision guidance while holding the research question and main rubric fixed. Judges see only the question and idea texts, in both display orders. The rubric rewards substantive departures from known work. An aligned generation instruction requiring a component beyond combining known methods increased preference across two generator families and two judge families. Giving this instruction only to GPT-5 mini reversed the preference for GPT-5. When the generator was asked to plan verification before revising a draft, the judge more often preferred the version revised without that instruction, even though the proposed verification was never carried out. Retrieving prior work and giving the generator feedback on possible overlap before each review round produced ideas the judge preferred to those from general review. To assess how these judgments relate to scientific novelty, we also examined consistency and agreement with existing expert ratings. Reversing display order changed the judge's preferred idea in nearly one fifth of pairs answered in both orders, and its choices only partly matched expert preferences. The judge's preference alone does not establish that the favored ideas make more original scientific contributions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM-as-a-judge, research ideation, research agent, novelty evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NMGeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors run a series of controlled experiments on LLM judges for research ideation systems, and find effects such as order sensitivity and measure agreement with human ratings from existing papers.

Critical strengths

There is growing interest in LLM-based ideation, and clearly the rapid advances in AI-for-science (and math) results suggest that more is coming in this space. Understanding LLM as a judge is important if we want to scale up evaluations or if we want to use the judges as verifier targets for test-time-search.

Critical weaknesses

It is not clear to me that this paper adds much beyond the existing literature on llm-as-a-judge's biases, as well as various more controlled human evaluation studies on the quality of llm-as-a-judge methods when compared to humans.

While the paper has a human eval section, this is actually reusing human eval data from past studies, and so the main contribution of the paper is in studying various forms of biases (like order sensitivity) and measuring changes as a function of architecture interventions (like retrieval).

However, the former is well studied in past work on llm judgements, and the latter is ungrounded, and it's unclear to me what the 'ground truth' should be in this case.

For all these reasons, it's not clear that there's much of a clear contribution beyond collecting a bunch of experimental observations. The paper is also written in this way, with sections essentially serving as collections of observations. If the observations are robust and timeless, maybe this is fine, but the paper does not do much to convince me of the novelty and robustness of these findings.

Questions for authors

For the llm as a judge biases parts, are you finding any new phenomena?

What are some more general, concrete prescriptions that you have for llm-as-a-judge for ideation as a result of your experiments? do you have any stronger falsifiable claims that you're making about the practice of llms for ideation?

Reviewer_hZmSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies what pairwise LLM novelty judgments of machine-generated research ideas reward, holding the question and rubric fixed while manipulating generation instructions, revision guidance, retrieved contexts, and generator settings (GPT-5 mini/GPT-5, Qwen3 4B–32B; Claude Sonnet 4.5 as primary judge). A one-sentence instruction requiring a "nonstandard component" flips the GPT-5-vs-mini ranking (84.8% under Claude) and requires generation-time changes; requiring a verification plan lowers judged novelty (D_yield = −0.289 to −0.500) though never executed; overlap feedback raises it (+0.328); 18.6% of pairs flip under reversed display order; expert agreement is partial (75.2% decisive RINoBench, 68.8% balanced accuracy on Si). The authors conclude that LLM-judged novelty measures preference under a rubric, not scientific originality. Overall, the question is important and parts of the measurement practice are careful, but the experimental program is fragmented and confounded (W1), the manipulated ideas are never validated by human experts, and the headline reversal carries a generation-time/seed confound. As a result the paper does not establish what LLM-judged novelty rewards.

Critical strengths
  1. Disciplined methodology. Both display orders, pre-registered vs. post-hoc separation, Holm correction, compliance checks, length-matched subsets, and honest "inconclusive" statements (e.g., p=0.69).

  2. A decision-relevant reversal. One rubric-aligned sentence flips the GPT-5 vs. mini comparison and replicates across generators, judges, a plainer rubric, and length-matched pairs; the decomposition (component-only 95.5%, explanation-only 71.4%, inserted-explanation ≈ chance) is genuinely informative.

  3. Valuable counterintuitive finding and honest scope. Verification planning reduces judged novelty across three judge families, and the paper never claims the interventions change actual novelty.

Critical weaknesses
  1. The experimental program is fragmented rather than a coherent controlled design. Conditions drift across sections (28/64/68 questions; four generators and three judge families; two rubrics; 5–397 pairs per condition) with no unified design matrix; key comparisons change several factors at once (the feedback condition adds both papers and an instruction, relevant-vs-unrelated differs by 30% in length, Qwen3 size is collinear with length); instructed ideas are compared with defaults stored from an earlier run, without a default-vs-default control; validation depth is uneven (the verification-plan mechanism is unquantified, execution feedback is weakly instantiated, overlap scores are never checked against the source sentences); and expert-agreement evidence comes from external datasets with unmatched, post-hoc human baselines rather than from within the design.
  2. The construct-validity gap is fundamental. All validity evidence comes from external datasets not produced under these interventions, so the paper cannot tell whether instructed ideas are substantively more novel or merely emit rewarded textual markers.
  3. No human expert experiment on the generated ideas. None of the ideas produced under the controlled conditions is rated by domain experts; all expert-agreement evidence comes from external datasets. A small blind study (e.g., 2–3 experts × 100 pairs from Section 4) would directly test whether the instructed ideas are genuinely more novel, rather than only more preferred by the judge.
  4. Judge validity is modest and the paper is purely diagnostic. 65.3% agreement with RINoBench labels (75.2% on decisive pairs, interval [58.9, 88.3]) and an unmatched 91.8% human baseline computed post hoc; no mitigation is proposed.
Questions for authors
  1. Did you verify that regenerating the default prompt later with the same seeds reproduces the stored defaults' win rates (a temporal control)?
  2. Can you quantify revision substance (edit distance, mechanism changes) to test whether verification planning mainly induces conservative edits?
  3. What fraction of flagged overlaps were true overlaps on manual inspection? If often spurious, the +0.328 gain suggests the judge rewards revision activity rather than de-overlapping.
  4. Can you provide a single design matrix listing questions, generators, judges, rubrics, seeds, and sample sizes per condition, and re-run the key comparisons under one fixed setting?
Reviewer_wqUhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the factors that influence LLM-based judgments of novelty in automatically generated scientific research ideas. Through controlled experiments, the research investigated that how generation instructions, retrieved literature, iterative revision strategies, and model configurations affect the novelty preferences of LLM judges. It found that: 1. explicitly requiring an original, nonstandard component in a generated research idea substantially increases its judged novelty. 2. verification-planning requirements reduce judged novelty relative to general revision. 3. providing retrieved research papers generally increases judged novelty, although the effects of relevance and diversity are less conclusive.

Critical strengths
  1. The paper addresses a fundamental concern: optimization against an LLM novelty judgement may improve the judge's preference but without improving genuine scientific originality.
  2. Several major findings are reproduced across different generators and evaluation models to ensure the quality of the findings.
Critical weaknesses
  1. The first concern is the distinction between an LLM judge preferring a proposal and that proposal containing a genuinely original scientific contribution. The experiments demonstrate substantial changes in judged novelty, but the newly generated ideas are not independently evaluated by domain experts. In addition, the judge receives only the research question and proposal descriptions, without an explicit literature search or the source papers needed to verify whether a claimed contribution is actually new.
  2. Several important comparisons involve interventions that change multiple factors simultaneously.
  3. The expert ratings are drawn from external datasets rather than the generated ideas used in the main experiments. Moreover, the benchmark labels differ in their construction and are not necessarily direct measures of originality relative to all relevant prior work.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.