acceptodds
Under review as a conference paper at ICLR 2027

Constructive Divergence: A Judge-Free Reward for Creative Language Generation

Abstract

Creativity is a core capability for autonomous research agents, yet optimizing it remains challenging: verifiable reward signals are scarce, human annotations and LLM-as-a-judge can be costly or contaminated, and heuristic rewards are vulnerable to hacking. To address this challenge, we propose Constructive Divergence, a scalable creativity score requiring only the token probabilities of two reference models. Grounded in the psychological view that creativity requires both novelty and value, we operationalize novelty as the surprisal of an output and value as the recoverability of its original intent. Concretely, we compute surprisal as the negative log-likelihood of the output under a strong reference model, and recoverability as the conditional log-likelihood of the prompt given the output under a second reference model. Notably, Constructive Divergence serves as both a reliable evaluation metric and a cost-effective training reward. As an evaluator, Constructive Divergence is deterministic and robust to reference-model choice, with 70% agreement across model configurations. On paper-abstract generation, optimizing models with DPO and GRPO using Constructive Divergence improves creativity with an average increase of 25%. Our trained models achieve creativity gains up to 4.0 times larger than the average gain of the baselines, and their outputs are preferred by LLM-as-a-judge models in up to 61% of pairwise comparisons, as further examined with human annotators.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Creativity, Reward modeling, Preference optimization, Scientific idea generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RLdnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces Constructive Divergence, which uses two frozen models' probabilities to score surprisal and recoverability via conditioned log-probability. First, when applying Constructive Divergence as an evaluator, CD is more stable and better aligned with human judges. Second, when applying Constructive Divergence as a reward in DPO and GRPO, they show improved creativity under LLM-as-a-judge and human evaluation. Detailed analysis suggests CD-based RL decreases the recombination ratio in ideas and improves the distance to existing papers.

Critical strengths
  • The method constrains small designs, like paraphrasing, to ensure the reliability of the metric.
  • The analysis is interesting.
  • Human evaluation and various LLM judge evaluations are included.
Critical weaknesses
  • As a metric (evaluator), the absolute scores are conditional on the choice of the two reference models, and only setting a fixed choice can ensure that it can be fairly reused in future work. I suggest the authors 1) discuss which is the optimal choice of reference models and why; 2) if presenting it as a model-agnostic evaluator (L086), you should demonstrate high agreement across reference model choices on scenarios other than your improving method's performance.
  • Fundamentally, I am concerned that the design of this metric is, by nature, biased toward the two reference models' familiarity with potential outputs. However, reference models may not always reflect human familiarity or sense of novelty and may exhibit bias. For example, a default reference model is approaching more average human knowledge, which should be largely different from the PhD annotators' group's knowledge distribution. In that case, the metric has limitations in broader usage.
  • The method doesn't seem to be computationally cheaper than LLM-as-a-judge, which makes me concerned about how much motivation other people will still have to use this to replace LLM-as-a-judge.
  • The paper does not consider the validation of the research idea as part of the evaluation. A surprising and recoverable idea can be an invalid or even wrong idea.
  • The math part in Section 2, from my perspective, doesn't bring in too much novel knowledge or information compared with the literature.
Questions for authors
  • Why can log-likelihood reflect the concepts of surprisal and recoverability?
  • How can other researchers directly use your metric (CD) out-of-the-box? And what information can they gain in addition to existing creativity metrics?
  • Regarding weakness 2, people often assess creativity on a detailed downstream use case; for example, evaluating music idea targeting with top musicians as judges. In these cases, is it still true that we can use any general-purpose LLM as a reference? If not, how can we find a good reference model?
6 comments

First of all, thank you for the comprehensive and thoughtful feedback on our work. We sincerely appreciate the reviewer's constructive comments and questions, which provided valuable insights. We address all concerns below.

1. Which reference models are the optimal choice, and is the evaluator really "model-agnostic"? (Weakness 1)

Yes, we select one configuration in the paper (V = gpt-oss-120B, D = Llama-3.1-70B-Instruct, K = 8, σ_S = 0.37, σ_R = 0.90, with the instructions and thresholds; Appendix A.11) and scores are compared under this fixed configuration and for the same prompt.

The configuration selection procedure was only in the appendix and is now summarized in the main text. Before any training, we compared four candidate models (gpt-oss-120B, Llama-3.1-70B, Qwen3.5-122B, Mistral-Large) on four probes that a reference model for novelty should pass (Table 10, Appendix A.7):

  • Probe 1: is unseen work more surprising? We took arXiv abstracts published after the training cutoff of every candidate (March–August 2026) and abstracts from July 2022–June 2023, which the candidates may have seen in training, and measured how often a candidate assigns the higher surprisal to the post-cutoff abstract (AUC; 0.5 means it cannot tell the two groups apart). Only gpt-oss-120B passes: 0.743, against 0.48–0.56 (chance level) for the other three.
  • Probe 2: are well-known ideas less surprising? We measured how often a candidate ranks classic ideas above post-cutoff ones (AUC, lower is better). All four candidates pass (0.08–0.29), so this probe does not separate them.
  • Probe 3: is the score stable under rewording? We measured how much a score changes when the text is paraphrased, relative to the spread of scores across ideas (lower is better). The four candidates are similar (0.32–0.42), so this probe does not separate them either.
  • Probe 4: does the score respond to the argument, not only to the vocabulary? Shuffling the sentences of an abstract destroys its argument but keeps its vocabulary, so a shuffled abstract should not score higher than the original. Only gpt-oss-120B passes: the score rises for 16% of abstracts, against 59–70% for the other three.

gpt-oss-120B is thus the only candidate that passes the two probes on which the candidates differ, and both test whether the model responds to new content.

Also, we define the term "model-agnostic" in the revision paper: the score is computed from token probabilities alone and is not tied to a particular model, so any model that meets the selection criteria above can serve as a reference. The term "model-agnostic" does not mean that absolute scores coincide across any model. Two results support this reading. First, we re-scored the same held-out outputs after swapping one reference model for another: gpt-oss-120B (V) was replaced in turn by each of seven other models with everything else fixed, and Llama-3.1-70B (D) likewise. The replacements range from 9B to 123B parameters across four families (Qwen, GLM, Mistral, Llama). The surprisal gain of the trained policies over Base reappears under every replacement V (d = 0.58–1.36), and the recoverability gain under every replacement D. Second, we ran the test the reviewer suggests: every method of Table 2 plus four controls, re-scored under a replacement pair (Qwen3.5-9B, Llama-3.1-8B-Instruct; new Table 13, Appendix B.2).

Spearman rank correlation of method means between the two pairsSRC
11 rows of Table 20.880.980.81
15 rows (with the four controls)0.760.990.80

The set of methods that raise both terms over Base is identical under both pairs. Where the pairs disagree, the cause is the capability of the reference model, not the definition of the score. The 9B model separates new content from noisy wording less well (Base at temperature = 1.8 gains +0.39 in S against +0.19). And when we retrain the policy with the replacement pair as its reward and score it under the original pair, the recoverability gain transfers fully (+0.39, d = 1.19) while the surprisal gain transfers at about 60% (+0.28, d = 1.22): the loss is concentrated in the term computed by the weaker model (new Table 12). The bottleneck is therefore how well the reference model knows the domain, which is what the probes measure, and a stronger open model can replace V without any change to the method.

2. The metric is biased toward the reference models' familiarity; which reference suits a specific domain? (Weakness 2, Question 3)

This is the same bottleneck as in point 1: the novelty term inherits what the reference model knows, so the bias is a property of the reference model, and it can be measured. On 60 arXiv abstracts written before the training cutoff of V and 60 written after it, with Base outputs generated for the same prompts (Appendix B.7):

arXiv abstractsPublishedBase (same prompts)dPublished higher on
Pre-cutoff (V may have seen them)2.302.45−0.2120/60 prompts
Post-cutoff (V cannot have seen them)2.982.42+0.6946/60

(Context-calibrated surprisal.) The effect of familiarity is therefore measurable, and we control for it by comparing scores only across outputs for the same prompt, where every output is subject to the same familiarity. The conclusions about the trained policies do not depend on one model's familiarity, since they are reproduced under the alternative reference models of point 1.

For a detailed downstream use case such as music, we would not use an arbitrary general-purpose LLM. The reference model must (i) model the domain's text well and (ii) pass the same probe on that domain: items that experts regard as new, and that postdate the model's cutoff, must be more surprising than earlier ones. This probe is how we selected V here, and it requires only dated items, not expert labels. Where no model passes, we would not recommend the score for that domain.

3. The method is not computationally cheaper than LLM-as-a-judge (Weakness 3)

We measured the cost and converted it at public list prices (Appendix A.11):

EvaluatorCost per abstract
Constructive Divergence, K = 8 (10.5–13.3 GPU-seconds on an A100)$0.009–0.012
Constructive Divergence, K = 2 (5.6–6.6 GPU-seconds)$0.005–0.006
One pointwise judge call (Gemini 3.6 Flash / GPT-5.2 / GPT-5.6 Sol)/ $0.0014 / $0.0040 / $0.0057
Pairwise ranking of 8 candidates in both orders (GPT-5.2, 56 calls)at least $0.028

GPU time is priced at $3.19 per GPU-hour, the on-demand list price of an A100 80GB at Lambda. Judge calls are priced at each provider's list API price, with token counts measured on 105 held-out abstracts (about 500 input and 180–260 output tokens per call, reasoning tokens included).

A single judge call is cheaper than our score. The score is nevertheless the cheaper way to obtain a reliable reward, and its cost can be reduced further:

  • A single judge call is not a usable reward. The judges reverse their verdict on 30.0% of pairs when the presentation order is swapped (Table 1), so a judge-based reward needs both orders, and ranking n candidates by comparison grows quadratically. Our score is two to three times cheaper than that protocol and grows linearly.
  • Reducing K lowers the cost further. K is the number of continuations that V samples for the context baseline, and it is the main cost of the score. We re-scored the 650 held-out Base outputs with K = 2 on the same hardware: surprisal scoring took 901 s instead of 2,630 s (34%), and the scores agree with the K = 8 scores at r = 0.99 (means 2.46 vs. 2.47). In an offline recomputation on 1,950 held-out outputs, even K = 1 correlates at r = 0.996 with K = 8, and the effect sizes of both trained policies are unchanged at K = 2. At K = 2 the score costs about the same as one GPT-5.6 Sol call and about one fifth of the pairwise protocol.
  • The score is deterministic. The same output under the same configuration always receives the same score: there is no presentation order, no sampling of a verdict, and no dependence on which other outputs it is compared with. Each output is therefore scored exactly once, stored scores remain valid when a new method is added, and anyone with the open model weights can reproduce them exactly. This matters most when the score is used as a training objective. Training queries the reward for every candidate (8,000 candidates in our DPO run alone), and the reward must stay the same from the first step to the last. Our score is computed locally from frozen weights, so it is unaffected by network latency, rate limits, price changes, or a provider updating or retiring a model, any of which can slow, interrupt, or silently alter a judge-based reward in the middle of training.

The revision reports these figures in the main text, so that the statement that judges "can be costly" refers to the measured cost of an order-balanced protocol rather than to a single call.

4. A surprising and recoverable idea can still be invalid (Weakness 4)

We agree that the CD dose not cover execution-level validness, which requires complex implementation thus beyond an ideation-stage score. Instead, we test the case that an idea that keeps the given keywords and scores high only because it is unusual, with no sound scientific basis. We ran a blind audit in which three LLM judges (Gemini 3.6 Flash, GPT-5.2, GPT-5.6 Sol) rated 650 abstracts per method one at a time on an absolute 1–10 rubric (Table 17, Appendix B.6). Differences from Base, range over the three judges:

MethodNoveltyFeasibilitySoundnessIncoherent claims
DPO_CD+0.06 to +0.13−0.03 to +0.08 (n.s.)−0.01 to −0.09 (n.s., n.s., p = 0.02)0.6–2.1% (Base 1.1–2.6%)
GRPO_CD−0.05 to −0.06 (n.s.)+0.01 to +0.02 (n.s.)−0.07 to −0.13 (p < 0.01)0.1–1.2%
DPO_LLM+0.6 to +0.8−0.17 to −0.250.00 to +0.10up to 5.1%
Creativity Instruction+1.1 to +1.7−1.1 to −1.5−0.6 to −1.35.5–16.4%
Base, t = 1.8−0.3 to −0.8−0.9 to −1.7−1.5 to −2.429–37%

The failure the reviewer describes is exactly what the prompting, temperature, and judge-reward alternatives show: they look more novel to the judges and lose feasibility or soundness. The policies trained with our reward do not: both stay within 0.13 points of Base in feasibility and soundness, and GRPO_CD produces fewer incoherent claims than Base. This is the role of the recoverability term: without it, recoverability stays at the Base level instead of rising (Table 2, w/o Recoverability).

In short, the audit shows that training on the score raises novelty-related signals without the loss of soundness that the alternatives incur.

5. The math does not bring much new knowledge (Weakness 5)

In fact, the entropy–KL decomposition and the mutual-information lower bound explain what the two terms respond to, thus they are presented as motivation rather than as a theoretical contribution. In the main text we introduced the decomposition as "distributional motivation" (Section 3.2), the derivations are in Appendix A.1. The effectiveness of our objective is demonstrated in the subsequent experimental sections.

Reviewer_PsEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Constructive Divergence (CD), a reward mechanism for creative text generation that does not rely on an "LLM-as-a-judge" approach, but is instead derived from the probabilities of a frozen language model. The method combines "surprisal" (the degree of unexpectedness) with "recoverability" (the extent to which the original input information can be reconstructed from the output), encouraging generated content to deviate from conventional expressions while preserving the intent conveyed by the input keywords. This reward is then utilized for training via DPO and GRPO. In the task of generating scientific research abstracts, the authors report that CD improves automated reward scores and outperforms baselines in specific evaluations concerning bias detection and creative generation.

Critical strengths
  1. The research problem is significant. Open-domain creative generation lacks reliable ground-truth answers, and relying on an LLM-based judge can introduce issues such as position bias or model-specific preferences. This paper proposes a reward mechanism that does not require an online judge model, which is of practical value.

  2. The reward design is grounded in both intuition and theory. "Surprisal" encourages deviations from the typical generation distribution of a reference model, while "Recoverability" ensures that the output preserves the original intent. The authors provide a formal interpretation of these components rather than relying solely on heuristic scoring.

  3. The design specifically addresses the issue of "reward hacking." A semantic paraphrase correction is incorporated into the Surprisal component to mitigate the risk of models achieving high scores simply by using obscure or unusual phrasing. Furthermore, ablation studies demonstrate that when Recoverability is removed, the Surprisal score may increase, but the recoverability of key information drops significantly.

Critical weaknesses
  1. The link between the core reward components and genuine creativity remains insufficiently validated. The two components of the "CD" (Creativity-Driven) reward do not directly assess the novelty, feasibility, or scientific merit of an idea. For instance, a model could generate ideas that deviate from the norm but lack a sound scientific basis, all while retaining the original input keywords. The paper currently lacks sufficient evidence to establish a robust connection between high CD scores and high-quality scientific innovation.

  2. Theoretical analysis does not directly imply the effectiveness of the reward. While the decomposition of the surprisal distribution clarifies the relationship between "unexpectedness," entropy, and KL divergence, deviating from a reference model does not inherently equate to enhanced creativity. Similarly, the lower bound on mutual information used for "recoverability" relies on specific probabilistic modeling assumptions. Thus, while these theoretical insights explain the properties of the metrics, they do not prove that maximizing them will optimize creativity as perceived by humans.

  3. Evidence from human evaluations is weak and reveals a discrepancy with automated evaluation results. In the human preference experiments reported, the DPO+CD method achieved a win rate of 54.5% against the baseline, while GRPO+CD reached 40.0%; furthermore, the significance tests reported by the authors do not indicate a statistically significant advantage. Thus, the observed gains in automated CD scores are insufficient to conclude that the method reliably improves creativity as perceived by humans.

  4. The reliance on reference models lacks adequate robustness testing. The "surprisal" component utilizes GPT-oss-120b, while "recoverability" employs Llama-3.1-70b-instruct. Both components are potentially sensitive to the training corpora, linguistic styles, and knowledge distributions of these models. The paper fails to sufficiently demonstrate whether the reward rankings and training benefits remain stable when the reference models are replaced; therefore, the transferability of the proposed method remains unclear.

Questions for authors

Please see weakness.

2 comments

First of all, thank you for the comprehensive and thoughtful feedback on our work. We sincerely appreciate the reviewer's constructive comments, which provided valuable insights and led to several new experiments. We address all concerns below.

1. The link between the reward components and genuine creativity is insufficiently validated (Weakness 1)

We agree that the value term we used in main paper dose not cover execution-level merit, which requires complex implementation thus beyond an ideation-stage score. Instead, we test the case that an idea that keeps the given keywords and scores high only because it is unusual, with no sound scientific basis. We ran a blind audit in which three LLM judges (Gemini 3.6 Flash, GPT-5.2, GPT-5.6 Sol) rated 650 abstracts per method one at a time on an absolute 1–10 rubric (Table 17, Appendix B.6). Differences from Base, range over the three judges:

MethodNoveltyFeasibilitySoundnessIncoherent claims
DPO_CD+0.06 to +0.13−0.03 to +0.08 (n.s.)−0.01 to −0.09 (n.s., n.s., p = 0.02)0.6–2.1% (Base 1.1–2.6%)
GRPO_CD−0.05 to −0.06 (n.s.)+0.01 to +0.02 (n.s.)−0.07 to −0.13 (p < 0.01)0.1–1.2%
DPO_LLM+0.6 to +0.8−0.17 to −0.250.00 to +0.10up to 5.1%
Creativity Instruction+1.1 to +1.7−1.1 to −1.5−0.6 to −1.35.5–16.4%
Base, t = 1.8−0.3 to −0.8−0.9 to −1.7−1.5 to −2.429–37%

The failure the reviewer describes is exactly what the prompting, temperature, and judge-reward alternatives show: they look more novel to the judges and lose feasibility or soundness. The policies trained with our reward do not: both stay within 0.13 points of Base in feasibility and soundness, and GRPO_CD produces fewer incoherent claims than Base. This is the role of the recoverability term: without it, recoverability stays at the Base level instead of rising (Table 2, w/o Recoverability).

In short, the audit shows that training on the score raises novelty-related signals without the loss of soundness that the alternatives incur.

2. The theoretical analysis does not imply the effectiveness of the reward (Weakness 2)

In fact, the entropy–KL decomposition and the mutual-information lower bound explain what the two terms respond to, thus they are presented as motivation rather than as a proof of effectiveness. In the main text we introduced the decomposition as "distributional motivation" (Section 3.2), the derivations are in Appendix A.1. The effectiveness of our objective is demonstrated in the subsequent experimental sections.

3. Human evidence is weak and differs from the automatic results (Weakness 3)

To strengthen the human evidence, we analyze all 9,221 ICLR 2025 submissions with a decision and compared the score with what their reviewers decided.

DecisionOralSpotlightPosterReject
Mean surprisal (n)2.43 (213)2.40 (378)2.34 (3,108)2.30 (5,011)

Surprisal rises monotonically with the decision tier (Kruskal–Wallis p = 9e-10; AUC oral vs. reject 0.585) and correlates with the mean reviewer rating after controlling for length (partial Spearman ρ = 0.10, p = 7e-23), whereas abstract length shows no such ordering (AUC 0.50–0.52).

4. Reliance on the reference models lacks robustness testing (Weakness 4)

The submission re-scored outputs with seven alternative models for each reference model (9B to 123B parameters, four families; all reproduce the gains, d = 0.58–1.36), but only in the appendix. We have moved this to the main text (Section 4.4, Table 3) and added the two tests the reviewer asks for.

Training benefits: retraining with a different reference pair (new Table 12, Appendix B.2). We trained a policy with the same recipe but with rewards from Qwen3.5-9B and Llama-3.1-8B-Instruct, and scored it with the original pair:

Policy (gain over Base under the original pair)ΔS (d)ΔR (d)ΔC (d)
DPO_CD, original pair as reward+0.45 (1.90)+0.38 (1.20)+1.65 (2.10)
DPO_CD′, replacement pair as reward+0.28 (1.22)+0.39 (1.19)+1.20 (1.55)

A policy that never saw the original reference models still improves both terms under them, with about 70% of the full gain. The recoverability gain transfers fully and the surprisal gain at about 60%, so the loss is concentrated in the term computed by the weaker 9B model.

Moreover, in the new Table 13, we report that the Spearman rank correlations between the method means for the two pairs, computed over the 11 rows of Table 2, are 0.88 for S, 0.98 for R, and 0.81 for C, showing that the set of methods that improve both terms over Base is identical for the two pairs. The main disagreement is informative: the 9B model rewards temperature noise more than the 120B model does, while recoverability still falls, which is why the score has two terms and why we recommend a strong V.

Thank you to the authors for their response. These additional details help strengthen the assessment of the work's soundness. However, I still have some concerns about the evaluation methodology. Even papers accepted at ICLR are not necessarily reliable proxies for creativity and may introduce other forms of bias. I believe a more rigorous approach would involve human annotators directly evaluating the generated content.

Overall, I tend to remain my score.

Reviewer_END3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a judge-free creativity score out of two LMs. Creativity/novelty is measured by how surprising an output is to one model and value is how well a second model can recover the prompt’s keywords from that output. The authors use this score as an evaluator and a DPO/GRPO reward for writing research abstracts.

Critical strengths

The paper documents judge biases carefully, and the corrections and ablations are well done.

Critical weaknesses
  • The proposed metric fails basic sanity checks, with real published abstracts scoring below every model. So, this means accepted ICLR papers are the least creative text in the study.

  • The headline gains are mostly on the training objective

  • The human evidence is null

Questions for authors
  • Why would a creativity metric rank published research below base-model outputs?

  • Do the results hold with a larger sample of participants in a human study?

2 comments

First of all, thank you for the careful and candid feedback on our work. We sincerely appreciate the reviewer's comments, which led us to run new experiments. We address all concerns below.

1. Why would a creativity metric rank published research below base-model outputs? (Weakness 1, Question 1)

We agree that this needs an additional experiment. The problem arises when model text is compared with human text, especially when paraphrase correction is used. The paraphrase correction itself is essential; without it, surprisal rewards unusual wording as much as unusual content:

Surprisal gain over BaseWithout correctionWith correction
Base at t = 1.8 (29–37% incoherent claims)+1.08+0.19
DPO_CD+0.40+0.45
GRPO_CD+0.49+0.59

Thus, without the paraphrase correction, temperature noise would outscore every trained policy.

Moreover, it can accurately capture differences in creativity by comparing model-generated texts with other model-generated texts and human-written texts with other human-written texts. On 60 arXiv abstracts written before the training cutoff of the same V(GPT-OSS-120B) used in main paper and 60 written after it, with five Base outputs for the same prompts, we computed the score before and after the paraphrase correction (Appendix B.7):

TextScore before correctionScore after correction (reported)Shift (after − before)
Published abstracts, pre-cutoff2.312.480.16
Published abstracts, post-cutoff2.983.080.10
Base, pre-cutoff keywords2.453.711.26
Base, post-cutoff keywords2.423.651.23

Before the correction, abstracts that V cannot have seen score above Base outputs (2.98 vs. 2.42), and among published abstracts the post-cutoff ones score higher than the pre-cutoff ones, as expected. The reversal comes from the paraphrase shift: paraphrasing lowers the log-probability of a Base sentence by about 1.25 nats per token, but that of a human sentence by only 0.10–0.16. This is the known property that machine-generated text lies near a local maximum of a model's probability, so that any rewrite lowers it, whereas human text does not [1]. It does not depend on V paraphrasing its own text: with Llama-3.1-70B or Qwen3.5-9B as the paraphraser, the shift is 1.2–1.5 for Base against 0.15–0.27 for published abstracts. The correction therefore adds an offset to machine text that is nearly equal across the policies we compare (Base 1.24, DPO_CD 1.29, GRPO_CD 1.34) and absent from human text.

[1] E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. ICML 2023.

Within real human papers, the score moves in the expected direction. Over all 9,221 ICLR 2025 submissions with a decision, surprisal rises monotonically with the decision tier (Oral 2.43 > Spotlight 2.40 > Poster 2.34 > Reject 2.30; Kruskal–Wallis p = 9e-10) and correlates with the mean reviewer rating after controlling for length (partial Spearman ρ = 0.10, p = 7e-23).

In short, this result does not mean that accepted papers are the least creative text. The paraphrase correction is necessary for comparing model outputs, where it shifts every policy alike, but it makes model text and human text incomparable. The new Limitations section states this, and the row is reported only in the appendix as "Published abstracts (own keywords)".

2. The headline gains are mostly on the training objective (Weakness 2)

In fact, we already report several additional signals in the paper. Three LLM judges prefer the trained outputs in 56–61% of pairs. The distance to the nearest published abstract increases (d = 0.38/0.47), whereas the judge-reward policy moves closer (d = −0.30). The rate of recombination among extracted core ideas decreases (0.529 → 0.448), and the gains are replicated across seven alternative models for each reference model.

Nevertheless, we agree that additional experiments are needed to establish the objective’s effectiveness. Three LLM judges read every abstract in isolation, without knowing which method wrote it and without seeing any other output, and rated it on 1–10 scales (650 abstracts per method; Table 17, Appendix B.6):

  • Novelty: originality of the core idea relative to the literature.
  • Feasibility: whether the study can be carried out with typical academic resources.
  • Soundness: whether the claims and the mechanism are consistent and well-founded.
  • Incoherent claims: share of abstracts with a self-contradictory or impossible claim.

Differences from Base, range over the three judges:

MethodNoveltyFeasibilitySoundnessIncoherent claims
DPO_CD+0.06 to +0.13−0.03 to +0.08 (n.s.)−0.01 to −0.09 (n.s., n.s., p = 0.02)0.6–2.1% (Base 1.1–2.6%)
GRPO_CD−0.05 to −0.06 (n.s.)+0.01 to +0.02 (n.s.)−0.07 to −0.13 (p < 0.01)0.1–1.2%
DPO_LLM+0.6 to +0.8−0.17 to −0.250.00 to +0.10up to 5.1%
Creativity Instruction+1.1 to +1.7−1.1 to −1.5−0.6 to −1.35.5–16.4%
Base, temp = 1.8−0.3 to −0.8−0.9 to −1.7−1.5 to −2.429–37%

The alternatives that look more novel to the judges (DPO_LLM, Creativity Instruction) pay for it: DPO_LLM loses 0.17–0.25 points of feasibility and raises incoherent claims to up to 5.1%, and Creativity Instruction loses 1.1–1.5 points of feasibility and 0.6–1.3 points of soundness. Base at temperature = 1.8 loses on every axis (up to 2.4 points). Our policies stay close to Base on all three: feasibility within 0.08 points, soundness within 0.13 points, and fewer incoherent claims than Base under every judge.

3. Do the results hold with a larger sample of participants? (Weakness 3, Question 2)

In our response to Weakness 1, we evaluate the score on 9,221 ICLR 2025 submissions with actual reviewer ratings. The score ranks submissions consistently with both their decision tiers and their reviewer ratings. In a study of 27 pairs used in main paper, LLM judges agree with the human majority on 65–80% of pairs for which a majority decision is reached.

Comments on the paper

2 comments

Dear reviewers.

A gentle reminder: We sincerely appreciate your participation in the review of our research and the valuable feedback you provided. With the discussion period ending in about two day, we would appreciate it if you could check back for our responses and confirm if your concerns have been addressed properly at your convenience.

Sincerely, the Authors

AuthorsOpenReview ↗

Review Summary

We express our sincere gratitude to all reviewers for their time and effort in reviewing our paper and providing valuable feedback.


The reviewers highlighted several strengths:

Reviewer PsEf emphasized the significance of evaluating open-ended creative generation without relying on an online LLM judge, recognizing the practical value of our approach and the intuitive and formal motivation for combining surprisal with recoverability. They also highlighted our explicit safeguards against reward hacking, including paraphrase correction and the recoverability component.

Reviewer RLdn appreciated the design choices supporting the metric’s reliability, the accompanying analyses, and the inclusion of both human and multiple LLM-judge evaluations. The other reviewer specifically praised our careful documentation of judge biases and the corrections and ablation studies.

However, the reviewers raised related concerns: (1) whether improvements in CD reflect meaningful creativity rather than optimization of the training objective, particularly given the limited human evidence; (2) why published abstracts score below model-generated outputs; and (3) sensitivity to reference-model selection and familiarity.


In response, we provided the following additional experiments and clarifications:

(1) We added a blind, pointwise audit by three LLM judges covering 650 abstracts per method and separately assessing novelty, feasibility, soundness, and incoherent claims. CD-trained policies remain close to Base in feasibility and soundness, with fewer incoherent claims, whereas several alternatives achieve larger novelty gains at the expense of feasibility or coherence. We also analyzed the relationship between surprisal and expert assessments of ICLR 2025 submissions: surprisal increases monotonically across decision tiers and shows a modest positive association with reviewer ratings after controlling for length (partial Spearman ρ = 0.10). These results provide complementary evidence.

(2) We added controlled comparisons of abstracts published before and after the reference model’s training cutoff, together with model outputs generated from the same prompts. The analysis identifies a substantially larger paraphrase-induced score shift for machine-generated text, explaining the reversal between published and generated abstracts. We also show that paraphrase correction suppresses rewards for noisy wording. We therefore explicitly restrict comparisons to comparable outputs under a fixed configuration and clarify that corrected scores should not be used to rank human text against model text.

(3) We clarified the reference-selection probes and added comparisons across a replacement reference pair, obtaining rank correlations of 0.88, 0.98, and 0.81 for surprisal, recoverability, and the combined score across the 11 methods. Retraining with the replacement pair retains approximately 70% of the combined-score improvement when evaluated under the original pair.


We deeply appreciate all the constructive suggestions provided. They have helped us strengthen the empirical analysis, clarify the intended scope and limitations of CD, and improve the presentation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.