- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces Constructive Divergence, which uses two frozen models' probabilities to score surprisal and recoverability via conditioned log-probability. First, when applying Constructive Divergence as an evaluator, CD is more stable and better aligned with human judges. Second, when applying Constructive Divergence as a reward in DPO and GRPO, they show improved creativity under LLM-as-a-judge and human evaluation. Detailed analysis suggests CD-based RL decreases the recombination ratio in ideas and improves the distance to existing papers.
Critical strengths
- The method constrains small designs, like paraphrasing, to ensure the reliability of the metric.
- The analysis is interesting.
- Human evaluation and various LLM judge evaluations are included.
Critical weaknesses
- As a metric (evaluator), the absolute scores are conditional on the choice of the two reference models, and only setting a fixed choice can ensure that it can be fairly reused in future work. I suggest the authors 1) discuss which is the optimal choice of reference models and why; 2) if presenting it as a model-agnostic evaluator (L086), you should demonstrate high agreement across reference model choices on scenarios other than your improving method's performance.
- Fundamentally, I am concerned that the design of this metric is, by nature, biased toward the two reference models' familiarity with potential outputs. However, reference models may not always reflect human familiarity or sense of novelty and may exhibit bias. For example, a default reference model is approaching more average human knowledge, which should be largely different from the PhD annotators' group's knowledge distribution. In that case, the metric has limitations in broader usage.
- The method doesn't seem to be computationally cheaper than LLM-as-a-judge, which makes me concerned about how much motivation other people will still have to use this to replace LLM-as-a-judge.
- The paper does not consider the validation of the research idea as part of the evaluation. A surprising and recoverable idea can be an invalid or even wrong idea.
- The math part in Section 2, from my perspective, doesn't bring in too much novel knowledge or information compared with the literature.
Questions for authors
- Why can log-likelihood reflect the concepts of surprisal and recoverability?
- How can other researchers directly use your metric (CD) out-of-the-box? And what information can they gain in addition to existing creativity metrics?
- Regarding weakness 2, people often assess creativity on a detailed downstream use case; for example, evaluating music idea targeting with top musicians as judges. In these cases, is it still true that we can use any general-purpose LLM as a reference? If not, how can we find a good reference model?
6 comments
2. The metric is biased toward the reference models' familiarity; which reference suits a specific domain? (Weakness 2, Question 3)
This is the same bottleneck as in point 1: the novelty term inherits what the reference model knows, so the bias is a property of the reference model, and it can be measured. On 60 arXiv abstracts written before the training cutoff of V and 60 written after it, with Base outputs generated for the same prompts (Appendix B.7):
| arXiv abstracts | Published | Base (same prompts) | d | Published higher on |
|---|---|---|---|---|
| Pre-cutoff (V may have seen them) | 2.30 | 2.45 | −0.21 | 20/60 prompts |
| Post-cutoff (V cannot have seen them) | 2.98 | 2.42 | +0.69 | 46/60 |
(Context-calibrated surprisal.) The effect of familiarity is therefore measurable, and we control for it by comparing scores only across outputs for the same prompt, where every output is subject to the same familiarity. The conclusions about the trained policies do not depend on one model's familiarity, since they are reproduced under the alternative reference models of point 1.
For a detailed downstream use case such as music, we would not use an arbitrary general-purpose LLM. The reference model must (i) model the domain's text well and (ii) pass the same probe on that domain: items that experts regard as new, and that postdate the model's cutoff, must be more surprising than earlier ones. This probe is how we selected V here, and it requires only dated items, not expert labels. Where no model passes, we would not recommend the score for that domain.
3. The method is not computationally cheaper than LLM-as-a-judge (Weakness 3)
We measured the cost and converted it at public list prices (Appendix A.11):
| Evaluator | Cost per abstract |
|---|---|
| Constructive Divergence, K = 8 (10.5–13.3 GPU-seconds on an A100) | $0.009–0.012 |
| Constructive Divergence, K = 2 (5.6–6.6 GPU-seconds) | $0.005–0.006 |
| One pointwise judge call (Gemini 3.6 Flash / GPT-5.2 / GPT-5.6 Sol) | / $0.0014 / $0.0040 / $0.0057 |
| Pairwise ranking of 8 candidates in both orders (GPT-5.2, 56 calls) | at least $0.028 |
GPU time is priced at $3.19 per GPU-hour, the on-demand list price of an A100 80GB at Lambda. Judge calls are priced at each provider's list API price, with token counts measured on 105 held-out abstracts (about 500 input and 180–260 output tokens per call, reasoning tokens included).
A single judge call is cheaper than our score. The score is nevertheless the cheaper way to obtain a reliable reward, and its cost can be reduced further:
- A single judge call is not a usable reward. The judges reverse their verdict on 30.0% of pairs when the presentation order is swapped (Table 1), so a judge-based reward needs both orders, and ranking n candidates by comparison grows quadratically. Our score is two to three times cheaper than that protocol and grows linearly.
- Reducing K lowers the cost further. K is the number of continuations that V samples for the context baseline, and it is the main cost of the score. We re-scored the 650 held-out Base outputs with K = 2 on the same hardware: surprisal scoring took 901 s instead of 2,630 s (34%), and the scores agree with the K = 8 scores at r = 0.99 (means 2.46 vs. 2.47). In an offline recomputation on 1,950 held-out outputs, even K = 1 correlates at r = 0.996 with K = 8, and the effect sizes of both trained policies are unchanged at K = 2. At K = 2 the score costs about the same as one GPT-5.6 Sol call and about one fifth of the pairwise protocol.
- The score is deterministic. The same output under the same configuration always receives the same score: there is no presentation order, no sampling of a verdict, and no dependence on which other outputs it is compared with. Each output is therefore scored exactly once, stored scores remain valid when a new method is added, and anyone with the open model weights can reproduce them exactly. This matters most when the score is used as a training objective. Training queries the reward for every candidate (8,000 candidates in our DPO run alone), and the reward must stay the same from the first step to the last. Our score is computed locally from frozen weights, so it is unaffected by network latency, rate limits, price changes, or a provider updating or retiring a model, any of which can slow, interrupt, or silently alter a judge-based reward in the middle of training.
The revision reports these figures in the main text, so that the statement that judges "can be costly" refers to the measured cost of an order-balanced protocol rather than to a single call.
4. A surprising and recoverable idea can still be invalid (Weakness 4)
We agree that the CD dose not cover execution-level validness, which requires complex implementation thus beyond an ideation-stage score. Instead, we test the case that an idea that keeps the given keywords and scores high only because it is unusual, with no sound scientific basis. We ran a blind audit in which three LLM judges (Gemini 3.6 Flash, GPT-5.2, GPT-5.6 Sol) rated 650 abstracts per method one at a time on an absolute 1–10 rubric (Table 17, Appendix B.6). Differences from Base, range over the three judges:
| Method | Novelty | Feasibility | Soundness | Incoherent claims |
|---|---|---|---|---|
| DPO_CD | +0.06 to +0.13 | −0.03 to +0.08 (n.s.) | −0.01 to −0.09 (n.s., n.s., p = 0.02) | 0.6–2.1% (Base 1.1–2.6%) |
| GRPO_CD | −0.05 to −0.06 (n.s.) | +0.01 to +0.02 (n.s.) | −0.07 to −0.13 (p < 0.01) | 0.1–1.2% |
| DPO_LLM | +0.6 to +0.8 | −0.17 to −0.25 | 0.00 to +0.10 | up to 5.1% |
| Creativity Instruction | +1.1 to +1.7 | −1.1 to −1.5 | −0.6 to −1.3 | 5.5–16.4% |
| Base, t = 1.8 | −0.3 to −0.8 | −0.9 to −1.7 | −1.5 to −2.4 | 29–37% |
The failure the reviewer describes is exactly what the prompting, temperature, and judge-reward alternatives show: they look more novel to the judges and lose feasibility or soundness. The policies trained with our reward do not: both stay within 0.13 points of Base in feasibility and soundness, and GRPO_CD produces fewer incoherent claims than Base. This is the role of the recoverability term: without it, recoverability stays at the Base level instead of rising (Table 2, w/o Recoverability).
In short, the audit shows that training on the score raises novelty-related signals without the loss of soundness that the alternatives incur.
5. The math does not bring much new knowledge (Weakness 5)
In fact, the entropy–KL decomposition and the mutual-information lower bound explain what the two terms respond to, thus they are presented as motivation rather than as a theoretical contribution. In the main text we introduced the decomposition as "distributional motivation" (Section 3.2), the derivations are in Appendix A.1. The effectiveness of our objective is demonstrated in the subsequent experimental sections.
First of all, thank you for the comprehensive and thoughtful feedback on our work. We sincerely appreciate the reviewer's constructive comments and questions, which provided valuable insights. We address all concerns below.
1. Which reference models are the optimal choice, and is the evaluator really "model-agnostic"? (Weakness 1)
Yes, we select one configuration in the paper (V = gpt-oss-120B, D = Llama-3.1-70B-Instruct, K = 8, σ_S = 0.37, σ_R = 0.90, with the instructions and thresholds; Appendix A.11) and scores are compared under this fixed configuration and for the same prompt.
The configuration selection procedure was only in the appendix and is now summarized in the main text. Before any training, we compared four candidate models (gpt-oss-120B, Llama-3.1-70B, Qwen3.5-122B, Mistral-Large) on four probes that a reference model for novelty should pass (Table 10, Appendix A.7):
gpt-oss-120B is thus the only candidate that passes the two probes on which the candidates differ, and both test whether the model responds to new content.
Also, we define the term "model-agnostic" in the revision paper: the score is computed from token probabilities alone and is not tied to a particular model, so any model that meets the selection criteria above can serve as a reference. The term "model-agnostic" does not mean that absolute scores coincide across any model. Two results support this reading. First, we re-scored the same held-out outputs after swapping one reference model for another: gpt-oss-120B (V) was replaced in turn by each of seven other models with everything else fixed, and Llama-3.1-70B (D) likewise. The replacements range from 9B to 123B parameters across four families (Qwen, GLM, Mistral, Llama). The surprisal gain of the trained policies over Base reappears under every replacement V (d = 0.58–1.36), and the recoverability gain under every replacement D. Second, we ran the test the reviewer suggests: every method of Table 2 plus four controls, re-scored under a replacement pair (Qwen3.5-9B, Llama-3.1-8B-Instruct; new Table 13, Appendix B.2).
The set of methods that raise both terms over Base is identical under both pairs. Where the pairs disagree, the cause is the capability of the reference model, not the definition of the score. The 9B model separates new content from noisy wording less well (Base at temperature = 1.8 gains +0.39 in S against +0.19). And when we retrain the policy with the replacement pair as its reward and score it under the original pair, the recoverability gain transfers fully (+0.39, d = 1.19) while the surprisal gain transfers at about 60% (+0.28, d = 1.22): the loss is concentrated in the term computed by the weaker model (new Table 12). The bottleneck is therefore how well the reference model knows the domain, which is what the probes measure, and a stronger open model can replace V without any change to the method.