acceptodds
Under review as a conference paper at ICLR 2027

Evaluating LLM Creativity through Long-Tail Performance

Abstract

Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text. We ground this proposition in the novelty-quality product formulation and evaluate it empirically across 15 benchmarks and 26 open-weight models, using n-gram absence from the pretraining data as a novelty proxy. Our results show that creativity benchmarks lie further in the tail regions than standard benchmarks. Moreover, on both creativity and standard benchmarks, model rankings from higher-tail subsets of each benchmark align more closely with human creativity judgments in LM-Arena. Building on these findings, we introduce q@Tail(), a lower-cost creativity diagnostic that measures mean task quality on the tail fraction of an existing benchmark. Applying it to creativity-promoting methods reveals substantial room to improve LLM creativity through stronger tail performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Creativity, Natural Language Generation, Analysis, Evaluation, Long-Tail Distribution

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xrJ6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the lack of a unified evaluation standard for assessing creativity in large language models (LLMs). It proposes conceptualizing creativity as a model’s ability to generate high-quality results within the long-tail regions of reference input-output distributions. Through a survey of 82 research papers and an analysis of 15 benchmarks and 26 open-source models, the authors examine the relationship between performance on long-tail tasks and human preferences for creativity. They introduce a metric, q@Tail(α), which evaluates a model’s creativity by measuring task quality on highly novel samples drawn from existing benchmarks. The core contribution of this work is the proposal of a unified evaluation perspective.

Critical strengths
  1. The problem is significant, and the authors propose an insightful perspective. In the assessment of creativity, "novelty" and "quality" are often examined separately. This paper links the two, emphasizing a model's ability to maintain quality even when dealing with atypical inputs or outputs. This can help explain why different benchmarks may lead to conflicting conclusions.

  2. The theoretical framework is robust. Starting from a reference distribution, the paper distinguishes between the "long tails" of inputs and outputs and derives a formal connection between novelty-weighted creativity and "tail-conditioned quality," ensuring that the core arguments are grounded in more than just intuition.

  3. The experimental scope is extensive. The authors evaluate 15 benchmarks, 26 open-source models, and a variety of task domains. They further test the robustness of their primary findings through diverse n-gram configurations and supplementary analyses, rather than relying on a single benchmark. The metric q@Tail(α) leverages samples from existing benchmarks to determine whether a model performs well only on common patterns. Furthermore, the experiments reveal a clear trade-off between quality and novelty, demonstrating that novelty alone is an insufficient criterion for evaluating creativity.

Critical weaknesses
  1. The scope of the core theory’s extrapolation is insufficiently justified. While the paper derives the importance of "long-tail quality" from the product of novelty and quality, this does not automatically establish that long-tail performance serves as a common, fundamental metric across diverse creative tasks. Specifically, in the analysis presented in Section 6, the Spearman correlation between "tailness" and human ranking alignment is only rho=0.26, with p=0.53. It is unclear whether this result reaches statistical significance? This lack of significance potentially weakens the empirical support for the claim that the "long tail" constitutes a unified evaluation objective.

  2. There is a misalignment of evaluation objectives in the human preference validation. The paper uses creative writing rankings from LM Arena as a reference to analyze model ranking alignment on non-writing tasks, such as GPQA-Diamond. Since quality rankings for mathematical reasoning tasks represent a different evaluation objective than human preferences for creative writing, such alignment cannot be straightforwardly interpreted as a valid confirmation of "creativity" in domains like mathematics.

  3. The choice of the tail proportion parameter α for the metric q@Tail(α) lacks a robust rationale. The metric relies on a manually selected α, yet varying this proportion could alter the relative performance rankings of models. Although the paper primarily employs this as a diagnostic tool, it fails to provide clear guidance on how to select an appropriate α or whether model rankings remain stable across different values of α. So comparisons across different papers or benchmarks may lack consistency. Furthermore, the methodological comparisons in Section 7 are based exclusively on 7–8B parameter open-source models. Conclusions from this have not been verified for larger or more-scale models. The limitations regarding model scale and the scope of methods examined cast doubt on the generalizability of these findings.

Questions for authors

Please see weakness.

Overall, I believe the concept and motivation behind this paper are sound. However, the experimental section could be more detailed and thorough to verify the reliability of the underlying theory and to further refine it. I look forward to the authors' response during the rebuttal phase and would be open to adjusting my score accordingly.

9 comments

We thank all reviewers for their valuable feedback. We appreciate their recognition of the importance of the problem (xrJ6), the clear and insightful motivation (ukBK, c5gj), and the robust theoretical framework (xrJ6). We are also grateful for their positive assessments of the experiments’ soundness (ukBK) and breadth (c5gj, xrJ6), as well as the helpful discussion and clear organization (c5gj). We address their questions and concerns below.

Choosing the tail proportion and assessing ranking stability. (W3: alpha)

We chose α\alpha = 0.25 as a practical operating point that balances focusing on highly novel examples with retaining enough observations for reliable evaluation, rather than as a theoretically optimal cutoff.

  • α\alpha choice shows low sensitivity. We evaluated α∈{0.10,0.20,0.25,0.30,0.40,0.50}\alpha \in \{0.10, 0.20, 0.25, 0.30, 0.40, 0.50\} across the datasets. Changing \alpha does not significantly alter model rankings in terms of their agreement with human judgments. Rankings of other α\alpha values were broadly similar near α\alpha = 0.25. For example, Kendall \tau-b with the α\alpha = 0.25 ranking ranged from 0.821–0.979, and at \alpha = 0.20 and 0.856–0.994 at \alpha = 0.30.

Using q@Tail to diagnose creativity-promoting methods. (W3: scale)

We recognize that our main intervention experiments focus on 7-8B models due to computational constraints, and we limit our claims to this scope. The methods' performance at larger scales remains an open question, as the literature also has limited results on this. We are working on this to provide some pilot findings:

  • For frontier closed-source models, test-time intervention also does not show improvements on q@Tail. In the sanity check of the Section 7 experiments (Appendix E), we evaluate Claude Opus 4.6 High and GPT-5.2 Pro on a smaller data scale. Likewise, it showed no consistent improvement from the test-time intervention methods, with relative changes in q@Tail ranging from −7.49% to +0.28%.
  • Open-weight model scaling. We agree that scaling is an interesting direction and will conduct a small pilot across sizes within an open-weight model family to assess whether the observation persists. We will update the results shortly.

A shared framework for understanding creativity evaluation. (W1)

q@Tail measures model quality on a shared set of prompts selected for high input and aggregate output novelty. Our theory connects this diagnostic to novelty-weighted task quality through an exact decomposition into a weighted integral of quality across novelty thresholds. q@Tail provides a coarse-grained summary of this profile at a selected threshold, making performance in high-novelty regions directly inspectable. We evaluate its diagnostic value through agreement with the human-perceived creativity ranking. Shared novelty weights ensure that models are compared on the same prompts. The resulting score characterizes performance in a shared task region, rather than the novelty–quality product of each model’s individual responses. We will also clarify that output n-gram absence measures corpus-relative lexical novelty, rather than conditional novelty P(y|x).

We agree that our theoretical framework does not establish tail performance as a universal creativity metric. Our empirical findings provide initial evidence that tailness can be informative across different tasks, including creative writing, math reasoning, and question answering. Although we observe evidence of this relationship within benchmarks, the weak across-benchmark association leaves its generalization across task distributions unresolved. This motivates future work on a more fine-grained framework that explicitly accounts for differences in task distributions, enabling clearer tests of our theory across heterogeneous tasks. We will clarify this distinction and the scope of our claims.

Human-perceived creativity ranking as a cross-task reference. (W2)

We agree that the LM Arena Creative Writing ranking cannot directly validate creativity in domains such as scientific reasoning. Our goal is instead to examine whether tail performance across different task formats is associated with a common measure of human-perceived creativity.

This LM Arena category leaderboard includes varied tasks, such as humor, philosophical responses, stories, and poetry, making it a useful, though imperfect, reference. Our GPQA analysis examines whether performance on novel scientific question answering is associated with this same ranking, rather than claiming to measure scientific creativity directly. Such an association could be consistent with shared creativity-related capabilities, but it does not by itself establish a common or comprehensive measure of creativity.

We will clarify this distinction and the limitations in the manuscript.

Thank you

We thank the reviewer again for their feedback and willingness to engage in further discussion during this period.

AuthorsOpenReview ↗

Follow-up Response to Reviewer xrJ6: Our Finding holds when Scaling Model Size

Thank you for asking whether our findings extend to larger open-weight models. We conducted additional pilot evaluations with Qwen3-8B, -14B, and -32B.

  • Scaling does not consistently make the tested test-time methods more effective. Vanilla q@Tail increased or remained unchanged with model size, but the methods did not consistently narrow their q@Tail gaps relative to vanilla across tasks and scales.
TaskModelVanillaMin-pTemp. ×1.5BSMMeta
Writing8B0.58050.60920.58640.39010.2429
Writing14B0.65350.62880.67190.44260.1584
Writing32B0.68480.67320.68790.52890.4782
Math8B0.91670.88330.86670.83330.8500
Math14B0.91670.95000.93330.90000.8667
Math32B0.95000.95000.95000.91670.8500
All entries report q@Tail.

These results reinforce the limitation discussed in Section 7. Current creativity-promoting methods often achieve novelty gains alongside lower quality on uncommon inputs or outputs. The scaling results show no consistent evidence that larger models resolve the tail-quality limitation of these methods.

We are still looking forward to discussing this with you within this period!

I have carefully reviewed the authors' response and appreciate the detailed feedback and additional experiments provided. I noticed that the results for different values ​​of α\alpha are not presented in detail. The authors state that they "were broadly similar near α=0.25\alpha = 0.25." Could the authors provide more granular results to facilitate an analysis of how the choice of α\alpha influences the outcomes? Furthermore, the authors focus on the quality of outputs in the "long tail", why not directly evaluate the novelty of high-quality outputs? Novelty seems like a frequent metric for assessing creativity, and the two concepts appear to be somewhat dual to each other. How do the authors view the relationship between the "long tail" and novelty, and could they offer further insights into this connection? Finally, is there a way to aggregate a model's long-tail performance across various benchmarks, such as GPQA or the LM Arena, to derive a proxy for its overall level of creativity?

Reviewer_mXJKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a formal definition of creativity as the product of the expected novelty and expected quality of the outputs of a model under some distribution of inputs and/or outputs. It uses this definition to make sense of existing empirical findings, with the biggest output being a taxonomy of creativity evals.

Critical strengths

The paper provides a formal definition.

Critical weaknesses

The paper is clear, but otherwise just formally defines existing empirical results without much else. I am a fan of formalizations, but only if those formalizations teach us something new.

Questions for authors

What exactly can the community learn from this formalization you propose? To me, it seems difficult to answer this.

1 comment

We thank all reviewers for their valuable feedback. We appreciate their recognition of the importance of the problem (xrJ6), the clear and insightful motivation (ukBK, c5gj), and the robust theoretical framework (xrJ6). We are also grateful for their positive assessments of the experiments’ soundness (ukBK) and breadth (c5gj, xrJ6), as well as the helpful discussion and clear organization (c5gj). We address their questions and concerns below.

Our contribution to the community

Beyond the literature review and taxonomy, our framework provides three concrete insights:

  • Tailness reveals a shared pattern across creativity benchmarks. Across writing, mathematics, and QA, creativity benchmarks contain more novel inputs and/or outputs than standard benchmarks (§5).
  • Tailness helps identify where evaluation is most informative. Performance on higher-novelty examples often aligns more closely with human-perceived creativity rankings (§6), motivating q@Tail as a practical diagnostic for existing benchmarks.
  • Tail-based evaluation reveals limitations of creativity-promoting methods. Most methods we tested increase novelty but reduce quality on novel examples (§7), highlighting why increasing novelty alone is insufficient.

We will revise the manuscript to better articulate these insights and clarify what our framework contributes beyond the literature review and taxonomy.

Reviewer_c5gjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether different LLM creativity benchmarks share a common evaluation target. The authors propose viewing creativity evaluation through model performance in the long tail of a reference input-output distribution. Based on a novelty-quality formulation, they use n-gram absence from the Pile as a proxy for novelty and examine 15 benchmarks across 26 open-weight models. The main finding is that creativity benchmarks tend to contain more long-tail examples, and model rankings on these examples often agree better with human creative-writing preferences. The paper further introduces q@Tail as a diagnostic that evaluates task quality on high-novelty subsets of existing benchmarks, and uses it to analyze several creativity-promoting methods.

Critical strengths
  1. The motivation of this work is clear. Creativity evaluation is quite fragmented, and trying to find a common perspective across different benchmarks is a meaningful problem. The taxonomy covering 82 papers also provides a useful overview of existing evaluation practices.

  2. The empirical study is fairly extensive. Evaluating 26 models across 15 benchmarks from different task domains gives a reasonably broad picture. I also find the within-benchmark analysis in Section 6.1 interesting, especially the observation that higher-tail subsets can show stronger agreement with human preferences.

  3. The paper is generally well organized, and I appreciate that the authors report results that do not fully support their hypothesis. The additional analyses on difficulty and the limitations discussed in the appendix are helpful for understanding the scope of the proposed metr

Critical weaknesses
  1. I am not fully convinced that the proposed metric captures creativity rather than general model capability. This is my main concern. The key validation in Section 6 relies on agreement with the LM Arena creative-writing ranking. However, Table A2 shows that this ranking is already highly correlated with the overall LM Arena ranking (Spearman ρ = 0.944). This makes it difficult to tell whether better agreement on tail subsets actually reflects creativity, or simply better discrimination between generally stronger and weaker models. I understand the authors acknowledge this issue in Appendix B.2 and do not claim to isolate creativity completely. Still, I think this matters quite a lot for a paper about evaluating creativity. Some comparison against overall capability rankings, or an analysis among models with similar general capabilities, would help clarify what additional signal q@Tail provides.

  2. There seems to be a gap between the theoretical formulation and the actual novelty measurement. Section 3 motivates the framework using a reference joint distribution and the novelty-quality product. However, the experiments in Section 4 approximate novelty through 4-gram absence from the Pile. I am not sure this is sufficient for capturing the semantic or conceptual novelty that creativity often requires. Unusual wording does not necessarily imply a creative idea. There is also a distinction between the theoretical creativity score and q@Tail. The latter measures average quality on a selected subset, where the tail selection depends partly on output novelty averaged across evaluated models. A model can therefore receive a high q@Tail score even when its own responses are fairly conventional. I do not see a mathematical problem with the derivation in Section 3.2, but the connection to creativity measurement seems more limited than the overall framing suggests. The authors acknowledge some of these limitations, though I would like to see a clearer justification of why this particular approximation is informative beyond lexical rarity.

  3. The empirical relationship is less consistent when looking beyond the main examples. Figures 4 and 5 provide interesting observations, but the broader results are not equally convincing. In Section 6.2 and Table A7, the correlation between benchmark-level tailness and agreement with human rankings is only ρ = 0.26 (p = 0.53). This is quite weak and not statistically significant. Table A6 also shows that the most-tail subsets do not always give better ranking agreement. For example, StoryGen and TinyStories show negative endpoint differences, while Alpaca shows essentially no difference. I think the within-benchmark findings are worth reporting, but the evidence is not yet strong enough to establish tail performance as a broadly reliable creativity diagnostic. It would help to make clearer under which benchmark conditions the proposed metric is expected to work well. Reporting uncertainty around the bin-level ranking correlations would also make the results easier to interpret.

Questions for authors
  1. Given the strong correlation between LM Arena creative-writing and overall rankings (Table A2), have the authors examined whether q@Tail provides additional predictive value after accounting for general model capability?

  2. The practical novelty score is based on n-gram absence and output novelty averaged across evaluated models. How sensitive is tail selection to the chosen model set? Also, how should this be interpreted relative to the conditional novelty in the theoretical formulation?

  3. How do the authors reconcile the relatively weak across-benchmark correlation in Table A7 with the stronger within-benchmark results? Are there specific properties of a benchmark that determine when q@Tail is likely to be informative?

  4. Are uncertainty estimates available for the per-bin Kendall correlations in Section 6.1? In particular, I would like to understand how stable the reported ranking improvements are when the benchmark is divided into relatively small subsets.

3 comments
AuthorsOpenReview ↗

Response to Reviewer c5gj [1/2]

We thank all reviewers for their valuable feedback. We appreciate their recognition of the importance of the problem (xrJ6), the clear and insightful motivation (ukBK, c5gj), and the robust theoretical framework (xrJ6). We are also grateful for their positive assessments of the experiments’ soundness (ukBK) and breadth (c5gj, xrJ6), as well as the helpful discussion and clear organization (c5gj). We address their questions and concerns below.

The contribution of tail performance to creativity evaluation beyond general capability. (W1, Q1)

We agree that the human creativity ranking in LM Arena and overall model rankings for the models evaluated in our study are strongly correlated. However, our results suggest that evaluating model performance on novel, long-tail examples reveals creativity-relevant differences that overall capability ratings alone do not capture.

  • Tail performance provides additional information beyond general capability. While overall model ratings strongly predict human-perceived creativity, including tail performance improves these predictions. We conducted an exploratory leave-one-family-out analysis, excluding the held-out family from both regression fitting and novelty calibration. Adding tail performance to overall ratings reduced prediction RMSE for models’ percentile positions in the human-perceived creativity ranking from 11.96 to 9.32, a 22.1% reduction, which is statistically significant. This suggests that tail performance captures creativity-relevant differences that overall ratings alone do not fully explain.
  • Tail performance helps distinguish models with similar general capabilities. Models with similar overall ratings can differ in human-perceived creativity, and these differences become more apparent when we focus on novel examples. Among model pairs with similar overall ratings but opposing overall and creativity rankings, the human-perceived higher-creativity model’s mean per-prompt quality-score win rate increased from 71.4% on the full benchmark to 79.0% on the tail subset. For example, Gemma-3-4B and OLMo-3-32B-Think differ by only 2.48 overall-rating points, yet the human-perceived creativity ranking favors Gemma. Gemma’s per-prompt quality-score win rate against OLMo increases from 66% on the full benchmark to 90% on the tail subset.

We will revise the manuscript to clarify how tail performance provides additional information beyond general capability ratings and helps distinguish models with similar overall performance.

AuthorsOpenReview ↗

Response to Reviewer c5gj [2/2]

A shared calibration panel enables comparable and robust tail evaluation. (Model-set part of Q2)

Our results suggest that tail selection is robust to changes in the calibration model set. We also propose using a fixed calibration panel to ensure consistent comparisons as new models are evaluated.

  • Robustness to calibration-panel changes. We tested how removing individual models or entire model families affects tail selection and model rankings. Across 448 tests, at least 83.3% of tail examples were retained, and model rankings remained highly consistent (Kendall \tau-b ≥ 0.838). In 274 tests, tail membership was completely unchanged. For example, on Arena-Hard, the improvement from tail evaluation remained positive under every removal (+0.094 to +0.129, compared with +0.105 using the full panel).
  • Ensuring consistent comparisons in future evaluations. We have published a fixed calibration panel and its selected tail subset as a versioned benchmark. Researchers can evaluate new models on the same prompts without changing tail selection. As new models are added to the calibration panel, we will release updated benchmark versions and document any changes in tail selection and model rankings.

Theoretical interpretation and the novelty proxy. (W2; conditional-novelty part of Q2)

q@Tail measures model quality on a shared set of prompts selected for high input and aggregate output novelty. Our theory connects this diagnostic to novelty-weighted task quality through an exact decomposition into a weighted integral of quality across novelty thresholds. q@Tail provides a coarse-grained summary of this profile at a selected threshold, making performance in high-novelty regions directly inspectable. We evaluate its diagnostic value through agreement with the human-perceived creativity ranking. Shared novelty weights ensure that models are compared on the same prompts. The resulting score characterizes performance in a shared task region, rather than the novelty-quality product of each model’s individual responses. We will also clarify that output n-gram absence measures corpus-relative lexical novelty, rather than conditional novelty P(y|x).

Scope and Robustness of Tail Evaluation. (W3, Q3, Q4)

We agree that the benefits of tail evaluation vary across benchmarks, and we will clarify when our approach is informative and how robust the findings are. Our strongest evidence comes from comparisons within individual benchmarks, while the across-benchmark results are less conclusive.

  • Understanding variation across benchmarks. The across-benchmark correlation is positive but weak and not statistically significant (\rho = 0.26, p = 0.53), so we cannot claim a reliable relationship. Benchmarks differ in prompt source, length, format, scoring, sample size, and complexity, which may help explain the variation. More specifically, benchmark format may shift the baseline distribution and interpretation of lexical novelty: uncommon n-grams in language-heavy tasks can reflect unusual phrasing or concepts, whereas those in symbol- or number-heavy tasks may instead reflect variations in numerical values or notation, making average novelty less directly comparable across benchmarks. Differences in prompt and response length may further affect both measured novelty and task demands, potentially obscuring relationships that are clearer within individual benchmarks. We will expand the paper’s discussion of these possible sources of heterogeneity.
  • Quantifying statistical uncertainty. Our appendix shows that the positive trends in the two main examples persist when using ten rather than five bins. Moreover, our new paired prompt-bootstrap analysis shows higher point-estimate ranking stability for Tail than for equally sized random subsets in five of six creative-writing tasks under fixed membership, and pointwise 95% confidence intervals consistently show positivity.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new analysis showing that tail performance provides additional predictive value beyond general model capability. The calibration robustness results and clarification of the theoretical scope are also helpful.

Overall, I find the rebuttal convincing enough to address my main concerns, although some limitations remain. I am therefore willing to raise my score from 2 to 3 (Weak Accept).

That said, creativity evaluation is not my primary area of expertise, so I would also encourage the Area Chair to carefully consider the assessments of other reviewers, particularly those with more specialized expertise, when making the final decision.

Reviewer_ukBKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes q@Tail, an evaluation metric for how well language models perform on unusual inputs. It ranks prompts using the rarity of four-word sequences in both prompts and model responses, then measures task quality on a shared subset of the rarest prompts. These subsets often produce model rankings closer to human creative-writing preferences, although the relationship is inconsistent across benchmarks. Decoding experiments also show that making outputs more unusual can reduce their quality. The paper provides a diagnostic of performance in rare regions, q@Tail alone does not measure whether an individual answer is creative.

Critical strengths

The proposed metric makes intuitive sense.

The experiments and evaluation setup is sound.

The paper reports where the proposed metric fails. The appendix includes benchmarks where tail selection does not improve agreement with human rankings, and a weak overall association across eight writing variants.

Critical weaknesses

Rare wording might not be a proper measure of originality. The proxy counts four-word sequences absent from the Pile but it does not directly measure where there's an actual new idea or how surprising an answer is given its prompt.

The signal may largely reflect general model ability. The creative-writing and overall Arena rankings is high, better tail performance could associate with stronger models without identifying a distinct creative capability.

Questions for authors

Would the result still hold if we paraphrase the same ideas? The score uses four-word sequences, so different wording can change measured novelty.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ukBK [1/2]

We thank all reviewers for their valuable feedback. We appreciate their recognition of the importance of the problem (xrJ6), the clear and insightful motivation (ukBK, c5gj), and the robust theoretical framework (xrJ6). We are also grateful for their positive assessments of the experiments’ soundness (ukBK) and breadth (c5gj, xrJ6), as well as the helpful discussion and clear organization (c5gj). We address their questions and concerns below.

Using n-gram absence as the measure of novelty. (W1)

We measure novelty using n-gram absence, the fraction of a text’s n-grams that do not occur in a reference corpus. Below, we discuss two alternatives we considered, their strengths and limitations, and our rationales for choosing n-gram absence.

  • Semantic matching. Semantic retrieval can identify similar content despite different wording, potentially capturing conceptual similarities that n-gram matching misses. However, indexing a Pile-scale corpus is computationally expensive, and the results depend on the choice of embedding model. Illustratively, indexing the same Pile text with 1 KiB passages and 1,024-dimensional float32 embeddings requires 5.23 TB for vectors alone, versus the reported 588 GB lexical index. Each semantic query also requires neural encoding and, with exhaustive search, 1.28 billion vector comparisons, where we might need severe compression and approximate search to reduce these costs.
  • Base-model perplexity. Perplexity measures how unexpected a text is under a reference model, reflecting both the model’s training data and how well it has learned from those data. High perplexity may reflect either unfamiliar content or poor fluency, making it difficult to interpret as a measure of novelty. Furthermore, our Appendix F.3 analysis found that perplexity did not consistently align with human judgments of creativity.

We chose n-gram absence for three reasons: it directly measures lexical novelty relative to a corpus, is reproducible without relying on a particular model, and is straightforward to interpret. At the same time, we recognize that lexical novelty does not necessarily imply conceptual originality. To examine this distinction, we will assess how n-gram absence relates to conceptual originality through the semantic matching and paraphrase checks described below.

  • N-gram measure in comparison to semantic matching. To explore this alternative, we are conducting a smaller-scale experiment using semantic matching to measure novelty. We will share the detailed results shortly.
  • N-gram measure’s sensitivity to paraphrasing. (Q1) We conducted a paraphrase-sensitivity check on a subset of data (n=200). Lexical novelty scores changed slightly, but the selected top-quarter subsets were largely unchanged, and the resulting model rankings showed high point-estimate agreement (Kendall \tau = 0.965).

We will incorporate the results into the revised paper to provide stronger empirical justification for our choice of n-gram absence as the primary novelty measure.

AuthorsOpenReview ↗

Response to Reviewer ukBK (continue) [2/2]

The contribution of tail performance to creativity evaluation beyond general capability. (W2)

We agree that the human creativity ranking in LM Arena and overall model rankings for the models evaluated in our study are strongly correlated. However, our results suggest that evaluating model performance on novel, long-tail examples reveals creativity-relevant differences that overall capability ratings alone do not capture.

  • Tail performance provides additional information beyond general capability. While overall model ratings strongly predict human-perceived creativity, including tail performance improves these predictions. We conducted an exploratory leave-one-family-out analysis, excluding the held-out family from both regression fitting and novelty calibration. Adding tail performance to overall ratings reduced prediction RMSE for models’ percentile positions in the human-perceived creativity ranking from 11.96 to 9.32, a 22.1% reduction, which is statistically significant. This suggests that tail performance captures creativity-relevant differences that overall ratings alone do not fully explain.
  • Tail performance helps distinguish models with similar general capabilities. Models with similar overall ratings can differ in human-perceived creativity, and these differences become more apparent when we focus on novel examples. Among model pairs with similar overall ratings but opposing overall and creativity rankings, the human-perceived higher-creativity model’s mean per-prompt quality-score win rate increased from 71.4% on the full benchmark to 79.0% on the tail subset. For example, Gemma-3-4B and OLMo-3-32B-Think differ by only 2.48 overall-rating points, yet the human-perceived creativity ranking favors Gemma. Gemma’s per-prompt quality-score win rate against OLMo increases from 66% on the full benchmark to 90% on the tail subset.

We will revise the manuscript to clarify how tail performance provides additional information beyond general capability ratings and helps distinguish models with similar overall performance.

AuthorsOpenReview ↗

Follow-up Response to Reviewer ukBK: Semantic Matching

Following up on our earlier response, we have completed a small-scale semantic-matching experiment.

  • Semantic matching. We evaluated held-out English prompts and corresponding model responses. Both used 10,000 Pile-validation reference documents to keep the comparison computationally feasible. The semantic variant retained exact matches and added embedding retrieval with bidirectional NLI.

  • Substantial ranking agreement between lexical and semantic matching. Novelty rankings achieved Spearman ρ = 0.823 (95% CI [0.554, 0.949], from 5,000 paired prompt-bootstrap draws) and Kendall τ = 0.668. The top-quarter tail selections overlapped by 67%.

These results strengthen the empirical justification for n-gram absence as a practical novelty proxy. Its strong agreement with this semantic-matching implementation shows that novelty rankings and tail selections remain broadly consistent when matching extends to the semantic level.

We are still looking forward to discussing this with you within this period!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Thank You and a Warm Invitation to Read Our Responses

Hi Reviewers,

First of all, we sincerely appreciate the effort you have devoted to reviewing our work.

As a gentle reminder, the ICLR discussion period closes in approximately 53 hours. We would be grateful if you could review our responses and share any further thoughts. We would be more than happy to discuss. We've also updated the manuscript accordingly, with revisions marked in blue.

Thank you again.

Respectfully,

The Authors

AuthorsOpenReview ↗

Thank You and a Warm Invitation to Read Our Responses

Hi Reviewers,

First of all, we sincerely appreciate the effort you have devoted to reviewing our work.

As a gentle reminder, the ICLR discussion period closes in approximately 30 hours. We would be grateful if you could review our responses and share any further thoughts. We would be more than happy to discuss. We've also updated the manuscript accordingly, with revisions marked in blue.

Thank you again.

Respectfully,

The Authors

AuthorsOpenReview ↗

Thank You All and a Final Invitation to Discuss Our Updates

Hi SAC, AC, and all reviewers,

With approximately 4 hours remaining, we have posted another round of responses sharing additional evidence gathered over the past few days.

We had hoped for a more interactive discussion and regret that limited follow-up has left little opportunity to exchange views. We hope these updates will inform your final assessments and would be glad to discuss any remaining concerns.

Thank you again for your time and consideration.

Respectfully,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.