- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the lack of a unified evaluation standard for assessing creativity in large language models (LLMs). It proposes conceptualizing creativity as a model’s ability to generate high-quality results within the long-tail regions of reference input-output distributions. Through a survey of 82 research papers and an analysis of 15 benchmarks and 26 open-source models, the authors examine the relationship between performance on long-tail tasks and human preferences for creativity. They introduce a metric, q@Tail(α), which evaluates a model’s creativity by measuring task quality on highly novel samples drawn from existing benchmarks. The core contribution of this work is the proposal of a unified evaluation perspective.
Critical strengths
-
The problem is significant, and the authors propose an insightful perspective. In the assessment of creativity, "novelty" and "quality" are often examined separately. This paper links the two, emphasizing a model's ability to maintain quality even when dealing with atypical inputs or outputs. This can help explain why different benchmarks may lead to conflicting conclusions.
-
The theoretical framework is robust. Starting from a reference distribution, the paper distinguishes between the "long tails" of inputs and outputs and derives a formal connection between novelty-weighted creativity and "tail-conditioned quality," ensuring that the core arguments are grounded in more than just intuition.
-
The experimental scope is extensive. The authors evaluate 15 benchmarks, 26 open-source models, and a variety of task domains. They further test the robustness of their primary findings through diverse n-gram configurations and supplementary analyses, rather than relying on a single benchmark. The metric q@Tail(α) leverages samples from existing benchmarks to determine whether a model performs well only on common patterns. Furthermore, the experiments reveal a clear trade-off between quality and novelty, demonstrating that novelty alone is an insufficient criterion for evaluating creativity.
Critical weaknesses
-
The scope of the core theory’s extrapolation is insufficiently justified. While the paper derives the importance of "long-tail quality" from the product of novelty and quality, this does not automatically establish that long-tail performance serves as a common, fundamental metric across diverse creative tasks. Specifically, in the analysis presented in Section 6, the Spearman correlation between "tailness" and human ranking alignment is only rho=0.26, with p=0.53. It is unclear whether this result reaches statistical significance? This lack of significance potentially weakens the empirical support for the claim that the "long tail" constitutes a unified evaluation objective.
-
There is a misalignment of evaluation objectives in the human preference validation. The paper uses creative writing rankings from LM Arena as a reference to analyze model ranking alignment on non-writing tasks, such as GPQA-Diamond. Since quality rankings for mathematical reasoning tasks represent a different evaluation objective than human preferences for creative writing, such alignment cannot be straightforwardly interpreted as a valid confirmation of "creativity" in domains like mathematics.
-
The choice of the tail proportion parameter α for the metric q@Tail(α) lacks a robust rationale. The metric relies on a manually selected α, yet varying this proportion could alter the relative performance rankings of models. Although the paper primarily employs this as a diagnostic tool, it fails to provide clear guidance on how to select an appropriate α or whether model rankings remain stable across different values of α. So comparisons across different papers or benchmarks may lack consistency. Furthermore, the methodological comparisons in Section 7 are based exclusively on 7–8B parameter open-source models. Conclusions from this have not been verified for larger or more-scale models. The limitations regarding model scale and the scope of methods examined cast doubt on the generalizability of these findings.
Questions for authors
Please see weakness.
Overall, I believe the concept and motivation behind this paper are sound. However, the experimental section could be more detailed and thorough to verify the reliability of the underlying theory and to further refine it. I look forward to the authors' response during the rebuttal phase and would be open to adjusting my score accordingly.
9 comments
Follow-up Response to Reviewer xrJ6: Our Finding holds when Scaling Model Size
Thank you for asking whether our findings extend to larger open-weight models. We conducted additional pilot evaluations with Qwen3-8B, -14B, and -32B.
- Scaling does not consistently make the tested test-time methods more effective. Vanilla q@Tail increased or remained unchanged with model size, but the methods did not consistently narrow their q@Tail gaps relative to vanilla across tasks and scales.
| Task | Model | Vanilla | Min-p | Temp. ×1.5 | BSM | Meta |
|---|---|---|---|---|---|---|
| Writing | 8B | 0.5805 | 0.6092 | 0.5864 | 0.3901 | 0.2429 |
| Writing | 14B | 0.6535 | 0.6288 | 0.6719 | 0.4426 | 0.1584 |
| Writing | 32B | 0.6848 | 0.6732 | 0.6879 | 0.5289 | 0.4782 |
| Math | 8B | 0.9167 | 0.8833 | 0.8667 | 0.8333 | 0.8500 |
| Math | 14B | 0.9167 | 0.9500 | 0.9333 | 0.9000 | 0.8667 |
| Math | 32B | 0.9500 | 0.9500 | 0.9500 | 0.9167 | 0.8500 |
| All entries report q@Tail. |
These results reinforce the limitation discussed in Section 7. Current creativity-promoting methods often achieve novelty gains alongside lower quality on uncommon inputs or outputs. The scaling results show no consistent evidence that larger models resolve the tail-quality limitation of these methods.
We are still looking forward to discussing this with you within this period!
I have carefully reviewed the authors' response and appreciate the detailed feedback and additional experiments provided. I noticed that the results for different values of are not presented in detail. The authors state that they "were broadly similar near ." Could the authors provide more granular results to facilitate an analysis of how the choice of influences the outcomes? Furthermore, the authors focus on the quality of outputs in the "long tail", why not directly evaluate the novelty of high-quality outputs? Novelty seems like a frequent metric for assessing creativity, and the two concepts appear to be somewhat dual to each other. How do the authors view the relationship between the "long tail" and novelty, and could they offer further insights into this connection? Finally, is there a way to aggregate a model's long-tail performance across various benchmarks, such as GPQA or the LM Arena, to derive a proxy for its overall level of creativity?
We thank all reviewers for their valuable feedback. We appreciate their recognition of the importance of the problem (xrJ6), the clear and insightful motivation (ukBK, c5gj), and the robust theoretical framework (xrJ6). We are also grateful for their positive assessments of the experiments’ soundness (ukBK) and breadth (c5gj, xrJ6), as well as the helpful discussion and clear organization (c5gj). We address their questions and concerns below.
Choosing the tail proportion and assessing ranking stability. (W3: alpha)
We chose α = 0.25 as a practical operating point that balances focusing on highly novel examples with retaining enough observations for reliable evaluation, rather than as a theoretically optimal cutoff.
Using q@Tail to diagnose creativity-promoting methods. (W3: scale)
We recognize that our main intervention experiments focus on 7-8B models due to computational constraints, and we limit our claims to this scope. The methods' performance at larger scales remains an open question, as the literature also has limited results on this. We are working on this to provide some pilot findings:
A shared framework for understanding creativity evaluation. (W1)
q@Tail measures model quality on a shared set of prompts selected for high input and aggregate output novelty. Our theory connects this diagnostic to novelty-weighted task quality through an exact decomposition into a weighted integral of quality across novelty thresholds. q@Tail provides a coarse-grained summary of this profile at a selected threshold, making performance in high-novelty regions directly inspectable. We evaluate its diagnostic value through agreement with the human-perceived creativity ranking. Shared novelty weights ensure that models are compared on the same prompts. The resulting score characterizes performance in a shared task region, rather than the novelty–quality product of each model’s individual responses. We will also clarify that output n-gram absence measures corpus-relative lexical novelty, rather than conditional novelty P(y|x).
We agree that our theoretical framework does not establish tail performance as a universal creativity metric. Our empirical findings provide initial evidence that tailness can be informative across different tasks, including creative writing, math reasoning, and question answering. Although we observe evidence of this relationship within benchmarks, the weak across-benchmark association leaves its generalization across task distributions unresolved. This motivates future work on a more fine-grained framework that explicitly accounts for differences in task distributions, enabling clearer tests of our theory across heterogeneous tasks. We will clarify this distinction and the scope of our claims.
Human-perceived creativity ranking as a cross-task reference. (W2)
We agree that the LM Arena Creative Writing ranking cannot directly validate creativity in domains such as scientific reasoning. Our goal is instead to examine whether tail performance across different task formats is associated with a common measure of human-perceived creativity.
This LM Arena category leaderboard includes varied tasks, such as humor, philosophical responses, stories, and poetry, making it a useful, though imperfect, reference. Our GPQA analysis examines whether performance on novel scientific question answering is associated with this same ranking, rather than claiming to measure scientific creativity directly. Such an association could be consistent with shared creativity-related capabilities, but it does not by itself establish a common or comprehensive measure of creativity.
We will clarify this distinction and the limitations in the manuscript.
Thank you
We thank the reviewer again for their feedback and willingness to engage in further discussion during this period.