- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies whether LLMs prefer to select an existing joke or create a new joke. The authors present JEST, a dataset consisting of news headlines in English, Spanish, and Chinese, each paired with four LLM-generated jokes. Eleven LLMs are then evaluated, tasked with either choosing one of the provided jokes or creating a new one. The authors set up two main experiments, through which they analyze the creative capabilities of the models that opted for selection, and vice versa.
Critical strengths
1) The underlying research question is interesting
When alternatives are already available, is an LLM capable of deciding whether to reuse one of them or create a better one? I believe that distinguishing between the decision to generate new content and the ability to improve upon existing content is potentially useful for collaborative writing and other workflows that rely on pre-existing options. However, I do not find the current experimental evidence and protocol sufficiently compelling and sound to establish how reliably these models make such a decision.
Critical weaknesses
1) The motivation for using humor and the reliability of its evaluation are not sufficiently established.
I like the underlying question of whether a model prefers to select existing content or create something new. However, I did not find the motivation for studying this question exclusively through humor sufficiently convincing. The introduction mentions collaborative writing, but I would have appreciated a clearer explanation of why humor is particularly suitable for isolating the selection–creation problem. In this setting, a model’s decision may also depend on whether it understands the joke and its cultural references, making it difficult to separate selection–creation behavior from humor-specific limitations.
This becomes especially important when humor is evaluated by other LLMs. I am not convinced that these evaluators can reliably determine which joke is better without some validation against human judgments. In this regard, I suggest discussing Phunny ("What do you call a dog that is incontrovertibly true? Dogma": Testing LLM Generalization through Humor, ACL 2025), which reports difficulties in LLM pun comprehension, resolution, and generation relative to human performance. Although these findings do not directly establish the reliability of the evaluators used here, they motivate checking whether the models understand and appreciate the humor they are asked to assess. Using multiple evaluators and comparing evaluator panels are useful controls, but agreement among LLMs does not establish agreement with human readers. A human evaluation on a subset of the data, covering all three languages, would therefore provide important support for the quality comparisons.
The use of three languages makes this question even more important. What readers find funny can depend on cultural background and familiarity with the news being discussed. I would therefore like to understand which audiences the evaluation is intended to represent and whether the judgments are reliable within each language. The authors acknowledge the need for human evaluation, but this remains a substantial limitation because the conclusions about quality gains depend on these scores.
Finally, I also could not find an explanation of how the four candidate jokes were checked during dataset construction. Were they verified to be relevant to the headline and meaningfully different from one another? Were there systematic differences in length, style, or joke structure? It would be useful to examine whether higher scores or more frequent selection correlate with joke length, since such patterns could affect the interpretation of the results.
2) It is unclear how much the selection–creation findings depend on the experimental setup.
The results are obtained with four candidates and a single select-or-create prompt formulation. My concern is that the observed behavior may depend substantially on these choices. For example, increasing the number of candidates could make selection more likely simply because the model has more opportunities to find an acceptable joke. What would happen with only one candidate, particularly if that candidate were the highest-scoring of the original four? Comparing different candidate-pool sizes and quality levels would help clarify whether the reported preferences remain stable.
The prompt itself is another important factor. Models are instructed to select if a candidate is sufficiently good and create otherwise. However, different models may interpret “sufficiently good” differently, and their decisions could change with a different formulation of the same task. Since only one formulation is tested, I would like to see whether the main findings persist across alternative prompts before interpreting the selection rates as stable model preferences.
I am also concerned about the repeatability of these decisions. Each model makes only one initial decision per headline at temperature 0.5. Therefore, we do not know whether repeating the same input would lead the model to select the same candidate, choose another candidate, or switch from selection to creation. The three forced generations in Experiment A address a different question and do not establish the stability of the initial decision.
Finally, I did not understand the motivation for consistently disabling reasoning, or using the lowest available level when it cannot be disabled. Enabling reasoning could change how models compare the candidates and decide whether to create.
The candidate-source analysis is a useful starting point, but I would also like to see a finer-grained breakdown of which generating models are preferred by each selecting model. Beyond reporting the aggregate proportions by language, the paper should try to explain whether these choices reflect candidate quality, stylistic differences, or model-family preferences.
Questions for authors
- Regarding the evaluator subsets described around lines 178–186, I understand that excluding evaluators from the same model family is intended to reduce self-preference. However, I found the procedure difficult to follow. Could you explain more intuitively how these subsets are constructed and how comparisons remain meaningful when different jokes are scored by different evaluator groups?
- Regarding dataset construction, how did you verify that the four candidates are relevant jokes for the headline and meaningfully different from one another?
- Is there any correlation between joke length and the quality scores or selection frequency? I would find this analysis useful to understand whether models prefer particular jokes because of their humorous content or whether length and structural style also influence their decisions.
- Why did you choose to disable reasoning whenever possible and otherwise use the lowest available level? I would appreciate a clearer motivation, since reasoning could affect both candidate evaluation and the decision to select or create. Have you checked whether the main patterns persist when reasoning is enabled?
- How stable are the initial decisions when the same input is evaluated more than once? In particular, how often does a model select the same candidate, switch to another candidate, or move from selection to creation? The three forced generations are useful for evaluating generation quality, but they do not answer this question about the initial decision.
- The analysis of candidate sources by language is a useful starting point. Could you also show which generating models are preferred by each selecting model? I would like to understand whether the higher selection frequency of GLM-5 jokes is shared across models and whether it can be explained by quality scores, length, or stylistic characteristics.
9 comments
Rebuttal - Part II
2. LLM evaluators and human judgment (W1, paragraphs 2–3)
We take your concern about LLM evaluators seriously. To address it, we added two checks, both reported per language: inter-evaluator agreement and a pairwise comparison experiment that involves no scoring. In every language, the four evaluators from different families agree in direction, and when they compare two jokes directly instead of scoring them, they still choose the highest-scoring candidate over the model's creation more often. These checks still rely on LLM evaluators: the quality comparisons in our paper are made by LLM evaluators, and we did not run a separate human evaluation.
Inter-evaluator agreement within each language: the four evaluators from different families agree in direction, and item-level agreement is moderate. Using the weighted scores, we computed two measures per language over the 6 evaluator pairs formed by the four evaluators: the Spearman correlation between two evaluators' weighted scores, and pairwise ordering agreement, i.e., the proportion of candidate pairs for the same headline that the two evaluators order the same way. As the table below shows, agreement is similar across the three languages.
| Range over the 6 evaluator pairs | English | Spanish | Chinese |
|---|---|---|---|
| Spearman correlation of weighted scores | 0.48–0.62 | 0.53–0.64 | 0.52–0.63 |
| Pairwise ordering agreement | 63.2%–68.8% | 68.3%–72.3% | 61.6%–67.2% |
Moderate item-level agreement is the norm in humor evaluation, among human annotators as well, and reliability comes from aggregation. SemEval-2026 Task 1, the source of our headlines, covers the same three languages and uses human evaluation. Its overview [1] reports that among the 288 pairwise comparisons with votes from multiple annotators, annotators agreed in 46.8% of cases (Fleiss' κ = 0.15), whereas system rankings computed separately from two disjoint halves of the annotators correlate at 0.79, 0.90 and 0.79 (Spearman) for English, Spanish and Chinese, which the authors summarize as “noisy individual judgments, stable system-level rankings”. Our quality conclusions are drawn at the same aggregate level: each text's score is the mean over multiple evaluators, and each comparison is averaged over all headlines evaluated in a model–language group.
Pairwise comparison experiment: with no scoring and a direct comparison of two jokes, the conclusion is the same in every language. We added a pairwise comparison experiment in which the same four LLM evaluators no longer assign scores and only judge which of two jokes is better. They see the headline and two jokes, shown in random order without source labels: the model's creation and the highest-scoring candidate as determined by the scores. Evaluators from the same model family as the model that produced the creation do not take part. In all three languages, the evaluators chose the highest-scoring candidate more often: the win rate of the creation for the eight models is 16.4%–40.3% in English, 12.9%–33.8% in Spanish and 18.0%–39.8% in Chinese, and the highest upper bounds of the 95% CIs are 44.8%, 38.2% and 44.6%, all below 50%. For this pairwise format, prior work shows that LLM evaluators are no worse than crowd workers: taking rankings from large-scale crowd-sourced ratings as the reference, Zhang et al. [2] find that GPT-4-family evaluators judge which of two captions is better with 60.5%–67% accuracy and crowd workers with 60.8%–61.7%.
Rebuttal - Part III
2. LLM evaluators and human judgment (W1, paragraphs 2–3), continued
On the three languages: our scores are assigned by LLM evaluators following the scoring rubric in each language, which does not specify a particular group of readers; every quality comparison is made within one language and one headline, and the quality conclusions are reliable within each language. Specifically, the rubric (Prompt 4 in the appendix) has English, Spanish and Chinese versions, and the evaluators use the version in the language of the headline. The creation and the candidates address the same headline in the same language, so the content and background of the news are the same on both sides. Quality conclusions are reported per language, and scores are not compared across languages. As for whether the judgments are reliable within each language: the two checks above are both reported per language. In every language, the four evaluators from different families agree in direction, and the pairwise comparison without scoring gives the same conclusion, indicating that the quality conclusions within each language depend neither on any single evaluator nor on scoring as the evaluation method. You point out that what readers find funny depends on cultural background and familiarity with the news. If a human evaluation is carried out on top of the current experiments, the annotators' native language, cultural background and familiarity with the news will need to be taken into account, and we will follow your suggestion in designing the annotation.
Thank you for recommending Phunny. It reports gaps between LLMs and humans in pun comprehension, resolution and generation. Section 2 of the paper already discusses how LLM-generated humor compares with human humor, and we will add Phunny to this discussion.
3. Candidate quality and length (W1, paragraph 4; Q2; Q3)
We computed statistics for each of the points you raise. The candidates are relevant to the headlines and distinct from one another; candidates from different sources differ in length; and length correlates positively, but weakly, with scores and with selection.
Relevance and distinctness of the candidates (Q2). The four candidates for each headline are generated independently by Qwen3-Max, DeepSeek-V3.2, Kimi-K2.5 and GLM-5 with the same prompt (Prompt 1 in the appendix), one per model, and are used unchanged. Candidate quality therefore varies, which is exactly what a model faces when judging whether the existing candidates are sufficiently good. We ran the following checks on these candidates. On relevance: our scoring includes a news relevance dimension. The rubric asks the evaluators to score strictly and gives verbal anchors: 1 for “keywords only”, 3 for “weak and clumsy”, 5 for “surface-level”, 7 for “good use of news” and 9 for “tightly interwoven”. Among the scored candidates of Experiments A and B, across the 12 language–source combinations, the mean relevance score is 5.90–7.92 and the median is 6.1–8.2, i.e., between “surface-level” and “good use of news” or higher. In every combination at least three quarters of the candidates score 5 or above, i.e., at the “surface-level” anchor or higher, above the “keywords only” and “weak and clumsy” levels. On distinctness: none of the 3,923 candidate pools contains duplicate candidates, and the character-level ROUGE-L F1 between two candidates in the same pool averages 0.437 (English), 0.456 (Spanish) and 0.230 (Chinese). On length and style: in non-whitespace characters, the mean length of the four sources is 86.7–129.3 in English, 100.7–139.0 in Spanish and 48.3–60.8 in Chinese. Style and joke structure vary with the generator; we kept each model's output as written and did not quantify them separately.
Rebuttal - Part I
Dear Reviewer,
We are pleased that you find the underlying research question interesting and potentially useful for collaborative writing and other workflows that rely on pre-existing options, and we thank you for your detailed comments on the motivation for using humor, the reliability of the evaluation, and the dependence of the findings on the experimental setup. To address them, we have added several analyses of the evaluation and the candidate pool and two new experiments. We hope that these new results, together with the clarifications below, address your concerns.
1. Why a humor task (W1, paragraph 1)
Thank you for raising these two questions: why we study the select-or-create decision through humor, and whether a model's understanding of the jokes could be entangled with its selection-creation behavior.
To isolate the selection-creation problem, a task must leave room for a new output to be better even when the candidates are already good. Humor has this property. Humor is open-ended: many different jokes can be written for the same headline, so a new joke can still improve on a good candidate. Tasks with a reference answer, such as math problems or factual questions, are different: a new answer cannot improve on a correct candidate, and the choice between adopting and creating reduces to judging whether the candidate is correct. The task is also convenient to measure. The headline is the shared input of the candidates and the new joke, so they can be compared headline by headline; the headlines cover English, Spanish and Chinese, allowing cross-lingual checks; and jokes are short, so all four candidates fit in a single prompt. Selecting among candidates or writing a new joke is also a real step in systems built for this task: in SemEval-2026 Task 1, the source of our headlines, 18 of the 28 systems with description papers first generate a pool of candidate jokes and then select from it [1].
On the models' understanding of the jokes. Before deciding whether to adopt or create, a model has to read and evaluate the candidates. The same holds in collaborative writing, where deciding whether to keep or rewrite a passage requires understanding it first. Section 3 of the paper designs the task around this scenario, with the model acting as both evaluator and creator. Our approach is to measure these components separately: the selection rate records which action the model takes, the top-score hit rate and the candidate selection gap record whether it identifies the highest-scoring candidate when it selects, and the score differences record the outcome of creation. Two results bear directly on the entanglement you describe. First, all 11 models receive the same headlines, candidates and instruction, so the cultural background of the headlines and jokes is the same for every model. The three languages differ in headlines and cultural background, yet the model ranking is consistent across them (Spearman correlations of 0.945–0.964), i.e., the differences between models appear consistently in three different linguistic and cultural settings.
Second, selection-preferring and creation-preferring models identify candidates about equally well. In Table 1 of the paper, the four selection-preferring models hit the highest-scoring candidate in 47.0%–61.8% of their selections, and their selected candidates score on average 0.33–0.65 points below it. For the four creation-preferring models (Kimi-K2.5, Kimi-K2.6, Qwen3.6-27B and Qwen3.6-Flash), we added a forced selection experiment: on the Experiment B instances, i.e., headlines where the model's initial decision was CREATE, we gave the same model the same headline and the same four candidates and required it to select the best one, with the same sampling settings as the initial decision. In all 12 model–language groups, the model's initial creation scores on average below the candidate it selected (by 0.06–0.95 points), significantly so in 8 groups. Pooled over the three languages, the four models' hit rates are 49.0%–61.5%, and their selected candidates score on average 0.34–0.61 points below the highest-scoring candidate (the hit rate under random selection is 25%). That is, the two groups identify candidates about equally well, while their selection rates differ widely.