acceptodds
Under review as a conference paper at ICLR 2027

Select or Create? Behavioral Preferences and Quality Gains in LLM Humor Tasks

Abstract

Do large language models (LLMs) prefer to select existing content or create new alternatives, and are such behavioral preferences associated with quality gains? We investigate selection-creation preferences in a news-based humor task, in which models evaluate candidate jokes and decide whether to adopt one unchanged or generate a new alternative. We introduce JEST, a multilingual dataset comprising 3,923 news headlines in English, Spanish, and Chinese, each paired with four jokes generated independently by distinct LLMs. We further propose an evaluation framework that conditions quality comparisons on models’ initial decisions. Across 11 state-of-the-art LLMs, we uncover distinct behavioral preferences, with average selection rates ranging from 33.7% to 99.8% and model rankings proving highly consistent across language datasets. Subsequent quality evaluation reveals that frequent selection does not guarantee accurate identification of the highest-scoring candidate. Forcing selection-preferring models to create does not consistently yield jokes that outscore their initial selections. For GPT-5.4, forced generations score higher than its selections but still lower than the highest-scoring candidates on average. Similarly, although some creation-preferring models generate jokes that score above the candidate mean, none achieves an average score as high as that of the highest-scoring candidates in any language. Overall, our findings highlight the need to jointly evaluate models' abilities to generate new content, identify high-quality candidates, and decide when to create.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Computational Humor, Selection-Creation Preferences, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_C7BPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether LLMs prefer to select an existing joke or create a new joke. The authors present JEST, a dataset consisting of news headlines in English, Spanish, and Chinese, each paired with four LLM-generated jokes. Eleven LLMs are then evaluated, tasked with either choosing one of the provided jokes or creating a new one. The authors set up two main experiments, through which they analyze the creative capabilities of the models that opted for selection, and vice versa.

Critical strengths

1) The underlying research question is interesting

When alternatives are already available, is an LLM capable of deciding whether to reuse one of them or create a better one? I believe that distinguishing between the decision to generate new content and the ability to improve upon existing content is potentially useful for collaborative writing and other workflows that rely on pre-existing options. However, I do not find the current experimental evidence and protocol sufficiently compelling and sound to establish how reliably these models make such a decision.

Critical weaknesses

1) The motivation for using humor and the reliability of its evaluation are not sufficiently established.

I like the underlying question of whether a model prefers to select existing content or create something new. However, I did not find the motivation for studying this question exclusively through humor sufficiently convincing. The introduction mentions collaborative writing, but I would have appreciated a clearer explanation of why humor is particularly suitable for isolating the selection–creation problem. In this setting, a model’s decision may also depend on whether it understands the joke and its cultural references, making it difficult to separate selection–creation behavior from humor-specific limitations.

This becomes especially important when humor is evaluated by other LLMs. I am not convinced that these evaluators can reliably determine which joke is better without some validation against human judgments. In this regard, I suggest discussing Phunny ("What do you call a dog that is incontrovertibly true? Dogma": Testing LLM Generalization through Humor, ACL 2025), which reports difficulties in LLM pun comprehension, resolution, and generation relative to human performance. Although these findings do not directly establish the reliability of the evaluators used here, they motivate checking whether the models understand and appreciate the humor they are asked to assess. Using multiple evaluators and comparing evaluator panels are useful controls, but agreement among LLMs does not establish agreement with human readers. A human evaluation on a subset of the data, covering all three languages, would therefore provide important support for the quality comparisons.

The use of three languages makes this question even more important. What readers find funny can depend on cultural background and familiarity with the news being discussed. I would therefore like to understand which audiences the evaluation is intended to represent and whether the judgments are reliable within each language. The authors acknowledge the need for human evaluation, but this remains a substantial limitation because the conclusions about quality gains depend on these scores.

Finally, I also could not find an explanation of how the four candidate jokes were checked during dataset construction. Were they verified to be relevant to the headline and meaningfully different from one another? Were there systematic differences in length, style, or joke structure? It would be useful to examine whether higher scores or more frequent selection correlate with joke length, since such patterns could affect the interpretation of the results.

2) It is unclear how much the selection–creation findings depend on the experimental setup.

The results are obtained with four candidates and a single select-or-create prompt formulation. My concern is that the observed behavior may depend substantially on these choices. For example, increasing the number of candidates could make selection more likely simply because the model has more opportunities to find an acceptable joke. What would happen with only one candidate, particularly if that candidate were the highest-scoring of the original four? Comparing different candidate-pool sizes and quality levels would help clarify whether the reported preferences remain stable.

The prompt itself is another important factor. Models are instructed to select if a candidate is sufficiently good and create otherwise. However, different models may interpret “sufficiently good” differently, and their decisions could change with a different formulation of the same task. Since only one formulation is tested, I would like to see whether the main findings persist across alternative prompts before interpreting the selection rates as stable model preferences.

I am also concerned about the repeatability of these decisions. Each model makes only one initial decision per headline at temperature 0.5. Therefore, we do not know whether repeating the same input would lead the model to select the same candidate, choose another candidate, or switch from selection to creation. The three forced generations in Experiment A address a different question and do not establish the stability of the initial decision.

Finally, I did not understand the motivation for consistently disabling reasoning, or using the lowest available level when it cannot be disabled. Enabling reasoning could change how models compare the candidates and decide whether to create.

The candidate-source analysis is a useful starting point, but I would also like to see a finer-grained breakdown of which generating models are preferred by each selecting model. Beyond reporting the aggregate proportions by language, the paper should try to explain whether these choices reflect candidate quality, stylistic differences, or model-family preferences.

Questions for authors
  1. Regarding the evaluator subsets described around lines 178–186, I understand that excluding evaluators from the same model family is intended to reduce self-preference. However, I found the procedure difficult to follow. Could you explain more intuitively how these subsets are constructed and how comparisons remain meaningful when different jokes are scored by different evaluator groups?
  2. Regarding dataset construction, how did you verify that the four candidates are relevant jokes for the headline and meaningfully different from one another?
  3. Is there any correlation between joke length and the quality scores or selection frequency? I would find this analysis useful to understand whether models prefer particular jokes because of their humorous content or whether length and structural style also influence their decisions.
  4. Why did you choose to disable reasoning whenever possible and otherwise use the lowest available level? I would appreciate a clearer motivation, since reasoning could affect both candidate evaluation and the decision to select or create. Have you checked whether the main patterns persist when reasoning is enabled?
  5. How stable are the initial decisions when the same input is evaluated more than once? In particular, how often does a model select the same candidate, switch to another candidate, or move from selection to creation? The three forced generations are useful for evaluating generation quality, but they do not answer this question about the initial decision.
  6. The analysis of candidate sources by language is a useful starting point. Could you also show which generating models are preferred by each selecting model? I would like to understand whether the higher selection frequency of GLM-5 jokes is shared across models and whether it can be explained by quality scores, length, or stylistic characteristics.
9 comments
AuthorsOpenReview ↗

Rebuttal - Part I

Dear Reviewer,

We are pleased that you find the underlying research question interesting and potentially useful for collaborative writing and other workflows that rely on pre-existing options, and we thank you for your detailed comments on the motivation for using humor, the reliability of the evaluation, and the dependence of the findings on the experimental setup. To address them, we have added several analyses of the evaluation and the candidate pool and two new experiments. We hope that these new results, together with the clarifications below, address your concerns.

1. Why a humor task (W1, paragraph 1)

Thank you for raising these two questions: why we study the select-or-create decision through humor, and whether a model's understanding of the jokes could be entangled with its selection-creation behavior.

To isolate the selection-creation problem, a task must leave room for a new output to be better even when the candidates are already good. Humor has this property. Humor is open-ended: many different jokes can be written for the same headline, so a new joke can still improve on a good candidate. Tasks with a reference answer, such as math problems or factual questions, are different: a new answer cannot improve on a correct candidate, and the choice between adopting and creating reduces to judging whether the candidate is correct. The task is also convenient to measure. The headline is the shared input of the candidates and the new joke, so they can be compared headline by headline; the headlines cover English, Spanish and Chinese, allowing cross-lingual checks; and jokes are short, so all four candidates fit in a single prompt. Selecting among candidates or writing a new joke is also a real step in systems built for this task: in SemEval-2026 Task 1, the source of our headlines, 18 of the 28 systems with description papers first generate a pool of candidate jokes and then select from it [1].

On the models' understanding of the jokes. Before deciding whether to adopt or create, a model has to read and evaluate the candidates. The same holds in collaborative writing, where deciding whether to keep or rewrite a passage requires understanding it first. Section 3 of the paper designs the task around this scenario, with the model acting as both evaluator and creator. Our approach is to measure these components separately: the selection rate records which action the model takes, the top-score hit rate and the candidate selection gap record whether it identifies the highest-scoring candidate when it selects, and the score differences record the outcome of creation. Two results bear directly on the entanglement you describe. First, all 11 models receive the same headlines, candidates and instruction, so the cultural background of the headlines and jokes is the same for every model. The three languages differ in headlines and cultural background, yet the model ranking is consistent across them (Spearman correlations of 0.945–0.964), i.e., the differences between models appear consistently in three different linguistic and cultural settings.

Second, selection-preferring and creation-preferring models identify candidates about equally well. In Table 1 of the paper, the four selection-preferring models hit the highest-scoring candidate in 47.0%–61.8% of their selections, and their selected candidates score on average 0.33–0.65 points below it. For the four creation-preferring models (Kimi-K2.5, Kimi-K2.6, Qwen3.6-27B and Qwen3.6-Flash), we added a forced selection experiment: on the Experiment B instances, i.e., headlines where the model's initial decision was CREATE, we gave the same model the same headline and the same four candidates and required it to select the best one, with the same sampling settings as the initial decision. In all 12 model–language groups, the model's initial creation scores on average below the candidate it selected (by 0.06–0.95 points), significantly so in 8 groups. Pooled over the three languages, the four models' hit rates are 49.0%–61.5%, and their selected candidates score on average 0.34–0.61 points below the highest-scoring candidate (the hit rate under random selection is 25%). That is, the two groups identify candidates about equally well, while their selection rates differ widely.

AuthorsOpenReview ↗

Rebuttal - Part II

2. LLM evaluators and human judgment (W1, paragraphs 2–3)

We take your concern about LLM evaluators seriously. To address it, we added two checks, both reported per language: inter-evaluator agreement and a pairwise comparison experiment that involves no scoring. In every language, the four evaluators from different families agree in direction, and when they compare two jokes directly instead of scoring them, they still choose the highest-scoring candidate over the model's creation more often. These checks still rely on LLM evaluators: the quality comparisons in our paper are made by LLM evaluators, and we did not run a separate human evaluation.

Inter-evaluator agreement within each language: the four evaluators from different families agree in direction, and item-level agreement is moderate. Using the weighted scores, we computed two measures per language over the 6 evaluator pairs formed by the four evaluators: the Spearman correlation between two evaluators' weighted scores, and pairwise ordering agreement, i.e., the proportion of candidate pairs for the same headline that the two evaluators order the same way. As the table below shows, agreement is similar across the three languages.

Range over the 6 evaluator pairsEnglishSpanishChinese
Spearman correlation of weighted scores0.48–0.620.53–0.640.52–0.63
Pairwise ordering agreement63.2%–68.8%68.3%–72.3%61.6%–67.2%

Moderate item-level agreement is the norm in humor evaluation, among human annotators as well, and reliability comes from aggregation. SemEval-2026 Task 1, the source of our headlines, covers the same three languages and uses human evaluation. Its overview [1] reports that among the 288 pairwise comparisons with votes from multiple annotators, annotators agreed in 46.8% of cases (Fleiss' κ = 0.15), whereas system rankings computed separately from two disjoint halves of the annotators correlate at 0.79, 0.90 and 0.79 (Spearman) for English, Spanish and Chinese, which the authors summarize as “noisy individual judgments, stable system-level rankings”. Our quality conclusions are drawn at the same aggregate level: each text's score is the mean over multiple evaluators, and each comparison is averaged over all headlines evaluated in a model–language group.

Pairwise comparison experiment: with no scoring and a direct comparison of two jokes, the conclusion is the same in every language. We added a pairwise comparison experiment in which the same four LLM evaluators no longer assign scores and only judge which of two jokes is better. They see the headline and two jokes, shown in random order without source labels: the model's creation and the highest-scoring candidate as determined by the scores. Evaluators from the same model family as the model that produced the creation do not take part. In all three languages, the evaluators chose the highest-scoring candidate more often: the win rate of the creation for the eight models is 16.4%–40.3% in English, 12.9%–33.8% in Spanish and 18.0%–39.8% in Chinese, and the highest upper bounds of the 95% CIs are 44.8%, 38.2% and 44.6%, all below 50%. For this pairwise format, prior work shows that LLM evaluators are no worse than crowd workers: taking rankings from large-scale crowd-sourced ratings as the reference, Zhang et al. [2] find that GPT-4-family evaluators judge which of two captions is better with 60.5%–67% accuracy and crowd workers with 60.8%–61.7%.

AuthorsOpenReview ↗

Rebuttal - Part III

2. LLM evaluators and human judgment (W1, paragraphs 2–3), continued

On the three languages: our scores are assigned by LLM evaluators following the scoring rubric in each language, which does not specify a particular group of readers; every quality comparison is made within one language and one headline, and the quality conclusions are reliable within each language. Specifically, the rubric (Prompt 4 in the appendix) has English, Spanish and Chinese versions, and the evaluators use the version in the language of the headline. The creation and the candidates address the same headline in the same language, so the content and background of the news are the same on both sides. Quality conclusions are reported per language, and scores are not compared across languages. As for whether the judgments are reliable within each language: the two checks above are both reported per language. In every language, the four evaluators from different families agree in direction, and the pairwise comparison without scoring gives the same conclusion, indicating that the quality conclusions within each language depend neither on any single evaluator nor on scoring as the evaluation method. You point out that what readers find funny depends on cultural background and familiarity with the news. If a human evaluation is carried out on top of the current experiments, the annotators' native language, cultural background and familiarity with the news will need to be taken into account, and we will follow your suggestion in designing the annotation.

Thank you for recommending Phunny. It reports gaps between LLMs and humans in pun comprehension, resolution and generation. Section 2 of the paper already discusses how LLM-generated humor compares with human humor, and we will add Phunny to this discussion.

3. Candidate quality and length (W1, paragraph 4; Q2; Q3)

We computed statistics for each of the points you raise. The candidates are relevant to the headlines and distinct from one another; candidates from different sources differ in length; and length correlates positively, but weakly, with scores and with selection.

Relevance and distinctness of the candidates (Q2). The four candidates for each headline are generated independently by Qwen3-Max, DeepSeek-V3.2, Kimi-K2.5 and GLM-5 with the same prompt (Prompt 1 in the appendix), one per model, and are used unchanged. Candidate quality therefore varies, which is exactly what a model faces when judging whether the existing candidates are sufficiently good. We ran the following checks on these candidates. On relevance: our scoring includes a news relevance dimension. The rubric asks the evaluators to score strictly and gives verbal anchors: 1 for “keywords only”, 3 for “weak and clumsy”, 5 for “surface-level”, 7 for “good use of news” and 9 for “tightly interwoven”. Among the scored candidates of Experiments A and B, across the 12 language–source combinations, the mean relevance score is 5.90–7.92 and the median is 6.1–8.2, i.e., between “surface-level” and “good use of news” or higher. In every combination at least three quarters of the candidates score 5 or above, i.e., at the “surface-level” anchor or higher, above the “keywords only” and “weak and clumsy” levels. On distinctness: none of the 3,923 candidate pools contains duplicate candidates, and the character-level ROUGE-L F1 between two candidates in the same pool averages 0.437 (English), 0.456 (Spanish) and 0.230 (Chinese). On length and style: in non-whitespace characters, the mean length of the four sources is 86.7–129.3 in English, 100.7–139.0 in Spanish and 48.3–60.8 in Chinese. Style and joke structure vary with the generator; we kept each model's output as written and did not quantify them separately.

Reviewer_cvxHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how large language models choose between adopting an existing joke and creating a new one, and how these choices relate to output quality. The authors introduce JEST, containing 3,923 news headlines in English, Spanish, and Chinese, each paired with four jokes from different models. They measure selection rates for 11 models and evaluate output quality for eight models conditional on their initial decisions. Experiment A examines forced generation following an initial selection, while Experiment B evaluates jokes produced during an initial creation decision. The results show substantial variation in selection rates and highly consistent model rankings across the three language datasets. Under the proposed LLM-based scoring procedure, selected candidates frequently fall below the highest-scoring available candidate. Some generated outputs exceed the selected candidate or candidate mean on average, while average scores remain below the highest-scoring candidate across all evaluated model and language combinations.

Critical strengths

Clear evaluation framework. Comparing generated outputs with the selected candidate, the candidate mean, and the highest-scoring candidate provides useful distinctions. In particular, the GPT-5.4 results illustrate how generation can improve upon a model’s actual selection while leaving higher-scoring existing content unused. This framework helps identify which capabilities deserve further evaluation.

Multilingual evidence. The behavioral study covers 11 models and 43,153 decisions, using common candidate pools for each headline and randomized candidate order. The high correlations between model rankings across language datasets are an informative empirical finding. The additional analysis using common evaluators within each comparison also addresses an important concern about comparing scores produced by different evaluator subsets.

Critical weaknesses

Limited quality validation. All quality comparisons rely on LLM scores, with no human evaluation confirming their relevance to readers’ humor preferences. Using the same scores to identify and evaluate the highest-scoring candidate may also exaggerate its advantage through scoring error. The common-evaluator analysis leaves these concerns unresolved.

Limited evidence on decision benefits. The experiments provide conditional quality comparisons, but the benefit of choosing between selection and creation remains unclear. In particular, Experiment B lacks a comparison with the candidate the same model would actually select if required to choose. Comparing free choice, mandatory selection, and mandatory creation on common inputs would clarify this benefit.

Questions for authors
  1. Do human evaluations across the three languages confirm the reported quality differences?

  2. Do the results persist when candidate selection and final comparison use independent assessments?

  3. On initial CREATE cases, how does the generated joke compare with the candidate the same model selects when required to choose?

5 comments
AuthorsOpenReview ↗

Rebuttal - Part I

Dear Reviewer,

We are glad that you found the evaluation framework clear and the multilingual evidence informative, and we thank you for your comments on quality validation and decision benefits. In response, we added one analysis and two new experiments, and the paper's conclusions hold under all three. We hope the responses below address your concerns.

1. Quality validation (W1, Q1, Q2)

Thank you for these two questions. We answer Q2 first and then Q1. On Q2: the results still hold. When candidate selection and the final comparison are assigned to two different evaluators, in none of the 24 model–language groups does the generated joke score significantly above the reference candidate, and in 22 groups it remains lower. You noted that using the same scores to identify and evaluate the highest-scoring candidate may exaggerate its advantage through scoring error. To address this, we had two different evaluators do the selecting and the scoring.

For each headline, we first use GPT-5.5's weighted score to select the highest-scoring of the four candidates, and then compare the generated joke with this candidate using only Gemini 3.1 Pro Preview's weighted score. Gemini's scores play no part in the selection, so this candidate's Gemini score is not inflated by being selected. Neither evaluator belongs to the same model family as any of the four candidate generators. The table below gives the mean score difference "generated joke − this candidate" with 95% bootstrap CIs (* marks CIs that exclude zero). Negative values mean the generated joke scores below the reference. For the first four models, the generated joke's score is the mean over the three forced generations in Experiment A; for the last four, it is the score of the initial creation in Experiment B.

ModelEnglishSpanishChinese
DeepSeek-V4-Flash−0.773* [−0.928, −0.612]−0.758* [−0.920, −0.596]−0.619* [−0.777, −0.460]
DeepSeek-V4-Pro−0.149 [−0.304, +0.010]−0.058 [−0.220, +0.105]−0.144 [−0.288, +0.003]
Qwen3-Max−0.605* [−0.770, −0.436]−0.447* [−0.615, −0.278]−0.315* [−0.461, −0.164]
GPT-5.4−0.091 [−0.296, +0.113]+0.138 [−0.094, +0.366]−0.045 [−0.251, +0.165]
Kimi-K2.5−0.095 [−0.282, +0.090]−0.277* [−0.496, −0.058]+0.014 [−0.224, +0.248]
Kimi-K2.6−0.002 [−0.204, +0.200]−0.024 [−0.226, +0.178]−0.131 [−0.353, +0.089]
Qwen3.6-27B−1.091* [−1.295, −0.880]−1.006* [−1.202, −0.810]−0.594* [−0.798, −0.390]
Qwen3.6-Flash−1.162* [−1.388, −0.937]−1.104* [−1.290, −0.915]−0.910* [−1.186, −0.632]

As the table shows, in 22 of the 24 groups the mean score difference is negative, in 13 the CI lies entirely below zero, and in none does the CI lie above zero. For DeepSeek-V4-Flash, Qwen3-Max, Qwen3.6-27B and Qwen3.6-Flash, the generated joke is significantly lower in all three languages (−1.162 to −0.315). For the other four models, the generated joke is at most on par with this reference (−0.277 to +0.138). That is, when selecting and scoring are done independently by different evaluators, the conclusion that no model's generated jokes significantly exceed the highest-scoring candidate does not change.

AuthorsOpenReview ↗

Rebuttal - Part II

1. Quality validation (W1, Q1, Q2), continued

We now turn to Q1. The quality comparisons in our paper are made by LLM evaluators, and we did not run a separate human evaluation. On human judgments of humor, two findings from prior work bear directly on our evaluation method. First, human judgments of humor are themselves highly inconsistent at the item level, and reliability comes from aggregation. SemEval-2026 Task 1, the source of our headlines, also covers English, Spanish and Chinese. Its overview [1] reports that among the 288 pairwise comparisons with votes from multiple annotators, annotators agreed in 46.8% of cases (Fleiss' κ = 0.15), whereas system rankings computed separately from two disjoint halves of the annotators correlate at 0.79, 0.90 and 0.79 (Spearman) for English, Spanish and Chinese, which the authors summarize as "noisy individual judgments, stable system-level rankings". Our quality conclusions are likewise relative comparisons at the aggregate level.

Second, when evaluators assign no scores and only judge which of two texts is better (pairwise comparison), prior work shows that LLM evaluators are no worse than crowd workers. Taking rankings from large-scale crowd-sourced ratings as the reference, Zhang et al. [2] find that GPT-4-family evaluators judge which of two captions is better with 60.5%–67% accuracy and crowd workers with 60.8%–61.7%. Using this format, we added a pairwise comparison experiment, and the quality difference between the generated joke and the highest-scoring candidate persists in all three languages.

In this experiment, the same four LLM evaluators no longer assign scores and only judge which of two jokes is better. They see the headline and two jokes, shown in random order without source labels: the model's generated joke and the highest-scoring candidate as determined by the scores. Evaluators from the same model family as the model that produced the generated joke do not take part. In all three languages, the evaluators chose the highest-scoring candidate more often: the win rate of the generated joke for the eight models is 16.4%–40.3% in English, 12.9%–33.8% in Spanish and 18.0%–39.8% in Chinese, and the upper bounds of all 95% CIs are below 50% (the highest is 44.8%). That is, when the evaluators compare two jokes directly, without dimensions, weights or a scale, their judgments agree with the scores in all three languages.

2. Gains from the decision (W2, Q3)

Following your suggestion, we added a forced selection experiment: on headlines where the model's initial decision was CREATE, we required the same model to select one of the candidates instead. In all 12 model–language groups, the model's creation scores on average below the candidate selected by the same model, significantly so in 8 groups, and in no group is the creation significantly higher.

Specifically, on the instances of Experiment B, we gave the same model the same headline and the same four candidates, and required it to select the best one, with the same sampling settings as in the initial decision. We then compared, headline by headline, the model's initial creation with the candidate it selected. The model, headline and candidates are the same for both, and the only difference is whether the action is creation or selection. The table below gives the mean score difference "initial creation − selected candidate" with 95% bootstrap CIs (* marks CIs that exclude zero). Negative values mean the creation scores below the candidate the model itself selected.

ModelEnglishSpanishChinese
Kimi-K2.5−0.090 [−0.243, +0.063]−0.192* [−0.355, −0.025]−0.143 [−0.342, +0.046]
Kimi-K2.6−0.062 [−0.231, +0.108]−0.090 [−0.226, +0.045]−0.273* [−0.434, −0.117]
Qwen3.6-27B−0.728* [−0.884, −0.572]−0.735* [−0.910, −0.556]−0.787* [−0.944, −0.632]
Qwen3.6-Flash−0.947* [−1.111, −0.783]−0.889* [−1.060, −0.716]−0.931* [−1.152, −0.707]

As the table shows, all 6 groups of the two Qwen models are significant, with the creation 0.73–0.95 points below the selected candidate. In the 6 groups of the two Kimi models, the creation is also lower on average, by 0.06–0.27 points, significantly so in 2 of them. Table 4 of the paper shows that the two Kimi models' creations score above the candidate mean. Here we further see that their creations still do not exceed the candidates they themselves select. That is, on the headlines where these models judged no candidate good enough and created a new joke, selecting instead gives an output that is better or at least no worse.

AuthorsOpenReview ↗

Rebuttal - Part III

2. Gains from the decision (W2, Q3), continued

When required to select, these four models identify the highest-scoring candidate at rates similar to those of the selection-preferring models. We examined how well they select under forced selection using the two metrics of Table 1 of the paper: the top-score hit rate, i.e., the proportion of cases where the selected candidate is the highest-scoring candidate, and the candidate selection gap, i.e., the mean score difference between the highest-scoring candidate and the selected candidate. Across languages, the four models' hit rates are 43.5%–65.0%, all well above the 25% of random selection, and the candidates they select score on average 0.27–0.69 points below the highest-scoring candidate. Pooled over the three languages, the hit rates are 49.0%–61.5% and the candidate selection gaps are 0.34–0.61 points, similar to those of the four selection-preferring models in Table 1 of the paper (47.0%–61.8% and 0.33–0.65 points). Each cell below gives top-score hit rate / candidate selection gap (points).

ModelEnglishSpanishChinese
Kimi-K2.558.5% / 0.40863.0% / 0.38257.0% / 0.335
Kimi-K2.656.5% / 0.43063.0% / 0.32865.0% / 0.266
Qwen3.6-27B43.5% / 0.69453.5% / 0.62856.0% / 0.388
Qwen3.6-Flash50.0% / 0.57049.0% / 0.67747.3% / 0.570

You suggested comparing free choice, forced selection and forced generation on common inputs. Experiment A provides the comparison between free choice and forced generation, and the new forced selection experiment provides the comparison between free choice and forced selection, with each direction covering four models. On headlines where the model freely selects, its choice is already selection, so the other action to compare is forced generation. Experiment A makes this comparison for the four selection-preferring models (the "vs. selected candidate" column of Table 3 of the paper): in 8 of the 12 groups forced generation scores significantly below the candidate the model itself selected, and in 2 groups significantly above it (GPT-5.4 in Spanish and Chinese).

On headlines where the model freely creates, its choice is already creation, so the other action to compare is forced selection. The forced selection experiment makes this comparison for the four creation-preferring models, with the results shown in the score-difference table above. Both directions are run on the same model, the same headline and the same set of candidates. As for the benefit of the decision, under our quality measure, for the selection-preferring models other than GPT-5.4, adopting a candidate is better than creating a new joke in most groups, and for the four creation-preferring models, setting the candidates aside to create a new joke brings no quality gain.

Planned revisions

We will make the following additions in the final version. Section 5.3 will include the forced selection experiment, with per-group score differences, top-score hit rates and candidate selection gaps in Table 4 of Appendix B (W2, Q3). Appendix A (Robustness of Quality Evaluation) will include the analysis with separate evaluators for selecting and scoring, as well as the pairwise comparison experiment, and Section 5.4 will point to these results (W1, Q1, Q2). Appendix C will include the prompts used for forced selection and the pairwise comparison.

Thank you again for your suggestions. The two checks you proposed in Q2 and Q3 have strengthened the paper's evidence on the quality comparisons and decision benefits. If you have any remaining questions, we would be happy to answer them during the discussion period.

References

[1] Castro et al. SemEval-2026 Task 1: MWAHAHA, Models Write Automatic Humor And Humans Annotate. SemEval 2026.

[2] Zhang et al. Humor in AI: Massive scale crowd-sourced preferences and benchmarks for cartoon captioning. NeurIPS 2024.

Reviewer_uiYzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates “selection-creation preference” in LLMs, i.e., whether an LLM tends to “adopt existing content” vs. “create new content” in the context of news-based humor. They characterize models by whether they tend to select vs. create, and then analyze how this correlates with quality in three ways:

  1. When models select, do they select the highest-quality candidate?
  2. If a model would select but you make them create anyways, or does it improve upon the initial selection?
  3. If a model chose to create, does it outperform the all four candidates?

They create JEST, a ~4k sample humor dataset in 3 languages with one headline paired with four “independently generated candidate jokes”.

Their findings over 8 or 11 models (depending on which stage of the analysis) are that:

  1. Models’ preference for selection vs. creation differ but they tend to have consistent rankings across languages
  2. When models select, they are not always good at choosing the “best” candidate, and forced generation doesn’t really improve over that candidate much
  3. When models choose to create, they don’t always do better than the maximum of the four provided candidates
Critical strengths

None

Critical weaknesses

My primary concern: I think this paper fails to make a case for why it is important. After reading the full paper, I still am unclear on why this particular testbed (humor from news article) is particularly appropriate. Moreover, I do not find either this setting or the "stakes" of the findings to be inherently consequential, and the paper does not refute this impression in either its framing or its experiments. For instance, the abstract seems to pitch an “evaluation framework that conditions quality comparisons on models’ initial decisions”. But this "framework" just means that you compare the models’ generation either against the selected candidate (if the model selected a joke) or the mean of candidates (if it chose to create). I also don’t believe the findings are particularly surprising -- the main takeaway is that some models would select and some would choose, and behavioral variance between models is a known phenomenon.

My secondary concern is in the rigor of the experimental design, specifically in scoring the quality of either candidate jokes or model generations. It’s unclear to me how carefully the humor evaluators were constructed (e.g. their agreement/disagreement? variance? etc.), how the four dimensions of humor (relevance, hymor, creativity, and naturalness) were selected, or how the authors arrived at the weighting of [0.1, 0.4, 0.3, 0.2] for the humor dimensions. Section 5.4’s heading “How Robust are the Quality Comparisons?” suggests that it should discuss these decisions, but is instead primarily concerned with excluding models from the same family of the evaluated model from the panel of LLM-as-a-judge. Without detail on why I should trust this quality metric, I find it hard to take the findings reported in the paper at face value.

Questions for authors
  1. What have you done (e.g. human evaluation) to established that your humor quality metric is both robust and correlates with human judgement?

  2. Your instructions ask the model to decide between selecting and creating. However, the instructions say to select if the candidate is “sufficiently good”, so it’s unclear to me if variance in model behavior is a result of different interpretations of this instruction or indicative of an actual selection-creation preference. Could you explain further on how you differentiate between the two?

5 comments
AuthorsOpenReview ↗

Rebuttal - Part I

Dear Reviewer,

Thank you very much for your review, and for your comments on the significance of the study and the reliability of the quality metric. To address your questions about the quality metric, we have added two analyses and one new experiment, and the paper's quality conclusions remain unchanged under all of them. We respond to each point below, and we hope these clarifications and new results resolve your concerns.

1. Research question and significance (W1)

We appreciate your concern regarding the significance of this study. Models can now both evaluate and create, and in practice they often have existing content in front of them: a draft written earlier in collaborative writing, or several answers to the same question produced by different models. In such cases a model can either adopt the existing content or create something new, so the quality of the final output depends not only on how well the model can generate or evaluate, but also on the decision it makes between these two actions. In SemEval-2026 Task 1, the source of our headlines, 18 of the 28 systems with description papers first generate a pool of candidate jokes and then select from it, and the task overview notes that this makes “reliable selection at least as important as generation quality” [1].

Existing methods, however, usually specify in advance whether the model selects or generates. Which action models favor when the decision is left to them, and whether this preference is accompanied by quality gains, have remained insufficiently characterized; these are the two questions posed at the start of our abstract. Our results show that models differ widely in this decision and that the preference by itself does not guarantee quality: under identical headlines, candidates, and instructions, the average selection rates of the 11 models range from 33.7% to 99.8%, and “neither frequent selection nor creation alone guarantees better use of available content”.

We chose the news humor task primarily because both actions are reasonable in it. Humor is open-ended: many different jokes can be built around the same headline, so creating a new one can still be better even when the candidates are good. By contrast, in tasks with a standard answer, such as math problems or factual QA, creating a new answer brings no gain when a candidate is correct, and the problem reduces to judging whether candidates are correct. The task also makes measurement straightforward: the headline is the shared input of the candidates and the new joke, which allows paired comparison on each item; the headlines cover English, Spanish, and Chinese, which allows cross-lingual testing; and jokes are short, so four candidates fit in one prompt and all 11 models were run on all 3,923 headlines.

The evaluation framework goes beyond the two comparisons you mention: it breaks quality down by the model's initial decision. When a model selects, we test whether it identified the highest-scoring candidate (top-score hit rate and candidate selection gap). When a model creates, we compare its joke with the candidate mean and the highest-scoring candidate. For selection-preferring models, we also measure the action they did not take: on headlines where they selected, we require them to create a new joke (Experiment A) and compare it with their own selected candidate and the highest-scoring candidate. These comparisons must be read together. Take GPT-5.4: its forced generation scores on average 0.06–0.26 points above its own selected candidate, so on this comparison alone creation pays off. But its selected candidate is on average 0.649 points below the highest-scoring candidate, and its forced generation is still 0.36–0.68 points below it. We summarize this case as “gains from generation can coexist with unrealized gains from better candidate selection”.

AuthorsOpenReview ↗

Rebuttal - Part II

1. Research question and significance (W1), continued

Taken together, the main takeaway of our paper is not that models differ, but that behavioral preferences alone do not establish quality gains. Even for the preferences themselves, we find more than variance: the model ranking is consistent across the three languages (Spearman correlations of 0.945–0.964). Building on this, the three analyses you list in your summary give the quality results. The four selection-preferring models identify the highest-scoring candidate in 47.0%–61.8% of their selections, and when they are required to create a new joke on the same headlines, 8 of the 12 model–language groups score significantly lower than their own selected candidates and only 2 significantly higher (Table 3 in the paper). Among the four creation-preferring models, the two Kimi models' jokes score on average above the candidate mean in all three languages (+0.47 to +0.85 points), but all four models' jokes score on average below the highest-scoring candidate in all three languages (Table 4 in the paper).

Frequent selection thus does not guarantee identifying the highest-scoring candidate, and choosing to create does not yield jokes that surpass it; neither can be read off from behavioral differences alone. Our overall conclusion is therefore that models' abilities to generate new content, identify high-quality candidates, and decide when to create need to be evaluated jointly. For models that can both evaluate and create, knowing how often they adopt or create is not enough to tell how effectively they use the available candidates, and this is where we see the significance of this study.

2. Reliability of the quality metric (W2, Q1)

Thank you for your question on why this quality metric can be trusted. Below we first explain how the evaluators, scoring dimensions, and weights were set, and then present two added analyses (weight sensitivity and inter-evaluator agreement) and an added pairwise comparison experiment. The quality conclusions do not depend on this set of weights, this set of evaluators, or scoring as the evaluation method: across five weightings, two evaluator compositions (Section 5.4 of the paper), and two evaluation methods, no model's generated jokes surpass the highest-scoring candidate.

Evaluators, scoring dimensions, and weights. The four evaluators (GPT-5.5, Gemini 3.1 Pro Preview, Qwen3.7-Max, and GLM-5.2) come from four different model families. Each evaluator sees one headline and one joke at a time, without knowing the joke's source, and scores the four dimensions separately on a 0–10 scale following a rubric with verbal anchors (Prompt 4 in the appendix). Evaluators from the same model family as a text's generator do not score that text, and the weighted score is the mean over the remaining evaluators (Eq. 1). The four dimensions correspond to four requirements for a news joke: being about the given headline, being funny, having a novel idea, and reading naturally. The first three are the criteria for “sufficiently good” in the instruction (Prompt 2: “funny, creative, and relevant to the news”). The weights reflect the task's focus on humorous content, which the models both evaluate and create, so humor (0.4) and creativity (0.3) receive higher weights, while news relevance (0.1) and naturalness and fluency (0.2), which are basic requirements for a news joke, receive lower weights.

AuthorsOpenReview ↗

Rebuttal - Part III

2. Reliability of the quality metric (W2, Q1), continued

Weight sensitivity: the conclusions are unchanged under five weightings. To test whether the conclusions depend on these weights, we kept the evaluators' original scores, changed only the weights, and recomputed all comparisons in Tables 1, 3, and 4 of the paper, re-identifying the highest-scoring candidate for each headline under the new weighted scores. Under all five weightings, the generated jokes of all 24 model–language groups score below the highest-scoring candidate, with every 95% CI entirely below zero; this holds even when only the humor dimension is scored. The top-score hit rates of the individual model–language groups are also essentially unchanged (43.0%–68.5% across the five weightings, 44.0%–68.0% under the original setting), indicating low sensitivity to the weights. The table below gives the range, across model–language groups, of the mean difference between the generated joke and the highest-scoring candidate; negative values mean the generated joke scores lower.

Weights (news relevance / humor / creativity / naturalness and fluency)Experiment A: forced generation − highest-scoring candidateExperiment B: creation − highest-scoring candidate
Original 0.10 / 0.40 / 0.30 / 0.20−1.152 to −0.364−1.566 to −0.419
Equal weights 0.25 / 0.25 / 0.25 / 0.25−1.161 to −0.411−1.512 to −0.425
Higher humor 0.10 / 0.50 / 0.20 / 0.20−1.198 to −0.396−1.623 to −0.448
Higher creativity 0.10 / 0.30 / 0.40 / 0.20−1.108 to −0.334−1.510 to −0.392
Humor only 0 / 1 / 0 / 0−1.314 to −0.435−1.766 to −0.493

Inter-evaluator agreement and score dispersion: the four evaluators from different families agree in direction, and item-level agreement is moderate. Using weighted scores under the original weights, we computed two metrics per language over the 6 evaluator pairs formed by the four evaluators: the Spearman correlation between two evaluators' weighted scores, and pairwise ordering agreement, i.e., for each pair of candidates under the same headline, the proportion in which the two evaluators give the same ordering. On the variance you mention, the standard deviation of the evaluators' weighted scores for the same text averages 0.6–1.0 points (10-point scale). The table below gives the ranges of the two metrics in each language.

Range over the 6 evaluator pairsEnglishSpanishChinese
Spearman correlation of weighted scores0.48–0.620.53–0.640.52–0.63
Pairwise ordering agreement63.2%–68.8%68.3%–72.3%61.6%–67.2%

Limited item-level agreement is the norm in humor evaluation, among human annotators as well, and reliability comes from aggregation. SemEval-2026 Task 1, the source of our headlines, uses human evaluation. Its overview [1] reports that among the 288 pairwise comparisons with votes from multiple annotators, annotators agreed in 46.8% of cases (Fleiss' κ = 0.15), whereas system rankings computed separately from two disjoint halves of the annotators correlate at 0.79, 0.90 and 0.79 (Spearman) for English, Spanish and Chinese, which the authors summarize as “noisy individual judgments, stable system-level rankings”. Our quality conclusions likewise rest on aggregation: each text's score is the mean over multiple evaluators, and each comparison is averaged over all headlines evaluated in a model–language group.

Reviewer_7JbEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. A behavioral study, not a method. JEST pairs 3,923 news headlines (EN / ES / ZH) with four LLM-written jokes each; 11 LLMs either adopt one candidate if it is "sufficiently good" or write a new one.
  2. Quality is a weighted score from four LLM judges (Eq. 1). Experiment A forces selection-preferring models to generate three jokes and compares the mean with their selection and with the best candidate; Experiment B compares creation-leaning models' jokes with the candidate mean and best candidate.
  3. Selection rates span 33.7% to 99.8%, selectors pick the top-scoring candidate 47% to 62% of the time, and no model's generations beat the best candidate on average.
Critical strengths
  1. The three-way split (recognize, generate, decide when to generate) and the three references (own pick, pool mean, pool max) separate "beat your own choice" from "beat the pool" cleanly (Sec. 3).
  2. Scale: 43k decisions over 11 models and three languages, with rankings stable across languages.
  3. Paired bootstrap intervals over headlines and a shared-panel re-scoring (Sec. 5.4).
Critical weaknesses
  1. The objective is undefined. "Sufficiently good" has no threshold (Sec. 3), and Sec. 6 concedes that a forced generation beating the selection does not show the model should have created. Selecting "good enough", maximizing quality, and predicting whether generation would help are three different objectives; the selection rate is a property of the prompt as much as the model.
  2. The generation condition is not isolated. Forced generation sees the four candidates and is asked to explain how its joke differs (Prompt 3), so it can recombine them, and it is sampled at temperature 0.5 vs 0.85 for the candidates (Sec. 4). No overlap analysis is reported. A headline-only arm is needed.
  3. The judging criterion is asserted. The weights in Eq. 1 are author-chosen and never varied; Sec. 5.4 varies only the panel and already flips several near-zero signs. No inter-judge agreement and no human evaluation, although the source task (SemEval-2026 Task 1) has human pairwise judgments. Sec. 6 itself says human judgments are needed.
  4. The comparisons are not budget-matched. A mean of three (Exp. A) or a single joke (Exp. B) is compared with the max of four candidates (Eq. 2b). A max of four wins on judge noise alone, so "no model beats the best candidate" is expected even at equal quality. The cost of producing the four candidates is not counted.
  5. Select vs. create is never tested on the same cases. Experiments A and B use disjoint models and decision-conditioned instances (Sec. 4), so preference type is confounded with model capability. The underlying phenomenon is the generator-validator gap [1], not cited.
Questions for authors
  1. Leave-one-out null: for each headline, how far does one candidate fall below the max of the other three? And best-of-3 generations vs best-of-4 candidates.
  2. For one or two models, always-select, always-generate, and own decision on the same headlines under a stated objective and equal budget.
  3. A headline-only generation arm for Experiment A, with overlap between generations and candidates. Inter-judge agreement or human judgments on a subset, and sensitivity of Tables 1 to 4 to the weights.
  4. Qwen3-Max, DeepSeek-V3.2 and Kimi-K2.5 are both candidate authors and selectors. How often does each pick its own joke, against the 25% base rate?
7 comments
AuthorsOpenReview ↗

Rebuttal - Part I

Dear Reviewer,

We are encouraged by your assessment that the three-way split and the three references separate “beat your own choice” from “beat the pool” cleanly, and we thank you for your careful reading and concrete comments. Prompted by these comments, we have added several analyses and two new experiments, most of them the checks you suggest in your questions, and the paper's conclusions hold under all of them. We hope the point-by-point responses below answer your questions.

1. Task objective (W1)

We appreciate your question about the task objective. As we understand it, your concern is that the paper does not state the objective of the select-or-create decision, i.e., the standard by which a decision counts as appropriate. The objective of our task is explicit: to make effective use of the available candidates. If a candidate is sufficiently good, the model adopts the best one; only if none is does it create a new joke that improves on them. The instruction states this objective in two steps. In the first step, the model acts as evaluator: given a headline and four candidate jokes, it judges whether any candidate is sufficiently good, i.e., funny, creative and relevant to the headline. In the second step, it acts on this judgment and either selects the best candidate (“select the best one”) or, as creator, writes a better joke (“create a better joke”). The paper judges the model's output by this objective: whether selection hits the highest-scoring candidate, and whether creation exceeds the candidates. “Sufficiently good” is the criterion for the first step and determines which action the model takes. It is not itself the objective. It has no numeric threshold, by design: this judgment and the action that follows are exactly the behavior we study, and fixing a threshold in advance would make the judgment for the model.

The sentence you cite from Section 6 follows from this objective. In the first step, the instruction asks the model to judge whether the candidates in front of it are sufficiently good, not to compare them with a joke it has not yet written (Section 6: “does not require a prediction about the quality of a later generation”). When a candidate is sufficiently good, adopting the best one is therefore the action the instruction asks for, and a forced generation that scores above the selected candidate does not show that the model should have created. Accordingly, the paper evaluates the outputs of the two actions at the level of model–language groups (Experiments A and B) and does not label individual decisions right or wrong. None of the three objectives you list is, on its own, the objective of our task. In “selecting good enough”, “good enough” is the criterion for the first step, and when a candidate meets it, the instruction asks the model to select the best one. “Maximizing quality” applies in our task only conditionally: since the objective is to make effective use of the available candidates, the model adopts the best candidate when one is sufficiently good, and the task does not ask it to weigh this against creating a new joke and take whichever would yield the higher quality. “Predicting whether generation would help” is not part of the task, and the paper does not study this predictive ability of models. As for the selection rate, all 11 models receive the same instruction, candidates and sampling settings, so the differences in selection rate observed in our experiments (33.7%–99.8%) are differences between models, i.e., their preferences between selection and creation.

2. Generation condition (W2; Q3: headline-only arm, overlap)

Thank you for the close examination of the generation conditions. We first clarify the task setup: in our task, creation is by design performed after the model has seen the candidates. The instruction asks the model to write a joke better than the candidates and does not prohibit borrowing from or recombining them. Seeing the candidates is therefore inherent to the task, and what needs to be confirmed is whether the model delivers a new joke. To this end, following your suggestion, we ran an overlap analysis between the generated jokes and the candidates. The results show that the newly generated jokes are not simple recombinations of the candidates: they are as close to the candidates as jokes written independently by different models for the same headline are to each other.

Specifically, for all 7,200 forced generations in Experiment A, we computed the character-level ROUGE-L F1 between each generation and the four candidates and took the closest candidate.

AuthorsOpenReview ↗

Rebuttal - Part II

2. Generation condition (W2; Q3: headline-only arm, overlap), continued

As a control, for each candidate we also computed its overlap with the closest of the other three candidates for the same headline. These candidates were written independently by different models. As the table below shows, the two rows are nearly identical, and no forced generation exactly duplicates a candidate.

Mean ROUGE-L F1EnglishSpanishChinese
Forced generation vs. closest candidate0.5010.5140.297
Candidate vs. closest other candidate in the same pool0.4860.5040.289

A headline-only generation arm measures a different quantity. On the overall design (W2): the paper studies whether a model adopts an existing candidate or creates a new joke when candidates are already available. Both actions therefore take place after the model has seen the candidates, and the instruction asks for a joke better than the candidates. Writing from the headline alone is neither of the two actions in this task, and it measures how well the model writes from scratch. Specifically for Experiment A (Q3): Experiment A tests whether, on headlines where the model selected, creating a new joke instead can improve on the candidate it selected. The forced generation therefore keeps the same headline and the same candidates as the initial decision and changes only the action, so that its difference from the selected candidate can be attributed to the action. With a headline-only arm, both the action and the input would change, and the difference could no longer be read this way.

On sampling temperature: 0.85 is used only to build the candidate pool and 0.5 for all target-model behavior, so the two actions compared in Experiment A are taken at the same temperature. When the dataset was built, each of the four candidate generators wrote one joke per headline at 0.85, which fixed the candidate pool. The pool is the existing content given in the task and is the same input for both actions. The target models' initial decisions and the forced generation in Experiment A all use 0.5. This follows the principle of controlling variables: in Experiment A, the same model adopts a candidate or creates a new joke on the same candidates at the same temperature.

AuthorsOpenReview ↗

Rebuttal - Part III

3. Robustness of the evaluation criteria (W3; Q3: agreement, weights)

We thank you for pointing out that, beyond the evaluator composition tested in Section 5.4, the weights and inter-evaluator agreement also need to be checked. We have now completed both as requested in Q3, and added a pairwise comparison experiment that involves no scoring. These checks show that the paper's conclusions do not depend on our chosen weights, this set of evaluators, or scoring as the evaluation method: across five weightings, two evaluator compositions and two evaluation methods, no model's creations exceed the highest-scoring candidate.

Weights: the conclusions are unchanged under five weightings. We set the weights in Eq. 1 with humor as the focus of the task. To test whether the conclusions depend on this choice, we kept the evaluators' original scores, changed only the weights, and recomputed all comparisons in Tables 1, 3 and 4 of the paper, re-identifying the highest-scoring candidate for each headline under the new weighted score. Under all five weightings, creation scores below the highest-scoring candidate in all 24 model–language groups, with every 95% CI below zero. This holds even when three dimensions are dropped and only humor is scored. The top-score hit rate and the candidate selection gap are also essentially unchanged (43.0%–68.5% and 0.296–0.793 points across the five weightings, 44.0%–68.0% and 0.313–0.736 points under the original setting), indicating low sensitivity to the weights. The table below gives the comparisons with the highest-scoring candidate. Each cell is the range of mean score differences across model–language groups. Weights are listed as news relevance / humor / creativity / naturalness and fluency. Negative values mean the generated joke scores below the highest-scoring candidate.

WeightsExperiment A: generation − highest-scoring candidateExperiment B: creation − highest-scoring candidate
Original 0.10 / 0.40 / 0.30 / 0.20−1.152 to −0.364−1.566 to −0.419
Equal weights 0.25 / 0.25 / 0.25 / 0.25−1.161 to −0.411−1.512 to −0.425
Higher humor 0.10 / 0.50 / 0.20 / 0.20−1.198 to −0.396−1.623 to −0.448
Higher creativity 0.10 / 0.30 / 0.40 / 0.20−1.108 to −0.334−1.510 to −0.392
Humor only 0 / 1 / 0 / 0−1.314 to −0.435−1.766 to −0.493

Only comparisons that were already near zero change sign. No main conclusion changes sign. You note that Section 5.4 already shows near-zero sign reversals when the evaluator panel is changed. Under different weights, only two such comparisons change sign: the difference from the selected candidate for GPT-5.4 in English (+0.057 in the original setting) and the difference from the candidate mean for Qwen3.6-27B in Spanish (−0.002). Their CIs include zero both before and after the change. The three comparisons that change sign in Section 5.4 likewise have CIs that include zero after the change. No comparison changes from significantly negative to significantly positive, or the reverse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.