acceptodds
Under review as a conference paper at ICLR 2027

TwistBench: Benchmarking Transformational Creativity in LLMs via Literary Plot Twists

Abstract

Benchmarks and tests of creativity for Large Language Models (LLMs) are now abundant—it has been shown that LLMs can realize aspects of combinatorial creativity, exploratory creativity, and that they exceed human performance on psychometric tests of creativity. However, these evaluations have failed to address more complex types of creativity recognized in cognitive science, such as *transformational creativity*. Widely held to underpin major scientific, artistic, and technological breakthroughs, transformational creativity describes the ability to *radically modify an existing conceptual space*, such as a scientific paradigm, a musical genre, or, as we show, the plot of a short story. To resolve this gap in the benchmarking landscape, we introduce **TwistBench**, a benchmark of literary plot twists for assessing transformational creativity. We collect a set of stories with plot twists written by famous human authors and compare them against stories written by 71 LLMs, across scoring dimensions *diversity, surprise, coherence*, and *realism*. Our main findings are that: (1) Even the latest frontier LLMs underperform expert humans in transformational creativity, though elaborate prompting strategies like in-context regeneration can raise the performance of select models to match expert humans. (2) Two dominant failure modes plague LLMs when applied to open-ended, transformative tasks: (a) *mode collapse*, where frontier models are capable of generating high quality twists, but these twists are largely the same and lack diversity; and (b) *breaking the world model,* where models give surprising and coherent plot twists, but do so in a way that measurably departs from physical reality (e.g., time travel, ghosts), violating literary "fair play." (3) An analysis of the creative processes employed by reasoning models reveals process-level homogeneity among LLMs. Namely, models always choose the plot twist *first*, then retroactively determine the plot conditioned on the twist. In summary, while today's LLMs struggle to surpass the transformational creativity of expert humans, we introduce a task and framework to assess progress towards this ability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
creativity, creative writing, benchmark, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3gVbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark for plot twists by testing "transformational creativity". It evaluates 71 LLMs on a set of self-designed metrics – diversity, surprise, coherence, and realism – by using three LLM judges, and compare with a curated set of 18 published human short stories. The paper finds that human stories outperform all LLM stories overall, narrowly ahead of the strongest model. The authors also identify two model failure modes: repeating similar twist designs and breaking real-world rules. For the ablation study, the authors compared two prompting strategies and different reasoning effects. They also report that the models always plot choose the twist first, and then complete the story.

Critical strengths
  1. The task design is interesting and specific, focusing on plot twists and designing an operational evaluation framework.
  2. The study presents a broad evaluation of 71 models.
  3. The qualitative analysis of the failure modes are helpful to pinpoint model weaknesses and enlighten model improvement. Analyzing model reasoning processes also makes the paper more complete.
Critical weaknesses
  1. The realism metric is not very convincing. In many cases, e.g., science fictions, breaking the real-world rules is precisely where creativity emerges. And the fact that realism serves as a preceding constraint in this paper makes the whole evaluation more problematic: if the paper only counts model stories that achieve full scores on realism, then how many stories are left? Are the results and discussion only based on full-realism stories?
  2. The human score and the best model score are very close (+2.00 vs. +1.9), but the authors did not report whether that gap is significantly different.

These are the most important issues that the authors need to address.

Questions for authors
  1. The datasets seem imbalanced: not each of the 71 models writes 30 stories, because that would be 2130 in total instead of 2035 (Section 3.2). Does the 2035 stories include human stories? Also, the human sample only contains 18 stories, so I doubt if the human scores are statistically reliable.
  2. As for the ablation study, it would be good to show the specific scores on all metrics for the prompting strategies. For example, by adding the prompt "Be creative", perhaps the model would break the physical rules even more and lead to a lower realism score?
  3. Mode collapse is reported in the paper, but it would be interesting to see if the human data also show such a tendency if the human sample is big enough.
  4. Although the abstract states that transformational creativity is understudied compared to the other two creativity types, the Background and Related Work section does not show the gap exists and why the authors choose transformational creativity as the focus of this study.
Reviewer_gh9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

TwistBench proposes literary plot twists as a testbed for Boden's transformational creativity. 71 LLMs are given a single prompt to write a 2,000–3,000-word story with a plot twist (30 samples across three temperatures).

  • Scoring. Each story is scored on 1–5 surprise, coherence and realism by a three-judge LLM ensemble (Claude Sonnet 4, GPT-4o, Gemini 2.5 Flash; median). Surprise and coherence count only if realism = 5. Diversity is the mean pairwise cosine distance of gpt-4o-mini twist summaries. The overall score averages z-scores of gated surprise, gated coherence and diversity.
  • Human baseline. 18 public-domain twist stories by famous authors (O. Henry, Maupassant, Bierce, etc.).
  • Findings. Humans rank first; frontier models fail via mode collapse or "breaking the world model" (unrealistic twists). In-context regeneration lifts one Claude model to human level. Reasoning traces show a uniform "choose twist first, then build the plot" process.
Critical strengths
  1. Timely question, broad coverage, useful qualitative findings. Moving creativity evaluation beyond divergent-thinking tests is worthwhile. Evaluating 71 models from 19 providers is broad. The twist-cluster analysis (Fig. 4, e.g., 10/30 Opus-4.5 stories collapsing onto "dead-spouse grief-denial") and the two failure modes are concrete and likely useful to the community even if the headline construct is contested.
Critical weaknesses
  1. Construct validity: what is measured is neither clearly "transformational creativity" nor what models were asked to do.

    • In Boden's sense, transformational creativity changes the generative rules of a domain. Writing a story with a twist ending is a well-established genre convention, arguably exploratory creativity within the "twist story" space. The model authors both the conceptual space and its reversal, which is standard narrative craft.
    • The overall score is dominated by the realism gate: surprise and coherence count only when Real = 5. Yet the task prompt (Fig. 3) never asks for realism or "fair play."
    • Models are thus penalized for an unstated constraint. Claude-opus-4.8 has the highest surprise (4.73) and near-perfect coherence (4.98) but ranks 52nd because of realism 2.10, which rewards genre conservatism rather than creativity.
    • Similarly, "choose the twist first, then plan backwards" (Sec. 5) is standard advice to human twist writers. Presenting it as a homogeneity failure is unsupported without comparable human process data.
  2. Evaluation validity and the human comparison are too weak to support the main claim ("humans outperform all 71 LLMs").

    • Judges are unvalidated against human readers (acknowledged).
    • A Claude judge is used while Claude models dominate the top ranks, and family-level self-preference is not tested.
    • The judges likely recognize the 18 canonical human stories.
    • The human set is small (n=18). Its lengths range 906–21,718 words despite the claim that the 2–3k constraint controls length. Human diversity pools 18 authors versus one model.
    • No confidence intervals or significance tests are reported. The human vs. claude-sonnet-4.5 gap (+2.00 vs +1.90) is very likely within noise.
    • The text is internally inconsistent: the contribution list says in-context regeneration lifted Claude Opus 4.5 to human level, while Sec. 4.2/Fig. 6 say Sonnet-4.5. Table 3 lists 72 populations. Fig. 2 states no model reaches humans.
Questions for authors

Clarifications (could change my assessment):

  1. Why is realism enforced as a hard gate when the task prompt never asks for a realistic twist? How do rankings change if the prompt states the fair-play constraint, or if realism is reported separately rather than gating?
  2. Have the judge scores been validated against blind human ratings, even on a subset (including the human stories)? Do Claude-family models score higher under the Claude judge than under the other two?
  3. What are bootstrap CIs on the overall scores, particularly humans vs. the top-5 models? How sensitive are human scores to story length and to judges recognizing the canonical stories?
  4. Which model does in-context regeneration lift to human level (Opus 4.5 or Sonnet 4.5)? Is the gain driven only by the diversity term, which the procedure directly optimizes?

Suggestions: 5. Collect contemporary human stories under the identical prompt and length constraint, and report reasoning-process data for human writers before framing "twist-first" planning as an LLM limitation.

Reviewer_nXsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TwistBench: a simple prompt asks a model to write short stories with a plot twist, followed by LLM-as-a-judge evaluation that finds that current LLMs underperform famous authors at writing plot twists. The authors also analyze whether more advanced prompting or longer reasoning helps, and do analysis of the chain of thought of the studied models.

Critical strengths

Studying the creative capabilities of AI systems is important, and the use of literary plot twists could definitely be a useful tool for this.

Critical weaknesses
  1. The methodology proposed is extremely thin, making the contribution minimal. All the choices are essentially the first thing one would think of. For a paper that studies plot twists, this is rather ironic: (1) the paper simply prompts LLMs to generate stories with the simplest prompt one could think of, (2) the evaluation is LLM-as-a-judge and none of it is validated whatsoever with human annotation.
  2. The results essentially contradict the points argued by the paper: the paper argues there's a significant gap between expert writers ability to propose creative plot twists and LLM-based prompting, yet one of the systems (Claude 4.5 Sonnet) essentially gets very close to humans (1.9 vs. 2.0), clearly indicating (to me) that either the evaluation methodology is unsound, or the interpretation of the metrics is poorly explained. Is the 1.9 to 2.0 gap significant?
  3. This paper's setting and evaluation protocol are fully automated (LLM experiments judged by LLMs). It seems like if the premise of the paper that LLMs have limited capabilities when it comes to creativity, that the work would require human annotation and data curation. It is unclear how this is reconciled.
Questions for authors
  1. Do you believe your use of LLM-as-a-judge for evaluation of plot-twist quality along the metrics you propose to be ecologically valid? How do you reconcile it with the fact that the work argues that such models are not capable of generating high quality tasks?
  2. You argue that the very strong correlation between LLM judges is a good sign ("high reliability"). Another interpretation is that these models do not meaningfully diverge since they are trained on similar corpora, and that using 1 judge rather than 3 gets the same signal. Do you think using 3 judges is significantly better than using 1, when they are so correlated?
  3. More importantly, how does LLM-as-a-judge compare to expert opinion of plot twist quality (say, a professional writer with experience teaching creative writing). Isn't such a validation necessary to judge the quality of the evaluation protocol you propose?
  4. The embedding-based metric seems equally unevaluated. Can you validate why cosine similarity with a generic embedding model (typically used for retrieval tasks) is adequate to evaluate diversity of literary plot twists. It seems (to me) like it would not, by default.
  5. A rather large missed opportunity (imo): you have prefix stories of high quality, and know the point in which an expert author introduced a plot twist. Why not cut the story then, and ask an LLM for a continuation, and judge LLMs vs. authors on a common set of prefix stories. It feels like the comparison would be much more sound, as if the underlying stories are different, the judgement of twist quality seems like comparing apples-vs.-oranges. Of course, these famous stories are in pretraining corpora, so in order to perform this experiment, you'd need to collect high-quality stories with plot twists that aren't readily published online. Do you think that such a comparison (sharing the story prefix where the task is solely to come up with the plot twist) would be better/worse, and in what ways?
  6. According to your experimental results, Claude 4.5 Sonnet is epsilon below the best human authors at writing creative plot twists (setting aside the diversity issue). Looking at the trend from Figure 2 (which is never discussed anywhere in the paper), can we imagine then that the latest generation of models have likely surpassed the "expert human ceiling"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.