- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark for plot twists by testing "transformational creativity". It evaluates 71 LLMs on a set of self-designed metrics – diversity, surprise, coherence, and realism – by using three LLM judges, and compare with a curated set of 18 published human short stories. The paper finds that human stories outperform all LLM stories overall, narrowly ahead of the strongest model. The authors also identify two model failure modes: repeating similar twist designs and breaking real-world rules. For the ablation study, the authors compared two prompting strategies and different reasoning effects. They also report that the models always plot choose the twist first, and then complete the story.
Critical strengths
- The task design is interesting and specific, focusing on plot twists and designing an operational evaluation framework.
- The study presents a broad evaluation of 71 models.
- The qualitative analysis of the failure modes are helpful to pinpoint model weaknesses and enlighten model improvement. Analyzing model reasoning processes also makes the paper more complete.
Critical weaknesses
- The realism metric is not very convincing. In many cases, e.g., science fictions, breaking the real-world rules is precisely where creativity emerges. And the fact that realism serves as a preceding constraint in this paper makes the whole evaluation more problematic: if the paper only counts model stories that achieve full scores on realism, then how many stories are left? Are the results and discussion only based on full-realism stories?
- The human score and the best model score are very close (+2.00 vs. +1.9), but the authors did not report whether that gap is significantly different.
These are the most important issues that the authors need to address.
Questions for authors
- The datasets seem imbalanced: not each of the 71 models writes 30 stories, because that would be 2130 in total instead of 2035 (Section 3.2). Does the 2035 stories include human stories? Also, the human sample only contains 18 stories, so I doubt if the human scores are statistically reliable.
- As for the ablation study, it would be good to show the specific scores on all metrics for the prompting strategies. For example, by adding the prompt "Be creative", perhaps the model would break the physical rules even more and lead to a lower realism score?
- Mode collapse is reported in the paper, but it would be interesting to see if the human data also show such a tendency if the human sample is big enough.
- Although the abstract states that transformational creativity is understudied compared to the other two creativity types, the Background and Related Work section does not show the gap exists and why the authors choose transformational creativity as the focus of this study.