- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ScreenWritingBench, a Chinese-language benchmark comprising 279 tasks constructed and validated by professional screenwriters to evaluate LLMs across the end-to-end screenwriting workflow: proposals, story synopses, character profiles/biographies, scene breakdowns, and scripts, covering both initial generation and revision; 219 tasks include multi-turn dialogue history.
The authors propose a three-layer, five-dimensional LLM-as-a-judge evaluation framework: (1) Task Layer: Task Completion (TC) and Contextual Consistency (CC) using query-aware rubrics. (2) Norm Layer: Artifact Compliance (AC) and Basic Lexical Defects (BLD) using category-aware rubrics. (3) Quality Layer: Dramatic Quality (DQ), which ranks a model's response relative to eight expert-ordered anchor responses per task. The authors also demonstrate an anchor-extension protocol using an additional extension: a 9th anchor, A9, to prevent ceiling effects.
They evaluate 18 recent LLMs and analyze cross-dimension trade-offs.
The paper tackles an important, practical gap in creative writing evaluation: moving beyond single-turn story generation or script continuation to evaluate multi-turn, multi-artifact revision and planning workflows. Decomposing evaluation is well-motivated, and the empirical trade-offs showing that surface format compliance in AC and defect-free text in BLD are orthogonal to Dramatic Quality are insightful. However, the paper suffers from severe methodological and reproducibility issues regarding judge-model circularity, weak human-judge agreement on its core novel dimensions, unexplained evaluation pipelines ("Segment Parsing" in AC), and an unacknowledged restriction to Chinese, which is only mentioned in the Appendix.
Critical strengths
-
Unlike existing benchmarks that only evaluate script continuation (DramaBench) or summary-to-script generation (CML-Bench), ScreenWritingBench captures 10 task dimensions and 172 tags across planning, character design, scene outlines, and multi-turn revision.
-
Distinguishing query-aware constraint checking (TC, CC), format/surface compliance (AC, BLD), and reference-anchored listwise ranking (DQ) prevents LLM judges from conflating formatting neatness with creative merit.
-
The evaluation of 18 models clearly validates the multi-dimensional design
-
Appendices C and D convincingly show that 8 anchors prevent the score compression seen in pointwise/single-anchor scoring, and adding a synthetic-then-verified 9th anchor (A9) shifts absolute scores down by 4.2 points while preserving 100% of the relative model rankings.
Critical weaknesses
-
In Section 5.1, the authors use DeepSeek-V4-pro as the judge for Dramatic Quality (and DeepSeek-V4-flash for TC/CC, Qwen3.7-plus for AC/BLD), while simultaneously evaluating DeepSeek-V4-pro and four Qwen models on the leaderboard. Furthermore, in Appendix D, Claude-Opus-4.6 and DeepSeek-V4-pro are used to select A9. There is no analysis of self-preference bias or family bias.
-
Table 2 (Section 5.5) reports human-evaluator agreement of only 58.33% for Contextual Consistency (CC) and 58.89% for Dramatic Quality (DQ), and 64% Overall. Moreover, the paper never defines what metric "Agreement" represents (e.g., exact match, pairwise accuracy, Cohen's kappa, or Spearman's rank correlation), how many items/experts were involved, or what the inter-annotator agreement (IAA) among human experts is. If human-LLM agreement on DQ is ~59% (barely above 50% when using pairwise comparisons), the reliability of the DQ leaderboard is questionable.
-
It is never stated that ScreenWritingBench is a Chinese-language benchmark. Readers only discover this incidentally in Figure 2, Appendix A.4, and Appendix E. This also explains why Chinese-centric models (Qwen, GLM, Kimi) perform so strongly on CC and AC relative to Gemini-3.1-pro.
-
Figure 3 depicts a "Segment Parsing and Type Identification" step and a "Line-by-Line Judge" for AC, and Section 4.1.2 mentions "violation levels from 0 to 3" and "crossing the red line" (Dstr, Dcnt), but the exact deduction formula and parser details are completely omitted from both the main text and Appendix A.3. Similarly, for DQ, it is unclear whether the LLM judge ranks R against all 8 anchors in a single mega-prompt or via pairwise binary search.
-
BLD is scored as a strict binary 100 or 0 across an entire response, meaning a single typo or punctuation error (B4) in a 3,000-word screenplay drops the BLD score from 100 to 0, instantly costing 20 points on the overall score. Equal weighting (0.2 each, Eq. 1) gives surface typo detection (BLD) the exact same weight as TC and DQ.
Questions for authors
-
What is the exact mathematical definition of the "Agreement" percentage in Table 2? How many tasks and human experts were used to compute it, and what is the inter-annotator agreement between human screenwriters on CC and DQ?
-
How is the 8-anchor comparison executed in practice? Are all 8 anchors plus the dialogue history and candidate response fed into a single prompt (and if so, how do you control for positional bias and context length), or is it done via pairwise comparisons?
-
Did you check whether DeepSeek-V4-pro (as DQ judge) or Qwen3.7-plus (as AC/BLD judge) systematically inflated scores for the DeepSeek and Qwen model families?
-
Can you provide the exact mathematical mapping from violation levels (0-3) across the 10-20 rules per artifact to the deduction scores Dstr and Dcnt?
-
How were the initial 8 anchors per task generated? If generated by specific LLMs, does that bias DQ toward those model families?
5 comments
Q2.
How is the 8-anchor comparison executed in practice? Are all 8 anchors plus the dialogue history and candidate response fed into a single prompt (and if so, how do you control for positional bias and context length), or is it done via pairwise comparisons?
Response: DQ evaluation uses a single joint comparison rather than a sequence of separate pairwise comparisons. The judge receives one prompt containing, in order, the dialogue history, the current instruction, the eight expert-ordered anchor responses, and the response being evaluated. For a given task, every evaluated model is assessed using the same anchor order, with its response always placed last. This standardizes candidate placement across models, avoiding differences in candidate presentation position. In the current evaluation, the context, eight anchors, and candidate response together do not exceed 200,000 tokens. No truncation was used.
Q3.
Did you check whether DeepSeek-V4-pro (as DQ judge) or Qwen3.7-plus (as AC/BLD judge) systematically inflated scores for the DeepSeek and Qwen model families?
Response: Our evaluation is primarily rubric-based, including AC and BLD, with explicit criteria intended to reduce the scope for model-family preferences. For DQ, the reported results show no obvious indication of model-family preference. T
Q1.
Response: We sampled 143 response pairs to evaluate human–model agreement, with each pair consisting of two model responses to the same task. Human–model agreement is calculated as follows:
Tasks requiring contextual interpretation or subjective judgment can elicit disagreement not only between humans and the evaluator, but also among experts. Human–human agreement is 67% for CC and 63% for DQ, while the corresponding human–model agreement rates are 58.33% and 58.89%; Overall human–model agreement is 64%. These results provide an inter-expert reference for interpreting disagreement on judgment-intensive dimensions such as CC and DQ.