acceptodds
Under review as a conference paper at ICLR 2027

ScreenWritingBench: Evaluating LLMs Across the Screenwriting Workflow with Dimension-Specific Evaluation

Abstract

Professional screenwriting develops through planning, drafting, and revision across several artifact types. Existing writing benchmarks sample only parts of this workflow, making it difficult to assess whether large language models (LLMs) can support the successive tasks of screenwriting. Dramatic quality also lacks an objective ground truth and calls for a different form of judgment from task completion or artifact compliance. To address these gaps, we introduce ScreenWritingBench, a collection of 279 tasks constructed and validated by professional screenwriters. The tasks span artifacts from proposals to scripts and operations from initial generation to revision; 219 include dialogue history. We evaluate responses along five dimensions: Task Completion, Contextual Consistency, Artifact Compliance, Basic Lexical Defects, and Dramatic Quality. The first four use rubrics, whereas Dramatic Quality places each response relative to eight expert-ordered anchors. An evaluation of 18 models reveals substantial variation across dimensions. We report both an overall leaderboard and dimension-level scores to characterize these differences.

Primary area:
datasets and benchmarks
Keywords:
Artificial intelligence.Large language models.Evaluation.Benchmark.Screenwriting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9sHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ScreenWritingBench, a Chinese-language benchmark comprising 279 tasks constructed and validated by professional screenwriters to evaluate LLMs across the end-to-end screenwriting workflow: proposals, story synopses, character profiles/biographies, scene breakdowns, and scripts, covering both initial generation and revision; 219 tasks include multi-turn dialogue history.

The authors propose a three-layer, five-dimensional LLM-as-a-judge evaluation framework: (1) Task Layer: Task Completion (TC) and Contextual Consistency (CC) using query-aware rubrics. (2) Norm Layer: Artifact Compliance (AC) and Basic Lexical Defects (BLD) using category-aware rubrics. (3) Quality Layer: Dramatic Quality (DQ), which ranks a model's response relative to eight expert-ordered anchor responses per task. The authors also demonstrate an anchor-extension protocol using an additional extension: a 9th anchor, A9, to prevent ceiling effects.

They evaluate 18 recent LLMs and analyze cross-dimension trade-offs.

The paper tackles an important, practical gap in creative writing evaluation: moving beyond single-turn story generation or script continuation to evaluate multi-turn, multi-artifact revision and planning workflows. Decomposing evaluation is well-motivated, and the empirical trade-offs showing that surface format compliance in AC and defect-free text in BLD are orthogonal to Dramatic Quality are insightful. However, the paper suffers from severe methodological and reproducibility issues regarding judge-model circularity, weak human-judge agreement on its core novel dimensions, unexplained evaluation pipelines ("Segment Parsing" in AC), and an unacknowledged restriction to Chinese, which is only mentioned in the Appendix.

Critical strengths
  1. Unlike existing benchmarks that only evaluate script continuation (DramaBench) or summary-to-script generation (CML-Bench), ScreenWritingBench captures 10 task dimensions and 172 tags across planning, character design, scene outlines, and multi-turn revision.

  2. Distinguishing query-aware constraint checking (TC, CC), format/surface compliance (AC, BLD), and reference-anchored listwise ranking (DQ) prevents LLM judges from conflating formatting neatness with creative merit.

  3. The evaluation of 18 models clearly validates the multi-dimensional design

  4. Appendices C and D convincingly show that 8 anchors prevent the score compression seen in pointwise/single-anchor scoring, and adding a synthetic-then-verified 9th anchor (A9) shifts absolute scores down by 4.2 points while preserving 100% of the relative model rankings.

Critical weaknesses
  1. In Section 5.1, the authors use DeepSeek-V4-pro as the judge for Dramatic Quality (and DeepSeek-V4-flash for TC/CC, Qwen3.7-plus for AC/BLD), while simultaneously evaluating DeepSeek-V4-pro and four Qwen models on the leaderboard. Furthermore, in Appendix D, Claude-Opus-4.6 and DeepSeek-V4-pro are used to select A9. There is no analysis of self-preference bias or family bias.

  2. Table 2 (Section 5.5) reports human-evaluator agreement of only 58.33% for Contextual Consistency (CC) and 58.89% for Dramatic Quality (DQ), and 64% Overall. Moreover, the paper never defines what metric "Agreement" represents (e.g., exact match, pairwise accuracy, Cohen's kappa, or Spearman's rank correlation), how many items/experts were involved, or what the inter-annotator agreement (IAA) among human experts is. If human-LLM agreement on DQ is ~59% (barely above 50% when using pairwise comparisons), the reliability of the DQ leaderboard is questionable.

  3. It is never stated that ScreenWritingBench is a Chinese-language benchmark. Readers only discover this incidentally in Figure 2, Appendix A.4, and Appendix E. This also explains why Chinese-centric models (Qwen, GLM, Kimi) perform so strongly on CC and AC relative to Gemini-3.1-pro.

  4. Figure 3 depicts a "Segment Parsing and Type Identification" step and a "Line-by-Line Judge" for AC, and Section 4.1.2 mentions "violation levels from 0 to 3" and "crossing the red line" (Dstr, Dcnt), but the exact deduction formula and parser details are completely omitted from both the main text and Appendix A.3. Similarly, for DQ, it is unclear whether the LLM judge ranks R against all 8 anchors in a single mega-prompt or via pairwise binary search.

  5. BLD is scored as a strict binary 100 or 0 across an entire response, meaning a single typo or punctuation error (B4) in a 3,000-word screenplay drops the BLD score from 100 to 0, instantly costing 20 points on the overall score. Equal weighting (0.2 each, Eq. 1) gives surface typo detection (BLD) the exact same weight as TC and DQ.

Questions for authors
  1. What is the exact mathematical definition of the "Agreement" percentage in Table 2? How many tasks and human experts were used to compute it, and what is the inter-annotator agreement between human screenwriters on CC and DQ?

  2. How is the 8-anchor comparison executed in practice? Are all 8 anchors plus the dialogue history and candidate response fed into a single prompt (and if so, how do you control for positional bias and context length), or is it done via pairwise comparisons?

  3. Did you check whether DeepSeek-V4-pro (as DQ judge) or Qwen3.7-plus (as AC/BLD judge) systematically inflated scores for the DeepSeek and Qwen model families?

  4. Can you provide the exact mathematical mapping from violation levels (0-3) across the 10-20 rules per artifact to the deduction scores Dstr and Dcnt?

  5. How were the initial 8 anchors per task generated? If generated by specific LLMs, does that bias DQ toward those model families?

5 comments

Q1.

What is the exact mathematical definition of the "Agreement" percentage in Table 2? How many tasks and human experts were used to compute it, and what is the inter-annotator agreement between human screenwriters on CC and DQ?

Response: We sampled 143 response pairs to evaluate human–model agreement, with each pair consisting of two model responses to the same task. Human–model agreement is calculated as follows:

DimensionAssessmentAgreement calculation
Task Completion (TC)Two screenwriters judge whether each response completes the task.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Contextual Consistency (CC)Two screenwriters judge whether each response contains contextual consistency problems.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Artifact Compliance (AC)Humans and the model independently identify problematic points under the artifact-compliance rules.F1 of model-identified problem points against human annotations as ground truth.
Basic Lexical Defects (BLD)Humans and the model independently identify basic-text defect points.F1 of model-identified problem points against human annotations as ground truth.
Dramatic Quality (DQ)Compare the screenwriters' preference between two responses with their anchor-based rank order.Comparisons with matching orders / all evaluated comparisons. Equal anchor-based ranks count as disagreement.
OverallCompare the screenwriters' holistic preference with the aggregate-score order.Comparisons with matching orders / all evaluated comparisons. This is not an average of dimension-level agreement rates; human preference is not computed with the benchmark's aggregation formula.

Tasks requiring contextual interpretation or subjective judgment can elicit disagreement not only between humans and the evaluator, but also among experts. Human–human agreement is 67% for CC and 63% for DQ, while the corresponding human–model agreement rates are 58.33% and 58.89%; Overall human–model agreement is 64%. These results provide an inter-expert reference for interpreting disagreement on judgment-intensive dimensions such as CC and DQ.

Q2.

How is the 8-anchor comparison executed in practice? Are all 8 anchors plus the dialogue history and candidate response fed into a single prompt (and if so, how do you control for positional bias and context length), or is it done via pairwise comparisons?

Response: DQ evaluation uses a single joint comparison rather than a sequence of separate pairwise comparisons. The judge receives one prompt containing, in order, the dialogue history, the current instruction, the eight expert-ordered anchor responses, and the response being evaluated. For a given task, every evaluated model is assessed using the same anchor order, with its response always placed last. This standardizes candidate placement across models, avoiding differences in candidate presentation position. In the current evaluation, the context, eight anchors, and candidate response together do not exceed 200,000 tokens. No truncation was used.

Q3.

Did you check whether DeepSeek-V4-pro (as DQ judge) or Qwen3.7-plus (as AC/BLD judge) systematically inflated scores for the DeepSeek and Qwen model families?

Response: Our evaluation is primarily rubric-based, including AC and BLD, with explicit criteria intended to reduce the scope for model-family preferences. For DQ, the reported results show no obvious indication of model-family preference. T

Reviewer_FkmjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ScreenWritingBench, a benchmark for evaluating LLMs on professional screenwriting tasks. It covers multiple stages, including story development, character design, scene planning, script generation, and revision. It evaluates five dimensions: Task Completion, Contextual Consistency, Artifact Compliance, Basic Lexical Defects, and Dramatic Quality. The first four use rubric-based evaluation, while Dramatic Quality uses eight expert-ordered anchors. It evaluates 18 LLMs and analyzes their performance across dimensions and task categories.

Critical strengths
  1. Existing creative-writing benchmarks mainly focus on story generation, text continuation, or specific screenwriting tasks, while professional screenwriting involves multiple stages and artifact types. The proposed benchmark covers a broader workflow and involves professional screenwriters in task construction and validation.
  2. The multi-dimensional evaluation framework is reasonably designed and easy to follow.
  3. The experimental evaluation and analyses are relatively comprehensive.
Critical weaknesses
  1. The validity of the Dramatic Quality evaluation remains insufficiently established. Table 2 reports only 58.89% agreement for Dramatic Quality and 64% overall. The no-anchor baselines also use different criteria or scoring scales, making it difficult to isolate the contribution of anchors. Moreover, they do not clearly compare these results with inter-expert agreement.

  2. The evaluation of multi-turn screenwriting remains limited. Although 219 tasks include dialogue history, the maximum is four turns, with a median of two. Because professional screenwriting involves long-term revision and stage-to-stage dependencies, this setting provides limited coverage of long-term consistency and iterative revision.

  3. The scoring rules and dimension weights are not sufficiently justified. All five dimensions have equal weights despite using different scoring mechanisms. For example, Basic Lexical Defects uses binary scoring, while Dramatic Quality uses 12.5-point intervals. These choices may affect the overall rankings, but this study lacks a sensitivity analysis of the scoring rules and weights.

Questions for authors
  1. How do longer multi-turn screenwriting scenarios affect cross-stage revision and long-term consistency evaluation?

  2. How is Agreement in Table 2 calculated? What is the inter-expert agreement? Why is the agreement for Dramatic Quality relatively low?

  3. Why are all five dimensions equally weighted, and why does Basic Lexical Defects use binary scoring? Would different settings significantly affect the leaderboard?

4 comments

Q1.

How do longer multi-turn screenwriting scenarios affect cross-stage revision and long-term consistency evaluation?

Response: ScreenWritingBench targets stage-specific collaboration within the screenwriting workflow, including revisions within a stage and transitions between stages, such as converting a proposal into an outline or an outline into a script. Its 219 history-conditioned tasks assess whether a model can understand user requirements that remain applicable across the supplied history and fulfill the specific writing or revision request at the current stage. Here, longer-term needs refer to requirements established in the history that still apply to the current task, while immediate needs refer to the specific requirements of the current turn. Understanding and satisfying both does not mean that a single model must complete an entire long-running creative project from start to finish. Moreover, dialogue turn count alone does not capture the richness of the context, as requests may also include documents and other writing materials.

Q2.

How is Agreement in Table 2 calculated? What is the inter-expert agreement? Why is the agreement for Dramatic Quality relatively low?

Response: We sampled 143 response pairs to evaluate human–model agreement, with each pair consisting of two model responses to the same task. Human–model agreement is calculated as follows:

DimensionAssessmentAgreement calculation
Task Completion (TC)Two screenwriters judge whether each response completes the task.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Contextual Consistency (CC)Two screenwriters judge whether each response contains contextual consistency problems.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Artifact Compliance (AC)Humans and the model independently identify problematic points under the artifact-compliance rules.F1 of model-identified problem points against human annotations as ground truth.
Basic Lexical Defects (BLD)Humans and the model independently identify basic-text defect points.F1 of model-identified problem points against human annotations as ground truth.
Dramatic Quality (DQ)Compare the screenwriters' preference between two responses with their anchor-based rank order.Comparisons with matching orders / all evaluated comparisons. Equal anchor-based ranks count as disagreement.
OverallCompare the screenwriters' holistic preference with the aggregate-score order.Comparisons with matching orders / all evaluated comparisons. This is not an average of dimension-level agreement rates; human preference is not computed with the benchmark's aggregation formula.

Tasks requiring contextual interpretation or subjective judgment can elicit disagreement not only between humans and the evaluator, but also among experts. Human–human agreement is 67% for CC and 63% for DQ, while the corresponding human–model agreement rates are 58.33% and 58.89%; Overall human–model agreement is 64%. These results provide an inter-expert reference for interpreting disagreement on judgment-intensive dimensions such as CC and DQ.

Q3.

Why are all five dimensions equally weighted, and why does Basic Lexical Defects use binary scoring? Would different settings significantly affect the leaderboard?

Response: The current study uses equal weights for the five evaluation dimensions. A more differentiated weighting framework is a direction for future work. Such a framework would account not only for dimension weights, but also for differences in the importance of benchmark tasks and evaluation rules. Developing this multilevel framework requires systematic design and validation and lies beyond the scope of this paper. BLD measures surface-level textual usability, covering garbled characters, repetition, inappropriate language mixing, typographical or grammatical errors, and truncation. We include this dimension for two reasons. First, some of these defects can directly obstruct reading or use of a response. Second, alongside complex creative abilities, we also assess whether models reliably satisfy basic textual requirements, maintaining readability and linguistic correctness.

We also tested the ranking effect of changing the BLD weight while keeping the recorded per-response dimension scores and valid response sets fixed. We set the BLD weight to 10%, 15%, 25%, or 30%, with each other dimension receiving (1 − w_BLD)/4, so the weights always sum to 100%.

BLD weightWeight of each other dimensionMaximum rank changeSpearman correlation with the original ranking
10%22.5%10.9959
15%21.25%01.0000
25%18.75%20.9876
30%17.5%30.9794

The top-three model set remains unchanged across these settings, although the first-place model switches from Claude-Opus-5 to GPT-5.6-sol at BLD weights of 25% and 30%.

We additionally tested a count-based BLD score, max(0, 100 − 20n), where n is the number of defect instances recorded by the judge. Holding the other four dimension scores, evaluated responses, and equal weights fixed, its ranking has a Spearman correlation of 0.9917 and a maximum change of two positions relative to the binary BLD (0/100) baseline on the same records.

Thank you for the detailed responses and additional analyses. Overall, the rebuttal has addressed several of my concerns. Given the benchmark's useful task coverage and reasonably motivated evaluation framework, I will maintain my original slightly positive rating.

Reviewer_jPfCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ScreenWritingBench, a collection of 279 tasks constructed and validated by professional screenwriters. Each response is scored on five dimensions: Task Completion, Contextual Consistency, Artifact Compliance, Basic Lexical Defects and Dramatic Quality. The first four use rubrics, whereas Dramatic Quality places each response relative to eight expert-ordered anchors. The overall score is the average of the five scores. The authors test 18 models and find substantial variation across dimensions. Agreement between the judges and expert screenwriters is 64% overall.

Critical strengths
  1. Broad coverage of screenwriting. The benchmark covers many kinds of screenwriting work. It has both writing from scratch and revising, and most tasks come with earlier dialogue turns, so the model also needs to stay consistent with them. This is closer to how screenwriters really work than earlier screenwriting benchmarks, which mostly focus on one kind of artifact or task, such as script continuation.

  2. Detailed analysis over diverse, well-designed metrics. The five metrics look at quite different things: whether the task is done, whether the response fits the dialogue history, whether it follows the rules of the document type, surface errors like typos, and dramatic quality. Each metric uses a judging method that fits it. The results are also broken down by metric, by task action and by artifact type, which shows differences that one single score would hide.

Critical weaknesses
  1. Many details about the data construction are missing. The paper says the tasks are "constructed and validated by professional screenwriters," but how the human experts are really involved is pretty vague. We don't know how many screenwriters took part, what they did in each step, and how carefully people checked the model generated artifacts. This really affects the reality of the data. Also, there is no concrete data example in either the main text or the appendix, so the reader cannot see what a task actually looks like.

  2. Dramatic quality dominates the overall score and ranking. The five metrics are calculated in different ways, so their variance across models is very different. (DQ spreads much wider than the others.) So when the paper directly averages the raw scores, DQ mostly decides the final ranking. I understand the authors explain that DQ is the most discriminating metric, but if it should count more, this should be shown by a clear weight on normalized scores, not by using unnormalized numbers. Also, a small change of the weights can already change the top of the ranking.

  3. The analysis of the leaderboard is kind of superficial. Section 5 spends many words repeating the numbers in the tables and figures (who is first, who is last, the score range), but there is not much deeper qualitative analysis. For example, there are no model outputs that show typical failures, and no statistics of the error types. Some claims also go further than the numbers without providing evidence, e.g., the score range of models on script tasks is used to say "most models can not produce reasonably adequate screenplays."

Questions for authors
  1. Could you explain more about the data construction, e.g., which parts are written by screenwriters and which by models? How are the eight DQ anchors sampled, and how does the judge see the response and the anchors? A complete data example would help a lot.
  2. How is the agreement rate in Table 2 computed, and on how many samples? Why is the overall rate (64%) much lower than the average of the five metrics?
  3. Are all numbers in Table 1 computed on all 279 tasks? The averages do not fit 279 tasks and seem to use different task counts for different models. If some tasks were dropped, which ones and why?
  4. Why do the metrics use different judge models (DeepSeek or Qwen)? Since both families are also among the tested models, could a judge favor outputs from its own family?
3 comments

Q1.

Could you explain more about the data construction, e.g., which parts are written by screenwriters and which by models? How are the eight DQ anchors sampled, and how does the judge see the response and the anchors? A complete data example would help a lot.

Response: Screenwriters contributed in three roles: constructing task data in a simulated writing environment, reviewing task quality, and conducting human evaluation across the evaluation dimensions. Screenwriters involved in data construction also participated in evaluating the corresponding tasks, while other screenwriters cross-reviewed task quality after construction. The project involved ten screenwriters over four months: three junior screenwriters and seven with three to ten years of domain-specific professional experience.

Screenwriters authored all user requests in the dialogue history and the task requirements in the final turn. Models generated the assistant responses in the history. Data construction used the latest versions of GPT, Gemini, and Claude available during data collection. At the start of each conversation, the system randomly assigned one of these models, which remained fixed throughout that conversation.

For each task, we construct eight DQ anchor responses using a 4+4 design. Four responses are generated by GPT-5.4 under four distinct system prompts, each specifying a different persona with a distinct writing ability and style. The other four are sampled from Claude-Opus-4.6, Gemini-3.1-preview, Qwen3.6-plus, and DeepSeek-V4-pro, with one response from each model. This design is intended to reduce similarity among the anchors and increase the diversity and discriminative range of the reference set.

Experts then rank the eight candidate responses by quality. For anchor ordering, two screenwriters first ranked each task's candidates independently, discussed conflicting judgments, and referred unresolved disagreements to a third experienced screenwriter.

At evaluation time, the judge receives a single prompt containing, in order, the dialogue history, the current instruction, the eight anchor responses in their expert-defined order, and the response being evaluated. The judge identifies the candidate response's insertion position while preserving the relative order of the anchors; this is not a sequence of separate pairwise judge calls.

For a given task, every evaluated model is assessed using the same anchor order, with its response always placed last. This standardizes candidate placement across models, avoiding differences in candidate presentation position. In the current evaluation, the context, eight anchors, and candidate response together do not exceed 200,000 tokens. No truncation was used.

Section 3.1 describes task construction and anchor generation. Appendix G presents an anonymized, author-adapted example illustrating dimension-specific checks and anchor-based DQ scoring; the scenario is fictionalized, the preceding dialogue is summarized, and the response is shortened.

Q2.

How is the agreement rate in Table 2 computed, and on how many samples? Why is the overall rate (64%) much lower than the average of the five metrics?

Response: We sampled 143 response pairs to evaluate human–model agreement, with each pair consisting of two model responses to the same task. Human–model agreement is calculated as follows:

DimensionAssessmentAgreement calculation
Task Completion (TC)Two screenwriters judge whether each response completes the task.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Contextual Consistency (CC)Two screenwriters judge whether each response contains contextual consistency problems.Responses where the model matches the human consensus / responses on which both screenwriters agree.
Artifact Compliance (AC)Humans and the model independently identify problematic points under the artifact-compliance rules.F1 of model-identified problem points against human annotations as ground truth.
Basic Lexical Defects (BLD)Humans and the model independently identify basic-text defect points.F1 of model-identified problem points against human annotations as ground truth.
Dramatic Quality (DQ)Compare the screenwriters' preference between two responses with their anchor-based rank order.Comparisons with matching orders / all evaluated comparisons. Equal anchor-based ranks count as disagreement.
OverallCompare the screenwriters' holistic preference with the aggregate-score order.Comparisons with matching orders / all evaluated comparisons. This is not an average of dimension-level agreement rates; human preference is not computed with the benchmark's aggregation formula.

We use dimension-specific human–model agreement measures because we found that direct holistic evaluation leaves considerable room for subjective judgment.

Q3.

Are all numbers in Table 1 computed on all 279 tasks? The averages do not fit 279 tasks and seem to use different task counts for different models. If some tasks were dropped, which ones and why?

Response: Table 1 does not use all 279 tasks for every dimension-level mean. Not every task contributes to every evaluation dimension; tasks are routed to dimensions according to their type. A small number of non-creative tasks, such as evaluation tasks, are not assessed for Basic Lexical Defects (BLD), because their outputs are not used as inputs to subsequent creative work. Condensation and distillation tasks do not involve Artifact Compliance (AC) evaluation. In addition, a very small number of model responses encounter failures during generation or evaluation. These records are not assigned a score of zero and are excluded from the denominator of the corresponding mean. Consequently, the number of valid tasks can differ across models even within the same dimension.

Q4.

Why do the metrics use different judge models (DeepSeek or Qwen)? Since both families are also among the tested models, could a judge favor outputs from its own family?

Response: Judge selection balanced evaluation cost and effectiveness, with different models chosen to match the requirements of each dimension. For Basic Lexical Defects (BLD) and Artifact Compliance (AC), evaluation consists mainly of well-defined rubric checks. We therefore selected the lower-cost Qwen3.7-plus. We expected explicit rule constraints to limit the influence of family preferences. For Task Completion (TC) and Contextual Consistency (CC), we initially used Claude-Opus-4.6. To reduce evaluation cost, we subsequently adopted DeepSeek-V4-flash, whose judgments were close to those of Claude-Opus-4.6. For Dramatic Quality (DQ), we also initially used Claude-Opus-4.6 and subsequently selected DeepSeek-V4-pro to balance cost and effectiveness. Among the candidate judges compared, its rankings were closest to those of Claude-Opus-4.6. A dedicated family-bias test has not been conducted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.