- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper shows how to scale story up to 100K length while maintaining consistency and writing quality. It also suggest NSTAgent, a training free framework that keeping the character information, past events, and unresolved narrative requirements. The NSTAgent dynamically adapt to the current story, so the content inside NSTAgent changes during the story. This framework also allows the writer to retrieve and revise earlier passages. The paper also propose ConStory-Bench to evaluate contradictions within story, using the preceding story as evidence.
Critical strengths
-
The paper shows how to evaluate long-form stories for contradictions, even at substantial lengths.
-
Tracking unresolved narrative requirements is the paper’s greatest strength, and this approach could be widely adopted in other works.
-
The paper is very well organized and easy to understand. I could easily follow its central contributions, including the generation process and evaluation metrics. The experimental results also show clear improvements over the baselines.
-
The appendix shows that NSTAgent’s performance can be improved through GRPO.
-
The paper provides an online link to the generated stories.
Critical weaknesses
-
I understand why the authors choose to check for contradictions only in the final 10K tokens, but is this the best approach? What if the distribution of contradictions differs from that of the full text? The paper does not compare contradictions in the full text with those in the final 10K tokens. (I later found that Appendix B.2 addresses this issue, but it examines only one type of error: “abandoned plot elements.”)
-
The paper does not include a human evaluation. Story generation should be evaluated by humans after all, since automatic metrics are still unreliable for assessing a story’s creativity, quality, and engagement. The authors should conduct a human evaluation, at least on a subset of shorter samples.
-
Related to the second weakness, the paper relies heavily on LLM-as-a-judge evaluation, which is vulnerable to several biases. This raises serious concerns about the reliability of the experimental results.
-
The paper states that the main experimental results do not align with those obtained using a smaller judge in Appendix B.1. However, it does not compare the results with other strong judges, such as GPT-5.6 or Claude, or provide further analysis of this discrepancy.
Questions for authors
-
I wonder whether the experimental results would hold when using different LLM judges. I strongly recommend evaluating the method with at least two additional LLMs.
-
I also wonder how human readers perceive the generated stories. I expect that stories produced using your method may be more engaging and interesting to readers. Even if this is not the case, analyzing the reasons would still be valuable.
-
I recommend comparing the method with other story-generation baselines on stories of up to 10K tokens, such as DOC or following works building on it.
If you can show that your evaluation results are stable across different LLM judges or human evaluations, and that your method yields significant improvements over other strong baselines, I would give you a higher overall score.
7 comments
Response to Reviewer BvZT (2/3)
Q1. Do the results hold under different LLM judges?
We re-scored the 20 10K DeepSeek-V4-Flash stories with three representative judges: Claude Opus 5.5, Gemini 4 Argon and gpt-6.1-sol. (Difference = NstAgent - RollSum)
Narrative Consistency:
| Judge | Subtype CED ↓, NstAgent | Subtype CED ↓, RollSum | Difference [95% CI] | Instance CED ↓, NstAgent | Instance CED ↓, RollSum | Difference [95% CI] |
|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro (paper judge) | 3.87 | 5.79 | −1.91 [−3.00, −0.78] | 4.87 | 8.72 | −3.85 [−5.57, −2.10] |
| claude-opus-5-5 | 7.62 | 9.56 | −1.94 [−3.11, −0.90] | 13.02 | 19.61 | −6.58 [−9.36, −3.87] |
| gemini-4-argon | 5.00 | 6.65 | −1.65 [−2.80, −0.59] | 7.54 | 12.22 | −4.68 [−7.06, −2.44] |
| gpt-6.1-sol | 4.10 | 4.48 | −0.38 [−1.24, +0.41] | 6.64 | 7.38 | −0.74 [−2.76, +1.17] |
Writing quality:
| Judge | WritingBench ↑, NstAgent | WritingBench ↑, RollSum | Difference [95% CI] |
|---|---|---|---|
| DeepSeek-V4-Pro (paper judge) | 9.18 | 8.87 | +0.31 [+0.12, +0.50] |
| claude-opus-5-5 | 6.72 | 6.53 | +0.19 [+0.02, +0.37] |
| gemini-4-argon | 9.41 | 9.24 | +0.17 [−0.03, +0.37] |
| gpt-6.1-sol | 6.85 | 6.92 | −0.08 [−0.21, +0.06] |
How the judges differ. We compared the judges error by error. The judges differ mainly in how much they report and they largely agree on what an error is. GPT differs in one specific way. In NstAgent stories it files the same passage under several error types more often: 2.07 entries per passage, against 1.36 in RollSum stories. Both CED metrics count such repeats, which hides the gap under GPT. Counting each erroneous passage once, all four judges find significantly fewer in NstAgent stories:
| Judge | Distinct erroneous passages per 10K words ↓, NstAgent | Distinct erroneous passages per 10K words ↓, RollSum | Difference [95% CI] | Entries per passage, NstAgent | Entries per passage, RollSum |
|---|---|---|---|---|---|
| DeepSeek-V4-Pro (paper judge) | 2.73 | 5.82 | −3.10 [−4.04, −2.17] | 1.67 | 1.44 |
| claude-opus-5-5 | 7.68 | 11.29 | −3.60 [−5.28, −2.05] | 1.65 | 1.64 |
| gemini-4-argon | 4.32 | 7.76 | −3.44 [−4.82, −2.17] | 1.75 | 1.57 |
| gpt-6.1-sol | 2.95 | 4.67 | −1.72 [−2.75, −0.75] | 2.07 | 1.36 |
The gain also holds for passages on which judges agree. Passages flagged by at least two judges number 3.76 per 10K words for NstAgent against 6.52 for RollSum (−2.75 [−4.05, −1.55]). Passages flagged by all four number 0.93 against 2.10 (−1.17 [−1.86, −0.48]).
Both stories in each comparison come from the same backbone, so family preferences affect both sides equally.
All three judges reproduce the consistency gain.
Q2. How do human readers perceive the stories?
We conducted preliminary human evaluation (see W2) and results suggest that NstAgent produces more favored stories.
Qualitatively, some readers say they are amazed that LLM can generate such lengthy stories, and it is cool that they can use their own prompt. Our simple but effective state design also helps readers to better keep up the plot. However, some feedback indicates that the stories lacks in variety compared to human stories due to the limit of backbone model.
Response to Reviewer BvZT (3/3)
Q3. Comparison with DOC and follow-up work at up to 10K
We ran the official code of DOC [1] (specifically DOC Story Generation V2) and of ConWriter [2], a recent state-tracking writer with verification and repair. Both use our prompts and ±20% length gate, and we changed only configuration. For ConWriter we also turned off its option to stop the whole story when a scene still fails a check after repair, because most stories then ended early.
Consistency at 10K under the paper judge (difference = NstAgent - baseline):
| Baseline | Backbone | Subtype CED ↓, NstAgent | Subtype CED ↓, baseline | Difference [95% CI] | Instance CED ↓, NstAgent | Instance CED ↓, baseline | Difference [95% CI] |
|---|---|---|---|---|---|---|---|
| DOC | DeepSeek-V4-Flash | 3.85 | 10.47 | −6.62 [−7.67, −5.53] | 4.76 | 19.60 | −14.84 [−17.56, −12.06] |
| ConWriter | DeepSeek-V4-Flash | 3.94 | 9.12 | −5.18 [−6.53, −3.91] | 4.99 | 16.16 | −11.17 [−14.22, −8.50] |
| ConWriter | GPT-5.6 Luna | 4.42 | 5.54 | −1.12 [−2.40, +0.07] | 6.24 | 8.00 | −1.76 [−3.53, −0.09] |
Writing quality at 10K under the paper judge:
| Baseline | Backbone | WritingBench ↑, NstAgent | WritingBench ↑, baseline | Difference [95% CI] |
|---|---|---|---|---|
| DOC | DeepSeek-V4-Flash | 9.22 | 5.12 | +4.11 [+3.61, +4.61] |
| ConWriter | DeepSeek-V4-Flash | 9.18 | 5.59 | +3.59 [+3.01, +4.18] |
| ConWriter | GPT-5.6 Luna | 9.19 | 6.91 | +2.28 [+1.64, +3.00] |
Note: DOC was run only at 10K on DeepSeek-V4-Flash. It is designed for stories of a few thousand words, one 10K story takes several hours, and its candidate scoring needs token log-probabilities, which GPT-5.6 Luna does not return.
The same comparison under three other judges. claude-opus-5-5, gemini-4-argon and gpt-6.1-sol re-scored the DOC and ConWriter stories written with DeepSeek-V4-Flash, using unchanged prompts and rubrics.
Consistency at 10K:
| Judge | Baseline | Subtype CED ↓, NstAgent | Subtype CED ↓, baseline | Difference [95% CI] | Instance CED ↓, NstAgent | Instance CED ↓, baseline | Difference [95% CI] |
|---|---|---|---|---|---|---|---|
| claude-opus-5-5 | DOC | 7.57 | 14.06 | −6.50 [−7.08, −5.94] | 12.62 | 46.19 | −33.57 [−38.36, −28.81] |
| claude-opus-5-5 | ConWriter | 7.62 | 11.66 | −4.04 [−4.99, −3.12] | 13.02 | 29.40 | −16.38 [−20.62, −12.26] |
| gemini-4-argon | DOC | 5.02 | 12.45 | −7.43 [−8.48, −6.38] | 7.91 | 32.29 | −24.38 [−27.17, −21.19] |
| gemini-4-argon | ConWriter | 5.16 | 11.38 | −6.22 [−7.24, −5.28] | 8.03 | 23.04 | −15.01 [−17.67, −12.55] |
| gpt-6.1-sol | DOC | 4.11 | 6.83 | −2.71 [−4.05, −1.30] | 6.79 | 13.88 | −7.10 [−10.69, −3.72] |
| gpt-6.1-sol | ConWriter | 4.10 | 6.22 | −2.11 [−3.22, −1.03] | 6.64 | 14.83 | −8.19 [−11.67, −5.09] |
Writing quality at 10K:
| Judge | Baseline | WritingBench ↑, NstAgent | WritingBench ↑, baseline | Difference [95% CI] |
|---|---|---|---|---|
| claude-opus-5-5 | DOC | 6.79 | 3.69 | +3.09 [+2.85, +3.35] |
| claude-opus-5-5 | ConWriter | 6.72 | 3.99 | +2.73 [+2.46, +3.01] |
| gemini-4-argon | DOC | 9.43 | 3.08 | +6.35 [+5.98, +6.67] |
| gemini-4-argon | ConWriter | 9.41 | 3.39 | +6.02 [+5.53, +6.47] |
| gpt-6.1-sol | DOC | 6.93 | 4.08 | +2.84 [+2.65, +3.03] |
| gpt-6.1-sol | ConWriter | 6.85 | 4.34 | +2.51 [+2.20, +2.82] |
All four judges find NstAgent significantly better than both baselines in consistency and in writing quality.
DOME [3], which the paper already compares with at 10K, follows DOC's evaluation setting and reports improvements over DOC. We also compare StoryWriter, which reports improvements over DOC. (Table 2 in paper)
Thank you again for your time and insightful comments. We would be glad to address any further questions or concerns.
[1] Kevin Yang et al. DOC: Improving Long Story Coherence With Detailed Outline Control. ACL 2023.
[2] Jindong Li et al. ConWriter: Transition-Constrained Stateful Long-Form Story Generation with Lightweight Neuro-Symbolic Consistency Control. Findings of EMNLP 2026.
[3] Qianyue Wang et al. Generating Long-form Story Using Dynamic Hierarchical Outlining with Memory-Enhancement. NAACL 2025.
[4] Haotian Xia et al. StoryWriter: A Multi-Agent Framework for Long Story Generation. CIKM 2025.
Response to Reviewer BvZT (1/3)
We thank the reviewer for the detailed review and for naming the key concerns. We answer in the order of the review.
W1. Contradictions in the final window versus the full text
For 20 prompts we scored windows of the same size at the beginning, middle and end of the 50K and 100K DeepSeek-V4-Flash stories (Differences = NstAgent - RollSum):
The terminal-window result holds at every position. The beginning window scores lower for both methods, as expected for text with nothing earlier to contradict. Apart from the opening, the terminal window is therefore representative of the story. We chose the terminal window to avoid "lost-in-the-middle" as much as possible.
W2. Human evaluation
We ran a blind pairwise study on 20 pairs of 10K DeepSeek-V4-Flash stories, NstAgent against RollSum. The two stories are shown side by side, following the pairwise design of DOC's own human evaluation [1]. Annotators do not know which method wrote which story.
These results suggest that NstAgent produces more favored stories. We know this evaluation is a small subset of samples and are working to expand it.
W3. Reliance on LLM-as-a-judge
LLM judges can carry biases, including a preference for their own model family. We added several checks to address this.
Three judges from other providers re-scored the stories (Q1).
An extended injection study. The original injection study had only 24 template controls. We raised template controls to 30 per length, added 15 natural contradictions and 15 consistent controls per length written in the story's own voice.
Precision does not differ significantly between the methods (Fisher p=0.20) and does not fall with length. Scaling each setting's CED by its own precision (unsure counted as false) leaves the gap almost unchanged:
We will add these checks into the revision.
W4. The smaller judge in Appendix B.1, and strong judges
That judge was used only in an early calibration of recall on injected contradictions, never to compare methods. Its recall dropped from 93% to 53% at long lengths, which is why we chose DeepSeek-V4-Pro.
We compared and analyzed our results against strong judges including Claude Opus 5.5, Gemini 4 Argon and gpt-6.1-sol. (see Q1).