- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes NarraWorld, a structured memory system for long-form writing. It maintains four connected views over a shared narrative graph: world facts, character-specific beliefs, ongoing narrative developments, and hypothetical continuations. It organizes events into scenes, plotlines, and plots while retaining links to their underlying evidence. At retrieval time, a planner uses the writing context and a memory preview to identify implicit information needs and assemble relevant records within a token budget.
Critical strengths
-
The distinction between world facts and character-specific beliefs is very important to assess narrative coherence, which is not widely covered by previous works. It is also related to the 'theory-of-mind' task, where language models still shows weakness.
-
The combination of hierarchical memory and dependency planning is well motivated, and I think it can be adopted by other story generation methods.
-
The evaluation includes plenty number of baselines, ablations studies, and human assessments.
Critical weaknesses
-
Is there any problem with averaging the scores for the final summary and the 14 chapters? The quality of the overall summary may differ from that of the individual chapters, and including it in the average could inflate the overall score.
-
This is the most serious concern. What is the practical improvement over the Full Context baseline? NarraWorld outperforms Full Context on LSGC quality and overall TMAS quality, but the gains do not seem substantial enough, considering additional inference cost. Moreover, the paper does not evaluate the same method across stories of different lengths.
-
Is atomic closure accurate? The paper proves that links exist between higher-level and lower-level nodes, but does not establish whether those links are accurate or meaningful. Further analysis is needed to assess their quality.
Questions for authors
-
Why didn’t the authors measure agreement among the LLM judges? Although the paper reports inter-human and human–LLM agreement, it does not report agreement among the LLM judges. I think this should also be measured. I understand that the inter-human and human–LLM agreement scores are relatively low, but this is common in story evaluation, so I will not focus on those values in my assessment.
-
Providing an anonymous link to the generated stories would be very helpful, as it would allow readers to assess their quality directly.
I can give you higher overall score if you appropriately prove that NarraWorld has significant improvements than Full Context baseline even considering the huge amount of additional cost, and that these improvements hold across various lengths of stories. I recommend analyzing your method from other perspectives, such as its effect on diversity, or training models using your method.
8 comments
Response to Reviewer TUzp (2/3)
W2: Advantages over Full Context: Long-Form Quality, Originality, and Diversity (Continued)
Originality and Diversity. Following your suggestion, we use the publicly released MoPS collection (Ma et al., 2024) and adopt its Originality and Fascination metrics to assess originality and narrative appeal, complementing the long-story quality and cost comparisons above. MoPS provides varied premises for open-ended story generation, synthesized from components such as themes, settings, characters, and plots. We sample 200 premises from this collection and generate one story with each method per input, yielding 200 matched pairs. DeepSeek-V4-Pro and Gemini evaluate the generated stories on Originality, the perceived novelty relative to familiar stories, and Fascination, the perceived appeal of the narrative. Both use a 0-100 scale, with higher scores indicating better performance.
Both judges favor NarraWorld on these two dimensions. The table reports paired mean differences (NarraWorld minus Full Context); brackets give 95% bootstrap intervals for Originality.
| Dimension | DeepSeek difference | Gemini difference |
|---|---|---|
| Originality | +15.67 [13.02, 18.29] | +11.21 [9.31, 13.16] |
| Fascination | +1.08 | +1.15 |
We additionally measure semantic diversity across the stories generated from these inputs using BGE-M3 embeddings (Chen et al., 2024). Pairwise distance and distance to the collection center measure overall semantic spread; nearest-neighbor distance measures local distinctiveness. Larger distances indicate greater diversity under this representation. NarraWorld improves all three collection-level measures:
| Collection diversity measure | Change relative to Full Context |
|---|---|
| Mean pairwise cosine distance | +6.8% |
| Distance to the group center | +7.3% |
| Mean 3-nearest-neighbor distance | +11.4% |
Taken together, these results show an increasingly favorable quality-efficiency trade-off as stories grow, alongside gains in originality, narrative appeal, and collection diversity. NarraWorld is therefore particularly well suited to long-form writing.
Revision: Section 3.2 (p. 8, lines 382–394) summarizes the length-dependent comparison and additional evaluation. Appendix I.2 (p. 40, lines 2120–2147, Table 25) reports quality, Appendix F.7 (pp. 35–36, lines 1884–1908, Figure 6) reports the chapter-position diagnostic, and Appendix F.6 (p. 35, lines 1844–1881, Table 24) reports full-story human evaluation. Appendix I.1 (p. 40, lines 2108–2118) defines resource accounting, Appendix I.3 (pp. 41–42, lines 2185–2220, Table 27) reports resource use, and Appendix L (pp. 48–49, lines 2540–2594, Tables 34–35) reports originality and collection diversity.
W3: Atomic closure accuracy and meaning
Thank you for asking us to verify the quality of atomic closure. We audited hierarchical memory from 15 complete NarraWorld LSGC stories, sampling 30 scenes, 30 plotlines, and 30 plots across early, middle, and late snapshots, together with 846 node-member associations.
Correctness and Meaningfulness. Correctness requires the linked record to be supported by the source text and correctly associated with that scene, plotline, or plot. Meaningfulness requires the association to provide substantive evidence for its content or development; merely sharing a character or background is insufficient. Both requirements must be met to pass the association check. We additionally report reference resolution and summary fidelity.
Evaluation Procedure. Two annotators checked memory against the source prefix available at each snapshot. Reference resolution and association accuracy use sampled links as their denominator; summary fidelity uses sampled nodes. Before discussion, annotator agreement on closure associations was 94.2%, with Cohen's kappa of 0.63.
| Level | References resolving | Correct and meaningful associations | Node summaries passing |
|---|---|---|---|
| Scene | 254/254 (100.0%) | 243/254 (95.7%) | 28/30 (93.3%) |
| Plotline | 290/292 (99.3%) | 271/292 (92.8%) | 26/30 (86.7%) |
| Plot | 297/300 (99.0%) | 272/300 (90.7%) | 25/30 (83.3%) |
| Overall | 841/846 (99.4%) | 786/846 (92.9%) | 79/90 (87.8%) |
Overall, 92.9% of sampled links are both correct and meaningful, with pass rates above 90% at every level. These links provide substantive evidence for the corresponding abstractions, enabling higher-level memory to be traced back to the events and states that support it. Together with the summary-fidelity results, this supports the reliability and practical role of atomic closure in evidence reconstruction.
Revision: Section 3.2 (p. 8, lines 396–400) summarizes the audit. Appendix J.1 (p. 43, lines 2270–2303) describes sampling and annotation; Appendix J.3 (pp. 44–45, lines 2339–2380, Table 30) reports closure accuracy and summary fidelity.
Response to Reviewer TUzp (1/3)
Q1: Agreement among LLM judges
Thank you for suggesting this consistency check. We added Gemini (gemini-3.1-pro-preview) alongside DeepSeek-V4-Pro to evaluate stories generated on the LSGC dataset at 10k, 20k, 40k, and 80k words by Full Context, StructMem, and NarraWorld. Both judges evaluate the same anonymized chapter texts with identical prompts and the same 16-criterion rubric. Story scores average chapter scores on a 0-100 scale and exclude final summaries.
Agreement Metrics. Spearman correlation measures agreement in story rankings; MAE measures the average absolute score difference; mean offset is Gemini minus DeepSeek. The last column reports how often the two judges agree on the sign of the NarraWorld-minus-Full Context score difference across 15 matched starter–run pairs; this reports agreement in comparison direction.
| Length | Spearman ρ | MAE | Mean offset | NW vs. FC direction agreement |
|---|---|---|---|---|
| 10k | 0.73 | 1.82 | +1.47 | 13/15 (86.7%) |
| 20k | 0.70 | 1.95 | +1.48 | 14/15 (93.3%) |
| 40k | 0.68 | 2.12 | +1.49 | 14/15 (93.3%) |
| 80k | 0.63 | 2.39 | +1.48 | 15/15 (100.0%) |
At each length, these metrics cover stories from all three methods. The two judges show substantial positive correlation in story rankings and small absolute score differences. Ranking methods by their mean story scores also yields the same ordering at every length: NarraWorld, StructMem, then Full Context. These results provide consistent support from both evaluators for our main comparative findings.
Revision: Appendix F.5 (pp. 34–35, lines 1782–1842, Tables 22–23) reports cross-judge agreement. The blue-marked addition in Section 3.2 (p. 8, lines 379–380) directs readers to this analysis.
Q2: Anonymous access to generated stories
Thank you for requesting access to the generated stories. They are available through the following anonymous repository:
Anonymous generated-story examples
Finally, thank you again for your thoughtful feedback! Your questions helped us sharpen the evaluation and clarify how the results should be interpreted. We learned a great deal from your suggestions, particularly about examining narrative quality from complementary perspectives, and explored more useful and interesting datasets&benchmarks in the field of AI Writing. The additional analyses and revisions have strengthened the paper, and we appreciate the time and care you have devoted to reviewing our work.
Response to Reviewer TUzp (1/3)
We thank you for highlighting the distinction between world facts and character beliefs, the memory hierarchy, and the evaluation coverage. We address the three weaknesses and two questions below, incorporating the additional diversity suggestion into W2.
W1: Chapter and final-summary score aggregation
Thank you for raising this concern. For LSGC, where summary evaluation contributes to the overall score, we report chapter-only, final-summary, and combined scores separately for all eight methods below. Scores are on a 0-100 scale, with higher values indicating better quality.
Final-summary scores are lower than chapter-only scores for every method, so including the summary does not inflate the absolute Overall scores in this comparison. The ranking is unchanged across all three columns. We retain the original combined score in the primary 40k table and report the decomposition alongside it; the new cross-length study uses chapter-only scores throughout.
Revision: Appendix I.2 (pp. 40–41, lines 2149–2175, Table 26) reports chapter-only, final-summary, and combined scores for all eight methods on LSGC.
W2: Advantages over Full Context: Long-Form Quality, Originality, and Diversity
Thank you for asking whether the gains over Full Context justify the inference cost. We directly compare both methods on stories generated on the LSGC dataset at 10k, 20k, 40k, and 80k words, using the same starters and three runs per setting.
Quality Across Lengths. DeepSeek-V4-Pro and Gemini (gemini-3.1-pro-preview) evaluate the same anonymized chapter texts using the same 16-criterion rubric. Scores are chapter-only averages on a 0-100 scale, excluding final summaries; higher is better. Δ is NarraWorld minus Full Context.
NarraWorld's quality advantage grows with story length. From 10k to 80k, its lead over Full Context increases from 2.39 to 8.63 points under DeepSeek and from 2.36 to 8.60 under Gemini. Both judges therefore show a widening advantage as the narrative horizon expands. The separate human evaluation gives NarraWorld the highest mean Human Overall at all four lengths; Appendix F.6 reports its sample sizes and results.
Chapter-Position Diagnostic. We additionally examined the chapter-only means by position for the 14-chapter 40k and 28-chapter 80k conditions. The late-minus-early change is −6.32 and −12.72 for Full Context, −3.73 and −7.50 for StructMem, and −1.48 and −3.53 for NarraWorld, respectively. This position-wise diagnostic shows a smaller decline for NarraWorld and a widening late-story margin over Full Context; the appendix reports the per-chapter points and descriptive LOWESS trends.
Computational Cost. Resource values are means per completed story, covering writing, memory construction, and planning, and excluding evaluation-judge usage. Tokens include input and output; runtime is end-to-end wall-clock time.
NarraWorld's relative token and runtime costs decline as stories grow. From 10k to 80k, the token-cost ratio relative to Full Context falls from 3.24 to 0.68, and the runtime ratio from 1.50 to 0.95. At shorter lengths, the quality gains come with additional token and runtime costs; at 80k, NarraWorld delivers its largest quality gain while using 31.8% fewer tokens and taking 5.4% less time. Full Context repeatedly supplies the entire accumulated history for each writing step, whereas NarraWorld reconstructs the relevant memory context for the current request. The measured total costs include both context processing and memory construction and planning.