acceptodds
Under review as a conference paper at ICLR 2027

NovelAudit: A Benchmark for Open-Ended Narrative Consistency Auditing in Full-Length Novels

Abstract

AI-generated stories and scripts create a practical need for narrative consistency auditing. We begin with complete human-authored novels, where apparent contradictions may have distant explanations. Existing tasks emphasize supplied claims, predefined error categories, or detected errors, and largely omit explained reversals from required discovery outputs. We introduce NovelAudit, which asks agents to discover explained reversals and unresolved inconsistencies without supplied cases and support them with role-structured quotations. We construct references for 13 principal and 11 extended novels; evaluation separates task completion, case discovery, and explanatory sufficiency. On the principal novels, we compare DeepSeek across six runtimes, with supplementary Qwen and GPT–Codex conditions. Differences in runtime and model–runtime adaptation are clearest in completion rate, while single attempts discover limited, changing case subsets. Complementary discoveries across rounds expand coverage at increasing input cost per new match. Human audits link Explanation-Path F1 to explanatory sufficiency and identify valid predicted cases missed by matching, distinguishing discovery from evidence quality.

Primary area:
datasets and benchmarks
Keywords:
language-model agents, benchmark evaluation, narrative consistency, contradiction auditing, long-context reasoning, evidence retrieval, full-length novels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W55AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NovelAudit, a benchmark for open-ended narrative consistency auditing in full-length novels. Agents must autonomously identify apparent inconsistencies, determine whether they are explained by the narrative, and provide supporting evidence. The benchmark includes 24 novels with annotated reference cases, and evaluates several LLM-based agent configurations on discovery recall, labeling, evidence quality, and task completion.

Critical strengths
  1. It's an interesting task formulation. Open-ended inconsistency discovery is more challenging than conventional long-context QA or claim verification, requiring agents to autonomously identify and investigate potential contradictions.

  2. The benchmark provides structured annotations and evaluates not only case discovery but also the correctness and sufficiency of supporting evidence.

Critical weaknesses
  1. NovelAudit combines long-context reasoning, agentic exploration, and harness-dependent execution without sufficiently disentangling their impacts. In particular, the extremely low recall may reflect incomplete document coverage or premature stopping ("laziness") rather than limitations in consistency reasoning. While the task has practical relevance, its academic contribution remains unclear to me. Is the benchmark primarily intended to evaluate long-context reasoning, long-horizon agentic exploration, or agent harness effectiveness?
  2. Since reference cases are collected through model-assisted discovery and human review, they may not cover all valid inconsistencies. Consequently, correct predictions missing from the reference may be penalized. The paper also reports valid predictions missed by automatic matching, raising concerns about the reliability of the reported precision and recall.
Questions for authors
  1. How does agentic auditing compare against directly providing the full novel to a long-context LLM, at least for novels that fit within the model's context window? Especially recent LLMs with reasonable long context capabilities (e.g. MRCR).
  2. Can the authors evaluate same-session continuation after agents voluntarily terminate? Would asking agents to continue searching, while preserving their prior discoveries and context, substantially improve recall?
Reviewer_CF3NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NovelAudit studies open-ended consistency auditing in full-length novels. Agents discover state reversals, distinguish explained reversals from unresolved inconsistencies, and provide endpoint and explanatory quotations. Experiments use 13 novels with 1,032 reference cases; references for 11 additional novels are also constructed. DeepSeek experiments across six runtimes show substantial completion-rate differences but limited single-attempt coverage. Repeated attempts discover complementary cases. Human audits examine limitations of automatic matching and explanation scoring.

Critical strengths
  1. The task separates case identity, endpoint support, and complete explanation. Structured evidence routes identify outputs that contain relevant quotations without establishing the full explanation, providing a more detailed object of analysis than an error list.

  2. Core findings have complementary support: even perfect matching limits DeepSeek’s single-attempt recall to 9.49%–14.06%, while merging three formal GPT rounds increases recall from 17.06% to 32.79%. Limited individual output and complementary discoveries are therefore supported beyond a single composite metric.

Critical weaknesses
  1. Automatic matching misses valid, adequately supported discoveries. Among 216 audited unmatched outputs, both reviewers endorse 66 as existing qualified cases; 37 also have supported endpoints and explanations, yet fail the matching gates. Composite also changes substantially across matching rules. Consequently, precise score interpretation and system ordering remain insufficiently validated, directly limiting the benchmark’s use for automatic evaluation.

  2. Adjudication stability and source neutrality remain uncertain. The unmatched audit contains 104/216 identity or category disagreements, while 82.2% of the principal reference cases are GPT-only discoveries. Existing source-expansion analyses support limited coverage, but do not resolve inconsistent qualification boundaries or establish neutrality of reference construction across systems.

Questions for authors

Continue from Weakness,

  1. Based on the existing human audits, which system-level comparisons remain supported after accounting for missed and incorrect matches? Please distinguish validated conclusions from descriptive score differences.

  2. Which qualification or identity disagreements can be resolved through clearer rules, which remain unresolved, and how materially do they affect system comparisons?

  3. Are the complete references, scorer, raw predictions, and version records currently accessible for verifying these measurement issues?

Reviewer_KWY7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper intriduces NovelAudit, a benchmark for open-ended narrative consistency auditing of full-length novels. This task requires the agent to discover potential contradictory state reversals, classify them as explained or unresolved, and provide supporting textual evidence. The authors construct a reference of 24 novels and evaluate multiple model runtime configurations on 13 major novels. The proposed task is interesting and practically relevant. Large-scale evaluations and extensive human audits are valuable. However, there are currently several issues that limit the reliability and interpretability of the benchmark results.

Critical strengths
  1. The definition of evaluation task has certain innovation, especially NovelAudit emphasizes the self-discovery and evidence interpretation in complete works, which has clear application motivation.
  2. This benchmark uses 24 complete novels, 13 of which are used for formal evaluation, so the data scale is still large.
  3. Manual review is added to the evaluation process, and the reliability is better than the full automation process.
Critical weaknesses

1.This paper mainly relies on GPT and DeepSeek to find candidate cases, and then constructs reference answers through manual review. However, most of the cases in the final reference set are from GPT, while some cases unique to DeepSeek are excluded. This may lead to reference answers that are more in line with GPT's discovery preferences, thus affecting the fairness of the comparison between different models. 2. The paper mainly uses text matching to judge whether the model has successfully found cases, but manual inspection finds that some predictions that are actually correct and provide sufficient evidence still fail to match the reference answer. This suggests that the low recall reported by the paper may be partly due to the evaluation methodology itself, rather than solely due to insufficient model power. 3. This paper mainly compares different models and Agent running frameworks, but lacks basic methods such as chapter analysis, retrieval enhancement and structured state tracking, so it is difficult to judge the specific reasons for the poor performance of the model. In addition, the paper constructed contradictions by deleting explanations in the novel, but manual inspection found that many revised cases still retained complete explanations. This makes the relevant experiments difficult to accurately reflect the ability of the model to identify contradictions.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.