- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces NovelAudit, a benchmark for open-ended narrative consistency auditing in full-length novels. Agents must autonomously identify apparent inconsistencies, determine whether they are explained by the narrative, and provide supporting evidence. The benchmark includes 24 novels with annotated reference cases, and evaluates several LLM-based agent configurations on discovery recall, labeling, evidence quality, and task completion.
Critical strengths
-
It's an interesting task formulation. Open-ended inconsistency discovery is more challenging than conventional long-context QA or claim verification, requiring agents to autonomously identify and investigate potential contradictions.
-
The benchmark provides structured annotations and evaluates not only case discovery but also the correctness and sufficiency of supporting evidence.
Critical weaknesses
- NovelAudit combines long-context reasoning, agentic exploration, and harness-dependent execution without sufficiently disentangling their impacts. In particular, the extremely low recall may reflect incomplete document coverage or premature stopping ("laziness") rather than limitations in consistency reasoning. While the task has practical relevance, its academic contribution remains unclear to me. Is the benchmark primarily intended to evaluate long-context reasoning, long-horizon agentic exploration, or agent harness effectiveness?
- Since reference cases are collected through model-assisted discovery and human review, they may not cover all valid inconsistencies. Consequently, correct predictions missing from the reference may be penalized. The paper also reports valid predictions missed by automatic matching, raising concerns about the reliability of the reported precision and recall.
Questions for authors
- How does agentic auditing compare against directly providing the full novel to a long-context LLM, at least for novels that fit within the model's context window? Especially recent LLMs with reasonable long context capabilities (e.g. MRCR).
- Can the authors evaluate same-session continuation after agents voluntarily terminate? Would asking agents to continue searching, while preserving their prior discoveries and context, substantially improve recall?