acceptodds
Under review as a conference paper at ICLR 2027

Consistency Checking in Interactive Narratives at Million-Token Scale: A Benchmark and a Dual-Route Evaluator

Abstract

Large language models are increasingly used in text-adventure games and role-playing applications, where narrative histories can span hundreds or thousands of turns. Maintaining factual consistency over such long histories remains challenging; failures to do so can introduce contradictions that disrupt gameplay and reduce immersion. Reliable evaluation is therefore essential, yet the effectiveness of consistency checkers at this scale remains underexplored. To address this gap, we introduce LinConBench (Long Interactive Narrative Consistency Benchmark), to our knowledge the first benchmark for contradiction detection and turn-level localization in interactive narratives extending to thousands of turns. It comprises 2,000 error-bearing narratives derived from 400 human-authored novels across five source-length buckets, ranging from below 64K to 512K–1M tokens. Constructing such a benchmark requires reliable error annotations, yet exhaustively identifying naturally occurring contradictions is labor-intensive and prone to omissions or inconsistent judgments. We therefore adopt a two-stage pipeline that converts novels into multi-turn interactions while preserving their narration, then injects controlled contradictions with annotated error locations and supporting historical evidence. Each sample contains a single injected contradiction expressed in one or more turns and is validated by three reviewers. We further propose DRPV (Dual-Route Proposition Verification), which combines dual-route candidate discovery with proposition consolidation and full-prefix verification. Our experiments with nine baselines show that existing evaluators struggle to localize all erroneous turns, particularly in longer narratives and when historical evidence is more distant, whereas DRPV performs better across multiple metrics.

Primary area:
datasets and benchmarks
Keywords:
Meta-Evaluation, Narrative Self-Consistency, Long-Context Interactive Narratives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cWZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies semantic consistency checking in long interactive narratives, focusing on detecting contradictions with previously established facts and locating the corresponding erroneous turns. The authors introduce LINCONBENCH, a benchmark constructed from 50 Chinese novels, containing 2,000 instances across five context-length ranges up to 1M tokens. They also propose DRPV, a framework combining local discovery, historical retrieval, proposition consolidation, and verification. Experiments with nine evaluators show that long-context consistency checking remains challenging, while DRPV achieves the best overall Turn Recall, Turn F1, and Complete Hit.

Critical strengths
  1. Relevant and well-defined research problem. The paper addresses an important challenge in long-context understanding by requiring both contradiction detection and precise error localization while accounting for historical facts and state changes.

  2. Useful benchmark and empirical evaluation. LINCONBENCH provides annotated contradictions and supporting evidence across different context lengths. The evaluation includes multiple baselines, localization metrics, and ablation studies, offering useful insights into the limitations of existing approaches.

Critical weaknesses
  1. Limited benchmark diversity and generalizability. Although LINCONBENCH contains 2,000 instances, they originate from only 50 Chinese novels, potentially introducing substantial source-level dependence. The exclusive use of Chinese novels and artificially injected single-root contradictions also limits generalizability to other languages and naturally occurring consistency errors. Furthermore, potential pretraining exposure to the publicly available source novels is not adequately discussed.

  2. Insufficient control of test-time computation. Different evaluators use different numbers of requests, context inputs, and inference procedures. Although some settings are standardized, the overall computational budgets are not matched. Consequently, it remains unclear whether DRPV's improvements primarily result from its methodological design or additional candidate search and verification calls.

  3. Low precision and substantial false-alarm rates. Despite achieving higher recall, DRPV reports only 18.48% Turn Precision and a 54% false-alarm rate on clean histories, while approximately 71.09% of annotated error turns remain undetected. These results raise serious concerns about its practical reliability. The reported improvements appear to favor error coverage at the expense of precision and false-positive control.

Questions for authors

How do the authors address source-level dependence, potential pretraining exposure, and the limited diversity of artificially injected contradictions in LINCONBENCH?

Does DRPV maintain its advantage when compared with other evaluators under matched total test-time computation budgets?

How do the authors justify DRPV's practical reliability given its low precision and high false-alarm rate?

3 comments

W1.

First, our uncertainty analysis accounts for shared-source dependence. We performed a source-grouped paired bootstrap, resampling novels within each length bucket while retaining variants from the same novel together. The differences relative to ConStory are:

MetricDRPV − ConStory (pp)95% CI
Turn-P+3.88[2.31, 5.49]
Turn-R+8.84[7.20, 10.52]
Turn-F1+1.38[1.05, 2.85]
CH+7.42[5.68, 9.21]

Using frozen predictions, we also examined progressively larger subsets. From 1,625 examples onward, at least 95% of randomized subsets place Turn-R, Turn-F1, and CH simultaneously within ±1 percentage point of their full-benchmark values. These analyses support stability within the evaluated source distribution; they do not increase source diversity or establish cross-domain generalization.

Second, our focus is Chinese interactive narratives, a meaningful application setting in its own right. We acknowledge that Chinese novels cannot represent all languages or narrative domains, and cross-language generalization remains unverified.

Third, a single underlying contradiction can require localization across multiple turns. Section 3.2 allows the same erroneous fact to appear repeatedly. For example, Appendix J presents an encounter at turn 4 that contradicts later denials of prior sparring at both turns 260 and 261; evaluators must recover both erroneous turns. Controlled injection provides explicit error locations and inspectable historical evidence, although it does not fully represent naturally occurring errors or interactions among multiple independent contradictions.

Finally, we cannot rule out pretraining exposure to the source novels. The injected contradictions are absent from the original texts, but source familiarity may still influence evaluation. Low localization scores across the evaluated methods show that the task remains challenging; they do not establish that pretraining exposure has little effect.

W2.

We audited full-pipeline token usage, including candidate discovery, consolidation, and verification. For methods with complete token records, the per-example averages are:

MethodRecorded requestsInput (M tokens)Output (M tokens)Total (M tokens)
NFS1.000.1550.0240.179
Flawed2.240.5550.0310.586
ConStory8.812.2930.1902.483
DRPV18.181.6550.4842.139

DRPV uses more requests but approximately 13.9% fewer total tokens than ConStory, while consuming substantially more tokens than NFS and Flawed. Request count alone therefore does not characterize computational expenditure. These figures describe retained usage records, rather than matched FLOPs or monetary budgets; retry and cache accounting are not fully standardized across methods.

Existing ablations also support the contribution of the design. Combining both discovery routes yields 22.55% Turn-F1, versus 17.42% and 16.92% for block-only and retrieval-only discovery. With selected clusters held fixed, verification improves Turn-P from 15.23% to 18.48% and Turn-F1 from 20.27% to 22.55%.

These results provide evidence for the usefulness of the components, but do not fully separate their benefits from computational expenditure. We have not performed a strictly budget-matched comparison and therefore do not claim superiority under matched total test-time computation.

W3.

DRPV improves error recovery while using verification to filter false-positive candidates. Table 4 shows that, with selected clusters held fixed, verification increases Turn-P from 15.23% to 18.48% and Turn-F1 from 20.27% to 22.55%, while reducing Turn-R from 30.27% to 28.91%. This demonstrates a deliberate precision–recall trade-off rather than simply maximizing coverage.

Figure 5 and Appendix H further show that this trade-off can be adjusted through the verification cap. At K = 3, Turn-P reaches 22.01% and Turn-F1 reaches 23.08%, compared with 18.48% and 22.55% at the default K = 5. The default provides broader error coverage, whereas the smaller cap favors precision.

Nevertheless, substantial limitations remain: 71.09% of annotated error turns are missed, and 54% of reviewed clean histories receive false alarms. The cap analysis concerns localization on error-bearing histories and does not establish clean-history false-alarm control. We therefore present DRPV as an improvement in localization, with further advances in precision, recall, and false-alarm control needed for practical reliability.

Q1. As detailed in W1, we account for shared-source dependence through story-grouped bootstrap and sample-size stability analyses. Controlled injection provides explicit error annotations and historical evidence, but broader generalization beyond Chinese narratives and injected contradictions remains unverified. We also acknowledge that pretraining exposure cannot be ruled out and its effect has not been isolated.

Q2. We have not established whether DRPV maintains its advantage under strictly matched total test-time computation budgets. W2 provides a full-pipeline token audit and component ablations, which characterize computational expenditure and support the usefulness of the design.

Q3. We do not claim deployment-ready reliability. As detailed in W3, verification improves precision and F1 with selected candidates held fixed, and the verification cap adjusts the coverage–precision trade-off. These improvements do not resolve the substantial missed errors and 54% clean-history false-alarm rate.

After I went through it repeatedly, thinking it through and reviewing it, I think there are still many problems with this kind of work, so I will stick to my original position

Reviewer_BrELOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces LinConBench, a benchmark for detecting contradictions and identifying erroneous turns in long interactive narratives. It contains 2,000 variants derived from 50 human-authored Chinese novels, with controlled contradictions and human-verified error annotations across five source-length buckets reaching approximately one million tokens. The paper also proposes DRPV, which combines block-based and retrieval-based candidate discovery, proposition consolidation, and verification against the complete narrative prefix.

Critical strengths
  1. The benchmark proposes an effective and interesting evaluation setting by combining long narrative histories with explicit annotations of erroneous turns and supporting evidence.

  2. The human annotators review the contradiction generation process, which gives additional reliability of the research.

  3. The analysis distinguishes detecting an inconsistent narrative from accurately localizing its errors, offering a more informative assessment of evaluator performance.

  4. Comparisons with plenty number of baselines, component ablations, an alternative judge model provide a broad effectiveness of the proposed method. And the paper also provides sufficient additional analysis of the results.

Critical weaknesses
  1. Is there any risk of cherry-picking, if there is only one reviewer who check each of the error injections? Can I ask how many rejections are during reviewing error injection and the most typical reasons of rejection?

  2. The 2000 error examples are made from only 50 stories. This fact may limit generalization of the findings in this paper. Also, Appendix I reports a history-level false-alarm rate of 54% for DRPV on the 50 reviewed clean histories. This raises concerns about its practical reliability.

  3. There are more complex types of errors such as loosing objective or change of characteristics. This paper only deals with factual errors, which is relatively easier to notice.

Yet the paper primarily aims to detect factual conflicts in stories, I can give you higher overall score if you demonstrate your method is also effective at detecting other types of errors.

Questions for authors
  1. Is the BM25 the best retriever? What about to try other retrievers? There could have been bottleneck in the retriever's performance.

  2. Is there any other options rather using 'complete-prefix verification'? What if you verify only the state-transition history? Can this method reduce computational cost, while maintaining the performance?

3 comments

W1.

Human validation preceded detector evaluation, and reviewers had no access to any method’s predictions or scores. Acceptance was based on contradiction validity and annotation correctness.

Of 2,247 candidates entering human review, 247 were rejected, corresponding to approximately 11.0%. Typical rejection reasons included:

  1. No valid contradiction: the injected edit did not establish a genuine conflict with the historical evidence under our semantic-contradiction criterion.
  2. Inaccurate annotations: the annotated error turns or supporting evidence did not accurately identify or substantiate the injected contradiction.
  3. Ambiguity: the edited narrative admitted plausible alternative interpretations, preventing an unambiguous judgment that a contradiction had occurred.

Each variant was assessed by one reviewer, with uncertain or disputed cases resolved through discussion. This review process filtered candidates according to semantic validity and annotation quality, independently of detector performance. However, single-reviewer assessment can still introduce subjective judgments and does not provide independent inter-annotator agreement; we will clarify this limitation.

W2.

We added two complementary analyses to examine uncertainty and sample-size stability. First, a source-grouped paired bootstrap resamples stories within each length bucket, retaining variants from the same story together. The corrected differences relative to ConStory are:

MetricDifference (percentage points)95% CI
Turn-P+3.88[2.31, 5.49]
Turn-R+8.84[7.20, 10.52]
Turn-F1+1.38[1.05, 2.85]
CH+7.42[5.68, 9.21]

Second, using frozen predictions and 5,000 randomized, length-stratified source-story orders, we progressively increased the evaluated subset toward the full benchmark:

Stability criterionSample size from which the criterion remains satisfied
At least 95% of subsets rank DRPV first simultaneously in Turn-R, Turn-F1, and CH900
At least 95% of subsets place all three metrics within ±1 percentage point of their full-benchmark values1,625

These analyses support improvements and stable rankings within the evaluated source distribution while accounting for shared-story dependence. They do not increase the number of independent sources or establish generalization to new narrative domains, and we will state this limitation explicitly.

DRPV’s improved localization is accompanied by a lower observed history-level false-alarm rate than ConStory’s. Table 20 reports false alarms on 27 of 50 reviewed clean histories for DRPV (54%), compared with 31 of 50 for ConStory (62%), an 8-percentage-point difference. Across these complementary evaluations, stronger error localization therefore coexists with less frequent flagging of clean histories. This comparison is descriptive and does not establish statistical significance. The remaining 54% false-alarm rate is substantial, so we distinguish these relative improvements from deployment-ready reliability.

W3.

Our scope is evidence-grounded contradictions, which can overlap with goal and character consistency when explicit narrative claims are involved. For example, a later statement that a character “never pursued revenge” conflicts with an earlier scene explicitly documenting that pursuit. In contrast, a character abandoning revenge may represent legitimate development. The distinction depends on what the narrative establishes and whether subsequent events support the change.

Our factual-consistency setting also requires more than spotting mismatched attributes: evaluators must recover distant evidence, account for intervening state changes, and locate multiple turns expressing the same contradiction. The results show that these requirements remain challenging for the evaluated methods.

We recognize the value of broader goal-persistence and characterization errors. However, the current experiments do not establish effectiveness on those categories as a whole. We will clarify both their overlap with our contradiction criterion and the boundary of the benchmark’s validated coverage.

Q1.

We additionally tested BGE-M3 dense retrieval and BM25+BGE-M3 fusion using RRF. Neither alternative showed a consistent advantage over BM25-style retrieval across discovery-stage precision, recall, and F1. The following results measure candidate coverage before final verification; all scores are percentages.

RetrieverRetrieval-route RRetrieval-route PRetrieval-route F1Dual-route RDual-route PDual-route F1
BM25-style25.9212.3416.7242.4712.2118.96
BGE-M3 dense27.1512.1016.7443.1312.1018.90
BM25+BGE-M3 RRF26.4112.3016.7942.4712.0618.78

Q2.

A state-transition history could reduce verification input length if it accurately preserves relevant facts and intervening changes. For example, a key may initially be left in a lock, later retrieved, and then placed in a pocket. If the state history omits the retrieval event, the final location may incorrectly appear contradictory.

Complete-prefix verification allows the verifier to inspect these events in the original narrative, reducing dependence on an intermediate representation, at the cost of greater input length. Whether state-history verification maintains performance remains to be evaluated, and its total cost should include history construction and maintenance.

I appreciate your detailed response. I believe your research is valuable, but even with the rebuttal, the paper still lacks sufficient discussion of its validity and how it can be adapted to different cases. I will maintain my score, but I hope to see an improved version of the paper at a future conference.

Reviewer_XjpTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LinConBench, a human-verified benchmark for semantic contradiction detection and error-turn localization in long interactive narratives. It converts 50 Chinese novels into multi-turn interaction histories spanning up to 1M tokens, injects controlled contradictions, and annotates the erroneous turns and supporting evidence. The paper also proposes DRPV, a dual-route proposition verification pipeline that combines block-level and retrieval-based candidate discovery, proposition consolidation, and complete-prefix verification. Experiments with nine baselines show that DRPV achieves the highest Turn-R, Turn-F1, and CH, but precision remains low and performance degrades sharply with longer evidence distance and source length.

Critical strengths

A useful and non-trivial benchmark. LinConBench fills a gap by combining human-authored long narratives, multi-turn interaction, million-token scale, controlled contradiction injection, and turn-level error and evidence annotations. This enables systematic evaluation of long-range narrative consistency checking.

Reasonable and well-ablated method design. DRPV's dual-route discovery plus complete-prefix verification is a sensible engineering pipeline. Ablations show that the two routes are complementary, proposition consolidation helps, and prefix verification trades recall for precision. The paper also reports distance-stratified, length-stratified, and alternative-judge results.

Thorough empirical analysis. The evaluation covers nine baselines, multiple metrics, evidence-distance and source-length breakdowns, component ablations, verification-cap sensitivity, and false-alarm analysis. The negative results are informative: even the best method recovers only about 29% of annotated error turns, and performance drops substantially at long distances.

Critical weaknesses

My core concern is three-fold.

  1. The benchmark primarily measures long-context analysis/detection, not long-range generation consistency. This is my main concern. long-range generation consistency as the more central and valuable challenge for LLMs, while long-context analytical detection is increasingly commoditized. Framing detection as the primary target risks measuring a capability whose marginal value is limited.

  2. Detection is not connected to correction, limiting practical value. In interactive narrative settings, once an early erroneous turn has been generated, the history is already fixed and cannot be easily revised retroactively. The benchmark evaluates offline localization but does not study whether detected errors can be prevented, repaired, or used to improve generation. As a result, the benchmark may identify failures without offering an actionable path forward.

  3. Performance is still far from usable, and the source of difficulty is not fully disentangled. DRPV reaches only 22.55% Turn-F1 and 19.90% CH, with CH falling to 7.59% at 256+ turns and Turn-F1 to 10.61% at 512K–1M tokens. These results are far from my understanding of the capability of frontier LLMs. Have you tried to evaluate these models with a proper harness? For example, claude code? I think currently we are less and less likely to directly feed 1m tokens into the LLMs for a judge.

Questions for authors

Have you tried using any harness for this evaluation?

2 comments

W1.

We maintain that reliable evaluation is a foundation for systematic improvement. The reviewer emphasizes consistency in long-horizon generation, whereas our work addresses the equally important task of detecting inconsistencies in long interaction histories. Without reliable detection, improvements in generation cannot be assessed with confidence.

This need is reflected in practical applications. Anthropic’s Demystifying evals for AI agents describes how evaluations guide improvements in Claude Code, support quality benchmarking and regression testing for Descript’s video-editing agent, and assess Bolt’s application-building agent. The same principle applies to interactive narratives: before determining whether a narrative engine has become more consistent, we need evaluators that reliably identify its contradictions. LinConBench addresses this need through human-validated contradictions and error-turn annotations in histories extending to the million-token scale.

W2.

The difficulty of revising an already delivered interaction does not remove the practical value of offline diagnosis. Localization turns a general consistency failure into a concrete debugging target. For example, identifying a later statement that two characters have never met, together with an earlier passage describing their encounter, tells developers which claim conflicts with the established history and why. This information can guide targeted inspection and subsequent correction without requiring the original user interaction to be rewritten.

LinConBench evaluates whether checkers can reliably provide such localization. End-to-end repair is a further research question; its absence does not make identifying the erroneous turns and their conflicting evidence non-actionable.

W3.

The low scores are not confined to single-call full-context judging: substantial localization errors persist across the evaluated auditing, retrieval, and tool-assisted search approaches. Section 5.1 and Table 3 evaluate nine baselines using different ways of organizing and examining narrative histories. Our additional Claude Code harness experiment also permits adaptive search, selective reading, and note-taking, yet achieves only 17.23% Turn-F1 overall (see the harness results below). Thus, the observed difficulty extends beyond directly feeding the complete history into one judging request.

The paper also provides evidence about where difficulties arise. Table 4 shows that block-only and retrieval-only discovery achieve 17.42% and 16.92% Turn-F1, respectively, versus 22.55% when combined. With selected candidates held fixed, verification increases precision while reducing recall, showing that it removes false positives but also rejects some correct localizations. Figures 3–4 further document poorer localization at greater evidence distances and longer source lengths.

These analyses reveal complementary discovery needs and verification trade-offs, but do not fully isolate retrieval omissions, state-change interpretation, and turn-localization errors. The distance and length comparisons likewise establish associations rather than independent causal effects. We will make this distinction clearer. Our contribution is to expose and systematically measure these limitations across multiple evaluation strategies; we do not claim that DRPV has achieved deployment-ready reliability.

W4.

Following the reviewer’s suggestion, we evaluated Claude Code on our dataset using DeepSeek-V4-Pro, the same underlying model used by DRPV and all nine baselines. The agent autonomously reads selected turns, searches the history, and maintains notes, without access to gold annotations or a prescribed exploration sequence. The overall results are:

MethodTurn-P (%)Turn-R (%)Turn-F1 (%)CH (%)
Claude Code harness14.6320.9717.2317.10
DRPV18.4828.9122.5519.90

Comparing the shortest and longest source-length buckets further reveals substantial degradation across all four metrics for the Claude Code harness:

Source lengthTurn-P (%)Turn-R (%)Turn-F1 (%)CH (%)
Below 64K20.4828.4623.8223.25
512K–1M8.1411.819.648.50

These results show that substantial localization difficulty persists under adaptive, tool-assisted exploration. The challenge therefore extends beyond single-call full-context judging, supporting the benchmark’s relevance to harness-based evaluation.

Thank you for you response. I keep my score as is.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.