- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies semantic consistency checking in long interactive narratives, focusing on detecting contradictions with previously established facts and locating the corresponding erroneous turns. The authors introduce LINCONBENCH, a benchmark constructed from 50 Chinese novels, containing 2,000 instances across five context-length ranges up to 1M tokens. They also propose DRPV, a framework combining local discovery, historical retrieval, proposition consolidation, and verification. Experiments with nine evaluators show that long-context consistency checking remains challenging, while DRPV achieves the best overall Turn Recall, Turn F1, and Complete Hit.
Critical strengths
-
Relevant and well-defined research problem. The paper addresses an important challenge in long-context understanding by requiring both contradiction detection and precise error localization while accounting for historical facts and state changes.
-
Useful benchmark and empirical evaluation. LINCONBENCH provides annotated contradictions and supporting evidence across different context lengths. The evaluation includes multiple baselines, localization metrics, and ablation studies, offering useful insights into the limitations of existing approaches.
Critical weaknesses
-
Limited benchmark diversity and generalizability. Although LINCONBENCH contains 2,000 instances, they originate from only 50 Chinese novels, potentially introducing substantial source-level dependence. The exclusive use of Chinese novels and artificially injected single-root contradictions also limits generalizability to other languages and naturally occurring consistency errors. Furthermore, potential pretraining exposure to the publicly available source novels is not adequately discussed.
-
Insufficient control of test-time computation. Different evaluators use different numbers of requests, context inputs, and inference procedures. Although some settings are standardized, the overall computational budgets are not matched. Consequently, it remains unclear whether DRPV's improvements primarily result from its methodological design or additional candidate search and verification calls.
-
Low precision and substantial false-alarm rates. Despite achieving higher recall, DRPV reports only 18.48% Turn Precision and a 54% false-alarm rate on clean histories, while approximately 71.09% of annotated error turns remain undetected. These results raise serious concerns about its practical reliability. The reported improvements appear to favor error coverage at the expense of precision and false-positive control.
Questions for authors
How do the authors address source-level dependence, potential pretraining exposure, and the limited diversity of artificially injected contradictions in LINCONBENCH?
Does DRPV maintain its advantage when compared with other evaluators under matched total test-time computation budgets?
How do the authors justify DRPV's practical reliability given its low precision and high false-alarm rate?
3 comments
Q1. As detailed in W1, we account for shared-source dependence through story-grouped bootstrap and sample-size stability analyses. Controlled injection provides explicit error annotations and historical evidence, but broader generalization beyond Chinese narratives and injected contradictions remains unverified. We also acknowledge that pretraining exposure cannot be ruled out and its effect has not been isolated.
Q2. We have not established whether DRPV maintains its advantage under strictly matched total test-time computation budgets. W2 provides a full-pipeline token audit and component ablations, which characterize computational expenditure and support the usefulness of the design.
Q3. We do not claim deployment-ready reliability. As detailed in W3, verification improves precision and F1 with selected candidates held fixed, and the verification cap adjusts the coverage–precision trade-off. These improvements do not resolve the substantial missed errors and 54% clean-history false-alarm rate.
After I went through it repeatedly, thinking it through and reviewing it, I think there are still many problems with this kind of work, so I will stick to my original position
W1.
First, our uncertainty analysis accounts for shared-source dependence. We performed a source-grouped paired bootstrap, resampling novels within each length bucket while retaining variants from the same novel together. The differences relative to ConStory are:
Using frozen predictions, we also examined progressively larger subsets. From 1,625 examples onward, at least 95% of randomized subsets place Turn-R, Turn-F1, and CH simultaneously within ±1 percentage point of their full-benchmark values. These analyses support stability within the evaluated source distribution; they do not increase source diversity or establish cross-domain generalization.
Second, our focus is Chinese interactive narratives, a meaningful application setting in its own right. We acknowledge that Chinese novels cannot represent all languages or narrative domains, and cross-language generalization remains unverified.
Third, a single underlying contradiction can require localization across multiple turns. Section 3.2 allows the same erroneous fact to appear repeatedly. For example, Appendix J presents an encounter at turn 4 that contradicts later denials of prior sparring at both turns 260 and 261; evaluators must recover both erroneous turns. Controlled injection provides explicit error locations and inspectable historical evidence, although it does not fully represent naturally occurring errors or interactions among multiple independent contradictions.
Finally, we cannot rule out pretraining exposure to the source novels. The injected contradictions are absent from the original texts, but source familiarity may still influence evaluation. Low localization scores across the evaluated methods show that the task remains challenging; they do not establish that pretraining exposure has little effect.
W2.
We audited full-pipeline token usage, including candidate discovery, consolidation, and verification. For methods with complete token records, the per-example averages are:
DRPV uses more requests but approximately 13.9% fewer total tokens than ConStory, while consuming substantially more tokens than NFS and Flawed. Request count alone therefore does not characterize computational expenditure. These figures describe retained usage records, rather than matched FLOPs or monetary budgets; retry and cache accounting are not fully standardized across methods.
Existing ablations also support the contribution of the design. Combining both discovery routes yields 22.55% Turn-F1, versus 17.42% and 16.92% for block-only and retrieval-only discovery. With selected clusters held fixed, verification improves Turn-P from 15.23% to 18.48% and Turn-F1 from 20.27% to 22.55%.
These results provide evidence for the usefulness of the components, but do not fully separate their benefits from computational expenditure. We have not performed a strictly budget-matched comparison and therefore do not claim superiority under matched total test-time computation.
W3.
DRPV improves error recovery while using verification to filter false-positive candidates. Table 4 shows that, with selected clusters held fixed, verification increases Turn-P from 15.23% to 18.48% and Turn-F1 from 20.27% to 22.55%, while reducing Turn-R from 30.27% to 28.91%. This demonstrates a deliberate precision–recall trade-off rather than simply maximizing coverage.
Figure 5 and Appendix H further show that this trade-off can be adjusted through the verification cap. At K = 3, Turn-P reaches 22.01% and Turn-F1 reaches 23.08%, compared with 18.48% and 22.55% at the default K = 5. The default provides broader error coverage, whereas the smaller cap favors precision.
Nevertheless, substantial limitations remain: 71.09% of annotated error turns are missed, and 54% of reviewed clean histories receive false alarms. The cap analysis concerns localization on error-bearing histories and does not establish clean-history false-alarm control. We therefore present DRPV as an improvement in localization, with further advances in precision, recall, and false-alarm control needed for practical reliability.