acceptodds
Under review as a conference paper at ICLR 2027

Overlap, Unique and Conflict: Can LLMs Extract What They Can Recognize?

Abstract

Understanding multi-perspective alternative narratives requires identifying how their information agrees, conflicts, or differs across sources. Existing work on crosstext relations largely focuses on categorizing relations between predefined text pairs, such as entailment or contradiction, rather than directly extracting such information from full narratives. To address this gap, we introduce **Overlap–Unique–Conflict (OUC)** extraction, a cross-narrative task that extracts all overlapping, conflicting, and unique clauses from two narratives. To support this study, we construct a benchmark of approximately 22K narrative pairs and 140K OUC instances spanning factual, argumentative, and political discourse. Evaluating 14 opensource LLMs (0.6B–35B), we find that unique information is far easier to extract than overlap and conflict: the strongest model, Gemma-4-31B, reaches only 61.13% F1-score on overlap and 48.58% on conflict, against more than 75% on unique. Further diagnostic analysis reveals that this difficulty does not stem from relation recognition alone, but rather from a failure to pair and extract the corresponding clauses from full narratives, especially in smaller models. Nevertheless, learning these extractions with task-specific supervision narrows the gap considerably: a fine-tuned Qwen-3-8B gains ∼**15–28%** absolute over its baseline and surpasses models roughly four times its size (e.g., Qwen-3.6-35B) on several tasks. Even so, overlap and conflict remain well below satisfactory, leaving cross-narrative clause extraction an open challenge.

Primary area:
datasets and benchmarks
Keywords:
Cross-narrative extraction, large language models, LLM evaluation, fine-tuning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BNHeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Overlap–Unique–Conflict (OUC) extraction, a task that requires a model to identify overlapping, conflicting, and narrative-specific information from two complete narratives. Unlike conventional relation classification settings, OUC requires models to discover relevant clauses, establish cross-narrative correspondences, classify their relations, and identify information unique to each narrative. The authors construct a benchmark of 22,558 narrative pairs from peer reviews, factual news, and political discourse using an LLM-assisted annotation pipeline with subsequent human validation. They evaluate 14 open-source LLMs ranging from 0.6B to 35B parameters under zero-shot, few-shot, and chain-of-thought prompting. Their diagnostic experiments separate relation understanding, clause alignment, and end-to-end extraction to investigate the sources of model errors. They further compare supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO), and examine the effect of supervision size and cross-domain generalization. The results show that overlap and conflict extraction are substantially more difficult than unique-clause extraction. Task-specific SFT improves performance considerably, enabling Qwen-3-8B to outperform larger models on several tasks. The paper provides a broad empirical study of cross-narrative extraction, although the novelty of the task relative to existing cross-document relation extraction work and the reliability of the automatically curated benchmark require further scrutiny.

Critical strengths
  1. A useful task formulation and systematic analysis of extraction bottlenecks. The paper formulates OUC extraction as a task that requires models to identify overlapping, conflicting, and unique information directly from complete narratives. This goes beyond classifying predefined clause pairs and requires models to discover relevant information and establish cross-narrative correspondences. The diagnostic experiments separately examine relation understanding, pair alignment, and end-to-end extraction, providing useful evidence that the main difficulty lies not only in recognizing semantic relations but also in discovering and correctly aligning the relevant clauses. This analysis offers a meaningful perspective on the limitations of current LLMs.

  2. Broad experimental coverage and evidence that task-specific supervision improves performance. The paper evaluates 14 open-source LLMs across multiple model families and sizes, compares zero-shot, few-shot, and chain-of-thought prompting, and investigates SFT, DPO, and GRPO. The additional supervision-scaling, cross-domain, and error analyses provide a relatively comprehensive empirical study. In particular, SFT substantially improves the performance of Qwen-3-8B, allowing it to outperform the much larger Gemma-4-31B baseline on Conflict, Unique-A, and Unique-B, while remaining competitive on Overlap. These findings provide useful evidence that task-specific supervision can improve cross-narrative extraction, even though the generality of the results and the novelty of the task formulation require further examination.

Critical weaknesses
  1. The novelty of the OUC task is not sufficiently distinguished from closely related prior work. The paper presents OUC as a new task for extracting overlapping, conflicting, and unique information from complete narratives. However, prior work has already studied semantic overlap between alternative narratives, including the Semantic Overlap Summarization (SOS) task introduced by Bansal et al. (COLING 2022) and subsequent work on synthetic-data training and semantic evaluation. Cross-document structure theory and sentence-alignment research also address relations between information units from different documents. Although OUC extends these directions by jointly extracting overlap, conflict, and narrative-specific information, the paper does not sufficiently establish the conceptual and empirical boundaries between OUC and these existing tasks. In particular, it remains unclear whether the proposed formulation enables fundamentally new research questions or primarily combines established capabilities in a unified extraction setting. A more direct comparison with existing datasets, task definitions, and evaluation protocols is needed to substantiate the claimed novelty.

  2. The benchmark has unresolved annotation-reliability and statistical-reporting issues. The benchmark is constructed primarily through an LLM-assisted pipeline, with candidate generation, model-based validation, and subsequent unique-clause extraction. Although the authors provide instance-level human validation, the complete-set evaluation covers only 100 narrative pairs and relies on a single annotator to extract the full set of relations. This provides limited evidence that the automatically curated annotations are exhaustive, particularly for subtle conflicts and implicit semantic correspondences. Moreover, the reported benchmark statistics are inconsistent: the abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, totaling approximately 1.10 million instances. The authors should clarify this discrepancy and establish the counting conventions. Until these issues are resolved, the reliability, completeness, and reproducibility of the benchmark remain difficult to assess.

  3. The ROUGE-L-based evaluation is insufficiently justified for semantic relation extraction. The primary evaluation uses ROUGE-L similarity with a threshold of 0.6 to match predicted and reference clauses, followed by greedy one-to-one matching. However, lexical overlap does not necessarily indicate that two clauses express the same underlying fact, nor does it establish that a pair represents a genuine contradiction. Conversely, valid semantic correspondences can use substantially different wording. This concern is particularly important for the Conflict task, where two clauses can share many words while making incompatible claims, or discuss related topics without contradicting each other. Closely related work on semantic overlap summarization has already identified limitations of ROUGE and proposed SEM-F1 as a more suitable alternative for that task. Although the authors report threshold sensitivity and evaluate a human-annotated subset, they do not sufficiently establish that the chosen metric reliably measures semantic correctness across all four OUC categories. Additional evaluation using semantic matching and human-judgment-based comparisons would strengthen confidence in the reported model rankings and performance differences.

  4. The experimental setup does not fully establish generalization or isolate the effects of post-training. The main benchmark is divided into training, validation, and test sets using a random 75/5/20 split at the narrative-pair level. The paper does not clearly establish whether duplicate or near-duplicate source documents, overlapping events, or closely related narratives are excluded across splits. Such overlap could inflate performance if models encounter related content during training and evaluation. The leave-one-domain-out experiments are useful, but they do not fully resolve potential leakage or establish generalization to unseen source collections and event distributions. In addition, the comparison between SFT, DPO, and GRPO involves substantially different computational costs and optimization procedures, making it difficult to attribute performance differences solely to the learning objectives. More explicit source-level deduplication, grouped split validation, and controlled comparisons of training budgets would be needed to support the broader claims about generalization and the relative effectiveness of the post-training methods.

Questions for authors
  1. How does OUC differ fundamentally from existing work on semantic overlap summarization, cross-document structure, sentence alignment, and contradiction detection? In particular, what capabilities or research questions does OUC enable that cannot be adequately addressed by existing tasks and benchmarks? Could the authors clarify the specific contribution beyond unifying overlap, conflict, and unique-information extraction in a single framework?

  2. The abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, which sum to approximately 1.10 million instances. Could the authors explain this discrepancy and clarify the counting conventions used? In addition, given that the complete-set human evaluation covers 100 narrative pairs and uses one annotator, what evidence supports the completeness of the automatically curated annotations, particularly for implicit conflicts and semantically related clauses with different wording?

  3. The paper uses ROUGE-L-based matching with a threshold of 0.6 as the primary evaluation protocol. How do the authors justify this metric for evaluating semantic overlap and conflict relations, where lexical similarity does not necessarily imply semantic equivalence or contradiction? Have the authors compared the reported results with semantic matching metrics or human-judgment-based evaluation, and do the main model rankings remain stable under alternative evaluation protocols?

  4. The benchmark is split at the narrative-pair level using a random 75/5/20 partition. Can the authors clarify whether duplicate or near-duplicate source documents, overlapping events, and closely related narratives are prevented from crossing the training, validation, and test splits? Furthermore, how much of the observed improvement from SFT over DPO and GRPO can be attributed to the learning objective rather than differences in computational budgets and optimization configurations? Additional evidence on source-level separation and controlled comparisons would help establish the robustness of the reported conclusions.

3 comments

We thank the reviewer for the positive assessment of the task formulation, diagnostic analysis, and broad experimental study. We address the concerns below in Questions 1 to 4.

Response to Questions

[Question 1a.] How does OUC differ fundamentally from existing work on semantic overlap summarization, cross-document structure, sentence alignment, and contradiction detection? Could the authors clarify the specific contribution beyond unifying overlap, conflict, and unique-information extraction in a single framework?

We see two main differences from prior work. (i) Contradiction detection, semantic similarity, sentence alignment, and cross-document structure usually assume that the relevant units have already been identified or paired. The OUC extraction task instead starts from two complete narratives and requires the model to discover and extract all sentence pairs that satisfy a target relation. (ii) Semantic overlap summarization is inherently compressive: its goal is to summarize what is shared, so multiple source sentences may be merged, omitted, or abstracted into a shorter output. On the other hand, OUC asks a different question: whether a model can extract the complete set of underlying sentence-level relations themselves, including shared, conflicting, and source-specific information, without collapsing them into a summary. This distinction is important because recognizing a relation for a given pair is much easier than discovering all such pairs from full narratives. The contribution is therefore not simply to combine three relation types, but to formulate and benchmark exhaustive extraction of relation sets directly from complete narratives. We will clarify this distinction more clearly in the revised paper.

[Question 1b.] In particular, what capabilities or research questions does OUC enable that cannot be adequately addressed by existing tasks and benchmarks?

We thank the reviewer for this question. The structured OUC representation enables the study of questions that are difficult to address with existing pairwise or summarization benchmarks. In particular, it allows us to examine whether a model can explicitly separate what multiple sources agree on, where they conflict, and what each source contributes uniquely before those sources are synthesized. This is especially relevant to real-time decision support, where information from multiple sources may arrive within a limited time window and may contain both confirmed and disputed claims. The extracted OUC sets can guide the synthesis process by preserving which information is jointly supported, which remains conflicting, and which is reported by only one source, rather than merging everything into a single account. The resulting synthesized narrative can therefore give a decision-maker a clearer view of what information is relatively reliable and where uncertainty or disagreement still remains before taking action. The same OUC structure can also serve as a reference for evaluating whether that synthesized narrative faithfully preserves the underlying agreement, disagreement, and source-specific information.

[Question 2a.] The abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, which sum to approximately 1.10 million instances. Could the authors explain this discrepancy and clarify the counting conventions used?

Thank you for pointing out this discrepancy. The 140K figure in the abstract is a reporting error; the correct total is approximately 1.1M OUC instances across 22,558 narrative pairs, and we will correct this in the revision.

[Question 2b.] In addition, given that the complete-set human evaluation covers 100 narrative pairs and uses one annotator, what evidence supports the completeness of the automatically curated annotations, particularly for implicit conflicts and semantically related clauses with different wording?

Thank you for raising this concern. We do not claim that the automatically curated annotations are guaranteed to include every possible valid relation. Exhaustive annotation is inherently difficult in this task because annotators must understand both narratives in full and identify all valid cross-narrative pairs, including implicit relations and cases expressed with very different wording. To reduce this risk, we use explicit annotation guidelines and two complementary human evaluations. A three-annotator study over 120 narrative pairs checks the validity and consistency of the automatically curated instances, while a separate 100-pair complete-set study requires an annotator to independently extract the full OUC sets from the original narratives rather than only verify existing candidates. The strong agreement in the first study and high recall in the second provide substantial evidence that the curation pipeline captures the large majority of valid relations. At the same time, we acknowledge that a small number of difficult relations may still be missed and will explicitly note this residual annotation incompleteness as a limitation in the revised paper.

[Question 3a.] The paper uses ROUGE-L-based matching with a threshold of 0.6 as the primary evaluation protocol. How do the authors justify this metric for evaluating semantic overlap and conflict relations, where lexical similarity does not necessarily imply semantic equivalence or contradiction?

We believe there may be a misunderstanding about the role of ROUGE-L in our evaluation. The model is explicitly instructed to extract the exact sentence or sentence pair from the input narratives, so ROUGE-L is not used to decide whether two sentences are semantically overlapping or conflicting. That relation is already defined by the gold annotation. Instead, ROUGE-L is used only to check whether the model selected the correct gold sentence(s), while allowing small surface-form differences. For Overlap and Conflict, we compute ROUGE-L on both sides of the predicted pair and use the minimum score with a threshold of 0.6. This ensures that both extracted sentences must sufficiently match the corresponding gold pair.

[Question 3b.] Have the authors compared the reported results with semantic matching metrics or human-judgment-based evaluation, and do the main model rankings remain stable under alternative evaluation protocols?

We do not believe that a generic semantic similarity metric would be better aligned with this extraction objective. In Conflict, two sentences can be highly similar while differing in only one crucial detail, such as a number, entity, or polarity. In Overlap, several sentences may discuss the same topic and exhibit high semantic similarity even though they are not a correctly aligned pair. A semantic matcher could therefore reward an incorrect extraction, and choosing a single similarity threshold that reliably separates the correct sentence from merely related sentences would itself be difficult. For the same reason, Human-based matching would also add limited value here because the outputs are already defined at the sentence level and the model is asked to extract those source sentences directly rather than generate a free-form answer. For these reasons, we use ROUGE-L as the primary matching criterion and will clarify the rationale for this evaluation in the revised paper.

[Question 4a.] The benchmark is split at the narrative-pair level using a random 75/5/20 partition. Can the authors clarify whether duplicate or near-duplicate source documents, overlapping events, and closely related narratives are prevented from crossing the training, validation, and test splits?

We control this during sampling from the original datasets, before creating the train/dev/test partitions. Each sampled example retains the unique identifier of its original source, and we do not sample multiple examples associated with the same source ID. In addition, PeerSum contributes only one narrative pair per paper, WCEP factual news only one article pair per event instance, and the AllSides narrative pairs are unique. Therefore, once the sampled examples are partitioned, the same underlying paper, event, or source document cannot appear in more than one split. We will make this source-level separation more explicit in the revised paper, since Section 4.1 currently describes these sampling choices mainly as part of data collection rather than as a safeguard against train–test leakage.

[Question 4b.] Furthermore, how much of the observed improvement from SFT over DPO and GRPO can be attributed to the learning objective rather than differences in computational budgets and optimization configurations? Additional evidence on source-level separation and controlled comparisons would help establish the robustness of the reported conclusions.

Thank you for raising this point. We cannot attribute the full improvement of SFT over DPO and GRPO to the learning objective alone, because the methods are not strictly matched in computational budget or optimization configuration. However, we believe the learning objective contributes meaningfully to the observed gap because OUC requires predicting a structured set of exact sentence-level targets, and SFT directly supervises this complete target output. In contrast, DPO provides preference-level supervision between complete outputs, whereas GRPO optimizes a scalar reward over the entire extraction, providing less direct feedback on individual missing or incorrect pairs. For each method, we use optimization choices and hyperparameter ranges commonly adopted in prior work. Further tuning could improve DPO or GRPO, but these methods already require substantial computation in our experiments, and exhaustive compute-matched tuning would require significantly greater resources. We therefore interpret the results as evidence that the learning objective is an important factor behind SFT’s stronger performance under the configurations we study, without claiming that it is the only factor.

Reviewer_UTFNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Overlap–Unique–Conflict (OUC) extraction, which requires finding aligned overlapping and conflicting sentence/clause pairs, as well as information unique to either of two full narratives. It constructs an automatically annotated, partially human-validated benchmark of 22,558 narrative pairs from peer reviews, factual news, and political discourse. The authors evaluate 14 open-source LLMs (0.6B–35B) using prompting, analyze relation classification versus candidate alignment and full extraction, and compare SFT, DPO, and GRPO for smaller models. They find that conflict extraction is substantially harder than unique-information extraction and that task-specific SFT substantially improves performance.

Critical strengths
  1. The motivation of this paper is intuitive. Traditional NLI tasks typically focus on identifying semantic relationships between predefined text pairs. In contrast, the proposed OUC task requires models to directly discover, align, and extract relevant information from two complete narratives. By integrating Overlap, Unique, and Conflict relations into a unified cross-text extraction framework, this task extends the scope of traditional text relation understanding and offers meaningful research value.

  2. The experiment is comprehensive. The authors evaluate 14 open-source language models of different sizes and architectures, comparing multiple prompting strategies, including Zero-shot, Few-shot, and Chain-of-Thought, as well as training methods such as SFT, DPO, and GRPO.

Critical weaknesses
  1. The reliability and annotation granularity of the benchmark are worth considering. First, the reported instance counts are inconsistent, and reusing annotation models for validation may introduce correlated errors. Second, annotation completeness is assessed on only 100 narrative pairs by a single annotator, raising concerns about reliability. Third, sentence-level prompts may fail to capture multiple relations within a sentence, despite the task's emphasis on clause-level extraction.

  2. Explaining the performance bottlenecks of the model still requires more substantial experimental evidence. Although the diagnostic experiments in Table 2 reveal substantial performance gaps, it remains unclear whether the poor OUC extraction performance primarily stems from difficulties in discovering and aligning relevant information or from limitations of direct generative extraction itself. An end-to-end baseline combining embedding retrieval and NLI classification would help distinguish these factors and strengthen the authors' conclusions.

Questions for authors
  1. How is the ~140K instance count reconciled with the 1,098,899 OUC entries in Appendix Table 4? Precisely what is the gold annotation unit, and how are sentences with multiple independently matchable propositions represented? Please provide a worked example of sub-sentence overlap and conflict coexisting with unique content.

  2. Could the authors report an end-to-end system using automatic sentence splitting, Qwen3-Embedding top-k retrieval, and a pairwise NLI/LLM classifier with a one-to-one or many-to-many matching step, evaluated under the same macro-F1 protocol? Such a comparison would help clarify whether the poor performance on OUC extraction primarily stems from difficulties in discovering and aligning relevant information or from limitations inherent in direct generative extraction.

  3. Section 4.2 excludes narrative pairs lacking validated Overlap or Conflict. How many pairs are removed for each reason, and what happens to model precision and macro F1 on a randomly sampled, unfiltered evaluation set that includes zero-conflict and zero-overlap cases? How are duplicate source events/documents prevented from crossing train/test partitions?

  4. How were positive and negative candidate pairs sampled for the relation-recognition experiment in Table 2, with what class priors and difficulty controls? How sensitive is the gap to negative sampling and the number of candidate sentences?

6 comments

We thank the reviewer for the positive assessment of the task motivation and the comprehensiveness of our experiments. We address the concerns below in Questions 1- 4.

Response to Questions

[Question 1a.] How is the ~140K instance count reconciled with the 1,098,899 OUC entries in Appendix Table 4?

Thank you for pointing this out. This is a reporting error in the abstract. The correct benchmark contains approximately 1.1M OUC instances across 22,558 narrative pairs, and we will correct this in the revised version.

[Question 1b.] Precisely what is the gold annotation unit, and how are sentences with multiple independently matchable propositions represented?

Thank you for raising this point. The gold annotation unit in our benchmark is the complete sentence rather than an atomic proposition. We intentionally use sentence-level units because proposition-level extraction introduces an additional segmentation problem: the same sentence can often be decomposed into different numbers and boundaries of claims, and different LLMs may produce different but semantically valid decompositions. This makes it difficult to define a consistent gold representation and reliably compare predictions across models. Also, during our manual validation, we did not encounter cases where different propositions within the same sentence clearly required separate OUC labels. We nevertheless recognize that such cases are possible, particularly in other narrative domains or when the relevant OUC boundary falls below the sentence level. Our current benchmark does not separately represent these sub-sentence relations, and we will clarify this limitation in the revised paper.

[Question 1c.] Please provide a worked example of sub-sentence overlap and conflict coexisting with unique content.

We appreciate this suggestion. In our manual inspection, we did not observe a clear benchmark example in which a single sentence simultaneously required separate Overlap, Conflict, and Unique labels at the sub-sentence level. Since our benchmark is explicitly constructed and validated using complete sentences as the annotation unit, we would prefer not to introduce a hypothetical example that is not representative of the annotated data. We will clarify in the revision that such mixed proposition-level cases are possible in principle, and that developing a finer-grained proposition-level annotation and evaluation protocol would be an important direction for future work.

[Question 2.] Could the authors report an end-to-end system using automatic sentence splitting, Qwen3-Embedding top-k retrieval, and a pairwise NLI/LLM classifier with a one-to-one or many-to-many matching step, evaluated under the same macro-F1 protocol?.......

Following the reviewer’s suggestion, we added an end-to-end retrieval-and-classification baseline. We first split each narrative into sentences and use Qwen3-Embedding-0.6B to retrieve the top-k candidates from the opposite narrative. Each retrieved pair is then classified as Overlap, Conflict, or Invalid using either Qwen-3-8B or Gemma-4-31B under zero-shot prompting. Since our benchmark allows a single sentence to correspond to multiple sentences in the opposite narrative, we retain all pairs classified as valid rather than enforcing a one-to-one match. Unique-A and Unique-B are derived when no retrieved counterpart is classified as Overlap or Conflict. We evaluate both k=1 and k=5 using the same macro-F1 score as in our main experiments.

ClassifierkOverlap F1Conflict F1Unique-A F1Unique-B F1
Qwen-3-8B132.1930.4737.1549.37
Qwen-3-8B515.9930.3819.0838.11
Gemma-4-31B148.4534.7859.8765.27
Gemma-4-31B541.9235.1153.4660.28

The results show that retrieval alone does not resolve the difficulty of OUC extraction. At k=5, the retriever already recovers 92.48% of gold Overlap pairs and 79.97% of gold Conflict pairs, yet the final end-to-end scores remain substantially below direct Gemma-4-31B extraction. The main difficulty lies in correctly filtering the retrieved candidates. For example, with Gemma-4-31B at k=5, Overlap recall after classification remains high at 89.53%, but precision drops to 25.54%, indicating that many incorrect candidates are still retained as valid relations. Increasing k therefore improves candidate coverage, but also introduces a much larger set of candidates that must be filtered accurately.

Overall, these results suggest that the challenge is not limited to finding potentially relevant sentence pairs; deciding which retrieved pairs actually express Overlap or Conflict also remains difficult. We nevertheless view this pipeline mainly as a diagnostic baseline. Our primary goal in this work is to probe whether open LLMs can directly discover, align, and extract OUC relations from complete narratives. Adding a separate embedding-based retrieval stage changes the problem into a multi-stage system that requires many additional cross-narrative comparisons and inference calls. Such a system could certainly be improved further with a separately fine-tuned relation classifier, or with stronger retrieval, but optimizing that architecture would address a different question from the one we study here. We therefore use this baseline to test whether a simple retrieval-based decomposition is sufficient to explain or close the observed performance gap, rather than using it as the main formulation for OUC extraction.

[Question 3a.] How many narrative pairs were removed by the filtering step?

Thank you for asking for this breakdown. After validation, we removed narrative pairs for which Overlap was empty, Conflict was empty, or both were empty. Specifically, 1,883 pairs had no valid Conflict relation, 629 had no valid Overlap relation, and the remaining removed pairs had neither relation. This reduced the dataset from approximately 25.3K narrative pairs to 22,558 pairs. We will report this breakdown explicitly in Section 4.2 of the revised paper.

[Question 3b.] What happens to model precision and macro F1 on an unfiltered evaluation set that includes zero-Conflict and zero-Overlap cases?

Thank you for suggesting this analysis. To examine the effect of retaining zero-relation cases, we sampled 500 pairs from the pre-filter pool and evaluated Gemma-4-31B using the same ROUGE-L matching protocol as in our main experiments (threshold 0.6). We compare performance before and after validation, where unsupported relations are removed, and some samples consequently have an empty gold Overlap or Conflict set.

Gold annotationsOverlap POverlap F1Conflict PConflict F1
Before validation52.3856.1718.2316.36
After validation50.3749.525.424.70

The drop is driven largely by the model continuing to predict relations when the validated gold set is empty. After validation, 446 of the 500 samples have no Conflict relation, but Gemma-4-31B returns an empty set for only 112 of them, producing at least one spurious Conflict pair in the remaining 334 cases. Similarly, 70 samples have no Overlap relation, but the model abstains in only 15 and predicts at least one Overlap pair in the other 55. This substantially reduces precision and macro F1, particularly for Conflict. Importantly, this does not change our main finding: models already struggle to discover and extract valid Overlap and Conflict relations from complete narratives, and the unfiltered analysis shows that they also struggle to abstain when no valid relation exists. We will report this complementary analysis and clarify the role of the filtering step in the revised paper.

[Question 3c.] How are duplicate source events/documents prevented from crossing train/test partitions?

Thank you for raising this point. We control for source leakage before creating the train/dev/test splits. During sampling, each example retains the unique identifier of its original source, and we do not sample multiple examples associated with the same source ID. In addition, PeerSum contributes only one narrative pair per paper, WCEP factual news only one article pair per event instance, and the AllSides narrative pairs are unique. Therefore, the same underlying source cannot contribute examples to more than one partition. We will make this source-level separation more explicit in the revised paper.

Reviewer_83vnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Overlap–Unique–Conflict (OUC) extraction, a cross-narrative task that requires identifying overlapping and conflicting sentence pairs, as well as information unique to each of two narratives. The authors construct a silver-standard benchmark of 22,558 narrative pairs covering peer reviews, factual news, and political news, using an LLM-assisted annotation pipeline with additional human validation. The paper evaluates 14 open-source LLMs under zero-shot, few-shot, and chain-of-thought prompting. Its central finding is that models perform substantially better at recognizing semantic relations between given sentence pairs than at exhaustively extracting such relations from complete narratives. The authors further compare SFT, DPO, and GRPO on smaller models, finding that SFT provides the most consistent improvements. A fine-tuned Qwen-3-8B achieves 59.98% F1 on Overlap and 50.82% on Conflict, outperforming several larger models without task-specific fine-tuning.

Critical strengths
  1. A practically relevant task formulation with a reasonably broad empirical study. Moving from predefined sentence-pair classification to exhaustive cross-narrative extraction addresses an important limitation of conventional NLI-style evaluations. The joint consideration of overlapping, conflicting, and narrative-specific information is relevant to multi-document analysis, where the information requiring comparison is not known beforehand. The evaluation across 14 models and three discourse domains makes the empirical observations potentially useful beyond a single model family. The authors also make a meaningful effort to validate their automatically curated benchmark through human annotation.

  2. The diagnostic experiments and supervision results provide useful empirical insights. The separation of relation recognition, pair alignment, and end-to-end extraction is a useful starting point for understanding where model performance deteriorates. The large recognition-to-extraction performance gap, particularly for smaller models, is an interesting observation. Moreover, the comparison of SFT, DPO, and GRPO shows that task-specific supervision can substantially improve extraction performance. These results suggest that scaling model parameters is not the only route to improving cross-narrative extraction.

Critical weaknesses
  1. The benchmark construction and annotation granularity raise fundamental questions about validity and representativeness.

The paper claims to extract all overlapping, conflicting, and unique information, but the operationalized task largely uses complete sentences as extraction units (Sections 3–4 and Appendix A.9). A sentence may simultaneously express a claim that overlaps with the other narrative and an additional claim that is unique. Under the current definition, a sentence participating in an Overlap or Conflict pair is excluded from Unique extraction, potentially losing unique information embedded within that sentence. It is therefore unclear whether the benchmark actually measures exhaustive information extraction at the proposition level.

More importantly, Section 4.2 explicitly excludes narrative pairs whose validated Overlap or Conflict set is empty. Consequently, the benchmark does not adequately test whether a model can correctly return an empty set of conflicts or overlaps—an important capability in real-world applications. This selection procedure may introduce a substantial distributional bias, particularly for Conflict extraction.

The LLM-based curation pipeline also creates a potential annotation-completeness problem: relations missed by the candidate extractors cannot be recovered by subsequent validation, and errors in the validated pairs can propagate into Unique annotations. Although the human validation is encouraging, the fully extracted human subset comprises only 100 narrative pairs and relies on one annotator. It does not fully resolve concerns about annotation completeness across the entire benchmark.

There are also concrete cases where the distinction between contradiction and temporal updates appears ambiguous. For example, Appendix Table 17 labels reports of no confirmed flood-related deaths versus subsequently confirmed fatalities as conflicting, although both claims could be valid at different reporting times. A more explicit treatment of temporal context is needed for factual news.

These issues are consequential because the benchmark itself is the paper's primary contribution. If the reference annotations do not faithfully represent the intended information relations, both the reported model performance and conclusions about extraction difficulty become harder to interpret.

  1. The main bottleneck claim is insufficiently isolated, and important end-to-end baselines are missing.

Section 6.2 compares relation understanding using provided sentence pairs, alignment using ground-truth candidate sentences, and extraction from complete narratives. However, these settings differ not only in the capability being tested but also in their search spaces, output requirements, and potentially the distributions of positive and negative examples. The construction and class balance of the relation-recognition evaluation are insufficiently specified. Consequently, the F1 differences are suggestive but do not rigorously establish which component is primarily responsible for the degradation in end-to-end performance.

A particularly important omission is a competitive modular baseline combining sentence segmentation, candidate retrieval, and pairwise relation classification. The authors already report Recall@5 of 92.48% for Overlap and 79.97% for Conflict using an embedding retriever. These results make a retrieval-and-classification pipeline a natural baseline, yet its end-to-end performance is not evaluated. Without this comparison, it is difficult to determine whether the benchmark fundamentally requires task-specific generative fine-tuning or whether a relatively simple decomposition could close much of the gap.

Similarly, the comparisons between a fine-tuned 8B model and larger prompting-only models conflate model capacity with access to task-specific supervision. They demonstrate the value of supervision, but not necessarily superior underlying extraction capability at smaller scales.

There are also reproducibility concerns. The abstract reports approximately 140K OUC instances, while Appendix Table 4 lists 179,790 Overlap, 48,857 Conflict, 450,349 Unique-A, and 419,903 Unique-B instances, totaling 1,098,899. Additionally, the Unique-B instructions in Appendices A.11.2 and A.11.4 appear to reference Narrative B where Narrative A should be used. If these instructions were executed as written, the prompting comparisons may be affected.

Overall, the paper provides interesting descriptive evidence, but the central diagnostic conclusions and comparative claims require stronger controls and baselines before they are fully convincing.

Questions for authors

Q1. Benchmark representativeness and annotation granularity. Why were narrative pairs with no validated Overlap or Conflict relations removed? Can the authors report results on the original unfiltered pairs, including examples with genuinely empty relation sets? How does the annotation scheme handle a sentence containing both overlapping and unique propositions? I would also appreciate clarification of the discrepancy between the approximately 140K instances claimed in the abstract and the more than 1M instances reported in Appendix Table 4.

Q2. Reliability of Conflict annotations. How are temporal updates distinguished from genuine contradictions in factual news? For example, how should the conflicting fatality reports in Appendix Table 17 be interpreted when they may refer to different reporting times? Were the annotators in the 100-pair exhaustive validation study blind to the automatically generated labels? A targeted manual audit of temporally ambiguous Conflict cases would materially increase my confidence in the benchmark.

Q3. Bottleneck isolation and missing baselines. How were negative examples sampled for the relation-understanding experiments in Section 6.2, and what were their class distributions? Are the F1 scores directly comparable across the three diagnostic settings? More importantly, can the authors provide an end-to-end baseline using the existing embedding retriever followed by a cross-encoder or NLI classifier to label candidate pairs? Given the reported Recall@5, this seems essential for substantiating the main bottleneck claim.

Q4. Prompt reproducibility. The documented Unique-B instructions in Appendices A.11.2 and A.11.4 appear to contain incorrect references to Narrative B. Were the actual experimental prompts identical to those shown in the appendix? If so, could the affected prompting results be rerun with corrected instructions? If these are transcription errors, please provide the exact executed prompts.

6 comments

We thank the reviewer for the constructive feedback and for recognizing the relevance of the task and the value of our empirical analysis. We address the concerns below through Q1–Q4.

Response to Questions

[Question 1a–1b.] Why were narrative pairs with no validated Overlap or Conflict relations removed, and can the authors report results on the original unfiltered pairs, including genuinely empty relation sets?

We thank the reviewer for raising this point. We originally removed pairs without validated Overlap or Conflict relations because the benchmark was designed to evaluate extraction in the presence of cross-narrative relations. Determining that no such relation exists is a related but distinct case that was outside the scope of the original benchmark design. However, as the reviewer suggested, we examined this case directly by sampling 500 pairs from the previously excluded pool and evaluating whether the models correctly returned an empty set when the corresponding gold relation was empty. Among these samples, 430 had no Conflict relation, 54 had no Overlap relation, and 16 had neither Overlap nor Conflict. We evaluated Gemma-4-31B and Qwen-3-8B models using the same zero-shot prompting setup. We found that Gemma-4-31B correctly returned an empty set for 112/446 Conflict cases (25.11%) and 15/70 Overlap cases (21.43%), while Qwen-3-8B did not return an empty set in any of these cases. These results illustrate that models often generate relation pairs even when no valid relation exists. This is consistent with the precision errors observed in the main extraction task, where models may recover correct pairs while also producing unsupported ones. One possible reason is the tendency of autoregressive LLMs to generate plausible outputs rather than abstain. We will clarify the rationale and criteria for excluding empty-relation pairs more explicitly in the revised paper.

[Question 1c.] How does the annotation scheme handle a sentence containing both overlapping and unique propositions?

Thank you for raising this point. In our annotation scheme, the complete sentence is treated as the extraction unit rather than being decomposed into atomic propositions. We chose this because proposition-level extraction would introduce an additional segmentation problem: the same sentence can often be divided into different numbers or boundaries of claims, making it difficult to define a consistent gold representation and reliably evaluate outputs across different LLMs. We therefore assign the OUC relation at the sentence level based on the information expressed by the sentence. Besides that, during manual validation, we did not observe cases where different propositions within the same sentence clearly required different OUC labels. However, such cases are possible, and the current formulation would not separately represent these finer-grained relations. We will clarify this design choice and acknowledge proposition-level OUC extraction as a limitation and future direction.

[Question 1d.] Why does the abstract report approximately 140K instances while Appendix Table 4 reports more than 1M?

Thank you for catching this. This is a reporting error in the abstract. The correct total is approximately 1.1M OUC instances across 22,558 narrative pairs, and we will correct it in the revised version.

[Question 2a.] Reliability of Conflict annotations. How are temporal updates distinguished from genuine contradictions in factual news?....

Thank you for raising this distinction. In our current formulation, Conflict is defined based on the information presented in the two narratives: if they report incompatible states of the same fact or aspect, we extract the corresponding sentence pair as a Conflict. The task does not additionally determine whether that disagreement could later be reconciled through temporal progression or newly available evidence. The fatality example in Appendix Table 17 illustrates this point: one narrative reports that no flood-related fatalities had been confirmed, while the other reports confirmed fatalities. These are incompatible reported states of the same fact and are therefore captured as a Conflict under the current definition. Determining whether this difference reflects a later temporal update would require additional temporal annotation beyond the current OUC schema, which is outside the scope of the present task. Explicitly separating temporal updates from other conflicts would therefore require a finer-grained relation schema, which we will clarify as a limitation and future direction.

[Question 2b.] Were the annotators in the 100-pair exhaustive validation study blind to the automatically generated labels?

Yes, for the 100-pair exhaustive validation, the annotator independently extracted the complete OUC sets from the narrative pairs without seeing the automatically curated OUC labels. The automatic and human sets were compared only after the human extraction was completed. Thus, the reported agreement reflects independent extraction rather than verification of the automatically generated labels.

[Question 3a.] How were negative examples sampled for the relation-understanding experiments, and what were their class distributions?

Thank you for asking for clarification. For the relation-understanding diagnostic, we curated negative examples from model-generated candidate pairs rather than relying only on random sentence mismatches. Specifically, we used a smaller LLaMA-3.2-1B model, which is not part of our main model evaluation, to generate Overlap and Conflict candidates for all 4,513 test narrative pairs. These candidates were then matched against the gold pairs using the same ROUGE-L criterion as in our main evaluation (sentence-min, threshold 0.6). Any predicted pair that failed to match a gold pair for the corresponding relation was treated as an invalid candidate.

TaskPositivesNegativesTotal
Overlap35,806 gold Overlap12,357 invalid Overlap + 8,502 gold Conflict = 20,85956,665
Conflict10,943 gold Conflict9,890 invalid Conflict + 4,513 sampled gold Overlap = 14,40325,346

More specifically, for Overlap, the positive examples are gold Overlap pairs. The negative set contains two types of examples: (1) invalid Overlap candidate pairs that do not match any gold Overlap pair, and (2) gold Conflict pairs. For Conflict, the positive examples are gold Conflict pairs, while the negatives consist of (1) invalid Conflict candidate pairs and (2) one randomly sampled gold Overlap pair from each narrative pair, using a fixed seed of 42. We include the opposite OUC relation as a negative so that the diagnostic tests whether the model can distinguish Overlap from Conflict, rather than only separating valid pairs from invalid candidates. We will make this construction and the resulting class distributions explicit in the revised paper.

[Question 3b.] Are the F1 scores directly comparable across the three diagnostic settings?

Although we used the same F1 metric across the three diagnostics, the scores reflect progressively different requirements. Relation understanding tests classification of a provided pair; pair alignment additionally requires identifying the correct correspondence among candidate sentences; and end-to-end extraction further requires discovering those candidates from the full narratives. We designed the diagnostics this way to progressively introduce the components required for full OUC extraction. Therefore, we use the score differences to identify where performance begins to degrade, rather than as an exact decomposition of each component's contribution.

Comments on the paper

1 comment

Dear Reviewers, as the rebuttal deadline is approaching, please actively engage in the discussion based on the authors' responses. Thank You!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.