- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Overlap–Unique–Conflict (OUC) extraction, a task that requires a model to identify overlapping, conflicting, and narrative-specific information from two complete narratives. Unlike conventional relation classification settings, OUC requires models to discover relevant clauses, establish cross-narrative correspondences, classify their relations, and identify information unique to each narrative. The authors construct a benchmark of 22,558 narrative pairs from peer reviews, factual news, and political discourse using an LLM-assisted annotation pipeline with subsequent human validation. They evaluate 14 open-source LLMs ranging from 0.6B to 35B parameters under zero-shot, few-shot, and chain-of-thought prompting. Their diagnostic experiments separate relation understanding, clause alignment, and end-to-end extraction to investigate the sources of model errors. They further compare supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO), and examine the effect of supervision size and cross-domain generalization. The results show that overlap and conflict extraction are substantially more difficult than unique-clause extraction. Task-specific SFT improves performance considerably, enabling Qwen-3-8B to outperform larger models on several tasks. The paper provides a broad empirical study of cross-narrative extraction, although the novelty of the task relative to existing cross-document relation extraction work and the reliability of the automatically curated benchmark require further scrutiny.
Critical strengths
-
A useful task formulation and systematic analysis of extraction bottlenecks. The paper formulates OUC extraction as a task that requires models to identify overlapping, conflicting, and unique information directly from complete narratives. This goes beyond classifying predefined clause pairs and requires models to discover relevant information and establish cross-narrative correspondences. The diagnostic experiments separately examine relation understanding, pair alignment, and end-to-end extraction, providing useful evidence that the main difficulty lies not only in recognizing semantic relations but also in discovering and correctly aligning the relevant clauses. This analysis offers a meaningful perspective on the limitations of current LLMs.
-
Broad experimental coverage and evidence that task-specific supervision improves performance. The paper evaluates 14 open-source LLMs across multiple model families and sizes, compares zero-shot, few-shot, and chain-of-thought prompting, and investigates SFT, DPO, and GRPO. The additional supervision-scaling, cross-domain, and error analyses provide a relatively comprehensive empirical study. In particular, SFT substantially improves the performance of Qwen-3-8B, allowing it to outperform the much larger Gemma-4-31B baseline on Conflict, Unique-A, and Unique-B, while remaining competitive on Overlap. These findings provide useful evidence that task-specific supervision can improve cross-narrative extraction, even though the generality of the results and the novelty of the task formulation require further examination.
Critical weaknesses
-
The novelty of the OUC task is not sufficiently distinguished from closely related prior work. The paper presents OUC as a new task for extracting overlapping, conflicting, and unique information from complete narratives. However, prior work has already studied semantic overlap between alternative narratives, including the Semantic Overlap Summarization (SOS) task introduced by Bansal et al. (COLING 2022) and subsequent work on synthetic-data training and semantic evaluation. Cross-document structure theory and sentence-alignment research also address relations between information units from different documents. Although OUC extends these directions by jointly extracting overlap, conflict, and narrative-specific information, the paper does not sufficiently establish the conceptual and empirical boundaries between OUC and these existing tasks. In particular, it remains unclear whether the proposed formulation enables fundamentally new research questions or primarily combines established capabilities in a unified extraction setting. A more direct comparison with existing datasets, task definitions, and evaluation protocols is needed to substantiate the claimed novelty.
-
The benchmark has unresolved annotation-reliability and statistical-reporting issues. The benchmark is constructed primarily through an LLM-assisted pipeline, with candidate generation, model-based validation, and subsequent unique-clause extraction. Although the authors provide instance-level human validation, the complete-set evaluation covers only 100 narrative pairs and relies on a single annotator to extract the full set of relations. This provides limited evidence that the automatically curated annotations are exhaustive, particularly for subtle conflicts and implicit semantic correspondences. Moreover, the reported benchmark statistics are inconsistent: the abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, totaling approximately 1.10 million instances. The authors should clarify this discrepancy and establish the counting conventions. Until these issues are resolved, the reliability, completeness, and reproducibility of the benchmark remain difficult to assess.
-
The ROUGE-L-based evaluation is insufficiently justified for semantic relation extraction. The primary evaluation uses ROUGE-L similarity with a threshold of 0.6 to match predicted and reference clauses, followed by greedy one-to-one matching. However, lexical overlap does not necessarily indicate that two clauses express the same underlying fact, nor does it establish that a pair represents a genuine contradiction. Conversely, valid semantic correspondences can use substantially different wording. This concern is particularly important for the Conflict task, where two clauses can share many words while making incompatible claims, or discuss related topics without contradicting each other. Closely related work on semantic overlap summarization has already identified limitations of ROUGE and proposed SEM-F1 as a more suitable alternative for that task. Although the authors report threshold sensitivity and evaluate a human-annotated subset, they do not sufficiently establish that the chosen metric reliably measures semantic correctness across all four OUC categories. Additional evaluation using semantic matching and human-judgment-based comparisons would strengthen confidence in the reported model rankings and performance differences.
-
The experimental setup does not fully establish generalization or isolate the effects of post-training. The main benchmark is divided into training, validation, and test sets using a random 75/5/20 split at the narrative-pair level. The paper does not clearly establish whether duplicate or near-duplicate source documents, overlapping events, or closely related narratives are excluded across splits. Such overlap could inflate performance if models encounter related content during training and evaluation. The leave-one-domain-out experiments are useful, but they do not fully resolve potential leakage or establish generalization to unseen source collections and event distributions. In addition, the comparison between SFT, DPO, and GRPO involves substantially different computational costs and optimization procedures, making it difficult to attribute performance differences solely to the learning objectives. More explicit source-level deduplication, grouped split validation, and controlled comparisons of training budgets would be needed to support the broader claims about generalization and the relative effectiveness of the post-training methods.
Questions for authors
-
How does OUC differ fundamentally from existing work on semantic overlap summarization, cross-document structure, sentence alignment, and contradiction detection? In particular, what capabilities or research questions does OUC enable that cannot be adequately addressed by existing tasks and benchmarks? Could the authors clarify the specific contribution beyond unifying overlap, conflict, and unique-information extraction in a single framework?
-
The abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, which sum to approximately 1.10 million instances. Could the authors explain this discrepancy and clarify the counting conventions used? In addition, given that the complete-set human evaluation covers 100 narrative pairs and uses one annotator, what evidence supports the completeness of the automatically curated annotations, particularly for implicit conflicts and semantically related clauses with different wording?
-
The paper uses ROUGE-L-based matching with a threshold of 0.6 as the primary evaluation protocol. How do the authors justify this metric for evaluating semantic overlap and conflict relations, where lexical similarity does not necessarily imply semantic equivalence or contradiction? Have the authors compared the reported results with semantic matching metrics or human-judgment-based evaluation, and do the main model rankings remain stable under alternative evaluation protocols?
-
The benchmark is split at the narrative-pair level using a random 75/5/20 partition. Can the authors clarify whether duplicate or near-duplicate source documents, overlapping events, and closely related narratives are prevented from crossing the training, validation, and test splits? Furthermore, how much of the observed improvement from SFT over DPO and GRPO can be attributed to the learning objective rather than differences in computational budgets and optimization configurations? Additional evidence on source-level separation and controlled comparisons would help establish the robustness of the reported conclusions.
3 comments
[Question 2a.] The abstract describes approximately 140K OUC instances, whereas Appendix Table 4 reports 179,790 overlap pairs, 48,857 conflict pairs, 450,349 Unique-A clauses, and 419,903 Unique-B clauses, which sum to approximately 1.10 million instances. Could the authors explain this discrepancy and clarify the counting conventions used?
Thank you for pointing out this discrepancy. The 140K figure in the abstract is a reporting error; the correct total is approximately 1.1M OUC instances across 22,558 narrative pairs, and we will correct this in the revision.
[Question 2b.] In addition, given that the complete-set human evaluation covers 100 narrative pairs and uses one annotator, what evidence supports the completeness of the automatically curated annotations, particularly for implicit conflicts and semantically related clauses with different wording?
Thank you for raising this concern. We do not claim that the automatically curated annotations are guaranteed to include every possible valid relation. Exhaustive annotation is inherently difficult in this task because annotators must understand both narratives in full and identify all valid cross-narrative pairs, including implicit relations and cases expressed with very different wording. To reduce this risk, we use explicit annotation guidelines and two complementary human evaluations. A three-annotator study over 120 narrative pairs checks the validity and consistency of the automatically curated instances, while a separate 100-pair complete-set study requires an annotator to independently extract the full OUC sets from the original narratives rather than only verify existing candidates. The strong agreement in the first study and high recall in the second provide substantial evidence that the curation pipeline captures the large majority of valid relations. At the same time, we acknowledge that a small number of difficult relations may still be missed and will explicitly note this residual annotation incompleteness as a limitation in the revised paper.
[Question 3a.] The paper uses ROUGE-L-based matching with a threshold of 0.6 as the primary evaluation protocol. How do the authors justify this metric for evaluating semantic overlap and conflict relations, where lexical similarity does not necessarily imply semantic equivalence or contradiction?
We believe there may be a misunderstanding about the role of ROUGE-L in our evaluation. The model is explicitly instructed to extract the exact sentence or sentence pair from the input narratives, so ROUGE-L is not used to decide whether two sentences are semantically overlapping or conflicting. That relation is already defined by the gold annotation. Instead, ROUGE-L is used only to check whether the model selected the correct gold sentence(s), while allowing small surface-form differences. For Overlap and Conflict, we compute ROUGE-L on both sides of the predicted pair and use the minimum score with a threshold of 0.6. This ensures that both extracted sentences must sufficiently match the corresponding gold pair.
[Question 3b.] Have the authors compared the reported results with semantic matching metrics or human-judgment-based evaluation, and do the main model rankings remain stable under alternative evaluation protocols?
We do not believe that a generic semantic similarity metric would be better aligned with this extraction objective. In Conflict, two sentences can be highly similar while differing in only one crucial detail, such as a number, entity, or polarity. In Overlap, several sentences may discuss the same topic and exhibit high semantic similarity even though they are not a correctly aligned pair. A semantic matcher could therefore reward an incorrect extraction, and choosing a single similarity threshold that reliably separates the correct sentence from merely related sentences would itself be difficult. For the same reason, Human-based matching would also add limited value here because the outputs are already defined at the sentence level and the model is asked to extract those source sentences directly rather than generate a free-form answer. For these reasons, we use ROUGE-L as the primary matching criterion and will clarify the rationale for this evaluation in the revised paper.
[Question 4a.] The benchmark is split at the narrative-pair level using a random 75/5/20 partition. Can the authors clarify whether duplicate or near-duplicate source documents, overlapping events, and closely related narratives are prevented from crossing the training, validation, and test splits?
We control this during sampling from the original datasets, before creating the train/dev/test partitions. Each sampled example retains the unique identifier of its original source, and we do not sample multiple examples associated with the same source ID. In addition, PeerSum contributes only one narrative pair per paper, WCEP factual news only one article pair per event instance, and the AllSides narrative pairs are unique. Therefore, once the sampled examples are partitioned, the same underlying paper, event, or source document cannot appear in more than one split. We will make this source-level separation more explicit in the revised paper, since Section 4.1 currently describes these sampling choices mainly as part of data collection rather than as a safeguard against train–test leakage.
[Question 4b.] Furthermore, how much of the observed improvement from SFT over DPO and GRPO can be attributed to the learning objective rather than differences in computational budgets and optimization configurations? Additional evidence on source-level separation and controlled comparisons would help establish the robustness of the reported conclusions.
Thank you for raising this point. We cannot attribute the full improvement of SFT over DPO and GRPO to the learning objective alone, because the methods are not strictly matched in computational budget or optimization configuration. However, we believe the learning objective contributes meaningfully to the observed gap because OUC requires predicting a structured set of exact sentence-level targets, and SFT directly supervises this complete target output. In contrast, DPO provides preference-level supervision between complete outputs, whereas GRPO optimizes a scalar reward over the entire extraction, providing less direct feedback on individual missing or incorrect pairs. For each method, we use optimization choices and hyperparameter ranges commonly adopted in prior work. Further tuning could improve DPO or GRPO, but these methods already require substantial computation in our experiments, and exhaustive compute-matched tuning would require significantly greater resources. We therefore interpret the results as evidence that the learning objective is an important factor behind SFT’s stronger performance under the configurations we study, without claiming that it is the only factor.
We thank the reviewer for the positive assessment of the task formulation, diagnostic analysis, and broad experimental study. We address the concerns below in Questions 1 to 4.
Response to Questions
[Question 1a.] How does OUC differ fundamentally from existing work on semantic overlap summarization, cross-document structure, sentence alignment, and contradiction detection? Could the authors clarify the specific contribution beyond unifying overlap, conflict, and unique-information extraction in a single framework?
[Question 1b.] In particular, what capabilities or research questions does OUC enable that cannot be adequately addressed by existing tasks and benchmarks?