acceptodds
Under review as a conference paper at ICLR 2027

ROEC: Record-Oriented Evidence Construction for Document-Level Structured Information Extraction

Abstract

Targeted structured information extraction organizes information according to a predefined schema. Extending this setting to the document level is challenging because target information is often sparse and distributed across the document, while non-target content can interfere with extraction. Beyond recovering relevant information, extracted outputs must satisfy predefined requirements and remain supported by the source text. We propose Record-Oriented Evidence Construction (ROEC), which replaces one-step record extraction with Two-Stage Structured Information Extraction. ROEC first extracts core fields and then completes the associated contextual fields, while retaining supporting source sentences throughout extraction. It further incorporates Record-Level Refinement to improve record granularity and conformity with task requirements and reduce redundancy, and Claim Verification and Support Scoring to evaluate extracted claims against retained supporting source sentences and derive a score reflecting the degree of source-text support. Experiments on agricultural research papers show that ROEC improves Relaxed and Strict F1 by 11.83 and 5.86 points over the strongest evaluated baseline under each criterion. The two-stage configuration improves recall and F1 over one-step extraction, while Record-Level Refinement further reduces false positives and increases true positives. Human assessment indicates that the support score reflects differences in source-text support and helps prioritize outputs for manual review of source-support errors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
information extraction, document-level extraction, large language models, evidence grounding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6hTYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the engineering challenge of building ROEC, an implemented system for extracting structured findings from research papers. Its task is defined specifically for the agriculture domain: each record captures an intervention-outcome relation, or a factor-outcome relation, in agricultural practice and research, along with applicable context such as crop, location, research methodology, and so on. This is a classic information-extraction problem: information about a finding may be scattered across a document, in natural language, yet must be brought together into a coherent record, placed under the right fields, and clearly supported by the source text. ROEC uses an LLM in a typical two-stage process: first extracting core relations, and then completing their contextual fields in a round of deeper reasoning. It further refines records in order to split combined findings, merge duplicates (e.g., entity resolution), filter invalid results, and assign confidence scores by checking claims against retained source sentences (using NLI). Experiments compare the system with several extraction baselines and report improved extraction performance.

Critical strengths
  1. The paper addresses an important, practically relevant problem. As research activity and publication volume continue to grow, it becomes increasingly valuable to systematically collect findings from a large and expanding literature in a uniform format. This is true for many fields, especially agriculture, where research findings can inform decisions with broad societal consequences.

  2. The paper is well written and well organized. Its presentation is clear, making it easy to read and follow.

Critical weaknesses
  1. While the paper addresses an interesting engineering challenge, the contribution is primarily the design and evaluation of a standard extraction pipeline. The paper does not develop a sufficiently deep conceptual, methodological, or analytical results to constitute a substantial research contribution.

  2. Information extraction from natural text is a longstanding, classic problem, and the paper does not clearly establish what is novel in its approach. Related work in biomedical and medical research, for instance, has addressed the challenge of extracting and synthesizing findings from a rapidly growing literature, and dealt with very similar challenges. The database community has also developed general approaches and principles for information extraction, including IBM’s SystemT and the formal framework of document spanners. The paper does not convincingly explain how ROEC advances these lines of work or what new insight its specific combination of techniques provides.

  3. The paper focuses on a domain-specific information-extraction task in agriculture. Although it uses LLMs, it does not make a clear contribution to machine learning or offer insights into learning methods. Its connection to ICLR’s research focus is therefore weak.

Questions for authors

None.

Reviewer_QfEPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is about extracting information from natural language text into a structured format. In particular, the paper proposes a pipeline for extracting structured information from scientific articles in agricultural science. The target task is extracting Agricultural Evidence Records (AER), which have two types: (1) Intervention-Effect: "describing outcomes associated with an intervention, management practice, technology, or policy"; and (2) Factor-Impact: "describing outcomes associated with an agricultural problem, risk, constraint, or factor". It is a multi-stage pipeline that consists of prompting, rule-based components and NLI scoring (no fine-tuning or training is involved). The authors annotated a gold-standard dataset and evaluated their method on this data with metrics that combine token-overlap and embedding similarity. They show that, according to their metrics, the method outperforms simpler baselines.

Critical strengths
  • This paper could be useful for domain-specific information extraction. For the domain, the authors contribute: (1) a new method for extracting information from agricultural science articles; (2) a dataset for evaluation, which the authors intend to release if the paper is accepted.
Critical weaknesses
  • Paper scope. The paper proposes a method for information extraction for agricultural science. In the experiments, the authors test only on this domain, and do not test transfer to other domains. Therefore, the paper is limited to this niche subfield and provides no evidence of generality. Moreover, the paper seems to compile a pipeline from already existing parts, which makes it a heavily engineered method, and leaves the reader wondering what exactly can be learned from it?
  • Evaluation methodology. The authors propose new metrics that combine token-overlap scoring with semantic similarity, which are not validated manually. Therefore, I find it hard to interpret the results, as I don't know (1) what exactly they capture; (2) whether the metric is of sufficient quality?
Questions for authors

No further questions.

Reviewer_iLxfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Record-Oriented Evidence Construction (ROEC), a framework for document-level targeted structured information extraction from agricultural research papers. The task aims to extract Agricultural Evidence Records (AERs), each representing a specific intervention–effect or factor–impact finding together with its relevant contextual information. ROEC consists of three main components. First, Two-Stage Structured Information Extraction identifies core relations and then completes contextual fields using relevant source sentences. Second, Record-Level Refinement splits records containing multiple findings, merges duplicates, and removes invalid records. Third, Claim Verification and Support Scoring decomposes extracted records into verifiable claims, evaluates them against retained evidence sentences using a natural language inference model, and aggregates the results into an AER-level support score. The authors evaluate ROEC on a dataset of agricultural research papers, using 15 documents for development and 30 documents for testing. The test set contains 421 gold AERs annotated by domain experts. ROEC is compared with four baselines: Full, Chunk, Retrieval, and LangExtract. It achieves a Relaxed F1 of 45.05 and a Strict F1 of 21.98, improving over the strongest evaluated baseline by 11.83 and 5.86 percentage points, respectively. Ablation experiments show that two-stage extraction improves performance over one-step extraction, while record-level refinement provides further gains. A human assessment of 871 extracted records also indicates that the support score is associated with source-text support and can help prioritize outputs for manual review. The paper contributes a structured extraction framework and an empirical study of record construction and evidence-based support assessment, although the evaluation is currently limited to agricultural research papers.

Critical strengths
  1. A clear problem formulation and a coherent extraction framework. The paper addresses the challenge of associating information distributed across different sections of a document while ensuring that all fields in an extracted record refer to the same underlying finding. The two-stage extraction design separates core-relation identification from contextual completion, and the retention of supporting source sentences improves traceability. This provides a reasonable and practically relevant framework for document-level scientific information extraction.
  2. Empirical improvements supported by progressive ablation studies. ROEC achieves a Relaxed F1 of 45.05 and a Strict F1 of 21.98, improving over the strongest evaluated baselines by 11.83 and 5.86 percentage points, respectively. The ablation results further show that two-stage extraction improves performance over one-step extraction, while record-level refinement provides additional gains by reducing false positives and increasing true positives. The human assessment of the support score also suggests its potential to prioritize records for manual review. These results provide useful empirical support for the proposed framework within the evaluated setting.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework combines staged information extraction, context selection, record-level refinement, and NLI-based support assessment. Although this combination is reasonable, the paper does not sufficiently establish a distinct methodological contribution beyond integrating existing techniques. The two-stage extraction strategy is related to progressive field-filling approaches such as SudokuFill, while record refinement and claim-level verification build on established ideas in event individuation and factuality assessment. The paper would benefit from a clearer explanation of the specific methodological advance introduced by ROEC and evidence showing why its improvements cannot be explained primarily by additional LLM processing and refinement.

  2. Limited evaluation scale and domain coverage. The main evaluation uses 30 test documents containing 421 gold Agricultural Evidence Records, all drawn from agricultural research papers. Although the annotation process involves domain experts, this test set provides limited evidence about the generality of the proposed framework. The method may behave differently on other scientific disciplines, document structures, and extraction schemas. Consequently, the results support the usefulness of ROEC in the evaluated setting but do not yet establish its broader applicability to document-level structured information extraction.

  3. Insufficient comparison with strong and closely related baselines. The evaluation compares ROEC with Full, Chunk, Retrieval, and LangExtract. These baselines provide useful reference points, but the comparison does not sufficiently establish the advantage of ROEC over a broader range of strong document-level extraction methods, particularly approaches that progressively complete structured records or explicitly associate evidence across documents or sections. Without more direct comparisons with closely related methods, it remains difficult to determine whether the gains arise from a distinct methodological contribution or from combining multiple LLM calls and refinement stages.

  4. Lack of uncertainty estimates for the main results. The paper reports substantial improvements in Relaxed and Strict F1, but does not provide repeated-run variability, confidence intervals, or statistical significance tests for the principal comparisons. Since the evaluation involves LLM-based extraction and refinement, document-specific variation and model behavior may affect the measured gains. Without uncertainty estimates, the robustness of the reported improvements is difficult to assess, especially given the relatively small number of test documents.

  5. Limitations of the evaluation and record-matching protocol. The evaluation uses partial matching based on token-level similarity and embedding similarity, followed by threshold-based eligibility checks and one-to-one bipartite matching. Although this protocol accommodates variations in field wording, the results depend on the similarity thresholds and the definitions used to determine a valid match. The paper does not clearly report the numerical values of the Relaxed and Strict matching thresholds in the main evaluation description, making it harder to reproduce the reported results directly. Further validation of the thresholds and matching protocol would strengthen confidence that the measured improvements reflect better record extraction rather than evaluation-specific effects.

  6. Insufficient validation of the support score as a measure of record reliability. The proposed support score measures how strongly constructed claims are supported by retained source sentences. However, entailment between a claim and selected evidence does not necessarily establish that the entire record is correct, that all fields refer to the same finding, or that relevant information has not been omitted. The human evaluation demonstrates an association between the score and human-assessed source support, but the reported results also show that many records requiring review remain in the highest-score interval. The score is therefore more convincingly established as a review-prioritization signal than as a reliable detector of incorrect records.

  7. Insufficient isolation of individual design choices. The progressive ablation demonstrates improvements from two-stage extraction and record-level refinement. However, the framework also relies on several specific design choices, including field-specific lexicons, sentence-window selection, contextual candidate limits, similarity-based duplicate detection, and the aggregation of claim-level support scores. The paper does not fully isolate the effects of these individual choices. Additional ablations would help clarify which components are essential to the observed gains and whether simpler alternatives could achieve comparable performance with less complexity.

  8. Limited analysis of computational efficiency and practical trade-offs. ROEC involves multiple stages of LLM-based extraction and refinement, followed by NLI-based verification. While the appendix provides computational details for post-training and inference, the paper does not offer a sufficiently systematic end-to-end comparison of latency, computational cost, and resource requirements against the baselines. This omission makes it difficult to assess whether the observed improvements justify the additional processing overhead in practical scientific evidence extraction workflows.

Questions for authors
  1. How does ROEC differ technically from existing progressive information extraction methods, particularly SudokuFill? Could the authors clarify which specific component introduces a distinct methodological contribution, and provide evidence that the improvements are not primarily attributable to additional LLM calls or generic refinement?

  2. Given that the test set contains only 30 agricultural research papers, how confident are the authors that the reported improvements generalize beyond this particular sample? Can the authors provide uncertainty estimates or additional evaluation on independently sampled documents to establish the robustness of the main results?

  3. Why were the selected baselines considered sufficient to establish the effectiveness of ROEC? Can the authors provide a more direct comparison with closely related progressive extraction or structured information extraction methods, using the same schema, underlying LLM, and evaluation protocol where possible?

  4. What are the exact numerical values of the Relaxed and Strict matching thresholds, and how sensitive are the main results to these choices? Would the relative ranking of ROEC and the baselines remain stable under alternative matching thresholds or matching strategies?

  5. The human evaluation shows that the support score can prioritize records for review, but some records requiring review still receive high scores. Can the authors quantify how well the score distinguishes fully supported, partially supported, and unsupported records, for example using AUROC, precision–recall curves, or review performance at different score thresholds? How should the score be interpreted given that it measures support from retained evidence rather than overall record correctness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.