acceptodds
Under review as a conference paper at ICLR 2027

SDA: Separating Discovery from Annotation for LLM-Based Structured Extraction

Abstract

Structured extraction organizes semantic elements and their relations in free-form text into predefined records. Large language models (LLMs) support this task through in-context learning (ICL), yet often fall short of task-specific supervised models. Prior work addresses this gap through better task instructions and demonstration selection. Our analysis shows that demonstration patterns change predicted record counts for unchanged inputs, suggesting that examples may constrain the model's interpretation of source content. Inspired by this finding, we propose SDA, a structured extraction framework that separates source-based semantic discovery from demonstration-guided annotation. Specifically, the framework comprises three modules. Semantic Discovery constructs natural-language candidates through complementary source readings without labeled demonstrations. Candidate Projection uses relevant precedents from an annotation memory to normalize candidates and delimit admissible annotation choices. Joint Compilation jointly resolves labels and record selection while preserving candidate meanings and previously fixed fields. Extensive experiments on three structured extraction tasks across five datasets show that SDA with GPT-4o-mini outperforms the strongest evaluated same-backbone LLM baseline on each dataset, with an average gain of 20.78 F1 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large language models, information extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BSw4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that in few-shot LLM-based structured extraction, the number of records in the demonstrations shifts the number of records the model predicts even for fixed input text, suggesting that demonstrations bias what content is discovered rather than merely teaching annotation conventions. To address this, the authors propose SDA, which first discovers natural-language candidates from the source text without labeled examples (Semantic Discovery), then projects them onto the dataset schema using retrieved training precedents (Candidate Projection), and finally resolves labels and record merges with K examples (Joint Compilation). Experiments on three tasks and five datasets (ASQP on Rest15/Rest16, RTE on NYT/WebNLG, ED on ACE2005) with GPT-4o-mini show that SDA outperforms all same-backbone LLM baselines by 20.78 F1 on average, while still lagging supervised models and costing considerably more tokens and time. The paper also provides ablations on multi-view reading, projection evidence, and sensitivity to K and data size.

Critical strengths
  1. The motivating controlled analysis is a clean and useful observation. Holding the input text fixed and varying only the record cardinality of the demonstrations, the authors show that the predicted record count shifts systematically, and that single-record and multi-record inputs behave differently. The follow-up comparison between direct ICL and the label-free discovery stage is also informative. Whether or not the proposed framework fully solves it, the problem itself is clearly demonstrated and is worth attention.

  2. The framework is technically complete and presented clearly. The three stages have well-defined inputs and outputs , the overview figure and case studies make the pipeline concrete, and the prompt templates, example-selection scoring, and evaluation rules are provided in the appendices, with a reproducibility statement promising code, prompts, and per-sentence annotations. This is more transparent than most prompt-based submissions.

  3. The analysis is fairly thorough: it ablates multi-view reading and the evidence used in projection, tracks candidate evolution across stages (where the coverage/precision trade-offs become visible), and examines sensitivity to K and to the training fraction. The 10%-training result on Rest15 (40.21 F1 over a fully supervised baseline) is also practically relevant for low-resource settings.

Critical weaknesses
  1. In this paper, both the baselines and SDA are evaluated on GPT-4o-mini, and I think the authors should also run experiments on GPT-5.6/GPT-6 to demonstrate the effectiveness. The paper does evaluate SDA with GPT-5.6-Luna to examine how backbone choice affects the framework, but there is no analysis of why the results differ between GPT-5.6-Luna and GPT-4o-mini, and the baselines are not run on GPT-5.6-Luna at all. Testing SDA alone on GPT-5.6-Luna is simply meaningless.

  2. Most of the baselines used here are from before 2025, and the authors should at least include one or two methods from 2025–2026. The rise of agents and the improvements in post-training have shifted the reasoning paradigm of current models, so comparing only against older methods is no longer meaningful.

  3. The proposed method samples multiple times, which greatly increases input/output token consumption and latency — in Table 3, SDA's token usage and runtime are over ten times those of the baselines. Meanwhile, prior work has shown that models tend to perform better on benchmarks when they are allowed to produce more output tokens. I would therefore suggest re-running the comparison under the same token budget to make it fair.

  4. A question about the method: in Candidate Projection, the authors retrieve gold-annotated precedents semantically similar to each candidate from the training set, put them into the prompt, and let the backbone decide whether to keep or drop a candidate based on whether a similar precedent exists in the precedent bank. This still lets the precedent bank determine which content is extracted. In other words, the decision of "what to extract" is reintroduced through precedent retrieval, and the coupling between source-side discovery and dataset-specific annotation conventions is not truly removed.

Questions for authors

Please See Weakness.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer BSw4

REVIEWER BSW4

Q1. The purpose of the Luna results

The Luna rows answer a different question from the GPT-4o-mini baseline comparison. Table 2 uses GPT-4o-mini to compare methods on one backbone. The Luna rows examine whether a stronger backbone narrows SDA's gap to task-specific supervised references; they are not presented as evidence of a relative gain over Luna-based LLM baselines. The revision will make that purpose and the 300-input ASQP/RTE versus full-test ACE2005 scopes explicit. Lack of Luna baseline rows therefore limits a second-backbone comparative claim, but does not make the reported backbone experiment meaningless.

Q2. Recent baselines

The claim that the comparison contains only pre-2025 methods is factually incorrect. Table 2 includes SimRP-ICL (AAAI 2025), ProgCo (ACL 2025), SR-LLM (ACL 2025), and KeyCP++ (ECAI 2025), in addition to foundational prompting controls. The request to include at least one or two 2025–2026 methods is already met.

Q3. Tokens, latency, and budget matching

SDA's additional calls have a real latency cost, but the gain is not explained by output-token volume alone. On the same 100 Rest15 inputs, SDA records 48.25 F1 with 336 output tokens and about 7,132 total tokens per input; ProgCo records 29.35 F1 with 685 output tokens and 8,991 total tokens. SDA is slower (48.82 versus 22.09 seconds of summed request time). Table 3 reports both sides of this tradeoff; no claim of superior inference efficiency is made. Multiple source readings and annotation decisions are distinct operations, rather than repeated sampling of one final answer.

Q4. Whether retrieval reintroduces coupling

The review conflates finding a source-supported candidate with deciding whether that candidate belongs in a dataset's output schema. SDA separates these decisions, not all influence of annotation examples on the final record set. Semantic Discovery fixes the natural-language candidate space without labeled examples. Projection may then reject or normalize candidates under the annotation policy, and Compilation labels the recordable ones. This later dependence on annotation evidence is required by the task and is exactly where SDA places it; it does not retroactively make the initial source reading demonstration-conditioned.

The authors doesn't fully adressed my concerns.

  1. I think authors should use GPT5.6/GPT6 to test all baselines and SDA. It is unconvinced to use GPT-4o-mini as the backbone in 2026.

  2. These papers are accepted by the conference in 2025, but are submitted in 2024 (e.g. SimRP-ICL, ProgCo). So, why not use the work wirtten in 2025 or 2026 as baselines?

  3. I want to know whether SDA can maintain its advantage when other baselines are given the same number of tokens. The authors can compare the baseline and SDA methods under the same total tokens. Only such an experiment can eliminate the doubt about whether the token count affects the experimental results.

Reviewer_sdu5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SDA, an LLM-based framework for structured information extraction that separates identifying semantic information in text from assigning task-specific annotation labels. The authors find that the number of records in demonstrations influences how many records the model predicts, even when the input text remains unchanged. To address this issue, SDA introduces a three-stage framework. The authors evaluate SDA on five datasets and gain improvements.

Critical strengths
  1. Well-motivated problem formulation and an intuitive, generalizable framework, identifies an interesting limitation of ICL
  2. Strong empirical improvements across multiple tasks, supported by informative ablations.
Critical weaknesses
  1. Novelty is limited, the pipeline is heavily engineered. Candidate generation followed by LLM verification, multi-view reading, and per-candidate example retrieval all have been already addressed by various previous studies
  2. Figure 3(a,b) shows that adding demonstrations to Discovery raises final F1, which is kind of against the central design choice of keeping Discovery example-free. I also noticed that appendices A show that example-free Discovery’s counts and coverage also shift with demonstrations
  3. the baselines compared use fixed demonstrations, which is kind of unfair, as stronger baselines with retrieval such as GPT-RE exist
Questions for authors
  1. as NYT has a well known heavy overlap between training and test, I question whether that improvement is close to a direct lookup of labels for previously seen entity pairs. The paper’s own case study also illustrates this (capital, contains). And the reported average improvement comes largely from NYT. if the authors do not want to remove this benchmark they should consider report results split by whether a test triple’s entity pair appears in the training data.
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer sdu5

Q1. Novelty and engineering

The shared pipeline shape does not capture the contribution. SDA starts from a measured ICL failure mode: examples can change the semantic scope of extraction, not only its annotation. It therefore generates natural-language relation candidates from the source before introducing labeled examples, then uses annotation evidence to decide their dataset-compatible realization. Existing candidate verification, multi-view reading, and retrieval methods address extraction accuracy or schema alignment without imposing this separation. The multi-view ablation tests the coverage role directly: discovery broad recall rises from 53.5 to 57.3 on Rest15 and from 78.4 to 87.9 on NYT.

Q2. Why exclude examples from Discovery if final F1 can rise?

The F1 gain is compatible with the design goal because exact annotation matching and semantic relation coverage are different outcomes. On Rest15, adding examples to Discovery raises final F1 by 1.6 points but lowers discovery broad recall from 57.3 to 56.0. Some of the gain comes from expressing an already found relation with a boundary closer to the gold annotation, effectively moving normalization earlier. On the same fixed NYT inputs, changing each demonstration from one to two relations moves direct ICL broad recall from 22.60 to 33.33, but Discovery broad recall only from 90.25 to 90.40. Thus the measured coverage dependence is far smaller in Discovery than in direct ICL. Appendix A's None condition is example-free; its One, Two, and Balanced conditions deliberately add examples and should not be described as example-free.

Q3. Retrieval baselines

The updated direct-ICL control uses retrieved examples on the same GPT-4o-mini backbone. On Rest15/16 it receives SDA's exact five examples and remains below it by 5.87/2.50 F1 points. Input-level BM25 ICL is below SDA by 22.05/2.48 on NYT/WebNLG, although SDA still uses candidate-linked evidence for RTE. The original table also included retrieval-based SimRP-ICL, so the claim that every baseline used fixed demonstrations was inaccurate. TableIE, however, exceeds SDA on WebNLG (72.46 versus 71.42); the revised comparison states this explicitly.

Q4. Previously seen NYT entity pairs

Dataset overlap is not a method-specific defect. NYT and WebNLG are established benchmarks, and their published train–test overlap applies to every method evaluated on the same splits. It does not, by itself, show that SDA copies labels for seen pairs. SDA's gains on sentiment extraction and event detection further show that its observed benefit is not confined to the two relation datasets. We will state the overlap property without treating it as proof of lookup.

Reviewer_JieyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that in-context structured extraction fails for two separable reasons, noticing the semantic content and reproducing the dataset's annotation conventions, and that one prompt with a handful of fixed demonstrations does neither well. SDA splits the task into three calls to a frozen LLM: discovery reads the input several times without demonstrations and emits over-complete natural-language candidates; projection retrieves training records whose anchors match each candidate and asks the LLM to keep, drop or rewrite it so that it follows those precedents; compilation renders the kept candidates into the dataset format with a few retrieved complete examples. The training set is pre-processed offline into an annotation memory. Experiments cover ASQP (Rest15/16), relation triple extraction (NYT, WebNLG) and event detection (ACE05) with GPT-4o-mini and a second backbone, and the headline claim is about +21 F1 on average over the strongest same-backbone LLM baselines.

Critical strengths
  • The decomposition is well motivated, and the diagnostics in Figure 1 and Appendix A (count shift as a function of records per demonstration; broad recall of demo-free discovery) make a concrete, falsifiable case that fixed demonstrations bias how many records the model emits.
  • Figure 4 is a genuinely useful view of the pipeline: it shows exactly where the precision comes from (projection) and what it costs (recall).
  • Three task families and five datasets with one method and one prompt family, and an unusually complete appendix (prompts, example-selection scoring, relation cards), so the work is reproducible in principle.
  • The ablations in Figures 3 and 5 are the right experiments to run, and the authors report the ones that hurt them.
Critical weaknesses

1. The pipeline shape is not new, and the closest prior work is not cited. Extract first with a frozen LLM, then align to the schema against retrieved schema elements, is exactly EDC (Zhang & Soh, EMNLP 2024), which was evaluated on WebNLG. For ASQP, Li et al. (Findings of EMNLP 2025) already run a candidate-then-revise loop with a black-box LLM on Rest15/16. ChatIE, AgentRE, the OneKE agent system, GoLLIE and kNN-RE are all uncited. Once these are on the table, what is specific to SDA is the ordering (demo-free discovery first, precedent retrieval second) and retrieving per candidate by anchor string instead of once per sentence. Both are reasonable engineering choices, not a conceptual contribution at the level the introduction suggests.

2. The main comparison is confounded by retrieval, and the one ablation that could separate it points the other way. Every generic LLM baseline in Table 2 uses the same fixed random demonstrations; SDA retrieves anchor-matched precedents and nearest complete examples. Similarity-based demonstration retrieval has been the default since KATE, and GPT-RE, which the paper cites but does not run, already does it for RE. The evidence ablation in Figure 3 is the closest thing to a control: on NYT, replacing anchor matching with plain whole-sentence similarity retrieval still gives two to three times the F1 of the fixed-demo baselines, and the authors themselves write that this ablation does not isolate retrieval. The headline is also dominated by NYT; without it the average gain roughly halves, and the SimRP paper the submission cites reports that plain retrieval ICL with GPT-4o on ASQP reaches the same numbers as SDA. The missing baseline is simple: vanilla ICL with K retrieved demonstrations at the same backbone. Without it I cannot attribute the gain to the decomposition.

3. Projection on NYT and WebNLG looks like retention of training triples rather than extraction. The projection prompt keeps a candidate only when a retrieved annotated precedent supports it. Lee et al. (NAACL 2022) measured that about 90% of test triples in these exact-match versions of NYT and WebNLG are entirely seen in training (same entity pair and relation). On such data the KEEP/DROP rule approximates "keep iff the same pair is already annotated in training". Three results fit that reading: random precedents collapse NYT F1 to near zero; projection trades a large amount of broad recall for precision; and the stronger backbone pushes NYT precision to about 99 while recall falls, so the better model scores lower F1. The paper never mentions memorisation or leakage. I would want F1 split by seen / partially seen / unseen triples (Taillé et al., 2021) before believing the NYT number.

4. Several numbers do not agree across the paper or with their sources, and the baselines are weak. The 100% endpoints of the supervised curves in Figure 5 are below the corresponding Table 2 entries, so Table 2 presumably copies published numbers while Figure 5 shows re-runs; this has to be stated and the re-runs used. SimRP-ICL differs by five points between Table 2 and the authors' own run in Table 3. The ED supervised rows do not match the IEPile table they are cited from, and two of them have P = R = F1 to two decimals, which does not happen for a real system. Figure 3 reports a with-demonstration discovery configuration on Rest15 that beats the main result, without explanation. More broadly, Wadhwa et al. (cited) report GPT-3 few-shot on the same NYT split far above the ICL and TableIE numbers here, which says more about the baseline prompts than about ICL, and SDA remains well behind supervised models on four of five datasets.

5. Cost, hidden human effort, an under-specified backbone, and no variance. On Rest15 SDA uses roughly 17× the tokens and 30× the latency of ICL, and the offline cost of running discovery over the whole training set is not counted at all. Appendix D lists hand-written relation-family question cards for NYT and WebNLG and a hand-tuned seven-term example-selection score; neither is ablated, although GoLLIE showed that guidelines alone move zero-shot IE substantially. The second backbone is given without snapshot or reasoning-effort setting, so half of Table 2 cannot be reproduced. Every number is a single run with a single demonstration draw, while the paper's own Appendix Table 4 shows that changing the demonstrations moves NYT recall by up to 25 points.

Questions for authors
  1. Please add a vanilla-ICL baseline with K retrieved demonstrations (sentence-embedding retrieval, and entity-aware retrieval for NYT/WebNLG) at the same backbone. What is the gap to SDA then?
  2. For NYT and WebNLG, what is SDA's F1 on the seen / partially seen / unseen partition, and what fraction of the correct candidates dropped at projection are unseen?
  3. Which Table 2 numbers are re-runs and which are copied, and where do the ED supervised numbers come from?
  4. How much of the discovery recall is due to the hand-written relation cards, and how were the example-selection weights chosen?
  5. Exactly which configuration of the second backbone was called?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Jiey

Q1. Novelty and related work

The review misidentifies SDA's contribution. SDA identifies a failure mode of ICL: demonstrations intended to teach annotation conventions can also change which relations the model finds in unchanged source text. The proposed framework separates these two decisions. EDC addresses schema size and canonicalization; its first extraction prompt already contains few-shot triples. AgentRE and OneKE use memory and retrieved cases to improve extraction while it is taking place. Li et al. instead use supervised small models to propose and evaluate ASQP answers. ChatIE studies zero-shot question answering, GoLLIE is trained to follow guidelines, and kNN-RE builds on supervised relation extraction. These are relevant related works, but a shared extract–retrieve–revise outline does not make them solutions to the demonstrated ICL failure mode. SDA's source-first natural relation discovery is therefore a substantive decision boundary, not merely the removal of examples from an ordinary structured-output prompt.

Q2. Retrieval-matched ICL and the claimed gain

The fixed-random-demonstration premise no longer holds, and retrieval alone does not close the ASQP gap. The updated direct-ICL control uses the same GPT-4o-mini backbone, training pool, five examples, and exact-record scorer. On Rest15/16 it receives the same retrieved examples as SDA and reaches 37.79/47.83 F1, versus SDA's 43.66/50.33 (+5.87/+2.50). On NYT/WebNLG, input-level BM25 direct ICL reaches 61.45/68.94, versus SDA's 83.50/71.42 (+22.05/+2.48); SDA's candidate-linked evidence selection remains a method difference in those RTE comparisons. The revised WebNLG TableIE result is 72.46, above SDA's 71.42, so the claim is not superiority to every retrieved method. Two comparisons in the review are invalid: Figure 3 reports entity-pair F1, while Table 2 reports complete triple F1; and the cited SimRP numbers use GPT-4o, while SDA uses GPT-4o-mini. SimRP's published GPT-4o-mini five-shot scores are 33.33/40.44 on Rest15/16, versus SDA's 43.66/50.33.

Q3. NYT/WebNLG overlap and lookup

Train–test overlap in public NYT and WebNLG splits does not establish that SDA performs label lookup. The same benchmark splits and overlap apply to the compared methods. The proposed equivalence between precedent-based filtering and “keep iff the pair appeared in training” has not been demonstrated by the cited observations. SDA also improves over the strongest evaluated GPT-4o-mini baselines on Rest15 and ACE2005, where the disputed NYT/WebNLG entity-pair overlap is not the explanation. Overlap should be reported as a benchmark property and interpreted consistently across methods; it cannot selectively invalidate SDA's scores.

Q4. Score provenance and numerical objections

The assertion that P = R = F1 cannot come from a real system is mathematically false. On the common ACE2005 test set there are 293 gold events. Official-checkpoint reproductions produce 293 predictions each: YAYI-UIE gets 202 correct (68.94 for P, R, and F1), and InstructUIE gets 229 correct (78.16 for all three). The prediction files, checkpoint records, and scorer outputs were audited. The other cited differences compare different scopes: Figure 5's supervised curves are reproductions, while some Table 2 supervised references are published scores; Table 3 uses 100 shared inputs, while Table 2 reports its stated test split. These sources and scopes will be marked per row. Figure 3 already reports the Rest15 demonstration result: final F1 rises while discovery broad recall falls. Wadhwa et al.'s GPT-3 results are not a same-backbone control for GPT-4o-mini.

Q5. Cost, retrieval specification, backbone, and variance

Table 3 reports an online inference cost tradeoff, not an efficiency advantage over direct ICL. It uses the same 100 inputs per method and explicitly excludes one-time annotation-memory construction. More tokens alone do not explain all performance differences: on the Rest15 cost sample, ProgCo uses 8,991 total tokens for 29.35 F1; SDA uses about 7,132 for 48.25 F1, though SDA has greater latency. The former seven-term retrieval formula was an obsolete description and has been removed; the revision states the actual task-specific selection procedure. Relation-family cards are source-reading cues, not gold records. The Luna requests used the model alias gpt-5.6-luna at temperature 0 with no explicit reasoning-effort parameter; the provider did not expose a versioned snapshot. Finally, Appendix A's large recall shift is a controlled change to demonstration composition, not variance across repeated calls under a fixed support set. We will make these protocol and cost scopes explicit.

Thank you for the detailed response and additional experiments. The retrieval-matched ASQP results substantially address my concern about comparisons against fixed demonstrations. I also acknowledge the differences in metrics and evaluation scopes that you clarified.

However, my concern about the interpretation of the NYT/WebNLG gains remains unresolved. Using the same benchmark splits does not establish whether SDA and the baselines benefit similarly from training–test overlap. I agree that overlap alone does not demonstrate label lookup, but the requested seen/partially seen/unseen analysis is still needed to assess this concern. Moreover, the new RTE comparisons retain different evidence-selection procedures, so they do not fully separate the contribution of decomposition from candidate-linked retrieval.

Please also clarify which retrieval procedure produced the original and revised results, given that the seven-term formula is now described as obsolete. The revised method description should correspond to the implementation used for the reported experiments.

The response strengthens the empirical case, particularly for ASQP, but these remaining questions limit my confidence in the interpretation of the main gains. I therefore maintain my original rating for now, while remaining open to revisiting it if these points are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.