- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper observes that in few-shot LLM-based structured extraction, the number of records in the demonstrations shifts the number of records the model predicts even for fixed input text, suggesting that demonstrations bias what content is discovered rather than merely teaching annotation conventions. To address this, the authors propose SDA, which first discovers natural-language candidates from the source text without labeled examples (Semantic Discovery), then projects them onto the dataset schema using retrieved training precedents (Candidate Projection), and finally resolves labels and record merges with K examples (Joint Compilation). Experiments on three tasks and five datasets (ASQP on Rest15/Rest16, RTE on NYT/WebNLG, ED on ACE2005) with GPT-4o-mini show that SDA outperforms all same-backbone LLM baselines by 20.78 F1 on average, while still lagging supervised models and costing considerably more tokens and time. The paper also provides ablations on multi-view reading, projection evidence, and sensitivity to K and data size.
Critical strengths
-
The motivating controlled analysis is a clean and useful observation. Holding the input text fixed and varying only the record cardinality of the demonstrations, the authors show that the predicted record count shifts systematically, and that single-record and multi-record inputs behave differently. The follow-up comparison between direct ICL and the label-free discovery stage is also informative. Whether or not the proposed framework fully solves it, the problem itself is clearly demonstrated and is worth attention.
-
The framework is technically complete and presented clearly. The three stages have well-defined inputs and outputs , the overview figure and case studies make the pipeline concrete, and the prompt templates, example-selection scoring, and evaluation rules are provided in the appendices, with a reproducibility statement promising code, prompts, and per-sentence annotations. This is more transparent than most prompt-based submissions.
-
The analysis is fairly thorough: it ablates multi-view reading and the evidence used in projection, tracks candidate evolution across stages (where the coverage/precision trade-offs become visible), and examines sensitivity to K and to the training fraction. The 10%-training result on Rest15 (40.21 F1 over a fully supervised baseline) is also practically relevant for low-resource settings.
Critical weaknesses
-
In this paper, both the baselines and SDA are evaluated on GPT-4o-mini, and I think the authors should also run experiments on GPT-5.6/GPT-6 to demonstrate the effectiveness. The paper does evaluate SDA with GPT-5.6-Luna to examine how backbone choice affects the framework, but there is no analysis of why the results differ between GPT-5.6-Luna and GPT-4o-mini, and the baselines are not run on GPT-5.6-Luna at all. Testing SDA alone on GPT-5.6-Luna is simply meaningless.
-
Most of the baselines used here are from before 2025, and the authors should at least include one or two methods from 2025–2026. The rise of agents and the improvements in post-training have shifted the reasoning paradigm of current models, so comparing only against older methods is no longer meaningful.
-
The proposed method samples multiple times, which greatly increases input/output token consumption and latency — in Table 3, SDA's token usage and runtime are over ten times those of the baselines. Meanwhile, prior work has shown that models tend to perform better on benchmarks when they are allowed to produce more output tokens. I would therefore suggest re-running the comparison under the same token budget to make it fair.
-
A question about the method: in Candidate Projection, the authors retrieve gold-annotated precedents semantically similar to each candidate from the training set, put them into the prompt, and let the backbone decide whether to keep or drop a candidate based on whether a similar precedent exists in the precedent bank. This still lets the precedent bank determine which content is extracted. In other words, the decision of "what to extract" is reintroduced through precedent retrieval, and the coupling between source-side discovery and dataset-specific annotation conventions is not truly removed.
Questions for authors
Please See Weakness.
2 comments
The authors doesn't fully adressed my concerns.
-
I think authors should use GPT5.6/GPT6 to test all baselines and SDA. It is unconvinced to use GPT-4o-mini as the backbone in 2026.
-
These papers are accepted by the conference in 2025, but are submitted in 2024 (e.g. SimRP-ICL, ProgCo). So, why not use the work wirtten in 2025 or 2026 as baselines?
-
I want to know whether SDA can maintain its advantage when other baselines are given the same number of tokens. The authors can compare the baseline and SDA methods under the same total tokens. Only such an experiment can eliminate the doubt about whether the token count affects the experimental results.
Author Response to Reviewer BSw4
REVIEWER BSW4
Q1. The purpose of the Luna results
The Luna rows answer a different question from the GPT-4o-mini baseline comparison. Table 2 uses GPT-4o-mini to compare methods on one backbone. The Luna rows examine whether a stronger backbone narrows SDA's gap to task-specific supervised references; they are not presented as evidence of a relative gain over Luna-based LLM baselines. The revision will make that purpose and the 300-input ASQP/RTE versus full-test ACE2005 scopes explicit. Lack of Luna baseline rows therefore limits a second-backbone comparative claim, but does not make the reported backbone experiment meaningless.
Q2. Recent baselines
The claim that the comparison contains only pre-2025 methods is factually incorrect. Table 2 includes SimRP-ICL (AAAI 2025), ProgCo (ACL 2025), SR-LLM (ACL 2025), and KeyCP++ (ECAI 2025), in addition to foundational prompting controls. The request to include at least one or two 2025–2026 methods is already met.
Q3. Tokens, latency, and budget matching
SDA's additional calls have a real latency cost, but the gain is not explained by output-token volume alone. On the same 100 Rest15 inputs, SDA records 48.25 F1 with 336 output tokens and about 7,132 total tokens per input; ProgCo records 29.35 F1 with 685 output tokens and 8,991 total tokens. SDA is slower (48.82 versus 22.09 seconds of summed request time). Table 3 reports both sides of this tradeoff; no claim of superior inference efficiency is made. Multiple source readings and annotation decisions are distinct operations, rather than repeated sampling of one final answer.
Q4. Whether retrieval reintroduces coupling
The review conflates finding a source-supported candidate with deciding whether that candidate belongs in a dataset's output schema. SDA separates these decisions, not all influence of annotation examples on the final record set. Semantic Discovery fixes the natural-language candidate space without labeled examples. Projection may then reject or normalize candidates under the annotation policy, and Compilation labels the recordable ones. This later dependence on annotation evidence is required by the task and is exactly where SDA places it; it does not retroactively make the initial source reading demonstration-conditioned.