acceptodds
Under review as a conference paper at ICLR 2027

LLM Extractor Errors Are Not Random: Protagonist Bias in Knowledge Graph Extraction

Abstract

Large language models have become the prevalent tool for building knowledge graphs from scientific literature. It turns unstructured text into entity-relation graphs which enhance literature-based discovery, biomedical question answering, and drug repurposing. However, the resulting graphs remain far from complete and accurate, and common remedies such as sampling and voting implicitly treat extraction errors as independent noise. Through cross-corpus error analyses, counterfactual experiments, and mechanistic analyses, we show that these errors are structured: relations between a document's supporting entities are extracted systematically worse than those involving its most-mentioned entity, its protagonist. We trace this to document-level salience: an entity's prominence across the whole document, not just in the sentence that states the relation, shapes whether the model opens a relation with it, and also inflates the model's confidence, so frequency- or confidence-based selection reinforces the bias. We therefore separate proposing relations from verifying them: an evidence-anchored extractor re-reads each document sentence by sentence to propose candidates, and a selector keeps only those supported by local evidence. On BioRED, this improves Qwen3.5-27B by 0.07 F1 over standard fine-tuning and raises supporting-entity precision from 0.54 to 0.68 without loss of recall.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge Graph Construction, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jv8POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the limitations of LLM-based biomedical knowledge graph construction, focusing on the disparity in relation extraction performance between prominent and supporting entities. The authors benchmark several open-weight and closed-source LLMs across 3 datasets and identify a consistent performance gap. Through controlled interventions and mechanistic analyses, they investigate how an entity's document role influences relation extraction, suggesting that the disparity primarily arises during entity selection rather than relation completion. To address this limitation, the authors propose a two-stage approach that separates relation candidate generation from verification. An evidence-anchored extractor processes the document sentence by sentence to generate candidate relations while a subsequent selection step retains only those supported by local textual evidence.

Critical strengths
  • The paper investigates an important limitation of LLM-based knowledge graph construction, particularly the underrepresentation of relations involving less prominent entities. Beyond reporting extraction performance, the authors attempt to identify where the observed disparity originates through controlled interventions, activation patching, and attention manipulation.
  • The paper proposes a two-stage approach that separates relation candidate generation from verification. An evidence-anchored extractor processes the document sentence by sentence to generate candidate relations while a subsequent selection step retains only those supported by local textual evidence.
Critical weaknesses
  • The main weakness of this work is the limited evidence establishing protagonist bias as the underlying cause of the observed performance disparity. The authors attribute the lower extraction performance on supporting entities to an entity-prominence bias (mention frequency, title position, and other document-level properties). However, this disparity could instead reflect greater model uncertainty due to the limited evidence available for less frequently mentioned entities. Although the attention interventions show that increasing attention to supporting entities makes them more likely to appear in extracted relations, this does not establish that protagonist prominence, rather than differences in supporting evidence, causes the original disparity. Furthermore, the authors acknowledge in Section 2.2 (“Simple alternatives do not account for the gap”) that masking protagonist mentions reduces protagonist relations without improving supporting-entity recall. This suggests that the observed gap cannot be explained by simple competition for attention or output capacity and may instead reflect more general limitations in the model ability to perform relation extraction.

  • The paper lacks comparisons with existing approaches that propose novel methods for biomedical relation extraction. For example, GLiM: Integrating Graph Transformer and LLM for Document-Level Biomedical Relation Extraction with Incomplete Labeling and PLRTE: Progressive Learning for Biomedical Relation Triplet Extraction Using Large Language Models are relevant methodological baselines.

  • The paper does not provide a comparison with specialized biomedical BERT-based relation extraction models. Previous studies like “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks. Including such baselines would provide a more comprehensive assessment of how LLMs are positioned against BERT performance for biomedical relation extraction tasks.

  • The paper does not analyze the additional computational overhead introduced by the two-stage candidate generation and local evidence selection procedure. Reporting inference time and computational resources, would help assess the practicality and scalability of the approach.

Questions for authors
  • Figure 2: The caption begins with “Recall of protagonist relations” but the figure reports F1 scores. The terminology should be corrected to avoid confusion.

  • Interpretation of Figure 2: The performance gap between protagonist and supporting entities is not necessarily surprising. Central entities naturally occur more frequently and may have more relations and supporting contextual evidence, making their relations easier to extract. How do the authors distinguish a genuine entity-prominence bias from differences in evidence availability and relation difficulty?

  • In the controlled experiment for “Changing document role changes whether fixed evidence is reported”, the authors replaced the protagonist and the supporting entity name everywhere outside of the sentence that states the scored relation. How are these replacements performed while preserving the semantic coherence of the document? If the substitutions introduce inconsistencies or alter contextual evidence, the resulting performance changes could reflect increased model uncertainty rather than entity prominence alone.

  • The best configuration “Fine-tuned, evidence-anchored target + sentence pass + local-evidence selector” was only evaluated on one dataset, would this improvement generalize to other dataset? For instance, what about the other datasets BC5CDR and Re-DocRED?

  • The authors claim using both open-source and closed-source models for their experiments but they only use Qwen3.5-9B, Qwen3.5-27B and DeepSeek V4 Pro which are all open-source models.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jv8P (1/2)

We thank the reviewer for their constructive feedback!

Weakness 1 & Question 2: Protagonist bias or weaker evidence?

Thank you for raising the central question of the paper. The gap remains when the evidence is held fixed, so it cannot be reduced to weaker evidence. We agree that Figure 2 alone cannot separate these explanations, and we should have made our controls easier to find.

First, the swap holds the evidence fixed and changes only the role. The sentence that states the relation stays word for word the same, so its evidence and difficulty do not change. Yet the model writes the same relation 24 to 31 points less often once its entity is no longer the protagonist (Section 2.2). Over all swapped records, the drop is still 19 to 23 points (Appendix D.9). Our reply to Question 3 shows that the edits themselves do not cause this drop.

Second, the gap remains after controlling for evidence and difficulty. A regression holds fixed the number of shared sentences, mention count, position and relation type. Involving the protagonist still multiplies the odds of recovery by 2.0 to 3.6 (Table 6). We also added a control for how clearly the evidence states each relation, which we use as a proxy for difficulty. A classifier scores each relation from its shared sentence alone, and we compare the two roles at similar scores. The recall gap is still 15 to 28 points and significant in all eight systems (Table 10).

Third, the gap is not a general inability to extract these relations. We agree that masking rules out simple competition for output. In the fine-tuned extractors, making a supporting entity the protagonist raises the write-out of its relation by 22 to 26 points (Section 2.2). The same model can extract the same relation from the same sentence. What changes is the entity’s role.

Revision: Appendix A reports the new evidence-strength control in Table 10

Weakness 2: Comparison with GLiM and PLRTE Thank you for pointing us to GLiM and PLRTE. GLiM solves a different problem in a different setting. It is given the gold entities and targets incomplete training labels, with an LLM re-checking a BERT model’s borderline pairs. Our pipeline starts from raw text, and its LLM proposes candidates that a BERT model then filters. Following your suggestion, we ran GLiM’s released code on BioRED with gold entities and scored it with the scorer of Table 2. It reaches 0.45 F1, against 0.53 for our 27B pipeline, which receives no gold entities. Both baselines stay below our full pipeline. PLRTE is closer to our setting, because it also generates triplets from raw text. It reports no BioRED results and has not released its weights, so we compare on BC5CDR, where its 7B model reports 0.50 F1. Fine-tuned on BC5CDR, our pipeline reaches 0.62 F1 with Qwen3.5-9B, a model of similar size, and 0.645 with Qwen3.5-27B. PLRTE does not describe its scoring rule, so we report these numbers side by side without a significance test.

Revision: Related Work now discusses GLiM and PLRTE. Table 2 now includes the GLiM result, and Appendix C adds the comparison with PLRTE.

Weakness 3: Biomedical BERT models. Our pipeline is itself a specialized biomedical system, and end to end it outperforms a biomedical BERT classifier. Its LLM is fine-tuned on BioRED, and its final filter is BioLinkBERT, a biomedical BERT model. The studies you cite compare such fine-tuned models with general-purpose LLMs used without fine-tuning, on entity pairs given in advance. Our results agree with them. Without fine-tuning, the open-weight LLMs in Table 2 reach 0.24 to 0.42 F1. A PubMedBERT pair classifier, which labels every entity pair, reaches 0.45 on PubTator3 entities and 0.46 on the entities from our 27B pipeline, with the same scorer. Our 27B pipeline reaches 0.53, and both gaps are significant.

Weakness 4: Computational cost. Thank you for this practical point, and we agree that the cost should be reported. On one H200 GPU, the full pipeline takes 2.3 to 2.5 times as long as a single pass. This is about 10 seconds per abstract for Qwen3.5-9B and 20 seconds for Qwen3.5-27B, including model loading. The sentence pass sends 11.5 prompts per abstract, but each prompt and its output are short. It therefore takes only 1.3 to 1.5 times as long as the whole-document pass. The selector needs a few seconds for the whole test set.

(Continued in Part 2 below.)

AuthorsOpenReview ↗

Response to Reviewer jv8P (2/2)

Question 1: Figure 2 caption. We corrected the caption of Figure 2, which now refers to F1.

Revision: Figure 2 caption

Question 3: Swap procedure and coherence. The drop in write-out after the swap comes from the change of protagonist, not from confusion caused by the edits. We exchange every mention of the protagonist and a supporting entity of the same type outside the stating sentence, including the title, and keep that sentence word for word. For example, one abstract becomes a study of acetaldehyde instead of ethanol, while "Long-term (4-week) exposure to 25 mM ethanol upregulated the Oct4 and Nanog proteins" stays unchanged. The fine-tuned Qwen3.5-27B writes the ethanol–Oct4 relation for the original but not after the swap, and it still writes it after a matched-null swap. Four checks show that the edits themselves do not cause such drops. Matched-null swaps edit as many mentions but exchange two other entities, and they change write-out by at most 2 points. In the same swapped documents, relations that involve neither swapped entity are written at least as often as before. Perplexity under an untuned Qwen3.5-9B, a direct measure of uncertainty about a text, rises by the same amount for real and matched-null swaps. Finally, we read 30 randomly drawn swapped documents. Apart from the intended name change, 23 read coherently, 7 had minor local oddities, and none contradicted the stating sentence.

Revision: Appendix D now describes the procedure with this example.

Question 4: Other datasets. The improvement carries over to both datasets. We applied the same recipe to each one. We fine-tuned Qwen3.5-27B with evidence-anchored targets on its training set, trained the selector on the same set, and froze the threshold on its validation set. Each test set was scored once with the scorer of Table 2. On both datasets, the full pipeline improves over a single pass of the same model.

DatasetSingle passFull pipelineGain [95% CI]
BC5CDR0.620.65+0.03 [+0.01, +0.04]
Re-DocRED0.610.62+0.01 [+0.01, +0.02]

Revision: Appendix E (Table 41) now reports both datasets.

Question 5: Closed-source results We agree this was unclear, because we accessed DeepSeek-v4-pro only through its API and grouped it with closed-source models. For completeness, Tables 1 and 2 now include the closed-source Claude Sonnet 5.5, which also recovers at most 41% of BioRED relations and shows the same protagonist gap.

Revision: Table 1 in Section 2 and Table 2 in Section 4 now displays experiment result with Claude.

I appreciate the authors efforts in clarifying my questions and completing the study with additional experiments. Most of my concerns have been addressed so I will change my score from 2 to 3.

For W3. it would be great if the authors also include results for replacing the first stage (mention detection) with a biomedical BERT baseline to clearly demonstrate the proposed pipeline is doing better at each stage of the relation extraction pipeline.

For Q3. I think it would be nice if the authors add a concrete example of swap procedure so that it's clear to readers that the coherence is not affected.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for carefully reading our responses and raising your recommendation. In the final version, we will add the results of our pipeline with entities recognized by a biomedical BERT model and a concrete swap example. Thank you again for your time and helpful suggestions.

Reviewer_KJghOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the protagonist bias in document-level knowledge graph extraction, where the LLMs recover relations involving the most frequently mentioned entities more reliable than relations between two supporting entities. Through cross-dataset analyses, entity-swapping experiments that preserve the local evidence sentence, activation patching, and attention interventions, the authors examine how document-level salience influences the choice of entity that initiates a relation. Motivated by these findings, the authors propose a two-stage framework that separates the generation of candidate relationships from the verification of local evidence. On BioRED, the pipeline improves Qwen3.5-27B’s relation F1 from 0.46 with standard fine-tuning to 0.53. The main contributions are a systematic empirical analysis of extraction bias and a candidate-generation and selection pipeline motivated by that analysis.

Critical strengths
  1. The paper provides a valuable diagnosis of structured extraction errors, supported by multiple complementary experiments. Beyond comparing recovery rates for protagonist and supporting-entity relations, the authors investigate matched swaps, bystander and identity controls, gold-entity inputs, a discriminative classifier, and interventions on training targets. This experimental breadth takes the work beyond a simple correlation between mention frequency and recall.

  2. The paper reports important limitations and negative results. The authors use document-clustered uncertainty estimates and distinguish internal scores, participation in free generation, and extraction quality. The appendices report heterogeneous generation responses across model families, degraded cross-sentence performance, and unsuccessful methodological variants. They do not simply equate attention steering with improved recovery of correct relations. These disclosures help establish which findings may generalize and which remain specific to particular models and experimental settings.

Critical weaknesses
  1. The swap experiments provide compelling evidence for the composite intervention: changing document context while preserving the local sentence affects relation reporting. However, mention frequency, title identity, and other contextual associations change together. Although Section 2.2 and Appendix D.8 appropriately qualify the interpretation, the abstract and introduction would benefit from clarifying that “document role” is operationalized through these jointly manipulated cues, rather than identified separately from them.

  2. Section 3.2 and Appendix B clearly state that the selector does not receive the relation type, which is supplied by the extractor. The concern is therefore not an unclear implementation but the scope of “verification” in the high-level framing. The selector makes a pair-level relatedness decision; although the complete pipeline improves typed-relation F1, this does not establish that the selector independently verifies whether the proposed relation type is supported by the local evidence. The paper should distinguish pair-level filtering from verification of a complete relation triple.

Questions for authors
  1. How should the contributions of mention frequency, title identity, and broader context be interpreted given the existing controls? Please clarify what the matched-null swaps control for and which sources are included in the joint “all cues” condition in Table 20. Since the source knockouts intervene on attention at selected layers, rather than separately manipulating the textual components of the swap, what conclusions can they support about the factors driving the behavioral effect?

  2. The 71% figure appears to be based on endpoint co-occurrence (704/990). Please clarify whether it measures relations whose endpoints share a sentence or relations whose supporting evidence is contained within a single sentence. Does the selector use the generated anchor S? Appendix B and Algorithm 1 indicate that it independently selects the first co-occurrence sentence, or the first sentence mentioning each endpoint, whereas Section 3.1 uses the nearest two sentences for non-co-occurring endpoints. Please explain the rationale for these different rules and whether their effect on selection performance has been evaluated.

  3. Under the final protocol, without gold annotations for evidence localization, does the local selector still outperform the whole-document selector?

  4. The paper already reports that the evidence-anchored 9B pipeline outperforms the same pipeline with standard targets by 0.04 F1 and that a 17-sample pool with the same selector reaches 0.49 F1 versus 0.53 for the two-pass pipeline. To better isolate the value of the generative candidate proposal, could you provide an exhaustive entity-pair classification baseline using the same predicted entity set and training split, without additional labeled data, and evaluate it with the same typed-relation scorer?

3 comments
AuthorsOpenReview ↗

Response to Reviewer KJgh

We thank the reviewer for their detailed comments!

Weakness 1: What “document role” means. We agree that the swap changes mention frequency, title identity and context together, and that the abstract and introduction should say so. We have revised both.

Revision: The introduction now reads: "A swap keeps the sentence that states a relation unchanged and exchanges the protagonist with a supporting entity everywhere else. Title identity, mention frequency and context therefore change together, and we use 'document role' for this joint change."

Weakness 2: Pair-level filtering rather than verification We agree that the selector makes a pair-level decision and does not verify the relation type, which comes from the extractor.

Revision: We have revised the abstract, introduction, Section 3.2 and the Figure 4 caption to describe it as a pair-level filter, and we no longer use "verification."

Question 1: Separating the three cues. Thank you for this careful question. We agree with your reading in Weakness 1. The swap changes title identity, mention frequency and context together, so we interpret its effect as their joint change, which we call document role, and not as separate factors. The existing controls rule out the other explanations. Matched-null swaps edit the same number of mentions but exchange two other entities, so editing the text does not explain the drop. Bystander records score a relation that involves neither swapped entity, and these relations are written at least as often as before, so the swap does not disrupt the document as a whole. In Table 23 (Table 20 in the submission), “all cues” masks the title together with every mention of the two scored entities outside the evidence sentence. The knockouts block attention without changing the text. They show where the decision reads, namely the title and those mentions, but not which textual factor causes the behavioral effect.

Revision: Appendix D.4 now defines each masked region, and states that the swap changes these cues together.

Question 2: The 71% figure and the evidence-sentence rules. The selector does not read the anchor S, and this is deliberate design. S is the sentence number the extractor writes after each relation and it shapes generation. Listing relations sentence by sentence makes the extractor cover the whole document, which raises the 27B's recall from 0.42 to 0.52. The selector has a different job. It checks each candidate against the text, so it finds the sentence from the two entity names instead of trusting the extractor's choice. This also lets it work for extractors that write no sentence number. We tested the alternative rules on validation. Reading S, or reading the two nearest sentences when the entities never share one, changes F1 by at most 0.003. Scoring every sentence that mentions both entities and keeping the highest score adds about 0.01 F1 on test. The 71% counts relations whose two entities share a sentence. It does not mean their evidence is one sentence, because BioRED has no evidence labels.

Revision: We corrected this wording and Section 3.1, and Appendix B now reports these comparisons.

Question 3: Local or whole-document selector. We reran the comparison under the final protocol, where neither selector uses gold annotations. The whole-document selector was trained and tuned the same way as the local one. On all four candidate pools, the two selectors show no significant difference since every 95% CI includes zero. We keep the local selector because it reaches the same F1 while reading only one sentence.

Revision: Appendix C now reports this comparison.

Question 4: Exhaustive pair-classification baseline. We added this baseline. It uses the entities predicted by our pipeline and pairs every two of them whose types match a BioRED relation. A PubMedBERT pair classifier trained on the BioRED training split labels each pair with a relation type or none. No extra labeled data is used, and the same scorer evaluates all systems. With 27B entities, the pipeline outperforms this baseline by 0.07 F1. With 9B entities, the pipeline is higher by 0.01 F1(0.45 to 0.46).

Revision: This baseline result is also shown in table 2.

AuthorsOpenReview ↗

Looking forward to your feedback

Thank you again for your thoughtful review. We hope our response has addressed your concerns. We would be happy to clarify anything further before the discussion ends. We look forward to your feedback.

I thank the authors for their clarifications and additional experiments, which address my original concerns. However, I remain concerned that the methodological contribution is incremental and that evidence for the full pipeline’s effectiveness remains concentrated on BioRED. While I value the systematic analysis of extraction bias, I find the overall contribution insufficient for acceptance and will therefore maintain my score of 2.

Reviewer_jNA3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at whether the mistakes LLMs make when extracting knowledge graphs from scientific text are random. The authors call the most-mentioned entity in a document its protagonist and show that models find relations involving the protagonist much more often than relations between the other entities. This holds across three datasets, different model sizes and both open and closed models. By swapping entity names in the document while keeping the key sentence the same, they show the cause is how prominent an entity is in the whole document, and they trace where this happens inside the model. Based on this, they propose a two-step method. First, a fine-tuned model reads the whole document and then each sentence on its own to collect candidate relations. Then a separate classifier keeps a candidate only if the sentence it comes from supports it. On BioRED this improves over standard fine-tuning, but only for relations stated within a single sentence.

Critical strengths
  1. The main finding is interesting and holds up well. Models find relations involving the protagonist much more often than relations between the other entities. This shows up on all three datasets, for every model, with and without examples in the prompt. On BioRED, the gap stays even after accounting for things like mention count, position and relation type.

  2. The causal evidence is convincing. The authors swap entity names everywhere except the sentence that states the relation, with proper controls. This shows that an entity's role in the document changes what the model reports. They also try simple fixes, like hiding the protagonist, telling the model not to focus on it, or giving it the list of entities. None of these does much to help the model find the missing relations.

  3. The mechanistic part is careful and honest. It shows the effect happens when the model picks which entity starts the next relation. The authors also say openly that steering changes whom the model reports, but doesn't clearly add correct relations.

  4. The method follows naturally from the diagnosis. It clearly improves precision on relations between the less-mentioned entities. The authors also report the variants that didn’t work, and that the gain does not extend to cross-sentence relations.

Critical weaknesses
  1. The method's gain is small and comes from one small test set. The full pipeline is only evaluated on BioRED, with 100 test abstracts, and the one transfer test, the sentence pass on BC5CDR, lowers F1. The paper says two runs of the same model can differ by up to 0.03 F1, so the 0.03 gains over the single-pass model and on DeepSeek-v4-pro are no larger than that run-to-run noise. I also couldn't find confidence intervals for the relation-F1 gains, even though the paper says they are reported for all method comparisons.

  2. The final pipeline doesn't find more of the missed relations than its first stage. The sentence pass finds more relations between the less-mentioned entities, but after the selector their recall drops to 0.44, below the 0.48 of the single anchored pass. For the 9B model it stays flat (0.39 to 0.38), and for DeepSeek-v4-pro it falls (0.41 to 0.34). So the abstract's 'without loss of recall' only holds against standard fine-tuning. I see the argument that the selector removes unmatched predictions and helps under the stricter scoring, but the recall loss matters given that missed relations are the paper's main concern.

  3. Cross-sentence relations get worse. They make up 29% of BioRED's test relations, and their F1 drops for all three models, for example from 0.40 to 0.35 for Qwen3.5-27B. The main text only says the gain is confined to same-sentence relations, and the conclusion calls this an open challenge. Only the appendix shows that these relations actually get worse.

  4. The comparisons are limited. The paper says BERT-style pair classifiers are still the strongest systems on BioRED, but none is scored end to end in Table 2, so it's hard to tell where 0.53 F1 stands. The idea of running extraction on evidence sentences alongside the full document is also close to EIDER (Xie et al., 2022), which isn't cited.

My weak accept is mainly for the analysis, which I see as the main contribution. I'm less convinced by the method, for the reasons above.

Questions for authors

Questions

  1. Is the protagonist effect a bias or a prior? Table 8 shows that protagonist pairs sharing a sentence are annotated as related much more often than other pairs (0.57 vs 0.19 on test). Removing about half of the protagonist training targets also nearly closes the gap (from 14 to 1 point in recall gap, on validation, Table 9). Table 7 suggests the preference doesn't fully track truth, but how do you tell a harmful bias apart from the model correctly using this base rate?

  2. The evidence anchor is derived from co-occurrence rather than annotated, and the selector reads the first sentence where both entities appear. On a manual sample, how often is that not the sentence that actually states the relation?

  3. Were the extractors trained with a single seed? The 0.03 run-to-run noise comes from re-decoding one checkpoint, so training variance is not covered by it.

  4. How does the method differ from EIDER (Xie et al., 2022), which also runs extraction on evidence sentences alongside the full document and combines the two?

  5. All three datasets are abstracts or short Wikipedia paragraphs, and on BioRED the protagonist is a title entity in 93 of 100 abstracts. Does the effect still hold in full-length papers, where mentions are spread across sections? And since the sentence pass already takes 11.5 prompts per abstract, how would its cost scale to a full paper?

Experiments I would like to see

  1. An end-to-end baseline under the same scorer. Right now every system in Table 2 is built on an LLM extractor, so it's hard to know whether 0.53 F1 is actually good. The paper itself says BERT-style pair classifiers are the strongest systems on BioRED, and it already trained one for Table 5. Running that classifier end to end, with an entity recognition step in front and the same scorer, would show whether the LLM pipeline is competitive or still behind.

  2. Repeated runs with confidence intervals. The paper says two runs of the same checkpoint can differ by up to 0.03 F1, and several of the reported gains are exactly that size. A few repeated runs of the standard fine-tune, the anchored pass and the full pipeline, for both Qwen models, with intervals for the F1 differences, would show which gains are real and make the main result much more convincing.

  3. A breakdown of what the selector removes. For Qwen3.5-27B, the sentence pass raises recall of supporting-entity relations to 0.61, but after the selector it is 0.44. Showing which correct relations are dropped and why, for example a wrong evidence sentence or the threshold, would show whether the recall loss can be fixed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jNA3 (1/2)

We thank the reviewer for the insightful comments!

Weakness 1, Question 3 and Experiment 2: Repeated runs, seeds and intervals. Thank you for these points. You are right that the submission used one training seed and gave no intervals for relation F1. We now report both. For both Qwen3.5-9B and Qwen3.5-27B, we also trained two more seeds of the standard and the anchored extractor and decoded each of them four times. The intervals come from a two-level bootstrap over test documents and runs or seeds.

BioRED test F1, averaged over repeats:

ModelRepeatsStandard fine-tuneFull pipelineΔF1 [95% CI]
Qwen3.5-9B3 seeds × 4 runs0.370.47+0.10 [+0.03, +0.16]
Qwen3.5-27B3 seeds × 4 runs0.470.53+0.06 [+0.02, +0.12]

Every run and every seed ranks the systems in the same order: full pipeline, anchored pass, standard fine-tune. The full pipeline beats the anchored pass in every run and every seed, by 0.011 to 0.049 F1, so this gain is not run-to-run noise. On 100 test abstracts, however, its interval includes zero. BC5CDR and Re-DocRED have 500 test documents each. There, the full pipeline beats a single pass in all six settings, including every DeepSeek-v4-pro setting, and every interval excludes zero.

Full pipeline − single pass, F1 [95% CI]:

DatasetExtractorSingle passFull pipelineΔF1 [95% CI]
BC5CDRQwen3.5-27B, fine-tuned on BC5CDR0.620.65+0.03 [+0.01, +0.04]
BC5CDRDeepSeek-v4-pro, few-shot0.630.66+0.03 [+0.01, +0.06]
BC5CDRDeepSeek-v4-pro, 0-shot0.590.64+0.05 [+0.03, +0.07]
Re-DocREDQwen3.5-27B, fine-tuned on Re-DocRED0.610.62+0.01 [+0.01, +0.02]
Re-DocREDDeepSeek-v4-pro, few-shot0.260.30+0.04 [+0.04, +0.05]
Re-DocREDDeepSeek-v4-pro, 0-shot0.190.21+0.02 [+0.02, +0.03]

Each comparison uses one extraction run, but each of the four selector seeds also beats the single pass in every setting. The BC5CDR drop in the submission came from scoring the candidate pool without a selector.

Revision: Section 4 now reports the repeated runs, and Appendix E gives both tables.

Weakness 2 and Experiment 3: Precision–recall trade-off. You are right that “without loss of recall” holds only against standard fine-tuning. Over repeated runs and seeds, the selector raises precision on supporting-entity relations by 0.16 to 0.19 over the single anchored pass. Their recall changes by only −0.04 to −0.01, which is not significant. The selector removes 94 correct supporting-entity pairs for Qwen3.5-27B. Of these, 37% join entities that never share a sentence, 15% fall just below the threshold, and 48% score low on a shared sentence. For example, the first sentence that mentions both iopamidol and iodixanol says that “no direct comparisons exist”. The selector scores the pair 0.20 and drops it, but the next sentence describes a trial comparing them. This happens because the selector reads only the first sentence the two entities share. Revision: The abstract no longer claims "without loss of recall". Appendix C reports the breakdown of removed relations and the gain from scoring every shared sentence.

Weakness 3: Cross-sentence relations Thank you for pointing this out. We agree that the main text should say directly that cross-sentence relations get worse. The cause lies in the design. The sentence pass adds a relation only when both entities appear in the same sentence. The selector also judges each pair from local sentences only. Cross-sentence pairs therefore gain no new candidates, and the selector removes some correct ones.

Revision: Section 4 now states that cross-sentence relations get worse, for example from 0.40 to 0.35 F1 for Qwen3.5-27B.

Question 4 and Weakness 4: Comparisons and EIDER. Thank you for pointing us to EIDER. Our method is designed around a failure that EIDER does not study. LLM extractors miss relations between supporting entities, because the protagonist shapes which relations they write. Each part of our method targets this failure. Evidence-anchored fine-tuning and the sentence pass give supporting entities more chances to open a relation. The selector then keeps a relation based only on its local evidence. The sentence pass is the part closest to EIDER, but it plays a different role. EIDER uses the local view to re-score pairs, and we use it to add new candidates. On Qwen3.5-27B, an EIDER-style variant of our pipeline, in which the selector only re-scores the whole-document candidates, reaches 0.49 F1, against 0.53 for our full pipeline. The gap is significant for every extractor we tested. The end-to-end BERT baseline is in our reply to Experiment 1.

Revision: EIDER is now cited in the Related Work part and the result is also shown in the Section 4 table 2. (Continued in Part 2 below.)

AuthorsOpenReview ↗

Response to Reviewer jNA3 (2/2)

Question 1: Bias or prior. Thank you for this sharp question. The preference starts as a prior, but the model applies it even when the evidence is clear, so it becomes a bias. The prior is real. Protagonist pairs are related more often (Table 8), and removing protagonist training targets nearly closes the gap (Table 9). Two results show that it goes beyond a prior. First, a correct prior would raise all protagonist relations equally, so true ones would still rank above fabricated ones. Instead, among protagonist pairs the model scores true and fabricated relations almost the same (AUC 0.55 to 0.60, against 0.67 to 0.74 for other pairs, Table 7). Second, when the stating sentence is very clear (selector probability at least 0.9), a reader that uses the base rate correctly reports both roles equally. Our extractors still recover fewer supporting-entity relations there, by 6 to 33 points, significantly in five of eight systems.

Question 2: Accuracy of the evidence sentence. For most sampled relations, the first shared sentence states the relation. We sampled 100 relations whose two entities share at least two sentences: 50 involving the protagonist and 50 between supporting entities. Two authors annotated the sample, with one author starting from language-model draft labels. They agreed on all 100 labels (100% inter-rater agreement). The first shared sentence states the relation in 76 cases; in the remaining 24, only a later shared sentence states it. The first-sentence rate is identical across groups, at 38 of 50 each. These findings support evaluating every shared sentence and retaining the highest selector score, whose benefit Appendix B reports.

Question 5: Full-length papers and cost. Thank you for this forward-looking question. The cost grows linearly with document length. Each sentence prompt holds only the title, one sentence and the fixed instructions, so its size does not depend on the length of the document. At our measured speed on one H200 GPU, a paper of 250 sentences would take about 2 minutes for Qwen3.5-9B and 4.5 minutes for Qwen3.5-27B, and the selector adds a few seconds. The protagonist is defined by mention counts, so it can be defined for a document of any length. We agree that full-length papers need their own test. There, the whole-document pass must run on sections, and each section may have its own most-mentioned entity. We leave this test to future work.

Experiment 1: End-to-end BERT baseline. Thank you for this suggestion. We ran it, and the LLM pipeline is competitive. We ran the PubMedBERT pair classifier from Table 5 end to end, with PubTator3 recognizing the entities, and scored it with the scorer of Table 2. It reaches 0.45 F1. Our 27B pipeline reaches 0.53 (difference 0.07, 95% CI 0.03 to 0.13), and our 9B pipeline reaches 0.46. Given the entities predicted by our 27B pipeline instead, the classifier reaches 0.46, again significantly below. Table 2 now includes both baselines.

Thanks for the thorough reply. I’m glad to see you’ve covered almost everything, and the corrected abstract, the BERT baseline and the repeated runs helps. To me the analysis is still the real strength of the paper, and that’s what my weak accept is based on, so I’ll keep my score as it is.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for your helpful suggestions and for acknowledging our response. We will keep all the changes in the final version, such as the abstract revision, the repeated experiment results with confidence intervals, and the BERT baseline. Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.