- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates the limitations of LLM-based biomedical knowledge graph construction, focusing on the disparity in relation extraction performance between prominent and supporting entities. The authors benchmark several open-weight and closed-source LLMs across 3 datasets and identify a consistent performance gap. Through controlled interventions and mechanistic analyses, they investigate how an entity's document role influences relation extraction, suggesting that the disparity primarily arises during entity selection rather than relation completion. To address this limitation, the authors propose a two-stage approach that separates relation candidate generation from verification. An evidence-anchored extractor processes the document sentence by sentence to generate candidate relations while a subsequent selection step retains only those supported by local textual evidence.
Critical strengths
- The paper investigates an important limitation of LLM-based knowledge graph construction, particularly the underrepresentation of relations involving less prominent entities. Beyond reporting extraction performance, the authors attempt to identify where the observed disparity originates through controlled interventions, activation patching, and attention manipulation.
- The paper proposes a two-stage approach that separates relation candidate generation from verification. An evidence-anchored extractor processes the document sentence by sentence to generate candidate relations while a subsequent selection step retains only those supported by local textual evidence.
Critical weaknesses
-
The main weakness of this work is the limited evidence establishing protagonist bias as the underlying cause of the observed performance disparity. The authors attribute the lower extraction performance on supporting entities to an entity-prominence bias (mention frequency, title position, and other document-level properties). However, this disparity could instead reflect greater model uncertainty due to the limited evidence available for less frequently mentioned entities. Although the attention interventions show that increasing attention to supporting entities makes them more likely to appear in extracted relations, this does not establish that protagonist prominence, rather than differences in supporting evidence, causes the original disparity. Furthermore, the authors acknowledge in Section 2.2 (“Simple alternatives do not account for the gap”) that masking protagonist mentions reduces protagonist relations without improving supporting-entity recall. This suggests that the observed gap cannot be explained by simple competition for attention or output capacity and may instead reflect more general limitations in the model ability to perform relation extraction.
-
The paper lacks comparisons with existing approaches that propose novel methods for biomedical relation extraction. For example, GLiM: Integrating Graph Transformer and LLM for Document-Level Biomedical Relation Extraction with Incomplete Labeling and PLRTE: Progressive Learning for Biomedical Relation Triplet Extraction Using Large Language Models are relevant methodological baselines.
-
The paper does not provide a comparison with specialized biomedical BERT-based relation extraction models. Previous studies like “Benchmarking Large Language Models for Biomedical Natural Language Processing Applications and Recommendations” and “Exploring Transformer Models: Fine-Tuning vs. Inference on Relation Extraction from Biomedical Texts” report that specialized biomedical models can outperform general-purpose LLMs for biomedical relation extraction tasks. Including such baselines would provide a more comprehensive assessment of how LLMs are positioned against BERT performance for biomedical relation extraction tasks.
-
The paper does not analyze the additional computational overhead introduced by the two-stage candidate generation and local evidence selection procedure. Reporting inference time and computational resources, would help assess the practicality and scalability of the approach.
Questions for authors
-
Figure 2: The caption begins with “Recall of protagonist relations” but the figure reports F1 scores. The terminology should be corrected to avoid confusion.
-
Interpretation of Figure 2: The performance gap between protagonist and supporting entities is not necessarily surprising. Central entities naturally occur more frequently and may have more relations and supporting contextual evidence, making their relations easier to extract. How do the authors distinguish a genuine entity-prominence bias from differences in evidence availability and relation difficulty?
-
In the controlled experiment for “Changing document role changes whether fixed evidence is reported”, the authors replaced the protagonist and the supporting entity name everywhere outside of the sentence that states the scored relation. How are these replacements performed while preserving the semantic coherence of the document? If the substitutions introduce inconsistencies or alter contextual evidence, the resulting performance changes could reflect increased model uncertainty rather than entity prominence alone.
-
The best configuration “Fine-tuned, evidence-anchored target + sentence pass + local-evidence selector” was only evaluated on one dataset, would this improvement generalize to other dataset? For instance, what about the other datasets BC5CDR and Re-DocRED?
-
The authors claim using both open-source and closed-source models for their experiments but they only use Qwen3.5-9B, Qwen3.5-27B and DeepSeek V4 Pro which are all open-source models.
4 comments
Response to Reviewer jv8P (2/2)
Question 1: Figure 2 caption. We corrected the caption of Figure 2, which now refers to F1.
Revision: Figure 2 caption
Question 3: Swap procedure and coherence. The drop in write-out after the swap comes from the change of protagonist, not from confusion caused by the edits. We exchange every mention of the protagonist and a supporting entity of the same type outside the stating sentence, including the title, and keep that sentence word for word. For example, one abstract becomes a study of acetaldehyde instead of ethanol, while "Long-term (4-week) exposure to 25 mM ethanol upregulated the Oct4 and Nanog proteins" stays unchanged. The fine-tuned Qwen3.5-27B writes the ethanol–Oct4 relation for the original but not after the swap, and it still writes it after a matched-null swap. Four checks show that the edits themselves do not cause such drops. Matched-null swaps edit as many mentions but exchange two other entities, and they change write-out by at most 2 points. In the same swapped documents, relations that involve neither swapped entity are written at least as often as before. Perplexity under an untuned Qwen3.5-9B, a direct measure of uncertainty about a text, rises by the same amount for real and matched-null swaps. Finally, we read 30 randomly drawn swapped documents. Apart from the intended name change, 23 read coherently, 7 had minor local oddities, and none contradicted the stating sentence.
Revision: Appendix D now describes the procedure with this example.
Question 4: Other datasets. The improvement carries over to both datasets. We applied the same recipe to each one. We fine-tuned Qwen3.5-27B with evidence-anchored targets on its training set, trained the selector on the same set, and froze the threshold on its validation set. Each test set was scored once with the scorer of Table 2. On both datasets, the full pipeline improves over a single pass of the same model.
| Dataset | Single pass | Full pipeline | Gain [95% CI] |
|---|---|---|---|
| BC5CDR | 0.62 | 0.65 | +0.03 [+0.01, +0.04] |
| Re-DocRED | 0.61 | 0.62 | +0.01 [+0.01, +0.02] |
Revision: Appendix E (Table 41) now reports both datasets.
Question 5: Closed-source results We agree this was unclear, because we accessed DeepSeek-v4-pro only through its API and grouped it with closed-source models. For completeness, Tables 1 and 2 now include the closed-source Claude Sonnet 5.5, which also recovers at most 41% of BioRED relations and shows the same protagonist gap.
Revision: Table 1 in Section 2 and Table 2 in Section 4 now displays experiment result with Claude.
I appreciate the authors efforts in clarifying my questions and completing the study with additional experiments. Most of my concerns have been addressed so I will change my score from 2 to 3.
For W3. it would be great if the authors also include results for replacing the first stage (mention detection) with a biomedical BERT baseline to clearly demonstrate the proposed pipeline is doing better at each stage of the relation extraction pipeline.
For Q3. I think it would be nice if the authors add a concrete example of swap procedure so that it's clear to readers that the coherence is not affected.
Thank you for your feedback
Thank you for carefully reading our responses and raising your recommendation. In the final version, we will add the results of our pipeline with entities recognized by a biomedical BERT model and a concrete swap example. Thank you again for your time and helpful suggestions.
Response to Reviewer jv8P (1/2)
We thank the reviewer for their constructive feedback!
Weakness 1 & Question 2: Protagonist bias or weaker evidence?
Thank you for raising the central question of the paper. The gap remains when the evidence is held fixed, so it cannot be reduced to weaker evidence. We agree that Figure 2 alone cannot separate these explanations, and we should have made our controls easier to find.
First, the swap holds the evidence fixed and changes only the role. The sentence that states the relation stays word for word the same, so its evidence and difficulty do not change. Yet the model writes the same relation 24 to 31 points less often once its entity is no longer the protagonist (Section 2.2). Over all swapped records, the drop is still 19 to 23 points (Appendix D.9). Our reply to Question 3 shows that the edits themselves do not cause this drop.
Second, the gap remains after controlling for evidence and difficulty. A regression holds fixed the number of shared sentences, mention count, position and relation type. Involving the protagonist still multiplies the odds of recovery by 2.0 to 3.6 (Table 6). We also added a control for how clearly the evidence states each relation, which we use as a proxy for difficulty. A classifier scores each relation from its shared sentence alone, and we compare the two roles at similar scores. The recall gap is still 15 to 28 points and significant in all eight systems (Table 10).
Third, the gap is not a general inability to extract these relations. We agree that masking rules out simple competition for output. In the fine-tuned extractors, making a supporting entity the protagonist raises the write-out of its relation by 22 to 26 points (Section 2.2). The same model can extract the same relation from the same sentence. What changes is the entity’s role.
Revision: Appendix A reports the new evidence-strength control in Table 10
Weakness 2: Comparison with GLiM and PLRTE Thank you for pointing us to GLiM and PLRTE. GLiM solves a different problem in a different setting. It is given the gold entities and targets incomplete training labels, with an LLM re-checking a BERT model’s borderline pairs. Our pipeline starts from raw text, and its LLM proposes candidates that a BERT model then filters. Following your suggestion, we ran GLiM’s released code on BioRED with gold entities and scored it with the scorer of Table 2. It reaches 0.45 F1, against 0.53 for our 27B pipeline, which receives no gold entities. Both baselines stay below our full pipeline. PLRTE is closer to our setting, because it also generates triplets from raw text. It reports no BioRED results and has not released its weights, so we compare on BC5CDR, where its 7B model reports 0.50 F1. Fine-tuned on BC5CDR, our pipeline reaches 0.62 F1 with Qwen3.5-9B, a model of similar size, and 0.645 with Qwen3.5-27B. PLRTE does not describe its scoring rule, so we report these numbers side by side without a significance test.
Revision: Related Work now discusses GLiM and PLRTE. Table 2 now includes the GLiM result, and Appendix C adds the comparison with PLRTE.
Weakness 3: Biomedical BERT models. Our pipeline is itself a specialized biomedical system, and end to end it outperforms a biomedical BERT classifier. Its LLM is fine-tuned on BioRED, and its final filter is BioLinkBERT, a biomedical BERT model. The studies you cite compare such fine-tuned models with general-purpose LLMs used without fine-tuning, on entity pairs given in advance. Our results agree with them. Without fine-tuning, the open-weight LLMs in Table 2 reach 0.24 to 0.42 F1. A PubMedBERT pair classifier, which labels every entity pair, reaches 0.45 on PubTator3 entities and 0.46 on the entities from our 27B pipeline, with the same scorer. Our 27B pipeline reaches 0.53, and both gaps are significant.
Weakness 4: Computational cost. Thank you for this practical point, and we agree that the cost should be reported. On one H200 GPU, the full pipeline takes 2.3 to 2.5 times as long as a single pass. This is about 10 seconds per abstract for Qwen3.5-9B and 20 seconds for Qwen3.5-27B, including model loading. The sentence pass sends 11.5 prompts per abstract, but each prompt and its output are short. It therefore takes only 1.3 to 1.5 times as long as the whole-document pass. The selector needs a few seconds for the whole test set.
(Continued in Part 2 below.)