acceptodds
Under review as a conference paper at ICLR 2027

Risk-Controlled Biomedical Claim Verification over Heterogeneous Evidence Sources

Abstract

Verifying a biomedical claim is adjudication across an ecosystem rather than lookup in a corpus. One assertion that a compound inhibits a target is recorded at once in abstracts, curated pharmacology records, interaction graphs and solved structures, each with its own provenance and update cadence, and these records disagree for structural rather than accidental reasons: a curated entry may assert an interaction that a later abstract negates under a different assay. Existing resources assume one corpus with one gold rationale, so they cannot express such conflict, and the verifiers built on them return unauditable labels alongside confidences that do not bound error. We introduce BioMSVerify, claims over four types with gold evidence spanning at least two source modalities, with conflicting sources, and Evidict, a verifier fusing a generative energy over typed evidence, a reliability-weighted and direction-preserving contradiction score, and a bidirectional process reward model multiplying each step's grounding by its value-to-go, then turning the fused score into a commit-or-abstain decision by group-conditional split conformal prediction. Three measurements motivate the design. Closed-book PubMedBERT reaches macro- overall and on multi-hop drug–target–pathway–phenotype claims, so accuracy cannot certify grounding. Given an identical evidence pool, a frontier reasoning LLM reaches while a M retrieval-grounded encoder reaches , so scale is not the bottleneck. And confidence is not risk: committing on temperature-scaled scores leaves a realized error of on conflicting-source claims, twice a target, whereas the conformal procedure keeps realized risk at or below for every claim type and on the conflict subset at three target levels. Evidict improves macro- by points on conflicting-source claims over the strongest baseline, is the most robust tested system to five counterfactual perturbation families, and costs as much at inference as GPT-4o.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Biomedical Claim Verification, Multi-Source Evidence Retrieval, Conformal Risk Control, Process Reward Models, Selective Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GvR1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a biomedical claim verification benchmark named BioMSVerify. It uses evidence from multiple sources. The dataset consists of 10,206 claims across 4 types, including 813 claims with conflicting evidence sources. Each claim has gold evidence spanning at least 2 modalities. The authors also propose EVIDICT, a verification method that includes generative scoring, cross-source contradiction detection, and a process reward model. Additionally, it uses conformal prediction to determine whether to return a label or abstain. EVIDICT is evaluated on BioMSVerify and 5 other benchmarks. It achieves 89.21 macro-F1 on BioMSVerify and 77.94 on the conflict test subset, outperforming the strongest baseline by 8.14 points on the conflict subset.

Critical strengths
  1. BioMSVerify is a meaningfull benchmark in the field of biomedical claim verification. It studies conflicting evidence across different sources, which is not well covered by existing benchmarks.

  2. The paper provides a comprehensive empirical evaluation, including comparisons across several benchmarks and model families. The ablation studies are informative, especially the large performance drop on conflicting claims as the contradiction score is removed.

Critical weaknesses
  1. The validity of Benchmark: The benchmark relies heavily on synthetic negative claims. Some claims are generated by replacing pathways with plausible but unsupported ones. However, a lack of supporting evidence does not necessarily mean that a biomedical claim is false. Although the authors report human annotation, it is unclear how they distinguish REFUTED from INSUFFICIENT in these cases. Also, the relatively high performance of closed-book PubMedBERT suggests that the dataset may contain shortcuts. These issues raise concerns about how well the benchmark evaluates evidence-based verification.

  2. Experimental comparisons: Generally speaking, the experiments do not fully isolate the performance of EVIDICT. Although the baselines use the same evidence pool, EVIDICT uses specialized encoders for different modalities and additional training. Therefore, the performance gains may come from these differences rather than the proposed verification components. In addition, the authors do not apply the same conformal prediction method to the strongest baselines. This makes it hard to determine whether EVIDICT provides better risk control than existing methods under a fair comparison.

Questions for authors
  1. How do the authors distinguish REFUTED from INSUFFICIENT claims when generating synthetic negative claims? For example, if a pathway is replaced with a plausible but unsupported one, how is it determined that the new claim is false rather than simply lacking evidence?

  2. The closed-book PubMedBERT achieves relatively high performance on several claim types. How do the authors evaluate the impact of shortcuts introduced by synthetic negative claims? Would the main results still hold on claims that require evidence to verify?

  3. Although the baselines use the same evidence pool, EVIDICT uses additional modality-specific encoders. Could the authors compare it with a simpler baseline using the same evidence representations? Also, could the same conformal prediction method be applied to the strongest baselines to compare their risk and abstention rates?

7 comments
AuthorsOpenReview ↗

Response to Reviewer GvR1 (1/3)

Dear Reviewer GvR1,

Thank you for your review. We answer your three questions in order with new experiments, all incorporated into the revised PDF (changes in blue; section and table numbers below refer to the revised version). In brief:

  • the main ranking holds after removing the disputed negative families (Conflict margin 6.19);
  • without evidence, EVIDICT's Conflict F1 falls to the closed-book level, so its advantage comes from the evidence;
  • with EVIDICT's own frozen representations, a simple model still trails by 4.82 Conflict points;
  • under an identical conformal protocol, EVIDICT abstains least among the strongest baselines.

Q1. How are REFUTED and INSUFFICIENT distinguished for synthetic negatives?

You are right that replacing a pathway with a plausible but unsupported one does not, by itself, make a claim false. In BioMSVerify, an operator only proposes a candidate claim; the label is decided by the evidence. REFUTED requires counterevidence under comparable qualifiers, and evidence that is merely silent yields INSUFFICIENT. This is why the same operator can yield either label: affinity perturbation gives REFUTED only when assay conditions are comparable and INSUFFICIENT otherwise (Section 3), and qualifier corruption gives INSUFFICIENT precisely because no record addresses the stated condition. Every generated claim was then validated by domain annotators against its gold evidence pointers, to which they could add items (Appendix A.5.1).

The one-line description of pathway substitution in Section 4.5 of the submission ("a plausible but unsupported pathway") describes how the operator picks the substitute, not the labeling criterion, but we agree it reads as if lack of support were sufficient. The revised version separates the two in Section 3 and gives the labeling criterion for each operator in a new table (Table 1).

To test whether our results depend on these disputed negatives, we rescored the saved predictions on two fixed subsets. Subsets are defined from construction logs only, never from model correctness or confidence, and all models are scored on the same IDs.

Test populationNSUP / REF / INSConflict N
Full test set1,868623 / 623 / 622149
w/o pathway-substitution negatives1,721623 / 476 / 622132
w/o entity-, pathway- and residue/interaction-generated negatives1,438623 / 193 / 622106
Overall / Conflict macro-F1BioLinkBERTDeepSeek-V4EVIDICT
Full test set87.82 / 60.2881.34 / 69.8089.21 / 77.94
w/o pathway substitution87.91 / 61.3281.82 / 70.1589.27 / 78.21
w/o entity, pathway, residue/interaction88.04 / 63.0182.47 / 70.4288.97 / 76.61

EVIDICT is best in every row. After the broadest exclusion, it still leads by 0.93 overall (over BioLinkBERT) and 6.19 on Conflict (over DeepSeek-V4), so the ranking does not rest on the disputed cases. Because the label distribution changes across rows, differences between rows should not be read as gains or losses, and the analysis is a sensitivity check rather than a validation of the retained labels. Both tables are in Appendix A.14.4 (Table 20) and summarized in Section 5.4. In addition, the dataset release marks the generating operator of every claim, so users can exclude any operator family, and we will re-audit the pathway-substitution REFUTED labels against the counterevidence criterion before release (Appendix A.6).

AuthorsOpenReview ↗

Response to Reviewer GvR1 (2/3)

Q2. Closed-book PubMedBERT and shortcuts: do the results hold on claims that need evidence?

We report the closed-book row precisely because of this risk; Appendix A.6 treats it as a benchmark artifact. The effect is not uniform across claim types. On DT, entity substitution yields counterfactuals that are as plausible a priori as the originals, and closed-book PubMedBERT is near chance (38.27); on Conflict it scores 36.41. These are the subsets where evidence is required, and EVIDICT is best on both: 90.74 vs 89.59 on DT, and 77.94 vs 69.80 on Conflict. On GDP, DTPP and ME (closed-book ≈ 70), label F1 partly measures prior knowledge, which is why Ev-F1 is reported alongside it.

We also ran an evidence intervention: each claim's evidence is replaced by items from other relations of the same type (same per-modality counts, no labels used) or removed, and all downstream scores are recomputed.

EvidenceBioLinkBERT Overall / ConflictEVIDICT Overall / Conflict
Original87.82 / 60.2889.21 / 77.94
From other relations62.74 / 39.8258.43 / 38.56
None55.86 / 35.7454.12 / 35.07

Without evidence, EVIDICT's Conflict F1 falls to the closed-book level, so its conflict advantage requires the evidence. These are agreements with the original labels after the intervention: they measure evidence sensitivity, not correctness under the replaced evidence. Together with the exclusion analysis in Q1, this supports a bounded claim that the main ranking does not depend on the disputed negatives. It does not certify an artifact-free benchmark; prior-matched counterfactuals remain the planned fix. The intervention results are in Appendix A.14.3 (Table 19) and summarized in Section 5.4.

Q3a. Same evidence pool, but different encoders and training: where does the gain come from?

An identical evidence pool does not mean identical representations or supervision, so we gave lightweight baselines EVIDICT's own frozen modality representations and detector outputs.

VerifierOverall F1Conflict F1
BioLinkBERT + retrieval87.8260.28
BioLinkBERT + retrieval + simple contradiction87.6968.36
Frozen EVIDICT representations + MLP88.1867.84
Frozen EVIDICT representations + MLP + simple contradiction88.6673.12
Full EVIDICT89.2177.94

"Simple contradiction" averages item stances uniformly and penalizes the polar verdicts by mean pairwise contradiction, with no learned reliability weights and no PRM. Your concern is partly borne out: the representations explain some of the gain, since an MLP on frozen features already beats BioLinkBERT overall. With the same representations, however, the scoring components still add 4.82 Conflict points (0.55 overall). This is Table 6a in Section 5.4 of the revised version (details in Appendix A.14.1). The ablations (Table 5), which keep the encoders fixed, point the same way: without the contradiction score Conflict F1 is 66.31, and with a plain classifier head in place of the generative energy it is 73.88.

AuthorsOpenReview ↗

Response to Reviewer GvR1 (3/3)

Q3b. The same conformal procedure for the strongest baselines

We applied split conformal prediction at α=0.10\alpha = 0.10 to BioLinkBERT, DeepSeek-V4 and the strongest lightweight control, using one protocol for all: the same calibration IDs, score 1−p(y)1 - p(y), finite-sample quantile rule, and claim-type × source-profile groups. Groups come from a shared frozen contradiction/evidence pipeline and never use the gold conflict label. Each model gets its own quantiles; temperatures and fusion weights are set on the selection half only. For DeepSeek-V4 we use its recorded three-class verbalized distributions (the self-consistency vote is used only for its F1). A singleton set commits; any other set, including the empty set, counts as abstention.

OverallMiscoverageAbstentionSelective errorAnswered correctly*
BioLinkBERT + retrieval0.0890.1020.08682.1%
DeepSeek-V4 + retrieval0.0870.2140.09571.1%
Frozen reps + MLP + simple contradiction0.0860.0720.07985.5%
EVIDICT0.0830.0540.08186.9%
ConflictMiscoverageAbstentionSelective errorAnswered correctly*
BioLinkBERT + retrieval0.0980.3980.10453.9%
DeepSeek-V4 + retrieval0.0940.2760.09265.7%
Frozen reps + MLP + simple contradiction0.0950.2310.08470.4%
EVIDICT0.0960.1860.09973.3%

*Share of all claims on which the method commits to the correct label, (1−abstention)(1−selective error)(1-\text{abstention})(1-\text{selective error}), computed from the rounded rates.

All four methods meet the miscoverage target. That is what the theory predicts: the guarantee comes from the procedure, not from EVIDICT, and the revised version states this explicitly. The difference lies in how many claims each verifier has to give up to meet it. EVIDICT abstains least (5.4% overall; 18.6% on Conflict vs 27.6% for DeepSeek-V4 and 39.8% for BioLinkBERT) and answers the most claims correctly. There is a cost: among committed predictions its selective error is slightly above the lightweight control overall (0.081 vs 0.079) and above DeepSeek-V4 on Conflict (0.099 vs 0.092). We read this as a better abstention–accuracy trade-off, not dominance on every safety metric.

On terminology: miscoverage counts prediction sets that miss the gold label, and selective error counts mistakes among committed predictions. Conformal prediction bounds miscoverage, and hence the probability of a wrong commitment, within each group under exchangeability; it does not directly bound selective error by α\alpha. The gold Conflict subset is not a calibration group, so its numbers are empirical. The revised version defines both terms in Section 4.5, uses them throughout (renaming "Risk" in Table 7 and Figure 2b to miscoverage), and adds the results above as Table 8 in Section 5.5, with the protocol in Appendix A.14.5.


If these additions address your concerns about benchmark validity and fair comparison, we would appreciate it if you would consider raising your score.

Best regards,

Authors

Reviewer_M48QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces EVIDICT, a biomedical claim-verification framework designed to adjudicate evidence from heterogeneous sources when those sources may conflict. The method combines a generative energy-based verifier, a reliability-weighted and direction-sensitive contradiction score, a bidirectional process reward model (PRM) intended to evaluate both evidence grounding and progress toward a verdict, and group-conditional split conformal prediction to support selective prediction through abstention. The authors also introduce BioMSVerify, a benchmark of 10,206 biomedical claims across four claim types, with evidence drawn from at least two modalities and 813 claims marked as involving conflicting sources.

The experiments compare EVIDICT with encoder-based verifiers, retrieval-augmented methods, and instruction-tuned or reasoning-oriented language models. EVIDICT achieves 89.21 macro-F1 on BioMSVerify, compared with 87.82 for the strongest overall baseline reported in the main table, and obtains a larger improvement on the conflict subset: 77.94 versus 69.80 for the strongest baseline on that subset. Ablation studies suggest that the contradiction score and PRM are particularly important for handling conflicting evidence. The conformal layer also reduces the reported overall risk by allowing the system to abstain on some claims.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important challenge in biomedical claim verification: evidence from heterogeneous sources may conflict, whereas existing resources and methods often assume a single corpus or a single gold rationale. The explicit inclusion of multiple evidence modalities and a conflict subset is a valuable contribution.

  2. Practical commit-or-abstain mechanism. The proposed mechanism is useful for biomedical applications, where making a definitive claim without sufficient supporting evidence may have potentially harmful consequences.

  3. Comprehensive framework. The energy model scores candidate verdicts, the contradiction score handles source disagreement, the PRM assesses reasoning steps, and conformal prediction controls when the system commits. This is a coherent architecture for a task where ordinary classification accuracy is not enough .The ablation studies provide evidence that the contradiction score and PRM contribute substantially to performance, particularly on claims involving conflicting evidence.

  4. Broad experimental evaluation. The paper evaluates the proposed method on six benchmarks and compares it with a range of baselines. The use of multiple training seeds and paired-bootstrap significance testing further strengthens the experimental evaluation.

Critical weaknesses
  1. Shortcut exploiting. Since REFUTED and INSUFFICIENT examples are generated using counterfactual operators and qualifier corruption, the benchmark may contain patterns that models can exploit without performing robust evidence-grounded verification. Although the paper evaluates several perturbation families, these overlap with the operators used during training. Evaluation on naturally occurring negative claims and unseen counterfactual transformations would provide stronger evidence of generalization.

  2. Potential cross-split leakage. The paper reports in its annex that splits are made at the level of the relation between conflict claims, but it doesn't explain whether related claims, entities or source documents can appear across different splits. If near-duplicate claims derived from the same underlying biomedical relation are distributed across splits, test performance may overestimate generalization to genuinely novel claims.

  3. Lack of a comprehensive human evaluation. The authors report a validation with domain annotators, with a Krippendorff’s αK = 0.81 on the validated subset. However, no details are provided about the process they used to validate them nor how the quality has been assessed. This is specially relevant because it doesn't allow us to see possible artifacts or problems during the creation of the benchmark. Moreover, a human assessment of step-level support, relevance, and inference validity would help substantiate the claim that EVIDICT produces auditable reasoning.

  4. Hallucination-risk during tuple generation. The pipeline relies on LLM-based claim normalization and entity linking before retrieval. Errors in this stage could propagate through the entire verification process. A direct evaluation of tuple accuracy, qualifier preservation, and entity-linking accuracy would help quantify this risk.

  5. No code, data or prompts have been released, reducing the reproducibility and transparency of this work.

Questions for authors
  1. Can you provide further details on the train/dev/test splitting procedure?
  2. Can you clarify the annotation protocol underlying the reported Krippendorff’s, including the size and selection of the validated subset, the number of annotators, and whether agreement varies across claim types or labels?
4 comments
AuthorsOpenReview ↗

Response to Reviewer M48Q (1/3)

Dear Reviewer M48Q,

Thank you for your review. We answer your two questions first, then the remaining weaknesses. All new analyses and clarifications below are incorporated into the revised PDF (changes in blue; section and table numbers below refer to the revised version).

Q1 / W2. Split procedure and possible cross-split leakage

The whole benchmark is split at the level of the underlying relation (train/dev/test = 6,794 / 1,544 / 1,868). Dev is further divided into 772 selection and 772 calibration claims; the calibration half is used only by the conformal layer. The 813 conflict claims are stratified across all three splits in proportion to split size, and every Conflict number we report uses the 149 in test.

To check for leakage, we audited four kinds of overlap:

  • parent-claim IDs, which group a supported claim with all variants generated from it;
  • canonical underlying-relation keys, taken from construction provenance, so a perturbed tuple stays linked to its source relation instead of counting as a new one;
  • normalized claim text, flagging test claims whose token 5-gram Jaccard similarity to any train/dev claim is ≥ 0.80;
  • source provenance (document or curated-record IDs).
Overlap checkCount
Parent-claim IDs in more than one split0
Canonical relation keys in more than one split0
Identical normalized claim strings across splits0
Test claims with 5-gram similarity ≥ 0.80 to train/dev14
Test claims sharing a source document/record ID with train/dev386

No relation and no generated variant crosses splits, so the near-duplicate scenario you describe (variants of one biomedical relation landing in different splits) does not occur. Removing the 14 lexically similar test claims (1,854 left) gives 89.08 for EVIDICT and 87.65 for BioLinkBERT, a gap of 1.43, essentially unchanged from the 1.39 on the full set.

Shared provenance (last row) has a different cause. A single curated record or abstract typically describes several distinct relations; for example, one DrugBank entry lists a compound's activity against multiple targets. Two claims citing the same record can therefore concern different relations with different labels. Sharing a record does not let a model memorize the test relation, but it does mean the split is relation-disjoint rather than document- or entity-disjoint. The revised version states this in Section 3 and adds the audit as Appendix A.5.2 (Table 11).

AuthorsOpenReview ↗

Response to Reviewer M48Q (2/3)

Q2 / W3. Annotation protocol behind αK=0.81\alpha_K = 0.81

You are right that Section 3 described two different things in one sentence: (i) the validation applied to every generated claim, in which domain annotators checked the claim against its gold evidence pointers and could add evidence items, and (ii) the multiply annotated subset on which agreement was computed. The revised version separates the two stages (Section 3, Appendix A.5.1). We will add the full protocol documentation in the final version: the annotators' background, the checks applied in the first stage, the size and sampling of the agreement subset, the adjudication procedure, and agreement broken down by claim type and by label.

Agreement alone does not show that every negative label follows the intended evidence criterion, which is also the core of your shortcut concern (W1 below). For that reason, Section 3 now states the labeling criterion for each operator (Table 1), and the dataset release includes the generating operator and the label provenance of every claim, so that users can filter or re-weight by construction type.

On step-level human evaluation: we agree it would be the right evidence for strong claims about reasoning quality, and we have scoped our auditability claim accordingly. What EVIDICT provides is inspectable intermediate steps, each tied to evidence pointers, together with the contested steps when it abstains (Figure 2a). Ev-F1 measures whether those pointers are correct, and EVIDICT leads every baseline on it (68.40 vs 63.70 for the best baseline and 44.80 for BioLinkBERT). The PRM grounding filter (pback>0.5p^{\text{back}} > 0.5) is a heuristic and carries no guarantee. The revised version defines "auditable" in exactly this sense in Section 4.4: every retained step can be checked against a concrete source, not that every step has been validated.

W4. Error propagation from LLM normalization and entity linking

We compared the pipeline's cached normalized tuples with the tuples stored at construction time, after applying the logged perturbation. Entity aliases are mapped to canonical IDs; relation/direction and explicit qualifier fields are compared separately.

Consistency with construction metadataEligible test claimsMatchAgreement
All entity IDs1,8681,79395.99%
Relation and direction1,8681,81797.27%
All explicit qualifiers preserved1,1561,06792.30%
Exact full tuple set1,8681,67089.40%

Entity linking and relation direction are reliable. Qualifier preservation (92.30%) is the weakest stage, which matters because qualifiers such as organism, assay and cell type decide whether two sources are comparable. Two remarks put these numbers in context. First, all reported results are end-to-end, so normalization errors are already reflected in EVIDICT's scores rather than hidden by gold tuples. Second, these figures measure agreement with construction metadata, which can itself be incomplete, so they are a consistency check rather than an independently annotated accuracy. The revised version reports this table in Appendix A.5.3 (Table 12) and names qualifier extraction as an upstream error source in the Limitations (Section 6).

AuthorsOpenReview ↗

Response to Reviewer M48Q (3/3)

W1. Shortcuts from counterfactual negatives; natural negatives and unseen transformations

We agree that the robustness study (Table 6 in the submission, now Table 17) uses the same operator families as training, as the submission noted, so it measures resistance to known perturbations rather than generalization to new ones. We added three analyses that bear directly on shortcuts.

(a) Excluding operator families from the test set. (Appendix A.14.4, Table 20.) Using construction logs only (never model outputs), we rescore saved predictions after removing (i) pathway-substitution negatives and (ii) all negatives produced by the entity, pathway or residue/interaction operators. Cells are Overall / Conflict macro-F1.

Test population (N)BioLinkBERTDeepSeek-V4EVIDICT
Full (1,868)87.82 / 60.2881.34 / 69.8089.21 / 77.94
w/o pathway substitution (1,721)87.91 / 61.3281.82 / 70.1589.27 / 78.21
w/o entity, pathway, residue/interaction (1,438)88.04 / 63.0182.47 / 70.4288.97 / 76.61

EVIDICT remains best in every row. With these three operator families removed, its Conflict margin over the strongest baseline is still 6.19 points.

(b) Replacing or removing evidence (Appendix A.14.3, Table 19). With evidence drawn from unrelated relations, EVIDICT's Conflict F1 falls from 77.94 to 38.56; with no evidence, to 35.07, close to closed-book PubMedBERT (36.41). A model exploiting claim-side artifacts would keep much of its score without evidence. EVIDICT does not, so its conflict gains come from reading the evidence.

(c) Naturally occurring negatives. Among the external benchmarks, PubHealth (claims from fact-checking and news-review sites) and HealthVer (real-world health claims) contain naturally occurring false claims, and EVIDICT improves over the strongest baseline on both (+1.30 and +1.80 F1). These are text-only, so they test transfer to natural negatives but not the multi-source setting.

A train-time leave-one-operator-out study would be the most direct test of unseen transformations. The revised version lists it, together with prior-matched counterfactuals (Appendix A.6), as the main open item in the Limitations (Section 6). Analyses (a) and (b) are summarized in Section 5.4.

W5. Code, data and prompts

As stated in the Reproducibility Statement, the code is included in the supplementary material. Upon publication, we will release the benchmark files with splits, per-claim operator and label provenance, all prompts (claim normalization, LLM judge and baseline prompts, with closed-model versions and access dates), and the evaluation configurations. In the revised version, the Reproducibility Statement lists each of these artifacts individually.

Other additions you may find relevant

Following the other reviews, the revised version also applies the same conformal protocol to the strongest baselines (Table 8) and adds same-representation controls (Table 6). Under identical calibration at α=0.10\alpha = 0.10, all methods meet the coverage target, and EVIDICT abstains least (18.6% on Conflict vs 27.6% for DeepSeek-V4 and 39.8% for BioLinkBERT). An MLP on EVIDICT's frozen representations with simple contradiction features reaches 73.12 on Conflict vs 77.94 for the full model. Details are in our responses to Reviewers GvR1 and tecB.


Thank you again for the constructive review. We hope the additions strengthen your assessment of the paper, and we would appreciate it if you would consider raising your score.

Best regards,

Authors

Thank you for the detailed clarifications and the additional analysis.

Q1/W2. I still believe that sharing source documents across splits (386 test claims) may pose a concern regarding the generalization assessment. I would suggest considering a document-level split, ensuring that claims derived from the same source document or curated record are assigned to the same partition. Although this would constitute a more challenging evaluation setting, it would provide stronger evidence of the model's ability to generalize to genuinely unseen sources.

Q2/W3. Thank you for the additional information regarding the annotation protocol. However, I still believe that a step-by-step human evaluation of the pipeline would be valuable to better assess the reliability of its individual components and the potential propagation of errors between stages.

W4. I appreciate the authors' transparency on this point, but I find the 92.30% agreement in qualifier preservation somewhat concerning.

Overall, although the additional analyses improve the transparency of the methodology and address some of my initial concerns, I believe that the reliability of the individual pipeline components remains insufficiently established. Therefore, I will maintain my original score.

Reviewer_tecBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces BioMSVerify, a new benchmark for biomedical claim verification using evidence from multiple sources, including scientific papers, databases, knowledge graphs, and molecular structures. The benchmark contains claims with conflicting evidence. The authors also propose EVIDICT, a verification framework that combines energy, contradiction scoring, and a reward model to evaluate claims and their supporting evidence.

The method uses conformal prediction to control prediction risk and abstain when the evidence is uncertain. Experiments on benchmarks show that EVIDICT outperforms most baselines, with larger improvements on claims involving conflicting evidence.

Critical strengths
  • The paper studies biomedical claim verification using evidence from multiple sources, which is an important and challenging task.
  • The authors introduce BioMSVerify, a benchmark with conflicting evidence.
  • EVIDICT achieves strong performance, especially on claims with conflicting evidence.
Critical weaknesses
  • The readability of this paper is a major concern. Many sentences are long and difficult to follow. I strongly suggest improving the clarity and readability of the paper.
  • Many negative claims are generated by using counterfactual perturbations, which may introduce shortcuts. Therefore, the benchmark cannot truly evaluate evidence-based reasoning.
  • It is unclear whether multiple sources are actually necessary to verify a claim. A single reliable source may already be sufficient in many cases. I suggest adding experiments that compare single-source and multi-source verification.
  • The proposed method combines many existing techniques, which may lack novelty. Moreover, this method is very complex. The paper lacks comparisons with simpler methods that may achieve similar performance.
Questions for authors
  • a BioLinkBERT can achieve very competitive performance, which undermines the key contribution of this work. How to explain this
  • the improvement of EVIDICT is larger on conflicting claims, it is unclear whether similar results could be achieved by adding a simple contradiction detection to a fine-tuned encoder
4 comments
AuthorsOpenReview ↗

Response to Reviewer tecB (1/3)

Dear Reviewer tecB,

Thank you for your review. We address each point below with new controls, all incorporated into the revised PDF (changes in blue; section and table numbers below refer to the revised version). They reuse the frozen checkpoints of the five submitted runs (only small classification heads are newly trained), and macro-F1 is computed on all test claims before abstention. In brief:

  • Simpler alternatives (W4, Q2): the contradiction baseline you suggested helps BioLinkBERT on Conflict (+8.08), but EVIDICT still leads it by 9.58 there; even with EVIDICT's own frozen representations, a simple model trails by 4.82.
  • Multiple sources (W3): at the same 8-item budget, combining modalities beats the best dev-selected single source by 12.94 Conflict points.
  • Shortcuts (W2): without evidence, EVIDICT's Conflict F1 falls to the closed-book level, and its ranking holds after removing the disputed negative families.
  • BioLinkBERT (Q1): the overall gap is small, but EVIDICT is +17.66 on Conflict and +23.60 on evidence selection.

W1. Readability

Agreed. The abstract and Section 4 pack too many qualifiers into single sentences, and several terms are used before they are defined. In the revised version:

  • Section 4 is reorganized around the four things the system does, one subsection each (Sections 4.2–4.5): (1) score each candidate verdict, (2) aggregate disagreement between sources, (3) check each reasoning step, (4) return a verdict or abstain. Each subsection opens with one plain sentence saying what it computes and why.
  • Retrieval and encoder specifications and the full training objective are moved to Appendix A.3, and a notation table is added (Table 9).
  • Conformal prediction is presented as the established procedure it is (Sections 1 and 4.5), so that what is ours (the evidence and step scoring) is easier to see.
  • Long sentences are split throughout. For example, the first sentence of the abstract changes from "Verifying a biomedical claim is adjudication across an ecosystem rather than lookup in a corpus" to "The same biomedical claim is often recorded in several places at once (papers, curated databases, interaction graphs and protein structures), and these sources sometimes disagree."

Q1. BioLinkBERT is competitive. Does this undermine the contribution?

On overall F1 the margin is small (+1.39), and the paper says so (Sections 1 and 5.2). But overall F1 is the column where BioMSVerify is least able to separate verifiers, because most non-conflict claims can be labeled correctly from a single source. The differences appear where the benchmark was built to discriminate:

BioLinkBERT (+R)EVIDICTΔ
Overall macro-F187.8289.21+1.39
Conflict macro-F160.2877.94+17.66
Evidence-selection F1 (Ev-F1)44.8068.40+23.60
Abstention on Conflict, same conformal protocol (α=0.10\alpha=0.10)39.8%18.6%−21.2 pts

Two points. First, Ev-F1: BioLinkBERT often reaches the right label while citing the wrong evidence. For a verifier whose output a curator is meant to check, that is a failure a label-only metric cannot see. Second, the last row is new (Table 8; protocol in our response to Reviewer GvR1): under an identical calibration procedure, BioLinkBERT has to abstain on about 40% of conflict claims to meet the same risk target, against 18.6% for EVIDICT.

We also think the strength of a 110M encoder is worth reporting in its own right. Given the identical evidence pool, it beats every frontier LLM we tested (best: 81.34). We present this as one of the paper's findings.

AuthorsOpenReview ↗

Response to Reviewer tecB (2/3)

W4 + Q2. Novelty, complexity, and whether simple contradiction detection on a fine-tuned encoder is enough

We ran the baseline you suggested, plus a stronger control (Section 5.4, Table 6a; details in Appendix A.14.1).

  • BioLinkBERT + retrieval + simple contradiction. It receives the same frozen stance and contradiction predictions as EVIDICT, averages item stances uniformly, and penalizes the two polar verdicts in proportion to mean pairwise contradiction. The fusion weight is chosen on the selection half of dev. No learned source reliability, no PRM.
  • Frozen EVIDICT representations + MLP. We freeze EVIDICT's modality encoders and projections and train a small MLP on the claim representation, pooled per-modality representations and missing-modality indicators. A second variant adds the same simple stance/contradiction summaries. Both use the original training labels.
VerifierOverall F1Conflict F1
BioLinkBERT + retrieval87.8260.28
BioLinkBERT + retrieval + simple contradiction87.6968.36
Frozen EVIDICT representations + MLP88.1867.84
Frozen EVIDICT representations + MLP + simple contradiction88.6673.12
Full EVIDICT89.2177.94

Simple contradiction detection adds 8.08 Conflict points to BioLinkBERT (overall F1 changes by −0.13). With EVIDICT's own representations, the simple version reaches 73.12. The remaining gap is 0.55 overall and 4.82 on Conflict. In other words, the extra machinery pays off almost only on conflicting evidence, and the revised version narrows our claim to exactly this.

The ablations (Table 5) agree while keeping the encoders fixed: removing the contradiction score lowers Conflict F1 from 77.94 to 66.31, and removing only the PRM's value-to-go term lowers it to 71.85. Neither control reproduces the full backbone or the auxiliary objectives, so we do not claim EVIDICT is the only design that works. Section 5.4 reports these controls and recommends the strongest one as a baseline for BioMSVerify.

On novelty: we agree that conformal prediction, DPO and generative verifiers are existing tools, and the revised version presents them that way. What we consider new is:

  1. The benchmark setting. Disagreement between sources is part of the ground truth. No existing claim-verification benchmark can express this: SciFact, HealthVer and PubHealth pair each claim with one text corpus, and SciVer draws its evidence from a single paper.
  2. The contradiction score. It is reliability-weighted and direction-preserving (an item reporting activation counts against an inhibition claim instead of as a generic "association"). Its conflict mass κ\kappa penalizes only the two polar verdicts, so disagreement between two reliable sources pushes the decision toward INSUFFICIENT.
  3. The PRM factorization. Each step is scored as evidence grounding × value-to-go, so a grounded step that leads nowhere is still penalized. The value-to-go term alone is worth 6.09 Conflict points.

On complexity at inference: the full test split costs $0.51 at 0.56 s per claim on one A100, against $41.20 for GPT-4o (Table 18).

AuthorsOpenReview ↗

Response to Reviewer tecB (3/3)

W3. Is more than one source needed?

We restrict the evidence given to the same frozen verifier, under a common budget of at most eight items in total, so "more sources" never means "more items". All conditions draw from the same retrieval candidates: single-modality conditions take that source's top-ranked items, and the multi-modality condition interleaves the ranks in a fixed order. As the strongest single-source policy, we also choose the best modality per claim type on the selection half of dev and fix it for test. Claims with no evidence from the chosen source stay in the evaluation with an empty input.

Evidence available (≤ 8 items)Overall F1Conflict F1
Text only83.9462.16
Curated records only84.8264.35
Knowledge graph only80.7360.28
Structure only76.4853.74
Best modality per claim type (chosen on dev)85.3663.91
All modalities88.7476.85

At equal budget, combining modalities beats the dev-selected single-source policy by 3.38 overall and 12.94 on Conflict. Your point holds in part: one good source gets within about 3–4 points overall. It falls well short when sources disagree. Two caveats: this restricts the inputs of one verifier rather than training separate single-source models, and the budget differs from the submitted configuration (top 8 per source), so its all-modality number is not the main-table number. This is Table 6b in the revised version (details in Appendix A.14.2).

W2. Counterfactual negatives and shortcuts

The paper already flags one artifact of this kind (Section 5.2, Appendix A.6). We added two tests.

(a) Does the verifier use the evidence? (Appendix A.14.3, Table 19.) We replace each claim's evidence with items from other relations of the same claim type (same number of items per modality, no labels used), or remove it, and recompute everything downstream.

EvidenceBioLinkBERT Overall / ConflictEVIDICT Overall / Conflict
Original87.82 / 60.2889.21 / 77.94
From other relations62.74 / 39.8258.43 / 38.56
None55.86 / 35.7454.12 / 35.07

Without evidence, EVIDICT falls to 35.07 on Conflict, about the closed-book PubMedBERT level (36.41), so its Conflict advantage comes from the evidence and not from the claim text. It also drops more than BioLinkBERT under replacement (−39.4 vs −20.5 on Conflict), i.e., it relies more on what it is given. These numbers are agreement with the original labels after an intervention, so they show evidence sensitivity, not correctness under the new evidence.

(b) Do the results depend on the disputed negatives? (Appendix A.14.4, Table 20.) Using construction logs only (never model outputs), we drop all negatives produced by entity substitution, pathway substitution and residue/interaction corruption, leaving 1,438 test claims. EVIDICT still leads by 0.93 overall over BioLinkBERT and by 6.19 on Conflict over DeepSeek-V4, the strongest Conflict baseline. The full table is in our response to Reviewer GvR1.

Both analyses are summarized in Section 5.4. Neither shows generalization to operators unseen in training, which the revised version states as a limitation (Section 6).


These controls narrow our claim to where the method adds value: conflict adjudication and evidence selection. If they resolve your concerns, we would appreciate it if you would reconsider your score.

Best regards,

Authors

Thank you so much for the response. From what I can see, the major issue is that the benefit of using EVIDICT is not clear. Simple baselines such as BioLinkBERT can achieve very competitive performance while the proposed method is complicated, introducing some complexity. I am also not convinced by the benchmark design, which simply uses counterfactual perturbations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

Dear Reviewers and Area Chair,

We thank all reviewers for their feedback. Several concerns were shared across reviews: whether synthetic negatives create shortcuts (all reviewers), whether the gains come from the verifier or from encoders and simpler alternatives (tecB, GvR1), whether risk control is compared fairly (GvR1), and how the benchmark was built and split (M48Q). We ran new analyses for each and have uploaded a revised PDF. All changes are highlighted in blue, and the main text stays within the 10-page limit for the discussion phase. The analyses reuse the frozen checkpoints of the five submitted runs; the only newly trained models are small classification heads.

Summary of new analyses (locations refer to the revised PDF)

AnalysisQuestionMain resultWhereReviewers
Simple contradiction baseline and same-representation MLPCan a simpler verifier recover the gains?Partly. The best lightweight control reaches 88.66 / 73.12 (Overall / Conflict) vs 89.21 / 77.94 for EVIDICTSec. 5.4, Table 6atecB, GvR1
Single- vs multi-source at a fixed 8-item budgetAre multiple sources needed?All modalities beat the best dev-selected single source by +3.38 overall and +12.94 on ConflictSec. 5.4, Table 6btecB
Evidence replacement and removalDoes the verifier use the evidence?EVIDICT Conflict F1: 77.94 → 38.56 (mismatched) → 35.07 (none)App. A.14.3, Table 19all
Excluding disputed negative familiesDo results depend on questionable negatives?Ranking unchanged; Conflict margin 8.14 → 6.19App. A.14.4, Table 20all
Same conformal protocol for baselinesIs risk control compared fairly?All methods meet the target; EVIDICT abstains least (18.6% vs 27.6–39.8% on Conflict), with slightly higher selective errorSec. 5.5, Table 8GvR1, tecB
Cross-split overlap auditIs there leakage?0 shared relations or parent claims; 386 test claims share a source documentApp. A.5.2, Table 11M48Q
Normalization consistencyDo LLM tuples propagate errors?Entities 95.99%, relation/direction 97.27%, qualifiers 92.30%App. A.5.3, Table 12M48Q

How the claims are narrowed in the revised version

  • The contribution is evidence-sensitive conflict adjudication and evidence selection. The overall-F1 margin over a fine-tuned encoder is small, and the abstract, introduction and conclusion say so.
  • Conformal prediction is presented as an established procedure (Sec. 4.5). Its guarantee applies to any score; what EVIDICT changes is how often it must abstain. Miscoverage and selective error are defined separately and used consistently.
  • The split is relation-disjoint, not document- or entity-disjoint (Sec. 3).
  • "Auditable" means inspectable steps with evidence pointers, not validated reasoning (Sec. 4.4).
  • Section 3 separates the operator that proposes a negative from the evidence criterion that assigns its label, with the criterion for each operator in Table 1.
  • A Limitations paragraph (Sec. 6) lists the open items: unseen perturbation operators, document-level overlap, qualifier extraction, and step-level human evaluation.

Presentation changes

Section 4 is reorganized around four operations, one subsection each (Secs. 4.2–4.5: score verdicts, aggregate disagreement, check steps, commit or abstain). Retrieval and encoder details and the full training objective move to Appendix A.3, with a notation table (Table 9). The two annotation stages are now described separately (A.5.1; full protocol documentation will follow in the final version), and a split audit (A.5.2), a normalization check (A.5.3) and an itemized Reproducibility Statement are added. To stay within 10 pages, the robustness table moves to Appendix A.11 (Table 17); its findings are summarized in Section 5.6.

Detailed responses are posted under each review.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.