- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a biomedical claim verification benchmark named BioMSVerify. It uses evidence from multiple sources. The dataset consists of 10,206 claims across 4 types, including 813 claims with conflicting evidence sources. Each claim has gold evidence spanning at least 2 modalities. The authors also propose EVIDICT, a verification method that includes generative scoring, cross-source contradiction detection, and a process reward model. Additionally, it uses conformal prediction to determine whether to return a label or abstain. EVIDICT is evaluated on BioMSVerify and 5 other benchmarks. It achieves 89.21 macro-F1 on BioMSVerify and 77.94 on the conflict test subset, outperforming the strongest baseline by 8.14 points on the conflict subset.
Critical strengths
-
BioMSVerify is a meaningfull benchmark in the field of biomedical claim verification. It studies conflicting evidence across different sources, which is not well covered by existing benchmarks.
-
The paper provides a comprehensive empirical evaluation, including comparisons across several benchmarks and model families. The ablation studies are informative, especially the large performance drop on conflicting claims as the contradiction score is removed.
Critical weaknesses
-
The validity of Benchmark: The benchmark relies heavily on synthetic negative claims. Some claims are generated by replacing pathways with plausible but unsupported ones. However, a lack of supporting evidence does not necessarily mean that a biomedical claim is false. Although the authors report human annotation, it is unclear how they distinguish REFUTED from INSUFFICIENT in these cases. Also, the relatively high performance of closed-book PubMedBERT suggests that the dataset may contain shortcuts. These issues raise concerns about how well the benchmark evaluates evidence-based verification.
-
Experimental comparisons: Generally speaking, the experiments do not fully isolate the performance of EVIDICT. Although the baselines use the same evidence pool, EVIDICT uses specialized encoders for different modalities and additional training. Therefore, the performance gains may come from these differences rather than the proposed verification components. In addition, the authors do not apply the same conformal prediction method to the strongest baselines. This makes it hard to determine whether EVIDICT provides better risk control than existing methods under a fair comparison.
Questions for authors
-
How do the authors distinguish REFUTED from INSUFFICIENT claims when generating synthetic negative claims? For example, if a pathway is replaced with a plausible but unsupported one, how is it determined that the new claim is false rather than simply lacking evidence?
-
The closed-book PubMedBERT achieves relatively high performance on several claim types. How do the authors evaluate the impact of shortcuts introduced by synthetic negative claims? Would the main results still hold on claims that require evidence to verify?
-
Although the baselines use the same evidence pool, EVIDICT uses additional modality-specific encoders. Could the authors compare it with a simpler baseline using the same evidence representations? Also, could the same conformal prediction method be applied to the strongest baselines to compare their risk and abstention rates?
7 comments
Response to Reviewer GvR1 (2/3)
Q2. Closed-book PubMedBERT and shortcuts: do the results hold on claims that need evidence?
We report the closed-book row precisely because of this risk; Appendix A.6 treats it as a benchmark artifact. The effect is not uniform across claim types. On DT, entity substitution yields counterfactuals that are as plausible a priori as the originals, and closed-book PubMedBERT is near chance (38.27); on Conflict it scores 36.41. These are the subsets where evidence is required, and EVIDICT is best on both: 90.74 vs 89.59 on DT, and 77.94 vs 69.80 on Conflict. On GDP, DTPP and ME (closed-book ≈ 70), label F1 partly measures prior knowledge, which is why Ev-F1 is reported alongside it.
We also ran an evidence intervention: each claim's evidence is replaced by items from other relations of the same type (same per-modality counts, no labels used) or removed, and all downstream scores are recomputed.
| Evidence | BioLinkBERT Overall / Conflict | EVIDICT Overall / Conflict |
|---|---|---|
| Original | 87.82 / 60.28 | 89.21 / 77.94 |
| From other relations | 62.74 / 39.82 | 58.43 / 38.56 |
| None | 55.86 / 35.74 | 54.12 / 35.07 |
Without evidence, EVIDICT's Conflict F1 falls to the closed-book level, so its conflict advantage requires the evidence. These are agreements with the original labels after the intervention: they measure evidence sensitivity, not correctness under the replaced evidence. Together with the exclusion analysis in Q1, this supports a bounded claim that the main ranking does not depend on the disputed negatives. It does not certify an artifact-free benchmark; prior-matched counterfactuals remain the planned fix. The intervention results are in Appendix A.14.3 (Table 19) and summarized in Section 5.4.
Q3a. Same evidence pool, but different encoders and training: where does the gain come from?
An identical evidence pool does not mean identical representations or supervision, so we gave lightweight baselines EVIDICT's own frozen modality representations and detector outputs.
| Verifier | Overall F1 | Conflict F1 |
|---|---|---|
| BioLinkBERT + retrieval | 87.82 | 60.28 |
| BioLinkBERT + retrieval + simple contradiction | 87.69 | 68.36 |
| Frozen EVIDICT representations + MLP | 88.18 | 67.84 |
| Frozen EVIDICT representations + MLP + simple contradiction | 88.66 | 73.12 |
| Full EVIDICT | 89.21 | 77.94 |
"Simple contradiction" averages item stances uniformly and penalizes the polar verdicts by mean pairwise contradiction, with no learned reliability weights and no PRM. Your concern is partly borne out: the representations explain some of the gain, since an MLP on frozen features already beats BioLinkBERT overall. With the same representations, however, the scoring components still add 4.82 Conflict points (0.55 overall). This is Table 6a in Section 5.4 of the revised version (details in Appendix A.14.1). The ablations (Table 5), which keep the encoders fixed, point the same way: without the contradiction score Conflict F1 is 66.31, and with a plain classifier head in place of the generative energy it is 73.88.
Response to Reviewer GvR1 (3/3)
Q3b. The same conformal procedure for the strongest baselines
We applied split conformal prediction at to BioLinkBERT, DeepSeek-V4 and the strongest lightweight control, using one protocol for all: the same calibration IDs, score , finite-sample quantile rule, and claim-type × source-profile groups. Groups come from a shared frozen contradiction/evidence pipeline and never use the gold conflict label. Each model gets its own quantiles; temperatures and fusion weights are set on the selection half only. For DeepSeek-V4 we use its recorded three-class verbalized distributions (the self-consistency vote is used only for its F1). A singleton set commits; any other set, including the empty set, counts as abstention.
| Overall | Miscoverage | Abstention | Selective error | Answered correctly* |
|---|---|---|---|---|
| BioLinkBERT + retrieval | 0.089 | 0.102 | 0.086 | 82.1% |
| DeepSeek-V4 + retrieval | 0.087 | 0.214 | 0.095 | 71.1% |
| Frozen reps + MLP + simple contradiction | 0.086 | 0.072 | 0.079 | 85.5% |
| EVIDICT | 0.083 | 0.054 | 0.081 | 86.9% |
| Conflict | Miscoverage | Abstention | Selective error | Answered correctly* |
|---|---|---|---|---|
| BioLinkBERT + retrieval | 0.098 | 0.398 | 0.104 | 53.9% |
| DeepSeek-V4 + retrieval | 0.094 | 0.276 | 0.092 | 65.7% |
| Frozen reps + MLP + simple contradiction | 0.095 | 0.231 | 0.084 | 70.4% |
| EVIDICT | 0.096 | 0.186 | 0.099 | 73.3% |
*Share of all claims on which the method commits to the correct label, , computed from the rounded rates.
All four methods meet the miscoverage target. That is what the theory predicts: the guarantee comes from the procedure, not from EVIDICT, and the revised version states this explicitly. The difference lies in how many claims each verifier has to give up to meet it. EVIDICT abstains least (5.4% overall; 18.6% on Conflict vs 27.6% for DeepSeek-V4 and 39.8% for BioLinkBERT) and answers the most claims correctly. There is a cost: among committed predictions its selective error is slightly above the lightweight control overall (0.081 vs 0.079) and above DeepSeek-V4 on Conflict (0.099 vs 0.092). We read this as a better abstention–accuracy trade-off, not dominance on every safety metric.
On terminology: miscoverage counts prediction sets that miss the gold label, and selective error counts mistakes among committed predictions. Conformal prediction bounds miscoverage, and hence the probability of a wrong commitment, within each group under exchangeability; it does not directly bound selective error by . The gold Conflict subset is not a calibration group, so its numbers are empirical. The revised version defines both terms in Section 4.5, uses them throughout (renaming "Risk" in Table 7 and Figure 2b to miscoverage), and adds the results above as Table 8 in Section 5.5, with the protocol in Appendix A.14.5.
If these additions address your concerns about benchmark validity and fair comparison, we would appreciate it if you would consider raising your score.
Best regards,
Authors
Response to Reviewer GvR1 (1/3)
Dear Reviewer GvR1,
Thank you for your review. We answer your three questions in order with new experiments, all incorporated into the revised PDF (changes in blue; section and table numbers below refer to the revised version). In brief:
Q1. How are REFUTED and INSUFFICIENT distinguished for synthetic negatives?
You are right that replacing a pathway with a plausible but unsupported one does not, by itself, make a claim false. In BioMSVerify, an operator only proposes a candidate claim; the label is decided by the evidence. REFUTED requires counterevidence under comparable qualifiers, and evidence that is merely silent yields INSUFFICIENT. This is why the same operator can yield either label: affinity perturbation gives REFUTED only when assay conditions are comparable and INSUFFICIENT otherwise (Section 3), and qualifier corruption gives INSUFFICIENT precisely because no record addresses the stated condition. Every generated claim was then validated by domain annotators against its gold evidence pointers, to which they could add items (Appendix A.5.1).
The one-line description of pathway substitution in Section 4.5 of the submission ("a plausible but unsupported pathway") describes how the operator picks the substitute, not the labeling criterion, but we agree it reads as if lack of support were sufficient. The revised version separates the two in Section 3 and gives the labeling criterion for each operator in a new table (Table 1).
To test whether our results depend on these disputed negatives, we rescored the saved predictions on two fixed subsets. Subsets are defined from construction logs only, never from model correctness or confidence, and all models are scored on the same IDs.
EVIDICT is best in every row. After the broadest exclusion, it still leads by 0.93 overall (over BioLinkBERT) and 6.19 on Conflict (over DeepSeek-V4), so the ranking does not rest on the disputed cases. Because the label distribution changes across rows, differences between rows should not be read as gains or losses, and the analysis is a sensitivity check rather than a validation of the retained labels. Both tables are in Appendix A.14.4 (Table 20) and summarized in Section 5.4. In addition, the dataset release marks the generating operator of every claim, so users can exclude any operator family, and we will re-audit the pathway-substitution REFUTED labels against the counterevidence criterion before release (Appendix A.6).