- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BioTraceBench, a benchmark for evaluating AI systems on biomedical research reasoning. It is built from 60 expert-curated studies that were decomposed into about a thousand local conclusion nodes. The benchmark evaluates three capabilities as separate tasks: reproducing a paper's computational analyses, inferring local conclusions from figures and essential context, and synthesizing a study's main conclusion from its local conclusions. Each task has its own inputs, references and scoring. The claimed contributions are this unified representation, expert-reviewed task construction and traceable scoring. The experiments report descriptive model comparisons on each task, with Task 1 evaluated on a 15-task subset. The LLM judges are compared with blinded expert ratings for Task 2, and an ablation shows that models almost always abstain when the required figures are removed.
Critical strengths
-
Problem decomposition. The paper evaluates computational execution, evidence interpretation and conclusion synthesis as separate tasks, whereas existing benchmarks often combine them into a single score. This is the main contribution I see, as it makes the results easier to interpret.
-
Expert involvement. The tasks are constructed and reviewed by domain researchers, which grounds the benchmark in real biomedical studies rather than synthetic problems.
-
Scoped claims. The authors are explicit about what their scores measure and what they do not, and they avoid overstating the results.
Critical weaknesses
-
Limited judge validation. Human validation covers only Task 2, so the Task 3 judge decisions behind the primary metric cannot be distinguished from judge error. The judges in both tasks are also evaluated models, and in Task 2 they agree with each other more than with the experts, which raises concerns about shared bias.
-
Dependence on reference quality. The judges treat the reference as the only correct target, so any overclaim in a source paper becomes part of the target and a more cautious answer is penalized. The construction checks intended to prevent this are not documented in enough detail to assess.
-
Limited support for the diagnostic claim. Each task evaluates a different set of models and the paper provides no error analysis, so the benchmark does not yet show where specific models fail across tasks. In addition, Task 1 covers only 15 of its 60 tasks and no significance testing is reported, so the rankings are descriptive only.
Questions for authors
-
How are reference conclusions handled when a source paper's claim goes beyond its evidence? Could the authors report how many references were revised from the original claims, and clarify whether an answer that is more cautious than the reference is penalized?
-
Could the authors explain the choice of judge models? All three Task 2 judges are also evaluated models and score below the top two, and the reason for excluding the earlier Claude judging is not given.
-
Do the authors plan to release the full benchmark, including the figures and task packages? If so, under what license, given the third-party rights noted in the reproducibility statement?
-
Evaluating the same models on all three tasks, and auditing a sample of Task 3 judge decisions against expert ratings, would strengthen the diagnostic claim.
4 comments
Authors‘s Comments (2/2)
Q1. Reference handling under overclaim; how many revised; are more-cautious answers penalized?
Construction-side: Task 2 requires experts to confirm each reference is supported by evidence and passes a 3-person masked answerability check ; we will count and report how many references were revised or weakened during this process. Evaluation-side: a more cautious answer is not penalized on correctness — boundary only penalizes explicit overreach, and volunteering extra caution is not deducted; only a gold-stated limitation left uncovered affects coverage. We will illustrate this rule with an example in the main text to remove the "more cautious = more penalized" misreading.
Q2. Judge-model choice; lower-ranked models as judges; excluding earlier Claude judging
Judge selection balanced availability, self-evaluation avoidance, and judging capability; critically, the panel was not arbitrary — it was validated through a 20-item judge-selection pilot (balanced ground truth, three experts applying the same rule as the judges). Results:
| Candidate judge | Valid judgments | Key behavior | Outcome |
|---|---|---|---|
| GPT-5.6 Sol | 4/20 | biosafety refusals (16/20) | excluded |
| Opus 5 | 8/20 | biosafety refusals (12/20) | excluded |
| DeepSeek v4.1 Flash | 20/20 | over-lenient: 18 passed, 9 false accepts (90% FPR) | excluded as judge |
| Kimi K3 | 20/20 | precision 0.89 / recall 0.80 | selected |
| GLM 5.3 | 20/20 | precision 0.75 / recall 0.90 | selected |
| Qwen 3.8 Max | 20/20 | precision 0.91 / recall 1.00 | selected |
The three selected judges are complementary (Qwen high precision, GLM high recall, Kimi balanced), so the two-of-three majority rule balances false accepts against false rejects.
On "lower-ranked models serving as judges": judging capability (not refusing, not over-lenient, faithful discrimination) is distinct from a model's task score as an answerer; the two need not coincide. Moreover, self-evaluation bias is a known but direction-uncertain threat — a model may score its own outputs higher or judge same-style outputs more harshly; because the direction cannot be predicted, we mitigate it structurally with multiple judges and a two-of-three majority rather than relying on any single judge.
On excluding the earlier Claude judging (line 327): two reasons. (i) High refusal rate — in a Task 2 run over 160 items, only 54 returned valid judgments; (ii) it used a deprecated one-pass prompt whose scale is not comparable to the current three-pass, median-aggregated rubric, so mixing it would contaminate the judge–human agreement figures. We will document both the selection pilot and these exclusion reasons in the revision.
Q3. Full benchmark release; license under third-party rights
We will release all task definitions, references, prompts, scoring code, and aggregate data. For third-party paper figures/tables, we adopt a pointer + checksum model: releasing each material's stable identifier, source locator, and SHA-256. The benchmark's own code and annotations are released under a permissive open-source license; redistribution rights for third-party materials remain with their holders.
Thank you for the detailed response. However, my main concerns remain insufficiently addressed, so I will maintain my original score.
The new results in the rebuttal can’t be reconciled with the corresponding entries in Tables 1–3, so I don’t know which numbers to trust. The added expert validation is for the Task 2 judges. Task 3’s primary score comes from a different panel. Several of the remaining points are plans for the revision.
Authors‘s Comments
Thank you for flagging this. Tables 1–3 report the full benchmark: 15 Task 1, 384 Task 2, and 60 Task 3 questions. For the rebuttal’s cross-task comparison, we used the same 15 papers across all tasks, yielding 15, 116, and 15 questions, respectively.
We understand your main concern to be how closely the Task 3 panel’s majority verdict agrees with human judgment. The 20 questions reported in Q2 were drawn from Task 3 itself. Three experts applied the same rubric, and the three selected judges achieved precision of 0.75–0.91 and recall of 0.80–1.00 against the expert labels. This provides preliminary evidence of agreement for the individual judges on the selection set.
To measure that agreement directly, each of the three experts is now reviewing the remaining 40 Task 3 questions, with approximately 800 words to assess per question. That evaluation is still in progress.
If you believe we have addressed your concern, we would appreciate it if you could consider raising your score.
Authors‘s Comments (1/2)
Response to Reviewer BHK3
We thank the reviewer for the thoughtful comments and constructive suggestions. We provide our point-by-point responses below.
W1. Judge validation covers only Task 2; inter-judge correlation exceeds judge–expert correlation (shared bias)
Point 1 — the Task 3 judge panel was itself human-validated, and we will extend it. Our underlying question is: since experts cannot score every answer each time, is evaluation-by-model reliable? The Task 3 panel was itself human-validated through a 20-item selection pilot . We commit a ~100-judgment stratified human audit in the revision, sampling across models, pass/fail outcomes, and Strict/Core/Cited match types, reported as an agreement rate and confusion matrix against blinded expert re-judgment.
Point 2 — we own the shared-bias concern and support it with a new experiment. We agree that the inter-judge correlation (0.913–0.919) is substantially higher than judge–expert correlation, so high mutual agreement cannot by itself rule out shared bias among judges trained on similar distributions (line 462 already notes "cannot rule out shared errors or self-evaluation bias"). We state this explicitly as a limitation rather than treating judge consensus as validity, and we add our completed fully-overlapped validation:
On a fully-overlapped 122-item subset (all 3 experts × all 3 judges; 6 answer models; 42 papers; 10,000× paper-cluster bootstrap), judge–consensus agreement (0.705–0.733) falls within the inter-expert range (0.699–0.739), and judges show no systematic over- or under-scoring (bias +1.4/−5.5/+7.5).
W2. Dependence on reference quality (source-paper overclaim becomes the target)
Task 1 focuses on whether the source paper's bioinformatics analysis is correctly executed, and is thus unrelated to whether a conclusion is overclaimed.
Task 2's reference conclusions are expert-confirmed to be supported by experimental evidence plus essential context — this excludes the influence of source-paper overclaim; three senior doctoral students then blindly re-derived each item with conclusions masked to check answerability (line 140, App. B line 630). Both construction steps are designed to reduce the influence of source-paper overclaim.
Task 3 focuses on the ability to integrate a study-level conclusion from local scientific concepts; it could in principle be affected by overclaim, but the papers we curated are relatively solid studies whose scientific claims are not deliberately exaggerated — they are reasonable claims supported by experimental evidence and expert judgment.
W3. Limited support for the diagnostic claim
Our completed same-model cross-task analysis directly supports the diagnostic claim: the three tasks are not interchangeable, and the same model shows divergent profiles across layers.
Cross-task correlations are near zero: execution×interpretation ρ=−0.074 [−0.343, 0.224]; interpretation×synthesis ρ=0.039 [−0.145, 0.218]. The revision adds a case-study section analyzing one model's concrete errors across all three tasks on a single paper. For significance, we label Task 1's 15-paper subset as preliminary and add a paired sign test / bootstrap for Task 3 rankings.