acceptodds
Under review as a conference paper at ICLR 2027

BioTraceBench: Evaluating Computational Execution and Evidence-Based Reasoning in Biomedical Research

Abstract

Assessing AI for biomedical research requires distinguishing the ability to execute computational analyses from the ability to interpret experimental evidence and synthesize scientific conclusions. We introduce BioTraceBench, a benchmark that evaluates these capabilities through three independently administered tasks: computational execution, local evidence interpretation, and study-level conclusion synthesis. The benchmark comprises 60 curated study entries and 1,061 local conclusion nodes spanning three biomedical research domains. Twenty-four domain researchers constructed structured nodes through schema-guided annotation and expert review, linking local conclusions to experimental designs, results, essential context, and source evidence. Task 1 requires models to execute bioinformatics analyses and produce verifiable computational artifacts. Task 2 provides 384 expert-reviewed, minimally sufficient evidence packages combining experimental figures or tables with essential textual context, from which models infer local conclusions. Task 3 requires models to integrate local conclusion nodes and study background into self-contained study-level conclusions. During construction, three senior doctoral researchers conducted a blinded review of all 60 entries to assess whether the supplied nodes support the target inference using domain knowledge. Each task separates model-visible inputs from evaluation references and uses task-specific scoring. Expert ratings additionally assess correspondence between Task 2 automated judgments and human assessments. Evaluation reveals remaining limitations in computational execution, local evidence interpretation, and study-level conclusion synthesis. The tiered Task 3 evaluation distinguishes recovery of core conclusions, preservation of appropriate inference scope, and citation of sufficient supporting nodes, showing that recovering a core conclusion does not necessarily satisfy the accompanying scope and support requirements. By relating task performance to explicit input conditions and scoring requirements, BioTraceBench provides a traceable basis for diagnosing specific capability gaps in biomedical AI and guiding model improvement.

Primary area:
datasets and benchmarks
Keywords:
scientific reasoning, biomedical research, benchmarks, large language models, AI agents, bioinformatics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BHK3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BioTraceBench, a benchmark for evaluating AI systems on biomedical research reasoning. It is built from 60 expert-curated studies that were decomposed into about a thousand local conclusion nodes. The benchmark evaluates three capabilities as separate tasks: reproducing a paper's computational analyses, inferring local conclusions from figures and essential context, and synthesizing a study's main conclusion from its local conclusions. Each task has its own inputs, references and scoring. The claimed contributions are this unified representation, expert-reviewed task construction and traceable scoring. The experiments report descriptive model comparisons on each task, with Task 1 evaluated on a 15-task subset. The LLM judges are compared with blinded expert ratings for Task 2, and an ablation shows that models almost always abstain when the required figures are removed.

Critical strengths
  1. Problem decomposition. The paper evaluates computational execution, evidence interpretation and conclusion synthesis as separate tasks, whereas existing benchmarks often combine them into a single score. This is the main contribution I see, as it makes the results easier to interpret.

  2. Expert involvement. The tasks are constructed and reviewed by domain researchers, which grounds the benchmark in real biomedical studies rather than synthetic problems.

  3. Scoped claims. The authors are explicit about what their scores measure and what they do not, and they avoid overstating the results.

Critical weaknesses
  1. Limited judge validation. Human validation covers only Task 2, so the Task 3 judge decisions behind the primary metric cannot be distinguished from judge error. The judges in both tasks are also evaluated models, and in Task 2 they agree with each other more than with the experts, which raises concerns about shared bias.

  2. Dependence on reference quality. The judges treat the reference as the only correct target, so any overclaim in a source paper becomes part of the target and a more cautious answer is penalized. The construction checks intended to prevent this are not documented in enough detail to assess.

  3. Limited support for the diagnostic claim. Each task evaluates a different set of models and the paper provides no error analysis, so the benchmark does not yet show where specific models fail across tasks. In addition, Task 1 covers only 15 of its 60 tasks and no significance testing is reported, so the rankings are descriptive only.

Questions for authors
  1. How are reference conclusions handled when a source paper's claim goes beyond its evidence? Could the authors report how many references were revised from the original claims, and clarify whether an answer that is more cautious than the reference is penalized?

  2. Could the authors explain the choice of judge models? All three Task 2 judges are also evaluated models and score below the top two, and the reason for excluding the earlier Claude judging is not given.

  3. Do the authors plan to release the full benchmark, including the figures and task packages? If so, under what license, given the third-party rights noted in the reproducibility statement?

  4. Evaluating the same models on all three tasks, and auditing a sample of Task 3 judge decisions against expert ratings, would strengthen the diagnostic claim.

4 comments
AuthorsOpenReview ↗

Authors‘s Comments (1/2)

Response to Reviewer BHK3

We thank the reviewer for the thoughtful comments and constructive suggestions. We provide our point-by-point responses below.


W1. Judge validation covers only Task 2; inter-judge correlation exceeds judge–expert correlation (shared bias)

Point 1 — the Task 3 judge panel was itself human-validated, and we will extend it. Our underlying question is: since experts cannot score every answer each time, is evaluation-by-model reliable? The Task 3 panel was itself human-validated through a 20-item selection pilot . We commit a ~100-judgment stratified human audit in the revision, sampling across models, pass/fail outcomes, and Strict/Core/Cited match types, reported as an agreement rate and confusion matrix against blinded expert re-judgment.

Point 2 — we own the shared-bias concern and support it with a new experiment. We agree that the inter-judge correlation (0.913–0.919) is substantially higher than judge–expert correlation, so high mutual agreement cannot by itself rule out shared bias among judges trained on similar distributions (line 462 already notes "cannot rule out shared errors or self-evaluation bias"). We state this explicitly as a limitation rather than treating judge consensus as validity, and we add our completed fully-overlapped validation:

ComparisonSpearman ρ [95% CI]
DeepSeek vs expert consensus0.705 [0.565, 0.813]
Kimi vs expert consensus0.733 [0.603, 0.834]
MiniMax vs expert consensus0.733 [0.608, 0.831]
Expert 1 vs Expert 20.701 [0.549, 0.816]
Expert 1 vs Expert 30.739 [0.627, 0.832]
Expert 2 vs Expert 30.699 [0.574, 0.787]

On a fully-overlapped 122-item subset (all 3 experts × all 3 judges; 6 answer models; 42 papers; 10,000× paper-cluster bootstrap), judge–consensus agreement (0.705–0.733) falls within the inter-expert range (0.699–0.739), and judges show no systematic over- or under-scoring (bias +1.4/−5.5/+7.5).


W2. Dependence on reference quality (source-paper overclaim becomes the target)

Task 1 focuses on whether the source paper's bioinformatics analysis is correctly executed, and is thus unrelated to whether a conclusion is overclaimed.

Task 2's reference conclusions are expert-confirmed to be supported by experimental evidence plus essential context — this excludes the influence of source-paper overclaim; three senior doctoral students then blindly re-derived each item with conclusions masked to check answerability (line 140, App. B line 630). Both construction steps are designed to reduce the influence of source-paper overclaim.

Task 3 focuses on the ability to integrate a study-level conclusion from local scientific concepts; it could in principle be affected by overclaim, but the papers we curated are relatively solid studies whose scientific claims are not deliberately exaggerated — they are reasonable claims supported by experimental evidence and expert judgment.


W3. Limited support for the diagnostic claim

Our completed same-model cross-task analysis directly supports the diagnostic claim: the three tasks are not interchangeable, and the same model shows divergent profiles across layers.

ModelExecution: Task1 /100Interpretation: Task2 /100Synthesis: Strict Top1 %
Kimi41.6648.0541.67
DeepSeek39.3846.9176.67
Qwen34.5650.4746.67
GPT43.2451.6260.00

Cross-task correlations are near zero: execution×interpretation ρ=−0.074 [−0.343, 0.224]; interpretation×synthesis ρ=0.039 [−0.145, 0.218]. The revision adds a case-study section analyzing one model's concrete errors across all three tasks on a single paper. For significance, we label Task 1's 15-paper subset as preliminary and add a paired sign test / bootstrap for Task 3 rankings.

AuthorsOpenReview ↗

Authors‘s Comments (2/2)


Q1. Reference handling under overclaim; how many revised; are more-cautious answers penalized?

Construction-side: Task 2 requires experts to confirm each reference is supported by evidence and passes a 3-person masked answerability check ; we will count and report how many references were revised or weakened during this process. Evaluation-side: a more cautious answer is not penalized on correctness — boundary only penalizes explicit overreach, and volunteering extra caution is not deducted; only a gold-stated limitation left uncovered affects coverage. We will illustrate this rule with an example in the main text to remove the "more cautious = more penalized" misreading.


Q2. Judge-model choice; lower-ranked models as judges; excluding earlier Claude judging

Judge selection balanced availability, self-evaluation avoidance, and judging capability; critically, the panel was not arbitrary — it was validated through a 20-item judge-selection pilot (balanced ground truth, three experts applying the same rule as the judges). Results:

Candidate judgeValid judgmentsKey behaviorOutcome
GPT-5.6 Sol4/20biosafety refusals (16/20)excluded
Opus 58/20biosafety refusals (12/20)excluded
DeepSeek v4.1 Flash20/20over-lenient: 18 passed, 9 false accepts (90% FPR)excluded as judge
Kimi K320/20precision 0.89 / recall 0.80selected
GLM 5.320/20precision 0.75 / recall 0.90selected
Qwen 3.8 Max20/20precision 0.91 / recall 1.00selected

The three selected judges are complementary (Qwen high precision, GLM high recall, Kimi balanced), so the two-of-three majority rule balances false accepts against false rejects.

On "lower-ranked models serving as judges": judging capability (not refusing, not over-lenient, faithful discrimination) is distinct from a model's task score as an answerer; the two need not coincide. Moreover, self-evaluation bias is a known but direction-uncertain threat — a model may score its own outputs higher or judge same-style outputs more harshly; because the direction cannot be predicted, we mitigate it structurally with multiple judges and a two-of-three majority rather than relying on any single judge.

On excluding the earlier Claude judging (line 327): two reasons. (i) High refusal rate — in a Task 2 run over 160 items, only 54 returned valid judgments; (ii) it used a deprecated one-pass prompt whose scale is not comparable to the current three-pass, median-aggregated rubric, so mixing it would contaminate the judge–human agreement figures. We will document both the selection pilot and these exclusion reasons in the revision.


Q3. Full benchmark release; license under third-party rights

We will release all task definitions, references, prompts, scoring code, and aggregate data. For third-party paper figures/tables, we adopt a pointer + checksum model: releasing each material's stable identifier, source locator, and SHA-256. The benchmark's own code and annotations are released under a permissive open-source license; redistribution rights for third-party materials remain with their holders.

Thank you for the detailed response. However, my main concerns remain insufficiently addressed, so I will maintain my original score.

The new results in the rebuttal can’t be reconciled with the corresponding entries in Tables 1–3, so I don’t know which numbers to trust. The added expert validation is for the Task 2 judges. Task 3’s primary score comes from a different panel. Several of the remaining points are plans for the revision.

AuthorsOpenReview ↗

Authors‘s Comments

Thank you for flagging this. Tables 1–3 report the full benchmark: 15 Task 1, 384 Task 2, and 60 Task 3 questions. For the rebuttal’s cross-task comparison, we used the same 15 papers across all tasks, yielding 15, 116, and 15 questions, respectively.

We understand your main concern to be how closely the Task 3 panel’s majority verdict agrees with human judgment. The 20 questions reported in Q2 were drawn from Task 3 itself. Three experts applied the same rubric, and the three selected judges achieved precision of 0.75–0.91 and recall of 0.80–1.00 against the expert labels. This provides preliminary evidence of agreement for the individual judges on the selection set.

To measure that agreement directly, each of the three experts is now reviewing the remaining 40 Task 3 questions, with approximately 800 words to assess per question. That evaluation is still in progress.

If you believe we have addressed your concern, we would appreciate it if you could consider raising your score.

Reviewer_fUX6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes BioTraceBench, a benchmark for evaluating LLMs in biomedical research. The authors argue that executing an analysis, interpreting experimental evidence, and drawing study-level conclusions are different abilities, and that mixing them makes benchmark scores hard to interpret. The benchmark addresses this with three separately scored tasks built on the papers from three biomedical domains. Task 1 asks models to run bioinformatics analyses. Results are verifiable by code. Task 2 asks them to infer conclusions from figures or tables, and Task 3 asks them to combine conclusions into the main conclusion of the study. The last two are not directly verifiable and require expertise. Hence, the materials are constructed by 24 domain researchers and reviewed by senior doctoral researchers. The authors argue that the current models are still limited in all three tasks (However, it is not clear to me what is “good” and why). The automated judges also agree to a certain degree with humans. The paper concludes that the benchmark provides a traceable way to diagnose which part of scientific reasoning a model fails at.

Critical strengths
  1. The paper has a good motivating gap in the literature. Rigorous generation of a benchmark is something desperately needed in the field. With expert review sessions and separating failure types, this is a good step forward.
  2. Unlike many benchmark papers, which assume the published conclusion is correct, the reference conclusions here are checked by experts against the evidence supplied to the models. This makes the ground truth more reliable.
  3. Although not major, a minor strength is that in Task 1, credit for a conclusion depends on executing the analysis, so a model cannot get full credit by copying values from the paper.
Critical weaknesses
  1. The 60 papers are selected by two principal investigators, based on criteria such as innovation and data accessibility. This is a small and curated set, which may not represent biomedical research in general, and the possible selection bias and its effect on the results are not discussed.
  2. The paper claims to help diagnose specific capability gaps, but it does not provide an analysis of where and why models fail. The implications of the results for model development are not discussed.
  3. The paper is difficult to read. The text is very dense, and the story is not very coherent.
  4. The agreement with human ratings is limited, and the analysis does not account for the variation between experts. The experts themselves agree only moderately with each other, and their scores for the same answer differ by around 20 to 28 points out of 100 on average. Hence, each expert’s score is a noisy reference, and comparing the judges with each expert separately using single agreement values does not show whether a disagreement comes from the judge or from the noise in the human scores. I think validating the judges with human ratings is the right approach, however, the expert variation should be modeled explicitly. For example, a mixed-effects model with answer and expert effects could estimate the underlying quality of each answer, together with its uncertainty, against which the judges can then be compared. Alternatively, the judge–expert agreement could be corrected for the reliability of the expert scores, or at least reported together with the inter-expert reliability, such as the intraclass correlation.
  5. The main argument of the paper is the joint evaluation of the three tasks; however, this is not used in the results. Each task is evaluated with a different set of models, and only Kimi K3 appears in all three. Therefore, it is not possible to see how the same model performs across the layers, and the benchmark is closer to three separate benchmarks reported together. Unless I am missing something, can the authors address this point by either explaining why or running more examples?
  6. The paper does not show how much the results would change if the evaluation were repeated. Only Task 2 reports confidence intervals, and these treat every item as independent, although several items come from the same paper, so the intervals are likely too narrow. Tasks 1 and 3 are based on a single run with no intervals. Since the differences between models are often small (in Task 3, sometimes one or two papers), it is not clear whether the model rankings are reliable.
  7. Since all papers are published, the models may have seen them during training. The authors reduce the cues, for example by hiding titles and abstracts, and they acknowledge the risk, but memorization is not tested. If possible, an evaluation on papers published after the models’ training cutoff would address this.
Questions for authors
  1. Since each task uses a different set of models, how can the three tasks be compared using the same model? Could you evaluate the same models on all three tasks and show whether performance in one task relates to performance in another?
  2. Could the individual tasks be compared with existing benchmarks of the same type, for example, BixBench3 for computational execution, to show whether the model rankings agree?
4 comments
AuthorsOpenReview ↗

Authors's Comments (1/2)

Response to Reviewer fUX6

We thank Reviewer for affirming the motivating gap ("rigorous generation of a benchmark is desperately needed") and the reliability of expert-checked references.

W1. The 60 papers are a small, PI-curated set; possible selection bias

We agree the 60 entries are an expert-curated set, not a random sample, and we provide a descriptive breakdown of their composition:

DimensionComposition and coverage
Scale & time60 papers, published 2022–2026; 55 (91.7%) from 2024–2026
Research topics25 oncology, 16 neuroscience, 19 other diseases and foundational atlases
Venue distribution28 journals; Nature Communications 14 (23.3%), Cancer Cell 6 (10.0%)
Primary data typessingle-cell / single-nucleus transcriptomics 23; genomics / genetic association 28; proteomics / antibody profiling 23
Other data typesspatial transcriptomics, metabolomics, epigenomics, medical imaging, immune repertoires, functional screens

The corpus spans the major scenarios of bioinformatics analysis and is therefore reasonably representative of the research settings the benchmark targets.


W2 + W5. Diagnostic claim lacks failure analysis; the three tasks use different model sets

These two points are two sides of one concern: the diagnostic decomposition must be supported by same-model, cross-layer evidence.

Acknowledge and complete. The original three tasks used different model panels — a consequence of task input requirements (multimodality / long-context / code execution) and provider availability, not design. We agree the original data could not show cross-layer behavior. During the rebuttal we completed a same-model cross-task experiment: 4 models (Kimi K3, GPT-5.6 Sol, Qwen 3.8 Max, DeepSeek v4.1 Flash) now have scores on all three tasks.

The data show two things. (1) The three tasks are not interchangeable — cross-task correlations are near zero: execution×interpretation ρ=−0.074 [−0.343, 0.224]; interpretation×synthesis ρ=0.039 [−0.145, 0.218].

(2) The same model shows divergent profiles across layers:

ModelExecution: Task1 /100Interpretation: Task2 /100Synthesis: Strict Top1 %
Kimi-k341.6648.0541.67
DeepSeek-v4.1 flash39.3846.9176.67
Qwen-3.8 Max34.5650.4746.67
GPT 5.6-sol43.2451.6260.00

For example, DeepSeek scores lowest on interpretation yet highest on synthesis; different models have different bottleneck layers, so a single aggregate score cannot characterize how models differ.

Framing and case study. "Three parallel tasks" is by design, not a defect: the three tasks use independent task packages, different inputs, and different cognitive operations, and thus measure three distinct abilities; separate scoring is precisely what makes failures attributable to a specific layer, and the near-zero correlations confirm that one task's score cannot substitute for another's. The revision adds a case-study section dissecting one model's concrete errors across all three tasks on a single paper.


W4. Expert agreement is only moderate; model the noise explicitly

Fully agreed, and we have completed it during the rebuttal period. On a fully-overlapped 122-item subset (all 3 experts × all 3 judges; 6 answer models; 42 papers), which removes the item-difficulty confounding the partially-overlapped design could not separate from rater effects: (1) individual experts are noisy references (ICC(2,1)=0.685 [0.591, 0.758]), but the three-expert consensus is highly reliable (ICC(2,3)=0.867 [0.812, 0.904]), so we evaluate judges against the consensus;

(2) judges correlate with the consensus at Spearman 0.705–0.733, statistically indistinguishable from inter-expert agreement (0.699–0.739), with overlapping 10,000× paper-cluster bootstrap CIs — the disagreement between a judge and an expert is no larger than between two experts;

(3) the variance the reviewer noted is dominated by a systematic leniency offset (Expert 2 scores 13.4 points higher than Experts 1/3, who show no bias between themselves), not random quantization noise, and our rank-based metrics are robust to such constant offsets. We will add the mixed-effects latent-quality model and per-dimension agreement in the revision.

ComparisonSpearman ρ [95% CI]
DeepSeek vs expert consensus0.705 [0.565, 0.813]
Kimi vs expert consensus0.733 [0.603, 0.834]
MiniMax vs expert consensus0.733 [0.608, 0.831]
Expert 1 vs Expert 20.701 [0.549, 0.816]
Expert 1 vs Expert 30.739 [0.627, 0.832]
Expert 2 vs Expert 30.699 [0.574, 0.787]
AuthorsOpenReview ↗

Authors's Comments (2/2)


W7. All papers are published; models may have seen them; memorization untested

We acknowledge this is an inherent boundary of the benchmark (line 369, 449 already note "possible memorization bound interpretation" and that the ablation "does not demonstrate absence of memorization"). During the extended evaluation we observed that DeepSeek v4.1 Flash scored notably higher on Task 3, which prompted us to investigate. We ran a memorization probe on two papers and found that the model could indeed recall their content, indicating that memorization is a real effect for at least some papers and models. We are committed to quantifying this transparently: in the camera-ready we will extend the memorization probe across all models and papers and report its impact on the affected scores, rather than leaving the issue unexamined.


Q2. Comparison with BixBench3

A good suggestion. Task 1 and BixBench3 both measure computational execution, but they are complementary rather than redundant: BixBench3 grades data artifacts (e.g., peak-call matrices, DE tables) against the original study, whereas Task 1 additionally scores the *computational process, evidence lineage, and semantic conclusion BixBench3 evaluates 13 models and our Task 1 evaluates 4, but the strictly same-version overlap is only two models (GPT 5.6 Sol, Kimi K3) — too few for a meaningful rank correlation. This comparison therefore provides only limited evidence of consistency across the two benchmarks.

ModelBixBench3 ScoreBixBench3 RankTask 1 ScoreTask 1 RankNotes
GPT 5.6 Sol0.481 / 1343.241 / 4Ranks first on both benchmarks
Kimi K30.472 / 1341.663 / 4Ranks among the top models on both benchmarks

Thanks for taking the time to respond to the concerns.

For W1, both the diversity of the fields and high-impact journals such as Nature make a good argument for the quality of the dataset. Although, this increases the memorization possibility, as long as it is acknowledged and memorization probe is done properly as mentioned in W7's response, this would be sufficient.

For W2 + W5, the authors' response to provide a diagnosis is a good step forward, and the paper would benefit from a more in-depth discussion and reasoning behind the results. However, the current written response looks very AI generated, raises concern as how serious the authors considers concerns.

For W4, the new results are similar to the agreement between the experts and address the concern.

Lastly, I thank the authors for providing the experiment for Q2.

Overall, the responses are sufficient and mostly address the concerns. However, I would appreciate more consideration in the written text of the rebuttal. Academia would benefit more from human discussions. Nevertheless, I might be wrong in my interpretation, and maybe all texts are written by humans. Hence, both because of how much we need rigorous benchmarks in this field and because of how they answer my concerns about soundness, I will raise my score to weak acceptance. In my decision, due to the field I give more weight to "soundness," although my average score is closer to 2.

AuthorsOpenReview ↗

Authors‘s comments

Thank you very much for your reply, and for raising your score.

We strongly agree that academia would benefit more from human discussion. We kept the writing fairly tidy mainly to reduce the burden on reviewers.

Personally, I have learned a great deal from this rebuttal process, and I care deeply about the case study in particular. I believe that only by examining closely what AI still cannot do, and by understanding why, can research keep moving forward — toward the level I hope for, which I think of as “We Code Life.”

Thank you again for your time. Reviews like this genuinely help us improve the work.

Reviewer_DVoaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces BioTraceBench, a benchmark built on expert-provided traces of biomedical and bioinformatics tasks. The authors examine three separate capabilities: computational execution, interpretation of local experimental evidence, and synthesis of study-level conclusions. The benchmark organizes expert annotations into structured analysis nodes and evaluates multiple LLMs across these tasks. Overall, the focus on trace-level evaluation and the collection of richer expert annotations are useful contributions, but I have concerns about the clarity of the presentation and whether the scoring methodology is too restrictive for scientific analyses that can be completed through different valid approaches.

Critical strengths
  1. The authors correctly identify the importance of trace-level analyses of biomedical tasks. Examining how models execute analyses and interpret evidence is an important problem for evaluating AI systems in biomedical research.
  2. The richer data extraction from experts is a valuable contribution. In particular, organizing expert annotations into structured analysis nodes provides useful information about the intermediate evidence and conclusions within a study.
  3. It’s appreciated that the authors released the benchmark along with the paper. Thank you for including this as it helps reviewers distinguish reproducibility.
Critical weaknesses
  1. The paper is very difficult to follow from a communication perspective. The scoring choices, metric definitions, and relationships between the different task evaluations need to be explained more clearly. This makes it difficult to understand exactly what the reported scores measure and to assess the contribution of the benchmark.
  2. Equation 3 adds little beyond describing averages. The formal presentation does not help clarify the evaluation, and I would prefer a clearer explanation of the scoring procedure and the motivation for its design choices.
  3. The design of the benchmark seems to assume that human-annotated reference traces are a gold-standard, but I believe this is too restrictive to allow for alternative ways of reaching the same conclusions. The benchmark doesn’t seem to allow for these flexibilities with the semantic judgement scoring being a significant component. The authors should clarify how their evaluation distinguishes an incorrect analysis from a valid alternative approach that differs from the reference trace.
  4. While the automated evaluation portion of the benchmark scales well, I worry that the expert judgement components will scale poorly and will be hard to reproduce if anyone desires to run a new model on this benchmark.
  5. Finally, I don’t believe this work is a very novel contribution. The scoring seems to be quite ad hoc and not very principled. The benchmark itself is a good contribution, but this doesn’t move beyond previous works in benchmarking agents on biomedical tasks. The human annotations of traces is valuable, but I think as a benchmark, adherence to human annotated traces has structural problems. I think the authors could help me better understand the work and contribution and I could be convinced of the novelty and its fit for the conference.
Questions for authors
  1. There are several decisions in the paper that seem hardcoded without justification. In Equation 2, what motivates the weights of 25, 55, 15, and 5 in the score S? How were these values chosen? Similarly, what motivates the 25/45/30 allocation to engineering and resource evidence, computational process evidence, and conclusion evidence described around line 161?
  2. Around line 270, Task 1 refers to the 25 September 2026 scoring release and several component version numbers. What is the relationship between this release and the submitted benchmark, and why is this particular release used for the reported evaluation?
  3. Around line 309, the paper reports that some planned judge calls were missing. What caused these calls to be missing or invalid, and what kinds of failures does this category include?
  4. Why does Task 3 use Kimi, GLM, and Qwen as judges, while Task 2 uses DeepSeek, Kimi, and MiniMax? What motivated using different judge sets across the tasks?
  5. Why are different sets of LLMs evaluated in Task 1 and Task 2? Could the authors explain how these model sets were selected?
3 comments
AuthorsOpenReview ↗

Authors's Comments (1/2)

Response to Reviewer DVoa

We thank Reviewer DVoa for recognizing two key contributions — the importance of trace-level analysis and the richer expert annotations organized into structured analysis nodes.


W3. Treating human-annotated traces as gold-standard is too restrictive for alternative valid approaches

This is the point we most need to clarify.

Task 1 provides an explicit task package and focuses on whether the agent can carry out conventional bioinformatics analyses and produce numerical results consistent with human ones.

Task 2's judge sees only two texts — gold_conclusion and answer — and focuses on whether the model, like a human, can derive the scientific conclusion supported by the experimental results and principles. The model may use any capability except web search in forming its answer, but we do not score the process, so the concern about penalizing alternative reasoning paths does not arise.

Task 3 focuses further on the ability to integrate local scientific concepts into a study-level main conclusion; the judge scores whether the model's final conclusion is sufficiently consistent with the main conclusion and whether the cited evidence is appropriate.

Therefore, an answer that reaches a semantically equivalent conclusion through a different evidence combination or wording is scored correct, not penalized.


W4. The expert-judgment components scale poorly and are hard to reproduce

We wish to emphasize an important division of labor. Construction is deliberately labor-intensive — it serves two purposes: guaranteeing reference quality, and validating that the rubric-based scoring itself is solid. On the latter, our judge-validation analysis (a fully-overlapped 122-item subset, all 3 experts × all 3 judges) shows judges agree with the expert consensus at Spearman 0.705–0.733, statistically indistinguishable from inter-expert agreement (0.699–0.739, overlapping bootstrap CIs) — i.e., the rubric can substantially replace humans for relative discrimination. Crucially, this heavy human effort is one-time, while evaluation of any new model is fully automatic.

ComparisonSpearman ρ [95% CI]
DeepSeek vs expert consensus0.705 [0.565, 0.813]
Kimi vs expert consensus0.733 [0.603, 0.834]
MiniMax vs expert consensus0.733 [0.608, 0.831]
Expert 1 vs Expert 20.701 [0.549, 0.816]
Expert 1 vs Expert 30.739 [0.627, 0.832]
Expert 2 vs Expert 30.699 [0.574, 0.787]

W5. Not novel; ad hoc

We understand the "another biomedical benchmark" impression, but our benchmark focuses on helping AI researchers understand the fundamental abilities an AI bioinformatics scientist should possess. End-to-end benchmarks are valuable, but finer-grained evaluation helps the field better understand and improve. Three aspects go beyond prior work:

(1) Joint organization — BixBench3 measures execution and Sci-MMR/SciRigor measure evidence reasoning, but all mix the three layers into single scores; BioTraceBench is the first to place all three in one study-to-node representation with independent, separately-scored tasks. Our cross-task analysis confirms the three are not interchangeable (see table below), and the same model shows inconsistent profiles across layers — empirically validating that the three must be measured separately, and showing researchers where current AI is weak and worth improving.

(2) Tiered task design targeting expert-like abilities — Task 1 asks the agent to execute complex analyses like a human expert, emphasizing engineering management, process execution, and result consistency;

Task 2 asks the agent to draw reasonable, evidence-based scientific conclusions from experiments — an ability requiring research training, neither overly conservative/defensive nor exaggerated;

Task 3 asks the agent to integrate prior scientific findings into a main conclusion like an expert. Task 3 separates "recovering the core conclusion / preserving scope / citing sufficient node support" going beyond "beyond final score" work.

(3) Extensibility to the full scientific workflow — the benchmark also supports further analyses, e.g.: (i) using each paper's local conclusions in Task 3 to test whether new scientific discoveries can be proposed (a scientist's ability to form hypotheses from prior work); (ii) given only a paper's main conclusion and its raw data, testing whether the agent can design a sound logical chain of experiments to support it (a scientist's ability to design experiments after forming a hypothesis).

Thus BioTraceBench can evaluate an AI Scientist across the full pipeline — from hypothesis generation, to experimental design, to execution, to local-conclusion analysis, to integrating conclusions into discoveries.


W1 & W2. Scoring choices, metric definitions, and the relationships between tasks are unclear

The responses above should address both concerns.


AuthorsOpenReview ↗

Authors's Comments (2/2)

Q1. What motivates the weights 25/55/15/5 in Eq. 2 and 25/45/30 around line 161?

Not hardcoded without justification; three grounds plus a new analysis.

(1) Human–machine same-weight: the Task 2 weights 25/55/15/5 are exactly the weights human experts use (Eq. 8, line 748), so judge–human agreement in §5.4 is on one interpretable scale.

(2) Correctness dominating (55) is a domain prior, not tuned: a scientifically wrong conclusion must cost more than an incomplete-but-correct one; the hard gate (any scientific-correctness component zero zeroes the whole item) enforces this prior.

(3) Task 1's 25/45/30 weights Process highest (0.45) because the task measures executable reproduction — process evidence is the precondition for trusting a conclusion.

  1. Weight-sensitivity analysis (completed): we perturbed each weight within ±10 percentage points of its baseline (renormalized to sum to 1, one-point grid) and recomputed every model's ranking. The main orderings are stable:
TaskPerturbationValid weight combosComplete order retainedKendall's WMax rank shift
Task 2±10 pp4,856100.0%1.000
Task 1±10 pp33171.9%0.921

Q2. The 25 September 2026 scoring release and version numbers

We use it because it is the only stable version that passed all release checks at the evaluation cutoff. Its relationship to the submitted benchmark: the benchmark content is unchanged; the release fixes the scorer. In the main text we will unify the version numbers and no longer expose this discrepancy.


Q3. Missing judge calls around line 309

Of 20,736 planned calls, 20,662 are valid and 74 are missing (line 309). Missing = API/format failures (timeouts, HTTP errors, JSON failing schema validation even after repair),


Q4. Why does Task 3 use Kimi/GLM/Qwen while Task 2 uses DeepSeek/Kimi/MiniMax?

The panels were chosen via a documented judge-selection pilot, not ad hoc. For Task 3 we evaluated six candidate judges (GPT-5.6 Sol, DeepSeek v4.1 Flash, Opus 5, plus the final Kimi K3, GLM 5.3, Qwen 3.8 Max) on 20 items with balanced human-verified ground truth (10 pass / 10 fail), three experts applying the same conjunction rule.

Candidate judgeValid judgmentsKey behaviorOutcome
GPT-5.6 Sol4/20biosafety refusals (16/20)excluded
Opus 58/20biosafety refusals (12/20)excluded
DeepSeek v4.1 Flash20/20over-lenient: 18 passed, 9 false accepts (90% FPR)excluded as judge
Kimi K320/20precision 0.89 / recall 0.80selected
GLM 5.320/20precision 0.75 / recall 0.90selected
Qwen 3.8 Max20/20precision 0.91 / recall 1.00selected

GPT-5.6 Sol and Opus 5 triggered biosafety refusals on 16/20 and 12/20 items — too few valid judgments to serve. DeepSeek v4.1 Flash as a judge was systematically too lenient (18 passed, of which 9 were false accepts — 90% FPR). The selected three neither refuse nor are over-lenient, and are complementary: Qwen (0.91/1.00), Kimi (0.89/0.80), GLM (0.75/0.90) — so two-of-three majority balances false accepts against false rejects. This also explains excluding Claude/Opus (refusals, not a judgment of their answer quality). Task 2's panel reflects the same availability / capability / self-evaluation-avoidance trade-offs, plus its need for image-grounded comparison — which is why GLM, lacking VLM capability, was not evaluated there. We will document this selection procedure in the revision.


Q5. Why do Task 1 and Task 2 use different model sets? How were they selected?

We agree this limited cross-task diagnosis, and we have completed a same-model cross-task analysis: 4 models (Kimi K3, GPT-5.6 Sol, Qwen 3.8 Max, DeepSeek v4.1 Flash) now have scores on all three tasks. The three tasks are not interchangeable (execution×interpretation ρ=−0.074; interpretation×synthesis ρ=0.039), and the same model shows inconsistent profiles across layers — validating the premise that the three must be measured separately. The revision adds a case-study section analyzing one model's errors across all three tasks on a single paper.

ModelExecution: Task1 /100Interpretation: Task2 /100Synthesis: Strict Top1 %
Kimi41.6648.0541.67
DeepSeek39.3846.9176.67
Qwen34.5650.4746.67
GPT43.2451.6260.00
Reviewer_DVoaOpenReview ↗

Work has adequate motivation but poor execution

Thank you for your responses and clarifications. I had misunderstood some components of the work, and your answers helped resolve some of my concerns. Your contrast to previous biomedical AI benchmarks is especially helpful for drawing contrast between your work and previous benchmarks. I think the overall motivation for your work and the construction of your benchmark is interesting and an important contribution.

However, there are still some significant concerns. I think some of the experiments and decisions are fairly ad hoc and not done rigorously, evidenced by my questions that I do not think have been addressed adequately. I would also recommend greatly improving the communication of the paper as many of these points were not clear upon my initial review. The contribution of the paper is not clearly defined, and more succinct communication could help convey your points significantly.

Based on this assessment, I will raise my score to a 2. I believe this work needs a more precise definition of the benchmarking methodology and more rigor in your experiments. Succinctly, my interpretation is that this work’s motivation is interesting but it is currently not executed at a level of rigor that would push it to an accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.