acceptodds
Under review as a conference paper at ICLR 2027

BioRelay: Multi-Agent Orchestration with Evidence Continuity for Biomedical Knowledge Discovery

Abstract

Biomedical deep-research agents retrieve evidence from literature and databases, reason over it, and synthesize conclusions into static reports. Yet whether the evidence needed to support those conclusions survives the intermediate stages from retrieval to final output remains largely unmeasured. We formalize this property as evidence continuity and introduce ProteInsight, a benchmark of 2,796 tasks over 699 disease–protein cases with typed evidence assembled from biomedical databases and literature. We find that evidence omission, rather than unsupported generation, is the dominant failure mode: GPT-4o produces grounded answers in 98.7% of cases but with omission accounting for 90.7% of failures and worsening as tasks require more heterogeneous evidence. To preserve evidence across stages, we develop BioRELAY, a multi-agent system that maintains a shared typed evidence state across search, reasoning, and synthesis, with deterministic gates enforcing evidence linkage and state consistency, supporting narrative report and interactive interface. BioRELAY performs strongly on standardized biomedical QA, challenging protein-function reasoning, and end-to-end evidence synthesis. Ablations further show that evidence continuity is distinct from schema validity and model capability, establishing evidence continuity as a measurable failure for multi-agent system and providing a mechanism to localize where evidence is lost.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
multi-agent orchestration, Evidence Continuity, harness-level engineering, disease-associated protein interpretation, biomedical NLP, deep research

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7CP5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper propose BioRELAY. BioRELAY is a biomedical research system that maintains a shared typed evidence state across retrieval, reasoning, and report generation. The paper also introduces ProteInsight to measure whether required evidence survives these stages. Experiments cover biomedical QA, protein interpretation, and disease reports, with further analyses of evidence loss and gate behavior.

Critical strengths

S1. A useful distinction between answer correctness and evidence preservation. In the interaction subset, at least 51.4% of valid answers omit a required record, while only 2 of 658 tasks contain interchangeable evidence. This provides a concrete example of a failure that final-answer accuracy can miss.

S2. The shared state records claims, supporting evidence, qualifications, and provenance in a form that can be checked across agent transitions. The linkage checks identify 28 schema-valid reasoning outputs that ignore their supplied evidence (Table 14), showing value beyond format validation.

S3. ProteInsight provides 2,796 evidence-annotated tasks over 699 disease protein cases. The experiments examine typed inputs, stage-wise retention, model substitution, and expert report assessments.

Critical weaknesses

W1. The interpretation of ECR is unclear. ECR requires all reference identifiers, including interchangeable evidence, while grounding checks citation resolution rather than claim support. These definitions weaken the claim that omission is the dominant failure mode.

W2. The gate’s benefit is not established. All 28 failed reasoning outputs were retained after unsuccessful revision (Table 15). The ablation measures detection on fixed outputs, leaving the gate’s effect on final evidence continuity unclear.

W3. BioRELAY is not evaluated on the central metric. ECR is reported only for GPT-4o (Section 4.3). A matched comparison with a typed single-agent baseline is needed to establish whether BioRELAY improves required-evidence completeness.

W4. The interface comparison is not well controlled. BioRELAY uses deterministic state projection, while baseline interfaces are generated from reports (Table 1). Using the same projector would help isolate the benefit of the proposed orchestration.

Questions for authors

1.What are the exact ECR scores under the declared alternative-evidence rules?

2.How much does the gate improve final continuity in a matched gate-on/gate-off comparison?

3.Do the findings hold with matched projection methods and expert evaluation of the reported outputs?

4 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank the reviewer for the careful and constructive review, and for recognizing the paper’s soundness and the value of our framework and benchmark. We also appreciate the thoughtful comments. Below, we would like to provide more detailed analyses and experiments.

  1. For Q1, we rescored the same outputs using the declared any_of/all_of rules. The results are listed below. | Scoring definition | ECR@20 (%) | Grounded-but-incomplete among failures (%) | | :--- | :---: | :---: | | Strict ECR | 40.3 | 94.6 | | ECR under alternative-evidence rules | 44.3 | 93.1 | | ECR-sem | 40.2 | 64.7 |

As shown above, ECR@20 increased from 40.3% (strict) to 44.3% (under alternative-evidence rules), and we further find that the 4% gain primarily concentrated in protein-function tasks (+15.0 points). After accounting for alternative evidence rules, the remaining incomplete cases lacked mandatory all_of records, contained no records from at least one interchangeable evidence group, or both. Thus, although strict ECR penalizes some valid alternatives, most observed incompleteness reflects missing required evidence.

Moreover, we also analyzed the evidence-group results, which further supports this conclusion: of 4,349 required evidence groups judged incomplete, 4,276 (98.3%) were never cited, 32 were cited without adequate support, and 41 were supported but incorrectly aligned. These results identify evidence omission as the dominant bottleneck, while the support and alignment errors further highlight the value of statement-level citation verification for integrating heterogeneous evidence.

  1. For W1, we agree that strict ECR may penalize valid alternative evidence and that citation resolution alone does not establish claim-level support. Therefore, we introduced ECR-sem, which checks whether cited records support their associated statements and whether required evidence is correctly aligned with the relevant claims by Claude Sonnet 5.5. As listed in the table below, evidence omission is the largest single failure category overall, accounting for 54.3% of ECR-sem failures. Besides, as the task requirements integrate more heterogeneous evidence, the subset failure mechanism of unsupported citations become comparably prevalent.
Task type#NumOmission-only (%) Val∧Gnd∗∧¬Cmp∗{Val} \wedge {Gnd}^* \wedge \neg {Cmp}^*Grounded-but-incomplete (%) Gnd∗∧¬Cmp∗{Gnd}^* \wedge \neg {Cmp}^*
PPI65870.3 [66.5, 74.1]83.7
Protein function55059.3 [54.5, 64.2]62.2
Function + interaction56635.4 [31.4, 39.6]48.8
All247354.3 [51.8, 56.9]64.7
  1. For W3, we agree that evaluating BioRELAY on the central evidence-continuity metric is essential. We therefore conducted a matched comparison against the typed GPT-4o single-agent baseline on ProteInsight. We report ECR@20 and introduce ECR-sem@20, which counts a required record only when it supports the claim for which it is cited. | Method (ECR@20 / ECR-sem@20) | Disease association (n=699) | PPI (n=658) | Protein function (n=550) | Function + interaction (n=566) | All (n=2,473) | |:--:|:--:|:--:|:--:|:--:|:--:| | GPT-4o | 97.0 / 98.4 | 34.5 / 19.5 | 11.1 / 24.6 | 2.8 / 7.6 | 39.7 / 40.2 | | BioRELAY (Ours) | 100.0 / 100.0 | 93.8 / 93.8 | 46.7 / 24.8 | 29.4 / 23.5 | 69.1 / 61.8 |

These results show that BioRELAY improves required-evidence completeness under matched conditions, achieving the highest overall ECR and ECR-sem scores among the evaluated systems. BioRELAY achieves its largest gain on PPI, where tasks require many evidence records: BioRELAY raises ECR@20 from 34.5 for the typed single-agent GPT-4o baseline to 93.8.

We have reported the evaluation settings separately and clarify the metric definitions and limitations in the revision.

AuthorsOpenReview ↗

Author Response (2/2)

We thank the reviewer for the thoughtful comments. For W4 and Q3, we have added a controlled comparison using the same deterministic projector(PDP_D) across all systems to distinguish the contribution of the structured evidence state from the effect of projection method. To match projection, we convert each report, including BioRELAY's, into a structured state containing its stated entities, relations, protein–pathway memberships, and paragraph-level citation links. All states are then rendered using the same (PDP_D) and evaluated with the same extractor and scorer, reporting percentages with 95% intervals. Here is the comparison table.

SystemRep.UentU_{ent} ↓UrelU_{rel} ↓UcitU_{cit} ↓ncitn_{cit}CovprotCov_{prot} ↑CovpathCov_{path} ↑CovpapCov_{pap} ↑
GPT-5.4 (Typed)R21.41 [13.87, 32.46]39.53 [27.97, 55.93]7.66 [2.71, 13.54]293.8 [175.0, 432.8]35.06 [25.48, 48.23]45.76 [37.20, 54.29]18.71 [16.03, 21.40]
D: model-generated (Table 1)36.18 [23.35, 48.59]5.71 [0.00, 17.14]09.00 [0.00, 27.00]13.97 [2.05, 32.50]28.55 [15.19, 45.82]1.88 [0.00, 5.65]
D′: PD(R)P_D(R)4.68 [1.43, 10.71]03.36 [1.53, 4.81]169.6 [110.4, 232.0]25.35 [14.53, 42.90]37.96 [33.41, 44.29]11.77 [10.06, 13.64]
BioRELAY (GPT-5.4)R6.14 [5.34, 6.80]27.27 [19.98, 36.33]0382.0 [236.8, 543.6]42.21 [27.62, 62.15]10026.83 [22.95, 30.72]
D′: PD(R)P_D(R)5.61 [2.61, 11.18]0099.80 [62.80, 156.2]17.99 [5.92, 39.17]37.96 [33.41, 44.29]7.33 [4.90, 10.06]
D: PD(S∗)P_D(S^*) (Table 1)000212.8 [137.2, 306.4]37.32 [29.21, 49.03]37.96 [33.41, 44.29]15.18 [12.16, 18.10]

Our controlled comparisons clarify how the projection method and the structured evidence state contribute to interface quality:

  1. Projection method substantially affects interface quality. Applying PDP_D to the GPT-5.4 baseline's report reduces unsupported entities from 36.18% to 4.68% and increases protein and paper coverage from 13.97% to 25.35% and 1.88% to 11.77%, respectively. Thus, part of the original interface-quality gap between BioRELAY and the baseline stems from how the baseline converts its report into interface elements.
  2. The structured evidence state provides an advantage beyond projection alone. Under the same PDP_D, BioRELAY's state-derived interface achieves 37.32% protein coverage and 15.18% paper coverage, compared with 17.99%/7.33% for its own report-derived interface and 25.35%/11.77% for the baseline's report-derived interface. It also achieves zero unsupported entities and citation mismatches. The state-derived interface retains higher paper coverage across all five diseases and higher protein coverage in four, with one tie.

Under the same report-based projection method, the baseline report yields greater interface coverage than BioRELAY’s report. This further indicates that BioRELAY’s interface advantage under matched projection comes from directly using its structured evidence state, rather than from greater coverage yielded by its narrative report.

For the results with matched projection methods reported above, the expert evaluation is pending, we have conduct an automated evaluation using the same end-to-end scoring pipeline as in our original study, with Claude Sonnet 5.5 as the judg. We will assess Coverage (Cov.), Validity (Val.), Traceability (Trac.), and Ranking (Rank.) across the matched GPT-5.4 baseline and BioRELAY (GPT-5.4), using identical evaluation criteria.

Rep.SystemCov.Val.Trac.Rank.Avg.
RBioRELAY4.253.94.153.553.96
GPT-5.4 (Typed)43.63.53.13.55
D′:PD(R)P_D(R)BioRELAY4.43.754.23.654
GPT-5.4 (Typed)4.23.64.053.553.85

Thank you for the detailed response. It resolves some of my concerns, but given the conference standards and the extent of revision the paper needs, I will be keeping my score as is.

We sincerely thank the reviewer 7CP5 again for your thoughtful review and constructive feedback.

Reviewer_FKMEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BioRELAY, a multi-agent system for biomedical evidence synthesis, and ProteInsight, a benchmark of 2,796 tasks from 699 disease–protein cases. The main idea is to track evidence across retrieval, reasoning, and report generation using a shared evidence state and deterministic checks. The authors evaluate evidence completeness, biomedical QA, protein reasoning, and end-to-end synthesis.

Critical strengths
  1. The paper raises a useful problem: valid citations do not necessarily mean that all relevant evidence has been preserved. Table 8 shows a large gap between grounding (98.7%) and completeness (44.0%) for GPT-4o. I think this is an interesting finding that conventional QA metrics can easily miss.

  2. The shared evidence state and validation gates are practical design choices. Table 14 shows that evidence-linkage checks detect 28 failures missed by schema validation. The implementation and expert evaluation are also fairly detailed, making the system easier to understand and inspect.

Critical weaknesses
  1. My main concern is that the paper does not directly show whether BioRELAY improves the proposed evidence-continuity metric. Table 8 evaluates GPT-4o, not BioRELAY. Also, Table 15 shows that none of the 28 failed Reasoning artifacts were repaired, despite being detected by the gates. The end-to-end results are mixed: Table 1 shows better publication coverage (5.48% to 21.95%), but lower protein coverage (51.17% to 31.76%) and more unresolved relations (0.36% to 19.27%). A matched comparison is needed to establish the actual benefit of the proposed mechanism.

  2. I also question whether ECR accurately measures missing evidence. It requires individual references to be retained even when several sources support the same conclusion. Table 9 shows that 74.9% of protein-function tasks contain alternative evidence, with a possible identification range of 11.1%–86.0%. This could substantially change the reported omission rate. Furthermore, reference validity alone does not establish that a cited source supports the claim.

Questions for authors
  1. Could you evaluate BioRELAY directly on ProteInsight using ECR, with matched models and evidence budgets? An ablation separating the shared evidence state and validation gates would help.

  2. How much does ECR change when the alternative-evidence rules in Table 9 are applied? Does evidence omission remain the dominant failure mode?

  3. In Table 15, all 28 failed Reasoning artifacts were retained. Did these failures affect the final reports, and what practical benefit did the gates provide beyond detecting them?

  4. How do you explain the lower protein coverage and higher unresolved-relation rate in Table 1? Would the publication-coverage improvement remain under matched output budgets?

6 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank Reviewer FKME for the thoughtful questions and constructive feedback, and for recognizing the value of the problem we raised and the practical role of our shared evidence state and validation gates. Below, we would like to provide more detailed analyses and experiments.

W1.1 My main concern is that the paper does not directly show whether BioRELAY improves the proposed evidence-continuity metric. Table 8 evaluates GPT-4o, not BioRELAY.
Q1. Could you evaluate BioRELAY directly on ProteInsight using ECR, with matched models and evidence budgets? An ablation separating the shared evidence state and validation gates would help.

We thank the reviewer for highlighting the distinction between citation validity and evidence completeness, and for recognizing the practical value of the shared evidence state and validation gates. To directly evaluate evidence continuity and quantify each component’s contribution, we have added a controlled ECR evaluation and ablation study to the revised paper.

All configurations use the same backbone (GPT-4o), the same evidence packet per task (b=20b=20), and the same judge model (Claude Sonnet 5.5). We compare BioRELAY against a single GPT-4o call and two ablations that separately remove the shared evidence state and evidence continuity gate check. We report ECR@20 and introduce ECR-sem@20 (see detailed description in Author Response (2/n) ), which counts a required record only when it supports the claim for which it is cited.

Method (ECR@20 / ECR-sem@20)Disease association (n=699)PPI (n=658)Protein function (n=550)Function + interaction (n=566)All (n=2,473)
GPT-4o97.0 / 98.434.5 / 19.511.1 / 24.62.8 / 7.639.7 / 40.2
BioRELAY (Ours)100.0 / 100.093.8 / 93.846.7 / 24.829.4 / 23.569.1 / 61.8
w/o shared state80.0 / 80.087.5 / 87.560.0 / 20.029.4 / 17.664.7 / 52.9
w/o gate revision100.0 / 100.068.8 / 62.546.7 / 20.029.4 / 11.863.2 / 51.5
  1. BioRELAY improves evidence continuity under matched conditions. BioRELAY achieves the highest overall scores on ECR and ECR-sem metrics. BioRELAY achieves its largest gain on PPI, where tasks require many evidence records, raising ECR@20 from 34.5 for a single GPT-4o call to 93.8.
  2. Both shared state and gate check contribute to this improvement. Removing the shared state reduces overall ECR-sem@20 from 61.8 to 52.9, while removing gate revision reduces it to 51.5. Without gate check, misplaced or unsupported citations are not returned for correction, leading to PPI ECR-sem@20 consequently dropping from 93.8 to 62.5. The shared state provides explicit requirements for preserving evidence through synthesis. It is logged that without shared state, rewrites replaced a cohort record with a URL in 174 of 699 disease-association tasks, and these outputs passed the gate check before the Synthesis Agent. What's more, each component alone nevertheless improves on the single-call baseline.
  3. The protein-function results highlight the importance of claim-level support. Here, the w/o state ablation achieves higher ECR@20 than BioRELAY (60.0 vs. 46.7), while BioRELAY performs best on ECR-sem@20. In the protein-function type, 74.9% of tasks contain multiple branches, each task requires an average of 5.23 records, and UniProt, GO, and Reactome frequently reference the same PMID. Without the shared state, outputs can repeatedly cite overlapping records without better supporting the associated claims. ECR does not check this claim–evidence relationship. But on the newly-introduced metric, ECR-sem, counts a record only when it supports its claim, BioRELAY performs best.

The new results table and component ablations are included in the revised manuscript.

AuthorsOpenReview ↗

Author Response (2/n)

W2. I also question whether ECR accurately measures missing evidence. It requires individual references to be retained even when several sources support the same conclusion. Table 9 shows that 74.9% of protein-function tasks contain alternative evidence, with a possible identification range of 11.1%–86.0%. This could substantially change the reported omission rate. Furthermore, reference validity alone does not establish that a cited source supports the claim.
Q2. How much does ECR change when the alternative-evidence rules in Table 9 are applied? Does evidence omission remain the dominant failure mode?

We thank the reviewer for these valuable questions. We agree that completeness should account for interchangeable evidence and that a valid reference alone does not establish support for the associated claim.

At the same time, ECR remains useful for diagnosing missing evidence for two reasons. 1. Strict ECR provides a conservative bound on evidence coverage. It reports the lower bound of the coverage range permitted by the gold annotations, corresponding to the largest acceptable omission rate. The alternative-evidence rules in Table 9 are designed to establish this range, making explicit how coverage changes under different assumptions about evidence substitutability. For protein function, the resulting range is 11.1%–86.0%. Strict ECR therefore provides a diagnostic bound, while Table 9 characterizes the uncertainty introduced by alternative evidence. 2. Evidence omission remains the dominant failure mode when we applied the alternative-evidence rules to ECR evaluation on ProteInsight. Under the Table 9 rules, only 114 of the 793 exposed ProteInsight tasks can be completed with a single record; the remaining 679 require multiple distinct evidence branches. Interchangeable sources can satisfy the same requirement, but cannot substitute for evidence needed to support other branches. Measuring whether these requirements survive synthesis thus remains meaningful.

Strict ECR credits a required record when it appears anywhere in the answer. To address the limitations, we introduce ECR-sem. ECR-sem incorporates the Table 9 alternative-evidence rules and additional requires each credited record to support the claim to which it is attached. We updated two aspects as follows:

  • Gnd → Gnd*. Gnd checks whether a cited identifier resolves to a supplied record and is nearly saturated at 98.7%. Gnd* additionally checks whether that record supports the statement to which it is attached. It detects misplaced citations, swapped records, blanket citation of the entire evidence packet, and citations attached to statements declaring the record irrelevant.
  • Cmp → Cmp*. Cmp* treats interchangeable records as a single evidence requirement and credits that requirement only when a supporting record is attached to the relevant claim.

We validated Gnd* and Cmp* against known failure cases. To assess the reliability of ECR-sem evaluation, two domain experts independently annotated Gnd* and Cmp* decisions on 90 tasks sampled across all four task categories, including 30 tasks with alternative evidence. Inter-expert agreement was 92.22%. Therefore, ECR-sem evaluates both substitutability and claim-level support. Then, we reevaluate GPT-4o on all 2,473 tasks at b=20b=20. All three ECR-sem components are evaluated by Claude Sonnet 5.5. The table below compares strict ECR and ECR-sem for GPT-4o at b=20b=20 across 2,473 scorable tasks, reporting percentages with 95% intervals.

Task typeExposedAvg. BoundValGndCmpECR@20Gnd*Cmp*ECR-sem@20
Disease association0[97.0, 100.0]97.199.999.997 [95.7, 98.1]98.410098.4 [97.4, 99.3]
Protein–protein interaction2[34.5, 34.8]71.699.740.634.5 [30.9, 38.3]89.523.919.5 [16.4, 22.5]
Protein function412[11.1, 86.0]86.497.614.511.1 [8.5, 13.6]75.039.424.6 [21.2, 28.5]
Function + interaction379[2.8, 55.7]55.797.37.82.8 [1.6, 4.2]57.415.57.6 [5.5, 9.9]
All793[39.7, 68.5]78.498.744.039.7 [38.4, 41.1]81.547.040.2 [38.8, 41.8]

As shown in the table, evidence omission remains the dominant failure mode under ECR-sem. Cmp* remains the lowest-scoring component overall, even after allowing interchangeable evidence and requiring claim-level support. Meanwhile, Gnd* decreases from 98.4 to 89.5, 75.0, and 57.4 across increasingly heterogeneous evidence settings, revealing citation-support errors that identifier validity misses.

We have added ECR-sem, the expert assessment, and the evaluation results to the revised manuscript.

AuthorsOpenReview ↗

Author Response (3/n)

W1.2. Also, Table 15 shows that none of the 28 failed Reasoning artifacts were repaired, despite being detected by the gates.
Q3. In Table 15, all 28 failed Reasoning artifacts were retained. Did these failures affect the final reports, and what practical benefit did the gates provide beyond detecting them?

All 28 failed Reasoning artifacts are Protein_Function_Analysis artifacts, and the error code is L.packet_unused. In each case, the supplied evidence packet contained no publication attributable to the target protein, so revision within that packet could not supply the missing support. The system therefore recorded these evidence gaps in SS and routed them back to the Reasoning Agent for re-reasoning. All 28 failed artifacts were withheld from the reporting interface DD, and no citations from other proteins were substituted to fill the gaps. This handling preserved evidence attribution and kept the missing support explicit.

The gates provide three practical benefits beyond detection:

  1. Control which artifacts enter the report while preserving traceability. Our log audit confirms that all 28 failed artifacts were excluded from the reporting interface DD. Their corresponding claims were retained in SS as low-certainty entries without evidence links, keeping the evidence gaps explicit and traceable. We have clarified this handling in Section 4.6 and Appendix G.
  2. Improve evidence continuity through revision. Each finding is typed and logged, allowing the system to route revisions and control which artifacts proceed to reporting. The ablation (see the table below, also reported in our response (1/n)) demonstrates the benefit of gate-triggered revision: it improves overall ECR-sem@20 by 10.3 points (61.8 versus 51.5) and ECR@20 by 5.9 points (69.1 versus 63.2).
  3. Guide selection among partially validated outputs. For tasks that never fully passed all checks, the system retained the attempt with the fewest findings, reducing the aggregate finding count from 528 to 265. On this same task subset, BioRELAY (GPT-4o) also achieved higher ECR@20 than the GPT-4o baseline.
Method (ECR@20 / ECR-sem@20)Disease association (n=699)PPI (n=658)Protein function (n=550)Function + interaction (n=566)All (n=2,473)
GPT-4o97.0 / 98.434.5 / 19.511.1 / 24.62.8 / 7.639.7 / 40.2
BioRELAY (Ours)100.0 / 100.093.8 / 93.846.7 / 24.829.4 / 23.569.1 / 61.8
w/o gate revision100.0 / 100.068.8 / 62.546.7 / 20.029.4 / 11.863.2 / 51.5
Reviewer_uus2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether evidence used by biomedical reasoning agents survives through the reasoning process: that is, whether the outputs provided are grounded in evidence that was seen in the literature or databases. They claim that omission of evidence is one of the larger issues, rather than hallucination. To address this issue, they developed BioRELAY, a multi-agent system that maintains evidence throughout reasoning stages.

Critical strengths

I think holding on to evidence during the reasoning process is an important attribute for reasoning systems to have, so I applaud the authors for working on this space.

Critical weaknesses

-Using GPT 5.5 as a sole judge in the ProteInsight task is an issue, especially given information on model self-preference (and family preference) and the fact that GPT 5.4 is used for BioRELAY. -Concern around train/test leakage. While there is definitely a novel dataset here, there are a lot of public benchmarks used along with models that may have "seen" this data. -There are no standard deviations on the human review in the supplement (Table 5). It seems that multiple experts were reviewing here, so there should be some can of measure of spread on the scores. Hard to tell if humans really felt one model was better than another. -Table 5 also appears to show that human review ranked BioRELAY worse than GPT-5.5 review, which seems to support the concerns I had above about self-preference in the LLM-judge.

Questions for authors

-Can you make Figure 1 more readable? -How do you define the six domain experts? What is their experience? -The crux of my concerns is that I'm not sure how to interpret the claims of improved performance given how the evaluations were set up (see above).

3 comments
AuthorsOpenReview ↗

Author Response (1/n)

We thank the reviewer uus2 for recognizing the importance of preserving evidence throughout the reasoning process. We appreciate all the thoughtful and valuable concerns. We have conducted additional controlled analyses and will clarify the scope of our claims accordingly.

W1. Using GPT 5.5 as a sole judge in the ProteInsight task is an issue, especially given information on model self-preference (and family preference) and the fact that GPT 5.4 is used for BioRELAY.

To address the concern of reliance on GPT-5.5 as the sole judge, we provide two complementary analyses. First, we introduce ECR-sem, which assesses whether cited evidence supports the associated claims, using Claude Sonnet 5.5 as the judge. This provides an additional assessment from a different model family. Second, we conduct a matched-backbone comparison on 2,473 ProteInsight tasks, using the same backbone GPT-4o for BioRELAY and free-form multi-agent baselines, with identical evidence packets and evaluation criteria. This comparison holds the backbone constant across the two methods.

Method (ECR@20 / ECR-sem@20)Disease association (n=699)PPI (n=658)Protein function (n=550)Function + interaction (n=566)All (n=2,473)
GPT-4o97.0 / 98.434.5 / 19.511.1 / 24.62.8 / 7.639.7 / 40.2
BioRELAY (Ours)100.0 / 100.093.8 / 93.846.7 / 24.829.4 / 23.569.1 / 61.8

As shown above, BioRELAY improves ECR@20 from 39.7% to 69.1% (+29.4 points) and ECR-sem@20 from 40.2% to 61.8% (+21.6 points). These results support improved evidence continuity without relying on GPT-5.5 as the sole evaluator or comparing different backbones.

W2. Concern around train/test leakage. While there is definitely a novel dataset here, there are a lot of public benchmarks used along with models that may have "seen" this data.

We agree that potential pretraining exposure should be considered when interpreting results on public benchmarks such as BioASQ. On BioASQ, we compare BioRELAY with multiple open-source models and agentic deep-research systems to contextualize its standard biomedical QA performance.

And we would like to clarify that our central evidence continuity claims are additionally evaluated on the ProteInsight dataset, newly-constructed by our own and have not been publicly released now (We're going to open source the ProteInsight dataset and code in the camera-ready). Its protein–protein interaction, protein function, and function + interaction tasks remain challenging for single-model baselines and other agentic biomedical systems. Furthermore, our matched comparisons use the same backbone and identical evidence packets across systems, reducing differences in pretraining exposure and directly assessing their ability to preserve and integrate task-specific evidence.

Q1. Can you make Figure 1 more readable?

We have re-organized Figure 1 in the revision to clarify the research problem and the BioRELAY workflow. The revised figure emphasizes evidence continuity across the search–reasoning loop, the roles of the shared evidence states SS and gate check, and the distinction between report synthesis and interface generation.

AuthorsOpenReview ↗

Author Response on (2/n)

W3. There are no standard deviations on the human review ... (Table 5) ... multiple experts were reviewing ... be some can of measure of spread on the scores. Hard to tell if humans really felt one model was better than another.

We thank the reviewer for thoughtful comments on reporting variability in the human evaluation. In the Appendix B.4, we reported ICC(1,6) = 0.76 for the six-expert mean ratings. To better characterize variability in expert ratings and quantify differences between models, we calculated each model’s per-disease mean score and standard deviation across the six experts. As presented in the tables below, experts assigned BioRELAY the highest mean scores for prioritization and research depth, and readability, with mean differences as +0.28 (95% CI: [0.12, 0.45]) and +0.17 ([0.03, 0.33]), respectively, with both intervals above zero.

SystemCov.Val.Trac.Rank.Read.
BioRELAY3.82 ± 0.934.40 ± 0.764.25 ± 0.904.18 ± 0.744.57 ± 0.63
GPT-5.5 + Web Search4.37 ± 0.854.77 ± 0.454.35 ± 0.873.90 ± 0.814.40 ± 0.72
Gemini Deep Research4.20 ± 1.114.10 ± 1.143.60 ± 0.993.90 ± 0.964.30 ± 0.84
DR-Tulu-8B2.73 ± 1.123.20 ± 1.212.93 ± 1.272.87 ± 1.142.73 ± 1.06

W4. Table 5 ... ranked BioRELAY worse than GPT-5.5 review, ... support ... self-preference in the LLM-judge.

The original human evaluation compares GPT-5.4-based BioRELAY with GPT-5.5 (different backbones). To isolate the workflow contribution, we added a matched GPT-5.4 comparison evaluated by Claude Sonnet 5.5, with four evaluations per system–disease pair. Each cell reports in the format of BioRELAY / GPT-5.4 matched .

DiseaseCov.Val.Trac.Rank.
AD4.25 / 4.004.00 /3.504.25 / 3.503.75 / 3.00
CKD4.00 / 4.003.50 / 3.504.00 / 3.503.50 / 3.00
MDD4.50 / 4.004.00 / 3.504.00 / 3.503.50 / 3.00
RA4.50 / 4.004.00 / 3.504.50 / 3.503.50 / 3.00
AF4.00 / 4.004.00 / 4.004.00 / 3.503.50 / 3.50
Avg.4.25 / 4.003.90 / 3.604.15 / 3.503.55 / 3.10

Across five diseases, BioRELAY achieves higher mean scores on all four dimensions, averaging 3.96 vs. 3.55. Moreover, we have evaluated ECR and ECR-sem on ProteInsight using Claude Sonnet 5.5, a non-GPT judge (Tab. 2 in the revision). With the backbone held fixed, BioRELAY achieves higher scores. These complementary evaluations provide cross-family support for BioRELAY’s benefits in evidence-centered biomedical synthesis.

Q2. How do you define the six domain experts? What is their experience?

All outputs were anonymized and presented in randomized order. The six domain experts were independent researchers who:

  • Held a PhD in biological sciences or a closely related biomedical or medical field.
  • Had published research on protein function, disease mechanisms, or pathway biology.
  • Were blinded to the systems producing each report and interface.

Q3. The crux of my concerns is that I'm not sure how to interpret the claims of improved performance given how the evaluations were set up.

We thank the reviewer for recognizing the importance of evidence preservation. We summarize the evidence supporting our performance claims below:

  1. Better evidence continuity under matched models and evidence. On the same GPT-4o backbone and with Claude Sonnet 5.5 as the judge, BioRELAY increases ECR@20 from 39.7% to 69.1% and ECR-sem@20 from 40.2% to 61.8. Removing gate actions or the shared state reduces ECR-sem from 61.7% to 51.5% and 52.9%, respectively, supporting the contribution of the shared state and gate check.
  2. Across five diseases, compare to the matched baseline use GPT-5.4, BioRELAY increases protein coverage from 35.06% to 42.21% and publication coverage from 18.71% to 26.83%, while reducing unresolved entities from 20.70% to 6.14% and unresolved citations from 7.66% to zero. These comparisons support improved evidence retention under a matched output-length comparison.
  3. Interface quality under matched projection. With the same deterministic projector PDP_D and display budget, BioRELAY’s state-derived interface achieves 37.32% protein coverage and 15.18% publication coverage, compared with 17.99%/7.33% from its own report and 25.35%/11.77% from the baseline report. It also has zero unresolved entities and citation mismatches. This comparison identifies the structured evidence state as a source of interface advantage beyond the projection method alone.
  4. Expert evaluation with reported variability and agreement. Six biomedical experts independently evaluated anonymized outputs presented in randomized order. We report human scores as mean ± SD, together with ICC(1,6) = 0.76 for the six-expert mean ratings.

All new experiments and implementation details have been incorporated into the final revision.

We sincerely thank you again for your thoughtful review and constructive feedback. We hope our rebuttal and the additional experiments help clarify the revisions made in response to your comments.

As the discussion period draws to a close, we would be grateful if you could take a moment to review our rebuttal. If any point remains unclear or you have further questions or suggestions, we would be very happy to clarify and discuss them.

If you feel that our responses have adequately addressed your concerns, we would sincerely appreciate your reconsideration of the overall assessment. Thank you again for your time and valuable feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.