- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper propose BioRELAY. BioRELAY is a biomedical research system that maintains a shared typed evidence state across retrieval, reasoning, and report generation. The paper also introduces ProteInsight to measure whether required evidence survives these stages. Experiments cover biomedical QA, protein interpretation, and disease reports, with further analyses of evidence loss and gate behavior.
Critical strengths
S1. A useful distinction between answer correctness and evidence preservation. In the interaction subset, at least 51.4% of valid answers omit a required record, while only 2 of 658 tasks contain interchangeable evidence. This provides a concrete example of a failure that final-answer accuracy can miss.
S2. The shared state records claims, supporting evidence, qualifications, and provenance in a form that can be checked across agent transitions. The linkage checks identify 28 schema-valid reasoning outputs that ignore their supplied evidence (Table 14), showing value beyond format validation.
S3. ProteInsight provides 2,796 evidence-annotated tasks over 699 disease protein cases. The experiments examine typed inputs, stage-wise retention, model substitution, and expert report assessments.
Critical weaknesses
W1. The interpretation of ECR is unclear. ECR requires all reference identifiers, including interchangeable evidence, while grounding checks citation resolution rather than claim support. These definitions weaken the claim that omission is the dominant failure mode.
W2. The gate’s benefit is not established. All 28 failed reasoning outputs were retained after unsuccessful revision (Table 15). The ablation measures detection on fixed outputs, leaving the gate’s effect on final evidence continuity unclear.
W3. BioRELAY is not evaluated on the central metric. ECR is reported only for GPT-4o (Section 4.3). A matched comparison with a typed single-agent baseline is needed to establish whether BioRELAY improves required-evidence completeness.
W4. The interface comparison is not well controlled. BioRELAY uses deterministic state projection, while baseline interfaces are generated from reports (Table 1). Using the same projector would help isolate the benefit of the proposed orchestration.
Questions for authors
1.What are the exact ECR scores under the declared alternative-evidence rules?
2.How much does the gate improve final continuity in a matched gate-on/gate-off comparison?
3.Do the findings hold with matched projection methods and expert evaluation of the reported outputs?
4 comments
Author Response (2/2)
We thank the reviewer for the thoughtful comments. For W4 and Q3, we have added a controlled comparison using the same deterministic projector() across all systems to distinguish the contribution of the structured evidence state from the effect of projection method. To match projection, we convert each report, including BioRELAY's, into a structured state containing its stated entities, relations, protein–pathway memberships, and paragraph-level citation links. All states are then rendered using the same () and evaluated with the same extractor and scorer, reporting percentages with 95% intervals. Here is the comparison table.
| System | Rep. | ↓ | ↓ | ↓ | ↑ | ↑ | ↑ | |
|---|---|---|---|---|---|---|---|---|
| GPT-5.4 (Typed) | R | 21.41 [13.87, 32.46] | 39.53 [27.97, 55.93] | 7.66 [2.71, 13.54] | 293.8 [175.0, 432.8] | 35.06 [25.48, 48.23] | 45.76 [37.20, 54.29] | 18.71 [16.03, 21.40] |
| D: model-generated (Table 1) | 36.18 [23.35, 48.59] | 5.71 [0.00, 17.14] | 0 | 9.00 [0.00, 27.00] | 13.97 [2.05, 32.50] | 28.55 [15.19, 45.82] | 1.88 [0.00, 5.65] | |
| D′: | 4.68 [1.43, 10.71] | 0 | 3.36 [1.53, 4.81] | 169.6 [110.4, 232.0] | 25.35 [14.53, 42.90] | 37.96 [33.41, 44.29] | 11.77 [10.06, 13.64] | |
| BioRELAY (GPT-5.4) | R | 6.14 [5.34, 6.80] | 27.27 [19.98, 36.33] | 0 | 382.0 [236.8, 543.6] | 42.21 [27.62, 62.15] | 100 | 26.83 [22.95, 30.72] |
| D′: | 5.61 [2.61, 11.18] | 0 | 0 | 99.80 [62.80, 156.2] | 17.99 [5.92, 39.17] | 37.96 [33.41, 44.29] | 7.33 [4.90, 10.06] | |
| D: (Table 1) | 0 | 0 | 0 | 212.8 [137.2, 306.4] | 37.32 [29.21, 49.03] | 37.96 [33.41, 44.29] | 15.18 [12.16, 18.10] |
Our controlled comparisons clarify how the projection method and the structured evidence state contribute to interface quality:
- Projection method substantially affects interface quality. Applying to the GPT-5.4 baseline's report reduces unsupported entities from 36.18% to 4.68% and increases protein and paper coverage from 13.97% to 25.35% and 1.88% to 11.77%, respectively. Thus, part of the original interface-quality gap between BioRELAY and the baseline stems from how the baseline converts its report into interface elements.
- The structured evidence state provides an advantage beyond projection alone. Under the same , BioRELAY's state-derived interface achieves 37.32% protein coverage and 15.18% paper coverage, compared with 17.99%/7.33% for its own report-derived interface and 25.35%/11.77% for the baseline's report-derived interface. It also achieves zero unsupported entities and citation mismatches. The state-derived interface retains higher paper coverage across all five diseases and higher protein coverage in four, with one tie.
Under the same report-based projection method, the baseline report yields greater interface coverage than BioRELAY’s report. This further indicates that BioRELAY’s interface advantage under matched projection comes from directly using its structured evidence state, rather than from greater coverage yielded by its narrative report.
For the results with matched projection methods reported above, the expert evaluation is pending, we have conduct an automated evaluation using the same end-to-end scoring pipeline as in our original study, with Claude Sonnet 5.5 as the judg. We will assess Coverage (Cov.), Validity (Val.), Traceability (Trac.), and Ranking (Rank.) across the matched GPT-5.4 baseline and BioRELAY (GPT-5.4), using identical evaluation criteria.
| Rep. | System | Cov. | Val. | Trac. | Rank. | Avg. |
|---|---|---|---|---|---|---|
| R | BioRELAY | 4.25 | 3.9 | 4.15 | 3.55 | 3.96 |
| GPT-5.4 (Typed) | 4 | 3.6 | 3.5 | 3.1 | 3.55 | |
| D′: | BioRELAY | 4.4 | 3.75 | 4.2 | 3.65 | 4 |
| GPT-5.4 (Typed) | 4.2 | 3.6 | 4.05 | 3.55 | 3.85 |
Thank you for the detailed response. It resolves some of my concerns, but given the conference standards and the extent of revision the paper needs, I will be keeping my score as is.
We sincerely thank the reviewer 7CP5 again for your thoughtful review and constructive feedback.
Author Response (1/n)
We thank the reviewer for the careful and constructive review, and for recognizing the paper’s soundness and the value of our framework and benchmark. We also appreciate the thoughtful comments. Below, we would like to provide more detailed analyses and experiments.
any_of/all_ofrules. The results are listed below. | Scoring definition | ECR@20 (%) | Grounded-but-incomplete among failures (%) | | :--- | :---: | :---: | | Strict ECR | 40.3 | 94.6 | | ECR under alternative-evidence rules | 44.3 | 93.1 | | ECR-sem | 40.2 | 64.7 |As shown above, ECR@20 increased from 40.3% (strict) to 44.3% (under alternative-evidence rules), and we further find that the 4% gain primarily concentrated in protein-function tasks (+15.0 points). After accounting for alternative evidence rules, the remaining incomplete cases lacked mandatory all_of records, contained no records from at least one interchangeable evidence group, or both. Thus, although strict ECR penalizes some valid alternatives, most observed incompleteness reflects missing required evidence.
Moreover, we also analyzed the evidence-group results, which further supports this conclusion: of 4,349 required evidence groups judged incomplete, 4,276 (98.3%) were never cited, 32 were cited without adequate support, and 41 were supported but incorrectly aligned. These results identify evidence omission as the dominant bottleneck, while the support and alignment errors further highlight the value of statement-level citation verification for integrating heterogeneous evidence.
These results show that BioRELAY improves required-evidence completeness under matched conditions, achieving the highest overall ECR and ECR-sem scores among the evaluated systems. BioRELAY achieves its largest gain on PPI, where tasks require many evidence records: BioRELAY raises ECR@20 from 34.5 for the typed single-agent GPT-4o baseline to 93.8.
We have reported the evaluation settings separately and clarify the metric definitions and limitations in the revision.