- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BIOEVIDENCE supplies frozen LLMs with compact, precomputed structural and evolutionary annotations for protein variants. The paper studies ranking on 651 fixed ProteinGym episodes, varying evidence coverage, modality, candidate correspondence, and reasoning effort. For GPT-5.6 Sol, increasing evidence coverage raises matched nested-macro Spearman correlation from about 0.322 to 0.433; shuffled evidence instead reduces performance. Paired analyses use protein-level resampling, and additional experiments compare specialists, evolutionary rank fusion, six LLMs, and 25 post-cutoff assays. The study distinguishes controlled evidence effects from incomplete-coverage and cross-channel comparisons.
Critical strengths
- The central evidence effect has meaningful controls. R/S/E/SE conditions share task inputs, the coverage subsets are nested, and SH preserves candidate identifiers, true mutation-position headers, and evidence length/content while disrupting biological payload correspondence. On common episodes, SE improves over R by 0.113 with a paired confidence interval excluding zero, while both evidence modalities contribute. This supports evidence access as a useful experimental variable.
- The analysis treats several statistical and operational confounders seriously. It uses protein-level paired resampling, separately reports response failures and matched coverage, checks served-model identity, and compares against all-site evolutionary fusion. The finding that weaker models can underperform simple fusion is especially informative and avoids a uniformly favorable account of LLM evidence use.
Critical weaknesses
- The R-to-SE gain does not isolate biological annotations from mutation-localization assistance. Evidence blocks explicitly identify substituted positions and residues, whereas R presents complete wild-type and candidate sequences without the same mutation list. The SH control retains true mutation-position headers and disrupts biological payload correspondence, so SE-SH does support the value of correctly assigned biological information. However, without a location-only condition, the R-to-SE gain still combines biological annotations with mutation-localization assistance. This limits attribution of the total gain rather than eliminating the evidence for biological information use.
- The specialist-gap and generalization results remain conditional on incomplete, non-random coverage and unequal comparison conditions. GPT-6 Astra scores only 267/651 episodes and has no R counterpart; other failures depend on the condition. The temporal study selects 25/59 assays and crosses channels, while its specialist comparison is sensitive to rebuilt alignment quality. The paper discloses these issues, but stronger population-wide conclusions or equivalence to specialists are not established by the current panels.
Questions for authors
- How much of the R-to-SE gain remains when R receives the exact mutation positions and residue substitutions, with neutral or missing fields replacing biological annotations? This would distinguish biological information from mutation-location assistance.
- What do coverage-aware sensitivity analyses show for the specialist comparison, especially the unfinished GPT-6 panel? Please distinguish uncollected responses from refusals and show how the conclusion changes under plausible performance on missing episodes.
- On the post-cutoff assays, is a same-model, same-channel R/SE contrast available? That comparison would materially change the assessment of evidence-driven temporal transfer. Flag For Ethics Review
3 comments
Additional Temporal Validation: Matched R/SE Comparisons
We thank the reviewer for highlighting the need for temporally eligible, fully matched R/SE comparisons.
The original 25-assay panel comes from the PG-LLM temporal benchmark, with first-public score dates spanning February–July 2026. We additionally assembled a new 69-assay panel covering 19 proteins, with experimental scores first released in October–November 2026, to test generalization to more recent outcomes.
We apply two criteria to all comparisons:
(a) Same model, same channel, same settings. Both R and SE arms were run by us with identical model identifiers, reasoning effort, thinking settings, API channels, candidate sampling, prompt versions, and evidence pipelines. The evidence blocks are the only difference. For open-weight models, OpenRouter providers were pinned with fallback disabled (Qwen: Venice FP8; Kimi: AtlasCloud) to avoid differences between hosting configurations.
(b) Model-specific temporal eligibility. Each comparison includes only assays whose exact scores became public after the model's temporal reference. The original panel's dates (2026-02-09 to 2026-07-10) are recorded in study-group-mapping.json from the PG-LLM evaluator bundle. Open-weight models use their publicly verifiable weight-release dates, which bound their training dates; closed models use reported knowledge cutoffs. For Opus 5, we conservatively use 2026-05-31 as the end of its reported cutoff month.
Four models satisfy both criteria, yielding five paired comparisons:
| Model / panel | Channel | Eligible assays | Common episodes | R → SE (ρ) | Δρ, 95% CI |
|---|---|---|---|---|---|
| Qwen3-235B / original | OpenRouter, Venice FP8 | 25/25 | 64 | 0.0655 → 0.2290 | +0.1635 [+0.0927, +0.2479] |
| Opus 4.8 / original | Anthropic Batches | 25/25 | 59 | 0.1027 → 0.2383 | +0.1356 [+0.0613, +0.2116] |
| Kimi K2.5 / original | OpenRouter, AtlasCloud | 25/25 | 75 | 0.1835 → 0.2892 | +0.1057 [+0.0500, +0.1631] |
| Opus 5 / original | Anthropic Batches | 15/25 | 39 | 0.3126 → 0.3690 | +0.0565 [−0.0120, +0.1299] |
| Opus 5 / new | Anthropic Batches | 69/69 | 133 | 0.2452 → 0.2854 | +0.0401 [+0.0216, +0.0592] |
Four of five comparisons show statistically detectable improvements. The remaining Opus 5 comparison is positive but inconclusive on the original panel (39 episodes across 7 study clusters). On the new panel, the same model shows a significant improvement across 133 common episodes from 19 clusters. The two panels are evaluated separately, without pooling.
These results include two pre-measurement open-weight models and provide evidence that SE improvements extend beyond potentially memorized benchmark outcomes.
Limitations. Absolute correlations are not comparable across panels due to different candidate sampling and difficulty. All results are conditional on jointly completed episodes: Qwen experienced output truncation, while Opus 5 completed 137/207 R and 197/207 SE episodes on the new panel. The paired R score (0.2452) is close to its score on all completed R episodes (0.2466), although selection bias cannot be excluded. Results use temporal study-group aggregation with 10,000 paired bootstrap resamples.
Response
Thanks for the response. I would like to maintain my original rating.
Response to Reviewer: Localization and GPT-6 Coverage
We thank the reviewer for recognizing the experimental controls and statistical analysis, and for the specific suggestions regarding mutation localization and incomplete GPT-6 coverage.
1. Separating mutation localization from biological information
We added a location-only condition (L) that provides mutation positions on top of R, without structural or evolutionary fields. These position rows are identical to those in the S and E evidence blocks. L deliberately omits substituted residue identities, as do S and E, ensuring that localization is the only additional information. Complete wild-type and candidate sequences are provided in all conditions.
On the pre-registered 50% stratified assay subset (108/217 assays), we evaluated R, L, and SE using Opus 5 medium under identical settings, with 284 jointly completed episodes from 88 proteins:
The localization-only gain is not statistically detectable, whereas adding biological evidence after controlling for localization yields a substantial improvement. Approximately 85% of the total point-estimate gain cannot be explained by mutation-localization assistance.
2. GPT-6 coverage and specialist-comparison sensitivity
Of 651 targeted GPT-6 Astra episodes, 300 were collected: 267 scorable, 10 blocked, and 23 unparseable. The remaining 351 were uncollected, not refused. Provider access was suspended at the account level following a biological-content flag (
401 token_revoked), pending institutional verification. This was an account-level interruption rather than 351 individual episode refusals, although the missing outcomes remain unknown.We compared GPT-6 with two specialists on matched observed coverage: VenusREM (structure + MSA, first on the full leaderboard) and GEMME (alignment only).
To assess sensitivity to the 351 uncollected episodes, we conducted a pattern-mixture analysis. The 33 collected failures remain excluded. Each hypothetical penalty δ represents a per-episode disadvantage for GPT-6 relative to the specialist on uncollected episodes (δ = 0 assumes a tie):
These are assumption-based sensitivity scenarios, not independent evidence about missing outcomes. Imputed values carry no uncertainty, so intervals reflect only observed between-protein variability and are correspondingly narrow. A penalty of approximately 0.02 makes GPT-6's disadvantage relative to VenusREM detectable; at approximately 0.03, its advantage over GEMME is no longer statistically detectable.
We therefore do not claim equivalence to specialist models. On observed shared coverage, GPT-6 outperforms alignment-only GEMME and is not detectably different from VenusREM, but incomplete coverage limits generalization. We will narrow the manuscript's specialist-comparison claims accordingly. Statistical aggregation follows the paper's nested macro evaluation with 10,000 paired protein-level bootstrap resamples.