acceptodds
Under review as a conference paper at ICLR 2027

Evidence Scaling for Zero-Shot Protein Reasoning with Large Language Models

Abstract

Large language models (LLMs) show emerging zero-shot capability for protein variant prediction, yet still lag behind specialized protein models. We ask whether this gap can be reduced by scaling access to biological evidence rather than adapting model parameters. We introduce BioEvidence, a training-free and model-agnostic interface that converts structural and evolutionary information from standard biological tools into compact evidence for frozen LLMs. On the ProteinGym benchmark, we observe evidence scaling: performance improves as evidence becomes richer. Structural and evolutionary evidence each improve performance, and combining them yields further gains, while mismatching the same evidence to the wrong variants degrades performance below the no-evidence baseline. Notably, BioEvidence enables zero-shot ranking to reach strong specialized protein predictors on matched evaluations, and the improvement persists on post-cutoff data released after the model's knowledge cutoff. Evidence also interacts with conventional scaling: for GPT-5.6 Sol, evidence at low reasoning effort outperforms the no-evidence condition at medium effort, while a six-model analysis associates stronger no-evidence performance with larger margins over evolutionary rank fusion. These results identify external evidence as a complementary scaling axis for scientific prediction alongside model capability and inference effort.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
large language models, protein variant effect prediction, zero-shot learning, evidence grounding, protein language models, ProteinGym, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_h6AYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

BIOEVIDENCE supplies frozen LLMs with compact, precomputed structural and evolutionary annotations for protein variants. The paper studies ranking on 651 fixed ProteinGym episodes, varying evidence coverage, modality, candidate correspondence, and reasoning effort. For GPT-5.6 Sol, increasing evidence coverage raises matched nested-macro Spearman correlation from about 0.322 to 0.433; shuffled evidence instead reduces performance. Paired analyses use protein-level resampling, and additional experiments compare specialists, evolutionary rank fusion, six LLMs, and 25 post-cutoff assays. The study distinguishes controlled evidence effects from incomplete-coverage and cross-channel comparisons.

Critical strengths
  1. The central evidence effect has meaningful controls. R/S/E/SE conditions share task inputs, the coverage subsets are nested, and SH preserves candidate identifiers, true mutation-position headers, and evidence length/content while disrupting biological payload correspondence. On common episodes, SE improves over R by 0.113 with a paired confidence interval excluding zero, while both evidence modalities contribute. This supports evidence access as a useful experimental variable.
  2. The analysis treats several statistical and operational confounders seriously. It uses protein-level paired resampling, separately reports response failures and matched coverage, checks served-model identity, and compares against all-site evolutionary fusion. The finding that weaker models can underperform simple fusion is especially informative and avoids a uniformly favorable account of LLM evidence use.
Critical weaknesses
  1. The R-to-SE gain does not isolate biological annotations from mutation-localization assistance. Evidence blocks explicitly identify substituted positions and residues, whereas R presents complete wild-type and candidate sequences without the same mutation list. The SH control retains true mutation-position headers and disrupts biological payload correspondence, so SE-SH does support the value of correctly assigned biological information. However, without a location-only condition, the R-to-SE gain still combines biological annotations with mutation-localization assistance. This limits attribution of the total gain rather than eliminating the evidence for biological information use.
  2. The specialist-gap and generalization results remain conditional on incomplete, non-random coverage and unequal comparison conditions. GPT-6 Astra scores only 267/651 episodes and has no R counterpart; other failures depend on the condition. The temporal study selects 25/59 assays and crosses channels, while its specialist comparison is sensitive to rebuilt alignment quality. The paper discloses these issues, but stronger population-wide conclusions or equivalence to specialists are not established by the current panels.
Questions for authors
  1. How much of the R-to-SE gain remains when R receives the exact mutation positions and residue substitutions, with neutral or missing fields replacing biological annotations? This would distinguish biological information from mutation-location assistance.
  2. What do coverage-aware sensitivity analyses show for the specialist comparison, especially the unfinished GPT-6 panel? Please distinguish uncollected responses from refusals and show how the conclusion changes under plausible performance on missing episodes.
  3. On the post-cutoff assays, is a same-model, same-channel R/SE contrast available? That comparison would materially change the assessment of evidence-driven temporal transfer. Flag For Ethics Review
3 comments
AuthorsOpenReview ↗

Response to Reviewer: Localization and GPT-6 Coverage

We thank the reviewer for recognizing the experimental controls and statistical analysis, and for the specific suggestions regarding mutation localization and incomplete GPT-6 coverage.

1. Separating mutation localization from biological information

We added a location-only condition (L) that provides mutation positions on top of R, without structural or evolutionary fields. These position rows are identical to those in the S and E evidence blocks. L deliberately omits substituted residue identities, as do S and E, ensuring that localization is the only additional information. Complete wild-type and candidate sequences are provided in all conditions.

On the pre-registered 50% stratified assay subset (108/217 assays), we evaluated R, L, and SE using Opus 5 medium under identical settings, with 284 jointly completed episodes from 88 proteins:

ConditionSpearman ρ
R0.3464
L0.3612
SE0.4434
ContrastΔρ95% CIShare of total gain
Localization L − R+0.0148[−0.0059, +0.0345]15.2%
Biological evidence SE − L+0.0822[+0.0529, +0.1083]84.8%
Total SE − R+0.0970[+0.0659, +0.1313]100%

The localization-only gain is not statistically detectable, whereas adding biological evidence after controlling for localization yields a substantial improvement. Approximately 85% of the total point-estimate gain cannot be explained by mutation-localization assistance.

2. GPT-6 coverage and specialist-comparison sensitivity

Of 651 targeted GPT-6 Astra episodes, 300 were collected: 267 scorable, 10 blocked, and 23 unparseable. The remaining 351 were uncollected, not refused. Provider access was suspended at the account level following a biological-content flag (401 token_revoked), pending institutional verification. This was an account-level interruption rather than 351 individual episode refusals, although the missing outcomes remain unknown.

We compared GPT-6 with two specialists on matched observed coverage: VenusREM (structure + MSA, first on the full leaderboard) and GEMME (alignment only).

ContrastCommon episodesΔρ95% CI
GPT-6 − VenusREM261−0.007[−0.045, +0.032]
GPT-6 − GEMME267+0.075[+0.034, +0.115]

To assess sensitivity to the 351 uncollected episodes, we conducted a pattern-mixture analysis. The 33 collected failures remain excluded. Each hypothetical penalty δ represents a per-episode disadvantage for GPT-6 relative to the specialist on uncollected episodes (δ = 0 assumes a tie):

Penalty δGPT-6 − VenusREMGPT-6 − GEMME
0−0.002 [−0.015, +0.012]+0.029 [+0.014, +0.047]
0.02−0.014 [−0.027, −0.000]+0.017 [+0.001, +0.035]
0.03−0.020 [−0.033, −0.006]+0.011 [−0.006, +0.030]
0.05−0.032 [−0.045, −0.018]−0.001 [−0.019, +0.019]

These are assumption-based sensitivity scenarios, not independent evidence about missing outcomes. Imputed values carry no uncertainty, so intervals reflect only observed between-protein variability and are correspondingly narrow. A penalty of approximately 0.02 makes GPT-6's disadvantage relative to VenusREM detectable; at approximately 0.03, its advantage over GEMME is no longer statistically detectable.

We therefore do not claim equivalence to specialist models. On observed shared coverage, GPT-6 outperforms alignment-only GEMME and is not detectably different from VenusREM, but incomplete coverage limits generalization. We will narrow the manuscript's specialist-comparison claims accordingly. Statistical aggregation follows the paper's nested macro evaluation with 10,000 paired protein-level bootstrap resamples.

AuthorsOpenReview ↗

Additional Temporal Validation: Matched R/SE Comparisons

We thank the reviewer for highlighting the need for temporally eligible, fully matched R/SE comparisons.

The original 25-assay panel comes from the PG-LLM temporal benchmark, with first-public score dates spanning February–July 2026. We additionally assembled a new 69-assay panel covering 19 proteins, with experimental scores first released in October–November 2026, to test generalization to more recent outcomes.

We apply two criteria to all comparisons:

(a) Same model, same channel, same settings. Both R and SE arms were run by us with identical model identifiers, reasoning effort, thinking settings, API channels, candidate sampling, prompt versions, and evidence pipelines. The evidence blocks are the only difference. For open-weight models, OpenRouter providers were pinned with fallback disabled (Qwen: Venice FP8; Kimi: AtlasCloud) to avoid differences between hosting configurations.

(b) Model-specific temporal eligibility. Each comparison includes only assays whose exact scores became public after the model's temporal reference. The original panel's dates (2026-02-09 to 2026-07-10) are recorded in study-group-mapping.json from the PG-LLM evaluator bundle. Open-weight models use their publicly verifiable weight-release dates, which bound their training dates; closed models use reported knowledge cutoffs. For Opus 5, we conservatively use 2026-05-31 as the end of its reported cutoff month.

Four models satisfy both criteria, yielding five paired comparisons:

Model / panelChannelEligible assaysCommon episodesR → SE (ρ)Δρ, 95% CI
Qwen3-235B / originalOpenRouter, Venice FP825/25640.0655 → 0.2290+0.1635 [+0.0927, +0.2479]
Opus 4.8 / originalAnthropic Batches25/25590.1027 → 0.2383+0.1356 [+0.0613, +0.2116]
Kimi K2.5 / originalOpenRouter, AtlasCloud25/25750.1835 → 0.2892+0.1057 [+0.0500, +0.1631]
Opus 5 / originalAnthropic Batches15/25390.3126 → 0.3690+0.0565 [−0.0120, +0.1299]
Opus 5 / newAnthropic Batches69/691330.2452 → 0.2854+0.0401 [+0.0216, +0.0592]

Four of five comparisons show statistically detectable improvements. The remaining Opus 5 comparison is positive but inconclusive on the original panel (39 episodes across 7 study clusters). On the new panel, the same model shows a significant improvement across 133 common episodes from 19 clusters. The two panels are evaluated separately, without pooling.

These results include two pre-measurement open-weight models and provide evidence that SE improvements extend beyond potentially memorized benchmark outcomes.

Limitations. Absolute correlations are not comparable across panels due to different candidate sampling and difficulty. All results are conditional on jointly completed episodes: Qwen experienced output truncation, while Opus 5 completed 137/207 R and 197/207 SE episodes on the new panel. The paired R score (0.2452) is close to its score on all completed R episodes (0.2466), although selection bias cannot be excluded. Results use temporal study-group aggregation with 10,000 paired bootstrap resamples.

Reviewer_h6AYOpenReview ↗

Response

Thanks for the response. I would like to maintain my original rating.

Reviewer_5bgqOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

BioEvidence tests whether extra biological information/features helps general-purpose natural language LLMs rank protein variants pathogenicity/VEP. It adds structural and evolutionary features to the input, without further training. Evaluation used sampled ProteinGym variants plus a smaller 25 assay temporal set. Both feature types help, and combining them works best, with correlation increasing from 0.322 to 0.434 in the main experiment, outperforming the vanilla llms’ results and some protein models.

Critical strengths
  1. Practical example of prompt/feature engineering, easily applied to other scenarios. E.g. could add other features also.

  2. The extra features improve results over the basic approach (for the nlp LLMs) and are more competitive with some biological language models. In the main experiment, Spearman correlation rises from 0.322 to 0.434. Combining structural and evolutionary features also beats either alone (makes sense).

Critical weaknesses
  1. Possible Leakage/memorization of labels. This is my main concern. The LLMs may have encountered these public assay measurements during training. Adding or shuffling features does not test that possibility, as extra features could still be helping recognize a variant and recall a known result. The authors do evaluate 25 assays from PG-LLM’s temporal set (I note that this could be made clearer in the text re background/context). However, the paper does not document whether those measurements first became public after each evaluated model’s cutoff. And furthermore, commercial LLMs are notoriously opaque about minor updates or training cutoffs or retrieval, making it harder to independently verify. The benchmark improvement is useful, but it does not yet establish generalization to unseen experimental outcomes or a clean comparison with specialist predictors.

  2. The added value of having the LLM combine the features is unclear. Averaging an evolutionary ranking with the LLM’s original ranking already reaches 0.405, compared with 0.434 when GPT-5.6 Sol receives the extra features. But that simple baseline omits structure, while the LLM receives both feature types. A comparison giving the simple baseline of using both kinds of information (seperately) could help establish what the LLM adds (Table 7;).

  • Seperately, optionally, a feature concatenation baseline compared with the protein models (e.g., combining protein model embeddings with the same structural and evolutionary features ) would be helpful in separating the contribution of the extra features from the llm (E.g. a simple experiment would be a text/llm models embeddings of the evolutionary and structural features, and concatenating that with the feature vector of a protein model, and comparing that to said models baseline or the llm results).
Questions for authors
  1. Temporal validation. Please provide 25 assays’ first public measurement dates and the exact model versions and cutoffs. Does the improvement hold on genuinely post-cutoff measurements with identical settings and features? A fixed open-weight LLM released before those measurements (with a reasonable margin of time) would be the clearest way to test this, and is mandatory in my view.
  2. Simple baseline. Does the LLM with extra features outperform a simple predictor using both evolutionary and structural information, alone or combined with the original LLM ranking?
  3. Did you do any other tests for memorization; or giving the text features or annotations to the protein models, even as a simple embedding concat/linear model? e.g llm leakage test (giving it inputs that would only let it rank if it had the result memorized, without the actualy variant/sequence data for example).
7 comments
AuthorsOpenReview ↗

Response to Reviewer 5bgq

We thank the reviewer for the specific suggestions. To minimize the llm leakage/memorization, we added three experiments (clean temporal set evaluation; evaluation of open-weight models released before the test set; evaluation on new assays released after October 2026). To ensure a fair fusion comparison, we added structural evidence to the baseline. Data sources and each model’s reported knowledge cutoff or weight-release date are documented in Appendix A (Temporal Audit) of the revised PDF.

1. Temporal validation and open-weight model evaluation

First, we verified the earliest public release date of each assay’s scores and repeated the R/SE comparison using only assays released after each model’s temporal reference date.

Second, we added two open-weight models, Qwen3-235B-A22B-Thinking-2507 and Kimi K2.5, released on 25 July 2025 and 27 January 2026. Both predate the earliest score release in the panel (9 February 2026).

Third, we built a new panel of 69 assays across 19 proteins from three studies. All scores first became public after October 2026, five to six months after the reported Opus 5 cutoff. We checked Crossref, Zenodo and preprint histories and found no earlier versions.

Within each row, R and SE differ only in the presence of evidence blocks. All comparisons use matched R and SE runs from the same model, and inference channel.

ModelPanelEligible assaysCommon episodesR → SEΔρ, 95% CI
Qwen3-235B-Thinking (open weights)original 25-assay25/25640.066 → 0.229+0.164 [+0.093, +0.248]
Claude Opus 4.8 (no extended thinking)original 25-assay25/25590.103 → 0.238+0.136 [+0.061, +0.212]
Kimi K2.5 (open weights)original 25-assay25/25750.184 → 0.289+0.106 [+0.050, +0.163]
GPT-5.6 Sol maxoriginal 25-assay23/25690.351 → 0.387+0.036 [+0.011, +0.062]
Claude Opus 5 maxoriginal 25-assay15/25390.313 → 0.369+0.057 [-0.012, +0.130]
Claude Opus 5 maxnew 69-assay69/691330.245 → 0.285+0.040 [+0.022, +0.059]

Five of six confidence intervals exclude zero. The exception is Opus 5 on the original panel, where temporal filtering leaves only 39 episodes from 7 study clusters. On the new panel, the same model improves by +0.040 across 133 episodes. We therefore view the original result as underpowered rather than null.

Some Qwen outputs were truncated, so we report only its 64 jointly completed episodes. On the new panel, Opus 5 completed 137/207 R and 197/207 SE episodes, with R hitting the output budget on the rest. We use only paired episodes. The R score on these episodes (0.245) is close to its score on all completed R episodes (0.247), though selection bias cannot be ruled out.

These results strengthen the evidence that BioEvidence improves predictions on experimental measurements released after model training or release, rather than relying solely on potentially memorized benchmark labels.

2. Fusion baseline

To make the fusion comparison fair, we added structural evidence to the baseline. All weights are fixed at 0.5, nothing is fitted, and structural values come from the same prompt seen by the model.

The first five models used medium effort for both R and SE; Opus 5 used max effort.

ModelRSE_roR + SE_roBioEvidence SESE − (R + SE_ro), 95% CI
Claude Haiku 4.50.1600.3850.3550.302-0.053 [-0.080, -0.030]
Claude Opus 4.70.2000.3840.3610.312-0.048 [-0.070, -0.027]
GPT-5.50.2450.3820.3880.403+0.015 [-0.007, +0.036]
Claude Opus 4.80.3140.3870.4230.395-0.028 [-0.046, -0.011]
GPT-5.6 Sol0.3260.3830.4340.449+0.015 [-0.002, +0.031]
Claude Opus 50.4170.3890.4800.517+0.037 [+0.022, +0.051]

GPT-5.5 and GPT-5.6 Sol remain positive (+0.015 each), but their confidence intervals cross zero. Opus 5 significantly outperforms the new fusion baseline (+0.037). Haiku 4.5, Opus 4.7 and Opus 4.8 perform significantly worse than mechanical fusion.

This is consistent with our original claim that the benefit of LLM evidence integration is model-dependent. The association between no-evidence performance and the fusion margin remains positive (six models: Pearson +0.81, Spearman +0.83), but we do not interpret it as a capability threshold, especially given differences in episode coverage (514–592).

In conclusion, the open-weight models and the new temporal panel provide further checks against label memorization. Qwen's checkpoint predates the original panel's measurements by several months, and the new 69-assay panel tests generalization to newly released experimental results. These controls cannot fully exclude contamination, but they strengthen the evidence for temporal generalization.

AuthorsOpenReview ↗

Temporal Audit: Original 25-Assay Panel

Full details are documented in Appendix A

AuthorsOpenReview ↗

Temporal Audit: New 69-Assay Panel

Full details are documented in Appendix A

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author Response: Scientific Contribution and Evidence Validation (1/2)

We thank the reviewer for the constructive feedback and thoughtful suggestions.

1. Positioning and scientific contribution

Our primary contribution is not a new prompting algorithm or learning architecture, but a systematic investigation of external biological evidence as a complementary scaling dimension for frozen general-purpose LLMs, alongside model capability and inference-time computation.

Our paper is positioned within applications to physical sciences, specifically protein variant-effect prediction. Beyond demonstrating that domain information improves prediction, we identify three important empirical findings:

  1. Evidence availability and correspondence matter. Performance increases with evidence coverage, whereas assigning the same evidence to incorrect variants substantially reduces performance. This demonstrates that evidence utilization depends on correct biological correspondence, not simply additional input information.

  2. Evidence utilization depends on model capability. Stronger models can outperform simple evolutionary rank fusion, whereas weaker models may underperform it. This suggests that accessing biological information and effectively exploiting it are distinct capabilities.

  3. Evidence complements inference-time scaling. In a controlled setting, evidence-assisted low-effort inference outperforms higher-effort inference without evidence, demonstrating that external knowledge can provide substantial predictive benefits beyond increasing reasoning effort alone.

Together, these findings establish evidence availability as an experimentally controllable dimension of scientific prediction and characterize when frozen LLMs benefit from it. BioEvidence enables these investigations without modifying model parameters or introducing task-specific training.

2. Alternative evidence representations

To examine whether the improvements depend on the specific evidence format, we additionally tested two alternative representations: a unified table containing both evidence modalities and candidate-level evidence cards placed beside the corresponding sequences.

Using identical biological evidence, Claude Opus 5 at medium effort, the same API channel, and the same output cap, we obtained:

Representation comparisonShared episodesΔρ95% CI
Table − original SE118+0.0052[−0.0120, +0.0187]
Card − original SE116+0.0142[−0.0039, +0.0350]

Neither difference was statistically detectable. Although these results do not establish equivalence between representations, they suggest that the observed benefits are not restricted to our original evidence layout, supporting the broader applicability of evidence-conditioned inference rather than the superiority of a particular prompt format.

3. Interpretation of "reasoning" and "evidence scaling"

We agree that improved ranking alone does not establish mechanistic biological reasoning. Our use of "reasoning" refers to evidence-conditioned predictive inference, rather than validated causal or mechanistic explanations.

Our controlled coverage, modality, and shuffled-evidence experiments establish how external evidence affects prediction in the tested model and effort setting. The additional open-weight experiments below provide complementary evidence that these benefits extend across model families.

We will clarify these distinctions and describe evidence scaling as an empirically observed coverage-dependent relationship, rather than a universal scaling law.

AuthorsOpenReview ↗

Author Response: Open-Weight Validation and Cost Analysis (2/2)

4. Open-weight model validation

To address generalizability beyond proprietary models, we evaluated two additional open-weight models: Kimi K2.5 and Qwen3-235B-A22B-Thinking-2507, released on January 27, 2026 and July 25, 2025, respectively.

Both model releases precede the earliest public availability of the exact variant-level score tables in our original 25-assay temporal panel (February 9, 2026, according to the benchmark registry). This temporal separation reduces the risk of memorization of publicly available experimental labels, although it cannot completely exclude data contamination.

Each model was evaluated under matched R/SE settings using a pinned provider:

ModelShared episodesΔρ (SE − R)95% CI
Kimi K2.575+0.1057[+0.0500, +0.1631]
Qwen3-235B-A22B-Thinking-250764+0.1635[+0.0927, +0.2479]

Kimi completed all 75 episodes in both conditions. Qwen produced scorable rankings for 74/64 episodes under R/SE, respectively, with output truncation contributing to incomplete coverage. Its paired comparison is therefore conditional on the 64 shared episodes.

Both models show substantial positive evidence gains with confidence intervals excluding zero. These findings extend evidence-conditioned improvements to two independent open-weight model families on temporally newer experimental measurements, although they do not establish model-size scaling effects.

5. Cost and efficiency

We additionally measured actual token usage and API charges using Claude Opus 5 at medium effort. The following totals cover the same 324 attempted requests per condition, including refusals. Output tokens include thinking tokens, and charges reflect the Batch API rates used for these runs. Latency was measured separately through a direct-API probe.

MeasurementRSE
Input tokens (millions)6.00811.688
Output tokens (millions)1.0990.713
API charge (USD)$28.76$38.14
Median latency (seconds)34.828.0

Evidence augmentation increased API charges by 32.6% in this setting. The latency estimates are descriptive, based on only 10 completed, unmatched requests per condition, and do not establish a latency advantage.

We also quantified evidence-preprocessing requirements:

Evaluation panelEvidence constructionAllocated CPU core-hours
Main ProteinGymReused MSAs and structures; incremental evidence extraction and prompt construction~5.4
Original 25-assay temporal panelNew MSA searches, reused structures, and evidence construction~225

The main-benchmark estimate excludes the upstream cost of generating reused MSAs and structures. Thus, these measurements characterize the resource requirements of evidence augmentation under the evaluated workflows.

We agree that the original low-effort evidence comparison demonstrates a predictive performance advantage, not necessarily an end-to-end efficiency advantage. We will clarify this distinction and report the observed computation and monetary costs.

Overall, we hope these additional experiments and clarifications better communicate the scientific contribution of BioEvidence: not optimizing a particular prompt format, but systematically establishing how external biological evidence affects the predictive capabilities of frozen LLMs, under what conditions it provides benefits, and where its limitations lie.

Reviewer_SVLTOpenReview ↗

Response to the Authors

I thank the authors for their helpful clarifications and additional experiments, which have addressed most of my concerns. I will therefore raise my score to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.