acceptodds
Under review as a conference paper at ICLR 2027

BioStitch: Gated Cross-Attention for Biological-Language Modeling

Abstract

Integrating biological foundation models with large language models (LLMs) is crucial for a broad range of biological tasks, from sequence understanding to text-guided biological generation. However, existing approaches typically connect these models through input-space fusion, without fully exploiting the rich representations distributed across their internal layers. To understand how pretrained representations can be better connected beyond input-space fusion, we systematically probe the layer-wise interaction between biological models and LLMs. We find that, when transferring DNA representations into LLMs, intermediate layers provide more effective features on average than conventionally used final-layer representations. Guided by this finding, we propose BioStitch, a versatile framework that stitches biological foundation models and LLMs through multi-level gated cross-attention, allowing hidden states in one model to selectively query internal representations from the other. On biological-sequence-to-text tasks, BioStitch delivers strong improvements on DNA variant-effect benchmarks, protein question answering, and zero-shot single-cell annotation, without multimodal pre-training. On text-conditioned protein generation, BioStitch substantially improves text-sequence alignment over conventional input-space conditioning. BioStitch is also highly efficient in the measured long-DNA setting, reducing peak training memory by up to and LLM prompt KV-cache usage by . All these results indicate that BioStitch is an effective and efficient framework for connecting specialized biological and language representations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Multimodal Large Language Models, AI for Science, Gated Cross-Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aqS6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BioStitch, a framework that connects biological foundation models and LLMs through multi-level gated cross-attention. The framework supports biology-to-language tasks, including DNA variant-effect prediction, protein question answering, and single-cell annotation, as well as language-conditioned protein generation. The paper also reports reductions in training memory and prompt KV-cache usage for long biological inputs.

Critical strengths
  • Proposes a potentially reusable interface supporting both biological foundation models and LLMs for efficient integration.

BioStitch aims to integrate biological foundation models and LLMs without requiring the biological input to be represented entirely as text. The unified formulation for both biology-to-language and language-to-biology tasks provides a reusable interface for future biological applications rather than a collection of task-specific pipelines.

  • Demonstrates substantial improvements in a relatively rigorous DNA variant-effect evaluation.

The improvements in the DNA variant-effect experiments are impressive. BioStitch outperforms BioReason across four dataset–model configurations, with an average F1 improvement of 15.27 percentage points, under a strict held-out evaluation protocol. The authors also investigate potential confounding factors, including the original baseline's evaluation protocol and the effect of connector warm-up. This makes the DNA experiments more convincing than results based only on comparisons with published scores.

  • Reports meaningful reductions in training memory and prompt KV-cache usage for long biological inputs.

BioStitch reduces the need to place long biological representations directly into the LLM's self-attention sequence. In the reported long-DNA setting, peak training memory decreases from 79.9 GB to 16.8 GB, while the LLM prompt KV cache decreases from 476.9 MB to 6.65 MB. These are meaningful efficiency gains for applications involving long biological sequences.

Critical weaknesses
  • The distinction between BioStitch and existing cross-attention-based multimodal architectures needs to be established more clearly.

Cross-attention and gated cross-attention are established mechanisms, including in architectures such as Flamingo and Evolla. The main contribution of BioStitch appears to be its use of multi-level gated cross-attention to connect pretrained biological and language models, together with its analysis of source-layer selection and its application to both biology-to-language and language-to-biology tasks. However, the current results do not fully isolate the contributions of multi-level integration, gating, and source-layer selection. I encourage the authors to conduct controlled ablation experiments comparing single-layer cross-attention, multi-layer cross-attention without gates, and the full BioStitch architecture under matched training conditions. This would clarify whether the proposed design offers a substantive advantage over simpler alternatives and better establish the methodological contribution of the work.

  • BioStitch’s median-source-layer choice is motivated by experiments on DNA-to-text tasks; it should not be interpreted as a universally validated optimum for every experiment.

The median layer is best understood as a practical default motivated by the DNA experiments, not an established optimum across all modalities. As DNA sequences, protein representations, and single-cell expression profiles have different structures and representational properties, a layer-selection strategy that works well for a DNA encoder may not be optimal for a protein model or a transcriptomic encoder. In the task of text-to-protein generation, BioStitch demonstrates that a shared gated cross-attention interface can be applied in both directions. However, the evidence that its layer-selection strategy is appropriate in both directions is weak. The reverse text-to-protein experiment supports the framework’s applicability, but does not by itself validate the generality of the median-source-layer principle.

  • The paper presents four application settings, but the experimental comparisons are not equally controlled.

For DNA variant-effect prediction, the authors conduct a relatively rigorous comparison against BioReason using matched test sets and a strict held-out protocol. For protein question answering, however, the comparison with Protein2Text involves substantially different training resources and language models. Although BioStitch's performance is encouraging, this is not a controlled comparison of the architectures. The same distinction matters when interpreting comparisons with published systems in single-cell annotation and protein generation.

  • The computational improvements are promising, but different efficiency metrics measure different aspects of the system.

For the reported long-DNA setting, BioStitch substantially reduces peak training memory and prompt KV-cache usage. These results support the claim that BioStitch can reduce particular training and prompt-state costs. They do not establish a general improvement in inference efficiency across all workloads. It would also be useful to report how these metrics scale with biological input length and the number of cross-attention layers. Such results would make the practical advantages and limitations of the architecture clearer.

Questions for authors
  • Because gated cross-attention and intermediate-layer fusion have precedents in multimodal learning, what are the differences between BioStitch and the closest existing approaches, particularly Evolla?
  • What is the essential architectural contribution of BioStitch? How much of the observed gain comes from gated cross-attention itself, as opposed to using intermediate biological representations or inserting cross-attention at multiple layers?
  • The gains in DNA variant-effect prediction are substantial, while it raises a further question: What information does BioStitch extract from the biological representations that enables it to identify these additional pathogenic variants?
  • What evidence supports the biological quality of generated proteins? Beyond text–sequence alignment, do the generated sequences exhibit improvements in independent structural or functional evaluations?
7 comments
AuthorsOpenReview ↗

Response to Reviewer aqS6 (Part 1/5)

We thank Reviewer aqS6 for the careful and constructive review. We appreciate the suggestions on architectural positioning, source-layer selection across task directions, the comparability of the experiments, and the interpretation of computational efficiency and biological results. We address these points below with additional experiments and analyses, which we have incorporated into the revised manuscript.

Weakness W1: The distinction between BioStitch and existing cross-attention-based multimodal architectures needs to be established more clearly.
Question Q2: What is the essential architectural contribution of BioStitch? How much of the observed gain comes from gated cross-attention itself, as opposed to using intermediate biological representations or inserting cross-attention at multiple layers?

Response:

Thank you for your suggestion. BioStitch makes source extraction and receiver injection depths explicit design choices and applies a common internal gated cross-attention formulation to biological understanding and language-guided generation. We have clarified the architectural positioning in the Introduction and Related Work and added the component ablations to Section 4.2 and Appendix N.1.

To examine these choices, we retrained seven VE-NonSNV configurations using NT-v2-500M and Qwen3-1.7B with the same data splits, initialization, training seed, and two-stage training budget. The full model uses middle-layer NT features and injects them at shallow, middle, and deep Qwen layers, corresponding to source layer 15 and injection layers 6, 14, and 22. The no_gate variant retains all three connections but fixes the effective coefficient to 1. The single-point variants retain the middle source features and use only one injection point. Finally, source_low and source_high retain all three injection points but use shallow or deep NT features, corresponding to source layers 6 and 23. All configurations use the same final-checkpoint decoding and answer-scoring protocol.

ConfigurationAccuracy (%)Pathogenic F1 (%)Balanced accuracy (%)
no_gate72.96777.23772.240
single_low74.68577.77874.950
single_mid73.88377.00174.186
single_high69.75973.17169.988
source_low68.95873.93867.901
source_high72.96777.54371.909
BioStitch75.94579.84675.056

The full configuration achieves the highest scores across all three metrics. Its pathogenic F1 is 2.609 percentage points above the fixed-gate variant and 2.068 points above the strongest single-point variant. Changing to shallow or deep source features gives F1 scores of 73.938% and 77.543%, respectively. These matched comparisons support the combination of learned gating, middle-layer source features, and multi-layer injection, with the full configuration outperforming each of the simpler alternatives considered.

AuthorsOpenReview ↗

Response to Reviewer aqS6 (Part 2/5)

Weakness W2: BioStitch’s median-source-layer choice is motivated by experiments on DNA-to-text tasks; it should not be interpreted as a universally validated optimum for every experiment.

Response:

Thank you for highlighting this distinction. We added a text-to-protein source-layer comparison, now reported in Appendix O.1 and referenced from Section 4.2. The revised manuscript treats intermediate extraction as a practical default and reports the metric-dependent generation results rather than a universal optimum. Following the original text-to-protein setup, we used Qwen3-1.7B as the frozen text source and ProGen2-base as the frozen protein receiver, training only the connector on the ProDVa Molinst-SwissProtCLAP corpus. Evaluation used PDFBench with the held-out Mol-Instructions protein-design prompts. We varied the Qwen extraction layer across 8, 14, 20, and 28 while keeping injection at ProGen2 blocks 5, 14, and 22, counted from 1. The backbone checkpoints, connector architecture, and training setup were held fixed across the four runs.

MetricL08L14L20L28
ProTrek ×100 ↑8.71928.81728.83458.1840
Normal 4-way selection (%) ↑39.264036.556636.664737.4509
Normal 10-way selection (%) ↑30.763627.545227.446928.4935
Repeat (%) ↓16.088316.690815.969516.3382
Rep-2 (%) ↓55.923756.369255.690156.3429
Rep-5 (%) ↓16.026516.724715.902116.6012
ProGen2 PPL ↓ (decoder diagnostic)4.987544.950915.016394.98048
ProtGPT2 PPL ↓245.98177246.78696241.14229247.34379
RITA PPL ↓6.064436.076376.063106.11106

The preferred source depth varies across metrics. The two intermediate settings, layers 14 and 20, obtain higher ProTrek scores than the final layer, and layer 20 has the lowest repetition measures and perplexity under ProtGPT2 and RITA. The shallower layer 8 performs best on the Normal selection metrics, while the middle layer 14 has the lowest ProGen2 perplexity, which serves as a decoder-likelihood diagnostic.

The reverse-direction results therefore support intermediate extraction as a practical choice for balancing text–protein alignment and sequence quality. They also provide a direct empirical basis for refining source depth according to the generation objective, extending the layer-selection analysis beyond DNA-to-text.

Weakness W3: The paper presents four application settings, but the experimental comparisons are not equally controlled.

Response:

Thank you for your suggestion. In the protein question-answering experiment, the systems share the official Protein2Text-QA test set, but use different training data and language models. The public release provides evaluation subsets rather than the complete training corpus used by the reference system. We therefore constructed our own adaptation set from the released IntroductionQA and DiscussionQA subsets, using substantially fewer training examples than the reference system.

To check for training–test overlap, we audited the training and test proteins by exact amino-acid sequence identity and protein name, finding no shared proteins under these criteria. The complete official test set was retained. The evaluation therefore compares systems trained with their respective resources on the same complete official test set.

We have made this comparison scope explicit in Sections 4.4–4.6 for protein question answering, single-cell annotation, and protein generation, distinguishing the published-system references from the separately controlled comparisons. The ProGen2 generation experiment separately includes a prefix control with the same pretrained backbones, training corpus, and training budget. For protein question answering, retraining the systems on a common training set with matched backbones and budgets would provide a more direct architectural comparison and is a useful direction for future work.

AuthorsOpenReview ↗

Response to Reviewer aqS6 (Part 3/5)

Weakness W4: The computational improvements are promising, but different efficiency metrics measure different aspects of the system.

Response:

Thank you for distinguishing these efficiency measures. To examine the scaling behavior more directly, we added measurements at biological input lengths of 512, 2048, and 4096 NT tokens, comparing BioReason with single-point and three-point BioStitch. We have included the sweep in Appendix O.3 and revised Section 4.7 to distinguish training memory, prompt self-KV, prefill, decoding, and end-to-end latency. The results are shown below:

Biological tokensMetric ↓BioReasonBioStitch single-pointBioStitch three-point
512Peak training memory (GiB)15.25312.77513.462
512Training time per sample (ms)244.0308.9367.8
512Prefill (ms)58.360.164.8
512Time to first token (ms)86.3111.1116.4
512Decode (ms/token)51.7352.9254.68
512Total time for 64 generated tokens (s)3.3463.4503.563
512Peak inference memory (GiB)9.1729.2129.494
512Prompt self-KV (MiB)88.7584.1024.102
2048Peak training memory (GiB)23.46312.81613.571
2048Training time per sample (ms)354.7285.0292.5
2048Prefill (ms)88.359.663.0
2048Time to first token (ms)135.1114.8117.6
2048Decode (ms/token)51.0252.0954.23
2048Total time for 64 generated tokens (s)3.3513.4003.537
2048Peak inference memory (GiB)9.4419.2329.513
2048Prompt self-KV (MiB)340.7584.1024.102
4096Peak training memory (GiB)34.34412.86713.692
4096Training time per sample (ms)627.5346.9385.2
4096Prefill (ms)162.160.362.4
4096Time to first token (ms)283.4189.1191.9
4096Decode (ms/token)51.1851.6453.95
4096Total time for 64 generated tokens (s)3.5093.4453.593
4096Peak inference memory (GiB)10.0359.3169.539
4096Prompt self-KV (MiB)676.7584.1024.102

The strongest benefits appear in training memory and prompt self-attention KV storage as biological input length increases. At 4096 tokens, three-point BioStitch uses 13.692 GiB of peak training memory, compared with 34.344 GiB for BioReason, while prompt self-KV storage is 4.102 versus 676.758 MiB. This cache measurement covers the receiver's prompt self-attention states.

These measurements show that BioStitch keeps training-memory and prompt self-KV costs nearly constant as biological input length increases, while reducing prefill time and time to first token for longer inputs. Comparing single-point and three-point injection also quantifies the additional cost of multi-layer integration, directly addressing how efficiency scales with input length and the number of cross-attention layers.

Reviewer_AdUROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose BioStitch, a new method to fuse biological foundation models with large language models that allows LLMs to reason over biological sequences and vice versa, generation of biological sequences based on LLM input. The method replaces the simpler final layer representations as input into the other language model approach used in prior work with a gated cross-attention approach that enables use of intermediate layer representations and injection into several receiver layers, which the authors show improves performance on benchmarks compared to BioReason and also is computationally more efficient.

Critical strengths

The method is sound and the benefit of using intermediate representations to fuse language models is well established from other domains, so it is believable that this also translates well to this settings, and the benchmarks of the authors seem to support this assessment (although see concerns below).

Critical weaknesses

I see limited methodological novelty in the proposed method. The core idea is well established from other domains, and the key contribution of the authors appears to be to test this in this novel setting. As such the generally positive evaluation results are promising, but also not very surprising.

I have major concerns about the utility of the proposed model and the evaluation. The method is primarily compared against other language-model-based or multimodal biological approaches, and evaluations with simpler established baselines that are commonly used to solve the same tasks are missing (e.g., traditional VEP scores or probes on genome foundation models). Without these comparisons, it is unclear to me if these models have practical utility.

Furthermore, some of the reported improvements in performance are small, and CIs, statistical significance tests, etc. are missing in many analyses, and it is thus unclear to me if and which of the results are significant.

Questions for authors

What concrete real-world applications benefit from this approach? Are there benchmarks demonstrating advantages over existing biological prediction methods?

Why are biological foundation models frozen throughout? Would we not expect the biological foundation models to benefit from additional task specific training as well?

Why are relatively small LLMs used? Is this primarily due to computational constraints, or is there evidence that large models don't provide additional benefits?

6 comments
AuthorsOpenReview ↗

Response to Reviewer AdUR (Part 1/4)

We thank Reviewer AdUR for the detailed and constructive review. We appreciate the suggestions on clarifying BioStitch's contribution and practical utility, strengthening the baseline comparisons and statistical analysis, and explaining our backbone adaptation and model-size choices. We address each point below and have incorporated the additional results and clarifications into the revised manuscript.

Weakness W1: I see limited methodological novelty in the proposed method. The core idea is well established from other domains, and the key contribution of the authors appears to be to test this in this novel setting. As such the generally positive evaluation results are promising, but also not very surprising.

Response:

Thank you for this assessment. Building on cross-attention and gating, BioStitch contributes two design advances for connecting pretrained biological and language models to support both sequence understanding and language-guided generation.

First, our layer-wise analysis provides evidence for where the two models should be connected. By varying source extraction and receiver injection depths on DNA-to-text tasks, we find that source depth has the larger effect on performance and that intermediate source layers provide the strongest aggregate results across the tested DNA configurations. These findings guide BioStitch's selection of the source representations exposed to the receiver, making layer selection an empirically informed design choice.

Second, research on fusion between pretrained biological and language models remains relatively limited, and cross-attention-based approaches in this setting are still developing. Against this background, BioStitch explores an internal gated cross-attention interface for connecting biological information with text. For biological understanding, biological representations condition a language model to generate textual answers. For text-guided protein generation, language representations condition a pretrained protein decoder to generate protein sequences. These settings use a common connection formulation, with separately trained connectors.

Together, these contributions address where pretrained models should be connected and how the same internal connection design can support both biological understanding and language-guided generation. We have revised the introduction and related work to make this contribution explicit, including the use of separately trained connectors for biological-sequence-to-text tasks and text-guided protein generation.

AuthorsOpenReview ↗

Response to Reviewer AdUR (Part 2/4)

Weakness W2: I have major concerns about the utility of the proposed model and the evaluation. The method is primarily compared against other language-model-based or multimodal biological approaches, and evaluations with simpler established baselines that are commonly used to solve the same tasks are missing (e.g., traditional VEP scores or probes on genome foundation models). Without these comparisons, it is unclear to me if these models have practical utility.

Question Q1: What concrete real-world applications benefit from this approach? Are there benchmarks demonstrating advantages over existing biological prediction methods?

Response:

Thank you for highlighting the importance of simpler baselines. We added a probe based on the frozen Nucleotide Transformer (NT) to evaluate its predictive performance without an LLM. Specifically, we extracted representations from NT layers 6, 15, and 23, mean-pooled the reference and variant representations, and concatenated them with their difference. We then trained the same one-hidden-layer MLP with GELU activation for each layer, using the same VEC and VE-NonSNV data splits as our main experiments. Results are reported as mean ± standard deviation across three seeds, with the full setup and results provided in Appendix N.2.

MethodVEC accuracy (%)VEC F1 (%)VE-NonSNV accuracy (%)VE-NonSNV F1 (%)
NT+MLP, shallow L663.56 ± 1.4041.89 ± 1.9766.63 ± 1.8366.11 ± 0.92
NT+MLP, middle L1569.61 ± 0.8144.86 ± 1.3268.80 ± 2.7566.74 ± 0.74
NT+MLP, deep L2366.04 ± 0.5040.43 ± 2.2868.61 ± 1.6065.67 ± 0.75
BioReason: NT+Qwen3-1.7B58.8856.5655.2161.40
BioReason: NT+Qwen3-4B58.1551.4155.9055.28
BioStitch: NT+Qwen3-1.7B66.5966.1772.9777.39
BioStitch: NT+Qwen3-4B64.9664.0674.3478.12

The middle-layer probe achieves the best results among the three NT layers on both datasets, with 69.61% accuracy on VEC and 68.80% on VE-NonSNV. Compared with this probe, BioStitch achieves higher F1 on both tasks (66.17%/64.06% vs. 44.86% on VEC, and 77.39%/78.12% vs. 66.74% on VE-NonSNV). Its accuracy is also higher on VE-NonSNV (72.97%/74.34% vs. 68.80%), although the probe achieves higher accuracy on VEC (69.61% vs. 66.59%/64.96%).

AuthorsOpenReview ↗

Response to Reviewer AdUR (Part 3/4)

Weakness W3: Furthermore, some of the reported improvements in performance are small, and CIs, statistical significance tests, etc. are missing in many analyses, and it is thus unclear to me if and which of the results are significant.

Response:

Thank you for your suggestion. We added 95% confidence intervals (CIs) for the evaluated scores and the differences between BioStitch and BioReason. Section 4.3 now reports the paired uncertainty results and briefly describes the sequence-pair group bootstrap, while Appendices E and F provide the complete DNA and KEGG interval tables. For VEC and VE-NonSNV, the first table reports accuracy and pathogenic-class F1 for each method.

TaskQwen3MethodAccuracy (%) [95% CI]Pathogenic F1 (%) [95% CI]
VEC1.7BBioStitch66.59 [63.08, 70.07]66.17 [61.54, 70.63]
VEC1.7BBioReason58.88 [55.72, 62.26]56.56 [52.05, 61.21]
VEC4BBioStitch64.96 [61.75, 68.47]64.06 [59.43, 68.61]
VEC4BBioReason58.15 [54.60, 61.67]51.41 [46.03, 56.69]
VE-NonSNV1.7BBioStitch72.97 [69.87, 75.95]77.39 [74.31, 80.15]
VE-NonSNV1.7BBioReason55.21 [51.66, 58.30]61.40 [57.71, 64.72]
VE-NonSNV4BBioStitch74.34 [71.48, 77.32]78.12 [75.30, 80.88]
VE-NonSNV4BBioReason55.90 [52.69, 59.11]55.28 [51.31, 59.21]

To assess the improvements directly, the next table reports the paired differences, defined as BioStitch minus BioReason. Across all four configurations, the 95% CIs for both accuracy and pathogenic F1 lie entirely above zero, supporting positive gains in each evaluated setting. The F1 gains range from 9.62 to 22.84 percentage points.

TaskQwen3ΔAccuracy (pp) [95% CI]ΔPathogenic F1 (pp) [95% CI]
VEC1.7B+7.70 [3.63, 11.70]+9.62 [4.54, 14.65]
VEC4B+6.81 [2.70, 11.13]+12.65 [6.89, 18.73]
VE-NonSNV1.7B+17.75 [13.52, 22.11]+15.99 [12.12, 20.06]
VE-NonSNV4B+18.44 [14.09, 22.57]+22.84 [18.33, 27.28]

We also examined the paired differences on KEGG using the merged validation–test cohort used for checkpoint selection. Here, the weighted-, macro-, and micro-F1 gains are 7.50, 9.81, and 11.10 percentage points, respectively, with all three 95% CIs excluding zero. The accuracy difference is 3.79 percentage points, although its interval reaches zero at the lower bound, so the evidence for an accuracy gain is less conclusive.

KEGG metricBioStitch − BioReason [95% CI], pp
Accuracy+3.79 [0.00, 7.61]
weighted-F1+7.50 [3.35, 11.03]
macro-F1+9.81 [3.37, 15.50]
micro-F1+11.10 [6.15, 14.67]

These analyses directly address which improvements are statistically supported: the accuracy and pathogenic-F1 gains on both DNA tasks and the weighted-, macro-, and micro-F1 gains on KEGG all have 95% confidence intervals above zero.

Reviewer_6kFeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes BioStitch, a unified framework that connects models through multi-level gated cross-attention. By extracting intermediate representations from the source model and injecting them into multiple layers of the receiver, BioStitch supports both biological sequence understanding and text-conditioned protein generation. However, the evaluation lacks direct comparisons with closely related cross-attention-based fusion methods, and the ablation studies do not fully isolate the contributions of individual architectural components, particularly the gating mechanism.

Critical strengths
  • Well-written. The paper is clearly written and well organized, making the proposed method and experimental findings easy to follow.

  • Diverse experimental settings. The method is evaluated across multiple tasks, demonstrating its generalization across different modalities and task settings.

Critical weaknesses
  • Limited novelty. The proposed Gated Cross-Attention mechanism largely builds upon existing cross-attention-based fusion approaches.

  • Limited comparison with existing fusion methods. The experiments mainly compare BioStitch with input-space fusion approaches, while lacking direct comparisons with closely related cross-attention-based methods.

  • Incomplete ablation studies. Although the paper investigates source-layer selection and multi-level injection, it lacks ablations isolating the contributions of key architectural components, particularly the gated mechanism.

Questions for authors
  • Comparison with existing fusion methods. Could the authors provide a controlled comparison with existing cross-attention-based fusion methods, to better demonstrate the advantages of BioStitch over prior fusion architectures?

  • Incomplete ablation studies. Could the authors provide additional ablation studies to isolate the contributions of individual architectural components, particularly the gated mechanism, and clarify whether each component is necessary for the reported performance improvements?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6kFe (Part 1/2)

We thank Reviewer 6kFe for the thoughtful and constructive review. We appreciate the suggestions on clarifying BioStitch's novelty, comparing it with closely related cross-attention methods, and examining the contributions of individual architectural components. We address these points below through additional comparisons and ablations, which we have incorporated into the revised manuscript.

Weakness W1: Limited novelty. The proposed Gated Cross-Attention mechanism largely builds upon existing cross-attention-based fusion approaches.

Response:

Thanks for the comment. We recognize that cross-attention and gating are established mechanisms, as exemplified by Flamingo and Evolla. Evolla is a particularly close architectural reference because, like BioStitch, it uses gated cross-attention to connect protein representations with a language model. For protein representation extraction, Evolla uses last-layer features, while BioStitch uses intermediate-layer features. Building on these established mechanisms, the novelty of BioStitch lies in two complementary aspects of how pretrained biological and language models are connected. We have clarified this positioning in the Introduction and Related Work of the revised manuscript.

First, we make layer selection an explicit design choice supported by empirical analysis. By examining both source extraction and receiver injection depths on DNA-to-text tasks, we find that source depth has the stronger effect on performance and that intermediate source layers provide the strongest aggregate performance across the tested DNA configurations. These findings provide an empirical basis for choosing which pretrained source representations should be exposed to the receiver.

Second, research on fusion between pretrained biological and language models remains relatively limited, and cross-attention-based approaches in this setting are still developing. Against this background, BioStitch explores an internal gated cross-attention interface for connecting biological information with text. For biological understanding, biological representations condition a language model to generate textual answers. For text-guided protein generation, language representations condition a pretrained protein decoder to generate protein sequences. These settings use a common connection formulation, with separately trained connectors.

Weakness W2: Limited comparison with existing fusion methods. The experiments mainly compare BioStitch with input-space fusion approaches, while lacking direct comparisons with closely related UI cross-attention-based methods.

Question Q1: Comparison with existing fusion methods. Could the authors provide a controlled comparison with existing cross-attention-based fusion methods, to better demonstrate the advantages of BioStitch over prior fusion architectures?

Response:

Thank you for this assessment. We added a comparison with Evolla, a particularly close architectural reference that, like BioStitch, uses gated cross-attention to connect protein representations with a language model. In terms of protein representation extraction, EvoLLA uses the last-layer representations, while BioStitch uses intermediate-layer representations.

We compare BioStitch and Evolla on the Mol-Instructions datasets (https://proceedings.iclr.cc/paper_files/paper/2024/file/d346d91999074dd8d6073d4c3b13733b-Paper-Conference.pdf) using their official training and test splits.

We have added the setup and complete result table to Section 4.4 of the revised manuscript. The table below reports the experimental results, where ↑ indicates that higher values are better and ↓ indicates that lower values are better.

MetricEvollaBioStitch
ROUGE-1 ↑39.94940.042
ROUGE-2 ↑20.23920.952
ROUGE-L ↑36.38036.610
BLEU-2 ↑25.79426.175
BLEU-4 ↑14.32314.648
METEOR ↑37.70537.847
Answer cross-entropy ↓0.4041090.403490

BioStitch achieves higher scores on all six text-generation metrics. For example, ROUGE-2 increases from 20.239 to 20.952, while BLEU-4 increases from 14.323 to 14.648. Answer cross-entropy also decreases slightly, from 0.404109 for Evolla to 0.403490 for BioStitch. These results provide the requested comparison with a closely related cross-attention method and support the use of intermediate protein representations for conditioning text generation.

AuthorsOpenReview ↗

Response to Reviewer 6kFe (Part 2/2)

Weakness W3: Incomplete ablation studies. Although the paper investigates source-layer selection and multi-level injection, it lacks ablations isolating the contributions of key architectural components, particularly the gated mechanism.

Question Q2: Incomplete ablation studies. Could the authors provide additional ablation studies to isolate the contributions of individual architectural components, particularly the gated mechanism, and clarify whether each component is necessary for the reported performance improvements?

Response:

Thank you for highlighting this gap in the ablation studies. We evaluated the full model and six variants on VE-NonSNV using NT-v2-500M and Qwen3-1.7B. Section 4.2 now summarizes the component ablations, with the complete setup and results in Appendix N.1. All seven configurations were retrained with the same data splits, initialization, training seed, sample order, and two-stage budget, and their final checkpoints were evaluated using greedy decoding.

The full configuration extracts middle-layer NT features at layer 15 and injects them after shallow, middle, and deep Qwen layers 6, 14, and 22. To examine learned gating, no_gate retains the same source features and all three connections but fixes the effective gating coefficient to 11. The single_low, single_mid, and single_high variants retain the middle source features and use only one injection point. Finally, source_low and source_high retain all three injection points but extract shallow or deep NT features at layers 6 and 23. The table reports accuracy, pathogenic F1, and balanced accuracy (BAcc), the mean of benign and pathogenic recall.

ConfigurationAccuracy (%)Pathogenic F1 (%)Balanced accuracy (%)
no_gate72.96777.23772.240
single_low74.68577.77874.950
single_mid73.88377.00174.186
single_high69.75973.17169.988
source_low68.95873.93867.901
source_high72.96777.54371.909
BioStitch (full)75.94579.84675.056

Learned gating increases accuracy from 72.967% to 75.945% and pathogenic F1 from 77.237% to 79.846%, a gain of 2.609 percentage points in F1. The full configuration also exceeds the strongest single-point variant, which reaches 77.778% F1, while shallow and deep source extraction yield 73.938% and 77.543%, respectively. The full model achieves the highest scores across all three metrics. These ablations directly address the missing component comparisons and support the combination of learned gating, middle-layer source features, and multi-level injection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.