- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes how to fine tune biological language models from different modalities to create representations for interacting systems. The paper connects pretrained models with cross attention adapters, and uses the conditional log likelihood to apply a contrastive objective. They benchmark this on binding prediction, drug resistance, and mutation effect scoring, and show modest to significant iimprovement across benchmarks.
Critical strengths
- The method is reasonable, clearly defined, and seems to work. For a methodological contribution this is quite important.
- While I've seen individual components of this before (log likelihoods for contrastive fine-tuning), I haven't seen it specifically applied to multimodal/interacting systems.
Critical weaknesses
When evaluating a paper like this, I'm mostly considering whether I think the methodology is something the ai x bio community will find interesting. I think this can occur in two main ways: a clever method that we didn't think of, or very strong results that will inform future work.
My main issue is I don't see that this work satisfies either of these approaches. There isn't anything very new in the method; the primary advance there seems to be applying contrastive objectives to different modalities. I think this is a natural thing to do, but somewhat unsurprising. Turning to the results, they seem to be fairly modest. Given these points, I have a hard time understanding what methodological value this would provide to the machine learning community. I qualify this by noting I'm not an expert on any of the three tasks the authors use, so I invite discussion on this point.
I think if the authors could demonstrate that this approach is useful for a more substantive task (e.g., binder design), that would be compelling. There are databases like STRING for protein-protein interactions where the contrastive objective might be useful, though converting it to a generative approach seems challenging and potentially beyond this work. The authors do explicitly note that this approach could be used for generation, but this doesn't seem to be present in the main text.
Questions for authors
- Can this be used for generation? if the generative results are significant, I would change my score to an acceptance.
6 comments
R2. Methodological novelty and significance for AI × Biology
"I'm mostly considering whether I think the methodology is something the ai x bio community will find interesting. I think this can occur in two main ways: a clever method that we didn't think of, or very strong results that will inform future work."
We really appreciate this framing. Evaluating methodological significance through either a new conceptual angle or results strong enough to steer future work is exactly what motivated LogiCA. We believe our revised work provides evidence for both.
1. Conceptual novelty: Aligning native logits rather than pooled representations
The core idea isn't just "applying contrastive learning to biology." The bottleneck we wanted to tackle is where and how biological models interact. Standard CLIP-style models align pooled representations to produce scalar compatibility scores. While effective for retrieval, these scores no longer retain the native token-probability interface. Once interaction is reduced to scalar similarity, we cannot directly ask: "What residue should go here in the presence of this molecule?"
This distinction has practical consequences. On LOCO-30%, pooled latent methods such as Boltz-2, DrugCLIP, and DrugBAN remain near random () under their evaluated adaptation protocols. LogiCA instead aligns models through context-conditioned native token log-likelihoods, allowing a biological partner to reshape the original token distributions:
- Native vocabularies and probabilities remain intact, enabling heterogeneous BioFMs to interact without a shared decoder or tokenizer.
- Mutation scoring, Gibbs generation, and cross-modal dependency maps emerge from the same model, without separate task-specific heads or generative decoders.
- Contrastive supervision directly optimizes token likelihoods, discriminating subtle local changes while preserving the pretrained probabilistic interface.
This is not simply conditional MLM with a different loss. Matched LogiMLM and latent-contrastive controls isolate the contributions of contrastive supervision and native-logit scoring, showing that both matter, especially under scarce supervision.
2. Empirical results that guide future modeling
What excites us most is seeing this distinction translate into substantial improvements across biological settings, revealing when native-logit alignment matters most.
- Out-of-distribution clusters (LOCO-30%): LogiCA reaches top-1% enrichment, versus SiteFuse () and LogiMLM (), while leading in correlation (). With only 1% of held-in DMS supervision, LogiCA already exceeds LogiMLM trained on 100%.
- Unseen chemical scaffolds (LOLO): LogiCA achieves enrichment, versus SiteFuse () and LogiMLM (), demonstrating transfer beyond ligand scaffolds seen during adaptation.
- Clinical stress test (Oncology LOO): Where unconditional sequence models and evaluated latent baselines remain below random enrichment (), LogiCA achieves , demonstrating the value of drug-conditioned mutation scoring.
- Native sequence generation: Without an additional decoder, LogiCA supports iterative Gibbs design, reaching enrichment on LOCO-30% and on LOLO, versus SiteFuse (/) and LogiMLM (/). Beyond single mutations, on the TtgR combinatorial landscape, 15-step generation reaches measured fitness, versus for ESM-2 and for SiteFuse, without generative training.
These findings reveal an important pattern: although positive-only MLM can recover much of the global fitness landscape with sufficient data, contrastive alignment retains a substantial advantage in prioritizing rare, high-functioning variants. Moreover, the same token distributions support ranking, structural interpretation, and iterative design without separately trained downstream heads.
The broader takeaway we hope the AI Bio community takes from this work is simple: what we choose to align matters as much as how we align it. Aligning native token likelihoods rather than pooled representations provides a richer probabilistic interface connecting multimodal discrimination to mutation-level interpretation and sequence design. When the goal is not merely to classify interacting pairs, but to engineer biological sequences under specific molecular contexts, preserving and contrastively aligning native logits offers a distinct and practically valuable modeling strategy.
R3. Broader applications and future impact of LogiCA
There are databases like STRING for protein-protein interactions where the contrastive objective might be useful. Yes! Applying LogiCA to large-scale interactomes like STRING-DB is a natural and exciting direction, and extending our framework to protein–protein interactions (PPI) is very much on our active roadmap. Still, our deliberate priority in this submission was to first tackle what we view as the hardest conceptual testbed: coupling foundation models that speak entirely different biological languages.
Pairings with disjoint vocabularies and distinct tokenizers—such as ESM-2 with SELFormer (protein–chemical) and TCRLang with ESM-2 (immune repertoire–antigen)—are precisely where standard multimodal alignment fails most visibly. In those settings, collapsing representations into pooled scalar compatibility scores completely strips away token-level distributions, making conditional sequence generation impossible. Demonstrating that native logits can be preserved and directly aligned across fundamentally heterogeneous representations was the critical methodological bottleneck we set out to resolve.
By contrast, PPI operates in an ESM–ESM regime where both interacting partners share the same amino-acid vocabulary. While conceptually more homogeneous than cross-modal pairings, this setting unlocks distinct advantages that make it a compelling next step:
- Bidirectional interface co-design: Because both partners expose native amino acid logits, Gibbs sampling can operate symmetrically—mutations sampled on protein dynamically update the likelihood surface of binding partner , enabling joint, iterative optimization of complex interfaces.
- Complex-conditioned DMS ranking: It provides a path to predict mutation effects on binding affinity across the interactome directly from sequence pairs, bypassing the need for resolved 3D structures.
More broadly, this highlights what excites us most about LogiCA: it offers a general, modular recipe for converting standalone masked language models into context-conditioned design engines. The exact same formulation applies whenever a sequence must be optimized against external biological constraints:
- Single-cell & Perturbation Biology: Coupling masked gene models (e.g., GenePT, scGPT) with chemical encoders to model drug-induced transcriptional shifts and predict perturbation responses at single-gene resolution.
- RNA–Protein Recognition: Conditioning RNA language models on RNA-binding proteins to design synthetic untranslated regions (UTRs), guide RNAs, or translation-regulating motifs.
- Antibody–Antigen Affinity Maturation: Contextualizing antibody CDR loops directly against target antigen sequences for guided in-silico optimization.
We share the reviewer's vision for this broader impact. Aligning native token probabilities rather than discarding them for scalar scores transforms pretrained biological foundation models into generative, context-aware design systems, and we are eager to see how the community builds on this foundation.
R1. New quantitative evidence of context-conditioned sequence generation
We thank the reviewer for this constructive and decisive feedback. We agree that while iterative Gibbs sampling was noted as an inherent capability of LogiCA in the initial submission, its empirical validation was not demonstrated adequatly. In response, we have carried out a comprehensive set of generative experiments across both protein–ligand and TCR–peptide systems, demonstrating that context-conditioned sequence design emerges directly from LogiCA's preserved logits without auxiliary decoders or generative training.
1. Benchmark design and generalization regimes
Crucially, Gibbs sampling in LogiCA performs iterative multi-step sequence design, not merely single-point mutation ranking. Starting from a query sequence, LogiCA samples substitutions from partner-conditioned token distributions and accepts updates guided by local likelihood scores, navigating combinatorial fitness landscapes directly.
To benchmark this systematically beyond our original oncology resistance screens, we curated 103 protein–ligand deep mutational scanning (DMS) datasets totaling 530,565 measured variants across three evaluation regimes:
2. Generative enrichment results
Generated sequences were evaluated retrospectively against experimental fitness landscapes, measuring top-10% enrichment over random sampling:
Values report top-10% enrichment relative to random baseline (1.0×). On LOCO-30%, LogiCA's margin over LogiMLM is statistically significant under gene-collapsed paired testing (p=1.55×10−3).
3. Multi-mutant rescue on combinatorial landscapes (TtgR)
To test whether LogiCA can navigate combinatorial fitness landscapes rather than merely prioritize individual substitutions, we evaluated 15-step Gibbs trajectories on TtgR (15,216 experimentally characterized variants, up to four simultaneous mutations). Starting from deleterious multi-mutants, LogiCA reaches +1.289σ experimental fitness and 1.92× top-10% enrichment, compared with +0.674σ / 1.56× for ESM-2 and +0.184σ / 1.15× for SiteFuse.
Together, these results confirm that LogiCA’s logit preservation provides a faithful, partner-conditioned energy surface capable of directed combinatorial sequence optimization, fulfilling the generative promise of the architecture without task-specific retraining.
We have now integrated all these new results into revised Sections 2.5 and 3.1–3.2, Tables 2–3, and expanded Appendix Sections D–E. accordingly