acceptodds
Under review as a conference paper at ICLR 2027

PPI2Text: Captioning Protein-Protein Interactions with Coordinate-Aligned Pair-Map Decoding

Abstract

Protein-protein interaction (PPI) modeling has been widely studied as a binary or multi-label classification task. While emerging multimodal large language models (LLMs) can now describe single proteins, they remain unable to generate free-form descriptions of interactions between protein pairs. Moving beyond controlled vocabulary annotations, we propose to model PPI using free-text description, enabling richer expressiveness, improved interpretability, and better integration with literature knowledge base. We present PPI2Text, a multimodal LLM for free-form PPI captioning from amino acid sequences, that encodes each protein using ESM3 encoder, constructs a pair map from the two representations to capture interactions across all residue pairs, and autoregressively generates descriptions using a Qwen3 language decoder. We further introduce PaCo-RoPE, a coordinate-aligned positional encoding that aligns each axis of the pair grid with the residue positions of the corresponding protein. In addition, we release PPI2Text-Dataset, a 351k-pair corpus of free-form PPI descriptions aggregated from ten curated biological databases and further synthesized with Gemini under evidence-tiered prompting. PPI2Text consistently outperforms strong baselines across multiple ablation settings and evaluation protocols. It not only achieves higher scores on linguistic metrics against synthesized references, but also excels on factuality metrics, where an LLM-based judge evaluates outputs against raw biological evidence.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein-Protein Interaction, Multimodal Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_wjCjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PPI2Text, a multimodal language model for generating free-form descriptions of protein-protein interactions from amino acid sequences. The method uses ESM3 to encode individual proteins, constructs a residue-level pair map to capture interactions between protein representations, and introduces PaCo-RoPE to align the positional information of the two proteins and their pairwise representations in the language decoder. The authors also construct PPI2Text-Dataset, containing 351K protein pairs with descriptions synthesized from curated biological evidence. Experiments on temporal holdout and C3-hard splits show improvements over the compared baselines in both text generation and evidence-based factuality metrics.

Critical strengths
  1. The paper addresses a meaningful problem by extending conventional PPI classification to free-form interaction descriptions. The proposed architecture is well motivated, particularly the pairwise residue modeling and PaCo-RoPE, which go beyond simply combining individual protein embeddings.

  2. The construction of a large-scale PPI description dataset is a valuable contribution. The integration of evidence from multiple biological databases, together with temporal holdout and C3-hard evaluations, provides a reasonably comprehensive experimental setup. The ablation studies also support the effectiveness of the proposed components.

Critical weaknesses
  1. The effectiveness of the pair-map architecture is not sufficiently isolated or validated. The paper does not examine whether the learned representations correspond to actual residue contacts or interaction interfaces. Moreover, there seems to be no baseline using the same ESM3 backbone that feeds the cross-attended 1D representations directly to the decoder without constructing the 2D pair map, making it difficult to attribute the improvements specifically to pair-map modeling and PaCo-RoPE.
  2. The reliability and practical applicability of the generated descriptions require further investigation. Although the pipeline uses evidence-tiered control tags to constrain the labeler LLM, the training data still relies on LLM-synthesized descriptions that may contain hallucinated biological details, while the factuality evaluation primarily uses an LLM judge validated on only 50 protein pairs. Additionally, the assumption that all input protein pairs interact limits the framework's applicability to real-world settings.
Questions for authors
  1. Can the authors provide evidence that the learned pair-map representations correspond to actual residue-level interaction patterns? For example, could they evaluate the pair-map features against known protein interaction interfaces or contact maps?

  2. How does PPI2Text behave when given a pair of proteins that do not interact? Does it still generate a confident interaction description, or can it express uncertainty about the existence of an interaction?

  3. Could the authors provide a more extensive human expert evaluation of the generated descriptions, particularly regarding mechanistic correctness and hallucination?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer wjCj (1/3)

Dear Reviewer:

We sincerely thank you for the positive and constructive assessment of our work. We are very much encouraged by your recognition of the well-motivated architecture, the value of our large-scale dataset, and the comprehensive experimental evaluation supporting our proposed components. We greatly appreciate your insightful suggestions, which provide valuable directions for further improving our work. Below, we address each of the weaknesses (W) and questions (Q) raised in your review in detail.

W1 & Q1 (pair map effectiveness)

We would like to clarify that the pair map is a learned feature grid, and is not guaranteed to be a strict "contact map". It is trained only through the captioning loss, and has no supervision on interface positions. Nevertheless, to measure how much interface information the pair map actually carries, we trained a linear classifier on top of each kind of pair map representation to predict the interface location between two proteins:

FeatureAUROC
Pair map0.741
Pair map after 32x32 pooling0.726

The result suggests that the learned pair map indeed encodes coarse-grained interface information even though it was never supervised to do so, and that pooling to 32x32 preserves most of the information. We will make this clearer in the paper. Adding an explicit interface supervision signal to the pair map could be a promising direction for future work.

We also conducted the ablations as the reviewer suggested to further demonstrate the contribution of the pair map architecture:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
SingProt-Only (XA,XBX_A,X_B)43.3527.6158.4431.8836.0089.5483.714.758.262.615.21
SingProt-Only (HA,HBH_A,H_B)46.4329.6460.1634.7337.1390.1185.806.528.203.646.12
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

HA,HBH_A,H_B are the outputs of the bidirectional cross-attention module, as shown in our Figure 2

Using the output of cross-attention instead of the compressed output of ESM3 does improve the metrics, but the pair map still provides substantially better performance with our PaCo-RoPE design.

We also ablated our PaCo-RoPE design:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
1D-RoPE PPI2Text39.1723.6154.6327.4532.5088.5281.663.007.312.014.11
Unaligned 2D-RoPE PPI2Text46.9230.4460.6834.3236.7589.7485.236.406.973.855.74
Blockwise PaCo-RoPE PPI2Text47.8031.6061.4935.3837.5290.5986.367.358.075.727.04
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

The results demonstrate that both coordinate alignment and interleaved frequency allocation are important design choices in our PaCo-RoPE.

W2.1 (hallucination risk in synthesis)

We agree with the reviewer that, despite the evidence-tiered filtering and constraints applied in our data synthesis pipeline, the risk of hallucinations cannot be entirely ignored. To assess such risk, we conducted a new round of human auditing on the synthesized text, where every claim or statement in the synthesized descriptions was individually cross-checked against the sources. 60 PPIs were sampled and examined, with about 14 claims in each description on average. We found that 82% of the statements were directly supported by at least one source, whereas 17% of the claims were biologically plausible generic inferences made by Gemini based on the available evidence. Most importantly, we found only 2 claims (0.2%) that contradict either the source evidence or established biological principles. This further suggests that, with well-designed instruction prompting and comprehensive evidence sources, the synthesis pipeline minimizes factual hallucinations and greatly reduces the risk of introducing misleading information into our model training.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer wjCj (2/3)

[CONTINUED]

W2.2 & Q3 (extended human evaluation)

Following the reviewer's suggestion, we conducted a claim-level human audit of the descriptions generated for 59 C3-hard pairs, cross-checking each claim against the raw evidence in a model-blinded way. We distinguished between two categories of claims: (1) sequence-inferable (SI) claims, which could in principle be inferred from protein sequences (e.g., family, domain, function, localization, and interaction type), and (2) non-sequence-inferable (nSI) claims, which involve literature-specific knowledge not directly recoverable from sequences alone (e.g., named complexes, diseases, drugs, and other specific proteins).

C3-hard, 59 pairsSI (wrong) ↓nSI (supported) ↑nSI (contradicted) ↓Mec.Hal. ↓Certainty vs. evidence (ρ) ↑
Retrieval (ESM3)31%20%17%46%0.18
Qwen3-32B, 3-shot, seq only59%2%27%85%0.09
PPI2Text21%40%12%16%0.43

SI: sequence-inferable; nSI: non-sequence-inferable; Mec.Hal.: rate of unsupported mechanism claims, where the model asserts a mechanism absent from the source evidence.

PPI2Text consistently outperforms both baselines in factually grounded and uncertainty-aware generation. It achieves the lowest error rate on sequence-inferable claims (21% vs. 31% and 59%) while producing fewer hallucinations on literature-specific claims (12% vs. 17% and 27%). More importantly, its mechanism hallucination rate is only 16%, compared with 46% for retrieval and 85% for Qwen3-32B. Furthermore, the certainty expressed in its descriptions exhibits a substantially stronger correlation with the underlying evidence strength (ρ = 0.43 vs. 0.18 and 0.09), suggesting that PPI2Text better adjusts the confidence of its language to the available evidence rather than presenting uncertain hypotheses as established facts.

Nevertheless, hallucinations are not entirely eliminated: 12% of PPI2Text's literature-specific claims contradict the source evidence, corresponding to an average of 0.47 hallucinated claims per description. We acknowledge this remaining limitation and will incorporate both the new evaluation and a discussion of these findings into the revised manuscript.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer wjCj (3/3)

[CONTINUED]

W2.3 & Q2 (non-interacting inputs and uncertainty)

We would like to clarify that, as a deliberate simplification of data preparation and model training, PPI2Text is trained to assume that the input protein pair interacts, rather than to predict whether an interaction exists. This is an acknowledged limitation as we already discussed in Section 6. Our engineering choice is mainly motivated by the fact that binary PPI prediction (whether two proteins will interact) is a well-studied task, and recent state-of-the-art models reported 95%-99% accuracy [1-3]. Therefore, in practical real-world settings, PPI2Text can be integrated into a two-stage pipeline, where a binary PPI prediction model first screens input protein pairs for potential interactions, and PPI2Text then generates detailed descriptions for pairs predicted to interact. This allows PPI2Text to focus on its main objective of characterizing interactions and to leverage existing specialized models for interaction pre-screening.

However, this does not mean that PPI2Text is unable to express uncertainty when encountering unfamiliar or weakly supported protein pairs. Our data synthesis pipeline explicitly enforced four evidence-tiered labels (length, verb, mechanism and silence) to control the level of confidence strength and expressiveness of the synthesized descriptions. Since PPI2Text is trained on such data with varying levels of evidence support and confidence, it can learn to adapt its language and express uncertainty when encountering weakly supported cases. Following the reviewer's question, we sampled 300 non-interacting pairs from Negatome 2.0 [4] plus 300 random pairs of C3-hard proteins with no interaction recorded in IntAct or STRING. We fed these pairs to PPI2Text and analyzed the generated descriptions:

Non-interacting pairsInteracting pairs
Asserts a direct physical interaction7.8%37.7%
Opens with hedged wording95.7%60.7%
Raises location mismatch40.8%6.3%

We observed a significant shift toward more cautious and less assertive claims for those non-interacting inputs. Notably, PPI2Text raised potential subcellular location mismatches much more often, suggesting that the model can recognize concrete biological constraints and adapt its wording accordingly. Rather than uniformly asserting interactions with full confidence, PPI2Text can adapt the strength and specificity of its descriptions, and highlight potential biological limitations when definitive conclusions cannot be reached. This is particularly relevant to real-world applications, where many candidate PPIs are poorly characterized and their interaction mechanisms remain unknown. These observations suggest that, with its confidence-aware training, PPI2Text retains its ability to communicate uncertainty, rather than presenting unsupported details as established facts.

Again, we sincerely appreciate the reviewer's constructive feedback, which has helped us improve the clarity and completeness of our work. We hope that our detailed responses and additional evidence adequately address your concerns and further demonstrate the effectiveness and potential of PPI2Text. We will reflect these improvements in the revised paper.


[1] Dang, Thanh Hai, and Tien Anh Vu. "xCAPT5: protein–protein interaction prediction using deep and wide multi-kernel pooling convolutional neural networks with protein language model." BMC bioinformatics 25.1 (2024): 106.

[2] Hu, An, Linai Kuang, and Dinghai Yang. "LPBERT: a protein–protein interaction prediction method based on a pre-trained language model." Applied Sciences 15.6 (2025): 3283.

[3] Zhou, Senyu, et al. "Predicting protein–protein interactions in microbes associated with cardiovascular diseases using deep denoising autoencoders and evolutionary information." Frontiers in Pharmacology 16 (2025): 1565860.

[4] Blohm, Philipp, et al. "Negatome 2.0: a database of non-interacting proteins derived by literature mining, manual annotation and protein structure analysis." Nucleic acids research 42.D1 (2014): D396-D400.

Reviewer_p4LAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PPI2Text generates free-form descriptions of protein–protein interactions (PPIs). The paper constructs a 351k-pair corpus from ten biological databases, using evidence-based filtering and Gemini to synthesize descriptions. The model combines frozen ESM3 features, sequence compression, bidirectional cross-attention, and a pooled pair map with a LoRA-adapted Qwen3 decoder. PaCo-RoPE encodes positional relationships between the two protein streams and the pair map. Evaluation includes temporal and protein-decontaminated splits, text metrics, and an evidence-based LLM judge with a small human validation study.

Critical strengths
  1. A useful task and substantial dataset. Free-form descriptions can express interaction mechanisms, regulatory direction, complex membership, and uncertainty beyond binary labels. The synthesis pipeline documents source versions and licenses and constrains descriptions by evidence strength and mechanism availability. The resulting corpus could be valuable independently of the proposed model.
  2. Consistent gains across evaluation settings. The full model improves both text metrics and evidence-based judge scores over the reported baselines and ablations. Its average judge score is 7.21 on the temporal holdout, compared with 5.86 for No-Cross and 5.21 for SingProt-Only, and it also leads on C3-hard with 6.31. The protein-decontaminated evaluation and small human agreement study strengthen the evidence for usefulness within this benchmark.
Critical weaknesses
  1. The evaluation does not establish prospective, context-specific generalization.
    • Although the PPI pairs used for evaluation are strictly split by a temporal cutoff (May 1, 2025), the evidence sources used to construct the descriptions include 2026 releases and live services. As a result, the temporal split applies only to the pairs, not to the evidence: descriptions of older training pairs may still incorporate findings published after the cutoff, and it is unclear whether training descriptions, filtering scores, and control tags rely solely on pre-cutoff evidence. The temporal holdout therefore does not rule out information leakage from the future.
    • In addition, the targets contain conditions, cofactors, modifications, and evidence-dependent certainty that are not supplied to the predictor, so it remains unclear whether these details are inferred for new interactions or merely learned as typical descriptions. The human agreement study covers only PPI2Text outputs, and the handling of NA mechanism scores is unspecified, which further weakens the interpretation of model rankings.
  2. Coordinate alignment and fine-grained interaction modeling are insufficiently validated.
    • Appendix D.2 defines an already-compressed token index in Eq. (18), then divides it by the stride again in Eq. (19). Pair-map coordinates use the compressed-length scale, while Eq. (23) bounds the distance from the token index rather than its assigned coordinate.
    • The 1D-RoPE ablation supports the complete PaCo-RoPE design, but does not separate the benefits of retaining two-dimensional structure, aligning coordinates across representations, and interleaving frequency channels. Moreover, the pair map is a learned feature grid, not a validated contact map. Average pooling can dilute sparse signals and remove within-patch positions; coordinate encoding cannot recover that lost information. Caption-level gains alone therefore do not establish precise interface localization or mechanistic interpretability.
Questions for authors
  1. What is the accuracy–cost trade-off of reducing sequence length by a factor of 4?
  2. How sensitive are the results to the fixed 32 × 32 grid and average pooling? Would a higher-resolution grid or learned pooling preserve sparse interaction signals better, and is there evidence that the pooled features retain interface-specific information?
  3. Please reconcile the compressed index in Eq. (18) with the additional stride division in Eq. (19). Beyond the existing 1D-RoPE ablation, can an unaligned 2D-RoPE control and an interleaved-versus-blockwise comparison distinguish coordinate alignment from two-dimensional encoding and frequency allocation?
  4. Were all evidence fields used to construct training descriptions and filtering decisions restricted to information available before May 1, 2025? If not, how should the temporal experiment's prospective interpretation be qualified?
  5. How are NA mechanism scores aggregated, and would blinded human assessment of both full-model and baseline outputs reproduce the judge's ranking?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer p4LA (1/3)

Dear Reviewer:

We sincerely thank you for the thoughtful and constructive evaluation of our work. We appreciate very much your recognition of the significance of our PPI captioning task, the value of our large-scale dataset, and the consistent improvements achieved across different evaluation settings. Your insightful comments and suggestions have helped us further strengthen our work. We address each of the weaknesses (W) and questions (Q) raised in your review in detail below.

W1.1 & Q4 (temporal cutoff)

We thank the reviewer for pointing out that all PPI descriptions come from the same 2026 snapshot of sources. The "temporal holdout" (May 2025) is a guarantee on the PPI pairs. We further examined the actual leakage due to this simplification in the data pipeline:

Leakage levelTemporal holdoutC3-hard
Single protein97.8%0%
Pair info15.6%0%
PubMed item0.1%0%
Protein Pair0%0%

pair-info-level leakage means a protein seen in training contains at least one source annotation about the PPI evaluated in test

Moreover, PPI2Text performs slightly below average on those pair-info-leaked PPIs (BLEU-2 46.95 vs 48.33). This suggests that the leaked information has not been incorporated into the synthesized description of the corresponding training PPIs (only 1 out of 50 cases we manually examined), possibly because the data synthesis pipeline considered it irrelevant to the training PPI being described.

Thus, PPI2Text is not able to see or memorize much leaked information from synthesized descriptions in training, but is instead forced to learn from patterns in the training data and to draw inspiration from the generic biological knowledge of the pretrained Qwen3 model (knowledge cutoff March 2025).

In any case, we constructed C3-hard as a strictly guaranteed test split with zero overlap, and we honestly reported the difference between these two splits.

W1.2 (detailed or generic descriptions)

We would like to clarify that PPI2Text is designed to predict specific details rather than generic descriptions.

First, our data synthesis is instructed to cover details instead of making broad statements. As a consequence, linguistic metrics against synthesized text in Table 1 faithfully reflect the level of detail and punish broad predictions.

Second, we introduced two new retrieval baselines based on MMseqs2 and ESM3 embeddings:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
Retrieval (MMseqs2)34.2517.7354.1625.0227.3387.5581.503.174.302.223.23
Retrieval (ESM3)36.4418.9655.0325.1228.1287.8682.013.555.182.823.85
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

PPI2Text outperforms both retrieval baselines, indicating that retrieving descriptions from similar PPIs is insufficient, which shows the importance of predicting information specific to the input pair.

Third, we conducted a counterfactual input-perturbation test. For each test PPI, we replaced one protein with its most similar counterpart in the training set, and measured the change in perplexity on different parts of the reference description:

PartsHoldoutC3-hard
Modifications+140%+152%
Cofactors+201%+239%
Conditions+73%+83%
Certainty+12%+18%
All other words+67%+81%

Replacing one input protein substantially increases the perplexity of the original reference description, indicating that PPI2Text output depends strongly on the specific protein pair. Notably, the increase is particularly significant for aspects requiring biological details (modifications and cofactors).

In summary, both our PPI2Text-Dataset and PPI2Text model cover detailed information rather than broad descriptions.

W1.3 & Q5 (NA handling and human study)

The judge uses NA only when the evidence has no mechanism and the prediction does not invent one. These cases are excluded from the average. However, if the prediction invents a fake mechanism, the score is forced to "LOW". We will add this detail to the paper.

We would also like to explain that our human evaluation study in Table 2 validates the LLM-judge as a measuring instrument, rather than any particular model. Both the LLM-judge and human experts are blind to the model, and only see the evidence with a prediction. Furthermore, all baselines and ablations use the same decoder and the same training targets, so they all share the same output format and style. The only difference is how well the prediction matches the evidence, which is exactly what our Table 2 validates.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer p4LA (2/3)

[CONTINUED]

W2.1 & Q3 (correction to Eq. 19)

Thank you for catching this! In Eq. (18), rr is already the index of a compressed token r=⌊i/s⌋r=\lfloor i/s \rfloor, so Eq. (19) should read pθ(r)=rp^\theta(r) = r. With this correction, the coordinate assigned to a protein token is rr itself, so Eq. (23) bounds the distance between the two assigned coordinates. Our code implementation is correct, so all the results stand. We will correct Appendix D.2.

W2.2 & Q3 (PaCo-RoPE vs 2D-RoPE)

We agree with the reviewer and conducted the ablations as suggested:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
1D-RoPE PPI2Text39.1723.6154.6327.4532.5088.5281.663.007.312.014.11
Unaligned 2D-RoPE PPI2Text46.9230.4460.6834.3236.7589.7485.236.406.973.855.74
Blockwise PaCo-RoPE PPI2Text47.8031.6061.4935.3837.5290.5986.367.358.075.727.04
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

The results demonstrate that both coordinate alignment and interleaved frequency allocation are important design choices in our PaCo-RoPE.

W2.2 & Q2 (grid size and average pooling)

We thank the reviewer for raising questions on grid size and pooling, and we conducted two ablations as suggested:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
SingProt-Only PPI2Text43.3527.6158.4431.8836.0089.5483.714.758.262.615.21
PPI2Text (8x8)47.3830.8363.2235.6038.1990.6186.087.258.474.916.88
PPI2Text (32x32)48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

The results show that a smaller grid degrades most metrics. We chose not to go beyond 32x32 because of the computational cost.

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
PPI2Text + Attention Pooling48.1331.5263.6636.3138.7790.7086.197.348.485.667.16
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21

We found that replacing our adaptive average pooling with learnable pooling strategies such as attention pooling does not improve the performance consistently.

We also agree with the reviewer that the pair map is a learned feature grid, and is not guaranteed to be a strict "contact map". It is trained only through the captioning loss, and has no supervision on interface positions. Nevertheless, to measure how much interface information the pair map actually carries, we trained a linear classifier on top of each kind of pair map representation to predict the interface location between two proteins:

FeatureAUROC
Pair map0.741
Pair map after 32x32 pooling0.726

The result suggests that the learned pair map indeed encodes coarse-grained interface information even though it was never supervised to do so, and that pooling to 32x32 preserves most of the information. We will make this clearer in the paper. Adding an explicit interface supervision signal to the pair map could be a promising direction for future work.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer p4LA (3/3)

[CONTINUED]

Q1 (compression factor)

The compression factor of 4 is a compromise between training efficiency and performance. As stated in Appendix E, the intuition behind a 4x compression comes from the common resolution of alpha-helix ss in protein sequences. We also compared different compression factors:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvgTpSGPU-M
PPI2Text (2x compr.)48.5331.8662.9636.4838.9290.7886.297.628.045.917.191.8228.4
PPI2Text (4x compr.)48.3331.8662.1536.5639.9690.7986.337.728.325.597.211.1921.5
PPI2Text (8x compr.)47.5331.0362.1335.8139.0089.8185.357.668.025.407.030.9419.0

TpS: training time per step (seconds); GPU-M: peak GPU memory consumption (GB); both measured on NVIDIA RTX A6000 48GB

The results demonstrate that a higher compression factor harms the performance, whereas reducing it does not bring consistent improvements. Considering the training and inference efficiency, we selected the factor of 4 as a balanced trade-off.

After all, we thank the reviewer again for the valuable feedback. We believe that our clarifications and additional analyses help address the concerns raised, strengthen the empirical evidence for our approach, and clarify the scope of our contributions. We will incorporate these improvements into the revised manuscript.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer GG5H (1/6)

Dear Reviewer:

We sincerely thank you for the exceptionally detailed and thoughtful assessment of our work. We particularly appreciate your recognition of the novelty of the PPI captioning task, the value of our carefully constructed dataset, the elegance of our proposed architecture, and the rigor and transparency of our evaluation design. We also greatly value your critical observations regarding factual reliability and generalization, which have motivated us to conduct further analyses and better clarify the capabilities and limitations of our approach. We address each of the weaknesses (W) and questions (Q) raised in your review in detail below.

W1.1 (self-referential evaluation)

We agree with the reviewer that the reference descriptions are synthesized by Gemini from the aggregated evidence and that PPI2Text is trained on them, so the first link of the loop to verify is whether the synthesized text stays faithful to the sources. To this end, we conducted a new round of human auditing on the synthesized text, where every claim or statement in the synthesized descriptions was individually cross-checked against the sources. 60 PPIs were sampled and examined, with about 14 claims in each description on average. We found that 82% of the statements were directly supported by at least one source, whereas 17% of the claims were biologically plausible generic inferences made by Gemini based on the available evidence. Most importantly, we found only 2 claims (0.2%) that contradict either the source evidence or established biological principles. This further suggests that, with well-designed instruction prompting and comprehensive evidence sources, the synthesis pipeline minimizes factual hallucinations and greatly reduces the risk of introducing misleading information into our model training.

We would like to clarify that both our synthesized-data auditing and human evaluation of the LLM-judge are specifically designed to break the self-referential loop. We agree that PPI2Text, since it is fine-tuned on Gemini-synthesized descriptions, may inherit certain linguistic styles from Gemini. This is precisely why our evaluation in Table 1 compares against not only Gemini-generated reference descriptions but also raw source evidence independently as the ground truth. Similarly, our human evaluation independently verifies generated claims against the raw evidence. This design ensures that evaluation is based on factual consistency with external evidence rather than stylistic similarity to the training data, mitigating the risk of self-referential evaluation bias.

Regarding the scope of the knowledge sources, we acknowledge that PPI2Text's knowledge is primarily bounded by the ten sources integrated into our data synthesis pipeline. However, these sources were deliberately selected for their complementary coverage of different aspects of protein–protein interactions, including protein functions, structural properties, biological processes, subcellular localization, and interaction mechanisms. Together, they provide a broad and multifaceted evidence base for generating informative PPI descriptions. While we do not claim that these sources exhaust all available biological knowledge, we argue that their combined coverage is sufficiently comprehensive to support the key aspects of PPI characterization targeted by our task.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer GG5H (2/6)

[CONTINUED]

W1.2 & Q2 & Q3 (extended human evaluation)

Following the reviewer's suggestion, we conducted a claim-level human audit of the descriptions generated for 59 C3-hard pairs, cross-checking each claim against the raw evidence in a model-blinded way. We distinguished between two categories of claims: (1) sequence-inferable (SI) claims, which could in principle be inferred from protein sequences (e.g., family, domain, function, localization, and interaction type), and (2) non-sequence-inferable (nSI) claims, which involve literature-specific knowledge not directly recoverable from sequences alone (e.g., named complexes, diseases, drugs, and other specific proteins).

C3-hard, 59 pairsSI (wrong) ↓nSI (supported) ↑nSI (contradicted) ↓Mec.Hal. ↓Certainty vs. evidence (ρ) ↑
Retrieval (ESM3)31%20%17%46%0.18
Qwen3-32B, 3-shot, seq only59%2%27%85%0.09
PPI2Text21%40%12%16%0.43

SI: sequence-inferable; nSI: non-sequence-inferable; Mec.Hal.: rate of unsupported mechanism claims, where the model asserts a mechanism absent from the source evidence.

PPI2Text consistently outperforms both baselines in factually grounded and uncertainty-aware generation. It achieves the lowest error rate on sequence-inferable claims (21% vs. 31% and 59%) while producing fewer hallucinations on literature-specific claims (12% vs. 17% and 27%). More importantly, its mechanism hallucination rate is only 16%, compared with 46% for retrieval and 85% for Qwen3-32B. Furthermore, the certainty expressed in its descriptions exhibits a substantially stronger correlation with the underlying evidence strength (ρ = 0.43 vs. 0.18 and 0.09), suggesting that PPI2Text better adjusts the confidence of its language to the available evidence rather than presenting uncertain hypotheses as established facts.

Nevertheless, hallucinations are not entirely eliminated: 12% of PPI2Text's literature-specific claims contradict the source evidence, corresponding to an average of 0.47 hallucinated claims per description. We acknowledge this remaining limitation and will incorporate both the new evaluation and a discussion of these findings into the revised manuscript.

We would also like to explain that our human evaluation study in Table 2 validates the LLM-judge as a measuring instrument, rather than any particular model. Both the LLM-judge and human experts are blind to the model, and only see the evidence with a prediction. Furthermore, all baselines and ablations use the same decoder and the same training targets, so they all share the same output format and style. The only difference is how well the prediction matches the evidence, which is exactly what our Table 2 validates.

W1.3 & W2 (judge axes and Entity-axis drop)

We would like to clarify that the three evaluation axes are designed to capture distinct and complementary aspects of PPI captioning. The interaction axis appears relatively flat only on the temporal holdout split, where the model can make use of information about proteins already encountered during training. In contrast, on the C3-hard split, where test proteins are unseen during training, the interaction axis shows substantially greater discrimination among different models and architectural variants, highlighting its importance for evaluating generalization to unseen proteins.

The entity axis dropped notably for PPI2Text from temporal holdout to C3-hard also because of this fact, while the interaction and mechanism axes remain relatively unaffected. This suggests that PPI2Text's ability to characterize protein interactions is not solely dependent on recognizing the identities of individual proteins, highlighting the robustness of its learned interaction representations.

We agree with the reviewer that the mechanism axis consistently yields the lowest scores across nearly all evaluated models and ablations, indicating that mechanistic characterization remains the most challenging aspect of free-text PPI description generation. Importantly, however, PPI2Text achieves its largest relative improvement over the strongest baseline (MINT+Qwen3) on this axis on both splits (5.59 vs. 1.63 on the temporal holdout and 5.33 vs. 1.50 on C3-hard), suggesting that our architectural design is particularly effective at addressing this challenging aspect of the task. Rather than indicating a weakness in the evaluation framework, these results highlight the complementary roles of the three axes in capturing different levels of difficulty and revealing meaningful differences in model capabilities.

[CONTINUED BELOW]

AuthorsOpenReview ↗

Author Response to Reviewer GG5H (3/6)

[CONTINUED]

W1.4 (prediction or database-to-text reconstruction)

We agree with the reviewer that our evaluation measures consistency with curated biological evidence rather than direct experimental validation, and we acknowledge this distinction as a limitation. However, we would like to clarify that our task goes beyond database-to-text reconstruction. PPI2Text takes only two amino-acid sequences as input, without access to any evidence associated with the test pair. Moreover, on the C3-hard split, no proteins homologous to the test proteins appear in training. Therefore, the model must infer interaction-related information from the input sequences and its learned representations, rather than retrieve or reproduce previously seen descriptions. Achieving agreement with the curated evidence under these conditions demonstrates the model's ability to generalize to unseen proteins.

We also emphasize that our factuality evaluation is grounded in raw evidence from curated biological databases, not Gemini-synthesized reference descriptions. The LLM-judge evaluates generated claims directly against the original source evidence, without access to the Gemini-generated training labels. This separation is specifically designed to mitigate potential self-referential evaluation bias.

Furthermore, several of our new experiments provide complementary evidence beyond LLM-based judgments, including an expert claim-level human audit on C3-hard, comparisons with retrieval baselines, a perturbation test, and a linear probing experiment. Notably, the linear probe achieves an AUROC of 0.741 in predicting interaction interface locations from the learned pair map, despite the absence of explicit interface supervision during training. This provides additional evidence that PPI2Text learns biologically meaningful interaction representations, rather than merely reproducing textual patterns from the synthesized training data.

Nevertheless, we agree that consistency with curated evidence should not be considered experimentally established biological correctness. To make the scope of our claims more precise, we will revise the manuscript to state explicitly that our factuality scores measure consistency with curated evidence, and we will expand the limitation on the lack of wet-lab validation that is already noted in Section 6.

W2.1 & Q1 (memorization vs prediction)

We introduced two new retrieval baselines based on MMseqs2 and ESM3 embeddings:

HoldoutB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
Retrieval (MMseqs2)34.2517.7354.1625.0227.3387.5581.503.174.302.223.23
Retrieval (ESM3)36.4418.9655.0325.1228.1287.8682.013.555.182.823.85
PPI2Text48.3331.8662.1536.5639.9690.7986.337.728.325.597.21
C3-hardB-2B-4R-1R-2R-LRBTBBTEntIntMecAvg
Retrieval (MMseqs2)34.4315.8351.4821.2525.3886.5080.053.685.413.744.28
Retrieval (ESM3)34.6416.0551.8021.6225.4986.5679.863.115.583.804.16
PPI2Text37.2620.5755.0225.4829.7488.2382.575.737.885.336.31

PPI2Text outperforms both retrieval baselines, indicating that retrieving descriptions from similar PPIs is insufficient, which shows the importance of predicting information specific to the input pair.

We also conducted a counterfactual input-perturbation test. For each test PPI, we replaced one protein with its most similar counterpart in the training set, and measured the change in perplexity on different parts of the reference description:

PartsHoldoutC3-hard
Modifications+140%+152%
Cofactors+201%+239%
Conditions+73%+83%
Certainty+12%+18%
All other words+67%+81%

Replacing one input protein substantially increases the perplexity of the original reference description, indicating that PPI2Text output depends strongly on the specific protein pair. Notably, the increase is particularly significant for aspects requiring biological details (modifications and cofactors).

[CONTINUED BELOW]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.