- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates whether including or excluding certain features derived from the representation spaces of different models can enhance performance or adversely affect predictions.
The authors frame their study within partner-conditioned predictions and analyze the effects of adding or removing frozen components in data representation pipelines.
The first setting and task presented involves producing pair scores and separate antigen- and antibody-node probabilities to predict which residues participate in antibody-antigen interactions and which residue pairs are in contact.
The authors quantify the effect of additional features on the correct residues, the effect of shuffling features across residues, and the correspondence sensitivity.
Subsequently, a retraining experiment is conducted, examining both training and evaluation with shuffled features, as well as training with aligned features while validating on shuffled ones.
The evaluation considers the sensitivity measure—a relative comparison between aligned and shuffled features when applied to the correct interaction residues—focusing specifically on whether additional structural features improve performance.
The analysis of ESM-IF1 and ProteinMPNN yields inconclusive generalization results, as the technique is highly beneficial in the former case but provides only negligible improvements in the latter.
Further analysis compares aligned versus shuffled features to measure the difference in performance advantage. Here too, the results are not entirely conclusive; while the advantage was substantial during development, it failed to generalize to the test set.
The final experimental setting considers shuffling features across all residues rather than just the key interacting ones. In this scenario, the models consistently degrade in performance.
The authors then provide insights into potentially adding a contact loss and a partner-ranking loss in antibody-antigen prediction. The results indicate that supervision of the actual contact points between chains is the primary driver of improvement, whereas the ranking component harms model performance. Even with scheduling or gradient inspections, the authors do not clearly address this issue.
Critical strengths
Overall, the paper is clear, and the underlying idea is interesting. Given the increasing prevalence of complex architectures that merge intermediate representations from different networks—often without a clear understanding of each component's contribution—this is an important question to address. The experimental section is thorough and includes training losses. In the discussion, the authors also analyze limitations and offer observations on the method.
Critical weaknesses
The methodology lacks clear overall organization. The authors' intent is difficult to follow, and the manuscript lacks a cohesive narrative that would better motivate and frame the underlying problem. Some sections are oversimplified, lacking formal notation as well as helpful tables or visual representations that would render the analysis clearer. The figures are uninformative and lack clarity, even when reading the captions.
The authors should include additional benchmarks and incorporate more models into the analysis to better validate the method's generalization capabilities. Moreover, the gains obtained by applying the method across the evaluated cases are not consistently substantial. This highlights a lack of compelling evidence for the method's effectiveness and an absence of analysis explaining the reasons behind the reported results.
Questions for authors
a. What is the motivation that led the authors to pick this specific task? The evidence is fairly limited due to sticking with a single task and a small number of chosen models.
b. For the training procedure, there is a lack of continuous ablation of the loss component weights. The absence of such interpolation greatly limits the conclusions that can be drawn regarding the proposed losses.
c. The analysis regarding the insertion of regularization later in training is not sufficiently formalized or sound. It lacks clear criteria for insertion timing as well as ablation of the approach.
d. The analysis of the gradients lacks an explanation of the underlying intuition behind this approach.
e. The related work section lacks clarity; it could be expanded and revised to include more baselines for comparison.