- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MotifMatch for protein–glycan binding response prediction. It combines explicit, linkage-aware glycan motif counts with frozen ESM2 protein features through projected factors and Hadamard matching. The authors derive an overlap–alignment factorization of gradient coupling and investigate cross-glycan transfer using fixed-budget support experiments. On GlycanML, the method achieves the highest mean ranking scores across protein-cold, glycan-cold, and double-cold settings, with lower training and inference times under the reported implementation.
Critical strengths
The controlled representation–scoring comparisons provide a useful empirical insight. The paper goes beyond comparing glycan encoders under a single prediction head: it crosses motif and circular representations with concatenation and Hadamard scoring, and includes multiplicity and bilinear controls. On the official protein-cold split, Hadamard scoring improves NDCG@20 from 0.4330 to 0.4708 with motif counts, whereas it provides no corresponding improvement with circular features. This supports the consequential observation that the effectiveness of a molecular representation depends on how the interaction between partners is modeled.
The fixed-budget support experiment meaningfully strengthens the empirical investigation. Replacing motif-negative observations with motif-containing support observations while preserving the total measurement budget and each protein’s observation count is a thoughtful way to investigate cross-glycan transfer. Together with the competitive protein-cold ranking results, this makes the proposed approach a useful and computationally lightweight contribution, beyond a straightforward benchmark comparison.
Critical weaknesses
The evidence for an advantage in the additional cold-start settings is not sufficiently conclusive. Although MotifMatch achieves the highest mean ranking scores, several margins are very small relative to the reported variability. For example, glycan-cold Protein Spearman is 0.1971 versus 0.1967 for GIFFLAR, and double-cold NDCG@20 is 0.4751 versus 0.4702 for LectinOracle. Only three runs are reported, with split and training seeds varied together in these settings, and no paired uncertainty analysis is provided. Moreover, double-cold NMSE is 1.0030 and Protein Spearman is 0.0991, indicating limited absolute predictive performance. These results establish competitiveness more clearly than a reliable generalization advantage.
The mechanistic contribution is less established than the paper’s framing suggests. The overlap–alignment factorization follows directly from the outer-product gradient of the shared linear count projection and, as Appendix A.2 acknowledges, also applies to more general scoring functions. The support experiments do not directly measure response alignment or establish its relationship with observed transfer. Adding motif-containing glycans also changes coverage of other co-occurring structures, while the reported support gains have substantial seed variability. Consequently, the experiments support the usefulness of the proposed combination, but do not yet convincingly isolate the proposed mechanism. This matters because the architecture otherwise combines established components, making the strength of the explanatory contribution important to the paper’s novelty.
Questions for authors
Statistical clarification: Can you provide paired differences and confidence intervals for the principal ranking metrics against the strongest baselines in the glycan-cold and double-cold settings? Please specify the independent evaluation units and account for dependence among observations sharing proteins or glycans. This would help determine whether the small mean improvements are reliable.
Mechanism clarification: Is there direct evidence connecting the response-alignment term in Proposition 1 to the observed support-transfer gains? In particular, do overlap and alignment predict actual target-loss changes, and how does the support-selection protocol account for changes in co-occurring motifs? Please distinguish what the existing experiments establish from what remains an interpretation.
Experimental fairness clarification: What hyperparameter-search budgets and pretrained components were used for the baselines? For the efficiency comparison, were independently computed entity representations cached and reused for baselines wherever their architectures permit this? These details would materially affect my interpretation of both the accuracy comparisons and the reported speedups.
3 comments
The factorization alone does not establish the mechanism behind variable support gains. Can response alignment predict actual loss changes beyond shared structure, while accounting for co-occurring motifs? Please distinguish direct evidence from interpretation. (W2, Q2)
Our theoretical analysis links motif overlap and protein-conditioned response alignment to local target-loss changes (Proposition 1), but as you point out, we needed to connect this theoretical account more directly to the experimental transfer results. We therefore added a controlled local-update test and an analysis of the existing complete-training gains.
Across 36 motif–run conditions (12 motifs × three seeds), we compared small updates from the same k = 0 checkpoint using the original training set or its k = 16 support replacement, and measured the difference in held-out target loss. Only shared motif-projection columns were updated, with total measurements and per-protein observation counts fixed. We included added-minus-removed contributions across all motif coordinates, so changes in co-occurring structures entered the loss-change prediction.
To isolate the contribution of response direction in the diagnostic, the control preserves motif overlaps, residuals, and sensitivity magnitudes, replacing only response inner products with products of norms. Both scores use the same support and target observations and structural information. Retaining direction reduces relative L1 error from 32.05% to 0.212%. This supports the distinction made by our theory: motif overlap identifies shared-parameter connections, while response alignment and residuals characterize how updates through those connections affect target loss.
| Evaluation | Full score | Without directional alignment |
|---|---|---|
| Local update: Spearman with measured gain ↑ | 1.000 | 0.891 |
| Local update: correct change direction ↑ | 36/36 | 30/36 |
| Local update: aggregate relative L1 error ↓ | 0.212% | 32.05% |
| Training endpoint: Spearman with standardized MSE gain ↑ | 0.930 | 0.846 |
Endpoint gain is the target MSE reduction normalized by the k = 0 training-label variance; definitions are in Appendix C.7.
We then asked whether initial coupling relates to the k = 0→16 complete-training gains. Across 12 motif means, each averaging three seeds, the correlation is 0.930 for the full score versus 0.846 without direction. These endpoints come from separately trained models, connecting the initial diagnostic to differences in the eventual benefit of shared-motif support. The local experiment directly tests the predicted loss change; the endpoint analysis establishes an association with complete-training transfer. Together, they provide complementary empirical support for our theoretical account of how supervision is shared across glycans in a protein-dependent manner. Thank you for helping us make this experimental validation more complete!
What search budgets and pretrained components were used for the baselines? Were entity representations cached and reused wherever their architectures permit in the efficiency comparison? (Q3)
We apologize for not making these protocols sufficiently clear in the submission. Thank you for highlighting how they affect the interpretation of both accuracy and speed.
Our guiding principle was to follow the original authors’ released implementations and published experimental settings as closely as possible when adapting each baseline to the common protein–glycan regression task. We evaluated each baseline using its adopted configuration, without conducting additional hyperparameter search. Table 6 in Appendix C.2 lists the adopted configurations and per-run training limits for the official protein-cold benchmark. Each configuration was evaluated over three training seeds, with validation-based checkpoint selection.
To maintain comparability on the shared task, all benchmark methods use the same frozen, mean-pooled ESM2-650M protein features. Glycan encoders and prediction heads are trained on the binding task without loading additional pretrained glycan weights.
We also use the entity-reuse mechanisms available in the evaluated implementations. All methods use precomputed protein features and prepared glycan inputs. During training, learned encodings are recomputed as parameters change. The per-method caching procedures are detailed in Appendix C.5.
Timing begins after input preparation and model setup, and excludes ESM2 feature extraction. Training includes optimization, validation, and early stopping; inference uses prepared inputs and loaded models over the same 63,364 test pairs. These measurements therefore compare the evaluated implementations with their actual reuse procedures. Under this protocol, MotifMatch requires 25.63 s for training and 0.291 s for inference, with 1.016 M trainable parameters. We hope these details provide the context needed to assess the accuracy and efficiency comparisons.
Dear Reviewer dv87,
Thank you again for your detailed feedback. We have shared our responses along with the additional statistical and mechanistic analyses above. As the discussion deadline is approaching, we would be very grateful to hear whether our replies have addressed your concerns or if any questions remain.
Best regards,
The Authors
Thank you for reading our work so carefully and for suggesting concrete ways to strengthen the evidence. We appreciate your recognition of the insight from the controlled representation–scoring comparisons, the value of the fixed-budget support design, and the practical contribution of a lightweight method with competitive protein-cold ranking. Your questions helped guide our expanded cold-start evaluation with paired uncertainty analyses (§§4.1–4.2; Appendix C.3, Table 10) and the new controlled response-alignment experiments (§4.3, Figure 4; Appendix C.7); we also provide clearer explanations of the training, pretraining, caching, and timing protocols below (§4.5, Table 5; Appendices C.2 and C.5). These additions and clarifications are included in the revised manuscript and marked in blue, so that you can readily check them against your concerns. We hope these additional results and clarifications help address your concerns about the reliability of the ranking gains, the connection between response alignment and transfer, and the fairness of the comparisons.
We understand your concern. In these challenging cold-start settings, small mean margins need closer examination in light of the variation across runs. Double-cold response prediction is particularly difficult: all evaluated predictive models have mean NMSE close to the constant-reference value of 1 (Table 9). Following your suggestion, we therefore examined whether the ranking gains persist across data splits and remain supported after accounting for dependence among observations sharing proteins or glycans.
We expanded the evaluation to four models × five splits × three training seeds, totaling 60 model runs. Comparators were fixed before the expansion using the strongest submitted baseline for each metric, and paired methods were evaluated on identical test observations. Protein Spearman and NDCG@20 are averaged over eligible proteins, while Global Spearman uses all test pairs. We average the three training seeds within each split, then use the five paired split differences to construct a t-based 95% interval. The split is the replication unit for this interval.
As you noted, shared entities introduce a further source of dependence. We therefore also resample protein similarity clusters and glycan similarity clusters jointly, recomputing the metrics in each replicate. The same cluster weights are shared across methods and runs, preserving the links among repeated entities. These intervals assess sensitivity to test-entity composition conditional on the fitted models, complementing the split-level analysis. The full six comparisons are reported below.
Δ denotes MotifMatch minus the baseline, computed before rounding. Intervals are pointwise 95% intervals without multiplicity adjustment; positive splits are counted after averaging training seeds. Computational details are in Appendix C.3.
The additional analysis gives us a clearer answer for the hardest setting: the double-cold NDCG@20 gain is +0.0147, positive in four of five splits, with both intervals above zero. This supports a modest but robust improvement in prioritizing top candidates. The submitted double-cold NDCG@20 also exceeds the constant reference (0.4751 versus 0.3928, Table 9), illustrating the practical ranking value alongside the response-accuracy evaluation. We hope these experiments help address your concerns!