- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether allowing variable binding-core lengths improves peptide–MHC-II classification for the chicken allele Gaga-BLB2*002:01. It compares an existing register-scoring model permitting 9- and 10-residue cores with a parameter-matched variant restricted to 9-residue cores. The dataset contains 62,590 records, including 4,184 BLB2 positives and 58,406 negatives. Validation includes five outer folds, subgroup analyses, supervised core-annotation recovery, and an external comparison with MixMHC2pred.
The flexible model does not improve the reported classification metrics: both models have pooled ROC-AUC 0.920, while pooled PR-AUC is 0.583 for the flexible model and 0.597 for the restricted model. The flexible model matches annotated core length on 74.3% of held-out positives and the exact annotated core on 30.7%. The intended contribution is an empirical distinction between classification performance and compatibility with heterogeneous core annotations, rather than a new scoring mechanism. The paper also investigates possible BLB1-related label contamination as a competing explanation.
Critical strengths
S1. A focused and biologically motivated question. Testing the fixed 9-residue assumption on an allele with structural support for a 10-residue core is a sensible experimental setting. Distinguishing classification from core recovery can reveal a limitation of relying on classification metrics alone. This supports acceptance in principle even if the method is not new and does not outperform existing predictors.
S2. Targeted diagnostics and transparent negative findings. The paper examines the 10-mer subgroup that should plausibly benefit, distinguishes length prediction from exact core recovery, reports negative controls, and acknowledges inferred annotations, synthetic negatives, and single-seed uncertainty. These analyses make the work more informative than a single aggregate performance comparison and provide a basis for a useful, bounded empirical conclusion.
Critical weaknesses
W1. The central experiment does not isolate length flexibility (Sections 3–6; Appendix Table 7). Masking 10-mer candidates also removes auxiliary core supervision for 10-mer-annotated positives. Moreover, the selected flexible model uses mildly weighted BCE while the restricted model uses ordinary BCE. Equal parameter counts do not control these differences. Consequently, the observed performance gap cannot be attributed specifically to candidate-length support. The missing 10-mer- only control also leaves open whether an allele-appropriate fixed length would suffice. This is central because the paper’s main contribution is a controlled empirical conclusion, not the introduction of a new mechanism.
W2. Evaluation independence and the interpretation of the findings remain insufficiently established (Sections 4, 6–10). If model choices were selected using the inner folds of outer round 1 and then reused across all five outer folds, later test folds may already have influenced those choices. For example, tuning on F2–F5 while testing F1 does not leave F2–F5 untouched for later evaluation. This is a conditional concern requiring clarification. Separately, agreement with supervised computational annotations does not establish faithful biological explanations, and low scores on double-labelled peptides do not establish contamination: a genuine BLB1/BLB2 dual binder missed by the model produces the same observation. These gaps limit both the reliability and the scope of the claimed insight.
Questions for authors
Q1. Required clarification and targeted control – Can you report a comparison using the same classification loss, class weights, splits, and selection protocol, with auxiliary core supervision disabled for all variants? Including 9-mer-only, 10-mer-only, and flexible variants would distinguish changing the preferred fixed length from requiring per-peptide flexibility. An additional comparison could supervise the same 9-mer examples in both the restricted and flexible models. Allowing 10-mer outputs does not require providing 10-mer auxiliary targets. The original fully supervised model can remain a reference. These controls would directly affect my assessment of W1.
Q2. Required clarification – Which data selected the architecture, loss, training duration, and threshold for each outer fold? Was there a development set independent of every reported test fold, or was selection nested separately within each outer training set? If choices from outer round 1 were reused after exposure to later test folds, please provide an evaluation that accounts for this selection. Protein-disjoint training and test sets alone do not establish independence of model selection.
Q3. Required clarification and missing comparison – Can you report exact core-match and start- position performance for both models, overall and by annotated length? In Table 4, 1257/2050 = 61.3% measures 10-mer length sensitivity, not exact recovery of a 10-mer core. Please distinguish this from the 30.7% overall exact-match result. The 74.3% length accuracy is a nontrivial supervised prediction result, but it validates agreement with the annotation procedure rather than independently establishing structural truth. If the claim concerns explanation of classification, what shows that the reported argmax core faithfully represents a classifier using soft pooling and additional whole-sequence branches? Otherwise, narrowing the claim to supervised annotation recovery would be appropriate.
Q4. Required clarification and sensitivity analysis – What evidence distinguishes incorrect BLB2 assignment from genuine dual binding or model failure? The reported enrichment of BLB1-positive labels in the 9-mer group is compatible with contamination but is not a contamination-rate estimate or lower bound. For example, a correctly labelled dual binder with a low BLB2 prediction would be counted in the same pattern. Please separate the biological hypothesis motivated by Halabi et al. from the present paper’s prediction about classification scores. Reporting results separately for double-positive and BLB2-only records, and repeating the subgroup and model comparisons after excluding double-positive records, would be useful sensitivity analyses without assuming those records are wrong. Also explain why contamination would differentially affect the two models, rather than merely make the 9-mer subgroup harder for both.
Q5. Required clarification and uncertainty analysis – How should the claim of statistically indistinguishable ranking performance be reconciled with a paired bootstrap PR-AUC difference of 0.013 and a 95% interval [0.006, 0.021]? Equal rounded ROC-AUC values are not an equivalence test. The mean fold-wise PR-AUC gap is about 0.003, whereas the pooled gap is about 0.014; please report paired fold-wise differences and discuss the role of cross-fold score scales. Protein-cluster resampling and a small number of independent seeds would better characterize the uncertainty relevant to this small effect. Please also distinguish the overall bootstrap result from subgroup-specific uncertainty and the tuning-stage negative controls from final outer-fold evidence.
Q6. Clarification of the comparison – Why was DeepMHCII, identified as closely related, not evaluated or discussed as an adaptable baseline under the same BLB2 splits? I am not requesting every recent predictor or assuming human-only tools support chicken alleles. Please explain applicability and adaptation constraints. MixMHC2pred v2.0.2 should be associated with the corresponding 2023 work rather than only the 2019 citation. A fixed 9-mer core alone does not make an evaluation zero-shot or out-of-distribution; this depends on training coverage, which also needs reconciliation with the stated possibility of overlapping chicken data. Scope of requests and minor corrections. Q1–Q5 concern the existing central claims, not a demand for a broad new research program. Multi-species validation and extensive new structural experiments are possible future work, not prerequisites here. Please also state the class ratio unambiguously as positive:negative approximately 1:14; check the boldface for pooled precision in Table 2 (0.694 versus 0.698); and provide the negative-sampling procedure, weighted-BCE weights, PR-AUC definition, and fold-specific selection details. These reporting corrections alone would not change my recommendation.
Thanks for the very thorough review, we’ll go through this point by point.
On W1/Q1 (confound + requested control): we reran it with the exact grid you described — 9mer-only, 10mer-only, and flexible, all with auxiliary supervision disabled, same loss/weights/splits/selection across the board. Length alone isn’t significant between flexible and 9mer-only. It is significant for 10mer-only vs flexible though (flexible wins there). Separately, turning auxiliary supervision ON for the flexible model significantly hurts its PR-AUC (-0.025, 95% CI [-0.0395,-0.0110]). So the real effect in this whole thing is the auxiliary loss, not candidate-length support like we originally claimed. Rewriting that section.
On W2/Q2 (selection independence): you and yyLV both caught the same thing — selection was done once on round 1’s inner folds and reused for rounds 2-5, so later test folds could’ve influenced the choice. We reran selection independently per outer round. Result flips: flexible now slightly ahead (+0.0084 PR-AUC, CI [-0.0061,0.0229]) instead of behind like originally reported (-0.0133, CI also crossing zero). So this wasn’t just a mild leakage issue, the original gap direction itself wasn’t even stable.
On Q3 (metric clarification): fair catch, 61.3% is length-match not exact-core-match, we’ll fix the table/text to make that distinction explicit and clarify the 74.3%/30.7% numbers are agreement with the annotation procedure, not an independent ground-truth check.
On Q4 (contamination): excluded all dual BLB1/BLB2-labeled positives and reran the subgroup comparison, gap barely moves (0.236 to 0.239, flexible model), actually slightly wider. We also broke it out by dual-positive-only vs single-positive-only (9mer dual n=196, 10mer dual n=14, too small to report a separate PR-AUC on its own). So contamination doesn’t look like it’s driving the effect the way we suggested, we’re softening that claim.
On Q5 (uncertainty/bootstrap unit): you’re right, we were resampling at peptide level not protein level, which is the actual grouping unit for the splits. Redone with protein-clustered resampling, the original PR-AUC CI [0.006,0.021] becomes [-0.001,0.028], crosses zero now. Also reran at 2 more seeds (43,44), gap direction holds (+0.013/+0.014/+0.011) but given the CI above that’s “consistently not significant” not confirmation. And on fold-wise vs pooled: mean fold-wise diff is +0.004 (SD 0.025) vs pooled +0.013, difference comes from 3 of 5 folds favoring the ablation plus pooling being prevalence-weighted, not a uniform per-fold effect.
On Q6: DeepMHCII baseline is a fair ask but we can’t responsibly implement and tune a second published method before the deadline, we’ll flag it as the clearest future work item and note our current comparison should be read as an ablation study not a SOTA comparison. Also fixing the MixMHC2pred citation year, the class ratio wording (1:14), and the Table 2 boldface error you caught (0.698 should’ve been bold, not 0.694) — already corrected in our working draft.