- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark for class-incremental protein function prediction under frozen pretrained encoders and proposes ProtoCIL, a training-free prototype classifier. The benchmark covers two enzyme datasets, five sequence/structure encoders, and eleven CIL baselines. ProtoCIL maintains class prototypes while sharing covariance estimates among adaptively formed class groups. Its target number of groups is derived from effective embedding rank and sample size; groups are refined incrementally using held-out validation samples, Ledoit–Wolf shrinkage, and a Mahalanobis classifier with log-determinant normalization. Across the ten encoder–dataset combinations, ProtoCIL reports the best average incremental accuracy (74.3% on Enzyme Reaction and 91.8% on EC Number) and low forgetting (3.8% and 1.8%). Ablations, alternative phase constructions, and efficiency measurements are also provided.
Critical strengths
-
The benchmark is broad and useful: it evaluates multiple CIL paradigms across five heterogeneous protein encoders, two class scales, and several phase constructions. The consistent advantage across all encoder–dataset combinations makes the empirical result more convincing than a single-backbone demonstration.
-
The method is technically coherent and well motivated for the sample-starved frozen-feature regime. Adaptive covariance sharing, shrinkage, and log-determinant normalization directly address the instability of class-wise covariance estimates. The component ablations and grouping-strategy comparisons substantiate the design choices.
Critical weaknesses
-
The biological validity of the evaluation is not yet sufficiently established. Both datasets are originally multi-label, but the benchmark converts them to single-label problems by argmax/first annotation. More importantly, the paper analyzes sequence identity across phases, but does not report a homology-controlled train/test split. Cross-phase dissimilarity does not rule out highly homologous train/test proteins within the same class. The near-99.5% joint-MLP accuracy on the 4,788-class EC dataset makes this concern especially important. If train/test near-neighbors are present, the very large gains may not reflect generalization to genuinely novel proteins.
-
The resource and comparison accounting needs clarification. ProtoCIL reserves 10% of every class for repeated split validation and stores dense group covariances, while most baselines use globally fixed hyperparameters and only a subset receive per-encoder tuning. It is unclear whether the reported storage includes retained validation embeddings and whether all competitors receive comparable validation and memory budgets. This matters to the claims of a practical, tuning-free continual learner.
Questions for authors
-
What is the maximum sequence identity between training and test proteins, and how does ProtoCIL perform under a cluster-based homology split (e.g., 30% or 50% identity)? This result would materially affect my confidence in the benchmark.
-
Does the reported storage include all held-out validation embeddings? Must old validation samples remain accessible after each phase, or can they be discarded? Please compare against baselines under matched total memory and validation budgets.
-
How robust are the conclusions when preserving the original multi-label annotations and evaluating with appropriate multi-label metrics?
We thank the reviewer for the careful review. All revisions are included in the revised PDF, and new text and new results are highlighted in blue.
Q1 (homology and the high joint accuracy).
The reviewer's concern was justified. The training and test sets we built from CLEAN's data overlapped, so some test proteins had already been seen in training. This is why joint accuracy was so high. We rebuilt the split by removing from training every protein that also appears in the test set (64,161 train, 8,106 test, 3,932 classes). All EC Number results in the revised paper use this new split, and the Joint MLP now reaches 68.9%.
On the new split, the maximum sequence identity of each test protein to the training set (MMseqs2) is:
The reviewer's concern is about close homologs within the same class. A class always belongs to one phase, so the Same phase row covers this case, and there only 2.8% of test proteins have a training protein with 70% or more identity. Following the reviewer, we also evaluated two homology-controlled test sets that keep only the test proteins whose maximum identity to any training protein is below 50% or 30% (4,482 and 1,018 of the 8,106 test proteins). Average accuracy (%), mean over the five encoders and 3 seeds, including Deep SLDA and KLDA, two baselines we added following Reviewer yQH4:
The 30% test set contains only proteins without close homologs in training, so all methods drop sharply. ProtoCIL remains the most accurate on average and stays well above joint training.
Q2 (storage, held-out samples and matched budgets).
The storage of ProtoCIL in the revised Table 2 includes the frozen embeddings of past training samples, about 313 MB on EC Number with ESM2 650M (64,161 embeddings of dimension 1,280 in float32), so ProtoCIL needs 557.5 MB in total, still below the 879.5 MB of RanPAC.
Old held-out samples must remain accessible under the current design. When a group that contains classes from earlier phases is considered for a split, candidates are scored on the held-out samples of all classes in the group, and the final prototypes and covariances are refit on all training samples. ProtoCIL stores no raw proteins and performs no replay training or gradient updates.
On matched memory, the Joint MLP keeps the same embeddings as ProtoCIL and retrains on them at every phase. Under the efficiency setting of Table 2, ProtoCIL reaches 80.0% against 71.5% for the Joint MLP and is also faster, 212.8 s against 220.5 s. In the main experiments, ProtoCIL is more accurate than it on both datasets, 74.9% against 66.4% on Enzyme Reaction and 72.9% against 68.9% on EC Number.
On validation budgets, ProtoCIL uses the same fixed settings for every encoder, and its held-out samples are used only for split decisions, not for tuning any hyperparameter. Baselines given per-encoder hyperparameter tuning reach at most 71.3% on Enzyme Reaction and 60.0% on EC Number, both below ProtoCIL without tuning.
Q3 (multi-label annotations).
We have not evaluated multi-label metrics, so we cannot claim robustness under them. Two facts limit the effect of the single-label conversion. First, more than 88% of proteins in every split carry a single EC annotation, so the conversion changes the label of a small fraction of proteins only. Second, the retained label is always one of the protein's true labels, and the same conversion is applied to every method. A prediction of another true label is counted as wrong, so the single-label accuracy is a lower bound on any-label accuracy for all methods alike. Single-label prediction is also the standard setting for class-incremental learning, while multi-label class-incremental learning is a different problem that we leave to future work.