- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MisLocus is a benchmark for detecting variant-induced protein mislocalization in single-cell images. From the public VarChAMP collection, it curates about 3.3 million four-channel cell crops covering 250 reference proteins and 1,291 variants (1,055 reference–variant pairs), with plate-level splits, control wells, and an evaluation pipeline; imaging and CellProfiler features are inherited. Nine feature sets (CellProfiler, Cytoself, frozen and fine-tuned SubCell, MorphEM) are evaluated on three tasks: separating variants from references (per-pair XGBoost AUROC and retrieval mAP, with hits called against a control-based threshold), agreement with ClinVar and variant-effect predictors, and clustering of reference proteins by Human Protein Atlas location. The paper concludes that CellProfiler is competitive with a supervised classifier, while learned embeddings are better under retrieval.
Critical strengths
-
This benchmark study is valuable. The variant-induced mislocalization is a known disease mechanism, but existing variant-imaging data, including the VarChAMP source, were not organized for comparing representations. MisLocus provides a shared cell set, plate-level splits, control wells, and released crops and features, so new representations can be compared under identical conditions.
-
Control-based calibration shows why mean scores can mislead. Thresholding against same-protein control wells reveals that mean AUROC and hit rate can disagree. The paper also traces CellProfiler's high control scores to GFP intensity features.
Critical weaknesses
-
The hit rate in the main results uses a lenient rule. A variant is counted as a hit if it passes in at least one of the two replicate batch. With a 95th-percentile threshold, a variant without real effect can become a. it with up to 10% chance, but the paper does not report the control hit rate under this rule. With the stricter rule (both batches, 680 pairs), CellProfiler-ALL decreases from 29.4% to 10.9%, while MorphEM-GFP and Cytoself stay around 23% (Table 1, 4). So the paper's claim that CellProfiler is competitive with the bests learned models is true only under the lenient rule.
-
I think the advantage of learned embeddings is overstated. First, fine-tuned SubCell and Cytoself are trained with allele labels on the same alleles used for evaluation (Section A.2.3, A.2.5), and the main text does not mention this. Part of their good performance may come from this supervision. Second, the abstract and Section 4.3 say learned embeddings group proteins by location better than CellProfiler. However, CellProfiler-GFP has the same number of significant locations as fine-tuned SubCell (8/21), and its mean score (0.061) is higher than MorphEM (0.051–0.053) and Cytoself (0.034) (Fig. 3, Table 6).
Questions for authors
-
What is the hit rate of the control comparisons under the any-batch rule? If the thresholds are adjusted so that controls have a 5% hit rate after aggregation, does the ranking of representations stay the same?
-
How do fine-tuned SubCell and Cytoself perform on alleles not used in their training, for example by holding out a set of genes? This would show how much of their advantage comes from allele-label supervision.
-
For the HPA analysis (A.3.5), are the reference profiles built from all plates or only from the held-out T4 plates?
Response to Reviewer az7x
Thank you for your careful analysis of the hit-calling rule. Your feedback prompted us to measure how often replicate-matched controls are called as hits and to report calibrated hit calls alongside agreement with external references.
Weakness 1 / Question 1. The lenient hit rule and the control hit rate. Thank you for raising the effect of scoring a variant across multiple batches. Under the any-batch rule, 6–10% of control comparisons were called hits under XGBoost (5–13% under retrieval). We now use a per-batch threshold corrected for the number of batches a variant is scored in with Šidák correction and filter control comparisons exactly like variants. Control hit rates are now 3.6–4.2% under XGBoost and 3.8–7.6% under retrieval (Table 1, Appendix A.4.1). Appendix A.4.3 shows the control-null distribution behind each representation's threshold, and Appendix A.4.2 reports every setting under both rules (Table 6). To your question on rankings of representations: with calibrated controls the hit-rate ranking barely changes (Kendall τ = 0.94 under XGBoost and 0.89 under retrieval for the nine submitted settings).
Your observation about CellProfiler holds under the strict both-batch rule, where CellProfiler-ALL drops to 18% (Table 6), and the hits it gains from the non-protein stains include none of the expert-called mislocalizations (Appendix A.5.5). More fundamentally, hit rates mix variant signal with each representation's control noise and under XGBoost track agreement with none of our references (|ρ| ≤ 0.26 across 19 settings). We therefore skipped comparing representations by hit rate in the main tables. Tables 1,2 and Fig. 2 now compare their continuous scores against the expert’s visual annotation of protein mislocalization, variant’s clinical significance and protein localization references, and hits are reported for screening. The CellProfiler claim is narrowed accordingly (Discussion): "per-variant supervised classifiers yield similar agreement with visual and clinical references across all 19 representations (mean AUROC within 0.1 Tables 1 and 2), allowing interpretable CellProfiler features to remain competitive despite their sensitivity to intensity and non-protein stains."
Weakness 2 / Question 2. Allele-label supervision. We now state this in Section 3.3: fine-tuned SubCell and Cytoself are trained with reference and variant labels on the T1–T2 plates, so we test whether they generalize to unseen proteins. We retrained fine-tuned SubCell and Cytoself, and every other model trained on a subset of MisLocus, while withholding all cells from 162 protein variants across 24 genes. On those held-out variant proteins, their mislocalization scores are unchanged (fine-tuned SubCell ViT 0.796 against 0.799, Cytoself 0.695 against 0.692), and their HPA localization retrieval of the reference proteins is essentially unchanged (Cytoself 0.035 against 0.034, fine-tuned SubCell ViT 0.077 against 0.079, Tables 10 and 11).
Weakness 2. Agreement with HPA localization annotations. Thank you for asking us to distinguish the performance of fine-tuned and frozen representations. Fine-tuned SubCell has the highest localization score (0.079 against 0.061 for CellProfiler-GFP, a difference of +0.017 [0.000, 0.037]), and frozen learned embeddings perform comparably to protein-channel CellProfiler (Section 4.3, Table 8). Both have 8 of 21 significant locations, but three of CellProfiler-GFP's are the overlapping sperm-tail labels. Section 4.3 now has a neutral heading, and the abstract and introduction state the narrower claim.
Question 3. HPA reference profiles. We clarified in our text that only the held-out T4 plates are used, for every representation (Section 4.3, Appendices A.3.5 and A.3.7).