acceptodds
Under review as a conference paper at ICLR 2027

MisLocus: A single-cell imaging benchmark for protein mislocalization

Abstract

Genetic variants that alter the location of proteins within cells can disrupt protein function and contribute to disease. Microscopy can detect protein mislocalization, but benchmarking image representations to automate this task requires data for reference (healthy) proteins plus protein variants, with a replicate structure that separates biological signal from technical variation. We introduce MisLocus, a single-cell imaging benchmark for variant-induced protein mislocalization. It contains ∼3.3 million single-cell images, covering 250 reference proteins and 1,291 variants, with experimental replicates spanning imaging wells, plates, and batches. We evaluate nine image representations from CellProfiler, Cytoself, SubCell, and MorphEm, including handcrafted features, pretrained learned representations, and models retrained or fine-tuned on MisLocus. Representations are assessed along three complementary axes: detecting variant-induced mislocalizations, predicting each variant’s clinical pathogenicity based on mislocalization phenotype, and discriminating annotated reference protein localization patterns. We find no single representation dominates all evaluation axes. Engineered features, paired with supervised models trained to predict localization shifts, detect variant-induced mislocalization competitively with the best learned representations, whereas learned representations better organize proteins according to shared localization patterns. MisLocus establishes a controlled and reproducible framework for benchmarking single-cell microscopy representations on protein mislocalization phenotyping.

Primary area:
datasets and benchmarks
Keywords:
cellular imaging, single-cell profiling, protein mislocalization, computer vision, biomedical imaging, subcellular localization, fluorescence microscopy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_az7xOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MisLocus is a benchmark for detecting variant-induced protein mislocalization in single-cell images. From the public VarChAMP collection, it curates about 3.3 million four-channel cell crops covering 250 reference proteins and 1,291 variants (1,055 reference–variant pairs), with plate-level splits, control wells, and an evaluation pipeline; imaging and CellProfiler features are inherited. Nine feature sets (CellProfiler, Cytoself, frozen and fine-tuned SubCell, MorphEM) are evaluated on three tasks: separating variants from references (per-pair XGBoost AUROC and retrieval mAP, with hits called against a control-based threshold), agreement with ClinVar and variant-effect predictors, and clustering of reference proteins by Human Protein Atlas location. The paper concludes that CellProfiler is competitive with a supervised classifier, while learned embeddings are better under retrieval.

Critical strengths
  1. This benchmark study is valuable. The variant-induced mislocalization is a known disease mechanism, but existing variant-imaging data, including the VarChAMP source, were not organized for comparing representations. MisLocus provides a shared cell set, plate-level splits, control wells, and released crops and features, so new representations can be compared under identical conditions.

  2. Control-based calibration shows why mean scores can mislead. Thresholding against same-protein control wells reveals that mean AUROC and hit rate can disagree. The paper also traces CellProfiler's high control scores to GFP intensity features.

Critical weaknesses
  1. The hit rate in the main results uses a lenient rule. A variant is counted as a hit if it passes in at least one of the two replicate batch. With a 95th-percentile threshold, a variant without real effect can become a. it with up to 10% chance, but the paper does not report the control hit rate under this rule. With the stricter rule (both batches, 680 pairs), CellProfiler-ALL decreases from 29.4% to 10.9%, while MorphEM-GFP and Cytoself stay around 23% (Table 1, 4). So the paper's claim that CellProfiler is competitive with the bests learned models is true only under the lenient rule.

  2. I think the advantage of learned embeddings is overstated. First, fine-tuned SubCell and Cytoself are trained with allele labels on the same alleles used for evaluation (Section A.2.3, A.2.5), and the main text does not mention this. Part of their good performance may come from this supervision. Second, the abstract and Section 4.3 say learned embeddings group proteins by location better than CellProfiler. However, CellProfiler-GFP has the same number of significant locations as fine-tuned SubCell (8/21), and its mean score (0.061) is higher than MorphEM (0.051–0.053) and Cytoself (0.034) (Fig. 3, Table 6).

Questions for authors
  1. What is the hit rate of the control comparisons under the any-batch rule? If the thresholds are adjusted so that controls have a 5% hit rate after aggregation, does the ranking of representations stay the same?

  2. How do fine-tuned SubCell and Cytoself perform on alleles not used in their training, for example by holding out a set of genes? This would show how much of their advantage comes from allele-label supervision.

  3. For the HPA analysis (A.3.5), are the reference profiles built from all plates or only from the held-out T4 plates?

1 comment
AuthorsOpenReview ↗

Response to Reviewer az7x

Thank you for your careful analysis of the hit-calling rule. Your feedback prompted us to measure how often replicate-matched controls are called as hits and to report calibrated hit calls alongside agreement with external references.

Weakness 1 / Question 1. The lenient hit rule and the control hit rate. Thank you for raising the effect of scoring a variant across multiple batches. Under the any-batch rule, 6–10% of control comparisons were called hits under XGBoost (5–13% under retrieval). We now use a per-batch threshold corrected for the number of batches a variant is scored in with Šidák correction and filter control comparisons exactly like variants. Control hit rates are now 3.6–4.2% under XGBoost and 3.8–7.6% under retrieval (Table 1, Appendix A.4.1). Appendix A.4.3 shows the control-null distribution behind each representation's threshold, and Appendix A.4.2 reports every setting under both rules (Table 6). To your question on rankings of representations: with calibrated controls the hit-rate ranking barely changes (Kendall τ = 0.94 under XGBoost and 0.89 under retrieval for the nine submitted settings).

Your observation about CellProfiler holds under the strict both-batch rule, where CellProfiler-ALL drops to 18% (Table 6), and the hits it gains from the non-protein stains include none of the expert-called mislocalizations (Appendix A.5.5). More fundamentally, hit rates mix variant signal with each representation's control noise and under XGBoost track agreement with none of our references (|ρ| ≤ 0.26 across 19 settings). We therefore skipped comparing representations by hit rate in the main tables. Tables 1,2 and Fig. 2 now compare their continuous scores against the expert’s visual annotation of protein mislocalization, variant’s clinical significance and protein localization references, and hits are reported for screening. The CellProfiler claim is narrowed accordingly (Discussion): "per-variant supervised classifiers yield similar agreement with visual and clinical references across all 19 representations (mean AUROC within 0.1 Tables 1 and 2), allowing interpretable CellProfiler features to remain competitive despite their sensitivity to intensity and non-protein stains."

Weakness 2 / Question 2. Allele-label supervision. We now state this in Section 3.3: fine-tuned SubCell and Cytoself are trained with reference and variant labels on the T1–T2 plates, so we test whether they generalize to unseen proteins. We retrained fine-tuned SubCell and Cytoself, and every other model trained on a subset of MisLocus, while withholding all cells from 162 protein variants across 24 genes. On those held-out variant proteins, their mislocalization scores are unchanged (fine-tuned SubCell ViT 0.796 against 0.799, Cytoself 0.695 against 0.692), and their HPA localization retrieval of the reference proteins is essentially unchanged (Cytoself 0.035 against 0.034, fine-tuned SubCell ViT 0.077 against 0.079, Tables 10 and 11).

Weakness 2. Agreement with HPA localization annotations. Thank you for asking us to distinguish the performance of fine-tuned and frozen representations. Fine-tuned SubCell has the highest localization score (0.079 against 0.061 for CellProfiler-GFP, a difference of +0.017 [0.000, 0.037]), and frozen learned embeddings perform comparably to protein-channel CellProfiler (Section 4.3, Table 8). Both have 8 of 21 significant locations, but three of CellProfiler-GFP's are the overlapping sperm-tail labels. Section 4.3 now has a neutral heading, and the abstract and introduction state the narrower claim.

Question 3. HPA reference profiles. We clarified in our text that only the held-out T4 plates are used, for every representation (Section 4.3, Appendices A.3.5 and A.3.7).

Reviewer_sdcKOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new single-cell imaging benchmark caled Mislocus. Mislocus is a benchmark curated from the public VarChAMP collection. Protein mislocalization is a task where a genetic variant changes a protein’s subcellular location relative to its reference (healthy) form. The authors score 9 different representations on three different axes.

Critical strengths
  1. The paper has a strong case for why protein mislocalization is an important task to model and is well motivated.
  2. The benchmark has four replicates which is the core strucutre here. The supervised/unsupervised feature handling is useful as it tells us how much supervision is needed for this particular task.
Critical weaknesses
  1. The benchmark is not posed as an ML task. The contribution is a curated benchmark built from existing data (VarChAMP images and CellProfiler features), evaluated with established tools on off-the-shelf or conventionally fine-tuned encoders. The resulting comparison is useful for practitioners choosing a representation for mislocalization screens, but it does not identify a new challenge for machine learning. The task has properties that could define such challenges: learning representations of the difference between a variant and its paired reference, separating localization signal from technical confounds, and generalizing to unseen genes or variants. None of these is posed as a task with its own protocol, metric, or failing baselines. Relatedly, the paper argues that mislocalization detection differs from localization classification but does not demonstrate it. A direct comparison of representation rankings on the two tasks would establish whether MisLocus measures something existing localization benchmarks do not. As it stands, the paper reads as a well-executed application study, which may be better suited to a computational biology venue or workshop.
  2. No held-out gene or held-out variant split. All splits differ only by physical plate, so the benchmark measures robustness to technical replicates, not generalization to unseen proteins or variants. More importantly, Cytoself and fine-tuned SubCell are trained with allele-identity supervision on T1/T2 for the same alleles later scored on T4. These models have effectively been trained to separate the exact variant–reference pairs they are evaluated on, which likely explains part of fine-tuned SubCell’s advantage and would let any future submission do the same. A held-out-gene or held-out-variant track is needed for the benchmark to measure generalization.
  3. Frozen and trained representations are compared in one ranking. Retrained or fine-tuned methods (which see MisLocus images and labels) should be reported separately from frozen feature extractors. The comparison is further confounded by preprocessing: SubCell inputs are min–max normalized per cell, removing absolute intensity, while CellProfiler retains it. The baseline set is also limited. A general natural-image encoder such as DINOv3 would show how far non-microscopy foundation models get. A simple intensity-only baseline would also indicate how much performance is achievable trivially.
  4. The CellProfiler top-K analysis is not unsupervised. Features are selected by importance in supervised variant-vs-reference classifiers, and the resulting “unsupervised” PA score is then used to argue that the signal is present but diluted. This makes the conclusion partly circular, and the same selection is not applied to learned embeddings for parity.
Questions for authors
  1. Can you report results on a held-out-gene or held-out-variant split, at least for the trained and fine-tuned models? How do fine-tuned SubCell and Cytoself perform on alleles never seen during training?
  2. Can you report frozen and trained representations in separate tables, and a natural-image encoder (e.g., DINOv3), and an intensity-only baseline?
  3. How much of the CellProfiler vs. SubCell difference is due to per-cell intensity normalization? What happens if CellProfiler features are intensity-normalized, or SubCell inputs are not?
4 comments
Reviewer_sdcKOpenReview ↗

No response on reviews - lowering score from 2 to 1.

The authors have not participated in the rebuttal phase. Hence, I have to lower the score from a 2 to 1 as the paper as it currently stands cannot be accepted without proper peer review process.

We sincerely apologize for the delayed response and for not updating you sooner. We undertook substantial additional analyses and validations to address your concerns thoroughly, which took longer than anticipated. Our detailed responses are now ready, and we will post them shortly. Thank you for your time and patience.

AuthorsOpenReview ↗

Response to Reviewer sdcK (part 1/2)

Thank you for suggesting ways to make the machine-learning tasks more explicit. Your feedback helped shape the revised evaluation protocol for generalization across replicate plates and held-out genes, and for robustness to variation in protein intensity. Weakness 1. Posing MisLocus as an ML task. Thank you for this crucial observation. We now explicitly formulate MisLocus as a paired population-level distribution-shift detection problem under structured technical and biological confounders: detecting subtle shifts between empirical single-cell distributions (variant vs. paired reference) while remaining invariant to nuisance shifts (protein expression/intensity, cell density, and physical plate effects). To turn the challenges you identified into concrete ML tasks, the revised paper defines three standardized ML evaluation tracks (Section 5 & Appendix A.3.7), each with its own protocol, metrics, andfailing trivial baselines (4 intensity-only and cell-count baselines):

  • Track 1 (Out-of-replicate generalization): Generalizing variant–reference distribution shifts across independently prepared physical replicate plates (T1–T3 → held-out T4).
  • Track 2 (Unseen-gene generalization): Generalizing to unseen proteins via a fixed stratified split of 24 held-out genes (162 variants) withheld entirely from representation training and validated against both expert calls and blinded 3-rater visual annotations (Appendix A.5.3, A.5.8, Tables 10 & 15).
  • Track 3 (Confound-controlled distribution-shift detection): Disentangling spatial localization shifts from expression-level confounds by matching variant and reference single-cell populations on protein intensity (plus protein-only vs. context-stain ablations; Appendix A.5.5–A.5.6, Table 13 and 14).

We also formalize the open ML representation-learning challenge in Section 5 and Appendix A.3.7: current methods embed single cells independently and compare populations post hoc; learning paired or set-level population representations that directly parameterize variant–reference distribution shifts while remaining invariant to expression level and plate effects is a natural next frontier enabled by MisLocus. The benchmark also yields a result that matters for representation learning: once a supervised classifier is trained on top, all 19 image representations agree with expert, clinical and sequence-based labels within 0.1 AUROC (Tables 1, 2 and 7), so classifier-based evaluation hides differences that unsupervised retrieval exposes.

Mislocalization detection and standard protein-localization benchmarks are distinct problems, yet we still consider a model's capacity to group proteins by localization important. We therefore tested how rankings correlate across tasks for the 19 image settings. Representations that organize reference proteins better by HPA localization (our HPA localization retrieval, Section 4.3, the analysis closest to a standard protein-localization benchmark) also agree better with expert, clinical and blinded visual labels under retrieval (Spearman ρ = 0.54–0.75, a moderate correlation), but not under supervised scoring (ρ = 0.14–0.37) (Appendix A.5.1). A protein-localization benchmark would therefore likely pick similar representations for retrieval but not for supervised scoring, so MisLocus, as a mislocalization benchmark, measures something that protein-localization benchmarks alone do not.

AuthorsOpenReview ↗

Response to Reviewer sdcK (part 2/2)

Weakness 2 / Question 1. Held-out genes and allele-identity supervision. This was an important point. The allele-identity training signal is now stated in Section 3.3. We retrained every model on MisLocus (fine-tuned SubCell ViT and MAE, two- and three-channel SubCell, and Cytoself) while withholding all cells from 24 genes (162 variants), chosen by a stratified rule fixed before training and containing all 9 expert-called positives. On these genes the retrained models score almost the same as those trained on all genes (fine-tuned SubCell ViT 0.796 against 0.799, MAE 0.791 against 0.794, Cytoself 0.695 against 0.692), agree equally well with the blinded labels, and keep their localization structure (Tables 10, 11 and 15). The similar results on held-out genes suggest that exposure to those genes during MisLocus training is not required to obtain the reported performance. The revised protocol includes this split as a standard evaluation setting.

Weakness 3 / Question 2. Separate reporting, DINOv3 and trivial baselines. Table 1 now reports adapted, frozen and baseline settings separately. We added DINOv3 ViT-S/16 on the protein channel and on all channels. On the protein channel DINOv3 performs on par with MorphEM, and below fine-tuned SubCell in localization structure. The intensity-only baselines reach hit rates similar to learned representations (23–27%), but these and the cell-count baseline agree with expert, clinical and sequence-based labels no better than the weakest image representation (Tables 1 and 2, Appendix A.5.7). These baselines are now included in every evaluation.

Question 3. Per-cell intensity normalization. We tested both directions. Re-extracting frozen SubCell with one fixed intensity scale per plate, so that intensity differences between cells are kept, changes its XGBoost AUROC by only +0.004 to +0.008, and CellProfiler-GFP stays ahead by 0.038. Removing all intensity features from CellProfiler lowers its AUROC from 0.768 to 0.736, and the hits it loses come from the non-protein stains (Appendix A.5.6, Table 14). Per-cell normalization therefore does not explain the difference between SubCell and CellProfiler.

Weakness 4. The top-K analysis. Thank you for pointing out the importance of separating feature selection from evaluation. We now rank features using classifiers trained and validated on T1–T3, evaluate them on T4, and compare with 20 random feature subsets of the same size. We apply the same protocol to CellProfiler and fine-tuned SubCell (Figure 8). The top 10–50 CellProfiler-GFP features raise its retrieval score from 0.106 to 0.30–0.31, above every random subset, while fine-tuned SubCell stays near its full-feature value. We now present this as signal recoverable with the help of supervision, not as an unsupervised result.

Reviewer_xUqLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MisLocus, a single-cell imaging benchmark derived from the public VarChAMP collection for evaluating image representations on variant-induced protein mislocalization. It contains approximately 3.3 million quality-controlled single-cell images covering 250 reference proteins and 1,291 variants, with 1,055 paired reference–variant comparisons across 125 reference proteins. The authors evaluate nine representation settings from CellProfiler, Cytoself, SubCell, and MorphEM along three axes: reference–variant separation, agreement with clinical pathogenicity annotations and variant-effect predictors, and preservation of annotated reference-protein localization structure. No representation dominates across all settings: handcrafted CellProfiler features remain competitive under supervised evaluation, whereas learned representations generally perform better in unsupervised retrieval. The main contribution is a standardized evaluation framework with curated single-cell crops, harmonized annotations, replicate-aware splits, and technical-control calibration.

Critical strengths

A biologically meaningful standardized evaluation framework. This work focuses on variant-induced changes in protein localization and integrates large-scale single-cell imaging, paired reference–variant measurements, replicate-aware data splits, and technical-control calibration to provide a unified platform for comparing different image representations.

Critical weaknesses
  • The biological interpretation of the main detection task is insufficiently validated. The benchmark operationalizes mislocalization detection as reference–variant image separability. However, such separability can also arise from changes in protein abundance, intensity, or other phenotypes without a localization change. Although the main text briefly reports agreement with manual annotations on a small subset, it does not establish how reliably the reported hit calls identify actual mislocalization. This leaves a gap between the quantity measured by the benchmark and the biological interpretation of its main results.

  • Cross-representation comparisons are not fully controlled for training exposure and input channels. For example, frozen SubCell-portable uses a three-channel input, while fine tuned SubCell uses all four channels. In addition, some models are retrained or fine-tuned on MisLocus, whereas others are evaluated as frozen pretrained models. As a result, the comparisons conflate differences in representation-learning methods with additional input information and dataset-specific adaptation, limiting the conclusions that can be drawn about the relative merits of the methods themselves.

Questions for authors
  1. The main benchmark measures reference–variant separability without sufficiently establishing whether the detected differences reflect actual localization changes. Could the authors provide explicit localization-change annotations for a representative subset, specifying the reference and variant localization patterns and the observed spatial redistribution, supported by paired images? Please report the number of manually annotated positive and negative variants, variant-level AUROC or AP against these labels, and precision and recall of the calibrated hit calls. This would help distinguish genuine mislocalization detection from sensitivity to intensity, abundance, or other imaging phenotypes.

  2. The number of variants per reference protein is highly imbalanced. Could the authors report a reference-protein-balanced evaluation, for example by macro-averaging over reference proteins or using variant-balanced sampling? Do the main conclusions remain unchanged under such an evaluation?

  3. Could the authors provide controlled comparisons using the same input channels and matched MisLocus-specific training exposure across representations? When assessing the effect of fine-tuning, other experimental conditions should remain fixed. Do the main conclusions and representation rankings hold under these matched conditions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xUqL (part 1/2)

Thank you for a careful review. You identified a central question for this benchmark, whether reference-variant separability reflects protein localization shifts or a broader collection of changes. We now explicitly clarify that the strategy could capture broader impacts than just localization; we had oversimplified by using the term ‘localization’. It also motivated us to run some direct tests to validate that our strategy does, in practical terms, mainly capture localization changes. We implemented two controlled analyses: excluding the protein channel to confirm a major loss in sensitivity when relying on the remaining morphology-only channels, and confirming that mislocalization signals are retained even when analyzing the subset of cells with matched single-cell protein-channel intensity between reference and variant pairs to rule out changes in fluorescence intensity driving the results. Furthermore, we added independent validation of mislocalization against expert annotations and blinded visual inspections. We elaborate below.

Weakness 1 / Question 1. Do the detected differences reflect localization changes? We agree that changes in protein abundance or cell morphology can also make variant and reference cells distinguishable; we neglected to explain this, or include data about this, in the original version. To assess how much of the measured signal reflects relocalization, we include new analyses that examine the effects of the protein channel and protein intensity and compare the scores with independent manual localization annotations.

  • Protein-channel features show stronger agreement with external references than morphology-channel features. We compared protein-only (GFP), all-channel, and context-only features (DNA, AGP and Mito, protein channel removed) for CellProfiler, MorphEM and DINOv3. Context-only features still produce hits, but these carry little clinical signal (within-gene ClinVar AUROC 0.56–0.60), call at most 1 of the 9 expert positives, and have no significant power to distinguish HPA protein-localization patterns (0 of 21 locations significant, mean mAP_norm 0.013; Table 13). Protein-only features match or exceed all-channel features in clinical agreement in 6 of 6 comparisons (Appendix A.5.1, Table 7). The hits CellProfiler gains from the context stains include none of the expert positives (Appendix A.5.5), and adding context channels to fine-tuned SubCell raises its average score but not its clinical agreement (Table 12). Across tasks, the hits that matter for clinical and localization biology are protein-channel-driven.
  • Intensity is not the driver. Selecting a subset of reference and variant cells that are matched for protein intensity removes no more hits than a random subset of the same size (Appendix A.5.6, Table 14).
  • Expert annotations. We use the per-variant calls of Lacoste et al. (2024) for the 64 variants that both studies imaged: 9 mislocalized and 55 not. Table 9 lists each positive with its annotated reference and variant compartments, and Figure 9 shows the paired single-cell images. Every image representation detects 8–9 of the 9 positives, and the imaging phenotype score derived from each representation clearly separates the mislocalized vs unchanged variants (XGBoost AUROC 0.83–0.92, average precision 0.38–0.73 against 0.14 by chance). The precision of the hit calls is 0.22–0.33, because the representations also call variants the authors scored as unchanged (Section 4.1, Table 1 and 6).
  • Blinded annotation of a representative set. Three raters, blinded to variant identity, labels and model calls, judged whether the GFP localization pattern differed between reference and variant in paired images for all 162 variants of 24 held-out genes. Against the clear relocalizations identified by a majority of the three raters (27 labeled mislocalized of the 154 variants), learned scores reached AUROC 0.91–0.94 under XGBoost, and the hit calls recovered 78–93% of them at precision 0.32–0.66 (Appendix A.5.8, Figure 10, Table 15).

Still, many hits identified through computational approaches lack an expert-positive mislocalization label; this is expected. Some may reflect subtler changes that human eyes can’t identify, such as partial redistribution. Others may reflect altered abundance. The available annotations do not establish which is the case for these additional hits. Appendix A.3.1 now defines the measured phenotype explicitly, and we clearly state the limitations in the Discussion.

AuthorsOpenReview ↗

Response to Reviewer xUqL (part 2/2)

Weakness 2 / Question 3. Controlled comparisons for channels and training exposure. We added frozen SubCell with the same four input channels as fine-tuning (gene-balanced AUROC 0.784, against 0.808 fine-tuned), and we fine-tuned SubCell with two, three and four channels. The fine-tuning gain is consistent at every channel count (Table 12). To control training exposure, we retrained every model while withholding all cells from 24 genes (162 variants), and their performance on those genes is unchanged (Table 10). Table 1 now reports adapted, frozen and baseline settings separately. Under these matched conditions fine-tuning gives a modest, consistent gain, and the main conclusions hold. Rankings among the strongest representations are not stable under supervised scoring, because they agree with the labels within 0.10 AUROC (Table 1,2 and 7). For this reason the paper reports an evaluation profile rather than a single ranking.

Question 2. Reference-protein-balanced evaluation. We now average all scores by reference genes with gene-bootstrap confidence intervals, and clinical agreement is tested within genes. Table 5 compares pooled and gene-balanced values, and gene-balanced within-gene ClinVar AUROC is 0.71–0.83 for the image settings against 0.48–0.62 for trivial baselines. The main conclusions do not change under gene balancing: fine-tuned SubCell still scores highest under both XGBoost and PA, CellProfiler remains low under PA, and the trivial baselines stay near chance against ClinVar (Table 5).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (part 1/2)

We thank the reviewers for their thorough and constructive feedback. All reviewers recognized the importance of the biological task and the value of this standardized, replicate-aware benchmark. We have addressed their concerns through additional analyses and narrowed our claims where appropriate.

Specifically, the reviewers raised three shared concerns: (1) whether the detected differences reflect genuine protein mislocalization versus a broader set of image differences, (2) whether comparisons between representations are fairly controlled for input channels and training exposure, and (3) whether the benchmark is framed around explicit machine-learning tasks with properly calibrated mislocalization hit calling across replicate batches. In response, we expanded the MisLocus evaluation with independent validation of protein mislocalization, strictly controlled comparisons across input channels and training settings, and a formalized evaluation protocol. We appreciate this guidance to substantially strengthen the benchmark, and the revised paper now establishes three main results (our responses to individual reviewers elaborate on these).

  1. MisLocus captures real protein localization changes. We validated the detected phenotypes against three new independent references. First, we showed every image representation recovers 8–9 of the 9 mislocalizations that experts called in a separate study (Lacoste et al., 2024). Second, on 162 variants across 24 genes held out of MisLocus training, three image analysts–blinded to variant identity and model output–visually inspected single-cell images of these variants and their respective reference proteins for mislocalization events, and the computational mislocalization scores agreed closely with their visual annotations (AUROC 0.91–0.94). On the same held-out test images, models finetuned or retrained with the cell images from those proteins excluded, performed comparably to models trained with those proteins included. Third, within genes, mislocalization scores of the ClinVar pathogenic variants are significantly higher than benign ones. Crucially, the mislocalization signal persists after matching reference and variant cells with the same distributions of protein-channel intensity, and weakens when the protein channel is removed (Section 4.1, Appendix A.5), confirming that scores capture shifts in protein spatial distribution, instead of only changes in protein abundance and cell morphology, or technical artifacts such as fluorescence intensity.

  2. The benchmark reveals where representations actually differ. When we train a classifier to distinguish variant cells from reference cells using each representation’s features, the mislocalization scores from all 19 image representations agree with the expert’s visual annotation of mislocalization, and clinical significance from ClinVar database and protein sequence-based variant-effect predictors to a similar degree (within 0.10 AUROC), and above trivial baselines. Performance varies more across representations on every evaluation axis under unsupervised retrieval without supervised model training. The SubCell model fine-tuned on MisLocus leads in the agreement with experts’ mislocalization annotations, clinical significance, and localization structure. MisLocus now reports an evaluation profile against independent references rather than a single leaderboard of hit counts (Tables 1 and 2).

AuthorsOpenReview ↗

General response (part 2/2)

  1. MisLocus provides a clearer machine-learning evaluation protocol and calibrated hit calling. As requested by Reviewer sdcK to make the machine-learning challenges explicit, we formalized three standardized evaluation tracks: generalization to new replicate plates, generalization to unseen proteins (24 genes and 162 variants held out of MisLocus retraining), and robustness to variation in protein-channel intensity, benchmarked alongside new vision foundation models (DINOv3) and trivial intensity and cell-count baselines (Appendix A.3.7). Models retrained without the 24 held-out genes perform almost identically on those unseen genes as models trained on the full dataset, confirming that performance generalizes to unseen proteins. In addition, to address Reviewer az7x’s concern about multiple-testing inflation when calling discrete hits across replicate batches, we introduced a batch-adjusted threshold that calibrates false-positive hit rates on replicate-matched negative controls to near the nominal 5% level (3.6–4.2% under XGBoost, 3.8–7.6% under retrieval) while barely altering representation rankings across different evaluation tasks.

Summary of detailed revisions. Beyond these core additions, we made further methodological and presentation revisions in response to all three reviews: we matched input channel counts (2, 3, and 4 channels) between frozen and fine-tuned SubCell to verify that fine-tuning provides a consistent gain independent of channel count; we reported all metrics both pooled and macro-averaged across reference genes to prevent variant-count imbalance from skewing results; we separated adapted, frozen, and baseline models across all tables; and we refined our text to state training signals and empirical claims with precision. Together, these revisions strengthen the benchmark through additional analyses and narrower claims where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.