acceptodds
Withdrawn from ICLR 2027

MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language–Image Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined concepts from closed-set data, limiting transfer across tasks and biological contexts. In this work, we introduce MicroCLIP, a vision–language model that learns generalizable cellular and subcellular representations from natural language supervision for open-set image classification and retrieval. To pretrain MicroCLIP, we construct MicroPool-2M, a dataset of approximately 2.5 million fluorescence microscopy image–text pairs spanning diverse cellular structures, cell lines, and imaging conditions. Aligning microscopy images with biological concepts in natural language is challenging because most images lack descriptive captions and biological knowledge annotations. We therefore progressively establish image–text associations at three levels to provide MicroCLIP with richer semantic supervision: (1) category-level alignment links images to standardized labels defined by subcellular structure–cell-line combinations; (2) knowledge-level alignment incorporates biological priors derived from these labels to capture relationships across species and cell types; and (3) detail-level alignment uses a multimodal large language model to recaption and verify image–text pairs at scale, emphasizing fine-grained visual characteristics. Extensive experiments demonstrate that MicroCLIP achieves leading performance in zero-shot classification and text–image retrieval on microscopy images. It retrieves images with specific cellular characteristics from thousands of candidates and generalizes to unseen biological concepts without additional supervision. These results highlight MicroCLIP’s potential as a foundation model for fluorescence microscopy. Models, training code, and datasets will be publicly released.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI4Science, Fluorescence Microscopy, Vision-Language Learning, Zero-Shot Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5oXdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MicroPool-2M, a dataset of about 2.5M single-channel fluorescence microscopy images labeled by (structure, cell line). Each image has three types of text: a category template, GPT-generated knowledge, and a detailed caption generated by Qwen using both the image and its label. MicroCLIP fine-tunes DFN2B-CLIP ViT-B/16 with standard InfoNCE. Training samples one caption type per image, uses MetaCLIP-style balancing, and includes 0.6M natural image-text pairs. The paper reports large gains over general-purpose CLIPs in zero-shot classification across held-out splits and in image-text retrieval.

Critical strengths

The dataset is sizable. Holding out compositions, cell lines, structure sets, and sources makes the evaluation useful for examining generalization. The training recipe is also clearly specified and follows established CLIP data practices.

Critical weaknesses
  1. My main concern is the lack of evaluation and justification for the synthetic captions, since the three-level caption supervision is the paper's central contribution.

    The abstract says that the captions are verified, but the paper gives no verification procedure. For the detailed captions, there is no human audit, hallucination rate, or diversity analysis, and the examples are highly templated. The GPT-generated knowledge captions have no reported factual checks either.

    Label leakage is another concern. The generator receives the category name, and the examples repeat that label in the caption. Without a comparison between label-conditioned and image-only generation, I cannot tell whether the gains come from visual descriptions or from the labels.

    The choice of Qwen3.5 also needs justification. It is a general-purpose MLLM, and both the paper's Section 3 and prior work in μ-Bench show that such models are unreliable on microscopy images. The captions may therefore contain little visual detail or include hallucinations. The paper compares neither alternative MLLMs and model scales nor domain VLMs. It also does not compare the knowledge captions with curated sources such as UniProt or Cellosaurus.

    Finally, retrieval queries come from the same generator as the training captions. The retrieval results therefore partly measure how well the model fits that generator's style.

  2. The baseline comparisons omit several relevant microscopy-capable CLIPs: BiomedCLIP, BMCA-CLIP, CellCLIP, and MorphoCLIP. CellCLIP is cited but not evaluated. The general-purpose CLIPs used as baselines have never seen cell-line names and score near chance. Category-only fine-tuning is the fair reference here, with a Compositional score of 56.47 compared with 67.70. The paper also needs fine-tuned baselines, such as BiomedCLIP or BMCA-CLIP trained with the same recipe, and a factorized supervised classifier.

  3. The evaluation protocol inflates the results. Classification ranks candidates only within each subset, rather than using generalized compositional evaluation. The retrieval baselines score 0.0 to 0.7% R@1, close to the 0.1% chance level. There is no evaluation on an external benchmark such as μ-Bench and no check for catastrophic forgetting. It is also unclear whether the splits are grouped by field of view or plate.

Questions for authors

Results that need clarification

  • Retrieval without auxiliary data is 39.4 in Table 2, 32.4 in Table 3, and 60.1 in Table 4.
  • Cell-DINO scores 0.98% on Compositional Shift, below the 1.56% chance level for 64 classes.
  • Adding data hurts performance on Cell-Line and Source Shift. The full recipe is not the best on Source Shift, scoring 36.53 compared with 40.59.

Other questions

  1. How was caption quality verified? Please report a human audit, hallucination rate, label-mention rate, and diversity.
  2. How do the results change with image-only detailed captions, other caption generators, or curated knowledge sources?
  3. How do BiomedCLIP and BMCA-CLIP perform both zero-shot and after fine-tuning with the same recipe?
  4. What are the results under generalized compositional evaluation and on μ-Bench?
  5. How are duplicate texts within a batch handled, and which configuration does each row of Tables 2 to 4 use?

I would be willing to raise my score to 3 if the authors could solve my concerns.

Reviewer_ehcQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MicroCLIP, which is a CLIP-style encoder for microscopy data (specifically in the immunofluorescent imaging domain) that can generalise to two sets of biological concepts unseen during training: 1. Subcellular structure or marker type, and 2. Cell line.

The authors propose three contributions.

  1. the identification/observation that existing open set methods using general purpose image-text encoders, and also general purpose MLLMs such as GPT-6 fail to generalise to unseen structures and cell lines out of the box. This motivates the need for a domain specific encoder that can generalise as such.
  2. the 2.5M image-text dataset and benchmarking scheme. For each microscopy image there are 3 levels of captions, where first is the ground truth (cell, structure) pair parsed as text, and the second is a knowledge distillation through an LLM on what the cell and structure properties are, and the third being text describing the visual properties of the image, where the image is shown to an MLLM (Qwen3.8) and a detailed visual description is obtained.
  3. the encoder trained using CLIP loss. In addition to the original method, the authors implement random sampling of text levels as supervision, composition aware sampling, and adding auxillary general image-text data.

The authors show that their encoder (MicroCLIP) performs better than the comparative methods on both ID and OOD tasks, where the OOD tasks are generalisation to unseen cell and structure pairings.

Critical strengths

Motivation and general results: The authors have motivated this work well, and the writing is clear. The propsed encoder MicroCLIP can generalise between unseen cell lines or structures. This is a good step in the direction of designing encoders for microscopy data, where generally, models cannot identify concepts across cell lines. However, please see below (weaknesses) on why I think this is not argued well in the paper.

Dataset: The introduction of the MicroPool dataset is a good contribution as this can act as a multi cell-line and multi-structure dataset which can later be utilised for microscopy image pre-training and as a standard benchmark to evaluate the generalisation to cell line and structure.

Methodological components: The grouped application of data balancing, detail and knowledge level captions, and use of general purpose data (VLM-0.7M) shows improvement over category-level only labels.

Critical weaknesses

Impact of the work: The authors have correctly motivated the problem by mentioning that existing works do not consider the variation of physical conditions (i.e.: sub-cellular structures and cell lines beyond the training set). However, why this is and important problem for microscopy is not clear to me. Without this argument, one may simply view this as, "we have label sets A1 and B1 belonging to concepts A and B in domain XX, so how do we build a model that can generalise to A2 and B2 within that domain, where XX can be any domain?". This now gets reduced to a standard composition learning problem, and therefore should be compared with methods in Section 2: Generalization Evaluation.

Missing implementation details: Details about the baseline fully supervised encoder are not mentioned; if it is from a pre-trained checkpoint or not. It would be unfair to train MicroCLIP starting from DFN2B-CLIP and the baseline fully supervised from scratch. Model type and parameter count are also not mentioned. Given that it is the second best perfoming method on IID (also by a margin of less than 1%), this information is necessary. Training time should also be considered as the fully supervised method only trains a vision encoder, while MicroCLIP trains both image and text encoders.

Weak baselines: See above (missing implementation details) for the fully supervised classifer. However, even if they were fine-tuned from pre-trained model, as both Cell line and Structure can be seen as 2 different label sets, it would make more sense to have 2 prediction heads for the fully supervised method also. Why the Cell-DINO is probed like that, but the fully supervised is not trained like that, is suspicious to me. Also, the fully supervised method and Cell-DINO can be evaluated on OOD using a kNN classifier on the feature maps or cls tokens. Why this is not performed is unclear. Furthermore, the Open-set methods have not seen any data from MicroPool, therefore it is an unfair comparison. It would be fair to atleast train projection heads on those models using MicroPool.

The only fair comparison appears to be the fully supervised classifier, where the implementation of that is also not mentioned, and can be argued as not fair. For a truly fair comparison, a fully supervised classifier with data balancing will have to be implemented and compared with MicroCLIP without VLM-0.7M supervision. This would highlight if the detail level captions and knowledge level captions (which are the main contributions of this paper).

Questions for authors

1- What is the argument for unseen structure and cell line generalisation for microscopy? For example, the generalisation across Mechanism-of-Action (MoA) of chemcial compounds which cells are perturbed with, or even the compound structure itself, have valid generalisation arguments with respect to drug discovery, where relationships between cell morphology and compounds/MoA have to be modelled. Similarly, what is the argument for structure and cell line?

2- What happens when both label sets are unseen during training? are there methods in other domains that account for all label sets being unseen during training? Although this makes little sense in microscopy (i.e.: predicting a nucleus in a new cell line without ever seeing a nucleus before is infeasible), it would be interesting to see if the performance is random, or if it is closer to the current OOD metrics than we think it is. We can predict two nucleus images to be clustered together in the feature space of a trained encoder although nucelus hasn't been seen in training.

3- Why is (c, s) viewed as a joint label for supervised fine-tuning? Why cannot it be viewed as separate labels? In reality, they are independent identities and should be treated as such.

4- Why are methods like CellCLIP (NeurIPS '25) and CP-CLIP (from CP-Agent, ICLR '26) not compared here?

Reviewer_Wok8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper describes a dataset that pairs microscopy images of cells with natural language caption descriptions, and uses it to train a CLIP-like model.

Critical strengths
  • A dataset collected from public sources.
  • A model and experimental evaluation.
  • Well conducted experiments.
Critical weaknesses
  • It is unclear why language is necessary or at least useful for describing microscopy images. Microscopy images of cells are carefully acquired scientific measures of cellular activity and not pieces of art that need to be searched with language. Language is imprecise, subjective, and incomplete to describe the full quantitative and biological richness of cellular images.
  • The annotations used to generate natural language captions comes from well documented and routine metadata that is known in biological studies. It does not need to be predicted or expanded with natural language because all of these properties are chosen in an experiment. Natural language descriptions are not needed. What's needed is quantitative metrics that report phenotypic variations.
  • The observation that multimodal models fail at interpreting cellular images is artificial. This does not reflect a real world case application of how biologists use microscopy images.
  • The paper does not present any new technical innovation. The CLIP framework for representation learning is used with minor adaptations to support the artificially created tasks. Predicting cell line or what cellular compartment is observed in the image are not relevant in biology. Again, these choices are known before the images are acquired and the images can be searched deterministically using metadata.
  • The experimental design leaves out real downstream biological tasks. It is not clear that this model helps with interpreting images in a new experimental setting for a real biological task. For instance, can the representations learned by this model be used to predict protein localization or mechanism of action? Benchmarks such as HPA, JUMP-CP, and CHAMMI have made efforts to collect biologically meaningful annotations for real world biological challenges, and are not evaluated in this work.
Questions for authors

See weaknesses above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification on Biological Motivation and Additional Downstream Evaluation

We thank the reviewers for their careful assessment of our work. We would like to clarify the intended role of language and provide additional evidence connecting our approach to biological applications.

Biological motivation and the role of language.

We agree that microscopy images contain rich quantitative information. Our aim is to explore whether language can serve as a complementary interface that helps researchers specify visual features they wish to retrieve. For example, knowing the cell line and staining target does not determine the morphology or state of every individual cell. A description such as “condensed chromatin” can express a visual criterion that acquisition metadata may not directly capture. This motivates our exploration of joint image–text representations.

Additional downstream evaluation.

The reviewers’ comments highlight the importance of distinguishing between evaluating generalization across categories and their combinations and addressing biologically meaningful tasks. To provide additional evidence beyond the original recognition benchmarks, we conducted downstream experiments on protein localization, cell-state recognition, and cross-protein retrieval, alongside classification of projected 3D cell images. The results are summarized below.

MethodProtein localization (mAP ↑)Cell-state recognition (Macro-F1 ↑)3D classification (Accuracy ↑)Cross-protein retrieval (mAP ↑)
Fully supervised———69.45
Cell-DINO———48.66
CLIP (OpenAI)49.3162.6112.5065.21
SigLIP56.4654.4322.5068.67
SigLIP 253.2762.3418.7567.67
PE-Core64.3860.0441.8869.57
DFN2B-CLIP59.4960.3917.5069.11
BiomedCLIP50.6858.2434.3867.32
BMCA-CLIP59.0357.7348.1269.29
CellCLIP60.7647.3723.1270.99
MicroCLIP77.6567.4188.1273.19

All values are percentages; “—” indicates an unevaluated setting. Protein localization and cell-state recognition use text-encoder-only fine-tuning, while cross-protein retrieval uses frozen visual features. The 3D task classifies 2D maximum-intensity projections using text prompts.

These results complement our motivation for using language as an interface to microscopy images by demonstrating that the learned representations also transfer to protein localization, cell-state recognition, and cross-protein retrieval. They provide additional evidence connecting our representation-learning approach to the practical biological applications raised by the reviewers.

We appreciate the reviewers’ feedback, which has helped us clarify both the motivation and the scope of the evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.