acceptodds
Withdrawn from ICLR 2027

UniCell: Unifying Cell Image Understanding and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Biological discovery increasingly relies on two complementary capabilities from microscopy images: understanding cellular phenotypes and generating images that simulate morphological responses to perturbations. Existing models treat these tasks in isolation, leaving their potential synergy underexplored. We introduce UniCell, a unified mixture-of-transformers model that accommodates microscopy understanding and perturbation-conditioned generation within a single architecture. This unified structure enables two synergistic designs: a two-stage training strategy, where the model first learns visual question answering and object localization and then transfers these representations to image generation; and text phenotype conditioning, where natural-language descriptions of expected morphological changes further improve generation quality. UniCell achieves state-of-the-art perturbation-conditioned generation and competitive performance on microscopy understanding benchmarks. We further show that understanding-first training directly improves generation quality, and that phenotype descriptions inferred from related perturbations further improve generation on unseen perturbations. Together, these findings establish joint understanding and generation as a viable paradigm for data-driven biological discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Unified Model, Generative Model, Microscopy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_o3ZZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper (UniCell) adapts the pretrained BAGEL model to microscopy images to combine visual question answering, object localization and image generation. It has understanding and generation modules within a shared backbone, which are trained sequentially -- understanding adaptation is performed first, and then generation adaptation is performed with understanding part frozen. The generator is conditioned on "phenotype-text" which is either inferred using understanding part or taken from external-VLM annotations checked on samples by experts.

Evaluation consists of two benchmarks for understanding (Micro-Bench and MicroVQA), one for localization (Micro-Bench), and one for generation (BBBC021). Understanding results nearly match results from finetuned Qwen2.5-VL-7B but are slightly overall worse, while outperforming selected models without benchmark-specific finetuning. For localization they show improvement over Qwen2.5-VL-7B and selected methods without benchmark-specific finetuning. For generation they show reported improvement compared to other methods on seen and held-out compounds.

Critical strengths
  1. The phenotype-text conditioning is a neat idea and shows improvement over baselines on both seen and held-out compounds across image-distribution and MoA-classification results. These gains are strongest empirical contribution (subject to clarification of baseline conditioning and evaluation details).
Critical weaknesses
  1. Limited methodological novelty and scope. The paper adapts an existing unified architecture through sequential finetuning and phenotype-text conditioning on different data domain. The integration between components (understanding and generation) is limited and one-directional: inferred phenotype descriptions and representations are used directly for generation, and generation doesn't feed back into understanding.

  2. Empirical evaluation leaves much to be desired

    • understanding + localization: Qwen2.5-VL-7B finetuned model is the only direct comparison which was also tuned on these datasets, but Qwen2.5-VL-7B is quite old, results should be shown for newer models like finetuned Qwen3.5/Qwen3.8 or Gemma4. Comparison to non-finetuned models is a useful complementary result, but they receive less benchmark-specific supervision, so improvement over them does not establish an advantage of the architecture or training method. Also non-finetuned baselines should be up-to-date, e.g. Gemini-3.8-Flash instead of Gemini-2.5-Flash, GPT-6-Astra instead of GPT-4o, Qwen3.8 instead of Qwen2.5-VL. These models are 1-2 years old. No repeats/cross-validation were reported and uncertainty estimates are not shown in the table. The custom 90/10 splits are not specified and don't have sufficient specification about image grouping, exact evaluation counts, or the ordering of splitting and answer-option rotation. Because multiple questions can share an image, inappropriate splitting could cause the leakage, so it doesn't establish whether evaluation measures generalization to unseen images or new questions about familiar images.
    • generation: there's only one generation dataset, involving one cell line and eight held-out compounds whose mechanisms are represented in training. Again no reruns and uncertainty estimates were reported.
  3. Insufficient evidence that unification/biological understanding explains the gains. The Stage 1 ablation supports the usefulness of understanding-first adaptation, but doesn't separate improvements in predicted phenotype descriptions from improvements in the internal representations used during generation, or establish an advantage over a modular approach. There are no ablations where generation uses conditioning (either text, or text+representations) from independent Stage 1 adaptation but finetuned from BAGEL directly, not from Stage 1. The experiments demonstrate an effective overall recipe more clearly than they establish benefits of unification.

  4. Insufficient biological validation of phenotype-conditioned generation. The paper doesn't systematically evaluate whether inferred descriptions for held-out compounds are correct or generated images show their predicted morphological changes. Expert validation covers only sampled training-annotation observations, rather than predictions. FID/KID and MoA classification are useful evidence but don't establish morphological correctness. This leaves the central phenotype interface insufficiently validated.

Questions for authors
  1. Generation comparison and reproducibility. What exact compound representations/conditioning did DDIB, IMPA, and CellFlux receive, including dose? How were held-out compounds represented? Specify configuration, checkpoint sources, training splits, and whether checkpoints were retrained or reused. An anonymous package containing generation/evaluation scripts and manifests would be perfect. These details would help verify the improvements and assess fairness.

  2. Understanding splits and independence. How were custom 90/10 splits selected? Were examples grouped by original image/source before splitting and answer-option rotation? Please provide image/question counts, split manifests, and overlap checks.

  3. Held-out-compound exposure. Were images or compound-specific phenotype annotations of the eight held-out compounds present in understanding-stage training? Was overlap with BBBC021's held-out images or conditions checked? Please clarify which parts of your pipeline exclude those compounds. This concerns evaluated images and phenotype-conditioning, rather than ensuring that the pretrained model has no general knowledge of these compounds.

  4. Baseline settings. Please provide prompts, inference settings (including Gemini 2.5 Flash's thinking budget), and handling of failed/truncated responses. Did you run anything across multiple seeds / cross-validation (including training where appropriate) to get any measure of uncertainty/noise of results? Do you have comparisons with newer VLMs (Qwen3.5/3.8, Gemini-3.8-Flash, GPT-6-Astra/6.1-Sol) or generation models such as CellFluxV2? An evaluation of a strong generation baseline, such as CellFlux, adapted to receive the same phenotype descriptions and explicit dose information as UniCell, would help separate the benefit of conditioning from the benefit of architecture and adaptation. The request is for representative evidence, not for all possible new published models/methods.

  5. Stage 1 and unification. For unseen compounds in Table 4, were descriptions/conditions generated separately by each variant, or generated once and reused across variants? Could you add to ablation a setup in which a Stage-2 generation model is trained from base BAGEL but descriptions/conditions are passed from Stage-1 (best same as in final model)? This would test if access to Stage-1 trained weights adds value beyond the phenotype-text part.

  6. Biological validation. Can you provide validation of the predicted phenotype descriptions for held-out compounds, and check whether generated populations show the predicted morphological changes?

Reviewer_X4w4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new framework UniCell, combining microscopy image understanding and conditional image generation within a single model. The authors build it using BAGEL's MoT architecture with a two stage training. First stage trains VQA answering and object localization components while the second stage focuses on training the generative part, conditioned on control images, perturbation embeddings and natural language phenotype descriptions. UniCell shows strong performance on microscopy understanding and improvements in conditioned generation on BBBC021 dataset additionally with an unseen perturbation generalization analysis.

Critical strengths
  • Interesting and well motivated overall framework. Combining visual understanding with image generation is a promising direction. The use of natural language is good because it can provide a natural interface for describing expected phenotypes and potentially interpreting the information used by the generator.
  • The paper reports strong results both on distributional metrics, MoA classification and visual understanding benchmarks.
  • Thorough experimental effort. I see the phenotype annotation pipeline as a standalone contribution to the field and appreciate authors' efforts on collecting expert defined attributes.
Critical weaknesses
  • For the generation quality concerns, the language-based understanding is not sufficiently justified. From an interpretability perspective, framework makes a lot of sense as it allows users to check and control the model through natural language. But for generation quality (although it clearly improves the generative performance), it is not easy to tell the design choices are sufficiently justified and if it can be achieved without these particular design choices (keeping a simpler alternative). For example, [1] incorporates pretrained biological representations through REPA-style diffusion alignment using OpenPhenom features. Although it operates on purely generative setting, an adaptation of its representation alignment strategy to the proposed flow matching framework would be a very good baseline.

  • The claims including visual understanding and semantic links are too bold given the experimental evidence. The VQA part is tested on the same questions that it is trained on. Given the extensive line of work studying VLMs' semantic failures outside their training distributions [2, 3, 4, 5], it ishard to claim that generation is improved by incorporating semantic understanding. An independent evaluation of the question answering part would be beneficial to check generalizability of the features. Authors can generate additional synthetic tasks for VQA by using the existing labels for this. Additionally, assuming that question answering part is not trained with the questions asking perturbations, an interesting study would be to test whether the learned transformer features can capture biological relationships between perturbations. For example, the authors could compare similarities between phenotype-text embeddings with similarities between real perturbation distributions (pairwise FID(p1, p2), computed using training images, could be one candidate).

  • Biological fidelity evaluation is limited. Although a perturbation classification study is presented, classifiers can be sensitive to spurious correlations. Therefore, generated images may reproduce spurious features associated with perturbations, giving the impression that they are faithful to the conditioning. FID and KID cannot easily detect that as the Inception network is trained using natural data. For this reason, I would strongly encourage the authors to include an independent morphological analysis using a tool such as CellProfiler.

  • Limited generation dataset and metrics. All generation experiments are on BBBC021 using 96x96 resolution from a single cell line and small number of perturbations. A second dataset involving different cell types or perturbation types would strengthen the evidence for the framework's generalizability (e.g., RxRx1, RxRx3, or Rohban5/12).

Questions for authors
  • Could the authors provide FID_o(training_real, test_real) and FID_c(traininig_real_c, test_real_c) for the empirical lower bounds. In the finite data settings these numbers are valuable and would help the readers understand if the pipeline is ready to be used under practical settings.
  • Could the authors provide memorization and generation diversity analyses?
  • Could the authors repeat generations with different random seeds for metric evaluations for uncertainty estimates?
  • Could the authors clarify Figure 2 in terms of which token streams are active across stages? In particular, it looks like MoT Joint Self-attention is mixing VAE control and noisy latent target no matter what but they are obviously not used in stage 1.

Minor:

  • The authors refer to reductions in FID as better generation error in multiple places. FID is a distributional distance metric and involves uncertainty errors in finite sample cases. Even when you split your data into two halves FID will be nonzero. Calling it a generation error I think is inaccurate.
  • I think it would be helpful to distinguish perturbation conditioned distribution generation ([1], [6]) from the settings where a control image is also provided to the generator, when discussing the baselines.

References:

[1] Demirel, Berker, et al. "MorphGen: Controllable and morphologically plausible generative cell-imaging." arXiv preprint arXiv:2510.01298 (2025).

[2] Fu, Stephanie, et al. "Hidden in plain sight: Vlms overlook their visual representations." arXiv preprint arXiv:2506.08008 (2025).

[3] Tong, Shengbang, et al. "Eyes wide shut? exploring the visual shortcomings of multimodal llms." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Rahmanzadehgervi, Pooyan, et al. "Vision language models are blind." Asian Conference on Computer Vision. Singapore: Springer Nature Singapore, 2024.

[5] Luo, T., Cao, A., Lee, G., Johnson, J. & Lee, H. Probing Visual Language Priors in VLMs. Proceedings of the 42nd International Conference on Machine Learning, PMLR, 2025.

[6] Navidi, Zeinab, et al. "Morphodiff: Cellular morphology painting with diffusion models." International Conference on Learning Representations. Vol. 2025. 2025.

Reviewer_epQcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UniCell, a unified framework for microscopy image understanding and perturbation-conditioned Cell Painting image generation. The proposed approach combines understanding-first training with natural-language phenotype descriptions to improve the generation of cellular morphological responses, particularly for unseen perturbations.

Critical strengths

The paper is generally well-written, well-motivated, and addresses an important problem in Cell Painting image generation. The proposed integration of visual understanding and phenotype-guided generation is interesting, and the empirical results demonstrate substantial improvements over existing baselines in both image quality and biological fidelity.

Critical weaknesses

Insufficient evidence of visual grounding and its contribution to generation. The authors associate strong performance on Micro-Bench following supervised fine-tuning (SFT) with improved visual understanding. However, prior work has shown that high VQA accuracy does not necessarily indicate strong visual grounding, as models may exploit linguistic priors or dataset-specific shortcuts [1,2]. Similar concerns have been raised for microscopy VQA benchmarks [3]. Although Table 4 shows that understanding-first training improves generation quality, it remains unclear whether these gains arise from better visually grounded representations or simply additional supervised training.

Insufficient validation of the think-then-generate strategy. Related to the above, the authors attribute improvements on unseen perturbations to biological reasoning over a reference library of related compounds. However, the current experiments do not sufficiently isolate the contribution of the proposed reasoning mechanism from the additional phenotype information available in the reference library. In particular, it remains unclear whether the gains arise from meaningful biological reasoning or simply transferring phenotype descriptions from chemically similar compounds.

Limited empirical validation and baseline comparisons. The generation experiments are conducted exclusively on BBBC021, involving only 26 seen and 8 unseen compounds from a single cell line, which limits the evidence for generalizability. The authors should evaluate UniCell on more diverse datasets, such as RxRx1/RxRx3 [4] for genetic perturbations and JUMP [5] for chemical and genetic perturbations. Furthermore, the comparisons omit some relevant approaches, e.g., MorphoDiff [6].

Reliability of phenotype descriptions. UniCell relies heavily on VLM-generated phenotype descriptions to guide image generation. However, Appendix A.4 reports only 63.1% exact-state agreement between VLM annotations and expert judgments, raising concerns about the reliability of these conditioning signals. Moreover, the authors do not directly evaluate the accuracy of inferred phenotype descriptions for unseen perturbations or whether the generated images faithfully reflect these descriptions.

Minor weakness

Missing related works: The authors should better discuss recent work incorporating natural-language descriptions and text-guided representation learning for cellular microscopy [7,8].

Reference


  1. Shrestha, Robik, Kushal Kafle, and Christopher Kanan. "A negative case analysis of visual grounding methods for VQA." ACL. 2020.
  2. Goyal, Yash, et al. "Making the v in vqa matter: Elevating the role of image understanding in visual question answering." CVPR. 2017.
  3. Burgess, James, et al. "Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research." CVPR, 2025.
  4. Sypetkowski, Maciej, et al. "Rxrx1: A dataset for evaluating experimental batch correction methods." CVPRW, 2023.
  5. Chandrasekaran, Srinivas Niranj, et al. "Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations." Nature Methods 2024
  6. Navidi, Zeinab, et al. "Morphodiff: Cellular morphology painting with diffusion models." ICLR 2025
  7. Lu, Mingyu, et al. "CellCLIP-Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning." NeurIPS 2025
  8. Alrubyli, Yazeed, and Alessandro Bevilacqua. "CellVLM: Hierarchical Text-Guided Vision Transformers for Cell Segmentation and Classification." IEEE, 2025.
Questions for authors

Why does adding phenotype text at inference yield negligible FID improvement (6.92 vs. 6.93, Table 3)? Does this suggest that the model does not effectively leverage phenotype descriptions during inference?

Reviewer_ACukOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces UniCell, a unified framework for microscopy image understanding and perturbation-conditioned image generation. This method is built upon BAGEL's mixture-of-transformers architecture and employs a two-stage training procedure: microscopy understanding tasks are learned before adapting the model for image generation through flow matching.

The paper reports competitive performance on microscopy understanding tasks and substantial improvements over existing generative baselines on BBBC021, particularly when incorporating phenotype descriptions.

Critical strengths
  • Unifying microscopy understanding and generation is a meaningful research direction, and using interpretable morphological descriptions to connect these capabilities is appealing.

  • UniCell achieves substantial improvements over the evaluated baselines on BBBC021, especially in terms of FID and mechanism-of-action classification accuracy.

  • The approach is well motivated, and the methodology is generally understandable. The authors provide useful implementation details and several ablation experiments, including investigations of phenotype conditioning and understanding-first training.

Critical weaknesses
  • The main limitation is that perturbation-conditioned generation is evaluated exclusively on BBBC021, using a single train/test partition with only eight held-out compounds. Although two datasets are used for microscopy understanding, this does not establish the generalizability of the generative framework across different cellular contexts, imaging modalities, or perturbation spaces. The paper does not report results across multiple training seeds or alternative perturbation splits, nor does it provide uncertainty estimates or statistical significance tests. Given the relatively small number of held-out compounds, this makes it difficult to assess the robustness of the reported improvements.

I consider stronger validation essential. At minimum, I would expect multiple compound-level splits, repeated runs, and appropriate statistical analyses. Evaluation on an additional dataset, such as RxRx1, for instance, would considerably strengthen the paper.

  • The proposed understanding-first training procedure introduces an interesting inductive bias: learning to understand cellular morphology should facilitate subsequent generation. However, its optimality and underlying mechanism are not sufficiently investigated. Table 4 demonstrates that including the understanding stage improves generation performance, which is useful evidence. Nevertheless, this comparison does not fully disentangle the benefits of task-specific understanding from additional microscopy supervision, training compute, or generic domain adaptation. Comparisons against alternative training strategies, such as joint or reversed-stage training, or a compute-matched generation-only adaptation, would help establish whether the proposed curriculum is genuinely advantageous.

  • The text-conditioning mechanism is one of the most interesting aspects of the paper, but its evaluation leaves several important questions unanswered. For unseen perturbations, the model predicts phenotype descriptions by comparing chemical structures with a reference library. However, chemical similarity does not necessarily imply similarity in biological activity, and the paper does not establish that the proposed reasoning procedure is superior to simpler retrieval- or similarity-based approaches. The authors do not adequately isolate the contribution of the two-pass reasoning strategy itself. A comparison against direct phenotype prediction without the intermediate reasoning step, as well as a simple chemical-similarity-based phenotype transfer baseline, would be informative.

  • The generative baselines include DDIB, IMPA, and CellFlux. While these are relevant approaches, the comparison would benefit from more recent methods -- e.g. SimuSDB (CVPR 2026) and CellFluxV2 (2026 preprint; seem very closely relevant and would provide valuable additional context, although I recognise that comparison with non-peer-reviewed preprints is not mandatory). BAGEL provides a substantial pretrained understanding-and-generation backbone. It is therefore difficult to determine how much of UniCell's advantage arises from the proposed training and conditioning strategies rather than the capabilities of the pretrained model itself. Finally, on the understanding side, UniCell remains competitive but underperforms fine-tuned Qwen2.5-VL-7B on both VQA benchmarks, particularly MicroVQA. Consequently, the evidence more strongly supports improved generation while preserving reasonable understanding performance than a general improvement across both capabilities.

Questions for authors
  • Q1: Can the authors provide results across additional compound-level train/test splits and multiple random seeds, with standard deviations or confidence intervals and appropriate statistical comparisons? Can they evaluate the framework on a second perturbation-imaging dataset?

  • Q2: How does the proposed sequential curriculum compare with joint training or a compute-matched generation-only adaptation of BAGEL? Can the authors clarify which aspects of the understanding stage are primarily responsible for the improvements?

  • Q3: How accurately do the inferred phenotype descriptions match experimentally observed morphological changes for unseen compounds? Does the two-pass reasoning procedure outperform direct prediction or a straightforward chemical-similarity-based retrieval baseline? An oracle experiment using observed phenotype descriptions would also help quantify the limitations of the inference component.

Overall, I think the method is interesting, and it's got potential, but I am not at all persuaded by the soundness of the evaluation. I shall be happy to reconsider my score should the authors address my concerns.

Comments on the paper

3 comments

(1/3)

Dear Reviewer,

Thank you very much for your careful reading of our submission and for your detailed and constructive comments. We genuinely appreciate the time and effort you invested in identifying the limitations of our evaluation, the missing analyses, and the claims that were not yet sufficiently supported.

Given the extent of the revisions required, we have decided to withdraw this version and substantially improve the work. In particular, we plan to:

  • Clarify the scope of our claims: we present understanding-first training as an effective training recipe, rather than claiming that improved visual understanding causes better generation, and we position the understanding results as competitive rather than superior.
  • Add implementation details, including data splits, baseline configurations and conditioning, and inference settings, to ensure full reproducibility.
  • Evaluate generation on multiple seen/unseen compound splits with repeated runs and uncertainty estimates, and, if feasible, on additional datasets (e.g., RxRx1, JUMP). However, extending to these datasets is non-trivial. On BBBC021, one Stage 2 training run already takes about 150–200 H100 GPU-hours per split, and these datasets have far more perturbations. The phenotype text also costs human time: every perturbation needs a description that biology experts must review, so this effort grows with the number of perturbations.
  • Assess biological fidelity with CellProfiler-based morphological analyses.
  • Compare the reasoning step with direct phenotype prediction and similarity-based retrieval, and validate the inferred descriptions for unseen compounds against their observed phenotypes.
  • Provide further evidence of the effectiveness of phenotype text, including whether generated images reflect the conditioning descriptions.
  • Add more recent generative and vision-language baselines.
  • Validate and refine the VLM annotation process with biology experts to improve the accuracy of phenotype annotations.
  • Conduct comprehensive ablations on how the understanding and generation components should be trained, disentangling the effects of understanding-first training, additional supervision, and the pretrained backbone.

Results already obtained (BBBC021, three held-out compound folds)

Several of these items are already done. We re-ran the BBBC021 experiments on three disjoint held-out compound folds (each holds out 8 of the 34 compounds, 24 held out in total) and report mean ± std over folds. One MoA classifier scores every method and fold. We trained it with the official CellFlux script on the training-split images of all 34 compounds and score only test images; the generator never sees the held-out compounds. The submitted version used the released CellFlux classifiers, which are tied to a single split; the one for unseen compounds has 7 classes (the 6 MoAs of that split's 8 held-out compounds, plus DMSO). Our classifier has 13 classes, all 12 MoAs in BBBC021 plus DMSO, which covers the MoAs of every split, so a single classifier scores all splits. With more classes to choose from, it is a harder test: on fold 1, real unseen images score 73% with it and 88% with the 7-way released classifier. Unseen MoA accuracies are therefore lower than in the submitted version, and the numbers are not directly comparable.

Generation vs. baselines. We trained DDIB, IMPA and CellFlux on each fold and scored all methods with the same metric code. Our model is best on all metrics in both splits, reducing FID_o by 62% on seen and 30% on unseen compounds relative to CellFlux. CP-Δr (×100) is the correlation, in CellProfiler features, between the generated and the real morphological change from control; for real images, it is the correlation between two halves of the real images, an upper bound. Real images are shown as a reference and are not bolded.

SplitMethodFID_o ↓KID_o (×10²) ↓MoA acc ↑MoA macro-F1 ↑CP-Δr ↑
SeenDDIB56.89 ± 1.104.29 ± 0.2538.87 ± 4.4633.00 ± 3.5163.13 ± 2.71
SeenIMPA25.20 ± 8.861.88 ± 0.8560.07 ± 2.0353.49 ± 3.7476.59 ± 2.72
SeenCellFlux19.48 ± 2.501.64 ± 0.2460.37 ± 1.4553.81 ± 0.2978.10 ± 1.75
SeenOurs7.38 ± 1.070.33 ± 0.1264.88 ± 0.6459.24 ± 1.0184.96 ± 1.26
SeenReal images––72.68 ± 1.4068.47 ± 0.7093.48 ± 0.30
UnseenDDIB106.31 ± 26.207.69 ± 3.1619.22 ± 17.5718.56 ± 13.8137.27 ± 21.19
UnseenIMPA64.23 ± 10.404.51 ± 0.6121.96 ± 6.2121.75 ± 4.6850.35 ± 2.41
UnseenCellFlux32.21 ± 4.542.17 ± 0.4431.01 ± 5.4629.75 ± 4.9562.96 ± 9.59
UnseenOurs22.64 ± 6.321.17 ± 0.5937.78 ± 10.5238.41 ± 8.7667.60 ± 6.62
UnseenReal images––74.07 ± 5.0474.74 ± 5.3493.64 ± 1.08

(2/3)

In CellProfiler features, our model reproduces the real morphological change better than every baseline in every fold, for both seen and unseen compounds. Per compound, it beats DDIB and IMPA for at least 32 of 34 seen and 22 of 24 unseen compounds, and CellFlux, the strongest baseline, for 28 of 34 seen and 16 of 24 unseen compounds; all of these differences are significant (Wilcoxon p < 0.05).

Simple retrieval for unseen compounds. The submitted version inferred the phenotype description of an unseen compound with in-context reasoning by the model. We now show that a simpler step works better: retrieving the description of the most structurally similar seen compound (Morgan fingerprint top-1). With the generator unchanged, retrieval gives lower FID_o, higher MoA accuracy and smaller variation across folds than in-context reasoning. When the MoA is known, restricting retrieval to same-MoA compounds improves generation further; this oracle row is shown in the last row and is not bolded. Every row gives the generator the compound's Morgan fingerprint and dose, and None removes only the text. The text barely changes FID_o (22.30 vs. 22.64, within the std over folds) because FID_o pools all images and measures overall realism, not whether each compound shows its own phenotype: even the description of each compound's own real images gives 22.46. The effect of the text shows in MoA accuracy, which rises from 28.78% to 37.78%.

Description for unseen compoundsFID_o ↓KID_o (×10²) ↓MoA acc ↑MoA macro-F1 ↑
None (fingerprint + dose only)22.30 ± 5.331.11 ± 0.4928.78 ± 7.2429.53 ± 5.12
In-context reasoning (submitted version)27.74 ± 14.391.82 ± 1.5932.85 ± 12.7133.85 ± 11.61
Structure retrieval (ours)22.64 ± 6.321.17 ± 0.5937.78 ± 10.5238.41 ± 8.76
Same-MoA retrieval (MoA known)21.88 ± 6.461.08 ± 0.6040.19 ± 13.9241.09 ± 13.73

We also checked the retrieved descriptions against each held-out compound's observed description, taken from its own real images in the folds where it is seen. On the 11 compounds whose observed description reports a change, the retrieved description is as useful for generation as the observed one: MoA accuracy on these compounds rises from 28.1% without text to 52.7%, vs. 47.0% with the observed description (no significant difference). The Jaccard overlap of their stated changes (e.g., nuclear size increased) is 0.29, about 70% of the overlap reached by a same-MoA donor, which requires the true MoA (0.41), and seven times that of a different-MoA donor (0.04).

Understanding-first training. We trained the same generator, with the same data and backbone, without Stage 1 (understanding training). Stage 1 helps across the board (seen CP-Δr is tied), but most significantly where the generator must generalize: on unseen compounds, FID_o falls by 20% and MoA accuracy rises, both in every fold.

SplitTrainingFID_o ↓MoA acc ↑MoA macro-F1 ↑CP-Δr ↑
SeenStage 2 only7.80 ± 1.1163.33 ± 2.2257.13 ± 1.8284.96 ± 0.92
SeenStage 1 → 27.38 ± 1.0764.88 ± 0.6459.24 ± 1.0184.96 ± 1.26
UnseenStage 2 only28.17 ± 10.9735.44 ± 10.6636.46 ± 9.6566.52 ± 9.07
UnseenStage 1 → 222.64 ± 6.3237.78 ± 10.5238.41 ± 8.7667.60 ± 6.62

(3/3)

Phenotype text. We experimented with changing only the description given to the trained generator at inference time, keeping the compound's fingerprint and dose fixed. In particular, we consider changing the description to that of a decoy, where the decoy is the most structurally similar seen compound with a different MoA. Own is the description annotated from the compound's own real images. → Decoy MoA is the share of images classified as the decoy's MoA.

SplitDescription givenMoA acc ↑→ Decoy MoA (%)CP-Δr ↑
UnseenNone28.78 ± 7.247.40 ± 3.2165.24 ± 6.41
UnseenRetrieved (ours)37.78 ± 10.528.51 ± 0.5367.60 ± 6.62
UnseenDecoy (different MoA)21.53 ± 10.6723.39 ± 8.3560.33 ± 6.46
SeenNone62.53 ± 0.822.90 ± 0.4884.28 ± 1.04
SeenOwn (ours)64.88 ± 0.642.73 ± 0.2584.96 ± 1.26
SeenDecoy (different MoA)44.88 ± 1.5114.42 ± 1.0176.70 ± 1.03

The text adds information that the fingerprint lacks: removing it lowers MoA accuracy for 18 of 24 unseen and 23 of 34 seen compounds. The generator follows its content: a decoy description shifts the images toward the decoy's MoA for 22 of 24 unseen and all 34 seen compounds. All of these changes are significant (Wilcoxon p < 0.05).

The text is also finer-grained than the MoA label. In a separate experiment on seen compounds, not shown in the tables, we gave each of 17 seen compounds the description of its nearest same-MoA donor, a compound with the same MoA but a different description. On these compounds, the MoA classifier detects no change (73.96% vs. 70.09%, not significant), because the MoA stays the same. The images still move toward the donor: KID to the donor's real images falls by 34% (4.98 to 3.31; 15 of 17 compounds, Wilcoxon p < 0.05), and KID to the compound's own real images rises from 0.86 to 1.41.

Your feedback has been extremely helpful and will directly guide the next iteration of this project. Thank you again for your thoughtful review.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.