- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CPGENeval, an evaluation framework for Cell Painting image generation, to systematically select how multi-channel fluorescence images are rendered to RGB and which VAE encodes the RGB images into a latent space. The design is a factorial benchmark: seven rendering protocols (RECURSION, IMPA, MORPHGEN, MORPHDIFF, CELLCLIP, CELLFLUX, FREEBRIDGE) are crossed with five frozen natural-image VAEs (SD1.4, SD ft-MSE, SDXL, SD3.5, FLUX.1) over two datasets (HUVEC512, IMPA96). The study shows that the conventional reconstruction FID (rFID) and generation FID (gFID) are deficient for evaluating rendering protocols and VAEs, in that they do not reflect the original cellular features, and accordingly proposes native readout and retrieval as more reliable complementary measures. The paper also examines how latent-space and pixel-space generation differ in training cost and retrieval performance, finding that under a sensible rendering, latent-space generation not only shortens training time substantially but also attains better retrieval performance.
Critical strengths
The paper names a blind spot the field has been living with. FID is not only pursued blindly as a target; the score itself moves with the rendering, and it is decoupled from the biological features the images are meant to carry. Quantifying that rather than asserting it is what this paper does.
The sweep is broad enough to carry the claim: seven published rendering protocols crossed with five frozen VAEs on two datasets, with both the reconstruction stage and the generation stage covered under one design.
The paper does not stop at criticism. Native readout and retrieval give practitioners something to act on, and the guidance that follows from the experiments is concrete — report the rendering alongside every FID, fix it when comparing models, and check it against a biological readout. For a field that compares numbers across papers, this is directly usable.
Critical weaknesses
-
The scores and the VAEs all sit in a natural-image feature space. rFID and gFID use frozen Inception features, trained on natural images and insensitive to cell morphology; would the ranking changes reported here survive if they were recomputed with a microscopy-pretrained encoder? Retrieval does not settle the question, since mAP is computed over the same features, and the disagreement between gFID and retrieval is then one between two aggregations of a single representation — "gFID can favor weaker siRNA retrieval" need not hold in a biological feature space. Section 4 states that finding without this qualification. The frozen VAEs are natural-image models too, which the limitation already concedes.
-
Native readout audits reconstruction but not generation. Section 3 uses it to show that rFID fails; Section 4 then evaluates generation with Inception features alone. Is that because generated samples make the ridge regression unreliable? If so, the generation-side conclusions still rest on the space the paper argues against. Mapping CellProfiler profiles of generated samples into a library built from real images, and checking whether they retrieve the right siRNA perturbation, would give a readout outside Inception space without needing a paired ground truth.
Questions for authors
-
Would the reversals reported in Section 4 survive if gFID and mAP were recomputed with an encoder pretrained on microscopy images, or with CellProfiler features?
-
Is native readout left out of Section 4 because generated samples have no paired native image, or because the ridge reader is unreliable on generated images? If the latter, how should the generation-side conclusions be read?
-
The reversal looks established on IMPA96 — the significant contrasts are there, while both HUVEC512 intervals in Table 10 include zero. Is there a HUVEC512 contrast whose interval excludes zero, and how far should "best gFID, weaker retrieval" be read?
-
Pixel space is trained to 300k updates, with retrieval still rising when it stops; latent space to 100k. Does pixel-space retrieval keep improving past 300k, and would it close the gap?
8 comments
Response to Reviewer W6VG (2/3)
Weakness 2:
"Native readout audits reconstruction but not generation. Section 3 uses it to show that rFID fails; Section 4 then evaluates generation with Inception features alone. Is that because generated samples make the ridge regression unreliable? If so, the generation-side conclusions still rest on the space the paper argues against. Mapping CellProfiler profiles of generated samples into a library built from real images, and checking whether they retrieve the right siRNA perturbation, would give a readout outside Inception space without needing a paired ground truth."
Question 2:
"Is native readout left out of Section 4 because generated samples have no paired native image, or because the ridge reader is unreliable on generated images? If the latter, how should the generation-side conclusions be read?"
Response:
Because generated samples have no paired native image, not because the reader is unreliable on them: the readout is a paired measure, and a generator samples from a distribution. Without pairs, generation can only be scored at the distribution and perturbation level, so Section 4 starts from the accepted gFID and retrieval mAP and audits them. The CellProfiler retrieval you proposed works on six-channel outputs, where the released reversal holds.
The readout scores each object against its own native CellProfiler features, which a generated image, sampled for a perturbation, lacks. A real image of the same perturbation cannot stand in: its cells sit elsewhere, and batch effects change the same perturbation across batches [1]. Moreover, the readout depends on the layout of each cell: permuting 8×8-pixel blocks keeps every pixel yet drops native radial below zero (Table 2 (Reviewer W6VG); Appendix Table 5).
Table 2 (Reviewer W6VG): native , intact → 8×8 blocks permuted | Intensity | Radial | Texture | Granularity |
|---|---|---|---|---|
| HUVEC512 | 0.90 → 0.01 | 0.66 → −1.05 | 0.47 → −0.17 | 0.25 → −0.01 |
| IMPA96 | 0.86 → −0.15 | 0.55 → −0.69 | 0.36 → −0.13 | 0.25 → −0.13 |
Scoring unpaired generated samples against the native stains is therefore an open problem, which the revision now names as future work (Limitations).
CPGENeval is thus a combination: the readout, validated in Section 3, judges renderings and reconstructions, and gFID and retrieval compare generators (Table 1). Even so, the readout still reaches generation through the rendering: on held-out HUVEC512 batches, renderings it ranks higher tend to train generators that retrieve better, in all five feature spaces including OpenPhenom (Table 3).
Your proposal is interesting and promising: it scores generation outside Inception space without paired ground truth. Unfortunately, it leaves out every generator that outputs RGB. A rendering keeps at most three of the six stains, selected or mixed, so any profile of an RGB image, measured or predicted, comes from the rendering, and mixing renderings leave no pure DNA channel to segment nuclei. As its title From Stains to RGB says, this paper asks what the rendering keeps of the native stains, so such profiles lie outside its scope, although they are an interesting direction for future work.
For this check we use a dedicated set of six-channel latent generators: the same five frozen VAEs encode each stain separately or in two groups of three. On these and the six-channel pixel generators, profiles on the 108 native targets query a library of real images, scored by mAP as in Section 4. First, on images regenerated from the released checkpoints, CellFlux, the model gFID prefers, does not retrieve above chance (p = 0.34), and IMPA leads by 6.8 points (Table 1 (Reviewer W6VG)). Second, among our own generators, CellProfiler retrieval picks the same best HUVEC512 model as gFID.
[1] M. Sypetkowski et al. RxRx1: A dataset for evaluating experimental batch correction methods. CVPR Workshops, 2023.
Response to Reviewer W6VG (3/3)
Thank you again for reading our generation results this closely.
Question 3:
"The reversal looks established on IMPA96 — the significant contrasts are there, while both HUVEC512 intervals in Table 10 include zero. Is there a HUVEC512 contrast whose interval excludes zero, and how far should "best gFID, weaker retrieval" be read?"
Response:
Yes: on HUVEC512, 62 within-rendering contrasts exclude zero, and 61 agree with gFID (Appendix Table 9). "Best gFID, weaker retrieval" is a case that occurs, not a rule.
The one exception is a near-tie in gFID (14.27 vs. 14.28). Thus, within our training recipe on HUVEC512, gFID and retrieval mostly agree; the reversal that holds in every feature space is between released models (Table 1 (Reviewer W6VG)). Accordingly, the revised Finding says the lowest-gFID model can still retrieve worse.
Question 4:
"Pixel space is trained to 300k updates, with retrieval still rising when it stops; latent space to 100k. Does pixel-space retrieval keep improving past 300k, and would it close the gap?"
Response:
Yes, slowly. We extended both routes to 500k updates: the gap narrows to about one point, with pixel retrieval at 4.01% against 5.09% for SD1.4 under Recursion.
Table 3 (Reviewer W6VG): mAP (%), Recursion | 300k | 500k |
|---|---|---|
| Pixel | 3.93 | 4.01 |
| SD1.4 latent | 5.50 | 5.09 |
Past 300k, pixel retrieval peaks about half a point higher (4.47% at 440k), whereas SD1.4 retrieval stays at 5.0–5.6% from 200k on, and each pixel update costs about five times the H100 hours (Section 5). The trade-off of Section 5 thus holds at 500k: latent generators retrieve better sooner and cheaper, and pixel generators keep the six channels that native features can measure.
Overall, CPGENeval reports gFID and retrieval in several feature spaces to show where they disagree; our response to Reviewer Lt2d adds how each depends on sample size. Your questions name key open problems of the field, and the revision and this response now state each explicitly. These are an expert's questions, and we thank you for raising them. We would be glad to discuss any further questions.
Response to Reviewer W6VG (1/3)
Dear Reviewer
W6VG,Thank you for a review that shows a deep understanding of the problem: it names points to the problem that the field has not yet widely discussed, that generated images can be scored only in feature spaces, which the paper itself shows can mislead. To address it, we ran five new sets of experiments:
Together they address all four questions, and Section 4 now carries the qualification you asked for.
Weakness 1:
Question 1:
Response:
Yes for the released models, not always for ours. The released IMPA–CellFlux reversal holds in every feature space we tested, including the microscopy encoder uniDINO and CellProfiler features, whereas reversals among our own generators persist in only 7 of 15 cases in ConvNeXt, CLIP and DINOv2, and in 1 of 5 in uniDINO.
For the microscopy encoders, OpenPhenom [1] is built for 256-pixel tiles, so we use it on HUVEC512 (Table 3) and uniDINO [2] on rendered planes of the 96-pixel IMPA96 crops, where the reversals lie (Table 1 (Reviewer
W6VG)).W6VG): IMPA96CP: six-channel images regenerated from the released checkpoints and scored against Inception gFID, so there retrieval and gFID no longer share a representation. In contrast, each of our reversals involves an RGB latent generator, which has no native CellProfiler profile (–; Weakness 2).
Following your comment, Section 4 now qualifies the finding: our reversals depend on the feature space, and the Finding rests on the released reversal, which does not. In addition, the VAE ranking change survives OpenPhenom: on HUVEC512 every reconstruction score ranks VAEs against generator retrieval (τb −0.60 to −0.71; Appendix Table 8). Finally, we agree that the frozen VAEs are natural-image models; Cell Painting-specific VAEs remain untested (Limitations).
[1] O. Kraus et al. Masked autoencoders for microscopy are scalable learners of cellular biology. CVPR, 2024.
[2] F. M. Morelli et al. uniDINO: Assay-independent feature extraction for fluorescence microscopy images. Computational and Structural Biotechnology Journal, 2025.