acceptodds
Under review as a conference paper at ICLR 2027

From Stains to RGB: Revisiting Rendering for Cell Painting Generation

Abstract

Generating Cell Painting images requires balancing computational cost with biological utility. To reduce training costs, some existing approaches render multichannel fluorescence images into RGB and compress them using variational autoencoders (VAEs) pretrained on natural images for latent-space generation. Yet the separate effects of rendering and VAE compression on cellular feature recoverability and their implications for generative training remain unclear. Here we introduce CPGENeval, a benchmark that evaluates seven RGB rendering protocols and five VAEs across two datasets and compares pixel- and latent-space generation under a common training setup. Alongside FID, we measure whether cellular features computed from the original fluorescence channels can be recovered from RGB images and their VAE reconstructions, and whether generated images can be matched to real images of the same siRNA perturbation (perturbation retrieval). We find that (i) renderings that rank best by reconstruction FID (rFID) can leave native cellular features poorly recoverable; (ii) changing the rendering substantially changes generation FID (gFID) for the same generated samples, and even within a fixed rendering, gFID and perturbation retrieval can rank generators differently; and (iii) with appropriate rendering, RGB latent-space generation achieves stronger perturbation retrieval than pixel-space generation under most protocols, in substantially less training time. These findings call for reporting the rendering protocol with every FID, holding it fixed across compared models, and validating it with feature recovery and perturbation retrieval.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Cell Painting, generative model evaluation, Fréchet Inception Distance, latent generative models, variational autoencoders, benchmark, microscopy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W6VGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CPGENeval, an evaluation framework for Cell Painting image generation, to systematically select how multi-channel fluorescence images are rendered to RGB and which VAE encodes the RGB images into a latent space. The design is a factorial benchmark: seven rendering protocols (RECURSION, IMPA, MORPHGEN, MORPHDIFF, CELLCLIP, CELLFLUX, FREEBRIDGE) are crossed with five frozen natural-image VAEs (SD1.4, SD ft-MSE, SDXL, SD3.5, FLUX.1) over two datasets (HUVEC512, IMPA96). The study shows that the conventional reconstruction FID (rFID) and generation FID (gFID) are deficient for evaluating rendering protocols and VAEs, in that they do not reflect the original cellular features, and accordingly proposes native readout and retrieval as more reliable complementary measures. The paper also examines how latent-space and pixel-space generation differ in training cost and retrieval performance, finding that under a sensible rendering, latent-space generation not only shortens training time substantially but also attains better retrieval performance.

Critical strengths

The paper names a blind spot the field has been living with. FID is not only pursued blindly as a target; the score itself moves with the rendering, and it is decoupled from the biological features the images are meant to carry. Quantifying that rather than asserting it is what this paper does.

The sweep is broad enough to carry the claim: seven published rendering protocols crossed with five frozen VAEs on two datasets, with both the reconstruction stage and the generation stage covered under one design.

The paper does not stop at criticism. Native readout and retrieval give practitioners something to act on, and the guidance that follows from the experiments is concrete — report the rendering alongside every FID, fix it when comparing models, and check it against a biological readout. For a field that compares numbers across papers, this is directly usable.

Critical weaknesses
  1. The scores and the VAEs all sit in a natural-image feature space. rFID and gFID use frozen Inception features, trained on natural images and insensitive to cell morphology; would the ranking changes reported here survive if they were recomputed with a microscopy-pretrained encoder? Retrieval does not settle the question, since mAP is computed over the same features, and the disagreement between gFID and retrieval is then one between two aggregations of a single representation — "gFID can favor weaker siRNA retrieval" need not hold in a biological feature space. Section 4 states that finding without this qualification. The frozen VAEs are natural-image models too, which the limitation already concedes.

  2. Native readout audits reconstruction but not generation. Section 3 uses it to show that rFID fails; Section 4 then evaluates generation with Inception features alone. Is that because generated samples make the ridge regression unreliable? If so, the generation-side conclusions still rest on the space the paper argues against. Mapping CellProfiler profiles of generated samples into a library built from real images, and checking whether they retrieve the right siRNA perturbation, would give a readout outside Inception space without needing a paired ground truth.

Questions for authors
  1. Would the reversals reported in Section 4 survive if gFID and mAP were recomputed with an encoder pretrained on microscopy images, or with CellProfiler features?

  2. Is native readout left out of Section 4 because generated samples have no paired native image, or because the ridge reader is unreliable on generated images? If the latter, how should the generation-side conclusions be read?

  3. The reversal looks established on IMPA96 — the significant contrasts are there, while both HUVEC512 intervals in Table 10 include zero. Is there a HUVEC512 contrast whose interval excludes zero, and how far should "best gFID, weaker retrieval" be read?

  4. Pixel space is trained to 300k updates, with retrieval still rising when it stops; latent space to 100k. Does pixel-space retrieval keep improving past 300k, and would it close the gap?

8 comments
AuthorsOpenReview ↗

Response to Reviewer W6VG (1/3)

Dear Reviewer W6VG,

Thank you for a review that shows a deep understanding of the problem: it names points to the problem that the field has not yet widely discussed, that generated images can be scored only in feature spaces, which the paper itself shows can mislead. To address it, we ran five new sets of experiments:

  1. Six feature spaces beyond Inception: ConvNeXt, CLIP, DINOv2, two microscopy encoders (OpenPhenom, uniDINO) and CellProfiler features (W1, Q1).
  2. CellProfiler retrieval of all 22 six-channel generators and three released checkpoints, as you proposed (W2, Q2).
  3. A spatial permutation test of the readout (W2, Q2).
  4. Every within-rendering contrast on HUVEC512 (Q3).
  5. Both generation routes extended to 500k updates (Q4).

Together they address all four questions, and Section 4 now carries the qualification you asked for.


Weakness 1:

"The scores and the VAEs all sit in a natural-image feature space. rFID and gFID use frozen Inception features, trained on natural images and insensitive to cell morphology; would the ranking changes reported here survive if they were recomputed with a microscopy-pretrained encoder? Retrieval does not settle the question, since mAP is computed over the same features, and the disagreement between gFID and retrieval is then one between two aggregations of a single representation — "gFID can favor weaker siRNA retrieval" need not hold in a biological feature space. Section 4 states that finding without this qualification. The frozen VAEs are natural-image models too, which the limitation already concedes."

Question 1:

"Would the reversals reported in Section 4 survive if gFID and mAP were recomputed with an encoder pretrained on microscopy images, or with CellProfiler features?"

Response:

Yes for the released models, not always for ours. The released IMPA–CellFlux reversal holds in every feature space we tested, including the microscopy encoder uniDINO and CellProfiler features, whereas reversals among our own generators persist in only 7 of 15 cases in ConvNeXt, CLIP and DINOv2, and in 1 of 5 in uniDINO.

For the microscopy encoders, OpenPhenom [1] is built for 256-pixel tiles, so we use it on HUVEC512 (Table 3) and uniDINO [2] on rendered planes of the 96-pixel IMPA96 crops, where the reversals lie (Table 1 (Reviewer W6VG)).

Table 1 (Reviewer W6VG): IMPA96InceptionConvNeXtCLIPDINOv2uniDINOCP
Released IMPA − CellFlux, mAP (pp)+7.4 ✓+9.7 ✓+4.5 ✓+6.6 ✓+13.7 ✓+6.8 ✓
Our reversals that hold (of 5)53131–

CP: six-channel images regenerated from the released checkpoints and scored against Inception gFID, so there retrieval and gFID no longer share a representation. In contrast, each of our reversals involves an RGB latent generator, which has no native CellProfiler profile (–; Weakness 2).

Following your comment, Section 4 now qualifies the finding: our reversals depend on the feature space, and the Finding rests on the released reversal, which does not. In addition, the VAE ranking change survives OpenPhenom: on HUVEC512 every reconstruction score ranks VAEs against generator retrieval (τb\tau_b −0.60 to −0.71; Appendix Table 8). Finally, we agree that the frozen VAEs are natural-image models; Cell Painting-specific VAEs remain untested (Limitations).

[1] O. Kraus et al. Masked autoencoders for microscopy are scalable learners of cellular biology. CVPR, 2024.

[2] F. M. Morelli et al. uniDINO: Assay-independent feature extraction for fluorescence microscopy images. Computational and Structural Biotechnology Journal, 2025.

AuthorsOpenReview ↗

Response to Reviewer W6VG (2/3)

Weakness 2:

"Native readout audits reconstruction but not generation. Section 3 uses it to show that rFID fails; Section 4 then evaluates generation with Inception features alone. Is that because generated samples make the ridge regression unreliable? If so, the generation-side conclusions still rest on the space the paper argues against. Mapping CellProfiler profiles of generated samples into a library built from real images, and checking whether they retrieve the right siRNA perturbation, would give a readout outside Inception space without needing a paired ground truth."

Question 2:

"Is native readout left out of Section 4 because generated samples have no paired native image, or because the ridge reader is unreliable on generated images? If the latter, how should the generation-side conclusions be read?"

Response:

Because generated samples have no paired native image, not because the reader is unreliable on them: the readout is a paired measure, and a generator samples from a distribution. Without pairs, generation can only be scored at the distribution and perturbation level, so Section 4 starts from the accepted gFID and retrieval mAP and audits them. The CellProfiler retrieval you proposed works on six-channel outputs, where the released reversal holds.

The readout scores each object against its own native CellProfiler features, which a generated image, sampled for a perturbation, lacks. A real image of the same perturbation cannot stand in: its cells sit elsewhere, and batch effects change the same perturbation across batches [1]. Moreover, the readout depends on the layout of each cell: permuting 8×8-pixel blocks keeps every pixel yet drops native radial R2R^2 below zero (Table 2 (Reviewer W6VG); Appendix Table 5).

Table 2 (Reviewer W6VG): native R2R^2, intact → 8×8 blocks permutedIntensityRadialTextureGranularity
HUVEC5120.90 → 0.010.66 → −1.050.47 → −0.170.25 → −0.01
IMPA960.86 → −0.150.55 → −0.690.36 → −0.130.25 → −0.13

Scoring unpaired generated samples against the native stains is therefore an open problem, which the revision now names as future work (Limitations).

CPGENeval is thus a combination: the readout, validated in Section 3, judges renderings and reconstructions, and gFID and retrieval compare generators (Table 1). Even so, the readout still reaches generation through the rendering: on held-out HUVEC512 batches, renderings it ranks higher tend to train generators that retrieve better, in all five feature spaces including OpenPhenom (Table 3).

Your proposal is interesting and promising: it scores generation outside Inception space without paired ground truth. Unfortunately, it leaves out every generator that outputs RGB. A rendering keeps at most three of the six stains, selected or mixed, so any profile of an RGB image, measured or predicted, comes from the rendering, and mixing renderings leave no pure DNA channel to segment nuclei. As its title From Stains to RGB says, this paper asks what the rendering keeps of the native stains, so such profiles lie outside its scope, although they are an interesting direction for future work.

For this check we use a dedicated set of six-channel latent generators: the same five frozen VAEs encode each stain separately or in two groups of three. On these and the six-channel pixel generators, profiles on the 108 native targets query a library of real images, scored by mAP as in Section 4. First, on images regenerated from the released checkpoints, CellFlux, the model gFID prefers, does not retrieve above chance (p = 0.34), and IMPA leads by 6.8 points (Table 1 (Reviewer W6VG)). Second, among our own generators, CellProfiler retrieval picks the same best HUVEC512 model as gFID.

[1] M. Sypetkowski et al. RxRx1: A dataset for evaluating experimental batch correction methods. CVPR Workshops, 2023.

AuthorsOpenReview ↗

Response to Reviewer W6VG (3/3)

Thank you again for reading our generation results this closely.


Question 3:

"The reversal looks established on IMPA96 — the significant contrasts are there, while both HUVEC512 intervals in Table 10 include zero. Is there a HUVEC512 contrast whose interval excludes zero, and how far should "best gFID, weaker retrieval" be read?"

Response:

Yes: on HUVEC512, 62 within-rendering contrasts exclude zero, and 61 agree with gFID (Appendix Table 9). "Best gFID, weaker retrieval" is a case that occurs, not a rule.

The one exception is a near-tie in gFID (14.27 vs. 14.28). Thus, within our training recipe on HUVEC512, gFID and retrieval mostly agree; the reversal that holds in every feature space is between released models (Table 1 (Reviewer W6VG)). Accordingly, the revised Finding says the lowest-gFID model can still retrieve worse.


Question 4:

"Pixel space is trained to 300k updates, with retrieval still rising when it stops; latent space to 100k. Does pixel-space retrieval keep improving past 300k, and would it close the gap?"

Response:

Yes, slowly. We extended both routes to 500k updates: the gap narrows to about one point, with pixel retrieval at 4.01% against 5.09% for SD1.4 under Recursion.

Table 3 (Reviewer W6VG): mAP (%), Recursion300k500k
Pixel3.934.01
SD1.4 latent5.505.09

Past 300k, pixel retrieval peaks about half a point higher (4.47% at 440k), whereas SD1.4 retrieval stays at 5.0–5.6% from 200k on, and each pixel update costs about five times the H100 hours (Section 5). The trade-off of Section 5 thus holds at 500k: latent generators retrieve better sooner and cheaper, and pixel generators keep the six channels that native features can measure.


Overall, CPGENeval reports gFID and retrieval in several feature spaces to show where they disagree; our response to Reviewer Lt2d adds how each depends on sample size. Your questions name key open problems of the field, and the revision and this response now state each explicitly. These are an expert's questions, and we thank you for raising them. We would be glad to discuss any further questions.

Reviewer_Lt2dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CPGENeval, a unified benchmark for assessing how RGB rendering and VAE compression affect the preservation of cellular information and model evaluation in Cell Painting image generation. Across two data settings, the authors compare 7 rendering protocols and 5 pretrained VAEs, evaluating 70 reconstruction configurations and 72 generators. The results show that a lower FID does not necessarily indicate better preservation of cellular features; even with rendering held fixed, FID and perturbation retrieval can yield opposite model rankings. Well-chosen latent-space models train faster, although longer training of pixel-space models can narrow the performance gap. The main contribution is a systematic analysis of how rendering and evaluation metrics influence model selection, together with a unified comparison protocol.

Critical strengths
  1. The paper addresses a practically important evaluation problem. By showing that rendering affects both model inputs and evaluation outcomes, it makes an often implicit experimental choice explicit and provides a useful basis for more reliable comparisons in cellular image generation.

  2. The controlled experiments provide convincing support for the conclusion that FID alone is insufficient for model selection. Rendering the same generated images in different ways and comparing metrics under a fixed rendering protocol help distinguish the effects of rendering from disagreements between evaluation criteria.

Critical weaknesses
  1. The novelty lies primarily in the systematic evaluation. The sensitivity of FID to preprocessing and the distinction between overall distributional similarity and correct correspondence to conditioning inputs are already recognized issues. Examining these issues in cellular image generation is useful, but the methodological and conceptual advances appear limited. This limits the strength of the paper's contribution beyond its value as an empirical benchmark.

  2. The biological utility of the proposed evaluation framework is not yet sufficiently established. The experiments show that evaluation choices can change model rankings, but do not adequately demonstrate that models selected using CPGENeval more accurately capture perturbation-induced biological changes. This leaves the practical value of the benchmark for biological research less clear than its value for comparing metrics.

Questions for authors
  1. Which findings go beyond the known sensitivity of FID to preprocessing and the distinction between distributional similarity and correspondence to conditioning inputs? Please identify the main new insight specific to Cell Painting and explain how it changes model selection in this setting.

2.What evidence shows that selecting a model using CPGENeval leads to more faithful recovery of perturbation-induced biological changes? In particular, how do the reported cellular-feature or perturbation-retrieval results establish a biologically meaningful advantage for the selected models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Lt2d (1/2)

Dear Reviewer Lt2d,

Thank you for noting that our paper "addresses a practically important evaluation problem" and "makes an often implicit experimental choice explicit". For your questions we added Table 1, which states what each measure can certify, and an Implication for Sections 3 and 4 (W1, Q1); a test of whether the readout's choice of rendering predicts generator retrieval (Table 3; W2, Q2); the released reversal in five feature spaces including CellProfiler features (Appendix Table 9; W2, Q2); and how FID and mAP depend on sample size (W2, Q2).


Weakness 1:

"The novelty lies primarily in the systematic evaluation. The sensitivity of FID to preprocessing and the distinction between overall distributional similarity and correct correspondence to conditioning inputs are already recognized issues. Examining these issues in cellular image generation is useful, but the methodological and conceptual advances appear limited. This limits the strength of the paper's contribution beyond its value as an empirical benchmark."

Question 1:

"Which findings go beyond the known sensitivity of FID to preprocessing and the distinction between distributional similarity and correspondence to conditioning inputs? Please identify the main new insight specific to Cell Painting and explain how it changes model selection in this setting."

Response:

The Cell Painting-specific insight is that the RGB reference is itself a choice: fidelity metrics are largely hacked by saturated renderings, while the radial native readout of the original draft detects them (Section 3 Implication).

This question showed us that the original draft understated its claims; the revision states them in the new Table 1 and, following Reviewer dboH, tests them with LPIPS, other image sizes, nonlinear readers and 108 CellProfiler targets.

For natural images, rFID, PSNR, SSIM and LPIPS are the standard reconstruction metrics because the RGB input is the ground truth. In Cell Painting, however, the RGB input is only a rendering of six stains. These metrics therefore measure fidelity to the rendering and cannot tell whether it kept the cellular signal. They are hacked by saturation: a rendering that pushes most pixels to black or white keeps little biological information and is the easiest reference to reconstruct, so PSNR, SSIM and LPIPS select it under 4–5 of 5 VAEs on both datasets. Moreover, saturation is not an artificial case: each published rendering is calibrated for its own data and is rarely reported. Applied to another dataset, it can saturate, as Recursion does on IMPA96 crops (64% near-white pixels, Figure 1).

The radial native readout of the original draft closes this gap. It predicts the radial CellProfiler features of the native stains from the RGB image, so its reference is the stains rather than the rendering. The rendering is thus a model choice of its own. The reconstruction part of the new Table 1 shows which measure serves which choice (Table 1 (Reviewer Lt2d)):

Table 1 (Reviewer Lt2d): revised Table 1, reconstructionCPGENeval (native readout)Fidelity metrics (rFID, PSNR, SSIM, LPIPS)Image statistics (entropy, near-black/white)
Reference (recon. target)Native stainsRendered RGBRendered RGB
Picks each dataset's published rendering✓✗✗
Ranks saturated renderings lowest✓✗Entropy only
Ranks VAEs (fixed rendering)✓✓✗

Table 2 checks each row on both datasets: the readout picks the published rendering, ranks saturated renderings last, and ranks VAEs in the same direction as PSNR, SSIM and LPIPS once the rendering is fixed.

Implication of the CPGENeval readout. In Cell Painting a low rFID or a high PSNR does not certify a rendering. Evaluate the rendering first against the native stains with the readout, then choose the VAE for reconstruction with the readout and fidelity metrics. The revision states this as the Implication of Section 3.

It also changes how generators are selected. Prior generators are strong. All 11 generators surveyed in Appendix Table 13 report FID, and three state that they select or stop training by it. In any feature space, a generated condition should lie closer to real wells of its own perturbation than to those of others; when real images retrieve their perturbation there but generated ones do not, the generator has missed the condition. FID pools all conditions and cannot see this, and a biological check run after selecting by FID sees only the selected model. For example, the released CellFlux has the lowest gFID of the released models, yet in CellProfiler space it retrieves at chance (p = 0.34; Appendix Table 9). Retrieval in several feature spaces can therefore screen settings and checkpoints before the large, time-consuming biological validation downstream.


AuthorsOpenReview ↗

Response to Reviewer Lt2d (2/2)

Weakness 2:

"The biological utility of the proposed evaluation framework is not yet sufficiently established. The experiments show that evaluation choices can change model rankings, but do not adequately demonstrate that models selected using CPGENeval more accurately capture perturbation-induced biological changes. This leaves the practical value of the benchmark for biological research less clear than its value for comparing metrics."

Question 2:

"What evidence shows that selecting a model using CPGENeval leads to more faithful recovery of perturbation-induced biological changes? In particular, how do the reported cellular-feature or perturbation-retrieval results establish a biologically meaningful advantage for the selected models?"

Response:

On the generation side the Cell Painting-specific insight is that the native readout links the two stages: the rendering it chooses before any training trains generators that retrieve better on held-out batches, while gFID can still favor the weaker model once the rendering is fixed (Section 4 Implication).

That pooled FID ignores whether a sample matches its condition is known from natural images. Yet Cell Painting generation still relies on it: as CellFluxRL puts it, "to date, focus has been on image generation quality scores, like FID and KID, rather than biological validity" [1]. The generators are strong, but they are judged by a metric built for natural images. In Cell Painting, however, the condition is a perturbation, so this gap can be measured against biology. Indeed, for the released models it is not an artifact of natural-image encoders: gFID favors the released CellFlux over IMPA under the same rendering, yet IMPA retrieves better in every feature space we tested, including CellProfiler features measured directly on six-channel generations (Appendix Table 9).

A second finding has no counterpart in natural images. The rendering is fixed before training, and the readout chooses it from real images alone. On held-out HUVEC512 batches, renderings it ranks higher tend to train generators that retrieve better, in all five feature spaces including the microscopy encoder OpenPhenom.

These results answer Q2 with the siRNA retrieval of the original draft. The revision makes its biological meaning explicit: retrieval mAP is how image-based profiling tests whether profiles of one perturbation form a consistent signature [2], and on HUVEC512 it is computed on batches unseen in training. Accordingly, on HUVEC512, selecting with CPGENeval gives generators that recover perturbations better. In contrast, PSNR and LPIPS choose the CellFlux rendering for every VAE, and its generators retrieve at most 0.44% mAP, against 3.9% on average for those of the readout's choice, the Recursion rendering.

FID and mAP also depend on sample size in different ways (Table 2 (Reviewer Lt2d)). mAP rises with the images that form each condition's profile, because averaging reduces noise, while the ground truth stays fixed. By comparison, FID estimates a 2048-dimensional covariance and is biased at small samples: between real train and eval images of the same siRNA, its median is 12.6 times the pooled value. The two scores are therefore complementary: CPGENeval reports both, in five feature spaces, so that their disagreement can be checked (see our response to Reviewer W6VG).

Table 2 (Reviewer Lt2d): real HUVEC512 images, 20 draws1 image4 images16 imagesAll (≈32)
mAP, % of full28%60%91%100%
Variation across draws5.8%3.8%1.1%–
mAP / chance10×21×33×36×

Implication of CPGENeval for generation. A low gFID does not certify a generator. Choose the rendering by the readout before training generators, then compare the generators under that rendering by siRNA retrieval as well as gFID (Section 4 Implication).

Thank you again for a careful review and for recognizing our work. Your summary of the paper is one we would gladly adopt, and your two questions pushed us to state what the original draft left implicit. We hope the revision now shows what is specific to Cell Painting and why it matters for biology. We would be glad to discuss any further questions.

[1] D. Wu et al. CellFluxRL: Biologically-Constrained Virtual Cell Modeling via Reinforcement Learning. arXiv:2603.21743, 2026.

[2] A. A. Kalinin et al. A versatile information retrieval framework for evaluating profile strength and similarity. Nature Communications, 2025.


AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer Lt2d,

As the discussion period is drawing to a close, we would like to kindly follow up on our responses. We sincerely hope they have addressed your concerns, and we would appreciate it if you could let us know whether any issues remain. We would be happy to provide any further clarification.

Thank you very much for your time and consideration.

Best regards, The Authors

AuthorsOpenReview ↗

Additional Follow up to reviewer Lt2d

Dear Reviewer Lt2d,

We would like to share one addition made during the discussion. It relates to your Weakness 2 and Question 2 on biological utility.

A concurrent preprint [1] appeared on arXiv in October 2026. This is after our submission, so the original draft could not discuss it. We think it is worth mentioning here.

  • It proves by construction that pooled and conditional FID can both prefer the model with the wrong biological label. It calls this an FID ranking reversal.
  • Its authors note that this is not an empirical result on cell datasets.
  • CPGENeval supplies the measurement.

Motivated by this work we applied CPGENeval to BBBC021. It serves as a case study of the benchmark on a practical Cell Painting question. The question is which released generator better recovers the mechanism of action (MoA) of a compound. MoA is a biological label beyond the identity of the perturbation.

The main line of our paper is six-stain data, where the stains must be rendered to RGB. BBBC021 has three stains that fill the RGB channels directly. The same evaluation applies there. This shows that CPGENeval is not limited to six-stain data.

We regenerated the released CellFlux and IMPA models on its 7,119 test images of 26 compounds. For retrieval by MoA the wells of the query compound are removed. The positives are real wells of other compounds with the same MoA.

Table 3 (Reviewer Lt2d): BBBC021InceptionConvNeXtCLIPDINOv2uniDINO
FD, CellFlux18.619.732.8230.3117.6
FD, IMPA35.727.3115.8311.4485.9
mAP by MoA (%), CellFlux56.757.549.143.371.4
mAP by MoA (%), IMPA57.173.332.945.269.8
IMPA − CellFlux, mAP (pp)+0.4+15.8−16.1+1.9−1.6

Bold means that the paired 95% interval excludes zero. Chance is 12.1%.

  • FD favors CellFlux in all five feature spaces by 1.4 to 4.1 times.
  • Retrieval by MoA stays within 2 points in three of the five spaces. In ConvNeXt IMPA retrieves 15.8 points better.
  • A several-fold FD gap therefore need not come with a gap in recovering the mechanism of action.
  • On IMPA96 the released CellFlux has the lowest FD and the lowest retrieval of the three released models in all five feature spaces.

The theory of [1] and the measurement of CPGENeval are complementary. Both support the Implication of Section 4 that a low gFID does not certify a generator.

The full tables are in our follow-up responses to Reviewer W6VG.

If our responses have addressed your concerns, we would be grateful if you would consider raising your assessment. Thank you again for your careful review.

[1] PerturbBridge: Direct phenotypic supervision for cellular perturbation generation. arXiv:2610.17066, 2026.

Reviewer_dboHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CPGENeval, a benchmark for Cell Painting image generation that isolates the multichannel-fluorescence-to-RGB rendering step from VAE reconstruction and from generator training/evaluation. The authors reimplement seven rendering protocols crossed with five frozen VAEs (70 reconstruction configurations and 72 generators), and add evaluations whose ground truth does not depend on the rendering: 24 native CellProfiler radial intensity readouts and siRNA retrieval mAP. The central finding is that selecting renderings by rFID can favor options that have already discarded native cellular information, and that within a fixed rendering, gFID and retrieval can rank generators in opposite orders.

Critical strengths
  • The problem is well chosen: the survey shows at least 6 of the 11 reviewed Cell Painting generators compute FID on RGB renderings, yet the rendering protocol is almost never reported. Explicitly examining this step has practical value.

  • Large experimental scale with controlled comparisons: the 70 configurations and 72 generators cover a 7×5 protocol-VAE grid, with the two routes sharing a backbone and training setup. Re-rendering the same generated samples alone moves gFID from 8.8 to 70.6, a clean design.

  • Concretely designed evaluation protocol: the native readout uses 24 fixed targets, a frozen reader, and leave-one-batch-out folds. Retrieval mAP comes with chance-level controls.

  • Specific key evidence: transferred to IMPA96, the RECURSION protocol leaves 63.68% of pixels near-white with R²_rec of only 0.007, yet still ranks first by rFID.

  • Transparent documentation and cost accounting: each of the seven protocols is documented down to source commits, mixing matrices, and calibration details. Training hyperparameters and the ~2,000 GPU-hour total are transparent.

Critical weaknesses
  • The experiments cover only two fixed resolutions: everything runs on the 512×512 and 96×96 settings, and resolution is never treated as a variable. Since resolution is directly coupled to the rendering calibrations, all three practical recommendations rest on unexamined single-resolution settings.

  • The critique of FID is one-sided, with per-image fidelity metrics missing: FID is a distribution-level measure whose standard complement is per-image fidelity (PSNR, SSIM, LPIPS), yet these never enter the main analysis and generated samples receive no per-image evaluation. Judging existing evaluation practice after examining only half of the standard suite is under-argued.

  • The operationalization of native cellular features is too narrow: the headline finding rests on 24 radial intensity fractions and a low-capacity linear reader, excluding texture and morphology. The paper itself concedes these scores do not measure information loss, leaving genuine loss indistinguishable from insufficient reader capacity.

Questions for authors
  • Could you add experiments at at least one additional resolution (e.g., 256×256, or the 1024×1280 BBBC021 images) to test whether the conclusions change with resolution? If they change, how should the current conclusions be qualified?

  • Could you move PSNR/SSIM (already in Appendix Table 6) into the main analysis, add LPIPS, and report per-image fidelity for generated samples? Does the conclusion that selecting by FID favors weaker biological signal still hold once these are included?

  • If the 24 radial intensity fractions are extended to broader CellProfiler features with a higher-capacity nonlinear reader, do the relative conclusions between saturated and three-channel renderings change?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dboH (1/3)

Dear Reviewer dboH:

You listed five strengths of the paper, above all that "the problem is well chosen" and that re-rendering the same generated samples is "a clean design". You also raised three concerns: whether the conclusions hold with per-image fidelity metrics, at other image sizes and with a broader native readout. Each concern is well founded. For these concerns we added LPIPS and moved PSNR and SSIM into the main table as selectors (W2, Q2), extended the readout to 108 targets in four families with MLP and RFF readers and a spatial-permutation control (W3, Q3), and repeated the analysis at five image sizes (W1, Q1).

Thank you for appreciating the work and for raising these concerns. The original draft held these three settings fixed so that each comparison changes one factor, which left open whether the conclusions survive when they change.

The original conclusions hold, and fidelity metrics do not replace the native readout. Among the 108 targets, the radial targets of the original draft give the most stable choice of rendering across readers and image sizes (Table 2). The native readout of the original draft is this radial readout; with four families, the revision marks the family as a subscript (Rrec,radial2R^2_{\mathrm{rec,radial}}), and "the readout" below means the radial one.

The following are our detailed responses in three parts. We answer them out of order: the main table first and image size last, so the main result is tested before its setting changes.


Weakness 2:

"The critique of FID is one-sided, with per-image fidelity metrics missing: FID is a distribution-level measure whose standard complement is per-image fidelity (PSNR, SSIM, LPIPS), yet these never enter the main analysis and generated samples receive no per-image evaluation. Judging existing evaluation practice after examining only half of the standard suite is under-argued."

Weakness 3:

"The operationalization of native cellular features is too narrow: the headline finding rests on 24 radial intensity fractions and a low-capacity linear reader, excluding texture and morphology. The paper itself concedes these scores do not measure information loss, leaving genuine loss indistinguishable from insufficient reader capacity."

Question 2:

"Could you move PSNR/SSIM (already in Appendix Table 6) into the main analysis, add LPIPS, and report per-image fidelity for generated samples? Does the conclusion that selecting by FID favors weaker biological signal still hold once these are included?"

Question 3:

"If the 24 radial intensity fractions are extended to broader CellProfiler features with a higher-capacity nonlinear reader, do the relative conclusions between saturated and three-channel renderings change?"

Response:

The revised Table 2 widens the main experiment in both directions you asked for: PSNR, SSIM and LPIPS join rFID as selectors, and the readout grows from 24 radial targets to 108 targets in four families, checked with nonlinear readers. The conclusions hold and become stronger. Furthermore, revised Table 2 now reports, for every metric, which rendering and VAE it selects and how often it agrees with the readout, choosing the rendering for each VAE and then the VAE for each rendering, as a practitioner would. Table 1 (Reviewer dboH) below reports what it picks, whether the pick is saturated and how well its VAE ranking agrees with the readout.

Table 1 (Reviewer dboH): choices of each metricHUVEC512: most chosenRadial Rrec2R^2_{\mathrm{rec}} of choiceSaturated picksVAE τˉb\bar\tau_bIMPA96: most chosenRadial Rrec2R^2_{\mathrm{rec}} of choiceSaturated picksVAE τˉb\bar\tau_b
Native readoutRecursion 5/50.640/5–IMPA 4/50.520/5–
rFID ↓MorphGen 2/50.501/5+0.54Recursion 4/50.144/5+0.29
PSNR ↑CellFlux 5/50.175/5+0.74Recursion 4/50.124/5+0.63
SSIM ↑CellFlux 4/50.244/5+0.74Recursion 5/50.045/5+0.60
LPIPS ↓CellFlux 5/50.175/5+0.66Recursion 4/50.144/5+0.49

PSNR, SSIM and LPIPS make the original finding stronger. When choosing the rendering, they pick a saturated one under 4–5 of 5 VAEs on both datasets, at least as often as rFID, while the readout never does. Across the 11 reader and image-size settings, the radial readout also ranks the saturated renderings last in all 11 and picks the published rendering in 10 (Table 2c).


AuthorsOpenReview ↗

Response to Reviewer dboH (2/3)

Weakness 2 and Question 2 (continued)

Response:

Table 1 (Reviewer dboH), continued lists the values you asked for, averaged over the five VAEs (every configuration: Appendix Tables 21–22). The fidelity metrics and the readout disagree. On HUVEC512 the saturated CellFlux rendering has the best rFID, PSNR and LPIPS and ties for the best SSIM, yet the lowest readout (0.17, against 0.64 for Recursion). On IMPA96 the saturated Recursion rendering is best on all four metrics, yet its readout is 0.04, against 0.51 for IMPA.

Table 1 (Reviewer dboH), continued: mean over 5 VAEsRenderingrFID ↓PSNR ↑SSIM ↑LPIPS ↓Radial Rrec2R^2_{\mathrm{rec}} ↑
HUVEC512Recursion5.0830.950.8290.0450.64
(Rank by radial Rrec2R^2_{\mathrm{rec}})CellCLIP4.5231.330.8390.0410.63
MorphGen2.8034.110.8650.0350.60
MorphDiff3.0237.010.8940.0320.56
FreeBridge4.4628.870.7990.0270.53
IMPA2.4640.830.9280.0190.53
CellFlux (saturated)2.3350.730.9280.0040.17
IMPA96IMPA27.9727.100.6790.0520.51
(Rank by radial Rrec2R^2_{\mathrm{rec}})FreeBridge34.2124.630.6770.0500.49
MorphDiff32.6025.450.6710.0700.48
MorphGen38.5426.140.6820.0820.47
CellFlux31.9427.220.6980.0640.42
CellCLIP (saturated)21.5527.080.8390.0330.15
Recursion (saturated)20.1329.290.8930.0230.04

As for generated samples, per-image metrics need a paired real image, so they do not apply. Section 4 tests them on the generation side instead. Used to choose the rendering on HUVEC512, PSNR and LPIPS pick the CellFlux rendering for every VAE, and its generators retrieve at most 0.44% mAP, against 3.9% on average for the readout's choice, the Recursion rendering (Section 4).


Weakness 3 and Question 3 (continued)

Response:

Stronger readers do not recover what saturation removes. With MLP and RFF, saturated renderings stay far below every other rendering on the radial targets, so the gap lies in the image, not in reader capacity.

We would like to clarify the original draft. It notes that no reader gives an upper bound, so an absolute score is not a measure of information. A large gap under the same frozen reader compares what two renderings keep. If the gap came from reader capacity, the MLP would close it. It reads more from every non-saturated rendering, yet the gap between the best and the saturated rendering grows, from 0.47 to 0.59 on HUVEC512 and from 0.41 to 0.45 on IMPA96 (Appendix Table 4).

The relative conclusion of Q3 does not change. Both three-stain renderings, IMPA and FreeBridge, stay above every saturated rendering on the radial targets in all six reader and dataset settings (Table 2 (Reviewer dboH)).

Table 2 (Reviewer dboH): six reader settingsThree-stain renderings above every saturated oneSmallest gap
Radial6/6+0.36
Granularity6/6+0.06
Intensity5/6−0.05
Texture4/6−0.06

Morphology. We also read 17 nuclear shape features (CellProfiler AreaShape) with the same three readers, and the conclusion is the same. Saturated renderings lose shape under every reader: on HUVEC512 the CellFlux rendering reaches 0.02 with ridge and 0.04 with the MLP, against 0.33–0.53 and 0.70–0.84 for the other six renderings. Both three-stain renderings again stay above every saturated one in all six reader and dataset settings (smallest gap +0.37).

Texture and granularity, the two CellProfiler families that describe the fine structure of each stain, are now part of the readout. Radial is also the most reliable family: the other three pick the published rendering in at most 4 of 11 settings, radial in 10 (Table 3 (Reviewer dboH)).

Table 3 (Reviewer dboH): 11 reader and size settingsNative R2R^2 (HUVEC512 / IMPA96)Picks the published renderingRanks saturated last
Radial0.66 / 0.5510/1111/11
Intensity0.90 / 0.864/115/11
Granularity0.25 / 0.253/1111/11
Texture0.47 / 0.363/114/11

The native column suggests why. Intensity is the easiest family to read (native 0.90), and even the stains barely predict granularity (0.25). Radial lies in between and changes with channel selection and mixing.


AuthorsOpenReview ↗

Response to Reviewer dboH (3/3)

Weakness 1:

"The experiments cover only two fixed resolutions: everything runs on the 512×512 and 96×96 settings, and resolution is never treated as a variable. Since resolution is directly coupled to the rendering calibrations, all three practical recommendations rest on unexamined single-resolution settings."

Question 1:

"Could you add experiments at at least one additional resolution (e.g., 256×256, or the 1024×1280 BBBC021 images) to test whether the conclusions change with resolution? If they change, how should the current conclusions be qualified?"

Response:

The conclusions hold at all five image sizes. Only reconstruction changes: at lower resolution the VAEs lose radial signal and diverge. But the native radial readout captures this pattern.

BBBC021 has three stains, so it has no six-to-RGB step to test. A new dataset would also change the cells and the targets. We therefore keep the clean design you praised and change one factor only. HUVEC512, its objects and its native targets stay fixed, and only the image size changes, in two ways. Both operations are how these datasets are made: RxRx1 itself was downsampled to 1024 pixels and cropped to its central 512 [1], RxRx3-core keeps 512 center crops of 2048 images [2], and microscopy encoders train on 256 crops [3]. First, tiles of 256 and 128 keep the pixel scale and shrink the field of view. Second, block means of 256 and 128 lower the resolution itself. Altogether, the test uses all 39,272 sites of the 16 batches, and 157,088 and 628,352 tiles. All sizes share the 747,406 objects whose window fits a 128-pixel tile.

At any size, the readout picks the saturated CellFlux rendering for at most 1 of 5 VAEs. In contrast, PSNR, SSIM and LPIPS pick it for 4–5 of 5 at every size, although they still rank VAEs with the readout (Table 4 (Reviewer dboH)).

Table 4 (Reviewer dboH): HUVEC512 sizesSaturated picks: readoutrFIDPSNRSSIMLPIPSVAE τˉb\bar\tau_b (PSNR, SSIM, LPIPS)
5120/51/55/54/55/50.69–0.77
Tile 2560/53/55/55/55/50.74–0.94
Tile 1280/55/55/55/55/50.69–0.86
Block mean 2560/54/55/55/55/50.86–0.91
Block mean 1281/55/55/55/55/50.80–0.86

What changes is reconstruction. Tiling barely moves the readout. Block means, however, change the pixel statistics that the calibrations act on. Even so, the rendering keeps its readout (0.63 to 0.65), so the loss comes from the VAEs: after reconstruction, Recursion falls to 0.48 on average at 256 and to 0.03 at 128, where the five VAEs spread from −0.17 to 0.37 (Table 5 (Reviewer dboH)).

Table 5 (Reviewer dboH): radial R2R^2 before and after the VAERecursion renderingCellFlux rendering (saturated)Recursion after the VAE: mean [min, max] of 5 VAEs
5120.650.180.64 [0.62, 0.65]
Tile 2560.650.180.64 [0.62, 0.65]
Tile 1280.650.180.62 [0.58, 0.65]
Block mean 2560.650.170.48 [0.32, 0.62]
Block mean 1280.630.150.03 [−0.17, 0.37]

Generation. The native readout has no paired native image for generated samples, so it drops out on the generation side (see our response to Reviewer W6VG). What can be tested there is the recommendation on gFID, and it holds at every size: re-rendering the same generated samples changes gFID 8.0 times at 512, 12.0 and 20.8 times in tiles of 256 and 128, and 6.1 and 3.5 times at block means of 256 and 128.

Qualification. Lower resolution weakens reconstruction, yet the readout's choice of rendering keeps its direction at all five sizes, so the three recommendations stand (Appendix F.3).

In short, resolution is now a variable, the standard suite is complete, and a stronger reader separates genuine loss from reader capacity. These were the concerns behind your Soundness score, and they turned out to be addressable with the experiments you proposed. Together they took at least 1,300 CPU core-hours and 50 GPU-hours, and the readout of the original draft passed every test you proposed. We would be glad to discuss any further questions. And we kindly ask you to raise our score if you think our revision has addressed all your concerns.

[1] M. Sypetkowski et al. RxRx1: A dataset for evaluating experimental batch correction methods. CVPR Workshops, 2023.

[2] O. Kraus et al. RxRx3-core: Benchmarking drug-target interactions in high-content microscopy. arXiv:2503.20158, 2025.

[3] O. Kraus et al. Masked autoencoders for microscopy are scalable learners of cellular biology. CVPR, 2024.


AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer dboH,

As the discussion period is drawing to a close, we would like to kindly follow up on our responses. We sincerely hope they have addressed your concerns, and we would appreciate it if you could let us know whether any issues remain. We would be happy to provide any further clarification. If our responses have resolved them, we would be grateful if you would consider updating your assessment.

Thank you very much for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

general response

We thank the PCs, SACs, AC and all reviewers for their time and their careful reviews.

Summary: This paper introduces CPGENeval, a benchmark, evaluation suite and analysis pipeline for the RGB rendering step in Cell Painting generation. We systematically analyse seven published renderings and five VAEs with a native radial readout, which predicts CellProfiler features of the original six stains from the RGB image, and compare pixel-space and RGB latent-space generation under a unified setup. Our work shows three things: 1. Fidelity metrics favor saturated renderings, and our radial native readout detects them. 2. gFID and siRNA retrieval, which the benchmark adds as a perturbation-level score, can disagree, so generators are evaluated with both scores in several feature spaces. Renderings that the readout ranks higher train generators that retrieve better. 3. Pixel and RGB latent generation each have their own regime: latents reach strong retrieval with less training, and pixel generators keep all six channels and close most of the gap with longer training.

Reviewers agree on the following strengths:

  • An important, overlooked problem: "the problem is well chosen" (Reviewer dboH); "a practically important evaluation problem" that "makes an often implicit experimental choice explicit" (Reviewer Lt2d); the paper "names a blind spot the field has been living with" (Reviewer W6VG).
  • Broad, controlled experiments: "large experimental scale with controlled comparisons" and "a clean design" (Reviewer dboH); "the sweep is broad enough to carry the claim" (Reviewer W6VG).
  • Convincing evidence: "specific key evidence" (Reviewer dboH); the controlled experiments "provide convincing support for the conclusion that FID alone is insufficient for model selection" (Reviewer Lt2d).
  • Actionable evaluation: a "concretely designed evaluation protocol" with "transparent documentation and cost accounting" (Reviewer dboH); native readout and retrieval "give practitioners something to act on", and the guidance "is directly usable" (Reviewer W6VG).

The reviews raised two types of questions. Reviewer dboH asked whether the conclusions survive more variables. They do: with more features and other resolutions, the key results of the original draft hold. Reviewers Lt2d and W6VG asked what the results mean for Cell Painting: what is specific to it, and whether the generation findings hold outside natural-image features. The revision answers both with the following additions, marked in blue, and states the claims that the original draft left implicit:

Native readout and fidelity metrics

  • PSNR, SSIM and the new LPIPS join rFID as selectors in the main Table 2, which now reports which rendering and VAE each metric selects and how often it agrees with the readout (Reviewer dboH).
  • The readout grows from 24 radial targets to 108 CellProfiler targets in four families, read by ridge, MLP and RFF readers (Appendix Table 4), with a spatial-permutation control (Appendix Table 5). The native readout of the original draft is the radial family, and scores now carry the family as a subscript, e.g. Rrec,radial2R^2_{\mathrm{rec,radial}} (Reviewers dboH, W6VG).
  • The reconstruction analysis is repeated at five image sizes, by tiles and by block means (Appendix F.3) (Reviewer dboH).

Generation

  • gFID and siRNA retrieval are recomputed in five more feature spaces: ConvNeXt, CLIP, DINOv2 and the microscopy encoders OpenPhenom and uniDINO. The released IMPA–CellFlux reversal is also tested with CellProfiler retrieval on six-channel images regenerated from the released checkpoints (Appendix Tables 8 and 9) (Reviewers W6VG, Lt2d).
  • The new Table 3 tests whether the readout's choice of rendering predicts generator retrieval on held-out batches (Reviewers Lt2d, W6VG).
  • All 105 within-rendering contrasts on HUVEC512 are now tested against gFID (Appendix Table 9) (Reviewer W6VG).

Presentation

  • The new Table 1 states which measure serves each choice of rendering, VAE and generator (Reviewer Lt2d).
  • Sections 3 and 4 end with an explicit Implication, and Section 4 now states which reversals depend on the feature space (Reviewers Lt2d, W6VG).

The individual responses also report CellProfiler retrieval of our 22 six-channel generators and both generation routes trained to 500k updates (Reviewer W6VG).

We are grateful to the reviewers for their time and insight. Each point is answered in the individual responses, which clarify what is specific to Cell Painting, how the native readout separates signal loss from reader capacity, and how far the generation results extend across feature spaces. We are glad to discuss with anyone interested in this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.