acceptodds
Withdrawn from ICLR 2027

FIRM: Morphology-Guided Residual Experts for Fine-Grained Pathology Image Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained pathology image synthesis requires modeling both diagnostic distinctions across categories and morphological diversity within each category. However, conventional category conditioning specifies the target diagnosis while leaving within-class variation largely implicit. We introduce Feature-Informed Residual Modulation (FIRM), a framework that structures pathology representations into inter-class diagnostic variation and intra-class morphology for conditional generator adaptation. FIRM first learns a shared pathology prior and constructs a diagnostic and morphological geometry from fixed training-image features, decomposing within-class morphology into discrete modes and continuous residuals. These structured conditions modulate class-routed low-rank residual experts, allowing each diagnostic category to specialize the shared generator while preserving fine-grained morphological variation. We further introduce paired pathology feature refinement, which aligns predicted clean images with their corresponding real training crops in feature space without additional sampling trajectories. Experiments on BRACS and SICAPv2 show consistent improvements in class-conditional distribution matching and feature-space coverage over existing generative baselines. Controlled interventions further demonstrate discrete and continuous morphology manipulation, while synthetic augmentation improves downstream classification.

Primary area:
generative models
Keywords:
Histopathology Image Synthesis, Flow Matching, Structured Condition Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1TQkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The core question addressed by this paper is how to preserve within-class variation during digital pathology image generation. The paper used a method called Feature-Informed Residual Modulation framework. The propoed framework adds image features as conditions, and further decomposes these image features to inter-class, inner-class (including one discrete morphology mode and one residual morphology) modes.

The proposed FIRM framework is built upon three things:

  1. A fronzen generic image generator G0, whose backbone is FLUX, is trained to generate a general pathology image.
  2. An pre-defined orthogonal feature space is computed with directions of diagnostic axis B and a morphology axis P. Then UNI-h features m are projected into these two directions. Additionally, for each morphology vector, the model decompose it into mode (categorical) and continuous features.
  3. Gated LoRA modules are then finetuned upon G0 using features computed from step 2.

Experiment-wise, the paper compared FIRM with FLUX, LDM, PixCell methods, and showed improved FID, KID, and Coverage with these methods. Especially, Figure 5 showed that the method generate images more similar to the original image manifold.

Critical strengths
  1. The question discussed in this paper is meaninful and interesting. The motivation is well-defined, and the limitation for class-conditioned pathology generation is real.

  2. The conditioning mechanism, although convolutedly written, is technically conherent and interpretable. And the continouse controlling of the generated image is interesting.

  3. The experiments are comprehensive, including useful ablations and downstream classification tasks showing the utility of the generated images.

Critical weaknesses
  1. The term “morphology” is used too generic and is not sufficiently validated. For example, in Figure 7(a), changes induced by switching the discrete mode appear to involve substantial intensity variation, rather than obvious morphological changes. I am not necessarily opposed to using “morphology” as an umbrella term for non-diagnostic variation, potentially including both biological morphology and acquisition-related factors such as scanner or site effects. However, doing so actually weakens the interpretation of the imposed orthogonality constraint between B and P. Additionally, morphology itself is often diagnostic, while nuisance variation may also occupy the residual subspace. The paper would therefore benefit from stronger semantic validation of what the learned morphology dimensions actually represent.

  2. One of the most interesting contributions of the paper is the decomposition of UNI2-h features into inter-class diagnostic directions, discrete within-class modes, and continuous residual variation. The subsequent use of these quantities to condition a generative model feels somewhat downstream of this representation analysis. If the proposed geometry indeed captures meaningful diagnostic and within-class structure, its quality should arguably be demonstrable directly in feature space, independently of the generator. For example, the authors could evaluate how informative the diagnostic coordinates q_y, morphology coordinates s_i, or their combination are for classification, class separation, or other downstream tasks. This would help clarify whether the contribution comes primarily from discovering meaningful structure in the pathology representation or from the generative modeling architecture built on top of it. A direct evaluation of using decomposed feature representation for classification similar as Table 4 would be useful.

  3. Following 2, the proposed decomposition depends heavily on the UNI2-h feature space, but the choice of encoder is not sufficiently justified. As a result, the proposed decomposition may reflect properties specific to UNI2-h rather than a general structure of pathology images.

Questions for authors
  1. How do the authors validate that the subspace P specifically represents morphology?

  2. Why was UNI2-h selected to construct the diagnostic and morphological geometry? Have the authors performed any comparative analysis among different foundation model features?

  3. Can the authors evaluate the proposed decomposition directly, without the generative model? For example, how well do q_y and s_i, or their combination perform in predicting diagnostic labels or separating diagnostic categories?

  4. An important ablation appears to be missing: directly conditioning the generator on the original UNI2-h features without the proposed decomposition.

  5. [Nit] The feature-space overlap shown in Figure 5 is partly expected from the design of FIRM because it uses real image features as conditions.

Reviewer_wzkPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FIRM, which is a morphology-guided fine-grained image generation model with residual experts for the task of generating pathology images. For a chosen diagnosis, the model controls how the tissue can look within each category and generates variations of it. They first train a shared generator for pathology images. Then they use a feature extractor on real images, identify the appearance and patterns, and use them as guidance for the shared generator model to train small, diagnosis-specific expert adapters.

Critical strengths
  • The paper introduces an interesting approach for fine-grained generation and explicitly controlling the variations in pathology images with residual experts.
  • FIRM shows improvements over the two datasets and also shows nice ablations to support the discrete and continuous conditioning
  • The authors also show matched repeated-real and FLUX-based augmentations and improvements compared to the FLUX baseline.
Critical weaknesses
  • The results and experiments are interesting to see. The model does not directly establish whether the generated images still belong to the intended diagnosis as the morphology changes. The paper lacks expert verification by pathologists, which is important.

  • The controllability results for generating images within the mode are limited to three categories. It would also be helpful to clearly explain which tissue properties change, which would help in better understanding and strengthen the paper.

  • The proposed model is not sufficiently compared with simpler conditioning methods. Orthogonality to class-separating directions does not necessarily establish that the remaining variations capture the correct morphology. Comparisons to UNI2-h features and within-class PCA could help establish this.

  • The role of the geometric category code is unclear. If each category has its own expert and conditioning network that receives the same category code that gets the same cateogory code, it produces a constant category-gate contribution? Comparing this component with a directly learned constant gate would help establish its benefit.

  • Do the baseline training and ablations have similar training budgets and steps? In particular, PPFR should be compared with a similar baseline with the same number of additional flow-matching updates without the feature-matching loss, to separate its benefit from that of further training.

  • The PPFR mode trains to Inception features as an objective, and the evaluation metrics (FID, KID, etc.) are using the same inception features. It would be helpful to show that PPFR improves meaningfully in generating the tissue details beyond just the features it is directly optimising.

  • PPFR resembles existing perceptual objectives that compare predicted and real images through feature representations. The paper should clarify what is new about its objective or training procedure and compare it with a suitable existing perceptual loss.

  • The paper states the evaluation is done against the “entire real image pool." Does it include training images too, or is it evaluated against a held-out set?

  • Since FIRM uses training-image-derived conditions during generation, a comparison with the source images and nearest training examples would help assess how much additional variation the outputs provide. Does it generate images similar in appearance with variations or a near-duplicate?

  • The downstream classification task is limited to SICAPv2. An additional experiment on the BRACS dataset would establish whether the generated images benefit the difficult breast-lesion classification task used to motivate the paper.

  • The paper is too domain-specific for a general track of generative models. The qualitative results are hard to interpret without pathology expertise. Although the paper briefly discusses cellular detail and tissue organization, clearer annotations and explanations of the relevant visual features would help readers interpret Figure 4 and the morphology-control examples. The discussion should identify what changes, what is preserved, and how these observations support the claimed improvements.

Questions for authors

Please see the weakness section

Reviewer_4Sr1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a new flow based generative model specifically designed to generate pathology images. The argument is that this domain of images has certain specific properties that are difficult for existing methods to capture; e.g., the categorical nature (presence of disease or not) and the morphological diversity within each category. The overall method is a class conditional generative model which learns to visual concepts in a hierarchical way; by first learning a shared pathological prior and then the later stages learn the fine-grained features within each category. Qualitative and quantitative results are shown which depict the improved performance compared to baselines on two different datasets BRACS and SICAPv2.

Critical strengths
  • The overall idea of breaking down the generation process into a hierarchical structure is a logical step, where the general prior of the pathological images is learnt first (irrespective of the category) and then the finegrained details are filled in.

  • The proposed method scores better compared to the baselines on all the metrics reported in the paper. The authors have also extended the generic evaluation metrics like FID into their pathological version where the pre-trained network used to extract the representations is trained on pathological images. This version gives further credence to the quality of generated images.

  • The authors have also conducted the ablation study for the various components of the proposed method and have justified the use of all of them.

Critical weaknesses
  • The overall method is not that simple. It seems like a method designed for a very niche problem setup (generating pathological images). The overall stated goals of generating visual concepts in a hierarchical way seem generic enough, so I would imagine that this method should be usable for other domains as well. Ideally, those other domains could include non-pathological images as well. But even if the authors intended to remain within the realms of medical imaging, surely there are other datasets beyond BRACS and SICAPv2 where the proposed method could be compared to the same baselines? As of right now, it is difficult to see how general purpose this method is.

  • Regarding the quantitative results

    • While the authors have shown the results on FID and Coverage/Recall using two types of pre-trained networks – one using the default ImageNet pretrained network and one using a network trained on pathological images – the more relevant metrics are the one computed on network trained on pathological images, i.e., Path-F and Path-R. This is because the default FID and recall (coverage) scores are not going to be that meaningful since that network never saw pathological images. Now, when we consider Path-F, the metrics of all the proposed baselines are very high (between 300-1000s). So, while the method does achieve the best results, I am not that confident looking at those high FID scores. Furthermore, for Path-R, the method is almost the same as Flux. In short, the quantitative numbers are not completely convincing.
    • The authors have considered the recall metric, but they should also consider its counterpart, i.e., precision, which measures the fraction of generated images that are within the space of real image distribution.
  • While this is not a critical weakness, the authors should do a computational cost analysis of their method compared to other baselines. This will be especially helpful since the overall method is a little bit complicated.

Questions for authors
  • In what other domains should the proposed method be helpful, beyond the pathological images?

  • In figure 4, the images from Flux and the proposed method seem roughly aligned. What is the cause of this seeming alignment (e.g., what do #0, #1, #200, #201 mean)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.