- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a framework for distribution-to-distribution generative modeling of scientific images, with particular emphasis on OOD robustness (different experimental settings) and uncertainty quantification. Stochastic flow matching, where standard flow matching is supplemented with a diffusion component and sinusoidal noise schedule is adopted as the generative model, while the authors derive the decomposition of uncertainties and use MC-Dropout for approximate posterior inference using the trained network. The method is evaluated on 3 benchmark image datasets and compared against several baselines, and appears to work well.
Critical strengths
- important and general problem in distribution-to-distribution (image) generative modeling
- the topic is somewhat dense technically, but the authors do a good job in exposing the background material in a didactic manner, with a clear progression of ideas, especially in the intro, related works, and preliminary sections. The setup (section 3) is an especially clear combination of mathematical notations and plain English explanations.
- the proposed solution seems reasonable, but I am not an expert at all on uncertainty decomposition other than having a conceptual understanding of the problem, so I cannot comment on the validity or novelty of the proposed method. That being said, the sequence of methodology introduced in 4.2-4.4 is again very well motivated and logical, in particular the necessity for AVUQ with the mirrored noise instances (though I also cannot comment on why this should work)
- empirically, the method seems to work very well compared to baselines, often by a significant amount compared to the next best (e.g., SDEdit)
Critical weaknesses
- the authors define reliability as specific instances of generalization (across unseen perturbations and experimental setups), and accountability as the ability to report uncertainty under unreliable settings. Is this standard? It makes sense to me but appears somewhat arbitrary, both the definition and the specific desiderata of optimizing these (and not other considerations) in distributional modeling.
- Table 1 does not report std / uncertainty on performance metrics so it's hard to evaluate how consistent the superior performance is.
- the experimental evaluation is comprehensive with multiple real data benchmarks and OOD scenarios, but very little space was left to describe and interpret the results. The main table benchmark numbers speak for themselves, but the example generated samples in Figure 3 are not very informative as I’m not sure what features, or differences between methods, are the relevant to focus on. More guidance on interpreting these results (either qualitatively or quantitatively) would be appreciated. In particular, the cell imaging benchmarks seem standard but it's unclear why this particular MRI dataset is included, and what the takeaway is here.
- it is taken (with cited works) that OOD inputs induces a variance reduction, but this seems to be a critical assumption
Questions for authors
- could you report variability over multiple seeds / runs in performance for Table 1?
- could you clarify or provide further context for why reliability and accountability are defined as such, and what aspects the current framework does NOT consider or prioritize in robustness and generalization for image generative modeling?
- could you explain in more detail the assumption that OOD → variance reduction, scenarios where this would be violated, and how the proposed framework would deal with such cases?
- could you expand on the qualitative interpretation of the main results on the 3 benchmarks, as well as the justification for the choice of benchmarks? In particular, there are hundreds of cognitive neuroimaging datasets out there, why this particular one and why would one assume low TOM cohort to be OOD in the brain scans? The example images also show anatomical scans without clear indication of neural activity, even though the task is to transform from resting to task activity.
2 comments
W4 / Q3: The assumption that OOD inputs induce a variance reduction: when is it violated, and how does the framework deal with it?
This is an empirical observation, not an assumption of the method: the estimators (Eqs. 14–20) do not depend on it; only the sign of the scalar anomaly score does (Sec. 4.4). It can be violated when a shift induces genuine posterior disagreement. To handle both directions without assuming either, we added a two-sided, label-free score, fixed in advance: each component's log-trace is standardized by the mean and standard deviation of the ID inputs, and an input is scored by its absolute z-score, so it is flagged when its uncertainty is unusual in either direction. No OOD input, label or direction is used. AUROC / AUPR:
| Method | BBBC021 Unseen Pert. | BBBC021 Intensity | JUMP Unseen Cell Lines | JUMP Intensity |
|---|---|---|---|---|
| MAP Aleatoric | 0.532 / 0.363 | 0.552 / 0.547 | 0.526 / 0.534 | 0.581 / 0.505 |
| SWAG Aleatoric | 0.639 / 0.517 | 0.681 / 0.711 | 0.534 / 0.540 | 0.636 / 0.576 |
| SWAG Epistemic | 0.639 / 0.502 | 0.677 / 0.701 | 0.532 / 0.539 | 0.624 / 0.566 |
| LA Aleatoric | 0.620 / 0.503 | 0.679 / 0.706 | 0.531 / 0.547 | 0.612 / 0.569 |
| LA Epistemic | 0.583 / 0.400 | 0.651 / 0.632 | 0.520 / 0.520 | 0.356 / 0.396 |
| MCD Aleatoric | 0.581 / 0.391 | 0.602 / 0.572 | 0.568 / 0.570 | 0.457 / 0.436 |
| MCD Epistemic | 0.556 / 0.371 | 0.576 / 0.549 | 0.553 / 0.554 | 0.446 / 0.429 |
| AVUQ Aleatoric | 0.554 / 0.381 | 0.585 / 0.569 | 0.581 / 0.579 | 0.592 / 0.512 |
| AVUQ Epistemic | 0.684 / 0.607 | 0.689 / 0.724 | 0.545 / 0.543 | 0.667 / 0.586 |
Same per-input scores, inputs and labels as Table 5. As expected, not assuming a direction reduces power relative to the one-sided score (e.g., AVUQ Epistemic 0.764 → 0.684 on BBBC021 Unseen Pert.), but AVUQ remains the best detector in all four cohorts, with the same component as under the one-sided score: Epistemic on BBBC021 Unseen Pert., BBBC021 Intensity and JUMP Intensity (AUROC 0.684 / 0.689 / 0.667), and Aleatoric on JUMP Unseen Cell Lines (AUROC 0.581).
We will discuss this explicitly in the paper, and we thank the reviewer again for the helpful suggestions.
We thank the reviewer for the positive assessment and the constructive comments.
Thanks for asking. These terms are not standardized for generative models; we use them as operational definitions motivated by deployment in scientific imaging. In standard ML terms:
The framework does not address (1) calibration of the uncertainty estimates, which requires paired source–target observations that are unavailable here; (2) shift types beyond those tested; (3) biological validation beyond the MoA classifier (Table 11). We have added these points to the Limitations (Sec. 6).
We agree that Table 1 should report variability across runs. Given the time constraints of the rebuttal period, we could not complete multiple training seeds for every baseline. We will report the corresponding multi-seed results (mean ± std) in the final revision.
Quantitative interpretation. FID and KID measure distributional similarity; whether generated cells carry the right biology is measured quantitatively by classifying the MoA of the generated images (Table 11).
Fig. 3. The relevant features are the compound-induced changes in cell and nuclear morphology and in the cytoskeleton. In the BBBC021 columns, the targets show enlarged, spread cells with a large nucleus (blue) and actin concentrated at the cell periphery (red). Under both the intensity shift and the combined shift, SFM generates this morphology, whereas SDEdit and UNSB largely keep the crowded layout of the source and CellFlux shows color artifacts. On JUMP Unseen Cell Lines, SFM preserves the sparse cell layout of the targets, whereas SDEdit and CellFlux generate densely packed cells; on JUMP Unseen Plates, the mildest shift, all methods produce plausible images. Under the intensity shift, both source and target test images are rescaled, so the question is whether the compound's morphology is still generated from inputs with unfamiliar intensities.
Benchmarks. BBBC021 and JUMP are standard benchmarks for perturbation-response generation (CellFlux, IMPA). We include fMRI to test whether the approach carries over to a second imaging modality with a subject-level cohort shift.
Why ds000228. It provides three properties this test requires: (i) source and target states within the same scan (an annotated baseline period and annotated ToM events), so source and target come from the same subject, analogous to control and perturbed cells; (ii) an independent behavioral measure (false-belief task performance) that defines the cohorts, so the shift is set by a non-imaging variable rather than by acquisition; and (iii) a movie-viewing paradigm suitable for young children, so the OOD cohort is also younger, an age range over which ToM-network responses become increasingly specialized (Richardson et al., 2018). Many cohort-shift datasets in medical imaging provide a single image type per subject and therefore do not define a source-to-target task.
Why low-ToM is OOD. The INC+FAIL children are younger (4.7 vs. 7.6 years) and have lower ToM scores (0.54 vs. 0.88). A model trained on PASS children therefore faces a population shift defined by an external, non-imaging variable rather than by image statistics.
Anatomical appearance. The images are single-TR BOLD slices. BOLD contrast is dominated by static tissue contrast, and task-evoked changes are only a few percent of the signal, so rest and ToM frames look similar by eye.
We have added this interpretation to Sec. 5.3 and the fMRI motivation to Appendix B.