- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper explores whether conditional flow models can correctly recover non-additive interactions when extrapolating to unseen conditions. Here, the authors argue that aggregate metrics can be misleading, as interaction effects may be much smaller yet highly scientifically relevant. Decomposing predicted interactions into true interaction recovery and an orthogonal hallucinated component, the authors study a simluated setting, drug response data, and interacting single-cell perturbations.
Critical strengths
Strength 1. The paper reads well and presents the problem along with good perspectives. Both the settings and results were communicated effectively.
Strength 2. The study fits timely well. Both flow matching and interactions are increasingly relevant in scientific modelling. This makes understanding the properties of the models important.
Critical weaknesses
Weakness 1. The overall significance is somewhat limited. There are only few interesting results in the paper. Several main observations follow fairly directly from the interaction and identifiability setup. The strongest evidence comes from the controlled simulator while real data provided less convincing examples. I find it unclear whether the paper currently provides enough substance required for acceptance.
Weakness 2. It is unclear how specific the results are to the particular flow-matching recipe considered here (the authors also acknowledge this in the ending paragraph). It would be interesting to see a more general setting.
Questions for authors
Question 1. Beyond evaluation, could the paper guide changes to model design or training?
For a concrete example -- but it is fine to think of this question more generally -- could we make changes in objectives or architectural biases to avoid hallucinated effects?
Question 2. When estimating the interaction targets from finite experimental samples, how sensitive are the metrics for interaction recovery and hallucination to the empirical interaction estimates? For example, the double perturbations seen in Norman et al range from 49 to 1030 cells per condition, so we might expect empirical estimates to greatly vary.
5 comments
Reply 2/2
Can the paper guide model design or training? (Q1)
We appreciate this question and tested both modelling changes and new data, using the PPG LL corner:
| Change | Recovery | Hallucination | |
|---|---|---|---|
| Endpoint flow (manuscript) | .76 | .42 | .48 |
| Velocity flow | .67 | .40 | .30 |
| Diffusion | .78 | .42 | .51 |
| Regressor | .83 | .34 | .50 |
| Additional training and/or data: | |||
| Endpoint flow, trained further without new data | .82 | .31 | .43 |
| Endpoint flow + training on 16 new conditions, chosen at random | .50 | .80 | .39 |
| Endpoint flow + training on 16 new conditions, targeting response | .39 | .75 | .30 |
| Endpoint flow + training on 16 new conditions, targeting interaction | .32 | .81 | .26 |
- We find that modeling changes can lead to reductions in hallucination moreso than recovery. We attempted the requested objectives but were not able to achieve consistent gains: interaction-aware contrast and readout losses, and condition embeddings which process each factor separately. We will report these negative results in the appendix. We interpret these preliminary experiments to indicate that extrapolation is mainly determined by data.
- Indeed, only new data in the held-out region raised recovery (: .31 → .80). For the bottom row, we apply the paper's framing to data collection: at each step, we pick the condition that most reduces uncertainty about the interaction component, rather than the response (which yields instead of ). Here we use a Gaussian process over the two condition coordinates (only where training data exists). The decomposition shows that the advantage over random placement is due to lower hallucination ().
Sensitivity to the empirical interaction estimates (Q2)
We thank the reviewer for this suggestion and agree this is an important control. We estimated the noise in each measured interaction from 200 random half-splits of that pair's cells, and provide a noise floor (via a model predicting the true means). We then split all 131 pairs into thirds by the number of measured cells:
| Measured cells per double | Pairs | Model | Noise-floor | Model | Noise-floor |
|---|---|---|---|---|---|
| 49–205 | 44 | .87 | .42 | .46 | .76 |
| 208–291 | 43 | .90 | .35 | .60 | .86 |
| 292–1030 | 44 | .93 | .29 | .69 | .91 |
- Hallucination does not come from the poorly measured pairs, as is fairly stable across each third, while more cells leads to a lower noise floor.
- Recovery is lower for poorly measured pairs, which results from target noise diluting the measured interaction.
- We find the number of generated cells to not matter strongly. Moving from 192 to 768 changes by .02 and not at all (kindly see reply to reviewer x7eF for the full table).
Thanks, I actually think these are some interesting experiments. I however believe you need to work hard on making the additions more rigorous across recipes, as they don't yet feel completely like general takeaways yet. I believe it might be possible to do so for a camera ready, so I now lean towards weak accept.
Thanks for the effort, I hope you will continue working on refining it more!
Reply 1/2
We sincerely thank the reviewer for their evaluation. We have run new experiments for each of the raised points, which we address one by one below. First, we highlight the main additions:
Significance, and real data being less convincing (W1)
We agree that some observations follow from the setup: interactions are harder to learn than main effects, and less support makes extrapolation worse. However, two findings are not explained by distance from the training data alone:
For real data, we added a second system (molecular conformers) and a matched design for single-cell (Norman), in which the same held-out targets are scored under decreasing training support.
In both settings, the interaction error rises as support is removed while the targets stay fixed. As only the training support changes between conditions, these provide matched tests on real data.
How specific are the results to this flow-matching recipe? (W2)
We varied the recipe in both the simulator and the single-cell setting:
We therefore now present these failures as arising from training data that incompletely constrain the interaction, rather than from one particular recipe. We propose to keep flow models as the main subject of the paper, as they are what is deployed in the fields we study, and because they fail in a distinct way: unlike a regressor, they produce sharp, realistic samples whose interactions are wrong.