acceptodds
Under review as a conference paper at ICLR 2027

A Flow Matching Framework for Neural Representational Dissimilarity

Abstract

Neural representational dissimilarity quantifies differences between neural response distributions, and is essential for comparing neural codes across stimuli, brain areas, tasks, and models. Commonly used distance metrics involve different assumptions and are estimated with separate methods. Here, we show that a variety of distance metrics can be unified under a flow matching framework developed in deep generative models. That is, these distances arise as Jeffreys divergences under different velocity constraints. We find that flow matching has advantages for estimating distances involving complicated distributions and continuous variables. Furthermore, this framework enables the design of new distance metrics in a principled way. Together, flow matching provides a unified approach for understanding, estimating, and designing neural representational dissimilarity metrics.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Flow matching, Representational similarity analysis, Computational neuroscience, NeuroAI, neural coding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KwoaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a flow-matching framework that connects several neural representational distances through constraints on the velocity field. It also applies the method to Fisher information and time-dependent representational distances in the mouse visual cortex.

Critical strengths
  1. I find the unification of representational distances the strongest part of the paper. The authors show how constraints on the velocity field recover familiar distances, which is useful because they capture which features of the response distributions each distance cares about. This provides a useful way to think about metric choice in neural data analysis and is the main reason I lean toward acceptance.

  2. The continuous-condition applications give the framework practical value. Estimating Fisher information and time-varying distances from limited neural recordings is a relevant problem, and the simulation results are promising. The neural-data examples also show how the method could be used, though my concerns about the evaluation splits still need to be addressed.

Critical weaknesses
  1. I am not fully convinced that there is a benefit to the flow matching itself. In the non-gaussian experiments, most baselines assume gaussian responses, so they cannot capture the structure being tested. A conditional normalizing flow would be a useful comparison. Without this, it is hard to tell how much of the gain comes from flow matching itself. In other words, a conditional normalizing flow might also capture the higher-order structure in the gaussian mixture simulations and outperform the gaussian baselines without using flow-matching training.

  2. The Allen dataset's test-train splits raise concerns about the reported held-out performance. Wouldn't placing temporally correlated observations from the same trial in both training and test sets make the evaluation optimistic? Also, for the neuropixels dataset, PCA was fitted before splitting, unlike the other datasets. I would like to see trial or block-disjoint evaluation with training-only PCA, since these issues weaken the evidence for generalization to new trials.

Minor clarity issue: 3. Figure 1 is great, but in the caption, V\mathcal{V} is mentioned (line 39) without explaining what it is. Without the definition, that sentence is meaningless until a reader gets to Section 3 line 124 where it is defined as "velocity class", which is, by itself, still not clear what it means. Only after I got to Figure 2, I understood what it meant. The readability could be improved.

  1. Line 88: pθ(x)p\theta (x) was introduced without defining what p and x are.

  2. Table 1: "Auxiliary tau-dependent distance". What is this? What is this τ\tau?

Questions for authors

Can HSIC or CKA also be derived from the author's framework?

1 comment

We thank the reviewer for recognizing the value of our unification and the practical relevance of our continuous-condition applications, and for the constructive feedback. Please see our response below.

Q3.1:

I am not fully convinced that there is a benefit to the flow matching itself. In the non-gaussian experiments, most baselines assume gaussian responses, so they cannot capture the structure being tested. A conditional normalizing flow would be a useful comparison. Without this, it is hard to tell how much of the gain comes from flow matching itself. In other words, a conditional normalizing flow might also capture the higher-order structure in the gaussian mixture simulations and outperform the gaussian baselines without using flow-matching training.

R3.1:

Thank you for this suggestion. We have now compared flow matching with a conditional neural spline flow (Durkan et al., 2019; https://arxiv.org/abs/1906.04032) trained by maximum likelihood, using matched datasets and splits across five runs (Supplementary Figure S16, page 52, lines 2766–2775). Flow matching generally had lower estimation error. It is worth emphasizing that neural spline flow serves as one example method, and broader comparisons with other non-Gaussian estimators remain for investigation in future.

Our main contribution is the theoretical framework and for neuroscience applications. In neuroscience applications, popular estimators such as the Wishart process (for fisher information) impose explicit distributional assumptions that may lead to wrong result (Figure 4D to F). This is where unconstrained flow-based methods may have an advantage.

Q3.2:

The Allen dataset’s test-train splits raise concerns about the reported held-out performance. Wouldn’t placing temporally correlated observations from the same trial in both training and test sets make the evaluation optimistic? Also, for the neuropixels dataset, PCA was fitted before splitting, unlike the other datasets. I would like to see trial or block-disjoint evaluation with training-only PCA, since these issues weaken the evidence for generalization to new trials.

R3.2:

This is a good point. We agree that the original split could give optimistic held-out performance. We therefore re-ran all four methods across all 32 Neuropixels sessions using whole-presentation (“trial”) splits within each condition (12 training and 3 test trials when 15 were available), with the 70-component PCA fitted only on training responses (lines 1887–1892). Mean held-out likelihood is lower for all four methods, but their ranking is unchanged: unconstrained flow, affine flow, GKR, then Bin+LW. This preserves the conclusion of the likelihood comparison. Figure S14 shows the results (page 50, lines 2661–2666).

Q3.3:

Minor clarity issue: 3. Figure 1 is great, but in the caption, [V\mathcal{V}] is mentioned (line 39) without explaining what it is. Without the definition, that sentence is meaningless until a reader gets to Section 3 line 124 where it is defined as “velocity class”, which is, by itself, still not clear what it means. Only after I got to Figure 2, I understood what it meant. The readability could be improved.

Line 88: [pθ(x)p_{\theta}(x)] was introduced without defining what p and x are.

Table 1: “Auxiliary tau-dependent distance”. What is this? What is this [τ\tau]?

R3.3:

We clarified all three definitions, with changes marked in red: V\mathcal{V} is the set of allowed velocity fields, such as all fields affine in xx (Figure 1 and Section 3, lines 38–39 and 125); xx is a neural population response vector, and pθ(x)p_{\theta}(x) is its density conditional on a continuous stimulus or behavioral variable θ\theta (lines 88–89); τ\tau is an additional observed variable, such as time within a trial, at which conditions are compared (Table 1, lines 217–218 and 231).

Q3.4:

Can HSIC or CKA also be derived from the author’s framework?

R3.4:

CKA operates at a different level from the condition-to-condition distances studied in our framework. Our framework estimates distances between condition-specific response distributions to construct an RDM. Given two populations’ responses to matched conditions, CKA compares their centered kernel similarity matrices. This comparison across representations is not directly covered by our framework.

HSIC is also not directly covered by our current framework because it addresses a different comparison task. Our framework compares condition-specific response distributions, pa(x)p_{a}(x) and pb(x)p_{b}(x), defined on the same neural response space, so our results concern distributional dissimilarity across conditions. HSIC instead measures statistical dependence between paired variables XX and YY, which may take values in different spaces.

We thank the reviewer for highlighting this distinction and have added it to the Discussion (page 10, lines 493–498).

Reviewer_4d3yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers the problem of comparing neural response distributions (assuming responses are stochastic). The authors show that existing methods (eg euclidean distance or mahalanobis distance) can be unified under a flow-matching framework wherein restricting the kinds of movements the field can express imposes a choice about which distributional properties to retain. They show how the Jeffreys divergence between the fitted clouds after flow matching provides this unification: under different velocity constraints, it matches different representational distances. On simulated data with known fisher information, they show that flow matching can estimate it accurately. On real neural data (from Allen neuropixels), they recover intuitive findings such as the distances between grating conditions increasing after stimulus onset and decreasing after offset.

Critical strengths
  • The main strength of the paper is a useful mathematical framework for thinking about representational distances. the framework allows connecting several familiar representational distances through a unified flow-based formulation.
  • It also makes the interpretation of different metrics clearer as the velocity constraints make explicit which properties of neural representations each metric retains.
  • The paper is clearly written and easy to follow
  • Validation on synthetic data serves as a useful estimation check
Critical weaknesses
  1. Figures 4 and 6 are quite dense, and I found it difficult to follow what each set of panels is intended to establish. Clearer labels distinguishing the quantities being estimated, more guidance in the captions, and a brief statement of the main takeaway would make these results easier to understand.

  2. I found the real-data validation less convincing than the theoretical and simulation results. Better held-out likelihood supports improved density estimation, but does not directly establish more accurate Fisher information or representational distances. I view the mathematical unification as the main contribution, so stronger neuroscientific validation is not essential to the paper’s value. However, the distinction between better density fitting and better distance estimation should be clearer.

  3. The authors raise the intriguing possibility of designing new distances using their formulation which seems appealing, but I would appreciate more intuition about how to choose a source distribution and velocity constraints for a particular scientific question. A more directly motivated neural example would help eg what property should a new distance capture or ignore, and how would the proposed constraints achieve this?

  4. As described in Sections D, observations from the same trial can appear in both training and test sets. Given temporal correlations, particularly in calcium-imaging data, this could make held-out performance appear higher than it actually is? The Neuropixels analysis also fits PCA before splitting. Holding out entire trials and fitting PCA only on training data would provide a stronger assessment of generalization.

  5. the authors should try adding better baselines for the non-Gaussian experiments. The mixture simulations introduce higher-order structure that Gaussian estimators cannot capture, so an advantage for the unconstrained flow is expected. These experiments demonstrate the value of modeling non-Gaussian structure, but provide less evidence that flow matching is preferable to other flexible estimators. Although a density-ratio baseline is included in the categorical benchmark, comparable flexible baselines in the continuous-condition experiments could strengthen the empirical claims.

Questions for authors

See weaknesses above

1 comment

We thank the reviewer for recognizing our framework’s value in unifying representational distances, and for the constructive suggestions. Please see our response below.

Q2.1:

Figures 4 and 6 are quite dense, and I found it difficult to follow what each set of panels is intended to establish.

R2.1:

Thank you for pointing this out. We agree and have clarified the labels and revised the captions (Figure 4, lines 340–350; Figure 6, lines 394–405).

Q2.2:

I found the real-data validation less convincing than the theoretical and simulation results. Better held-out likelihood supports improved density estimation, but does not directly establish more accurate Fisher information or representational distances.

R2.2:

We agree that higher held-out likelihood alone does not establish more accurate Fisher information or representational distances.

We examined this relationship in simulations by comparing held-out log likelihood with estimation error (Figure S15, page 51, lines 2738–2747). Higher likelihood accompanies lower error for all methods in the Gaussian simulations and for unconstrained flow in the mixture simulations. These results are encouraging, but higher likelihood does not guarantee more accurate estimates. We now state this explicitly in Section 5.3 (page 9, lines 467–468).

Q2.3:

The authors raise the intriguing possibility of designing new distances using their formulation which seems appealing, but I would appreciate more intuition about how to choose a source distribution and velocity constraints for a particular scientific question.

R2.3:

Thank you for this comment. One possible application is studying how running speed affects grid-cell population responses. One can collect grid cell population responses within a speed bin but different spatial locations. The collected data can be used as a source distribution (which might have interesting toroidal topology, Gardner et al., 2022; https://doi.org/10.1038/s41586-021-04268-7). Translation and scaling-only velocity field can reflect gain changes, while rotation constraints can reflect orientation changes by running speed. Applying our framework to this complicated case may require non-trivial additional work, but we hope our framework set up a good starting point.

Another possible non-neuroscience application is comparing noisy point-cloud scans of a manufactured part with a known contour. To capture changes in position, orientation, or overall size while preserving shape, we can use a noise-broadened contour as the source and restrict the velocity to translation, rotation, and uniform scaling. Our square-template demonstration illustrates this approach (Supplementary Section C and Figure S12).

Q2.4:

As described in Sections D, observations from the same trial can appear in both training and test sets. Given temporal correlations, particularly in calcium-imaging data, this could make held-out performance appear higher than it actually is? The Neuropixels analysis also fits PCA before splitting. Holding out entire trials and fitting PCA only on training data would provide a stronger assessment of generalization.

R2.4:

Thank you for this suggestion. We agree that the original split could give optimistic held-out performance. We therefore reran all four methods across all 32 Neuropixels sessions using whole-presentation (“trial”) splits within each condition (12 training and 3 test trials when 15 were available), with the 70-component PCA fitted only on training responses (lines 1887–1892). Mean held-out likelihood is lower for all four methods, but their ranking is unchanged: unconstrained flow, affine flow, GKR, then Bin+LW. This preserves the conclusion of the likelihood comparison. Figure S14 shows the results (page 50, lines 2661–2666).

Q2.5:

the authors should try adding better baselines for the non-Gaussian experiments.

R2.5:

Following the reviewer’s suggestion, we have now compared flow matching with a conditional neural spline flow (Durkan et al., 2019; https://arxiv.org/abs/1906.04032) trained by maximum likelihood, using matched datasets and splits across five runs (see the newly added Supplementary Figure S16, page 52, lines 2766–2775). We find that flow matching generally had lower estimation error. It is worth emphasizing that neural spline flow serves as one example method, and broader comparisons with other non-Gaussian estimators remain for investigation in future.

Our main contribution is the theoretical framework and for neuroscience applications. In neuroscience, popular estimators such as the Wishart process (for fisher information) impose explicit distributional assumptions that may lead to wrong result (Figure 4D to F). This is where unconstrained flow-based methods may have an advantage.

Reviewer_vY8JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a common framework for neural representational dissimilarity: fit condition-dependent response distributions using flow matching under different velocity-field constraints, then compute Jeffreys divergence between the fitted distributions. It establishes theoretical connections between constrained velocity classes, existing distance metrics, and Fisher information. The experiments assess the estimation of distances and Fisher information on synthetic data, and apply the framework to estimate Fisher information and time-resolved representational dissimilarity matrices (RDMs) on real neural recording data.

Critical strengths
  1. The constrained-velocity interpretation provides a useful unification of existing distance metrics with solid mathematical justification. Sections 3–4 and Table 1 connect various modeling restrictions to the properties retained by different representational distances. This perspective is informative even where specialized estimators remain competitive, and further enables the design of new metrics. For practitioners, affine flows with a Gaussian source allow Gaussian means and covariances to be calculated without generating samples from the fitted model, providing a deterministic way to evaluate Jeffreys divergence.

  2. The synthetic experiments provide a quantitative assessment of estimation accuracy. Figures 4 and 6 report magnitude-sensitive errors across sample sizes and response dimensions with repeated experiments. The Gaussian comparisons provide evidence of improvements within the Gaussian-family estimators, and the mixture experiments demonstrate the value of capturing non-Gaussian structure. The neural recording datasets further demonstrate that the framework can be applied to realistic neural responses.

Critical weaknesses
  1. The connection between the theoretical contribution and the main applications is unclear to me. The real-data applications of constrained flows focus on affine classes for linear Fisher information and Gaussian Jeffreys divergence; the other constraints are demonstrated only on synthetic data. Meanwhile, for unconstrained flows, the procedure appears to combine existing flow-matching training with standard sampling and density evaluation to estimate Jeffreys divergence. It would help to clarify what is methodologically new in the unconstrained case, and how the velocity–distance correspondences contribute to it beyond providing a theoretical interpretation.

  2. The real-data evaluation in Section 5.3 needs stronger validation. For Figure 7B, individual time-bin observations are randomly split, so training and test data can come from the same presentation (Appendix D.3.3). For example, unusually high activity during one presentation could affect both subsets. PCA is also fitted using all observations, allowing test responses to influence preprocessing, unlike the training-only PCA in Section 5.2. Repeating the comparison with whole presentations held out and PCA fitted only to training data would strengthen the evaluation. For Figure 7D, attributing GKR’s high pre-stimulus correlations to excessive smoothing needs further support. High correlations can arise from tiny absolute errors that persist across time and do not, by themselves, establish excessive smoothing; lower correlations do not necessarily indicate more accurate estimates. For example, suppose there are three motion directions with identical pre-stimulus response distributions, so all three true pairwise distances are zero. However, estimating three pairwise distances as (0.0001, 0.0002, 0.0003) at two time points would produce a correlation of 1.

Questions for authors
  1. Contribution to practical estimation: For unconstrained flows, what does the proposed framework add beyond fitting conditional response distributions and estimating Jeffreys divergence from their samples and densities? How do the theoretical correspondences inform the applications?

  2. Evaluation clarification: Does the Figure 7B likelihood comparison give similar results when entire stimulus presentations are held out and PCA is fitted only to training data? What evidence supports attributing GKR’s high pre-stimulus correlations to excessive smoothing in Figure 7D?

  3. Likelihood and estimation accuracy: Figures 5B and 7B compare held-out likelihoods. Could the simulations in Figures 4 and 6 also report likelihood alongside errors in Fisher-information or distance estimation, to examine whether higher likelihood tends to accompany more accurate estimates? I appreciate the qualification in Appendix D.3.1 that likelihood does not directly measure Fisher-information accuracy; this additional check would help clarify the relationship between likelihood and estimation accuracy.

2 comments

We thank the reviewer for recognizing the value and mathematical grounding of our unification of representational distances, and for the constructive feedback. Please see our response below.

Q1.1:

Contribution to practical estimation: For unconstrained flows, what does the proposed framework add beyond fitting conditional response distributions and estimating Jeffreys divergence from their samples and densities? How do the theoretical correspondences inform the applications?

R1.1:

Unconstrained flow does not assume Gaussian responses, so with sufficient data it can capture distributional structure that Gaussian models would miss. For example, Ohiorhenuan et al. (2010) reported higher-order correlations among nearby visual-cortical neurons that changed with visual input (https://doi.org/10.1038/nature09178). Unconstrained flow can model non-Gaussian structure of this kind.

We agree that further examples would be needed in future, to demonstrate how the theoretical correspondences inform applications.

Our framework separates distance estimation into three components: a source distribution that defines the “distribution template,” a velocity constraint that defines the allowed transformations, and Jeffreys divergence between the fitted distributions. Users can choose the source and velocity constraint according to the scientific question and prior knowledge. For example, correlation emphasizes response-pattern direction after centering and normalization, Gaussian Jeffreys divergence also accounts for the covariance structure, and unconstrained flow can capture the non-Gaussian structure.

To strongly demonstrate how the theoretical correspondences inform applications, more complicated cases would be needed. One possible complicated application is studying how running speed affects grid-cell population responses. One can collect grid cell population responses within a speed bin but different spatial locations. The collected data can be used as a source distribution (which might have interesting toroidal topology, Gardner et al., 2022; https://doi.org/10.1038/s41586-021-04268-7). Translation and scaling-only velocity field can reflect gain changes, while rotation constraints can reflect orientation changes by running speed. Applying our framework to this more complicated case may require non-trivial additional work, but we hope our framework provides a solid starting point.

Q1.2:

Evaluation clarification: Does the Figure 7B likelihood comparison give similar results when entire stimulus presentations are held out and PCA is fitted only to training data?

R1.2:

Yes. We reran all four methods across all 32 Neuropixels sessions using whole-presentation (“trial”) splits within each condition (12 training and 3 test trials when 15 were available), with the 70-component PCA fitted only on training responses (lines 1887–1892). The method ranking is unchanged: unconstrained flow, affine flow, GKR, then Bin+LW. Figure S14 shows this comparison (page 50, lines 2661–2666). Thank you for raising this question.

Q1.3:

What evidence supports attributing GKR’s high pre-stimulus correlations to excessive smoothing in Figure 7D?

R1.3:

We agree that high pre-stimulus correlations alone do not establish excessive smoothing. We therefore softened our statement (page 9, lines 482–485):

“GKR, however, yields high RDM correlations even before stimulus presentation. The correlation rises by only 0.036 from before to during stimulation, compared with 0.652 for unconstrained flow and 0.411 for Bin+LW (Supplementary Fig. S13). This smaller increase suggests that GKR might oversmooth temporal changes in representational geometry.”

Q1.4:

Likelihood and estimation accuracy: Figures 5B and 7B compare held-out likelihoods. Could the simulations in Figures 4 and 6 also report likelihood alongside errors in Fisher-information or distance estimation, to examine whether higher likelihood tends to accompany more accurate estimates? I appreciate the qualification in Appendix D.3.1 that likelihood does not directly measure Fisher-information accuracy; this additional check would help clarify the relationship between likelihood and estimation accuracy.

R1.4:

To compare held-out likelihood with estimation error, we added Supplementary Figure S15 (page 51, lines 2738–2747). Higher likelihood accompanies lower error for all methods in the Gaussian simulations and for unconstrained flow in the mixture simulations which is encouraging.

Nevertheless, higher likelihood does not guarantee more accurate Fisher information or distance estimates. We now state this explicitly in Section 5.3 (page 9, lines 467–468).

Thank you for the thoughtful response and additional analyses. I appreciate the authors’ efforts, which have largely addressed my concerns about the evaluation.

  1. Contribution to neuroscience applications with unconstrained flow: I value the theoretical unification and appreciate the proposed grid-cell example, which helps clarify how the theoretical correspondences could inform applications. My original question was what the theoretical correspondences add to unconstrained estimation beyond standard flow matching with a Gaussian source, sampling, and density evaluation. The response explains the flexibility of unconstrained flows but does not identify an additional methodological contribution in this setting. Given the authors’ clarification to other reviewers that the “main contribution is the theoretical framework and for neuroscience applications”, I remain uncertain how much applying standard unconstrained flow to estimate Fisher information and representational distances contributes to neuroscience. Specifically, (i) held-out likelihood does not directly validate the accuracy of these estimates on real neural data (as the authors acknowledge); and (ii) my limited familiarity with the neuroscience literature makes it difficult to assess the novelty and significance of applying unconstrained flow to neural data.
  2. Data splitting and PCA: The whole-presentation splits and training-only PCA in Figure S14 largely address my concern about the likelihood comparison. However, Figure 7B–D still use the original protocol. For the revision, I suggest updating Figure 7B with the stricter comparison and checking whether the distance patterns and time-to-time correlations in Figure 7C–D remain similar under the revised protocol.
  3. Interpretation of GKR correlations: I appreciate the more cautious oversmoothing explanation and the comparison of changes in correlation. One minor numerical point: the reported means give an increase of 0.73-0.055=0.675 (lines 477–478), whereas the text reports 0.652 (line 483).
  4. Likelihood and estimation accuracy: Thank you for adding Figure S15 in response to my suggestion, though its implications deserve fuller discussion. In the mixture experiments, panels B and D show positive estimated likelihood–error correlations for Gaussian estimators. These results suggest that improved density fit can accompany worse estimation of the desired quantity. I’d appreciate an explicit discussion of this finding in the caption. Also, as the caption notes, S15 uses a “sample-size sweep”: the negative correlations may partly reflect improvements in both likelihood and estimation accuracy as more data become available. This does not by itself establish that, when comparing methods on the same dataset at a fixed sample size, higher likelihood indicates more accurate Fisher-information or distance estimates. I regard this mainly as a clarification now that the limitation is acknowledged. In addition, I suggest including the same caveat in Section 5.2 for the other neural-data experiment.

The additional analyses and clarifications have largely resolved my soundness concerns, so I will raise my soundness score to 3. My main remaining uncertainty concerns the contribution of using standard unconstrained flows in neuroscience applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.