Towards Interpretable Brain-Region Similarity Benchmarks

Comparing Deep Neural Network (DNN) representations to neural activity is a common way of evaluating computational models of the brain. Yet, design choices such as the alignment measure critically influence conclusions drawn from such benchmarks, which is a problem if we have no ground truth to validate a benchmark against. We propose to first sanity-check whether the benchmark is sensitive to the axes that differentiate the brain region under study, using region matching as a diagnostic test, and to require feature specificity of the similarity measure if the goal is to infer functional or mechanistic similarity from benchmark scores. We evaluate 10 similarity measures on both criteria using two large-scale fMRI datasets, the Natural Scenes Dataset and BOLD Moments. We find that region matching performance varies substantially by brain region and dataset. Region matching exposes preprocessing and measure implementation choices that degrade measure sensitivity. For example, applying PCA impairs performance of some measures and using Pearson-correlation distance reduces matching accuracy with RSA. Further, the test surfaces that region identification on fMRI data strongly depends on the univariate signal component, a component that has questionable relevance for identifying functional similarity to DNNs. Also, many commonly used measures lack feature specificity: They cannot distinguish a region from overcomplete supersets or incomplete subsets of the region. We recommend region matching as a sanity check before interpreting brain-DNN similarity scores, and to test for feature specificity when searching for precise model–region matches.