Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model–Brain Alignment

Alignment between vision models and the human brain is often evaluated by how accurately model representations predict brain responses. When two models achieve similar prediction accuracy, this measure does not establish whether they recover the same brain-response structure. We propose a framework for comparing models through their predictive subspaces in the target subject's response space. For each target subject and visual region, we construct a weighted reference from functional magnetic resonance imaging (fMRI) responses across multiple trials of the same images. In the main analysis, this reference averages predictive-subspace projectors obtained by ridge-regularized reduced-rank regression between trial-group means. A recovery profile measures each model's subspace overlap with the weighted reference directions. Differences between profiles obtained from other subjects' responses, matched for prediction accuracy within each target, provide a brain-to-brain benchmark for model differences. We analyzed responses to 515 shared natural images from eight Natural Scenes Dataset (NSD) subjects in early and intermediate visual areas (V1, V2, V3, and hV4). We compared ImageNet-pretrained and randomly initialized models of the same architecture, matched within 0.01 in held-out prediction accuracy. Pretrained models had higher mean recovery-profile scores in 25 of 29 pairs. The mean difference remained positive with an alternative reference based on signal covariance estimated by Generative Modeling of Signal and Noise (GSN). These results show that recovery profiles can reveal differences associated with pretraining, even when prediction accuracy is closely matched.