Do fMRI Encoding Models Preserve Cross-Subject Transfer? Diagnosing and Reducing the Brain–Model Transfer Gap

Cross-subject transfer is important for fMRI encoding models, but whether model-predicted responses preserve the transfer capability of measured brain responses remains unclear. We propose brain-referenced transfer evaluation, which compares transfer from measured source-subject responses with transfer from model predictions under matched target adaptation. On NSD, model predictions from 4 backbones $\times$ 3 predictors transfer less effectively than measured responses, showing that training models to predict source responses does not ensure full transfer preservation. We probe this gap in the primary DINOv2–TBEn configuration, finding that predicted responses concentrate variance in fewer principal components (PCs) than measured responses and that leading PCs of measured responses support more transfer than the same number of model PCs. Correcting model predictions with a mapping learned from measured source responses narrows the transfer gap. These findings motivate Population Subspace Supervision (PSS), which upweights training errors in the leading source-population subspace. PSS closes 39.5% of the brain–model transfer gap, improves mean transfer across all 12 NSD configurations, and improves mean correlation in 11 of 12 BOLD5000 configurations. Together, these results highlight cross-subject transfer as a design objective for fMRI encoding model training alongside source-response prediction.