The No Free Lunch of Self-Supervised Brain Models: Behavior Prediction, Learning Objective, and Data

Deep neural networks provide leading quantitative models of sensory systems. By circumventing the need for massive labeled datasets, self-supervised pretrained models are increasingly viewed as a free lunch for sensory system modeling. In this study, we evaluate this assumption by applying four categories of computational models—four acoustic, three semantic, three supervised, and eight self-supervised models—to predict human auditory cortical responses and perceptual behavior. Our results demonstrate that this lunch is far from free. In auditory cortex, model-class advantages shifted across the cortical hierarchy: supervised models generally showed stronger alignment in primary and secondary regions, whereas selected self-supervised models surpassed the strongest supervised baselines by approximately 25–30% in tertiary regions. Within the self-supervised class, cortical predictivity varied substantially across learning objectives and pretraining datasets, including among closely matched model families. This cortical advantage did not extend to perceptual behavior, where the strongest supervised DNN exceeded the strongest self-supervised model by approximately 54% and 57% in the sound- and word-judgment tasks, respectively. Self-supervised models also tended to peak at earlier network depths. Together, these findings reveal distinct trade-offs between cortical and behavioral alignment and highlight the importance of learning objective and pretraining data in sensory system modeling.