Relative Jacobian Nonlinearity for Interpreting Neural Encoding Models

Similar neural responses can accompany different local dependencies on visual features. Describing these dependencies requires a measure of how an encoding model's mapping changes across images. We introduce relative Jacobian nonlinearity (RJN), which compares two local mappings on the scale of their combined sensitivity before averaging over images. This pairwise construction connects overall variation to magnitude disparity, weighted directional disagreement, and the stimuli contributing to each. In visual encoding models, the decomposition locates a fusiform contrast in directional variation, while stimulus scores distinguish mapping profiles among images with closely matched responses. Pair-relative and energy-based scores give almost the same voxel ordering but different image rankings; total RJN profiles are more consistent across alternative fits in the evaluated models. Regional contrasts vary with output reliability and sensitivity, and changing the feature perturbations can reverse the relation to predictive fit. RJN characterizes this relative mapping variation through its amount, form and distribution over images.