acceptodds
Under review as a conference paper at ICLR 2027

Relative Jacobian Nonlinearity for Interpreting Neural Encoding Models

Abstract

Similar neural responses can accompany different local dependencies on visual features. Describing these dependencies requires a measure of how an encoding model's mapping changes across images. We introduce relative Jacobian nonlinearity (RJN), which compares two local mappings on the scale of their combined sensitivity before averaging over images. This pairwise construction connects overall variation to magnitude disparity, weighted directional disagreement, and the stimuli contributing to each. In visual encoding models, the decomposition locates a fusiform contrast in directional variation, while stimulus scores distinguish mapping profiles among images with closely matched responses. Pair-relative and energy-based scores give almost the same voxel ordering but different image rankings; total RJN profiles are more consistent across alternative fits in the evaluated models. Regional contrasts vary with output reliability and sensitivity, and changing the feature perturbations can reverse the relation to predictive fit. RJN characterizes this relative mapping variation through its amount, form and distribution over images.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural encoding, interpretability, Jacobian, nonlinearity, visual neuroscience, fMRI, encoding models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_EmkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Relative Jacobian Nonlinearity (RJN), a measure of how an encoding model’s local feature sensitivity varies across stimuli. RJN compares geometry-weighted Jacobians using a pairwise discrepancy normalized by the combined sensitivity of each pair. It provides an additive decomposition into magnitude disparity and weighted directional disagreement, together with stimulus-level scores whose average recovers the overall measure.

The evaluation uses neural encoding models for four NSD participants, two visual representations, and twelve layers. The main findings concern regional differences, stimulus rankings beyond response amplitude, and the dependence of the relationship between predictive fit and mapping variation on the chosen perturbation geometry.

Critical strengths
  • Explicit treatment of relative Jacobian variation. RJN normalizes each pairwise Jacobian discrepancy by the combined sensitivity of the two mappings before aggregation. This provides a useful alternative to normalization at the level of the entire Jacobian field and makes the scale of each comparison explicit.
  • Useful decomposition of mapping variation. Decomposing the score into magnitude disparity and weighted directional disagreement provides a helpful framework for interpreting how local mappings change across stimuli. The additive structure connects these components to the overall score and supports more informative analyses than a single scalar summary.
  • Distinct stimulus-level results with improved rank stability. Despite nearly identical voxel rankings, RJN and (V_C) produce different stimulus-level rankings. RJN also shows higher overall rank consistency across the evaluated refits, suggesting that pair-relative normalization provides a complementary and potentially more reproducible view of how mapping variation is distributed across images.
Critical weaknesses

Major weaknesses

  • The directional term remains coupled to relative norm imbalance.. For positive Jacobian norms, writing m=kmag,aθ=1−cos⁡θ2m=k_{\mathrm{mag}}, a_\theta=\frac{1-\cos\theta}{2}, gives kdir=(1−m)aθk_{\mathrm{dir}}=(1-m)a_\theta. Thus, the directional term depends on both angular and norm difference. The authors explicitly acknowledge this weighting, and the decomposition is mathematically valid. However, a contrast carried by the directional term does not alone establish a difference in angular variability independently of norm difference.
  • After reading the paper, I think the new interpretive value that RJN can provide is not very clear. As the paper acknowledges, (V_C) also supports magnitude/direction decomposition and a mean-conserving stimulus allocation. Empirically, RJN and this baseline have mean voxelwise Pearson and Spearman correlations above 0.999, and the energy baseline reproduces the principal regional ordering, including the FFA contrast. The distinction is more apparent at the stimulus level: total-score rankings have median Spearman correlation 0.921 and top-100 overlap 0.84. RJN also improves median refit rank correlation from 0.898 to 0.931, although both methods have median top-100 overlap of 0.76. These are positive differences, but the paper could more clearly demonstrate what substantive interpretation becomes available through the changed ranking. A focused example explaining the significance of images selected differently by the two methods would strengthen the contribution. This does not require RJN to universally outperform (V_C), but it does require a clearer account of when the distinction matters.

Minor weaknesses:

  • The results section is difficult to follow in places. Text placement around figures and tables sometimes interrupts the reading order, making it difficult to locate the continuation of an argument. Several paragraphs also have many numerical results without a sufficiently explicit takeaway. A clearer progression from question to comparison to conclusion, together with a brief concluding sentence in each subsection, would improve readability.
  • Figure 4 needs more explainations.
  • The legend texts in figure 5G overlap with panel above and are partially clipped.
Questions for authors

Questions

  1. Which substantive conclusion or stimulus-level interpretation is enabled by pair-relative normalization but is not recovered by the corresponding energy allocation?
  2. Does the FFA-2 versus FFA-1 difference persist for unweighted angular disagreement (aθ=1−cos⁡θ2a_\theta=\frac{1-\cos\theta}{2} as mentioned above, solely angular difference) when comparing pairs with similar norm ratios?

Overall assessment

I am currently leaning toward weak rejection, although I consider this a borderline paper. The construction appears technically coherent, the presentation is careful, and the empirical controls are a strength. My main reservation is the significance of the contribution beyond the closely related energy baseline, together with the limited interpretive validation of the differences between them.

Reviewer_wPPPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how to describe changes in a fitted fMRI encoding model's local dependence on input features across different images, recognizing that two stimuli can produce nearly identical predicted neural responses while relying on distinct combinations of visual features. The authors argue that local mapping disagreements should be judged relative to the combined sensitivity of the specific pair of mappings being compared, rather than against the derivative scale of the entire stimulus field. This ensures that relative changes in weakly sensitive mappings contribute proportionally alongside those in strongly sensitive ones. To address this, the proposed method—Relative Jacobian Nonlinearity make—evaluates local feature sensitivity across reference images using a pairwise distance that splits mapping differences into magnitude disparity and directional disagreement.

Across cortical voxels, RJN orders outputs almost identically to normalized gradient-energy variation and produces a regional ranking distinct from sensitivity energy, with local mapping variation being overwhelmingly directional (averaging 89.8%89.8\% across visual regions). Regionally, FFA-2 consistently displays higher RJN than FFA-1 across all 96 model conditions, though this regional contrast shrinks or reverses when controlling for noise ceiling, predictive fit, and sensitivity energy, or when applying a sensitivity floor to the denominator. At the stimulus level, mapping scores correlate weakly with recorded fMRI responses, yet image pairs with nearly identical measured and predicted cortical responses can exhibit distinct local feature mappings. Finally, while higher predictive fit correlates with lower RJN under natural feature covariance geometry, switching to isotropic raw-coordinate geometry flips this relationship from negative to positive across most higher visual areas, demonstrating that the observed fit-nonlinearity link is heavily confounded by voxel noise ceilings and output sensitivity.

Critical strengths
  • The kernel derivation is clean, providing an exact additive split between magnitude disparity and directional disagreement. Discrepancies aggregate consistently from pairs to images to global scores.
Critical weaknesses
  • The paper fails to demonstrate that RJN provides meaningful added value over the existing baseline (VCV_C). On the evaluated models, the two measures are functionally interchangeable. Voxel rankings within visual regions are virtually identical because RJN mathematically reduces to a scaled version of VCV_C whenever sensitivity norms are equal. The magnitude component that captures these norm differences accounts for only ~10% of the overall score. Moreover, the core capabilities highlighted for RJN—decomposing scores into magnitude and direction, allocating variation to individual images, and driving the primary FFA regional contrast—are already fully supported by VCV_C. The remaining divergence reduces to minor shifts in image rankings and slightly higher repeatability across refitted models. However, consistency does not imply correctness, and the paper explicitly acknowledges that evaluating scientific accuracy would require an independent target benchmark. The toy analytic example demonstrates that the two metrics can disagree, but not that RJN yields a more useful answer. To establish a genuine contribution, the authors need a ground-truth validation—such as synthetic encoders with planted feature dependencies, targeted image edits, or alignment with measurable visual properties—showing that RJN recovers true functional structure where VCV_C and existing methods fail.

  • Normalizing by pair sensitivity gives noisy gradients full weight, causing the paper’s key findings to reflect optimization noise rather than cortical organization. Because near-zero Jacobians receive equal weight to high-sensitivity ones, estimation noise dominates in poorly fit voxels—as evidenced by gradient drift between refits exceeding the gradient magnitudes themselves. Consequently, regional RJN scores inversely mirror predictive fit (R2R^2), a relationship heavily confounded by voxel noise ceilings. This makes the headline FFA contrast exceedingly fragile: matching outputs on reliability and sensitivity shrinks the effect by ~75%, while applying a sensitivity floor or evaluating fixed voxel subsets frequently reverses it. Furthermore, image-level score variations correlate weakly with neural responses, and cross-participant consistency may simply stem from shared pretrained feature spaces. The authors should reframe their findings as properties of specific fitted network heads and restrict or drop the FFA claim.

Minor

  • Robustness checks evaluate 3- or 32-output ROI subsets, whereas primary results use full ROIs. These subsets fail to replicate the full-ROI FFA contrast direction consistently.
Questions for authors
  • Do FFA or PPA image scores correlate with any measurable, independent image properties (e.g., face presence, clutter, or object category)?

  • What specific research question regarding an encoding model can RJN answer that VCV_C cannot?

Reviewer_njRPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Not applicable.

Critical strengths

Not applicable.

Critical weaknesses

Despite having extensive experience with encoding models, I found most of the paper impossible to follow. I had to read sentences, including in the abstract, many times and still could not be sure what they meant.

Questions for authors

see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.