- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a method to align brain encoding models to individual-level similarity judgements. Their approach is straightforward: using a subset of the training data used to fit a baseline encoding model, fine-tune a perceptual similarity model (DreamSim) on individual similarity judgements. Then fine-tune the brain encoding model to reproduce both brain-similarity and perceptual similarity scores for said individual. The authors then valide their approach showing that indeed the correlation between encoder output responses and perceptual similarity increases in a statistically significant way. They further explored how image change the response categories in which they are before and after alignment (between all combinations of being similar/dissimilar in baseline/aligned models).
Critical strengths
The main strengths are that the method is clear simple and thus easy to follow. The authors also evaluate the approach in an overall rigorous way, showing that indeed perceptual similarity correlation between the aligned brain-encoding models increase.
Critical weaknesses
The main weakness I see is that there is no analysis of the potential trade-offs when aligning the brain-encoding models. Indeed, because the models are fine-tuned, they must necessarily induce some degradation in the ability of the encoders to predict brain activity. This is important because it establishes key questions about perception similarity, the underlying representations that support them, and the ability of encoding models to capture them.
Furthermore, the correlations between the model's predictions and the ground-truth human similarities remain relatively low (around .33), even if the increase with respect to the baseline models is statistically significant. This is probably because the noise ceiling is low, but there is no analysis of this either.
These weaknesses make me lean towards rejection.
Questions for authors
Apart from how to address the weaknesses above I have two questions:
- Have the authors thought about the implications that this research has towards the broader topic of using representation similarity to asses mechanistic process of information in the brain?
- The method seems very similar to an empirical Bayes approach: use the whole population data to fit a prior model, then compute per-participant conditional posteriors. In this case, the optimization approach creates an amortized version of this. How would this approach compare against this bayesian perspective?
Response to Reviewer RF3c
We thank the reviewer for taking the time to give us helpful feedback and ask us questions about our work. We address all the comments below:
We report this trade-off in voxel level accuracy as well as correlation to human judgments in absolute as well as relative terms. Please see Figure 7 for this.
For a small cost in median voxel accuracy (from 0.303 to 0.277, −7.4%), fine-tuning toward the personalized model raises alignment with the subject's judgments by 0.237 (from 0.205 to 0.443, +194%). Alignment increases in all 8 subjects (by 0.09 to 0.35), while the accuracy change per subject ranges from −0.056 to +0.007. Fine-tuning toward the population model gives a smaller gain of 0.143 for a similar accuracy cost (0.024). Going from the population target to the personalized target costs almost no accuracy (0.002, n.s.) but adds 0.094 in alignment, so personalizing these encoders is nearly free.
The loss weights and number of epochs are chosen explicitly at the knee of this trade-off on validation data. Therefore, in weighing the theoretical cost of making brain encoders human aligned, we think that the tradeoff for a small level of voxel level accuracy loss is well worth the upside of making sure the brain response space is more human aligned. Please see section 4.4, and appendix figure 7 for more clarity.
With AligNet as the perceptual model, the correlations the reviewer mentions (around 0.33) are now higher: 0.461 for Personalized AligNet and 0.443 for the personalized encoder. We agree that the noise ceiling is needed to interpret them. Each participant completed only one arrangement session, so within-subject reliability cannot be measured in this dataset. However, a large Meadows study using a similar arrangement task with faces found that test–retest reliability for single participants averaged only ρ = 0.25 (Xu et al., 2026, Nature Human Behaviour). Individual similarity judgments are therefore inherently noisy.
As a reference in our own data, we computed a group noise ceiling: each subject's held-out judgments correlate with the average judgments of the other 7 subjects at ρ = 0.375 (0.347–0.399 across the three cross-validation repeats), the lower bound of the noise ceiling (Nili et al., 2014). Generic AligNet is close to this value (0.361), while Personalized AligNet exceeds it (0.461). The personalized encoder also exceeds it (0.443), while the population encoder stays below it (0.348). This shows that the personalized models capture structure specific to each person that a population or group model cannot.
A true within-subject ceiling would require repeated sessions, which we note as a limitation. Please refer to section 4.2, and appendix table 1 for more clarity.
We appreciate the question of considering the implications of this project in studying brain mechanisms with representational similarity. To this, we look towards the ROI analysis found in Appendix Figure 8 to see that alignment gains are much higher in higher level regions. We believe that future work can be done in studying how each ROI relates to perceptual similarity and if further tuning at an ROI level can allow for brain encoding models that better represent the human brain’s readings.
We agree that empirical Bayes is a very similar approach to our method and a helpful way to understand our methodology. As you noted, there are several key similarities. Each subject's reweighting vector is shrunk toward a population prior, generic AligNet (w = 1). The penalty in Eq. 2 acts like a Gaussian prior centered on w = 1, so our fit is close to a maximum a posteriori (MAP) estimate, with early stopping adding further shrinkage. Each subject's vector is optimized separately rather than amortized across subjects, and the strength of this shrinkage is learned from data by nested cross-validation, as one setting shared across subjects (λ = 0.1, about 20 steps).
However there is a key difference in the methods. In empirical Bayes, the prior mean would be estimated from the participants in the study, but our prior is a general population model trained on other people's judgments. AlignNet is trained using the THINGS odd one out dataset, and DreamSim is trained using the NIGHTS dataset. These are not only different datasets with different subjects, images and training methodologies, but the behavioral task is not the same as well. Please refer to section 3.2 for clarity about our methodology.