acceptodds
Under review as a conference paper at ICLR 2027

Alignment of visual brain encoding models with individual perceptual similarity judgments

Abstract

Encoding models that predict brain activation responses to various stimuli, including images, have traditionally been trained to maximize accuracy when predicting the observed brain data. However, in doing so, their internal representations may not reflect individual human perception. For example, two images that have very different predicted brain responses may be perceptually similar to a specific person. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with specific individuals' similarity judgments. This improved perceptual alignment may be useful in downstream tasks using the brain encoding model, particularly decoding of seen or imagined images. Using measured similarity judgments of 100 images from the 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each NSD participant to obtain their personalized perceptual similarity models. These personalized perceptual models are then used to obtain subject-specific similarity structure across the images in their encoding model training set. We fine-tune each participant's baseline encoder in two separate conditions, aligning its representations with either the generic or the personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and strongly preserve intersubject correlations present in measured fMRI, but fail to reflect the intersubject correlations of human perceptual similarity judgments. Fine-tuning brain encoding models with generic, population-level perceptual similarity information results in even better alignment with human perceptual judgments compared to baseline encoders, but these models still fail to capture intersubject correlation of human perceptual judgments. However, fine-tuning brain encoding models with personalized perceptual similarity information results in models that are overall the most aligned with human perceptual similarity judgments out of all the brain encoders, and also reflect intersubject correlation of human perceptual similarity judgments. Our framework provides a robust approach for bootstrapping human similarity judgment data into creating more individualized, human-aligned brain encoding models of vision.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain encoding models, behavioral alignment, perceptual similarity, fMRI, human visual perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RF3cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a method to align brain encoding models to individual-level similarity judgements. Their approach is straightforward: using a subset of the training data used to fit a baseline encoding model, fine-tune a perceptual similarity model (DreamSim) on individual similarity judgements. Then fine-tune the brain encoding model to reproduce both brain-similarity and perceptual similarity scores for said individual. The authors then valide their approach showing that indeed the correlation between encoder output responses and perceptual similarity increases in a statistically significant way. They further explored how image change the response categories in which they are before and after alignment (between all combinations of being similar/dissimilar in baseline/aligned models).

Critical strengths

The main strengths are that the method is clear simple and thus easy to follow. The authors also evaluate the approach in an overall rigorous way, showing that indeed perceptual similarity correlation between the aligned brain-encoding models increase.

Critical weaknesses

The main weakness I see is that there is no analysis of the potential trade-offs when aligning the brain-encoding models. Indeed, because the models are fine-tuned, they must necessarily induce some degradation in the ability of the encoders to predict brain activity. This is important because it establishes key questions about perception similarity, the underlying representations that support them, and the ability of encoding models to capture them.

Furthermore, the correlations between the model's predictions and the ground-truth human similarities remain relatively low (around .33), even if the increase with respect to the baseline models is statistically significant. This is probably because the noise ceiling is low, but there is no analysis of this either.

These weaknesses make me lean towards rejection.

Questions for authors

Apart from how to address the weaknesses above I have two questions:

  1. Have the authors thought about the implications that this research has towards the broader topic of using representation similarity to asses mechanistic process of information in the brain?
  2. The method seems very similar to an empirical Bayes approach: use the whole population data to fit a prior model, then compute per-participant conditional posteriors. In this case, the optimization approach creates an amortized version of this. How would this approach compare against this bayesian perspective?
1 comment
AuthorsOpenReview ↗

Response to Reviewer RF3c

We thank the reviewer for taking the time to give us helpful feedback and ask us questions about our work. We address all the comments below:

  1. We report this trade-off in voxel level accuracy as well as correlation to human judgments in absolute as well as relative terms. Please see Figure 7 for this.

    For a small cost in median voxel accuracy (from 0.303 to 0.277, −7.4%), fine-tuning toward the personalized model raises alignment with the subject's judgments by 0.237 (from 0.205 to 0.443, +194%). Alignment increases in all 8 subjects (by 0.09 to 0.35), while the accuracy change per subject ranges from −0.056 to +0.007. Fine-tuning toward the population model gives a smaller gain of 0.143 for a similar accuracy cost (0.024). Going from the population target to the personalized target costs almost no accuracy (0.002, n.s.) but adds 0.094 in alignment, so personalizing these encoders is nearly free.

    The loss weights and number of epochs are chosen explicitly at the knee of this trade-off on validation data. Therefore, in weighing the theoretical cost of making brain encoders human aligned, we think that the tradeoff for a small level of voxel level accuracy loss is well worth the upside of making sure the brain response space is more human aligned. Please see section 4.4, and appendix figure 7 for more clarity.

  2. With AligNet as the perceptual model, the correlations the reviewer mentions (around 0.33) are now higher: 0.461 for Personalized AligNet and 0.443 for the personalized encoder. We agree that the noise ceiling is needed to interpret them. Each participant completed only one arrangement session, so within-subject reliability cannot be measured in this dataset. However, a large Meadows study using a similar arrangement task with faces found that test–retest reliability for single participants averaged only ρ = 0.25 (Xu et al., 2026, Nature Human Behaviour). Individual similarity judgments are therefore inherently noisy.

    As a reference in our own data, we computed a group noise ceiling: each subject's held-out judgments correlate with the average judgments of the other 7 subjects at ρ = 0.375 (0.347–0.399 across the three cross-validation repeats), the lower bound of the noise ceiling (Nili et al., 2014). Generic AligNet is close to this value (0.361), while Personalized AligNet exceeds it (0.461). The personalized encoder also exceeds it (0.443), while the population encoder stays below it (0.348). This shows that the personalized models capture structure specific to each person that a population or group model cannot.

    A true within-subject ceiling would require repeated sessions, which we note as a limitation. Please refer to section 4.2, and appendix table 1 for more clarity.

  3. We appreciate the question of considering the implications of this project in studying brain mechanisms with representational similarity. To this, we look towards the ROI analysis found in Appendix Figure 8 to see that alignment gains are much higher in higher level regions. We believe that future work can be done in studying how each ROI relates to perceptual similarity and if further tuning at an ROI level can allow for brain encoding models that better represent the human brain’s readings.

  4. We agree that empirical Bayes is a very similar approach to our method and a helpful way to understand our methodology. As you noted, there are several key similarities. Each subject's reweighting vector is shrunk toward a population prior, generic AligNet (w = 1). The penalty in Eq. 2 acts like a Gaussian prior centered on w = 1, so our fit is close to a maximum a posteriori (MAP) estimate, with early stopping adding further shrinkage. Each subject's vector is optimized separately rather than amortized across subjects, and the strength of this shrinkage is learned from data by nested cross-validation, as one setting shared across subjects (λ = 0.1, about 20 steps).

    However there is a key difference in the methods. In empirical Bayes, the prior mean would be estimated from the participants in the study, but our prior is a general population model trained on other people's judgments. AlignNet is trained using the THINGS odd one out dataset, and DreamSim is trained using the NIGHTS dataset. These are not only different datasets with different subjects, images and training methodologies, but the behavioral task is not the same as well. Please refer to section 3.2 for clarity about our methodology.

Reviewer_FT1QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether brain encoding models, trained to predict fMRI responses, can also reflect how a specific person judges image similarity. Using the 8 NSD participants, the authors adapt DreamSim to each person from their judgments on 100 images (a feature reweighting fitted with cross-validation). They then use it to fine-tune that person's NRF encoder so that images the person sees as similar get similar predicted brain responses (an RSA loss plus a frozen-teacher MSE). Generic DreamSim serves as the comparison. On held-out images, the personalized encoder matches each person's judgments best, ahead of the population-DreamSim encoder and the baseline, and recovers the intersubject structure of the judgments, which the other two encoders do not. The cost is a modest drop in voxel prediction accuracy.

Critical strengths
  1. Clean experimental contrast. Fine-tuning the same initialized encoder with the same objective, changing only the source of the similarity targets (population vs. personalized), isolates the contribution of personalization. Combined with repeated cross-validation over held-out images and paired nonparametric tests, this makes the within-subject comparisons convincing and is the main reason the central claim is credible.

  2. Bootstrapping a small amount of individual behavioral data (100 images) through a personalized perceptual model to supervise an encoder trained on thousands of images is a sensible way around the cost of collecting judgments at scale. The finding that subject-specific encoders capture individual differences in brain responses but not in perception is itself a useful observation for the encoding-model community.

Critical weaknesses
  1. The personal benefit of personalization is not isolated from adaptation to the shared task and image set. Personalized DreamSim is fitted to the Meadows arrangement task on these 100 images as well as to the individual, so part of the personalized encoder's gain over the population-DreamSim encoder may be adaptation to the task and image set that all eight participants share. The model-by-subject matrix does not settle this, because each row changes both the encoder (trained on that person's brain data) and the target. This is a fixable gap rather than a flaw: the inter-subject result already shows that part of the benefit is personal; the open question is how much. Two controls fit the existing pipeline: fine-tune each subject's encoder with DreamSim adapted to the other seven subjects' pooled judgments, which isolates the shared component, and fine-tune it with another subject's personalized DreamSim (S1's encoder with S2's weights), which separates the target from the brain data. Evaluate both against the first subject's own judgments; the margin by which own-subject fine-tuning wins is the size of the personal benefit.
Questions for authors
  1. Voxel accuracy protocol. Is the accuracy drop measured as in the NRF paper (voxel-wise Pearson r on the held-out images, median across nsdgeneral voxels)? If not, which images and which summary statistic? Relatedly, the accuracy-vs-alignment tradeoff is reported as relative percentages of two different quantities; absolute per-subject changes (delta rho and delta r) would be more informative.

  2. Fine-tuning protocol. What determines the stopping epoch ("up to 15 epochs"), and were the held-out image pairs used to choose it or the loss weights? If so, the held-out alignment numbers would be optimistic; this is the one item here that bears on validity rather than interpretation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer FT1Q

We thank the reviewer for the helpful feedback and ideas for additional experiments. We address all the comments below:

  1. We ran both controls proposed by the reviewer. We fine-tuned each subject's NRF encoder toward two additional targets: (A) AligNet adapted to the pooled judgments of the other 7 subjects, and (B) another subject's personalized AligNet. The brain data, training images, loss and number of epochs were identical, so only the target changed, and every encoder was scored on the subject's own held-out judgments.

    Fine-tuning toward the subject's own personalized AligNet gave a +0.065 increase over the other-7-subjects target (8/8 subjects, p = 0.008). In contrast, the other-7-subjects target gave only +0.013 over the population target (p = 0.383). Splitting each subject's total gain, +0.157 comes from adapting to human perception in general, +0.013 from the shared task and images, and +0.065 from the individual, so adapting to the shared task and images explains little of the gain. The subject's own target also outperformed another subject's target (+0.087, 8/8, p = 0.008).

    The same pattern holds for AligNet alone, without brain data: adapting to the other 7 subjects adds nothing over generic AligNet (+0.001, p = 0.844), while adapting to the subject's own judgments adds +0.078 (8/8, p = 0.008). It also holds in the other three combinations: the own target beats the other-7-subjects target by +0.039 (NRF + DreamSim), +0.029 (linear + AligNet) and +0.022 (linear + DreamSim), all p < 0.05, while the other-7-subjects target is never significantly better than the population target (p ≥ 0.055). Please refer to section 4.2, the newly created Figure 3, and appendix Figure 9 for further confirmation of our results.

  2. We would like to confirm that we use the same the voxel-wise Pearson correlation between predicted and measured responses on held-out images, summarized by the median across voxels as the NRF paper. The voxels are those in the nsdgeneral mask, restricted to the 19 visual ROIs common to all eight subjects, and the images are the 100 Meadows images, which are not in the encoder's training set.

    As suggested, we now report absolute per-subject changes alongside the percentage changes. Fine-tuning toward the personalized model changes voxel accuracy by Δr = −0.026 (0.303 to 0.277; −7.4%) for a gain of Δρ = +0.237 in alignment (+194%). The population model costs about the same (Δr = −0.024; −6.4%) for a smaller gain (Δρ = +0.143; +127%). Going from the population to the personalized target costs no additional accuracy (Δr = −0.002, p = 0.148). This can be found in Appendix Figure 7 for relative and absolute changes per subject, as well as section 3.4 for a declaration of our definition for voxel-accuracy.

  3. We wanted to clarify our fine tuning protocol in this revision of the paper. To clarify, the personalization penalty λ and number of steps are chosen by nested cross-validation within the 80 training images of each fold, from 20 values of λ (0 to 100) at every step from 1 to 800 (chosen: λ = 0.1, about 20 steps).

    The loss-weight ratio λRSA/λteacher (0.05 to 2) and the number of fine-tuning epochs are chosen at the knee of the accuracy–alignment trade-off on validation data (chosen: ratio 0.2, 1 epoch). You can read more about these result sin section 3.2, section 3.3, and figure 6a-d.

Reviewer_ekwiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers if subject-specific encoding models can be fine-tuned in a way such that the representational geometry of their predicted voxel responses reflects an individual’s perceptual similarity judgments. A per-subject reweighting of DreamSim embeddings from individual judgments is learned on the subset of the NSD shared1000 dataset. Personalized fine-tuning roughly doubles within-subject alignment with held-out judgments and lets the encoders reproduce the intersubject structure of human judgments.

Critical strengths
  1. The paper proposes a clear and novel question about predicting a person’s brain responses versus reflecting their perception. It shows that baseline encoders do the first but not the second.

  2. The paper is well motivated, and the topic of neural encoding personalization is a new direction worth investigating.

  3. The personalization solution of applying a diagonal reweighting on top of frozen DreamSim is simple and data-efficient.

  4. The inter-subject structure of perception is reflected after fine-tuning at a negligible cost to encoding accuracy.

Critical weaknesses
  1. The paper combines standard techniques and components, so the contribution mainly lies in the empirical findings. However, its motivation for better decoding of seen or imagined images is never tested. All results also come from one dataset, one NRF encoder, and one perceptual model (DreamSim). An encoder-based identification experiment would strengthen the case considerably, for example, by checking whether errors become more perceptually similar to the target for that subject.

  2. Personalization is confounded with task adaptation. A DreamSim adapted to the average judgments of the other seven subjects would show how much of the within-subject gain comes from fitting the Meadows task and images rather than the individual.

  3. Fine-tuning design choices are neither explained nor ablated. The paper does not say which NRF parameters are updated or how the RSA term is computed over 4×1074\times10^7 pairs. It also does not test whether restricting the RSA term to higher-level ROIs would reduce the accuracy cost.

  4. The presentation is weak. There are no tables, so key results are scattered through the text, and the figures are poorly formatted, with numbers too small to read (e.g., Figure 3b and Figure 4).

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer ekwi

We thank the reviewer for offering essential criticisms and questions. We address each weakness here:

  1. We agree that the generality of our claims were limited by testing only one dataset, one brain encoder, and one perceptual model. After doing some experimentation, we switched our main perceptual model to AligNet (Muttenthaler et al., 2025) and added a second brain encoder, a linear CLIP-based encoder following BrainDiVE (Luo et al., 2023), and tested all four combinations.

    We are pleased to show that in every combination, fine-tuning toward the personalized model gives higher within-subject alignment than toward the population model (8/8 subjects, p = 0.008 in each): NRF + AligNet 0.443 vs. 0.348, NRF + DreamSim 0.367 vs. 0.290, linear + AligNet 0.353 vs. 0.306, and linear + DreamSim 0.314 vs. 0.270. Each subject's personalized encoder fits their own judgments best in 8/8 subjects for the NRF encoder and 5/8 for the linear encoder, vs. at most 2/8 for the other encoders. Personalized encoders are also the only ones that reproduce the intersubject structure of human judgments (Mantel ρ = 0.52, 0.41, 0.39 and 0.18; significant in three of four), while baseline and population encoders do not (−0.18 to 0.07, all n.s.).

    On the dataset, to our knowledge NSD is currently the only dataset that combines high-resolution fMRI for many images with similarity judgments from the same individuals. Please refer to section 3.5, section 4.6, and appendix table 1 for further clarification of these methods, and results.

  2. We agree that a decoding-based identification experiment would be the most direct test of our motivation. It requires building and validating a full decoding pipeline, which we believe is beyond the scope of this research project and is the next step for this work in creating human aligned brain decoders. We present decoding as motivation and future work.

    We believe that our image pair analysis shows a similar level of downstream practicality. Of the 7,600 held-out image pairs, fine-tuning toward Personalized AligNet moved almost twice as many pairs from disagreeing with the subject's judgments to agreeing (7.4%) as in the reverse direction (3.8%). For example, 177 pairs that humans rated as similar but the baseline encoder did not became similar for the fine-tuned encoder ("became similar" in Fig. 5b), and 385 pairs that the baseline encoder treated as similar but humans rated as dissimilar became dissimilar. Please refer to section 4.5 and figure 5 for more on this experiment.

  3. AligNet was adapted with the identical procedure to the pooled judgments of the other 7 subjects, so the subject's own judgments never enter. This model is no better than generic AligNet (+0.001, p = 0.844), while AligNet adapted to the subject's own judgments is +0.078 higher (8/8, p = 0.008).

    The NRF encoder shows the same pattern. Splitting each subject's gain, +0.157 comes from adapting to human perception in general, only +0.013 from the NSD Meadows task and images (p = 0.383), and +0.065 from the individual (8/8, p = 0.008). The own target also beats another subject's target (+0.087, 8/8, p = 0.008). In the other three combinations, the own target again beats the other-7-subjects target (+0.039, +0.029, +0.022; all p < 0.05), while the other-7-subjects target is never significantly better than the population target.

    As a result, we are able to confidently say that our models are able to adapt and personalize to the subject and not just adapt to the task or dataset. Please refer to section 4.2, Figure 3, and Appendix Figure 9 for further clarity.

  4. We also appreciate the reviewer bringing up clarity of tables, and figures as an issue in our paper. Since then, we have redrawn figures with larger fonts and added a table to consolidate results. Please check through all the figures and Appendix table to see our new work.

Thank you for your efforts in clarification and revision.

Some points still remain open after reading the author's response. The suggested identification test is encoder-based, so it does not necessarily need a decoding model. In addition, personalization lowers prediction accuracy and loses the fMRI inter-subject structure. For this reason, I'm not sure if the gain holds up against real brain responses.

I decided to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Remarks

We thank all the reviewers for their careful reading of our work and helpful feedback and criticisms. Based on this feedback, we have substantially revised our paper. The core method and the primary claims still remain the same:

  1. The baseline brain encoders predict each person's brain responses and preserve the intersubject structure of measured fMRI (ρ = 0.82), but do not reflect how individuals differ in their similarity judgments (ρ = −0.03 for the encoder, −0.19 for fMRI) (see section 4.1, and figure 2).
  2. A small amount of behavioral data is enough to personalize a perceptual model. Personalized AligNet is more aligned with each person's judgments than generic AligNet (0.461 vs. 0.361, 8/8 subjects, p = 0.008) (see section 4.2).
  3. Fine-tuning the brain encoder toward the personalized model raises within-subject alignment from 0.205 to 0.443, more than fine-tuning toward the population model (0.348; 8/8, p = 0.008), at a small cost in voxel accuracy (median r from 0.303 to 0.277) (see section 4.2, 4.4).
  4. Only the personalized models reproduce the intersubject structure of human judgments (Mantel ρ = 0.62 for Personalized AligNet and 0.52 for the personalized encoder; baseline and population encoders ≈ 0) (see section 4.3 and figure 4). 
  5. The gain comes from personalization, not from adapting to the task or the images (see new figure  3 and section 4.2).

The main changes are the following:

  1. First, our primary perceptual model is now AligNet (Muttenthaler et al., 2025). In response to Reviewer 1, we explored additional perceptual models and found that AligNet was more aligned with individual judgments than DreamSim (0.361 vs. 0.255 before personalization), so we switched to it. We also added a second brain encoder, a linear CLIP-based encoder following BrainDiVE (Luo et al., 2023). Across all four combinations of brain encoders and perceptual models, fine-tuning toward the personalized model beats the population model (8/8 subjects, p = 0.008 in each), and only personalized models reproduce the intersubject structure of human judgments (see section 4.5, section 4.6, and Appendix Table 1). Our original DreamSim results are essentially unchanged under the new protocol (personalized encoder 0.367 vs. 0.377 originally).
  2. Second, we added the two controls proposed by Reviewer 2 to separate personalization from task and dataset adaptation. These controls are AligNet adapted to the pooled judgments of the other 7 subjects, and another subject's personalized AligNet (see new Figure 3 and section 4.2). Adapting to the NSD Meadows task and images without the person adds almost nothing (+0.013, n.s.), while adapting to the person's own judgments adds a consistent +0.065 (8/8, p = 0.008). The same holds in all four combinations (see Appendix Fig. 9).
  3. Third, every hyperparameter is now chosen on validation data that never includes the held-out test images. These parameters are  the penalty λ (0.1) and number of optimization steps (about 20) by nested cross-validation, and the loss-weight ratio (0.2) and number of fine-tuning epochs (1) at the knee of the accuracy–alignment trade-off. (see section 3.2, section 3.3, and appendix figure 6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.