acceptodds
Under review as a conference paper at ICLR 2027

Do fMRI Encoding Models Preserve Cross-Subject Transfer? Diagnosing and Reducing the Brain–Model Transfer Gap

Abstract

Cross-subject transfer is important for fMRI encoding models, but whether model-predicted responses preserve the transfer capability of measured brain responses remains unclear. We propose brain-referenced transfer evaluation, which compares transfer from measured source-subject responses with transfer from model predictions under matched target adaptation. On NSD, model predictions from 4 backbones 3 predictors transfer less effectively than measured responses, showing that training models to predict source responses does not ensure full transfer preservation. We probe this gap in the primary DINOv2–TBEn configuration, finding that predicted responses concentrate variance in fewer principal components (PCs) than measured responses and that leading PCs of measured responses support more transfer than the same number of model PCs. Correcting model predictions with a mapping learned from measured source responses narrows the transfer gap. These findings motivate Population Subspace Supervision (PSS), which upweights training errors in the leading source-population subspace. PSS closes 39.5% of the brain–model transfer gap, improves mean transfer across all 12 NSD configurations, and improves mean correlation in 11 of 12 BOLD5000 configurations. Together, these results highlight cross-subject transfer as a design objective for fMRI encoding model training alongside source-response prediction.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI encoding model, cross-subject transfer, brain-referenced evaluation, representational geometry, population subspace supervision, NeuroAI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EkUVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the critical yet under-explored problem of whether deep fMRI encoding models preserve the cross-subject transfer capabilities inherent to biological brain responses. To formalize this, the authors introduce the Brain-Referenced Transfer Evaluation framework, directly benchmarking model-predicted transfer against measured source-subject neural transfer under matched target adaptation conditions. Evaluating across 12 model configurations (4 vision backbones ×\times 3 neural predictors) on the Natural Scenes Dataset (NSD), the authors identify a consistent Brain–Model Transfer Gap and attribute it to representational differences (e.g., lower effective rank and under-represented leading principal components in model predictions). To address this, they propose Population Subspace Supervision (PSS), a loss formulation that preferentially upweights reconstruction errors along the dominant source-population PCA subspace (k=64k=64). The authors demonstrate that PSS closes 39.5% of the transfer gap on NSD and yields empirical improvements across most configurations on NSD and BOLD5000.

Critical strengths

Clear Problem Formulation and Closed-Loop Narrative: The study presents a complete, rigorous research pipeline—from defining the transfer gap and establishing a principled benchmark to diagnosing geometric representation bottlenecks and proposing a tailored training objective.

Novel Diagnostic Perspective: The brain-referenced evaluation framework shifts the standard paradigm from solely optimizing source-response prediction accuracy to explicitly probing whether models capture shared functional topologies across individuals.

Thorough Diagnostic and Validation Experiments: The geometric analyses (effective rank, PC transfer power, coordinate calibration, oracle replacement) provide meaningful mechanistic grounding. The empirical evaluation spans diverse architectures, predictor types, target adaptation setups, and cross-dataset validation on BOLD5000.

Critical weaknesses

1.The most significant novelty of this paper lies in proposing Brain-referenced Transfer Evaluation, which uses measured brain responses as a reference to quantify the cross-subject transfer capability of models—this problem formulation and evaluation perspective offers genuine novelty. However, the core method, PSS, is essentially a PCA subspace-weighted MSE (performing PCA on the ground truth, selecting the top-64 directions, and applying them to the MSE error to amplify the loss along these directions). Although the authors have proven in Appendix D.3 that it acts to adjust the effective regularization, and despite showing certain empirical effects in the experimental results, the analysis of its actual underlying mechanism remains insufficiently clear and lacks persuasiveness. The connection between adjusting the effective regularization and preserving the model's cross-subject transfer capability during training still requires deeper deconstruction and dissection.

2.The experimental design includes extensive cross-experiments between backbones and predictors, which thoroughly demonstrates the problem across frameworks and the generalizability of the proposed solution. However, regarding the validation of the effectiveness of this problem-solving solution, it is still desirable to see comparisons with the performance of cutting-edge research in this specific direction to further showcase its superiority. If existing methods are not compared due to insufficient protocol and framework compatibility, an explicit explanation and justification should be provided. In addition, mechanistic validation experiments for PSS may still need to be supplemented, such as: random PC subspace supervision, non-dominant PC subspace supervision, and simple anisotropic regularization training controls. If these random weightings that lack meaningful PCA directions also yield certain effects, it would weaken the claimed role of ordinary directional weighting and regularization effects.

3.Please supplement the specific reasons for selecting k=64k=64 in PSS.

4.In the experimental results, such as in Table 1, there are several instances where the performance gains are not obvious. For example, with AugReg + Ridge, Model + PSS only improves by 0.002 on the NSD-Shared1000 dataset. With AugReg + TBEn on BOLD5000, performance with PSS even deteriorates. While there are configurations with relatively large improvements, there are also many instances with minimal or no improvement, and several gains are obscured by standard deviation fluctuations. This has not yet sufficiently proven that such marginal gains are statistically significant, making it difficult to justify the necessity of retraining the models. Furthermore, the external validation gains are also limited in certain areas (as mentioned in the Appendix); the full support-range AUC in Appendix Table 4 actually shows improvements in only 10 out of the 12 configurations, with performance dropping in the other two (minor point). This weakens the consistency of the method's generalization performance. In addition, although it may be due to dataset limitations, it is still desirable to see more subjects (there are only four in the paper). While the overall data volume is not small, having only four subjects throughout is very unfavorable for solidly supporting the paper's core claims regarding generalization.

Questions for authors

see the weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer EkUV (1/4)

We thank the reviewer for recognizing the problem formulation, brain-referenced evaluation framework, mechanistic diagnostics, and empirical validation. Using evidence from the original manuscript and additional experiments, we address the concerns regarding mechanism, positioning relative to recent methods, control experiments, statistical evidence, and participant count.


1. Response to Weakness 1: How does directional PSS preserve cross-subject transfer?

The core of PSS is Brain-guided, non-uniform supervision: it increases the learning weight on dominant population coordinates and retains shared information with transfer value. Below, we connect effective regularization to transfer preservation using theoretical derivations, mechanistic diagnostics, and additional controls.

1.1 Non-uniform supervision and its additional benefit after global-regularization tuning

Ridge and weight decay apply a uniform regularization coefficient across directions; PSS (Section 3.4, Eq. 9) instead assigns greater weight to prediction errors along population Brain directions, focusing the model on recovering these coordinates.

For linear Ridge (Appendix D.3), effective regularization along supervised Brain-PC directions decreases from α\alpha to α/(1+ρ)\alpha/(1+\rho), while other directions remain unchanged. From the per-source-subject perspective, the supervision target is also pulled toward the population mean (Appendix D.2). This reduces shrinkage along dominant population coordinates, preserving shared signals that vary across images and making them available for adaptation to new subjects. For nonlinear models, this direction selection appears as greater gradient weight on errors in the population subspace (Appendix D.3). Unlike globally reducing regularization, PSS concentrates the additional learning weight on specified directions.

We further test whether this selectivity provides additional benefits after global tuning:

  • Linear Ridge: Source-only tuning of α\alpha increases transfer AUC from 2.1512 to 2.3205, recovering approximately 95% of the original PSS gain; adding PSS further increases it to 2.3371 (+0.0166; 4/4 targets improve).

  • Nonlinear TBEn: After source-only weight-decay tuning, the baseline AUC is 2.3086; Brain-PC PSS reaches 2.4835 (+0.1749; 4/4 targets improve).

Thus, the non-uniform directional supervision of PSS provides an additional benefit beyond global regularization.

1.2 The benefit of non-uniform supervision depends on direction selection

The coordinate interventions in the original manuscript (Section 4.2) show that correcting dominant Brain coordinates narrows the gap (Fig. 3C), whereas adjusting the variance spectrum along existing Model directions is insufficient to restore transfer (Appendix C.9.1).

We further compare Brain-PC PSS with random-subspace, Model-PC, non-dominant Brain-PC, and anisotropic-regularization controls (see Response 2.2). Brain-PC PSS achieves the best transfer performance.

Thus, dominant Brain-population directions carry the greatest cross-subject transfer value, and the PSS gain comes from directionally recovering these coordinates.

1.3 The transfer gain cannot be explained by improved overall source-prediction accuracy

The original manuscript (Section 4.3 and Fig. 4B) shows that PSS increases the mean coordinate-recovery correlation across the 64 supervised PCs from 0.1120 to 0.2332 (Appendix C.9.2).

We further calculate held-out source-prediction accuracy for the primary DINOv2-TBEn configuration. PSS decreases source NC-r from 0.4406 to 0.4127 and increases MSE from 5.4322 to 5.5585, while increasing target-transfer AUC from 2.2863 to 2.4452 (+0.1589).

Population-coordinate recovery and transfer therefore improve even as overall source-prediction accuracy declines. This shows that transferable population structure is the key determinant of a representation's value for adaptation to new subjects; PSS improves transfer by recovering this structure.

In summary, PSS non-uniformly increases the supervision weight on dominant Brain-population coordinates, strengthening the learning and retention of transferable shared information. The linear derivation, coordinate interventions, training dynamics, and additional experiments jointly establish its additional benefit after global tuning, the importance of direction selection, and the separation between source prediction and transfer.


AuthorsOpenReview ↗

Response to Reviewer EkUV (2/4)

2. Response to Weakness 2: Positioning relative to recent methods and mechanistic controls for PSS

PSS is a directional auxiliary objective compatible with different predictors and target adaptors. The original manuscript validates its benefits with recent predictors TBEn and UBE and multiple adaptors; additional controls establish dominant Brain-population directions as the source of its transfer gain.

2.1 Validation across recent prediction architectures and multiple target adaptors

Validation on recent architectures. The 12 evaluated configurations include the Transformer Brain Encoder (TBEn; Adeli et al., 2025) and the multi-subject Universal Brain Encoder (UBE; Beliy et al., 2026), and PSS is validated on both. For example, DINOv2-UBE transfer AUC increases from 1.687 to 2.332 (+0.644).

Comparison across target-adaptation methods. Appendix E.2 compares Ridge, RBF-KRR, and Residual MLP. PSS improves transfer in the primary configuration under all three adaptors, showing that its benefit generalizes across prediction architectures and target-mapping mechanisms.

Protocol differences from other methods. BraInCoRL performs in-context conditional prediction using target support samples (Yu et al., 2025); Neural Response Functions (NRF) use continuous MNI-space coordinates and adapt through target-subject fine-tuning (Chen et al., 2026). Both alter source representation and target adaptation; our protocol fixes the target adaptor, support samples, and query conditions to isolate source-representation transfer preservation. Directly comparing final scores would conflate representation transfer with adaptation mechanisms. We therefore conduct controlled validation with TBEn, UBE, and multiple target adaptors, and will clarify these complementary protocols in Related Work.

2.2 Random-subspace, non-dominant-PC, and anisotropic-regularization controls

We add direction and regularization controls in the primary DINOv2-TBEn configuration. All subspace-supervision experiments use matched initialization and training data, fix K=64K=64 and λ=0.25\lambda=0.25, and train for 1,200 updates; the global weight-decay and anisotropic-regularization controls do not use PSS.

Experimental controlMechanismHeld-out Source NC-rTarget NC-r AUCControl beats PSS
Tuned Weight DecayGlobal weight decay=10−510^{-5}0.44102.30860/4 targets
Random-Subspace supervision64-dimensional random orthogonal subspaces from 3 seeds0.44592.35420/12 target-by-seed combinations
Model-PC supervisionSupervision along the top 64 PCs of frozen baseline predictions0.43102.46001/4 targets
Non-dominant-PC supervisionSupervision along Brain PCs 65–1280.31792.19050/4 targets
Anisotropic regularizationNon-uniform readout weight decay with mean strength matched to tuned WD0.44142.30630/4 targets
Brain-PC PSSSupervision along the top 64 PCs of the biological population0.41472.4835-
  • Supervision direction affects transfer. Non-dominant Brain-PC supervision reaches 2.1905 AUC, below the tuned baseline (2.3086). Although Random-Subspace achieves the highest source NC-r (0.4459), its transfer AUC is 0.1293 below Brain-PC PSS; PSS wins all 12 target-by-seed combinations.

  • Dominant Brain-population directions provide additional value. Model-PC supervision (2.4600 AUC) and anisotropic regularization (2.3063) both underperform Brain-PC PSS (2.4835).

These controls establish the mechanism of PSS: its transfer gain comes from directionally recovering dominant Brain-population coordinates.


3. Response to Weakness 3: Rationale for selecting K=64K=64

K=64K=64 is a stable default that covers a broad dominant population subspace; stable gains across KK show that PSS captures transferable structure broadly distributed throughout this subspace.

Based on the PC-transfer and coordinate-recovery analyses in the original manuscript (Section 4.2, Fig. 3B; Section 4.3, Fig. 4B), we fix λ=0.25\lambda=0.25 and all other settings in DINOv2-TBEn and compare K∈16,32,64,128,256K\in\\{16,32,64,128,256\\}:

KK163264128256
Target NC-r log-shot AUC2.47682.47432.48352.49352.4951

Transfer performance is similar across K=16K=16–256256, with slightly higher values at 128/256. Thus, the effect does not depend on K=64K=64, which serves as a stable default.

The cross-session analysis in Appendix C.5 further shows that, among the first 64 PCs, 39–48 PCs in each target-fold-hemisphere unit jointly satisfy population consistency, cross-session reliability, and visual predictability, supporting supervision over this broad subspace. We will add the complete KK-sensitivity results to the revision.

AuthorsOpenReview ↗

Response to Reviewer EkUV (3/4)

4. Response to Weakness 4: Statistical evidence and expansion in participant count

The ±\pmSD values in Table 1 describe variation across target subjects, not uncertainty in the paired PSS−Model gains; overlap between standard deviations therefore cannot establish that the gains are insignificant.

We additionally conduct paired bootstrap analysis and combine it with the BOLD5000 external validation and the full-data NSD experiments with four and eight subjects to evaluate the robustness of PSS across adaptors, datasets, and participant counts.

4.1 Paired statistical evidence

We compute PSS−Model within each subject and perform 20,000 paired bootstrap resamples across target subjects:

Target adaptorPSS−Model AUC and 95% paired bootstrap interval
Ridge+0.1574 [0.0997, 0.2118]
RBF-KRR+0.1517 [0.0984, 0.1996]
Residual MLP+0.1506 [0.0933, 0.2079]

All three intervals are above zero. All 12 NSD configurations achieve positive mean AUC gains, with 11 improving for 4/4 targets; for AugReg-Ridge, AUC over the full support range improves by 0.1826, with gains for 4/4 targets. On BOLD5000, PSS improves full-support raw-r in 11/12 configurations and full-support-range AUC in 10/12. Thus, the transfer gains of PSS are statistically significant in the paired analysis and reproduce across configurations, support ranges, and an independent dataset.

4.2 Full source data and eight-subject expansion

The original BOLD5000 experiment already provides external validation with four subjects and 2,952 shared images under independent stimuli and acquisition conditions. To examine training scale and participant count, we additionally conduct full-data NSD experiments with four and eight subjects.

Increasing source-training scale substantially narrows the transfer gap, showing that data scale affects its magnitude. However, low effective rank and a residual transfer gap remain stable in the full-data four- and eight-subject experiments; using shared stimuli comprising only 3.6% and 1.43% of the training images, respectively, PSS still improves effective rank and mean transfer. These results show that low effective rank and the transfer gap are structural properties of current encoding models trained for prediction, rather than artifacts of small-sample training; by directionally supervising the Brain-PC population structure, PSS remedies this structural deficiency and yields cross-subject transfer gains across data scales.

4.2.1 Full-source-data training with four subjects

This experiment retains the LOSO setting and uses all valid training images from the three source subjects. Each outer unit contains, on average, 21,475 distinct images and 57,605 trials. DINOv2-TBEn uses K=64K=64 and λ=0.001\lambda=0.001 selected by source validation and trains for 2,400 updates; hyperparameters for all three target adaptors are selected only within the target support set.

Target adaptorBrain AUCModel AUCModel+PSS AUCPSS gainImproved targets
Ridge2.67082.59212.6021+0.01004/4
RBF-KRR2.70132.57972.5918+0.01214/4
Residual MLP2.64152.55952.5664+0.00703/4
ConditionSource NC-rEffective rank (top 64 PCs)
Brain reference-21.1745
Model0.57766.2742
Model+PSS0.58006.5505

Full-source-data training increases the Model's mean relative transfer score from 0.8537 to 0.9687, showing that more data recover part of the transfer capacity. However, the Model effective rank remains 6.2742, substantially below the Brain value of 21.1745; using only 3.6% of the training images, PSS still increases effective rank and improves mean transfer under all three adaptors. This establishes the value of representation structure independently of data scale: more training data still leave low effective rank and missing population structure, whereas PSS restores the transferable population subspace and yields cross-subject transfer gains under both linear and nonlinear target adaptors.

4.2.2 Full-source-data expansion to all eight NSD subjects

We extend the experiment to all eight NSD subjects, training on seven source subjects and evaluating on the held-out subject in each fold. Each outer unit contains, on average, 46,636 distinct images and 119,294 trials. DINOv2-TBEn uses λ=0.0005\lambda=0.0005, selected through source-only validation.

ConditionSource NC-rEffective rank (top 64 PCs)Target NC-r log-shot AUCRelative Brain transfer ratio
Brain reference-16.89242.693821.00000
Model0.588657.16402.584980.95949 [0.9440, 0.9731]
Model+PSS0.587997.30522.589190.96106 [0.9461, 0.9743]
Reviewer_vvm1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether fMRI encoding models, trained to predict source subjects' responses, produce predictions that are as useful as the measured source responses for transferring to a new subject. It proposes "brain-referenced transfer evaluation": fit matched Ridge adaptors from either model predictions (Model) or averaged measured source responses (Brain) to a target subject on the same support images, and compare held-out query performance across support sizes. On NSD Shared1000, across four backbones and three predictors, Model transfer falls well short of Brain transfer. Diagnostics attribute this to variance concentration in model predictions (much lower effective rank than Brain) and to poorer transfer from leading Model PCs; replacing Model coordinates in the leading Brain-PC subspace with a learned map narrows the gap, and oracle coordinates close it. The authors then propose Population Subspace Supervision (PSS), an auxiliary loss upweighting prediction error in that subspace, which closes a substantial part of the gap, improves every NSD configuration, and gives small improvements on most BOLD5000 configurations.

Critical strengths
  1. The diagnostic chain is carefully controlled: random Brain-PC subsets, spectrum replacement in both directions, shuffled coordinate maps, and an oracle, which together rule out the alternative that the gap is only a variance-spectrum effect.
  2. Breadth of validation: twelve configurations, nonlinear adaptors, support-only alpha tuning, and an external dataset (BOLD5000).
  3. The appendix derivation makes the method interpretable: PSS is a reweighting of errors inside the subspace and, for a linear predictor with Ridge, a reduced effective regularization along the supervised directions.
  4. Transparent reporting: individual targets are shown alongside aggregates, paired differences come with bootstrap intervals, and the authors state explicitly that the oracle condition is a hybrid response.
Critical weaknesses
  1. The "transfer gap" is not separated from ordinary source-prediction error, and the training regime likely inflates it. The Brain reference is the quantity the encoder is trained to approximate, so some gap is expected from prediction error alone, and the claim that prediction training "does not ensure" transfer preservation is informative only if the shortfall exceeds that. The Limitations acknowledge this, but it is not tested, and the Model's held-out accuracy on the source subjects is not reported. In addition, encoders are trained only on Shared1000, roughly 800 distinct images after the holdouts, versus the roughly 9k images per subject that standard NSD encoders use, and a rank collapse of this kind is what regression on noisy targets with limited data tends to produce. The measurements are not in doubt; what is open is whether the gap is a general property of prediction-trained encoders, as the headline claim and the motivation for PSS assume, or a feature of this low-data regime. Three additions would settle this: held-out source-subject NC-r for Model and Model+PSS; a noisier Brain reference whose source accuracy matches the Model's (for example by adding noise), to see whether it still transfers better; and the primary configuration retrained with the base loss on each source's full image set (PSS term kept on the shared subset), to see whether the collapse and the gap persist.

  2. The PSS gain is not shown to be transfer-specific rather than a generic regularization effect, and the lambda selection procedure is unspecified. By the paper's own derivation, PSS up-weights errors inside the subspace by a factor that grows with the number of vertices, which for the Ridge predictor equals weaker regularization along those directions with the target pulled toward the population mean. Either effect could improve the encoder in general, including source prediction, in which case PSS is a better encoder rather than a transfer-specific objective. Separately, the configuration-specific lambda and warmup values come without the procedure used to choose them; if they were selected on target-query transfer, the all-configurations result is optimistically biased. The first point concerns attribution, the second validity. Four small additions would clarify this: lowering alpha for the Ridge predictor; the same lambda on a random subspace of the same size and on the Model's own leading PCs; source-subject accuracy for Model versus Model+PSS; and a statement of how lambda and warmup were selected and on which data.

Questions for authors
  1. How were the configuration-specific lambda and warmup settings chosen, and on which data?
  2. At query images the Brain reference is built from held-out-session trials. Typically how many source trials contribute per query image, and how reliable is that reference (for example its correlation with the same images' responses from another session), so that it can be compared with the Model's source-prediction accuracy?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vvm1 (1/3)

We thank the reviewer for recognizing the research question, diagnostic experiments, and cross-model validation, and for suggesting specific tests. The reviewer hypothesizes that the transfer gap reflects ordinary prediction error or training scale, and that the PSS gain reflects generic regularization or target-data hyperparameter selection. Our additional experiments test and rule out each hypothesis, further supporting the proposed structural mechanism.

PSS improves target transfer without higher source-prediction accuracy. After noise injection matches Brain-reference accuracy to the Model, Brain retains approximately 69% of its original transfer advantage. With full four- and eight-subject data, low effective rank and the transfer gap remain, while PSS improves effective rank and mean transfer using shared images comprising only 3.6% and 1.43% of training images. PSS also provides additional gains after thorough global-regularization tuning and outperforms random-subspace, Model-PC, non-dominant Brain-PC, and anisotropic-regularization controls. All hyperparameters use source-only selection.

These results strengthen our central mechanism: current encoding models insufficiently preserve cross-subject shared population structure in real brain responses, producing low effective rank and a transfer gap. Directional population-subspace supervision allows PSS to remedy this structural deficiency and provide transfer value beyond data scale and global regularization. Detailed responses follow.


1. Response to Weakness 1: Low effective rank and the transfer gap persist after accuracy matching and data expansion

1.1 Held-out source-prediction accuracy (Model vs. Model+PSS)

In the primary DINOv2-TBEn configuration (K=64K=64, λ=0.25\lambda=0.25), PSS improves transfer while slightly reducing overall source accuracy:

ConditionSource NC-rSource MSETarget NC-r log-shot AUC
Model0.44065.43222.2863
Model+PSS0.41275.55852.4452
Change−0.0279+0.1262+0.1589

PSS decreases source NC-r by 0.0279 and increases MSE by 0.1262, while increasing target-transfer AUC by 0.1589. These opposite changes show that PSS improves not generic source fit, but source-response structure valuable for cross-subject adaptation. This supports our mechanism: transfer depends on preserving transferable directions of population variation.

1.2 Accuracy-matched Brain reference

To test prediction error directly, we inject independent Gaussian noise into the Brain reference. Five-fold source-only cross-calibration, without target responses, matches its source NC-r to the Model with a mean residual of −0.00054:

RepresentationSource NC-rTarget NC-r log-shot AUCFull-support target NC-r
Model0.44062.31020.6105
Accuracy-matched Brain0.44012.55810.6958
Difference−0.00054+0.2478+0.0853

(This section reuses the Section 1.1 Model predictions and pairwise re-evaluates all conditions with another fixed support sample set and within-support splits; hence the slight Model-AUC difference.)

After matching overall source accuracy, Brain transfer AUC remains 0.2478 higher, retaining approximately 69% of the original gap; full-support NC-r is higher for all four targets. This 0.2478 AUC advantage quantifies the transfer value of population structure: structure retained in Brain yields substantially stronger cross-subject transfer at the same overall accuracy.

1.3 Full source data and eight-subject expansion

The original BOLD5000 experiment already provides external validation with four subjects and 2,952 shared images under independent stimuli and acquisition conditions. To address training scale and participant count, we add full-data NSD experiments with four and eight subjects.

Expanding source training substantially narrows the gap, confirming that data scale affects its magnitude. Yet low effective rank and a residual gap persist with full four- and eight-subject data; using shared stimuli comprising only 3.6% and 1.43% of training images, PSS still improves effective rank and mean transfer. Thus, low effective rank and the gap are inherent structural properties of current prediction-trained encoding models, not small-sample artifacts; directional Brain-PC supervision remedies this deficiency and consistently yields cross-subject transfer gains across data scales.

1.3.1 Full-source-data training with four subjects

This LOSO experiment uses all valid training images from three source subjects, averaging 21,475 distinct images and 57,605 trials per outer unit. DINOv2-TBEn uses K=64K=64, source-validated λ=0.001\lambda=0.001, and 2,400 updates; all target-adaptor parameters are selected only within target support.

AuthorsOpenReview ↗

Response to Reviewer vvm1 (2/3)

Target adaptorBrain AUCModel AUCModel+PSS AUCPSS gainImproved targets
Ridge2.67082.59212.6021+0.01004/4
RBF-KRR2.70132.57972.5918+0.01214/4
Residual MLP2.64152.55952.5664+0.00703/4
ConditionSource NC-rEffective rank (top 64 PCs)
Brain reference−21.1745
Model0.57766.2742
Model+PSS0.58006.5505

Full-source-data training raises the Model's mean relative transfer score from 0.8537 to 0.9687, showing that more data recover substantial transfer capacity. Yet Model effective rank remains 6.2742 versus Brain's 21.1745; with shared stimuli comprising only 3.6% of distinct training images, PSS still increases effective rank and mean transfer under all three adaptors. This establishes structure's data-scale-independent value: more data still leave low effective rank and missing population structure, whereas PSS restores the transferable population subspace and consistently yields cross-subject gains with linear and nonlinear target adaptors.

1.3.2 Full-source-data expansion to eight NSD subjects

We extend the experiment to all eight NSD subjects, training on seven sources and evaluating the held-out eighth subject. Each outer unit averages 46,636 distinct images and 119,294 trials. DINOv2-TBEn uses source-only-selected λ=0.0005\lambda=0.0005.

ConditionSource NC-rEffective rank (top 64 PCs)Target NC-r log-shot AUCRelative Brain transfer ratio
Brain reference−16.89242.693821.00000
Model0.588657.16402.584980.95949 [0.9440, 0.9731]
Model+PSS0.587997.30522.589190.96106 [0.9461, 0.9743]

Model effective rank remains below Brain (7.1640 vs. 16.8924), and all eight targets have relative transfer ratios below 1, averaging 0.95949 [0.9440, 0.9731]. Using only approximately 668 shared images (1.43%), PSS raises effective rank to 7.3052 for 8/8 targets; transfer AUC rises by 0.00421 for 6/8 targets, with a 95% paired-bootstrap interval of [0.00084, 0.00762].

At larger scale, the eight-subject experiment reproduces the mechanism: Model effective rank and transfer are below Brain, while PSS improves both. Extending low effective rank, the gap, and the PSS gain to all eight NSD subjects provides larger-scale direct evidence for our structural explanation.

1.4 Number and reliability of Brain-reference measurements—Response to Question 2

For each query image, the Brain reference contains three measurements from each of three sources (nine trials total). We first average within source, then equally across sources, with no missing repeats. Support and query images use all source repeats; the Encoder, PSS teacher, and PCA use only valid source-training responses. We will correct Appendix A.2's statement that these responses come “from the held-out query pool.”

We also construct independent population views from non-overlapping even- and odd-session trials, each with 3–6 trials per image. Mean cross-session raw-r is 0.3307 and Model-to-view raw-r is 0.3355, providing an independent repeated-measurement reference for source prediction. The revision will distinguish this population-view raw-r from the individual-trial NC-r in Sections 1.1–1.2.


2. Response to Weakness 2: The PSS transfer gain comes from supervision of the Brain-PC-defined population subspace

2.1 Global-regularization controls

The reviewer's alternative hypothesis is that PSS only changes effective regularization strength, so sufficient tuning of ordinary regularization should reproduce its gain. We test this theoretically and empirically.

The analytic equivalence applies only to a linear Ridge source predictor and cannot explain PSS in our primary nonlinear DINOv2-TBEn configuration. Empirically, we use nested source-only α\alpha selection for linear Ridge and source-only weight-decay tuning for nonlinear TBEn; neither tuned baseline reaches PSS transfer performance.

Regularization strength therefore explains only part of the PSS gain. The Brain-derived population subspace provides directional population-structure information absent from global regularization, allowing further cross-subject gains after thorough tuning.

For linear DINOv2 backbone + Ridge source predictor, we keep data, features, and evaluation unchanged and conduct nested source-only selection over α∈1,3,10,30,100,300,1000\alpha\in\\{1,3,10,30,100,300,1000\\}. Of 20 outer target/fold units, 14 select α=30\alpha=30 and 6 select α=100\alpha=100:

Source predictor settingTarget NC-r log-shot AUC
Ridge, α=1000\alpha=10002.1512
Source-selected Ridge α\alpha2.3205
PSS, α=1000\alpha=1000, λ=0.25\lambda=0.252.3299
Source-selected Ridge α\alpha + PSS2.3371
AuthorsOpenReview ↗

Response to Reviewer vvm1 (3/3)

Lowering α\alpha recovers approximately 95% of the original Ridge PSS gain, so regularization explains most of the linear gain. However, adding PSS to the source-selected baseline still raises AUC by 0.0166 for 4/4 targets, showing that Brain-PC provides transfer information beyond scalar regularization even linearly.

For nonlinear DINOv2-TBEn, the same source-only principle tunes weight decay. Baseline AUC is 2.3086 versus 2.4835 for Brain-PC PSS; source NC-r is 0.4410 versus 0.4147. Despite lower source accuracy, PSS raises AUC by 0.1749, showing value beyond global tuning: Brain-PC reweighting strengthens the directions most valuable for cross-subject adaptation and converts this structural improvement into substantial gains.

Together, both results establish Brain-PC supervision's independent role: PSS uses the Brain-derived population subspace to convert directional structure inexpressible by global regularization into cross-subject transfer capacity.

2.2 Supervision-direction and anisotropic-regularization controls

For DINOv2-TBEn, we fix K=64K=64 and λ=0.25\lambda=0.25 and match initialization, data, optimizer, and 1,200 updates. Random-subspace results average three seeds; global weight-decay and anisotropic-regularization controls omit PSS.

Supervision/controlHeld-out source NC-rTarget NC-r AUC
Source-selected weight decay0.44102.3086
Random-subspace supervision0.44592.3542
Model-PC supervision0.43102.4600
Non-dominant Brain-PC supervision0.31792.1905
Anisotropic regularization0.44142.3063
Brain-PC PSS0.41472.4835

Brain-PC PSS exceeds random-subspace by 0.1293 and wins all 12 target-by-seed combinations; it exceeds Model-PC by 0.0235 for 3/4 targets. Non-dominant Brain-PC and anisotropic regularization also underperform it.

Random-subspace has the highest source accuracy but substantially lower transfer than Brain-PC PSS; Model-PC, non-dominant Brain-PC, and anisotropic regularization also underperform PSS. Brain-PC achieves the highest transfer and wins all 12 target-by-seed comparisons against random subspaces. This establishes the mechanism: dominant population directions in real brain responses carry the greatest cross-subject transfer value, and aligning supervision with them converts Brain-derived population structure into stable transfer gains.

2.3 Hyperparameter-selection procedure—Response to Question 1

Table 1 λ\lambda and warmup values use source-only inner validation; outer target subjects and held-out images/sessions never enter selection. Within each outer target/fold, each of three sources serves as pseudo-target while the other two train the model. Selection uses mean raw-r log-shot AUC across three pseudo-targets and five support orders; ties favor smaller λ\lambda, then shorter warmup.

Configuration-specific λ\lambda candidates span 0–2. Most configurations omit warmup; AugReg-TBEn selects from 0,600\\{0,600\\} and AugReg-UBE from 0,450\\{0,450\\} updates. K=64K=64; TBEn and UBE use fixed final checkpoints at updates 1,200 and 900; Table 1 target-adaptor α=10\alpha=10. Appendix D.4 will include all candidate grids.

Full-source-data experiments also use source-only selection. For four subjects, source-prediction raw-r selects λ=0.001\lambda=0.001 from 0.0005,0.001,…,0.05\\{0.0005,0.001,\ldots,0.05\\}; for eight, source pseudo-target transfer AUC selects λ=0.0005\lambda=0.0005 from 0,0.00025,0.0005,0.001,0.002,0.004\\{0,0.00025,0.0005,0.001,0.002,0.004\\}. Outer-target tests serve only final evaluation and never model or hyperparameter selection.


In summary, the experiments establish a consistent chain: encoding models insufficiently preserve transferable population structure in real brain activity, producing lower effective rank and the Brain–Model Transfer Gap; PSS uses Brain PCs to recover these directions and improve effective rank and transfer. Accuracy matching and full-data four- and eight-subject experiments show that prediction error and data scale affect gap magnitude, while low effective rank and the gap persist across accuracy levels, scales, and participant counts.

Regularization and direction controls further establish the mechanism: the Brain-derived population subspace provides directional transfer information unavailable to global regularization or other subspaces. PSS adds gains to tuned linear Ridge and nonlinear TBEn and performs best against random-subspace, Model-PC, non-dominant Brain-PC, and anisotropic-regularization controls. Source-only hyperparameter selection shows that these gains reflect better preservation of transferable brain-population structure.

If the additional results and clarifications above have addressed the reviewer's main concerns, we respectfully ask the reviewer to reconsider and accordingly raise the score for our paper.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer vvm1,

We wanted to follow up on our response. We have added the experiments you suggested to the revised manuscript. After matching source accuracy, Brain retained 69% of its original transfer AUC advantage (Appendix G.1). With full-source training, the transfer gap narrowed but persisted for all eight held-out subjects (Appendix E.4). We also added the subspace and regularization controls for PSS (Appendices G.2 and G.3).

We would appreciate your feedback on whether these additions address your main concerns and affect your overall assessment of the paper. We would be happy to clarify any remaining questions.

Reviewer_tzpdOpenReview ↗
Rating
3
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether model-predicted fMRI responses preserve the cross-subject transfer ability of measured responses.

It introduces a brain-referenced evaluation and observes a transfer gap across several model configurations. The analyses of response geometry motivate Population Subspace Supervision, which adds a loss on prediction errors in the leading principal-component subspace of source-population responses. Experiments show improved transfer in most evaluated settings.

Critical strengths
  • The quesiton of this paper studies is meaninngful.
  • The paper is easy to follow.
  • The experiments look comprehensive. It covers several backbone and models.
  • The proposed methods PSS is well motivated by the geometry analyses and coordinate-replacement experiments.
Critical weaknesses
  • The propose methods is kind of simple. It mainly reweights prediction errors within a PCA subspace.
  • I understand the effort of the experiments, but the number of participants is small, only four. Although the model coverage is broad, generalization to larger and more diverse populations remains uncertain.
Questions for authors

I have no further questions about this work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tzpd (1/2)

We thank the reviewer for recognizing the importance of the research question, the clarity of the presentation, the breadth of the experiments, and the mechanistic motivation for PSS. To address the concerns about the mechanistic specificity of PSS and participant count, we add supervision-direction and regularization controls, together with full-source-data experiments using four and all eight NSD subjects. We respond point by point below.


1. Response to Weakness 1: The simplicity and mechanistic specificity of PSS

PSS converts the diagnosed transfer mechanism into a simple, predictor-agnostic intervention. Its contribution lies in connecting brain-referenced evaluation and coordinate-level diagnosis to a targeted auxiliary objective that can be applied across predictors.

To test whether the gain depends specifically on the supervised subspace, we fix K=64K=64 and λ=0.25\lambda=0.25 in the primary DINOv2-TBEn configuration and match initialization, training data, optimizer, and the 1,200-update budget. Random-subspace results are averaged over three seeds; the global weight-decay and anisotropic-regularization controls do not use PSS.

Supervision/controlHeld-out source NC-rTarget NC-r AUC
Source-selected weight decay0.44102.3086
Random-subspace supervision0.44592.3542
Model-PC supervision0.43102.4600
Non-dominant Brain-PC supervision0.31792.1905
Anisotropic regularization0.44142.3063
Brain-PC PSS0.41472.4835

Brain-PC PSS exceeds random-subspace supervision by 0.1293 AUC and wins all 12 target-by-random-seed comparisons. It also exceeds Model-PC supervision by 0.0235, with improvements for 3/4 targets; non-dominant Brain-PC supervision and anisotropic regularization likewise underperform Brain-PC PSS. Random-subspace supervision achieves the highest source accuracy, yet its transfer performance remains substantially below PSS.

These controls establish the mechanistic specificity of PSS: dominant Brain-population directions yield the highest transfer performance despite lower source accuracy than random-subspace supervision. PSS therefore converts the population structure identified by our diagnostics into transferable representations through a simple, predictor-agnostic objective.


AuthorsOpenReview ↗

Response to Reviewer tzpd (2/2)

2. Response to Weakness 2: Full-source-data training and extension to eight subjects

The original BOLD5000 experiment already provides external validation with four subjects and 2,952 shared images under independent stimuli and acquisition conditions: PSS improves full-support raw-r in 11/12 configurations and full-support-range AUC in 10/12. To separate the effects of training scale and participant count, we additionally conduct full-data NSD experiments with four and all eight subjects.

Increasing source-data scale substantially narrows the transfer gap, confirming that data scale affects its magnitude. However, low effective rank and a residual transfer gap persist in both full-data settings; using shared images comprising only 3.6% and 1.43% of the training data, PSS still improves effective rank and mean transfer.

2.1 Four-subject full-source-data training

This experiment uses all valid training images from the three source subjects, averaging 21,475 distinct images and 57,605 trials per outer evaluation unit. DINOv2-TBEn uses K=64K=64, source-validated λ=0.001\lambda=0.001, and 2,400 updates; all target-adaptor parameters are selected only within target support.

Target adaptorBrain AUCModel AUCModel+PSS AUCPSS gainImproved targets
Ridge2.67082.59212.6021+0.01004/4
RBF-KRR2.70132.57972.5918+0.01214/4
Residual MLP2.64152.55952.5664+0.00703/4
ConditionSource NC-rEffective rank (top 64 PCs)
Brain reference-21.1745
Model0.57766.2742
Model+PSS0.58006.5505

Full-source-data training raises the Model's mean relative transfer score from 0.8537 to 0.9687, showing that more data recover substantial transfer capacity. However, Model effective rank remains 6.2742 versus Brain's 21.1745; using only 769 shared images (3.6%), PSS still increases effective rank and mean transfer under all three adaptors. This establishes the data-scale-independent value of representation structure: increasing training data still leaves low effective rank and missing population structure, whereas PSS restores the transferable population subspace.

2.2 Full-source-data extension to all eight NSD subjects

We extend the experiment to all eight NSD subjects, training on seven source subjects and evaluating on the held-out eighth subject. Each outer evaluation unit averages 46,636 distinct images and 119,294 trials. DINOv2-TBEn uses source-only-validated λ=0.0005\lambda=0.0005.

ConditionSource NC-rEffective rank (top 64 PCs)Target NC-r log-shot AUCRelative-to-Brain transfer
Brain reference-16.89242.693821.00000
Model0.588657.16402.584980.95949 [0.9440, 0.9731]
Model+PSS0.587997.30522.589190.96106 [0.9461, 0.9743]

Under the eight-subject full-data setting, Model effective rank remains substantially below Brain (7.1640 vs. 16.8924), and relative transfer remains below 1 for all eight targets. Using only 668 shared images (1.43%), PSS increases effective rank for 8/8 targets and raises transfer AUC by 0.00421 for 6/8 targets, with a 95% paired-bootstrap interval of [0.00084, 0.00762].

Together, these experiments establish that increasing data scale raises the overall transfer level but does not eliminate the structural gap: low effective rank and a residual transfer gap persist in full-data NSD, while PSS improves effective rank and transfer in both the four- and eight-subject cohorts. BOLD5000 further extends the transfer benefit to an independent dataset.


In summary, PSS is a simple but mechanism-specific intervention: it converts dominant Brain-population directions identified by our diagnostics into improved cross-subject transfer. The full-data NSD and BOLD5000 results extend this conclusion across training scales, participant cohorts, and datasets.

We hope these additional results and clarifications resolve the reviewer's main concerns and further strengthen the evidence for the paper's central conclusions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes to the Revised Manuscript

Dear Reviewers and Area Chair,

We have uploaded the revised paper PDF to OpenReview. Thank you for the suggestions. We have incorporated the following experiments and clarifications into the manuscript.

  • Full-source training with four and eight NSD subjects (Sections 4.1 and 4.3, Appendix E.4). Training on all available source images narrowed the Brain–Model transfer gap, and Brain still outperformed Model for all eight held-out subjects. Only 3.6% of source-training images in the four-subject experiment and 1.43% in the eight-subject experiment provided population targets for PSS. PSS improved mean Ridge transfer AUC by 0.0100 and 0.00421, respectively.
  • Source accuracy and target transfer (Appendices A.4 and G.1). We now report held-out source accuracy alongside target transfer. After we added noise to match the Brain reference to the model’s source accuracy, Brain retained 69% of its original transfer AUC advantage.
  • Supervision and regularization controls (Appendix G.2–G.3). The leading Brain PCs gave the highest mean transfer AUC among the tested subspaces, followed by leading Model PCs, random directions, and Brain PCs 65–128. We also added global and anisotropic weight-decay controls. After tuning the Ridge source predictor, PSS still improved transfer for all four targets.
  • PSS hyperparameters (Appendices D.1 and D.4). We added a sensitivity analysis for the number of supervised PCs and clarified how we selected the PSS weight and warmup using source subjects alone. We would appreciate hearing whether these revisions address your main concerns. If any questions remain, we would be happy to discuss them before the discussion period closes.

We would appreciate hearing whether these revisions address your main concerns. If any questions remain, we would be happy to discuss them before the discussion period closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.