- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the critical yet under-explored problem of whether deep fMRI encoding models preserve the cross-subject transfer capabilities inherent to biological brain responses. To formalize this, the authors introduce the Brain-Referenced Transfer Evaluation framework, directly benchmarking model-predicted transfer against measured source-subject neural transfer under matched target adaptation conditions. Evaluating across 12 model configurations (4 vision backbones 3 neural predictors) on the Natural Scenes Dataset (NSD), the authors identify a consistent Brain–Model Transfer Gap and attribute it to representational differences (e.g., lower effective rank and under-represented leading principal components in model predictions). To address this, they propose Population Subspace Supervision (PSS), a loss formulation that preferentially upweights reconstruction errors along the dominant source-population PCA subspace (). The authors demonstrate that PSS closes 39.5% of the transfer gap on NSD and yields empirical improvements across most configurations on NSD and BOLD5000.
Critical strengths
Clear Problem Formulation and Closed-Loop Narrative: The study presents a complete, rigorous research pipeline—from defining the transfer gap and establishing a principled benchmark to diagnosing geometric representation bottlenecks and proposing a tailored training objective.
Novel Diagnostic Perspective: The brain-referenced evaluation framework shifts the standard paradigm from solely optimizing source-response prediction accuracy to explicitly probing whether models capture shared functional topologies across individuals.
Thorough Diagnostic and Validation Experiments: The geometric analyses (effective rank, PC transfer power, coordinate calibration, oracle replacement) provide meaningful mechanistic grounding. The empirical evaluation spans diverse architectures, predictor types, target adaptation setups, and cross-dataset validation on BOLD5000.
Critical weaknesses
1.The most significant novelty of this paper lies in proposing Brain-referenced Transfer Evaluation, which uses measured brain responses as a reference to quantify the cross-subject transfer capability of models—this problem formulation and evaluation perspective offers genuine novelty. However, the core method, PSS, is essentially a PCA subspace-weighted MSE (performing PCA on the ground truth, selecting the top-64 directions, and applying them to the MSE error to amplify the loss along these directions). Although the authors have proven in Appendix D.3 that it acts to adjust the effective regularization, and despite showing certain empirical effects in the experimental results, the analysis of its actual underlying mechanism remains insufficiently clear and lacks persuasiveness. The connection between adjusting the effective regularization and preserving the model's cross-subject transfer capability during training still requires deeper deconstruction and dissection.
2.The experimental design includes extensive cross-experiments between backbones and predictors, which thoroughly demonstrates the problem across frameworks and the generalizability of the proposed solution. However, regarding the validation of the effectiveness of this problem-solving solution, it is still desirable to see comparisons with the performance of cutting-edge research in this specific direction to further showcase its superiority. If existing methods are not compared due to insufficient protocol and framework compatibility, an explicit explanation and justification should be provided. In addition, mechanistic validation experiments for PSS may still need to be supplemented, such as: random PC subspace supervision, non-dominant PC subspace supervision, and simple anisotropic regularization training controls. If these random weightings that lack meaningful PCA directions also yield certain effects, it would weaken the claimed role of ordinary directional weighting and regularization effects.
3.Please supplement the specific reasons for selecting in PSS.
4.In the experimental results, such as in Table 1, there are several instances where the performance gains are not obvious. For example, with AugReg + Ridge, Model + PSS only improves by 0.002 on the NSD-Shared1000 dataset. With AugReg + TBEn on BOLD5000, performance with PSS even deteriorates. While there are configurations with relatively large improvements, there are also many instances with minimal or no improvement, and several gains are obscured by standard deviation fluctuations. This has not yet sufficiently proven that such marginal gains are statistically significant, making it difficult to justify the necessity of retraining the models. Furthermore, the external validation gains are also limited in certain areas (as mentioned in the Appendix); the full support-range AUC in Appendix Table 4 actually shows improvements in only 10 out of the 12 configurations, with performance dropping in the other two (minor point). This weakens the consistency of the method's generalization performance. In addition, although it may be due to dataset limitations, it is still desirable to see more subjects (there are only four in the paper). While the overall data volume is not small, having only four subjects throughout is very unfavorable for solidly supporting the paper's core claims regarding generalization.
Questions for authors
see the weaknesses
5 comments
Response to Reviewer EkUV (2/4)
2. Response to Weakness 2: Positioning relative to recent methods and mechanistic controls for PSS
PSS is a directional auxiliary objective compatible with different predictors and target adaptors. The original manuscript validates its benefits with recent predictors TBEn and UBE and multiple adaptors; additional controls establish dominant Brain-population directions as the source of its transfer gain.
2.1 Validation across recent prediction architectures and multiple target adaptors
Validation on recent architectures. The 12 evaluated configurations include the Transformer Brain Encoder (TBEn; Adeli et al., 2025) and the multi-subject Universal Brain Encoder (UBE; Beliy et al., 2026), and PSS is validated on both. For example, DINOv2-UBE transfer AUC increases from 1.687 to 2.332 (+0.644).
Comparison across target-adaptation methods. Appendix E.2 compares Ridge, RBF-KRR, and Residual MLP. PSS improves transfer in the primary configuration under all three adaptors, showing that its benefit generalizes across prediction architectures and target-mapping mechanisms.
Protocol differences from other methods. BraInCoRL performs in-context conditional prediction using target support samples (Yu et al., 2025); Neural Response Functions (NRF) use continuous MNI-space coordinates and adapt through target-subject fine-tuning (Chen et al., 2026). Both alter source representation and target adaptation; our protocol fixes the target adaptor, support samples, and query conditions to isolate source-representation transfer preservation. Directly comparing final scores would conflate representation transfer with adaptation mechanisms. We therefore conduct controlled validation with TBEn, UBE, and multiple target adaptors, and will clarify these complementary protocols in Related Work.
2.2 Random-subspace, non-dominant-PC, and anisotropic-regularization controls
We add direction and regularization controls in the primary DINOv2-TBEn configuration. All subspace-supervision experiments use matched initialization and training data, fix and , and train for 1,200 updates; the global weight-decay and anisotropic-regularization controls do not use PSS.
| Experimental control | Mechanism | Held-out Source NC-r | Target NC-r AUC | Control beats PSS |
|---|---|---|---|---|
| Tuned Weight Decay | Global weight decay= | 0.4410 | 2.3086 | 0/4 targets |
| Random-Subspace supervision | 64-dimensional random orthogonal subspaces from 3 seeds | 0.4459 | 2.3542 | 0/12 target-by-seed combinations |
| Model-PC supervision | Supervision along the top 64 PCs of frozen baseline predictions | 0.4310 | 2.4600 | 1/4 targets |
| Non-dominant-PC supervision | Supervision along Brain PCs 65–128 | 0.3179 | 2.1905 | 0/4 targets |
| Anisotropic regularization | Non-uniform readout weight decay with mean strength matched to tuned WD | 0.4414 | 2.3063 | 0/4 targets |
| Brain-PC PSS | Supervision along the top 64 PCs of the biological population | 0.4147 | 2.4835 | - |
-
Supervision direction affects transfer. Non-dominant Brain-PC supervision reaches 2.1905 AUC, below the tuned baseline (2.3086). Although Random-Subspace achieves the highest source NC-r (0.4459), its transfer AUC is 0.1293 below Brain-PC PSS; PSS wins all 12 target-by-seed combinations.
-
Dominant Brain-population directions provide additional value. Model-PC supervision (2.4600 AUC) and anisotropic regularization (2.3063) both underperform Brain-PC PSS (2.4835).
These controls establish the mechanism of PSS: its transfer gain comes from directionally recovering dominant Brain-population coordinates.
3. Response to Weakness 3: Rationale for selecting
is a stable default that covers a broad dominant population subspace; stable gains across show that PSS captures transferable structure broadly distributed throughout this subspace.
Based on the PC-transfer and coordinate-recovery analyses in the original manuscript (Section 4.2, Fig. 3B; Section 4.3, Fig. 4B), we fix and all other settings in DINOv2-TBEn and compare :
| 16 | 32 | 64 | 128 | 256 | |
|---|---|---|---|---|---|
| Target NC-r log-shot AUC | 2.4768 | 2.4743 | 2.4835 | 2.4935 | 2.4951 |
Transfer performance is similar across –, with slightly higher values at 128/256. Thus, the effect does not depend on , which serves as a stable default.
The cross-session analysis in Appendix C.5 further shows that, among the first 64 PCs, 39–48 PCs in each target-fold-hemisphere unit jointly satisfy population consistency, cross-session reliability, and visual predictability, supporting supervision over this broad subspace. We will add the complete -sensitivity results to the revision.
Response to Reviewer EkUV (3/4)
4. Response to Weakness 4: Statistical evidence and expansion in participant count
The SD values in Table 1 describe variation across target subjects, not uncertainty in the paired PSS−Model gains; overlap between standard deviations therefore cannot establish that the gains are insignificant.
We additionally conduct paired bootstrap analysis and combine it with the BOLD5000 external validation and the full-data NSD experiments with four and eight subjects to evaluate the robustness of PSS across adaptors, datasets, and participant counts.
4.1 Paired statistical evidence
We compute PSS−Model within each subject and perform 20,000 paired bootstrap resamples across target subjects:
| Target adaptor | PSS−Model AUC and 95% paired bootstrap interval |
|---|---|
| Ridge | +0.1574 [0.0997, 0.2118] |
| RBF-KRR | +0.1517 [0.0984, 0.1996] |
| Residual MLP | +0.1506 [0.0933, 0.2079] |
All three intervals are above zero. All 12 NSD configurations achieve positive mean AUC gains, with 11 improving for 4/4 targets; for AugReg-Ridge, AUC over the full support range improves by 0.1826, with gains for 4/4 targets. On BOLD5000, PSS improves full-support raw-r in 11/12 configurations and full-support-range AUC in 10/12. Thus, the transfer gains of PSS are statistically significant in the paired analysis and reproduce across configurations, support ranges, and an independent dataset.
4.2 Full source data and eight-subject expansion
The original BOLD5000 experiment already provides external validation with four subjects and 2,952 shared images under independent stimuli and acquisition conditions. To examine training scale and participant count, we additionally conduct full-data NSD experiments with four and eight subjects.
Increasing source-training scale substantially narrows the transfer gap, showing that data scale affects its magnitude. However, low effective rank and a residual transfer gap remain stable in the full-data four- and eight-subject experiments; using shared stimuli comprising only 3.6% and 1.43% of the training images, respectively, PSS still improves effective rank and mean transfer. These results show that low effective rank and the transfer gap are structural properties of current encoding models trained for prediction, rather than artifacts of small-sample training; by directionally supervising the Brain-PC population structure, PSS remedies this structural deficiency and yields cross-subject transfer gains across data scales.
4.2.1 Full-source-data training with four subjects
This experiment retains the LOSO setting and uses all valid training images from the three source subjects. Each outer unit contains, on average, 21,475 distinct images and 57,605 trials. DINOv2-TBEn uses and selected by source validation and trains for 2,400 updates; hyperparameters for all three target adaptors are selected only within the target support set.
| Target adaptor | Brain AUC | Model AUC | Model+PSS AUC | PSS gain | Improved targets |
|---|---|---|---|---|---|
| Ridge | 2.6708 | 2.5921 | 2.6021 | +0.0100 | 4/4 |
| RBF-KRR | 2.7013 | 2.5797 | 2.5918 | +0.0121 | 4/4 |
| Residual MLP | 2.6415 | 2.5595 | 2.5664 | +0.0070 | 3/4 |
| Condition | Source NC-r | Effective rank (top 64 PCs) |
|---|---|---|
| Brain reference | - | 21.1745 |
| Model | 0.5776 | 6.2742 |
| Model+PSS | 0.5800 | 6.5505 |
Full-source-data training increases the Model's mean relative transfer score from 0.8537 to 0.9687, showing that more data recover part of the transfer capacity. However, the Model effective rank remains 6.2742, substantially below the Brain value of 21.1745; using only 3.6% of the training images, PSS still increases effective rank and improves mean transfer under all three adaptors. This establishes the value of representation structure independently of data scale: more training data still leave low effective rank and missing population structure, whereas PSS restores the transferable population subspace and yields cross-subject transfer gains under both linear and nonlinear target adaptors.
4.2.2 Full-source-data expansion to all eight NSD subjects
We extend the experiment to all eight NSD subjects, training on seven source subjects and evaluating on the held-out subject in each fold. Each outer unit contains, on average, 46,636 distinct images and 119,294 trials. DINOv2-TBEn uses , selected through source-only validation.
| Condition | Source NC-r | Effective rank (top 64 PCs) | Target NC-r log-shot AUC | Relative Brain transfer ratio |
|---|---|---|---|---|
| Brain reference | - | 16.8924 | 2.69382 | 1.00000 |
| Model | 0.58865 | 7.1640 | 2.58498 | 0.95949 [0.9440, 0.9731] |
| Model+PSS | 0.58799 | 7.3052 | 2.58919 | 0.96106 [0.9461, 0.9743] |
Response to Reviewer EkUV (1/4)
We thank the reviewer for recognizing the problem formulation, brain-referenced evaluation framework, mechanistic diagnostics, and empirical validation. Using evidence from the original manuscript and additional experiments, we address the concerns regarding mechanism, positioning relative to recent methods, control experiments, statistical evidence, and participant count.
1. Response to Weakness 1: How does directional PSS preserve cross-subject transfer?
The core of PSS is Brain-guided, non-uniform supervision: it increases the learning weight on dominant population coordinates and retains shared information with transfer value. Below, we connect effective regularization to transfer preservation using theoretical derivations, mechanistic diagnostics, and additional controls.
1.1 Non-uniform supervision and its additional benefit after global-regularization tuning
Ridge and weight decay apply a uniform regularization coefficient across directions; PSS (Section 3.4, Eq. 9) instead assigns greater weight to prediction errors along population Brain directions, focusing the model on recovering these coordinates.
For linear Ridge (Appendix D.3), effective regularization along supervised Brain-PC directions decreases from α to α/(1+ρ), while other directions remain unchanged. From the per-source-subject perspective, the supervision target is also pulled toward the population mean (Appendix D.2). This reduces shrinkage along dominant population coordinates, preserving shared signals that vary across images and making them available for adaptation to new subjects. For nonlinear models, this direction selection appears as greater gradient weight on errors in the population subspace (Appendix D.3). Unlike globally reducing regularization, PSS concentrates the additional learning weight on specified directions.
We further test whether this selectivity provides additional benefits after global tuning:
Linear Ridge: Source-only tuning of α increases transfer AUC from 2.1512 to 2.3205, recovering approximately 95% of the original PSS gain; adding PSS further increases it to 2.3371 (+0.0166; 4/4 targets improve).
Nonlinear TBEn: After source-only weight-decay tuning, the baseline AUC is 2.3086; Brain-PC PSS reaches 2.4835 (+0.1749; 4/4 targets improve).
Thus, the non-uniform directional supervision of PSS provides an additional benefit beyond global regularization.
1.2 The benefit of non-uniform supervision depends on direction selection
The coordinate interventions in the original manuscript (Section 4.2) show that correcting dominant Brain coordinates narrows the gap (Fig. 3C), whereas adjusting the variance spectrum along existing Model directions is insufficient to restore transfer (Appendix C.9.1).
We further compare Brain-PC PSS with random-subspace, Model-PC, non-dominant Brain-PC, and anisotropic-regularization controls (see Response 2.2). Brain-PC PSS achieves the best transfer performance.
Thus, dominant Brain-population directions carry the greatest cross-subject transfer value, and the PSS gain comes from directionally recovering these coordinates.
1.3 The transfer gain cannot be explained by improved overall source-prediction accuracy
The original manuscript (Section 4.3 and Fig. 4B) shows that PSS increases the mean coordinate-recovery correlation across the 64 supervised PCs from 0.1120 to 0.2332 (Appendix C.9.2).
We further calculate held-out source-prediction accuracy for the primary DINOv2-TBEn configuration. PSS decreases source NC-r from 0.4406 to 0.4127 and increases MSE from 5.4322 to 5.5585, while increasing target-transfer AUC from 2.2863 to 2.4452 (+0.1589).
Population-coordinate recovery and transfer therefore improve even as overall source-prediction accuracy declines. This shows that transferable population structure is the key determinant of a representation's value for adaptation to new subjects; PSS improves transfer by recovering this structure.
In summary, PSS non-uniformly increases the supervision weight on dominant Brain-population coordinates, strengthening the learning and retention of transferable shared information. The linear derivation, coordinate interventions, training dynamics, and additional experiments jointly establish its additional benefit after global tuning, the importance of direction selection, and the separation between source prediction and transfer.