acceptodds
Withdrawn from ICLR 2027

MindBank: Data-Efficient Cross-Subject fMRI Decoding by Leveraging Shared Semantic Structure Across Source Subjects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-subject functional magnetic resonance imaging (fMRI) visual decoding aims to adapt decoders pretrained on source subjects to a new target subject using only a limited amount of paired fMRI–image data. However, limited data and large differences in fMRI responses across individuals can cause the target representation to lose semantic information during adaptation. We recognize that despite individual variability in fMRI responses, the underlying structure of visual semantics is shared across individuals. Consistent with this view, we observe that the decoded target representation is more similar to the average decoded representation across source subjects. We propose MindBank, which addresses the challenges by leveraging this similarity and the variability through two complementary mechanisms. (1) MindBank aggregates representations from frozen source decoders into a shared semantic reference and adaptively integrates the reference with target features. This mechanism keeps target adaptation consistent with the semantic structure shared across subjects. (2) MindBank exploits variations across the source representations to generate controlled perturbations of target representations. The perturbations expand the coverage of the target feature space and alleviate the limitations imposed by scarce target data. Experiments on the Natural Scenes Dataset (NSD) under a unified one-hour adaptation protocol demonstrate that Mind Bank uses the fewest trainable parameters, yet outperforms state-of-the-art methods in both retrieval and reconstruction. Code is included in the supplement and will be released.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
fMRI Decoding, Transfer, Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HcuyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses multiple frozen source subject decoding paths for the fMRI input of the same target subject, uses the average representation of the path output to constrain target adaptation, and combines reliability weighting, feature perturbation and category constraints to improve cross-subject visual decoding under one hour of target data conditions.

Critical strengths
  1. Source model constraints are introduced during training, and additional branches can be removed during inference.

  2. Compared to MindTuner, which replicates the same protocol, some reconstruction metrics show actual improvements. In Appendix Table 6, AlexNet-2, AlexNet-5, Inception, and CLIP show improvements of approximately 1.63, 1.42, 1.82, and 2.24 percentage points, respectively.

  3. Component ablation, parameter sensitivity, and baseline reproduction are provided. These experiments form the basis for further evaluation of the method's contribution.

Critical weaknesses
  1. The paper interprets "output consistency across different model paths" as "shared semantics across subjects," but these two concepts are not distinguished experimentally. Section 3.3 explicitly states that the Source Bank does not aggregate the actual brain responses of the same stimulus to different subjects, but rather zeros-padding or truncating the input of the same target subject before sending it to the projection layers of different source subjects. The authors also acknowledge that this step does not establish anatomical or functional correspondences between voxels. Therefore, the so-called source subject differences may be a mixture of pre-training parameter differences, voxel index mismatches, and common network biases, and cannot be directly interpreted as natural inter-subject neural variations. Figure 2 shows that the cosine similarity between the target representation and the source mean is higher than its similarity with a single source representation, supporting the interpretation of "extracting shared semantics." For example, the correlation value increases from 0.568 to 0.729. However, vector averaging itself may eliminate components in different directions, highlight common directions, and increase cosine similarity; common directions may also come from shared backbones or representational anisotropy, rather than correct stimulus semantics. What is currently established is that multi-path representations have geometric commonalities, but what is not yet established is that this commonality corresponds to transferable visual semantics.

  2. The mathematical implementation of “expanding feature coverage using source divergence” is essentially radial scaling. Equation 8-11 first compresses the cross-source variance into a scalar, then performs the following operation:

z^t=μsrc+v(zt−μsrc)\hat z_t=\mu_{\mathrm{src}}+v(z_t-\mu_{\mathrm{src}}).

This operation preserves the direction of the target residual, only adjusting its length. It does not use the directional structure of the cross-source covariance, nor does it sample along the actual source difference direction; when (v<1)(v<1), it even exhibits contraction towards the source center. Therefore, “expanding the distribution along a reasonable cross-subject variation” is not a property naturally derived from the formula; ordinary norm perturbations or contraction regularization could also explain the benefit. Ablation by removing SDA cannot replace comparisons with these simple enhancements. The authors need to provide the value distribution, truncation rules, and realization boundaries, and cannot simply use the name “reliability calibration” to replace mathematical guarantees.

  1. In Table 3, Subject 2's baseline after disabling the new components reached a brain retrieval rate of 91.00%, while MindEye2 in Table 1 only achieved 67.18%. The complete MindBank sample reached 92.37%, only a 1.37 percentage point increase compared to a truly close fit baseline; image retrieval, however, decreased from 95.12% to 92.47%. Component results also did not consistently support the complete combination. After removing RC, brain retrieval was 92.49%, slightly higher than the complete model; after removing SDA, it was 92.42%; after removing CSA, image retrieval reached 94.62%, higher than the complete model. Current ablation methods mainly involve item-by-item deletion of the complete model, lacking controls for Source Bank alone, CSA alone, single-teacher constraints, and ordinary perturbations, and are primarily focused on Subject 2.

  2. Existing evidence is insufficient to support a strong ranking conclusion. The comparison criteria between the main text and the appendix are questionable. The main text emphasizes a 5.69 percentage point improvement in brain retrieval compared to MindTuner, but the same protocol reproduction in Appendix Table 6 improves MindTuner brain retrieval to 80.50%, at which point MindBank's advantage is only 1.14 percentage points, and the image retrieval gap is even smaller at only 0.03 percentage points. Fair reproduction is commendable, but conclusions should be organized around these results.

  3. Appendix A.4 sets (αbase=0.10)(\alpha_{\mathrm{base}}=0.10) for subjects 1 and 2 and 0.30 for subjects 5 and 7 based on the similarity between different target subjects and the source mean; Appendix C.1 cites corresponding similarity values ​​to support these choices. Meanwhile, C.4 states that the similarity analysis comes from brain responses corresponding to 1,000 independent retained images. The paper does not explicitly state whether these retained samples are the final test set in the main table. If so, the parameter selection at least utilizes the overall statistics of the test input, and its transitive setting needs to be explicitly stated; if the test reconstruction score is also referenced when selecting "balanced" parameters, the issue further involves the participation of test indicators in model selection. The existing text cannot prove that improper parameter tuning occurred, but it cannot be ruled out either.

  4. The category accuracy in Figure 1 improved from 20.3% for MindEye2 to 26.0%, but the strongest comparison object, Duala, reached 25.5%. The paper does not fully explain the category vocabulary, classifier, ground truth definition, and how multi-object images are mapped to single labels. CSA relies on "CLIP-assigned categories," and if the training pseudo-labels are highly coupled with the evaluator, an increase in category scores does not necessarily mean improved independent semantic correctness. "Data efficiency" is also only validated on one target session, lacking data volume curves and source participant number curves. The training parameters 68.45M are only about 0.93% less than Duala's 69.09M.

Questions for authors
  1. Can the source mean itself correctly retrieve the corresponding image? After removing the global mean, mismatched stimuli, or shuffling voxels, do the similarity advantage and final performance of Figure 2 still exist? Can multiple sub-teachers for the same subject achieve similar gains?

  2. Is SDA superior to ordinary radial scaling, Gaussian perturbation, and Mixup of the same intensity? What are the actual minimum and maximum values ​​of the reliability weights and αc\alpha_c?

  3. Why is the fully closed baseline in Table 3 significantly stronger than MindEye2 in Table 1? After fixing the low-rank adaptation, training budget, and number of parameters, how much independent gain do Source Bank and CSA each provide?

  4. Why doesn't the main text use the MindTuner protocol reproduced in the appendix as the primary comparison?

  5. Did the 1,000 hold-out samples used for similarity analysis participate in the final test? When and on which validation set were all parameters determined?

  6. How is class accuracy defined, and is an evaluator independent of the training pseudo-labels used? How much actual training time and GPU memory can the method save under different target data volumes and source subject numbers?

Reviewer_egDXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MindBank adapts a pretrained visual fMRI decoder to a target participant using one hour of paired target data. During adaptation, the target fMRI vector is padded or truncated and passed through frozen source-subject projections; their representations form a Source Bank. Source-target consistency losses and reliability-calibrated interpolation provide regularization, source disagreement rescales target latents, and a category-aware memory loss uses labels assigned to training images by frozen CLIP. Evaluation on NSD participants 1, 2, 5, and 7 reports average brain retrieval of 81.64% and image retrieval of 83.83%, with competitive reconstruction scores and 68.45M trainable parameters for Subject 1. The appendix includes a reproduced MindTuner comparison, component and hyperparameter analyses, and representation visualizations. The bank and auxiliary mechanisms operate during training, with the adapted decoder used directly at inference.

Critical strengths
  1. A practical adaptation objective with competitive reconstruction quality. Reducing target-participant data requirements is a meaningful problem, and MindBank combines frozen source models with a relatively small trainable adaptation component. The controlled MindTuner reproduction is particularly useful: Table 6 reports improved reconstruction metrics across the four target participants, with average gains of 1.63, 1.42, 1.82, and 2.24 percentage points in Alex(2), Alex(5), Inception, and CLIP scores. The training-only nature of the auxiliary machinery also avoids adding a source ensemble to the inference path (Appendix A.6).

  2. The proposed regularization is explicit and supported by component analyses. Equations and algorithms specify the bank, reliability weights, disagreement-based perturbation, and category margin loss. The component and interpolation analyses expose trade-offs between reconstruction and retrieval (Tables 3-5). These results make source-guided regularization a plausible empirical contribution even if the stronger interpretation as recovered shared semantic structure remains unresolved.

Critical weaknesses
  1. The evidence does not distinguish shared source semantics from generic ensemble regularization. The source representations are not measured source-brain responses to matched images: all source paths receive the same target fMRI vector after padding/truncation, which the paper explicitly states does not establish anatomical or functional correspondence (Section 3.3; Appendix C.4). Higher cosine similarity to their mean could reflect common backbone structure or a shared representation direction, rather than stimulus-specific semantic agreement across people. Matched-versus-mismatched image controls and randomized or voxel-permuted source-projection controls are therefore important to the central explanation. Table 3 also does not isolate a bank-only variant or compare against a matched generic regularizer. The empirical gains may be real, but their attribution to the claimed shared semantic structure is not yet demonstrated.

  2. The strength and independence of the evaluation need clearer support. Relative to Duala, Table 1 improves average brain retrieval by only 0.53 percentage points and decreases image retrieval by 0.66; the matched MindTuner reproduction improves those metrics by 1.14 and 0.03 points, rather than the 5.69-point brain-retrieval difference highlighted against the Table 1 baseline. The reconstruction gains deserve credit, but no run variability or confidence intervals are supplied to assess these small retrieval differences. There is also an unresolved selection issue: Appendix C.4 derives similarity statistics from 1,000 held-out images, and Appendices A.4/C.1 cite these statistics to motivate participant-specific interpolation strengths. The PDF does not clearly separate those analyses, validation choices, and final test evaluation. This needs clarification before the reported advantage can be interpreted as an independently evaluated improvement; it is not, by itself, evidence of leakage.

Questions for authors
  1. Clarification/control affecting the mechanism: Does the bank outperform otherwise matched random-source, voxel-permuted-source, or generic regularization controls? Does its agreement with target features remain stimulus-specific when compared with mismatched-image pairs or after removing the common feature mean? Please clarify what makes the padded/truncated source paths semantic references rather than simply an ensemble prior.
  2. Clarification of the evaluation protocol: Are the 1,000 held-out images in Appendix C.4 the final test set? Were their similarity statistics or results used to choose the subject groups, interpolation strengths, loss weights, or checkpoints? Please specify disjoint training/validation/test image IDs, treatment of repeated trials, exclusion of test images from source pretraining, and the retrieval candidate-pool protocol.
  3. Clarification of robustness: What is the variability across adaptation seeds and one-hour target sessions, especially for the 0.53-point brain-retrieval advantage over Duala and 1.14-point advantage over reproduced MindTuner? Which baselines were rerun under the same preprocessing and evaluation pipeline? Making the controlled Table 6 comparison primary would help separate methodological gains from protocol differences.
Reviewer_JXAGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work seeks to address cross-subject adaptation with limited data, while using fMRI subject-wise volumes, then with the voxels from visual cortex extracted. The authors completely bypass anatomical and functional alignment.

Critical strengths

There are none, this is a deeply deeply flawed paper.

Critical weaknesses

The method in this paper is so wrong, I do not know where to begin. I do not believe this paper can be salvaged, I strongly recommend for rejection. This paper is extremely technically invalid.

This is an example of why modern fMRI-to-image decoding has drifted into scientifically useless benchmark-chasing. The core approach of this work is biologically absurd. The authors are completely ignoring literally decades of work on anatomical & functional alignment in fMRI research.

In the NSD dataset, the 1000 shared images are usually used for the test set, so the 1-hour adaptation set (~750 trials) consists of unique images the 7 source subjects never saw. How do I get "source representations" when the source subjects never viewed the stimulus? In Section 3.3 (and Algorithm 1), the authors take Subject 2’s flattened 1D voxel vector, pad the end of the array with zeros or chop it off to match the array length of Subject 1, and multiply Subject 2’s unaligned brain data through Subject 1’s frozen ridge regression matrix...

The authors state in Line 204 that padding/truncation ensures dimensional compatibility but does not establish voxel-wise anatomical or functional correspondence.

Think about what that means - Voxel #5000 in Subject 2 is being multiplied by the weight learned for Voxel #5000 in Subject 1, which is in a completely different anatomical location (especially after taking the 1D vector from the cortical surface). Passing a single subject's brain scan through 7 mismatched linear matrices is not "leveraging shared cross-subject semantic structure". This work is feeding mismatched, scrambled projections into a frozen 1.9 billion param MLP hat acts as an attractor network, forcing any noisy input back toward the MS-COCO CLIP embedding space.

Because the 7 source matrices are being fed a padded/truncated target brain scan they were never trained to read, their outputs naturally disagree. The authors call this variance "natural inter-subject variation" and build the SDA module (Equations 8~11) to perturb the target latent using this variance. In reality, this variance is not biological inter-subject variability; it is the mathematical variance of multiplying a vector by 7 wrongly-indexed weight matrices. Calling this "plausible cross-subject variation" is a massive overclaim.

Figure 5's anatomical overlay is also completely wrong, just because other accepted papers also use a wrong mask, does not mean it is valid to use an incorrect mask. The ROI labels are so off, it is deeply obvious.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.