acceptodds
Under review as a conference paper at ICLR 2027

Concept Generalization in Visual Brain Decoding via Encoder-Generated fMRI

Abstract

Visual brain decoders are typically trained and evaluated on stimuli drawn from similar semantic distributions, leaving their ability to reconstruct unseen concepts poorly understood. We study this concept generalization problem by withholding semantic concepts from a target subject's training data. We find that three recent fMRI-to-image decoders degrade substantially for the held-out concepts, even when the concept remains available for multi-subject models. In contrast, a brain encoder trained to predict fMRI from images is markedly more robust to the same semantic holdout. Motivated by this asymmetry, we use the encoder to generate synthetic fMRI responses for new images and add the resulting image-fMRI pairs to decoder training. Across three decoders, four subjects, and twelve semantic concepts, synthetic data recovers a substantial fraction of the performance lost on unseen concepts without requiring additional, time-consuming fMRI acquisition. We further show that the same approach improves general decoding performance across a wide range of data budgets. Analyses of encoder training conditions and synthetic-data controls show that the gains depend on both encoder generalization and the semantic content of the synthetic supervision, rather than merely increasing the amount of training data. Together, our results show that encoder-generated fMRI can transfer semantic generalization to decoders.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Decoding, Brain Encoding, Semantic Generalization, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZpxzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper diagnoses the concept generalization gap of existing brain decoders by removing specific visual concepts from the training data of target subjects, and then uses brain encoders to generate synthetic fMRI for relevant images to supplement decoder training and analyze the sources of benefit.

Critical strengths
  1. Concept holdout assessment is of substantial value and has sufficient coverage. The experiment involved three decoders, four participants, and twelve COCO superclasses.

  2. The authors compared different encoder pre-training conditions, including adding an equal amount of non-target concept synthesis data, incorrect fMRI pairings, freezing DINOv2, and more granular category analysis. Correctly synthesized pairings were significantly better than incorrect pairings; the entire benefit cannot be hastily attributed to increasing the number of images or training sessions.

Critical weaknesses
  1. The main methodology follows the existing synthetic fMRI training workflow. Section 2.3 already acknowledges that encoder-generated brain signals for decoder training is not a new technology. The main workflow in Section 4.2 is to train the existing encoder, construct (u,E(u))(u,E(u)) pairings, and then train the existing decoder; there are no new encoders, decoders, or learning targets. The most explicit increments in this paper are actually concept holdout assessment, synthetic image pool selection, and mechanism analysis. In particular, page 5 explicitly disables the original external synthetic data training from Brain-IT. This is beneficial for measuring the intervention of "adding synthetic data," but it cannot replace comparison with the complete Brain-IT protocol.

  2. The enhanced model has already seen synthetic supervision of the target concept, thus addressing the adaptation of a known target, rather than the direct generalization of unknown concepts. Section 5.2 uses images of the target concept removed from the training set to generate synthetic responses; the appendix explains that the model is trained separately for each subject and each concept. Therefore, although the original real brain signal supervision does not have the target concept, the enhanced decoder has explicitly learned the image and synthetic pairing of that concept. This changes the deployment problem: the system needs to know what will be missing in the future before supplementing it. The missing element is a pre-fixed general image pool, training a single decoder, and then evaluating new concepts without targeted supplementation. General data budgeting experiments do not simultaneously maintain concept retention, thus failing to fill this gap.

  3. The recall metrics in Appendix B only check whether the target class appears in the reconstruction, while relaxed recall only requires the presence of the same superclass. They do not penalize incorrectly added targets, nor do they require the target location, quantity, or instances within the same class to correspond to the input. Each concept-specific model primarily scores on positive examples containing that concept; therefore, "easier generation of target concepts" and "more accurate reading of the current trial" are not completely separated. The authors have provided a mismatch control, but this control pairs non-target concept responses with target concept images and cannot replace the shuffling of pairings within the target concept. In Table 7, mismatch still increases recall from 0.080 to 0.122 and relaxed recall from 0.219 to 0.298, indicating that simply changing the concept generation tendency can explain part of the gain. Correct pairing reaches 0.301, which is clearly more effective, but whether its remaining advantage comes from coarse class information or sample-specific information still needs further investigation.

  4. The paper emphasizes that the encoder retains 87% of the target recall, while MindEye2 retains only 22%. However, the encoder receives real target images, and the resulting response is read out by a fully synthesized MindEye2 that has seen the target concept and undergone synthetic training; the decoder being compared starts from noisy measured fMRI and is not supervised by the target concept. Input information content, noise, visual representation, reader experience, and synthetic domain calibration all differ. Therefore, these two ratios describe two specific pipelines but cannot independently demonstrate an inherent generalization advantage in the encoding direction. The authors state on page 8 that the readout metric is primarily used to compare encoders, but the introduction and motivation still use it as direct evidence of encoder-decoder asymmetry.

  5. Table 3 shows that the proportion of data removed for different concepts varies greatly: appliance averages approximately 6.9%, person approximately 50.6%; when participant 1 removed person, 4,636 out of 9,000 training images were deleted. Whole-image deletion also reduces the coverage of other objects, background, and composition. The authors have demonstrated that the decline on non-target concept tests is significantly smaller, supporting evidence for concept specificity. However, comparing different test distributions cannot replace comparing equal-quantity random deletions and concept deletions on the same target concept test set. The equal-quantity synthesis control in Table 7 answers the effect of adding data and cannot completely decompose the original seen-unseen gap. Therefore, the paper attributes the entire performance gap and its recovery ratio to a lack of conceptual knowledge, which may still be influenced by changes in data quantity and contextual support domain.

  6. Appendix A explicitly states that the data budget is constructed by randomly sampling independent images and retaining all their repeated observations, rather than sampling by scan number. The equivalent budget fit in the appendix also maintains a constant baseline for over 9,000 images, without a truly larger budget as a reference. Therefore, this type of fit cannot support the experimental effect of "doubling the scan count" at the maximum budget endpoint. The equivalent factors also differ for different metrics; for example, MindEye2's CLIP is approximately 1.74, PixCorr's is approximately 2.94, and SSIM's corresponding factors are difficult to identify.

Questions for authors
  1. After retaining the original synthetic training from Brain-IT and matching the number of synthetic images, real data, and update counts, how much better is concept-oriented augmentation than the complete previous work? What are the independent methodological contributions and evaluation findings of this paper?

  2. When training only one model and using a pre-fixed, generalized graph pool that is not selected for the test concepts, does the concept generalization gain remain?

  3. How do the false positive rate and precision change on images where the target concept is not present? How much of the sample-specific gain remains after adding shuffling of intra-concept pairings, subclass mean responses, and similarly difficult negative examples?

  4. After using bidirectional mappings of the same visual representation, different encoders, and different reader, does the asymmetry implied by 87% versus 22% still exist?

  5. After matching the number of real training images and other class coverage as closely as possible, how much additional loss does concept deletion incur compared to random deletion?

  6. After using a real scan budget, repeated training, and an equal number of update counts as a baseline, is the equivalent data factor still close to 2? What is the empirical basis for the "doubling" of the maximum budget endpoint?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Zpxz (1/3)

We thank the reviewer for the constructive feedback. Each response below addresses the corresponding weakness and question together. New results are in Tables R1–R4 of the common response and Tables R5–R7 below.

W1/Q1. Contributions and comparison with the full Brain-IT protocol

We agree that encoder-generated fMRI augmentation is an existing technique. Our contribution is a systematic empirical study of concept generalization in modern brain decoders and of the conditions under which synthetic supervision addresses their failures. We will make this distinction clearer in the manuscript.

Comparison with Brain-IT. We disabled Brain-IT's external data so that the added synthetic data is the single variable across all three decoders. We could not retrain the complete Brain-IT protocol within the rebuttal period, and instead we provide some controlled experiments with the MindEye2 probe decoder that isolate the effect of an external synthetic-data pool.

Our holdout models a concept for which the target subject has no fMRI. Brain-IT's external images come from the same COCO distribution as NSD, which leaves two ways to run its protocol under the holdout. Each is matched by one of our controls:

  • Pool that excludes CC. This is the faithful stand-in for a truly novel concept, which such a pool would not contain. Our non-C control (Table 7, Appendix D.3) recovers almost none of the recall gap.
  • Pool that includes CC. This reintroduces CC, so it is no longer a baseline separate from our intervention but an untargeted version of it. Our fixed general pool (Table R2) recovers nearly as much as targeted synthesis.

In these controls, the benefit depends on whether the pool covers the missing concept, not on selecting the pool for it. This identifies both a capability and a limit of generic augmentation: it can supply missing neural supervision without advance targeting, but it does not help on concepts outside its pool. Neither control uses the Brain-IT decoder or its external pool; we expect the same to hold there, and will state this correspondence in Sec. 5.1.

Findings. With the new experiments, the study establishes:

(i) The problem. When a concept is withheld from the target subject, three recent decoders degrade on it, including two multi-subject decoders trained with the concept available from other subjects. Existing cross-subject training does not carry the concept over.

(ii) The solution works. Motivated by the finding that encoders built on a pretrained visual representation are more robust to the same holdout than decoders, we show that their synthetic fMRI for the held-out concept recovers 37–87% of the lost performance across three decoders (including multi-subject decoders built on large pretrained priors), four subjects, and twelve concepts.

(iii) What the synthetic data must contain. The gain is not a data-volume effect: it requires the held-out concept in the synthesis pool and correct pairing within it (Table 7, Table R2). In our test, the pool did not need to be selected for the concept, as long as it covered it.

(iv) What the encoder must provide. A pretrained visual representation with a direct readout to voxels sufficed in our tests: even a ridge readout helps, and DINOv2 and OpenCLIP backbones give similar gains, while a random-init backbone or SynBrain gives little (Table R1). An encoder's fidelity on a held-out concept tracks the value of its synthetic data better than its overall accuracy does (Table R4).

W2/Q2. Generalization without concept-targeted augmentation

Yes, the gain remains in the condition we tested (Table R2). We trained the animal-held-out decoder with synthetic data for a general image pool that was fixed in advance and not selected for the test concept: the full training set of another subject. It recovers nearly as much of the gap as targeted synthesis.

We agree with the reviewer's characterization: the decoder does not generalize to the concept unaided, and the synthetic data supplies the supervision it lacks. What this experiment changes is the deployment requirement. The concept does not have to be known and targeted in advance; it only has to appear in a broad synthesis pool, and images are cheap to add where fMRI is not. A pool without the concept gives no such gain (non-C control, Table 7). However, targeting the synthetic data pool can be still useful when the concept is known: it reaches a similar performance without needing to add as much training data, reducing the burden of decoder training.

AuthorsOpenReview ↗

Response to Reviewer Zpxz (2/3)

Continued from W2/Q2

For holding out multiple concepts at once, our construction already covers several concepts at once. A held-out supercategory removes multiple COCO categories jointly (e.g., zebra, dog, and bird for animal), and Object Recall, which is scored per category, improves in this setting. One decoder and one untargeted pool therefore recover several held-out categories at once. However since we tested this within one supercategory, holding out unrelated supercategories together is left for future work, and we will update the Limitations accordingly.

W3/Q3. Concept bias versus sample-specific reconstruction

We ran the two checks that were feasible within the rebuttal period: precision and false-positive rate, and shuffling within the held-out concept. Both indicate that the gains are not explained by a tendency to generate CC more often.

Precision and false-positive rate (Table R5). Precision recovers together with recall. The false-positive rate does rise relative to the holdout model, which rarely generates CC at all, but it stays below that of the decoder trained on real fMRI for CC. Decoders trained with synthetic data therefore do not produce CC on images without it more often than a decoder that has seen real data for CC.

Table R5. Precision and false-positive rate on held-out concepts.

VariantRecall ↑Relaxed Recall ↑Precision ↑Relaxed Precision ↑FPR ↓Relaxed FPR ↓
Baseline (all data)0.4110.6430.4610.6862.01%4.69%
Holdout0.0870.2280.2730.5731.23%3.13%
Transfer-concept0.2790.4860.4160.6801.58%3.82%
Transfer-blind0.2290.4410.3970.6711.50%3.60%
Scratch-blind0.2180.4270.3930.6581.56%3.74%

All metrics are computed over the full test set, including images without CC, with the same models and averaging as Table 2; the relaxed versions are at the supercategory level. Precision is undefined when no reconstruction contains the category (20% of subject–category pairs for the holdout model, 3–7% for the others), so it is averaged over the 246 of 320 pairs defined for all five variants.

Within-concept shuffling (Table R2). Each CC image is paired with the synthetic response to a different CC image, which keeps supercategory-level information but breaks the pairing. Adding shuffled synthetic data raises only relaxed recall, and other metrics stay similar or fall below the unseen baseline. Supercategory-level supervision alone is therefore not enough.

W4/Q4. Interpreting the encoder–decoder comparison

We agree that the two ratios compare fixed pipelines and do not isolate an intrinsic advantage of the encoding direction. Since the brain encoder and decoder convert information in opposite directions, it is difficult to directly compare the generalization capability of those two constructions with a unified metric. The comparison was intended as a motivation as to why we chose to use encoder-generated synthetic data for concept generalization, and the real utility of the method was empirically tested by applying it to decoders. We will revise the introduction and motivation to present them that way. Within that scope, the asymmetry persists under each variation the reviewer asks about:

  • Different readout. We repeated the analysis with a MindEye2 readout trained on real data only, removing the synthetic training the reviewer is concerned about (Table R6). The pattern is unchanged: the encoder retains most of its recall under the holdout, while the decoder retains almost none.
  • Different encoders. Retention is graded across encoder variants (Table R6). It is highest with cross-subject exposure to CC, and still about half for scratch-blind, which never saw fMRI for CC from any subject.
  • Same visual representation. A UE built on OpenCLIP ViT-bigG, the backbone MindEye2 uses, still yields synthetic data that closes about half of the gap (Table R1), while MindEye2 itself degrades under the holdout.

Table R6. Encoder retention with a real-data-only readout (subject 1, animal).

EncoderAnimals in encoder trainingRecallRel. RecallIncepCLIPRetained (Recall / Rel.)
Real test fMRI (no encoder)—0.4840.7710.9610.934–
Transfer-oracleseen0.5590.8100.9700.940–
Transfer-conceptheld out0.4710.7130.9640.93484% / 88%
Transfer-blindheld out0.3580.6630.9600.92964% / 82%
Scratch-oracleseen0.5350.7980.9580.935–
Scratch-blindheld out0.2620.5930.9390.91749% / 74%
Decoder (probe), vs. its own CC-seen model (Table R1)held out0.0150.0540.7270.7183% / 7%
AuthorsOpenReview ↗

Response to Reviewer Zpxz (3/3)

Continued from W4/Q4

Each encoder's retention is relative to its own oracle, decoded by the same readout. The oracles exceed real test fMRI, possibly due to encoder predictions being free of trial noise. The readout is intentionally trained with CC so that its own concept-generalization limits do not confound the measurement. When an encoder does produce an fMRI signal containing proper conceptual knowledge, the readout decoder has to properly convert it to an image containing CC, thus it needs to be trained with CC to perform this conversion without error.

W5/Q5. Concept removal versus reduced training-data quantity

We ran the requested comparison on the same target-concept test set (Table R7). From the full training set, we randomly removed as many images as the animal holdout does, drawing only from images without animals. The resulting set matches the concept-held-out set in total real training volume, and the full set in concept-specific training volume, so the distinction can be separated cleanly.

Table R7. Size-matched random deletion vs. concept deletion (subject 1, animal).

Training set (real data only)# imagesAnimal: RecallAnimal: Rel. RecallAnimal: IncepAnimal: CLIPNon-animal: IncepNon-animal: CLIP
Full (CC Seen)9,0000.4620.7480.8820.9190.9650.934
Random deletion, animals kept7,0660.4560.7670.9030.9150.9520.916
Concept deletion (CC Unseen)7,0660.0150.0540.7270.7180.9610.926

Size-matched random deletion costs essentially nothing on animal images, while concept deletion of the same size removes almost all recall. The loss of general training volume alone therefore does not explain the deficit, consistent with the non-C control (Table 7), where adding data without CC does not help. Note that a similar experiment cannot be constructed for a very large concept like person: it would require deleting 4,636 non-person images from subject 1's training set, which contains only 4,364.

W6/Q6. Data budgets and the equivalent-data factor

The equivalent-data factor summarizes the gain across the measured budgets; it was not meant as a claim about the full-budget endpoint. We agree that "roughly doubling" invited that reading, and we will replace it. We address the reviewer's points in turn:

  • The "doubling" summary. "Roughly doubling" was shorthand for the fitted factors in Table 13, not a claim about the full-budget endpoint. One factor is fitted per metric across all budgets, and it is determined by the budgets where the equivalent baseline lies within the measured range; at the largest budget the baseline is held at its 9,000-image value and does not inform the fit. Measuring the factor at that endpoint would require a baseline trained on roughly twice the full budget, which does not exist without new data collection from the NSD subjects. Across the seven identifiable metrics, the median equivalent data factor is 2.4 for Brain-Diffuser, 2.2 for MindEye2, and 1.9 for Brain-IT. We will replace the "doubling" wording with these per-decoder summaries, stated as equivalent numbers of real training images within the measured range.
  • Budgets by image, not by scan. NSD shows each image three times in random order across 40 sessions, so a session-based split exposes different images a different number of times and yields training sets of varying size. Therefore although previous work such as MindEye2 and Brain-IT subsample based on scan numbers as the reviewer suggested, we instead subsample by image and keep all repeats so that the exposure to each image is consistent.
  • Update counts. We matched real-data exposure (equal epochs), so every model sees each real sample equally often. Total updates could be matched in two ways. Cutting the augmented model's updates would reduce its exposure to real data. Training the baseline longer is the alternative; although we have not run it, the baseline follows MindEye2's schedule and its validation performance has plateaued by the end of training, so additional updates are unlikely to account for the gains.
  • Repeated training. Each condition was trained once per subject, as is common in this literature (e.g., MindEye2, Brain-IT), so our statistics reflect variation across subjects rather than training seeds.

Thank you for providing the supplementary experiments and clarifications, which answered some of my questions. I appreciate the value of the concept retention assessment and your clarification that this is primarily an empirical study. However, the new contributions based on existing synthetic fMRI enhancement techniques remain unproven. The full Brain-IT assay was not evaluated, and the MindEye2 surrogate control could not determine how much the reported concept generalization deficit would persist while retaining the existing enhancement procedures. Therefore, I maintain my original assessment.

Reviewer_F7ybOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper systematically investigates the benefits of synthetic fMRI data for fMRI-to-image decoding models. The authors first identify the discrepancy between encoding models and decoding models in their out-of-distribution conceptual generalization ability. They then leverage synthetic data generated by the encoding model to boost the decoding model’s generalization to out-of-distribution concepts. The experiments are fairly comprehensive, including full-set and few-shot experiments on the NSD dataset. The authors analyze the data-efficiency metric and additionally introduce analyses on video decoding. This work also conducts a relatively thorough exploration of why and when synthetic data works. Overall, this paper focuses more on analysis rather than proposing a concrete method. In my opinion, it offers certain inspirational value for future research.

Critical strengths
  • The paper identifies the discrepancy in out-of-distribution conceptual generalization between encoding and decoding models and provides a plausible explanation.
  • The experiments and analyses in this paper are reasonably comprehensive.
  • The paper quantitatively analyzes the benefits brought by synthetic data.
  • The paper attempts to explain why synthetic data yields gains, rather than limiting the discussion to the perspective of data augmentation.
Critical weaknesses
  • The main limitation of this paper is that, although it points out that existing evaluation metrics for encoding models cannot be directly used to judge whether synthetic data will be effective, it does not attempt to propose a discriminative metric, which I consider a shortcoming. Note that training the "Probe Decoder" used in the paper requires incorporating synthetic data; therefore, it cannot serve as a simple evaluation scheme. In practical usage, researchers would prefer a universal evaluation metric to determine whether synthetic data generated by their trained encoding models will bring gains for decoding or other downstream tasks.

  • The authors use multiple decoding models but only adopt the Universal Brain Encoder proposed by Beliy et al. as the encoding model throughout. Given that the encoding model appears to be the more central component of this paper, could the authors run the main experiments using more NSD BOLD Beta encoding models — such as the SynBrain mentioned in related work, MindSimulator (Bao et al., ICLR 2025), or BraInCoRL (Yu et al., NeurIPS 2025) — while keeping MindEye2 as the only decoder? This would help demonstrate that the conclusions of this paper do not depend on a specific encoding model or pretrained image embedding model.

Questions for authors
  • When analyzing the out-of-distribution conceptual generalization of encoding and decoding models, the authors appear to only use MindEye2 and the Universal Brain Encoder proposed by Beliy et al. While this set of models is adequate, comparisons across more encoding/decoding architectures would, in my opinion, strengthen the argument. Could the authors provide such additional comparisons?

  • Video reconstruction is typically based on raw BOLD signals rather than BOLD betas processed via GLMsingle. For video reconstruction, if betas for each time window are predicted independently, can this approach properly capture the hemodynamic delay and temporal dependencies present in the original raw BOLD signals?

2 comments
AuthorsOpenReview ↗

Response to Reviewer F7yb

We thank the reviewer for the constructive feedback. We address the weaknesses and questions below; the new results are in Tables R1–R4 of the common response.

W1. Evaluating synthetic-data utility without training a decoder

We agree this is an important question, and we added an experiment comparing new encoder variants and metrics to identify a candidate diagnostic (Table R4). Aside from voxel-wise correlation and decoded recall, which are already included in the paper (Table 2), we additionally tested decoded CLIP and within-concept identification.

Overall voxel-wise correlation is not enough, as encoders with similar correlation can have widely different utility. What separates the encoders in our experiment is their fidelity on a concept held out from their training. The two new metrics we have tested do not require training with synthetic data: (a) decoded CLIP, similar to decoded recall, but instead measures the CLIP metric of the image from the readout decoder that converts the encoder-predicted fMRI, and (b) within-concept identification, which needs no decoder at all and asks how often an encoder’s prediction is matched to the correct image among the training images of the held-out concept. While decoded recall agrees except for the two weakest encoders, the two added metrics rank the five tested encoders in the same order as their decoder benefit. In practice, both can be computed by holding out a proxy concept for which the subject has measured responses.

We see this as a first step. With five encoders and one held-out concept, it is an exploratory diagnostic rather than a validated metric, and a general-purpose metric for decoding and other downstream tasks remains future work. We will include the diagnostic and this discussion in the revised Sec. 6.

W2/Q1. Generalization across encoding architectures

Following the reviewer’s suggestion, we kept MindEye2 as the decoder and added four encoders (Table R1): a linear (ridge) encoder on frozen DINOv2, the UE with an OpenCLIP ViT-bigG backbone, the UE with a frozen random-init backbone, and SynBrain. We prioritized SynBrain because it is the closest to our setting and is cited in our related work. It is also reported to outperform MindSimulator on voxel-level and semantic-level metrics, and BraInCoRL targets a different setting (few-shot in-context prediction). We could not run MindSimulator and BraInCoRL within the rebuttal period.

The conclusion does not depend on the image embedding model. Replacing DINOv2 with OpenCLIP ViT-bigG (the backbone MindEye2 uses) gives similar gains in our main setting, while a random-init backbone gives almost none. It does not depend on the UE (the encoder we use) architecture either. A simple ridge readout from frozen DINOv2 features already produces useful synthetic data, at roughly half the benefit of the UE trained on the same subject. But it is not automatic for every encoder. SynBrain’s synthetic data gives little recovery, although we verified the correctness of our implementation (common response, Table R3). Under the holdout, its predictions for the held-out concept degrade and are less identifiable within the concept than the UE’s. One possible explanation is its VAE voxel decoder, trained only on seen-concept fMRI, although we have not isolated the cause.

On Q1: on the decoder side, Table 1 already covers three decoder architectures, all of which degrade under the holdout. On the encoder side, the tested encoders that map a pretrained visual representation directly to voxels (UE, ridge) are robust, while SynBrain is not. We will narrow our claim accordingly in the revision.

Q2. Hemodynamic delay and temporal dependencies in video reconstruction

We follow standard practice in video decoding and apply a fixed temporal offset between the video and the fMRI scans, using the same offset as the NEURONS model when generating synthetic responses. This aligns stimulus and response but does not model the hemodynamic response or temporal dependencies: each 2 s window is predicted independently from a single frame. It is a framewise approximation, not a model of the full BOLD dynamics.

We intended the video experiment as a preliminary check of whether the recipe carries over to a different stimulus modality without modification, and deliberately kept the setup simple: a still-image encoder, a single frame per 2 s window, and no temporal modeling. Synthetic data still improved the video decoder at both full-data and limited-data budgets, which establishes usefulness under this approximation. Whether an encoder that models temporal dynamics would improve synthetic-data utility remains to be tested. Video encoders such as TRIBE v2 are natural candidates, although they predict responses in a different space (cortical surface rather than cc2017 voxels) and would need fine-tuning to specific subjects. We leave this to future work, and will describe the approximation precisely in Appendix F.

Thanks to the authors for carefully and thoroughly addressing my review comments and adding multiple sets of experiments within the limited rebuttal window. I believe these new results and explanations have adequately addressed my major concerns raised earlier. Therefore, I have no further substantive concerns and maintain my original positive score.

Reviewer_i4sDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study shows decoding models do not generalize well to concepts outside of their training set and uses encoder-generated fMRI for images of the held-out concepts to recover much of the lost performance.

Critical strengths

The paper raises an important question that most decoding work ignores, whether decoders can reconstruct concepts they were not trained on and how to fix potential shortcomings.

Critical weaknesses

Prior work established the generalization shortcomings of decoding models to concepts outside of the training set, and the benefit of augmenting the training set using encoder predicted activity. This makes the overall conceptual and technical novelty somewhat limited. However, this study can distinguish itself more by examining where the benefit comes from. All experiments use only one encoder. The authors attribute the generalization to the pretrained backbone, but this is only in passing and never really tested. What happens with other backbones (CLIP, supervised ImageNet, random weights), a simple linear encoder, or an encoder never trained on NSD? Does the encoder accuracy or generalization capabilities predict how useful the predictions would be? Also a more large-scale training where the encoder is applied to a whole range of concepts to augment the training rather than just the exactly heldout concepts is needed to better understand if the effect could be achieved without knowing the heldout concepts in advance.

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer i4sD (1/2)

We thank the reviewer for the detailed and constructive feedback. We agree that examining where the benefit comes from is where the paper should stand out, and most of our new experiments (Tables R1–R4 in the common response) were designed around the reviewer's questions. We address the concerns in the order raised.

1. Novelty relative to prior work

“This makes the overall conceptual and technical novelty somewhat limited.”

We agree that using encoder-generated fMRI to train decoders is not new in itself, and we do not claim the method as our contribution. Our contribution is a systematic study of concept generalization in modern decoders, and of why encoder-generated data addresses it:

(i) When a concept is withheld from the target subject, three recent decoders degrade on it. This includes two multi-subject decoders trained with the concept available from other subjects, so existing cross-subject training methods do not fully carry the concept over.

(ii) Motivated by the finding that encoders built on a pretrained visual representation are more robust to the same holdout than decoders, we show that their synthetic fMRI for the held-out concept recovers 37–87% of the lost performance across three decoders (including multi-subject decoders built on large pretrained priors), four subjects, and twelve concepts.

(iii) The gain is concept-specific, not a data-volume effect: matched data without the concept recovers 2% of the gap, while correctly paired responses for the concept recover 64%. Within-concept shuffling (Table R2) further shows that supercategory-level signal alone is not enough: correct pairing within the concept is needed.

(iv) A pretrained visual representation with a direct readout to voxels, in other words a simple linear encoder, also showed meaningful improvements in our tests (points 2–4 below).

2. Dependence on the pretrained backbone

"The authors attribute the generalization to the pretrained backbone, but this is only in passing and never really tested."

We agree that the frozen-backbone ablation (Table 7) alone does not establish this. We ran three new tests (Table R1):

  • Different pretrained backbone. Replacing DINOv2 with OpenCLIP ViT-bigG (the backbone MindEye2 uses) gives similar gains.
  • Random weights. A UE with a frozen, randomly initialized ViT gives much smaller gains, close to the unseen baseline on strict recall.
  • Linear encoder. A ridge regression from frozen DINOv2 features, with no cross-subject data and no backbone tuning, recovers 18% / 32% of the decoder recall/relaxed recall gap. This is about half of what our scratch-blind encoder achieves (34% / 55%). This shows a very simple encoder can still perform reasonably well when given the backbone representations.

Together, these results support that the pretrained backbone plays a major role, and show that the benefit is not specific to DINOv2.

The OpenCLIP result also sharpens the asymmetry: built on the same backbone as MindEye2, the encoder remains robust to the holdout while MindEye2 does not. This is consistent with our explanation that the asymmetry depends on where the pretrained representation sits (encoder input vs. decoder output) rather than on which representation is used.

AuthorsOpenReview ↗

Response to Reviewer i4sD (2/2)

3. Other encoder variants and a simple encoder

"All experiments use only one encoder. What happens with ... a simple linear encoder, or an encoder never trained on NSD?"

The benefit is not specific to the Universal Encoder (UE) we have mainly used. As shown in point 2, a simple ridge encoder on frozen DINOv2 features already produces useful synthetic data, at roughly half the benefit of the UE trained on the same subject (Table R1).

Not every encoder howevery shares this property. SynBrain gives little recovery when its synthetic data is used for decoder training, although we verified our implementation against the authors' reported results and released checkpoint (common response, Table R3). Under the holdout, its predictions for the held-out concept degrade and are less identifiable within the concept than the UE's, both when the UE is matched on training data and when it is built on the same OpenCLIP backbone. One possible explanation is SynBrain's VAE voxel decoder, trained only on seen-concept fMRI, although we have not isolated the cause. Concept generalization is therefore not automatic across encoder designs, but it does not require a sophisticated encoder either.

On "an encoder never trained on NSD": we did not test an encoder trained entirely outside NSD. Any encoder used for synthesis must output the target subject's voxels, so it needs at least a readout fit on that subject's data. The ridge encoder is the closest case we tested: its representation was never exposed to fMRI, and its linear map is fit only on the target subject's training data, which excludes CC. Together with scratch-blind, this shows that the benefit does not depend on cross-subject pre-training. Pre-training on a different fMRI dataset is a natural extension, which we will note as future work.

4. Can encoder performance predict synthetic-data utility?

"Does the encoder accuracy or generalization capabilities predict how useful the predictions would be?"

Based on the result comparing different encoder variants (Table R4), direct encoder accuracy (voxel-wise correlation) is informative but not sufficient. Encoders with nearly the same correlation can differ widely in how much their synthetic data helps, both across encoder families (SynBrain vs. UE scratch-blind) and across UE variants (transfer-blind vs. transfer-concept).

However, measuring the conceptual fidelity for the held-out concept can predict the usefulness better. Decoded CLIP and within-concept identification both rank the five tested encoders correctly, and decoded recall agrees except for the two weakest encoders. With five encoders and one held-out concept, we present these as exploratory diagnostics rather than validated predictors, and will add the analysis to Sec. 6.

5. Augmentation without knowing the held-out concepts

"if the effect could be achieved without knowing the heldout concepts in advance."

We added this experiment (Table R2). Instead of images of the held-out concept, we synthesize fMRI for a general image pool that is fixed in advance and not selected for the test concept. It recovers nearly as much of the gap as targeted synthesis.

Our non-C control (Table 7) shows the other side: synthetic data for images that do not contain CC recovers almost nothing. Together, the two results indicate that what matters is whether CC is represented in the synthesis pool, not whether the pool was selected for it. A broad pool can therefore cover held-out concepts without knowing them in advance, as long as they appear in it. Note that we tested this for one held-out concept (animal), and will update the Limitations accordingly.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response to all reviewers (1/2)

We thank all reviewers for their constructive feedback. Several requests concerned the same experiments, so we summarize the new results here and refer to them as Tables R1–R4 in the individual responses.

New experiments.

  1. Encoder ablation (R-i4sD, R-F7yb): We test decoder performance after adding synthetic data from new encoder variations: a linear (ridge) encoder, a frozen random-init backbone, SynBrain, and the Universal Encoder (UE) with an OpenCLIP ViT-bigG backbone (the same one MindEye2 uses) (Table R1).
  2. Synthetic-data controls (R-i4sD, R-Zpxz): synthesis from a fixed general image pool not selected for the test concept, and shuffled pairings within the held-out concept (Table R2).
  3. SynBrain reproduction and diagnosis (R-F7yb, R-i4sD) (Table R3).
  4. Encoder metrics vs. synthetic-data value (R-i4sD, R-F7yb) (Table R4).
  5. Precision and false-positive rate, a real-data-only readout, and size-matched random deletion (R-Zpxz) (Tables R5–R7, in the response to R-Zpxz).

Setting. Unless stated otherwise, the new experiments use subject 1 with animal held out and the MindEye2 probe decoder of Sec. 6. Low-level metrics will be reported in the revised appendix. Gap rec. is the recovered decoder performance gap ((augmented − unseen) / (seen − unseen)).

Table R1. Encoder ablation (decoder performance after adding synthetic data).

Encoder for synthesisIncepCLIPEffNet↓SwAV↓RecallRel. RecallGap rec. (Recall / Rel.)
CC Seen (upper bound)0.8820.9190.6660.3650.4620.748–
CC Unseen (no synthetic)0.7270.7180.8980.5110.0150.054–
UE, frozen random-init ViT0.7300.7680.8860.5060.0280.1473% / 13%
Ridge, frozen DINOv20.7550.7970.8460.4740.0940.27518% / 32%
SynBrain0.7200.7300.8870.5050.0260.0932% / 6%
UE scratch-blind0.7970.8070.8100.4530.1670.43434% / 55%
UE transfer-blind0.7950.8510.7860.4330.2120.50444% / 65%
UE transfer-concept (DINOv2)0.8340.8640.7670.4220.2700.54357% / 70%
UE transfer-concept (OpenCLIP ViT-bigG)0.8110.8420.7820.4350.2530.48453% / 62%

Table R2. Synthetic-data controls (UE transfer-concept). Fixed general pool: synthesis images drawn from a pool fixed in advance and not selected for the test concept: the entire training set of subject 2 (9,000 images, of which 1,929 contain animals, vs. 1,934 held out from subject 1). This pool is larger than the main setting (9,000 vs. 1,934 synthetic images), but added volume alone does not help on CC (non-C control, Table 7).

Shuffled within CC: each CC image is paired with the synthetic response of a different CC image.

Synthetic dataIncepCLIPRecallRel. RecallGap rec. (Recall / Rel.)
None (CC Unseen)0.7270.7180.0150.054–
CC images (main setting)0.8340.8640.2700.54357% / 70%
Fixed general pool0.8330.8810.2340.54349% / 70%
Shuffled within CC0.6630.7040.0290.2093% / 22%

SynBrain reproduction and diagnosis. We reproduced SynBrain with the authors' official code and recipe. We made sure our implementation of SynBrain is correct by confirming our measured performance is comparable against the subject 1 performance reported by the paper and also the model checkpoint released by the authors. We then retrained SynBrain on subject 1 with all animal images removed (7,066 images remaining).

AuthorsOpenReview ↗

Common response to all reviewers (2/2)

Table R3. Decoding encoder-predicted test fMRI with a fixed MindEye2 readout of Sec. 6 (Incep / CLIP).

Encoder (subject 1)Animals in training?Animal test (n=266)Non-animal test (n=734)Within-concept identification*
SynBrain, released checkpointYes0.943 / 0.9370.981 / 0.956–
SynBrain, our retrainNo0.822 / 0.8000.980 / 0.96220%
UE scratch-blind (subject 1 only)No0.873 / 0.8860.986 / 0.96330%
UE transfer-blindNo (any subject)0.937 / 0.9460.991 / 0.97358%
UE transfer-blind, OpenCLIP ViT-bigG (same backbone as SynBrain)No (any subject)0.851 / 0.8410.988 / 0.96438%

*Top-1 identification among the held-out concept images from training, after subtracting the mean response over all of the images. In our case, they are the 1,934 animal images held out from training (not the 266 test images; chance 0.05%). It therefore reflects if encoder-generated fMRI could be properly differentiated and matched to the correct image.

Result. Under this holdout, SynBrain's predictions for the held-out concept degrade, while its predictions for seen concepts stay intact. Its synthetic data gives little recovery on animal decoding (Table R1), whereas every UE variant with a pretrained backbone recovers substantially more. Within-concept identification suggests SynBrain's predictions also carry less within-concept information than the UE’s (20% vs. 30–58%).

One possible explanation is that SynBrain generates voxels through a VAE decoder trained only on seen-concept fMRI, which may limit extrapolation beyond its training concepts; establishing this would require a matched architectural ablation in the future. SynBrain was designed for in-distribution fidelity and variability; our finding is specific to concept holdout and shows that useful concept generalization is not automatic across all encoder architectures.

Table R4. Encoder metrics vs. value of synthetic data.

EncoderVoxel-wise rDecoded CLIP, test set animalsDecoded Recall, test set animalsWithin-concept identification*Decoder gap rec. (Recall / Rel.)
SynBrain0.3340.8000.1420%2% / 6%
Ridge, frozen DINOv20.3620.8290.1128%18% / 32%
UE scratch-blind0.3420.8860.2630%34% / 55%
UE transfer-blind0.3980.9460.4258%44% / 65%
UE transfer-concept0.4000.9610.5862%57% / 70%

*Top-1 identification among the held-out concept images from training, after subtracting the mean response over all of the images. In our case, they are the 1,934 animal images held out from training (not the 266 test images; chance 0.05%). It therefore reflects if encoder-generated fMRI could be properly differentiated and matched to the correct image.

The four metrics (Voxel-wise r, decoded CLIP, decoded recall, within-concept identification) are evaluations for the encoder-predicted fMRI, while the decoder gap recovered measures the impact when the synthetic data is used for actual training. Overall voxel-wise correlation does not consistently capture differences in synthetic-data utility. For one example, SynBrain and UE scratch-blind have similar correlation but their impact on the decoder is very different. However, for the most part, the other three metrics correctly order the five encoder variants according to their decoder gap recovered, except for decoded recall which switches SynBrain and the ridge encoder. The cleanest separation comes from the decoded CLIP metric.

This complements Appendix G.2, where overall r and decoded recall are moderately associated with decoder benefit across UE variants (ρ = 0.56 and 0.58): overall fidelity is informative but not sufficient, especially across encoder families. With five encoders and one held-out concept, we present these as exploratory diagnostics, not validated predictors.

Planned revisions. (i) Sec. 6 will be revised around these results. We will add the direct tests of the backbone explanation (ridge, random-init, OpenCLIP; Table R1), show that an encoder built on the backbone MindEye2 uses (OpenCLIP) remains robust while MindEye2 does not, and complement the statement on encoder metrics with the held-out-concept diagnostics above. (ii) Because SynBrain's predictions degrade under holdout, we will narrow our claim from "brain encoders are more robust" to the tested encoders that map a pretrained visual representation directly to voxels (UE, ridge). (iii) The Limitations will note that, with a general synthesis pool, CC need not be known in advance in the tested condition, as long as it is included in the synthesis pool. (iv) Tables R1–R7 will be added to the appendix.

AuthorsOpenReview ↗

Paper revision and an invitation for discussion

Dear Reviewers and Area Chair,

We have uploaded a revised version of the paper according to our previous responses to the reviews. The changes are marked in blue.

Here is a summary of the key changes:

  • New experiments: Tables R1–R7 from our responses are now in the appendix as Tables 12, 11, 13, 14, 15, 8 and 7, respectively, with accompanying text in Appendices C.3, D.1, D.3, D.5 and D.6.
  • Main text: we narrow our claim to encoders that map a pretrained visual representation directly to voxels, present the encoder–decoder comparison as motivation rather than as evidence of an intrinsic advantage. We also clarify and reword the part with per-decoder equivalent-data factors, and revise Sec. 6 and the Limitations around the new results.
  • Numbering: because of the added tables, former Tables 7–13 are now Tables 9, 10 and 16–20 in the revised version.

We thank Reviewer F7yb for the follow-up. We also invite reviewers i4sD and Zpxz for a discussion on whether our responses and the revision address the concerns, and we would be grateful for additional feedback. We are happy to clarify anything further in the remaining time.

Thank you for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.