- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper diagnoses the concept generalization gap of existing brain decoders by removing specific visual concepts from the training data of target subjects, and then uses brain encoders to generate synthetic fMRI for relevant images to supplement decoder training and analyze the sources of benefit.
Critical strengths
-
Concept holdout assessment is of substantial value and has sufficient coverage. The experiment involved three decoders, four participants, and twelve COCO superclasses.
-
The authors compared different encoder pre-training conditions, including adding an equal amount of non-target concept synthesis data, incorrect fMRI pairings, freezing DINOv2, and more granular category analysis. Correctly synthesized pairings were significantly better than incorrect pairings; the entire benefit cannot be hastily attributed to increasing the number of images or training sessions.
Critical weaknesses
-
The main methodology follows the existing synthetic fMRI training workflow. Section 2.3 already acknowledges that encoder-generated brain signals for decoder training is not a new technology. The main workflow in Section 4.2 is to train the existing encoder, construct pairings, and then train the existing decoder; there are no new encoders, decoders, or learning targets. The most explicit increments in this paper are actually concept holdout assessment, synthetic image pool selection, and mechanism analysis. In particular, page 5 explicitly disables the original external synthetic data training from Brain-IT. This is beneficial for measuring the intervention of "adding synthetic data," but it cannot replace comparison with the complete Brain-IT protocol.
-
The enhanced model has already seen synthetic supervision of the target concept, thus addressing the adaptation of a known target, rather than the direct generalization of unknown concepts. Section 5.2 uses images of the target concept removed from the training set to generate synthetic responses; the appendix explains that the model is trained separately for each subject and each concept. Therefore, although the original real brain signal supervision does not have the target concept, the enhanced decoder has explicitly learned the image and synthetic pairing of that concept. This changes the deployment problem: the system needs to know what will be missing in the future before supplementing it. The missing element is a pre-fixed general image pool, training a single decoder, and then evaluating new concepts without targeted supplementation. General data budgeting experiments do not simultaneously maintain concept retention, thus failing to fill this gap.
-
The recall metrics in Appendix B only check whether the target class appears in the reconstruction, while relaxed recall only requires the presence of the same superclass. They do not penalize incorrectly added targets, nor do they require the target location, quantity, or instances within the same class to correspond to the input. Each concept-specific model primarily scores on positive examples containing that concept; therefore, "easier generation of target concepts" and "more accurate reading of the current trial" are not completely separated. The authors have provided a mismatch control, but this control pairs non-target concept responses with target concept images and cannot replace the shuffling of pairings within the target concept. In Table 7, mismatch still increases recall from 0.080 to 0.122 and relaxed recall from 0.219 to 0.298, indicating that simply changing the concept generation tendency can explain part of the gain. Correct pairing reaches 0.301, which is clearly more effective, but whether its remaining advantage comes from coarse class information or sample-specific information still needs further investigation.
-
The paper emphasizes that the encoder retains 87% of the target recall, while MindEye2 retains only 22%. However, the encoder receives real target images, and the resulting response is read out by a fully synthesized MindEye2 that has seen the target concept and undergone synthetic training; the decoder being compared starts from noisy measured fMRI and is not supervised by the target concept. Input information content, noise, visual representation, reader experience, and synthetic domain calibration all differ. Therefore, these two ratios describe two specific pipelines but cannot independently demonstrate an inherent generalization advantage in the encoding direction. The authors state on page 8 that the readout metric is primarily used to compare encoders, but the introduction and motivation still use it as direct evidence of encoder-decoder asymmetry.
-
Table 3 shows that the proportion of data removed for different concepts varies greatly: appliance averages approximately 6.9%, person approximately 50.6%; when participant 1 removed person, 4,636 out of 9,000 training images were deleted. Whole-image deletion also reduces the coverage of other objects, background, and composition. The authors have demonstrated that the decline on non-target concept tests is significantly smaller, supporting evidence for concept specificity. However, comparing different test distributions cannot replace comparing equal-quantity random deletions and concept deletions on the same target concept test set. The equal-quantity synthesis control in Table 7 answers the effect of adding data and cannot completely decompose the original seen-unseen gap. Therefore, the paper attributes the entire performance gap and its recovery ratio to a lack of conceptual knowledge, which may still be influenced by changes in data quantity and contextual support domain.
-
Appendix A explicitly states that the data budget is constructed by randomly sampling independent images and retaining all their repeated observations, rather than sampling by scan number. The equivalent budget fit in the appendix also maintains a constant baseline for over 9,000 images, without a truly larger budget as a reference. Therefore, this type of fit cannot support the experimental effect of "doubling the scan count" at the maximum budget endpoint. The equivalent factors also differ for different metrics; for example, MindEye2's CLIP is approximately 1.74, PixCorr's is approximately 2.94, and SSIM's corresponding factors are difficult to identify.
Questions for authors
-
After retaining the original synthetic training from Brain-IT and matching the number of synthetic images, real data, and update counts, how much better is concept-oriented augmentation than the complete previous work? What are the independent methodological contributions and evaluation findings of this paper?
-
When training only one model and using a pre-fixed, generalized graph pool that is not selected for the test concepts, does the concept generalization gain remain?
-
How do the false positive rate and precision change on images where the target concept is not present? How much of the sample-specific gain remains after adding shuffling of intra-concept pairings, subclass mean responses, and similarly difficult negative examples?
-
After using bidirectional mappings of the same visual representation, different encoders, and different reader, does the asymmetry implied by 87% versus 22% still exist?
-
After matching the number of real training images and other class coverage as closely as possible, how much additional loss does concept deletion incur compared to random deletion?
-
After using a real scan budget, repeated training, and an equal number of update counts as a baseline, is the equivalent data factor still close to 2? What is the empirical basis for the "doubling" of the maximum budget endpoint?
4 comments
Response to Reviewer Zpxz (2/3)
Continued from W2/Q2
For holding out multiple concepts at once, our construction already covers several concepts at once. A held-out supercategory removes multiple COCO categories jointly (e.g., zebra, dog, and bird for animal), and Object Recall, which is scored per category, improves in this setting. One decoder and one untargeted pool therefore recover several held-out categories at once. However since we tested this within one supercategory, holding out unrelated supercategories together is left for future work, and we will update the Limitations accordingly.
W3/Q3. Concept bias versus sample-specific reconstruction
We ran the two checks that were feasible within the rebuttal period: precision and false-positive rate, and shuffling within the held-out concept. Both indicate that the gains are not explained by a tendency to generate more often.
Precision and false-positive rate (Table R5). Precision recovers together with recall. The false-positive rate does rise relative to the holdout model, which rarely generates at all, but it stays below that of the decoder trained on real fMRI for . Decoders trained with synthetic data therefore do not produce on images without it more often than a decoder that has seen real data for .
Table R5. Precision and false-positive rate on held-out concepts.
| Variant | Recall ↑ | Relaxed Recall ↑ | Precision ↑ | Relaxed Precision ↑ | FPR ↓ | Relaxed FPR ↓ |
|---|---|---|---|---|---|---|
| Baseline (all data) | 0.411 | 0.643 | 0.461 | 0.686 | 2.01% | 4.69% |
| Holdout | 0.087 | 0.228 | 0.273 | 0.573 | 1.23% | 3.13% |
| Transfer-concept | 0.279 | 0.486 | 0.416 | 0.680 | 1.58% | 3.82% |
| Transfer-blind | 0.229 | 0.441 | 0.397 | 0.671 | 1.50% | 3.60% |
| Scratch-blind | 0.218 | 0.427 | 0.393 | 0.658 | 1.56% | 3.74% |
All metrics are computed over the full test set, including images without , with the same models and averaging as Table 2; the relaxed versions are at the supercategory level. Precision is undefined when no reconstruction contains the category (20% of subject–category pairs for the holdout model, 3–7% for the others), so it is averaged over the 246 of 320 pairs defined for all five variants.
Within-concept shuffling (Table R2). Each image is paired with the synthetic response to a different image, which keeps supercategory-level information but breaks the pairing. Adding shuffled synthetic data raises only relaxed recall, and other metrics stay similar or fall below the unseen baseline. Supercategory-level supervision alone is therefore not enough.
W4/Q4. Interpreting the encoder–decoder comparison
We agree that the two ratios compare fixed pipelines and do not isolate an intrinsic advantage of the encoding direction. Since the brain encoder and decoder convert information in opposite directions, it is difficult to directly compare the generalization capability of those two constructions with a unified metric. The comparison was intended as a motivation as to why we chose to use encoder-generated synthetic data for concept generalization, and the real utility of the method was empirically tested by applying it to decoders. We will revise the introduction and motivation to present them that way. Within that scope, the asymmetry persists under each variation the reviewer asks about:
- Different readout. We repeated the analysis with a MindEye2 readout trained on real data only, removing the synthetic training the reviewer is concerned about (Table R6). The pattern is unchanged: the encoder retains most of its recall under the holdout, while the decoder retains almost none.
- Different encoders. Retention is graded across encoder variants (Table R6). It is highest with cross-subject exposure to , and still about half for scratch-blind, which never saw fMRI for from any subject.
- Same visual representation. A UE built on OpenCLIP ViT-bigG, the backbone MindEye2 uses, still yields synthetic data that closes about half of the gap (Table R1), while MindEye2 itself degrades under the holdout.
Table R6. Encoder retention with a real-data-only readout (subject 1, animal).
| Encoder | Animals in encoder training | Recall | Rel. Recall | Incep | CLIP | Retained (Recall / Rel.) |
|---|---|---|---|---|---|---|
| Real test fMRI (no encoder) | — | 0.484 | 0.771 | 0.961 | 0.934 | – |
| Transfer-oracle | seen | 0.559 | 0.810 | 0.970 | 0.940 | – |
| Transfer-concept | held out | 0.471 | 0.713 | 0.964 | 0.934 | 84% / 88% |
| Transfer-blind | held out | 0.358 | 0.663 | 0.960 | 0.929 | 64% / 82% |
| Scratch-oracle | seen | 0.535 | 0.798 | 0.958 | 0.935 | – |
| Scratch-blind | held out | 0.262 | 0.593 | 0.939 | 0.917 | 49% / 74% |
| Decoder (probe), vs. its own -seen model (Table R1) | held out | 0.015 | 0.054 | 0.727 | 0.718 | 3% / 7% |
Response to Reviewer Zpxz (3/3)
Continued from W4/Q4
Each encoder's retention is relative to its own oracle, decoded by the same readout. The oracles exceed real test fMRI, possibly due to encoder predictions being free of trial noise. The readout is intentionally trained with so that its own concept-generalization limits do not confound the measurement. When an encoder does produce an fMRI signal containing proper conceptual knowledge, the readout decoder has to properly convert it to an image containing , thus it needs to be trained with to perform this conversion without error.
W5/Q5. Concept removal versus reduced training-data quantity
We ran the requested comparison on the same target-concept test set (Table R7). From the full training set, we randomly removed as many images as the animal holdout does, drawing only from images without animals. The resulting set matches the concept-held-out set in total real training volume, and the full set in concept-specific training volume, so the distinction can be separated cleanly.
Table R7. Size-matched random deletion vs. concept deletion (subject 1, animal).
| Training set (real data only) | # images | Animal: Recall | Animal: Rel. Recall | Animal: Incep | Animal: CLIP | Non-animal: Incep | Non-animal: CLIP |
|---|---|---|---|---|---|---|---|
| Full ( Seen) | 9,000 | 0.462 | 0.748 | 0.882 | 0.919 | 0.965 | 0.934 |
| Random deletion, animals kept | 7,066 | 0.456 | 0.767 | 0.903 | 0.915 | 0.952 | 0.916 |
| Concept deletion ( Unseen) | 7,066 | 0.015 | 0.054 | 0.727 | 0.718 | 0.961 | 0.926 |
Size-matched random deletion costs essentially nothing on animal images, while concept deletion of the same size removes almost all recall. The loss of general training volume alone therefore does not explain the deficit, consistent with the non-C control (Table 7), where adding data without does not help. Note that a similar experiment cannot be constructed for a very large concept like person: it would require deleting 4,636 non-person images from subject 1's training set, which contains only 4,364.
W6/Q6. Data budgets and the equivalent-data factor
The equivalent-data factor summarizes the gain across the measured budgets; it was not meant as a claim about the full-budget endpoint. We agree that "roughly doubling" invited that reading, and we will replace it. We address the reviewer's points in turn:
- The "doubling" summary. "Roughly doubling" was shorthand for the fitted factors in Table 13, not a claim about the full-budget endpoint. One factor is fitted per metric across all budgets, and it is determined by the budgets where the equivalent baseline lies within the measured range; at the largest budget the baseline is held at its 9,000-image value and does not inform the fit. Measuring the factor at that endpoint would require a baseline trained on roughly twice the full budget, which does not exist without new data collection from the NSD subjects. Across the seven identifiable metrics, the median equivalent data factor is 2.4 for Brain-Diffuser, 2.2 for MindEye2, and 1.9 for Brain-IT. We will replace the "doubling" wording with these per-decoder summaries, stated as equivalent numbers of real training images within the measured range.
- Budgets by image, not by scan. NSD shows each image three times in random order across 40 sessions, so a session-based split exposes different images a different number of times and yields training sets of varying size. Therefore although previous work such as MindEye2 and Brain-IT subsample based on scan numbers as the reviewer suggested, we instead subsample by image and keep all repeats so that the exposure to each image is consistent.
- Update counts. We matched real-data exposure (equal epochs), so every model sees each real sample equally often. Total updates could be matched in two ways. Cutting the augmented model's updates would reduce its exposure to real data. Training the baseline longer is the alternative; although we have not run it, the baseline follows MindEye2's schedule and its validation performance has plateaued by the end of training, so additional updates are unlikely to account for the gains.
- Repeated training. Each condition was trained once per subject, as is common in this literature (e.g., MindEye2, Brain-IT), so our statistics reflect variation across subjects rather than training seeds.
Thank you for providing the supplementary experiments and clarifications, which answered some of my questions. I appreciate the value of the concept retention assessment and your clarification that this is primarily an empirical study. However, the new contributions based on existing synthetic fMRI enhancement techniques remain unproven. The full Brain-IT assay was not evaluated, and the MindEye2 surrogate control could not determine how much the reported concept generalization deficit would persist while retaining the existing enhancement procedures. Therefore, I maintain my original assessment.
Response to Reviewer Zpxz (1/3)
We thank the reviewer for the constructive feedback. Each response below addresses the corresponding weakness and question together. New results are in Tables R1–R4 of the common response and Tables R5–R7 below.
W1/Q1. Contributions and comparison with the full Brain-IT protocol
We agree that encoder-generated fMRI augmentation is an existing technique. Our contribution is a systematic empirical study of concept generalization in modern brain decoders and of the conditions under which synthetic supervision addresses their failures. We will make this distinction clearer in the manuscript.
Comparison with Brain-IT. We disabled Brain-IT's external data so that the added synthetic data is the single variable across all three decoders. We could not retrain the complete Brain-IT protocol within the rebuttal period, and instead we provide some controlled experiments with the MindEye2 probe decoder that isolate the effect of an external synthetic-data pool.
Our holdout models a concept for which the target subject has no fMRI. Brain-IT's external images come from the same COCO distribution as NSD, which leaves two ways to run its protocol under the holdout. Each is matched by one of our controls:
In these controls, the benefit depends on whether the pool covers the missing concept, not on selecting the pool for it. This identifies both a capability and a limit of generic augmentation: it can supply missing neural supervision without advance targeting, but it does not help on concepts outside its pool. Neither control uses the Brain-IT decoder or its external pool; we expect the same to hold there, and will state this correspondence in Sec. 5.1.
Findings. With the new experiments, the study establishes:
(i) The problem. When a concept is withheld from the target subject, three recent decoders degrade on it, including two multi-subject decoders trained with the concept available from other subjects. Existing cross-subject training does not carry the concept over.
(ii) The solution works. Motivated by the finding that encoders built on a pretrained visual representation are more robust to the same holdout than decoders, we show that their synthetic fMRI for the held-out concept recovers 37–87% of the lost performance across three decoders (including multi-subject decoders built on large pretrained priors), four subjects, and twelve concepts.
(iii) What the synthetic data must contain. The gain is not a data-volume effect: it requires the held-out concept in the synthesis pool and correct pairing within it (Table 7, Table R2). In our test, the pool did not need to be selected for the concept, as long as it covered it.
(iv) What the encoder must provide. A pretrained visual representation with a direct readout to voxels sufficed in our tests: even a ridge readout helps, and DINOv2 and OpenCLIP backbones give similar gains, while a random-init backbone or SynBrain gives little (Table R1). An encoder's fidelity on a held-out concept tracks the value of its synthetic data better than its overall accuracy does (Table R4).
W2/Q2. Generalization without concept-targeted augmentation
Yes, the gain remains in the condition we tested (Table R2). We trained the animal-held-out decoder with synthetic data for a general image pool that was fixed in advance and not selected for the test concept: the full training set of another subject. It recovers nearly as much of the gap as targeted synthesis.
We agree with the reviewer's characterization: the decoder does not generalize to the concept unaided, and the synthetic data supplies the supervision it lacks. What this experiment changes is the deployment requirement. The concept does not have to be known and targeted in advance; it only has to appear in a broad synthesis pool, and images are cheap to add where fMRI is not. A pool without the concept gives no such gain (non-C control, Table 7). However, targeting the synthetic data pool can be still useful when the concept is known: it reaches a similar performance without needing to add as much training data, reducing the burden of decoder training.