- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a novel, brain-inspired approach to improve performance of decoding imagined images, bringing it closer to decoded viewed images.
The model architecture suggested is based on a prior work (Mirage from Kneeland et. al, 2026) – three ridge-regression models per-subject, predicting low level image features, CLIP text features and CLIP image features, and using all 3 with a stable cascade generator.
Their approach to imagined stimulus utilizes the much wider dataset of viewed images (NSD) and prior neuroscience results showing the neural difference between the 2 tasks. Their approach is made of two main elements: • Training the model on augmented fMRI, augmented by 3 hierarchical brain levels. • Training another model that given the fMRI signal predicts the relative importance of each embedding level in the image generation (embedding levels - low level image features, CLIP Text and CLIP image).
The paper shows their model’s result on both imagined and viewed stimuli, showing the technique improves the score over both settings. They then test each part with various ablations, showing each has an effect.
Critical strengths
• The data augmentation circumvents the main challenge in the imagery task – the data shortage.
• Improvement on key metrics of the task.
Critical weaknesses
• Partial and unclear report of qualitative and quantitative results:
- On a test-set of 18 images, one should include all 18 visual reconstructions. The paper shows only 6 out of the 18 test reconstructions. Why?
- It is not clear from the paper which subject is being evaluated in the main paper, and how the results (both in Fig. 6 and in Table 1) were selected: Are these reported reconstructions from the mean fMRI of all 16 repeats per stimulus? Or are these the results of the best repeat for each stimulus? Please clarify.
• Unclear/Unjustified choice of fMRI augmentations:
- It is unclear nor justified why rotation of the projected sub-fMRI vector is a reasonable augmentation for imagined data. It seems to me that this would just shift fMRI activations from some brain voxels to other brain voxels.
- It seems from Figure 5a that the small final test dataset (18 test samples) was also used as a validating set for choosing the augmentation parameters. If so, this is not a valid practice, as it leads to overfitting and leakage of the test data into the training (the same set is afterward used for testing the method).
• Not reproduceable: The method is not reproduceable from the paper; many important details are missing. See detailed questions below.
A COMMENT ABOUT MY FINAL RATING:
I was debating between Weak Accept & Weak Reject, but I was forced to choose one of the two. If the rebuttal provides satisfactory answers to some of my concerns, I may raise my final score.
Questions for authors
• In the rebuttal, please provide all 18 visual reconstructions of the 18 test stimuli.
• Why is rotation of the projected sub-fMRI vector a reasonable augmentation? Doesn’t it just shift fMRI values from some brain voxels to other brain voxels? What’s the neuroscientific justification of this?
• How were the actual augmentation parameters selected for each of the hierarchy levels? It seems from Fig. 5a that the small final test dataset was also used as a validating set for these choices. If so, this is not a valid practice, as it leads to overfitting and leakage of the test data into the training.
• The preference model itself is under specified. What is the actual model? 3 Ridges? MLP?
• Subspace Perturbation: What is the chosen rank in the subspace projection of each brain level? How was it chosen? How many (and which) fMRI scans were used to compute these subspaces?
• The normalization map on the projections of π is never specified (just says “maps values to [0,1]”). How is the normalization done?
• How were the qualitative image results chosen?
• I didn’t understand figure 5b. Is this an intuitive infographic or actual measured results? Why aren’t the axes marked? How were the 2 axes computed from the 3-tuple simplex?
• What is the difference between “no control” and “fixed control”? (isn’t ‘no control’ just fixed control with coefficients of 1?) What are the fixed values? Additionally, “adaptive mapping” is defined only on the appendix, so is unclear given the main text alone.
• For the “concept” test stimuli – How were the metrics computed? From the image showing the concept word, or were the subjects asked verbally to imagine the concept?
• I believe that in Fig 5a the legend has an error – “Vision” should be replaced with “Imagery”.
10 comments
Q2. [Weakness #1, Question #2] Neuroscientific motivation and interpretation of subspace rotation
We thank the reviewer for requesting a clearer explanation of this operation. We now distinguish its mathematical effect from its neuroscientific motivation.
What the rotation changes. The operation perturbs the coordinates of an activity pattern in a low-rank representation within a cortical group. Mapping the perturbed representation back to voxel space changes the resulting voxel activations, but it is not a permutation of voxel locations or a literal transfer of neural activity between anatomical sites. It is a training augmentation that changes combinations of distributed activity components while retaining the cortical-group assignment.
Neuroscientific motivation. The most directly relevant evidence is Saha Roy et al. [31], already cited in our manuscript. Their analysis of perception-to-imagery transformations found that imagery and vision occupy differently oriented subspaces in early visual cortex, alongside reduced imagery dimensionality. Alignment and dimensionality were more similar in higher visual regions. Related geometric differences were observed in a second spatial-imagery dataset. These findings motivate allowing hierarchy-dependent changes in representational geometry rather than modeling imagery solely as uniformly weakened vision.
We nevertheless distinguish this motivation from an exact biological model. Our sampled rotations are not estimates of the transformation directions measured in [31]. In particular, rotating coordinates within a fixed PCA subspace does not necessarily reorient that entire subspace in voxel space. We use this operation as a simplified augmentation inspired by representational variability, rather than claiming that the brain performs the same mathematical operation.
Empirical evidence and additional controls. Our existing rotation-strength ablation provides evidence of utility: removing rotation reduces CLIP similarity from 0.583 to 0.566 and overall visual cortex correlation from 0.169 to 0.158 under the evaluated configuration. This supports including the operation, but does not independently establish its biological validity.
We conducted matched-strength global-noise and shuffled-hierarchy-schedule comparisons to assess the benefits of structured augmentation beyond generic regularization. The schedule-shuffling control specifically tests the assignment of attenuation schedules to cortical levels; it does not validate the sampled rotation directions.
Reference (numbering follows the submitted manu)
[31] Saha Roy, T., Breedlove, J., St-Yves, G., Kay, K., & Naselaris, T. (2025). A transformation from vision to imagery in the human brain. bioRxiv preprint. https://doi.org/10.1101/2025.09.02.672180
Q3. [Weakness #2, Question #3] Augmentation parameter selection and potential use of imagery test data
We thank the reviewer for raising this concern: The imagery data in Fig. 5(a) were used only for post-hoc distributional analysis, not to select augmentation parameters. The configuration was fixed before this analysis, and the reported distances were not used to select or revise it.
The hierarchy-dependent perturbation design was motivated by prior evidence of greater perception–imagery differences in early visual regions [31]. The numerical settings, including PCA rank 8 and rotation strength 0.35, were predefined implementation choices rather than parameters fitted to imagery data.
The augmentation subspaces were fitted exclusively to vision-training inputs. The separate PCA basis used in Fig. 5(a) includes imagery observations only to define the evaluation coordinates; it is not used for training or augmentation.
Q4. [Weakness #3, Question #4] Architecture and training of the hierarchy-preference model
We thank the reviewer for identifying this omission. The hierarchy-preference predictor is a single nine-input, three-output affine ridge-regression model, not an MLP. It is distinct from the three decoder branches that predict CLIP text, CLIP image, and VDVAE features.
Input features. For each of the early-, mid-, and high-level cortical groups, we compute three statistics: mean absolute activation, standard deviation, and root-mean-square activation. Concatenating them produces a nine-dimensional vector. These features are standardized using participant-specific vision-training means and population standard deviations; standard deviations below are replaced by .
Training objective. The three-dimensional target is derived from normalized reference-decoder sensitivities to cortical-group ablation. With standardized input features , the predictor is
and is fitted using
where and the intercept is unregularized. The model has 27 regression weights and three intercepts, totaling 30 trainable parameters per participant.
Inference. Predicted outputs are clipped below at and normalized across the three groups:
The training-fitted standardization statistics and regression parameters are reused at inference. The resulting preference vector provides the input to the gating and reconstruction-control mappings. We have added the predictor architecture and objective explicitly to the method description and provide the sensitivity-target normalization in the implementation details.
Q5. [Question #5] PCA subspace ranks and fitting data
We use the same fixed PCA rank at all three cortical levels: . PCA is fitted separately for each participant and cortical group using mean-centered, preference-gated vision-training inputs, before applying perturbations or adding augmented copies. The first eight right singular vectors define the perturbation subspace; the orthogonal residual is retained.
For each participant (1, 2, 5, and 7), fitting uses all 27,000 vision-training scans in the 40-session configuration, corresponding to 9,000 distinct NSD images presented three times each. Repeats are retained as separate samples. No imagery data, held-out shared1000 stimuli, or scans from other participants are used.
Rank 8 is a fixed implementation choice, not selected by an explained-variance threshold or a documented cross-validation search. Appendix F.4 additionally evaluates ranks 4 and 16, showing trade-offs across metrics rather than uniform superiority of rank 8.
Q6. [Question #6] Normalization of the preference projections
We thank the reviewer for identifying this missing definition. We normalize each linear projection using its analytic range over the preference simplex.
Since and , a projection lies between and . We define
The bounds depend only on the predefined coefficients, not on training or test statistics. If the coefficient range is below , the implementation returns zero. We have added the analytic-range normalization formula and its degenerate-range handling in Appendix B.
Q7. [Question #8] Interpretation and coordinate construction of Fig. 5(b)
We thank the reviewer for requesting clarification. Fig. 5(b) is a two-dimensional visualization of model-predicted hierarchy preferences, with stimulus thumbnails replacing scatter points. For stimulus i, its position is
where the horizontal axis represents high-level preference and the vertical axis represents early-level preference.
The preferences are obtained from the per-stimulus early, mid, and high scores saved for participants. The axis labels indicate the preference components; numerical ticks were omitted for visual clarity, which made the coordinate interpretation insufficiently explicit. We have clarified in the Fig. 5(b) caption that thumbnail coordinates represent predicted high-level and early-level preferences on the horizontal and vertical axes, respectively.
Thank you for your thoughtful and constructive feedback. We greatly appreciate your recognition of our approach to addressing imagery data scarcity and the improvements on key evaluation metrics. We also acknowledge that several experimental and methodological details require clarification, particularly the reconstruction protocol, augmentation parameter selection, and implementation of hierarchy-conditioned control.
We have incorporated all new results and suggested changes in our revised manuscript uploaded, with revisions highlighted in blue.
Q1. [Weakness #1-2, Question #1&7] Complete reconstruction results and clarification of the evaluation and selection protocol
We thank the reviewer for requesting the complete reconstruction results and clarification of our evaluation and selection protocol.
Complete reconstruction results and example selection.
Figures 6 and 7now jointly present all 18 test stimuli, with reconstructions from WALLBREAKER and the comparison methods. The original six examples were selected to illustrate the three stimulus categories, with two examples per category. For simple stimuli, we selected “+” and “×” because their intersecting contours provide more challenging cases than individual line segments. The additional figure includes the remaining 12 stimuli, allowing readers to assess reconstruction quality across the complete test set.Participant and repeat handling. Following MIRAGE [15], the qualitative reconstructions in
Figures 6 and 7are shown for participant 1.Table 1reports aggregate results across participants 1, 2, 5, and 7, with models trained and evaluated separately for each participant. For mental imagery, we average all 16 fMRI repetitions of each stimulus voxel-wise before decoding; we do not select the best individual repetition. Vision reconstructions similarly use the voxel-wise mean of all eight available repetitions.Generation and quantitative evaluation. We follow the retrieval-pooling and repeated-reconstruction evaluation protocol of MIRAGE [15]. Each reconstruction run generates 16 candidate images from the same repeat-averaged fMRI input. One candidate is selected by cosine similarity to the fMRI-predicted retrieval embedding, without accessing the ground-truth target image. This generation-and-selection procedure is repeated 10 times, and quantitative metrics are averaged across the resulting reconstructions, stimuli, and participants. The qualitative figures illustrate individual reconstructions,
Table 1further reports the aggregate evaluation rather than scores computed only from the displayed examples.Reference (numbering follows the submitted manuscript)
[15] Kneeland, R., Torrico Villanueva, C. K., Chen, T., Ojeda, J., Khanna, S., Xu, J., Scotti, P. S., & Naselaris, T. (2026). MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery. PLOS Computational Biology, 22(5), e1014263. https://doi.org/10.1371/journal.pcbi.1014263