- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a method for explicit fmri alignment before doing fmri to seen image decoding. They first learn a subject-specific transform of fmri voxels designed for a single reference subject and optimized for seen image decoding. Then subsequent subjects have their fmri voxels projected onto this latent space while the pipeline adapts to cross subject data. The initial reference subject's space creates an alignment anchor point. They show that this improves performance across metrics in low data regimes.
Critical strengths
The paper is well motivated and shows improvements against baselines in low data regimes. They show that it works well at only 1 hour of scan time and can likely improve interpretability by forcing the models to explicitly align in the latent space.
Critical weaknesses
On full data the method only matches baselines like MindEye1. The alignment only helps in low data regimes and the gains in reconstruction performance close at higher data.
I would like to see more interpretability analyses for the claims around "allows for more precise post-hoc analysis after extracting embeddings from the common space" and "the resulting representations not only place different subjects in a shared space but also align them semantically" . The comparisons with baselines in appendix K and L only uses the low data versions. I would like to see some comparisons with the full models to see how much the injected inductive bias from explicit subject alignment helps at scale.
Questions for authors
How are the MindEye2 baselines calculated? The original ME2 pipeline uses multi subject training and single subject finetuning for low data results. Do the baselines also use multi subject pretrain or only single subject? The captions imply that subject 1 was used as pretrain and finetunes were from there.
Thank you again for your time and for the detailed review you provided on our paper. As the discussion period is ongoing, we wanted to check in and see if you have any follow-up questions or if there is any part of our response we could clarify further for you. We are ready to provide any additional information that might be helpful.