- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a test-time training-free method that improves cross-subject visual decoding consistency through two modules: MSC and QSC. Specifically, MSC predicts the source-subject fMRI from the current clean latent estimate to provide gradient-based guidance; QSC is applied during the image enhancement stage to pull the VGG features closer to the fMRI-derived blurry image.
Critical strengths
- Clear motivation. Existing representation alignment methods focus on transferring representations across subjects but do not explicitly ensure whether the generated samples are faithful to the observed fMRI. This is intuitive and clearly articulated.
- The proposed method is training-free and potentially applicable to other visual decoding frameworks.
Critical weaknesses
- The cross-subject setting assumes the source subject has quite limited training data. Yet MSC needs a subject-specific encoding model for this subject. If the encoding models (NeuroFlow and SynBrain) were trained on the source subject’s full 40 sessions, then DS-DC uses far more data than the MindAligner baseline (only one session). This could undermine the fairness of the comparison.
- Several prior studies have explored ideas similar to MSC. Brain-optimized inference improves reconstructions of fMRI brain activity [1] uses encoding models to optimize reconstruction results; while BrainDiVE [2] guides diffusion process with encoder gradients. The paper should discuss and compare these methods.
- Compared to using MSC alone, the full DS-DC performed worse on 6 out of 8 metrics. This suggests that QSC may primarily acts as a regulator that balances low-level visual fidelity with high-level semantic quality rather than "complementary roles".
- The theory adds little novelty and does not cite relevant classical literature.
[1] Kneeland, Reese, et al. "Brain-optimized inference improves reconstructions of fMRI brain activity." arXiv preprint (2023).
[2] Luo, Andrew, et al. "Brain diffusion for visual exploration: Cortical discovery using large scale generative models." Advances in Neural Information Processing Systems 36 (2023): 75740–75781.
Questions for authors
- For each source subject, what training data were used to train NeuroFlow and SynBrain?
- MindEye2/MindAligner already use blurred reconstructed images as the starting point for img2img. A simple baseline would be to adjust the img2img strength or the low-level mixing weights within the existing pipeline. Could the authors compare QSC against these alternatives?
3 comments
W2:
Thank you for highlighting these studies. We will expand Related Work to clarify their connections to MSC.
BOI [1] selects and iteratively regenerates candidates using encoder scores, whereas MSC directly refines latents through measurement-residual gradients. We evaluated BOI-inspired reranking and iterative search on MindAligner in both 1→2 and 2→1 directions, with a random-candidate control. The BOI-inspired variants achieve higher PixCorr and SSIM, while MSC—including its few-shot version—performs better on all six image-feature metrics, demonstrating different reconstruction trade-offs.
BrainDiVE [2] uses encoder gradients to generate images that maximize predicted activation in selected cortical regions, revealing their visual preferences. MSC instead seeks agreement with the measured response to a specific stimulus. We did not include BrainDiVE as a direct reconstruction plug-in because this would require changing its original objective. Its encoder-gradient guidance nevertheless provides important methodological inspiration for MSC.
Comparison with BOI-inspired search on MindAligner.
| Direction | Method | PixCorr ↑ | SSIM ↑ | Alex(2) ↑ | Alex(5) ↑ | Incep. ↑ | CLIP ↑ | Eff. ↓ | SwAV ↓ |
|---|---|---|---|---|---|---|---|---|---|
| 1→2 | MindAligner | 0.2214 | 0.4143 | 0.8783 | 0.9235 | 0.8382 | 0.8247 | 0.8093 | 0.4598 |
| + MSC (full-data) | 0.2317 | 0.4000 | 0.9052 | 0.9443 | 0.8512 | 0.8421 | 0.7789 | 0.4394 | |
| + MSC (few-shot) | 0.2348 | 0.4237 | 0.9064 | 0.9381 | 0.8502 | 0.8248 | 0.7772 | 0.4585 | |
| B1: Random candidates | 0.2631 | 0.4683 | 0.7621 | 0.8112 | 0.7617 | 0.7663 | 0.8647 | 0.5374 | |
| B2: BOI-inspired reranking | 0.2603 | 0.4673 | 0.7809 | 0.8439 | 0.7815 | 0.7727 | 0.8530 | 0.5264 | |
| B3: BOI-inspired iterative search | 0.2579 | 0.4700 | 0.7790 | 0.8418 | 0.7798 | 0.7744 | 0.8527 | 0.5316 | |
| 2→1 | MindAligner | 0.2157 | 0.4275 | 0.8756 | 0.9380 | 0.8351 | 0.8286 | 0.8106 | 0.4562 |
| + MSC (full-data) | 0.2167 | 0.4029 | 0.8848 | 0.9421 | 0.8580 | 0.8445 | 0.7854 | 0.4386 | |
| + MSC (few-shot) | 0.2217 | 0.4282 | 0.8881 | 0.9399 | 0.8546 | 0.8292 | 0.7797 | 0.4563 | |
| B1: Random candidates | 0.2314 | 0.4350 | 0.7407 | 0.8115 | 0.7638 | 0.7539 | 0.8707 | 0.5513 | |
| B2: BOI-inspired reranking | 0.2305 | 0.4354 | 0.7630 | 0.8372 | 0.7723 | 0.7665 | 0.8637 | 0.5419 | |
| B3: BOI-inspired iterative search | 0.2293 | 0.4383 | 0.7632 | 0.8328 | 0.7764 | 0.7664 | 0.8637 | 0.5451 |
All methods use MindAligner as the decoding backbone. B1 is the random-candidate control; B2 and B3 are BOI-inspired adaptations using one-round reranking and two-round iterative search, respectively. Best results within each direction are shown in bold.
[1] Kneeland, Reese, et al. "Brain-optimized inference improves reconstructions of fMRI brain activity." arXiv preprint (2023).
[2] Luo, Andrew, et al. "Brain diffusion for visual exploration: Cortical discovery using large scale generative models." Advances in Neural Information Processing Systems 36 (2023): 75740–75781.
W3:
Thank you for this observation. We agree that the ablation results support interpreting QSC as a structural regularizer. Adding QSC improves pixel and structural fidelity, but reduces several feature-based metrics relative to MSC alone. We will revise the “complementary roles” description to explicitly acknowledge this structure–feature fidelity trade-off and retain separate MSC-only and full DS-DC results.
W4:
Thank you for this comment. We will strengthen the connection to classical iterative regularization and cite the relevant literature.
The specific value of our analysis is to characterize the competing effects of refinement with a learned neural forward model. Our finite-step decomposition separates initial-error contraction, forward-residual-induced bias, and nonlinear accumulation. In particular, a nonzero measurement gradient can persist even at the true stimulus representation, so further loss reduction need not improve reconstruction. This connects the PLR iteration count to a concrete correction–bias trade-off and motivates finite refinement.
Q2:
Thank you for suggesting these comparisons. We evaluated them on 1,000 MindAligner test stimuli in the 1→2 direction. The baseline corresponds to an effective img2img strength of 0.48 and a final blurry-image weight of 0.25. We varied strength from 0.2 to 0.8 and blending weight from 0 to 0.5, both in increments of 0.1, changing one parameter at a time. The table also includes Direct-VGG and QSC without reliability weighting to examine the guidance design.
In the reported results, QSC alone achieves higher PixCorr and SSIM than every tested img2img strength. Blending at 0.5 yields higher PixCorr, but QSC provides comparable SSIM with better scores on all six feature metrics. QSC also achieves higher SSIM and better scores on most feature metrics than Direct-VGG and QSC without reliability weighting. These comparisons support selectively using reference features: local retrieval accommodates spatial offsets, while reliability weighting reduces the influence of uncertain matches. This offers finer structural guidance than global strength or blending adjustments. Complete results are provided in the supplementary PDF.
QSC ablations and comparison with simpler structural guidance on MindAligner (1→2, 1,000 test stimuli).
| Method | PixCorr ↑ | SSIM ↑ | Alex(2) ↑ | Alex(5) ↑ | Incep. ↑ | CLIP ↑ | Eff. ↓ | SwAV ↓ |
|---|---|---|---|---|---|---|---|---|
| MindAligner | 0.2214 | 0.4143 | 0.8783 | 0.9235 | 0.8382 | 0.8247 | 0.4598 | |
| + QSC | 0.2232 | 0.8693 | 0.9187 | 0.8411 | 0.8205 | |||
| + QSC w/o reliability | 0.2251 | 0.4091 | 0.8699 | 0.9161 | 0.8348 | 0.8074 | 0.8274 | 0.4816 |
| + Direct-VGG | 0.2255 | 0.4089 | 0.8702 | 0.9164 | 0.8343 | 0.8086 | 0.8278 | 0.4827 |
| Img2img strength = 0.2 | 0.2189 | 0.3837 | 0.8470 | 0.8114 | 0.8244 | 0.4762 | ||
| Img2img strength = 0.7 | 0.2192 | 0.4011 | 0.8600 | 0.9040 | 0.8063 | 0.7973 | 0.8351 | 0.4886 |
| Blending weight = 0.5 | 0.2599 | 0.4296 | 0.8652 | 0.9167 | 0.8341 | 0.8022 | 0.8243 | 0.4829 |
| + DS-DC | 0.4406 | 0.8992 | 0.9322 | 0.8215 | 0.7875 | 0.4683 |
Best results in each column are shown in bold, and second-best results are underlined.
W1 and Q1:
Thank you for highlighting this data-budget concern. The original NeuroFlow and SynBrain operators followed their respective full-data NSD protocols, which report 9,000 training images per subject, for subjects 1, 2, 5, and 7, with the 1,000 shared test images held out. We acknowledge that these operators used more new-subject supervision than the one-session MindAligner baseline.
To address this, we retrained both operators using exactly the same new-subject and known-subject training data as the corresponding MindAligner configuration. For each evaluated direction, the new subject is restricted to the same single session, while the known subject uses the same full training split already available to the baseline. Forward-model training and alignment reuse these training sets, introducing no additional paired fMRI–image samples. We report the revised few-shot results separately from the original full-data results and use the matched-budget comparisons to assess performance under limited-data cross-subject decoding.
Under the matched data budget, DS-DC retains improvements in pixel and structural fidelity across all four subject groups. The full-data operator remains stronger on some higher-level feature measures, potentially benefiting from broader subject-specific stimulus–response coverage. These results show that useful reconstruction gains persist without additional paired training data, although the trade-off across metrics changes.
DS-DC performance with full-data and matched-budget NeuroFlow operators.
Full-data denotes the original NeuroFlow operator; few-shot denotes the retrained operator using the same new-subject and known-subject training data as the corresponding MindAligner baseline. Best results within each subject group are shown in bold.