acceptodds
Under review as a conference paper at ICLR 2027

MindLoop: Closed-Loop Brain Supervision for fMRI-to-Image Reconstruction

Abstract

Decoding visual experience from brain activity offers a direct window into how the brain represents the visual world. Modern fMRI-to-image reconstruction decodes cortical activity into conditioning representations for large pretrained generators. However, the targets used to learn these representations are derived primarily from stimulus images and generator-specific objectives: trial-specific brain responses serve as inputs but provide little direct supervision. We refer to this imbalance as supervision asymmetry: it leaves the decoded representation only indirectly constrained by the measured cortical response, allowing a generator to produce plausible images that are weakly grounded in the underlying brain activity and may therefore fail to reflect the individual's own perceptual experience on a given trial. To address this, we introduce MindLoop, which makes the measured cortical response an explicit source of supervision for reconstruction, constraining decoding at three complementary levels: Brain Latent Back-Projection enforces consistency with the same-trial cortical state, Dual-Anchored Relational Alignment preserves stimulus geometry across visual and cortical representations, and Hierarchy-Aware Cortical Fields incorporate retinotopic organization into visual-token readout. On the Natural Scenes Dataset, MindLoop outperforms seven representative methods on seven of eight standard reconstruction metrics, averaged across four subjects. More importantly, brain-response substitution experiments show that reconstruction content tracks the measured cortical response: replacing the brain response with that elicited by a different image collapses identification accuracy to chance and shifts the reconstruction toward the image associated with the substituted response. Cortical fields learned without explicit retinotopic supervision independently recover retinotopic organization, and this organization's horizontal mapping degrades substantially when horizontal-flip augmentation breaks image–cortex correspondence. Together, these results show that balancing supervision from cortical responses and visual stimuli improves reconstruction fidelity, providing a more faithful route toward reconstructing individualized visual experience.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI-to-image reconstruction, brain decoding, supervision asymmetry, brain-derived supervision, diffusion models, retinotopy, representation alignment, Natural Scenes Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RcyoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MindLoop, a model that grounds brain-to-image generation in measured brain signals. It integrates three components: Brain Latent Back-Projection, Dual-Anchored Relational Alignment, and Hierarchy-Aware Cortical Fields. Experiments and analyses demonstrate the effectiveness of the proposed approach, while ablation studies highlight the contribution of each component.

Critical strengths
  1. The work addresses an important issue in brain-to-image generation: reducing reliance on details introduced by the generator rather than supported by measured brain signals. This issue has been largely overlooked by existing baselines.

  2. The proposed Hierarchy-Aware Cortical Fields (CF) module is effective and novel, incorporating domain-specific inductive biases into the reconstruction framework.

Critical weaknesses
  1. Although the method reconstructs brain signals from latent representations (Brain Latent Back-Projection), this does not guarantee that the generator will use the brain-derived information encoded in those representations or refrain from inventing details.

  2. The authors propose Dual-Anchored Relational Alignment. However, distances between brain signals do not imply corresponding structural similarities between generated images. Enforcing such a correspondence risks introducing artifacts.

  3. A central claim of the manuscript is that the proposed method encourages the generator to rely on brain-derived latent information rather than invent details. However, the experimental evaluation does not directly test this claim. Most experiments and analyses instead focus on general brain-to-image reconstruction performance.

Questions for authors

Figure 7 visualizes attention maps over cortical voxels. However, the spatial bias in the attention computation already encodes a prior favoring such spatial organization. How much additional benefit does learning provide beyond this imposed prior?

4 comments
AuthorsOpenReview ↗

Response to W1: Brain-derived information in generated images

We thank the reviewer for highlighting this distinction. We agree that constraining the decoded representation alone does not guarantee that the generator will preserve its brain-derived information or avoid adding unsupported details. We therefore evaluated correspondence with measured brain responses at the final-image stage, in addition to examining the decoded conditions.

The proposed constraints improve brain-response correspondence in the generated images themselves. Using an independent encoding model, we tested whether each measured fMRI response could be matched to its corresponding decoded condition or final reconstruction among 982 candidates. In the Subject 1 analysis reported in Fig. 3, the full model outperforms the plain decoder at both stages:

ModelDecoded condition (%)Final reconstructed image (%)
Plain decoder80.137.7
Full model87.743.7

The improvements are significant for both the decoded condition (paired bootstrap p < .001) and the final reconstruction (p = .002). The final-image result is particularly relevant to the reviewer’s concern: it evaluates the generated output, rather than only the representation supplied to the generator. The gain from 37.7% to 43.7% shows that the full model’s improved correspondence with measured responses remains detectable after image generation.

A complementary brain-swap control checks whether generated content follows the supplied brain input. We replaced the measured response with one evoked by a different image while keeping the random seeds and reconstruction settings fixed. Candidate selection continued to reference the originally viewed image. We then scored the same swapped reconstruction against both the original and donor images:

Brain inputScored againstPixCorr ↑CLIP identification ↑ (%)
OriginalViewed image.374593.05
SwappedViewed image.006947.93
SwappedDonor image.370289.57

Following the swap, the reconstruction matches the donor image substantially better than the originally viewed image. The plain decoder exhibits the same qualitative behavior, so we interpret this experiment as a check of input dependence, not as evidence of an advantage specific to the proposed constraints. That advantage is supported by the independent encoding comparison above.

Together, these results show that generated content follows the supplied fMRI response and that the proposed constraints improve the final output’s correspondence with measured brain responses. The reviewer’s comment also highlights an exciting next step: moving beyond overall brain–image correspondence to determine which specific generated details are supported by the measured response. Developing ways to assess and constrain this correspondence at the level of individual image details would be a valuable direction for future work, which we will discuss in the Conclusion.

AuthorsOpenReview ↗

Response to W2: Dual-Anchored Relational Alignment

We thank the reviewer for the opportunity to clarify how the two relational constraints are defined. Dual-Anchored Relational Alignment does not equate distances between brain responses with structural similarities between generated images. Each term uses reference relationships defined within its own representation space.

The visual anchor aligns the relational structure of brain-decoded CLIP tokens with that of the corresponding ground-truth image CLIP tokens. Its target geometry therefore comes from real images in CLIP space. The cortical anchor aligns relationships among predicted brain features with those among measured brain features within the cortical representation space. Neither term directly requires generated images to reproduce the distance structure of measured brain responses.

To assess the effect of this supervision on reconstruction, we compare the full model with the configuration that removes both relational anchors while retaining Brain-LBP and CF. We extended this comparison from Subject 1 to all four participants. The table below reports without relational anchors → full model:

SubjectInception-V3 ↑ (%)CLIP ↑ (%)EffNet-B ↓SwAV ↓
196.76 → 96.8194.03 → 94.12.6198 → .6058.3673 → .3477
295.56 → 95.8692.96 → 93.27.6329 → .6278.3672 → .3621
596.30 → 95.7693.12 → 93.19.6255 → .6247.3777 → .3700
792.41 → 93.1288.45 → 89.98.7014 → .6778.4159 → .4022

CLIP identification, EffNet-B, and SwAV improve in every participant, while Inception-V3 identification improves in three of four participants. These results support the joint contribution of the relational anchors to reconstruction quality under the tested settings.

The reviewer’s concern about artifacts also highlights the distinction between overall reconstruction quality and specific visual errors. Our subgroup analysis identifies human pose and anatomy as remaining difficulties, but does not isolate their relationship to the relational losses. We therefore interpret the ablation as evidence of improved reconstruction fidelity, without treating aggregate metrics as a direct measure of anatomical plausibility. We will clarify the spaces and reference relationships used by the two anchors in the method section and discuss the remaining human-reconstruction errors in the Limitations section.

AuthorsOpenReview ↗

Response to W3: Evidence beyond general reconstruction performance

We thank the reviewer for highlighting the need to connect our evaluation more explicitly to this claim. Our evaluation goes beyond general reconstruction performance: Section 4.4 and Fig. 3 directly assess whether the final generated images correspond to measured brain responses, alongside analyses of input dependence and measurement-specific information. We have expanded and clarified these analyses on pages 9–10 of the revised manuscript.

Most directly, Fig. 3b evaluates brain-response correspondence after image generation. Using an independent encoding model fitted on training images, we tested whether each measured response could be matched to its corresponding final reconstruction among 982 candidates. In Subject 1, the full model improves identification over the plain decoder from 37.7% to 43.7% (paired bootstrap p = .002). Identification from the decoded condition also improves, from 80.1% to 87.7% (p < .001). These results show that the benefit of the proposed constraints is observable in the generated images themselves, rather than being confined to the latent representations.

The brain-swap analysis provides complementary evidence of input dependence. With reconstruction settings and random seeds fixed, replacing the fMRI response shifts the output toward the specific donor image: CLIP identification is 89.57% against the donor image, compared with 47.93% against the originally viewed image. The plain decoder exhibits the same pattern, so this experiment checks whether the pipeline follows the supplied brain response; the independent encoding comparison above establishes the additional benefit of our constraints.

The four-participant repeated-presentation analysis further shows that the forward decoded conditions retain information distinguishing individual measurements of the same image. We distinguish this condition-level evidence from the final-image evaluation, which directly addresses the reviewer’s concern about the generated output.

Together, these analyses support both dependence on the supplied brain input and improved correspondence between generated images and measured responses. The reviewer’s comment also highlights an exciting next step: determining which specific generated details are supported by the brain signal and developing constraints at that finer level. We will discuss this direction in the Conclusion and make the connection between our central claim and the evidence in Section 4.4 more explicit.

AuthorsOpenReview ↗

Response to Q1: Spatial prior versus learned cortical organization

We thank the reviewer for raising this important distinction. The Gaussian prior encourages local cortical weighting, but does not specify which cortical location should correspond to each image query. Field centers are randomly initialized, and all fields start with the same width. Thus, neither contralateral correspondence nor a retinotopic ordering of query–cortex mappings is prescribed by the initialization.

To test whether the observed organization depends on learning from paired data, we compared models trained with and without horizontal-flip augmentation, keeping the architecture and spatial-prior formulation unchanged. Horizontal flipping disrupts the consistency of horizontal image–cortex correspondence while preserving the vertical correspondence.

Mapping correlationWithout horizontal flippingWith horizontal flipping
Horizontal image–cortex mapping−.48+.09
Vertical image–cortex mapping−.55−.56
Horizontal mapping in Brain-LBP predictor attention−.59−.07

The result is selective: the horizontal mapping is largely lost with flipping, while the vertical mapping remains nearly unchanged. Because both models use the same spatial-prior formulation, this difference shows that the specific image–cortex correspondence depends on the spatial consistency of the training pairs. Encouragingly, the Brain-LBP predictor’s attention exhibits the same horizontal effect without using the CF spatial bias, providing complementary evidence that this organization is learned rather than prescribed by the Gaussian prior.

The reconstruction ablations address a related but distinct question: whether the CF module benefits reconstruction. Holding Brain-LBP and both relational anchors fixed, adding CF improves all eight image metrics in Subjects 1 and 2 and seven of eight in Subjects 5 and 7. This establishes the utility of the complete CF module, although it does not isolate the performance gain from learning its parameters relative to a fixed-prior version.

Together, these analyses distinguish the roles of the prior and learning: the prior encourages locality, while training determines the specific correspondence between image queries and cortical locations. We will clarify this distinction in the discussion of Fig. 7 and explain how the horizontal-flip control complements the reconstruction ablations.

Reviewer_UzrFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MindLoop to address the weak grounding of existing fMRI-to-image reconstruction methods in measured brain responses due to their reliance on image-derived supervision. MindLoop introduces three brain-informed constraints: Brain Latent Back-Projection (Brain-LBP), Dual-Anchored Relational Alignment, and Hierarchy-Aware Cortical Fields (CF). Experiments on four NSD subjects show improved reconstruction metrics. The authors further show that reconstruction content tracks measured cortical responses under brain-response substitution and that the learned cortical fields recover retinotopic organization.

Critical strengths
  1. Important problem. The gap between "visually plausible" and "supported by current brain response" in fMRI-to-image reconstruction is worth studying.
  2. MindLoop integrates complementary constraints at the trial, batch, and cortical levels into a unified framework; and hierarchy-aware attention is a clear and reasonable inductive bias.
  3. The paper writing, illustrations, and experiment presentation are generally clear.
Critical weaknesses
  1. Brain-LBP's effect on the inference path is indirect, and it cannot capture trial-specific information. The backward branch of Brain-LBP takes ZgtZ^{gt} as input as Eq. 4. Because Hˉ\bar{H} is stop-gradient (Eq. 3) and \hat{H} depends only on ZgtZ^{gt}, the online cortical encoder EθE_\theta receives no gradient from the Brain-LBP objectives. These objectives reach the forward path only through the closure term Lcyc\mathcal{L}{cyc} via shared decoder. Moreover, H^\hat{H} is a deterministic function of the visual stimuli and cannot carry the "trial-specific cues" described in abstract and introduction.
  2. The baseline results in Tab. 1 are inconsistent with the published values. The published EffNet-B and SwAV scores of MindEye are 0.645 and 0.367, whereas Table 1 reports 0.6605 and 0.3829. Similarly, the published refined MindEye2 results report 0.619 and 0.344, compared with 0.6858 and 0.3770 in Table 1. The authors should clarify the differences. In addition, Brain-IT, which is discussed in the introduction and uses the same SDXL generator, could be included in the comparison.
  3. The ablation study does not support the claimed contribution. In Tab. 2 row 4, (LBP + RvisR_{vis} + RbrainR_{brain}, without CF) is worse than row 1 (plain decoder) on 7 metrics. Moreover, row 2 (RvisR_{vis} + CF, without LBP or RbrainR_{brain}) achieves performance quite close to the full model. These results suggest that the incremental contributions of the brain-derived components may be limited and weaken the claimed effectiveness of the proposed closed-loop supervision mechanism.
  4. Minor presentation issues:
    • DpixD_{pix} in Eq. 5 is never defined.
    • Line 273: a redundant equation reference: "Eq. equation 5".
Questions for authors
  1. Does Brain-LBP have unique advantages over a direct forward brain-consistency constraint? The paper's core argument is that a closed-loop design better preserves trial-specific brain information. To verify this, it would be better to compare Brain-LBP with a direct brain-consistency baseline (for example, map the forward prediction ZpredZ_{pred} through a brain encoding model and constrain the output to match the xx or Hˉ\bar{H}.
  2. Could the authors clarify the evaluation protocol used to obtain the baseline results in Table 1?
6 comments
AuthorsOpenReview ↗

Response to W1: Brain-LBP gradient paths and trial-specific information

We thank the reviewer for carefully tracing the gradient paths. Brain-LBP was designed as an auxiliary training task for the shared decoder: during training, its closure loss updates decoder parameters that, at inference, operate on the latent of the measured fMRI response. Its intended role is therefore to improve forward decoding through shared parameters, rather than to supply trial-specific information through the stimulus-driven backward prediction.

We agree with the reviewer on both technical points. First, because H^\hat H is a deterministic function of ZgtZ_{\mathrm{gt}}, it is identical across repetitions of the same image and cannot itself contain trial-specific information. Second, because Hˉ\bar H is a stop-gradient target, the Brain-LBP objectives do not update the online cortical encoder EθE_\theta. The closure loss Lcyc\mathcal{L}_{\mathrm{cyc}} instead updates the query embeddings, decoder layers, and output projection shared with the forward path. Trial-specific information enters through the measured fMRI latent HH; the question is whether Brain-LBP training helps the shared decoder preserve that information in its forward output. We have revised the abstract, introduction, and Section 3.2 to make this distinction explicit.

To test this directly, we held stimulus identity fixed and decoded the three fMRI measurements of each test image separately. An independent encoding model trained on real images mapped each decoded condition to a predicted cortical response. Trial ID measures how often a measured response matches the prediction from its own trial among the three repetitions, after removing session means (chance: 33.3%). Residual rr measures agreement between predicted and measured deviations across repetitions of the same image.

MetricSubject 1Subject 2Subject 5Subject 7
Trial ID (%), full SDXL model90.793.391.690.3
Trial ID (%), Brain-LBP ablation88.791.488.889.2
Residual rr, full SDXL model.232.240.252.266
Residual rr, Brain-LBP ablation.222.227.233.251

The full model improves Trial ID by 1.1–2.8 percentage points and residual correlation in all four participants (paired bootstrap p≤.001p \leq .001 for residual rr in each participant). This ablation removes Brain-LBP together with the cortical relational anchor.

To test Brain-LBP without the other proposed constraints, we additionally compared the plain decoder with and without Brain-LBP and its associated masking in the Versatile Diffusion pipeline. Adding this training module improves both measures in every participant:

Improvement over plain decoderSubject 1Subject 2Subject 5Subject 7
Trial ID (percentage points)+2.6+2.3+5.3+4.8
Residual rr+.019+.016+.026+.023

The residual-correlation improvements are significant in each participant (paired bootstrap p≤.001p \leq .001). Because these comparisons hold the viewed image fixed, they show that training with Brain-LBP and its associated masking improves the forward decoded condition’s correspondence with differences between individual fMRI measurements, even without the other proposed constraints.

Thus, Brain-LBP’s influence on inference is indirect by design, but produces measurable improvements in the forward path. These results concern measurement-specific information in the forward decoded condition. We have clarified the ablation configuration in Section 4.3 and added the repeated-measurement analysis to Section 4.4.

AuthorsOpenReview ↗

Response to W2 and Q2: Baseline reproduction and evaluation protocol

We thank the reviewer for pointing out this ambiguity and suggesting an additional baseline. The baseline rows in Table 1 are our reproductions, rather than values copied from the original papers. We re-ran the methods on the same four NSD participants using their released implementations and the same GPU model, and scored all reconstructions with the same evaluation script on the same 982 test images. We will explicitly state in Table 1 and the evaluation protocol that the baseline results are our reproductions.

For MindEye2, we attempted to reproduce the results using the released implementation and training configuration, but encountered numerical instability similar to that reported in the official repository’s GitHub issues (e.g., Issue #44, “Issue reproducing fine-tuning results – NaN loss after 40 epochs”). In our runs, the diffusion-prior loss became non-finite and training terminated before completion. To address this, we tested several stabilization measures, including bfloat16 precision, gradient clipping, and skipping updates with non-finite losses or gradients. Further adjustments were needed for runs with persistent instability, including reducing the maximum learning rate from 3×10−43 \times 10^{-4} to 5×10−55 \times 10^{-5}, reducing the prior-loss weight from 30 to 10, and shortening training from 150 to 100 epochs. These changes may affect reconstruction quality and are relevant to interpreting the discrepancy with the published scores. We will explicitly label the baseline rows as reproductions and describe these deviations in the evaluation protocol so that the scope of the comparison is clear.

Regarding Brain-IT, its training requirements exceed what we can complete with our available computing resources within the rebuttal period. Following the reviewer’s suggestion to strengthen the comparison with SDXL-based approaches, we reproduced NeuroFlow and evaluated it under the same four-participant, 982-image protocol:

SubjectPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑EffNet-B ↓SwAV ↓
1.3052.271596.1798.8496.6494.03.6562.3744
2.2843.271795.4198.7095.7693.00.6676.3816
5.2633.262893.2798.1996.0494.57.6553.3779
7.2478.256692.4497.6094.3892.16.6981.4024
Mean.2751.265694.3298.3395.7093.44.6693.3841

We have incorporated NeuroFlow into Table 1 and Fig. 2 of the revised manuscript, adding both quantitative results and qualitative comparisons with another recent SDXL-based method under the same four-participant, 982-image evaluation protocol.

AuthorsOpenReview ↗

Response to W3: Contributions of the proposed constraints (1/2)

We thank the reviewer for examining these comparisons closely. The two comparisons highlight the importance of evaluating the proposed components within their intended configuration. They address different questions: whether the remaining objectives are effective without cortical fields, and whether Brain-LBP and the cortical relational anchor add value when cortical fields are retained. We extended all five configurations to all four participants to examine both questions.

First, the results confirm the importance of cortical fields within the combined framework. For Subject 1, the configuration without CF (row 4) indeed performs worse than the plain decoder on seven of eight image metrics. We therefore do not claim that the remaining objectives alone reliably improve reconstruction. However, comparing this configuration with the full model holds LBP and both relational anchors fixed and isolates the addition of CF. Adding CF improves all eight image metrics in Subjects 1 and 2, and seven of eight in Subjects 5 and 7; the remaining metric, SSIM, decreases slightly in Subject 5 and is unchanged at the reported precision in Subject 7. Thus, the effectiveness of the complete framework depends substantially on the cortical-field component.

Second, Brain-LBP and the cortical relational anchor provide additional gains beyond the strong Rvis + CF configuration. Relative to row 2, the full model improves all eight image metrics in Subjects 1 and 2 and seven of eight in Subjects 5 and 7—30 of 32 comparisons overall. These improvements are modest in magnitude, as the reviewer notes, but their direction is consistent across participants. This comparison evaluates the joint contribution of Brain-LBP, its associated masking, and the cortical relational anchor; it does not isolate their individual effects.

To assess whether these additions also improve correspondence with individual brain measurements, we used the repeated-image analysis described in W1. Trial ID tests whether the decoded condition distinguishes individual fMRI measurements of the same image, while residual (r) measures agreement with their trial-to-trial deviations.

ComparisonMeasureSubject 1Subject 2Subject 5Subject 7
Row 4 − plain decoderTrial ID (pp)+3.6+2.1+6.4+5.3
Row 4 − plain decoderResidual (r)+.024+.015+.035+.034
Full model − row 2Trial ID (pp)+2.0+1.9+2.8+1.1
Full model − row 2Residual (r)+.010+.013+.019+.015

Both comparisons improve the two measurement-level metrics in all four participants. In particular, the full model’s gains over row 2 show that the additional supervision improves correspondence with individual measurements alongside its incremental reconstruction gains. The row 4 comparison also shows why these outcomes should be distinguished: better measurement correspondence does not necessarily translate into better image reconstruction.

Taken together, the expanded ablations support the combined framework, while qualifying claims about individual components. The full model improves all eight image metrics over the plain decoder in every participant, and improves 30 of 32 image metrics over Rvis + CF, with additional gains in both trial-level measures across all four participants. We will revise Section 4.3 to state these conclusions explicitly, describe the jointly ablated components accurately, and include the complete results below.

Reviewer_hpuSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper describes a set of techniques to better incorporate brain activation patterns as targets of optimization in fMRI-based natural image decoding/reconstruction analyses. Typically, these analyses have focused on building objective functions based on errors between image reconstructions (either in image space or other image-related latent spaces) and the ‘ground truth’ pixel representation or latent space embedding. Here, the work attempts to move beyond this by taking into account the measured activation pattern itself, with the logic that producing an image reconstruction more likely to generate a brain activation pattern similar to that measured is a better constraint than focusing on image-space loss alone. Results on 4 participants of the Natural Scenes Dataset support the utility of the approach, and in particular emphasize the importance of selectively weighting signals from the most relevant parts of cortex for retinotopically-corresponding regions of the image. This is overall a promising approach that may lead to improved natural image reconstruction performance in large fMRI datasets.

Critical strengths
  1. Cortical field analyses throughout are very compelling, align with well-established properties of visual system function (contralateral organization), and are likely to lead to additional principled voxel input restrictions for making image reconstruction processes more accurate and efficient
  2. Innovative set of new constraints for natural image reconstruction would lay the foundation for further optimizations, potentially including between-participant and/or between-dataset transfer
Critical weaknesses
  1. Some of the reconstructions, especially those containing people, appear quite distorted (too many limbs, other incoherent features; e.g. Fig. 5), which does not appear well-captured by the quantified performance metrics
  2. While the data are quite compelling, they are focused on a single densely-sampled and high-quality dataset, with no evidence presented that these methods generalize across datasets and/or stimulus types
  3. The ablation analyses discussed in Section 4.3/Table 2 are focused on a single participant rather than the full dataset of 4 participants considered throughout
  4. Analysis/experiment randomizing cortical input (described in lines 439-444) for a given image is poorly described/justified – it is unclear why this would ever be expected to accurately reconstruct aspects of the viewed image, regardless of the specifics of the novel analyses employed here. Perhaps using a similar experiment on other reconstruction methods (MindEye2) could bolster confidence in this analysis.
Questions for authors
  1. The cortical field analyses which accurately recover retinotopic encoding preferences are very compelling. Would the authors be able to report analyses testing whether cortical fields centered at image locations further from the center of the image are selective for larger spatial extents of the image? This would confirm the typical (and cited) result that receptive field size typically increases with receptive field eccentricity
  2. Can the results be extended to at least one additional dataset? (LAION-fMRI, THINGS, etc)? This would substantially bolster confidence that the approach yields generalizable improvements across datasets, image sets, etc.
  3. Can the authors comment on the remaining qualitative differences between reconstructed images and the ground truth images based on their approach? Are there any features that the method seems to reliably fail at accurately reconstructing? Are there insights that can be gleaned about further analysis improvements based on these remaining errors?
8 comments
AuthorsOpenReview ↗

Response to W1: Distortions in reconstructions of people

We thank the reviewer for this helpful observation about Fig. 5. We agree that some reconstructions preserve the overall scene layout while depicting implausible human poses or extra limbs, and that the aggregate metrics do not fully capture these local anatomical errors.

To assess performance specifically on images associated with people, we split the 982 NSD test images according to whether their COCO captions mention a person and evaluated both subsets across all four participants. Each cell reports person mentioned (349 images) / no person mentioned (633 images), averaged across participants.

MethodGeneratorPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑EffNet-B ↓SwAV ↓
MindLoop-SDXL (ours)SDXL.313 / .367.461 / .43296.5 / 96.498.1 / 98.493.4 / 95.087.2 / 93.9.647 / .627.372 / .370
MindLoop-VD (ours)VD.303 / .354.419 / .38993.6 / 92.796.0 / 96.692.4 / 94.091.0 / 94.5.666 / .653.383 / .387
MindEyeVD.285 / .335.352 / .32492.6 / 92.795.7 / 96.592.5 / 93.489.3 / 93.5.662 / .659.379 / .385
MindEye2SDXL.245 / .285.412 / .37995.3 / 95.198.1 / 97.991.6 / 92.080.8 / 90.6.685 / .686.368 / .382

The subgroup analysis shows that CLIP identification is lower for the person-mentioned subset across all four methods, indicating a shared challenge in identifying reconstructions of these images. Nevertheless, MindLoop retains an advantage over the corresponding baseline with both generators. On this subset, MindLoop-SDXL achieves higher CLIP identification than MindEye2 (87.2% versus 80.8%), and MindLoop-VD exceeds MindEye (91.0% versus 89.3%). MindLoop also achieves higher PixCorr and SSIM than the corresponding baseline with each generator.

Thus, MindLoop’s quantitative improvements extend to this challenging subset across both generator families. We will discuss the remaining human pose and anatomy errors, and the limitations of the reported metrics in capturing them, in the Conclusion of the revised manuscript.

AuthorsOpenReview ↗

Response to W2 and Q2: Evaluation beyond NSD

We thank the reviewer for suggesting an evaluation beyond NSD. We have extended the evaluation to all four BOLD5000 participants (CSI1–4), training and evaluating MindLoop-VD with the architecture, loss weights, training duration, and inference settings established on NSD, without dataset-specific tuning.

BOLD5000 provides a complementary evaluation setting: it uses 3T rather than 7T fMRI, contains fewer training images per participant (4,803 for CSI1–3 and 2,862 for CSI4, versus approximately 9,000 in NSD), and draws stimuli from COCO, ImageNet, and SUN. This allows us to test whether the reconstruction benefits extend to a different acquisition protocol, training-data scale, and stimulus mix. The test sets contain 113 images for CSI1–3 and 90 for CSI4.

Table A. BOLD5000 reconstruction results averaged across four participants. We reproduced all six baselines on BOLD5000 following their respective protocols. Identification accuracies are reported in percentages.

MethodPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑EffNet-B ↓SwAV ↓
MindLoop-VD (ours).1347.365674.8684.8277.0284.37.8658.5390
NeuroPictor.0615.347773.8884.5874.7683.13.8764.5492
MindEye.0989.259572.1081.8773.2980.21.9046.6099
Brain-Diffuser.0961.256974.8383.9569.0980.87.9109.5921
PRISM.0823.360957.5858.8954.3354.86.9739.6427
DREAM.0262.169159.8468.7365.1682.84.9495.6413
MindBridge.0243.233265.5868.5359.9963.55.9663.7012

MindLoop-VD achieves the best point estimates on all eight metrics. Relative to the strongest baseline for each metric, it improves PixCorr by .0358, Inception identification by 2.26 percentage points, and CLIP identification by 1.24 percentage points. Alex(2) is effectively tied with Brain-Diffuser (74.86% versus 74.83%).

The results also extend across participants: for CSI1–4, MindLoop-VD achieves PixCorr of .157, .127, .144, and .110, and CLIP identification of 86.5%, 84.0%, 84.8%, and 82.1%, respectively.

These findings extend the reconstruction benefits beyond NSD to an independent dataset with different acquisition conditions, fewer training examples, and a different stimulus mix. This is a replication through training on BOLD5000 with the NSD-established configuration, rather than zero-shot transfer of an NSD-trained model. We will report the full results in Appendix C and summarize the additional dataset validation in the main text.

AuthorsOpenReview ↗

Response to W3: Ablations across all four participants

W3 — Ablations across all four participants

Thank you for pointing out that Table 2 reported ablations for only one participant. We repeated the same ablations for Subjects 2, 5, and 7, using the same training and reconstruction protocol across configurations. Together with Subject 1 in Table 2, these results cover all four participants.

SubjectLBPRvisRbrainCFPixCorr ↑SSIM ↑Alex(2) ↑Alex(5) ↑Incep ↑CLIP ↑EffNet-B ↓SwAV ↓
2−−−−0.35190.446497.799.1595.6591.690.63880.3657
2−✓−✓0.36070.445897.9499.3395.5693.040.63180.3626
2✓−−✓0.35160.447497.7799.2395.5692.960.63290.3672
2✓✓✓−0.34120.436696.3998.6594.5291.340.64630.3786
2✓✓✓✓0.36530.448898.0299.4195.8693.270.62780.3621
5−−−−0.31150.429295.6897.4194.7792.950.6330.3886
5−✓−✓0.31860.427396.298.0395.6193.060.6270.3745
5✓−−✓0.31690.435595.7898.2796.393.120.62550.3777
5✓✓✓−0.31550.43494.597.894.4891.620.64210.3829
5✓✓✓✓0.31910.433295.9898.4295.7693.190.62470.37
7−−−−0.29690.429893.299792.489.190.70020.4023
7−✓−✓0.29830.427493.8197.1292.6889.310.67980.3941
7✓−−✓0.28980.426392.7795.8892.4188.450.70140.4159
7✓✓✓−0.29490.433893.7897.0691.8989.130.70230.415
7✓✓✓✓0.30380.433894.197.2793.1289.980.67780.4022

The full model outperforms the model without brain-derived constraints on all eight metrics for each of the three additional participants. Cortical fields make the most consistent contribution across participants. With all other constraints held fixed, adding cortical fields improves seven of the eight metrics for all three participants, with CLIP identification gains of 1.93, 1.57, and 0.85 percentage points for Subjects 2, 5, and 7, respectively. Together with Subject 1 in Table 2, these results show that the full set of constraints improves reconstruction over the unconstrained model across all four participants.

Comments on the paper

2 comments
Area_Chair_mVkgOpenReview ↗

Gentle Reminder: ICLR Rebuttal Discussion Deadline – November 18

Dear Reviewers,

Thank you very much for your time, thoughtful reviews, and valuable contributions to the evaluation process.

The authors have carefully addressed your comments and concerns, and the discussion period will conclude on November 18. We would greatly appreciate it if you could take a few moments to review their responses and results, engage in the discussion, and indicate whether their clarifications or additional evidence affect your assessment.

Your continued engagement is essential to ensuring a rigorous, constructive, and fair evaluation process.

Thank you again for your time, expertise, and dedication!

Best regards, Area Chair

We sincerely thank Area Chair for facilitating the discussion and the reviewers for their valuable time and very thoughtful feedback.We greatly appreciate the feedback received so far and would welcome any further comments or questions from the reviewers regarding our additional experiments and revised manuscript. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.