- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MindLoop, a model that grounds brain-to-image generation in measured brain signals. It integrates three components: Brain Latent Back-Projection, Dual-Anchored Relational Alignment, and Hierarchy-Aware Cortical Fields. Experiments and analyses demonstrate the effectiveness of the proposed approach, while ablation studies highlight the contribution of each component.
Critical strengths
-
The work addresses an important issue in brain-to-image generation: reducing reliance on details introduced by the generator rather than supported by measured brain signals. This issue has been largely overlooked by existing baselines.
-
The proposed Hierarchy-Aware Cortical Fields (CF) module is effective and novel, incorporating domain-specific inductive biases into the reconstruction framework.
Critical weaknesses
-
Although the method reconstructs brain signals from latent representations (Brain Latent Back-Projection), this does not guarantee that the generator will use the brain-derived information encoded in those representations or refrain from inventing details.
-
The authors propose Dual-Anchored Relational Alignment. However, distances between brain signals do not imply corresponding structural similarities between generated images. Enforcing such a correspondence risks introducing artifacts.
-
A central claim of the manuscript is that the proposed method encourages the generator to rely on brain-derived latent information rather than invent details. However, the experimental evaluation does not directly test this claim. Most experiments and analyses instead focus on general brain-to-image reconstruction performance.
Questions for authors
Figure 7 visualizes attention maps over cortical voxels. However, the spatial bias in the attention computation already encodes a prior favoring such spatial organization. How much additional benefit does learning provide beyond this imposed prior?
4 comments
Response to W2: Dual-Anchored Relational Alignment
We thank the reviewer for the opportunity to clarify how the two relational constraints are defined. Dual-Anchored Relational Alignment does not equate distances between brain responses with structural similarities between generated images. Each term uses reference relationships defined within its own representation space.
The visual anchor aligns the relational structure of brain-decoded CLIP tokens with that of the corresponding ground-truth image CLIP tokens. Its target geometry therefore comes from real images in CLIP space. The cortical anchor aligns relationships among predicted brain features with those among measured brain features within the cortical representation space. Neither term directly requires generated images to reproduce the distance structure of measured brain responses.
To assess the effect of this supervision on reconstruction, we compare the full model with the configuration that removes both relational anchors while retaining Brain-LBP and CF. We extended this comparison from Subject 1 to all four participants. The table below reports without relational anchors → full model:
| Subject | Inception-V3 ↑ (%) | CLIP ↑ (%) | EffNet-B ↓ | SwAV ↓ |
|---|---|---|---|---|
| 1 | 96.76 → 96.81 | 94.03 → 94.12 | .6198 → .6058 | .3673 → .3477 |
| 2 | 95.56 → 95.86 | 92.96 → 93.27 | .6329 → .6278 | .3672 → .3621 |
| 5 | 96.30 → 95.76 | 93.12 → 93.19 | .6255 → .6247 | .3777 → .3700 |
| 7 | 92.41 → 93.12 | 88.45 → 89.98 | .7014 → .6778 | .4159 → .4022 |
CLIP identification, EffNet-B, and SwAV improve in every participant, while Inception-V3 identification improves in three of four participants. These results support the joint contribution of the relational anchors to reconstruction quality under the tested settings.
The reviewer’s concern about artifacts also highlights the distinction between overall reconstruction quality and specific visual errors. Our subgroup analysis identifies human pose and anatomy as remaining difficulties, but does not isolate their relationship to the relational losses. We therefore interpret the ablation as evidence of improved reconstruction fidelity, without treating aggregate metrics as a direct measure of anatomical plausibility. We will clarify the spaces and reference relationships used by the two anchors in the method section and discuss the remaining human-reconstruction errors in the Limitations section.
Response to W3: Evidence beyond general reconstruction performance
We thank the reviewer for highlighting the need to connect our evaluation more explicitly to this claim. Our evaluation goes beyond general reconstruction performance: Section 4.4 and Fig. 3 directly assess whether the final generated images correspond to measured brain responses, alongside analyses of input dependence and measurement-specific information. We have expanded and clarified these analyses on pages 9–10 of the revised manuscript.
Most directly, Fig. 3b evaluates brain-response correspondence after image generation. Using an independent encoding model fitted on training images, we tested whether each measured response could be matched to its corresponding final reconstruction among 982 candidates. In Subject 1, the full model improves identification over the plain decoder from 37.7% to 43.7% (paired bootstrap p = .002). Identification from the decoded condition also improves, from 80.1% to 87.7% (p < .001). These results show that the benefit of the proposed constraints is observable in the generated images themselves, rather than being confined to the latent representations.
The brain-swap analysis provides complementary evidence of input dependence. With reconstruction settings and random seeds fixed, replacing the fMRI response shifts the output toward the specific donor image: CLIP identification is 89.57% against the donor image, compared with 47.93% against the originally viewed image. The plain decoder exhibits the same pattern, so this experiment checks whether the pipeline follows the supplied brain response; the independent encoding comparison above establishes the additional benefit of our constraints.
The four-participant repeated-presentation analysis further shows that the forward decoded conditions retain information distinguishing individual measurements of the same image. We distinguish this condition-level evidence from the final-image evaluation, which directly addresses the reviewer’s concern about the generated output.
Together, these analyses support both dependence on the supplied brain input and improved correspondence between generated images and measured responses. The reviewer’s comment also highlights an exciting next step: determining which specific generated details are supported by the brain signal and developing constraints at that finer level. We will discuss this direction in the Conclusion and make the connection between our central claim and the evidence in Section 4.4 more explicit.
Response to Q1: Spatial prior versus learned cortical organization
We thank the reviewer for raising this important distinction. The Gaussian prior encourages local cortical weighting, but does not specify which cortical location should correspond to each image query. Field centers are randomly initialized, and all fields start with the same width. Thus, neither contralateral correspondence nor a retinotopic ordering of query–cortex mappings is prescribed by the initialization.
To test whether the observed organization depends on learning from paired data, we compared models trained with and without horizontal-flip augmentation, keeping the architecture and spatial-prior formulation unchanged. Horizontal flipping disrupts the consistency of horizontal image–cortex correspondence while preserving the vertical correspondence.
| Mapping correlation | Without horizontal flipping | With horizontal flipping |
|---|---|---|
| Horizontal image–cortex mapping | −.48 | +.09 |
| Vertical image–cortex mapping | −.55 | −.56 |
| Horizontal mapping in Brain-LBP predictor attention | −.59 | −.07 |
The result is selective: the horizontal mapping is largely lost with flipping, while the vertical mapping remains nearly unchanged. Because both models use the same spatial-prior formulation, this difference shows that the specific image–cortex correspondence depends on the spatial consistency of the training pairs. Encouragingly, the Brain-LBP predictor’s attention exhibits the same horizontal effect without using the CF spatial bias, providing complementary evidence that this organization is learned rather than prescribed by the Gaussian prior.
The reconstruction ablations address a related but distinct question: whether the CF module benefits reconstruction. Holding Brain-LBP and both relational anchors fixed, adding CF improves all eight image metrics in Subjects 1 and 2 and seven of eight in Subjects 5 and 7. This establishes the utility of the complete CF module, although it does not isolate the performance gain from learning its parameters relative to a fixed-prior version.
Together, these analyses distinguish the roles of the prior and learning: the prior encourages locality, while training determines the specific correspondence between image queries and cortical locations. We will clarify this distinction in the discussion of Fig. 7 and explain how the horizontal-flip control complements the reconstruction ablations.
Response to W1: Brain-derived information in generated images
We thank the reviewer for highlighting this distinction. We agree that constraining the decoded representation alone does not guarantee that the generator will preserve its brain-derived information or avoid adding unsupported details. We therefore evaluated correspondence with measured brain responses at the final-image stage, in addition to examining the decoded conditions.
The proposed constraints improve brain-response correspondence in the generated images themselves. Using an independent encoding model, we tested whether each measured fMRI response could be matched to its corresponding decoded condition or final reconstruction among 982 candidates. In the Subject 1 analysis reported in Fig. 3, the full model outperforms the plain decoder at both stages:
The improvements are significant for both the decoded condition (paired bootstrap p < .001) and the final reconstruction (p = .002). The final-image result is particularly relevant to the reviewer’s concern: it evaluates the generated output, rather than only the representation supplied to the generator. The gain from 37.7% to 43.7% shows that the full model’s improved correspondence with measured responses remains detectable after image generation.
A complementary brain-swap control checks whether generated content follows the supplied brain input. We replaced the measured response with one evoked by a different image while keeping the random seeds and reconstruction settings fixed. Candidate selection continued to reference the originally viewed image. We then scored the same swapped reconstruction against both the original and donor images:
Following the swap, the reconstruction matches the donor image substantially better than the originally viewed image. The plain decoder exhibits the same qualitative behavior, so we interpret this experiment as a check of input dependence, not as evidence of an advantage specific to the proposed constraints. That advantage is supported by the independent encoding comparison above.
Together, these results show that generated content follows the supplied fMRI response and that the proposed constraints improve the final output’s correspondence with measured brain responses. The reviewer’s comment also highlights an exciting next step: moving beyond overall brain–image correspondence to determine which specific generated details are supported by the measured response. Developing ways to assess and constrain this correspondence at the level of individual image details would be a valuable direction for future work, which we will discuss in the Conclusion.