acceptodds
Withdrawn from ICLR 2027

Hi-DREAM: Hierarchical Diffusion with ROI-Aware Representations for fMRI-to-Image Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing natural images from fMRI requires bridging neural activity with both the structural and semantic representations used by modern generative models. Existing diffusion-based decoders often condition on a single global fMRI embedding, which limits their ability to exploit the hierarchical organization of the visual cortex and makes the contribution of different visual areas difficult to inspect. We propose Hi-DREAM, a brain-inspired hierarchical diffusion framework that structures fMRI conditioning according to early, middle, and late visual Regions of Interest (ROI) streams. A ROI adapter converts these streams into a multi-scale cortical pyramid, and a lightweight ROI-conditioned ControlNet injects the resulting anatomy-aware priors into matched U-Net depths during denoising. Rather than learning a single undifferentiated neural representation, Hi-DREAM preserves subject-specific cortical structure throughout the generator-side conditioning pathway. Experiments on the Natural Scenes Dataset (NSD) show that Hi-DREAM achieves state-of-the-art high-level semantic reconstruction while retaining strong low-level structure. Further ablation and attribution analyses show that the proposed hierarchy-aware conditioning is effective, and that different ROI streams provide complementary, inspectable contributions to reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
fMRI-to-image reconstruction, brain decoding, ROI-aware learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Qv8YOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Hi-DREAM organizes fMRI regions into early, middle, and late visual streams and injects their representations at corresponding diffusion-model depths. It reports strong NSD reconstruction metrics and uses ROI, hierarchy, and attention ablations to argue for structured cortical conditioning.

Critical strengths

The paper tests a concrete, interpretable conditioning hypothesis, including capacity-matched random and reversed hierarchy controls in Table 3. Table 1 reports strong semantic performance across four NSD subjects, while retaining competitive low-level reconstruction quality.

Critical weaknesses
  1. The evaluation inputs are unresolved in a way that could change the task. Figure 2 does not specify the source of evaluation text, and the supplied evaluation procedure requests stimulus-associated captions for conditioning. The available materials do not establish whether captions were used in the runs behind Table 1, but reconciling those runs with an explicitly neural-input protocol is necessary before the comparison can substantiate fMRI-based reconstruction.
  2. The spatial representation is not adequately connected to the information it is claimed to preserve. Section 3.2 reduces each ROI to a scalar response multiplied by a projected anatomical mask, removing within-ROI activity patterns. The paper correctly states that the masks do not encode stimulus edges, but does not establish how anatomical coordinates correspond to image coordinates or how this compressed input preserves stimulus-specific layout; this is central to the spatial-conditioning motivation.
  3. The hierarchy controls do not distinguish a useful anatomical bias from disadvantages imposed by alternative priors. Section 3.3 encourages early-to-shallow and late-to-deep mixing, while Table 3 reverses the prior and initialization together. These controls support the selected configuration, but a neutral initialization without the anatomical regularizer would better establish whether the preferred correspondence emerges from the data and contributes beyond a generally learnable multiscale conditioner.
  4. The module ablation does not isolate the claimed cross-stream mechanism. Table 4 shows a large improvement from adding MHLA, but removing attention changes both model capacity and information exchange. Comparing against a similarly sized generic mixer using the same ROI inputs would test whether structured cross-stream interaction explains the gain, rather than the additional processing capacity; this matters because the proposed mechanism is a principal methodological contribution.
  5. The quantitative ROI ablation is insufficient to establish the stated complementary roles. Table 2 compares Early, Early+Middle, and Early+Middle+Late, so each step also supplies additional neural information. The qualitative examples are suggestive, but Middle-only, Late-only, or matched leave-one-stream-out measurements on the existing test set would better distinguish the proposed functional specialization from the general benefit of observing more regions.
Questions for authors
  1. Were stimulus-associated captions supplied in the runs behind Table 1 and the ablations? Please reconcile the evaluation materials with those runs and establish the results with text that is empty, stimulus-independent, or derived solely from fMRI.
  2. How are anatomical coordinates related to visual-field coordinates, and what spatial information remains after ROI averaging? Does the hierarchy advantage persist against neutral ROI-to-depth mixing and a capacity-matched generic mixer?
  3. Do quantitative leave-one-stream-out results support the proposed structural and semantic specializations, beyond the improvement from adding more neural information?
Reviewer_7VhAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Hi-Dream, a model for fMRI-to-image generation that organizes brain representations into three levels: early, middle, and late. It hierarchically injects the corresponding conditioning signals at different U-Net depths and denoising steps, aligning the image generation process more closely with brain representations. Experimental results demonstrate the model’s effectiveness, while ablation studies further validate the proposed design.

Critical strengths

The paper introduces a useful inductive bias for fMRI-to-image reconstruction by organizing ROI features according to the cortical hierarchy and injecting them into corresponding U-Net depths/denoising timesteps with scale-specific strengths.

Critical weaknesses
  1. The proposed method improves high-level semantic metrics but does not outperform the strongest baseline on some low-level metrics, including PixCorr and SSIM. This trade-off is acceptable; however, the paper would benefit from a more detailed analysis of why semantic improvements do not translate into better structural fidelity.

  2. The ROI-stream ablation reveals a similar trade-off: SSIM decreases from .421 with Early-only conditioning to .402 with Early+Middle conditioning, and only partially recovers to .407 with all three streams. This warrants further discussion, particularly given the statement that adding Middle ROIs improves both low-level and semantic metrics.

  3. Figure A1 does not convincingly establish the claimed correspondence between ROI streams and reconstruction properties. Although the appendix associates Early, Middle, and Late streams with layout, parts, and semantics, respectively, these distinctions are difficult to verify from the single example shown. Additional examples under controlled sampling conditions, together with quantitative analyses of structural and semantic fidelity, would strengthen this interpretation.

Questions for authors

What motivates the choice of three hierarchical levels in Hi-Dream? Can the framework be extended to accommodate additional levels, or generalized to support an arbitrary number of levels?

Reviewer_BwhbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a method to reconstruct seen images from fMRI scans with a visual hierarchy aware ControlNet. The hierarchy introduces an inductive bias that seems to increase reconstruction performance across metrics compared to RoI-agnostic methods.

Critical strengths

The paper splits the fmri voxels into a hierarchy according to known biological early, mid, late processing sequence and trains a controlnet that conditions the fmri to image pipeline with this known processing sequence prior. The motivation for this line of work is well grounded and it can likely lead to more explainable neural decoders that can pinpoint which RoI contributes to what features in the generated images.

Critical weaknesses

The supplied code doesn't seem to fully match the method mentioned in the paper. The NSDCondDataset class always fetches the coco caption of each nsd image and there is a prepare_text_embeddings function that encodes the text. This text embedding goes into both the instantiated controlnet and unet classes. This can possibly be an information leakage and contradicts the paper body that says "Unless otherwise stated, classifier-free guidance is disabled".

The paper’s core advantage is interpretability but the RoI ablations do not compare with established protocols as used by IC-GAN, BrainDiffuser and MindEye2. The RoI-optimal analysis used in these papers showed that even with RoI-agnostic training RoI awareness emerges. I would like to see such kinds of RoI level ablations with synthetic fMRI to see if there is controllability in the image. There is also a possibility of using cross sample RoI (early from one sample, late from another) to see if any coherent pattern emerges.

Currently the single qualitative example in Figure A1 shows a clock tower with minor variations in all RoI levels. It doesn’t support the claim that the RoI’s are making qualitative changes.

Questions for authors

Please re-check if the submitted code is correct and if there is an information leakage from text prompts going into the Unet and Controlnet. Are the current Tables 1-4 generated with caption conditioning and CFG present or off?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.