acceptodds
Under review as a conference paper at ICLR 2027

Wallbreaker: Breaking the Vision-Imagery Barrier through Cortical Hierarchy-Driven Neural Decoding

Abstract

Decoding mental imagery from fMRI is fundamentally challenging: models trained on visual perception degrade sharply when applied to imagery. We argue that this gap is not merely a signal-to-noise problem, but stems from a hierarchy-structured representation shift in the visual cortex. Vision is primarily driven by bottom-up sensory input, whereas imagery is shaped by top-down internal generation, systematically redistributing information across cortical levels. This induces two forms of hierarchy-dependent variability: a distribution-level shift in the balance of cortical information between vision and imagery, and a stimulus-level preference where different stimuli rely to different degrees on structural versus semantic information. We propose Wallbreaker, a framework that explicitly models cortical hierarchy-dependent variability during neural decoding. We first introduce hierarchy-driven representation perturbation, which simulates realistic hierarchy shifts during training by redistributing and perturbing representations across cortical levels. We further develop hierarchy-conditioned reconstruction, which infers a stimulus-wise hierarchy preference to dynamically balance semantic guidance and structural anchoring during reconstruction. Wallbreaker improves key imagery reconstruction metrics while maintaining competitive vision reconstruction performance, effectively narrowing the vision-to-imagery gap. More broadly, our results highlight cortical hierarchy as a fundamental inductive bias for bridging vision and imagery in neural decoding.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
fMRI, neural decoding, image generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LNnGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a novel, brain-inspired approach to improve performance of decoding imagined images, bringing it closer to decoded viewed images.

The model architecture suggested is based on a prior work (Mirage from Kneeland et. al, 2026) – three ridge-regression models per-subject, predicting low level image features, CLIP text features and CLIP image features, and using all 3 with a stable cascade generator.

Their approach to imagined stimulus utilizes the much wider dataset of viewed images (NSD) and prior neuroscience results showing the neural difference between the 2 tasks. Their approach is made of two main elements: • Training the model on augmented fMRI, augmented by 3 hierarchical brain levels. • Training another model that given the fMRI signal predicts the relative importance of each embedding level in the image generation (embedding levels - low level image features, CLIP Text and CLIP image).

The paper shows their model’s result on both imagined and viewed stimuli, showing the technique improves the score over both settings. They then test each part with various ablations, showing each has an effect.

Critical strengths

• The data augmentation circumvents the main challenge in the imagery task – the data shortage.

• Improvement on key metrics of the task.

Critical weaknesses

• Partial and unclear report of qualitative and quantitative results:

  1. On a test-set of 18 images, one should include all 18 visual reconstructions. The paper shows only 6 out of the 18 test reconstructions. Why?
  2. It is not clear from the paper which subject is being evaluated in the main paper, and how the results (both in Fig. 6 and in Table 1) were selected: Are these reported reconstructions from the mean fMRI of all 16 repeats per stimulus? Or are these the results of the best repeat for each stimulus? Please clarify.

• Unclear/Unjustified choice of fMRI augmentations:

  1. It is unclear nor justified why rotation of the projected sub-fMRI vector is a reasonable augmentation for imagined data. It seems to me that this would just shift fMRI activations from some brain voxels to other brain voxels.
  2. It seems from Figure 5a that the small final test dataset (18 test samples) was also used as a validating set for choosing the augmentation parameters. If so, this is not a valid practice, as it leads to overfitting and leakage of the test data into the training (the same set is afterward used for testing the method).

• Not reproduceable: The method is not reproduceable from the paper; many important details are missing. See detailed questions below.


A COMMENT ABOUT MY FINAL RATING:

I was debating between Weak Accept & Weak Reject, but I was forced to choose one of the two. If the rebuttal provides satisfactory answers to some of my concerns, I may raise my final score.

Questions for authors

• In the rebuttal, please provide all 18 visual reconstructions of the 18 test stimuli.

• Why is rotation of the projected sub-fMRI vector a reasonable augmentation? Doesn’t it just shift fMRI values from some brain voxels to other brain voxels? What’s the neuroscientific justification of this?

• How were the actual augmentation parameters selected for each of the hierarchy levels? It seems from Fig. 5a that the small final test dataset was also used as a validating set for these choices. If so, this is not a valid practice, as it leads to overfitting and leakage of the test data into the training.

• The preference model itself is under specified. What is the actual model? 3 Ridges? MLP?

• Subspace Perturbation: What is the chosen rank klk_l in the subspace projection of each brain level? How was it chosen? How many (and which) fMRI scans were used to compute these subspaces?

• The normalization map on the projections of π is never specified (just says “maps values to [0,1]”). How is the normalization done?

• How were the qualitative image results chosen?

• I didn’t understand figure 5b. Is this an intuitive infographic or actual measured results? Why aren’t the axes marked? How were the 2 axes computed from the 3-tuple simplex?

• What is the difference between “no control” and “fixed control”? (isn’t ‘no control’ just fixed control with coefficients of 1?) What are the fixed values? Additionally, “adaptive mapping” is defined only on the appendix, so is unclear given the main text alone.

• For the “concept” test stimuli – How were the metrics computed? From the image showing the concept word, or were the subjects asked verbally to imagine the concept?

• I believe that in Fig 5a the legend has an error – “Vision” should be replaced with “Imagery”.

10 comments

Thank you for your thoughtful and constructive feedback. We greatly appreciate your recognition of our approach to addressing imagery data scarcity and the improvements on key evaluation metrics. We also acknowledge that several experimental and methodological details require clarification, particularly the reconstruction protocol, augmentation parameter selection, and implementation of hierarchy-conditioned control.

We have incorporated all new results and suggested changes in our revised manuscript uploaded, with revisions highlighted in blue.

Q1. [Weakness #1-2, Question #1&7] Complete reconstruction results and clarification of the evaluation and selection protocol

We thank the reviewer for requesting the complete reconstruction results and clarification of our evaluation and selection protocol.

Complete reconstruction results and example selection. Figures 6 and 7 now jointly present all 18 test stimuli, with reconstructions from WALLBREAKER and the comparison methods. The original six examples were selected to illustrate the three stimulus categories, with two examples per category. For simple stimuli, we selected “+” and “×” because their intersecting contours provide more challenging cases than individual line segments. The additional figure includes the remaining 12 stimuli, allowing readers to assess reconstruction quality across the complete test set.

Participant and repeat handling. Following MIRAGE [15], the qualitative reconstructions in Figures 6 and 7 are shown for participant 1. Table 1 reports aggregate results across participants 1, 2, 5, and 7, with models trained and evaluated separately for each participant. For mental imagery, we average all 16 fMRI repetitions of each stimulus voxel-wise before decoding; we do not select the best individual repetition. Vision reconstructions similarly use the voxel-wise mean of all eight available repetitions.

Generation and quantitative evaluation. We follow the retrieval-pooling and repeated-reconstruction evaluation protocol of MIRAGE [15]. Each reconstruction run generates 16 candidate images from the same repeat-averaged fMRI input. One candidate is selected by cosine similarity to the fMRI-predicted retrieval embedding, without accessing the ground-truth target image. This generation-and-selection procedure is repeated 10 times, and quantitative metrics are averaged across the resulting reconstructions, stimuli, and participants. The qualitative figures illustrate individual reconstructions, Table 1 further reports the aggregate evaluation rather than scores computed only from the displayed examples.

Reference (numbering follows the submitted manuscript)

[15] Kneeland, R., Torrico Villanueva, C. K., Chen, T., Ojeda, J., Khanna, S., Xu, J., Scotti, P. S., & Naselaris, T. (2026). MIRAGE: Robust multi-modal architectures translate fMRI-to-image models from vision to mental imagery. PLOS Computational Biology, 22(5), e1014263. https://doi.org/10.1371/journal.pcbi.1014263

Q2. [Weakness #1, Question #2] Neuroscientific motivation and interpretation of subspace rotation

We thank the reviewer for requesting a clearer explanation of this operation. We now distinguish its mathematical effect from its neuroscientific motivation.

What the rotation changes. The operation perturbs the coordinates of an activity pattern in a low-rank representation within a cortical group. Mapping the perturbed representation back to voxel space changes the resulting voxel activations, but it is not a permutation of voxel locations or a literal transfer of neural activity between anatomical sites. It is a training augmentation that changes combinations of distributed activity components while retaining the cortical-group assignment.

Neuroscientific motivation. The most directly relevant evidence is Saha Roy et al. [31], already cited in our manuscript. Their analysis of perception-to-imagery transformations found that imagery and vision occupy differently oriented subspaces in early visual cortex, alongside reduced imagery dimensionality. Alignment and dimensionality were more similar in higher visual regions. Related geometric differences were observed in a second spatial-imagery dataset. These findings motivate allowing hierarchy-dependent changes in representational geometry rather than modeling imagery solely as uniformly weakened vision.

We nevertheless distinguish this motivation from an exact biological model. Our sampled rotations are not estimates of the transformation directions measured in [31]. In particular, rotating coordinates within a fixed PCA subspace does not necessarily reorient that entire subspace in voxel space. We use this operation as a simplified augmentation inspired by representational variability, rather than claiming that the brain performs the same mathematical operation.

Empirical evidence and additional controls. Our existing rotation-strength ablation provides evidence of utility: removing rotation reduces CLIP similarity from 0.583 to 0.566 and overall visual cortex correlation from 0.169 to 0.158 under the evaluated configuration. This supports including the operation, but does not independently establish its biological validity.

We conducted matched-strength global-noise and shuffled-hierarchy-schedule comparisons to assess the benefits of structured augmentation beyond generic regularization. The schedule-shuffling control specifically tests the assignment of attenuation schedules to cortical levels; it does not validate the sampled rotation directions.

Reference (numbering follows the submitted manu)

[31] Saha Roy, T., Breedlove, J., St-Yves, G., Kay, K., & Naselaris, T. (2025). A transformation from vision to imagery in the human brain. bioRxiv preprint. https://doi.org/10.1101/2025.09.02.672180

Q3. [Weakness #2, Question #3] Augmentation parameter selection and potential use of imagery test data

We thank the reviewer for raising this concern: The imagery data in Fig. 5(a) were used only for post-hoc distributional analysis, not to select augmentation parameters. The configuration was fixed before this analysis, and the reported distances were not used to select or revise it.

The hierarchy-dependent perturbation design was motivated by prior evidence of greater perception–imagery differences in early visual regions [31]. The numerical settings, including PCA rank 8 and rotation strength 0.35, were predefined implementation choices rather than parameters fitted to imagery data.

The augmentation subspaces were fitted exclusively to vision-training inputs. The separate PCA basis used in Fig. 5(a) includes imagery observations only to define the evaluation coordinates; it is not used for training or augmentation.

Q4. [Weakness #3, Question #4] Architecture and training of the hierarchy-preference model

We thank the reviewer for identifying this omission. The hierarchy-preference predictor is a single nine-input, three-output affine ridge-regression model, not an MLP. It is distinct from the three decoder branches that predict CLIP text, CLIP image, and VDVAE features.

Input features. For each of the early-, mid-, and high-level cortical groups, we compute three statistics: mean absolute activation, standard deviation, and root-mean-square activation. Concatenating them produces a nine-dimensional vector. These features are standardized using participant-specific vision-training means and population standard deviations; standard deviations below 10−610^{-6} are replaced by 11.

Training objective. The three-dimensional target tit_i is derived from normalized reference-decoder sensitivities to cortical-group ablation. With standardized input features ϕ(xi)\phi(x_i), the predictor is

t^(xi)=B⊤ϕ(xi)+b,\widehat t(x_i)=B^\top\phi(x_i)+b,

and is fitted using

LP=∑i∥B⊤ϕ(xi)+b−ti∥22+λP∥B∥F2,\mathcal{L}_{P}=\sum_i\lVert B^\top\phi(x_i)+b-t_i\rVert_2^2+\lambda_P\lVert B\rVert_F^2,

where λP=1\lambda_P=1 and the intercept is unregularized. The model has 27 regression weights and three intercepts, totaling 30 trainable parameters per participant.

Inference. Predicted outputs are clipped below at ϵ=10−6\epsilon=10^{-6} and normalized across the three groups:

pℓ(x)=max⁡(t^ℓ(x),ϵ)∑mmax⁡(t^m(x),ϵ).p_\ell(x)=\frac{\max(\widehat t_\ell(x),\epsilon)}{\sum_m\max(\widehat t_m(x),\epsilon)}.

The training-fitted standardization statistics and regression parameters are reused at inference. The resulting preference vector provides the input to the gating and reconstruction-control mappings. We have added the predictor architecture and objective explicitly to the method description and provide the sensitivity-target normalization in the implementation details.

Q5. [Question #5] PCA subspace ranks and fitting data

We use the same fixed PCA rank at all three cortical levels: rE=rM=rH=8r_E=r_M=r_H=8. PCA is fitted separately for each participant and cortical group using mean-centered, preference-gated vision-training inputs, before applying perturbations or adding augmented copies. The first eight right singular vectors define the perturbation subspace; the orthogonal residual is retained.

For each participant (1, 2, 5, and 7), fitting uses all 27,000 vision-training scans in the 40-session configuration, corresponding to 9,000 distinct NSD images presented three times each. Repeats are retained as separate samples. No imagery data, held-out shared1000 stimuli, or scans from other participants are used.

Rank 8 is a fixed implementation choice, not selected by an explained-variance threshold or a documented cross-validation search. Appendix F.4 additionally evaluates ranks 4 and 16, showing trade-offs across metrics rather than uniform superiority of rank 8.

Q6. [Question #6] Normalization of the preference projections

We thank the reviewer for identifying this missing definition. We normalize each linear projection using its analytic range over the preference simplex.

Since πℓ≥0\pi_\ell\geq0 and ∑ℓπℓ=1\sum_\ell\pi_\ell=1, a projection c⊤πc^\top\pi lies between min⁡(c)\min(c) and max⁡(c)\max(c). We define

Nc(c⊤π)=clip⁡(c⊤π−min⁡(c)max⁡(c)−min⁡(c),0,1).\mathcal{N}_c(c^\top\pi)=\operatorname{clip}\left(\frac{c^\top\pi-\min(c)}{\max(c)-\min(c)},0,1\right).

The bounds depend only on the predefined coefficients, not on training or test statistics. If the coefficient range is below 10−810^{-8}, the implementation returns zero. We have added the analytic-range normalization formula and its degenerate-range handling in Appendix B.

Q7. [Question #8] Interpretation and coordinate construction of Fig. 5(b)

We thank the reviewer for requesting clarification. Fig. 5(b) is a two-dimensional visualization of model-predicted hierarchy preferences, with stimulus thumbnails replacing scatter points. For stimulus i, its position is

(xi,yi)=(πH(xi),πE(xi)),(x_i,y_i)=(\pi_H(x_i),\pi_E(x_i)),

where the horizontal axis represents high-level preference and the vertical axis represents early-level preference.

The preferences are obtained from the per-stimulus early, mid, and high scores saved for participants. The axis labels indicate the preference components; numerical ticks were omitted for visual clarity, which made the coordinate interpretation insufficiently explicit. We have clarified in the Fig. 5(b) caption that thumbnail coordinates represent predicted high-level and early-level preferences on the horizontal and vertical axes, respectively.

Reviewer_1QdkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WALLBREAKER, a cortical hierarchy-aware framework for transferring fMRI-to-image decoders from visual perception to mental imagery. The main insight is that the perception-to-imagery gap reflects a structured shift in the relative contributions of different cortical hierarchy levels, therefore this work introduces two components: hierarchy-driven representation perturbation during training and hierarchy-conditioned reconstruction at inference time. Experiments on NSD and NSD-Imagery show improvements over several existing reconstruction methods, including MIRAGE, on selected reconstruction metrics.

Critical strengths
  • Generalization from visual perception to mental imagery remains an important challenge in neural decoding. Investigating the role of cortical hierarchy is a reasonable direction, particularly given the differences in neural processing between externally driven perception and internally generated imagery.

  • The proposed modifications do not require training a new generative backbone or substantially increasing model capacity.

Critical weaknesses
  • The central contribution is the claim that modeling cortical hierarchy-specific distribution shifts improves perception-to-imagery transfer. However, the proposed perturbation combines level-dependent attenuation with PCA-based rotation and compression. These operations may also work as generic regularization mechanisms that reduce dependence on unstable or noisy input features. In other words, the hierarchy-dependent Beta distributions and PCA-based rotations are predefined, without demonstrating that their magnitudes or directions correspond to actual vision-to-imagery transformations.

  • An important missing comparison is to shuffle the predicted hierarchy preferences across stimuli while keeping the control mechanism unchanged. This would directly test whether the correspondence between an individual stimulus and its estimated preference contributes to reconstruction performance.

  • The hierarchy preference is derived from changes in the reference decoder's output when individual cortical groups are ablated. This provides a measure of model sensitivity, but it is not necessarily equivalent to the contribution of a cortical region to the underlying neural representation. In particular, the resulting sensitivity can depend on voxel dimensionality, learned regression weights, correlations between cortical regions, and the effects of zeroing an entire ROI. The paper does not sufficiently disentangle these factors.

  • The Wasserstein analysis in Figure 5(a) is underspecified. The paper does not explain how the distances are computed or how response scale and voxel count are handled in comparisons across cortical levels. A clearer description of these details would help interpret the reported cross-level differences and their relation to the proposed hierarchy-based explanation of the vision–imagery gap.

Questions for authors
  • Can the authors provide the exact reference–student training objective and the complete preference-to-control mapping, including all numerical coefficients, normalization rules, and fixed-control values?
  • Were the encoder weights and perturbation configurations held identical across all Table 3 control variants? Please provide a controlled factorial ablation separating training-time hierarchy perturbation from inference-time hierarchy-conditioned control.
  • How does the proposed perturbation compare with global noise, random hierarchy assignments, and distribution-matched perturbations of equivalent strength? Such controls are important for establishing that the gains come specifically from cortical hierarchy modeling.
  • Can the authors validate the estimated hierarchy preferences against independent imagery-derived neural measurements, and evaluate shuffled or fixed preference assignments to determine whether the adaptive policy genuinely benefits from stimulus-specific information?
9 comments

Thank you for your thoughtful and detailed feedback. We sincerely appreciate your recognition of the importance of perception-to-imagery transfer and the lightweight nature of our approach. These identify important opportunities to strengthen our methodological description and distinguish hierarchy-specific benefits from generic regularization. New results and clarifications are included in the revised manuscript and highlighted in blue.

Q1. [Weaknesses #1, Question #3] Generic regularization versus hierarchy-specific modeling, and correspondence to actual vision-to-imagery transformations

Overview. To address these concerns, we:

- 1. Explain the neuroscientific evidence motivating the perturbation structure and distinguish it from the choice of numerical parameters (Appendix G); - 2. Provide the numerical distributional-alignment results underlying Fig. 5(a) (Appendix G); - 3. Report matched-energy Gaussian-noise and shuffled-schedule controls to assess the contribution of hierarchy assignment (Appendix F.1,).

The additional controls support the benefit of matching attenuation schedules to cortical levels, while the distributional analysis shows improved within-group alignment with imagery representations under the evaluated metric.

1. Neuroscientific evidences motivate our perturbation structure, yet do not pre-define our implementations.

Our design was inspired by empirical findings cited in the manu:

  • Dijkstra et al. [8] found increased bottom-up coupling during perception relative to baseline and increased top-down coupling during both perception and imagery, with a substantially stronger increase during imagery.
  • Horikawa and Kamitani [11] reported temporal patterns consistent with top-down recruitment during imagery: prediction peaks in higher-level representations tended to precede those in lower-level representations.
  • Saha Roy et al. [31] found that early visual imagery subspaces had, on average, fewer than half as many dimensions as vision subspaces, with dimensionality closer to parity in higher regions. Vision and imagery also occupied differently oriented subspaces in early cortex, with alignment increasing toward higher regions. Similar differences were observed in a second, spatial-imagery dataset.

These findings motivate hierarchy-dependent rather than uniform perturbations: stronger perturbations of lower-level representations, more conservative perturbations of higher-level representations, and changes in representational geometry through compression and rotation.

2. Empirical alignment with measured imagery representations.

Beyond the above neuroscientific motivation, Fig. 5(a) evaluates whether the combined perturbation moves vision representations closer to measured imagery representations under the Wasserstein metric. We provide more explicit numerical results underlying the figure.

Table I. Wasserstein distance to imagery before and after the combined perturbation.

CorticalgroupOriginalvisionPerturbedvisionRelativereduction
Allvisualvoxels3.605±0.3072.488±0.13731.0%
Early2.076±0.1791.781±0.16514.2%
Mid1.875±0.3101.502±0.21319.9%
High1.489±0.1191.074±0.02127.9%

The combined perturbation reduces the mean distance for the full visual representation by 31.0%, with reductions also observed in each cortical group. Thus, the perturbation is associated with improved distributional alignment.

3. Controls separating hierarchy-specific effects from generic regularization.

We compared the proposed perturbation with (i) no perturbation, (ii) global Gaussian noise, and (iii) shuffled hierarchy schedules. Its comparison with the proposed schedule tests the contribution of matching attenuation schedules to cortical hierarchy.

Table II. Comparison of training perturbations.

TrainingperturbationSSIM↑Inception↑CLIP↑VisualCortex↑
Noperturbation0.3930.5230.5700.153
GlobalGaussiannoise0.3910.5150.5740.151
Shuffledhierarchyschedule0.3960.5300.5770.156
Proposedhierarchyschedule0.4010.5540.5830.169

These support an additional contribution from the correspondence between cortical levels and attenuation schedules, beyond the benefits retained under shuffled assignments.

References (numbering follows the submitted manu)

[8] Dijkstra, N., Zeidman, P., Ondobaka, S., van Gerven, M. A. J., & Friston, K. (2017). Distinct top-down and bottom-up brain connectivity during visual perception and imagery. Scientific Reports, 7, 5677.

[11] Horikawa, T., & Kamitani, Y. (2017). Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 8, 15037.

[31] Saha Roy, T., Breedlove, J., St-Yves, G., Kay, K., & Naselaris, T. (2025). A transformation from vision to imagery in the human brain. bioRxiv preprint. https://doi.org/10.1101/2025.09.02.672180

Q2. [Weaknesses #2, Question #4] Does hierarchy-conditioned reconstruction benefit from stimulus-specific preference assignments?

We have conducted additional experiments below, and have added the controlled mean-, shuffled-, and predicted-preference comparisons, together with paired performance differences, in Appendix F.2 (Tables 15–16).

Controlled preference-assignment experiments. We compared three generation-side preference assignments while keeping the trained encoder, preference predictor, input-gating policy, and full-model preference-to-control mapping unchanged:

  1. Mean preference: a single participant-specific preference vector obtained by averaging generation-side predictions over vision-training samples, producing sample-independent generation coefficients.
  2. Shuffled preference: complete predicted preference vectors permuted across test stimuli within each participant. The early-, middle-, and high-level components remained together, preserving the preference distribution while disrupting stimulus correspondence.
  3. Predicted preference: each stimulus’s own predicted preference vector.

For each stimulus, we held the gated input and decoded representations fixed and varied only the preference supplied to the generation-control mapping. This isolates generation-side assignment from changes to the decoder input.

Table III. Generation-side preference-assignment comparison.

MetricMean preferenceShuffled preferencePredicted preference
PixCorr ↑0.1020.1010.109
SSIM ↑0.3970.3950.401
AlexNet(2) ↑0.6130.6230.648
AlexNet(5) ↑0.6030.6120.620
Inception ↑0.5480.5520.560
CLIP ↑0.5700.5770.583
Early visual cortex correlation ↑0.1800.1910.193
Higher visual cortex correlation ↑0.1380.1370.143
Overall visual cortex correlation ↑0.1560.1580.169

Table IV. Paired performance differences (95% CIs).

MetricPredicted − Mean95% CIPredicted − Shuffled95% CI
PixCorr ↑+0.007[−0.00074, +0.01111]+0.008[+0.00007, +0.00944]
SSIM ↑+0.004[+0.00121, +0.00797]+0.006[+0.00124, +0.00746]
AlexNet(2) ↑+0.035[+0.01667, +0.04167]+0.025[+0.00716, +0.02554]
AlexNet(5) ↑+0.017[+0.00189, +0.04356]+0.008[+0.00394, +0.02008]
Inception ↑+0.012[+0.00758, +0.03864]+0.008[+0.00053, +0.02159]
CLIP ↑+0.013[−0.01269, +0.02386]+0.006[+0.00135, +0.01693]
Early visual cortex correlation ↑+0.013[−0.00006, +0.03110]+0.002[−0.00049, +0.00685]
Higher visual cortex correlation ↑+0.005[+0.00155, +0.01268]+0.006[+0.00309, +0.00776]
Overall visual cortex correlation ↑+0.013[+0.00037, +0.01580]+0.011[+0.00210, +0.01782]

Results and interpretation. Predicted preferences achieve higher mean performance than both controls across all nine metrics. Compared with shuffled preferences, the gains include +0.008 in PixCorr, +0.025 in AlexNet(2), and +0.011 in overall visual cortex correlation. The reported metric-wise 95% confidence intervals exclude zero for eight of the nine comparisons; the exception is early visual cortex correlation, where the improvement is small (+0.002). Compared with mean preferences, the intervals exclude zero for six metrics, while those for PixCorr, CLIP similarity, and early visual cortex correlation include zero.

The shuffled-preference comparison directly supports the contribution of stimulus–preference correspondence: preserving the preference distribution while disrupting its assignment reduces reconstruction performance. The mean-preference comparison further supports the benefit of stimulus-dependent generation controls over a constant participant-level assignment, although the evidence varies across metrics. These results complement the held-out preference-prediction evaluation by testing whether the predicted preferences are useful for reconstruction, rather than only whether sensitivity-derived targets are predictable.

Q3. [Weaknesses #3, Question #4] Interpretation and robustness of hierarchy preferences derived from ROI-ablation sensitivity

Overview. To clarify the interpretation and assess the robustness of hierarchy preferences, we:

- 1. Define the measure as decoder-specific reliance and explain its dependence on the learned decoding model (Appendix B); - 2. Summarize held-out preference prediction and its association with reconstruction gains (Appendix E.1), complemented by controlled utility tests (Appendix F.2); - 3. Test robustness to voxel-count matching and an alternative ablation operator (Appendices E.2–E.3); - 4. Evaluate associations with independent neural target-identification measurements (Appendix E.4).

These show strong consistency under voxel-count matching, partial robustness to the ablation operator, and positive average associations with independent neural measurements, supporting the usefulness of the preferences as a decoder-specific measure.

1. Hierarchy preference measures decoder reliance, not model-independent neural contribution.

As stated in Section 3.1, hierarchy preference is an “operational estimate of hierarchy dependence”: the change in reference-decoder output after ablating a cortical group. It estimates which groups the decoder relies on for each input to guide reconstruction.

Dependence on learned regression weights is intrinsic to this measure. For a linear decoder, ablating group ll changes the output by WlxlW_lx_l; removing dependence on WlW_l would redefine the quantity rather than eliminate a confound. Cross-group correlations likewise shape the learned decoding solution and information redundancy.

2. Existing evidence supports predictability and utility of the operational targets.

Our held-out evaluation measures whether the preference estimator predicts sensitivity-derived targets on samples excluded from its training:

Table V. Held-out prediction of sensitivity-derived hierarchy preferences.

MetricPerformance
Pearsoncorrelation↑0.5802
Spearmancorrelation↑0.6026
Meanabsoluteerror↓0.0309
R²↑0.5307

These support generalization to held-out sensitivity targets. The associations between preference-prediction accuracy and reconstruction gains provide complementary evidence.

3. Robustness to cortical-group size.

We randomly match voxel counts across cortical groups within each participant, leaving other voxels unchanged, and refit the reference decoder for each subsample. We then compare recomputed preferences with the original profiles on the same held-out imagery stimuli.

Table VI. Robustness to voxel-count matching.

EvaluationSpearman corr.Dominant-level agreement
Early0.950±0.025N/A
Mid0.927±0.038N/A
High0.944±0.013N/A
Full profileN/A85.42%±7.84%

High correlations and agreement in the highest-preference group suggest that unequal cortical-group sizes are not the primary driver of the estimated preferences.

4. Robustness to the ablation operator. We recomputed preferences using 20 replacement patterns from other vision-training stimuli within each participant, preserving within-group voxel patterns while fixing the decoder and other groups. Output changes were averaged before normalization, with both operators using the same 1,024 vision-training samples for normalization statistics.

Table VII. Robustness to the ablation operator.

MeasureMean±SD
Early—Spearmanρ0.786±0.205
Mid—Spearmanρ0.638±0.183
High—Spearmanρ0.800±0.149
Dominant-level agreement66.67%±11.11%

Correlations compare replacement-based and zero-ablation preferences across 18 held-out imagery stimuli per participant; dominant-level agreement measures whether the highest-preference group is unchanged. These results support partial robustness to the ablation operator, while retaining the interpretation of preference as decoder reliance under the specified intervention.

5. Association with independent imagery-derived neural measurements.

We evaluated ROI-wise vision–imagery target identification. Imagery repeats were split into 2 independent subsets: 8 for preference prediction and 8 for neural evaluation. Within each cortical group, the identification margin was the correlation with the correct vision template minus the mean correlation with the other 11 templates. This measure used fMRI responses without preference-based gating, decoder-ablation outputs, or generated images.

Table VIII. Association with vision–imagery target-identification margins.

Cortical groupSpearman correlation with predicted preference
Early0.644 ± 0.131
Mid0.554 ± 0.238
High0.635 ± 0.284

This complements the earlier imagery-discriminability analysis, which showed no consistent association. Neither analysis was used to train the preference estimator or select reconstruction-control parameters.

Reviewer_GydPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study addresses the problem of transferring fMRI-based image reconstruction models trained on visual perception to mental imagery. The authors identify a potential source of the performance gap in the different cortical processing pathways involved in vision (primarily bottom-up) and imagery (primarily top-down), leading to differences in the engagement of early, mid-level, and high-level visual regions. They further highlight stimulus-dependent variability in the relative importance of structural and semantic representations.

To address those challenges the authors propose WALLBREAKER, introducing two main components: (1) hierarchy-driven representation perturbation, which simulates distribution shifts across cortical levels as well as representational changes within each level during training, and (2) hierarchy-conditioned reconstruction, which estimates stimulus-specific hierarchy preferences to adaptively balance semantic and structural guidance during image generation. The proposed approach achieves improvements on several mental imagery reconstruction metrics while maintaining competitive performance on visual perception reconstruction.

Critical strengths
  • Well motivated problem formulation and intuitive methodological contributions. The paper clearly identifies the gap between vision and imagery decoding and proposes a neuroscience-grounded explanation based on differences in cortical hierarchy engagement. The distinction between distribution-level shifts and stimulus-specific hierarchy preferences provides a coherent motivation for the proposed method. Both hierarchy-driven perturbation and adaptive reconstruction follow naturally from this formulation and constitute relatively lightweight extensions to existing approaches.

  • Outstanding presentation and good positioning within related work. The paper is very well written, with a clear introduction to the field and an effective discussion of existing vision-to-imagery transfer methods. The authors clearly position their contributions relative to prior work, identifying limitations that motivate their approach. The experimental results are presented clearly, with useful component ablations and participant-level analyses.

Critical weaknesses
  • Relatively modest and inconsistent improvements over existing methods. Although the proposed method achieves the best results on several metrics, improvements over the strongest baseline MIRAGE are generally small. For instance, imagery PixCorr increases from 0.104 to 0.109, CLIP similarity from 0.575 to 0.583, and overall visual cortex correlation from 0.168 to 0.169. While the supplementary participant-level analyses provide evidence of consistent improvements, the practical significance of these gains remains unclear, particularly given the additional methodological components. Moreover, the improvements are inconsistent across metric categories, with negligible gains or regressions in some cases, including early visual cortex correlation and AlexNet(5) accuracy. The hierarchy-conditioned reconstruction ablation also shows that fixed conditioning outperforms the full adaptive approach on certain semantic metrics, raising questions about the benefits of stimulus-specific hierarchy conditioning.

  • Insufficient controls to establish the importance of hierarchy-specific perturbations. Although the ablations demonstrate the contributions of hierarchy degradation and subspace perturbation, it remains unclear whether these improvements specifically arise from the proposed cortical hierarchy modeling or from the general regularization effect of the perturbations.

Questions for authors
  • While the participant-level bootstrap analyses provide evidence of consistent improvements on several metrics, the absolute gains over MIRAGE remain relatively small. How do the authors interpret the practical significance of these improvements, particularly in terms of perceptually meaningful reconstruction quality?

  • Have the authors evaluated generic perturbations (augmentations) or alternative hierarchy-dependent perturbation schedules with comparable strength? How much of the improvement can be attributed specifically to modeling the hypothesized cortical hierarchy shift versus the impact of augmentation or regularization?

  • Not sure if line 242 contains a typo "text-time approximation" - should that be "test-time approximation" ?

5 comments

Thank you for the thoughtful and constructive review. We greatly appreciate your positive assessment of our well motivate problem formulation, distinct, neuroscience-grounded method, and outstanding presentation. Below, we address the concerns regarding 1) the practical significance of the improvements and 2) the specific contribution of cortical hierarchy modeling. New results and clarifications are included in the revised manuscript and highlighted in blue.

Q1. [Weakness #1, Question #1] Interpreting the practical significance of improvements and benefits of adaptive conditioning

Overview. To address these concerns, we:

- 1. Provide complementary bidirectional retrieval results (Appendix D.2); - 2. Clarify the interpretation of the original control ablation; - 3. Report a controlled comparison of predicted, shuffled, and mean preference assignments (Appendix F.2); - 4. Present reconstructions for all 18 imagery stimuli to illustrate both improvements and remaining limitations (Appendix H).

New analyses show improved target identification and support the contribution of stimulus-specific conditioning, while clarifying the mixed results across reconstruction metrics.

1. Complementary evidence from target retrieval. We evaluated bidirectional retrieval using decoded CLIP image representations, with the same candidate gallery and evaluation protocol for both methods:

Table I. Bidirectional Cross-Modal Retrieval Evaluation.

DirectionMethodR@1↑MRR↑MeanRank↓
fMRI→ImageMIRAGE0.5830.7741.542
fMRI→ImageWALLBREAKER0.7080.8401.375
Image→fMRIMIRAGE0.2960.5752.336
Image→fMRIWALLBREAKER0.5680.7481.651

For fMRI-to-image retrieval, the correct target is ranked first in 70.8% of cases, compared with 58.3% for MIRAGE—an increase of 12.5% points. Reverse retrieval improves by 27.2% points, with corresponding improvements in MRR and mean rank. Although retrieval does not directly measure human-perceived reconstruction quality, these provide complementary evidence of our decoding quality.

2. Mixed metrics and clarification of the control ablation. We agree that the gain in overall visual cortex correlation, and the regressions on imagery AlexNet(5) and early visual cortex correlation, are small compared to other measures. Therefore, we did not claim uniform improvement across reconstruction measures.

We further clarify that “Fixed Control” in Table 3 fixes the baseline mapping parameters, not the realized generation coefficients. Both Fixed Control and the full method use stimulus-dependent preferences. These do not isolate the value of stimulus-specific preference assignments.

3. Direct evaluation of stimulus-specific conditioning. We compared correctly assigned, shuffled, and training-mean preferences while holding others constant. The mean condition used a participant-specific preference vector averaged over vision-training samples. The shuffled condition permuted complete preference vectors across test stimuli within each participant, preserving their distribution while disrupting stimulus correspondence.

Table II. Effect of generation-side preference assignment on mental imagery reconstruction.

MetricMeanpreferenceShuffledpreferencePredictedpreference
PixCorr↑0.1020.1010.109
SSIM↑0.3970.3950.401
AlexNet(2)↑0.6130.6230.648
AlexNet(5)↑0.6030.6120.620
Inception↑0.5480.5520.560
CLIP↑0.5700.5770.583
Earlyvisualcortexcorrelation↑0.1800.1910.193
Highervisualcortexcorrelation↑0.1380.1370.143
Overallvisualcortexcorrelation↑0.1560.1580.169

Table III. Paired performance differences (95% CIs).

MetricPredicted−Mean95%CIPredicted−Shuffled95%CI
PixCorr↑+0.007[−0.00074,+0.01111]+0.008[+0.00007,+0.00944]
SSIM↑+0.004[+0.00121,+0.00797]+0.006[+0.00124,+0.00746]
AlexNet(2)↑+0.035[+0.01667,+0.04167]+0.025[+0.00716,+0.02554]
AlexNet(5)↑+0.017[+0.00189,+0.04356]+0.008[+0.00394,+0.02008]
Inception↑+0.012[+0.00758,+0.03864]+0.008[+0.00053,+0.02159]
CLIP↑+0.013[−0.01269,+0.02386]+0.006[+0.00135,+0.01693]
Earlyvisualcortexcorrelation↑+0.013[−0.00006,+0.03110]+0.002[−0.00049,+0.00685]
Highervisualcortexcorrelation↑+0.005[+0.00155,+0.01268]+0.006[+0.00309,+0.00776]
Overallvisualcortexcorrelation↑+0.013[+0.00037,+0.01580]+0.011[+0.00210,+0.01782]

Correctly assigned preferences achieved higher mean performance than both controls across all nine metrics. These support the contribution of stimulus–preference correspondence to reconstruction performance, with benefits that vary across metrics.

4. Qualitative reconstruction evidence. Figures 6 and 7 in the revised manuscript provide comparisons for all 18 imagery stimuli. These suggest improved overall reconstruction quality with WALLBREAKER.

Q2. [Weakness #2, Question #2] Evaluating the contribution of hierarchy-specific perturbations beyond generic regularization

To address these concerns, we conducted a controlled comparison of four training conditions (Appendix F.1):

  1. No perturbation: training without representation perturbation.
  2. Global Gaussian noise: adding unstructured Gaussian noise with matched overall perturbation energy.
  3. Shuffled hierarchy schedule: retaining the cortical groups and their PCA subspaces, but permuting the attenuation schedules assigned to early-, middle-, and high-level groups.
  4. Proposed hierarchy schedule: applying the intended correspondence between cortical levels and attenuation schedules.

The shuffled and proposed conditions used identical subspace-perturbation settings, so that their comparison specifically tests the contribution of the attenuation schedule’s correspondence to cortical hierarchy. All conditions used the same training data, training budget, augmentation sample count, fixed inference-control settings, and paired generation seeds.

We define the relative perturbation energy as:

Eperturb=Ex∼Dtrain[∥x~−x∥22∥x∥22+ϵ]E_{\mathrm{perturb}}=\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{train}}}\left[\frac{\lVert\widetilde{x}-x\rVert_2^2}{\lVert x\rVert_2^2+\epsilon}\right]

where xx and x~\widetilde{x} denote the original and perturbed representations. Perturbation strength is matched using training-data statistics only, without selecting configurations based on imagery test performance.

Table IV. Comparison of training perturbations at matched overall perturbation energy.

Training perturbationRelative perturbation energySSIM ↑Inception ↑CLIP ↑Visual Cortex ↑
No perturbation0.0000.3930.5230.5700.153
Global Gaussian noise0.3000.3910.5150.5740.151
Shuffled hierarchy schedule0.3000.3960.5300.5770.156
Proposed hierarchy schedule0.3000.4010.5540.5830.169

Global Gaussian noise slightly improves CLIP similarity over no perturbation but reduces SSIM, Inception, and visual cortex correlation. This result indicates that unstructured augmentation at the tested strength does not reproduce the proposed method’s gains. The shuffled schedule improves all four metrics over no perturbation, suggesting that some benefits remain even without the intended hierarchy assignment. Nevertheless, the proposed schedule further improves SSIM by 0.005, Inception by 0.024, CLIP similarity by 0.006, and visual cortex correlation by 0.013 over the shuffled condition.

These results support an additional contribution from the correspondence between cortical levels and attenuation schedules, beyond the benefits retained under shuffled assignments. This comparison specifically evaluates the hierarchy assignment of attenuation schedules; it does not independently validate the biological correspondence of the PCA perturbation directions.

Q3. [Question #3] Minor: typo

Thank you for catching this typo. The intended term is “test-time approximation”. We have corrected it in the revised manuscript.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the rebuttal period comes to an end, we hope to make the best use of the remaining time for further discussion. We have addressed all comments and concerns raised in your review with utmost care, and hope that our rebuttal and revisions have met your expectations.

We look forward to further discussion. Thank you again for your time and effort!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We greatly appreciate the valuable and detailed feedback from all reviewers to help further strengthen our work. In the updated manuscript PDF, we have incorporated new experimental results, additional clarifications, and suggested changes, revisions are highlighted in blue.

We summarize these changes as below:

  • Additional evaluation of decoding quality [GydP, LNnG]. We add bidirectional fMRI–image retrieval results to assess target identification before image generation (Appendix D.2), and provide qualitative comparisons for all 18 imagery stimuli (Appendix H; Figures 6–7).
  • Controls for hierarchy-specific perturbations [GydP, 1Qdk, LNnG]. We add comparisons with global Gaussian noise and shuffled hierarchy schedules at matched overall perturbation energy to assess the contribution of assigning attenuation schedules to the intended cortical levels (Appendix F.1).
  • Direct tests of stimulus-specific conditioning [GydP, 1Qdk]. We compare predicted, shuffled, and training-mean preferences while holding the encoder, input gating, and generation mapping constant, and report paired performance differences with confidence intervals (Appendix F.2). A separate factorial ablation distinguishes training-time perturbation from generation-time conditioning (Appendix F.3).
  • Robustness and independent validation of hierarchy preferences [1Qdk]. We consolidate held-out preference prediction, voxel-count-matching and alternative-ablation analyses, and associations with independent neural measurements in Appendices E.1–E.4. We clarify that hierarchy preferences measure decoder-specific reliance.
  • More complete methodological details [1Qdk, LNnG]. We specify the training objectives, preference-predictor architecture, sensitivity normalization, and preference-to-control mappings in Appendix B. We document PCA fitting data, rank, and augmentation settings in Appendix C, with parameter sensitivity analyses in Appendix F.4.
  • Clearer evaluation and figure descriptions [GydP, 1Qdk, LNnG]. We explain participant selection, fMRI repeat averaging, candidate-image selection, quantitative aggregation, and the treatment of conceptual stimuli in Appendix C. We clarify the Wasserstein computation and its post-hoc role in Appendix G, explain the coordinates and legend in Figure 5, and correct “text-time approximation” to “test-time approximation” in Section 3.2.
  • More precise interpretation of the evidence [GydP, 1Qdk, LNnG]. We clarify the scope of the distributional findings in Section 4.2 and Appendix G, assess hierarchy-specific benefits through the controls in Appendix F.1, and discuss metric-dependent benefits of generation-side conditioning in Appendices F.2–F.3.
Area_Chair_sYMEOpenReview ↗

Gentle Reminder: ICLR Rebuttal Discussion Deadline – November 18

Dear Reviewers,

Thank you very much for your time, thoughtful reviews, and valuable contributions to the evaluation process.

The authors have carefully addressed your comments and concerns, and the discussion period will conclude on November 18. We would greatly appreciate it if you could take a few moments to review their responses and results, engage in the discussion, and indicate whether their clarifications or additional evidence affect your assessment.

Your continued engagement is essential to ensuring a rigorous, constructive, and fair evaluation process.

Thank you again for your time, expertise, and dedication!

Best regards, Area Chair

AuthorsOpenReview ↗

Summary of Discussion

We thank AC and all reviewers for your engagement and constructive feedback that have helped substantially strengthen our paper. Below, we summarize key results from the author-reviewer discussion phase for your reference.

Scores updated (before --> after):

  • GydP: 3 (weak accept) --> 3 (weak accept)
  • 1Qdk: 2 (weak reject) --> 3 (weak accept)
  • LNnG: 2 (weak reject) --> between 2 and 3 (weak accept)

Strengths recognized:

  • Motivation and novelty. A novel, well-motivated, neuroscience-inspired modeling perspective on perception-to-imagery transfer. [GydP, 1Qdk]
  • Design. Lightweight methodological extensions without a new generative backbone or substantial additional capacity. [GydP, 1Qdk]
  • Effectiveness. Augmentation addressing limited imagery data and improvements on key decoding metrics. [LNnG]
  • Presentation. Outstanding presentation and good positioning with related work. [GydP]

Addressed concerns acknowledged as resolved during discussion:

  • Methodological controls (Appendices F.1–F.2). Matched-strength perturbation comparisons and preference-assignment experiments were acknowledged to substantially address the main methodological concerns and strengthen evidence for stimulus-specific conditioning. [GydP]
  • Complementary evaluation (Appendix D.2). Bidirectional retrieval was acknowledged to strengthen the empirical results. [GydP]
  • Methodological analysis Appendices E.1–E.4 and F.3 We additionally provide preference-robustness analyses and a factorial ablation in response to methodological questions. [1Qdk]
  • Visualization and methodological clarity (Appendix B, Appendix H, and Figure 5(a)). Explicitly acknowledged resolutions include all 18 imagery reconstructions, the preference-predictor specification, projection normalization, and the corrected legend and post-hoc interpretation of Fig. 5(a). Further additions document the evaluation and selection protocols and provide remaining-participant and all-run visualizations in Appendix C.1 and Figures 8–9. [LNnG]
  • Development protocol (Section 4.3, Table 1 caption, and Appendix F.4) We disclose participant 1’s development role and separately report participants 2, 5, and 7. The similar gains on these participants were acknowledged as suggesting that development did not substantially inflate the results. The comparison is now included in the main text, alongside retrospective selection analysis in Appendix F.4. [LNnG]

Further, the Abstract, Section 4.3, and Conclusion now (i) emphasize our central contribution: a novel, neuroscience-motivated modeling perspective combining hierarchy-informed perturbation with stimulus-specific reconstruction control, and (ii) soften claims to align explicitly with the observed performance. [GydP, LNnG]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.