acceptodds
Under review as a conference paper at ICLR 2027

Bridging Vision and Imagery: Latent Functional Alignment for visual imagery decoding from fMRI Data

Abstract

Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery–perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Decoding, Cognitive Science, Artificial Intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JY5KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts a frozen DynaDiff perception decoder to mental imagery using a voxel-space adapter supervised by latent reconstruction and contrastive losses. Retrieved perception responses provide training targets. Experiments on four NSD-Imagery participants assess reconstruction, stimulus specificity, and decoding from additional cortical regions.

Critical strengths

Adapting a frozen perception decoder with retrieved targets is a practical approach to scarce imagery data, evaluated quantitatively across participants. Tables 2–3 and Figure 5 show why aggregate similarity can reward generic reconstructions, motivating a useful stimulus-specific evaluation.

Critical weaknesses
  1. The experiment does not distinguish alignment from recognition of known stimuli. Training and testing use the same six stimuli per set, while Equation 1 explicitly fits stimulus-level target means and separates their labels. A six-way neural classifier followed by the corresponding retrieved prototype is therefore an important missing control: it would test whether the gains require functional alignment beyond supervised stimulus identification within the stated evaluation setting.
  2. The comparison does not isolate the proposed latent objective. Relative to voxel-space ridge, LFA changes model capacity, temporal processing, and supervision simultaneously. The reported linear/nonlinear ablation does not resolve this issue: the linear latent model is not better than ridge on AlexNet-5 or CLIP, while a matched nonlinear voxel-supervised adapter and an isolated contrastive-loss comparison are absent, leaving the central design motivation insufficiently tested.
  3. The evidence for imagery-specific regional information remains ambiguous. The task associates each stimulus with a fixed visual cue, so stimulus discrimination may include cue recognition or associated semantic retrieval. The regional results in Section 3.2 do not separate these sources from mental imagery; despite the acknowledged ROI overlap and baseline limitations, successful reconstruction alone cannot establish the functional interpretation offered for the additional cortical regions.
  4. The claimed methodological advantage depends on which semantic property is measured. LFA trails ridge on AlexNet-5 and SwAV in Table 1 and on conceptual target similarity in Table 2, while improving specificity in Table 3. This distinction is informative, but the specificity gain should be assessed against matched supervised controls before it supports the broader claim that latent alignment yields better imagery reconstructions.
  5. Statistical support is limited by dependence across repeated partitions. Section 3.1 tests ten overlapping partitions of the same trials, with held-out responses averaged into six reconstructions per set. These repetitions characterize sensitivity to partitioning rather than independent neural samples; inference respecting shared trials and participants is needed to determine how strongly the reported significance supports the method and regional comparisons.
Questions for authors
  1. How does LFA compare with a six-way neural classifier followed by its retrieved prototype on the same held-out trials and specificity metrics? What existing control distinguishes imagery-related information from the fixed cue?
  2. With nonlinear capacity and temporal processing matched, does latent supervision outperform voxel-space supervision, and what is the contribution of the contrastive term?
  3. Was K = 1000 selected exclusively on validation data? Do the principal conclusions persist under inference that accounts for reused trials and the participant structure?
4 comments
AuthorsOpenReview ↗

Response to Reviewer JY5K

We thank the reviewer for their careful and constructive assessment of our work. The comments brought additional control experiments and helped us refine the scope and precision of our claims. The main revisions and responses are summarized below:

W1

As noted by the reviewer and as we mentioned in the limitations of our article, our findings are related to the generalization of new trials for a fixed set of stimuli, instead of generalizing to unseen concepts and stimuli. Tab. 8 in Appendix C of the updated pdf compares LFA with six-way classifiers followed by the retrieved prototype, that we report in the following:

MethodTop-1 (%)Image spec.Text spec.CLIPIncepAlexNet-5
Oracle1000.1170.07180.693.787.8
LFA56.30.0660.03767.478.867.2
Logistic regression (6-way)42.50.0400.01958.175.363.3
MLP (6-way)31.60.0230.01353.970.858.2
DynaDiff baseline26.70.0280.01953.367.055.2

Comparing our LFA both with a multinomial logistic regression and with a MLP but with a cross-entropy loss, our LFA method outperforms both of them, regarding both semantic metrics and image and text specificity. LFA identifies the imagined stimulus more accurately than tuned six-way classifiers (top-1 56.3% vs. 42.5% for logistic regression with nested cross-validation and 31.6% for the MLP; chance 16.7%); its reconstructions score above classifier-plus-prototype approaches, while remaining well below the oracle that always receives the correct prototype (image specificity 0.066 vs. 0.117).

When a whole stimulus is excluded from training, performance falls to chance; we now state this explicitly in Sec. 4.1 (lines 492-498) and restrict our claim to generalization across trials within a fixed stimulus set.

W2

In order to further assess the contribution of our LFA respect to the Ridge baseline, we investigated both the contribution of the latent supervision respect to a matched voxel-supervised adapter, isolating the contribution of the contrastive-loss term. Results are reported in Tab. 7 in Appendix C of the uploaded version; they are also reported in the following table:

MethodImage spec.Text spec.CLIPIncepAlexNet-5
Latent (LFA)0.0660.03767.478.867.2
Latent, SupCon only0.0640.03567.277.865.5
Latent, MSE only0.0330.02062.271.669.4
Voxel supervision0.0390.02457.371.058.2
DynaDiff Baseline0.0280.01953.367.055.2

It clearly emerges the main contributions of both the latent supervision and the contrastive-loss term, that drives most of the performance reported in the main text. The MSE term alone is competitive on AlexNet-5 but halves specificity.

On the other hand, the same architecture trained with a voxel supervision improves performance over the DynaDiff baseline but it loses on both specificity and high-level accuracy with respect the latent supervision (Our LFA). Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.

W3

We acknowledge this observation of the reviewer. We added this consideration in the limitations of the updated file (l. 504-509). Disentangling the contribution of the cue letter from the related imagery activity is not possible for the experimental setup itself; as described in the Appendix A.1, a cue letter is displayed for each trial of each run; however, as possible evidence, we used the mismatched stimulus-letter vision trials from NSD-Imagery in order to evaluate on them performance of DynaDiff model, i.e, without our Alignment module; results are reported in next Table:

TaskPixCorr ↑SSIM ↑Alex2 ↑Alex5 ↑Incep ↑CLIP ↑Eff ↓SwAV ↓
Simple Stimuli (A), mismatch−.000±.023.473±.02344.3±8.471.6±2.541.7±4.061.4±4.51.002±.006.582±.017
Complex Stimuli (B), mismatch.170±.013.280±.00588.6±1.086.0±2.481.4±5.783.7±5.7.914±.010.535±.015

These trials only show that, during perception, decoding follows the displayed image rather than the cue; they do not address imagery. These results are reported only here.

AuthorsOpenReview ↗

Second Part Reviewer JY5K

W4

See W2: the voxel-supervised adapter with identical architecture, data and training budget is reported in Tab. 7, and the statistical comparison in Tab. 12. Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.

W5

We redid all statistics (Appendix D, Tab. 15) at three levels: (i) conditional on participants and stimuli, per-participant paired t-test over the ten partitions with the Nadeau–Bengio correction for overlapping resamples, combined with Stouffer's method; (ii) participants as random effects (df = 3); (iii) stimuli as random effects (df = 5). Holm correction is applied within each table and separately at each level. The four primary tests (LFA vs. DynaDiff on AlexNet-5, Inception and CLIP for complex stimuli, and caption–word similarity for conceptual stimuli) hold at the conditional level (p_Holm ≤ 0.005) and with stimuli as random effects (p_Holm ≤ 0.028; 5–6 of 6 stimuli improve). With participants as random effects, the gain holds for conceptual stimuli (p_Holm = 0.032) but not for complex stimuli (p_Holm = 0.157; Subject 7 shows no gain), so the complex-stimulus conclusion is restricted to the participants studied. For the ROI analyses, all 11 conceptual tests survive at the conditional level, but only 3 of 30 complex tests do (parietal alone on CLIP, driven by one participant; parietal ∪ nsdgeneral on Inception; temporal ∪ nsdgeneral on CLIP). At the random-effects levels, only the temporal lobe on conceptual stimuli survives. We therefore no longer claim single-region effects for complex stimuli. The dissociation between complex and conceptual stimuli is preserved at the conditional level.

Q1

As discussed in W1, results show that our LFA method outperforms six-neural way classification, indicating that LFA identifies the imagined stimulus more accurately than tuned six-way classifiers and its reconstructions score above classifier-plus-prototype approaches, while remaining below the oracle (Tab. 8); we do not claim information beyond stimulus identity.

Regarding the role of cue letter, we have added this point as an additional limitation of our work (l. 504-509) of the updated version, since it is part of the experiment design and its effect can not be disentangled from the whole brain activity. As additional control, we observed the results of the mismatched cue letter-image vision runs on the original model, the results reported for W3 show that, for vision task, the role of the cue letter seems to not interfere with main task; however we acknowledge that this control does not automatically transfers to imagery-related activity, so it remains overall an interesting question and a limitation due to experimental protocol.

Q2

With the results from the Table in W2, reported also in Tab. 7 in Appendix C in the updated file, we demonstrated the main contribution of the latent supervision with respect to the voxel supervision and the central role played by the contrastive loss for the results. Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.

Q3

The selection at K=1000 was done on metrics results from validation trials of subject 1; we decided to use this subject to calibrate the number of augmented vision-matched neural trials of the other three subjects, in order to preserve the same setting for each experiment, we added this information in the updated text (l. 932/933). The main claims survive at the conditional and stimulus levels; with participants as random effects, only the conceptual gain holds (see W5).

Thanks for the addtional experiments and clarifications, which addresses my original concerns. My remaining concern is the scope and interpretation of the results. The evidence supports improved decoding across trials within a small, fixed stimulus set, but does not establish information beyond stimulus identity, as the authors acknowledge. Also, the perception control cannot distinguish imagery-related information from cue-associated semantic retrieval during imagery, limiting the interpretation of the regional findings. Thus, I will keep my recommendation.

We thank the reviewer for the constructive discussion and agree on the scope of our claims. We would like to add two remarks. First, as shown in Appendix A.1 (Fig. 8), single imagery trials of the same stimulus are statistically indistinguishable from trials of different stimuli at the voxel level. Generalizing to held-out imagery trials is therefore not trivial, even within a fixed stimulus set. Second, we agree that the cue-letter association cannot be separated from imagery in this paradigm, and we report it as a limitation. However, the cue structure is the same for complex and conceptual stimuli. The divergence in regional performance between the two classes therefore arises under the same cue condition, and we consider it a useful and interesting indication.

Reviewer_KM7ROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains an adaptation module supervised by latent variable loss before the frozen DynaDiff perceptual decoder and constructs a supervision target using NSD perceptual data retrieved by stimulus semantics to improve image reconstruction for an imagery task on a fixed stimulus set in four participants, while exploring decoding performance in different brain regions.

Critical strengths
  1. Under the condition of freezing the perceptual model, an operable imagination task adaptation scheme is given. The current results demonstrate the experimental value of supervised adaptation of perceptual decoders.
  2. Use the same subject's sensory data to construct the supervision target, avoiding the need to collect additional large-scale imagination data. Section 2.1 explicitly excludes target images and their corresponding NSD trials, and the appendix also groups repeated observations of the same retrieved image. These designs reduce confusion caused by duplication of direct targets.
  3. Gaps between standard reconstruction metrics and stimulus specificity were identified. Figure 5 and Tables 2 and 3 reveal that even if voxel space alignment achieves high semantic similarity, it may produce results that lack target distinction.
Critical weaknesses

1. The experiment has not distinguished between "generating prototypes after classifying known stimuli" and "recovering imagined content", and the scope of evidence for the core task is too narrow. Section 4.1 explicitly acknowledges that training and testing used the same six stimuli per set, separated only by different trials. Section 3.1 further explains that test trials of the same stimulus are averaged into one input, so only six test reconstructions are produced per subject, per division, and per set of stimuli. This setting can verify trial-by-trial generalization within a fixed set of stimuli, but cannot verify generalization to unseen imagined content. More importantly, Section 2.1 establishes a retrieval pool based on known stimulus images or target words, and then copies, adds noise to the imagined input of the same stimulus, and pairs it with the perceptual target in the pool. These pairings do not contain trial-by-trial correspondence between real imagined content. Let the stimulus identity be ss, and its perceptual latent variable center be μs\mu_s=Ev∣sg(v)\mathbb{E}_{v|s}g(v), then when taking the expectation for the retrieval pool,

Ev∣s∣∣e(x)−g(v)∣∣2\mathbb{E}_{v|s}||e(x)-g(v)||^2 = ∣∣e(x)−μs∣∣2||e(x)-\mu_s||^2 + Ev∣s\mathbb{E}_{v|s} ∣∣g(v)−μs∣∣2||g(v)-\mu_s||^2

The latter term is not relevant for prediction, so this supervision naturally encourages the model to output latent variable prototypes under stimulus conditions. Supervisory contrast loss continues to reinforce discrimination between stimulus labels and does not require recovery of imagined differences within the same stimulus. The authors themselves admit in Section 2.2 that MSE returns to the mean target of the stimulus. The current experiments did not demonstrate that LFA learned beyond category recognition and semantic prototype generation. This directly limits its contribution as a general imaginal decoding method.

  1. Section 3.1 and Appendix Tables 6-9 use ten random divisions as samples for the Wilcoxon paired test. Tables 8 and 9 first average the four subjects and then divide the test across ten times. These divisions repeatedly use the same small batch of original trials, the same set of stimuli, and the same subjects and are not a substitute for repeated evidence of independent subjects or independent stimuli. Even if these tests are able to describe robustness to random partitioning on a fixed data set, they do not allow for the same strength of population inferences. As an intuitive reference, if an exact one-sided signed rank test is performed based on the effect of four independent subjects, even if all four people are in the same direction, the minimum attainable p value is only 0.0625. This does not mean that all analyzes must use this test, but it does mean that the large number of p=0.001 in Table 9 expresses another level of evidence and cannot be used directly to support general conclusions across individuals. There are also unresolved reporting gaps in hyperparameter selection. Appendix B selects K=1000 based on Subject 1’s performance curves on the three reconstructed indicators, but does not clearly indicate whether Figure 9 is from the independent validation set or the test trials used for the final reported results. The early stopping criterion mentioned in Appendix A.2 is the difference in latent variable center similarity, which is different from this selection basis.

3.LFA consists of temporal linear mixing, single hidden layer MLP, freeze-aware decoder, MSE, supervised contrastive loss and CLIP retrieval. The building blocks themselves are mature methods. Application combinations can form valuable contributions, but this article needs to prove their irreplaceability through strict comparison. The paper compares nonlinear LFA to voxel-space ridge regression while varying function capacity, nonlinearity, regularization, time processing, and training objectives. Appendix Table 5 has already provided part of the architecture ablation, and the results directly weaken the simple latent space advantage explanation: when using the same retrieval target, voxel ridge regression reached 67.73% and 64.24% on subject 1's AlexNet-5 and CLIP respectively; the linear latent space method was only 54.09% and 60.00%, and after adding time mixing, it was only 60.15% and 61.52%. The author clearly admits in the appendix that under the same linear function family, the latent space target is not dominant in these two indicators. The missing decisive control is to use the exact same nonlinear network, data, and training budget, only replacing the loss with a voxel-space target. At the same time, there is no independent separation of the effects of MSE and SupCon, nor a comparison of direct visual semantic supervision, retrieval latent variable centering, and item-by-item perceptual latent variable supervision.

  1. In Table 1, the voxel space aligned AlexNet-5 is 68.90%, which is higher than LFA’s 67.20%; SwAV is 0.581, which is also slightly better than LFA’s 0.583. In Table 2, the average text similarity for conceptual stimuli is also higher for voxel alignment, 0.188 versus 0.171. Therefore, what the paper actually shows is a trade-off on different metrics and cannot be generalized to a consistent transcendence of voxel alignment. The authors rely on the specificity analysis of Figure 5 and Table 3 to support LFA, but this is the quantity evaluated in the previous question that has not yet been fully validated. More importantly, the appendix significance test focuses on frozen DynaDiff and does not give corresponding inferences about the most critical differences between LFA and voxel alignment. Demonstrating superiority to an unadapted model is not a substitute for demonstrating superiority to a reasonably adapted method that obtains the same supervisory information. Appendix Table 14 directly quotes the results of MindEye1, MindEye2 and BrainDiffuser in existing studies, while LFA additionally makes use of labeled imagination trials and retrieval supervision under label conditions. The paper does not establish a comparison between these methods under the same training trials, supervision budget and test averaging method. Figure 3 also explicitly selects the visually best reconstruction among the ten divisions, which cannot support better reconstruction quality in general.

6. Neuroscientific conclusions are stronger than experiments. Section 3.2 clearly states that non-visual brain areas do not have their own misaligned baselines, and the comparison object is actually the results of LFA training in these brain areas and the frozen DynaDiff on the visual area. This changes both the input region and the training state, and cannot isolate the alignment gain or independent information contribution of a region. Appendix Tables 8 and 9 list these differences as regional performance gains, which can easily be given meaning beyond the experimental design. Section 4.1 also acknowledges that there is voxel overlap with nsdgeneral in the parietal and temporal lobes, and these two areas happen to be areas where all three indicators significantly exceed the frozen baseline on complex stimuli. Without removing the intersection, this result cannot be taken as evidence outside the strictly visual area. Although the frontal lobe and prefrontal lobe do not have this overlap, they still face confusion in task interpretation: Appendix A.1 explains that each target is fixedly bound to a cue letter, which is presented on the screen during imagination, and training and testing share these mappings. The model may exploit cue recognition, letter-to-semantic association, memory retrieval, or task identity information, and the paper has not demonstrated which of these signals belong to visual imagination. Finally, after merging other regions with nsdgeneral, the results are mostly close to LFA using nsdgeneral alone. This can only show that under the current data scale, model and training method, additional information is not stably extracted; it cannot "confirm" the main role of the posterior cortex, let alone the lack of complementary information in other areas.

Questions for authors
  1. Under identical data partitioning and generation settings, what performance level can the "six-classifier + latent variable center for each class" approach achieve? How much benefit does the method still offer when the entire stimulus is excluded, and the image, word, and conditional retrieval pool for that stimulus are not involved in adaptation training?

  2. After removing simple line interference, what are the pairwise recognition rate and six-choice accuracy within complex stimuli? For conceptual stimuli, is there still an advantage in performing three-choice recognition and within-group specificity assessment within each of the two semantic groups? What scores would be obtained with a fixed natural graph output, two sets of semantic prototypes, and a control group with shuffled brain input?

  3. What inference object does the test on ten random partitions specifically target? After considering subject and stimulus hierarchy report effects, handling duplicate partition dependencies, and performing multiple comparison corrections, which conclusions still hold?

  4. Under the same nonlinear architecture and training budget, is the latent variable target superior to the voxel target? How do the performances differ after removing SupCon, using only SupCon, and using category latent variable centers or direct visual semantic targets?

  5. Does LFA offer a statistically and practically significant advantage over voxel alignment using the same retrieval supervision on pre-specified key metrics? Does the ranking remain consistent after providing the same imaginative supervision and adaptation opportunities to other decoders? Does the qualitative advantage implied in Figure 3 still exist after using a fixed partition and uniform display rules?

  6. After removing the intersection with nsdgeneral from the parietal and temporal lobes and matching the number of voxels, network capacity, and training process, are there control conditions that can separate cue letters, semantic extraction, and visual imagination?

6 comments
AuthorsOpenReview ↗

Response to Reviewer KM7R

We thank the reviewer for a very thorough review. The comments led to several new control experiments and to a narrower, more precise statement of our claims. In summary:

  • (i) we added a six-way classifier control;
  • (ii) we added a voxel-supervised adapter with identical architecture and training budget, and an ablation of the two loss terms;
  • (iii) we redid all statistics with tests that account for overlapping partitions and for the participant and stimulus structure, with Holm correction;
  • (iv) we retrained the parietal and temporal ROIs without the voxels shared with nsdgeneral.

Accordingly, we (a) restrict our main claim to generalization across trials within a fixed stimulus set, (b) no longer claim single-region effects for complex stimuli, (c) report the LFA vs. ridge comparison as a trade-off on standard metrics and an advantage in stimulus specificity, and (d) list the cue-letter confound as a limitation. All changes are highlighted in red in the revised PDF.

W1

We agree. As the reviewer notes, the MSE term regresses towards the stimulus mean, and the SupCon term is a stimulus-discriminative objective; moreover, test trials are averaged into one input per stimulus, so our setting cannot assess within-stimulus variability. What Tab. 8 in Appendix C shows is that

  • A) LFA identifies the imagined stimulus more accurately than tuned six-way classifiers (top-1 56.3% vs. 42.5% for logistic regression with nested cross-validation and 31.6% for the MLP; chance 16.7%).
  • B) its reconstructions score above classifier-plus-prototype approaches, while remaining well below the oracle that always receives the correct prototype (image specificity 0.066 vs. 0.117). Part of this advantage may reflect LFA's higher identification accuracy; we do not claim that LFA's outputs carry information beyond stimulus identity.

When a whole stimulus is excluded from training, performance falls to chance; we now state this explicitly in Sec. 4.1 (lines 492-498) and restrict our claim to generalization across trials within a fixed stimulus set (see Q1).

W2

A) We recomputed all statistics (Appendix D, Tab. 15) at three levels: (i) conditional on participants and stimuli, per-participant paired t-test over the ten partitions with the Nadeau–Bengio correction for overlapping resamples, combined with Stouffer's method; (ii) participants as random effects (df = 3); (iii) stimuli as random effects (df = 5). Holm correction is applied within each table and separately at each level. The four primary tests (LFA vs. DynaDiff on AlexNet-5, Inception and CLIP for complex stimuli, and caption–word similarity for conceptual stimuli) hold at the conditional level (p_Holm ≤ 0.005) and with stimuli as random effects (p_Holm ≤ 0.028; 5–6 of 6 stimuli improve). With participants as random effects, the gain holds for conceptual stimuli (p_Holm = 0.032) but not for complex stimuli (p_Holm = 0.157; Subject 7 shows no gain), so the complex-stimulus conclusion is restricted to the participants studied. For the ROI analyses, all 11 conceptual tests survive at the conditional level, but only 3 of 30 complex tests do (parietal alone on CLIP, driven by one participant; parietal ∪ nsdgeneral on Inception; temporal ∪ nsdgeneral on CLIP). At the random-effects levels, only the temporal lobe on conceptual stimuli survives. We therefore no longer claim single-region effects for complex stimuli.

B) Regarding K: K = 1000 was selected on the validation trials of Subject 1 (Fig. 9) and fixed for the other subjects; Tab. 5 and 6 in the updated pdf are post-hoc analyses on the test partitions, not used for selection; we now state this in Appendix B. The early-stopping criterion differs because it is computed in latent space on validation data at every epoch, whereas the reconstruction metrics require image generation.

AuthorsOpenReview ↗

Second Part Reviewer KM7R

W3

Regarding the apparent contradiction between performance between voxel and latent supervision, we showed in Fig. 5 that there is no one-to-one correspondence between the values of metrics and the faithfulness of reconstructions, with a closed set of stimuli, as in our case, Functional Alignment via Ridge regression could reproduce collapsed reconstructions for different concepts, still obtaining good performance for the restricted set of distractors in the pool for the 2-way comparisons that the dataset provides to us. Our claim is that metrics can only give a partial sight to the reconstructions, for this reasons we also provided specificity of reconstructions, in order to add an additional control to the quality of reconstructions.

Tab. 7 in Appendix C (reported in Q4) shows that with the same architecture and training budget, but with a latent supervision (our LFA) outperforms the voxel supervision both in specificity (Image spec. 0.066 > 0.039, Text spec. 0.037 > 0.024) and for high-level metrics (CLIP 67.4 > 57.3, ALEX-5 67.2 > 58.2, INCEPTION 78.8 > 71.0). Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.

The table in the updated version also displays the role of each term of the loss, providing the fact that the contrastive-term alone provides most of the gains, the MSE term alone is competitive on AlexNet-5 but halves specificity.

Training with the stimulus centroid as target is equivalent, up to a constant, to the MSE term with retrieved targets; this is the 'Latent, MSE only' variant of Tab. 7. The oracle row of Tab. 8 gives the ceiling of directly outputting the centroid. Finally, an item-by-item perceptual latent variable supervision is not indicated for our setting, since the Versatile Diffusion was fine tuned on the output of the Brain Module and It was not trained on CLIP perceptual embeddings directly, as we reported in the main text.

W4

With our work we do not want just to adapt, as it could be with a Functional Alignment via linear Ridge, which it would have enabled obtaining good metrics but not corresponding quality, as Fig. 5 shows. Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain:

MetricSubj 1 ΔSubj 1 pSubj 2 ΔSubj 2 pSubj 5 ΔSubj 5 pSubj 7 ΔSubj 7 pAll ΔConditional (k)Rand. participantsRand. stimuli
Image spec.+0.0360.026+0.0250.057+0.0410.034+0.0060.390+0.0270.010 (2/4)0.0960.043 (6/6)
Text spec.+0.0210.005+0.0130.034+0.0160.093+0.0010.477+0.0130.010 (2/4)0.1170.043 (4/6)
CLIP+14.70.125+7.00.178+18.20.014+0.80.473+10.20.030 (1/4)0.1170.007 (6/6)
Inception+10.90.014+5.60.282+15.80.007-0.90.538+7.80.015 (2/4)0.1170.043 (5/6)
AlexNet-5+16.80.081+3.00.375+10.50.106+5.80.274+9.00.037 (0/4)0.1170.043 (6/6)

We added in the caption of Tab. 20 in Appendix E in the updated pdf the note on the different procedure for our LFA and the other decoders. The choice of the best visually inspected partition is purely comparative, since the images shown for the other three models, MindEye1, MindEye2 and Brain Diffuser have been selected in the same way, as reported in Reese Kneeland et al. "NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery". Other medium and worst reconstructions from other partitions were already reported in the Appendix of the main text.

AuthorsOpenReview ↗

Third Part Reviewer KM7

W6

We agree with the reviewer on the fact that the lack of direct baselines for non-visual brain areas limits the neuroscientific conclusions over the contribution of the posterior cortex, and we acknowledged it in the main text. However, the choice of DynaDiff model's prediction from voxels of visual cortex offers not a simple comparison for our LFA while using it with different areas of the brain, since the baseline is performing cross-task generalization from the same input on which was trained on originally, while we are adding, both separately both with the union of the visual cortex voxels, regions that were not seen by the original model. We also agree with the reviewer on conducting additional statistical analysis which takes into account both overlapping partitions and the shared voxels between visual cortex with temporal and parietal ROIs. After Holm correction over the 30 complex-stimulus tests, only the parietal ROI alone on CLIP survives (p_Holm = 0.015, one participant), and it does not survive once the shared voxels are removed (p_Holm = 0.418); among the unions, parietal ∪ nsdgeneral on Inception and temporal ∪ nsdgeneral on CLIP survive (Tab. 13). However, for conceptual stimuli all ROIs contribute significantly at the conditional level (with participants as random effects only the temporal lobe survives), unlike for complex stimuli. We interpret this in terms of task demands: complex stimuli involve recalling a seen image, whereas conceptual stimuli involve more abstract processes that may be distributed across the cortex.

CUE LETTER CONTROL: Finally the lack of a control for the cue letter association with the imagery content must be acknowledged as an additional limitations, and we added it in the updated version; the current NSD-Imagery dataset can not provide us an alternative way to disentangle its role in the imagery analysis, since each imagery-trial is associated with a cue letter. The only control that we can conduct, as done for the Reviewer JY5K, could be to measure the performance of the DynaDiff base model on the mismatched vision trials between cue-letter and associated image, the performance are reported here only:

TaskPixCorr ↑SSIM ↑Alex2 ↑Alex5 ↑Incep ↑CLIP ↑Eff ↓SwAV ↓
Simple Stimuli (A), mismatch−.000±.023.473±.02344.3±8.471.6±2.541.7±4.061.4±4.51.002±.006.582±.017
Complex Stimuli (B), mismatch.170±.013.280±.00588.6±1.086.0±2.481.4±5.783.7±5.7.914±.010.535±.015

These trials only show that, during perception, decoding follows the displayed image rather than the cue; they do not address imagery.

Q1

We added an additional control in order to compare our LFA with a six-way neural classifier; we tested our LFA against both a multinomial logistic regression with nested CV and to an architecture as LFA but with a cross-entropy loss. Our LFA method outperforms both of them, regarding both semantic metrics and image and text specificity.

MethodTop-1 (%)Image spec.Text spec.CLIPIncepAlexNet-5
Oracle1000.1170.07180.693.787.8
LFA56.30.0660.03767.478.867.2
Logistic regression (6-way)42.50.0400.01958.175.363.3
MLP (6-way)31.60.0230.01353.970.858.2
DynaDiff baseline26.70.0280.01953.367.055.2

As targets were used the mean predicted embeddings from the brain module, within each pool of retrieved perceptual stimuli. Results are also reported in Tab. 8 in Appendix C in the updated text.

When a whole stimulus is excluded from training, performance falls to chance. This is expected, since the supervision is stimulus-specific (Sec. 4.1 of revised pdf). It is also consistent with Shirakawa et al. (2025, "Spurious reconstruction from brain activity"), who show that with low stimulus diversity, reconstruction methods can reduce to classification followed by generation; with six stimuli per set, our setting is an extreme case of this, and we therefore make no claim beyond decoding across trials within a fixed stimulus set.

Reviewer_HZHJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for decoding visual signal from neural activities (fMRI) of imagined content. This fairly new task extends the older visual-decoding task, although in imagined-decoding the neural activity recorded results solely from imagination of the patient. The proposed method builds on a recent decoding model, DynaDiff, and introduces an alignment module to align the neural imagined signal with those of "corresponding" neural visuals signals, using a latent objective. The correspondences is obtained with a k-nn search, comparing either pairs of CLIP-image embedding for neural visual signal or pairs of (CLIP-text embedding, CLIP-image embeddings) for neural imagined signal. The pairs are then aligned in the voxel space but solely using a latent objective and a contrastive term.

Critical strengths

Overall, the paper is clear and the experiments are rather thorough (though quite repetitive, cf. W6). While the neuroscientific contribution is beyond my field of expertise, I have found its positioning of interest for the reader.

  • The task addressed is fairly new, and to my knowledge only captured by the NSD-Imagery dataset, and of general interest for both machine learning and neuroscience, which makes the problem setup significant for the community.

  • While the proposed method is rather straightforward wrt the literature (cf. weaknesses) the proposal is sound and demonstrate great performance. Further, while the evaluation is (logically) limited to the only NSD-Imagery dataset, the authors have put a great effort to thoroughly analyze the results and provide several readings of the performance qualitatively (Fig. 3), via reconstruction metrics (Tab. 1, Fig. 4), correlation of the generated image and target caption (Fig. 5, Tabs. 2,3) as well as by exploring the use of non-visual brain regions (Sec. 3.2).

  • I appreciate the honnest effort in reporting true limitations of the method.

Critical weaknesses

As the paper positions at the cross-road of neuroscience and machine learning while my expertise lies in the latter, I focus my review on the methodology and evaluation, rather than the neuroscience analysis.
While my current rating is slightly leaning towards acceptance, my judgment is largely driven by the originality of the problem setup, although the experiments are largely redundant (cf. W6). My final ratings may vary, in particular given the opinion of peer neuroscience reviewers.

W1. The proposed methodology consists of two components: an augmentation strategy and a "latent" alignment. Both of the proposed components (k-nn based augmentations and mse+contrastive alignments) are classical in the ML literature. In particular, brain signal alignment is very classical in most fMRI visual decoding methods as in MindEye (Scotti et al., NeurIPS 2023), DREAM (Xia et al., WACV 2024), etc. Similar, contrastive alignment as in Sec. 2.2 is also done in the literature such as DREAM (Xia et al., WACV 2024). Therefore, despite the effectiveness of the proposed method, its technical contribution seems fairly small.
W2. The effect of the Gaussian noise (ll.155-157) is not analyzed.
W3. While Sec. 2.2 imposes a latent objective to align the imagery responses to the DynaDiff representation. However, since the alignment operates in voxel space (l. 177) how do the obtained representations compare to those of the NSD dataset ? For example, considering semantically similar pairs of data in (NSD, NSD-Imagery) is it possible to conduct a spatial (eg, at voxel level) analysis to measure the remaining difference between the NSD voxel brain signal and the resulting aligned representation with the proposed method.
W4. Overall, the discussion (Sec. 4) appears mostly as a summary rather than an actual discussion, and only a small portion contributes to further understanding.
W5. While I appreciate the non-visual brain regions analysis, which takes more than 1 full page, its importance is not obvious as the observations seems to correlate with existing findings in the literature per authors statements (ll.372-374)
W6. Some experiments appear as duplicates. For example, Fig. 5 reports similarity between the generated image and the target caption, a semantic similarity already captured by high-level metrics of Tab. 1, and already dissected per-subject in Fig. 4. As it stands Tab. 1, Fig. 4, Fig. 5 appear quite repetitive and a large part could have been moved to the appendix.

References:
Scotti, Paul, et al. "Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors." NeurIPS 23.
Xia, Weihao, et al. "Dream: Visual decoding from reversing human visual system." WACV 24

Questions for authors

Please mostly refer to the weaknesses.

  • Could the authors clarify the originality of the technical contribution ? (cf. W1)
  • What is the effect of varying the Gaussian noise (W2) ?
  • Could the authors clarify if I understood correctly that the present method is trained per-subject? If so, given that several vision-decoding methods have now been extended to multi-subject (such as UMBRAE (ECCV, 2024) and others), could the current method be extended for generalization across subjects? How?
  • Is it possible to measure the effect of the LFA at the representation level (eg, voxel, spatial, etc.) ? (cf. also W3)
  • Could authors summarize the key findings of Sec. 4 with respect to the existing neuroscience literature ? (cf. W4)

References:
Xia, Weihao, et al. "Umbrae: Unified multimodal brain decoding." ECCV 24

5 comments
AuthorsOpenReview ↗

Response to Reviewer HZHJ

We thank the reviewer for the careful reading and the constructive questions. In response, we have (i) sharpened the statement of our contribution, (ii) added an analysis of the injected noise level, (iii) added a voxel-level and a latent-level analysis of the alignment module, and (iv) rewritten the Discussion to relate our findings to the imagery and semantic-memory literature. All changes are highlighted in red in the revised PDF.

W1 and Q1

We agree that k-NN retrieval, MSE regression and supervised contrastive losses are individually standard, and we do not claim novelty for these building blocks. However, to the best of our knowledge, this is the first time in which their application is not for "cross-subject" but for "cross-task" alignment, since our goal is to retrieve neural data from runs of vision task of the same participants in order to handle the out-of-distribution imagery shift domain with respect to vision perception's domain, on which DynaDiff was trained on.

We believe this is the main novelty, together with the alignment objective, that is in the latent space, which is in our perspective, the most natural choice space where this alignment could take place.

Our contribution lies in how the individual components are combined to address a problem where existing alignment methods do not directly apply:

  • A) Prior alignment methods (e.g., MindEye, DREAM, hyperalignment) map responses to the same stimuli across subjects or modalities. In NSD-Imagery, almost no matched perception data exist for the imagined stimuli (five of six complex images belong to the held-out shared1000 set, and conceptual targets have no image at all). We instead use perception responses to different but semantically related images, retrieved from the same participant's NSD sessions, as surrogate targets. Here, we aim for surrogate supervision without trial-matched data.
  • B) Cross-task alignment through a frozen decoder's latent space: The adapter outputs voxels but is trained only through the frozen brain module's conditioning space. A new control with identical architecture and training budget but a voxel-space target [Tab. 7 in Appendix C] with relative statistical analysis in Tab. 12 in Appendix D.
  • C) Along the way, we also found something on evaluation. Standard identification metrics can reward stimulus-generic reconstructions: voxel-space ridge alignment matches LFA on these metrics while its specificity remains well below that of LFA (Tab. 3: 0.040 vs. 0.067; baseline 0.028), and Fig. 5 shows the collapse for complex stimuli. We believe this is a useful message for imagery decoding beyond our method.

We have revised the text to state these contributions explicitly and added DREAM (Xia et al., 2024) to the Related Work.

W2 and Q2

We added an analysis varying the noise standard deviation for Subject 1 on complex stimuli (gains over the DynaDiff baseline, percentage points, mean over 10 partitions) computed on the test partitions; σ was not tuned on these results:

Valueσ / SDAlex-5IncepCLIP
00+14.5+17.0+16.1
0.0020.002+12.7+16.5+16.7
Paper (0.02)0.02+13.5+16.5+16.2
0.25≈ 0.24+11.5+17.4+15.2
1.0≈ 0.95+6.8+17.7+18.0

Differences are within the across-partition SD; this analysis cannot detect small effects, but shows no critical dependence on σ and only AlexNet-5 degrades at noise comparable to the data SD. The noise is therefore not a data augmentation and not a critical hyperparameter: its only role is to avoid exactly identical input rows when the few imagery training trials are replicated to match the K retrieved targets. We clarified this in Section 2.1 [l.143-145] and added in Tab. 6 in Appendix B.

AuthorsOpenReview ↗

Second Part

Q3

Yes we trained subject-specific models. Recent works in the field of visual perception brain decoding focus also on cross-subject alignment, however we left this design for future experiments since in this work our main goal is to address the problem of aligning two different brain processes with simple design choices; also DynaDiff itself, that is the state-of-the-art for vision decoding with time series fMRI data can be adapted for cross subject experiments, but first we want to validate results for individual subjects. Our method is quite general and in principle could be implemented with other state-of-the-art vision decoders. We added the citation to Xia, Weihao, et al. "Umbrae: Unified multimodal brain decoding." ECCV 24 in the conclusions, thanks for the suggestion.

W3 and Q4

We found this comment very interesting and we further investigated the effect of the Alignment module through a voxel-wise correlation analysis. For each voxel, we computed the Pearson correlation between imagery responses (before and after the alignment module) and perception responses to the same stimuli, across stimuli and the six time points, after removing the mean over stimuli at each time point. We used two perception references: the true NSD responses to the five shared1000 images, and the mean of the retrieved NSD targets. True NSD: NSD responses to the same images (5 stimuli) taken from the test set of NSD; Retrieved targets: mean of the retrieved training targets (6 stimuli), taken from the training set of NSD.

ReferenceStageSub. 1Sub. 2Sub. 5Sub. 7
True NSDBefore0.0230.011-0.003-0.018
True NSDAfter0.0110.0260.0210.005
Retrieved targetsBefore0.0280.0440.0370.023
Retrieved targetsAfter0.0430.0880.0550.017

Voxel-level correspondence increases in 6 of 8 cases, but the values remain very small and the effect is not consistent (Subject 1 decreases against the true NSD responses, Subject 7 against the retrieved targets). We interpret this as expected: the alignment module is trained only through the latent objective, so nothing forces its output to resemble perception at the voxel level.

Together, these results indicate that the alignment operates mainly on the semantic, stimulus-discriminative structure that imagery shares with perception, rather than making imagery voxel patterns perception-like. This is consistent with our motivation for a latent objective and with the poor specificity of voxel-space alignment. This table is also reported in the updated PDF in Tab. 9 in Appendix C.

AuthorsOpenReview ↗

Third Part

W4, W5 and Q5

We updated the Discussion section in order to provide further considerations on the obtained results. First, we discuss the relationship between imagery and perception: imagery trials are nearly uncorrelated at the voxel level yet decodable in the semantic space. This suggests that imagery shares with perception mainly semantic structure rather than the perceptual activity pattern. It also explains the limits of voxel-space alignment and of standard identification metrics.

Second, we interpret the different cortical distribution across stimulus classes. Non-visual regions, especially prefrontal and temporal cortex, contribute substantially more for conceptual than for complex stimuli. We relate this to the demands of each cue: reinstating a specific image relies on posterior cortex, whereas generating an exemplar from a concept draws on a distributed semantic network.

Revised Sec. 4 reports three key findings.

  • A) Imagery shares with perception mainly semantic rather than voxel-level structure. This is consistent with evidence that imagery preserves conceptual content more reliably than perceptual detail (Reddy et al., 2010; Pearson et al., 2015).
  • B) For complex stimuli, decoding is dominated by posterior cortex, in line with prior work (Ganis et al., 2004; Naselaris et al., 2014).
  • C) For conceptual stimuli, non-visual regions, especially temporal and prefrontal cortex, contribute substantially more. This extends prior findings and is consistent with distributed semantic representations (Patterson et al., 2007) and with prefrontal recruitment when mental content is constructed rather than recalled (Addis et al., 2007). These conceptual effects hold at the conditional level (Tab. 14, all 11 tests); with participants as random effects only the temporal lobe survives, so the prefrontal result should be read as indicative.

(Focus on W5) For complex stimuli, the agreement with prior literature supports the neuroscientific validity of our findings: posterior dominance confirms that our cross-task pipeline recovers established results (Ganis et al., "Brain areas underlying visual mental imagery and visual perception: an fMRI study", 2004; Wang et al., "Decoding visual experience and mapping semantics through whole-brain analysis using fMRI foundation models", 2024). In our view, a main contribution of the section is the dissociation between stimulus classes, which the revised Discussion (Sec. 4) now makes explicit and quantifies. Relative to the gain obtained from visual cortex, prefrontal cortex recovers less than 10% for complex stimuli but about 70% for conceptual ones. The temporal lobes retain over 90% of the conceptual gain even after removing voxels shared with posterior cortex (Tabs. 13–14 in Appendix D). Since frontal and prefrontal ROIs do not overlap with posterior cortex, they provide the cleanest evidence that imagery-related information extends beyond it specifically for conceptual content, although for these two ROIs the effect holds only at the conditional level (Tab. 14). We interpret this in terms of task demands. Reinstating a specific image relies on posterior cortex. Generating an exemplar from semantic knowledge draws on a distributed network (Patterson et al., "Where do you know what you know? The representation of semantic knowledge in the human brain", 2007), in line with evidence that constructing mental content recruits additional prefrontal regions (Addis et al., "Remembering the past and imagining the future: Common and distinct neural substrates during event construction and elaboration", 2007).

W6

Regarding the redundancy of the reported results, we choose to report separately in the main text Tab. 1, Fig. 4 and Fig. 5 because each of them show complementary aspects that we would like to stress out: Tab. 1 reports the averaged metrics among four subjects, in which appears the main contribution of our LFA but it still hides the collapse of the Functional Alignment model that is not captured in this table. For this reason we presented also Fig. 5 that is, in our opinion, a more evident representation of the specificity of LFA reconstructions with respect both the baseline and the Functional Alignment and shows clearly that the latter tends to collapse different categories in a single one for complex stimuli. The same effect is reported in Tab. 3 for conceptual stimuli. Lastly, Fig. 4 reports the single subject performance before and after the Alignment module, showing the inter-subject variability for imagery-related activity but it also shows transparently the contribution of each subject.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive feedback, which led to new control experiments, a revised statistical analysis and more precise claims. Main changes are in red in the PDF.

Our contribution

We adapt a state-of-the-art perception decoder (DynaDiff) to imagined content from NSD-Imagery with a latent functional alignment (LFA): a module placed before the frozen decoder, operating on voxels but trained only through the decoder's conditioning space (MSE + SupCon loss). As matched imagery–perception data are almost absent, we use the participants' own NSD perception trials, retrieved for semantic similarity to each stimulus, as surrogate targets. LFA improves high-level metrics over the frozen decoder, shows that standard identification metrics can reward stimulus-generic reconstructions, and reveals that non-visual regions contribute substantially more for conceptual than for complex stimuli.

Strengths identified by the reviewers

  • New, relevant problem setting for ML and neuroscience (HZHJ).
  • Practical adaptation of a frozen decoder to scarce imagery data using the same participants' perception data (KM7R, JY5K).
  • Sound method with good performance (HZHJ); targets excluded from retrieval and NSD repetitions grouped in the splits (KM7R).
  • Thorough evaluation, including specificity and non-visual regions (HZHJ).
  • Fig. 5 and Tabs. 2–3 show that aggregate similarity can reward generic reconstructions, motivating stimulus-specific evaluation (KM7R, JY5K).
  • Honest reporting of limitations (HZHJ).

Common weaknesses

  1. Alignment vs. recognition of the same six stimuli; a classifier-plus-prototype control was missing (KM7R W1, JY5K W1).
  2. The latent objective was not isolated from capacity and temporal processing; a matched voxel-supervised adapter and a contrastive-term ablation were requested (KM7R W3, JY5K W2); the building blocks are standard (HZHJ W1).
  3. Inference on overlapping partitions, participants and stimuli; selection of K (KM7R W2, JY5K W5).
  4. Regional conclusions: ROI overlap with visual cortex, no unaligned baseline, cue-letter confound (KM7R W6, JY5K W3); a deeper discussion was requested (HZHJ W4, W5).
  5. Effect of the injected noise and of the alignment at voxel level (HZHJ W2, W3).

How we addressed them

1. Classifier control (Tab. 8, App. C). On the same test inputs and prototypes, LFA identifies the imagined stimulus better than tuned six-way classifiers (top-1 56.3% vs. 42.5% logistic regression, 31.6% MLP; chance 16.7%), with higher specificity (0.066 vs. 0.040/0.023), below the correct-prototype oracle (0.117). We claim no information beyond stimulus identity and restrict our claim to generalization across trials within a fixed stimulus set; excluding a whole stimulus gives chance performance (Sec. 4.1).

2. Matched voxel supervision and loss ablation (Tab. 7, App. C; Tab. 12, App. D). With identical network, data and training, LFA beats voxel-space supervision on all five metrics at the conditional level (p_Holm = 0.010–0.037) and with stimuli as random effects (p_Holm ≤ 0.043), not with participants as random effects (p_Holm ≥ 0.096). The contrastive term drives most of the gain; MSE alone halves specificity. Our contribution lies not in the building blocks but in same-participant surrogate targets for cross-task alignment, supervised through the frozen decoder's latent space.

3. Statistics (App. D, Tabs. 10–15). Nadeau–Bengio corrected tests per participant with Stouffer combination, participants and stimuli as random effects, Holm correction. Primary tests hold at the conditional level (p_Holm ≤ 0.005) and with stimuli as random effects (p_Holm ≤ 0.028); with participants as random effects, only for conceptual stimuli (0.032 vs. 0.157), so the complex-stimulus claim is restricted to the participants studied. K was selected on Subject 1 validation trials; test analyses in App. B are post hoc.

4. Regions and Discussion (Tabs. 13–14, Sec. 4). After removing voxels shared with nsdgeneral, complex-stimulus gains shrink (only 3/30 complex tests survive overall), so we drop single-region claims for this class; conceptual gains persist (parietal +0.066, p_Holm = 0.010; temporal +0.068, p_Holm = 0.008). Relative to visual cortex, prefrontal cortex recovers <10% of the gain for complex but ~70% for conceptual stimuli; the rewritten Discussion relates this to task demands. The cue-letter confound and the missing baseline for non-visual ROIs are now limitations.

5. Further analyses. Gains are unchanged for noise from 0 to σ/SD ≈ 0.24 (Tab. 6, App. B); a voxel-wise analysis (Tab. 9, App. C) shows the alignment acts on stimulus-discriminative semantic structure rather than making voxel patterns perception-like. We also cite UMBRAE and DREAM and mark the comparison with Kneeland et al. (2025) as not like-for-like (Tab. 20).

Point-by-point answers follow below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.