- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper adapts a frozen DynaDiff perception decoder to mental imagery using a voxel-space adapter supervised by latent reconstruction and contrastive losses. Retrieved perception responses provide training targets. Experiments on four NSD-Imagery participants assess reconstruction, stimulus specificity, and decoding from additional cortical regions.
Critical strengths
Adapting a frozen perception decoder with retrieved targets is a practical approach to scarce imagery data, evaluated quantitatively across participants. Tables 2–3 and Figure 5 show why aggregate similarity can reward generic reconstructions, motivating a useful stimulus-specific evaluation.
Critical weaknesses
- The experiment does not distinguish alignment from recognition of known stimuli. Training and testing use the same six stimuli per set, while Equation 1 explicitly fits stimulus-level target means and separates their labels. A six-way neural classifier followed by the corresponding retrieved prototype is therefore an important missing control: it would test whether the gains require functional alignment beyond supervised stimulus identification within the stated evaluation setting.
- The comparison does not isolate the proposed latent objective. Relative to voxel-space ridge, LFA changes model capacity, temporal processing, and supervision simultaneously. The reported linear/nonlinear ablation does not resolve this issue: the linear latent model is not better than ridge on AlexNet-5 or CLIP, while a matched nonlinear voxel-supervised adapter and an isolated contrastive-loss comparison are absent, leaving the central design motivation insufficiently tested.
- The evidence for imagery-specific regional information remains ambiguous. The task associates each stimulus with a fixed visual cue, so stimulus discrimination may include cue recognition or associated semantic retrieval. The regional results in Section 3.2 do not separate these sources from mental imagery; despite the acknowledged ROI overlap and baseline limitations, successful reconstruction alone cannot establish the functional interpretation offered for the additional cortical regions.
- The claimed methodological advantage depends on which semantic property is measured. LFA trails ridge on AlexNet-5 and SwAV in Table 1 and on conceptual target similarity in Table 2, while improving specificity in Table 3. This distinction is informative, but the specificity gain should be assessed against matched supervised controls before it supports the broader claim that latent alignment yields better imagery reconstructions.
- Statistical support is limited by dependence across repeated partitions. Section 3.1 tests ten overlapping partitions of the same trials, with held-out responses averaged into six reconstructions per set. These repetitions characterize sensitivity to partitioning rather than independent neural samples; inference respecting shared trials and participants is needed to determine how strongly the reported significance supports the method and regional comparisons.
Questions for authors
- How does LFA compare with a six-way neural classifier followed by its retrieved prototype on the same held-out trials and specificity metrics? What existing control distinguishes imagery-related information from the fixed cue?
- With nonlinear capacity and temporal processing matched, does latent supervision outperform voxel-space supervision, and what is the contribution of the contrastive term?
- Was K = 1000 selected exclusively on validation data? Do the principal conclusions persist under inference that accounts for reused trials and the participant structure?
4 comments
Second Part Reviewer JY5K
W4
See W2: the voxel-supervised adapter with identical architecture, data and training budget is reported in Tab. 7, and the statistical comparison in Tab. 12. Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.
W5
We redid all statistics (Appendix D, Tab. 15) at three levels: (i) conditional on participants and stimuli, per-participant paired t-test over the ten partitions with the Nadeau–Bengio correction for overlapping resamples, combined with Stouffer's method; (ii) participants as random effects (df = 3); (iii) stimuli as random effects (df = 5). Holm correction is applied within each table and separately at each level. The four primary tests (LFA vs. DynaDiff on AlexNet-5, Inception and CLIP for complex stimuli, and caption–word similarity for conceptual stimuli) hold at the conditional level (p_Holm ≤ 0.005) and with stimuli as random effects (p_Holm ≤ 0.028; 5–6 of 6 stimuli improve). With participants as random effects, the gain holds for conceptual stimuli (p_Holm = 0.032) but not for complex stimuli (p_Holm = 0.157; Subject 7 shows no gain), so the complex-stimulus conclusion is restricted to the participants studied. For the ROI analyses, all 11 conceptual tests survive at the conditional level, but only 3 of 30 complex tests do (parietal alone on CLIP, driven by one participant; parietal ∪ nsdgeneral on Inception; temporal ∪ nsdgeneral on CLIP). At the random-effects levels, only the temporal lobe on conceptual stimuli survives. We therefore no longer claim single-region effects for complex stimuli. The dissociation between complex and conceptual stimuli is preserved at the conditional level.
Q1
As discussed in W1, results show that our LFA method outperforms six-neural way classification, indicating that LFA identifies the imagined stimulus more accurately than tuned six-way classifiers and its reconstructions score above classifier-plus-prototype approaches, while remaining below the oracle (Tab. 8); we do not claim information beyond stimulus identity.
Regarding the role of cue letter, we have added this point as an additional limitation of our work (l. 504-509) of the updated version, since it is part of the experiment design and its effect can not be disentangled from the whole brain activity. As additional control, we observed the results of the mismatched cue letter-image vision runs on the original model, the results reported for W3 show that, for vision task, the role of the cue letter seems to not interfere with main task; however we acknowledge that this control does not automatically transfers to imagery-related activity, so it remains overall an interesting question and a limitation due to experimental protocol.
Q2
With the results from the Table in W2, reported also in Tab. 7 in Appendix C in the updated file, we demonstrated the main contribution of the latent supervision with respect to the voxel supervision and the central role played by the contrastive loss for the results. Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.
Q3
The selection at K=1000 was done on metrics results from validation trials of subject 1; we decided to use this subject to calibrate the number of augmented vision-matched neural trials of the other three subjects, in order to preserve the same setting for each experiment, we added this information in the updated text (l. 932/933). The main claims survive at the conditional and stimulus levels; with participants as random effects, only the conceptual gain holds (see W5).
Thanks for the addtional experiments and clarifications, which addresses my original concerns. My remaining concern is the scope and interpretation of the results. The evidence supports improved decoding across trials within a small, fixed stimulus set, but does not establish information beyond stimulus identity, as the authors acknowledge. Also, the perception control cannot distinguish imagery-related information from cue-associated semantic retrieval during imagery, limiting the interpretation of the regional findings. Thus, I will keep my recommendation.
We thank the reviewer for the constructive discussion and agree on the scope of our claims. We would like to add two remarks. First, as shown in Appendix A.1 (Fig. 8), single imagery trials of the same stimulus are statistically indistinguishable from trials of different stimuli at the voxel level. Generalizing to held-out imagery trials is therefore not trivial, even within a fixed stimulus set. Second, we agree that the cue-letter association cannot be separated from imagery in this paradigm, and we report it as a limitation. However, the cue structure is the same for complex and conceptual stimuli. The divergence in regional performance between the two classes therefore arises under the same cue condition, and we consider it a useful and interesting indication.
Response to Reviewer JY5K
We thank the reviewer for their careful and constructive assessment of our work. The comments brought additional control experiments and helped us refine the scope and precision of our claims. The main revisions and responses are summarized below:
W1
As noted by the reviewer and as we mentioned in the limitations of our article, our findings are related to the generalization of new trials for a fixed set of stimuli, instead of generalizing to unseen concepts and stimuli. Tab. 8 in Appendix C of the updated pdf compares LFA with six-way classifiers followed by the retrieved prototype, that we report in the following:
Comparing our LFA both with a multinomial logistic regression and with a MLP but with a cross-entropy loss, our LFA method outperforms both of them, regarding both semantic metrics and image and text specificity. LFA identifies the imagined stimulus more accurately than tuned six-way classifiers (top-1 56.3% vs. 42.5% for logistic regression with nested cross-validation and 31.6% for the MLP; chance 16.7%); its reconstructions score above classifier-plus-prototype approaches, while remaining well below the oracle that always receives the correct prototype (image specificity 0.066 vs. 0.117).
When a whole stimulus is excluded from training, performance falls to chance; we now state this explicitly in Sec. 4.1 (lines 492-498) and restrict our claim to generalization across trials within a fixed stimulus set.
W2
In order to further assess the contribution of our LFA respect to the Ridge baseline, we investigated both the contribution of the latent supervision respect to a matched voxel-supervised adapter, isolating the contribution of the contrastive-loss term. Results are reported in Tab. 7 in Appendix C of the uploaded version; they are also reported in the following table:
It clearly emerges the main contributions of both the latent supervision and the contrastive-loss term, that drives most of the performance reported in the main text. The MSE term alone is competitive on AlexNet-5 but halves specificity.
On the other hand, the same architecture trained with a voxel supervision improves performance over the DynaDiff baseline but it loses on both specificity and high-level accuracy with respect the latent supervision (Our LFA). Against the same network supervised in voxel space (Tab. 12), LFA is better on all five metrics at the conditional level (p_Holm = 0.010 for image and text specificity, 0.015–0.037 for the high-level metrics) and with stimuli as random effects (p_Holm ≤ 0.043; 4–6 of 6 stimuli improve), but not with participants as random effects (p_Holm ≥ 0.096). For each metric, the advantage is individually significant in at most two of four participants, and Subject 7 shows little or no gain.
W3
We acknowledge this observation of the reviewer. We added this consideration in the limitations of the updated file (l. 504-509). Disentangling the contribution of the cue letter from the related imagery activity is not possible for the experimental setup itself; as described in the Appendix A.1, a cue letter is displayed for each trial of each run; however, as possible evidence, we used the mismatched stimulus-letter vision trials from NSD-Imagery in order to evaluate on them performance of DynaDiff model, i.e, without our Alignment module; results are reported in next Table:
These trials only show that, during perception, decoding follows the displayed image rather than the cue; they do not address imagery. These results are reported only here.