acceptodds
Under review as a conference paper at ICLR 2027

MindRefine: Language-Informed Semantics and Perceptual Refinement for Cross-Subject Brain Visual Decoding

Abstract

Cross-subject functional magnetic resonance imaging (fMRI) visual decoding remains challenging when only limited paired data are available for a target subject. Existing methods mainly align brain responses with pretrained visual representations, leaving semantic structure only implicitly constrained, and rely on reconstruction pipelines that lack sample-specific perceptual correction. We propose MindRefine, a framework that improves limited-data fMRI visual decoding through representation learning and reconstruction refinement. Language-informed semantic regularization uses structured descriptions generated by a vision-language model (VLM) exclusively from training images to provide gated multi-channel semantic supervision and semantic-neighborhood distillation. This introduces explicit semantic constraints without using language information at inference. Feature-space reconstruction refinement keeps the trained decoder frozen, estimates sample-specific perceptual and semantic targets from a paired training reference bank, and refines each reconstruction through stage-wise feature-space optimization. Extensive experiments on the Natural Scenes Dataset in the limited-data regime demonstrate that MindRefine improves both low-level reconstruction fidelity and high-level perceptual consistency while preserving competitive retrieval performance. Additional experiments under the full-data setting further confirm the effectiveness and robustness of the framework. The code is available at https://anonymous.4open.science/r/MindRefine.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, visual decoding, semantic regularization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_97grOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper utilizes structured language descriptions of training images to constrain brain representations, and then uses a training reference library to predict multi-network visual features of test samples. By optimizing and correcting the initial reconstruction image by image, the cross-subject visual decoding index is improved under limited data conditions.

Critical strengths
  1. Improvements are not limited to optimizing a single network. In Table 2, after adding FSR, Inception, which was not included in its optimization objectives, still saw an improvement from 87.2% to 87.8%, with PixCorr and SSIM also showing improvements.

  2. Post-processing is applied to another decoder. Figure 4 and the appendix demonstrate the application of similar modifications to MindEye2, illustrating that FSR has the potential to be used across at least two decoding systems, simultaneously covering both single-shot and full-data conditions.

Critical weaknesses
  1. Existing evidence is insufficient to explain the score improvement as a broad perceptual enhancement. Appendix C clearly states that the first stage of FSR optimizes CLIP, EfficientNet-B1, and SwAV, while the second stage adds AlexNet-2 and AlexNet-5. This directly overlaps with the reported five feature evaluations. Although the optimization target comes from the training reference library predictions, rather than the test ground truth, the final images are directly adjusted to conform to the representations of these networks, thus they cannot simultaneously serve as sufficiently independent effect validations. Table 2 particularly illustrates this coupling: from no FSR to the full method, Eff decreases from 0.797 to 0.758, and SwAV decreases from 0.468 to 0.416; when only CLIP correction is used, these two only reach 0.793 and 0.462. This result is effectively compatible with multi-network optimization, but it cannot alone prove that it is more consistent with human perception or more faithful to brain signals. Page 20 of the paper uses Eff and SwAV performance to support the "non-biased towards a single embedding space" explanation, ignoring that they are inherently part of the optimization target.

  2. LISR combines five VLM text channels, sample gating, manually set channel priors, and semantic neighborhood distillation. Compared to simple image description, uniform text averaging, or existing visual relation distillation, the paper doesn't provide a direct comparison of what this complex design actually adds. In Table 2, removing LISR actually increases CLIP from 83.6% to 83.9%, and brain retrieval from 81.8% to 82.4%. The complete system performs well on other reconstruction metrics, supporting a quality trade-off, but it doesn't directly prove that language constraints enhance semantic integrity. The authors have already proposed an ablation method of "removing the entire LISR," but it doesn't answer whether the five-channel structure, gating, and neighborhood distillation are necessary.

  3. The core of FSR is training feature regression plus image feature inversion, lacking a strong contrast sufficient to establish method incrementality and sample specificity. Equation 6 uses normalized brain embedding linear kernel ridge regression. For a fixed representation, this is equivalent to fitting a ridge regression mapping on that representation and then predicting different visual network features. Equation 7 then optimizes the image through feature loss, pixel anchoring, and total variation. This is a reasonable combination, but these operations themselves are not new learning principles. The key issue is that the paper does not independently evaluate whether the predicted features are closer to the correct stimulus than the initially reconstructed features, nor does it compare direct brain signals to multi-network feature regression, nearest neighbor reference, and general post-processing with the same budget. Figure 4 provides useful support for the transfer experiment of MindEye2, but it does not re-rank the data after assigning the same FSR to the strongest-fitting baseline.

  4. Section 3.3 refers to the FSR freezing decoder; the main text defines "not using FSR" as directly evaluating the output before correction, and "CLIP correction only" as only changing the feature constraints. Appendix L further clarifies that the retrieval branch and brain representation remain unchanged between the initial image, fixed fusion, and FSR. However, the average image retrieval rates for the three conditions in Table 2 are 84.0%, 84.0%, and 84.2%, respectively, and the brain retrieval rates are 81.6%, 81.3%, and 81.8%. In Table 5, Subject 2's "CLIP-only correction" brain retrieval rate was 90.4%, while the complete method and the method without FSR both achieved 92.4%, a difference of 2 percentage points, which cannot be explained by rounding.

  5. The appendix clearly states that each configuration was trained only once, with a fixed seed of 42; the thirty retrieval repetitions corresponded only to candidate sampling, not thirty independent training iterations. Therefore, the differences in Table 1, such as 83.6% for CLIP versus 83.5% for Duala and 81.8% versus 81.1% for brain retrieval, do not indicate any fluctuations in training or target data selection. The paper claims that hyperparameters are fixed across subjects, but this does not mean the selection process is independent of testing. Appendix K supports default semantic weights and correction strengths based on "balanced performance" across multiple evaluation metrics, but it does not specify which validation set these choices came from, or whether the default values ​​were determined before or after reviewing test results. The full-data experiment only compares MindEye2 and NeuroFlow, which is insufficient to demonstrate a broad advantage over all strongly fitted methods. The paper should clearly define which conclusions are descriptive results from a single configuration and which have been validated for stability.

  6. Algorithm 1 first freezes the trained decoder and residual label adapter, then fits a lightweight low-rank label adapter. The text and appendix do not fully explain the latter's rank, structure, training objective, number of fitting steps, or whether it participates in retrieval. The parameter table lists an additional 2.97M for the residual adapter, semantic gating, and projection head, but does not specify whether this post-fitting module is included. Similarly, the initial reconstruction comes from fixed fusion and "adaptive fusion of training calibration," but the appendix only gives a combined weight of 0.6 and 0.4, without fully defining the input, form, and calibration objective of the adaptive fusion.

Questions for authors
  1. If an evaluation network is removed from the FSR each time and then tested on that network, does the improvement remain?

  2. What levels of performance are achieved with single caption, five-channel uniform averaging, fixed gating, and neighborhood distillation only? Do the accuracy rates for objects, attributes, and relationships improve? What is the actual distribution of gating selection?

  3. How much is the error of predicted features reduced relative to the initial image features? What are the effects of direct ridge regression, nearest neighbor reference, and category prototype? Given the same FSR as the strongest baseline, how much does the training module contribute?

  4. Why does post-processing ablation of frozen brain representations change the retrieval score?

  5. Which independent validation or calibration set determines the semantic weights, corrected weights, and fusion coefficients?

  6. What is the complete definition of the post-fit adapter and adaptive fusion, and are they included in the parameter table? What are the end-to-end time, GPU memory, and storage costs after including annotations, reference libraries, and 64-step graph-by-graph optimization?

9 comments

We thank the reviewer for the careful evaluation and constructive suggestions. We have revised the manuscript, with changes highlighted in blue. Revision locations are given below. The response is organized into seven comments. Comments 1–4 address Questions 1–4, respectively. Question 5 is addressed in two comments. Comment 5 covers training-seed stability and cross-subject consistency, and Comment 6 covers validation and the expanded full-data comparison. Comment 7 addresses Question 6 on module definitions, parameter counts, and computational cost.

1. Evaluation after excluding a feature network from FSR

We agree that improvements in feature spaces included in the FSR objective require complementary evaluation. We therefore removed each evaluation network from both FSR stages and evaluated the resulting images in the omitted space. For AlexNet, both layers were removed together.

Network excluded from FSRHeld-out metricBefore FSR (Table 6)After exclusion
CLIPCLIP ↑83.0%84.0%
EfficientNet-B1EfficientNet-B1 ↓0.7810.775
SwAVSwAV ↓0.4530.443
AlexNetAlexNet(2) ↑91.1%90.9%
AlexNetAlexNet(5) ↑95.3%95.4%

Four of the five held-out metrics improve relative to the submitted pre-FSR reference, while AlexNet(2) decreases by 0.2 percentage points.

We also replaced the entire feature objective with DINOv2 ViT-B/14 CLS alignment in both stages, retaining the 40+24 optimization steps, pixel anchors, training reference bank, and blending settings. DINOv2 is outside the original evaluation suite.

MetricBefore FSR (Table 6)After DINOv2-only FSR
PixCorr ↑0.2640.276
SSIM ↑0.4120.422
AlexNet(2) ↑91.1%90.8%
AlexNet(5) ↑95.3%95.4%
Inception ↑88.4%88.7%
CLIP ↑83.0%84.6%
EfficientNet-B1 ↓0.7810.777
SwAV ↓0.4530.444
Image retrieval ↑93.8%93.8%
Brain retrieval ↑92.3%92.3%

This replacement improves seven of eight reconstruction metrics relative to the submitted reference. Inception and CLIP increase by 0.3 and 1.6 percentage points, respectively. Retrieval uses fixed embeddings and candidate sets and therefore remains unchanged. Together, these controls support improvements beyond the feature networks directly optimized by FSR. We have revised the discussion of EffNet-B1 and SwAV to identify their overlap with the original objective and have added the held-out evaluations to assess generalization across feature spaces.

We have added the independent-network comparison in Section 4.4, Table 3, and the complete network-exclusion and DINOv2-only results in Appendix L.2, Tables 8 and 9.

2. LISR components, semantic accuracy, and gate distributions

We evaluated caption-only supervision, five channels with uniform weights, five channels with fixed prior weights, and neighborhood distillation alone. The table reports Inception and CLIP identification before and after FSR, alongside the published full-method results from Table 6.

LISR configurationInception before / after FSRCLIP before / after FSR
Caption only87.1% / 87.7%81.6% / 84.2%
Five channels, uniform weights87.0% / 87.8%81.5% / 84.7%
Five channels, fixed prior weights87.3% / 87.8%81.6% / 84.1%
Neighborhood distillation only, uniform weights87.0% / 87.8%81.5% / 84.5%
Ours88.4% / 89.1%83.0% / 85.3%

Among the additional variants, five-channel uniform supervision improves post-FSR CLIP identification over caption-only supervision, and neighborhood distillation alone also provides effective semantic supervision. The full method combines direct alignment and neighborhood supervision with learned channel weights.

We additionally evaluated object, attribute, and relation accuracy using blinded GPT-6.1 Sol visual question answering on all 1,000 test images. We generated 6,000 questions from the original stimuli, with two questions per image for each category (2,000 object, 2,000 attribute, and 2,000 relation questions), and fixed the question bank before scoring reconstructions. The evaluator received anonymous reconstructed images and the same questions, with missing or unclear queried content counted as incorrect.

MethodObject ↑Attribute ↑Relation ↑
MindEye251.65%14.50%9.85%
Duala51.90%14.10%10.85%
w/o LISR52.35%16.15%13.00%
MindRefine (Ours)53.65%17.15%14.50%

With the stimulus-bank loss retained, adding LISR increases object, attribute, and relation scores by 1.30, 1.00, and 1.50 percentage points, respectively. These model-based semantic measurements complement the reconstruction metrics and directly address the content of the reconstructed images.

The following gate statistics were computed from the full Subject 1 checkpoint over the 1,000 test brain embeddings.

ChannelMean weight
Caption22.0%
Scene29.5%
Object20.3%
Attribute13.7%
Relation14.5%

The gate assigns substantial average weight to all five channels. Scene receives the largest mean weight (29.5%), followed by Caption (22.0%) and Object (20.3%). Caption and Scene together account for 51.5% of the average weight, while Object, Attribute, and Relation account for 48.5%. This distribution shows that the semantic supervision draws on both global context and detailed information about entities, their properties, and their relationships.

We have clarified the channel design in Section 3.2 and summarized the semantic findings in Section 4.3. Appendix N, Tables 14–16, reports the component ablations, semantic evaluation, and gate distributions.

3. Feature-target accuracy, alternative estimators, and baseline refinement

For a fixed normalized brain representation, linear-kernel ridge regression is equivalent to primal ridge regression. With training embeddings ZZ and training visual features YY, the primal mapping is W=(Z⊤Z+λI)−1Z⊤YW=(Z^\top Z+\lambda I)^{-1}Z^\top Y. The dual form in Eq. 6 makes prediction practical with a bank of 536 training examples. FSR uses this mapping to estimate sample-specific targets in several feature spaces, then optimizes an existing reconstruction with pixel anchoring in two stages.

To test whether FSR is guided toward more accurate features, we compare two feature vectors against the same ground-truth reference for each test stimulus. The first is extracted from the initial reconstruction before FSR. The second is the target predicted from the test brain embedding and the paired training bank. For each network ff, the reference is the feature f(Ii)f(I_i) of the true stimulus. We compute 1−cos⁡(f(I^i(0)),f(Ii))1-\cos(f(\hat I_i^{(0)}),f(I_i)) for the initial image and 1−cos⁡(t^i,f(Ii))1-\cos(\hat t_i,f(I_i)) for the predicted target, then average over the 1,000 test stimuli.

The Initial-image feature error column measures the error of features already present in the initial reconstruction. The next four columns measure the errors of target vectors supplied by the four estimators. All entries are cosine distances under the feature preprocessing used by FSR, and lower values indicate closer agreement with the true stimulus. Test stimulus features are used only to evaluate these errors. For example, CLIP error decreases from 0.386 for initial-image features to 0.261 for KRR targets.

Feature spaceInitial-image feature error ↓Raw-fMRI ridge ↓Nearest reference ↓Category prototype ↓Brain-embedding KRR (Ours) ↓
AlexNet(2)0.7430.6740.7870.6830.605
AlexNet(5)0.7800.7340.8360.7710.703
CLIP0.3860.2850.3860.3150.261
EfficientNet-B10.7640.6640.7580.7100.657
SwAV0.3180.2530.3490.2860.243

Brain-embedding KRR achieves the lowest mean target error among the four estimators in all five feature spaces. It also predicts features closer to the true stimulus than those extracted from the initial reconstruction, directly supporting the use of its predictions as refinement targets. Raw-fMRI ridge regularization was selected by five-fold cross-validation on training stimuli. Nearest-reference targets use brain-embedding similarity, while category prototypes average training features within categories and assign queries through training brain centroids.

We then used the same 64-step FSR budget with each target estimator.

Target estimatorInception ↑CLIP ↑EfficientNet-B1 ↓SwAV ↓
Raw-fMRI ridge88.4%81.8%0.7570.414
Nearest reference88.5%84.1%0.7590.415
Category prototype88.5%82.7%0.7610.416
Brain-embedding KRR (Ours)89.1%85.3%0.7420.401

The target-prediction and reconstruction results consistently favor brain-embedding KRR. Its lower target errors are accompanied by the highest Inception and CLIP identification and the lowest EfficientNet-B1 and SwAV distances among the four estimators. To test sample specificity under the same optimization budget, we also used a shared mean target and shuffled the sample-specific targets. Their Inception/CLIP scores were 88.3%/76.6% and 88.2%/77.9%, respectively, supporting the value of assigning each sample its own brain-predicted targets.

We further applied the same FSR procedure to Duala and MindTuner, using the same training-bank construction, feature objectives, ridge regularization, optimization steps, and blending.

Method and outputPixCorr ↑SSIM ↑Inception ↑CLIP ↑EfficientNet-B1 ↓SwAV ↓
Duala Initial0.2450.40885.9%80.5%0.8110.470
Duala Official Fixed Blend0.2530.41886.6%85.1%0.7680.433
Duala After FSR0.2640.42388.1%85.5%0.7460.417
MindTuner Initial0.2390.43284.1%79.3%0.8050.472
MindTuner Official Fixed Blend0.2620.42285.8%84.6%0.7740.433
MindTuner After FSR0.2670.42887.5%85.2%0.7540.416
MindRefine (Ours)0.2760.42589.1%85.3%0.7420.401

FSR improves Inception over Initial by 2.2 percentage points for Duala and 3.4 for MindTuner. MindRefine's w/o FSR ablation already reaches 88.4% Inception, and adding FSR raises it to 89.1%, retaining a lead over both refined baselines. These comparisons support contributions from both the decoding pipeline and FSR.

We have clarified KRR in Section 3.3 and added the target-quality, estimator, and baseline comparisons in Appendix L.3, Tables 10–12.

Reviewer_uUQ2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper targets cross-subject fMRI-to-image reconstruction on NSD when only about one scan session of paired data is available for the target subject. Building on MindEye2, it adds two components. Language-informed semantic regularization (LISR) prompts a VLM (Qwen3-VL-Plus) for five-channel structured descriptions of the training stimuli and uses them as training-time supervision: a gated multi-channel cosine alignment to CLIP text embeddings (Eq. 1-2), a KL term pulling the gate toward a fixed prior, and a semantic-neighborhood soft-label distillation whose targets are text-text similarities with the paired sample excluded (Eq. 3-4). No language input is used at inference. Feature-space reconstruction refinement (FSR) freezes the trained decoder, predicts per-sample feature targets from the brain embedding by linear-kernel ridge regression over a paired training reference bank (Eq. 5-6), and then optimizes the pixels of the already-decoded image in two stages with an MSE anchor, total variation, and cosine terms in several DNN feature spaces, blending the result back with coefficient alpha = 0.10 (Eq. 7-8). Experiments cover subjects 1, 2, 5, 7 on the 1,000-image shared test set in both the 1-session and 40-session settings, with eight reconstruction metrics plus image/brain top-1 retrieval, five 1-session baselines, and an ablation over LISR, L_stim, FSR and CLIP-only refinement.

Critical strengths
  1. The evaluation uses the full standard NSD protocol: all four subjects, the 1,000-image shared test set (p15), both the 1-session and 40-session settings (Tables 1 and 3), five 1-session baselines, and per-subject results (Tables 5 and 6, p17-18). The comparisons are therefore made on the same footing as prior work, and in the 1-session setting the gain over MindEye2 is large (PixCorr 0.239 vs 0.195, brain retrieval 81.8% vs 57.4%, Table 1).

  2. The anonymous code linked in the abstract implements the components described in the paper, and the hyperparameters match App. C, including the LISR losses and weights, the KRR ridge of 0.05, and both FSR stages. This makes the method reproducible.

Critical weaknesses
  1. FSR is optimized against the evaluation networks.

At test time, FSR changes each reconstructed image so that its features in AlexNet, CLIP, EfficientNet-B1 and SwAV move toward the predicted targets (App. C, p15). Five of the eight reported metrics are computed with these same networks (p6), and the released code uses the same layers for both. The paper does not mention this overlap.

Because of this, higher scores on these metrics do not by themselves show better images. The ablation is consistent with this concern. With FSR, EfficientNet distance drops from 0.797 to 0.758 and SwAV distance from 0.468 to 0.416, while Inception, the high-level metric that FSR does not optimize, only moves from 87.2% to 87.8% (Table 2, p8). Without FSR, the model is not better than Duala on most metrics (Table 2 vs. Table 1). FSR was also applied only to MindRefine and MindEye2, not to the other baselines. The advantage over the strongest baseline therefore rests on a step that targets the scoring networks and that the baselines did not receive.

  1. Some prior work on FSR is not cited Optimizing image pixels so that their DNN features match features decoded from brain activity is the main method of Shen et al., "Deep image reconstruction from human brain activity" (PLoS Comput Biol, 2019). Brain-Diffuser (arXiv:2303.05334) predicts feature targets from fMRI with ridge regression, Second Sight (arXiv:2306.00927) refines reconstructions iteratively, and MindDiffuser (arXiv:2303.14139) and BrainRAM (ACM MM 2024) refine a first reconstruction in a second stage using brain-decoded features or retrieved references. None of these is cited, and Related Work (p3) does not compare FSR with them. It is therefore unclear what FSR adds beyond using kernel ridge regression to set the targets and starting from an existing reconstruction.
Questions for authors
  1. Please report results with FSR restricted to feature networks that are not used for evaluation (e.g., DINOv2 or LPIPS), for all metrics in Tables 1 and 2. If the improvements persist under this setting, my main concern about the evaluation would be addressed.
  2. Please report results for Duala and MindTuner with the same FSR procedure applied. This comparison is needed to determine whether the advantage in Table 1 comes from the trained model or from the refinement step.
  3. L_stim is taken from Duala, and the released code derives its pseudo-labels from the VLM annotations. Could arthors report a variant trained without any VLM annotations, and clarify the contribution of LISR beyond the Duala losses?
  4. Please define the adaptive-fusion reconstruction and the second low-rank token adapter (p5), including their training objectives and the data they are fitted on. Please also discuss Shen et al. Deep image reconstruction from human brain activity. (2019), Brain-Diffuser, Second Sight, MindDiffuser and BrainRAM in Related Work, and state what FSR adds relative to each.
5 comments

We thank the reviewer for the careful evaluation and constructive suggestions. We have revised the manuscript, with changes highlighted in blue. Revision locations are given below. The response has three comments. Comment 1 addresses Questions 1–2, Comment 2 addresses Question 3, and Comment 3 addresses Question 4.

1. FSR with feature networks outside the evaluation suite

The original FSR objective overlaps with five evaluation metrics through CLIP, AlexNet(2), AlexNet(5), EfficientNet-B1, and SwAV. Targets are predicted from the training reference bank, with test stimulus features used only for scoring.

We replaced all feature losses in both stages with DINOv2 ViT-B/14 CLS alignment, outside the evaluation suite. We fixed the decoder, initial images, training bank, ridge (0.05), 40+24 steps, pixel anchors, and blend (0.10 per stage). The table compares the submitted w/o FSR reference with DINOv2-only FSR on Subject 1, one session, and 1,000 test images.

MetricBefore FSRAfter DINOv2-only FSR
PixCorr ↑0.2640.276
SSIM ↑0.4120.422
AlexNet(2) ↑91.1%90.8%
AlexNet(5) ↑95.3%95.4%
Inception ↑88.4%88.7%
CLIP ↑83.0%84.6%
EfficientNet-B1 ↓0.7810.777
SwAV ↓0.4530.444
Image retrieval ↑93.8%93.8%
Brain retrieval ↑92.3%92.3%

DINOv2-only FSR improves seven of eight reconstruction metrics over the submitted reference. CLIP and Inception rise by 1.6 and 0.3 percentage points, while AlexNet(2) decreases by 0.3. Retrieval uses fixed embeddings and candidates. These gains support refinement with a network outside the evaluation suite.

As a complementary control, we excluded each evaluation network from both FSR stages and scored its held-out features. Both AlexNet layers were excluded together.

Network excluded from FSRHeld-out metricBefore FSR (Table 6)After exclusion
CLIPCLIP ↑83.0%84.0%
EfficientNet-B1EfficientNet-B1 ↓0.7810.775
SwAVSwAV ↓0.4530.443
AlexNetAlexNet(2) ↑91.1%90.9%
AlexNetAlexNet(5) ↑95.3%95.4%

Four of five held-out metrics improve, with a 0.2-point decrease in AlexNet(2).

We have added the overlap discussion and DINOv2-only results in Section 4.4, Table 3, and both complete controls in Appendix L.2, Tables 8 and 9.

2. Applying the same FSR procedure to Duala and MindTuner

We applied the same two-stage FSR to Duala and MindTuner on Subject 1 in the one-session setting, using all 1,000 test images. We fixed the 536 paired training stimuli, KRR regularization, feature spaces and weights, 40+24 steps, pixel anchors, and blending. Official Fixed Blend reports the baselines’ 75:25 blends.

MetricDuala InitialDuala Official Fixed BlendDuala After FSRMindTuner InitialMindTuner Official Fixed BlendMindTuner After FSRMindRefine (Ours)
PixCorr ↑0.2450.2530.2640.2390.2620.2670.276
SSIM ↑0.4080.4180.4230.4320.4220.4280.425
AlexNet(2) ↑89.9%90.8%91.2%88.3%90.6%90.9%91.3%
AlexNet(5) ↑94.2%95.3%95.7%93.5%94.9%95.3%95.7%
Inception ↑85.9%86.6%88.1%84.1%85.8%87.5%89.1%
CLIP ↑80.5%85.1%85.5%79.3%84.6%85.2%85.3%
EfficientNet-B1 ↓0.8110.7680.7460.8050.7740.7540.742
SwAV ↓0.4700.4330.4170.4720.4330.4160.401
Image retrieval ↑94.8%94.8%94.8%94.2%94.2%94.2%93.8%
Brain retrieval ↑91.2%91.2%91.2%87.4%87.4%87.4%92.3%

Compared with Initial, FSR improved all eight reconstruction metrics for Duala and seven for MindTuner, with a small decrease in MindTuner’s SSIM. Inception increased by 2.2 and 3.4 percentage points, and CLIP increased by 5.0 and 5.9 percentage points, respectively. Image and brain retrieval scores stayed fixed because FSR only refines reconstructed images, while retrieval uses the same embeddings and candidate sets. Compared with Official Fixed Blend, After FSR scores were better on all eight reconstruction metrics for both baselines. Inception was higher by 1.5 percentage points for Duala and 1.7 for MindTuner, and CLIP was higher by 0.4 and 0.6 percentage points, respectively.

Without FSR, MindRefine reaches 88.4% Inception (Table 6), above Official Fixed Blend for Duala (86.6%) and MindTuner (85.8%). FSR adds 0.7 points to reach 89.1%, ahead of both refined baselines (88.1% and 87.5%). Therefore, both the decoding pipeline and FSR contribute to MindRefine’s reconstruction gains on Subject 1.

We have added the baseline comparison in Appendix L.3, Table 12; the MindRefine ablations appear in Table 6, Section 4.3, and Appendix I.

3. Training without VLM annotations and isolating LISR

VLM annotations enter training through two paths. The Duala-derived Lstim\mathcal{L}_{\mathrm{stim}} uses pseudo-labels derived from the annotations. LISR uses five-channel text embeddings directly for gated semantic alignment and semantic-neighborhood distillation.

We trained a No VLM variant that disables LISR and removes the annotation-dependent terms from Lstim\mathcal{L}_{\mathrm{stim}}. It retains visual supervision and exact-ID contrastive supervision indexed by training stimulus IDs.

MetricNo VLMw/o LISRw/o Lstim\mathcal{L}_{\mathrm{stim}}MindRefine (Ours)
PixCorr ↑0.2640.2660.2630.276
SSIM ↑0.4140.4220.4210.425
AlexNet(2) ↑89.8%91.0%91.1%91.3%
AlexNet(5) ↑95.6%95.3%95.5%95.7%
Inception ↑87.9%87.9%88.7%89.1%
CLIP ↑84.8%85.5%84.8%85.3%
EfficientNet-B1 ↓0.7550.7540.7470.742
SwAV ↓0.4020.4060.4010.401
Image retrieval ↑89.8%91.7%93.8%93.8%
Brain retrieval ↑87.4%91.2%93.8%92.3%

Without any VLM annotations, the model reaches 87.9% Inception and 84.8% CLIP identification. With Lstim\mathcal{L}_{\mathrm{stim}} and its annotation-derived pseudo-labels held fixed, adding LISR raises Inception from 87.9% to 89.1%, PixCorr from 0.266 to 0.276, and brain retrieval from 91.2% to 92.3%. These comparisons demonstrate the additional contribution of LISR beyond the Duala-derived auxiliary supervision.

We additionally evaluated an ablation that removed only LISR while retaining the stimulus-bank loss. Under a blinded, model-based visual question-answering evaluation, object, attribute, and relation scores increased from 52.35%, 16.15%, and 13.00% without LISR to 53.65%, 17.15%, and 14.50% with the full method. These point estimates provide complementary evidence of LISR’s contribution to the semantic content of the reconstructions.

We have added the annotation-free control in Appendix O, Table 17, and the semantic evaluation in Appendix N, Table 15. Section 4.3 summarizes the ablation and semantic findings.

4. Adaptive fusion, the post-fit token adapter, and prior work

Adaptive fusion. Let RR and BB be the raw diffusion and blurry reconstructions. For each of 100 target-subject training stimuli, we select the weight aa that maximizes the pixel correlation between aR+(1−a)BaR+(1-a)B and the corresponding training image over a grid from 0.50 to 0.95. A ridge regressor (λ=0.01\lambda=0.01) predicts this weight from 12 image-pair statistics, including discrepancy, sharpness, contrast, saturation, and intensity. At inference, the predicted weight aa is clipped to [0.50,0.95][0.50,0.95] and yields Iadapt=aR+(1−a)BI_{\rm adapt}=aR+(1-a)B. The input to FSR is 0.60(0.75R+0.25B)+0.40Iadapt0.60(0.75R+0.25B)+0.40I_{\rm adapt}. Calibration uses target-subject training pairs exclusively. At inference, the regressor uses only statistics computed from RR and BB.

Post-fit token adapter. After freezing the decoder and its training-time residual adapter, we fit a rank-32 residual adapter on 1,664-dimensional brain tokens. It applies affine-free layer normalization, a down-projection, GELU, and an up-projection, and adds the output with scale 0.20. The loss combines forward contrastive alignment to the training image-token bank, within-batch symmetric contrastive alignment, paired cosine distance, and an identity MSE penalty with respective weights 1, 8, 0.05, and 0.002. For Subject 1 in the one-session setting, we use 429 training stimuli for gradient updates and the other 107 for checkpoint selection, with 120 epochs and a batch size of 32. After checkpoint selection, the adapter is frozen before constructing the FSR reference bank and processing test samples. Its output tokens also enter retrieval calibration, which fits a low-rank residual mapping to align brain tokens with CLIP image tokens using target-subject training pairs. FSR updates reconstructed image pixels while these retrieval mappings remain fixed. We have clarified these modules and their fitting data in Section 3.3 and Appendix M.

Relation to prior work. We have added these five papers to Related Work and clarified how FSR builds on feature decoding and reconstruction refinement. FSR estimates sample-specific, multi-network feature targets from frozen brain embeddings using a paired target-subject training bank, then refines an existing reconstruction through two stages of anchored pixel-space optimization (40 and 24 steps).

  • Deep image reconstruction from human brain activity (Shen et al., PLoS Computational Biology, 2019) reconstructs images by matching decoded DNN features through pixel-space or generator-latent optimization. FSR starts from an existing fused reconstruction and applies two stages of anchored pixel-space optimization toward feature targets estimated from a paired training bank.
  • Brain-Diffuser (Ozcelik and VanRullen, arXiv:2303.05334, 2023) uses regression to predict VDVAE latents and CLIP conditioning for image-to-image diffusion. FSR performs pixel refinement after the frozen decoder has generated its image.
  • Second Sight (Kneeland et al., arXiv:2306.00927, 2023) refines a distribution by generating and selecting images with a voxelwise encoding model. FSR updates one image toward targets estimated from its brain embedding.
  • MindDiffuser (Lu et al., arXiv:2303.14139, 2023) uses decoded CLIP visual features to optimize diffusion latents and text-conditioning embeddings. FSR optimizes the already decoded image pixels using targets in several fixed visual feature spaces.
  • BrainRAM (Xie et al., ACM MM, 2024) retrieves external LAION image-text pairs to refine embeddings before diffusion. FSR estimates targets by kernel ridge regression over paired target-subject training stimuli and uses them after generation.

We have added the prior-work comparisons in Section 2 and clarified the modules in Section 3.3. Appendix M provides their definitions and fitting objectives, and Appendix Q, Table 21, specifies the calibration data and selection procedures.

Reviewer_ehXEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MindRefine, a framework for cross-subject fMRI visual decoding under limited-data settings. The framework consists of two main modules, LISR and FSR. LISR uses structured descriptions generated by a VLM from training images to enhance brain-side visual representations through gated semantic supervision and semantic-neighborhood distillation. FSR estimates sample-specific perceptual and semantic feature targets from a paired training reference bank and improves image reconstruction quality through stage-wise feature-space optimization. Experiments on NSD demonstrate that MindRefine improves overall reconstruction quality under both limited-data and full-data settings while preserving competitive retrieval performance.

Critical strengths
  1. The authors improve cross-subject fMRI visual decoding from both representation learning and reconstruction refinement. LISR enhances brain-side visual representations through explicit semantic constraints, while FSR improves the final reconstructions through feature-space optimization. The two modules complement each other within the proposed framework.
  2. The authors use a VLM to generate five semantic channels, including Caption, Scene, Object, Attribute, and Relation, and provide semantic supervision through a sample-dependent gate and semantic-neighborhood distillation. This approach introduces explicit semantic constraints beyond visual-token alignment and provides a new perspective on brain-side representation learning under limited-data settings. Overall, I consider these contributions meaningful and valuable. They demonstrate the effectiveness of the proposed approach and provide strong support for the paper's contributions. These strengths are the main reasons for my positive assessment of this work.
Critical weaknesses
  1. The paper does not sufficiently justify the selection of the five semantic channels in LISR or explain whether they adequately cover the semantics required for visual decoding. Moreover, LISR relies on VLM-generated annotations, making its effectiveness dependent on annotation quality. Experiments with different VLMs would help assess its robustness.
  2. Although MSE regularization and conservative blending are used, FSR applies the same refinement procedure to all test samples, which may degrade already accurate reconstructions. The authors should analyze its negative effects across samples. For weakness 1: LISR is one of the core components of the proposed method, but its key design choices lack sufficient justification and experimental validation. This makes it difficult to determine whether the current design is appropriate and whether it has inherent limitations. For weakness 2: This design choice appears relatively straightforward, but its effectiveness and limitations are not sufficiently discussed or validated. A more thorough explanation and analysis of this issue would make me more confident in the proposed method.
Questions for authors
  1. Why were Caption, Scene, Object, Attribute, and Relation selected as the five semantic channels in LISR? Are these channels sufficient to cover the semantic information required for visual decoding?
  2. Since LISR relies on structured semantic descriptions generated by a VLM, how does the choice of VLM affect the quality of semantic annotations and the final decoding performance? Would using different VLMs lead to significant differences in LISR's effectiveness and robustness?
  3. Does applying the same FSR refinement procedure to all test samples potentially degrade already accurate reconstructions? How do the authors assess this potential negative effect?
2 comments

We thank the reviewer for the careful evaluation and constructive suggestions. We have revised the manuscript, with changes highlighted in blue. Revision locations are given below.

1. Why these five semantic channels, and what do they cover?

Caption and Scene describe global content and setting; Object, Attribute, and Relation specify entities, properties, and interactions or spatial arrangements. These fields expose details a caption may omit; image supervision preserves visual detail. For example, PRISM's keyword search favored spatial layout and relative position (Huang et al., “Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI,” ICLR 2026).

Two-way identification before/after FSR on Subject 1 (one session):

LISR configurationInception before / after FSRCLIP before / after FSR
Caption only87.1% / 87.7%81.6% / 84.2%
Five channels, uniform weights87.0% / 87.8%81.5% / 84.7%
Five channels, fixed prior weights87.3% / 87.8%81.6% / 84.1%
Neighborhood distillation only, uniform weights87.0% / 87.8%81.5% / 84.5%
Ours88.4% / 89.1%83.0% / 85.3%

With uniform weights, five channels improved both post-FSR scores over caption-only supervision, especially CLIP. This supports including semantic cues that a single caption may omit.

We have clarified the channel design in Section 3.2 and added the ablations in Appendix N, Table 14.

2. How sensitive is the method to the choice of annotator?

To test annotator sensitivity, we used GPT-6.1 Sol (Sol) and Qwen3-VL-8B-Instruct (8B) on the same 536 training images with the original Qwen3-VL-Plus (Plus) prompt. We compared caption cosine similarities in CLIP ViT-L/14 text space.

Model pairSame-image cosineShuffled-image cosineDifference
8B / Plus0.77020.2150+0.5552
Sol / Plus0.75670.2366+0.5201
8B / Sol0.71550.2273+0.4882

The gaps show image-specific agreement, including with the 8B model, on this visual-description task. The code and prompt (Figure 7) are public. We will release the trained checkpoints and each VLM's captions.

GPT-6 Astra audited anonymized annotations on 100 randomly sampled training images, using the stimulus as primary evidence and all official COCO captions as references. Channel scores average per-image support rates; denominators include supported, contradicted, and unverifiable claims.

Channel8BSolPlus
Caption91.09%97.70%93.55%
Scene83.00%96.67%85.00%
Object95.50%98.82%94.95%
Attribute89.82%99.42%90.56%
Relation87.85%96.83%89.75%

8B and Plus differed by at most 2.5 pp across channels (all paired image-bootstrap 95% CIs include zero). Sol had higher support rates. Plus had more assessed claims (2,665 vs. 1,759) and more supported claims (2,434 vs. 1,723). Downstream reconstruction scores nevertheless remain close.

Subject 1 one-session results follow; Plus is from Table 6.

Metric8BSolPlus
PixCorr ↑0.2710.2740.276
SSIM ↑0.4230.4230.425
Inception ↑88.6%88.9%89.1%
CLIP ↑85.2%85.6%85.3%

Across the three annotators, the ranges were 0.005 for PixCorr, 0.002 for SSIM, 0.5 pp for Inception, and 0.4 pp for CLIP. VLM choice had a limited effect on reconstruction scores in this experiment.

We have added these comparisons in Appendix P, Tables 18–20; the prompt is in Appendix D, Figure 7.

3. Can FSR degrade an already accurate reconstruction?

FSR estimates targets from brain embeddings and paired training references via kernel ridge regression. An inaccurate target can draw a good reconstruction away from the stimulus, and joint optimization can help one feature space at another's expense. The pixel anchor and 0.10 blend per stage limit update size.

Across 1,000 test images, higher identification/similarity or lower distance after FSR counted as improvement; the reverse counted as degradation. DINOv2 and Inception are outside the FSR objective.

MetricImprovedDegradedTied
CLIP ↑861101 (10.1%)38
EfficientNet-B1 ↓772228 (22.8%)0
SwAV ↓95149 (4.9%)0
AlexNet(2) ↑685105 (10.5%)210
AlexNet(5) ↑612167 (16.7%)221
DINOv2 cosine812188 (18.8%)0
Inception identification79459 (5.9%)147

Applying the same FSR procedure to every test image can therefore make an already accurate reconstruction worse. However, in our paired analysis, improvements outnumbered declines on all seven metrics, including DINOv2 and Inception, which are outside the FSR objective. The aggregate comparison with w/o FSR in Table 2 also shows average gains across the reported reconstruction metrics.

We have added the sample-wise analysis in Appendix L.4, Table 13, complementing Section 4.4.

Dear Reviewer,

Thank you again for your encouraging assessment and thoughtful suggestions. Your feedback has helped us clarify the semantic-channel design, examine sensitivity to the annotation model, and assess the sample-wise benefits and potential drawbacks of FSR. We have addressed these points in our responses and revised manuscript. As the author-response deadline approaches, we would be very grateful if you could take a moment to review these additions.

Thank you sincerely for your time and consideration.

Comments on the paper

1 comment

We thank the reviewers and Area Chair for their careful evaluation and constructive suggestions. We have provided point-by-point responses under each review and incorporated the additional experiments and clarifications into the revised manuscript, with changes highlighted in blue. Below, we summarize the revisions addressing each reviewer.

1. Response to Reviewer ehXE — Semantic-channel design, VLM robustness, and sample-wise refinement. We have clarified the complementary roles of the five semantic channels and added component ablations. We have also compared Qwen3-VL-Plus, GPT-6.1 Sol, and Qwen3-VL-8B-Instruct through caption agreement, annotation factuality, and downstream decoding. Across annotators, Inception and CLIP identification vary by only 0.5 and 0.4 percentage points, respectively. Paired sample-wise analysis shows that improvements outnumber declines on all seven examined metrics, including DINOv2 and Inception, while explicitly acknowledging that some already accurate reconstructions can deteriorate. These additions appear in Section 3.2 and Appendices N, P, and L.4.

2. Response to Reviewer uUQ2 — Independent FSR evaluation, baseline comparisons, and methodological contributions. DINOv2-only FSR improves seven of eight reconstruction metrics using a network outside the evaluation suite. Applying the same FSR to Duala and MindTuner improves both baselines, while MindRefine retains its Inception advantage. We have added a fully annotation-free control and comparisons isolating LISR’s contribution beyond stimulus-bank supervision. We have also defined adaptive fusion and the post-fit adapter and expanded the discussion of the five requested prior works. See Sections 2, 3.3, and 4.4, Table 3, and Appendices L.2–L.3, M, and O.

3. Response to Reviewer 97gr — Component evidence, evaluation consistency, and experimental stability. We have added network-exclusion controls, semantic accuracy and gate analyses, and feature-target comparisons. We have corrected retrieval entries in Tables 2 and 6 with the full retrieval branch fixed; all eight reconstruction metrics remain unchanged. Across seeds 42, 43, and 44, MindRefine achieves 89.0 ± 0.2% Inception versus Duala’s 86.1 ± 0.4%, with gains at every seed. We have expanded the 40-session comparison with MindTuner and MindHier and clarified validation, parameter counts, and computational costs. See Sections 4.3–4.5, Table 4, and Appendices I, L.2–L.3, N, and Q–S.

We appreciate the reviewers’ feedback, which has strengthened the experimental evidence and presentation of MindRefine.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.