- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper utilizes structured language descriptions of training images to constrain brain representations, and then uses a training reference library to predict multi-network visual features of test samples. By optimizing and correcting the initial reconstruction image by image, the cross-subject visual decoding index is improved under limited data conditions.
Critical strengths
-
Improvements are not limited to optimizing a single network. In Table 2, after adding FSR, Inception, which was not included in its optimization objectives, still saw an improvement from 87.2% to 87.8%, with PixCorr and SSIM also showing improvements.
-
Post-processing is applied to another decoder. Figure 4 and the appendix demonstrate the application of similar modifications to MindEye2, illustrating that FSR has the potential to be used across at least two decoding systems, simultaneously covering both single-shot and full-data conditions.
Critical weaknesses
-
Existing evidence is insufficient to explain the score improvement as a broad perceptual enhancement. Appendix C clearly states that the first stage of FSR optimizes CLIP, EfficientNet-B1, and SwAV, while the second stage adds AlexNet-2 and AlexNet-5. This directly overlaps with the reported five feature evaluations. Although the optimization target comes from the training reference library predictions, rather than the test ground truth, the final images are directly adjusted to conform to the representations of these networks, thus they cannot simultaneously serve as sufficiently independent effect validations. Table 2 particularly illustrates this coupling: from no FSR to the full method, Eff decreases from 0.797 to 0.758, and SwAV decreases from 0.468 to 0.416; when only CLIP correction is used, these two only reach 0.793 and 0.462. This result is effectively compatible with multi-network optimization, but it cannot alone prove that it is more consistent with human perception or more faithful to brain signals. Page 20 of the paper uses Eff and SwAV performance to support the "non-biased towards a single embedding space" explanation, ignoring that they are inherently part of the optimization target.
-
LISR combines five VLM text channels, sample gating, manually set channel priors, and semantic neighborhood distillation. Compared to simple image description, uniform text averaging, or existing visual relation distillation, the paper doesn't provide a direct comparison of what this complex design actually adds. In Table 2, removing LISR actually increases CLIP from 83.6% to 83.9%, and brain retrieval from 81.8% to 82.4%. The complete system performs well on other reconstruction metrics, supporting a quality trade-off, but it doesn't directly prove that language constraints enhance semantic integrity. The authors have already proposed an ablation method of "removing the entire LISR," but it doesn't answer whether the five-channel structure, gating, and neighborhood distillation are necessary.
-
The core of FSR is training feature regression plus image feature inversion, lacking a strong contrast sufficient to establish method incrementality and sample specificity. Equation 6 uses normalized brain embedding linear kernel ridge regression. For a fixed representation, this is equivalent to fitting a ridge regression mapping on that representation and then predicting different visual network features. Equation 7 then optimizes the image through feature loss, pixel anchoring, and total variation. This is a reasonable combination, but these operations themselves are not new learning principles. The key issue is that the paper does not independently evaluate whether the predicted features are closer to the correct stimulus than the initially reconstructed features, nor does it compare direct brain signals to multi-network feature regression, nearest neighbor reference, and general post-processing with the same budget. Figure 4 provides useful support for the transfer experiment of MindEye2, but it does not re-rank the data after assigning the same FSR to the strongest-fitting baseline.
-
Section 3.3 refers to the FSR freezing decoder; the main text defines "not using FSR" as directly evaluating the output before correction, and "CLIP correction only" as only changing the feature constraints. Appendix L further clarifies that the retrieval branch and brain representation remain unchanged between the initial image, fixed fusion, and FSR. However, the average image retrieval rates for the three conditions in Table 2 are 84.0%, 84.0%, and 84.2%, respectively, and the brain retrieval rates are 81.6%, 81.3%, and 81.8%. In Table 5, Subject 2's "CLIP-only correction" brain retrieval rate was 90.4%, while the complete method and the method without FSR both achieved 92.4%, a difference of 2 percentage points, which cannot be explained by rounding.
-
The appendix clearly states that each configuration was trained only once, with a fixed seed of 42; the thirty retrieval repetitions corresponded only to candidate sampling, not thirty independent training iterations. Therefore, the differences in Table 1, such as 83.6% for CLIP versus 83.5% for Duala and 81.8% versus 81.1% for brain retrieval, do not indicate any fluctuations in training or target data selection. The paper claims that hyperparameters are fixed across subjects, but this does not mean the selection process is independent of testing. Appendix K supports default semantic weights and correction strengths based on "balanced performance" across multiple evaluation metrics, but it does not specify which validation set these choices came from, or whether the default values were determined before or after reviewing test results. The full-data experiment only compares MindEye2 and NeuroFlow, which is insufficient to demonstrate a broad advantage over all strongly fitted methods. The paper should clearly define which conclusions are descriptive results from a single configuration and which have been validated for stability.
-
Algorithm 1 first freezes the trained decoder and residual label adapter, then fits a lightweight low-rank label adapter. The text and appendix do not fully explain the latter's rank, structure, training objective, number of fitting steps, or whether it participates in retrieval. The parameter table lists an additional 2.97M for the residual adapter, semantic gating, and projection head, but does not specify whether this post-fitting module is included. Similarly, the initial reconstruction comes from fixed fusion and "adaptive fusion of training calibration," but the appendix only gives a combined weight of 0.6 and 0.4, without fully defining the input, form, and calibration objective of the adaptive fusion.
Questions for authors
-
If an evaluation network is removed from the FSR each time and then tested on that network, does the improvement remain?
-
What levels of performance are achieved with single caption, five-channel uniform averaging, fixed gating, and neighborhood distillation only? Do the accuracy rates for objects, attributes, and relationships improve? What is the actual distribution of gating selection?
-
How much is the error of predicted features reduced relative to the initial image features? What are the effects of direct ridge regression, nearest neighbor reference, and category prototype? Given the same FSR as the strongest baseline, how much does the training module contribute?
-
Why does post-processing ablation of frozen brain representations change the retrieval score?
-
Which independent validation or calibration set determines the semantic weights, corrected weights, and fusion coefficients?
-
What is the complete definition of the post-fit adapter and adaptive fusion, and are they included in the parameter table? What are the end-to-end time, GPU memory, and storage costs after including annotations, reference libraries, and 64-step graph-by-graph optimization?
9 comments
2. LISR components, semantic accuracy, and gate distributions
We evaluated caption-only supervision, five channels with uniform weights, five channels with fixed prior weights, and neighborhood distillation alone. The table reports Inception and CLIP identification before and after FSR, alongside the published full-method results from Table 6.
| LISR configuration | Inception before / after FSR | CLIP before / after FSR |
|---|---|---|
| Caption only | 87.1% / 87.7% | 81.6% / 84.2% |
| Five channels, uniform weights | 87.0% / 87.8% | 81.5% / 84.7% |
| Five channels, fixed prior weights | 87.3% / 87.8% | 81.6% / 84.1% |
| Neighborhood distillation only, uniform weights | 87.0% / 87.8% | 81.5% / 84.5% |
| Ours | 88.4% / 89.1% | 83.0% / 85.3% |
Among the additional variants, five-channel uniform supervision improves post-FSR CLIP identification over caption-only supervision, and neighborhood distillation alone also provides effective semantic supervision. The full method combines direct alignment and neighborhood supervision with learned channel weights.
We additionally evaluated object, attribute, and relation accuracy using blinded GPT-6.1 Sol visual question answering on all 1,000 test images. We generated 6,000 questions from the original stimuli, with two questions per image for each category (2,000 object, 2,000 attribute, and 2,000 relation questions), and fixed the question bank before scoring reconstructions. The evaluator received anonymous reconstructed images and the same questions, with missing or unclear queried content counted as incorrect.
| Method | Object ↑ | Attribute ↑ | Relation ↑ |
|---|---|---|---|
| MindEye2 | 51.65% | 14.50% | 9.85% |
| Duala | 51.90% | 14.10% | 10.85% |
| w/o LISR | 52.35% | 16.15% | 13.00% |
| MindRefine (Ours) | 53.65% | 17.15% | 14.50% |
With the stimulus-bank loss retained, adding LISR increases object, attribute, and relation scores by 1.30, 1.00, and 1.50 percentage points, respectively. These model-based semantic measurements complement the reconstruction metrics and directly address the content of the reconstructed images.
The following gate statistics were computed from the full Subject 1 checkpoint over the 1,000 test brain embeddings.
| Channel | Mean weight |
|---|---|
| Caption | 22.0% |
| Scene | 29.5% |
| Object | 20.3% |
| Attribute | 13.7% |
| Relation | 14.5% |
The gate assigns substantial average weight to all five channels. Scene receives the largest mean weight (29.5%), followed by Caption (22.0%) and Object (20.3%). Caption and Scene together account for 51.5% of the average weight, while Object, Attribute, and Relation account for 48.5%. This distribution shows that the semantic supervision draws on both global context and detailed information about entities, their properties, and their relationships.
We have clarified the channel design in Section 3.2 and summarized the semantic findings in Section 4.3. Appendix N, Tables 14–16, reports the component ablations, semantic evaluation, and gate distributions.
3. Feature-target accuracy, alternative estimators, and baseline refinement
For a fixed normalized brain representation, linear-kernel ridge regression is equivalent to primal ridge regression. With training embeddings and training visual features , the primal mapping is . The dual form in Eq. 6 makes prediction practical with a bank of 536 training examples. FSR uses this mapping to estimate sample-specific targets in several feature spaces, then optimizes an existing reconstruction with pixel anchoring in two stages.
To test whether FSR is guided toward more accurate features, we compare two feature vectors against the same ground-truth reference for each test stimulus. The first is extracted from the initial reconstruction before FSR. The second is the target predicted from the test brain embedding and the paired training bank. For each network , the reference is the feature of the true stimulus. We compute for the initial image and for the predicted target, then average over the 1,000 test stimuli.
The Initial-image feature error column measures the error of features already present in the initial reconstruction. The next four columns measure the errors of target vectors supplied by the four estimators. All entries are cosine distances under the feature preprocessing used by FSR, and lower values indicate closer agreement with the true stimulus. Test stimulus features are used only to evaluate these errors. For example, CLIP error decreases from 0.386 for initial-image features to 0.261 for KRR targets.
| Feature space | Initial-image feature error ↓ | Raw-fMRI ridge ↓ | Nearest reference ↓ | Category prototype ↓ | Brain-embedding KRR (Ours) ↓ |
|---|---|---|---|---|---|
| AlexNet(2) | 0.743 | 0.674 | 0.787 | 0.683 | 0.605 |
| AlexNet(5) | 0.780 | 0.734 | 0.836 | 0.771 | 0.703 |
| CLIP | 0.386 | 0.285 | 0.386 | 0.315 | 0.261 |
| EfficientNet-B1 | 0.764 | 0.664 | 0.758 | 0.710 | 0.657 |
| SwAV | 0.318 | 0.253 | 0.349 | 0.286 | 0.243 |
Brain-embedding KRR achieves the lowest mean target error among the four estimators in all five feature spaces. It also predicts features closer to the true stimulus than those extracted from the initial reconstruction, directly supporting the use of its predictions as refinement targets. Raw-fMRI ridge regularization was selected by five-fold cross-validation on training stimuli. Nearest-reference targets use brain-embedding similarity, while category prototypes average training features within categories and assign queries through training brain centroids.
We then used the same 64-step FSR budget with each target estimator.
| Target estimator | Inception ↑ | CLIP ↑ | EfficientNet-B1 ↓ | SwAV ↓ |
|---|---|---|---|---|
| Raw-fMRI ridge | 88.4% | 81.8% | 0.757 | 0.414 |
| Nearest reference | 88.5% | 84.1% | 0.759 | 0.415 |
| Category prototype | 88.5% | 82.7% | 0.761 | 0.416 |
| Brain-embedding KRR (Ours) | 89.1% | 85.3% | 0.742 | 0.401 |
The target-prediction and reconstruction results consistently favor brain-embedding KRR. Its lower target errors are accompanied by the highest Inception and CLIP identification and the lowest EfficientNet-B1 and SwAV distances among the four estimators. To test sample specificity under the same optimization budget, we also used a shared mean target and shuffled the sample-specific targets. Their Inception/CLIP scores were 88.3%/76.6% and 88.2%/77.9%, respectively, supporting the value of assigning each sample its own brain-predicted targets.
We further applied the same FSR procedure to Duala and MindTuner, using the same training-bank construction, feature objectives, ridge regularization, optimization steps, and blending.
| Method and output | PixCorr ↑ | SSIM ↑ | Inception ↑ | CLIP ↑ | EfficientNet-B1 ↓ | SwAV ↓ |
|---|---|---|---|---|---|---|
| Duala Initial | 0.245 | 0.408 | 85.9% | 80.5% | 0.811 | 0.470 |
| Duala Official Fixed Blend | 0.253 | 0.418 | 86.6% | 85.1% | 0.768 | 0.433 |
| Duala After FSR | 0.264 | 0.423 | 88.1% | 85.5% | 0.746 | 0.417 |
| MindTuner Initial | 0.239 | 0.432 | 84.1% | 79.3% | 0.805 | 0.472 |
| MindTuner Official Fixed Blend | 0.262 | 0.422 | 85.8% | 84.6% | 0.774 | 0.433 |
| MindTuner After FSR | 0.267 | 0.428 | 87.5% | 85.2% | 0.754 | 0.416 |
| MindRefine (Ours) | 0.276 | 0.425 | 89.1% | 85.3% | 0.742 | 0.401 |
FSR improves Inception over Initial by 2.2 percentage points for Duala and 3.4 for MindTuner. MindRefine's w/o FSR ablation already reaches 88.4% Inception, and adding FSR raises it to 89.1%, retaining a lead over both refined baselines. These comparisons support contributions from both the decoding pipeline and FSR.
We have clarified KRR in Section 3.3 and added the target-quality, estimator, and baseline comparisons in Appendix L.3, Tables 10–12.
We thank the reviewer for the careful evaluation and constructive suggestions. We have revised the manuscript, with changes highlighted in blue. Revision locations are given below. The response is organized into seven comments. Comments 1–4 address Questions 1–4, respectively. Question 5 is addressed in two comments. Comment 5 covers training-seed stability and cross-subject consistency, and Comment 6 covers validation and the expanded full-data comparison. Comment 7 addresses Question 6 on module definitions, parameter counts, and computational cost.
1. Evaluation after excluding a feature network from FSR
We agree that improvements in feature spaces included in the FSR objective require complementary evaluation. We therefore removed each evaluation network from both FSR stages and evaluated the resulting images in the omitted space. For AlexNet, both layers were removed together.
Four of the five held-out metrics improve relative to the submitted pre-FSR reference, while AlexNet(2) decreases by 0.2 percentage points.
We also replaced the entire feature objective with DINOv2 ViT-B/14 CLS alignment in both stages, retaining the 40+24 optimization steps, pixel anchors, training reference bank, and blending settings. DINOv2 is outside the original evaluation suite.
This replacement improves seven of eight reconstruction metrics relative to the submitted reference. Inception and CLIP increase by 0.3 and 1.6 percentage points, respectively. Retrieval uses fixed embeddings and candidate sets and therefore remains unchanged. Together, these controls support improvements beyond the feature networks directly optimized by FSR. We have revised the discussion of EffNet-B1 and SwAV to identify their overlap with the original objective and have added the held-out evaluations to assess generalization across feature spaces.
We have added the independent-network comparison in Section 4.4, Table 3, and the complete network-exclusion and DINOv2-only results in Appendix L.2, Tables 8 and 9.