acceptodds
Under review as a conference paper at ICLR 2027

Beyond Hierarchical Alignment: Distilling Teacher Diversity for Brain Visual Decoding

Abstract

A strong visual representation is not necessarily an effective target for brain decoding. Different visual models capture complementary stimulus properties, but their usefulness depends on what can be predicted from limited, noisy brain recordings. Fusing these representations provides shared supervision, yet still defines a single similarity geometry with its own representational bias. The fused space also need not retain each teacher's visual structure or match the conditioning space of a pretrained generator. We introduce BHA (Beyond Hierarchical Alignment) to learn shared brain–visual alignment while composing predictions across distinct visual spaces. BHA jointly learns a brain encoder and a fused visual target, allowing visual supervision to adapt to brain signals. To avoid relying exclusively on the resulting shared geometry, teacher-specific Group Mixture-of-Experts (MoE) readouts learn separate visual predictions on the frozen backbone. Learnable gating and query-dependent weighting compose their matching scores with the shared score, allowing complementary notions of visual similarity to contribute to retrieval. Separately trained semantic and spatial adapters reuse the same backbone for reconstruction without fine-tuning the pretrained generator. BHA improves within-subject retrieval on EEG and MEG, with gains persisting under reduced EEG training data. To examine what distinct visual spaces contribute, we analyze the learned representations and their matching behavior. Experts can assign opposing contributions to the same image regions, while learned model neurons exhibit semantic selectivity. These findings suggest that shared alignment preserves distinct visual matching functions that can be composed for brain decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Visual Decoding, EEG, Distillation, Visual Semantic Selectivity, Generative Model, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_p2sgOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a representational mismatch in brain visual decoding: features that are good for retrieval need not be compatible with a frozen pretrained generator. BHA first aligns a brain encoder to a fused target over multiple visual foundation models (Stage 1), then freezes the backbone and attaches task-specific readouts (Stage 2): teacher-specific Group MoE readouts whose matching scores are composed with the shared score via learnable gating for retrieval, and separate semantic/spatial adapters that condition a frozen diffusion generator in its native feature spaces for reconstruction. Two linear propositions motivate the design: complementary teachers reduce alignment estimation error, while adapting targets toward brain-predictability induces reconstruction discrepancy.

Validation on public datasets (THINGS-EEG1/EEG2, THINGS-MEG) shows within-subject retrieval gains (THINGS-EEG2 CSLS-2 Top-1: 90.95% to 93.90%) that persist with less training data, plus representation analyses showing that experts contribute opposing matching signals despite overlapping geometry. Notably, the gains do not transfer across subjects: under LOSO the gated readout underperforms the pretrained backbone (20.20 vs 22.50 Raw Top-1).

The retrieval-side score composition and the accompanying analyses are a real conceptual contribution, while the reconstruction branch is closer to existing fused-alignment pipelines; the honesty about negative results is a strength, but the method is best described as a well-motivated recombination rather than a fundamentally new mechanism.

Critical strengths
  1. The retrieval-side design is a genuine conceptual advance over single-space alignment. BHA is, to my knowledge, the first brain decoding work that retains teacher-specific prediction spaces and composes their matching scores with the shared score. The supporting empirical finding experts assign opposing signed contributions to the same image regions despite near-identical embedding geometry (Fig. 4) is a non-obvious result that substantiates the claim that shared geometry does not imply interchangeable matching functions, and it is supported by both occlusion attribution and CLIP-Discept concept analysis.

  2. The two propositions precisely delimit when complementary teachers help (estimation error) and when adaptation hurts (generator compatibility); the authors expose negative results that weaken their own method (LOSO degradation of the gated readout: 20.20 vs 22.50 Raw Top-1; adaptation lowering CLIP identification 0.754 to 0.706), which makes the positive claims more credible than the norm in this area.

Critical weaknesses
  1. The shared stem + teacher-specific readouts + router/gating structure follows SAK (ICLR 2025); the fused-target alignment with a frozen SDXL/IP-Adapter generator follows MBP (2026) and ENIGMA (2026); the reconstruction-side semantic/spatial adapters are a repackaging of the same conditioning pipeline with native spaces instead of a learned fusion prior. The retrieval score composition and joint brain-visual adaptation are new, but they are incremental rather than mechanism-level innovations, and the reconstruction branch's delta over MBP is an engineering choice whose benefit is not isolated (no head-to-head against an MBP-style fusion prior under identical teachers and backbone).

  2. The empirical gains are within-subject only and modest in magnitude. The main improvement is CSLS-2 Top-1 from 90.95% to 93.90% on THINGS-EEG2; under LOSO the proposed gating is worse than the frozen pretrained backbone, so the core mechanism does not transfer across subjects, which limits the significance for a field where cross-subject generalization is a stated goal. Reconstruction gains are inconsistent ("richer pretraining does not improve every reconstruction metric"), and the two propositions hold only for linear models, so their connection to the nonlinear pipeline is motivational rather than guaranteed.

Questions for authors
  1. Since the gated readout underperforms the pretrained backbone under LOSO (Table 3), is the within-subject gain partly attributable to subject-specific overfitting in the gating/routing parameters? Would per-subject calibration-free gating, or sharing readouts across subjects, recover LOSO performance?
  2. The reconstruction branch differs from MBP mainly by replacing the learned fusion prior with separate semantic and spatial adapters to native generator spaces. Can the authors provide an isolation experiment in same teachers, same backbone, same training budget, whicn can compare adapter-based native-space conditioning against an MBP-style fusion prior?
2 comments

Thanks for the reply. All my issues have been resolved, so I'll raise my rating.

Thank you for reconsidering your score and for the constructive questions. The LOSO explanation and the fusion-prior control are now in Section 5.4 and Table A.19.

Reviewer_FKZwOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BHA jointly learns a brain encoder and a fused visual target, then freezes this backbone and trains teacher-specific mixture-of-experts readouts. Query-dependent weights combine their retrieval scores with a shared score. Separate semantic and spatial predictors condition a frozen image generator. Experiments report improved within-subject EEG/MEG retrieval, EEG reconstruction, and analyses of learned matching functions and concept associations.

Critical strengths

1.The independent-query retrieval gains are substantial. BHA reports Raw Top-1 of 87.00% on EEG and 55.63% on MEG, exceeding the reported DMSA results of 82.6% and 48.0% (Table 1). The result therefore does not depend entirely on CSLS. Subject-wise results, reduced-data experiments, and independently trained readout controls provide useful supporting evidence.

2.The paper directly tests a meaningful distinction between alignment and generator compatibility. Under BiCLIP, adapting OpenCLIP targets raises retrieval from 30.05% to 37.80% while lowering CLIP reconstruction identification from 0.754 to 0.706 (Table A.15). The matched pretraining-source comparison also reports structural/semantic trade-offs rather than claiming uniform reconstruction improvement (Table A.16).

Critical weaknesses

1.The comparisons do not adequately establish the contribution of teacher diversity and query-dependent specialization.

Multi-encoder fusion and intermediate-layer supervision already appear in closely related decoding work [1, 2]. The consequential advance here is therefore the benefit of retaining and composing distinct prediction spaces. Table A.4 supports the combined readout, but its linear-head control already reaches 85.00% Raw and 93.45% CSLS-2, versus 87.00% and 93.90% for BHA. This comparison also changes the retrieval adapter and readout capacity, and provides no paired uncertainty for these differences. It does not isolate dynamic routing from a simpler score ensemble.

A matched static-weight composition and a repeated/same-teacher control are missing. The former would test query-dependent specialization; the latter would separate teacher diversity from additional prediction branches. Figure 1 does not supply a controlled fusion comparison, as Appendix D.1.3 acknowledges. Proposition 1 concerns averaging unbiased, commensurate noisy targets and does not establish these assumptions for the learned BiCLIP system. Likewise, opposing occlusion attributions and CLIP-Dissect associations [4] characterize learned functions without showing which teacher-specific distinctions cause correct retrievals that shared alignment misses. The evidence establishes a useful system more clearly than it establishes the proposed explanation of its gains.

2.The evaluation leaves the advantage over existing methods and its scope insufficiently resolved.

CSLS [3] raises within-subject EEG Top-1 from 87.00% to 93.90%, but the correction is not reported for the strongest baseline systems. Moreover, k=2 was assessed after benchmark inspection (Section 5.2; Appendix D.1.3). The disclosure is appropriate, but the headline corrected scores remain exploratory evidence rather than a comparison under a consistently applied, independently selected scoring rule. The Raw gains remain valid evidence in their stated setting.

The central Stage-2 gain also fails to transfer under EEG LOSO: Raw Top-1 falls from 22.50% for Stage 1 to 20.20% for the gated readout (Table 3). Reconstruction comparisons change conditioning and sampling settings, while the matched control gives higher CLIP identification with OpenCLIP-only pretraining at both α=0.75 and α=1 (Table A.16). Thus, the additional machinery has not established a consistent benefit beyond within-subject discrimination on the evaluated THINGS benchmarks. An NSD study [5], with an appropriate fMRI encoder and controlled visual/readout comparisons, would provide a valuable test on natural scenes and a different measurement modality. Its absence is a limitation of demonstrated scope, not itself a methodological error. Given the existing fusion and layer-selection literature, stronger attribution and evidence beyond the current benchmark configuration are needed to establish the significance of the advance.

References

[1] Zheng, J., Jia, H., Li, M., Zheng, Y., Zeng, Y., Gao, Y., & Liang, C. (2026). Learning brain representation with hierarchical visual embeddings. In Proceedings of the International Conference on Learning Representations.

[2] Du, Y., Dai, S., Song, Y., Thompson, P. M., Tang, H., & Zhan, L. (2026). Deep models, shallow alignment: Uncovering the granularity mismatch in neural decoding. Transactions on Machine Learning Research.

[3] Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jégou, H. (2018). Word translation without parallel data. In Proceedings of the International Conference on Learning Representations.

[4] Oikarinen, T., & Weng, T.-W. (2023). CLIP-Dissect: Automatic description of neuron representations in deep vision networks. In Proceedings of the International Conference on Learning Representations.

[5] Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., Hutchinson, J. B., Naselaris, T., & Kay, K. (2022). A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1), 116–126.

Questions for authors

Please refer to the Critical Weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer FKZw (1/4)

We thank the reviewer for a careful review. Your central question is whether BHA's gain comes from retaining and composing distinct teacher spaces, or from routing, capacity and extra branches. The controls below answer it: the gain comes from composing distinct spaces that EEG can predict, and none of the other factors contributes. Numbers are THINGS-EEG2 within-subject Raw Top-1, 10 subjects × 3 seeds.

[W1] "Multi-encoder fusion and intermediate-layer supervision already appear in [1, 2]. The advance must therefore be retaining and composing distinct prediction spaces."

Response. We agree that this is the contribution, and we now test it directly. MBP (Zheng et al., 2026) fuses several teachers into one target, and DMSA (Du et al., 2026) selects intermediate vision layers as brain targets. BHA builds on both: Stage 1 is such a fused target, and each teacher uses the layer DMSA identifies. What BHA adds is keeping a separate space per teacher at test time and composing their matching scores. With the same teachers and backbone, this raises Top-1 from 81.1 for the fused target to 86.5 (table in W1-1), the controlled fusion comparison that Figure 1 lacked.

Revision: Related Work now positions BHA against MBP and DMSA in these terms.

[W1-1] "It does not isolate dynamic routing from a simpler score ensemble. A matched static-weight composition ... is missing."

Response. Static learned weights give the full gain; dynamic routing and readout capacity add nothing.

Scoring rule, same trained modelTop-1
Stage 1, one fused space81.1
Shared space only83.3
Uniform weights over spaces83.6
Fixed learned weights86.4
Query-dependent weights (BHA)86.5
Capacity-matched MLP / linear readout, no MoE86.6

Revision: Table A.4 is replaced by a readout search within BHA that includes this control, and Section 5.4 cites it.

[W1-2] "A repeated/same-teacher control is missing ... to separate teacher diversity from additional prediction branches."

Response. Extra branches add nothing; the gain needs a space that is both distinct and predictable from EEG. BHA has four teacher branches (OpenCLIP-H final output, DINOv2-L, MAE, DFN-B), and Stage 1 fuses all four into one target. OpenCLIP-H stays fixed: it is the base of the fused target and the generator's conditioning space. Each control replaces one branch and retrains all stages at fixed size. Alone: the new branch on its own. Stage 1: fused space. Shared: shared space only. Gain = BHA − Shared.

1) Copies: a branch replaced by another BHA teacher.

ReplacedReplaced byAloneStage 1SharedBHAGain
Paper––81.183.386.5+3.2
DINOv2MAE30.780.482.986.6+3.7
DINOv2DFN-B73.580.882.986.3+3.4
DINOv2OpenCLIP-H37.279.682.386.5+4.2
MAEDINOv222.979.982.085.2+3.2
MAEDFN-B74.180.282.385.8+3.5
MAEOpenCLIP-H37.479.682.186.0+3.9
DFN-BDINOv227.075.777.877.4−0.4
DFN-BMAE25.776.078.577.7−0.8
DFN-BOpenCLIP-H24.775.077.376.7−0.6

Copies in the DINOv2-L or MAE branch leave BHA at 85.2–86.6, including a second DFN-B as predictable as the first (73.5 alone). Any copy in place of DFN-B removes the gain.

2) Other depths of OpenCLIP-H. The DFN-B branch is replaced by an intermediate OpenCLIP-H block (CLS or mean patch token); the OpenCLIP-H output branch stays:

BlockAloneStage 1SharedBHAGain
8, CLS58.378.380.683.4+2.8
9, CLS55.777.780.982.5+1.6
10, CLS57.977.780.082.2+2.2
11, CLS58.977.379.881.8+2.0
8, mean64.678.380.983.8+2.9
9, mean67.378.981.684.7+3.1
10, mean71.279.882.085.6+3.6
11, mean73.880.182.486.2+3.8

Unlike a copy of the output (table 1), intermediate blocks act as distinct spaces; mean pooling adds more.

3) Other models in place of DFN-B, at the depth DMSA reports as best:

TeacherAloneStage 1SharedBHAGain
DFN-B (paper)76.581.183.386.5+3.2
Qwen3-VL70.078.580.583.3+2.8
SigLIP-so400m67.478.881.283.2+2.0
InternViT-6B57.881.884.186.4+2.3
EVA-02-L56.474.177.178.1+1.0
DINOv3-L47.776.179.679.4−0.2
V-JEPA 2-L10.877.379.679.4−0.2
DINOv2-L, block 69.476.479.078.0−1.0

Gain tracks how well EEG predicts the new space (Spearman ρ = 0.88, Alone vs. Gain, 15 swaps in tables 2–3); self-supervised teachers add nothing, the predictability condition of Proposition 1.

Revision: all three controls are in the new Table A.12.

AuthorsOpenReview ↗

Response to Reviewer FKZw (2/4)

[W1-3] "... provides no paired uncertainty for these differences."

Response. Added. The composition gain is +3.2 ± 0.8 across seeds and positive in 27 of 30 subject–seed pairs; it, and every teacher-swap gain of 2 points or more, is significant under a paired Wilcoxon test across subjects (p ≤ 0.02). The gain also holds with fewer averaged test trials: +3.1 / +4.1 / +3.9 / +3.2 at 4 / 8 / 16 / 80 trials.

Revision: Tables A.4 and A.12 report three-seed means.

[W1-4] "Proposition 1 ... does not establish these assumptions for the learned BiCLIP system."

Response. We agree; revised Section 4.1 says the linear result "motivates, but does not guarantee" the learned system's gain. Its prediction, gains only from teachers EEG can predict, is what the teacher-swap table shows.

[W1-5] "... without showing which teacher-specific distinctions cause correct retrievals that shared alignment misses."

Response. Composition rescues more than twice as many queries as it loses, and dropping the DFN branch at test time undoes two thirds of the rescues. What each space carries is shown in the new Figure 7: OpenCLIP recovers object parts, DFN and MAE colour and texture, DINOv2 contours, and each EEG expert inherits its teacher's properties. Figure 8 shows that the adapters keep each teacher's geometry.

[W2-1] "CSLS ... is not reported for the strongest baseline systems. Moreover, k=2 was assessed after benchmark inspection."

Response. k is now selected on validation and comes out at k = 2 (Table A.14). Open-source baselines from 2024–2026 were run with official code on our split and scored with our CSLS code. CSLS subtracts each query's and each image's mean similarity to its k nearest neighbours. The CSLS column takes those neighbours from the test set, which is transductive; the Inductive column takes them from held-out validation queries and images, so no test query sees another.

MethodRawCSLSInductive
NICE (2024)27.538.632.9
ATM-S (2024)28.740.736.1
UBP (2025)46.457.149.8
MBP (2026)75.786.378.5
DMSA (2026)78.991.282.0
BHA86.593.988.6

CSLS narrows BHA's lead rather than creating it. DMSA's reported 82.6 uses test-loss epoch selection in its released code; we select on validation.

Revision: Section 5.2 now selects k on validation (Table A.14), Table A.15 adds the baselines, and Section 5.1 labels CSLS as transductive.

[W2-2] "The central Stage-2 gain also fails to transfer under EEG LOSO."

Response. Cross-subject transfer is not a claim of the paper. The Abstract states within-subject gains, and Section 5.2 names cross-subject transfer as an open problem. The bottleneck under LOSO is the brain encoder, which BHA does not aim to change: Stage 1 alone drops from 81.1 to 22.0 on unseen subjects. Composition helps wherever the encoder generalizes, including with one model for all ten subjects:

SettingStage 1Shared space onlyBHA
One model per subject81.183.386.5
One model for all 10 subjects87.587.589.4
Unseen subject (LOSO)22.021.120.6
LOSO, gate recalibrated without that subject22.021.120.7

On unseen subjects, the teacher-specific branches inherit the encoder's limit (the DFN branch drops from 76.5 to 13.5), so there is little signal left to compose.

Revision: Section 5.4 now states this explanation.

[W2-3] "... the matched control gives higher CLIP identification with OpenCLIP-only pretraining."

Response. This is the trade-off of Proposition 2: a target adapted for alignment drifts from the generator's native space. That is why BHA conditions the generator in native spaces. Two controls isolate it. In a minimal single-teacher model (revised Appendix D.3.2), the adapted target retrieves best, the native target reconstructs best, and no setting wins both. With conditioning and sampling matched, native conditioning beats an MBP-style fusion prior on 6 of 8 metrics (reply to p2sg, Q2).

Revision: the tables cited in the review are renumbered: A.15 is now A.18, and A.16 is now A.20.

AuthorsOpenReview ↗

Response to Reviewer FKZw (3/4)

[W2-4] "An NSD study ... would provide a valuable test."

Response. We ran it, together with TVSD (macaque multi-unit activity), changing only the input tokenizer. On NSD it ranks second to MindEye2 on retrieval while giving the highest CLIP identification, and on TVSD it leads recent methods on retrieval and, for each animal, on every reconstruction metric.

Setup. We reuse the THINGS-MEG recipe and the four teachers. NSD (7T fMRI, natural scenes) follows the MindEye protocol: single-trial training, the shared 1,000 images as test with responses averaged over three repetitions, and evaluation on subjects 1, 2, 5 and 7. One model is trained on all eight subjects, with a linear voxel tokenizer per subject. TVSD uses the zero-shot split of Song et al. (2026), removing the 100 test concepts from training (21,048 training images; 100 test images × 30 repetitions); one model per animal. Numbers are Raw Top-1, three-seed means.

Composition across datasets.

DatasetTestStage 1Shared onlyBHA
THINGS-EEG2 (paper)200-way81.183.386.5
NSD, image retrieval (brain query)1000-way84.685.485.5
NSD, brain retrieval (image query)1000-way85.386.387.2
TVSD, zero-shot, two animals100-way98.299.7100.0

On NSD the ordering of the paper holds, but composition adds less than on EEG: +0.9 for brain retrieval, positive in 10 of 12 subject–seed pairs, and +0.1 for image retrieval. TVSD is near ceiling for every stage, so it tests transfer to spiking data rather than the composition gain.

Comparison on NSD. The other rows are from MindEye2's Table 1, recomputed there on the full 40 sessions; retrieval is 300-way, averaged over 30 draws. The joint BHA model z-scores voxels per session over all trials (test trials included, without labels). The MindEye2-protocol row follows that table's setup: pretraining on the other seven subjects, fine-tuning on the target's 40 sessions, and z-scoring with training-trial statistics only.

MethodImage ret.↑Brain ret.↑PixCorr↑SSIM↑Alex(2)↑Alex(5)↑Incep↑CLIP↑EffNet-B↓SwAV↓
MindEye298.3%98.8%0.3220.43196.1%98.6%95.4%93.0%0.6190.344
MindEye184.1%90.0%0.3190.36092.8%96.9%94.6%93.3%0.6480.377
Brain Diffuser18.8%26.3%0.2730.36594.4%96.6%91.3%90.9%0.7280.421
Takagi & Nishimoto––0.2460.41078.9%85.6%83.8%82.1%0.8110.504
BHA, joint 8-subject92.9%94.1%0.2390.41988.3%95.9%94.1%94.7%0.6560.373
BHA, MindEye2 protocol93.2%94.1%0.2350.41988.1%95.7%94.1%94.6%0.6590.374

The two BHA rows differ by at most 0.3 points, so the joint results depend neither on test-trial normalization nor on per-subject fine-tuning. BHA has the highest CLIP identification and is second on retrieval, SSIM and SwAV. MindEye2 leads the other metrics, also under its own protocol, but fine-tunes SDXL into an unCLIP generator, whereas BHA trains none.

The TVSD comparison follows in the next comment.

AuthorsOpenReview ↗

Response to Reviewer FKZw (4/4)

[W2-4, continued] TVSD comparison.

Retrieval on TVSD. 100-way brain-to-image Top-1/Top-5, as reported (FuzzyAlign, Song et al. 2026: same zero-shot split, five seeds; †: official split, test concepts seen in training). BHA rows are three-seed means, and the two branch rows read out a single teacher space of the same BHA model.

MethodN Top-1F Top-1Mean Top-1Mean Top-5
FuzzyAlign61.2%64.4%62.8%87.7%
Ciferri et al.†––69.3%93.6%
BHA, OpenCLIP branch only70.3%70.0%70.2%93.7%
BHA, DFN branch only98.0%97.7%97.8%99.8%
BHA100.0%100.0%100.0%100.0%

Read through the OpenCLIP space alone, BHA is close to these methods; the mid-level DFN space, which multi-unit activity predicts well, accounts for most of the gap.

Reconstruction on TVSD (as reported; FuzzyAlign: better of two pipelines; † as above).

MethodAnimalPixCorr↑SSIM↑Alex(2)↑Alex(5)↑Incep↑CLIP↑EffNet-B↓SwAV↓
Perceptogram†N0.2180.41788.0%90.3%73.4%81.8%0.8670.538
FuzzyAlignN0.2070.42784.2%92.1%77.1%81.4%0.8500.549
BHAN0.2660.46691.1%96.1%85.8%86.2%0.7910.495
Perceptogram†F0.2140.41486.2%92.1%72.2%81.5%0.8590.520
FuzzyAlignF0.2350.42187.1%94.1%82.7%86.3%0.8170.526
BHAF0.2700.46491.5%96.6%87.1%89.7%0.7630.473
Ciferri et al.†mean0.1400.364––86.8%87.9%0.7920.492
BHAmean0.2680.46591.3%96.3%86.4%87.9%0.7770.484

BHA is best on all metrics per animal with an untrained generator; on the mean, Ciferri et al. lead Inception by 0.4 and tie CLIP.

With THINGS-EEG2 and THINGS-MEG, BHA now covers EEG, MEG, fMRI and intracortical recordings with one recipe.

Revision: Appendix D reports both datasets in full.

We believe these controls give the attribution your review asked for: the gain comes from composing distinct, EEG-predictable spaces, not from routing, capacity or branch count, and it holds within subjects. If this resolves your main concerns, we would be grateful if you would reconsider your score.

Reviewer_oxXcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BHA, a framework for brain-to-image retrieval and reconstruction using multiple visual teachers. It combines shared brain–visual alignment with teacher-specific MoE readouts for retrieval, while separately predicting generator-native conditions for reconstruction. Experiments show improved within-subject retrieval on EEG and MEG, supported by representation analyses and reconstruction evaluations.

Critical strengths
  1. The within-subject retrieval results are strong even without CSLS post-processing. BHA achieves Raw Top-1 accuracy of 87.00% on EEG and 55.63% on MEG, compared with the strongest reported baselines of 82.6% and 48.0%, respectively. The gains from teacher-specific readouts also persist across reduced training-data settings.

  2. I find the distinction between retrieval-friendly representations and generator-compatible conditions well motivated. The matched adaptation experiments in Appendix D.3.2 support this distinction, and the reconstruction permutation control provides useful evidence of stimulus-specific information. The paper also appropriately acknowledges that its gains do not consistently extend to cross-subject transfer.

Critical weaknesses
  1. The specific benefit of the MoE and learned routing is not fully isolated. Table A.4 provides useful readout controls, but linear heads already achieve 85.00% Raw Top-1 and 93.45% with CSLS-2, versus 87.00% and 93.90% for full BHA. Moreover, the linear/MLP controls omit the retrieval adapter, so these comparisons change more than the expert architecture. A capacity-matched non-MoE control retaining the adapter, together with fixed versus query-dependent score weighting, would clarify whether the additional complexity is necessary.

  2. The CSLS-enhanced results require a more controlled comparison. CSLS uses all 200 test queries as unlabeled references, and the paper explicitly states that (k=2) was assessed after benchmark inspection. I appreciate this transparency, but these results should be treated as exploratory rather than as a clean estimate under a validation-selected protocol. The comparison also does not establish how much of the advantage would remain if competing methods received the same post-processing. This concern does not invalidate the independently reported Raw improvements.

Questions for authors
  1. Does BHA retain its advantage over a capacity-matched non-MoE readout that keeps the retrieval adapter? How much does query-dependent score weighting improve over a fixed learned mixture?

  2. Could you report CSLS results with the neighborhood size selected exclusively on validation data, and apply the same procedure to a strong baseline under the same reference-query setting?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading. Reviewer p2sg calls score composition "a genuine conceptual advance", and oxXc and FKZw find the separation of retrieval and generation well motivated.

What BHA studies. Vision models trained with different objectives see the same image differently. BHA asks whether brain signals can be aligned to several of these views at once, and what each view contributes. We chose four teachers with distinct objectives: DINOv2 (self-distillation), MAE (masked pixel reconstruction), DFN (image–text contrastive learning) and OpenCLIP ViT-H (image–text contrastive learning, and the generator's conditioning space). The paper does not claim that these four are required. They are four views used to test a hypothesis, not a teacher set tuned for accuracy.

Contributions.

  • Multi-view distillation with learned composition. Instead of collapsing the teachers into one fused target, BHA keeps a brain-predicted space per teacher on one shared backbone and learns how to compose their scores. The composition finds the view that EEG can predict and that is distinct from the shared space: in all ten subjects, most of the teacher weight goes to DFN-B (0.89–0.98, Table A.4c), and copies of existing views add nothing in nine controls (Table A.12).
  • Interpretable brain–AI alignment beyond retrieval. Because the views stay separate, each brain-predicted space can be read through its own teacher. The EEG experts inherit their teachers' properties, such as contours (DINOv2) and colour and texture (MAE, DFN) (Figure 7), and the adapters preserve each teacher's geometry (Figure 8). Views that retrieval barely uses still carry this content, and the multi-teacher target improves reconstruction SSIM over OpenCLIP-only pretraining at every conditioning strength, though not every semantic metric (Table A.20). Retrieval favours high-level, discriminative features, so it shows only part of what the views provide.
  • One backbone, two tasks. Retrieval favours an adapted contrastive target and reconstruction the generator's native space (Table A.18). BHA serves both from one brain backbone, and native conditioning beats a learned fusion prior on 6 of 8 metrics without training the generator (Table A.19).

Shared questions. New numbers are means over 10 subjects × 3 seeds.

  • Composition or MoE, routing, capacity? Composition. With readout, capacity and branch count unchanged, learned weights over the teacher spaces add 3.2 points for every readout; Group MoE is one interchangeable readout (Table A.4; replies to oxXc and FKZw).
  • Fair CSLS comparison? With k chosen on validation and DMSA and MBP run with their official code on our split, BHA leads under Raw (86.5 vs. 78.9 and 75.7), CSLS and inductive CSLS (replies to oxXc and FKZw).
  • Cross-subject transfer (LOSO) was never a claim of the paper. It is limited by the brain encoder (Stage 1: 22.0 on unseen subjects vs. 81.1 within subjects), not by composition. Wherever the encoder generalizes, composition helps: +3.2 with one model per subject and +1.9 with one model for all ten subjects (replies to FKZw and p2sg).

Beyond THINGS. With the same recipe and teachers, BHA transfers to NSD (7T fMRI) and TVSD (macaque multi-unit activity). On NSD, one model for all eight subjects reaches 92.9% / 94.1% 300-way image / brain retrieval, second to MindEye2, and the highest CLIP identification in MindEye2's comparison (94.7% vs. 93.0%). Composition keeps the paper's ordering there, with a smaller gain than on EEG (reply to FKZw, W2-4).

Revisions to the PDF (highlighted in blue):

  • Abstract, Section 4.1: rewritten motivation; Proposition 1 is presented as motivation, not a guarantee.
  • Sections 5.5.3–5.5.4, Figures 7–8: feature-inversion case study and geometry analysis of the teacher views.
  • Table A.4 and Section 5.4: readout search (linear, MLP, low-rank linear and Group MoE, 0.47M–10.5M parameters) and per-subject composition weights (panel c).
  • Tables A.12, A.14, A.15, A.18, A.19: teacher swaps and nine copy controls; k selected on validation; CSLS for five open-source baselines; controlled single-teacher test; fusion-prior control.
  • Related Work: differences from SAK, MBP, DMSA and ENIGMA. Table A.16: full feature-inversion settings.
  • Appendices D.4–D.5 (new): NSD and TVSD (Tables A.22–A.25).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.