Beyond Hierarchical Alignment: Distilling Teacher Diversity for Brain Visual Decoding

A strong visual representation is not necessarily an effective target for brain decoding. Different visual models capture complementary stimulus properties, but their usefulness depends on what can be predicted from limited, noisy brain recordings. Fusing these representations provides shared supervision, yet still defines a single similarity geometry with its own representational bias. The fused space also need not retain each teacher's visual structure or match the conditioning space of a pretrained generator. We introduce BHA (Beyond Hierarchical Alignment) to learn shared brain–visual alignment while composing predictions across distinct visual spaces. BHA jointly learns a brain encoder and a fused visual target, allowing visual supervision to adapt to brain signals. To avoid relying exclusively on the resulting shared geometry, teacher-specific Group Mixture-of-Experts (MoE) readouts learn separate visual predictions on the frozen backbone. Learnable gating and query-dependent weighting compose their matching scores with the shared score, allowing complementary notions of visual similarity to contribute to retrieval. Separately trained semantic and spatial adapters reuse the same backbone for reconstruction without fine-tuning the pretrained generator. BHA improves within-subject retrieval on EEG and MEG, with gains persisting under reduced EEG training data. To examine what distinct visual spaces contribute, we analyze the learned representations and their matching behavior. Experts can assign opposing contributions to the same image regions, while learned model neurons exhibit semantic selectivity. These findings suggest that shared alignment preserves distinct visual matching functions that can be composed for brain decoding.