acceptodds
Withdrawn from ICLR 2027

MuCCA: Multilevel Human Color Representation-guided Complementary Alignment for Neural Visual Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural visual decoding links brain activity to visual content and supports the development of brain–computer interfaces. However, existing approaches to visual supervision adaptation do not explicitly account for the multilevel organization of color processing in human vision, potentially limiting neural–visual alignment. To address this limitation, we propose Multilevel Human Color Representation-guided Complementary Alignment (MuCCA) for visual decoding. MuCCA constructs RGB, LMS, DKL-inspired, and CIELUV views of each stimulus to provide supervision informed by cone responses, opponent relationships, and perceptual color attributes. Four independently trained Residual Projection Encoders align neural recordings with frozen CLIP ResNet-50 features of these views through symmetric contrastive learning. A separate original-image branch jointly trains an ATM neural encoder and a ViT-H-14 image encoder for generation. For retrieval and classification, the four RPE branches produce individual rankings, and their candidate union quantifies complementary decoding successes. For reconstruction, a diffusion prior predicts the primary condition from the ATM embedding, while three linear projections map the LMS-, DKL-, and CIELUV-supervised neural embeddings into the same conditioning space. Feature supervision and cross-space consistency regularize these projections. Four IP-Adapters integrate these conditions into a pretrained SDXL generator for reconstruction. Extensive experiments on THINGS-EEG and THINGS-MEG demonstrate that MuCCA outperforms state-of-the-art methods across retrieval, classification, and reconstruction. The code is available at https://anonymous.4open.science/r/MuCCA-8471-842D.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Electroencephalogram, Magnetoencephalography, Visual Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5X5uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MuCCA, a framework that leverages complementary neural-visual feature alignment guided by multilevel color representations to improve EEG/MEG-based image retrieval, classification, and reconstruction.

Critical strengths
  1. Physiology-inspired decoding of brain signal features is an interesting idea.

  2. The experiments cover a good range of tasks.

Critical weaknesses
  1. MuCCA uses four independent RPE branches to align with four types of visual representations. When evaluating Top-1, the four branches are collectively allowed to provide up to four candidates, whereas the Top-1 results of other baselines contain only one candidate. This makes the comparison clearly unfair.

  2. Similarly, regarding these four RPE branches, rather than taking the Top-1 from each branch, how about taking the Top-4 from each of the four branches separately and comparing which branch is more effective under the same retrieval budget?

  3. As the core innovation of this paper, the complementary multi-layer color representations and consistency-constrained auxiliary generation have not been fully justified through ablation studies, including but not limited to those involving the loss functions.

  4. Regarding the motivation of this paper, I have one concern: why must the color spaces be LMS, DKL, and CIELUV? These color spaces do describe different color properties, but is there really any clear, established correspondence between the features they extract and physiological mechanisms such as cone responses, as well as EEG/MEG signals? Without explicit support from physiological studies, this motivation seems somewhat unconvincing.

  5. The presentation needs improvement. For example, Figure 2 contains a large number of repetitive structures that provide limited additional information.

Questions for authors

please refer to the weaknesses

Reviewer_9FGcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MuCCA, a framework for EEG- and MEG-based visual decoding that uses RGB, LMS, DKL-inspired, and CIELUV views as complementary supervision. Four independently trained residual projection encoders align neural recordings with frozen CLIP ResNet-50 features. Retrieval and classification are evaluated through the union of branch-specific candidate sets. For reconstruction, a separate ATM–ViT-H-14 branch supplies a primary diffusion-prior condition, while projected embeddings from three color branches provide auxiliary conditions to a fixed SDXL generator through IP-Adapters.

The central contribution is the use of several color representations to diversify visual supervision. This is an interesting direction, and the reconstruction results are promising. However, the discriminative evaluation does not establish superiority under a comparable prediction budget, and the experiments do not adequately isolate the contribution of the proposed color representations.

Critical strengths
  1. Clear motivation and a plausible supervision strategy. The paper connects cone-response-like, opponent, and perceptual color coordinates to the design of neural decoding targets. It appropriately describes the opponent transform as “DKL-inspired” rather than equating it with calibrated DKL coordinates (Section 3.2).

  2. Broad empirical coverage. The evaluation covers two recording modalities, three decoding tasks, and both within-subject and cross-subject retrieval. The temporal, spatial, and spectral analyses provide useful information about which neural measurements support decoding.

  3. Encouraging reconstruction performance. Table 6 reports leading results on five of seven EEG reconstruction metrics and improvements over the listed ATM baseline on all seven MEG metrics. The EEG improvements in PixCorr and SSIM suggest benefits for low-level fidelity. The encoder-assignment comparison in Table 2 also supports using different primary and auxiliary encoder pairs, although it does not by itself establish the value of the color transformations.

Critical weaknesses
  1. The main retrieval and classification comparisons use different prediction budgets. Equation (12) defines success as the correct target appearing in the union of four branch-specific top-k sets. Consequently, the reported “Top-1” can contain up to four candidates, and “Top-5” can contain up to twenty. These are valid measures of complementary candidate coverage, but they are not equivalent to conventional Top-1 or Top-5 accuracy from a single ranked list. The manuscript discloses this definition; nevertheless, comparing these union-hit rates directly with standard baseline accuracies does not substantiate the claimed state-of-the-art advantage. This affects Tables 3–5 and the corresponding headline claims. A fair comparison requires a fused ranking evaluated with exactly k predictions, or baselines evaluated with the same candidate budget and aggregation protocol.

  2. The ablations do not separate color-specific gains from ensemble gains. Table 1 changes both the available color views and the number of branches. For fixed branch predictions, adding another candidate set cannot decrease union coverage, so improved coverage alone is insufficient evidence for a distinctive benefit from human-color-inspired supervision. Necessary controls include an equally sized ensemble of independently trained RGB branches and alternative transformations, such as random channel mixtures or other color spaces, under matched training and prediction budgets. The current evidence supports complementary predictions, but does not establish that the chosen color representations are responsible for the improvement.

  3. The biological interpretation remains insufficiently validated. All branches receive the same neural input, and transformed color coordinates are packed into three-channel inputs for a visual encoder pretrained on ordinary images. This is a computationally motivated construction, but it does not establish correspondence between the learned branches and successive stages of human color processing. The temporal and spatial analyses characterize overall decoding performance rather than branch-specific color mechanisms. Color-focused reconstruction metrics, chromatic perturbation experiments, or representational analyses would strengthen this interpretation. The demonstrated novelty is therefore primarily in supervision design and system integration rather than an established model of multilevel neural color processing.

  4. The reconstruction contribution needs more direct controls and uncertainty estimates. Table 1 retains at least one transformed view in every row, and Table 2 varies encoder assignments. Neither provides a direct comparison against the same ATM–ViT-H-14/SDXL pipeline with only the primary RGB condition. A matched control with additional RGB conditions is also needed to distinguish color-specific benefits from additional conditioning capacity. The contribution of the cross-space consistency loss is not isolated. These comparisons should use the same generator, inference settings, and evaluation protocol. Repeated training runs and paired analyses across participants or test stimuli would further clarify whether the reported gains are robust, since the current results lack uncertainty estimates.

  5. The evaluation leaves practical applicability and participant generalization insufficiently characterized. The test protocol averages 80 EEG repetitions and 12 MEG repetitions per image (Appendix A.1), so the results establish performance under repeated-stimulus averaging rather than single-trial decoding. This is a legitimate evaluation setting, but its implications for practical use should be made explicit. Performance as a function of the number of averaged repetitions, including a single-trial setting, would clarify signal requirements. Furthermore, the substantial within-subject versus cross-subject retrieval gap in Tables 3–4 indicates that participant transfer remains challenging. Reporting uncertainty for cross-subject results would help assess the consistency of this limitation.

  6. The scope of the performance claims should reflect the observed trade-offs. The abstract and Section 4.1 emphasize superiority across decoding tasks, but Tables 5–6 show important exceptions: EEG Inception and CLIP reconstruction scores trail the best listed baselines, and MEG classification remains below ATM even with the more permissive union-hit protocol. These results suggest that the proposed supervision does not consistently improve semantic decoding across modalities. The paper should qualify its headline claims and discuss why structural reconstruction gains coexist with weaker semantic or category-level performance. This would make the contribution more precise without diminishing the reconstruction improvements that the experiments do support.

Questions for authors

See above.

Reviewer_Gr4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents MuCCA, which draws inspiration from multi‑stage human color vision processing to construct four complementary supervision views: RGB, LMS, DKL‑inspired and CIELUV, for EEG‑ and MEG‑based neural visual decoding. The model employs Residual Projection Encoders (RPEs) for multi‑branch contrastive alignment. For discriminative tasks, the union of candidates from multiple branches is leveraged to boost recall. For image reconstruction, the ATM branch provides the primary diffusion condition, while embeddings from other color‑oriented branches are linearly projected and fed as auxiliary conditions to SDXL via IP‑Adapters. Experiments on THINGS‑EEG and THINGS‑MEG demonstrate that MuCCA outperforms state‑of‑the‑art methods on most retrieval and reconstruction metrics. Ablation studies validate the contributions of color representations as well as spatio‑spectral‑temporal neural information. Nevertheless, cross‑subject generalization remains a major bottleneck.

Critical strengths
  1. This work is motivated by the multi‑stage color‑processing mechanism in the human visual system, a largely overlooked prior in existing neural visual decoding. It translates cone‑response, opponent‑color and perceptually‑uniform color spaces into computable supervision views, elegantly incorporating cognitive neuroscience priors into deep learning pipelines.
  2. Extensive comparisons against many recent SOTA baselines are conducted on two public benchmarks (THINGS‑EEG, THINGS‑MEG) covering retrieval, classification and image reconstruction. Thorough ablations validate color‑view combinations and encoder choices. Systematic temporal‑spatial‑spectral analyses further reveal which neural components benefit decoding, providing solid empirical evidence.
  3. Anonymized code is provided. All datasets are publicly available. The appendix includes full metrics, implementation details and qualitative examples to facilitate reproduction and follow‑up research.
Critical weaknesses
  1. This work is motivated by human multi‑stage color‑processing pathways, yet LMS, DKL‑inspired, and CIELUV are purely fixed hand‑designed coordinate transforms. They act as a form of multi‑view image augmentation, conceptually similar to prior augmentation‑oriented supervision such as UBP, instead of a faithful physiological mapping onto brain color processing. Critically, the image encoders (frozen CLIP ResNet‑50) are pre‑trained on standard RGB natural images. There is no guarantee that these frozen encoders will extract representations that mimic human cone‑level, opponent‑color, or cortical color processing after feeding transformed color‑space inputs. No neurophysiological evidence is provided to confirm that the learned cross‑modal alignment actually corresponds to human biological color representations; performance improvements could be explained purely by diverse augmented visual views. Furthermore, these global fixed transformations cannot account for inter‑subject variability in human color perception.
  2. Experimental results show substantial performance drop under cross‑subject settings, a well‑known bottleneck in this research field. The paper only lists this limitation as future work and provides no targeted mitigation solutions. All major performance improvements are observed within intra‑subject scenarios.
  3. While MuCCA achieves competitive results for retrieval and reconstruction on THINGS‑MEG, its Top‑1 and Top‑5 classification accuracies are inferior to the ATM baseline. The paper lacks thorough analysis or targeted ablation studies to explain why multi‑view color supervision brings benefits for retrieval and reconstruction but fails to yield gains for category‑level classification.
Questions for authors
  1. Could you provide empirical or analytical evidence to rule out the alternative hypothesis that improvements are purely caused by multi‑view augmentation, rather than genuine alignment to human physiological color representations?
  2. Have you conducted control ablations using trivial random color‑space augmentations (not biologically motivated transforms) to compare against your biologically‑inspired color views?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.