acceptodds
Under review as a conference paper at ICLR 2027

Separating Readout Capacity, Feature Width, and Low-Level Stimulus Structure in Multimodal fMRI Encoding

Abstract

Multimodal fMRI encoders outperform unimodal ones, and the advantage is usually credited to the representation. We show that the measured multimodal gain depends on how the linear readout is restricted, not only on how much: below the free readout’s operating range, restricting by ridge penalty, by random projection, or by projection structure changes the gain by nearly a factor of two at approximately matched effective dimension, and the direction depends on the restriction. Holding one frozen feature set fixed, a concatenation of three unimodal backbones, we vary only the readout, from a free ridge down to two trainable parameters per parcel, across three low-level confound arms, and measure the fusion gain at every rung. Matching width leaves the gain within 3.5% of the free readout’s. In distribution, at approximately matched effective dimension, a ridge penalty retains 80% of it where a random projection retains 44%, or 50% with inputs standardized before projection. At a fixed 1,536-column budget a modality-wise projection retains 86% where a joint one retains 61%; standardizing the inputs narrows the gap to 92% against 85% but does not close it, while lowering every absolute score. The penalty ladder’s gain sits below the free readout’s through 1,024 effective dimensions and above it at 1,536 and 2,048. The gain is smallest in visual cortex (+0.0095); its inversion there under a 512-column joint projection (−0.0098) is gone at 1,536 columns, a budget effect rather than a regional one. A second feature set sharing two of three backbones reproduces all four registered comparisons. A reported fusion gain is therefore partly a property of the readout that measured it.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI encoding, computational neuroscience, multimodal representations, readout capacity, ridge regression, representation evaluation, low-level stimulus confounds

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dn4tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how readout capacity, feature dimensionality, and low-level stimulus covariates affect multimodal gains in fMRI encoding. Using frozen features from three pretrained unimodal backbones, the authors compare several linear readout configurations and report that feature-width matching largely preserves fusion gains, whereas different readout restrictions can substantially change them.

Critical strengths
  1. Relevant research question. The paper examines how readout capacity and feature dimensionality influence measured multimodal gains in fMRI encoding, addressing an important methodological consideration in representation evaluation.
  2. Systematic empirical comparisons. The authors investigate multiple readout configurations, including ridge regression, random projection, and capacity-matched settings, while also considering the influence of low-level stimulus covariates.
  3. Reproducibility and experimental transparency. The paper provides extensive experimental details, additional control analyses, and replication experiments. The authors also explicitly acknowledge several limitations of their findings.
Critical weaknesses
  1. Limited novelty and insufficient experimental isolation. While the paper provides a systematic empirical analysis of readout-dependent fusion gains, its findings remain closely related to established knowledge about regularization, feature projection, and representation probing. The study does not provide sufficient mechanistic understanding of why different readout restrictions produce different fusion gains. In particular, the effects of subspace restriction and regularization geometry are not fully disentangled. The magnitude of the reported effects is also sensitive to the choice of unimodal baseline. Furthermore, the experiments involve only four subjects and primarily evaluate concatenated unimodal representations, leaving the generalizability of the findings insufficiently established. Overall, the paper does not convincingly demonstrate a substantial advance beyond existing knowledge.
  2. Poor presentation and insufficient methodological clarity. The overall presentation is unnecessarily verbose and poorly organized. The Related Work section is surprisingly brief and does not adequately establish the distinction between this work and closely related studies. Despite the complexity of the experimental setup, the Method section lacks a methodological overview figure and a clear, unified mathematical formulation of the different readout configurations. The main figures are small and difficult to interpret, while the text is dominated by dense numerical comparisons and extensive implementation details. Substantial space is also devoted to scope qualifications and reproducibility procedures without sufficiently clarifying the central scientific contribution. These presentation issues make the paper difficult to follow and weaken the communication of its contribution.
Questions for authors

None.

1 comment

Dear Reviewer dn4t,

Can you please take a look at the authors' response and provide your further input? Thank you.

AC.

Reviewer_FmHTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In general, the paper studies how several factors affect multimodal fMRI encoding performance, including readout capacity, feature dimensionality, regularization, and low-level stimulus covariates. The experiments are based on pretrained vision, audio, and language models, which were commonly used in prior literature. Their main finding is that the measured multimodal encoding advantage depends on the choice of readout and cannot be attributed solely to the feature representations. They also find that feature width alone does not explain the advantage and that low-level stimulus covariates account for substantial predictive performance.

Critical strengths

The paper includes a fairly extensive set of controlled experiments. Holding the feature representations fixed is useful for studying readout effects, and the additional regularization and projection controls help clarify some of the initial comparisons. The authors also provide detailed robustness analyses and acknowledge several important limitations.

Critical weaknesses
  • My primary concern is the limited novelty and scientific contribution. The paper is essentially an extensive ablation study of existing brain encoding methods, examining how different implementation choices affect prediction accuracy. The findings are obvious that these design choices affect the encoding performance. I am not sure what fundamentally new understanding the study provides beyond quantifying these effects in one experimental setting.
  • The paper presents extensive quantitative comparisons but offers little explanation of why the observed differences arise. In particular, it does not establish a mechanism connecting readout capacity or feature structure to neural representations.
  • The conclusions are based on four subjects and a restricted set of pretrained backbones. The second feature configuration uses the same neural data and shares two of the three backbones. It is therefore unclear whether the findings generalize to other datasets, stronger encoding architectures, or genuinely multimodal representations.
  • The presentation is unnecessarily complicated for the scope of the contribution. I feel the paper is a stack of results without a focus. The main findings are buried under extensive implementation details and numerical results.
  • The entire work, including paper presentation and code repo, is heavily shaped by AI. The table is messay without a focus. The language is weird, such as "The root is verified against a SHA-256 manifest before every fit." (line 129), "Two results that bear directly on ours, neither comfortable." (line 421). And Section 1.2 reads more like a preemptive response to reviewer criticisms than a conventional statement of research scope. I don’t think authors have spent time polishing the work.
Questions for authors
  • The main finding is that multimodal encoding gains depend on readout configuration, regularization, and feature scaling. Given that these dependencies are already well established, what is the key new scientific or methodological insight beyond quantifying their effects in this particular encoding pipeline? How do the findings advance our understanding of multimodal neural representations rather than simply characterize the sensitivity of encoding performance to implementation choices?
  • The paper aims to disentangle readout capacity from feature information, but random projection changes both the accessible feature subspace and the number of trainable parameters. Although the ridge-based controls partially address this issue, differences in subspace selection and regularization remain. How can the authors attribute the observed differences specifically to readout capacity?
Reviewer_BLyhOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how readout capacity, feature width, and low-level stimulus covariates affect measured multimodal gains in fMRI encoding. Holding pretrained features fixed, it compares width matching, random projections, and ridge regularization. The results show that width alone does not explain the fusion advantage, but its measured magnitude depends substantially on how the readout is constrained.

Critical strengths
  1. The controlled comparisons provide a useful methodological finding: matching a scalar measure of capacity does not make readouts interchangeable. At approximately matched effective dimensions, ridge regularization retains around 80% of the in-distribution fusion gain, compared with 44% under random projection, or 50% after input standardization. Measuring the multimodal–unimodal contrast throughout the ladder makes this more informative than simply showing that absolute prediction accuracy changes.

  2. I appreciate the extensive robustness checks and careful interpretation. The paper examines input scaling, projection structure, low-level covariates, reliability thresholds, and alternative baselines. Paired episode-level intervals strengthen the main in-distribution comparisons, and the authors explicitly distinguish computational seed variation from uncertainty across participants.

Critical weaknesses
  1. The generality of the findings remains limited. Both feature sets use the same four participants and stimuli and share the video and audio backbones. The replication therefore supports robustness to feature construction, but provides limited evidence across independent representations or neural datasets. In particular, the study uses concatenated unimodal features rather than jointly pretrained multimodal representations. Since the contribution concerns how multimodal encoding gains should be interpreted, establishing whether the main comparisons extend beyond this configuration would substantially strengthen its relevance.

  2. The main readout comparison remains descriptive rather than explanatory. Even after harmonizing input scaling, R4 uses ordinary least squares on projected features, whereas R5 uses ridge regression on the full design. The comparison therefore changes both the accessible subspace and the regularization form, as Appendix O acknowledges. This supports the conclusion that effective dimension alone is insufficient, but does not establish how much of the gap comes from discarded predictive information versus different shrinkage. A ridge sweep on fixed projected features would help separate these effects and make the practical implications clearer.

Questions for authors
  1. Do the key width-matching and projection-versus-penalty results persist with a jointly pretrained multimodal representation or a substantially different set of backbones?

  2. What happens when ridge regularization is applied to the same fixed projected features used by R4? Can this control clarify how much of the R4–R5 gap reflects subspace restriction versus regularization?

1 comment

Dear Reviewer BLyh,

Can you please take a look at the authors' response and provide your further input? Thank you.

AC.

Comments on the paper

1 comment
Reviewer_FmHTOpenReview ↗

Reply to authors

Thank you for the response and additional experiments.

The proposed explanation largely follows existing results on random projections, ridge regression, and sketching variance. I agree that demonstrating these effects on the multimodal fusion-gain metric is useful, but I struggle to see what fundamentally new understanding emerges from applying these established principles to this particular fMRI encoding setting. Could you please explain more about this?

The new experiments help explain why different readout configurations produce different fusion gains, but the explanation is primarily about the statistical behavior of the estimator rather than the underlying neural representations. For example, averaging random projections recovers a fusion gain comparable to ridge regression, while the absolute encoding scores remain lower. Please correct me if I misunderstood it.

You has shwon that fusion gain is sensitive to readout choices, but does not demonstrate how these findings would change the interpretation of existing brain encoding studies or lead to better methodological practices beyond the expected importance of controlling for readout configuration. Do you have any thoughts about this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.