Separating Readout Capacity, Feature Width, and Low-Level Stimulus Structure in Multimodal fMRI Encoding

Multimodal fMRI encoders outperform unimodal ones, and the advantage is usually credited to the representation. We show that the measured multimodal gain depends on how the linear readout is restricted, not only on how much: below the free readout’s operating range, restricting by ridge penalty, by random projection, or by projection structure changes the gain by nearly a factor of two at approximately matched effective dimension, and the direction depends on the restriction. Holding one frozen feature set fixed, a concatenation of three unimodal backbones, we vary only the readout, from a free ridge down to two trainable parameters per parcel, across three low-level confound arms, and measure the fusion gain at every rung. Matching width leaves the gain within 3.5% of the free readout’s. In distribution, at approximately matched effective dimension, a ridge penalty retains 80% of it where a random projection retains 44%, or 50% with inputs standardized before projection. At a fixed 1,536-column budget a modality-wise projection retains 86% where a joint one retains 61%; standardizing the inputs narrows the gap to 92% against 85% but does not close it, while lowering every absolute score. The penalty ladder’s gain sits below the free readout’s through 1,024 effective dimensions and above it at 1,536 and 2,048. The gain is smallest in visual cortex (+0.0095); its inversion there under a 512-column joint projection (−0.0098) is gone at 1,536 columns, a budget effect rather than a regional one. A second feature set sharing two of three backbones reproduces all four registered comparisons. A reported fusion gain is therefore partly a property of the readout that measured it.