Rogue Dimensions Mask a Learned Cortical Trajectory of Visual Tokens in Vision–Language Decoders

Vision–language models (VLMs) process image tokens with an entire language decoder, yet brain-alignment studies usually score the visual encoder, one layer, or the final hidden state. We trace the representational geometry of image-token positions after every decoder block of 13 open checkpoints (256M–10B parameters) and compare it with seven visual-cortex regions in eight Natural Scenes Dataset (NSD) participants and, for 10 checkpoints, nine participants of the independent Natural Object Dataset (NOD). Two depth patterns hold under the frozen correlation-distance protocol and under three feature-scaling controls, and are absent from depth-matched untrained decoders: alignment with the lateral stream rises over the first three quarters of the decoder (+0.040 to +0.046 of the noise ceiling), and early-cortex alignment falls in the last quarter (negative in 13 of 13 NSD checkpoints; participant-by-stimulus intervals exclude zero in 7), as does alignment with the other streams. The large change at the returned final state, which at image positions no later computation reads, is instead a measurement effect: three channels carry a median 83% of the energy of the mean image token at the last read state and 34% after the output normalization, and standardizing features shrinks the final lateral gain from +0.030 to +0.005. Feature scaling also decides which checkpoints appear to move from early toward lateral geometry (6 of 13 raw, 11 standardized). Endpoint directions agree across datasets in 19 of 20 cells under the frozen metric and in 14 with standardized features, where the lateral rise transfers and the early decline is uncertain on NOD. The depth decomposition was defined after inspecting the curves and is exploratory. Layer-wise VLM–brain comparisons should standardize features and report the last read state alongside final-layer scores.