Matching Only Readouts Tells Incomplete Stories: Temporal Context Shapes Decoding Gains in an Invasive BCI Foundation Model

Frozen neural foundation models are often evaluated with readouts, which are small networks trained to predict behavior from a temporal window of neural signals or frozen features. The backbone can integrate activity outside the raw-signal readout's window, so matching readout architectures does not ensure the same temporal context. We study how much of the frozen-feature decoding gain depends on the readout's temporal context. We use a model pretrained by masked reconstruction on intracortical spikes and surface electrocorticography (ECoG) without behavioral supervision. On MC-RTT and MC-Maze, two datasets from macaque reaching tasks, we test whether additional temporal context reduces the decoding gain from frozen features. Each readout's input window is widened at fixed parameter count, while both receive inputs matched in time and the same explicit information about each prediction's position within the trial. We also examine whether Gaussian smoothing, which averages neighboring spike counts before decoding, reduces the gap without pretraining, selecting the kernel width on validation data. Widening reduces the $R^2$ gap between frozen-feature and raw-spike readouts by 89% on MC-RTT and 68% on MC-Maze, mainly through improved raw-spike decoding. At the widest window tested, the gap on MC-Maze averages 0.05 $R^2$ and is positive in every replicate. At that window, readouts using Gaussian-smoothed spikes perform nearly as well as those using frozen features on MC-Maze and better on MC-RTT. The results support temporal integration as a substantial source of the model's decoding gain on both tasks and leave open whether its frozen features also encode more abstract behavioral structure.