Contextual Control of Transformers

Computational neuroscience models show that gain modulation can alter neural computation while connectivity remains fixed. Inspired by this principle, we ask whether context can similarly provide an explicit control signal over computation in a pretrained Transformer while its weights remain fixed. We introduce contextual control, a lightweight learned pathway that maps contextual states to bounded gains over feed-forward computation. Because the control signal is explicit, we can manipulate it while holding the input and prediction target fixed. Across GPT-2 and Qwen, contextual control improves prediction beyond static and token-only controls. We then ask whether the learned signal is merely context-dependent or whether its assignment to a particular context matters. Moving control between occurrences of the same token in different contexts changes prediction, and in pre-tanh control coordinates its absolute effect exceeds that of perturbations matched in layer-wise magnitude on both backbones. The assigned control also contributes positively to prediction on average, as neutralizing it increases loss. Together, these interventions show that contextual assignment has causal consequences within the learned control interface, providing an explicit framework for studying how context regulates computation in frozen Transformers.