TTShift : Learning to Adapt at Test Time with Shared Shifts

Unlabeled test-time learning adapts LLMs using objectives available on target inputs, although these objectives only indirectly specify the reasoning behavior we ultimately care about. We first show that this distinction matters in practice: input next-token cross-entropy can continue to improve even as target accuracy, previously correct answers, and off-target behavior deteriorate. This motivates a complementary question: under such indirect supervision, how much update freedom is actually needed for useful adaptation? We introduce \method (Test-Time Shift), which freezes pretrained transformations and learns lightweight, dataset-shared additive shifts from unlabeled target inputs. Using Q/V projection-output shifts with a single fixed learning rate, \method improves mean accuracy over the corresponding Base model across seven reasoning tasks by $+3.38$ percentage points on Qwen3-8B-Base and $+0.94$ points on Qwen3.5-4B-Base. Controlled Q/V comparisons with rank-8 LoRA show that useful adaptation can persist after removing the learned input-dependent mapping correction: shared shifts exhibit a broader useful operating region and reach favorable gain–preservation operating points across the measured adaptation strengths. Further analysis shows that the shared direction can emerge from only partial cross-example gradient agreement and becomes directionally organized before its magnitude is fully accumulated, while downstream behavior continues to evolve. Together, these results suggest that effective test-time adaptation need not rely on highly expressive updates, and that restricting update freedom can provide a simple and useful way to adapt LLM reasoning behavior under indirect supervision.Anonymous code for reproducing the main experiments is available at https://anonymous.4open.science/r/review-code-FDE3/.