Cautious Context Steering for Language Model Personalization

User context can personalize a language model without per-user fine-tuning, but its influence is not equally helpful at every token. Context Steering (CoS) uses a fixed steering strength throughout generation and requires two LM forward passes per step. We propose Cautious Context Steering (CCS), a lightweight adapter that learns when and how strongly to steer a frozen LM using user context. We train the adapter through cautious distillation, using oracle targets built from predictions with and without context. It learns to match the steered distribution where context helps predict the user's preferred token and the context-free base distribution elsewhere. Trained only on PRISM, the adapter outperforms context-based and reward-guided baselines in both in-domain generation and evaluation on three out-of-domain benchmarks. We also introduce GSM8K-P to evaluate math accuracy and task-specific format preferences together. Trained separately on this benchmark, CCS achieves high format adherence while maintaining competitive task accuracy, offering a better balance than the baselines. The CCS adapter adds less than 1.5% of the backbone's parameters. At inference, each decoding step uses one context-conditioned LM forward pass and a lightweight adapter evaluation, reducing generation time on PRISM by approximately 29-39% compared with CoS. Our code and data are available at https://anonymous.4open.science/r/cautious-cos-0512