acceptodds
Under review as a conference paper at ICLR 2027

Cautious Context Steering for Language Model Personalization

Abstract

User context can personalize a language model without per-user fine-tuning, but its influence is not equally helpful at every token. Context Steering (CoS) uses a fixed steering strength throughout generation and requires two LM forward passes per step. We propose Cautious Context Steering (CCS), a lightweight adapter that learns when and how strongly to steer a frozen LM using user context. We train the adapter through cautious distillation, using oracle targets built from predictions with and without context. It learns to match the steered distribution where context helps predict the user's preferred token and the context-free base distribution elsewhere. Trained only on PRISM, the adapter outperforms context-based and reward-guided baselines in both in-domain generation and evaluation on three out-of-domain benchmarks. We also introduce GSM8K-P to evaluate math accuracy and task-specific format preferences together. Trained separately on this benchmark, CCS achieves high format adherence while maintaining competitive task accuracy, offering a better balance than the baselines. The CCS adapter adds less than 1.5% of the backbone's parameters. At inference, each decoding step uses one context-conditioned LM forward pass and a lightweight adapter evaluation, reducing generation time on PRISM by approximately 29-39% compared with CoS. Our code and data are available at https://anonymous.4open.science/r/cautious-cos-0512

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Preference Learning, Personalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yvnBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Cautious Context Steering (CCS), a small adapter trained to control how much LM relies on user context, uses one LM forward pass per decoding step.

On PRISM and three other datasets, CCS shows small gains in reference-based scores and generates faster than Context Steering (CoS). The authors also introduce GSM8K-P to test math accuracy alongside personalization. CCS is trained separately for this benchmark and evaluated on both answer correctness and whether it follows the user’s preferred format.

Critical strengths

The paper is generally clear and easy to follow. It connects the problem of unhelpful user context to the proposed method, making the motivation and main design choices understandable.

CCS offers a useful way to reduce the inference cost of context steering. By moving the context comparison into training, it reduces reported generation time by about 29–39% compared with CoS. This efficiency benefit gives the method practical value.

Critical weaknesses
  1. Evidence of personalization. The ROUGE and BERTScore results in Table 1 mainly used to measure reference similarity, but do not directly establish that users prefer the generated outputs. GSM8K-P provides evidence of format adherence, but preference-based evaluation or history-swapping controls on the naturalistic datasets would strengthen the central personalization claim.

  2. The contribution of cautious distillation is not sufficiently isolated. Figure 4 includes a History-SFT baseline on GSM8K-P, but Table 1 lacks a corresponding PRISM-trained SFT or simple adapter baseline evaluated under the same protocol. The small gains over ICL, together with the absence of uncertainty estimates, make it difficult to establish the advantage of the method.

Questions for authors
  1. The paper provides GSM8K-P's partitions and retrieval procedure, as well as the main adapter architecture and hyperparameters. However, for PRISM and the three OOD datasets, please report the numbers of users and examples, clarify the user-level splits, and specify how histories are selected. Please also list the prediction-distribution features used by the score and gate heads.

  2. The motivating failures are already known. AdaCAD [1] already discusses excessive adjustment from static decoding, while SteerX [2] already discusses history content that does not reflect user preferences. The paper should clarify what new insight it adds about these failures.

  3. Table 1 need simple SFT or adapter baseline trained on the same data with a similar budget, also the choice of λ = −0.5 for CoS also needs validation on the current tasks. All gains over ICL in Table 1 are below one point, For Qwen3-4B, BERTScore-F1 increases by only 0.08 points on UF-P-4, and ROUGE-1 by 0.12 points on Psoups.

  4. The loss ablations do not isolate the benefit of selective distillation. A comparison using the same adapter and training budget, but applying oracle targets at all eligible positions, would better test the core design.

  5. GSM8K-P uses a fixed mapping between math and extraction formats. CCS may learn this mapping to infer math preferences from extraction examples, which complicates the claim that it handles irrelevant history effectively.

[1] Wang et al. AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge. arXiv:2409.07394 [2] Zhao et al. SteerX: Disentangled Steering for LLM Personalization. arXiv:2510.22256

5 comments

Thank you for taking the time to review our paper. Your comments helped us see where we needed to explain our work more clearly. We address each of your concerns and questions below.

W1. The ROUGE and BERTScore results in Table 1 mainly used to measure reference similarity, but do not directly establish that users prefer the generated outputs.

We agree that high ROUGE and BERTScore do not necessarily mean that users prefer a model’s responses. Also, model can score well on both metrics simply by closely matching the reference responses. To address this concern, we conducted an additional LLM-as-a-judge evaluation using gpt-6-luna api. We compared CCS and the baselines, including History-SFT, against ICL responses in Qwen3-4B and 8B. We evaluated each pair twice, switching which response came first. This helps reduce bias from the order of the responses. If the two judgments did not agree, we counted the result as a tie.

Qwen3-8B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)42.2149.9274.5166.4141.82
SPL+RGD (vs ICL)27.5945.6063.0561.3432.68
History-SFT (vs ICL)42.3545.1771.7723.5948.88
CCS (Ours) (vs ICL)58.2055.7473.4749.3457.35
Qwen3-4B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)48.2251.9967.3458.7342.59
SPL+RGD (vs ICL)24.1945.6547.3835.0926.88
History-SFT (vs ICL)34.6242.8151.0424.5842.94
CCS (Ours) (vs ICL)55.9652.5359.8651.7353.61

Our CCS performs strongly in this preference-based evaluation. In particular, CCS achieves the highest win rates on PRISM, UF-P-4, and PersonalLLM for both model sizes among the evaluated methods. On PRISM, CCS outperforms the strongest evaluated baseline by 15.85% with Qwen3-8B. On PersonalLLM, the gains are 8.47% and 10.67% with Qwen3-8B and Qwen3-4B, respectively. On PSoups and Reddit TLDR, CCS outperforms the other evaluated training-based baselines.

These results show that CCS also performs well in the preference-based evaluation. This suggests that it captures user preferences and generates better personalized responses.

The judge results are reported in Table 1, and the evaluation protocol and prompts are provided in Appendix C.8.

W2. Table 1 lacks a corresponding PRISM-trained SFT or simple adapter baseline evaluated under the same protocol.

We have added the ROUGE and BERTScore results for History-SFT to Table 2 and 15. For the training-based methods, we now report results averaged over three random seeds, together with standard deviation. Representative results for Qwen3-8B are shown below. The remaining results are included in the revised paper.

Our central point is that fine-tuning an LM with user preference as context can be more than personalization actually requires. History-SFT directly optimizes the reference responses through cross-entropy loss, which can give it an advantage on the generation metrics in Table 2 and 15. However, the LLM-as-a-judge results in W1 and the GSM8K-P results show that it struggles with preference alignment and preserving the model’s original task performance.

CCS instead identifies the sparse set of tokens affected by user preference context and learns how much to adjust their probabilities. Its improvements on reference-based generation metrics can therefore be modest, while its gains in preference-based evaluation are much stronger. As shown in W1, CCS consistently achieves substantially higher win rates than History-SFT across both model sizes. Although History-SFT achieves higher generation metrics in some cases, CCS remains strong compared with the other methods. Together with the LLM-as-a-judge evaluation in W1 and the GSM8K-P results, these findings show that CCS can personalize responses effectively while preserving the model’s underlying capabilities. Given the slow generation of CoS and the weaknesses of History-SFT beyond reference-based metrics, CCS offers a highly competitive and practical approach to personalization.

DatasetMethodROUGE-1ROUGE-LBERTScore-F1
PRISM (ID)Base34.0719.2485.49
ICL36.1120.8486.74
CoS (-0.5)35.4220.3186.14
SPL+RGD34.03 ± 0.0919.24 ± 0.0585.42 ± 0.11
History-SFT35.52 ± 0.0323.03 ± 0.0487.96 ± 0.01
CCS (Ours)36.73 ± 0.0521.34 ± 0.0386.97 ± 0.01
UF-P-4 (OOD)Base36.5123.3685.55
ICL36.9523.9185.95
CoS (-0.5)37.0423.8985.82
SPL+RGD36.43 ± 0.2923.27 ± 0.1085.49 ± 0.09
History-SFT35.08 ± 0.0124.25 ± 0.0386.57 ± 0.01
CCS (Ours)37.12 ± 0.0024.07 ± 0.0286.13 ± 0.00
PSoups (OOD)Base34.6423.7186.00
ICL34.6124.3086.64
CoS (-0.5)35.8224.9386.53
SPL+RGD34.69 ± 0.0923.66 ± 0.0685.96 ± 0.04
History-SFT34.35 ± 0.0826.24 ± 0.0287.96 ± 0.02
CCS (Ours)35.04 ± 0.2024.94 ± 0.2086.97 ± 0.03
Reddit TLDR (OOD)Base26.9419.3988.19
ICL34.0725.3489.03
CoS (-0.5)29.3421.6888.48
SPL+RGD26.90 ± 0.0419.37 ± 0.0488.20 ± 0.02
History-SFT35.88 ± 0.0827.26 ± 0.0589.52 ± 0.01
CCS (Ours)34.31 ± 0.1025.52 ± 0.1489.12 ± 0.02
PersonalLLM (OOD)Base42.9023.2786.59
ICL43.6324.2386.96
CoS (-0.5)43.7724.0686.99
SPL+RGD42.68 ± 0.4323.21 ± 0.1486.52 ± 0.10
History-SFT35.98 ± 0.1021.82 ± 0.0586.75 ± 0.01
CCS (Ours)43.80 ± 0.1624.27 ± 0.0687.19 ± 0.04

Q1. PRISM and the three OOD datasets, please report the numbers of users and examples, clarify the user-level splits, and specify how histories are selected. Please also list the prediction-distribution features used by the score and gate heads.

The table below summarizes the datasets used in our experiments. Each example contains a prompt and a pair of responses (chosen and rejected).

DatasetPartitionUsers / preference groupsExamples
PRISMtraining1,07618,324
PRISMtest2603,285
UF-P-4OOD test4 preference profiles11,936
PSoupsOOD test6 dataset-defined user IDs2,674
Reddit TLDROOD test30 annotators6,537
PersonalLLMOOD test998 users4,990

The table below describes how we split each dataset and select user histories and evaluation queries. PRISM training and test participants are disjoint. For OOD evaluation, we use the PRISM-trained model without additional fine-tuning.

DatasetPartitioning procedure
PRISMWe split participants into training and held-out groups at an 80/20 ratio. For held-out participants, earlier conversations provide support histories, and later conversations are used as evaluation queries.
UF-P-4We draw support histories from each profile’s training pool and use its official test split for evaluation queries.
PSoupsWithin each user ID, we split the official test records into support and query sets in their original order.
Reddit TLDRWithin each annotator, we split the official validation records into support and query sets in their original order.
PersonalLLMFor each simulated user, we use the first four interactions as support history and the next five as evaluation queries, following the dataset’s interaction order.

For training, we use up to four preference pairs from preceding conversations. At evaluation time, we remove any overlap between the history and query examples, then select the first four distinct history examples in a fixed order. For a fair comparison, all context-based methods use the same user history for each evaluation example.

The score head uses each candidate’s standardized logit, probability, and normalized rank to decide how to adjust it. The gate head uses the sum and maximum of the active candidates’ probabilities to control the strength of the adjustment.

We have added these details to the our revised paper, including dataset construction and history selection in Appendix C.1 (Tables 7–8) and the score and gate features in Appendix C.3.

Q2. The motivating failures are already known. The paper should clarify what new insight it adds about these failures.

As we understand it, AdaCAD[1] studies excessive steering when the conflict between context and the model’s internal knowledge is small. It adjusts the steering strength based on the difference between the model’s prediction distributions with and without context. SteerX[2] finds that non-preferred information in history can weaken the personalization signal. It uses an LLM to select preference-related information and reconstruct the context used for steering.

In our Motivation section, we examine task accuracy and format adherence separately under controlled conditions. We find that increasing the influence of preference context can improve format adherence while reducing task accuracy. In the settings we studied, the steering strength that works well for one objective does not necessarily work well for the other.

These observations motivate our use of distillation and preservation. We aim to learn when and how strongly to adjust the model’s predictions, so that CCS can improve personalization while minimizing the loss of task performance.

We have added a comparison with these prior steering methods to the Related Work section of our revised paper (Appendix F).

[1] Wang, Han, et al. "Adacad: Adaptively decoding to balance conflicts between contextual and parametric knowledge." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

[2] Zhao, Xiaoyan, et al. "Steerx: Disentangled steering for llm personalization." arXiv preprint arXiv:2510.22256 (2025).

Reviewer_t6FHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Cautious Context Steering (CCS), a shared lightweight adapter that learns token-level logit corrections through selective distillation. CCS reduces inference cost relative to CoS and is evaluated on PRISM, three out-of-domain datasets, and a separately trained GSM8K-P setting.

Critical strengths
  1. CCS removes the second LM forward pass required by CoS, reducing measured generation time.
  2. A single PRISM-trained adapter improves over ICL across three OOD datasets without additional training.
Critical weaknesses
  1. Main results rely exclusively on ROUGE and BERTScore, which measure reference similarity rather than user preference satisfaction.
  2. Evaluation covers only Qwen3-0.6B/4B, while GSM8K-P uses only 0.6B with separate training. Main experiments lack matched-budget History-SFT or standard-distillation adapter baselines, and CoS uses a borrowed fixed coefficient rather than reported validation tuning.
Questions for authors

See Weaknesses.

4 comments

Thank you for taking the time to review our paper. Below, we discuss the results of the additional experiments we conducted to address the weaknesses you identified.

W1. Main results rely exclusively on ROUGE and BERTScore, which measure reference similarity rather than user preference satisfaction.

We agree that high ROUGE and BERTScore do not necessarily mean that users prefer a model’s responses. Also, model can score well on both metrics simply by closely matching the reference responses. To address this concern, we conducted an additional LLM-as-a-judge evaluation using gpt-6-luna api. We compared CCS and the baselines, including History-SFT, against ICL responses in Qwen3-4B and 8B. We evaluated each pair twice, switching which response came first. This helps reduce bias from the order of the responses. If the two judgments did not agree, we counted the result as a tie.

Qwen3-8B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)42.2149.9274.5166.4141.82
SPL+RGD (vs ICL)27.5945.6063.0561.3432.68
History-SFT (vs ICL)42.3545.1771.7723.5948.88
CCS (Ours) (vs ICL)58.2055.7473.4749.3457.35
Qwen3-4B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)48.2251.9967.3458.7342.59
SPL+RGD (vs ICL)24.1945.6547.3835.0926.88
History-SFT (vs ICL)34.6242.8151.0424.5842.94
CCS (Ours) (vs ICL)55.9652.5359.8651.7353.61

Our CCS performs strongly in this preference-based evaluation. In particular, CCS achieves the highest win rates on PRISM, UF-P-4, and PersonalLLM for both model sizes among the evaluated methods. On PRISM, CCS outperforms the strongest evaluated baseline by 15.85% with Qwen3-8B. On PersonalLLM, the gains are 8.47% and 10.67% with Qwen3-8B and Qwen3-4B, respectively. On PSoups and Reddit TLDR, CCS outperforms the other evaluated training-based baselines.

These results show that CCS also performs well in the preference-based evaluation. This suggests that it captures user preferences and generates better personalized responses.

The judge results are reported in Table 1, and the evaluation protocol and prompts are provided in Appendix C.8.

W2-1. Evaluation covers only Qwen3-0.6B/4B, while GSM8K-P uses only 0.6B with separate training. Main experiments lack matched-budget History-SFT or standard-distillation adapter baselines.

We have added the ROUGE and BERTScore results for Qwen3-8B and matched-budget History-SFT to Table 2. Representative results for Qwen3-8B are shown below.

DatasetMethodROUGE-1ROUGE-LBERTScore-F1
PRISM (ID)Base34.0719.2485.49
ICL36.1120.8486.74
CoS (-0.5)35.4220.3186.14
SPL+RGD34.03 ± 0.0919.24 ± 0.0585.42 ± 0.11
History-SFT35.52 ± 0.0323.03 ± 0.0487.96 ± 0.01
CCS (Ours)36.73 ± 0.0521.34 ± 0.0386.97 ± 0.01
UF-P-4 (OOD)Base36.5123.3685.55
ICL36.9523.9185.95
CoS (-0.5)37.0423.8985.82
SPL+RGD36.43 ± 0.2923.27 ± 0.1085.49 ± 0.09
History-SFT35.08 ± 0.0124.25 ± 0.0386.57 ± 0.01
CCS (Ours)37.12 ± 0.0024.07 ± 0.0286.13 ± 0.00
PSoups (OOD)Base34.6423.7186.00
ICL34.6124.3086.64
CoS (-0.5)35.8224.9386.53
SPL+RGD34.69 ± 0.0923.66 ± 0.0685.96 ± 0.04
History-SFT34.35 ± 0.0826.24 ± 0.0287.96 ± 0.02
CCS (Ours)35.04 ± 0.2024.94 ± 0.2086.97 ± 0.03
Reddit TLDR (OOD)Base26.9419.3988.19
ICL34.0725.3489.03
CoS (-0.5)29.3421.6888.48
SPL+RGD26.90 ± 0.0419.37 ± 0.0488.20 ± 0.02
History-SFT35.88 ± 0.0827.26 ± 0.0589.52 ± 0.01
CCS (Ours)34.31 ± 0.1025.52 ± 0.1489.12 ± 0.02
PersonalLLM (OOD)Base42.9023.2786.59
ICL43.6324.2386.96
CoS (-0.5)43.7724.0686.99
SPL+RGD42.68 ± 0.4323.21 ± 0.1486.52 ± 0.10
History-SFT35.98 ± 0.1021.82 ± 0.0586.75 ± 0.01
CCS (Ours)43.80 ± 0.1624.27 ± 0.0687.19 ± 0.04

History-SFT directly optimizes the reference responses through cross-entropy loss, which can give it an advantage on the generation metrics in Table 2. However, the LLM-as-a-judge results in W1 and the GSM8K-P results show that it struggles with preference alignment and preserving the model’s original task performance.

CCS instead identifies the sparse set of tokens affected by user preference context and learns how much to adjust their probabilities. Its improvements on reference-based generation metrics can therefore be modest, while its gains in preference-based evaluation are much stronger. As shown in W1, CCS consistently achieves substantially higher win rates than History-SFT across both model sizes. Although History-SFT achieves higher generation metrics in some cases, CCS remains strong compared with the other methods.

Also, we extended the GSM8K-P evaluation to Qwen3-4B. CCS and History-SFT were each trained on the same GSM8K-P training data. All methods were evaluated on the same questions with the same generation settings. The table below reports joint success (%), the percentage of responses that give the correct mathematical answer (task accuracy) and follow the user’s preferred output format (format adherence).

MethodHistory 4History 8History 16History 32
Base8.598.598.598.59
ICL85.0080.7876.4169.06
CoS (λ=−0.5\lambda=-0.5)47.9744.2233.2829.22
CoS (λ=+0.5\lambda=+0.5)80.7884.3874.6968.44
History-SFT75.9474.2268.9166.88
CCS (Ours)84.6983.2876.5674.38

CCS outperformed History-SFT at every history size. With histories of 16 and 32 examples, where the longer context includes more noise from other tasks, CCS achieved the highest joint success rate. It was also competitive at history sizes of 4 and 8. At History 32, CCS achieved 82.66% task accuracy and 89.84% format adherence. The corresponding scores were 81.25% and 82.34% for ICL, and 71.56% and 89.38% for History-SFT. These results support our claim that CCS can maintain high task accuracy while following the user’s preferred format, even with a larger model and longer histories.

Together with the LLM-as-a-judge evaluation in W1 and the GSM8K-P results, these findings show that CCS can personalize responses effectively while preserving the model’s underlying capabilities. Given the slow generation of CoS and the weaknesses of History-SFT beyond reference-based metrics, CCS offers a highly competitive and practical approach to personalization.

W2-2. CoS uses a borrowed fixed coefficient rather than reported validation tuning.

The CoS logits are given by

ℓCoS=ℓctx+λ(ℓctx−ℓbase)=(1+λ)ℓctx−λℓbase.\ell^{\mathrm{CoS}}=\ell^{\mathrm{ctx}}+\lambda(\ell^{\mathrm{ctx}}-\ell^{\mathrm{base}})=(1+\lambda)\ell^{\mathrm{ctx}}-\lambda\ell^{\mathrm{base}}.

When λ=−1\lambda=-1 corresponds to Base, and λ=0\lambda=0 corresponds to ICL. For −1<λ<0-1<\lambda<0, the CoS logits are a weighted average of the Base and ICL logits. Values below -1 move away from Base in the opposite direction of the context effect, while values above 0 increase the influence of context beyond ICL.

We used λ=−0.5\lambda=-0.5 across all model sizes and datasets. In the LaMP experiment in the original CoS paper, this value achieved the highest ROUGE metrics among the tested values. Also, λ=−0.5\lambda=-0.5 is mathematically equivalent to averaging the Base and ICL logits.

The amount of context that is useful can vary across datasets, user histories, queries, and even tokens within a response. The original CoS paper also notes that the choice of λ\lambda depends on the setting. Even if λ\lambda is tuned for each dataset, a single fixed value cannot account for all of these differences. Choosing a value for each input would require an additional selection criterion or quality evaluation.

We also examined how the CoS steering strength affects generation metrics. Using Qwen3-4B, we ran an experiment on PRISM test set using Qwen3-4B. We tested values from -1.5 to 1.5 in increments of 0.2, and include -1.0 and 0.0 as the Base and ICL reference points.

In this sweep, λ=−0.1\lambda=-0.1 gave the best ROUGE scores and BERTScore-F1. λ>0\lambda>0 did not improve the results, and the scores dropped as λ\lambda increased further.

λ\lambdaROUGE-1ROUGE-LBERTScore-F1
−1.532.4918.2784.76
−1.332.7418.5084.99
−1.133.4219.0385.29
−1.0 (Base)33.7719.2585.48
−0.933.9719.3885.64
−0.734.9120.0786.05
−0.535.7820.7886.54
−0.336.1821.0786.79
−0.136.2721.2086.87
0.0 (ICL)36.2621.1986.87
+0.136.0820.9886.86
+0.336.0721.0586.86
+0.535.9920.9486.87
+0.735.9020.8686.84
+0.935.6220.5786.78
+1.135.5320.4886.76
+1.335.3320.3586.68
+1.534.8020.0586.57

Our CCS also achieved higher mean scores on all three metrics than CoS at the best value found in this sweep.

MethodROUGE-1ROUGE-LBERTScore-F1
CoS ($\lambda=-0.5$)35.7820.7886.54
CoS ($\lambda=-0.1$)36.2721.2086.87
CCS36.5721.4187.09

We have explained our choice of λ = −0.5 and included the results of this additional experiment in Appendix B.2 of our revised paper (Figure 6 and Table 6).

Thank you again for reviewing our paper. We understand you are busy, but would appreciate your review of our responses and revised paper. If they address your concerns, we would be grateful if you could update your rating accordingly.

Reviewer_muJ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Cautious Context Steering (CCS), a lightweight adapter that is trained to decide whether and to what extent to use the user’s history and preference context when generating a response for a given task/query. The generation backbone itself is frozen and only the adapter (less than 1.5% of the backbone’s parameters) is trained. It extends the ideas introduced with the Context Steering (CoS) method from a training-free approach with a single steering coefficient across the response generation to a trained approach with a per-token coefficient. A new optimization objective “cautious distillation” is introduced to train the adapter. This training objective pushes the token probabilities towards an oracle-guided distribution when the context makes the user’s preferred token more likely and otherwise aligns with the context-free token probabilities. At inference time, it only requires a single forward pass per decoding step, which reduces inference time by about 29-39% compared to CoS (which requires two passes). The adapter is trained on PRISM and evaluated on PRISM and three out-of-domain benchmarks (UF-P-4, Psoups, and Reddit TLDR) with Qwen3-0.6B and Qwen3-4B. The paper’s findings suggest that this technique can modestly improve ROUGE and BERTScore over ICL on every metric and over CoS on most. They also present a new benchmark GSM8K-P to enable a more controlled evaluation of alignment with the user’s response format preference, separately from correctness.

Critical strengths

Overall, the paper addresses a clear limitation of Context Steering with a simple and practical method, as the generation model backbone stays frozen and a single adapter is shared across users (so no per-user training or weights required). Key strengths include:

  1. The motivation is clear and well supported. The paper presents detailed analysis on how if you use a single fixed steering strength as previously used by techniques like CoS, there can be accuracy - preference adherence trade-offs, and an oracle steering analysis suggests that steering strength is only relevant for a subset of tokens (although this analysis is limited to a single example). This clearly motivates the cautious distillation objective presented. Tricks and techniques like limiting the size of the plausible token set dynamically also make the method more efficient.
  2. The paper reports latency in addition to quality measurements on several benchmarks, demonstrating the method is efficient as compared to CoS. The adapter adds less than 1.5% of the backbone’s parameters and removes the second forward pass required by CoS, reducing inference time by approximately 29-39%. It is also 6-11x faster than reward-guided decoding (SPL+RGD) but still achieves better performance on almost all metrics.
  3. The approach is demonstrated to be generalizable. Although the adapter was trained only on PRISM, CCS also shows improvements across all three out-of-domain benchmarks reported in the paper for both generation backbone model sizes explored. This suggests the adapter learns to generalize how to use user context.
  4. The paper introduces an interesting new benchmark, GSM8K-P, as a more controlled method of personalization evaluation. Because the preference pairs differ only in answer format and the answers are verifiable for correctness, it enables a better separation of response accuracy / correctness from preference adherence.
Critical weaknesses
  1. The CCS adapter is trained on PRISM but the baselines are mostly training-free methods. The main goal of the adapter is to increase the probability of the preferred token based on the context when relevant. It seems like something similar could be achieved by supervised finetuning on the preferred PRISM responses. Perhaps consider adding a LoRA SFT with a similar number of adapter parameters as a baseline. This will help demonstrate the value of the cautious distillation objective introduced in the paper.
  2. The improvement over ICL seems small for most tasks, which brings up two concerns. First, whether from a real user perspective, the response quality improvement is actually noticeable and worth the extra inference cost (~20% per sample over ICL from Table 2). It would be useful to discuss whether this overhead is worth it, especially as this approach requires white-box access, so it only works for open-weight models. And second, whether the improvement is really significant or within the variance. I would recommend potentially reporting an averaged score across multiple adapter trainings for CCS, and confidence intervals for all methods.
  3. Benchmarks that use reward-model scores and LLM-judge win rates would be useful in addition to ROUGE and BERTScore against the chosen response, to understand whether users would actually prefer the CCS responses. This is especially relevant for PRISM, where the chosen responses come from many different LLMs, so overlap-based metrics may partly reward matching a particular model’s style.
  4. The average margin of improvement for the CCS approach over the ICL baseline generally drops as model size is scaled up from Qwen3-0.6B to Qwen3-4B, with Reddit TLDR the only exception. It is unclear if this trend will continue and thus whether the technique is less useful the more powerful the base model is. Evaluating a larger open-weight backbone (e.g. Qwen3-8B or larger) would help show whether the benefit holds where ICL is stronger.
  5. AdaCAD (Wang et al., NAACL 2025) also adapts the context-contrast strength per token in a training-free setting but is not cited.
Questions for authors
  1. Suggestion: As the goal of the paper is to develop a method that learns when to use context and shift distributions at a token-level, it would be useful to have more analysis of how the trained adapter shifts these probabilities, e.g. how often the gate is active and how its predicted strength compares to the oracle strength. There is really only a single example provided in the paper, and the benchmarks don't really measure things like the impact of irrelevant or contradictory context and other such issues that this approach should lessen the impact of.
  2. Suggestion: Add an adapter or LoRA baseline of similar size trained with cross-entropy on the chosen responses, and a preference-based evaluation/benchmark.
  3. Suggestion: More details can be provided on the adapter training cost.
  4. Clarification: Are the results in Table 1 from a single checkpoint from a single adapter training run? Consider reporting variance across multiple training runs, and confidence intervals for all methods, given the small differences between CCS and ICL.
  5. Clarification: Was the CoS steering coefficient tuned? If not, should it have been to maximie performance?
5 comments

Thank you for taking the time to review our paper. Below, we address each of your concerns and questions.

W1. It seems like something similar could be achieved by supervised finetuning on the preferred PRISM responses. Perhaps consider adding a LoRA SFT with a similar number of adapter parameters as a baseline. This will help demonstrate the value of the cautious distillation objective introduced in the paper.

W2-2. Whether the improvement is really significant or within the variance. I would recommend potentially reporting an averaged score across multiple adapter trainings for CCS, and confidence intervals for all methods.

and W4. Evaluating a larger open-weight backbone (e.g. Qwen3-8B or larger) would help show whether the benefit holds where ICL is stronger.

We have added the ROUGE and BERTScore results for matched-budget History-SFT(LoRA-based) to Table 2 and 15. For the training-based methods, we now report results averaged over three random seeds, together with standard deviation. The remaining results are included in the revised paper.

DatasetMethodROUGE-1ROUGE-LBERTScore-F1
PRISM (ID)Base34.0719.2485.49
ICL36.1120.8486.74
CoS (-0.5)35.4220.3186.14
SPL+RGD34.03 ± 0.0919.24 ± 0.0585.42 ± 0.11
History-SFT35.52 ± 0.0323.03 ± 0.0487.96 ± 0.01
CCS (Ours)36.73 ± 0.0521.34 ± 0.0386.97 ± 0.01
UF-P-4 (OOD)Base36.5123.3685.55
ICL36.9523.9185.95
CoS (−0.5)37.0423.8985.82
SPL+RGD36.43 ± 0.2923.27 ± 0.1085.49 ± 0.09
History-SFT35.08 ± 0.0124.25 ± 0.0386.57 ± 0.01
CCS (Ours)37.12 ± 0.0024.07 ± 0.0286.13 ± 0.00
PSoups (OOD)Base34.6423.7186.00
ICL34.6124.3086.64
CoS (−0.5)35.8224.9386.53
SPL+RGD34.69 ± 0.0923.66 ± 0.0685.96 ± 0.04
History-SFT34.35 ± 0.0826.24 ± 0.0287.96 ± 0.02
CCS (Ours)35.04 ± 0.2024.94 ± 0.2086.97 ± 0.03
Reddit TLDR (OOD)Base26.9419.3988.19
ICL34.0725.3489.03
CoS (-0.5)29.3421.6888.48
SPL+RGD26.90 ± 0.0419.37 ± 0.0488.20 ± 0.02
History-SFT35.88 ± 0.0827.26 ± 0.0589.52 ± 0.01
CCS (Ours)34.31 ± 0.1025.52 ± 0.1489.12 ± 0.02
PersonalLLM (OOD)Base42.9023.2786.59
ICL43.6324.2386.96
CoS (−0.5)43.7724.0686.99
SPL+RGD42.68 ± 0.4323.21 ± 0.1486.52 ± 0.10
History-SFT35.98 ± 0.1021.82 ± 0.0586.75 ± 0.01
CCS (Ours)43.80 ± 0.1624.27 ± 0.0687.19 ± 0.04

History-SFT directly optimizes the reference responses through cross-entropy loss, which can give it an advantage on the generation metrics in Table 2 and 15. However, the LLM-as-a-judge results in W2-1 and the GSM8K-P results show that it struggles with preference alignment and preserving the model’s original task performance.

CCS instead identifies the sparse set of tokens affected by user preference context and learns how much to adjust their probabilities. Its improvements on reference-based generation metrics can therefore be modest, while its gains in preference-based evaluation(LLM-as-a-judge) are much stronger. As we show in our response to the next weakness, our CCS consistently achieves substantially higher win rates than History-SFT across model sizes. Although History-SFT achieves higher generation metrics in some cases, CCS remains strong compared with the other methods.

W2-1. Whether from a real user perspective, the response quality improvement is actually noticeable and worth the extra inference cost (~20% per sample over ICL from Table 2). Especially as this approach requires white-box access, so it only works for open-weight models.

and W3. Benchmarks that use reward-model scores and LLM-judge win rates would be useful in addition to ROUGE and BERTScore against the chosen response, to understand whether users would actually prefer the CCS responses.

Regarding model access, CCS keeps the base model’s parameters fixed. It can be applied when the model provides the next-token logits and the last hidden state at each decoding step, and the logits can be adjusted before the next token is sampled.

CCS improves on ICL and generalizes to OOD datasets. However, we agree that ROUGE and BERTScore alone do not tell us whether users prefer its responses or whether the improvement justifies the additional inference cost.

To further assess response quality, we conducted an additional LLM-as-a-judge evaluation using the gpt-6-luna API. We compared CCS and the baselines, including History-SFT, against ICL using Qwen3-4B and Qwen3-8B. We evaluated each pair twice, switching which response came first. This helps reduce bias from the order of the responses. If the two judgments did not agree, we counted the result as a tie.

Qwen3-8B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)42.2149.9274.5166.4141.82
SPL+RGD (vs ICL)27.5945.6063.0561.3432.68
History-SFT (vs ICL)42.3545.1771.7723.5948.88
CCS (Ours) (vs ICL)58.2055.7473.4749.3457.35
Qwen3-4B (%)PRISMUF-P-4PSoupsReddit TLDRPersonalLLM
CoS (vs ICL)48.2251.9967.3458.7342.59
SPL+RGD (vs ICL)24.1945.6547.3835.0926.88
History-SFT (vs ICL)34.6242.8151.0424.5842.94
CCS (Ours) (vs ICL)55.9652.5359.8651.7353.61

Our CCS performs strongly in this preference-based evaluation. In particular, CCS achieves the highest win rates on PRISM, UF-P-4, and PersonalLLM for both model sizes among the evaluated methods. On PRISM, CCS outperforms the strongest evaluated baseline by 15.85% with Qwen3-8B. On PersonalLLM, the gains are 8.47% and 10.67% with Qwen3-8B and Qwen3-4B, respectively. On PSoups and Reddit TLDR, CCS outperforms the other evaluated training-based baselines.

These results show that CCS also performs well in the preference-based evaluation. This suggests that it captures user preferences and generates better personalized responses.

The judge results are reported in Table 1, and the evaluation protocol and prompts are provided in Appendix C.8.

W5. AdaCAD (Wang et al., NAACL 2025) also adapts the context-contrast strength per token in a training-free setting but is not cited.

AdaCAD[1] adjusts the steering strength at each token without training, based on the difference between the model’s predictions with and without context. CCS learns when and how strongly to adjust the model’s predictions through distillation and preservation, aiming to improve personalization while minimizing the loss of task performance. We have cited AdaCAD and discussed its relationship to CCS in the Related Work section of our revised paper (Appendix F).

[1] Wang, Han, et al. "Adacad: Adaptively decoding to balance conflicts between contextual and parametric knowledge." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Q1. Suggestion: As the goal of the paper is to develop a method that learns when to use context and shift distributions at a token-level, it would be useful to have more analysis of how the trained adapter shifts these probabilities, e.g. how often the gate is active and how its predicted strength compares to the oracle strength.

Thank you for the suggestion. We analyzed the gate and token-level corrections in the PRISM test set.

During generation, the average value of the continuous gate was 0.185. We also examined the gate and corrections using the same prefixes from the preferred responses. The table below groups token positions by the effect of context on the preferred token’s probability.

Effect of context on the preferred tokenMean gateTop-1 token change rate
Increases its probability0.2477.05%
Neutral0.1380.52%
Decreases its probability0.2436.56%

CCS therefore preserved the original top-1 prediction at 99.48% of neutral positions, while applying more corrections at positions where context had a stronger effect.

To assess how the corrections relate to the oracle, we compared the probability distributions before and after correction with the oracle distribution using the KL divergence from the training objective. All distributions were defined over the same candidate token set. CCS brought the predicted distribution closer to the oracle on average. For test tokens that met the training selection criterion, the mean KL divergence decreased from 0.2723 to 0.2430 (10.8%). Across all test tokens where the oracle could be computed, it decreased from 0.6936 to 0.4643 (33.1%). We have added these gate and oracle alignment analyses to our revised paper (Appendix D.3, Tables 13–14).

Q2. Add an adapter or LoRA baseline of similar size trained with cross-entropy on the chosen responses, and a preference-based evaluation/benchmark.

As we mentioned in W1, we added History-SFT to the comparison. Its LoRA adapter is trained with user context and has roughly the same number of trainable parameters as the CCS adapter.

We also report the results of a preference-based LLM-as-a-judge evaluation in W2-1 and W3 above.

Q3. More details can be provided on the adapter training cost.

We measured training costs using Qwen3-8B on the PRISM training split, with 4 RTX 4090 GPUs and an effective batch size of 64.

MethodGPU-hours / epochPeak allocated memory / GPU
SPL2.8719.40 GiB
History-SFT2.0518.69 GiB
CCS2.8820.06 GiB

We calculated GPU-hours per epoch by multiplying the elapsed training time by the number of GPUs and dividing by the number of completed epochs. Peak memory is the maximum memory allocated by PyTorch on any of the four GPUs. Although History-SFT trains faster, CCS remains competitive in training cost while achieving better personalization. We have included these training cost results in our revised paper (Appendix C.5, Table 9).

Q4. Are the results in Table 1 from a single checkpoint from a single adapter training run? Consider reporting variance across multiple training runs, and confidence intervals for all methods, given the small differences between CCS and ICL.

As explained in W2-2, we report standard deviation for the trained methods based on three runs with different seeds. For Base, ICL, and CoS, we use greedy decoding, so changing the seed does not affect their results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Key Revisions

Thank you for taking the time to review our paper. We have addressed all concerns and questions raised by the reviewers in our responses and revised the paper with additional experiments and clarifications. The revisions are highlighted in blue. We summarize the key revisions below.

  1. Preference-based evaluation beyond reference similarity (muJ9, t6FH, yvnB).

    We added LLM-as-a-judge comparisons against ICL using Qwen3-4B and 8B. CCS achieved the highest win rates on PRISM, UF-P-4, and PersonalLLM at both model sizes, and higher win rates against ICL than History-SFT in all model-dataset settings (Table 1, Appendix C.8).

  2. A comparable SFT baseline, larger-model evaluation, and training variability (muJ9, t6FH, yvnB).

    We added History-SFT, a LoRA baseline trained on the same data with a comparable number of trainable parameters, and extended evaluation to Qwen3-8B. We also report means and standard deviations over three training seeds for CCS, History-SFT, and SPL+RGD (Tables 2 and 15).

  3. Sensitivity to the CoS coefficient (muJ9, t6FH, yvnB).

    We use λ=−0.5\lambda=-0.5 across models and datasets, following the original CoS paper. This gives equal weight to the Base and ICL logits. However, the best steering strength can vary across user histories, queries, and tokens. We therefore added a coefficient sweep on PRISM with Qwen3-4B. CCS achieved higher scores on all three generation metrics even at its best-performing coefficient in the sweep (Appendix B.2, Figure 6 and Table 6).

  4. GSM8K-P evaluation on larger models and clarification of format assignments (t6FH, yvnB).

    We extended GSM8K-P to Qwen3-4B and 8B. CCS also achieved higher joint success than baselines on both backbones. CCS helps to follow task-specific preferences while preserving task accuracy (Appendix E.2, Figures 8–9). We clarified that the fixed cyclic format mapping applies only to evaluation (Appendix A.1).

  5. Selective distillation and token-level analysis (muJ9, t6FH, yvnB).

    On PRISM with Qwen3-4B, CCS achieved a 59.32% win rate in a direct LLM-as-a-judge comparison against a variant applying oracle distillation at all eligible positions (Appendix D.2, Table 12). We also compared CCS and ICL at the token level. CCS rarely changed top-1 prediction where user context had little effect on the preferred token’s probability. It also reduced the average KL divergence from the oracle distribution (Appendix D.3, Tables 13 and 14).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.