- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why activation steering works well for short generations but causes coherence degradation in longer ones. The authors propose a stochastic framework to model how concept and degradation scores evolve during generation. They use mean-field analysis to characterize score saturation and predict when degradation occurs under different steering strengths. The paper also examines the role of the KV cache and validates the framework across eight language models and seven persona traits.
Critical strengths
-
Important problem and well-motivated perspective. The paper addresses an important limitation of activation steering: interventions that work well for short generations may lead to coherence degradation during longer generations. By explicitly considering how previously generated tokens influence subsequent steering effects, the paper provides a useful perspective beyond existing fixed-context analyses.
-
A coherent theoretical framework for long-term steering dynamics. The paper develops a stochastic framework for modeling the evolution of concept and degradation scores under continuous steering. The mean-field analysis provides theoretical characterizations of score saturation, degradation-threshold crossings, and the dependence of exit times on steering strength.
-
A useful distinction between KV-cache effects and autoregressive feedback. The paper provides a theoretical bound on the direct contribution of historical steering interventions through the KV cache under explicit Transformer assumptions. The corresponding experiments hold the token sequence fixed to isolate this contribution.
-
Extensive empirical evaluation and connection to theoretical predictions. The experiments cover eight language models, seven persona traits, and multiple steering layers and coefficients. The authors provide a measurement methodology that connects the theoretical quantities to model activations and evaluate predictions of score trajectories, saturation, and degradation onset. The reported agreement between predicted and observed trajectories provides encouraging evidence for the framework's descriptive value.
Critical weaknesses
-
Appendix C.4 reports that, at the 1024-token horizon, the predicted coefficient limits agree with the observed limits in only 39 out of 168 configurations, with overestimation in 126 configurations. This discrepancy raises concerns about the reliability of the framework for guiding steering coefficient selection.
-
The degradation direction is constructed from activation differences between high-temperature and ordinary generations. Although the paper reports strong coherence discrimination and an association between threshold crossings and lower judged coherence, it does not separately evaluate how well the score captures different degradation patterns, such as repetition, semantic drift, and excessive persona expression.
-
Proposition 2 establishes a sequence-length-independent bound on the direct residual-stream contribution of historical steering interventions. However, the bound does not exclude bounded growth before saturation or characterize how cache-induced changes in token probabilities may influence subsequent autoregressive generation.
Questions for authors
- How well does the proposed framework generalize to unseen steering coefficients or longer generation horizons when its dynamics are estimated using a limited set of steering conditions or shorter generations?
- What explains the substantial overestimation of coefficient limits at the 1024-token horizon? Does the discrepancy primarily arise from errors in estimating the drift functions, degradation thresholds, or stochastic variability?
5 comments
Response to Reviewer kzcB (Part 2/4)
Q1
Thank you for raising this question. We agree that it is important to test whether the estimated dynamics remain predictive when the coefficient or generation length changes. To examine this, we tested unseen coefficients using a limited set of calibration coefficients, and later score trajectories using only short calibration text.
All predictions start from the observed context score after 16 tokens. To assess the trajectories, we compared predicted and observed mean context scores up to 1024 tokens, averaging over the same responses that reach each position.
Unseen steering coefficients. We calibrated only at , also excluding prefixes generated at stronger coefficients. From these measurements, we fitted the linear concept shift and quadratic degradation shift . Applying these shifts to the unsteered conditional-mean curve gives predictions at new coefficients without calibration continuations at those coefficients.
To test this, we evaluated four additional coefficients: lies within the calibrated range, while , , and lie progressively farther above it. For reference, we also predicted the same responses using a curve measured at each tested coefficient. Using the same expected score update for both lets us assess how well the fitted shifts replace calibration at the tested coefficient. The table reports median trajectory RMSE over the same 147 model-concept-layer settings with matching prompts. Lower values indicate better agreement with the observed mean trajectories.
| Tested | Concept: limited calibration | Degradation: limited calibration | Concept: tested | Degradation: tested |
|---|---|---|---|---|
| 0.75 | 0.128 | 0.037 | 0.122 | 0.039 |
| 1.25 | 0.177 | 0.086 | 0.144 | 0.084 |
| 1.75 | 0.207 | 0.133 | 0.132 | 0.094 |
| 2.25 | 0.206 | 0.216 | 0.099 | 0.098 |
At , both errors are close to those of the reference. Just outside the calibrated range, at , degradation error remains close, while concept error increases. The larger gaps at stronger coefficients therefore indicate a loss of accuracy as we move farther beyond the calibrated range.
Longer generations. We next tested whether short calibration text supports both later score predictions and coefficient selection. At each of the eight coefficients, we estimated a separate conditional mean and centered block residuals using the original calibration procedure. We compared calibration within the first 160 answer tokens with calibration from longer text. In both cases, we calibrated on one half of the questions and evaluated on the other, then swapped roles. The memory weight from the full calibration and the exit thresholds remain fixed, so the comparison isolates the effect of restricting the mean and noise data.
We then used the resulting score trajectories to predict mean degradation scores and select coefficients with Algorithm 1. The table reports median degradation trajectory RMSE over positions after token 160. The limit columns count correct predictions out of 168 at , with overestimates in parentheses. Both metrics are evaluated through the indicated response length.
| Response length | RMSE: short calibration | RMSE: longer calibration | Correct limits: short calibration | Correct limits: longer calibration |
|---|---|---|---|---|
| 256 | 0.033 | 0.032 | 114 (17) | 119 (19) |
| 512 | 0.050 | 0.047 | 102 (32) | 101 (36) |
| 1024 | 0.074 | 0.069 | 104 (33) | 99 (37) |
Across these lengths, restricting mean and noise calibration to the first 160 tokens leaves trajectory errors close to those from longer calibration, with comparable coefficient-limit accuracy. Short continuations can therefore be a useful choice when generating long calibration responses at every candidate is costly.
The coefficient comparison also shows useful transfer within or near the calibrated range, while extrapolating from low to much stronger steering is less reliable. A related limitation appears in Appendix C of [1], where behavior at large magnitudes departs from the linear steering response captured by their model. We added these comparisons to Appendix C.7 to clarify both forms of generalization.
[1] Bigelow et al., "Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering", arXiv:2511.00617, 2025.
Response to Reviewer kzcB (Part 3/4)
W2
Thank you for this suggestion. It motivated us to examine these patterns separately, using the existing degradation direction and score calibration.
We first screened generated responses by their fraction of repeated token 4-grams, using unsteered responses to set the cutoff. The table groups the 3,553 flagged responses by their existing coherence judgments and reports their mean degradation scores.
| Coherence group | Responses | Mean coherence | Mean degradation score |
|---|---|---|---|
| 3,353 | 3.25 | 0.786 | |
| 45 | 45.89 | 0.250 | |
| 155 | 97.48 | 0.059 |
Without further calibration for repetition, the score separates the low- and high-coherence groups well. Using higher degradation to identify the low-coherence group gives mean AUROC 0.991, computed within the same model, concept, and layer wherever both groups are represented, then averaged.
To examine topic drift, we then paired coherent answers to unrelated questions. We split each answer at a paragraph boundary into an opening and the remaining text. Keeping the first question as the prompt, we joined the first answer's opening to the remaining text of the second answer. We made the reverse combination for the second question. Each constructed answer therefore starts on topic and then moves to text written for the other question, with both pieces copied unchanged.
For this comparison, we used questions from the humorous and impolite evaluation sets. Each pair shared the same steering concept, model, layer, and coefficient. Thus, the topic changed while the target trait remained fixed.
Using the existing degradation direction, we then averaged token scores within each complete answer. The resulting answer scores distinguish the modified answers from the originals with mean AUROC 0.879, comparing responses under the same steering conditions.
The topic-switch test suggests that the same direction is also sensitive to a loss of relevance in otherwise readable text. We agree, however, that further work could improve how such directions distinguish coherent text from different forms of degradation in LLM hidden representations.
Note that our direction was constructed to capture prominent coherence breakdowns under strong steering, such as word salad and repetitive output. As shown in Appendix B.1 and the repetition table above, its scores agree well with LLM coherence judgments on steered answers. The topic-switch test extends that original focus to a form of degradation in which the text remains readable.
Regarding excessive persona expression, we agree that it can degrade an answer through repetition, topic drift, or loss of intelligibility. Strong expression of the trait need not itself imply a loss of coherence. We therefore focus on these consequences, which help us distinguish degradation from the expression of the intended trait. We added the repetition and topic-switch analyses to Appendix C.4.
Response to Reviewer kzcB (Part 1/4)
Reply to Reviewer kzcB
Thank you for the constructive review and helpful questions. Below, we respond to each point and present the additional analyses and clarifications.
W1/Q2
Thank you for the constructive question. Our stochastic recursion includes token-sampling variability, but the original coefficient limits were calculated from deterministic exit times. Individual answers can cross the degradation boundary even when the mean-field trajectory stays below it, with a higher chance of crossing over longer responses. This distinction matters because the coefficient limit depends on the fraction of answers that cross.
We therefore added the prediction procedure in Algorithm 1 described in our reply to tFkm (W2/Q2), retaining each answer's observed prefix score instead of the mean initial score used by the deterministic estimate. We added either Gaussian noise or block-resampled noise, obtained by sampling blocks of centered residuals from calibration continuations. Note that the measured drift curves and degradation thresholds remain fixed among them.
To examine drift estimation, we then changed the data used to estimate the conditional mean. The original curve primarily uses continuations of concept/contrast responses and unsteered answers. Continuations of steered prefixes fill bins with insufficient observations. We tested two alternatives: giving these steered-prefix continuations priority, or estimating the mean directly along recorded steered answers.
For each alternative, we split the questions into two groups, estimated the replacement curve from one group, and simulated scores for the other, then swapped roles. Where replacement data were insufficient, we used the original estimation procedure with the evaluated questions excluded. Both alternatives used the same original residual pool and block sampler. The steered-answer estimate serves as a reference because it uses full trajectories from other questions at the evaluated coefficients to calibrate the mean.
At 1024 tokens and crossing tolerance ρ=1/2, we compare coefficient limits across 168 configurations. "Exact matches" counts predictions equal to the limit observed in generated answers, and "Overestimates" counts predictions above it.
Accounting for variability reduces overestimates from 126 to 40 without changing the drift, and 159 limits are within one grid step of the observed value. Replacing the conditional mean reduces overestimation further, showing that drift estimation also matters. We interpret these replacements as drift diagnostics because threshold calibration and residual pools remain shared across the two question groups.
Regarding threshold calibration, we kept the original exit boundaries fixed for both predicted and observed crossings. This includes the 0.95 quantile used to estimate dmax and the estimated shifts gd(λ). The gains above therefore do not require threshold recalibration.
We added the variability and drift comparisons to Appendix C.5. We also added Algorithm 1 to Section 3.2 to specify how estimated crossing probabilities determine the coefficient limit.