acceptodds
Under review as a conference paper at ICLR 2027

When and Why Activation Steering Degrades Long Generations?

Abstract

Activation steering offers a computationally efficient mechanism for steering large language model output during inference. It has been observed that continuous residual-stream interventions, especially during long-form autoregressive decoding, induce semantic collapse and cause coherence loss. Although there have been recent attempts to mitigate this issue, it remains unclear why activation steering is robust over short generations but degrades rapidly during long-form generation. Existing analyses of activation steering primarily characterize its effects at individual generation steps, leaving their dependence on the evolving context over the entire generation trajectory insufficiently understood. We develop a theoretical framework that casts autoregressive generation with steering as a stochastic recursion for concept and degradation scores of the generation. At each step, steering modifies the next-token distribution, and the sampled token updates the context scores that govern subsequent expected token scores. A mean-field analysis characterizes score trajectory saturation and the dependence of degradation onset on steering strength. We describe how to estimate the framework's quantities from model activations and evaluate its predictions across eight instruction-tuned language models with 3B to 14B parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Activation Steering, Autoregressive Generation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kzcBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why activation steering works well for short generations but causes coherence degradation in longer ones. The authors propose a stochastic framework to model how concept and degradation scores evolve during generation. They use mean-field analysis to characterize score saturation and predict when degradation occurs under different steering strengths. The paper also examines the role of the KV cache and validates the framework across eight language models and seven persona traits.

Critical strengths
  1. Important problem and well-motivated perspective. The paper addresses an important limitation of activation steering: interventions that work well for short generations may lead to coherence degradation during longer generations. By explicitly considering how previously generated tokens influence subsequent steering effects, the paper provides a useful perspective beyond existing fixed-context analyses.

  2. A coherent theoretical framework for long-term steering dynamics. The paper develops a stochastic framework for modeling the evolution of concept and degradation scores under continuous steering. The mean-field analysis provides theoretical characterizations of score saturation, degradation-threshold crossings, and the dependence of exit times on steering strength.

  3. A useful distinction between KV-cache effects and autoregressive feedback. The paper provides a theoretical bound on the direct contribution of historical steering interventions through the KV cache under explicit Transformer assumptions. The corresponding experiments hold the token sequence fixed to isolate this contribution.

  4. Extensive empirical evaluation and connection to theoretical predictions. The experiments cover eight language models, seven persona traits, and multiple steering layers and coefficients. The authors provide a measurement methodology that connects the theoretical quantities to model activations and evaluate predictions of score trajectories, saturation, and degradation onset. The reported agreement between predicted and observed trajectories provides encouraging evidence for the framework's descriptive value.

Critical weaknesses
  1. Appendix C.4 reports that, at the 1024-token horizon, the predicted coefficient limits agree with the observed limits in only 39 out of 168 configurations, with overestimation in 126 configurations. This discrepancy raises concerns about the reliability of the framework for guiding steering coefficient selection.

  2. The degradation direction is constructed from activation differences between high-temperature and ordinary generations. Although the paper reports strong coherence discrimination and an association between threshold crossings and lower judged coherence, it does not separately evaluate how well the score captures different degradation patterns, such as repetition, semantic drift, and excessive persona expression.

  3. Proposition 2 establishes a sequence-length-independent bound on the direct residual-stream contribution of historical steering interventions. However, the bound does not exclude bounded growth before saturation or characterize how cache-induced changes in token probabilities may influence subsequent autoregressive generation.

Questions for authors
  1. How well does the proposed framework generalize to unseen steering coefficients or longer generation horizons when its dynamics are estimated using a limited set of steering conditions or shorter generations?
  2. What explains the substantial overestimation of coefficient limits at the 1024-token horizon? Does the discrepancy primarily arise from errors in estimating the drift functions, degradation thresholds, or stochastic variability?
5 comments
AuthorsOpenReview ↗

Response to Reviewer kzcB (Part 1/4)

Reply to Reviewer kzcB

Thank you for the constructive review and helpful questions. Below, we respond to each point and present the additional analyses and clarifications.

W1/Q2

Thank you for the constructive question. Our stochastic recursion includes token-sampling variability, but the original coefficient limits were calculated from deterministic exit times. Individual answers can cross the degradation boundary even when the mean-field trajectory stays below it, with a higher chance of crossing over longer responses. This distinction matters because the coefficient limit depends on the fraction of answers that cross.

We therefore added the prediction procedure in Algorithm 1 described in our reply to tFkm (W2/Q2), retaining each answer's observed prefix score instead of the mean initial score used by the deterministic estimate. We added either Gaussian noise or block-resampled noise, obtained by sampling blocks of centered residuals from calibration continuations. Note that the measured drift curves and degradation thresholds remain fixed among them.

To examine drift estimation, we then changed the data used to estimate the conditional mean. The original curve primarily uses continuations of concept/contrast responses and unsteered answers. Continuations of steered prefixes fill bins with insufficient observations. We tested two alternatives: giving these steered-prefix continuations priority, or estimating the mean directly along recorded steered answers.

For each alternative, we split the questions into two groups, estimated the replacement curve from one group, and simulated scores for the other, then swapped roles. Where replacement data were insufficient, we used the original estimation procedure with the evaluated questions excluded. Both alternatives used the same original residual pool and block sampler. The steered-answer estimate serves as a reference because it uses full trajectories from other questions at the evaluated coefficients to calibrate the mean.

At 1024 tokens and crossing tolerance ρ=1/2\rho=1/2, we compare coefficient limits across 168 configurations. "Exact matches" counts predictions equal to the limit observed in generated answers, and "Overestimates" counts predictions above it.

MethodMean calibration dataExact matchesOverestimates
Mean field (deterministic)Original39126
+ Gaussian noiseOriginal8470
+ Block-resampled noiseOriginal10040
+ Block-resampled noiseSteered-prefix continuations13122
+ Block-resampled noiseSteered answers (reference)14213

Accounting for variability reduces overestimates from 126 to 40 without changing the drift, and 159 limits are within one grid step of the observed value. Replacing the conditional mean reduces overestimation further, showing that drift estimation also matters. We interpret these replacements as drift diagnostics because threshold calibration and residual pools remain shared across the two question groups.

Regarding threshold calibration, we kept the original exit boundaries fixed for both predicted and observed crossings. This includes the 0.95 quantile used to estimate dmax⁡d_{\max} and the estimated shifts gd(λ)g_d(\lambda). The gains above therefore do not require threshold recalibration.

We added the variability and drift comparisons to Appendix C.5. We also added Algorithm 1 to Section 3.2 to specify how estimated crossing probabilities determine the coefficient limit.

AuthorsOpenReview ↗

Response to Reviewer kzcB (Part 2/4)

Q1

Thank you for raising this question. We agree that it is important to test whether the estimated dynamics remain predictive when the coefficient or generation length changes. To examine this, we tested unseen coefficients using a limited set of calibration coefficients, and later score trajectories using only short calibration text.

All predictions start from the observed context score after 16 tokens. To assess the trajectories, we compared predicted and observed mean context scores up to 1024 tokens, averaging over the same responses that reach each position.

Unseen steering coefficients. We calibrated only at λ∈0,0.25,0.5,1\lambda\in\\{0,0.25,0.5,1\\}, also excluding prefixes generated at stronger coefficients. From these measurements, we fitted the linear concept shift κsλ\kappa_s\lambda and quadratic degradation shift κλ2\kappa\lambda^2. Applying these shifts to the unsteered conditional-mean curve gives predictions at new coefficients without calibration continuations at those coefficients.

To test this, we evaluated four additional coefficients: 0.750.75 lies within the calibrated range, while 1.251.25, 1.751.75, and 2.252.25 lie progressively farther above it. For reference, we also predicted the same responses using a curve measured at each tested coefficient. Using the same expected score update for both lets us assess how well the fitted shifts replace calibration at the tested coefficient. The table reports median trajectory RMSE over the same 147 model-concept-layer settings with matching prompts. Lower values indicate better agreement with the observed mean trajectories.

Tested λ\lambdaConcept: limited calibrationDegradation: limited calibrationConcept: tested λ\lambdaDegradation: tested λ\lambda
0.750.1280.0370.1220.039
1.250.1770.0860.1440.084
1.750.2070.1330.1320.094
2.250.2060.2160.0990.098

At λ=0.75\lambda=0.75, both errors are close to those of the reference. Just outside the calibrated range, at λ=1.25\lambda=1.25, degradation error remains close, while concept error increases. The larger gaps at stronger coefficients therefore indicate a loss of accuracy as we move farther beyond the calibrated range.

Longer generations. We next tested whether short calibration text supports both later score predictions and coefficient selection. At each of the eight coefficients, we estimated a separate conditional mean and centered block residuals using the original calibration procedure. We compared calibration within the first 160 answer tokens with calibration from longer text. In both cases, we calibrated on one half of the questions and evaluated on the other, then swapped roles. The memory weight from the full calibration and the exit thresholds remain fixed, so the comparison isolates the effect of restricting the mean and noise data.

We then used the resulting score trajectories to predict mean degradation scores and select coefficients with Algorithm 1. The table reports median degradation trajectory RMSE over positions after token 160. The limit columns count correct predictions out of 168 at ρ=1/2\rho=1/2, with overestimates in parentheses. Both metrics are evaluated through the indicated response length.

Response lengthRMSE: short calibrationRMSE: longer calibrationCorrect limits: short calibrationCorrect limits: longer calibration
2560.0330.032114 (17)119 (19)
5120.0500.047102 (32)101 (36)
10240.0740.069104 (33)99 (37)

Across these lengths, restricting mean and noise calibration to the first 160 tokens leaves trajectory errors close to those from longer calibration, with comparable coefficient-limit accuracy. Short continuations can therefore be a useful choice when generating long calibration responses at every candidate λ\lambda is costly.

The coefficient comparison also shows useful transfer within or near the calibrated range, while extrapolating from low to much stronger steering is less reliable. A related limitation appears in Appendix C of [1], where behavior at large magnitudes departs from the linear steering response captured by their model. We added these comparisons to Appendix C.7 to clarify both forms of generalization.

[1] Bigelow et al., "Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering", arXiv:2511.00617, 2025.

AuthorsOpenReview ↗

Response to Reviewer kzcB (Part 3/4)

W2

Thank you for this suggestion. It motivated us to examine these patterns separately, using the existing degradation direction and score calibration.

We first screened generated responses by their fraction of repeated token 4-grams, using unsteered responses to set the cutoff. The table groups the 3,553 flagged responses by their existing coherence judgments and reports their mean degradation scores.

Coherence groupResponsesMean coherenceMean degradation score
≤25\le 253,3533.250.786
25<coherence<7525<\text{coherence}<754545.890.250
≥75\ge 7515597.480.059

Without further calibration for repetition, the score separates the low- and high-coherence groups well. Using higher degradation to identify the low-coherence group gives mean AUROC 0.991, computed within the same model, concept, and layer wherever both groups are represented, then averaged.

To examine topic drift, we then paired coherent answers to unrelated questions. We split each answer at a paragraph boundary into an opening and the remaining text. Keeping the first question as the prompt, we joined the first answer's opening to the remaining text of the second answer. We made the reverse combination for the second question. Each constructed answer therefore starts on topic and then moves to text written for the other question, with both pieces copied unchanged.

For this comparison, we used questions from the humorous and impolite evaluation sets. Each pair shared the same steering concept, model, layer, and coefficient. Thus, the topic changed while the target trait remained fixed.

Using the existing degradation direction, we then averaged token scores within each complete answer. The resulting answer scores distinguish the modified answers from the originals with mean AUROC 0.879, comparing responses under the same steering conditions.

The topic-switch test suggests that the same direction is also sensitive to a loss of relevance in otherwise readable text. We agree, however, that further work could improve how such directions distinguish coherent text from different forms of degradation in LLM hidden representations.

Note that our direction was constructed to capture prominent coherence breakdowns under strong steering, such as word salad and repetitive output. As shown in Appendix B.1 and the repetition table above, its scores agree well with LLM coherence judgments on steered answers. The topic-switch test extends that original focus to a form of degradation in which the text remains readable.

Regarding excessive persona expression, we agree that it can degrade an answer through repetition, topic drift, or loss of intelligibility. Strong expression of the trait need not itself imply a loss of coherence. We therefore focus on these consequences, which help us distinguish degradation from the expression of the intended trait. We added the repetition and topic-switch analyses to Appendix C.4.

Reviewer_UqpAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how activation steering changes an answer as LLM generation proceeds. The authors argue that previous works primarily study LLM steering at a fixed context for next-token predictions and cannot adapt to evolving generation dynamics. To address this gap, the authors propose to track token-wise concept and degradation scores with decaying memory, use conditional expected token scores to derive a continuous approximation of how these scores evolve. This mean-field approximation predicts when context scores stabilize or cross a reference boundary. Experiments on persona steering compare these predictions with actual observed scores and judged coherence.

Critical strengths
  1. The theory makes a useful feedback mechanism explicit: steering changes a token, which changes the context for later edits. The analysis of saturation and threshold crossing makes this idea precise under stated assumptions.

  2. The empirical study verifies the theory part extensively within the persona steering domain. It covers several model families, includes unsteered baselines, and evaluates the degradation score against different coherence judgments. The cases where observations depart from the predictions are also informative.

Critical weaknesses
  1. The theory still needs a clearer bridge to ordinary transformer steering: The local steering calculation relies on an exponential-tilt law from an idealized unconstrained-features model, where each context has its own independently learned feature vector instead of sharing the transformer encoder. Later transformer layers and layer normalization do not necessarily follow this law [1]. However, in this paper, the experiments edit intermediate-layer transformer states, leaving the applicability of this local derivation unclear. Also, Appendix B.2 checks the assumed score responses, but it does not directly test zero covariance between the degradation score and r(y)r(y), the score that reweights token probabilities, under the unsteered next-token distribution. This condition is needed for the quadratic degradation prediction in Proposition 1. Considering the good predictive fit in experiments, it is still possible that some theories may exist and we can build some similar models upon it. But in the current form, the connection of the theory part to ordinary transformer steering is not clear.

  2. Potentially over-engineered fitting process: the parameter fitting and calibration process still requires additional generated data and empirical estimation. This leaves an open question of how much the proposed structure adds beyond a predictor fitted to recent score history. In this view, the theory's practical value beyond this empirical fitting remains unclear.

  3. Synthetic Experiments and Limited Generalization: To quantify the degradation scores, the paper proposes to constructs a score from ordinary unsteered and temperature-2 responses. It is important to note that temperature-2 is a relatively weak baseline for detecting text degradation. As shown in Figure 6, high temperature can produce visibly incoherent text. But in many real-world applications, text degradation could be more subtle to detect. For example, LLMs nowadays can generate very fluent and locally coherent sentences, but the whole generation may remain uninformative or later sentences contradict earlier ones in long-form generation. All these bring significant challenges on the generalization of the proposed framework.

References:

[1] Taimeskhanov, Magamed, Samuel Vaiter, and Damien Garreau. "Towards understanding steering strength." ICML 2026

Questions for authors
  1. Section 4 of [1] includes normalization and downstream effects beyond the idealized model. What evidence supports applying Proposition 1's exponential-tilt premise to the intermediate-layer interventions tested here?

  2. Appendix B.1 makes the degradation direction orthogonal to the concept direction. How does this relate to the zero-covariance condition involving d(y)d(y) and r(y)r(y) in Proposition 1? After this projection, does the degradation score retain its intended semantics? How should we interpret the resulting score and its direction? Also, in real-world LLM generation, people could use multiple rubrics to judge the generation quality, and they even exhibit more complicated interaction patterns. How could the proposed theory handle that case?

  3. Why does the decay coefficient measured on copied continuations supply the memory weight for both concept and degradation scores?

  4. Clarify how temporary crossings and the choice of token-, word-, or sentence-level summaries affect coefficient selection.

  5. The comparisons with a fixed context score and with unsteered trajectories are useful. A simple predictor using recent scores and steering strength would further test what the proposed dynamics add to forecasting. Have you trained such models, or prompted an LLM for this task, using comparable fitting data?

  6. Beyond the long-answer experiments in Appendix C.4, which assumptions or calibration steps would need to change for story writing and agentic scenarios?

11 comments
AuthorsOpenReview ↗

Response to Reviewer UpqA (Part 1/4)

Reply to Reviewer UqpA

We thank you for your constructive review and questions. Below, we address each point and explain the corresponding additions and clarifications to the paper.

W1/Q1: Generalizing the idealized model to transformer steering

Thank you for the constructive question. We agree that Proposition 1 is based on an idealized model, where steering gives the context-independent logit shift in Eq. 2. In practice, an intervention at an intermediate transformer layer passes through subsequent attention, feed-forward, and normalization layers. These transformations can make the logit response depend on the context and vary nonlinearly with steering strength. We therefore generalized the analysis to the transformer's output logits lλ(c)\boldsymbol{l}_\lambda(c) in Proposition A.1 (Appendix A.2), recovering Proposition 1 as a special case.

Theoretical: For a fixed context cc, suppose the logits lλ(y∣c)l_\lambda(y\mid c) are three times continuously differentiable in λ\lambda near zero (as in transformers with smooth attention, feed-forward, and normalization). Let rc(y)r_c(y) and kc(y)k_c(y) denote their first and second derivatives at zero, and write rˉc=Ep0[rc]\bar{r}_c=\mathbb{E}_{p_0}[r_c]. For either score z∈s,dz\in\\{s,d\\}, as λ→0\lambda\to0,

\mathbb{E}_{p_\lambda}[z]-\mathbb{E}_{p_0}[z] &=\lambda\operatorname{Cov}_{p_0}(z,r_c)\\\\ &\quad+\frac{\lambda^2}{2}\left[\operatorname{Cov}_{p_0}\bigl(z,(r_c-\bar{r}_c)^2\bigr) +\operatorname{Cov}_{p_0}(z,k_c)\right]+O(\lambda^3). \end{aligned}$$ Here all expectations and covariances condition on $c$. Because the derivatives come from the full forward pass, the general result accounts for downstream layers and normalization, allowing both a linear degradation response and logit curvature. Under the idealized logit shift $l_\lambda(y\mid c)=l_0(y\mid c)+\lambda r(y)$, we have $r_c=r$ and $k_c=0$. With the additional condition $\operatorname{Cov}_{p_0}(d,r)=0$, the expansion recovers Proposition 1. **Empirical:** We next tested the first-order logit approximation underlying Proposition A.1, $l_\lambda(y\mid c)\approx l_0(y\mid c)+\lambda r_c(y)$, which yields the same linear expected-score term $\lambda\operatorname{Cov}_{p_0}(z,r_c)$. Keeping prefixes of unsteered answers fixed, we estimated $r_c$ by a central difference around zero and reused it at every tested coefficient. Applying softmax to these approximated logits gives $$\widetilde{p}_\lambda(y\mid c) =\frac{p_0(y\mid c)\exp\bigl(\lambda\widehat{r}_c(y)\bigr)} {\sum_{y'\in\mathcal{V}}p_0(y'\mid c)\exp\bigl(\lambda\widehat{r}_c(y')\bigr)}.$$ For comparison, we also evaluated the idealized tilt in Eq. 2. We measure each approximation's KL error against the actual steered distribution $p_\lambda$. For each model, concept, and layer, we average this error over the same prefixes and divide by the mean error from using $p_0$, then average these ratios. Values below 1 indicate a closer approximation than the unsteered reference. | Probability approximation | $\lambda=0.25$ | $\lambda=0.5$ | $\lambda=1$ | $\lambda=2$ | $\lambda=3$ | |---|---:|---:|---:|---:|---:| | Eq. 2, estimated $r(y)$ | 1.247 | 1.205 | 1.130 | 1.031 | 0.992 | | Local tilt, estimated $r_c(y)$ | **0.009** | **0.037** | **0.154** | **0.533** | **0.870** | Using the context-dependent logit response gives a closer approximation than the estimated Eq. 2 tilt at every tested coefficient. Its mean error ratios also remain below 1, though they increase with steering strength, consistent with the local scope of the first-order approximation. We added this comparison to Appendix C.2. Note, however, that this generalization leaves the main theoretical framework unchanged. The saturation and exit results use Assumptions 1-2 without requiring the exponential-tilt law or zero covariance. Our main trajectory and crossing predictions also use conditional means measured directly at each steering coefficient. The local calculation motivates the forms used to fit the measured equivalent increases $g_z$, including the quadratic fit for $g_d$ used to estimate $\lambda_c$. We examine the covariance condition and compare this fit with a linear-plus-quadratic alternative in our reply to Q2 below.
AuthorsOpenReview ↗

Response to Reviewer UpqA (Part 2/4)

Q2: Covariance, score interpretation, and multiple rubrics

Regarding the covariance condition, orthogonalizing the directions does not impose zero covariance between d(y)d(y) and r(y)r(y) under the unsteered next-token distribution. In Proposition 1, this statistical condition removes the linear degradation term. Our direct measurements find nonzero covariances (Appendix C.2), which Proposition A.1 accommodates by retaining the linear term.

To assess the practical effect of allowing a linear term, we compared the quadratic calibration of gdg_d with a linear+quadratic fit. Using the same measurements at 0<λ≤30<\lambda\le3, we fitted κλ2\kappa\lambda^2 and κ1λ+κ2λ2\kappa_1\lambda+\kappa_2\lambda^2 with nonnegative coefficients, then used each fit in m^d,0(q+gd(λ))\widehat{m}_{d,0}(q+g_d(\lambda)) to predict coefficient limits with Algorithm 1. For each mean, we recomputed and centered residuals from the same continuations for block resampling, keeping evaluation settings and thresholds fixed.

The table reports mean RMSE against measured gdg_d and counts predicted coefficient limits equal to those observed in generated answers at each length, across 168 settings and at ρ=1/2\rho=1/2. Best values in each column are bold, including ties.

Fit for gdg_dMean calibration RMSE64 tokens128 tokens256 tokens512 tokens1024 tokens
Quadratic0.0059124134119121114
Linear + quadratic0.0046124126114114114

Allowing a linear term improves calibration, while coefficient-limit accuracy remains close across response lengths. Both fits recover the same number of limits at 64 and 1024 tokens, and the quadratic fit recovers 5-8 more at intermediate lengths. We added these comparisons to Appendix C.2 alongside the covariance measurements.

Regarding the interpretation after projection, the difference between the mean residual vectors of high-temperature and ordinary responses can contain a component along the target concept direction. Before projection, its mean absolute cosine similarity with the concept direction is 0.203 (median 0.192) across our model, concept, and layer settings. We remove this component so that an activation change purely along the concept direction does not itself change the degradation score.

To check whether the resulting score still captures coherence loss, we evaluate it on steered answers excluded from its construction. The projected score separates answers with low judged coherence (≤25\le25) from those with high judged coherence (≥75\ge75) with mean AUROC 0.990 (Appendix B.1). These coherence judgments are not used to construct the direction, so they provide an external check of its interpretation after projection. We also added the repetition and topic-switch analyses in Appendix C.4, which use the same projected direction to examine these patterns separately.

Regarding multiple rubrics, our framework can accommodate their scores jointly. We focus here on coherence while steering one concept, following the common emphasis on single-concept interventions in the literature. Interactions in multi-concept steering remain an active research challenge [3].

To include several quality rubrics, we would collect their context scores together with the concept score into a vector z(c)\boldsymbol{z}(c). With joint expected next-token scores mλ(z(c))\boldsymbol{m}_{\lambda}(\boldsymbol{z}(c)), the same context update gives the mean-field equation

=\boldsymbol{m}_{\lambda}(\bar{\boldsymbol{z}})-\bar{\boldsymbol{z}}.$$ Each component of $\boldsymbol{m}_{\lambda}$ can depend on all the scores, allowing the interactions you describe. A theoretical analysis of these coupled dynamics is beyond the scope of this work.
AuthorsOpenReview ↗

Response to Reviewer UpqA (Part 3/4)

W2/Q5: Prediction from recent scores

Thank you for suggesting this comparison. To test what the dynamics add beyond fitting recent score history, we trained linear regression and a small neural network on the same calibration text. Both predict the next token's degradation score from the preceding 16 scores and the steering features λ\lambda and λ2\lambda^2.

To compare their usefulness for choosing a steering coefficient, we used each predictor in the score update and crossing calculation of Algorithm 1. Each predictor simulates subsequent scores from the observed 16-token prefix, using its own centered block-resampled noise from the same calibration continuations. We keep answer lengths, update weights, and exit thresholds fixed. The table compares these predictors with conditional means measured at each coefficient and with the unsteered curve shifted using the quadratic fit for gdg_d. Entries count predicted coefficient limits that equal those observed in generated answers, out of 168 settings at ρ=1/2\rho=1/2.

Predictor64 tokens128 tokens256 tokens512 tokens1024 tokens
Linear regression11560202020
Neural network1261241059294
Conditional mean per λ\lambda (Ours)128130124103100
Quadratic fit for gdg_d (Ours)124134119121114

The neural network is competitive at 64 tokens, but using conditional means measured at each coefficient gives more correct coefficient limits at every reported length. At 1024 tokens, shifting the unsteered curve using the quadratic fit for gdg_d recovers 114 limits, compared with 94 for the neural network. These results support using the context-score dynamics for coefficient selection, alongside their role in deriving saturation and exit conditions. We added this comparison to Appendix C.5.

W3: Forms of degradation under steering

Thank you for the question. Our choice of score was motivated by the prominent failures we observed under strong steering, including repetitive output and visible loss of coherence. The contrast between ordinary and high-temperature responses is intended to capture this loss of coherence in the model's activations. This focus is consistent with the use of n-gram diversity metrics to assess generation quality in the steering literature (e.g., [1, 2]).

We examined repetition directly in our steered responses. As detailed in our reply to kzcB (W2) and Appendix C.4, the degradation score separates repetitive responses with low versus high judged coherence with mean AUROC 0.991. The same direction also distinguishes answers with constructed topic switches from their coherent originals with mean AUROC 0.879, suggesting sensitivity to loss of relevance in otherwise readable text. These results support the score's relevance to the steering failures motivating our study and encourage further investigation of the informativeness and consistency concerns you raise.

Q3: A common memory weight for both scores

Thank you for the question. We keep the continuation tokens fixed so that differences in their concept scores measure the effect of the preceding concept or contrast span. Fitting the decay of this effect calibrates the exponential retention (1−a)k(1-a)^k in our context summary. We use the resulting aa as a common approximation of recency for both scores.

To check this choice, we repeated the measurement with ordinary and high-temperature spans, using a continuation sampled after the ordinary span for both. We then fitted the decay of their degradation-score difference. The resulting weight is typically larger than the original estimate (median 2.01x), so we next tested its effect on coefficient selection.

Because changing the weight changes the context scores, we recalibrated the conditional means, residuals, and boundaries using the same recorded token scores. We then applied Algorithm 1 with block-resampled noise. The table counts predicted limits equal to the observed limits under each calibration, across 168 settings at ρ=1/2\rho=1/2.

Update weight64 tokens128 tokens256 tokens512 tokens1024 tokens
Shared128130124103100
Separate for each score1271341259998

Coefficient-limit accuracy remains close across lengths, with no consistent gain from a separate weight. These results support the shared weight as an effective approximation for our coefficient-selection experiments. We clarified the calibration in Section 3.2 and Appendix B.1 and added this comparison to Appendix C.5.

Reviewer_tFkmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why activation steering can degrade generation coherence during long-context generation. It introduces a simplified theoretical model of autoregressive feedback using concept and degradation scores, derives theoretical results on score saturation and degradation thresholds, and evaluates the framework across eight language models.

Critical strengths
  1. Meaningful research problem. Understanding the relationship between steering strength, generation length, and coherence degradation is important.
  2. Theoretical effort. The paper provides a mathematical framework for analyzing long-generation dynamics with activation steering, including saturation, critical steering coefficients, and degradation onset.
  3. Extensive empirical evaluation. Experiments cover multiple model families, steering concepts, and intervention layers.
Critical weaknesses
  1. Insufficient empirical validation of the theoretical assumptions. The main trajectory predictions do not directly enforce the proposed context-independent shift assumption. Therefore, accurate predictions do not necessarily validate the central theoretical mechanism.
  2. Limited long-horizon predictive reliability. Though trajectory fitting results look good, the predicted steering-coefficient limits at 1024 tokens are still unsatisfying.
Questions for authors
  1. Can the authors compare a theory-constrained predictor with an unconstrained empirical dynamics predictor to establish the contribution of the proposed theoretical assumptions?
  2. How do the authors explain the poor predictive performance at 1024 tokens, and what guidance can the proposed framework provide for steered long-generation?
9 comments
AuthorsOpenReview ↗

Response to Reviewer tFkm (Part 1/2)

Reply to Reviewer tFkm

Thank you for your constructive questions. We address your points below and describe the additional analyses and revisions.

W1/Q1

We thank the reviewer for pointing out this distinction and suggesting a direct comparison. Our original trajectory predictions use a separately estimated conditional-mean curve at each steering coefficient, so their accuracy alone does not establish the context-independent shift assumption.

To examine this assumption, we compared the independently estimated curve m^z,λ(q)\widehat m_{z,\lambda}(q) with the constrained curve m^z,0(q+g^z(λ))\widehat m_{z,0}(q+\widehat g_z(\lambda)), where qq is the context score and zz denotes concept or degradation. We estimate g^z(λ)\widehat g_z(\lambda) separately at each coefficient and keep it constant across qq.

We first compared the curves at λ=0.25,0.5,…,3\lambda=0.25,0.5,\ldots,3, using additional calibration continuations at the intermediate coefficients. We added Figure 16 in Appendix C.7 to show the measured curves together and Figures 17 and 18 to compare each with a shifted version of the same unsteered reference. Across the 117 configurations supporting this comparison at every coefficient, median curve RMSE over the common measured range is 0.007 / 0.004 at λ=0.5\lambda=0.5 and 0.020 / 0.036 at λ=3\lambda=3 for concept / degradation. These results support the shift as an approximation across the tested coefficients, with closer agreement under milder steering. Because the shifts are estimated from these measurements, we next assess their predictive performance on generated responses.

To also test the dependence on steering strength, we included a third predictor that fits the measured shifts to g^s(λ)=κsλ\widehat g_s(\lambda)=\kappa_s\lambda and g^d(λ)=κλ2\widehat g_d(\lambda)=\kappa\lambda^2. This adds the linear/quadratic dependence on λ\lambda to the context-independent shift. We then predicted score trajectories with all three predictors, using the same calibration data, context-update rule, and observed 16-token prefixes. At each subsequent position up to token 1024, we averaged predictions and observations over the same responses reaching it. The table reports median RMSE between these mean trajectories as concept / degradation (lower is better), over the same 123 configurations. We omit configurations without the required empirical shift estimates.

To evaluate coefficient selection as well, we applied Algorithm 1 with each predictor's own block-resampled noise and the same fixed thresholds, as described in our reply to W2/Q2. The final column counts correctly predicted coefficient limits at 1024 tokens and ρ=1/2\rho=1/2 on the original coefficient grid, for these same 123 configurations.

Conditional-mean predictorλ=0.5\lambda=0.5λ=1\lambda=1λ=2\lambda=2λ=3\lambda=3Correct coefficient limits
Conditional mean per λ\lambda0.112 / 0.0370.136 / 0.0520.121 / 0.1150.109 / 0.09278 / 123
Unsteered curve + shift estimated per λ\lambda0.117 / 0.0350.153 / 0.0530.146 / 0.1820.169 / 0.18085 / 123
Unsteered curve + linear/quadratic shifts0.128 / 0.0330.153 / 0.0540.162 / 0.1630.191 / 0.18785 / 123

Using a separate shift at each coefficient retains similar trajectory accuracy at λ=0.5,1\lambda=0.5,1. Both shift-based predictors recover 85 of the 123 coefficient limits, compared with 78 for independently estimated curves. Although their trajectory errors are larger at λ=2,3\lambda=2,3, the coefficient-limit results show that context-independent shifts remain useful for selecting steering strength.

We added the curve comparisons to Appendix C.7 and the trajectory comparison to Appendix C.5, alongside coefficient-selection results across all 168 configurations. We also clarified the context-independent shift assumption in Section 3.2.

AuthorsOpenReview ↗

Response to Reviewer tFkm (Part 2/2)

W2/Q2

Thank you for highlighting this gap. Our stochastic recursion includes randomness from token sampling, which the mean-field approximation removes by replacing token scores with their conditional means. These fluctuations can push individual answers across the degradation boundary while the deterministic trajectory stays below it. Over longer responses, there are more opportunities for such crossings, so good trajectory fits need not imply accurate coefficient limits.

To account for this variability, we developed the prediction procedure in Algorithm 1, which draws residuals from the stored calibration continuations. For a position jj sampled at step tt, we set

=\underbrace{d(y_j)-\widehat{m}_{d,\lambda}\big(d(c_j)\big)}_{\text{measured token-score residual}} -\bar{\xi}_j,$$ where $\widehat{m}_{d,\lambda}$ is the measured conditional mean and $\bar{\xi}_j$ is the mean residual in the bin containing $d(c_j)$. Thus, $\widehat{\xi}_t$ is a sampled, centered residual. We draw consecutive residuals within a continuation to retain local dependence, choosing each block's start near the simulated score. With these values and the measured weight $a$, we simulate context scores $\widetilde{d}_t$ without generating new tokens: $$\widetilde{d}_{t+1}=(1-a)\widetilde{d}_t+a\big(\widehat{m}_{d,\lambda}(\widetilde{d}_t)+\widehat{\xi}_t\big).$$ For comparison, we also use independent zero-mean Gaussian draws for $\widehat{\xi}_t$, with token-score variance measured within context-score bins. Both prediction variants start from each answer's observed score after 16 tokens, whereas the mean-field estimate uses the mean initial score. We run 100 paths per answer and count crossings up to the evaluation horizon or the answer's observed end. Note that we keep the measured curves and thresholds fixed and do not fit to observed exit times. We compare these predictions with the coefficient limit observed in the 60 generated answers at each tested coefficient, using crossing tolerance $\rho=1/2$. In each cell, the first number counts configurations (out of 168) whose predicted limit equals the observed limit. The number in parentheses counts overestimates. To check how sensitive the observed limits are to the sampled questions, we also redraw the 20 questions with replacement within each configuration, keep all three answers per selected question, and recompute the limits. The “Resampled questions” row averages these same counts over 1000 repetitions, always comparing with the limits from the original answers. | Method | 64 tokens | 256 tokens | 512 tokens | 1024 tokens | | -------------------------- | ----------: | -----------: | -----------: | -----------: | | Mean field (deterministic) | 122 (35) | 68 (92) | 43 (122) | 39 (126) | | + Gaussian noise | **130 (25)** | 116 (37) | 89 (66) | 84 (70) | | + Block-resampled noise | 128 (6) | **124 (18)** | **103 (39)** | **100 (40)** | | Resampled questions | 154.8 (6.2) | 154.2 (6.0) | 150.8 (9.6) | 151.7 (9.3) | The predictor with Gaussian noise already improves agreement, indicating that score variability accounts for part of the discrepancy. Block resampling improves the 1024-token predictions further, with 100 correct limits and 40 overestimates, compared with 39 and 126 for the mean-field estimate. We added Appendix C.5 to examine the remaining errors and sensitivity to the conditional-mean estimate. These findings led us to propose Algorithm 1, a new procedure for selecting steering coefficients from estimated crossing probabilities. For a specified response length and crossing tolerance $\rho$, it scans candidate coefficients in increasing order and returns the last accepted candidate before the estimated crossing fraction reaches $\rho$. We added the algorithm to Section 3.2, giving the framework an explicit selection rule that accounts for both generation length and sampling variability. We thank the reviewer again for the constructive feedback. We hope our responses, including the new predictor comparisons and the improved coefficient-selection with the new Algorithm 1 provide a clearer perspective on our work and are taken into consideration in your final assessment.
AuthorsOpenReview ↗

Follow-up on Rebuttal

We thank Reviewer tFkm again for the constructive review and questions. As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal.

(W1/Q1) The new comparisons support the predictive value of the context-independent shift assumption: the constrained predictor gives more correct coefficient limits at 1024 tokens than independently fitted curves. We also assess the assumption directly by comparing measured steered curves with shifted versions of the unsteered curve, providing evidence beyond the original trajectory fits (Appendices C.5 and C.7, Figures 16-18).

(W2/Q2) We added Algorithm 1 to turn the stochastic framework into a coefficient-selection method that accounts for the sampling variability omitted by mean-field estimates. It selects a coefficient for a specified response length and crossing tolerance using score simulations from existing calibration data. At 1024 tokens, correct limits increase from 39 to 100 of 168 and overestimates fall from 126 to 40 (Section 3.2 and Appendix C.5).

We hope these responses address your concerns and would be grateful if you would take them, together with the additional results, into consideration in your final assessment. We are happy to clarify any remaining points or carry out further analyses or experiments before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.