acceptodds
Under review as a conference paper at ICLR 2027

Predicting Future Behaviors in Reasoning Models Enables Better Steering

Abstract

Developing white-box approaches to monitor and control Large Reasoning Model (LRM) behaviors requires understanding their decision-making process. Existing methods often rely on contrastive behavioral data that hides the probabilistic nature of LRM decision-making. We apply a forking-paths-style analysis to behavioral evaluations and find that as generation progresses, LRMs define evolving distributions over behavioral outcomes. To enable deployment-time prediction and intervention on these behavioral outcomes, we formally introduce the task of probabilistic future behavior prediction. We show that this prediction is possible in both white-box and text-only settings, and activation-based probes enable 64%-91% accuracy in predicting the most likely action from intermediate CoT steps. Finally, we show an application of probabilistic future behavior prediction by introducing a novel text-level steering method, Future Probe Controlled Generation. FPCG samples multiple candidate sentences and chooses the best one according to a probe predicting the future behavior likelihood. This enables steering with almost no output quality degradation. FPCG also enables steering in several evaluations where activation steering completely fails to produce coherent outputs.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, steering, large reasoning models, behavioral evaluation, CoT monitoring

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3586OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper predicts the probability of a model's eventual behavior from intermediate activations during reasoning. Repeated continuations provide probability labels for training the probes. The proposed FPCG method samples candidate sentences and chooses continuations using the predicted behavior probabilities. Experiments cover four models and six behaviors, compare steering strength and output quality with activation interventions, and examine differences between prediction and detection features.

Critical strengths
  1. Estimating conditional behavior probabilities from repeated continuations is more appropriate than treating the behavior of one completed response as a deterministic label for an intermediate state. FPCG changes behavior substantially across several models and tasks, showing that the prediction signal can be useful for steering.

  2. The evaluation includes formatting failures, perplexity, coherence, and general instruction following, alongside steering strength. Comparisons with detection features and text predictors, layer analyses, and compute measurements help characterize where the approach works and what it costs.

Critical weaknesses
  1. Appendix F.4 compares text-based FPCG, but budget-matched full-response reranking or an established guided decoder such as FUDGE is still missing. On sycophancy, ten candidates cost 16.6–16.9× FLOPs and 1.77–1.90× wall time per generated token on the two tested models (Table 5). This leaves the contribution of the extra search budget unresolved.

  2. Detection probes are trained only on final answers, while prediction probes also see CoT; their difference is confounded with training coverage. Likewise, agreement across ten base responses (Section 3.2) does not establish a decision made before reasoning or show that later CoT has no causal role.

  3. FPCG intervenes during both CoT and final-answer generation. Appendix F.2 compares probe training, but does not isolate the intervention phase. A CoT-only versus answer-only comparison is needed to establish how much of the gain requires prediction before the answer is written.

  4. Table 1 reports 15% filtering for DeepSeek's positive sycophancy steering and 18% for gpt-oss's positive wealth steering. Under negative SORRY-Bench steering (toward refusal), coherence win rates against unsteered DeepSeek/Qwen are 19.9%/15.5% among order-consistent judgments (Table 2). These exceptions qualify the quality-preservation claim; quality is also not compared at matched behavior rates.

  5. Appendix G.1's judge explicitly ignores correctness and helpfulness, while AlpacaEval uses separate general instructions. Neither establishes preserved performance on the steered tasks. In SEP, avoiding the injected string can still coincide with failing the legitimate task; its completion rate is not reported.

  6. Each behavior has 100 training and 100 test prompts; extra prefixes do not add independent problems. Stability across prompt splits and training seeds is not shown. MAE is reported, but neither MAE nor binarized accuracy alone establishes calibration against the underlying behavior probabilities estimated from finite continuations.

  7. The abstract's 64–91% accuracy claim refers to intermediate CoT, but Figure 4's evaluation includes final-answer positions. Also, gpt-oss's positive SORRY-Bench result—compliance, not refusal—is 43.3% in Table 1 versus 49.0% in Figure 28(a). Any difference in runs or settings should be stated.

Questions for authors
  1. Given that the detection probe is trained only on final answers and the prediction probe includes CoT positions, which results rule out training-position coverage as the explanation for their difference?

  2. Among SEP outputs that avoid the injected string, how often does the model complete the legitimate task specified by the system instruction? Was this measured separately?

  3. How are per-prompt behavior rates aggregated after filtering? For the settings with 15% or 18% filtered outputs, what fraction of all attempts both passes filtering and exhibits the target behavior?

  4. What causes the low coherence win rates under negative SORRY-Bench steering toward refusal—19.9% for DeepSeek and 15.5% for Qwen? How many comparisons survive the order-consistency filter?

  5. Why is gpt-oss's positive SORRY-Bench compliance rate 43.3% in Table 1 but 49.0% in Figure 28(a)? Do these come from different runs or configurations, and which corresponds to the main experiment?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3586 (1/2)

Thank you for the careful review. We are glad you find estimating conditional behavior probabilities "more appropriate" than single-response labels (this probabilistic future-behavior prediction is our central contribution) and that FPCG "changes behavior substantially across several models and tasks". Each point is answered below with new results.

W1: Budget-matched reranking and FUDGE

Do FPCG's gains come from prediction or from the extra search budget? We separate the two.

  • Same budget, different features. FPCG with a detection probe has an identical algorithm, candidate count and FLOPs; only the probe target differs. Prediction steers at least as far in 24/24 settings (DeepSeek, Qwen3-14B), significantly further in 14/24 (prompt-clustered CIs), by up to 17 pp. At a fixed budget, the prediction signal is what moves behavior.
  • FUDGE (revised App. F.6). We implemented FUDGE (Yang & Klein, 2021) with our prediction probe as the attribute classifier: at every token the top-17 next tokens are scored by the probe, and sampling follows p(t|x)·P(behavior|x,t). This matches the ~17× budget you cite. On the multiple-choice behaviors FUDGE matches FPCG. On free-form behaviors, where the outcome is decided over long reasoning, FPCG steers clearly further: Prompt Injection 28.2% vs 53.3% (−) and 82.1% vs 71.7% (+), Sycophancy 13.8% vs 28.2% (−) and 93.8% vs 88.0% (+), Refusal (+) 89.5% vs 53.0%. FUDGE steers further only on Refusal (−) (10.3% vs 29.0%), where 30% of its generations run to the 8192-token limit (FPCG: 2%). Overall FUDGE reaches 82% of FPCG's mean shift (DeepSeek, 12 settings).
  • Full-response reranking at ~17× FLOPs. Best-of-17 (revised App. F.5; 4 models, 48 settings; its scoring pass is not charged, which favors it). When suppressing free-form behaviors (Refusal, Prompt Injection, Sycophancy), FPCG wins 11/12 model–behavior pairs (e.g. DeepSeek Sycophancy 13.8% vs 28.8%); across all 48 settings the two are on par (mean shift 23.2 vs 22.1 pp). Reranking also must complete every response: on Prompt Injection (DeepSeek), best-of-17 executed the injection in 95% of prompts it returned clean, vs. 28% for FPCG, whose rejected candidates are single discarded sentences.

W2/Q1: Training-position coverage

Three controls (revised App. C.3; DeepSeek, layer 25; held-out MAE vs. future behavior):

  1. Matched count and stage coverage: the prediction probe's training data subsampled to exactly the detection probe's size, uniformly over reasoning stages (5 seeds): MAE rises by ≤.04 and stays below detection on all 6 behaviors.
  2. Matched positions: we apply the ground-truth behavior detector to every intermediate CoT prefix and train the detection probe on these labels, so it sees exactly the prediction probe's positions; only the target differs. It remains far behind on every free-form behavior where the detector applies to prefixes (Refusal .23 vs .11, Injection .40 vs .32, Sycophancy .34 vs .19). Controls 1 and 2 also hold on Qwen3-14B.
  3. Representations: LEACE concept erasure (Belrose et al., 2023) of all linearly decodable detection information collapses the retrained detection probe (worse than a constant predictor) but changes the retrained prediction probe by ≤.04 on all 6 behaviors; erasing the prediction concept leaves detection intact (≤.03).

With identical inputs, positions and sample counts the gap persists, and the two targets are carried by distinct features: training coverage does not explain it.

Sec. 3.2. We agree that agreement across ten responses is not a causal statement, and have rephrased Sec. 3.2 to state only that all sampled responses agree (revised App. C.4). Resampling independent of the 10 responses defining certainty: for certain prompts, fresh continuations (M = 10–30) from each intermediate prefix (0.9M prefixes, 4 models) leave the outcome exactly at 0 or 1 in 84% of prefixes and flip the majority behavior in only 3%. We do not claim the CoT has no causal role; on uncertain prompts we show the opposite (W3).

W3: CoT-only vs. answer-only intervention

We ran exactly this comparison (revised App. F.7) (6 behaviors × 2 directions). Probe-based selection is applied only before the end-of-thinking marker (CoT-only) or only after it (answer-only); in the other phase an unscored sampled sentence is committed. On DeepSeek, CoT-only FPCG recovers 86% of full FPCG's steering effect (mean shift 28.6 vs 33.3 pp on the same prompts), while answer-only recovers only 19% (6.4 vs 33.3 pp) and falls short of full FPCG in all 12 settings. This is not just a matter of how many interventions each phase gets: answer sentences are ~27% of FPCG's selection steps but yield 19% of the effect, while CoT steps (73%) yield 86%. Qwen3-14B replicates this (CoT-only 89%, answer-only 9% of the effect; 12 settings). Almost all of the gain requires predicting the behavior before the answer is written, which is what prediction features provide.

AuthorsOpenReview ↗

Response to Reviewer 3586 (2/2)

W4/Q3/Q4: Filtering, coherence, matched behavior

Aggregation (Q3, revised App. E.2). Per prompt we average behavior over parseable responses, drop prompts with none (0/100 prompts for DeepSeek Sycophancy +, 5/100 for gpt-oss Wealth +), then average over prompts. Counting every unparseable output as a failure, the fraction of all attempts that pass filtering and show the target behavior rises from 53.7% to 79.6% (DeepSeek Sycophancy +) and from 7.7% to 20.7% (gpt-oss Wealth +), and rises in 48/48 settings of Table 1. The 15% DeepSeek filtering is the model's own: unsteered DeepSeek filters 23% there.

Coherence (Q4). The low Refusal (−) win rates come from invisible padding tokens in the raw decoder output we judged, not from the text a user sees: FPCG pads finished candidates with end-of-sequence tokens (87% of DeepSeek Refusal (−) FPCG answers vs 0% unsteered), which the judge reads as "runaway repetition". Judging the detokenized text instead (all 12 settings; GPT-4o, both orders, 300 pairs each), FPCG wins 63% ± 8% of order-consistent pairs on DeepSeek Refusal (−) (131/300 consistent), ≥50% in 10/12 settings, and ≥50% in 11/12 when both responses reach the same outcome (matched behavior). An absolute per-response check (repetition, gibberish, truncation, contradiction) finds FPCG less degenerate than unsteered in 10/12 settings and never more than 1.6 pp above. On Qwen Refusal (−), the one setting below 50% even on behavior-matched pairs (31% overall, 34% matched), there is no increase in detectable degeneration (1.0% vs 1.3%); a residual pairwise preference for the unsteered model's longer, markdown-formatted refusals remains, shown in App. G.1. Against activation steering, the practical alternative, FPCG wins in 12/12 settings (69–100%), and steering loses to unsteered outputs in 11/12. The same padding also caused the one perplexity increase reported for FPCG, now corrected (Sec. 4); behavior labels, filtering and AlpacaEval are unaffected. Order-consistent pairs per cell: 35–180 of 300, listed in the corrected table (revised App. G.1); Sec. 4 now states both exceptions.

Matched behavior rates (revised App. E.3). We take the weakest steering multiplier that steers at least as far as FPCG. At matched behavior, activation steering filters more in 10/17 settings (DeepSeek mean 13.9% vs 3.2%) and less in only one (0.1% vs 0.7%). In 7/24 settings it reaches FPCG's level at no tested multiplier.

W5/Q2: Performance on the steered task

For SEP we measured it directly (revised App. G.3; GPT-4o: does the answer carry out the system instruction's task on the data?). Among outputs that avoid the injected string, FPCG completes the legitimate task as often as the unsteered model: 73.1% vs 73.6% (DeepSeek), 92.7% vs 95.2% (Qwen3-14B). The share of all outputs that are injection-free and complete the task doubles on DeepSeek (27.0% → 53.3%). Activation steering avoids the injection largely by abandoning the task: on Qwen3-14B its injection-free outputs complete it in only 7.5% (multiplier 1.0) and 0% (1.5) of cases. For the other behaviors the steered decision is the task output (an option, a verdict, compliance); output quality is covered by filtering, coherence, perplexity and AlpacaEval.

W6: Stability and calibration

(Revised App. C.5) We retrained the paper's Linear Probes (24 model–behavior pairs). Seeds (5): MAE sd ≤ .004. Prompt splits (5 random 100/100 re-splits of 200 disjoint prompts): MAE sd median .010 (max .033), accuracy sd median 1.3 pp. Calibration: each label is estimated from finitely many continuations, but its bin average is an unbiased estimate of the true probability, so reliability diagrams measure calibration against the underlying probabilities. Reliability is monotone in 22/24 pairs (the exceptions are Wealth Seeking on Qwen3-14B and gpt-oss, the rarest behavior, with sparse high-probability bins), with median ECE .07. FPCG selects the arg-max/arg-min candidate and uses no thresholds, so any monotone miscalibration leaves its choices unchanged.

W7/Q5: CoT accuracy

Restricted to prefixes before the end-of-thinking marker (79% of prefixes on average), binarized accuracy is 63.6–90.9% across all 24 model–behavior pairs. The abstract's 64–91% therefore holds on intermediate CoT alone. MAE changes by ≤.05; CoT-only numbers are in revised App. C.6. gpt-oss. Table 1 and Figure 28(a) report two independent runs of the same configuration (layer 20, 10 candidates). Their difference (43.3% vs 49.0%) is within sampling noise: the 95% CI of the Figure 28(a) run is [40.7, 57.7]. Revised Table 1 uses the Figure 28(a) run (CIs: App. E.2).

Summary

Every weakness and question now has a direct experiment. All of these are in the uploaded revision. If they resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any remaining questions.

AuthorsOpenReview ↗

Follow-up to the response to Reviewer 3586

Dear Reviewer 3586,

in the response above, we believe that we have addressed all your raised concerns and questions. We have also uploaded the revised manuscript containing the requested analyses and clarifications. Could you please indicate if your concerns are fully resolved now, or if any issues remain outstanding? We would greatly appreciate the opportunity to discuss them with you until the discussion period closes. Thank you!

Dear Reviewer 3586,

As the discussion period comes to an end, we would like to kindly ask you to indicate if your concerns were sufficiently addressed by our rebuttal. Thank you!

Reviewer_g5iqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the probability of a behavioral outcome evolves during reasoning. It estimates these probabilities by repeatedly sampling completions from intermediate sentence prefixes, then trains linear and MLP probes to predict them from residual-stream activations. Probes trained on intermediate reasoning outperform detectors trained only on final answers when forecasting subsequent behavior. Building on this observation, the authors introduce Future Probe Controlled Generation (FPCG): at each sentence boundary, the model generates multiple candidates and selects the one with the highest or lowest predicted probability of the target behavior. Experiments cover four reasoning models and six behavioral evaluations, with additional analyses of output quality, predictor choice, layer selection, and computational cost.

Critical strengths

Prediction and control are directly aligned. The probe estimates final behavior probabilities from partial responses, providing an outcome-oriented signal for sentence selection. Probabilistic supervision captures behavioral uncertainty, while the lightweight predictor avoids completing every candidate continuation at inference time.

Critical weaknesses
  1. Comparisons under matched computational budgets are missing. FPCG samples multiple candidates per sentence, but the paper lacks comparisons with full-response Best-of-N and related methods under the same budget. Consequently, it remains unclear whether the additional computation is used more effectively, and how much of the observed advantage comes from the predictor design versus additional sampling.

  2. The claim that prediction and detection features are distinct requires stronger validation. The prediction probe’s training data include intermediate prefixes, whereas the detection probe is trained only on final answers. The performance gap may therefore partly reflect differences in how well the training distributions match the evaluation task.

Questions for authors

Under matched total FLOPs or wall-clock time, does FPCG still achieve better behavioral control and output quality than full-response Best-of-N and related methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer g5iq

Thank you for the thoughtful review. We are glad you find that "prediction and control are directly aligned" and that "probabilistic supervision captures behavioral uncertainty": formalizing probabilistic future-behavior prediction is our central contribution, and FPCG puts it to use. As you note, the lightweight predictor "avoids completing every candidate continuation", which, as shown below, is also where FPCG differs most from reranking.

W1/Q:

"Comparisons under matched computational budgets" and "predictor design versus additional sampling."

  • Same sampling, different predictor. We replace FPCG's prediction probe with a detection probe, keeping the algorithm, candidate count and compute unchanged. Prediction steers at least as far in 24/24 settings (2 models × 6 behaviors × 2 directions), significantly further in 14/24 (prompt-clustered 95% CIs), by up to 17 pp. Additional sampling alone does not explain the gains.
  • Full-response Best-of-N at matched FLOPs. At FPCG's ~17× FLOPs overhead (App. I) we compare against Best-of-17 (revised App. F.5; 4 models, 48 settings; reranked by our prediction or detection probe; its scoring pass is not charged, which favors it). When suppressing free-form behaviors (Refusal, Prompt Injection, Sycophancy), FPCG wins in 11/12 model–behavior pairs (e.g. DeepSeek Sycophancy 13.8% vs 28.8%); across all 48 settings the two are on par (23.2 vs 22.1 pp mean shift).
  • Wall-clock time. FPCG runs at about 2× ordinary decoding (App. I), with one sequence in memory, and streams its output sentence by sentence.
  • Related methods (revised App. F.6). Token-level FUDGE (Yang & Klein, 2021) with the same probe, scoring the top-17 next tokens, matches FPCG on multiple-choice behaviors but gives weaker control in 5/6 free-form settings (e.g. injection suppression 53.3% vs FPCG's 28.2%). It is stronger only on Refusal suppression (10.3% vs 29.0%), where 30% of its generations run to the 8192-token limit (FPCG: 2%).

Importantly, reranking must also complete every candidate. This is problematic in some use-cases, as we show in App. F.5: on Prompt Injection (DeepSeek), Best-of-17 executed the injection in 95% of the prompts it eventually returned clean, vs. 28% for FPCG, whose rejected candidates are single discarded sentences, rather than an injection executed until the end.

"Better behavioral control and output quality" (revised App. G.3). Among injection-free answers, FPCG completes the legitimate task as often as the unsteered model (73.1% vs 73.6% on DeepSeek; 92.7% vs 95.2% on Qwen3-14B), so the share of all DeepSeek outputs that both avoid the injection and complete the task nearly doubles (27.0% → 53.3%). Stronger suppression yields more successful answers, not abandoned tasks.

W2:

"Prediction and detection features are distinct" and "training distributions match the evaluation task."

All three controls are in the revised App. C.3.

  1. Matched training distribution: we apply the ground-truth behavior detector to every intermediate prefix and train the detection probe on these labels, so it sees exactly the prediction probe's positions; only the target differs. Held-out MAE against future behavior (lower is better):
BehaviorDeepSeek: PredictionDeepSeek: DetectionQwen3-14B: PredictionQwen3-14B: Detection
Refusal.11.23.14.25
Prompt Injection.32.40.18.47
Sycophancy.19.34.13.30

This control covers the three free-form behaviors, whose detectors apply to prefixes.

  1. Matched sample counts: prediction data subsampled to the detection probe's size, uniformly across reasoning stages (5 seeds), keeps prediction's advantage on all six behaviors in both models.
  2. Representations: on all six DeepSeek behaviors, LEACE erasure (Belrose et al., 2023) of the final-answer detection concept makes a retrained detection probe fail (MAE ≥ .59), while the retrained prediction probe changes by ≤ .04 (e.g. Refusal .112 → .117). Conversely, erasing the prediction concept leaves the final-answer detection probe nearly unchanged (≤ .03).

Matching training positions and sample counts does not close the gap, and the two signals are linearly separable.

Summary

Both concerns now have direct experiments: prediction contributes beyond additional sampling, and its advantage persists after matching training positions and sample counts. All results are in the uploaded revision. If they resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any remaining questions.

AuthorsOpenReview ↗

Follow-up to the response to Reviewer g5iq

Dear Reviewer g5iq,

we hope that we have addressed all the raised concerns in our response above. If any issues remain outstanding, we kindly ask you to discuss them with us before the discussion period closes. If the raised concerns are resolved to your satisfaction, we would highly appreciate if you updated your original recommendation. Thank you!

Reviewer_g5iqOpenReview ↗

Thank you for the response

Thank you for the response, which has addressed some of my concerns. However, I still have reservations about the role of additional sampling and will maintain my original score.

Reviewer_w9xVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper estimates future behavior probabilities by repeatedly continuing partial reasoning traces and trains activation probes to predict those probabilities. It then introduces FPCG, which samples and selects sentences sequentially, and evaluates behavioral control and output quality across four reasoning models and six behavioral evaluations.

Critical strengths

Connecting probabilistic future behavior prediction with generation control is practically useful, and the experiments show that FPCG generally preserves output quality better than difference-in-means activation steering while effectively changing behavior. The appendices include comparisons involving text predictors, detection features, candidate counts, and intervention layers, alongside coherence, AlpacaEval, and computational cost evaluations.

Critical weaknesses

1.The central claim that detection and prediction features are fundamentally distinct remains insufficiently established: a probe trained only on final answers is evaluated during reasoning, whereas the other probe receives training on those stages, so the performance gap could reflect differences in training distribution and sample count. The results establish decodable predictive signal, but do not adequately establish distinct internal representations or explain how behavioral decisions emerge. 2.The method’s incremental benefit over strong baselines remains unclear. Detection-probe FPCG usually trails by only a few percentage points on DeepSeek, while text-predictor FPCG performs comparably or better; default FPCG requires approximately 17 times the FLOPs, yet lacks comparisons with compute-matched full-response reranking or other generation-selection methods, leaving the contributions of activation-based prediction and additional sampling unresolved. 3.The claim of almost no output quality degradation is stronger than the evidence. Under negative Refusal steering, coherence win rates against unsteered outputs are only 19.9% and 15.5% for the two models, and some configurations still filter out 15%–18% of outputs for formatting failures; these exceptions and the absence of confidence intervals for the main behavioral results make the robustness of quality preservation and smaller gains difficult to assess.

Questions for authors

1.Can the probes be compared with matched training sample counts and reasoning-stage coverage, supplemented by representation or intervention experiments that distinguish distribution shift from genuine feature differences? 2.At matched FLOPs or latency, what behavioral-control and quality advantages does FPCG offer over full-response reranking, detection-probe selection, and text-predictor selection? 3.Can you report prompt-clustered confidence intervals and valid target-behavior rates over all generations, and clarify whether the Refusal coherence decline reflects judge preferences or actual output degradation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer w9xV

Thank you for the review. We are glad you find connecting probabilistic future-behavior prediction with generation control "practically useful" (formalizing this task is our core contribution) and that FPCG "generally preserves output quality better than difference-in-means activation steering".

W1/Q1: Distribution shift vs. genuine feature differences.

We ran the requested controls (revised App. C.3; DeepSeek, layer 25; held-out MAE vs. future behavior):

  • Matched training sample counts and reasoning-stage coverage: we subsample the prediction probe's data to the detection probe's size, uniformly over reasoning stages (5 seeds).
  • Matched training distribution: we apply the ground-truth behavior detector to every intermediate prefix and train the detection probe on them, so its training distribution matches the evaluation task exactly; only the target differs.
BehaviorPredictionPrediction, matched NDetection, CoT prefixesDetection, final answers
Myopic Reward.27.28–.40
Wealth Seeking.26.28–.40
Survival Instinct.29.30–.36
Refusal.11.16.23.32
Prompt Injection.32.32.40.38
Sycophancy.19.21.34.45

(–: multiple-choice detectors need an answer.)

Both controls hold on Qwen3-14B (prefix detection vs prediction: Refusal .25 vs .14, Injection .47 vs .18, Sycophancy .30 vs .13). The gap reflects neither distribution shift nor sample count: features detecting a behavior in text predict it far worse than those learned from future-behavior labels.

Representation evidence. After LEACE erasure (Belrose et al., 2023) of linearly decodable detection information, the detection probe collapses, while the retrained prediction probe moves ≤ .04 on every behavior. Conversely, erasing prediction leaves detection intact (≤.03). The two signals are linearly separable.

W2/Q2: Prediction vs. sampling.

  • Detection-probe selection isolates prediction: same algorithm, candidates and FLOPs; only features differ. Prediction steers at least as far in 24/24 settings (2 models), significantly further in 14/24 (CIs); mean gain +4.0 / +6.3 pp (DeepSeek / Qwen), up to 17 pp.
  • Text-predictor selection is not prior work but FPCG with another predictor: a small LM trained on our future-behavior labels (revised App. F.4). That it steers comparably further supports our central contribution, the prediction task itself, which can be solved white-box or black-box: a flexibility opening new research directions. We use activation probes as they also reveal what the model represents (W1) and are cheaper: the text predictor needs a second LM in memory (a separate GPU here), its own training, and an extra forward pass per candidate; probes reuse the generator's activations.
  • Full-response reranking at the ~17× FLOPs you cite (revised App. F.5: best-of-17, probe-reranked, decoding cost only): when suppressing free-form behaviors (Refusal, Prompt Injection, Sycophancy), FPCG wins 11/12 model–behavior pairs; over all 48 settings they are on par (23.2 vs 22.1 pp). Latency: about 2× ordinary decoding (App. I), one sequence in memory.

Reranking selects among completed responses, which in agentic use contain irreversible tool calls. On Prompt Injection (DeepSeek), best-of-17 executed the injection in 95% of prompts it returned clean, vs. 28% for FPCG, whose rejected candidates are single sentences. FPCG commits sentence by sentence, so it streams and can halt early.

W3/Q3: Confidence intervals and valid rates

(Revised App. E.2) With prompt-clustered CIs, the steering effect excludes zero in 47/48 settings. Counting unparseable outputs as failures, the valid target-behavior rate over all generations rises in 48/48. The 15% filtering on DeepSeek Sycophancy is the unsteered model's own (23%), which FPCG reduces; even the most-filtered setting (gpt-oss Wealth +, 19%) nearly triples its valid target rate.

Refusal coherence. Finished FPCG candidates were padded with invisible EOS tokens, which the judge read as "runaway repetition". Re-judging detokenized text (GPT-4o, both orders), FPCG wins ≥50% in 10/12 settings and 63% ± 8% of order-consistent pairs (131/300) on DeepSeek Refusal (−). Per-response degeneration is 4.7% (FPCG) vs 27.7% (unsteered) on DeepSeek and 1.0% vs 1.3% on Qwen3-14B: no increase in detectable degeneration, though a residual preference for unsteered outputs remains on Qwen Refusal (−) (31% FPCG wins). Against activation steering, the practical alternative, FPCG wins 12/12 (69–100%), and steering loses to unsteered in 11/12. The padding also caused FPCG's one reported perplexity increase, now corrected (Sec. 4); behavior labels, filtering and AlpacaEval are unaffected. Corrected table: revised App. G.1; Sec. 4 now states both exceptions.

If these results resolve your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up to the response to Reviewer w9xV

Dear Reviewer w9xV,

we believe that we have addressed all your raised concerns in our response above. We kindly ask you to tell us whether any concerns still remain unresolved, and if those influence your recommendation. We would highly appreciate the opportunity to discuss them before the discussion period closes. Thank you!

Dear Reviewer w9xV,

As the discussion period comes to an end, we would like to kindly ask you to indicate if your concerns were sufficiently addressed by our rebuttal. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.