- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper predicts the probability of a model's eventual behavior from intermediate activations during reasoning. Repeated continuations provide probability labels for training the probes. The proposed FPCG method samples candidate sentences and chooses continuations using the predicted behavior probabilities. Experiments cover four models and six behaviors, compare steering strength and output quality with activation interventions, and examine differences between prediction and detection features.
Critical strengths
-
Estimating conditional behavior probabilities from repeated continuations is more appropriate than treating the behavior of one completed response as a deterministic label for an intermediate state. FPCG changes behavior substantially across several models and tasks, showing that the prediction signal can be useful for steering.
-
The evaluation includes formatting failures, perplexity, coherence, and general instruction following, alongside steering strength. Comparisons with detection features and text predictors, layer analyses, and compute measurements help characterize where the approach works and what it costs.
Critical weaknesses
-
Appendix F.4 compares text-based FPCG, but budget-matched full-response reranking or an established guided decoder such as FUDGE is still missing. On sycophancy, ten candidates cost 16.6–16.9× FLOPs and 1.77–1.90× wall time per generated token on the two tested models (Table 5). This leaves the contribution of the extra search budget unresolved.
-
Detection probes are trained only on final answers, while prediction probes also see CoT; their difference is confounded with training coverage. Likewise, agreement across ten base responses (Section 3.2) does not establish a decision made before reasoning or show that later CoT has no causal role.
-
FPCG intervenes during both CoT and final-answer generation. Appendix F.2 compares probe training, but does not isolate the intervention phase. A CoT-only versus answer-only comparison is needed to establish how much of the gain requires prediction before the answer is written.
-
Table 1 reports 15% filtering for DeepSeek's positive sycophancy steering and 18% for gpt-oss's positive wealth steering. Under negative SORRY-Bench steering (toward refusal), coherence win rates against unsteered DeepSeek/Qwen are 19.9%/15.5% among order-consistent judgments (Table 2). These exceptions qualify the quality-preservation claim; quality is also not compared at matched behavior rates.
-
Appendix G.1's judge explicitly ignores correctness and helpfulness, while AlpacaEval uses separate general instructions. Neither establishes preserved performance on the steered tasks. In SEP, avoiding the injected string can still coincide with failing the legitimate task; its completion rate is not reported.
-
Each behavior has 100 training and 100 test prompts; extra prefixes do not add independent problems. Stability across prompt splits and training seeds is not shown. MAE is reported, but neither MAE nor binarized accuracy alone establishes calibration against the underlying behavior probabilities estimated from finite continuations.
-
The abstract's 64–91% accuracy claim refers to intermediate CoT, but Figure 4's evaluation includes final-answer positions. Also, gpt-oss's positive SORRY-Bench result—compliance, not refusal—is 43.3% in Table 1 versus 49.0% in Figure 28(a). Any difference in runs or settings should be stated.
Questions for authors
-
Given that the detection probe is trained only on final answers and the prediction probe includes CoT positions, which results rule out training-position coverage as the explanation for their difference?
-
Among SEP outputs that avoid the injected string, how often does the model complete the legitimate task specified by the system instruction? Was this measured separately?
-
How are per-prompt behavior rates aggregated after filtering? For the settings with 15% or 18% filtered outputs, what fraction of all attempts both passes filtering and exhibits the target behavior?
-
What causes the low coherence win rates under negative SORRY-Bench steering toward refusal—19.9% for DeepSeek and 15.5% for Qwen? How many comparisons survive the order-consistency filter?
-
Why is gpt-oss's positive SORRY-Bench compliance rate 43.3% in Table 1 but 49.0% in Figure 28(a)? Do these come from different runs or configurations, and which corresponds to the main experiment?
4 comments
Response to Reviewer 3586 (2/2)
W4/Q3/Q4: Filtering, coherence, matched behavior
Aggregation (Q3, revised App. E.2). Per prompt we average behavior over parseable responses, drop prompts with none (0/100 prompts for DeepSeek Sycophancy +, 5/100 for gpt-oss Wealth +), then average over prompts. Counting every unparseable output as a failure, the fraction of all attempts that pass filtering and show the target behavior rises from 53.7% to 79.6% (DeepSeek Sycophancy +) and from 7.7% to 20.7% (gpt-oss Wealth +), and rises in 48/48 settings of Table 1. The 15% DeepSeek filtering is the model's own: unsteered DeepSeek filters 23% there.
Coherence (Q4). The low Refusal (−) win rates come from invisible padding tokens in the raw decoder output we judged, not from the text a user sees: FPCG pads finished candidates with end-of-sequence tokens (87% of DeepSeek Refusal (−) FPCG answers vs 0% unsteered), which the judge reads as "runaway repetition". Judging the detokenized text instead (all 12 settings; GPT-4o, both orders, 300 pairs each), FPCG wins 63% ± 8% of order-consistent pairs on DeepSeek Refusal (−) (131/300 consistent), ≥50% in 10/12 settings, and ≥50% in 11/12 when both responses reach the same outcome (matched behavior). An absolute per-response check (repetition, gibberish, truncation, contradiction) finds FPCG less degenerate than unsteered in 10/12 settings and never more than 1.6 pp above. On Qwen Refusal (−), the one setting below 50% even on behavior-matched pairs (31% overall, 34% matched), there is no increase in detectable degeneration (1.0% vs 1.3%); a residual pairwise preference for the unsteered model's longer, markdown-formatted refusals remains, shown in App. G.1. Against activation steering, the practical alternative, FPCG wins in 12/12 settings (69–100%), and steering loses to unsteered outputs in 11/12. The same padding also caused the one perplexity increase reported for FPCG, now corrected (Sec. 4); behavior labels, filtering and AlpacaEval are unaffected. Order-consistent pairs per cell: 35–180 of 300, listed in the corrected table (revised App. G.1); Sec. 4 now states both exceptions.
Matched behavior rates (revised App. E.3). We take the weakest steering multiplier that steers at least as far as FPCG. At matched behavior, activation steering filters more in 10/17 settings (DeepSeek mean 13.9% vs 3.2%) and less in only one (0.1% vs 0.7%). In 7/24 settings it reaches FPCG's level at no tested multiplier.
W5/Q2: Performance on the steered task
For SEP we measured it directly (revised App. G.3; GPT-4o: does the answer carry out the system instruction's task on the data?). Among outputs that avoid the injected string, FPCG completes the legitimate task as often as the unsteered model: 73.1% vs 73.6% (DeepSeek), 92.7% vs 95.2% (Qwen3-14B). The share of all outputs that are injection-free and complete the task doubles on DeepSeek (27.0% → 53.3%). Activation steering avoids the injection largely by abandoning the task: on Qwen3-14B its injection-free outputs complete it in only 7.5% (multiplier 1.0) and 0% (1.5) of cases. For the other behaviors the steered decision is the task output (an option, a verdict, compliance); output quality is covered by filtering, coherence, perplexity and AlpacaEval.
W6: Stability and calibration
(Revised App. C.5) We retrained the paper's Linear Probes (24 model–behavior pairs). Seeds (5): MAE sd ≤ .004. Prompt splits (5 random 100/100 re-splits of 200 disjoint prompts): MAE sd median .010 (max .033), accuracy sd median 1.3 pp. Calibration: each label is estimated from finitely many continuations, but its bin average is an unbiased estimate of the true probability, so reliability diagrams measure calibration against the underlying probabilities. Reliability is monotone in 22/24 pairs (the exceptions are Wealth Seeking on Qwen3-14B and gpt-oss, the rarest behavior, with sparse high-probability bins), with median ECE .07. FPCG selects the arg-max/arg-min candidate and uses no thresholds, so any monotone miscalibration leaves its choices unchanged.
W7/Q5: CoT accuracy
Restricted to prefixes before the end-of-thinking marker (79% of prefixes on average), binarized accuracy is 63.6–90.9% across all 24 model–behavior pairs. The abstract's 64–91% therefore holds on intermediate CoT alone. MAE changes by ≤.05; CoT-only numbers are in revised App. C.6. gpt-oss. Table 1 and Figure 28(a) report two independent runs of the same configuration (layer 20, 10 candidates). Their difference (43.3% vs 49.0%) is within sampling noise: the 95% CI of the Figure 28(a) run is [40.7, 57.7]. Revised Table 1 uses the Figure 28(a) run (CIs: App. E.2).
Summary
Every weakness and question now has a direct experiment. All of these are in the uploaded revision. If they resolve your concerns, we would be grateful if you would consider raising your score. We are happy to answer any remaining questions.
Follow-up to the response to Reviewer 3586
Dear Reviewer 3586,
in the response above, we believe that we have addressed all your raised concerns and questions. We have also uploaded the revised manuscript containing the requested analyses and clarifications. Could you please indicate if your concerns are fully resolved now, or if any issues remain outstanding? We would greatly appreciate the opportunity to discuss them with you until the discussion period closes. Thank you!
Dear Reviewer 3586,
As the discussion period comes to an end, we would like to kindly ask you to indicate if your concerns were sufficiently addressed by our rebuttal. Thank you!
Response to Reviewer 3586 (1/2)
Thank you for the careful review. We are glad you find estimating conditional behavior probabilities "more appropriate" than single-response labels (this probabilistic future-behavior prediction is our central contribution) and that FPCG "changes behavior substantially across several models and tasks". Each point is answered below with new results.
W1: Budget-matched reranking and FUDGE
Do FPCG's gains come from prediction or from the extra search budget? We separate the two.
W2/Q1: Training-position coverage
Three controls (revised App. C.3; DeepSeek, layer 25; held-out MAE vs. future behavior):
With identical inputs, positions and sample counts the gap persists, and the two targets are carried by distinct features: training coverage does not explain it.
Sec. 3.2. We agree that agreement across ten responses is not a causal statement, and have rephrased Sec. 3.2 to state only that all sampled responses agree (revised App. C.4). Resampling independent of the 10 responses defining certainty: for certain prompts, fresh continuations (M = 10–30) from each intermediate prefix (0.9M prefixes, 4 models) leave the outcome exactly at 0 or 1 in 84% of prefixes and flip the majority behavior in only 3%. We do not claim the CoT has no causal role; on uncertain prompts we show the opposite (W3).
W3: CoT-only vs. answer-only intervention
We ran exactly this comparison (revised App. F.7) (6 behaviors × 2 directions). Probe-based selection is applied only before the end-of-thinking marker (CoT-only) or only after it (answer-only); in the other phase an unscored sampled sentence is committed. On DeepSeek, CoT-only FPCG recovers 86% of full FPCG's steering effect (mean shift 28.6 vs 33.3 pp on the same prompts), while answer-only recovers only 19% (6.4 vs 33.3 pp) and falls short of full FPCG in all 12 settings. This is not just a matter of how many interventions each phase gets: answer sentences are ~27% of FPCG's selection steps but yield 19% of the effect, while CoT steps (73%) yield 86%. Qwen3-14B replicates this (CoT-only 89%, answer-only 9% of the effect; 12 settings). Almost all of the gain requires predicting the behavior before the answer is written, which is what prediction features provide.