acceptodds
Under review as a conference paper at ICLR 2027

When Is a Steering Direction Statistically Supported?

Abstract

Activation steering uses directions constructed from examples to steer model behavior and interpret representations. Recent work shows that variability across directions from small construction samples can obscure differences between prompt formats. When does variability in a steering vector translate into variability in model responses? In the evaluated A/B constructions, we find much of the sampling variability lies along an answer-position axis. For myopic reward, proportional stratification uses this structure to improve response repeatability over ordinary sampling at fixed target and sample size. In other experiments, an answer's probability can initially increase along an estimated direction but decrease along its target direction. To assess what construction data support about the target direction, we introduce SteerCert. It reports a verdict for claims holding throughout a confidence set for the population contrast, the expected activation difference. Valid confidence-set coverage controls false population claims. Our theory bounds the independent data needed to estimate the target direction precisely: necessary and sufficient sample-size scales match at fixed confidence under stated assumptions.

Primary area:
interpretability and explainable AI
Keywords:
Steering vector, Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WG7vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work looks into sampling uncertainty in steering vectors, which are usually constructed by averaging hidden-state differences between examples of two opposite behaviors. The paper proposes a method called SteerCert that constructs a confidence region around the average vector, and supports a claim only when the claim holds across the region. Empirically the paper finds that A/B answer-position explains much of this variation. The paper also shows that stratified sampling reduces such variance.

Critical strengths

S1. The variability in steering vector construction and the sampling analysis is interesting.

Critical weaknesses

W1. The spherical region may be overly conservative. Sampling error in the estimated steering vector can differ substantially across directions, but a sphere assumes the same error magnitude in every direction. Requiring a claim to hold throughout the sphere may therefore cause false positive abstentions. In the paper there are only 50 of 1200 evaluations receive positive guarantee, which might result from overly conservative confidence region.

W2. The experiment assesses whether an infinitesimally small intervention increases the probability of an A/B answer. However, they do not test what happens at a more practical setting where the intervention strengths that are actually used for steering or when the model generates its own answers.

Questions for authors

Q1. The certificate rate is quite low (50 out of 1200). What are the false positive and false negative rates if you consider the actual behavioral effect as the ground truth label?

Q2. Would it be possible (and helpful) to extend this work to covariance-aware confidence regions, given the complexity of the distribution in the latent space? I wonder if this might help improve the certificate rate.

2 comments

Thank you for the constructive suggestions on covariance-aware inference and finite-strength evaluation. We have added analyses addressing both points.

W1 and Q2: Conservatism and covariance-aware inference

A covariance-aware projection band recovers many correct local-response claims withheld by the ball in finite-source validation.

Following your suggestion, we compared the original confidence ball with a covariance-aware simultaneous projection band for the same predeclared local-response claims, using a multiplier bootstrap. Support increases from 50 to 788 of the original 1,200 evaluations, using the same construction data and nominal error allocation. Each simultaneous family contains 100 inputs for one behavior/layer combination.

The ball’s multiplier-bootstrap calibration allows for anisotropic sampling errors. Its spherical geometry can still be conservative for fixed local-response claims. This conservatism may be useful in steering analyses where limiting false support takes priority over recall.

These results suggest choosing the confidence construction to match the claim of interest. Both constructions apply the criterion in Sec. 3: a claim is statistically supported when it holds throughout the chosen confidence set. The projection band targets the 100 declared local-response claims using their joint uncertainty. The ball provides a common uncertainty set for assessing directional precision and local-response claims together, which can be useful when a steering direction is reused across inputs or analyses. The ball-based directional verdicts are unchanged.

We evaluated coverage and recall on expanded finite sources for sycophancy and hallucination. Across 2,000 reconstructions at n=1000n=1000:

Construction samplingBall recallProjection-band recallProjection-band simultaneous coverage
Ordinary9.05%82.03%95.07% [94.50, 95.63]
Answer-position stratified26.55%91.63%94.89% [94.33, 95.43]

Recall measures support among positive local signs at the known finite-source means. Coverage is averaged over six 100-input families; brackets give 95% bootstrap intervals conditional on the fixed sources and response functionals. Thus, the band recovered many correct claims withheld by the ball while achieving empirical family coverage near 95%.

For these two behaviors in the original panel, all 403 band-supported local signs also remained positive at the means of 24,570 unused source records.

We also checked whether the recall gain persisted after adjusting both methods toward the same coverage target. Using known finite-source truth, we adjusted their widths to target 95% simultaneous coverage of the response functionals. On held-out reconstructions at n=1000n=1000, the band retained recall gains of 18.9 percentage points under ordinary sampling and 9.2 under stratified sampling over the adjusted ball-derived bounds. Achieved coverages were close but not identical; this truth-based adjustment is a diagnostic, not a procedure for an unknown population. These results suggest that the support gain is not solely due to lower coverage.

At n=128n=128, band coverage was 93.8–93.9%, so small-sample calibration remains a limitation. Bootstrap settings differ in the number of Gaussian multiplier draws (1,999 in validation versus 19,999 in the public-panel comparison) and covariance denominators (sample size minus one versus sample size). These finite-source checks do not establish a 95% guarantee for the original 788 supports or an unknown generating population.

W2 and Q1: Finite-strength responses and false-positive/negative rates

In the current manuscript, Sec. 5.2 shows that answer-position stratification improves response repeatability at finite strength for myopic reward. We now complement this result with additional experiments connecting local support to finite responses across the original 1,200 evaluations at three nonzero strengths. Most band-supported cases show positive answer-probability changes at each tested strength, with agreement declining at larger strengths.

W2: “they do not test what happens at a more practical setting where the intervention strengths that are actually used for steering”

Sec. 5.2 of the current manuscript evaluates finite-strength interventions across reconstructed steering directions. For myopic reward, answer-position stratification yields 0.067 times the response variance of ordinary sampling in Llama and 0.425 times in Gemma (n=128n=128, κ=−1\kappa=-1), at the same target and sample size.

W2: “when the model generates its own answers”

Our behavioral evaluation uses answer-letter probabilities rather than scores of freely generated text. This lets us measure construction variability and compare local signs with finite-strength effects on the same response measure.

Q1: “What are the false positive and false negative rates if you consider the actual behavioral effect as the ground truth label?”

For the additional analysis, we kept support selections fixed on all 1,200 evaluations and intervened along the normalized file-mean directions at strengths κc\kappa c, where c=7.68c=7.68 and $\kappa\in \lbrace 0.25,0.5,1 \rbrace $. The reference norm and absolute strength coefficients are the same as in Sec. 5.2. We use positive coefficients here because the evaluated claims concern positive local responses. The endpoint is the change in the behavior-matching answer letter’s full-vocabulary probability.

Of the 788 band-supported evaluations, 784, 758, and 691 had positive finite responses. Across all 788, median changes were +3.1, +5.9, and +5.7 percentage points, respectively. All 50 ball-supported evaluations had positive responses at all three strengths.

For the requested confusion matrix, withholding is treated as a negative classification, without asserting a negative effect. FPR is the fraction of nonpositive finite responses that were supported; FNR is the fraction of positive finite responses that were withheld.

κ\kappaBall FPRBall FNRProjection-band FPRProjection-band FNR
0.250.00%94.76%1.63%17.82%
0.500.00%94.61%11.03%18.32%
1.000.00%94.25%29.39%20.57%

Among the 788 band-supported evaluations, 0.5%, 3.8%, and 12.3% had nonpositive finite responses at the three strengths, respectively. At the smallest tested strength, the band reduces FNR from 94.76% to 17.82%, with 1.63% FPR. These are descriptive rates on the fixed panel. Matched-count center selection gave 784, 761, and 696 positive responses out of 788; we found no consistent intervention-selection advantage on this panel.

At the largest strength, 97 supported evaluations had nonpositive finite responses despite positive local derivatives along the same file-mean directions. This measures the limit of extrapolating local signs to larger interventions, rather than false support for a population local sign.

Together, these results clarify both the gains in local-response support and the limits of extrapolating that support to finite-strength effects. We will revise Secs. 5.3–5.4 and the corresponding contribution, conclusion, and limitations; the appendix will provide calibration settings, effect distributions, and matched-count comparisons.

Reviewer_wQA9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how reliable steering vectors are when they are built by averaging activation differences over a sample of contrasting examples, as in Contrastive Activation Addition (CAA). They focus on Llama-2-7B-chat, Gemma-2-2B-IT, and Qwen2.5 models, using CAA behaviors and model-written evaluation datasets. Their experiments show that much of the variability between steering vectors comes from the mix of examples whose matching answer is “A” versus “B”, and that sampling these in fixed proportions makes vectors and model responses more consistent. They also propose SteerCert, a method that uses a confidence set to decide what can be claimed about a vector’s direction, and give theory on how much data is needed to estimate the direction precisely. Moreover, they find that a positive effect of steering along an estimated direction can be negative along the true direction, and that such positive effects are rarely statistically supported on public CAA datasets.

Critical strengths
  • The problem is timely and well-motivated. Activation steering is widely used both to control model behavior and to interpret representations. Asking which claims about a vector’s direction available data actually supports is an important question for interpretability research. The finding that a positive local steering effect along an estimated direction can be negative along the true direction (3.1% of positive estimated derivatives under ordinary sampling) shows that properties observed on one estimated vector should not be assumed to hold for its target.
  • The paper offers actionable insights researchers can adopt in experiments. Simply balancing “A” and “B” answers when sampling examples makes steering vectors much more stable (55–90% lower directional error).
  • The paper also provides ample theoretical grounding which further supports the method’s intuition and advances understanding of the approach. It shows how much data is needed to estimate a steering direction precisely, with matching lower and upper bounds, and how this grows with dimension and desired precision. It also shows that adding more examples from the same prompt templates cannot replace sampling more independent templates, which offers useful guidance for building steering datasets.
Critical weaknesses
  • Over-caution in SteerCert. It seems SteerCert might be too cautious; the paper finds that positive steering claims are rarely supported (e.g., only 50 of 1,200 cases on the public CAA datasets). This could mean the claims genuinely lack evidence, or that SteerCert is overly cautious. In 5.3, the experiments suggest that the latter might be true: most of the claims SteerCert rejected were actually correct. The authors acknowledges that their confidence sets may be wider than needed but some additional clarification/experiments would help strengthen the paper and validate the effectiveness of SteerCert.
Questions for authors

What causes the large share of variability along the answer-position axis? Since each contrast is read at the answer-letter token, part of it may reflect the letter identity (“A” vs. “B”) rather than the behavior itself, but the paper notes that answer position and question content are not separated. Constructing contrasts from each question in both answer orders could help separate these explanations. We understand that this changes the target contrast (as discussed for Gao et al., 2026), but even a diagnostic experiment would clarify what the answer-position axis captures and whether the benefit of stratification is primarily due to controlling for this answer-position effect.

2 comments

Thank you for suggesting the paired answer-order diagnostic and highlighting conservative withholding. We have investigated both.

What does the answer-position axis capture?

“What causes the large share of variability along the answer-position axis?”

Following your suggestion, we conducted an additional diagnostic by evaluating the same questions in both answer orders. This diagnostic shows that the original answer-position axis largely reflects answer-order sensitivity within the same questions.

We used 128 construction questions per behavior, covering four behaviors and three Llama-2-7B-chat layers. Aligning contrasts by behavioral meaning, we decomposed them into their half-sum and half-difference, representing order-averaged and order-sensitive components. The order-sensitive component captures the combined effect of the answer-order change, including possible A/B token effects and their interactions with context.

Cosine alignment with the order-sensitive component was 0.9983–0.9996 in three behaviors and 0.9632–0.9667 in sycophancy.

The paired diagnostic identifies the order sensitivity captured by the axis; the existing random-stratification control tests whether controlling answer-position composition improves repeatability. The control uses a fixed random partition with the same stratum sizes, allocation, and weighting rule. For myopic reward at n=128n=128, κ=−1\kappa=-1, response variance under answer-position stratification was 0.063 times that under random stratification in Llama and 0.405 times in Gemma, with 95% bootstrap intervals [0.037, 0.103] and [0.241, 0.669]. Together, these results support controlling answer-position composition as a source of the observed repeatability gain.

The paired-order experiment diagnoses the axis; using paired-order averages to construct a direction is a separate choice. At equal forward counts, paired averaging has lower modeled mean-contrast variance than random order assignment (0.35–0.57), but higher variance than balanced order assignment to independently sampled questions (1.29–1.63). Paired averaging changes the target to the order-averaged contrast used in these comparisons; proportional stratification preserves the original target and population weights. We will add the diagnostic to Sec. 5.1 and the full comparison to the appendix.

Does SteerCert withhold too many correct claims?

“This could mean the claims genuinely lack evidence, or that SteerCert is overly cautious.”

We compared the ball with a covariance-aware simultaneous projection band tailored to the predeclared local-response family. The band recovered many correct claims withheld by the ball. In the finite-source validation, its empirical simultaneous coverage was close to nominal at the larger sample size.

These results show that the ball is conservative for this response family. This conservatism may be useful in steering analyses where limiting false support takes priority over recall. The ball also provides a common uncertainty set for assessing directional precision and local-response claims together when a steering direction is reused across inputs.

On expanded finite sources for sycophancy and hallucination at n=1000n=1000, recall among positive local signs increased from 9.05% for the ball to 82.03% for the band under ordinary sampling and from 26.55% to 91.63% under stratified sampling. The band’s simultaneous coverage, averaged across six 100-input families, was 95.07% and 94.89%, respectively. Across 2,000 reconstructions, the band made 69 false local-support claims under ordinary sampling and 33 under stratified sampling, out of 1,200,000 total evaluations per design. No false local-support claims were observed for the ball in either design.

A covariance-aware simultaneous projection band increases support from 50 to 788 of the original 1,200 evaluations, with the same construction data and nominal error allocation. Simultaneous inference is within each 100-input behavior/layer family. The projection band applies the same confidence-set criterion for statistical support defined in Sec. 3, tailored to the declared local-response family. The ball-based directional verdicts are unchanged.

We also checked the supported signs against source records not used to construct the original directions. For sycophancy and hallucination in the original panel, all 403 band-supported local signs remained positive at the means of 24,570 unused source records.

Withholding is intended to limit false population claims: it means that the chosen confidence set does not support the claim, not that the direction is ineffective.

These finite-source checks do not establish a 95% guarantee for the original 788 supports or an unknown generating population. Small-sample calibration remains a limitation: coverage at n=128n=128 was 93.8–93.9%. See our response to WG7v (W1/Q2) for the coverage and width comparisons.

The ball complements the response-specific band by providing a common uncertainty set for assessing directional precision and local-response claims together. We will identify 50/1,200 as the spherical construction’s result and incorporate the projection-band comparison in Secs. 5.3–5.4, the contributions, and the conclusion. Together with the fixed-target sampling results and independent-data bounds, this clarifies what construction design can improve, what requires additional independent data, and which population claims the available data support.

Reviewer_bZCtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the statistical reliability of activation-steering directions estimated from finite samples and whether conclusions drawn from these estimates generalize to the underlying population directions. The paper shows the importance of accounting for sampling uncertainty in activation steering and provides practical methods for improving directional precision, behavioral consistency, and the statistical validity of conclusions about model representations. The authors find that a substantial portion of sampling variability is driven by answer-position effects in multiple-choice datasets and demonstrate that proportional stratification reduces angular estimation error significantly without requiring additional examples. They also introduce SteerCert, a statistical framework that uses confidence sets to assess whether a population steering direction is reliably identifiable, distinguishing between insufficient evidence, detectable signals, unoriented lines, and fully oriented directions.

Critical strengths
  • The paper addresses the reliability of activation-steering directions, which is an important research question.
  • The paper presents interesting empirical finfings, identifying answer-position bias as a major source of sampling variability.
  • They show that proportional stratification reduces directional estimation error without requiring more data, however, this concept has long existed in classical machine learning and classifier training.
  • They introduce SteerCert to evaluate which claims about population steering directions are statistically supported.
  • They show that improved directional precision can lead to more consistent model behavior, although this is very clear for whoever who has worked with steering vectors just once.
Critical weaknesses
  • Unfortunately, the paper is very difficult to read and I must say that it reads very much similar to AI-written documents, which does require substantial improvements.
  • The statistical guarantees rely on assumptions about activation distributions that may not hold in real-world settings.
  • The empirical SteerCert results rely on approximate bootstrap confidence sets.
  • SteerCert may reject useful steering directions and it is unclear how much of an issue this will be in practice. Additionally, the paper does not demonstrate whether statistically certified directions lead to more effective or reliable steering.
  • Experiments mainly focus on multiple-choice tasks and local responses rather than open-ended generation and realistic interventions.
Questions for authors

Please see the weaknesses. Please also explain what are the novel contributions of your paper.

2 comments

Thank you for recognizing the importance of construction uncertainty.

Steering directions are used both to control model behavior and to interpret representations. We examine which claims about their population targets are statistically supported by finite construction data.

Regarding novelty, our contributions are threefold.

In the evaluated A/B constructions, much of the sampling variability lies along an answer-position axis. Proportional stratification fixes the answer-position proportions while preserving the target and sample size. Compared with ordinary sampling, this improves directional precision across 152 persona and xrisk files at four model–layer settings (Sec. 5.1). For myopic reward, it also improves response repeatability under finite-strength interventions (Sec. 5.2).

Positive local responses at estimated directions can nevertheless reverse at known target directions. SteerCert assesses which population claims remain statistically supported despite this construction uncertainty (Secs. 3 and 5.3–5.4).

Our theory characterizes the independent data required for directional precision. Necessary and sufficient sample-size scales match at fixed confidence under the stated assumptions. The analysis also establishes limits to improving precision by adding examples within existing templates (Sec. 4).

“this concept has long existed in classical machine learning and classifier training.”

Stratification itself is classical; our contribution is identifying and quantifying the answer-position structure that makes it effective here. At the same target and sample size, proportional answer-position stratification reduces angular MSE by 54.8–89.8% relative to ordinary sampling across our 152-file model–family comparisons (Sec. 5.1). For myopic reward, response variance relative to random stratification is 0.063 in Llama and 0.405 in Gemma (n=128n=128, κ=−1\kappa=−1; Sec. 5.2). This comparison identifies the value of the particular partition, beyond stratification in general. See our response to wQA9 for the paired-order diagnostic.

“improved directional precision can lead to more consistent model behavior, although this is very clear …”

The nontrivial finding is that answer-position structure matters for response deviations even at fixed angular accuracy. This matters for evaluating and designing constructions: angular accuracy alone can miss response differences, while proportional answer-position stratification improves repeatability at the same target and sample size.

In an exploratory Llama myopic-reward diagnostic using the known finite-file reference, answer-position-axis removal yields 0.225 times the response MSE of random-axis removal, at matched angular error and intervention norm (Sec. 5.2).

Moreover, 3.117% of positive local derivatives at ordinarily sampled directions reverse sign at the finite-file target; stratification reduces this to 1.278% but does not eliminate it (Sec. 5.3). SteerCert assesses which population claims remain statistically supported despite this uncertainty.

“assumptions about activation distributions that may not hold in real-world settings.”

Our assumptions allow broad classes of activation distributions and are motivated by how steering vectors are actually constructed. They permit non-Gaussianity, anisotropy, and dependent coordinates, subject to finite second moments and the stated covariance restrictions. Gaussianity is used in lower-bound constructions; both upper bounds use second moments.

The sampling assumptions are motivated by the construction process: estimating directions from finite samples and distinguishing independent observations from examples sharing a template. Under these assumptions, matching sample-size scales at fixed confidence answer a concrete design question: what additional examples can achieve, and when more independent templates are needed. As reviewer wQA9 notes, this provides “useful guidance for building steering datasets.”

Specifically, Theorem 1 assumes iid sampling. Theorem 2 uses an additive model with mutually independent template effects and within-template errors; its upper bound also uses supplied magnitude, reference direction, and covariance information, with a locality condition.

“The empirical SteerCert results rely on approximate bootstrap confidence sets.”

The multiplier ball has fixed-dimension asymptotic justification under its stated assumptions. Section 5.4 reports the empirical results with bootstrap/approximate calibration status. In additional finite-source validation on two behaviors, we evaluated the ball at n=128n=128 and 1,000 under ordinary and answer-position-stratified sampling. Empirical coverage of the full mean contrast, averaged over six behavior/layer cells, ranged from 94.33% to 95.28% at nominal 95%. These finite-source checks do not establish a 95% guarantee for an unknown generating population.

“SteerCert may reject useful steering directions …”

Withholding concerns support for a population claim; it does not reject the vector as an intervention. Positive finite responses occur among withheld cases. For the same predeclared local-response claims, a simultaneous projection band increases support from 50 to 788 of 1,200 evaluations, with the same data and nominal error allocation within each 100-input family. The projection band applies the same confidence-set criterion for statistical support defined in Sec. 3, tailored to the declared local-response family. The ball represents uncertainty about the full mean contrast for directional verdicts; the band does not replace them.

For the additional projection-band comparison, finite-source validation on two behaviors gives simultaneous coverage near 95% at n=1,000 and 93.8–93.9% at n=128n=128, averaged over six 100-input families. These finite-source checks do not establish a 95% guarantee for the original 788 supports. See our response to WG7v (W1/Q2) for the validation and limitations.

The additional comparisons show that local-response support depends substantially on the confidence construction. We will incorporate this finding into the conclusions.

“whether statistically certified directions lead to more effective or reliable steering.”

Section 5.2 of the current manuscript shows that answer-position stratification improves response repeatability for myopic reward under finite-strength interventions. To connect local statistical support with finite behavioral effects, we additionally measured responses at three nonzero intervention strengths. We intervened along finite-file mean directions and measured changes in answer-letter probabilities rather than scores of freely generated text. This keeps the response measure consistent between the local analysis and the finite-strength experiments.

At κ=0.5\kappa=0.5, the ball withholds 878 of 928 positive finite responses, versus 170 for the projection band. Among band-supported evaluations, 758/788 have positive finite responses, versus 761/788 under matched-count center selection; we found no consistent intervention-selection advantage on this panel. See our response to WG7v (W2/Q1) for all three strengths and false-positive/negative rates.

We will sharpen the contribution statements, state each experiment’s question and comparator first, and separate theoretical assumptions from empirical calibration in the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of manuscript revisions

We have now updated the manuscript. The revised manuscript incorporates the additional analyses and clarifications presented in our responses. Below, we summarize the main changes, with page numbers referring to the revised manuscript.

  • Local-response support and conservatism — Sec. 5.4, pp. 9–10; Appendix I. We added the covariance-aware projection-band comparison, finite-source coverage and recall validation, and width-adjustment analysis. These results show that response-specific inference supports more local-response claims. We distinguish this support from the ball-based directional verdicts. (WG7v W1/Q2; wQA9 on conservatism; bZCt on withholding useful claims.)

  • Answer-position structure — Secs. 5.1–5.2, pp. 6–8; Appendix E.1. We added the paired answer-order diagnostic, which links the answer-position axis to order sensitivity within the same questions. We clarified its connection to the stratification controls and distinguished diagnosis of the axis from changing the construction target. (wQA9’s question; bZCt on the contribution beyond classical stratification.)

  • Finite-strength responses — Sec. 5.4, p. 10; Appendix I.6. We added answer-probability measurements at three nonzero strengths, matched-count selection comparisons, and false-positive/negative rates. These analyses examine how local statistical support relates to finite-strength effects. (WG7v W2/Q1; bZCt on steering effectiveness.)

  • Assumptions and calibration — Secs. 3–4, pp. 4–5; Appendix D.2, including Table 7. We clarified the assumptions for the theoretical results and confidence constructions, and separated theoretical guarantees from empirical calibration. (bZCt on distributional assumptions and approximate bootstrap confidence sets.)

  • Contribution statements and interpretation — p. 2 and p. 10. We updated the contributions, conclusion, and limitations to reflect the additional findings and distinguish construction-design benefits, statistical support, and finite-response behavior. (bZCt on novelty; all reviewers on the interpretation of low support.)

The supplementary archive, steercert-supplementary-material.zip, includes:

  • manuscript_changes_highlighted.pdf: the revised manuscript with revised text highlighted in red.
  • manuscript_original.pdf: the manuscript previously uploaded to OpenReview, included for comparison.
  • steercert-anonymous-code/: the unchanged code package, with reproduction instructions in its README.md.

The clean revised manuscript is provided as the main paper PDF.

AuthorsOpenReview ↗

Summary of our responses

We thank the reviewers for their thoughtful questions and constructive suggestions. As the discussion draws to a close, we summarize how our original results and additional analyses address three questions raised in the reviews.

  1. The ball was conservative for the declared local-response family: a projection band supported many claims that the ball withheld. In the original analysis, the ball supported a positive local response in 50 of 1,200 evaluations. On the same data, the covariance-aware simultaneous projection band supported a positive response in 788 evaluations, with the same nominal error allocation within each 100-input family. In a separate finite-source validation on two behaviors, the band recovered many correct claims that the ball withheld. Its empirical simultaneous coverage was near 95% at n = 1,000 and 93.8–93.9% at n = 128. When a steering direction is reused across inputs, the ball provides a common uncertainty set for assessing directional precision and local-response claims together. (Sec. 5.4; WG7v W1/Q2; wQA9 on withheld claims.)

  2. The answer-position axis largely reflects within-question answer-order sensitivity. In the additional diagnostic, we evaluated the same questions in both answer orders. The axis aligned closely with the order-sensitive component of the meaning-aligned contrasts. In the original comparison for myopic reward, proportional answer-position stratification yielded lower response variance than random stratification at the same sample size. Together, these results help explain why controlling answer-position composition improved response repeatability for myopic reward. (Secs. 5.1–5.2; wQA9 on the source of variability; bZCt on the contribution beyond classical stratification.)

  3. Most band-supported evaluations had positive finite responses at each tested strength, but agreement declined as intervention strength increased. In the additional analysis, we intervened along normalized file-mean directions at three nonzero strengths and measured changes in the probability of the behavior-matching answer letter. Of the 788 band-supported evaluations, 784 had positive responses at the smallest strength and 691 at the largest. For comparison, we ranked inputs by their normalized center scores and selected the same number as the band within each behavior/layer family. This matched-count center selection yielded similar counts of positive responses. We found no consistent intervention-selection advantage for the band on this panel. (Sec. 5.4; WG7v W2/Q1; bZCt on steering effectiveness.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.