acceptodds
Under review as a conference paper at ICLR 2027

Steering LLM Coding Behaviors with Concept Vectors

Abstract

Large language models follow coding-practice instructions, but prompt-only control neither reveals internal structure nor avoids repeating requirements. We test whether coding behaviors admit useful residual directions. Using factorized multilingual prompts, we generate and teacher-force code from six Qwen2.5-Coder-Instruct models (0.5B–32B) and estimate compliance-conditioned post-token mean differences for commenting, logging, detected defensive constructs, and programming language. The prompt-conditioned contrasts show coherent lexical and cross-language structure. Adding them after the first sampled token changes held-out outputs, but effects depend on model and layer, strong scaling harms validity and correctness, and axis specificity remains unresolved without matched-norm causal controls. Among endpoint-compliant tasks, explicit comment prompts shift outputs along the training-derived direction. Pre-summed vectors share one hook and one residual addition per steered decode call. In one correctness-screened 7B case, prompting and steering both reach 100% detected compliance; steering uses 17.6% fewer input but 5.1% more output tokens (9.0% fewer total), with 0.43 Python pass versus 0.44 for prompting. None of six tested positive strengths qualifies for a logging-plus-defensive profile. These empirical controls require model- and layer-specific calibration and executable evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, coding, steering, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uGDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether coding practices can be represented and controlled through residual activation directions. Using factorial prompts and compliance-conditioned mean differences across six Qwen2.5-Coder models, the authors analyze comments, logging, defensive constructs, and programming languages, then evaluate held-out steering, vector composition, prompt alignment, and token costs. The main contribution is an empirical characterization of when these directions correlate with and alter coding behavior, including their limited specificity and substantial correctness costs, rather than a new steering algorithm.

Critical strengths
  • The factorial multilingual design matches tasks and non-target behavior settings, providing a clearer extraction protocol than loosely contrasted prompts. Separate training, validation, and test tasks reduce direct tuning on the reported test outcomes.
  • The representation analysis combines variance decomposition, cross-language alignment, analogy retrieval, vocabulary projection, and random-direction lexical diagnostics. These complementary views offer useful descriptive evidence while acknowledging the limitations of each diagnostic.
Critical weaknesses
  • My main concerns lie in the gap between prompt-conditioned representation differences and behavior-specific causal mechanisms. The vectors are extracted from teacher-forced post-token states after selecting outputs that already satisfy behavior detectors, so they may encode the consequences of generated code and selection effects rather than a distinct variable that initiates the behavior. Intervening with such a direction establishes an effect on generation, but not the causal interpretation assigned to the direction itself.
  • The primary steering experiments lack norm-matched random and shuffled-vector intervention controls. For example, in the 14B model, logging steering increases comment detection by 0.36 but logging detection by only 0.08, leaving target specificity unresolved.
  • Validation selects interventions by detector induction without a correctness constraint, and some selected comment settings reduce Python test passing to zero. This establishes perturbation capacity but leaves the practical operating region much less convincing than the headline behavior changes suggest.
Questions for authors
  • Why do the logging directions reverse their effective steering sign at 1.5B and 3B?
  • Does a direction increase the probability of initiating a coding behavior, or mainly amplify that behavior after its first occurrence?
3 comments

Thank you for identifying the empirical characterization as the main contribution. Representation differences, intervention effects, and practical operating points require separate interpretations.

1. What the extracted direction represents

Section 3.2 estimates the average within-pair difference in teacher-forced post-token residuals, matching task, language, and the two non-target instruction settings. These states have already observed the generated token and still attend to the original instruction prompt. Examples are selected by syntax and target-detector compliance. Thus the contrast can contain prompt effects, output-content differences, and selection effects.

Matching reduces the specified nuisance differences; it does not identify an endogenous variable that initiates the behavior. Adding the direction to held-out neutral-prompt generations establishes an effect of that intervention, while its interpretation as a distinct behavior-specific mechanism remains unresolved. We will define "concept vector" operationally at first use and avoid assigning antecedent causal meaning to the extraction.

Removing comment/docstring spans before pooling yields cosine 0.45-0.66 with the full direction (Table 4). The pooled positions exclude literal comment spans, but remaining states can still encode preceding comments and the prompt. Neither conditioning nor selection is removed; neutral-context and pre-onset analyses are untested.

2. Specificity controls

We agree with the 14B counterexample: logging steering increases commenting by 36 percentage points and logging by 8 (Table 8). At 7B, comment and logging directions each increase commenting by 57 points. These observed off-target responses prevent interpreting the three directions as independently controllable axes.

Random lexical and alignment diagnostics do not substitute for matched-norm decoding interventions, which are absent. Equal coefficients do not match perturbation norms because Equation 4 includes the direction's extraction norm. Controls should match the injected delta norm, layer, coverage, and decoding, using multiple random and within-pair sign-shuffled directions with comparable validation-selection budgets. Held-out evaluation should include every detector and correctness. Specificity remains unresolved.

3. Why does logging reverse sign at 1.5B and 3B?

The extraction orientation is fixed as positive minus negative. Validation selects coefficient -4 at zero-based layer 14 for 1.5B and layer 18 for 3B (Table 7). Held-out logging increases by 19 and 23 percentage points, with 95% CIs [8, 29] and [12, 34], respectively (Table 2). We mark both as directional failures, rather than evidence that the extracted positive direction induces logging.

The experiment does not identify why the sign reverses. A mean difference among post-token states need not have the same signed effect when injected during generation. Prompt conditioning, selected output features, correlated behaviors, and nonlinear response are possible explanations, not established mechanisms. We will state the observation and this distinction directly, without attributing the reversal to model scale or sampling noise.

4. Initiation versus amplification

Our endpoint detectors cannot distinguish these mechanisms. "After the first token" means after the first output token, not after the first comment or logging occurrence. Subsequent steering is applied whether or not the target behavior has appeared, so it can affect a later first occurrence as well as continuation after an occurrence; the first output-token distribution itself is unchanged.

Common behavior-free prefixes could test first-onset probability; separate prefixes containing an occurrence could test continuation intensity. These analyses and coverage ablations are unperformed, so we make no initiation-specific claim.

5. Correctness-aware operating points

The primary sweep intentionally maximizes detector induction, then reports held-out behavior and correctness together. Its selected 7B/32B comment settings have 0% Python pass, and 14B logging has 4% (Section 6.1). These are destructive settings, not deployment recommendations.

Appendix G uses a separate validation screen: parsing/extraction at least 95%, and pass no more than five percentage points below the higher prompt/neutral pass rate. The held-out 7B concise-Python profile reaches 100% detected compliance, 99% parsing, and 43% pass (prompting: 100%, 100%, and 44%) and 9.0% fewer total tokens. The paired pass difference is -1 percentage point, 95% CI [-8, +6]; neither equivalence nor noninferiority is established. No tested positive logging-plus-defensive strength qualifies. The single calibrated suppression result does not establish a broad practical operating region.

Thank you for distinguishing a representation contrast from a selective causal control. We ran three checks to address that distinction directly. All rates below are over all requested tasks; intervals are paired task-bootstrap 95% intervals.

Does extraction require the original instruction to be present? We replayed exactly the same accepted 7B training programs under a common neutral prompt for each task and language. We retained the original matched pairs and matched the injected residual norm. Original/neutral-replay cosines are 0.81 for comments, 0.77 for logging, and 0.90 for defensive constructs. That geometric agreement did not imply equivalent interventions. At the same moderate injection norm, the original comment vector raised detection by 44 percentage points (95% CI [34, 54]), while the neutral-replay vector raised it by 5 points ([-2, 13]); the direct difference was -39 points ([-49, -29]). Both passed 45% of the same 100 Python tasks. In contrast, the neutral-replay defensive direction retained a 28-point detector gain ([20, 37]). Thus a common replay prompt preserves much of the geometry but changes some intervention effects. Both extracts remain post-token and outcome-conditioned; neither identifies an endogenous causal variable.

First occurrence versus continuation. At a moderate 7B setting, we held the assistant prefix fixed and compared decoding with and without the same comment direction. With a comment-free code fence as the prefix, first detected comment/docstring occurrence rose from 40% to 85% (+45 points [35, 55]). When the prefix already supplied one comment, further occurrences rose from 69% to 94% (+25 points [17, 34]); mean additional count rose from 2.65 to 4.48. In a separate ordinary generation ablation, the rates were 38% without steering, 83% for all positions, 83% for continuation only, 80% for only the first 32 continuation calls, and 48% when steering began after those calls. This demonstrates an effect on later first occurrence as well as on additional occurrences, under these prompts and this operating point. It does not tell us whether a model had already planned the behavior internally.

Logging sign and specificity. Holding layer and absolute coefficient fixed, the extraction-negative vs. extraction-positive directions produced logging on 26% vs. 20% of 1.5B tasks and 51% vs. 6% of 3B tasks. Across 100 random disjoint halves of the training tasks, median split-half cosine for extraction was 0.978 and 0.980, respectively. Thus the more effective negative injection is not explained simply by an unstable extraction sign; these experiments do not identify the underlying cause. Three Gaussian and three within-pair sign-shuffled directions matched the injected norm at each frozen setting. At 1.5B, the learned logging setting did not outperform the mean of either null family, which limits the selective-control interpretation. In a separate positive comment calibration at 7B, no tested learned coefficient met the five-point validation correctness screen against the stronger of prompting and the neutral reference. We will distinguish the reproducible extraction direction, the effect of an intervention, and the subset of settings supported as usable controls.

Thank you for considering our follow-up. We added neutral-context replay and fixed-prefix experiments that distinguish the first occurrence of a comment from further commenting. We’d welcome your updated assessment and would be grateful if you could reflect any improvement in your assessment in your rating.

Reviewer_Wc9jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates activation steering of coding-related behaviours in Qwen2.5-Coder-Instruct models using residual-stream concept vectors extracted from contrastive and factorized prompts. Results indicate that these extracted vectors can steer coding behaviours but effects strongly are model-layer dependent, can degrade code correctness and gives modest token savings for a 7B focused experiment.

Critical strengths
  1. Presentation Clarity: The paper is well organized and has detailed presentation of results alongside findings.

  2. Experimental Rigor: The paper presents results across various model scales with 3 languages with bootstrap confidence intervals and per-layer/per-scale sweeps. Specifically, The paper reports sign-reversed selections (logging at 1.5B and 3B), 0% pass rates at the 7B and 32B comment settings, a debug-mode profile for which no positive strength qualifies, and degeneration in the all-layer sweeps. These negative results are useful for practitioners who assume steering works out of the box.

Critical weaknesses

The paper has several weaknesses that weakens my confidence towards accepting this paper. Some of which are listed below.

  1. The observed steering effects do not consistently establish that the extracted vectors selectively control the intended coding behaviors. For ex: comment and logging vectors can induce similar changes in commenting, suggesting that the directions may capture correlated features rather than distinct behavioral concepts.

  2. The steering experiments lack random-direction or norm-matched controls. Random directions appear only in the descriptive analyses (vocabulary projection and prompt alignment) and not in the interventions. Without such controls, concept-specific effects cannot be separated from generic residual-stream perturbation. This matters because 14 of the 18 selected settings use ∣α∣≥4|\alpha| \ge 4, and at 7B and 32B all three behaviours select the same layer and strength (Table 7).

  3. The use of static/regex-based behavioural detectors without human validation for Java/C++ can potentially compromise the reliability of the reported steering effects.

  4. The evaluation is restricted to the Qwen2.5-Coder-Instruct family, leaving it unclear whether the observed steering effects generalize to other code LLM families.

  5. Functional correctness is evaluated only on Python tasks, leaving it unclear whether correctness trade-offs extend to Java and C++ code generation.

  6. The paper does not evaluate alternative token-position coverage, leaving the robustness of its chosen steering intervention strategy unclear.

Questions for authors
  1. How do results and effects generalize beyond Qwen2.5-Coder to other code LLM families ?

  2. Can the primary steering results be evaluated against matched-norm random directions or shuffled concept directions ?

  3. Could the authors assess the impact of activation steering on functional correctness using augmented test suites (e.g., HumanEval+) or more complex programming tasks ?

4 comments

Thank you for recognizing the value of the negative results. The study documents how induction-only calibration can destroy executable correctness and how a separate correctness screen can reject an entire profile. Your concerns clarify the scope of that contribution.

1. Specificity and matched intervention controls

We agree that the extracted directions do not consistently demonstrate selective control. Table 8 reports all three detector changes for each intervention. At 14B, logging steering increases comments by 36 percentage points but logging by only 8. At 7B, comment and logging steering both increase commenting by 57 points. Although the target has the largest absolute change in 14 of 18 settings, these exceptions prevent a general specificity claim.

The submitted study has no matched-norm random or shuffled-direction decoding interventions. The random directions in the lexical and prompt-alignment analyses are descriptive controls and do not resolve this objection. Consequently, our causal conclusion is that the specified injections change held-out generation; attribution of those changes to a distinct behavioral concept remains unresolved.

The shared layer/coefficient selections also need careful interpretation. Fourteen settings use an absolute coefficient of at least 4 (Section 6.1). Equation 4 multiplies each unit direction by its own extraction norm, so equal coefficients at 7B or 32B do not imply equal injected residual norms. We will make this distinction explicit.

Resolving this requires multiple random and within-pair sign-shuffled controls matched to the injected norm, layer, coverage, and decoding, with equal validation-selection opportunities and frozen held-out evaluation of all detectors, failures, and correctness over all requests. These experiments have not been run.

2. Detector reliability and language scope

One clarification is that the primary behavior-steering evaluation is Python-only: its neutral prompts specify Python and the expected function signature (Appendix B.1). Python comment detection uses tokenize, and docstrings and logical code lines use the AST (Appendix B.2). Java/C++ regex detectors affect multilingual extraction and C++ composition; Tables 2 and 8 use Python detectors.

Your reliability concern nevertheless remains relevant. Java/C++ patterns can mistake markers inside strings for comments, and their balanced-delimiter checks do not establish compiler validity. No human validation of the behavior detectors is reported in any language. Java/C++ also lack compiler and executable correctness checks. Even the Python behavior detectors measure construct presence, not the usefulness of comments, appropriateness of logging, or adequacy of input validation. We will separate these measurement claims from software-quality claims throughout.

3. Correctness and stronger test suites

The main sweep selects the largest detector induction on validation without a correctness constraint. The 0% pass rates for selected 7B/32B comment settings are genuine failures of that objective (Figure 2c); they should not be interpreted as usable operating points.

The separate cost experiment instead screens validation candidates for at least 95% parsing/extraction and pass rate no more than five percentage points below the higher prompt/neutral rate (Appendix G). On held-out concise-Python tasks, steering achieves 100% detected compliance, 99% parsing, and 43% Python pass; prompting achieves 100%, 100%, and 44%, respectively. Steering uses 9.0% fewer total tokens. The paired pass difference is -1 percentage point, 95% CI [-8, +6]; the point-estimate screen does not establish noninferiority. No tested positive debug-mode strength qualifies, so the suppression result does not validate positive logging-plus-defensive control.

Correctness uses the original MBPP Python assertions. We have no HumanEval+, augmented-test, or repository-scale results. Selecting longer multilingual MBPP reference solutions reduces one-line dominance but does not answer your robustness question. The correctness conclusion is limited to this test suite and language.

4. Generalization and token-position coverage

The six Qwen2.5-Coder-Instruct checkpoints support within-family recurrence under a shared protocol. Replication with re-extracted directions in another or newer family was not tested. The size sweep does not establish cross-family generalization.

Steering is applied on every continuation decode call after the first sampled output token. Prompt prefill and first-token logits are unchanged (Section 3.3; Appendix B.1). We have no alternative coverage ablation; the all-layer sweep varies depth and does not establish robustness to token-position choices. The evidence therefore applies to this precise intervention schedule.

I thank the authors for their responses. However without additional experimental results addressing the core concerns mentioned, I do not feel sufficiently confident to increase my score and will maintain my original rating

Thank you for identifying the missing intervention controls and executable tests. We added these experiments and report results over all requests, including malformed, capped, and failing generations.

Matched intervention controls. We tested ten frozen behavior settings across five Qwen2.5-Coder sizes. Each has three Gaussian and three within-pair sign-shuffled controls with the same injected L2 norm, layer, coverage, prompt, and decoding, on the same 100 held-out tasks. We also ran a one-clause prompting arm. The 32B logging vector beat both control families on logging in valid outputs: +21.0 percentage points vs. Gaussian (95% paired task CI [13.0, 29.0]) and +20.3 vs. shuffled ([12.7, 28.3]); pass differences vs. both controls were inconclusive. Prompting still reached 100% logging and 65% passage versus 21% and 55% for the vector. The 7B comment vector gave 97% raw detection vs. 51–68% for Gaussian controls, yet no learned output parsed or passed. At 14B, learned logging gave 3% logging detection, 81% commenting and 2% passage; the matched prompt gave 96% logging and 60% passage. In a separate 7B comment calibration with the same grid for learned and control directions, no learned positive coefficient qualified (95% valid extraction and passage within five points of the stronger baseline).

Executable correctness beyond the original Python tests. We took the same moderate 7B comment injection into four additional evaluations with no benchmark-specific tuning. Pass rates for neutral / explicit prompting / steering were:

Test suiteTasksNeutralPromptingSteering
HumanEval+ Python16476.8%79.3%73.8%
MBPP+ on the original test intersection3225.0%40.6%34.4%
MultiPL-E C++16169.6%67.7%64.0%
MultiPL-E Java15072.0%71.3%67.3%

The C++ and Java runs compile and execute the native test programs; the Java subset excludes tasks requiring an optional tuple library. Steering raised comment occurrence over neutral in each suite, but prompting reached higher comment rates and had a higher pass point estimate in every suite. These are measured tradeoffs, not a correctness guarantee.

Detector and coverage checks. On 799 C++ and 800 Java generated programs, an independent syntax-tree detector agreed with the original regex detector on 100% of comment decisions and 99.9% of logging decisions in both languages. Defensive-construct agreement was 86.2% for C++ and 97.6% for Java. This identifies a sensitivity of the C++ defensive metric; it is not human validation. At a moderate 7B comment strength, detection was 38% unsteered, 83% with all-position or continuation-only injection, 80% with the first 32 continuation calls only, and 48% when starting after call 32. Their test passage was 44–46%.

Model-family scope. We re-extracted three vectors in Gemma 4 E4B-IT. At validation-selected positive strengths the held-out comment/logging/defensive rates were 98%/2%/15%, compared with 99%/2%/13% unsteered. A separate exploratory negative comment direction reached 4% valid comment-free code vs. 94% for prompting. We will report this limit on transfer alongside the Qwen results.

Thank you for considering our follow-up. We added norm-matched random and sign-shuffled controls and tested executable correctness on HumanEval+, MBPP+, and native Java and C++ suites. If these checks address your concerns about specificity and correctness, we’d appreciate your considering a higher score.

Reviewer_Z3vtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an assessment of whether coding behaviors can be controlled through concept vectors added to the LLM's residual stream. The coding behaviors studied were: commenting, logging, defensive constructs, and programming language. The authors follow previous work and use factorized prompts to elicit these behaviors, generating positive and negative examples and extracting raw directions in the latent space through their average within-pair difference (which they call a concept vector). At generation time, they steer the model by adding this vector to the output residual of one decoder block on output tokens after the initial prompt. They find that this steering seem to alter coding behavior across model scale, the extracted concept vectors show signs of analogy-style relations, and their vocabulary projections decode to tokens related to the concept.

Critical strengths

The authors study the method from multiple dimensions, including the model's behavior, a descriptive analysis of the concept vectors and their combinations, how steering should be done for each configuration, and against controls. However, the evidence for each of these dimensions is not convincing (see Weaknesses).

Critical weaknesses
  • Practically all claims are made with significant caveats. This makes it very difficult to pinpoint the contributions to the field and the validity of the takeaways.
  • The results don't support a clear advantage against prompting, and the method's effectiveness on more complex prompts (with e.g. thousands of tokens, where an advantage would begin to be useful) isn't tested. The evidence is also bounded by a single model family and a single dataset of very simple programming problems.
  • Finally, the paper is very unclear. For example, the abstract only makes sense after reading the full paper. Notation and jargon is used before proper introduction, and they change across the manuscript (e.g., sometimes numbers are shown as percentages, sometimes in decimals). This makes it very difficult to analyze the soundness of each choice in the experimental setup, the results and their interpretation.
Questions for authors
  • Please highlight the paper's main contributions and their relevance to the community more clearly.
  • By now the models studied are considered outdated and the observed behavior may not transfer to newer models or other families. Please explain the model family choice, and why transfer was not tested.
4 comments

Thank you for the review. We agree that the main takeaway should be easier to identify, and that the empirical scope needs to be clear from the abstract.

1. Main contributions and relevance

We extract average hidden-state differences between code generated with and without a requested practice, then test how adding these differences changes generation and Python test passage. Our contribution is a coding-specific empirical audit using established difference-in-means steering. Relative to the cited language/library-control work (Section 2), we examine local practices, off-target effects, and executable correctness. We address three questions:

  • What contrasts emerge under a matched design? We match task, programming language, and the two non-target instruction factors in a complete factorial design, across six checkpoints and three languages (Sections 3.1-3.2). Separate sets of 100 training, validation, and test tasks separate extraction, intervention selection, and held-out evaluation.
  • Does changing a behavioral detector imply useful code control? The answer can be sharply negative: the selected 7B and 32B comment interventions increase detected commenting by 57 and 71 percentage points, respectively, but both yield 0% Python test passage (Table 2; Figure 2c). Logging sometimes changes commenting more than logging (Table 8). These results show why correctness and off-target effects must accompany detector gains.
  • What does correctness-screened calibration establish? A separate correctness-screened 7B suppression profile reduces token use on held-out tasks, whereas none of six tested positive strengths qualifies for the logging-plus-defensive profile (Section 7.4; Appendix G). This documents an observed token/compliance tradeoff and a profile for which calibration fails.

Cross-language comment-offset alignment of 0.78-0.92 supplies supporting descriptive evidence (Section 5.2), with no identification of independent causal mechanisms. We will foreground the three questions above.

2. Comparison with prompting

The direct comparison is limited to the 7B concise-Python profile on 100 held-out tasks (Table 14):

OutcomePromptingSteering
Detected joint compliance100%100%
Python test passage44%43%
Input tokens/request107.7688.76
Output tokens/request66.3169.68
Total tokens/request174.07158.44

This is a measured 9.0% reduction in total tokens, comprising 17.6% fewer input tokens and 5.1% more output tokens. The pass-rate difference is -1 percentage point, with a 95% paired-bootstrap interval of [-8, +6] percentage points; correctness equivalence is not established. The main intervention sweep has no matched prompt-only arm, and this one profile does not establish a general advantage over prompting. Reusable system-prefix instructions could also change the token comparison. We will state these boundaries alongside the result, rather than suggesting a general replacement for prompting.

3. Model choice, task complexity, and transfer

The six Qwen2.5-Coder-Instruct checkpoints provide a common-family design spanning 0.5B-32B, with accessible residual activations and the same task/prompt protocol. This supports a within-family characterization; it does not isolate parameter count from other checkpoint differences or establish applicability to newer families.

The evaluation covers a six-size within-family audit. Cross-family replication and direct vector reuse were outside that scope, leaving the newer-model concern empirically unresolved. Re-extracting and calibrating directions in another family would test the protocol's generality; reusing a vector across models would additionally require addressing their different representation spaces. These are separate evaluations.

We selected the 300 aligned multilingual MBPP tasks with the longest minimum reference-solution length across languages (Section 3.1), reducing domination by one-line tasks. They remain short program-synthesis tasks. Thousands-token input contexts and repository-scale coding are untested, and the 1,200-token output cap is not evidence about long-context effectiveness.

4. Presentation changes

We will simplify the abstract around the extraction procedure, held-out effects, correctness failures, and the single calibrated comparison. We will define a "concept vector" immediately as an average hidden-state difference between matched generated-code examples, identify its prompt/compliance conditioning, and distinguish the shared analysis layer from validation-selected intervention layers. Rates will use percentages, rate changes percentage points, and cosine values unitless decimals. Behavior changes and Python pass rates will be presented together so that the practical interpretation is visible without searching the appendices.

I thank the authors for the response. Unfortunately, without experiments supporting a clear advantage over prompting, transfer to other model families (especially newer ones), more robust controls like the other reviewers pointed out, or any kind of correctness guarantees (which would be crucial for coding applications), I'm unconvinced that this paper would provide much value to the community and should be accepted to ICLR.

Thank you for clarifying which evidence would change your assessment. We have now added matched prompting baselines, a longer-input comparison, a harder programming benchmark, and a replication in a newer model family. These results narrow our claim about when steering is useful.

Prompting and longer inputs. We reused the submitted 7B suppression profile, without tuning on the test tasks, on the same 100 tasks with a shared synthetic repository inventory. Prompting and steering each achieved 99% syntax-qualified joint compliance and 46% executable pass. Their paired pass difference was 0 percentage points (95% CI [-6, +6]). Steering saved 19 input tokens per request (3,145.59 vs. 3,164.59) and 17.38 total tokens (3,217.76 vs. 3,235.14), just 0.54% of total tokens here. Moving the requirements before task-specific text made the average input outside the common token prefix identical to steering (68.59 tokens) under ideal prefix reuse. This is a token-accounting result, not a measured latency improvement. The inventory isolates input length, so we separately tested actual contest problems.

Harder programming tasks. On 43 held-out Code Contests problems with six sampled private input/output tests each, we reused the comment-only suppression component and compared it directly with a no-comments instruction. Both passed all six tests on 3 of 43 problems. Prompting had fewer detected commented outputs (4/43 vs. 9/43) and used fewer total tokens (895.72 vs. 946.67 per request), despite steering's shorter input. This is a strict output-matched subset, not the official Code Contests score or a repository-level evaluation. For positive comment induction on HumanEval+, prompting also led steering on both comment detection (100% vs. 55.5%) and test passage (79.3% vs. 73.8%).

Replication in a newer family. We re-extracted all three vectors in Gemma 4 E4B-IT using a Python factorial grid, independently selected strengths on validation, and evaluated 100 held-out tasks. The neutral rates for comments, logging, and defensive constructs were 99%, 2%, and 13%; the corresponding steered rates were 98%, 2%, and 15%. Those settings passed 39%, 43%, and 23% of the tasks, versus 45% unsteered. Given the ceiling on positive comments, we ran an exploratory negative-direction comparison. Learned steering gave 4% valid comment-free outputs and 31% test passage; a no-comment prompt gave 94% and 47%. These results show that the extraction protocol can be run in another family, but usable control did not replicate at the tested settings.

We will foreground what these experiments establish: a controlled audit of how coding detectors, executable correctness, and transfer can diverge, with a narrow token-count benefit for one suppression profile. We do not claim a general advantage over prompting or guarantees about correctness beyond the evaluated suites. We will use percentages for rates, percentage points for changes, and define the extracted difference before introducing the term “concept vector.”

Thank you for considering our follow-up. We compared steering with prompting on longer inputs and harder programming tasks, and tested whether the results transfer to Gemma 4. We’d be grateful if you could reconsider your score in light of these results and the limits they clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.