acceptodds
Under review as a conference paper at ICLR 2027

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs

23 views

Abstract

In many settings, post-training need not create the target behavior from scratch: the base model can already produce it, but not reliably. This shifts part of preference alignment from capability acquisition to behavioral expression. This raises a central question: if target-supporting computation is already present, what prevents the preferred behavior from reliably dominating generation? At each generation step, many Transformer components write to the same residual stream, yet their effects are combined into a single next-token distribution. A target-supporting computation can therefore be present yet be outweighed by other computations. We hypothesize that reliable expression depends on this internal competition during inference. We introduce Residual Competition Maps (RCMs), which map a specified preference onto native residual computation by measuring signed causal effects relative to that preference. Across preference domains, RCMs reveal residual competition whose prevalence varies by task, distinct component roles relative to the target preference, and cases where a single native-component intervention reverses the preference outcome. DPO substantially reorganizes these effects and can weaken opposing effects, which may nevertheless persist. Beyond analysis, RCM provides causal guidance for where to intervene. To directly control preference formation during inference, we propose Direct Hidden-State Alignment (DHSA), which treats inference-time hidden states rather than base-model weights as the direct adaptation space. RCM-guided Causal Activation State Transition (CAST) instantiates this principle through local state interventions at a small number of preference-relevant interfaces while freezing the base model. With only 256–16,384 controller parameters, CAST reaches DPO-competitive operating points across three preference domains, can complement DPO-trained models, and can be enabled or removed at inference time.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Alignment, Mechanistic Interpretability, Residual Competition, Hidden-State Control, Causal Intervention, Activation Steering, Residual Stream, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v5cjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that when a preferred behavior is already reachable, alignment is partly a problem of expression: target-supporting computation may be outweighed by other components writing to the same residual stream ("residual competition").

RCM analysis.

  • It introduces Residual Competition Maps (RCMs). For each input, an RCM measures the signed change in a task-specific preference score when a single component's state is zeroed (RCM-Zero) or replaced by a prototype state (RCM-Patch).
  • On ConFiQA with Llama-3-8B, both-sign effects occur on most inputs (283/300).
  • DPO reorganizes these effects but does not remove all opposing ones.

DHSA and CAST.

  • It then proposes Direct Hidden-State Alignment (DHSA), the principle of adapting inference-time hidden states of a frozen model.
  • DHSA is instantiated as CAST: constant-vector or low-rank controllers at RCM-selected attention heads (Eq. 3), trained with a reference-relative DPO-form loss.

Validation.

  • With 256–16,384 controller parameters, CAST is reported to reach "DPO-competitive operating points" on ConFiQA, IMDb, and TL;DR across several model families.
  • It can also be added on top of released DPO checkpoints (Table 2, Fig. 3).
Critical strengths

Critical Strengths

  1. Unusually careful and candid reporting.

    • Selection and test splits are separated and documented (C.4).
    • Paired bootstrap intervals are given for the main contrasts (Table 10), and IMDb reward is always paired with KL.
    • Caveats are stated explicitly: DPO references are not data- or compute-matched (C.3), and RCM intervals are not multiplicity-corrected (A.2).
    • Negative results are kept, e.g., ITI beating RCM-Zero on ConFiQA Llama and CAST-all collapsing on Llama EM.

    This makes the paper's evidence easy to audit, and it is why I consider the empirical portion worth engaging with despite the concerns below.

Critical weaknesses

Critical Weaknesses

  1. The conceptual contributions largely relabel existing tools, and the benefit of RCM-guided localization is not isolated. RCM-Zero and RCM-Patch are single-component zero-ablation and mean activation patching (B.1). Components that oppose the final prediction are already documented, e.g., Negative Name Mover heads (Wang et al., 2022) and copy-suppression heads (McDougall et al., 2023). DHSA, "adapt hidden states of a frozen model at inference," describes existing methods such as ReFT, LoFiT, and BiPO. Empirically, RCM does not consistently beat ITI (ConFiQA Llama: 85.64 vs. 78.61). Its comparison is also confounded by different bank designs, which the authors acknowledge (C.3).

  2. The "DPO-competitive" claim is overstated. Table 2 contradicts it in several places. On ConFiQA Llama-3-8B, CAST-all has EM = 0.00, and on TL;DR LION it is significantly below DPO. On ConFiQA, CAST is below both BiPO and LoReFT. All DPO references are released checkpoints trained with different data and compute. DPO trained on the same pairs as CAST, the natural control for the paper's central thesis, is absent.

Questions for authors

All questions are clarifications that could change my assessment.

  1. How does CAST compare with DPO or LoRA-DPO trained on the same pairs and evaluated identically?
  2. With bank design equalized across selectors, does RCM still outperform ITI and random selection in Table 1?
  3. Does controlling only target-competing sites help beyond supporting-only or random sites? Without this, the "residual competition" analysis is not connected to the control method.
7 comments
AuthorsOpenReview ↗

Clarifying the Scientific Advance and Cross-Domain Performance

Thank you for recognizing transparent reporting, held-out evaluation and paired statistics.

W1 — Clarifying what advances beyond the existing tools

Thank you for asking us to clarify the advances over prior work. We will add these clarifications to Related Work and contributions.

1. Single-component zeroing and mean activation patching are measurement operators. RCM maps target–non-target preference advantage to the signed causal effects of native residual writes; CAST optimizes the same preference.

2. Opposing-head analysis extends from IOI/copy suppression to preference alignment. RCM reveals support and opposition toward specified preferences across domains, input-dependent roles and their reorganization after DPO (Fig. 2). We test competing sites as control locations (Q3 in our second reply).

3. CAST advances hidden-state adaptation to DPO-competitive preference alignment. ReFT/LoFiT establish adaptation; LoFiT learns head importance. RCM measures native effects before training; CAST reaches DPO-competitive preference alignment across three domains (W2). Similar interventions already exist, but BiPO and LoReFT do not reach DPO-competitive performance across all three domains in our evaluation: they are strong on ConFiQA but fall below DPO on IMDb and TL;DR. Establishing this cross-domain alignment level is itself part of our contribution.

4. Signed banks are part of RCM's control mechanism, not an extraneous complication. RCM supplies supporting/competing roles that ITI and Random cannot supply. Separate banks use these roles without extra examples or heads. Training data, total heads, actuator, objective, calibration and evaluation are otherwise matched. We do not claim RCM is the best selector in every setting. ITI's advantage on a particular dataset and model does not negate RCM's strong localization performance across preference domains or its contribution. See Q2 in our second reply for matched-bank attribution.

W2 — Clarifying cross-domain DPO competitiveness

Thank you for asking us to clarify cross-domain performance and the interpretation of EM.

1. CAST-all has two wins, two losses and one near tie across five settings. IMDb reaches DPO reward at lower KL; TL;DR is near-tied on GPT-J and lower on LION. Qwen and IMDb favor CAST; both losses are Llama-based (Tables 2/10; Fig. 3), supporting cross-domain competitiveness.

2. Stronger ConFiQA baselines do not imply stronger performance across domains. BiPO and LoReFT are strong on ConFiQA but fall below the same DPO references on IMDb and TL;DR. CAST competes with DPO across all three domains.

3. Semantic analysis explains EM=0 despite faithful answers. PC has high F1 for context-faithful final answers; EM misses faithful answers and has F1 relatively close to, though above, its random baseline. We will retain PC, remove EM from the revised evaluation, and include this semantic assessment in the appendix.

Semantic assessment adds to ConFiQA's official rule-based evaluation. Full assessment is costly, so we sampled 50 shared questions per model across six methods (600 responses). Method-blinded, AI-assisted assessment judges final answers against context and records explanation issues separately. The table below averages F1 over the 12 settings:

MetricMean F1 (%) [95% CI]
PC85.6 [80.3,90.0]
EM53.6 [47.2,59.2]

PC exceeds EM by 32.0 points. EM's F1 is relatively close to its random baseline (46.1%, preserving each setting's positive-prediction rate).

For this small sample, we use 95% CIs and paired comparisons to interpret method differences. Final-answer context faithfulness is summarized below (% [95% CI]):

MethodLlama-3-8BQwen2-7B
SFT36 [24.1,49.9]34 [22.4,47.8]
DPO94 [83.8,97.9]86 [73.8,93.0]
BiPO92 [81.2,96.8]100 [92.9,100]
LoReFT78 [64.8,87.2]96 [86.5,98.9]
CAST-all52 [38.5,65.2]78 [64.8,87.2]
CAST-prefill90 [78.6,95.7]72 [58.3,82.5]

In this semantic sample, Llama CAST-all trails DPO; the other CAST–DPO differences are not clearly separated. BiPO and LoReFT remain stronger overall on ConFiQA. The full-evaluation comparison is summarized in point 1 above.

4. RCM reveals unstable native effects in Llama that may explain its weaker control results. ConFiQA prompt rescans show unstable native-effect rankings and selected heads on Llama, while remeasured heads retain their mean signs. RCM detects this model-specific instability, which may explain why both CAST losses to DPO are Llama-based.

The table below compares RCM rescans on the same 300 ConFiQA inputs with Context/Question/Answer labels (F) or a context-use instruction (I):

ModelOriginal heads retained /8 (F; I)Same mean sign / remeasured heads (F; I)
Llama-3-8B5/8; 2/87/7; 3/3
Qwen2-7B7/8; 7/88/8; 8/8

We will clarify the cross-domain performance conclusion in Experiments and add the semantic and prompt-stability analyses to the appendix.

AuthorsOpenReview ↗

Clarifying Fair Comparisons and Control-Site Attribution

Q1 — Clarifying the DPO reference and data/compute controls

1. The original setup supports a fair performance comparison. Our blanket 'unmatched' wording obscured the controls already in place; thank you for flagging this. Test inputs, prompts and scoring match. CAST uses few ConFiQA pairs.

For IMDb and GPT-J TL;DR, CAST uses far fewer pairs than the published DPO budgets. The table below lists the data relationship for each setting:

SettingCAST pairsRelationship to DPO
ConFiQA240Same source/official prompt; checkpoint count/split undisclosed
IMDb, GPT-2-large1024Subset of public 10000-pair pool; 9.8× larger
TL;DR, GPT-J8192Same OpenAI source; DPO split has 18000 pairs, 2.2× larger
TL;DR, LION8192Shared SFT/evaluation; DPO uses a multi-source mixture

2. Our comparison supports DPO-competitive alignment across domains with few pairs (W2, first reply). Weight tuning can need more data than external control to reach representative alignment performance. Forcing one small or large set on both does not establish their representative performance or cost. We use established public DPO checkpoints; unreleased pairs/splits and recipes make exact retraining costly.

3. CAST, BiPO and LoReFT share training data and evaluation, with comparable preparation budgets. Each retains its native objective and preset tuning. The table below covers candidates to trained controllers (H100-minutes):

SettingCASTBiPOLoReFT
ConFiQA/Llama5251201
ConFiQA/Qwen8044157
IMDb423940
TL;DR GPT-J159160196
TL;DR LION266270295

We will clarify these matches in the setup and add the tables and data-construction rules to the appendix.

Q2 — Clarifying localization gains with bank design fixed

Thank you for suggesting a matched-bank comparison.

1. The original controls support attributing gains to RCM's selection and signed control. Separate banks are part of RCM, not an extraneous complication. Other conditions match: data, total heads, actuator, objective, calibration and evaluation. TL;DR also matches candidate, audit and final-head budgets. Original Table 1 shows RCM leading in three settings.

2. The new matched-bank tests retain location gains. At the same four-head budget and bank organization, RCM banks outperform ITI and Random in most comparisons. Results are in the shared table after Q3.

(a) ITI clearly exceeds the Random mean only on ConFiQA. It slightly exceeds competing RCM on ConFiQA/Llama, while both RCM banks exceed ITI on Qwen. On IMDb, the controller at sites with non-weak RCM effects, used alone, achieves a better reward–KL trade-off than ITI. On TL;DR, competing RCM scores higher than ITI on both models, and supporting RCM also scores higher on LION. Combining banks makes the separation clearer.

(b) RCM banks generally outperform Random. Both banks score above the Random mean on ConFiQA and TL;DR. On IMDb, the reward–KL curve for sites with non-weak RCM effects is better than Random 1/3 and crosses Random 2. KL comparisons use matched reward.

Q3 — Clarifying the control value of competing sites

Thank you for suggesting a direct test of competing sites.

1. Supporting-only and competing-only control perform comparably in the overall evaluation. Competing control is stronger on ConFiQA, the two are similar on TL;DR, and supporting control is stronger on IMDb.

2. Competing sites also provide gains over random sites. Competing RCM exceeds every Random seed on both ConFiQA models and the Random mean on both TL;DR models, as shown in the shared table below.

3. Joint control gives clearer separation at matched head budgets. On ConFiQA and TL;DR, Combined shows clearer separation from eight-head ITI/Random than single banks do from four-head ITI/Random.

Full ConFiQA and TL;DR results for Q2/Q3 are summarized below. Four-head conditions match actuator, data/objective, intervention timing and calibration; supporting/competing banks reuse Performance weights. TL;DR uses the same 8-to-4 audit for each pool. Combined uses eight heads.

Selector / compositionConFiQA Llama PCConFiQA Qwen PCTL;DR GPT-J WinTL;DR LION Win
Supporting60.8966.1152.8373.24
Competing76.8669.2454.8873.63
ITI79.9361.8253.5271.00
Random 158.5439.4551.3772.17
Random 258.4047.2752.4472.46
Random 361.5752.2952.9371.48
Combined (8 heads)81.9382.9657.1379.10

IMDb clarification. Even with extremely weak native negative effects, the corresponding bank's peak sentiment reward exceeds ITI, Random 1 and Random 3, but remains below Random 2 and the supporting bank with strong positive native effects.

We will release the IMDb reward–KL curves with the revision and add the analyses of location gains and joint residual control of supporting and competing components to Experiments and the appendix. TL;DR Win averages both orders against human references.

Thank you for the detailed responses. The compute-budget table, data-source disclosure, matched four-head comparison and competing-only control address several of my concerns. In particular, the finding that competing-only control outperforms Random and matches or exceeds supporting-only control connects the residual-competition analysis to the control method, which was my Q3. Remaining points:

  1. Same-pair DPO (most important). To support "DPO-competitive with few training pairs," a DPO run on the same pairs would help. Even one setting (e.g., LoRA-DPO on the 1,024 IMDb pairs) suffices. Alternatively, scoping the claim to "approaches DPO trained on larger data," without a data-efficiency claim, would also address my concern.
  2. Matched-bank results. In the four-head table, ITI exceeds both RCM banks on ConFiQA/Llama, and on TL;DR the differences are about 1–2 points. Could you add paired CIs? The Combined row is compared with eight-head ITI/Random, which still differ in bank organization, so I read the four-head rows as the matched evidence.
  3. Non-target capabilities. The results in your response to another reviewer show large drops for CAST on Llama (GSM8K 82.42 → 1.56, TriviaQA 63.67 → 5.08), while DPO stays close to SFT. Since this bears on "DPO-competitive," I suggest stating it alongside the main results, not only in Limitations.
  4. Qwen claim. In your semantic assessment CAST-all is below DPO on Qwen (78 vs. 86), so "Qwen favors CAST" should be revised. Please report PC and semantic results rather than only removing EM.

The Q3 result strengthens the paper. If item 1 is addressed (by experiment or by revising the claim), I would be glad to raise my score.

Reviewer_WgJBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Hidden-State Alignment (DHSA), a parameter-efficient alignment framework that modifies inference-time hidden states instead of updating base model weights.

The paper first introduces Residual Competition Maps (RCMs), which measure signed causal effects of Transformer components through targeted interventions, revealing competing computational pathways in the residual stream that support or oppose a specified preference.

Based on RCM-guided localization, the authors propose Causal Activation State Transition (CAST), which trains lightweight controllers on a small number of preference-relevant hidden interfaces while keeping the base model frozen.

Experiments across factual preference following, sentiment preference, and summarization demonstrate that CAST achieves DPO-competitive performance with only 256–16,384 additional parameters and can complement DPO-trained models.

Critical strengths

The paper introduces a novel alignment perspective by shifting from parameter updating to hidden-state control. This provides an interesting direction for understanding how preferences emerge and are expressed internally in LLMs.

RCM provides a bridge between mechanistic interpretability and practical model control. Compared with heuristic activation steering methods, RCM offers a causal mechanism for identifying intervention targets.

Critical weaknesses

The generalization ability of RCM is insufficiently demonstrated. It remains unclear whether identified components: remain stable across prompts, generalize across architectures, transfer to unseen preference domains.

The advantage of CAST is mainly demonstrated in terms of trainable parameters rather than total computational cost. Since RCM requires extensive component-level interventions, the overall pipeline may be more expensive than standard PEFT approaches. The paper should report: RCM discovery cost, CAST training cost, inference overhead.

Direct hidden-state manipulation may improve a target preference but potentially damage unrelated capabilities. Since alignment methods often involve trade-offs between objectives, evaluating only the target preference metric may not be sufficient.

Questions for authors

Q1: Are RCM-identified preference-supporting components stable across tasks and prompt distributions? Specifically, do the same heads preserve consistent causal directions?

Q2: Does CAST affect non-target capabilities, such as mathematical reasoning, factual accuracy, or safety refusal behavior?

Q3: Please report the complete computational cost of the pipeline, including RCM discovery and CAST training, rather than only controller parameter counts.

5 comments
AuthorsOpenReview ↗

Clarifying Component Stability and Transfer

Thank you for recognizing the alignment perspective and the measurement–control connection, and for asking which component properties remain stable.

Q1/W1 — Stability and transfer

1. Effects can generalize across samples, depending on the task.

ConFiQA heads have more consistent directions; most IMDb effects are small, while TL;DR shows more sign variation and cancellation across inputs. Supporting effects predominate above threshold, but all 18 supporting heads also show above-threshold reverse effects on some inputs. The table below reports frequencies over 300 discovery inputs per setting:

SettingHeadsPositive (>0) %Above +tau %Below -tau %Within ±tau %Positive if magnitude >tau %
ConFiQA/Llama473.242.06.251.887.2
ConFiQA/Qwen480.330.21.368.595.8
IMDb/GPT-2-large461.214.47.678.065.5
TL;DR/GPT-J254.314.28.877.061.6
TL;DR/LION455.636.626.836.657.7

Thresholds: ConFiQA 0.5 token-logit margin; IMDb 0.01 sentiment score; TL;DR 0.02 mean token log probability. The first four percentages use all inputs; the last uses effects with magnitude >tau. Mean CIs are positive for 17/18 heads (the exception is one GPT-J head). IMDb's low-effect bank has three negative-mean and one positive-mean head, all mean CIs crossing zero.

2. The same heads can be reused on an unseen dataset in the same domain.

On NQ knowledge-conflict data, all remeasured original ConFiQA heads retain their mean sign: 8/8 Qwen and 3/8 Llama heads; the other five Llama heads are outside the scan. Training controllers at these original sites gives performance similar to rescanned sites, as shown below. Both conditions share NQ pairs, four supporting/four competing heads, the official prompt and calibration. Official context-answer recall on 512 shared test questions:

ModelOriginal heads %Rescanned heads %Difference,points [paired 95% CI]
Llama-3-8B91.4189.84+1.56 [-0.98,4.10]
Qwen2-7B84.1883.79+0.39 [-1.76,2.54]

Rescanning uses 300 questions. Recall follows official normalized answer-in-response matching; paired 95% CIs for the differences span zero.

3. Mean directions persist across prompts, but selected heads differ.

The table below separates direction consistency from selection overlap. On the same 300 inputs, only wording changes: Fields uses Context/Question/Answer labels; Instruction adds "Answer the question using the supplied context."

Model / promptSame mean sign / remeasuredSupport overlap /4Compete overlap /4Total overlap /8
Llama / Fields7/7325
Llama / Instruction3/3202
Qwen / Fields8/8437
Qwen / Instruction8/8347

Remeasured heads retain their mean signs, and retained heads keep their roles. Llama's rankings and selections are unstable, which may help explain why both CAST losses to DPO are Llama-based. Its original competing heads are outside Instruction's selected parent layers.

4. Our results do not support direct cross-model mapping.

RCM/CAST works across Llama, Qwen, GPT-2 and GPT-J. Mapping corresponding relative-depth ranges within each dataset gives no consistent component correspondence: same-role final selections have no overlap on ConFiQA and one competing-layer correspondence on TL;DR. The table below shows partial overlap of scanned depth:

MappingOverlapping source layersOverlap / source scanned depth
ConFiQA: Llama → Qwen3/823.21%
ConFiQA: Qwen → Llama3/820.31%
TL;DR: GPT-J → LION3/828.13%
TL;DR: LION → GPT-J4/832.14%

Llama-specific properties may contribute; other mappings remain possible.

5. Transferred controllers can improve DPO models.

SFT-trained controller parameters transfer unchanged to DPO, with strength recalibration (Table 2). This tests controller reuse; NQ above tests site reuse with newly trained controllers. The table below reports DPO+CAST-T minus DPO, points [paired 95% CI]:

SettingMetricDifference
ConFiQA/LlamaPC0.00 [-0.88,0.88]
ConFiQA/QwenPC+10.01 [8.35,11.62]
TL;DR/GPT-JWin+6.45 [3.52,9.47]
TL;DR/LIONWin+0.88 [-1.17,2.93]

ConFiQA uses 2048 shared inputs; TL;DR uses 512 posts and human references in both orders. We will add these analyses to the revision and appendix, alongside the component roles and bilateral-effect evidence in Appendix B.2.

AuthorsOpenReview ↗

Clarifying Effects on Non-Target Capabilities and Computational Cost

Q2/W3 — Non-target capabilities

Thank you for requesting this evaluation. We tested the frozen ConFiQA controllers on unrelated tasks and will add the results and their interpretation to the appendix and Limitations.

1. ConFiQA control can affect unrelated tasks when left enabled. As shown in the table below, DPO stays close to SFT, whereas BiPO, LoReFT and CAST lose performance on at least one model. CAST's losses are larger on Llama. All methods share 256 GSM8K and 256 closed-book TriviaQA questions per model, retaining their ConFiQA settings.

MethodLlama GSM8KLlama TriviaQA EMQwen GSM8KQwen TriviaQA EM
SFT82.4263.6776.5645.31
DPO81.2565.2371.8844.92
BiPO17.1933.2074.6148.83
LoReFT80.0852.3445.7037.89
CAST1.565.0848.8326.56

Scores (%) use official GSM8K final-answer extraction and TriviaQA alias EM. Llama CAST's GSM8K format-failure rate is 91.80%.

2. The controller applies the target-preference intervention to every input. It is trained only for that preference and applies the same learned intervention rule even to unrelated tasks. This offers a possible explanation for the observed losses. As a separate module, it can be switched off for out-of-domain requests to restore the unmodified base model. We will clarify this task-specific use; retention objectives and domain-gated controllers are future directions.

Q3/W2 — Computational cost

Thank you for asking for discovery, training and inference costs rather than parameter counts alone.

1. RCM discovery is reusable, keeping preparation costs comparable to the baselines. Beyond very small training sets, discovery can cost about as much as one controller training run. RCM scores components without training a controller for each, and selected sites can be reused within a model and preference. The table below separates discovery and CAST training and compares preparation budgets (single-H100 minutes):

SettingRCM discoveryCAST trainingCAST budgetBiPOLoReFT
ConFiQA/Llama5025251201
ConFiQA/Qwen7828044157
IMDb384423940
TL;DR/GPT-J6257159160196
TL;DR/LION12451266270295

TL;DR budgets include 40/91 minutes of audit generation.

2. Online, CAST adds about 4–13% latency in these tests. The table below compares each method with its SFT reference. All use a dedicated H100, batch 8 and 512 input/128 output tokens; timings use 100 batches after 10 warmups. Native controller/runtime settings remain fixed, and DPO adapters are unmerged.

ModelSFT ms/batchCAST overhead % [95% CI]BiPO %LoReFT %DPO %
Llama21244.25 [4.17,4.32]7.080.2472.83
Qwen18644.29 [4.27,4.31]2.390.7471.34
GPT-J204813.44 [13.31,13.58]40.770.8551.10

Intervals are paired-batch 95% CIs. CAST produces 462/527/441 output tokens/s on Llama/Qwen/GPT-J; peak allocated memory is 15.94/15.02/15.49 GiB, matching SFT to two decimals. We will include the preparation-cost breakdown and full latency, throughput and memory results in the appendix.

Dear Reviewer WgJB,

We appreciate your time and constructive comments. We hope our responses have clarified the issues you raised, and would welcome your thoughts on whether any concerns remain. We would be glad to provide further clarification during the discussion.

Reviewer_iCvyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces Residual Competition Maps (RCMs), which use component interventions to identify effects that support or oppose a specified preference. It then trains small additive or low-rank controllers at selected locations while freezing the language model. Experiments check the context-faithful answering, the sentiment steering, and the summarization.

Critical strengths

The connection between preference-specific intervention measurements and localized control is useful, and the controllers require very few trainable parameters.

The evaluation covers several model families and preference domains, includes random-site and established steering baselines, and reports paired uncertainty alongside performance tradeoffs.

Critical weaknesses

The incremental contribution needs sharper justification. Frozen-model representation adaptation is established in ReFT and LoFiT, while GCM already combines causal head localization, including patching and zero ablation with representation steering. The potentially distinctive contribution is therefore the signed RCM analysis and selection procedure. Direct comparisons with the closest localization methods are needed to establish its value.

Some localization comparisons do not isolate the selection method. On ConFiQA and IMDb, RCM uses two separately trained four-head banks, whereas ITI and random selectors use one jointly trained eight-head bank (Appendix C.3). These differences can affect performance independently of which heads are selected. This limits the gains specifically resulted from RCM.

The practical advantage is sensitive to metrics and comparison settings. In Table 2, CAST-all achieves 81.93% target-answer occurrence but 0% exact match on ConFiQA/Llama. Since occurrences can count a negated mention and exact match is sensitive to formatting, a semantic evaluation is needed to interpret this discrepancy. BiPO and LoReFT also remain substantially stronger on some ConFiQA metrics. Moreover, the released DPO checkpoints use unmatched training data and compute, limiting conclusions about comparative efficiency.

Questions for authors

What does the signed RCM construction add beyond GCM’s causal localization and LoFiT’s learned head selection?

Do the localization gains persist when controller organization, training objective, active-head budget, and calibration are held fixed?

Can the authors evaluate semantic context-faithfulness and compare performance under matched training data and total computational budgets?

5 comments
AuthorsOpenReview ↗

Clarifying Contributions and Localization Gains

We thank the reviewer for recognizing preference-specific measurement and local control, parameter efficiency, and the breadth of the evaluation.

Q1/W1 — Clarifying the Advances Beyond GCM and LoFiT

1. RCM measures support for and opposition to a preference. GCM ranks heads by post-intervention log-likelihood contrasts of fixed response pairs. RCM measures component effects as changes in task-defined preference scores, including rewards on post-intervention generations. LoFiT learns head importance; RCM measures native causal effects before controller training and tests control utility.

2. Native components can support and oppose the same preference. RCM reveals this coexistence within a model, input-dependent component roles, and their reorganization after DPO (Sec. 3; Fig. 2).

3. Competing components can be effective control sites. RCM retains supporting and competing candidate pools. Fixed-alpha ConFiQA controls show competing-only control is competitive with supporting-only control in target-answer occurrence on Llama and Qwen. In the GPT-J TL;DR audit, head coordinates were replaced by audit IDs; six of eight retained heads came from the competing pool (Appendix C.3). See the attribution table in Q2.

4. CAST advances hidden-state adaptation to DPO-competitive preference alignment across three domains (Table 2; Fig. 3). Similar interventions already exist, but BiPO and LoReFT do not reach DPO-competitive performance across all three domains in our evaluation: they are strong on ConFiQA but fall below DPO on IMDb and TL;DR. Establishing this cross-domain alignment level is itself part of our contribution.

5. GCM's native evaluation is not directly comparable. It focuses on safety and concept behaviors rather than covering multiple heterogeneous preference domains. Direct comparison would require adapting its tasks and evaluation.

Thank you for requesting clearer distinctions from prior work. We will incorporate them into Related Work, the contribution statements and the experimental conclusions.

Q2/W2 — Clarifying Comparison Fairness and Internal Attribution

1. Using separate banks doesn't inherently make the comparison unfair. We fix training examples, total active heads, actuator, objective, calibration and evaluation. RCM trains signed banks separately without extra examples or heads; ITI and Random do not support separating banks by these signed roles. TL;DR also matches candidate, single-head audit and final eight-head budgets (Table 1; Fig. 3). These controls already support attributing the gains to RCM's selection and signed grouping.

2. RCM gains persist with matched four-head banks. Supporting and competing sites provide comparable control overall; combining them gives greater gains and clearer baseline separation in the paper's matched eight-head comparisons. The table below reports the single-bank results and their combinations.

Four-head comparisons match data, actuator, objective, intervention timing and calibration; TL;DR also matches the 8-to-4 audit. Combined uses eight heads.

Selector / compositionConFiQA Llama PCConFiQA Qwen PCTL;DR GPT-J WinTL;DR LION Win
Supporting60.8966.1152.8373.24
Competing76.8669.2454.8873.63
ITI79.9361.8253.5271.00
Random 158.5439.4551.3772.17
Random 258.4047.2752.4472.46
Random 361.5752.2952.9371.48
Combined (8 heads)81.9382.9657.1379.10

(a) ITI clearly outperforms the Random mean only on ConFiQA. It slightly outperforms competing RCM on ConFiQA/Llama, but both RCM banks outperform ITI on Qwen. On IMDb, the controller at sites with non-weak RCM effects, used alone, achieves a better reward–KL trade-off than ITI. On TL;DR, RCM scores higher overall, with clearer separation from ITI after combining the banks.

(b) Competing sites provide control comparable to supporting sites overall. Competing banks are stronger on ConFiQA, the two are similar on TL;DR, and supporting banks are stronger on IMDb.

(c) RCM banks generally outperform the Random mean in PC and TL;DR Win. On IMDb, the reward–KL curve for sites with non-weak RCM effects is better than ITI and Random 1/3, and crosses Random 2.

(d) Joint control gives clearer separation at matched head budgets. On ConFiQA and TL;DR, Combined shows clearer separation from eight-head ITI/Random than single banks do from four-head ITI/Random.

IMDb clarification. Even with extremely weak native negative effects, the corresponding bank's peak sentiment reward exceeds ITI, Random 1 and Random 3, but remains below Random 2 and the supporting bank with strong positive native effects.

TL;DR Win is order-balanced against human references; IMDb compares KL at matched reward. Thank you for examining location gains and joint residual control of supporting and competing components. We will add the controls, results, analyses and reward–KL curves to Experiments and Appendix C.3.

AuthorsOpenReview ↗

Clarifying Metrics, Model-Family Results, and Comparison Fairness

Q3/W3 — Clarifying ConFiQA Metrics and Data/Compute Comparisons

1. Semantic analysis supports retaining PC and explains why EM misses faithful answers. Thank you for suggesting this assessment beyond ConFiQA's official rule-based evaluation. Full semantic assessment is costly, so we sampled 50 shared questions per model across six methods (600 frozen responses). Method-blinded, AI-assisted assessment judges final answers against context and records explanation issues separately.

As shown in the table below, PC has high F1 against these semantic labels. F1 is averaged equally over the 12 model/method settings:

MetricMean F1 (%) [95% CI]
PC85.6 [80.3,90.0]
EM53.6 [47.2,59.2]

PC exceeds EM by 32.0 points; EM's F1 is relatively close to, though above, its random baseline. Exact matching misses faithful answers, so EM=0 does not mean all answers are wrong. We will retain PC, remove EM from the revised evaluation, and include the semantic assessment in the appendix.

For this small sample, we use 95% CIs and paired comparisons to interpret method differences. Final-answer context faithfulness is summarized below (% [95% CI]):

MethodLlama-3-8BQwen2-7B
SFT36 [24.1,49.9]34 [22.4,47.8]
DPO94 [83.8,97.9]86 [73.8,93.0]
BiPO92 [81.2,96.8]100 [92.9,100]
LoReFT78 [64.8,87.2]96 [86.5,98.9]
CAST-all52 [38.5,65.2]78 [64.8,87.2]
CAST-prefill90 [78.6,95.7]72 [58.3,82.5]

In this semantic sample, Llama CAST-all trails DPO; the other CAST–DPO differences are not clearly separated. BiPO and LoReFT remain stronger overall on ConFiQA.

2. The full evaluation supports DPO competitiveness across three domains. The original manuscript reports wins and losses against DPO. BiPO and LoReFT's stronger ConFiQA results do not extend across domains: they fall below the same DPO references on IMDb and TL;DR (Table 2; Fig. 3).

ConFiQA rescans preserve remeasured heads' mean signs but show unstable Llama rankings and selections, which may explain why both CAST losses to DPO are Llama-based. The table below compares two prompt variants on the same 300 inputs: Context/Question/Answer labels (F) and a context-use instruction (I):

ModelOriginal heads retained /8 (F; I)Same mean sign / remeasured heads (F; I)
Llama-3-8B5/8; 2/87/7; 3/3
Qwen2-7B7/8; 7/88/8; 8/8

We will add these results to the revision.

3. CAST, BiPO and LoReFT already use matched training data and evaluation. Our blanket 'unmatched' wording obscured these controls; thank you for flagging it. Each retains its native objective, with comparable compute budgets.

The table below reports preparation budgets from candidates to trained controllers (H100-minutes):

SettingCASTBiPOLoReFT
ConFiQA/Llama5251201
ConFiQA/Qwen8044157
IMDb423940
TL;DR GPT-J159160196
TL;DR LION266270295

4. Public DPO baselines provide established performance references. Test inputs, prompts and scoring match; we align official data sources where possible. CAST uses very few ConFiQA pairs with limited preset tuning.

For IMDb and GPT-J TL;DR, the published DPO data budgets are respectively 9.8× and 2.2× larger than ours. The table below states the data relationship for each setting:

SettingOur pairsRelationship to released DPO
ConFiQA240Same source and official prompt; checkpoint-specific training count/split undisclosed
IMDb, GPT-2-large1024Subset of the public 10000-pair pool; DPO pool is 9.8× larger
TL;DR, GPT-J8192Same OpenAI preference source; published DPO training split is 18000, or 2.2× larger
TL;DR, LION8192Shared SFT start and evaluation; DPO uses a multi-source mixture

Weight tuning can need more data than external control to reach representative alignment performance. Forcing the same small or large training set on both does not establish each method's representative performance and cost; matching CAST's scale would specifically test low-data DPO. We therefore use established public DPO checkpoints. Undisclosed pairs/splits prevent exact matching; unpublished recipes make exact retraining costly.

Together with the full evaluation (Table 2; Fig. 3), these controls support our contribution: DPO-competitive alignment across domains with few training pairs.

We will clarify the matched settings and DPO reference choice, and add these tables and data-construction rules to the appendix.

Dear Reviewer iCvy,

Thank you again for your thoughtful feedback. We would appreciate hearing whether our rebuttal has addressed your main concerns. If any points remain unclear or require further evidence, we would be happy to discuss them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.