- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that when a preferred behavior is already reachable, alignment is partly a problem of expression: target-supporting computation may be outweighed by other components writing to the same residual stream ("residual competition").
RCM analysis.
- It introduces Residual Competition Maps (RCMs). For each input, an RCM measures the signed change in a task-specific preference score when a single component's state is zeroed (RCM-Zero) or replaced by a prototype state (RCM-Patch).
- On ConFiQA with Llama-3-8B, both-sign effects occur on most inputs (283/300).
- DPO reorganizes these effects but does not remove all opposing ones.
DHSA and CAST.
- It then proposes Direct Hidden-State Alignment (DHSA), the principle of adapting inference-time hidden states of a frozen model.
- DHSA is instantiated as CAST: constant-vector or low-rank controllers at RCM-selected attention heads (Eq. 3), trained with a reference-relative DPO-form loss.
Validation.
- With 256–16,384 controller parameters, CAST is reported to reach "DPO-competitive operating points" on ConFiQA, IMDb, and TL;DR across several model families.
- It can also be added on top of released DPO checkpoints (Table 2, Fig. 3).
Critical strengths
Critical Strengths
-
Unusually careful and candid reporting.
- Selection and test splits are separated and documented (C.4).
- Paired bootstrap intervals are given for the main contrasts (Table 10), and IMDb reward is always paired with KL.
- Caveats are stated explicitly: DPO references are not data- or compute-matched (C.3), and RCM intervals are not multiplicity-corrected (A.2).
- Negative results are kept, e.g., ITI beating RCM-Zero on ConFiQA Llama and CAST-all collapsing on Llama EM.
This makes the paper's evidence easy to audit, and it is why I consider the empirical portion worth engaging with despite the concerns below.
Critical weaknesses
Critical Weaknesses
-
The conceptual contributions largely relabel existing tools, and the benefit of RCM-guided localization is not isolated. RCM-Zero and RCM-Patch are single-component zero-ablation and mean activation patching (B.1). Components that oppose the final prediction are already documented, e.g., Negative Name Mover heads (Wang et al., 2022) and copy-suppression heads (McDougall et al., 2023). DHSA, "adapt hidden states of a frozen model at inference," describes existing methods such as ReFT, LoFiT, and BiPO. Empirically, RCM does not consistently beat ITI (ConFiQA Llama: 85.64 vs. 78.61). Its comparison is also confounded by different bank designs, which the authors acknowledge (C.3).
-
The "DPO-competitive" claim is overstated. Table 2 contradicts it in several places. On ConFiQA Llama-3-8B, CAST-all has EM = 0.00, and on TL;DR LION it is significantly below DPO. On ConFiQA, CAST is below both BiPO and LoReFT. All DPO references are released checkpoints trained with different data and compute. DPO trained on the same pairs as CAST, the natural control for the paper's central thesis, is absent.
Questions for authors
All questions are clarifications that could change my assessment.
- How does CAST compare with DPO or LoRA-DPO trained on the same pairs and evaluated identically?
- With bank design equalized across selectors, does RCM still outperform ITI and random selection in Table 1?
- Does controlling only target-competing sites help beyond supporting-only or random sites? Without this, the "residual competition" analysis is not connected to the control method.
7 comments
Clarifying Fair Comparisons and Control-Site Attribution
Q1 — Clarifying the DPO reference and data/compute controls
1. The original setup supports a fair performance comparison. Our blanket 'unmatched' wording obscured the controls already in place; thank you for flagging this. Test inputs, prompts and scoring match. CAST uses few ConFiQA pairs.
For IMDb and GPT-J TL;DR, CAST uses far fewer pairs than the published DPO budgets. The table below lists the data relationship for each setting:
| Setting | CAST pairs | Relationship to DPO |
|---|---|---|
| ConFiQA | 240 | Same source/official prompt; checkpoint count/split undisclosed |
| IMDb, GPT-2-large | 1024 | Subset of public 10000-pair pool; 9.8× larger |
| TL;DR, GPT-J | 8192 | Same OpenAI source; DPO split has 18000 pairs, 2.2× larger |
| TL;DR, LION | 8192 | Shared SFT/evaluation; DPO uses a multi-source mixture |
2. Our comparison supports DPO-competitive alignment across domains with few pairs (W2, first reply). Weight tuning can need more data than external control to reach representative alignment performance. Forcing one small or large set on both does not establish their representative performance or cost. We use established public DPO checkpoints; unreleased pairs/splits and recipes make exact retraining costly.
3. CAST, BiPO and LoReFT share training data and evaluation, with comparable preparation budgets. Each retains its native objective and preset tuning. The table below covers candidates to trained controllers (H100-minutes):
| Setting | CAST | BiPO | LoReFT |
|---|---|---|---|
| ConFiQA/Llama | 52 | 51 | 201 |
| ConFiQA/Qwen | 80 | 44 | 157 |
| IMDb | 42 | 39 | 40 |
| TL;DR GPT-J | 159 | 160 | 196 |
| TL;DR LION | 266 | 270 | 295 |
We will clarify these matches in the setup and add the tables and data-construction rules to the appendix.
Q2 — Clarifying localization gains with bank design fixed
Thank you for suggesting a matched-bank comparison.
1. The original controls support attributing gains to RCM's selection and signed control. Separate banks are part of RCM, not an extraneous complication. Other conditions match: data, total heads, actuator, objective, calibration and evaluation. TL;DR also matches candidate, audit and final-head budgets. Original Table 1 shows RCM leading in three settings.
2. The new matched-bank tests retain location gains. At the same four-head budget and bank organization, RCM banks outperform ITI and Random in most comparisons. Results are in the shared table after Q3.
(a) ITI clearly exceeds the Random mean only on ConFiQA. It slightly exceeds competing RCM on ConFiQA/Llama, while both RCM banks exceed ITI on Qwen. On IMDb, the controller at sites with non-weak RCM effects, used alone, achieves a better reward–KL trade-off than ITI. On TL;DR, competing RCM scores higher than ITI on both models, and supporting RCM also scores higher on LION. Combining banks makes the separation clearer.
(b) RCM banks generally outperform Random. Both banks score above the Random mean on ConFiQA and TL;DR. On IMDb, the reward–KL curve for sites with non-weak RCM effects is better than Random 1/3 and crosses Random 2. KL comparisons use matched reward.
Q3 — Clarifying the control value of competing sites
Thank you for suggesting a direct test of competing sites.
1. Supporting-only and competing-only control perform comparably in the overall evaluation. Competing control is stronger on ConFiQA, the two are similar on TL;DR, and supporting control is stronger on IMDb.
2. Competing sites also provide gains over random sites. Competing RCM exceeds every Random seed on both ConFiQA models and the Random mean on both TL;DR models, as shown in the shared table below.
3. Joint control gives clearer separation at matched head budgets. On ConFiQA and TL;DR, Combined shows clearer separation from eight-head ITI/Random than single banks do from four-head ITI/Random.
Full ConFiQA and TL;DR results for Q2/Q3 are summarized below. Four-head conditions match actuator, data/objective, intervention timing and calibration; supporting/competing banks reuse Performance weights. TL;DR uses the same 8-to-4 audit for each pool. Combined uses eight heads.
| Selector / composition | ConFiQA Llama PC | ConFiQA Qwen PC | TL;DR GPT-J Win | TL;DR LION Win |
|---|---|---|---|---|
| Supporting | 60.89 | 66.11 | 52.83 | 73.24 |
| Competing | 76.86 | 69.24 | 54.88 | 73.63 |
| ITI | 79.93 | 61.82 | 53.52 | 71.00 |
| Random 1 | 58.54 | 39.45 | 51.37 | 72.17 |
| Random 2 | 58.40 | 47.27 | 52.44 | 72.46 |
| Random 3 | 61.57 | 52.29 | 52.93 | 71.48 |
| Combined (8 heads) | 81.93 | 82.96 | 57.13 | 79.10 |
IMDb clarification. Even with extremely weak native negative effects, the corresponding bank's peak sentiment reward exceeds ITI, Random 1 and Random 3, but remains below Random 2 and the supporting bank with strong positive native effects.
We will release the IMDb reward–KL curves with the revision and add the analyses of location gains and joint residual control of supporting and competing components to Experiments and the appendix. TL;DR Win averages both orders against human references.
Thank you for the detailed responses. The compute-budget table, data-source disclosure, matched four-head comparison and competing-only control address several of my concerns. In particular, the finding that competing-only control outperforms Random and matches or exceeds supporting-only control connects the residual-competition analysis to the control method, which was my Q3. Remaining points:
- Same-pair DPO (most important). To support "DPO-competitive with few training pairs," a DPO run on the same pairs would help. Even one setting (e.g., LoRA-DPO on the 1,024 IMDb pairs) suffices. Alternatively, scoping the claim to "approaches DPO trained on larger data," without a data-efficiency claim, would also address my concern.
- Matched-bank results. In the four-head table, ITI exceeds both RCM banks on ConFiQA/Llama, and on TL;DR the differences are about 1–2 points. Could you add paired CIs? The Combined row is compared with eight-head ITI/Random, which still differ in bank organization, so I read the four-head rows as the matched evidence.
- Non-target capabilities. The results in your response to another reviewer show large drops for CAST on Llama (GSM8K 82.42 → 1.56, TriviaQA 63.67 → 5.08), while DPO stays close to SFT. Since this bears on "DPO-competitive," I suggest stating it alongside the main results, not only in Limitations.
- Qwen claim. In your semantic assessment CAST-all is below DPO on Qwen (78 vs. 86), so "Qwen favors CAST" should be revised. Please report PC and semantic results rather than only removing EM.
The Q3 result strengthens the paper. If item 1 is addressed (by experiment or by revising the claim), I would be glad to raise my score.
Clarifying the Scientific Advance and Cross-Domain Performance
Thank you for recognizing transparent reporting, held-out evaluation and paired statistics.
W1 — Clarifying what advances beyond the existing tools
Thank you for asking us to clarify the advances over prior work. We will add these clarifications to Related Work and contributions.
1. Single-component zeroing and mean activation patching are measurement operators. RCM maps target–non-target preference advantage to the signed causal effects of native residual writes; CAST optimizes the same preference.
2. Opposing-head analysis extends from IOI/copy suppression to preference alignment. RCM reveals support and opposition toward specified preferences across domains, input-dependent roles and their reorganization after DPO (Fig. 2). We test competing sites as control locations (Q3 in our second reply).
3. CAST advances hidden-state adaptation to DPO-competitive preference alignment. ReFT/LoFiT establish adaptation; LoFiT learns head importance. RCM measures native effects before training; CAST reaches DPO-competitive preference alignment across three domains (W2). Similar interventions already exist, but BiPO and LoReFT do not reach DPO-competitive performance across all three domains in our evaluation: they are strong on ConFiQA but fall below DPO on IMDb and TL;DR. Establishing this cross-domain alignment level is itself part of our contribution.
4. Signed banks are part of RCM's control mechanism, not an extraneous complication. RCM supplies supporting/competing roles that ITI and Random cannot supply. Separate banks use these roles without extra examples or heads. Training data, total heads, actuator, objective, calibration and evaluation are otherwise matched. We do not claim RCM is the best selector in every setting. ITI's advantage on a particular dataset and model does not negate RCM's strong localization performance across preference domains or its contribution. See Q2 in our second reply for matched-bank attribution.
W2 — Clarifying cross-domain DPO competitiveness
Thank you for asking us to clarify cross-domain performance and the interpretation of EM.
1. CAST-all has two wins, two losses and one near tie across five settings. IMDb reaches DPO reward at lower KL; TL;DR is near-tied on GPT-J and lower on LION. Qwen and IMDb favor CAST; both losses are Llama-based (Tables 2/10; Fig. 3), supporting cross-domain competitiveness.
2. Stronger ConFiQA baselines do not imply stronger performance across domains. BiPO and LoReFT are strong on ConFiQA but fall below the same DPO references on IMDb and TL;DR. CAST competes with DPO across all three domains.
3. Semantic analysis explains EM=0 despite faithful answers. PC has high F1 for context-faithful final answers; EM misses faithful answers and has F1 relatively close to, though above, its random baseline. We will retain PC, remove EM from the revised evaluation, and include this semantic assessment in the appendix.
Semantic assessment adds to ConFiQA's official rule-based evaluation. Full assessment is costly, so we sampled 50 shared questions per model across six methods (600 responses). Method-blinded, AI-assisted assessment judges final answers against context and records explanation issues separately. The table below averages F1 over the 12 settings:
PC exceeds EM by 32.0 points. EM's F1 is relatively close to its random baseline (46.1%, preserving each setting's positive-prediction rate).
For this small sample, we use 95% CIs and paired comparisons to interpret method differences. Final-answer context faithfulness is summarized below (% [95% CI]):
In this semantic sample, Llama CAST-all trails DPO; the other CAST–DPO differences are not clearly separated. BiPO and LoReFT remain stronger overall on ConFiQA. The full-evaluation comparison is summarized in point 1 above.
4. RCM reveals unstable native effects in Llama that may explain its weaker control results. ConFiQA prompt rescans show unstable native-effect rankings and selected heads on Llama, while remeasured heads retain their mean signs. RCM detects this model-specific instability, which may explain why both CAST losses to DPO are Llama-based.
The table below compares RCM rescans on the same 300 ConFiQA inputs with Context/Question/Answer labels (F) or a context-use instruction (I):
We will clarify the cross-domain performance conclusion in Experiments and add the semantic and prompt-stability analyses to the appendix.