- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether a language model's verdict causally uses its representation of rule polarity, i.e., "whether a stated rule permits or prohibits an action, with the scenario held fixed." The authors deliberately study only this qualitative half of rule application and leave threshold and condition evaluation out of scope.
The main task is based on the anti-structuring rule of the US Bank Secrecy Act (BSA): the same deposit-splitting action is paired with either a permitting or a prohibiting rule, and the model outputs SATISFIED or EVADED as its next token. Experiments cover four instruction-tuned models (Llama, Qwen, Mistral, Gemma). The authors extract a permit-minus-prohibit direction by difference-of-means at a late decision layer and test it with ablation, projection-replacement, and head-level activation patching.
The paper makes four main claims:
- Many late-layer components write , while "a single direction provides a compact causal readout": replacing its coordinate recovers 73–93% of the opposite decision.
- Injecting a false premise and the conclusion it implies into a genuine violation moves the internal state toward the coordinate occupied by genuine permission.
- Using matched control tasks and two reference directions, the authors argue that the effect cannot be explained by a generic yes/no direction or by the answer tokens alone.
- The causal signature transfers to FAA drone-altitude rules: strongly in Llama and Gemma, weakly in Qwen, and not testable in Mistral.
Critical strengths
- Clear problem scoping. The paper restricts itself to the qualitative half of a rule: its polarity, i.e., whether the stated rule permits or prohibits an action. It explicitly excludes threshold and condition evaluation. The scope of the claims is stated clearly, and negative results are reported openly: Mistral fails the FAA eligibility gate, and Qwen fails the head-alignment reproduction gate.
- Well-motivated control design. The paper does not stop at random-direction baselines. It constructs the reference direction predicted by each deflationary account: a generic decision direction and a vocabulary direction . It also uses a task-content × answer-vocabulary crossed design (Tasks A–E). The results do not support the authors' conclusion (see Weakness 1), but the framework itself is the right one.
Critical weaknesses
1. The causal claims are questionable
"Causal" in this paper means do-interventions on activations: ablation for necessity and projection-replacement for sufficiency. This is standard practice in mechanistic interpretability. However, these experiments establish only a narrow chain:
explicitly stated exemption/prohibition clause → a late-layer direction → SATISFIED/EVADED logit difference
The central claim is that the model's verdict causally uses its representation of rule polarity. Supporting it would require three things the paper does not provide.
-
No high-level causal model and no alignment test. The paper never states an explicit high-level causal model, such as "rule polarity × condition satisfied → verdict". Nor does it test whether aligns with the high-level polarity variable using interchange interventions. The current experiments show that affects the output, not that it encodes rule polarity. Moreover, on BSA the model does not actually apply the rule:
- Every violating case is classified EVADED, and all errors are compliant cases flagged as violations (A.5).
- Changing the threshold flips the correct answer but barely moves the verdict: the logit gap changes by only 0.02–0.77, and both prompts remain on the EVADED side (A.10).
The permit/prohibit distinction therefore most plausibly comes from the exemption clause written explicitly in the rule text.
-
is not shown to differ from an output-token direction. is chosen where patching the full residual stream already recovers >90% of the verdict. At such a layer, any direction with a nonzero component along the answer-token difference will move the logit. The necessity and sufficiency tests therefore cannot distinguish a rule representation from an output-token direction.
The specificity analysis (§6) bases its conclusion on A vs. C/D, which varies task content and answer vocabulary at the same time. The two cells that separate them both point to the vocabulary account (Figs. 5 and 7):
- Task E (truth task with SATISFIED/EVADED answers): 's sensitivity is comparable to that on A (Fig. 7; Llama 0.66 vs. 0.78, Mistral 0.28 vs. 0.23). Its projection-patch recovery on E is at least as high as on A in Llama and Mistral (Fig. 5: 0.90 vs. 0.84; 1.09 vs. 1.02) and remains substantial in Gemma and Qwen (0.76; 0.44).
- Task B (the same rule task with YES/NO answers): 's sensitivity is near zero in all four models (≤ 0.09). The authors dismiss B as non-diagnostic. Yet in Llama, B is clearly steerable ( reaches 0.68), while reaches only 0.09. B is also absent from the stronger sufficiency test (Fig. 5).
This is the pattern the output-token hypothesis predicts, and it mirrors . The paper should report:
- the cosine similarity of with and with ;
- whether the component of orthogonal to the output-token direction still steers A and B.
-
The both-wrong experiment is observational, not interventional. §5 only reads 's projection after injecting a false argument. Contribution 2 nonetheless calls this "a causal lever". Supporting that would require, at minimum, patching the coordinate of the both-wrong prompt back to its violation value and checking whether the output reverts to EVADED. In addition:
- The injected text contains the conclusion itself ("…so it need not be reported"). The authors removed exactly this conclusion-copying confound from the extraction prompts ("Prompt provenance control", L270–272), and the experiment reintroduces it.
- The premise-only condition is not reported. This is the condition that could separate inference from copying, yet it appears only as "partial movement", with no numbers.
- Convergence at is close to tautological. Any input change that flips the output must push a verdict-correlated projection toward the permit end.
2. Most observations are either trivial or previously reported; novelty is limited
-
"A late-layer linear direction causally controls the decision" is already established. Arditi et al. (arXiv:2406.11717), Marks & Tegmark (arXiv:2310.06824) and Tigges et al. (arXiv:2310.15154), among others, have shown that difference-of-means directions causally control behavior under ablation and patching. The paper's method is a direct transfer of this approach, and the authors themselves validate it by reproducing the refusal direction (App. A.1).
-
"Many writers" (distributed writing) is the null hypothesis, not a finding. Given superposition and redundancy, many components writing to a shared direction is what one expects by default. A mechanistic account is constrained only by showing that some components contribute sparsely. The paper's distributed-writing evidence has three problems:
- No baseline. The paper does not report the same head-concentration analysis for an unrelated contrast or a random direction, so "top-5 heads account for only 15–39%" cannot be read as distributed or sparse. In Llama and Gemma, about 22–25 heads cover 70% of the effect, which could equally be read as relatively sparse.
- Instability. Under the matched contrast in A.3, Mistral drops from 121 heads to 35.
- Only node-level total effects. Compensation among components can spread effects across more nodes, and the authors themselves acknowledge that edge-level analysis would be needed.
-
"One reader" is contradicted by the appendix.
- A.7 concedes that is "not the only direction that can causally affect the decision".
- In A.9, fixing still leaves 40–73% of the writer-dependent decision variation (–). Writer influence is therefore "not confined to it".
- The existence of a compact late readout is also partly a consequence of the layer-selection rule. The paper lacks a same-layer control: can a direction for an unrelated binary concept achieve comparable rank-1 sufficiency?
-
The two genuinely novel experiments do not support their conclusions. The both-wrong and specificity experiments could have distinguished this work from prior steering studies. As detailed in Weakness 1, however, the former is confounded by the injected conclusion, and in the latter the diagnostic cells support the output-token account.
Besides, the manuscript reads like an unpolished draft.
- Revision history in the main text. Passages such as "Prompt provenance control" describe how the prompts were rewritten during the project. This is process documentation and does not belong in the main text. It could be moved to the appendix or removed.
- Repeated "not X, but Y" constructions and heavy parenthetical asides, for example:
- "A direction the model decides with, not one we can merely read"
- "a causal lever, not a change confined to the printed answer"
- "every claim about below rests on intervention, not probe accuracy"
- Inconsistent terminology and notation.
- "Crystallization band" appears in the caption of Figure 1 before "crystallizes" is defined in Table 1.
- The meaning of "coordinate" shifts throughout the paper: it refers variously to the axis (feature) itself, to the scalar projection onto it, and to a particular position or value along it.
- "Steering vector" appears only once in the paper, which suggests a leftover from an earlier draft.
- Unexplained signs. In the paragraph "Qwen is different", the paper does not explain what the signs of "−1.26 and −1.25" (or "−2.5 versus +7.3") denote, i.e., in which direction the permit and prohibit cases move.
These issues made the paper noticeably harder to follow, and they lowered my scores for Presentation.
Questions for authors
I would consider raising my score if the authors could address the following points.
-
Output-token direction. What is the cosine similarity of with and with ? Does the component of orthogonal to the output-token direction still steer Task A and Task B?
-
Tasks B and E. In Llama, Task B is clearly steerable (: 0.68), yet reaches only 0.09. Why is B considered non-diagnostic? Could B be added to the sufficiency test (Fig. 5)? Given that recovery on Task E matches or exceeds that on Task A in Llama and Mistral, how do the authors separate rule content from the answer-token channel?
-
Both-wrong experiment. Could the authors report numbers for the premise-only condition, add a conclusion-only condition, and patch the coordinate of the both-wrong prompt back to its violation value to test whether the output reverts to EVADED?
-
Rule application. Given that all violations are classified EVADED (A.5) and changing the threshold barely moves the verdict (A.10), can the authors show the same results on a contrast where permission must be inferred from the conditions rather than stated explicitly in the rule text?
-
Baselines. Could the authors provide (i) the same head-concentration analysis for an unrelated contrast or a random direction, and (ii) a same-layer control testing whether a direction for an unrelated binary concept achieves comparable rank-1 sufficiency?
3 comments
I appreciate the authors’ efforts, especially their honesty in reporting the recovery rates and baseline balanced accuracy for Tasks B and E.
However:
- I think the authors have shown that this direction captures the causal effect of false conclusions on the model’s verdicts. However, I had expected the authors to identify a higher-level causal model, incorporate some secondary causal nodes, and perform interchange interventions on both the causal model and the LLM to validate the plausibility of the causal model.
- I acknowledge that this direction is not entirely aligned with the output-token direction. However, the output-token direction may play an important role in this direction’s effect, as suggested by the approximately 50% change in sensitivity. The authors should use different target tokens—for example, replacing SATISFIED/EVADED—or vary the question format to obtain a semantic direction that genuinely represents whether the model is performing the task.
- If the authors consider their strongest new result to be, I quote, “controlled, bidirectional intervention on supplied-conclusion effects,” then the methodological novelty is limited. For example, Marks and Tegmark intervened on truth directions to make models treat false statements as true and vice versa, while Arditi et al. showed that erasing a refusal direction suppresses refusal and adding it induces refusal. The authors should clarify the main differences between their work and prior studies and articulate the incremental contribution of this specific application.
This third point is the most important to me. The authors must provide a clearer account of the paper’s novelty. Therefore, I do not intend to raise my score at this time.
Thank you for clarifying the novelty concern. Our contribution is empirical rather than a new intervention method. Marks and Tegmark establish causal interventions on truth-related directions, while Arditi et al. establish causal control of refusal and examine how adversarial text affects that mechanism. Our study asks whether a coordinate extracted from explicit permit/prohibit contrasts also controls verdict changes induced by separately supplied conclusions.
The new experiment connects these two manipulations without fitting the direction to the conclusion-induced errors. Restoring the clean coordinate repairs all 24 errors in each of the conclusion-only and combined conditions; reverse transfer induces errors in all 24 clean cases. Sham patches and five displacement-matched random controls do neither. The incremental finding is therefore that rule-text contrasts and supplied-conclusion effects share an interventionally effective coordinate. The conclusion-only control sharpens this result: the supplied judgment is sufficient to induce the errors, whereas the false numerical premise alone is not.
Regarding output-token dependence, the approximately 50% sensitivity reduction is substantial. The retained effect shows that the tested output-gradient direction does not explain the entire effect; it does not establish that the remaining component is vocabulary-independent. A successful test across answer vocabularies and question formats would address that stronger claim. We will distinguish this unresolved semantic question from the causal connection demonstrated here.
Your proposed higher-level causal model would test whether the coordinate corresponds to a variable in rule execution. Our present interventions instead establish a shared causal handle across the two prompt manipulations. We will make that distinction explicit and articulate the contribution around this experimentally supported connection, rather than the general existence of a steerable direction.
Response to AQZK
Thank you for identifying tests that distinguish causal influence from stronger claims about rule semantics. We completed two prospective Llama follow-ups: coordinate restoration and steering after removal of an output-gradient component.
Restoration and conclusion-only control. We tested 24 amount/split-count combinations across eight amounts. Identical false-premise and false-conclusion sentences appear alone or together. The direction was fitted on four separate permit/prohibit pairs, excluding evaluation cases. All evaluation cases have gold verdict EVADED.
Restoration replaces only the frozen direction's coordinate at block 22's final prompt position with its clean-case value. For conclusion-only and combined prompts, the SATISFIED-minus-EVADED gap falls by 3.766 and 3.964 logits, respectively. Bootstrap 95% intervals are [3.656, 3.859] and [3.927, 4.005], resampling eight amount groups. Reverse transfer of either injected coordinate into clean prompts induces errors in 24/24 cases. Sham patches and each of five direction-orthogonal random controls with matched requested displacement norms repair and induce zero errors. Saved states verify restoration within 0.08% relative error.
This establishes causal involvement in supplied-conclusion effects beyond activation convergence. It also narrows their interpretation: the false conclusion alone is sufficient; the false premise alone changes logits but no verdicts. We will describe a supplied-conclusion effect, not demonstrated inference from a false numerical premise. This does not establish exclusive mediation or an abstract rule variable; an output-related coordinate could also mediate the effect.
Output-direction control. Separately, we removed the rule direction's component along the normalized mean gradient of the SATISFIED-minus-EVADED logit difference, calibrated on factual-task extraction prompts. Fitting preceded evaluation. Rule-task baseline accuracy was 44/44. At perturbation scale 0.10 of the extraction residual norm, group-mean finite-difference sensitivity was 0.789 [0.749, 0.834] for the original direction and 0.405 [0.365, 0.454] for the unit perpendicular direction. It retains 51.3% of sensitivity, versus 50.8% at scale 0.05. The unnormalized perpendicular arm also retains a positive effect. All five sampled isotropic random directions have smaller absolute group-mean sensitivity. Intervals resample seven transaction groups. These perturbations change logits but do not flip rule-task answers. The result limits an explanation based solely on this calibrated gradient, not all output-channel explanations.
Historical cosines with the vocabulary-gradient/generic-decision references are Llama 0.158/0.208, Qwen 0.128/0.084, Gemma 0.202/0.063, and Mistral 0.202/0.095. Geometric separation alone is insufficient; the new intervention tests retained influence directly.
Tasks B and E. We agree these results must be reported. Historical Llama normalized recovery is A: 0.836, B: 0.254, E: 0.896; B was run and should appear in the presentation. New baseline balanced accuracy is A: 100%, B: 50%, E: 70.8%. B's system requests YES/NO while its question requests SATISFIED/EVADED; E leaves the mapping from factual truth to these labels implicit. Both fail our prespecified 80% baseline requirement for semantic comparisons. Their transfer effects cannot establish vocabulary-independent semantics. E's reduced sensitivity after orthogonalization is partly expected because its extraction prompts calibrated the removed gradient.
Scope and baselines. We will replace the exclusive “one reader” interpretation with “a substantial causal coordinate.” Fixing it leaves substantial writer-dependent variation, as the appendix shows. We have not completed the requested unrelated-concept rank-one sufficiency or head-concentration baselines, or a high-level interchange test requiring permission to be inferred from conditions. The five-head reference is hypothetical, not an empirical null. We will distinguish descriptive component concentration from evidence for a unique or unusually sparse mechanism and simplify the terminology and presentation. Our strongest new result is the controlled, bidirectional intervention on supplied-conclusion effects; it does not resolve every semantic objection.