acceptodds
Under review as a conference paper at ICLR 2027

What Attention Control Can Reach: Exact Output Geometry

Abstract

Editing where a transformer’s attention heads look is now a standard control, and the same patterns are read as evidence of what it computes. Both rest on an unchecked assumption: that pushing a head’s attention produces the intended change in its output. For one head at a fixed input we settle an edit’s reach and its cost. Reach is fixed before any budget is spent: a bias shared by a group of keys moves each row of the head’s output only inside the hull of one average value vector per group, a region the head’s values and unedited attention fix. Every interior point is reached by a closed-form bias, a target outside stays out at any budget, and a linear readout’s largest displacement follows from one number per group on a single unedited pass. Cost is charged only to edits that cross the softmax: a row’s distance from uniform and its first-order response to logit edits obey a sharp product bound, response vanishes as a row becomes deterministic, and a finite edit pays along its whole path, not at its start. In two live models the excluded directions do not move beyond the float32 floor of the head’s own output; in 7 pretrained transformers the upper tail of trained rows sits nearer the bound than at random initialization; and on one model’s pre-registered held-out split the routing law read at the spent budget, not differentiated at zero, orders real interventions its derivative misorders more often than not. An attention edit can therefore be screened before it is run; neither its reach nor its cost is legible in the pattern itself.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
attention steering, transformer interpretability, inference-time intervention, softmax geometry, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fBN1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies attention steering, i.e., changing where a head looks, e.g., by adding a bias to the logits of a chosen span of tokens. For a single head at a fixed input, it asks which output changes such an edit can reach at all (reach) and how strongly the output responds to it (cost).

Key contributions are:

  1. Exact reachable set of bias edits (Theorem 1, Corollaries 2–3). The theorem characterizes which outputs an edit can reach, for edits that add to the logits a bias that is constant within each group of keys. Because a shared bias rescales all weights in a group by the same factor, proportions inside a group never change, and each group acts as a single "super-token" whose value is a fixed centroid (the attention-weighted mean of the group's value vectors). The row output is therefore always a convex combination of the kk centroids, and the bias only changes the mixing weights. Conversely, every strictly positive mixture is attained by the closed-form bias bg=log⁡(qg/πg)b_g = \log(q_g / \pi_g), so the reachable set is exactly the relative interior of the convex hull of the centroids.

    Consequences: a larger budget never enlarges the reachable set, and kk groups expose at most k−1k - 1 output directions.

    The authors acknowledge that the underlying geometry is classical (exponential tilting of a categorical distribution); the novelty lies in its application to attention control.

  2. Decomposition of interventions by whether they pass through the softmax (Theorem 4). Theorem 4 decomposes any intervention on a head into three routes by which the output can change (through the logits, through the values, or added after the softmax), via a straightforward application of the product and chain rules. Its main takeaway is that the softmax sensitivity enters only the logit route, so attention-steering methods are limited by it, while steering vectors and value/MLP edits are not.

  3. Trade-off between attention sharpness and sensitivity (Theorem 6). Theorem 6 formalizes softmax saturation: the more concentrated an attention row, the less it responds to logit edits, with the product of sharpness and sensitivity bounded by approximately 0.35.

  4. Empirical validation and practical tools (see Strengths).

Critical strengths
  1. Theorem 1 is insightful, and Figure 1 makes it easy to understand. The figure shows the whole result in one picture: each key group G1,…,GkG_1, \dots, G_k is collapsed into a single centroid μg\mu_g, and the bias enters only through a kk-way softmax that mixes these fixed centroids into the output Y(b)Y(b). This makes it immediately clear that the edit changes only the mixing weights, while what can be reached is fixed by the centroids, i.e., by the values rather than by the attention pattern.

  2. The finite-edit result derived in Corollary 5 is experimentally verified in Figure 3 ("Path aperture closes the finite-budget gap").

  3. Practical consequence of Theorem 6: interpretability and steering pull in opposite directions. Although it is intuitively tempting to look at an attention heatmap and say "this head clearly focuses on token X", those clear, sharp heads are exactly the ones where pushing attention barely works, because their softmax is saturated.

  4. The theory may lead to a better head-selection rule for steering. On NQ-Swap (passages edited to contradict the model's memorized answer), steering only the heads chosen by centroid-contrast selection beats PASTA's head selection at matched scope and matched transfer.

  5. Formal (Lean) verification of the core results. The main identities and bounds are machine-checked in Lean, and the authors are transparent about what is not formalized. This substantially increases confidence in the correctness of the theory.

Critical weaknesses
  1. Theorem 4 is mathematically elementary. Its value lies in the resulting classification of steering methods rather than in the derivation.

  2. Limited theoretical novelty. Most results are, in some sense, elementary consequences of well-known facts:

    • Theorem 1 follows from the fact that an attention output is a weighted average of value vectors, so, by simple properties of the exponential function, a group-wise bias can only re-mix fixed group averages.
    • Theorem 4 is the product and chain rules applied to Y=softmax(L)U+ΔY = \mathrm{softmax}(L)U + \Delta.
  3. Major weakness: presentation.

    (a) Simple results presented in a complex way. The paper introduces a lot of terminology (aperture, payload contrast, saturation, certificates) and very dense notation, which makes intuitive ideas hard to follow. The main messages (pushing attention can only re-mix value averages; saturated softmax rows barely respond to logit edits) could be stated much more simply, which would make the contribution clearer and the paper accessible to a broader audience.

    (b) Most of the empirical work is relegated to the appendix. As a result, the reader cannot judge the strength of the empirical evidence from the main paper without going through roughly 20 pages of appendix. A compact main-text table summarizing each experiment's setup, baseline, and outcome would help.

Questions for authors

I have no questions.

2 comments

Thank you for a careful reading. Your summary states our results more plainly than our abstract does, which is the clearest evidence for your presentation point, and we agree with it. We propose local changes, each shown here as it would appear so you can judge it now. None changes a result, proof, experiment or number.

Novelty. We agree that the derivations are elementary. Theorem 1 is exponential tilting, and Theorem 4 is the chain rule; the paper attributes both. The contribution is what these facts make checkable for attention control, and not their proofs:

  • whether an edit can produce a given change at all, decided from one unedited forward pass and valid at every budget;
  • the inversion in your Strength 3: the sharp heads that look most interpretable are the ones least responsive to steering;
  • a finite routing law that predicts large edits where the zero-bias derivative fails (next paragraph).

We propose to rename Theorem 4 a Proposition, with statement and proof unchanged.

The result we placed too late. One result goes beyond checking the theory, and we put it at the end of Section 5 with no number in the abstract. The standard predictor of a steering edit, the derivative at zero bias, is local; Corollary 2 gives the routing at the budget actually spent in closed form. On development the derivative is steeper at zero bias for 382 of 398 records, yet delivers the smaller gain at the operating budget on 256 of them, which is the saturation Theorem 6 describes. On a pre-registered held-out split (Qwen2.5-3B, NQ-Swap, 398 records, no question shared with development), using the closed form in place of the derivative orders 77.1% of intervention pairs correctly, against 39.8% for the derivative and 60.2% for the development-fixed choice. Prediction error falls from 0.350 to 0.191 logits. In this result the elementary facts predict where a standard tool misranks interventions, and a held-out test confirms it. Scope: one model, layer, budget and task, with the downstream step first order.

Presentation (3a). We propose three local edits:

  1. Replace the two central sentences of the abstract and the overview paragraph of the introduction with your two messages, stated directly: a bias on a group of keys can only re-mix fixed averages of their values, and sharp, saturated attention rows barely respond to logit edits, plus the held-out numbers above (as in our response to Reviewer 2).
  2. Give one plain sentence before Theorem 1 and before Theorem 6 that says what the theorem means before its notation appears.
  3. Define each specialised term at first use: aperture is the norm of the softmax Jacobian of a row (its sensitivity to logit edits); payload contrast is the difference between the average value of the steered keys and that of the rest; saturation is the fraction of the sharp bound a row uses; a certificate is a check computed from one unedited pass.

Empirical evidence (3b). We propose to extend Table 2 with baseline and outcome columns. Every entry below is already in Section 5 or its appendix.

ExperimentModelsCompared againstOutcome
Reachable set (Thm 1, Cor 3)OLMo-2-1B, Qwen2.5-3Bmovement inside the reachable directionsmovement outside them is 1.5–1.7 × 10⁷ times smaller, at the float32 floor; two registered criteria (rank, ceiling overshoot) fail at float32 scale, reported as such
Steering vs. bypass (Thm 4)GPT-2, Qwen2, SmolLM2matched post-softmax bypasspre-softmax gain rises with sensitivity (slope 0.56–0.99), bypass is flat (slope ≤ 0.017)
Path bound (Cor 5)GPT-2, 1,728 pathssensitivity frozen at zero bias0 violations; the frozen value fails on 29.5% of paths and underestimates by up to 22.5×
Saturation (Thm 6)7 models, with training checkpointsuntrained initialisation99th-percentile saturation 0.80–0.95 trained vs ≤ 0.19 untrained
Span choicefour models, HaluEval-QAspan mass; always steering the questionabove chance on every model; does not beat always steering the question
Head selectionOLMo-2-1B, Qwen2.5-3B, NQ-SwapPASTA head ranking, same layer, head count and mass transfer+3.5 and +2.8 pp context-faithful accuracy, 3 of 4 comparisons clear zero; no advantage over the autodiff derivative (+0.25 pp, [−0.50, +1.26])
Finite responseQwen2.5-3B, held-out split, 398 recordszero-bias linear prediction; fixed choicepairs ordered: 77.1% vs 39.8% / 60.2%; error 0.350 → 0.191

Scope of the revision. The changes are limited to the abstract's central sentences, one introduction paragraph, two lead-in sentences, first-use glosses, one renamed result and two added table columns, offset by shortening the Section 5 prose that repeats the table's numbers. All statements, proofs and experiments stay as reviewed.

We would welcome your view on whether these changes address your concern.

Thank you for the response.

I will maintain my current rating. I think your work is valuable, but the local edits proposed are insufficient to improve the presentation quality. I struggled a lot to understand your work and I think the global edits are necessary. In particular, I think you should remove all the jargon (payload, range, reachable set, saturation) and rewrite the paper with much simpler and cleaner language (see my point 3 a).

Consider this sentence from your paper: "The routing factor in (2) is classical: a groupwise bias is an exponential tilt of the k-atom categorical family with base measure π, so b → w(b) is its natural-to-mean map (Amari, 2016), and the image of that simplex under the value map, the rank ceiling with it, is the family’s mean-map image (Wainwright & Jordan, 2008); the closed-form bias is in Kang et al. (2026, App. A.1) and the b = 0 factorisation in Su et al. (2025), with further antecedents in Appendix L."

There are 7-8 (depending on how you see it) claims here.

Reviewer_Eg4xOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper is about understanding the effect of changing attention weights via external interventions in a transformer model. Among the results, the following insights are given:

  1. If a bias is applied to groups of keys then the head's output moves in the convex hull of the average value vectors of the groups.
  2. Adding a linear bias can only have a linear effect on the head's output.
  3. When a head places almost all its weight on one or two tokens, it looks highly interpretable, but its softmax Jacobian vanishes.
Critical strengths

The paper does address an important problem: understanding the effects of steering from a theoretical and practical perspective. The theory developed seems sound (and was verified in lean) and the experiments provide an accurate complimentary perspective. The insights drawn are certainly interesting and can be of use to further studies in alignment and interpretability. I think the impact and potential this research program has is definitely high.

I also liked the figures the paper used to make its case, which aided in my understanding.

Critical weaknesses

The paper is extremely difficult for me to understand, and I am fairly familiar with the relevant literature. This has prevented me from being able to accurately assess the weaknesses and understand the mathematical content. The language is overly verbose and full of really hard to understand terminology.

Consider as one example, the following excerpt from lines 16-18: "Every interior point is reached by a closed-form bias, a target outside stays out at any budget, and a linear readout’s largest displacement follows from one number per group on a single unedited pass." The paper's language resembles the above example in many instances. While it is possible to digest the point of the above sentence, it takes 3-4 passes for me, which is also exacerbated when all the sentences are this dense. I would strongly advise the language be modified to highlight intuition and simplicity.

In technical terms, the analysis is done in terms of a single attention head. What about a multi-layer setting? The paper doesn't seem to address that.

As much as I like the direction and insights of the paper, I cannot accurately assess it because I do not understand it fundamentally. Were the language and presentation improved, this would likely be a very interesting paper. I acknowledge the variety of experimental and theoretical work put into this paper and my assessment does not try to diminish their value. I was just unable to follow the arguments being made. If other reviewers found the paper easier to read, I would gladly reconsider my evaluation.

Questions for authors

N/A

1 comment

Thank you for your honest review and for offering to look at the paper again. We agree that the abstract and introduction put too much into each sentence. Below is the plain wording we propose to use there, so you can judge it as written. In plain terms, the paper shows three things:

1. What an attention edit can reach. Suppose you add one bias to each group of keys. Each output row then moves only among weighted averages of a fixed set of points, one per group: the group's average value vector. We give the bias that reaches any point inside this region in closed form. A point outside is never reached, however large the bias. One consequence matters in practice: if a group's average value equals that of the other keys, attention can move onto it with no change in the output. One ordinary forward pass shows whether this is the case, before any steering is run.

2. What an edit costs. A single bias moves the output in one fixed direction, but the size of the move levels off as the bias grows. Sharp attention also responds weakly: as a row concentrates on a single key, its response to a logit edit goes to zero. A sharp bound makes this precise. As a result, the slope at zero misjudges large edits.

3. Evidence. On a pre-registered held-out split (Qwen2.5-3B, NQ-Swap), computing the levelling-off law at the bias actually applied ranks 77% of intervention pairs correctly. The slope at zero ranks 40% correctly.

Multi-layer setting. The exact results hold for one head. Proposition A.1 (Appendix A) bounds how an edit propagates through later layers, and Appendix C combines results across heads. Several experiments steer every layer, and one measures how far the edit's effect reaches as depth grows (Appendices E and J). These extensions are first order, and the predictive value fades within about one block of the edit. The main text points to this material in only one sentence.

Proposed changes. All three are local and change no result, proof, experiment or number:

  1. use the three-point wording above for the central sentences of the abstract and the overview in the introduction;
  2. add one table that maps each result to what it guarantees and to the evidence for it, built from material already in the paper;
  3. add two sentences to the Scope paragraph of the introduction that state the multi-layer bound and where it stops informing, with a pointer to Proposition A.1.

A short response can only show so much of a clearer paper. Yet we hope it shows that the obstacle is the wording, which we think can be fixed, and not the mathematics, which you found sound. We would welcome your view on whether this addresses the readability concern.

Reviewer_Ca97OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes attention interventions at a fixed input, separating the outputs an edit can reach from its local and finite-path sensitivity. For groupwise logit biases with fixed values, it characterizes the reachable row outputs as the relative interior of the convex hull of the groups' attention-weighted value centroids. The binary case factors output displacement into routing-mass change and a value-centroid contrast, yielding exact inertness and linear-readout ceiling conditions. A response-operator decomposition distinguishes logit, value, and post-softmax intervention paths, while a sharp product bound relates distance from uniform attention to the softmax Jacobian norm. Experiments check these identities and bounds, compare steering selectors, and evaluate a finite-routing predictor. On a held-out Qwen2.5-3B NQ-Swap split, that predictor improves intervention-pair ordering over both zero-bias linearization and a development-fixed constant choice. The paper also reports a Lean formalization with an explicit account of its coverage and remaining assumptions.

Critical strengths
  1. An exact characterization tied to a practically recognizable control class. Theorem 1 and Corollaries 2–3 identify what groupwise attention biases can change, give an explicit inverse for target group masses, and distinguish changing attention from changing the resulting output (Sections 3–4, pp. 3–6). The value-centroid contrast makes the obstruction concrete: a binary span can change its routing mass substantially while remaining output-inert. Although the underlying ingredients are classical, expressing the reachable set and inertness condition for this intervention class is informative and directly relevant to attention-steering analysis.

  2. The strongest empirical result goes beyond checking an algebraic identity. The held-out experiment in Appendix K fixes the intervention configuration and uses source questions absent from development. The finite-routing predictor reduces paired-difference prediction error from 0.350 to 0.191 logits of margin and orders 77.1% of scored intervention pairs correctly, versus 60.2% for the development-fixed constant choice and 39.8% for the linear prediction (p. 32). Because the predictors share their downstream gradients, the comparison usefully isolates the value of replacing linearized routing with its finite form. The explicit disclosure of failed criteria and the scope of formal verification also improves confidence in the paper's narrower claims.

Critical weaknesses
  1. The independent technical novelty is modest relative to the breadth of the theoretical presentation. The reachable-set result follows from groupwise exponential tilting and a linear image of the simplex; the interface theorem is a chain-rule decomposition with a block-operator norm; and the product bound maximizes a combination of the classical trace and one-half envelopes. The paper appropriately attributes these ingredients (p. 4, Section 4, and Appendix L), so this is a significance concern rather than an attribution concern. Nevertheless, the acceptance case depends mainly on the insight and utility of their synthesis. The number of theorems and formalized statements should not be taken as evidence of comparably many independent mathematical advances.

  2. The practical evidence supports a substantially narrower screening and prediction capability than the general motivation might suggest. The exact statements concern fixed values and immediate head outputs. In the OLMo-2 depth sweep, the ceiling's standalone ranking falls to approximately chance at four or more subsequent blocks, while the registered headroom-based depth prediction does not follow its proposed monotone pattern (Appendix J, p. 30). Span choice does not establish an improvement over always choosing the question (Appendix G, p. 26), and the selected-head method does not establish an accuracy advantage over the all-row derivative (Appendix K, p. 31). The positive held-out finite-response result is consequently important, but currently covers one model, layer, budget, task, and pair of intervention plans. These limitations constrain the practical significance of the framework even though they do not invalidate its local mathematics.

Questions for authors
  1. Assessment-critical analysis of existing outcomes: On the held-out NQ-Swap split, does choosing the intervention plan using the finite-routing predictor improve realized mean margin or task accuracy over the development-fixed constant plan? The 77.1% ordering accuracy is informative, but a decision-quality result would establish whether the correctly ordered cases outweigh errors on more consequential cases.

  2. Clarification of certification scope: Appendix E reports failures of the registered numerical rank and readout-ceiling criteria, including relative ceiling overshoots of approximately 0.0026 and 0.0060 (p. 21). Which implemented screening claims, if any, have a validated one-sided error guarantee in floating-point arithmetic? Please distinguish exact-real-arithmetic exclusions from numerical heuristics and explain how near-boundary cases are handled. This directly affects the advertised use of screening an intervention before running it.

  3. Assessment-critical robustness: Is the held-out advantage of the finite-routing predictor retained at nearby budgets or edited layers without retuning the selection rule? A narrowly scoped sensitivity analysis would help determine whether the strongest empirical contribution reflects a useful operating regime or a particular evaluated configuration. A comparison with a derivative recomputed along the path would also clarify the compute–accuracy tradeoff if the necessary measurements already exist.

1 comment

We thank the reviewer for a close reading and for three questions that point to exactly what would strengthen the paper. Two can be answered from the existing held-out run (Qwen2.5-3B, 398 records). We give those answers below and propose to include them in the final version, labelled as exploratory analyses.

Q1. Does choosing plans with the predictor improve outcomes? Yes, in margin. Choosing per record by the finite-routing predictor G raises the realised margin over the development-fixed plan by +0.078 [+0.057, +0.101] logits, closing 74% of the gap to an oracle. Its correct calls are the consequential ones: on the decile of records where the plans differ most, G orders every pair correctly (constant plan: 62.5%). Accuracy cannot separate the policies, because the two plans give different answers on only 4 of 398 records.

Q2. What does screening guarantee in floating point? We propose to separate three levels in the final version:

Exact: the reachable set, the inertness criterion and the readout ceiling hold in real arithmetic. Numerical: the implemented checks are tolerance-based screens without a validated floating-point guarantee. Cases within a tolerance band of the boundary would be reported as undecided. Downstream: the step to the model output is first order, so no level gives a task-level guarantee. Appendix J measures where it stops informing. The ceiling overshoots arise because edits actually reach the ceiling, so float32 rounding falls on both sides of it. Bounds that keep slack show no violations.

Q3. Robustness. Without retuning, G's ordering accuracy stays at 76–78% across the recorded dose grid. Its advantage over the linear prediction grows with dose, which is where the theory says finite routing matters. We propose to add this table to the final version, together with the same frozen protocol run at a neighbouring layer. G costs one forward and one backward pass, the same as the linear prediction.

Novelty. We agree: the ingredients are classical and the theorem count is not the contribution. In the final version we propose to state two contributions and present everything else as supporting tools. First, an exact criterion for when a routing change is inert, checkable from one unedited pass and not softened by budget; prior non-identifiability results show that such directions exist but give no test for them. Second, the routing-times-payload factorisation, which predicts why the zero-bias derivative misranks plans at the budgets steering uses, a prediction the held-out run confirms.

Scope of the evidence. We accept that the demonstrated capability is narrower than the motivation suggests, and we propose to match the framing to it: screening at the head output, and finite-routing prediction where routing saturates. The negative results mark that boundary. The ceiling informs within about one block of the edit, and the registered monotone depth prediction failed. Selection does not beat the all-row derivative on accuracy, and span choice does not beat always choosing the question. On the positive side, the final version would add decision quality (Q1), the dose range (Q3) and a second edited layer.

Readability. To lower the reader's load, we propose one table mapping each result to its guarantee and its empirical reach, and moving protocol detail to the appendix so the main argument reads in one pass.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.