acceptodds
Under review as a conference paper at ICLR 2027

Semiclassical Geometry of Attention Switching

Abstract

When one attention head is pushed toward a competing evidence span, do baseline quantities identify useful directions for controlling the answer margin? For a fixed-key, two-branch logit shift, the head's output change splits exactly into a branch-switching term and a total-evidence-mass term (a specialization of softmax differentiation, used as coordinates). Switching geometry identifies a task-enriched control subspace. Across eight Qwen3-8B examples (archived custom path), the three-dimensional switching normal subspace retains a mean of squared task-gradient norm versus for the median matched-random subspace (paired difference , 95% interval ; uses each example's gradient). Baseline-only prediction holds. On 50 ConflictQA examples and eight Gemma heads, finite attention recomputation from the baseline gives pooled versus for local linearization. In an exploratory native Qwen3-8B cohort (a 32-fact prefix of a 200-fact ConflictQA test set, path gate bypassed), a predictor frozen from baseline forward and backward passes achieves small median local-regime error () for all three steering arms—switching, fixed PCA basis, and static mean direction: geometrically grounded directions are collectively useful. A switching-specific advantage is not established. On those 32 facts, switching minus PCA at is and switching minus static is (not a test of equivalence); the 200-fact cohort favors switching over the fixed basis only. Switching geometry thus locates useful control directions but is not a privileged axis; path and attribution limits are in Section sec:discussion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_gpNwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper develops a geometric view of competing-evidence attention by defining a switching wall between two evidence branches, derives an exact decomposition of attention-output responses into branch-switching and outside-mass components and shows that baseline local information can predict finite intervention effects. The proposed switching-normal directions are behaviorally relevant and task-enriched, but the experiments do not establish a consistent advantage over strong static steering baselines.

Critical strengths

The idea sounds interesting.

Critical weaknesses

The manuscript is heavily over-engineered and difficult to read. It contains extensive internal filenames, code paths, hashes, audit records and implementation history, which makes it a strong AI-assisted log rather than a polished paper. The abstract is not polished at all, including ???. The paper also accumulates many experiments, reruns, controls and post-hoc analyses without clearly identifying which results are central. Some experimental evidence is explicitly unresolved or invalidated by the authors' own audits. At current status, I would not like to call it a paper.

Questions for authors

See weaknesses

1 comment

Thank you for your careful review and constructive comments.

Internal material. Filenames, paths, hashes, and audit records were removed; the revision is 10 main/13 total pages, with only proofs, protocol, criterion, and a per-head table in appendices.

Abstract/“???”. Rewritten without unresolved references or macros, it states one claim, result, and caveat.

Central result. One native-path experiment reports pre-specified R−L differences of −0.0177 [−0.0219,−0.0136] and −0.0248 [−0.0293,−0.0206] nats/token (Section 4.2; Table 2); secondary and unreported planned analyses are identified (Sections 4.3–4.5; Limitations).

Invalid evidence. All custom-path experiments and steering are withdrawn; both models fail the pre-specified all-width fp32 criterion, while weaker best-width and post hoc float64 checks pass but do not replace failure, so results remain descriptive (Sections 4.1, 4.5; Tables 1, 5).

Steering. No advantage over static baselines is shown; we make no steering/control claim (Limitations).

Remaining limits. Two models, one dataset, eight pre-selected heads/model, AtP* overlap, unsupported Eq. 11, and unchanged title are stated; no human line-by-line read has occurred.

Reviewer_BwcqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

When provided with two pieces of conflicting evidence, how does attention guide the model to choose one set or another? The paper uses the following approach:

  • Identify the tokens belonging to each competing evidence passage.
  • Find a direction in an attention head’s query space that shifts attention toward one passage and away from the other.
  • Use the resulting attention change, together with a downstream gradient, to predict how the model’s preference between two candidate answers changes.
Critical strengths

N/A

Critical weaknesses

The contribution is extremely modest and the paper is extremely hard to get through.

Lots of overlap with existing work, the performance improvement they report is not because of a methodological improvement (the authors don't really claim that it is to be clear). Recomputing attention before applying a frozen gradient that is downstream is a component of AtP*. The authors acknowledge this

Very poor writing. In the abstract: "Across eight Qwen3-8B examples (archived custom path)" What does archived custom path mean? Did a human look at this?

This paper is definitely not ready for ICLR

Questions for authors

N/A

1 comment

Thank you for your careful review and constructive comments.

Overlap/contribution. Recomputing attention before a frozen downstream gradient is AtP*’s attention correction; the abstract, introduction, and Section 3 now say so, and we claim no new estimator or improvement.

Proposition 1 only supplies switching/outside-mass coordinates; Eq. 8 is also in Huang et al. (2026), Eq. 9 is Zhang & Wang’s (2026) remainder, Eq. 10 is unusable because H is unestimated, and Eq. 11 is unsupported by measurements (Sections 2–3; Table 4).

Gemma/Qwen native results are descriptive: mean absolute L/R errors are 0.0324/0.0147 and 0.0450/0.0203 nats/token (Table 2), because both models fail the pre-specified all-width fp32 criterion; weaker best-width and post hoc float64 checks pass but cannot replace failure (Sections 4.1, 4.5; Tables 1, 5).

“Archived custom path.” This was our attention re-implementation; it fails validation, and every result using it is withdrawn (Limitations).

“Did a human look?” Not line by line; AI assistance and earlier unedited material are disclosed (AI Use Statement).

Readability. The 10-page main text has one claim/experiment; identifiers, paths, and audit records were removed.

Reviewer_uoYwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how an attention head shifts between two conflicting evidence spans (ConflictQA). It proposes to split the head's output change into two terms, predict its effect on the answer, and test whether the switching direction helps steer Gemma 4 E4B-it and Qwen3-8B. Writing problems and inconsistencies make the paper’s contributions hard to assess.

Overall, the paper reads like a research log rather than a paper. Its main results rely on an unvalidated attention implementation, and presentation issues make it difficult to assess what the paper establishes.

Critical strengths
  1. The paper proposes to study a relevant question for interpretability and knowledge-conflict research, namely how an attention head chooses between conflicting sources.

  2. The paper grounds its approach in a formal decomposition of the head's output change into two terms.

  3. The authors are transparent about their limitations.

Critical weaknesses
  1. The paper does not form a coherent argument as a whole. The title refers to a semiclassical theory that is used only as motivation, the abstract claims useful control directions that the results do not support, and the cohorts give conflicting outcomes. The authors should state one main claim and align the title, abstract, experiments, and conclusion with it.

  2. The paper is hard to read. Many terms are used before they are defined, defined only in the appendix, or never defined (e.g., CX-ID, CX-3, "commitment-sensitive transition window"), Fig. 1 is never cited in the text (only in Table 1 and the caption of Fig. 4), Fig. 3 is illegible, and key information appears only in the appendix.

  3. The paper describes five experiments, but only Experiments 1, 4, and 5 are presented in the main text, and the CX-ID analyses, which test control on Qwen3-8B, are placed inside the section on Experiment 1 (prediction on Gemma).

  4. The main measurements are not validated. Experiments 1 to 5 use a custom attention path whose audits failed, and the CX-ID validation check was bypassed. Results should be reported only on a path validated against the native model.

Questions for authors
  1. What is the single main claim of the paper, and which experiment supports it directly?

  2. Can you provide a passing validation of the custom attention path against the native model for Experiments 1 to 5?

  3. Why was the validation check in the CX-ID run bypassed, and why does the positive result on 200 facts not hold on the 32-fact subset?

1 comment

Thank you for your careful review and constructive comments.

W1/Q1. Proposition 1 splits the derivative into switching/outside-mass terms used by AtP*’s correction (Section 2; Appendix A); the experiment is descriptive (Section 4.2; Table 2) and title is unchanged.

W2. Listed terms were removed; notation defined, schematic added, submitted Fig. 3 removed, and validation/result/limits moved to main text (Sections 2–4, 6).

W3. The experiment covers validation, R vs L, shifts, mechanisms, and float64 (Sections 4.1–4.5); controls withdrawn.

W4/Q2. No custom-path validation passes; Experiments 1–5 withdrawn. Native no-op/batch checks pass, but both fail pre-set all-width fp32 criterion (h=0.01 maxima 0.062381/0.063526); weaker best-width and post hoc float64 checks pass but cannot replace failure (Tables 1, 5).

Q3. Bypass reason is unverified; rerun stopped at query consistency (1.55×10⁻⁵>10⁻⁵). Switching−PCA is +0.072 [0.058,0.087] on 200 facts but −0.003 [−0.016,0.005] on 32 shared rows, so the positive mean comes from 168 untested facts; cohorts/steering are withdrawn.

Limits: two models, one dataset, eight pre-selected heads/model, unsupported Eq. 11, and no human line-by-line read.

Comments on the paper

1 comment

We thank all reviewers for their careful and constructive reviews.

  1. Five experiments used unvalidated reimplemented attention; all results/steering are withdrawn (Limitations).
  2. Assistant-generated internal labels, audit notes, “???”, and undefined macros were removed/disclosed (AI Use Statement).
  3. We retain Proposition 1/chain rule, identify recomputation as AtP*’s correction, and add one native-path experiment (Sections 2–4; 10/13 pages).
  4. Both fail pre-set all-width fp32 criterion; weaker best-width and post hoc float64 checks pass but cannot replace it (Sections 4.1/4.5; Tables 1/5).
  5. Descriptive R−L: −0.0177/−0.0248 nats/token (Table 2).
  6. Eq. 11 is unsupported by measurements (Section 3; Table 4).
  7. Title unchanged; no semiclassical claim (page 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.