acceptodds
Under review as a conference paper at ICLR 2027

Identity-Preserving Attention Readouts for Deductive Validity

Abstract

Whether a conclusion follows from its premises can be read from where the conclusion's own attention goes, nearly as well as from the hidden states that a supervised probe reads. We train a logistic classifier, a readout, on a few attention values per head, such as the share of the conclusion's attention that lands on the premises. It reaches at least 89% of the probe's ProofWriter AUC in every one of the thirteen models where the readout is extracted, up to 27 billion parameters, without access to the residual stream. On Qwen3-1.7B it scores above LapEigvals, a trained detector of attention Laplacian features, on every benchmark we test, while reading fewer features per head. The reason is that statistics which stay the same when the tokens are relabeled, such as attention Laplacian spectra, cannot represent which token attends to which. We call this limit permutation blindness and state it as an elementary lemma. Spectra of the residual stream, the very states the probe reads, also stay near chance, so the limit lies in the summary and not in the tensor it reads. Even the entropy and the maximum of the conclusion's attention row, which ignore which token receives which share, beat every spectral summary of the whole attention matrix in each of those models, so detectors built on such summaries should read that row instead. The classifier also attributes the verdict to single heads, at about the per-item storage of a probe at one fixed layer, and needs more training labels than the probe. We also find that a single negation feature reads the labels of ProofWriter far above chance, invisibly to the usual length control, and release rebalanced splits and the code that regenerates every number. Code: https://anonymous.4open.science/r/latent-validity-8074/

Primary area:
interpretability and explainable AI
Keywords:
deductive reasoning, attention graphs, permutation invariance, probing, logical validity, benchmark artifacts, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x4gqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an attention-based anchored readout for predicting deductive validity. Evaluated across multiple models and six benchmarks (eg, ProofWriter, FOLIO), the method achieves competitive performance relative to hidden-state probes and outperforms several spectral attention-based baselines. The paper also identifies a significant negation artifact in ProofWriter and introduces the concept of permutation blindness to explain the limitations of invariant spectral summaries. The proposed approach is simple and interpretable, but further clarification is needed to convince its practical motivation, generalizability, and methodological contribution.

Critical strengths
  • The negation artifact identification in ProofWriter is valuable. The authors state this issue with rebalanced splits, while further experiments strengthen the evaluation.

  • The method is evaluated across diverse model sizes and benchmarks. Analyses of individual heads and attention regions provide useful insights into predictive signals and their potential role in model behavior.

Critical weaknesses
  • The main advantage is access to attention without requiring hidden states, but the practical importance of this capability is insufficiently established. Since hidden-state probes generally perform better and are more label-efficient in low-data settings, it would be good to know how authors can better justify the intended use case and demonstrate a concrete efficiency advantage.

  • The experiments predict deductive validity given premises and a candidate conclusion, rather than detecting hallucinations in generated answers. Given the small, mostly constructed benchmarks and variable cross-dataset transfer, it would be good for the authors to provide evidence on more realistic hallucination-detection tasks.

Questions for authors
  • Could the authors clarify the computational overhead of extracting attentions in practical inference systems?
2 comments
AuthorsOpenReview ↗

Response to Reviewer x4gq

Thank you, and for calling the negation artifact "valuable". The revision extends it to a surface-balanced split and four further benchmarks.

W1.

The main advantage is access to attention without requiring hidden states, but the practical importance of this capability is insufficiently established. Since hidden-state probes generally perform better and are more label-efficient in low-data settings, it would be good to know how authors can better justify the intended use case and demonstrate a concrete efficiency advantage.

The readout's value is explanatory, and the paper no longer argues a deployment advantage (Section 9). Per head, it shows which of six named attention quantities carry information about validity, with no hidden states and no learned direction. On the surface-balanced ProofWriter split it reaches 95 to 102% of a hidden-state probe's AUC.

W2.

The experiments predict deductive validity given premises and a candidate conclusion, rather than detecting hallucinations in generated answers. Given the small, mostly constructed benchmarks and variable cross-dataset transfer, it would be good for the authors to provide evidence on more realistic hallucination-detection tasks.

We agree, and Section 9 now states the scope. The paper studies the validity of a given premise-conclusion pair and does not claim hallucination detection in generated text.

On realism, ConTRoL is expert-written reasoning over real passages, news and examination text. Four further benchmarks, named before their data were opened, enter with their twelve-feature surface floors (text-only baselines, Table 11). On Qwen3-1.7B:

BenchmarkReadoutAgainst
ConTRoL0.636Lookback Lens, authors' code: 0.468 (Table 1, lower block)
Multi-LogiEval0.941surface floor 0.612
LogicBench0.760surface floor 0.663
ProverQA0.683surface floor 0.826, above both readouts, so the audit flags it

LogiQA 2.0 sits at chance for both readouts. We also applied the readout fitted on ProofWriter, unchanged, to these four benchmarks on five models. It matches the probe in 12 of 20 cells, exceeds it in 1 and trails it in 7 (Table 15).

Q1.

Could the authors clarify the computational overhead of extracting attentions in practical inference systems?

About one part in 5,000 of the forward pass, by our count.

Fused kernels do not return attention weights, so the readout recomputes the conclusion tokens' attention rows at each attentive layer from the cached queries and keys. Five of the six quantities need the final row alone. We counted from the Qwen3-1.7B configuration (28 layers, 16 query heads of dimension 128) and our span code (scripts/experiments/overhead_count.py in our repository):

Per ProofWriter itemMultiply-adds
Conclusion's attention rows (median)3×10⁷
Final row alone7×10⁶
Forward pass1.7×10¹¹

Section 4 gives our pipeline's measured times on the same model, 49 ms per item for the anchored pass and 46 ms for the probe path.

We hope the revision resolves your concerns, and we would appreciate it if you took it into account in your final score. We are happy to discuss any point further.

AuthorsOpenReview ↗

Follow-up for Reviewer x4gq

We evaluated the readout against hidden-state probes on a surface-balanced ProofWriter split, built so that lexical cues cannot solve it. There it reaches 9595 to 102%102\%% of the probe's AUC, using attention alone and no learned direction. We also added four further benchmarks, each reported beside its surface floor. Section 9 states the scope. We would be glad to hear whether these results resolve your concerns, and if so, we would be grateful if you would consider revising your rating.

Reviewer_oHmwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

General Comment: I think this paper contains potentially interesting research, particularly the analysis of the ProofWriter benchmark and the comparison between different attention-based representations. However, I have significant concerns about the writing, methodological presentation, novelty, and practical contribution.

The paper is difficult to understand, and the figures and tables do not sufficiently help the reader identify the main findings. Important decisions in the experimental protocol are explained mainly in supplementary material, even though they directly affect the interpretation of the results.

See below, what I think about the different sections:

Abstract and Introduction: I have several problems with the writing of this paper. I find the manuscript very difficult to follow, particularly because the connection between paragraphs is often unclear and many technical concepts are introduced without sufficient explanation. The abstract presents several findings almost as disconnected statements, making it difficult to identify the main contribution. The same happens in the introduction, where the authors do not clearly explain the task before moving to the proposed method and its results.

A clear example is the paragraph starting at line 052. The authors introduce three terms in the same paragraph, moving from "A readout is a logistic regression" to the definition of invariant summaries and then anchored readouts. I think these definitions need more explanation and a better connection with the motivation.

Another example of the presentation is Figure 1, which is referenced in the introduction before the reader has a clear understanding of the experimental setup. Why use connected lines in panel (b) when the horizontal axis represents different types of summaries rather than a continuous variable? What do the numerous grey lines represent? Although some information is included in the caption, the reader needs to make considerable effort to understand the figure. I feel the authors assume that the reader already understands several aspects of the methodology that have not yet been introduced.

Figure 2: The setup is also difficult to follow. The authors introduce many architectures, datasets, representation types, and experimental decisions in a very compressed section. I find the analysis of the negation artifact in ProofWriter interesting. Showing that a simple lexical feature can predict the labels with high accuracy is a relevant finding. However, Figure 2 is difficult to read and does not help much in understanding the problem. Why not include a concrete example showing how negation is correlated with the ground-truth label?. The authors explain the resampling procedure in the appendices, but I think the main paper should provide a clearer explanation of how the datasets are modified and why these modifications are necessary for the subsequent experiments.

Experimental Protocol: One of my main concerns is the selection of the best-performing layer for the hidden-state probes. The authors state that the results are reported at the best layer, but they do not sufficiently explain the selection procedure in the main paper. Appendix E clarifies that the best layer is selected using the same out-of-fold AUC results that are subsequently reported. This introduces optimistic selection bias because the evaluation results are used to select the configuration.

I understand that the authors acknowledge this limitation, quantify its effect, and that the bias favors the hidden-state probe rather than the proposed method. However, why use this procedure for a direct comparison instead of selecting the layer within the training folds?. I think this is an important methodological decision that should be discussed directly in the main paper. A reader should not need to go to the appendix to understand the exact implications of the evaluation protocol.

Anchored Readouts and Mathematical Contribution: The proposed method uses six attention statistics per head to predict deductive validity. The idea is understandable once the formulation is explained, but I find it difficult to identify what is fundamentally new compared with existing attention-based readouts, such as Lookback Lens (Chuang et al., 2024) and LapEigvals (Binkowski et al., 2025). Lookback Lens already uses attention ratios from individual heads to train a classifier, while LapEigvals uses spectral features extracted from attention matrices. Moreover, the authors acknowledge that the Lookback Lens ratio can be constructed using two of the six proposed statistics. So, what is the fundamental novelty of this method? Is it the selection of the conclusion's attention row, the combination of existing attention statistics, or the application to deductive validity? I understand that these are methodological differences, but I think the authors should better explain what new scientific understanding their approach provides beyond existing attention-based methods.

  • Chuang et al. (2024). Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. EMNLP 2024.
  • Binkowski et al. (2025). Hallucination Detection in LLMs Using Spectral Features of Attention Maps. EMNLP 2025.

Results and Tables: Tables 1 and 2 are particularly difficult to read. The proposed method is not easy to identify among the different baselines, and the abbreviations and large number of columns make direct comparisons unnecessarily complicated. The tables would benefit from better organization and clearer aggregate comparisons.

Another concern is the actual performance of the proposed method. In Table 1, the anchored readout performs below SAPLMA on all six benchmarks, while P(True) also performs better on some tasks. I understand that the authors focus on attention-only methods rather than hidden-state probing, but I think this makes the practical motivation even more important. Why should we prefer the proposed approach when conventional hidden-state probes achieve better results?

The paper demonstrates improvements over several attention-based spectral methods, but I am not convinced that it sufficiently establishes the importance of this improvement compared with stronger existing methods. Furthermore, the evaluations use relatively small datasets, with 240 examples for ProofWriter and 200 for FOLIO in the main analysis. Although the number of evaluated architectures is extensive, the limited size of the reasoning benchmarks raises questions about the generalization of the findings.

Computational Cost and Practical Application: I also find the discussion of computational efficiency confusing. The authors discuss storage, inference time, and label requirements, but the relationship between these measurements and the figures is not sufficiently clear. For example, Figure 3 evaluates the performance of the methods with different numbers of training labels, but it does not directly explain the computational costs discussed in the same section. More importantly, the proposed method does not demonstrate a clear computational advantage over hidden-state probing. On the anchor model, the anchored readout takes approximately 49 ms compared with 46 ms for the probe path, while also requiring more labeled examples to reach comparable performance.

The authors motivate the method as useful when attention information is available but hidden states are not. However, I would like a stronger explanation of how realistic this scenario is and what practical advantage the method provides when both representations are accessible.

Head Analysis and Conclusions: The head-selection and ablation experiments are interesting, but the interpretation of the results could be clearer. The authors report that removing the selected head produces effects similar to random-head ablations in 10 of 13 models. This is relevant because it limits what can be concluded about the functional importance of the selected heads. The ability to predict validity using attention information does not necessarily imply that those heads are causally responsible for the model's reasoning.

Critical strengths
  1. The paper investigates whether attention-based representations can provide useful information about deductive validity without accessing hidden states. This is an interesting direction for understanding internal representations.
  2. The authors evaluate several language-model families and sizes, including relatively large models. This provides useful evidence about the behavior of attention-based readouts across architectures.
  3. I find the analysis of the negation artifact in ProofWriter interesting. The authors demonstrate that a simple lexical feature can predict labels with high accuracy, which raises important concerns about evaluating reasoning and validity detection using this benchmark.
Critical weaknesses

See the summary section to get more details:

  1. Writing and presentation: The manuscript is difficult to follow, with unclear transitions between paragraphs and figures that require excessive interpretation.
  2. Experimental protocol: The best-layer probe results contain optimistic selection bias, and the implications of this procedure are not sufficiently explained in the main paper.
  3. Mathematical contribution: The novelty and scientific implications of the permutation-blindness formulation are not sufficiently established.
  4. Experimental results: The tables make comparisons difficult, and the proposed method does not outperform stronger hidden-state approaches.
  5. Practical relevance: The method does not establish clear advantages in computational cost, storage, or label efficiency compared with conventional hidden-state probing.
  6. Scientific interpretation: The connection between the different experimental findings and the central contribution is insufficiently explained.
Questions for authors
  • Why are the hidden-state probes reported using the best layer selected from the same out-of-fold evaluation scores? Why not use nested cross-validation or training-only layer selection consistently across methods in the main results?
  • What is the main new scientific understanding provided by the proposed anchored readout compared with existing attention-based methods such as Lookback Lens? Is the main contribution the selection of the conclusion's attention row, the proposed features, or the permutation-invariance analysis?
  • Since the entropy and maximum of the conclusion's attention row already outperform several spectral methods without preserving individual key identity, how much of the improvement comes from selecting the relevant row rather than preserving token-to-token identity?
  • In which realistic applications would attention representations be available while hidden states are inaccessible? Given the lower performance relative to supervised probes, higher label requirements, and similar computational cost, what makes the proposed approach preferable?
  • Why are the main benchmarks limited to relatively small samples? How confident can we be that the observed relationships generalize to more realistic reasoning tasks and different data distributions?
  • Could the authors reorganize Tables 1 and 2 to distinguish the proposed method and the different baseline families more clearly, including aggregate comparisons and a more direct discussion of their practical trade-offs?
  • Beyond the elementary property established by Lemma 1, what additional conclusions about the representations used by language models are supported by the experiments?
  • Could the authors improve Figures 1 and 2 by including clearer legends, larger fonts, and concrete examples illustrating the task, the representations, and the motivation for the proposed method?
6 comments
AuthorsOpenReview ↗

Response to Reviewer oHmw (1/2)

Thank you for a thorough, section-by-section critique, and for finding the negation analysis "interesting". The presentation changes follow your critique closely.

W1.

Writing and presentation: The manuscript is difficult to follow, with unclear transitions between paragraphs and figures that require excessive interpretation.

Done:

  • Abstract: one claim, then its evidence.
  • Section 1: the task before any method. Three short paragraphs define the three terms, each with an example.
  • Figure 1: cited first with the task. Its first row shows one item, the six quantities and the relabeling before any result, with no lines across categories.
  • Tables 1 and 2: our readout apart from each baseline family, with an average column, each detector's reads and labels, and a median row.
  • Figure 2: a concrete negated and non-negated example.
  • Section 3: how and why each dataset is resampled, now in the main text.
  • Figure 11 (Appendix E): the label study, captioned as label requirements, with cost reported separately.
  • Figure 3: the readout against Lookback Lens for every model on nine benchmarks.

W2.

Experimental protocol: The best-layer probe results contain optimistic selection bias, and the implications of this procedure are not sufficiently explained in the main paper.

We agree, and it is now in the main paper (Section 3). Nested layer selection exists for four small models on ProofWriter, the paired split, FOLIO and LogicNLI, and for the two 27B models on ProofWriter and FOLIO. Table 2 uses it there. Elsewhere Tables 1 and 2 mark the best-layer probe as an upper bound, which only makes the comparison more conservative for our readout.

W3.

Mathematical contribution: The novelty and scientific implications of the permutation-blindness formulation are not sufficiently established.

The implication is measurable. The signal sits mostly in the conclusion's row. That row's entropy and maximum ignore key order, yet reach 0.899 against the readout's 0.925 (Table 10). Permuting keys costs 0.008 (Appendix G).

LapEigvals reads the largest eigenvalues of each head's attention Laplacian. Lookback Lens reads, per head, mean attention on the context over mean attention on context and generated tokens, averaged over the generated span. So it already reads the generating rows. We add:

  • a measured account of why per-head spectra lose the signal (a permutation-invariant rule that is told the final row recovers a median 86% of the gap, Section 6),
  • six named quantities, five read from that row (Section 4)
  • the artifact audit (Sections 3 and 6).

Against Lookback Lens from its authors' code, the readout is higher by a paired interval on all six benchmarks of Table 1, by +0.12 to +0.25 AUC (Qwen3-1.7B, five are validity evidence). On the other twelve models it is never lower by a paired interval.

W4.

Experimental results: The tables make comparisons difficult, and the proposed method does not outperform stronger hidden-state approaches.

The probe is the stronger detector. The readout answers a different question. The probe says that the model encodes validity. The readout says which attention quantities, in which heads, carry that information, with no hidden states and no learned direction. Both come from the same forward pass, so the revision argues from explanation. On the surface-balanced ProofWriter split the readout reaches 95 to 102% of the nested probe's AUC on four models (Section 6). Stacked with the probe it adds nothing detectable (Section 4). The paper had not made this role clear. On the tables, see W1.

W5.

Practical relevance: The method does not establish clear advantages in computational cost, storage, or label efficiency compared with conventional hidden-state probing.

We agree. The anchored pass takes 49 against 46 ms for the probe and needs more labels, so it has no computational advantage. Section 4 says so plainly, and the paper no longer argues from cost.

W6.

Scientific interpretation: The connection between the different experimental findings and the central contribution is insufficiently explained.

The conclusion (Section 8) now states four results in turn:

  1. The validity signal is carried by the conclusion's own attention row.
  2. ProofWriter's labels are readable from negation and other surface features, so validity claims on it need a split balanced on all of them. The main, negation-balanced split still gives a twelve-feature surface model 0.943.
  3. On such a split both the readout and the probe keep a signal well above chance.
  4. The heads the readout ranks highest matter to the model together.
AuthorsOpenReview ↗

Response to Reviewer oHmw (2/2)

From your summary:

Head Analysis and Conclusions: The head-selection and ablation experiments are interesting, but the interpretation of the results could be clearer. The authors report that removing the selected head produces effects similar to random-head ablations in 10 of 13 models. This is relevant because it limits what can be concluded about the functional importance of the selected heads. The ability to predict validity using attention information does not necessarily imply that those heads are causally responsible for the model's reasoning.

We agree and removed the single-head verdicts. With ten controls no single-model test can reach significance, and the family-wise count matches chance (Appendix D). We reran with 99 controls per model (47 for the two smallest Qwen3.5 models) on 10 of the 11 models up to 8B. The confirmatory test across models gives p = 0.53, so the paper calls no single head required.

A registered joint ablation zeroed the readout's five most informative heads together, on the four models of the surface-balanced splits. It lowers the model's own True/False AUC more than 99 layer-matched random five-head sets do (p = 0.027 across models). The paper now says those heads matter together, though no single one alone (Section 7.1).

Q1.

Why are the hidden-state probes reported using the best layer selected from the same out-of-fold evaluation scores? Why not use nested cross-validation or training-only layer selection consistently across methods in the main results?

Answered with W2 above.

Q2.

What is the main new scientific understanding provided by the proposed anchored readout compared with existing attention-based methods such as Lookback Lens? Is the main contribution the selection of the conclusion's attention row, the proposed features, or the permutation-invariance analysis?

Answered with W3 above.

Q3.

Since the entropy and maximum of the conclusion's attention row already outperform several spectral methods without preserving individual key identity, how much of the improvement comes from selecting the relevant row rather than preserving token-to-token identity?

Answered with W3 above.

Q4.

In which realistic applications would attention representations be available while hidden states are inaccessible? Given the lower performance relative to supervised probes, higher label requirements, and similar computational cost, what makes the proposed approach preferable?

Answered with W4 above.

Q5.

Why are the main benchmarks limited to relatively small samples? How confident can we be that the observed relationships generalize to more realistic reasoning tasks and different data distributions?

We added four benchmarks and a ProofWriter split nearly four times larger.

The benchmarks were named before their data were opened, each with its lexical floors (text-only baselines, Qwen3-1.7B, Table 11):

BenchmarkTwelve-feature surface floorReadoutNested probe
LogicBench0.6630.7600.798
Multi-LogiEval0.6120.9410.970
ProverQA0.8260.6830.780

On ProverQA the surface features alone beat both, so the audit flags it as it flagged ProofWriter. On LogiQA 2.0, whose NLI release mixes in items converted from multiple-choice questions, neither separates the classes (0.512 each, intervals spanning 0.5).

On the main, negation-balanced ProofWriter split, a twelve-feature surface model reads 0.943, as the paper now states (Section 3). On splits balanced within every cell of those features, four models give:

AUC, four models520 items2,000 items
Readout0.685 to 0.7660.703 to 0.807
Nested probe0.707 to 0.8040.758 to 0.844
Strongest surface modelbelow each, by a paired interval0.531 [0.507, 0.557]

The registered split-level verdict is "survives" on all four.

Q6.

Could the authors reorganize Tables 1 and 2 to distinguish the proposed method and the different baseline families more clearly, including aggregate comparisons and a more direct discussion of their practical trade-offs?

Answered with W1 above.

Q7.

Beyond the elementary property established by Lemma 1, what additional conclusions about the representations used by language models are supported by the experiments?

Answered with W6 above.

Q8.

Could the authors improve Figures 1 and 2 by including clearer legends, larger fonts, and concrete examples illustrating the task, the representations, and the motivation for the proposed method?

Answered with W1 above.

We hope the revision resolves your concerns, and we would appreciate it if you took it into account in your final score. We are happy to discuss any point further.

AuthorsOpenReview ↗

Follow-up for Reviewer oHmw

Beyond the presentation changes, we ran new experiments on the soundness concerns. A surface-balanced split removes the negation artifact you identified, and on it the readout still matches a hidden-state probe. Four further benchmarks are each reported beside their lexical floors. The revised abstract, introduction, Figure 1 and tables present these results directly. We would be glad to hear whether the revised version addresses your concerns, and if so, we would be grateful if you would consider revising your rating.

Reviewer_JwTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a logistic readout of deductive validity on six attention scalars per head, primarily from the conclusion's final row. Across 13 models, these features outperform the tested spectral summaries; six-benchmark detector comparisons use Qwen3-1.7B. Additional contributions include an invariance analysis, head ablations, and a ProofWriter negation-artifact audit. Hidden-state probes generally remain stronger and more label efficient.

Critical strengths
  1. The broad comparison and feature decomposition are informative. The readout outperforms both spectral alternatives across all 13 models in Table 2. In Table 7, entropy and maximum alone reach mean ProofWriter AUC 0.899 versus 0.703 for per-head spectra. Storage and label-budget comparisons clarify the practical tradeoffs.

  2. The artifact audit and controls are valuable. A negation feature achieves 0.927 AUC while length achieves 0.509, exposing the inadequacy of length-only controls. Grouped splits, hypothesis-only controls, theory-matched pairs, and measured best-layer selection bias strengthen the evaluation, subject to the sampling clarification below.

Critical weaknesses
  1. Permutation invariance is not isolated as the cause of spectral failure. Lemma 1 is correct but does not bound prediction across different orbits, as Section 5 acknowledges. Appendix G's permutations generally leave causal attention support and move the fixed readout row by construction. In ideal dense causal attention, an invariant rule can identify the final row by its unique full support and return its entropy/maximum. Gram spectra also discard feature-space orientation, since X and XQ have identical Gram matrices for orthogonal Q. These experiments therefore leave other losses from spectral compression unresolved.

  2. The closest baseline needs clarification. Section 4 describes Lookback Lens as a final-row ratio of premise mass to premise-plus-claim mass. The original method normalizes each mass by span length and averages ratios over the judged span (Chuang et al., Section 2.1). The omitted normalization may be a description error, but it matters for assessing the six features' added value. Gains are only 0.007 on ProofWriter and 0.004 on ProntoQA, and Lookback Lens wins on ConTRoL; paired difference intervals would help.

  3. The head-ablation conclusions exceed the evidence. Table 4 assigns REQUIRED/ABSORBED labels using ranks against ten controls. An extreme rank among eleven is not itself conventional statistical significance, and no differential effect relative to controls does not demonstrate downstream self-repair. The results support relative ablation sensitivity more directly than necessity or compensation.

Questions for authors
  1. Can the invariance explanation account for structurally identifiable final rows and the additional information lost by Gram spectra, or should the claim be narrowed to the tested summaries?

  2. Which Lookback Lens implementation produced Table 1? Please compare the original length-normalized ratio with final-row and span-averaged aggregation, reporting paired confidence intervals.

  3. How does Appendix B obtain N=240 from equal depth-by-label-by-negation cells over depths 1-5? This requires 12 examples per cell, but depth 4 has only five False-positive and six True-negated examples. Please clarify cell counts, replacement/exclusion, and unique theories.

  4. What uncertainty estimates or evidence of downstream compensation support the REQUIRED/ABSORBED interpretation beyond the rank rule?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JwTV (1/2)

Thank you for a precise review, and for finding the feature decomposition "informative" and the artifact audit "valuable". Each of your four questions led to an analysis, and the first two reshaped the central claim.

W1.

Permutation invariance is not isolated as the cause of spectral failure. Lemma 1 is correct but does not bound prediction across different orbits, as Section 5 acknowledges. Appendix G's permutations generally leave causal attention support and move the fixed readout row by construction. In ideal dense causal attention, an invariant rule can identify the final row by its unique full support and return its entropy/maximum. Gram spectra also discard feature-space orientation, since X and XQ have identical Gram matrices for orthogonal Q. These experiments therefore leave other losses from spectral compression unresolved.

We agree, and the submission's own data show it: the signal comes mostly from the conclusion's row.

ProofWriter, mean of 13 modelsAUC
Per-head spectra0.703
Spectra plus the final row's entropy and maximum0.880
That entropy and maximum alone0.899
Anchored readout0.925

Told the final row, an invariant rule recovers a median 86% of the gap per model (Table 10, Section 6, contribution 2). Your orientation point also holds: Gram spectra discard orientation, so the Gram experiment locates the loss in the summary at large (Section 5).

W2.

The closest baseline needs clarification. Section 4 describes Lookback Lens as a final-row ratio of premise mass to premise-plus-claim mass. The original method normalizes each mass by span length and averages ratios over the judged span (Chuang et al., Section 2.1). The omitted normalization may be a description error, but it matters for assessing the six features' added value. Gains are only 0.007 on ProofWriter and 0.004 on ProntoQA, and Lookback Lens wins on ConTRoL; paired difference intervals would help.

The submitted row was our own variant. Lookback Lens from its authors' code is below the readout on all six benchmarks, with or without Bonferroni correction.

Our variant summed the final row's premise and claim masses, scored in an earlier run than the anchored values beside it. Rescored together, it has the higher point estimate on two of six benchmarks, ConTRoL and ProntoQA, and both paired intervals span zero ("Summed, final row" below, Table 12). Appendix J finds the same two on the stored extraction of the submitted Table 1.

On Qwen3-1.7B we ran the authors' code in their convention, with the conclusion fed as input (the context ends at its marker). We added your 2×22 \times 2 (summed or length-normalised masses, final row or span average), the final row's entropy and maximum, and the two whole-context variants. Over these eight registered variants the readout is higher by a 95% paired interval in 31 of the 48 comparisons (22 after Bonferroni correction). It is lower in one, the whole-context final-row variant on ConTRoL.

Anchored minus (paired 95% interval)Lookback Lens, authors' codeFinal row's entropy and maximumSummed, final rowSummed, span average
ProofWriter+0.154 [+0.105, +0.207]+0.012 [+0.002, +0.025]+0.008 [-0.004, +0.019]+0.057 [+0.027, +0.089]
FOLIO+0.247 [+0.176, +0.320]-0.002 [-0.026, +0.022]+0.057 [+0.007, +0.111]+0.214 [+0.143, +0.287]
FLD+0.218 [+0.146, +0.293]+0.023 [+0.003, +0.044]+0.047 [+0.014, +0.080]+0.178 [+0.109, +0.244]
LogicNLI+0.171 [+0.102, +0.239]+0.001 [-0.020, +0.022]+0.039 [+0.003, +0.076]+0.070 [+0.009, +0.125]
ConTRoL+0.168 [+0.087, +0.247]+0.031 [-0.010, +0.074]-0.055 [-0.112, +0.004]+0.103 [+0.020, +0.185]
ProntoQA+0.116 [+0.040, +0.190]+0.020 [-0.010, +0.049]-0.007 [-0.057, +0.045]+0.103 [+0.036, +0.172]
Anchored minus (length-normalised)Final rowSpan averageFinal row, whole contextSpan average, whole context
ProofWriter+0.014 [+0.001, +0.028]+0.074 [+0.043, +0.108]-0.004 [-0.015, +0.007]+0.072 [+0.040, +0.107]
FOLIO+0.043 [-0.007, +0.097]+0.198 [+0.126, +0.270]-0.008 [-0.047, +0.032]+0.222 [+0.153, +0.291]
FLD+0.056 [+0.020, +0.095]+0.173 [+0.104, +0.243]+0.069 [+0.034, +0.104]+0.177 [+0.111, +0.242]
LogicNLI+0.040 [+0.003, +0.076]+0.124 [+0.061, +0.188]+0.020 [-0.013, +0.052]+0.145 [+0.082, +0.209]
ConTRoL-0.055 [-0.118, +0.011]+0.060 [-0.023, +0.138]-0.068 [-0.126, -0.013]+0.155 [+0.076, +0.236]
ProntoQA+0.008 [-0.046, +0.063]+0.083 [+0.014, +0.156]+0.013 [-0.039, +0.063]+0.058 [-0.010, +0.124]

The final-row variants come closest, so the signal is in which row is read.

AuthorsOpenReview ↗

Response to Reviewer JwTV (2/2)

W3.

The head-ablation conclusions exceed the evidence. Table 4 assigns REQUIRED/ABSORBED labels using ranks against ten controls. An extreme rank among eleven is not itself conventional statistical significance, and no differential effect relative to controls does not demonstrate downstream self-repair. The results support relative ablation sensitivity more directly than necessity or compensation.

We agree and removed the per-model verdicts. With ten controls the smallest attainable two-sided p is 0.18. Over the 21 models of Appendix D, non-absorbed verdicts match chance (5 observed, 3.1 expected, binomial p 0.20).

We reran with 99 random-head controls per model on 10 of the 11 models up to 8B. Qwen3.5-0.8B and Qwen3.5-2B use all 47 other attentive heads. Qwen3.5-4B did not finish within its registered 150-minute limit. Your compensation question (Q4) added a registered joint test, since other heads can compensate for a single ablated head (McGrath et al., 2023). Both tests take the rank-sum of the targets' damage ranks across models. Each is one-sided toward damage, against its exact null.

Heads zeroedControls per modelModel's own True/False AUC
Selected head, 10 models99 random headsno larger drop, confirmatory p = 0.53
Five most informative, together, the four models of the surface-balanced splits99 layer-matched random sets of fivedrop of 0.02 to 0.11 on all four, more than 66 to 87 of the 99 sets, p = 0.027
Same joint test, per modelthe sameone-sided p 0.13 to 0.34

Pooled over the four models, the test shows that a few mid-depth heads matter to the model together more than random heads at the same depths. The paper calls no single head required (Section 7.1).

Q1.

Can the invariance explanation account for structurally identifiable final rows and the additional information lost by Gram spectra, or should the claim be narrowed to the tested summaries?

Answered with W1 above.

Q2.

Which Lookback Lens implementation produced Table 1? Please compare the original length-normalized ratio with final-row and span-averaged aggregation, reporting paired confidence intervals.

Answered with W2 above.

Q3.

How does Appendix B obtain N=240 from equal depth-by-label-by-negation cells over depths 1-5? This requires 12 examples per cell, but depth 4 has only five False-positive and six True-negated examples. Please clarify cell counts, replacement/exclusion, and unique theories.

It cannot. You found a real error, and we corrected Appendix B.

Corrected
Depths1, 2, 3 and 5. Depth 4 is excluded, as its pool holds only 5 False-positive and 6 True-negated items
Items per depth x label x negation cell15, drawn without replacement (seed 42) from the raw validation rows
Unique texts235, as the raw rows repeat five (theory, statement) pairs
Theory groups210 for the 240 items. Each repeat stays within one group, so none crosses folds

Q4.

What uncertainty estimates or evidence of downstream compensation support the REQUIRED/ABSORBED interpretation beyond the rank rule?

Answered with W3 above.

We hope the revision resolves your concerns, and we would appreciate it if you took it into account in your final score. We are happy to discuss any point further.

AuthorsOpenReview ↗

Follow-up for Reviewer JwTV

We ran the decomposition you asked for, and it identifies the paper's mechanism. Given the conclusion's row, an invariant rule recovers a median 86%86\%% of the gap (Table 10). The anchored readout reaches 0.9250.925 AUC against 0.7030.703 for per-head spectra. We also reran Lookback Lens with its original normalisation and paired intervals. The central claim is now stated in terms of these results. We would be glad to hear whether this resolves your concerns, and if so, we would be grateful if you would consider revising your rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers (1/2): main results

We thank the three reviewers. Two ask what an attention readout shows that a stronger hidden-state probe does not, and the new analyses answer that. The main results:

  • The conclusion's own attention row carries the validity signal. Six named attention quantities, none from the hidden states, reach 95 to 102% of a nested hidden-state probe's AUC on four models (surface-balanced ProofWriter split).
  • On Qwen3-1.7B (registered primary) the readout beats every published attention detector run from released code on all six benchmarks of Table 1 (five are validity evidence). Against Lookback Lens (Chuang et al., 2024), run with its authors' code in their row and marker convention, the gain is +0.12 to +0.25 AUC. Each difference stays above zero after Bonferroni correction across the six.
  • Across thirteen models up to 27B it significantly beats Lookback Lens, the closest attention-map detector, in 56 of 65 comparisons on five validity benchmarks (52 after Bonferroni correction). It is never significantly worse. The twelve other models span 0.6B to 27B and five families. In seven of them, both 27B models included, it is higher on all six (Table 13, registered per model).
  • The signal lives in which row is read. Lookback Lens variants that read the conclusion's final row come close to the readout. A rule that ignores token labels recovers a median 86% of the readout's gain over per-head spectra once told that row (submission data, now contribution 2).

Validity depends on which token attends to which. A whole-matrix summary that stays the same when the tokens are relabelled does not record it (Lemma 1). Even the entropy and maximum of the conclusion's row beat every spectral summary of the whole matrix, in every model we test.

New analyses.

Surface-balanced splits. ProofWriter splits balanced within every cell of twelve surface features, on Qwen3-0.6B, Qwen3-1.7B, Gemma-3-1B and OLMo-2-1B (Section 6, Table 5):

AUC, four models520 items2,000 items
Readout0.685 to 0.7660.703 to 0.807
Nested probe0.707 to 0.8040.758 to 0.844
Strongest surface modelbelow each, by a paired interval0.531

Both keep a signal well above chance once the shortcut is removed.

Role of the readout. It shows which heads carry information about validity. Stacked with the probe, it never improves on it by a paired interval (0 of 20 cells above zero, six models, median score correlation 0.81, Section 4).

Four new benchmarks. We named them before opening their data, and report each with its lexical floors (text-only baselines, Qwen3-1.7B, Table 11):

BenchmarkTwelve-feature surface floorReadoutNested probe
LogicBench0.6630.7600.798
Multi-LogiEval0.6120.9410.970
ProverQA0.8260.6830.780

On ProverQA the surface features alone beat both readouts, so the audit flags it as it flagged ProofWriter. On LogiQA 2.0 neither separates the classes (0.512 each, intervals [0.435, 0.583] and [0.440, 0.586]). Its NLI release includes items converted from multiple-choice questions (66 of 240 carry a question stem, a question mark before the hypothesis).

Head ablations. Zeroing the readout's five most informative heads at once lowers the model's own True/False AUC by 0.02 to 0.11 on all four models. That is more than 99 layer-matched random sets of five do (registered rank-sum test across models, p = 0.027). Zeroing one selected head does not (p = 0.53, Section 7.1).

AuthorsOpenReview ↗

Response to all reviewers (2/2): corrections and changes to the PDF

Scope of the readout. One logistic regression combines six quantities per head. It uses no hidden states and no learned direction:

  • the final conclusion token's attention mass on the premises, the conclusion and the first token
  • that row's entropy and maximum
  • the premise mass averaged over the conclusion's rows.

Corrections.

  • Lookback Lens in Table 1. The submitted row was our own variant (summed final-row masses), scored in a different run from the anchored values. Rescored together, it has the higher point estimate on two of six benchmarks, where the submission said one, and both paired intervals span zero (Table 12).
  • Spectral failure. The submission blamed relabelling invariance. The data show it is mostly which row is read, and we narrowed contribution 2 and Section 5.
  • Surface floors. The bound of 0.558 (p. 4, l. 206 of the submitted version) holds only for the four features tested. A twelve-feature surface model reaches 0.943 on the main, negation-balanced ProofWriter split (Section 3).
  • Conclusion marker. The submitted text put it in the claim span (l. 255), the code in the premise span. Section 4 now states the code's convention.
  • Ported baselines. The Table 1 caption and Appendix J overstated which baselines were ported from released code. LLM-Check and LapEigvals are. Our CCS follows the authors' notebook and agrees with their released class to 1.6e-4 in score. For SAPLMA and P(True) we found no release by their authors.
  • Head ablations. We removed the per-model verdicts. With ten controls no single-model test can reach significance, and the family-wise count is consistent with chance (Section 7.1, Appendix D).
  • ProofWriter sampling. We corrected Appendix B and the artifact explanation (depths 1, 2, 3 and 5, open-world split, a generation artifact, Section 3).
  • LogicNLI is a lexical shortcut for both the readout and the probe and no longer counts as validity evidence (Section 6, Appendix I).

Changes to the PDF (Blue marks results and content added in this revision. Changes of wording are not marked.):

  • Abstract, contribution 2 and Section 4: centred on reading the conclusion's row.
  • Section 6 and Figure 5: the surface-balanced split.
  • Table 1 (Section 4): the published Lookback Lens with paired intervals.
  • Section 6 and Table 11: the new benchmarks with their floors.
  • Tables 1 and 2 and Figures 1 and 2: reorganised for readability.
  • The corrections above, in the sections they name.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.