General Comment: I think this paper contains potentially interesting research, particularly the analysis of the ProofWriter benchmark and the comparison between different attention-based representations. However, I have significant concerns about the writing, methodological presentation, novelty, and practical contribution.
The paper is difficult to understand, and the figures and tables do not sufficiently help the reader identify the main findings. Important decisions in the experimental protocol are explained mainly in supplementary material, even though they directly affect the interpretation of the results.
See below, what I think about the different sections:
Abstract and Introduction: I have several problems with the writing of this paper. I find the manuscript very difficult to follow, particularly because the connection between paragraphs is often unclear and many technical concepts are introduced without sufficient explanation. The abstract presents several findings almost as disconnected statements, making it difficult to identify the main contribution. The same happens in the introduction, where the authors do not clearly explain the task before moving to the proposed method and its results.
A clear example is the paragraph starting at line 052. The authors introduce three terms in the same paragraph, moving from "A readout is a logistic regression" to the definition of invariant summaries and then anchored readouts. I think these definitions need more explanation and a better connection with the motivation.
Another example of the presentation is Figure 1, which is referenced in the introduction before the reader has a clear understanding of the experimental setup. Why use connected lines in panel (b) when the horizontal axis represents different types of summaries rather than a continuous variable? What do the numerous grey lines represent? Although some information is included in the caption, the reader needs to make considerable effort to understand the figure. I feel the authors assume that the reader already understands several aspects of the methodology that have not yet been introduced.
Figure 2: The setup is also difficult to follow. The authors introduce many architectures, datasets, representation types, and experimental decisions in a very compressed section. I find the analysis of the negation artifact in ProofWriter interesting. Showing that a simple lexical feature can predict the labels with high accuracy is a relevant finding. However, Figure 2 is difficult to read and does not help much in understanding the problem. Why not include a concrete example showing how negation is correlated with the ground-truth label?. The authors explain the resampling procedure in the appendices, but I think the main paper should provide a clearer explanation of how the datasets are modified and why these modifications are necessary for the subsequent experiments.
Experimental Protocol: One of my main concerns is the selection of the best-performing layer for the hidden-state probes. The authors state that the results are reported at the best layer, but they do not sufficiently explain the selection procedure in the main paper. Appendix E clarifies that the best layer is selected using the same out-of-fold AUC results that are subsequently reported. This introduces optimistic selection bias because the evaluation results are used to select the configuration.
I understand that the authors acknowledge this limitation, quantify its effect, and that the bias favors the hidden-state probe rather than the proposed method. However, why use this procedure for a direct comparison instead of selecting the layer within the training folds?. I think this is an important methodological decision that should be discussed directly in the main paper. A reader should not need to go to the appendix to understand the exact implications of the evaluation protocol.
Anchored Readouts and Mathematical Contribution: The proposed method uses six attention statistics per head to predict deductive validity. The idea is understandable once the formulation is explained, but I find it difficult to identify what is fundamentally new compared with existing attention-based readouts, such as Lookback Lens (Chuang et al., 2024) and LapEigvals (Binkowski et al., 2025). Lookback Lens already uses attention ratios from individual heads to train a classifier, while LapEigvals uses spectral features extracted from attention matrices. Moreover, the authors acknowledge that the Lookback Lens ratio can be constructed using two of the six proposed statistics. So, what is the fundamental novelty of this method? Is it the selection of the conclusion's attention row, the combination of existing attention statistics, or the application to deductive validity? I understand that these are methodological differences, but I think the authors should better explain what new scientific understanding their approach provides beyond existing attention-based methods.
- Chuang et al. (2024). Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps. EMNLP 2024.
- Binkowski et al. (2025). Hallucination Detection in LLMs Using Spectral Features of Attention Maps. EMNLP 2025.
Results and Tables: Tables 1 and 2 are particularly difficult to read. The proposed method is not easy to identify among the different baselines, and the abbreviations and large number of columns make direct comparisons unnecessarily complicated. The tables would benefit from better organization and clearer aggregate comparisons.
Another concern is the actual performance of the proposed method. In Table 1, the anchored readout performs below SAPLMA on all six benchmarks, while P(True) also performs better on some tasks. I understand that the authors focus on attention-only methods rather than hidden-state probing, but I think this makes the practical motivation even more important. Why should we prefer the proposed approach when conventional hidden-state probes achieve better results?
The paper demonstrates improvements over several attention-based spectral methods, but I am not convinced that it sufficiently establishes the importance of this improvement compared with stronger existing methods. Furthermore, the evaluations use relatively small datasets, with 240 examples for ProofWriter and 200 for FOLIO in the main analysis. Although the number of evaluated architectures is extensive, the limited size of the reasoning benchmarks raises questions about the generalization of the findings.
Computational Cost and Practical Application: I also find the discussion of computational efficiency confusing. The authors discuss storage, inference time, and label requirements, but the relationship between these measurements and the figures is not sufficiently clear. For example, Figure 3 evaluates the performance of the methods with different numbers of training labels, but it does not directly explain the computational costs discussed in the same section. More importantly, the proposed method does not demonstrate a clear computational advantage over hidden-state probing. On the anchor model, the anchored readout takes approximately 49 ms compared with 46 ms for the probe path, while also requiring more labeled examples to reach comparable performance.
The authors motivate the method as useful when attention information is available but hidden states are not. However, I would like a stronger explanation of how realistic this scenario is and what practical advantage the method provides when both representations are accessible.
Head Analysis and Conclusions: The head-selection and ablation experiments are interesting, but the interpretation of the results could be clearer. The authors report that removing the selected head produces effects similar to random-head ablations in 10 of 13 models. This is relevant because it limits what can be concluded about the functional importance of the selected heads. The ability to predict validity using attention information does not necessarily imply that those heads are causally responsible for the model's reasoning.
Response to Reviewer x4gq
Thank you, and for calling the negation artifact "valuable". The revision extends it to a surface-balanced split and four further benchmarks.
W1.
The readout's value is explanatory, and the paper no longer argues a deployment advantage (Section 9). Per head, it shows which of six named attention quantities carry information about validity, with no hidden states and no learned direction. On the surface-balanced ProofWriter split it reaches 95 to 102% of a hidden-state probe's AUC.
W2.
We agree, and Section 9 now states the scope. The paper studies the validity of a given premise-conclusion pair and does not claim hallucination detection in generated text.
On realism, ConTRoL is expert-written reasoning over real passages, news and examination text. Four further benchmarks, named before their data were opened, enter with their twelve-feature surface floors (text-only baselines, Table 11). On Qwen3-1.7B:
LogiQA 2.0 sits at chance for both readouts. We also applied the readout fitted on ProofWriter, unchanged, to these four benchmarks on five models. It matches the probe in 12 of 20 cells, exceeds it in 1 and trails it in 7 (Table 15).
Q1.
About one part in 5,000 of the forward pass, by our count.
Fused kernels do not return attention weights, so the readout recomputes the conclusion tokens' attention rows at each attentive layer from the cached queries and keys. Five of the six quantities need the final row alone. We counted from the Qwen3-1.7B configuration (28 layers, 16 query heads of dimension 128) and our span code (scripts/experiments/overhead_count.py in our repository):
Section 4 gives our pipeline's measured times on the same model, 49 ms per item for the anchored pass and 46 ms for the probe path.
We hope the revision resolves your concerns, and we would appreciate it if you took it into account in your final score. We are happy to discuss any point further.