acceptodds
Under review as a conference paper at ICLR 2027

Query–Key Ensembles for Reasoning Verification

Abstract

Large language models typically generate reasoning chains that are then judged from the outside, by solvers, reranking heuristics, or trained reward models. We ask whether the model has already formed a judgment of its own. Given a generated solution, we read which of two explicit verdicts, correct or erroneous, the model’s attention heads prefer, using raw query–key (QK) inner products taken before Q/K normalization, rotary position encoding, and the softmax. Building on QK readouts for multiple-choice selection and logical consistency, we turn these head-wise preferences into a verifier of complete chain-of-thought (CoT) solutions without updating the model. A calibration-free rule, CF-QK, weights each head by its agreement across read positions and its responsiveness across an unlabeled batch; a linear readout, L2-QK, needs only 50 labeled solutions. Across Qwen3 (8B–32B) and DeepSeek-R1-Distill-Llama-8B on MATH-500 and HLE-1/4, these ensembles outperform the model’s own verdict logits, solution likelihood, and prompted selfverification in most settings. Without labels, CF-QK raises balanced accuracy over verdict logits by up to 18 points. Across ten 50-label calibration splits, L2-QK gains 4–16 points over calibrated logits on every model and dataset. The fitted rules transfer without target labels to GSM8K and SVAMP, and each verdict costs one forward pass. Matched edits to solution content and activation patching show that the scores respond to the reasoning itself and that the selected heads shape the model’s verdict. Complementary experiments study multiple-choice selection with or without a “think-first” CoT preface.

Primary area:
interpretability and explainable AI
Keywords:
query-key alignment, attention heads, reasoning verification, chain-of-thought, large language models, label-free verification, head ensembles, activation patching, counterfactual analysis, mechanistic interpretability, self-verification, process reward models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TTrKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether query–key (QK) representations in the large language models contain useful signals for verifying the correctness of generated reasoning solutions. The authors propose extracting raw QK inner-product margins between two explicit verdict options, indicating that a candidate solution is either correct or contains an error. The paper proposes two main verification approaches: CF-QK and L2-QK, and conducts experiments on Qwen3 models and DeepSeek-R1-Distill-Llama-8B to demonstrate improvements over verdict logits and other scalar verification baselines on MATH-500 and an adapted HLE benchmark. Overall, the paper presents an interesting approach to extracting verification signals from frozen model representations, supported by reasonably extensive empirical evaluation.

Critical strengths
  1. A simple and effective verification approach with minimal supervision. The proposed QK-based methods is simple, without fine-tuning or training the models. The experiments also demonstrate the effectiveness of the methods.
  2. Careful empirical analysis beyond benchmark-level accuracy. The paper provides analyses that strengthen the empirical contribution in different aspects.
Critical weaknesses

The main weakness of this paper lies in its writing, where the sentences and logical flow are extremely difficult to follow. After reading this paper, I'm pretty sure that it was written almost entirely by AI, with little careful human reviewing and polishing. The reasons are as follows:

  1. Terrible introduction. In the introduction, the background and motivation are not clearly explained, making the paper extremely difficult to follow. Specifically, the first paragraph of the introduction does not clearly explain what "evaluating generated mathematical solutions" actually means, while the second paragraph abruptly shifts to QK alignment without any proper transition or background explanation. Such a logical progression makes the paper very frustrating to read. Furthermore, the authors do not provide a clear explanation of their proposed method before directly moving to the contributions. The standalone "1.1 Contributions" subsection also appears very weird without a subsequent "1.2".
  2. Lack of clear design principles in the methodology. When introducing the proposed method, the authors provide little explanation of the underlying design principles and motivations, instead directly piling up mathematical formulas. Moreover, the sentences are unusually awkward and rigid, making the methodology difficult to understand.
  3. Extremely disorganized analysis of experimental results. In Sections 3, 6, and 7, the analysis contains a large number of raw numerical results, frequently interspersed with references to the appendix, making it difficult for readers to focus on and reflect upon the experimental findings. Furthermore, the paper is filled with numerical results but lacks sufficient summarization and synthesis of the general conclusions that can be drawn from them.

Overall, I believe the writing of this paper falls below the acceptable standard for a research paper. In my view, an academic paper should prioritize a clear and systematic presentation of the research problem and limitations of existing approaches, the core principles underlying the proposed methodology and experimental design, and the evidence supporting the conclusions. Rather than piecing together various facts and results through extremely disorganized and awkward sentences.

Questions for authors

First, regarding my assessment that this paper was written almost entirely by AI, with little careful human review or polishing, I would like the authors to honestly answer the following questions:

  1. What's the proportion of this paper has actually been carefully reviewed by humans? If the proportion is greater than 0, please specify exactly which parts of the paper have undergone human review.
  2. Were humans sufficiently involved in conducting the experiments in this paper, or were the experiments entirely carried out by AI?

Second, I have the following concerns regarding the paper itself. I hope the authors can carefully revisit their own work and the manuscript, reflect on these issues in depth, and provide their responses.

  1. The title and task formulation of the paper focus on reasoning verification. However, the main analysis and experiments seem to primarily evaluate final answer correctness. These two concepts are fundamentally different. Could the authors clarify this gap and explain to what extent the proposed method is actually performing reasoning verification rather than outcome verification?
  2. The evaluation is conducted on a relatively limited set of models, which may not fully reflect the capabilities of current frontier reasoning models. Could the authors include experiments on more recent and stronger models (e.g., Qwen3.5)?
  3. Could the authors evaluate the proposed analysis and method on non-mathematical reasoning benchmarks? It would be valuable to understand whether the observed QK verification signals and the proposed methods generalize beyond mathematical reasoning tasks.
  4. The L2-QK method uses only 50 calibration examples. Given that the approach operates over a large number of attention heads and requires learning an ensemble of QK features, it is unclear whether such a small calibration set is sufficient to obtain stable and reliable estimates. Could the authors provide further analysis or discussion regarding calibration data requirements, robustness, and scaling behavior?
8 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We thank the reviewer for the detailed feedback. We first address the questions about manuscript preparation and the conduct of the experiments, then explain the revisions and additional results. We have uploaded the revised manuscript to OpenReview. We have not changed the proposed method, the original experimental protocols, or any of the original main results. The revisions improve the exposition and add separate, supplementary experiments.

Q1: "What proportion of the manuscript was carefully reviewed by humans?"

A: We wrote the manuscript and carefully reviewed every part of it ourselves (100%), including the main text, methodology, experimental analyses, tables, figures, and appendices.
As documented in our AI Use Statement, generative AI tools assisted with editing parts of the text, identifying related literature, formatting references, and suggesting alternative titles. We checked the scientific content and made all final editorial decisions. We also rewrote and carefully checked ourselves the introduction, methodology, and experimental discussion in the updated manuscript.

Q2: "Were humans sufficiently involved in conducting the experiments?"

A: We designed, implemented, and ran the experiments ourselves. We made all methodological decisions, established the experimental protocols, and analyzed, validated, and interpreted the results. For example, we manually audited 50 randomly selected Qwen3-32B MATH correctness labels.
In conducting the experiments, language models were used only as experimental subjects and, where documented by the paper, as evaluation tools. We conducted all the research ourselves.

W1: Introduction and motivation.

A: Section 1 (pp. 1–2) now begins with the practical question: when a model generates a candidate solution without access to the reference answer, how should we assess whether to accept it or seek further verification? We then ask whether the frozen model contains correctness information not fully reflected in its output verdict. This connects earlier QK-based answer selection to our extension to complete generated solutions.
Before the contributions, we explain the verification pipeline: the verifier receives a problem, a candidate solution, and two explicit verdict statements; each attention head expresses a relative QK preference between them; CF-QK aggregates these preferences without correctness labels, while L2-QK learns a regularized readout from a small labeled set. We removed the standalone Section 1.1 and placed the contributions at the end of the introduction.

W2: Methodological design principles.

A: Section 4 (pp. 3–5) now opens with the actual verification prompt and highlighted query/key read positions. Section 4.1 explains the signed head-level margin before introducing its mathematical definition. Section 4.2 motivates the aggregation rule through a concrete failure mode: a head can agree across read positions yet predict the same verdict for nearly every solution. CF-QK therefore considers both within-solution agreement and variation across an unlabeled reference batch. Section 4.3 explains feature selection, L2 regularization, validation, and threshold selection under the 30/10/10 source-label split. Each mathematical component is now accompanied by its rationale.

AuthorsOpenReview ↗

Rebuttal 2/3

W3: Analysis of experimental results.

A: Sections 3, 6, and 7 (pp. 3, 6–9) now lead with the experimental questions and conclusions. Section 3 follows a multiple-choice example showing how a head’s preference can differ from the model’s output. Section 6 separates verification without labels, verification with 50 labels, and transfer, including model-dependent exceptions. Section 7 explains what each analysis establishes. In particular, ranking an intact reasoning trace above a corrupted one is different from classifying both correctly at a fixed threshold. Detailed comparisons remain in the tables and appendices.

Q3: Does the method verify reasoning or merely outcomes?

A: The primary benchmark labels assess final-answer correctness, while the verifier reads the complete candidate solution (§5). We test sensitivity to intermediate reasoning separately in §7.3 and Appendices C.1–C.2. First, we modify an intermediate equality in 40 MATH solutions per model while preserving the problem and final answer. One-pass CF-QK ranks the intact solution above the edited one in 70% of pairs for Qwen3-8B and 85% for Qwen3-14B. At the fixed zero threshold, classification accuracy on the intact and edited solutions is 51.25% for Qwen3-8B and 58.75% for Qwen3-14B. Second, with the final answer again held fixed, we test 24 erroneous reasoning edits and 16 validity-preserving rewrites. On both models, CF-QK accepts the original and rejects the edit in all 24 error pairs, and accepts both versions in all 16 valid pairs. The results also hold when we remove the explicit final-answer declaration. Direct verdict logits give the same counts. These paired tests show that QK scores can respond to a change in the reasoning even when the final answer stays the same.

Q4: Can you evaluate newer and stronger models, such as Qwen3.5?

A: To address the request for stronger models, we extended our evaluation to Qwen3-Next-80B on MATH and HLE. Like Qwen3.5, this model combines Gated DeltaNet blocks with full-attention layers. It has 80B total parameters. Our readout compares a query at the verdict position with keys at the two verdict options. We evaluate all 192 heads in its 12 full-attention layers.

Table 23. Qwen3-Next-80B verification. AUROC and BA (%). Bold marks the highest value in each metric column.

MethodMATH AUROCMATH BAHLE AUROCHLE BA
CF-QK, batch reference91.7083.4685.2781.72
CF-QK, source-frozen89.9285.7781.3176.19
L2-QK94.2891.6288.0184.49
One-pass logits95.6474.7086.1682.01
Two-pass logits96.1274.1085.9381.30

On MATH, L2-QK achieves the highest BA among the evaluated methods, improving over one-pass logits by 16.92 percentage points. On HLE, L2-QK achieves the highest AUROC and BA, exceeding one-pass logits by 1.85 and 2.48 percentage points, respectively. For the camera-ready version, we will adapt the QK readout to Qwen 3.5 specific hybrid delta-net attention architecture.

AuthorsOpenReview ↗

Rebuttal 3/3

Q5: Can you evaluate the method on non-mathematical reasoning tasks?

A: We added two benchmarks. On LiveCodeBench (release_v6), a generated program is correct if it passes all released tests under the official checker. On ProofWriter (open-world setting, depth 5), the model generates a response ending in True, False, or Unknown. We check that conclusion against the published answer.

We evaluated Qwen3-8B and Qwen3-14B. For each model and benchmark, all methods scored the same 450 held-out responses. A separate set of 50 task-specific examples supplied the unlabeled reference for CF-QK and the 30/10/10 fitting, validation, and threshold-selection split for L2-QK. No test labels were used for fitting. CF-QK weights were frozen before evaluation.

Table 26. LiveCodeBench verification. Entries are AUROC / balanced accuracy (%). Bold marks the best value for each metric within a model.

MethodQwen3-8BQwen3-14B
CF-QK, one-pass81.06 / 73.6175.58 / 71.01
CF-QK, counterbalanced82.38 / 72.8779.99 / 70.12
L2-QK85.70 / 80.0984.16 / 73.05
Verdict logits, one-pass84.34 / 67.0483.29 / 69.09
Verdict logits, two-pass84.83 / 74.3583.94 / 71.84

Table 27. ProofWriter verification. Entries are AUROC / balanced accuracy (%) on the same 450 test responses per model.

MethodQwen3-8BQwen3-14B
CF-QK, one-pass64.83 / 61.2657.19 / 56.60
CF-QK, counterbalanced57.97 / 54.3758.57 / 55.87
L2-QK76.14 / 68.9860.69 / 58.58
Verdict logits, one-pass39.97 / 49.7559.16 / 50.30
Verdict logits, two-pass46.96 / 50.0759.51 / 50.33

Both QK methods use source examples from the task on which they are evaluated. These in-domain experiments show that QK scores carry information about correctness in code generation and deductive reasoning.

Q6: Are 50 examples sufficient for a readout over many attention heads?

A: In our experiments, 50 labeled source examples were sufficient to fit a useful L2-QK readout. We used 30 examples to select a pool of 128 heads and fit an L2-regularized classifier, 10 to select the regularization strength, and 10 to choose the decision threshold (§4.3). The underlying language model stayed frozen.

Repeated calibration. We repeated the full procedure on ten stratified source splits per model, averaging three search seeds within each split. Table 2 (§6.2) reports mean balanced accuracy across the eight model–dataset comparisons. Table 14 (Appendix C.5) reports the paired differences from verdict logits calibrated with the same label budget.

Label-budget scaling. For Qwen3-8B, we varied the source budget from 50 to 100, 200, and 400 labels while keeping the same 100 held-out MATH solutions. As the budget increased, mean balanced accuracy rose and the selected head pools overlapped more across splits (Appendix C.6, Figure 4).

Threshold sensitivity. We also held the original Qwen3-32B classifier fixed and resampled its ten threshold-calibration examples 1,000 times using stratified sampling. The resulting standard deviation in balanced accuracy was 7.30 percentage points on MATH and 5.77 on HLE. AUROC stayed constant because the underlying scores and their ranking were fixed.

Reviewer_xNPBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether Large Language Models (LLMs) internally form judgments about their own generated reasoning chains before external evaluation. By extracting raw Query-Key (QK) inner products from attention heads—prior to Q/K normalization, rotary position encoding, and softmax—the authors measure the model's internal preference between explicit "correct" and "erroneous" verdicts. They propose using these head-wise preferences to verify complete Chain-of-Thought (CoT) solutions without updating the model's weights. Two main ensemble methods are introduced: a calibration-free rule (CF-QK) that weights heads based on agreement and responsiveness using unlabeled data, and a linear readout (L2-QK) that requires only a tiny calibration set (50 labeled examples).

Critical strengths

The paper introduces a highly efficient, training-free reasoning verification mechanism by extracting raw Query-Key (QK) inner products from a frozen model via a single forward pass. It eliminates the need for reward models or fine-tuning.

Critical weaknesses

Despite strong performance on math reasoning, the limited scope of datasets and model scales weakens the method's generalizability. First, experiments max out at 32B models and focus narrowly on math and adapted multiple-choice tasks, leaving effectiveness on harder, open-ended tasks (e.g., code generation, complex logic) unverified. Second, for larger models (70B+), it remains empirically unclear whether the "internal judgment" mechanism still concentrates in later layers and if the unsupervised CF-QK heuristics remain robust.

Questions for authors

See Weaknesses

3 comments

We thank the reviewer for the constructive feedback.

W1: “First, experiments max out at 32B models and focus narrowly on math and adapted multiple-choice tasks, leaving effectiveness on harder, open-ended tasks (e.g., code generation, complex logic) unverified.”

A: We thank the reviewer for suggesting tests beyond mathematics. We added competitive programming on LiveCodeBench and deductive reasoning on ProofWriter (OWA, depth 5), using Qwen3-8B and Qwen3-14B. Coding correctness requires passing all released tests with the official checker. ProofWriter responses are generated freely and end with True, False, or Unknown; correctness is determined by the published answer.

For each benchmark and model, all verifiers score the same 450 test responses. A separate set of 50 examples supplies the unlabeled reference for CF-QK and the 30/10/10 fitting, validation, and threshold-calibration split for L2-QK.

Table 26. LiveCodeBench verification. AUROC / BA (%); bold marks the best value for each metric within a model.

MethodQwen3-8BQwen3-14B
CF-QK, one-pass81.06 / 73.6175.58 / 71.01
CF-QK, counterbalanced82.38 / 72.8779.99 / 70.12
L2-QK85.70 / 80.0984.16 / 73.05
Verdict logits, one-pass84.34 / 67.0483.29 / 69.09
Verdict logits, two-pass84.83 / 74.3583.94 / 71.84

Table 27. ProofWriter verification. AUROC / BA (%) on the same 450 test responses per model.

MethodQwen3-8BQwen3-14B
CF-QK, one-pass64.83 / 61.2657.19 / 56.60
CF-QK, counterbalanced57.97 / 54.3758.57 / 55.87
L2-QK76.14 / 68.9860.69 / 58.58
Verdict logits, one-pass39.97 / 49.7559.16 / 50.30
Verdict logits, two-pass46.96 / 50.0759.51 / 50.33

CF-QK weights are source-frozen. Counterbalanced CF-QK blends the one-pass and both-mapping scores. L2-QK has the highest AUROC and BA point estimates in all four settings.

W2: “Second, for larger models (70B+), it remains empirically unclear whether the ‘internal judgment’ mechanism still concentrates in later layers and if the unsupervised CF-QK heuristics remain robust.”

A: We extended the evaluation to Qwen3-Next-80B on MATH and HLE. The results below show that QK features remain predictive at this scale. All values are percentages; BA denotes balanced accuracy.

Table 23. Qwen3-Next-80B verification. AUROC and BA (%).

MethodMATH AUROCMATH BAHLE AUROCHLE BA
CF-QK, batch reference91.7083.4685.2781.72
CF-QK, source-frozen89.9285.7781.3176.19
L2-QK94.2891.6288.0184.49
One-pass logits95.6474.7086.1682.01
Two-pass logits96.1274.1085.9381.30

On MATH, L2-QK achieves the highest BA among the evaluated methods, improving over one-pass logits by 16.92 percentage points, while two-pass logits achieve the highest AUROC. On HLE, L2-QK achieves the highest AUROC and BA, exceeding one-pass logits by 1.85 and 2.48 percentage points, respectively.

Q: For 70B+ models, does the internal verification signal still concentrate in later layers?

A: Yes, within the full-attention layers of Qwen3-Next-80B. We evaluated all 192 available heads across its 12 full-attention layers before applying CF-QK’s later-layer restriction. The table counts heads with individual AUROC ≥ 80%; each half contains 96 heads.

Table 24. Layer concentration in Qwen3-Next-80B. Heads with individual AUROC ≥ 80%.

SignalDatasetLayers 1–24Layers 25–48
Raw QK marginMATH17/9652/96
Raw QK marginHLE5/9646/96
CF-style head voteMATH6/9646/96
CF-style head voteHLE2/9631/96

The strongest raw-QK head occurs in layer 36 on MATH and layer 44 on HLE, out of 48 layers. Strong heads also occur in layer 8, so the signal is concentrated in later layers but is not exclusive to them.

Q: Do the unsupervised CF-QK heuristics remain robust at larger model scales?

A: CF-QK retains substantial predictive performance at 80B with source-frozen weights. On MATH, source-frozen CF-QK achieves 89.92% AUROC and 85.77% balanced accuracy; on HLE, it achieves 81.31% AUROC and 76.19% balanced accuracy. Table 25 quantifies the effect of changing from evaluation-batch weighting to source-frozen weighting.

Table 25. CF-QK reference sensitivity at 80B.

DatasetΔ AUROCΔ BA
MATH−1.78+2.31
HLE−3.96−5.53

We believe that such sensibility can not be named as negligible, but it is acceptable for downstream performance. Furthermore, the source-frozen rule maintains substantial discrimination and balanced accuracy on both benchmarks. Thus, the results support robustness of the CF-QK aggregation principle to increased model scale.

AuthorsOpenReview ↗

Concluding remarks

We hope these additional experiments address the reviewer's concerns regarding generalization beyond mathematics and scalability to 70B+ models. We welcome any further questions or clarifications and would appreciate reconsideration of the overall rating in light of the new evidence.

AuthorsOpenReview ↗

Request for Feedback and Reassessment

We would like to draw your attention to the revised manuscript and our detailed rebuttal, including new evaluations on Qwen3-Next-80B and non-mathematical reasoning benchmarks (LiveCodeBench and ProofWriter), directly addressing your questions about scalability and generalization.

As the discussion period is approaching its end, we would greatly appreciate your consideration of these results and reconsideration of your overall rating and recommendation. Please let us know if any concerns remain unresolved.

Reviewer_4HVWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a frozen LLM already "knows" if its own chain-of-thought (CoT) solution is correct, and proposes to read that judgment directly from attention. Given a problem and candidate solution followed by two explicit verdict options ("A. correct" / "B. contains an error") and a Verdict: prompt, the method reads raw query-key (QK) inner products taken before Q/K normalization, rotary encoding, and softmax, and computes, per head, a signed margin d_h = q^T(k_correct - k_error). These head-wise preferences become a verifier in two regimes: (1) CF-QK, a calibration-free rule weighting each head by agreement across read positions, coverage, and a "responsiveness" factor 4p(1-p) estimated on an unlabeled batch, decided at a fixed zero threshold; and (2) L2-QK / SA-QK, lightweight supervised readouts (linear classifier or simulated-annealing subset search) fit on only 50 (or 400) labeled solutions and frozen for transfer. Across Qwen3-8B/14B/32B and DeepSeek-R1-Distill-Llama-8B on MATH-500 and an author-constructed four-choice HLE adaptation (HLE-1/4), the ensembles outperform the model's own verdict logits, solution likelihood, and prompted self-verification in most settings, transfer label-free to GSM8K/SVAMP, and cost one forward pass. Matched counterfactuals and activation patching are offered as evidence that the scores respond to reasoning content and that the selected heads causally shape the verdict.

Critical strengths
  1. Extending the QK-readout primitive from prompt-supplied, single-head multiple-choice / logical-consistency selection (Tulchinskii et al., 2024; 2025) to solution-level verification of a self-generated, heterogeneous reasoning trace, scored without a reference answer, is a genuine and non-trivial step. The commensurability argument in Appendix G (reading before rotary encoding removes the positional rotation that would otherwise differ between the two option keys, making head-wise signed votes aggregable) is the cleanest technical insight, and CF-QK's fixed zero threshold avoids per-dataset tuning of the headline method.
  2. The design is careful about what is and is not supervised, with disjoint search/validation/threshold partitions, frozen transfer of head indices/scales/thresholds, ten stratified 50-label splits times three seeds, and explicit separation of conditional-bootstrap SD (Table 1) from reference-batch SD (Table 8) from calibration-split SD (Fig. 4). The paired-difference analysis (Table 14; L2 wins 8-10 of 10 splits) is more trustworthy than the mean plus/minus SD comparison, and the weighting ablation (Table 6) convincingly isolates responsiveness as load-bearing (removing it collapses several MATH settings to roughly 50% BAcc).
  3. A verifier needing zero-to-50 labels, one forward pass, and no model updates is attractive relative to trained PRMs/outcome verifiers; the signal survives with an independent 32-50-example reference batch (Table 8) and transfers label-free across datasets (Table 3), lifting it above a pure in-distribution probing result.
  4. The paper reports where CF-QK loses (Qwen3-14B HLE), the near-chance threshold-level counterfactual accuracy (Sec. 7.3), the negative Q-only patching effect and sign-flip under an alternative control (Sec. 7.4, C.3), and the Qwen3-32B threshold sensitivity (Sec. C.5). Exact selected (layer, head) pairs, seeds, thresholds, prompts, and the deterministic dataset construction are given.
Critical weaknesses
  1. The headline tables pit QK only against self-produced signals (verdict logits, indirect logits, likelihood, prompted self-verification), baselines the paper's own related work labels weak (Huang et al., 2024). The one trained process reward model, Qwen2.5-Math-PRM-7B ("PRM-min"), appears only on transfer datasets (GSM8K/SVAMP, Table 10), never on the primary MATH/HLE comparison, and runs under a 2,048-token / 32-step budget while QK uses 4,096 tokens (Sec. 5, A.6). The cited label-free internal-signal baselines, Contrast-Consistent Search (Burns et al., 2022) and attention-entropy correctness signals (Ostmeier et al., 2026; Chuang et al., 2024), are discussed but never run head-to-head on the same candidates. The natural competitors are therefore either absent from the main benchmark, budget-mismatched, or untested.
  2. Default CF-QK-R1P computes head weights on the full unlabeled evaluation batch, including the scored examples (Sec. A.5), while the pointwise logit baselines see one example at a time; the authors measure this (Table 8: within 2.26 points for Qwen3, but about 7 points for DeepSeek-Llama-8B on MATH, 75.41 to 68.43), yet the inductive numbers are not the headline. Separately, the "4-16 points over calibrated logits" claim (abstract; Table 2/14, across all models and datasets) is, on HLE specifically, inflated because the logit baseline's MATH-fitted threshold transfers badly to HLE (calibrated verdict logits fall to 54% BAcc, Table 5); against uncalibrated logits the margin shrinks or reverses (Qwen3-14B: uncalibrated 80.48% vs L2 74.99%, Sec. 6.2). Both points belong in the abstract, not mid-section caveats.
  3. Sec. C.5 shows that changing only the Qwen3-32B threshold (0.3187 to 0.4906) moves BAcc 73.43 to 88.85 (MATH) and 68.19 to 80.00 (HLE). With 10 threshold-fitting examples, a roughly 15-point swing is threshold noise alone, which undermines every BAcc number fit on the 30/10/10 budget. The Sec. 7.3 result (CF-QK separating intact vs corrupted MATH traces at only 51.25% for 8B and 58.75% for 14B at its own zero threshold) is the same issue: the ranking responds to reasoning, the deployed decision barely does. AUROC (threshold-free) should be the primary metric, with BAcc reported alongside.
  4. Activation patching (Sec. 7.4, C.3) compares selected vs random heads without matching key norm or other activation statistics (stated explicitly), so the larger K/Q+K effects may reflect larger-magnitude keys rather than verdict-specific content; the negative Q-only MATH effect (Table 13) and the -0.1557 sign-flip under an alternative control reinforce this, and the analysis covers only Qwen3-8B. The aggregation rule is also a hand-designed heuristic the paper admits are "heuristics ... [that] do not estimate head accuracy" (Sec. 4.2); because p_h is the batch fraction of "correct" votes, on a class-imbalanced batch (HLE) a head that faithfully tracks correctness is down-weighted by 4p_h(1-p_h), conflating a constant head with an informative head on a skewed batch and plausibly contributing to the weaker HLE behavior.
  5. Q/K read positions are hard-coded; the paper concedes "some dependence on prompt format and tokenization likely remains" yet provides no cross-prompt-format robustness experiment. The method is white-box only (needs raw Q/K projections, unusable for API models) with no deployment-feasibility discussion. Calibrated L2-QK/SA-QK transfer is shown only across datasets within one model (MATH to GSM8K/SVAMP), never across model families. And where CF-QK loses (Qwen3-14B HLE, 5.69 points below verdict logits, Sec. 6.1), no failure-mode analysis is given.
Questions for authors
  1. Please report the fully inductive (source-frozen) CF-QK numbers as the headline, not only in Table 8, and clarify which number the abstract's "up to 18 points" refers to. How much of the Table 1 advantage is transductive for each model (DeepSeek's roughly 7-point gap on MATH is concerning)?
  2. Can you add Qwen2.5-Math-PRM-7B (and ideally a strong external LLM judge, not the model's own prompted self-check) to the main MATH/HLE tables under a token budget matched to QK's 4,096, and add a head-to-head comparison to CCS (Burns et al., 2022) and an attention-entropy verifier (Ostmeier et al., 2026) on the same candidates? Does the advantage survive?
  3. Given the 73 to 89 BAcc swing from one threshold change on Qwen3-32B, will you make AUROC the primary metric and report BAcc with an explicit threshold-selection-variance term?
  4. Can you add a key-norm-matched (or magnitude-ranked) random-head control, extend activation patching to at least one larger model (14B), and reconcile the negative Q-only MATH effect and the -0.1557 sign-flip? Relatedly, does labeling MATH correctness with a Qwen3-32B judge introduce circularity for the Qwen3-32B verifier results?
  5. Can you provide (a) an ablation measuring CF-QK and L2-QK sensitivity to small rewrites of the verdict prompt and to alternative token anchors, (b) a cross-model-family weight-transfer result (e.g., Qwen3-8B readout applied to DeepSeek), and (c) a qualitative failure analysis of the Qwen3-14B HLE setting where CF-QK falls below verdict logits?
2 comments

I would like to thank the reviewers for carefully respond to my review comments. Since my rating is already positive, I will keep my rating.

We thank the reviewer again for considering our detailed responses.

We have uploaded a revised manuscript incorporating the methodological clarifications and additional experimental evidence presented in the rebuttal.

We have addressed every point raised in the review with substantial additional evidence, and your follow-up did not identify any remaining concerns. We therefore believe the unchanged rating does not fully reflect the strengthened scientific evidence and respectfully urge you to reconsider and update your assessment accordingly.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal summary and request for reviewer reassessment

Dear Area Chair,

We have provided detailed point-by-point responses to all three reviews and uploaded a revised manuscript. We have addressed every concern raised with additional experiments, methodological clarifications, robustness analyses, and improvements to the presentation. The proposed method, original experimental protocols, and original main results have not changed.

The principal additions are:

  • Reviewer 4HVW — Stronger baselines and robustness. We added matched PRM, independent LLM judge, CCS, and HeadEntropy comparisons; clarified source-frozen versus transductive evaluation; quantified calibration and threshold sensitivity; and conducted further prompt, anchor, reference-batch, cross-model-transfer, and activation-patching experiments. We also explicitly documented negative findings and limitations.

  • Reviewer xNPB — Generalization and scale. We extended the evaluation to Qwen3-Next-80B, investigated the distribution of informative attention heads across layers, and added non-mathematical evaluations on LiveCodeBench and ProofWriter.

  • Reviewer TTrK — Presentation, scientific scope, and calibration. We revised the introduction, methodology, and experimental discussion, answered the questions about manuscript preparation and experimental conduct, clarified the distinction between outcome correctness and reasoning sensitivity, and provided further evidence from newer architectures, non-mathematical benchmarks, and calibration-stability analyses.

Reviewer 4HVW has acknowledged our detailed responses and confirmed that their rating remains positive. We appreciate this feedback.

Reviewers xNPB and TTrK have not yet provided follow-up assessments of our responses and the revised manuscript. As we have addressed their specific concerns with substantial new evidence and completed revisions, we would appreciate your assistance in inviting them to review the rebuttal and reconsider their ratings and recommendations accordingly.

We remain available to respond to any specific issues they believe are still outstanding.

Thank you for your consideration.

The Authors

AuthorsOpenReview ↗

Update on Reviewer TTrK's assessment

Dear Area Chair,

Following our earlier rebuttal summary, Reviewer TTrK has increased the soundness score after considering our responses and revised manuscript. We appreciate this acknowledgment.

However, the reviewer has maintained the original overall rating of 1, citing the extent of the editorial revisions and the presentation quality of the initial submission. Their latest comment does not identify any specific unresolved methodological or experimental concern.

Importantly, the original manuscript also received a presentation score of 3 from Reviewer xNPB, reflecting a favorable independent assessment of the manuscript clarity.

We emphasize that the proposed method, original experimental protocols, and main results have not changed. The revisions clarify the same scientific contribution, while the additional experiments provide further validation.

We respectfully ask that the final assessment distinguish the extent of editorial revisions from changes in scientific substance, and give full consideration to the original contribution, the additional evidence, and the improved soundness assessment.

Thank you for your consideration.

The Authors

AuthorsOpenReview ↗

Final Follow-up on Reviewer Assessments

Dear Area Chair,

As the discussion period approaches its end, we would like to draw your attention to two remaining points.

Reviewer 4HVW acknowledged our extensive rebuttal but maintained the original rating, without identifying any remaining concerns despite the substantial additional evidence addressing every point raised.

Reviewer xNPB has not provided a follow-up assessment of our rebuttal, including the new Qwen3-Next-80B, LiveCodeBench, and ProofWriter experiments that directly address the reviewer's concerns about scalability and generalization.

We have respectfully requested reassessment from both reviewers. We would appreciate your ensuring that the complete rebuttal, revised manuscript, and additional experimental evidence receive full consideration in the final assessment, particularly where the original ratings have not been updated.

Thank you for your consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.