acceptodds
Under review as a conference paper at ICLR 2027

Verbalising Model's Computation for Chain-of-Thought Faithfulness Detection

Abstract

Current Chain-of-Thought (CoT) verification methods detect reasoning faithfulness from model outputs or static activations, but are limited in explaining how unfaithfulness develops during generation. We introduce VERITAS to verbalise the model's computation: how different reasoning anchors emerge, transfer across layers, and surface in the output. VERITAS adopts the layer-wise Jacobian projection to produce a verbalised account of the model's internal reasoning evolution at each generated CoT token. By training a classifier on the extracted features along the traces, we show that these structured features provide insightful perspectives on how and where reasoning diverges toward unfaithfulness. Experiments on four datasets from FaithCoT-Bench and BonaFide show that VERITAS achieves a significant performance gain over LLM-as-Judge, with a 28.4% gain in balanced accuracy, and an average improvement of 16.0% over activation probing. Further analysis shows VERITAS can be generalised to different datasets and model architectures. Our work shows that verbalising model computation provides a richer and more interpretable lens on CoT faithfulness, moving beyond detection toward a mechanistic understanding of how reasoning fails.

Primary area:
interpretability and explainable AI
Keywords:
Chain-of-Thought faithfulness, Jacobian lens

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z4A2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce VERITAS to investigate how faithfulness develops during CoT via Jacobian lens. Three structured features are proposed: map, profile, and dynamic.

Critical strengths
  • The experiments and analysis are comprehensive and thorough.
  • The topic about CoT faithfulness is timely and important
  • Combining J-lens with CoT faithfulness is interesting.
Critical weaknesses
  1. I think the central claim seems to be overstated. In the abstract, the authors claim that VERITAS trace how unfaithfulness developes during generation, but the two models included in the experiments do not generate CoTs that they test. In other words, the hidden states of these evalated models are not the generating model's computation, particularly CoT faithfulness is highly model-dependent. This is also against the paper's motivation, as the authors critize the LLM-as-a-Judge approaches because they cannot access internal computation of the model, and the proposed approach suffers the same issue.
  2. It is questionable to use FaithCoT benchmark, as it tests CoT plausibility instead of genuine faithfulness.
  3. The baseline comparison may not be fair. For example, the HS baseline read only one fixed layer, while VERITAS reads all layers * 5 phases. It's unclear whether the gains are from J-lens verbalization.
  4. Since the anchor score is based on J-lens direction. In my opinion, ablation study of J-lens is required, e.g., with random directions or logit-lens directions.
Questions for authors
  1. Table 1 caption: "evalaution". Fig. 3: "Strcutured".
  2. l.196: there is a redundant "."
2 comments
AuthorsOpenReview ↗

Response to Reviewer Z4A2 (1/2)

We sincerely thank Reviewer Z4A2 for the careful reading and constructive comments. Below, we clarify our understanding of faithfulness and the “during generation” setting (W1–W2), and report two new experiments: a hidden-state baseline matched to VERITAS in layers, phases and classifier (W3), and a J-lens ablation with logit-lens and random directions (W4).


W1. Clarification on “during generation” and the evaluated model’s hidden states

Thank you for the comment. We would like to clarify a misunderstanding:

(1) Faithfulness is a characteristic of CoT. In BonaFide and FaithCoT-Bench, faithfulness is about “whether the CoT reflects how the answer was actually reached.” It’s model-independent. Faithfulness labels can be validated by human annotators without access to the generating model’s states. So, even when one model generates the CoT, another can serve as a feature extractor, capturing its own hidden-state features.

(2) “During generation” refers to capturing faithfulness-related features across CoT progression steps. This is technically grounded: with the causal attention mechanism, the hidden states a model produces when reading q+CoT are equivalent to those it would produce at each step when generating the CoT token-by-token given q. So, hidden states are used to capture the evaluated model’s dynamics along the CoT. This follows the standard probing protocol: [1-4] all probe an evaluation model on outputs it did not generate, using those models’ hidden states as features. Our contribution is to capture these dynamics along two axes: model depth and CoT progression.

(3) Our criticism of LLM judges is that LLM judges assess faithfulness from verbal cues and phrases in the CoTs, without analyzing hidden-state dynamics. However, VERITAS uses the evaluated model’s hidden states to construct Map, Profile, and Dynamics features, capturing how anchor evidence varies across model layers and CoT steps.

We will make the following changes:

  • We will revise wording that could imply that we analyze hidden states while real-time generating CoT:
    • Abstract: Replace “trace how unfaithfulness develops during generation” with “trace how faithfulness-related signals vary across model depth and CoT progression.”
    • Introduction: Replace “during generation” with “across CoT progression”. 
    • Introduction: Replace “shapes each generated CoT token” with “varies across successive CoT positions.”
  • Introduction, motivation, clarify: LLM-as-a-Judge assesses faithfulness from verbal cues in the CoT, while VERITAS analyzes intermediate representations across layers and CoT steps.
  • §3.1, clarify the definition of faithfulness: Add “Faithfulness concerns whether the CoT accurately states the information used and the reasoning actually performed to reach the answer.”
  • §5 (Related Work): Add the probing precedents [1–4] that analyze evaluated models’ hidden states on the given input text.
  • Appendix A Datasets: add the annotation criteria.

W2. FaithCoT-Bench as a faithfulness benchmark

As you may notice, CoT faithfulness has no unified definition, and existing benchmarks operationalise different facets of it. One facet is plausibility, i.e., whether the stated steps coherently lead to the answer. FaithCoT-Bench explicitly explores whether a CoT “faithfully reflects the internal reasoning process of the model”, and operationalizes this target through expert annotations of post-hoc rationalization and spurious reasoning chains. We use it to evaluate the detection of these observable indicators of unfaithfulness. 

References:

[1] Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. NeurIPS, 2023.
[2] The Internal State of an LLM Knows When It’s Lying. In Findings of the Association for Computational Linguistics: EMNLP 2023.
[3] Discovering Latent Knowledge in Language Models Without Supervision. ICLR, 2023.
[4] Detecting Strategic Deception Using Linear Probes. ICML, 2025.

AuthorsOpenReview ↗

Response to Reviewer Z4A2 (2/2)

W3. Fairness of the hidden-state baseline

To separate the effect of layers * 5 phases granularity from that of our method, we matched the HS baseline to VERITAS step by step. For every layer, we take the hidden states at the same sentence/clause delimiters as VERITAS and average them, either over the whole CoT (“mean”) or within each of the 5 phase bins (“5 phases”). The resulting vectors are concatenated across layers and phases and fed to the same classifier as VERITAS:

  • The original baseline with the same classifier as VERITAS:
    • HS_fixed × mean: one fixed layer, averaged over the whole CoT.
  • CoT-phase axis:
    • HS_fixed × 5 phases: the same layer, with the 5 phase averages concatenated.
  • Layer axis:
    • HS_all × mean: all layers, with their CoT-level averages concatenated.
  • Both axes:
    • HS_all × 5 phases: all layers × 5 phases concatenated.

Table 1: Balanced accuracy on FaithCoT-Bench (Qwen3-4B).

MethodAQuALogiQATruthfulQAHLE-BioAvg.
HS_fixed × mean0.6320.4700.6940.5440.585
HS_fixed × 5 phases0.7080.5360.7030.5410.622
HS_all × mean0.7260.6020.7290.5440.650
HS_all × 5 phases0.7360.5870.6990.5470.642
VERITAS0.7650.7190.7680.6410.723

Reading more layers and phases does help the HS probe on average. However, even with the same layer × phase grid, VERITAS remains 8.1 points higher on average and is higher on every dataset. This supports the benefit of its structured feature representation.


W4. Ablation of the J-lens.

We present three analyses: (1) the comparison between the J-lens and the logit lens and random directions; (2) the difference between J-lens and logit lens readout, to deepen the understanding; (3) the comparison showing the logit lens with structured features still outperforms the best hidden-state baseline, to locate the source of the performance gains.

J-lens ablation. We keep VERITAS_c unchanged and replace only the lens that defines the anchor directions. Each lens follows the same steps: it assigns every candidate a direction d; in each (layer ℓ\ell, phase p) cell, 8 anchors are selected; then compute the alignment of its mean hidden state with each anchor.

  • J-lens: A candidate word w has direction d=normalize(WU[w]Jℓ)d = normalize(W_U[w]\mathbf{J}_\ell). Anchors are therefore words, read with a layer-specific direction.
  • Logit lens: With d=normalize(WU[w])d = normalize(W_U[w]), anchors are still words, but read as if each layer were the output layer.
  • Random: The same number of candidates, but each dd is a random unit vector. Anchors carry no word identity.

Table 2: Balanced accuracy comparison with other Lens on FaithCoT-Bench (Qwen3-4B).

MethodAQuALogiQATruthfulQAHLE-BioAvg.
J-lens0.7650.7190.7680.6410.723
Logit lens0.7520.6880.7400.5840.691
Random0.7130.6670.7180.5480.661

The J-lens gives the best result on all four sources. The logit lens, which uses the same words, still loses 3.2 percentage points on average. The Jacobian transport is therefore what makes the word directions informative at the intermediate layers that VERITAS reads.

Interpretability. The J-lens also keeps the anchors interpretable in the middle layers (new Appendix D).

  • In layers 12–19, 39% of logit-lens top-10 readouts are not English words (subword fragments or code identifiers), compared with 24% for the J-lens.
  • On a concrete pair, at the sentence “Evaluate the hint”, the J-lens reads questionable, suspicious, dismissing, distrust from layer 12 onward, while the logit lens returns discount, Balance, ims, Hague, Halk and never reaches these words below the final layer.

Therefore, the logit lens is not adequate for the mid-layer predicted token content which our analysis relies on.

Contribution beyond the lens. The J-lens is only one way to obtain anchor directions. Our main contribution is Map, Profile, and Dynamics features. As shown in Table 3, even without the J-lens, logit-lens anchors with these structured features outperform the best hidden-state baseline across all four sources. This supports the benefit of its structured feature representation.

Table 3: Logit-lens VERITAS vs. the strongest hidden-state baseline. Balanced accuracy on FaithCoT-Bench (Qwen3-4B).

MethodAQuALogiQATruthfulQAHLE-Bio
HS_all × mean (best HS on average)0.7260.6020.7290.544
Logit lens0.7520.6880.7400.584

Q1 & Q2. We apologise for these typos. We will correct “evalaution” in the Table 1 caption and “Strcutured” in Fig. 3, remove the redundant “.” in l.196, and proofread the full manuscript.


We hope our clarifications can address your concerns. Please let us know if you have any further questions. If we have addressed your concerns, we would appreciate it if you could consider raising your rating.

Reviewer_7M6tOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VERITAS, a framework for detecting whether a chain-of-thought (CoT) explanation is faithful to a model’s internal computation. The central claim is that faithful and unfaithful reasoning induce different semantic trajectories along two axes: model depth and CoT generation progress. Rather than judging only the visible CoT text or training a static hidden-state probe, VERITAS “verbalises” intermediate model states into token-level concepts and tracks how those concepts evolve.

Critical strengths
  1. The paper introduces a genuinely new trajectory-based formulation of CoT faithfulness and operationalizes it in an interpretable, falsifiable way. The key move is to verbalise intermediate residual-stream states through a J-lens and then track token-level anchor alignment along two axes: model depth and CoT progress. This is consequential because prior work largely falls into two camps: judging the visible CoT text or probing static hidden states at a fixed layer/position. VERITAS instead characterizes how faithfulness-relevant evidence appears, which anchors carry it, and how it evolves. The Map/Profile/Dynamics decomposition is not an ad hoc feature stack: the ablation shows that each view contributes and that combining them is best across settings. This matters for my overall assessment because it turns the central hypothesis—faithful and unfaithful reasoning induce different semantic trajectories—into something empirically testable, and it gives the community a reusable analytical lens rather than just another classifier.

  2. The empirical scope is unusually broad for a method paper and directly supports the generality claims. The evaluation covers two benchmarks, four FaithCoT-Bench subsets, two backbones, multiple baselines, grouped out-of-fold prediction, cross-source transfer, cross-model transfer, and sensitivity analyses over CoT length, layer stages, phase count, and anchor count. This is consequential because the paper’s claims are comparative and generalization-oriented: without this breadth, the reported gains could plausibly be dataset-specific or model-specific artifacts. The transfer results—anchors retaining 97–99% of in-domain balanced accuracy, zero-shot transfer retaining 88–90%, and Dynamics retaining 91–96% under cross-model transfer—substantially increase confidence that the trajectory signals are not merely overfit to one setting. This strongly supports acceptance, even though the evidence remains primarily discriminative rather than causal.

Critical weaknesses

1. The paper’s central mechanistic claim is supported only by discriminative, correlational evidence, not by causal or intervention-based evidence.
The paper claims that faithful and unfaithful reasoning induce different semantic trajectories along model depth and CoT progress, and presents Map/Profile/Dynamics features that distinguish the two classes. But the experiments show that these trajectory features are predictive of faithfulness labels; they do not establish that the trajectory differences are the mechanism underlying faithfulness, nor do they rule out that the classifier is exploiting dataset-specific artifacts, annotation regularities, or contrasts induced by the anchor-discovery procedure itself. This is consequential because the paper’s main novelty is not merely a new detector but a “mechanistic account” of how unfaithfulness develops. Without intervention evidence—e.g., steering along anchor directions and observing whether faithfulness changes, or causal mediation analyses—the central hypothesis remains plausible but unproven. For a paper whose contribution is framed around mechanistic explanation, this gap is material, not cosmetic.

2. The empirical gains are difficult to interpret confidently because key implementation details are inconsistent and the strongest baseline is comparatively weak.
The main text uses κ=8\kappa=8 anchors per cell as the default, while Appendix B.2 states that five positively aligned tokens are retained; such inconsistencies in a core hyperparameter undermine reproducibility and make it unclear which configuration produced the reported results. In addition, the LLM-as-Judge baseline produces only hard labels under a single greedy judgment, with no reported sensitivity to prompt, model size, or sampling; this makes the headline 28.4% gain over LLM-as-Judge potentially overstated. This is consequential because the paper’s acceptance case rests heavily on comparative superiority and on the claim that the structured trajectory features, rather than a weak baseline or a particular hyperparameter choice, drive the improvement. Without resolving these issues, the reported gains are not yet robust enough to support the strength of the paper’s conclusions.

Questions for authors
  1. Anchor-discovery isolation and possible leakage.
    Please clarify exactly how the contrast direction Δℓ,p\Delta_{\ell,p} is estimated and how discovery questions are separated from the five evaluation folds. Are anchors frozen before any evaluation-fold labels are used? Can the same question (or near-duplicate CoT) appear in both discovery and evaluation? This is material because the paper’s central claim is that the anchors generalize; if discovery and evaluation are not strictly isolated, the reported gains could partly reflect leakage.

  2. Strength and tuning of the LLM-as-Judge baseline.
    The paper reports a 28.4% gain over LLM-as-Judge, but the judge appears to produce a single greedy hard label. Please specify the judge model, prompt, decoding settings, and whether the prompt was tuned. Could you also report a stronger judge baseline, e.g., multi-sample majority voting or a larger judge model? This is material because the headline comparative claim depends heavily on how strong this baseline is.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7M6t

We sincerely thank Reviewer 7M6t for the careful reading and concrete suggestions. We clarify your concerns below.


W1. Targeted intervention during the model’s own generation.

To test whether the anchor directions influence the model’s decision, we intervene on Qwen3-4B-Thinking’s hidden states during CoT generation on BonaFide prompts, using the original input context. We fix the top-8 anchors in each layer–phase cell before testing.

We first generate CoTs without steering. Each resulting CoT is truncated at the first delimiter beyond 20% of its full length, providing a shared prefix for all conditions. From this shared prefix, we steer newly generated tokens until generation ends. We steer the hidden state using h′=h+∥Jℓh∥2dDA+(α1),\mathbf{h}' = \mathbf{h}+\frac{\|\mathbf{J}_\ell\mathbf{h}\|_2}{\sqrt{d}}\mathbf{D}_A^{+}(\alpha\mathbf{1}), where DA\mathbf{D}_A stacks the cell’s anchor J-lens vectors as rows, DA+\mathbf{D}_A^{+} is its pseudoinverse, α\alpha is the target increase in each anchor readout coordinate, 1\mathbf{1} is the all-ones vector, and dd is the hidden-state dimension. The scale factor is computed before the update. As a control, we apply the same procedure to random words. All conditions use identical prefixes and sampling seeds, with 60 rollouts per condition.

Table 1. Hint acknowledgement rate of Qwen3-4B-Thinking.

ConditionAcknowledges reliance on the hint
No steering21 / 60 (35.0%)
Random words, α=3\alpha = 313 / 60 (21.7%)
Anchor words, α=3\alpha = 348 / 60 (80.0%)

Anchor steering increases hint acknowledgement from 35.0% without steering to 80.0%, while random-word steering yields 21.7%. These results provide causal evidence that the identified directions influence the model’s reporting of hint reliance.


W2 & Q2. Anchor-count consistency and LLM-as-Judge baseline strength

Anchor count. We apologise for the incorrect anchor count in Appendix B.2. The default is κ=8\kappa = 8. “Five” was copied from an ablation configuration not used for any reported result. We have corrected Appendix B.2.

LLM-as-Judge baseline. We test additional judge configurations addressing the three factors raised by the reviewer:

  • Sampling: Qwen2.5-7B with 8 samples and majority voting.
  • Prompt: Qwen2.5-7B with one labelled example from the anchor-discovery split, whose questions do not appear in the evaluation set.
  • Model scale / reasoning: Qwen3-8B with thinking enabled and gpt-oss-20b with high reasoning effort.

Table 2. Balanced accuracy comparison with LLM-as-Judge variants on FaithCoT-Bench.

MethodAQuALogiQATruthfulQAHLE-Bio
VERITAS0.7650.7190.7680.641
Qwen2.5-7B greedy0.6730.6060.5710.444
Qwen2.5-7B majority vote0.6270.5910.5420.457
Qwen2.5-7B one-shot prompt0.6850.5830.6330.465
Qwen3-8B (thinking)0.7080.6220.6300.558
gpt-oss-20b (high reasoning)0.6300.6530.6570.569

Larger, reasoning-enabled judges narrow the gap, but VERITAS remains best on all four subsets. Majority voting and one-shot prompting provide no consistent improvement. These results support the benefit of internal dynamic features for CoT faithfulness detection compared with the tested LLM-as-Judge configurations.


Q1. Anchor-discovery isolation

Thank you for the question. Anchor discovery and evaluation use disjoint questions, and anchors are frozen before cross-validation.

  • Question-level separation. Each subset is split by question_id into discovery and evaluation portions. All traces of a discovery question, including unpaired traces, are excluded from evaluation (App. B.6 ).
  • Estimating Δℓ,p\Delta_{\ell,p}. As described in §3.2, we form one-to-one faithful–unfaithful CoT pairs for the same question, without reusing traces. Each CoT is divided into PP relative-position bins, and its hidden states at sentence and clause boundaries are averaged within each layer–bin cell. Using discovery pairs only, Δℓ,p\Delta_{\ell,p} is then the average faithful-minus-unfaithful difference over pairs with valid readouts for both labels.
  • Freezing anchors. The top-8 candidates ranked by cos�0�cos�0� are selected once and never refit per fold or trace.
  • Evaluation. We use five-fold cross-validation grouped by question, with classifier fitting restricted to the training folds. Evaluation labels never influence anchor selection.

Revisions:

  • §3.2, end: We will add “Δℓ,p\Delta_{\ell,p} and the anchors are computed using discovery questions only and frozen before cross-validation.”

We hope these clarifications address your concerns. Please let us know if you have any further questions.

Thank you for the authors’ detailed response. All my questions have been addressed, and I think the response is excellent. I am inclined to keep my score, but that does not prevent me from thinking that this paper should be accepted.

Reviewer_5oA1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies chain-of-thought faithfulness detection,and proposes VERITAS to examine how hidden representations change across model depth and reasoning progress. The method uses the Jacobian Lens to associate intermediate representations with directions corresponding to vocabulary tokens. It selects anchors from representation differences between faithful and unfaithful CoTs for the same question. The authors then construct an alignment tensor over layers, reasoning phases, and anchors, and extract three feature sets. Map captures where alignment signals appear, Profile summarizes the responses of individual anchors, and Dynamics measures the directional consistency of changes between neighboring positions. A regularized logistic regression classifier uses these features to predict faithfulness labels. The experiments pass existing CoTs through replay backbones to extract activations and evaluate detection performance across several datasets. They also include feature ablations, visualizations, and transfer experiments across datasets and models.

Critical strengths
  1. The paper considers both model depth and CoT progress rather than focusing only on the final answer or a single hidden state. This preserves some of the structure in how representations change throughout reasoning. Map, Profile, and Dynamics have distinct roles in capturing signal locations, anchor identity, and local changes. Together, they provide a clear framework for studying representations associated with faithfulness and testing specific hypotheses.

  2. Anchor discovery uses differences between faithful and unfaithful CoTs for the same question, rather than directly comparing the overall means of the two label groups. This helps reduce the influence of question content on anchor selection. The evaluation keeps all CoTs for a given question in the same fold. The authors also state that anchor discovery uses a separate set of questions and that preprocessing and classifier fitting are performed within each training fold. These choices reduce the risk of leakage from different traces of the same question appearing in both training and test sets.

  3. The results across tasks and the feature ablations provide encouraging evidence for the method’s practical value. Table 1 shows that VERITASc achieves higher balanced accuracy than the listed non-VERITAS baselines across tasks and both replay backbones. In Table 4, the full feature combination achieves the highest point estimate in all four settings, providing initial evidence that the three feature sets are complementary.

Critical weaknesses
  1. There is an important gap between what the method measures and its claims about the computation that generated the original CoT. It states that existing CoTs from several generating models are passed through two replay backbones. The extracted activations therefore describe the replay model’s computation when processing an existing reasoning trace, which need not match the computation of the model that generated it. These representations may be useful for detection, but their predictive performance does not establish that the method tracks how unfaithful reasoning arose in the original model. The paper does not establish a correspondence between these two computations, leaving the scope of its mechanistic claims unclear.

  2. The anchors are selected from differences between faithful and unfaithful examples. Their ability to separate the two groups therefore shows that they are useful for classification, but does not by itself establish that the meanings of the corresponding tokens describe the computations the model actually performs. Figure 4 manually groups the anchors into Context, Explanation, Checking, and Reporting, and shows the composition of discriminative scores after removing tokens shared by the two anchor pools.

  3. The Raw-HS baseline uses a single predetermined layer, does not search over layers, and fits logistic regression without an intercept or class weighting. VERITAS uses multiple layers and reasoning phases, balanced class weights, and a different regularization setting. These differences may affect balanced accuracy, particularly on imbalanced datasets. The current comparison therefore does not clearly separate gains from J-lens, additional layer and phase information, feature compression, and the classifier configuration.

Questions for authors
  1. When the replay backbone differs from the original generating model, what justifies interpreting differences in replay activations as evidence of how the original model produced unfaithful reasoning? Could you validate the main findings using the original generating model and its original input context, and conduct targeted interventions on the identified anchors or layer and phase locations to test their relationship with the relevant decisions?

  2. How much of VERITAS’s advantage over a multi-layer temporal Raw-HS probe remains when layer coverage, CoT phase segmentation, normalization, class weights, and regularization tuning are matched? Can comparisons with a text classifier trained on the same labels rule out reliance on disclosure language and cues associated with the generating model?

  3. Please clarify the actual feature combination used by VERITAS_p, the default number of anchors, and the procedures used in the experiments. For cross-model transfer, how are the shared anchor coordinates and Profile feature dimensions constructed and aligned, and does this process use labels associated with the target model?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5oA1 (1/2)

We sincerely thank Reviewer 5oA1 for the careful reading and concrete suggestions. Below, we clarify the scope of replay analysis and present validation using the generating model, along with targeted interventions (W1 & Q1). We examine anchor interpretation through a case study (W2), add matched hidden-state and text-based baselines (W3 & Q2), and clarify the feature configuration and cross-model alignment (Q3).


W1 & Q1. Scope of replay analysis and validation with the generating model

Thank you for the comment and concrete suggestions. We will clarify below:

(1) We use the replay model as a feature extractor to capture diagnostic signals related to the faithfulness of CoT from the generating model. This approach is motivated by evidence of shared representational structures and transferable semantic features across language models [1, 2]. These findings suggest that different models may encode overlapping high-level semantic and reasoning-related information.

(2) Generating model vs. replay backbone. To test whether replay preserves faithfulness-related structures and signals, we evaluate CoTs generated by Llama-3.1-8B using both Llama-3.1-8B (the original generator) and Qwen3-4B (the replay backbone). We measure the alignment of the two models’ faithful directions (mean faithful − unfaithful states from the two models, compared after a label-free linear map between the two hidden spaces), faithful–unfaithful class separation using the between-class minus within-class cosine distance, and detection performance (BAcc).

The faithful directions of the two models are closely aligned (cosine similarity= 0.71, random directions: 0.23).

Table 1: Representation separation and detection performance of generator vs. replay backbone on Llama-generated FaithCoT CoTs.

BackboneΔDistanceAQuALogiQATruthfulQAHLE-BioAvg. bAcc
Generator (Llama-3.1-8B)0.0190.7220.7280.7660.6710.722
Replay (Qwen3-4B)0.0250.7610.7000.7590.6960.729

From Table 1, both models place faithful and unfaithful CoTs apart to a similar degree and achieve a similar average BAcc. Together with the aligned faithful directions, these results suggest that an independent replay model captures the same faithfulness-related signal as the generator and achieves comparable detection performance, without accessing the generating model's internal states.

(3) Targeted intervention during the model’s own generation. To test whether the anchor directions influence the model’s decision, we intervene on Qwen3-4B-Thinking’s hidden states during CoT generation on BonaFide prompts, using the original input context. We fix the top-8 anchors in each layer–phase cell before testing.

We first generate CoTs without steering. Each resulting CoT is truncated at the first delimiter beyond 20% of its full length, providing a shared prefix for all conditions. From this shared prefix, we steer newly generated tokens until generation ends. We steer the hidden state using h′=h+∥Jℓh∥2dDA+(α1),\mathbf{h}' = \mathbf{h}+\frac{\|\mathbf{J}_\ell\mathbf{h}\|_2}{\sqrt{d}}\mathbf{D}_A^{+}(\alpha\mathbf{1}), where DA\mathbf{D}_A stacks the cell’s anchor J-lens vectors as rows, DA+\mathbf{D}_A^{+} is its pseudoinverse, α\alpha is the target increase in each anchor readout coordinate, 1\mathbf{1} is the all-ones vector, and dd is the hidden-state dimension. The scale factor is computed before the update. As a control, we apply the same procedure to random words. All conditions use identical prefixes and sampling seeds, with 60 rollouts per condition.

Table 2. Hint acknowledgement rate of Qwen3-4B-Thinking.

ConditionAcknowledges reliance on the hint
No steering21 / 60 (35.0%)
Random words, α=3\alpha = 313 / 60 (21.7%)
Anchor words, α=3\alpha = 348 / 60 (80.0%)

Anchor steering increases hint acknowledgement from 35.0% without intervention to 80.0%, while random-word steering yields 21.7%. These results provide causal evidence that the identified directions influence the model’s reporting of hint reliance.

We will make the following changes:

  • We will revise wording that could imply that we analyze hidden states while real-time generating CoT:
    • Abstract: Replace “trace how unfaithfulness develops during generation” with “trace how faithfulness-related signals vary across model depth and CoT progression.”
    • Introduction: Replace “during generation” with “across CoT progression”.
    • Introduction: Replace “shapes each generated CoT token” with “varies across successive CoT positions.”
  • Appendix F: Add the anchor-steering experiment.

References:
[1] Position: The Platonic Representation Hypothesis. ICML, 2024.
[2] Transferring Linear Features Across Language Models With Model Stitching. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer 5oA1 (2/2)

W2. Whether anchor tokens describe the model’s computation

Thank you for the comment. We partially agree and clarify how the anchors are selected and why their tokens reflect the model’s computation.

  • Selection. Yes. Anchors are selected by alignment with the faithful–unfaithful hidden-state differences Δℓ,p\Delta_{\ell,p} computed on discovery questions.
  • Meaning. The J-lens Jacobian maps intermediate states to final-layer representations, which the model’s unembedding connects to vocabulary directions. Anchor tokens thus provide readouts of internal representations and need not appear in the CoT text.
  • Interpretation. The four categories in Fig. 4 are manually assigned for interpretation and are not used by the classifier. Shared tokens are excluded only from this visualization. The anchors used for detection remain unchanged.
  • Case study. On individual CoTs (new App. E), the faithful trace activates dismissing, caution, and skepticism while evaluating the hint. The unfaithful trace activates Nairobi, the hinted answer, seven sentences before stating it. Removing the hint while holding the CoT fixed eliminates most of these responses. Together with the steering results in W1 & Q1, these observations support the anchors’ role in hint-related information processing and reporting.

W3 & Q2. Sources of the gain: matched hidden-state and text baselines

Thank you for raising this. The original Raw-HS probes follow the settings of [1] as closely as possible. VERITAS uses the default L2-regularized logistic regression, without tuning the regularization. We add two comparisons to examine the contributions of layer and phase coverage, classifier settings, and textual cues to VERITAS’s performance.

(1) Matched hidden-state baseline. We match the HS baseline to VERITAS one factor at a time, averaging hidden states at the same delimiters, then concatenating across selected layers and phases. All variants except the original baseline use VERITAS’s classifier settings.

  • Original baseline. HS_fixed × mean (original), one fixed layer averaged over the whole CoT, with the classifier settings of baseline.
  • Classifier settings. HS_fixed × mean, the same features, with VERITAS’s classifier settings.
  • CoT-phase axis. HS_fixed × 5 phases, the same layer, with the 5-phase averages concatenated.
  • Layer axis. HS_all × mean, all layers, with their CoT-level averages concatenated.
  • Both axes. HS_all × 5 phases, all layers × 5 phases concatenated, the same cells as VERITAS.

Table 3. Balanced accuracy with HS baselines on FaithCoT-Bench (Qwen3-4B). All variants except the original baseline use VERITAS’s classifier settings.

MethodAQuALogiQATruthfulQAHLE-Bio
VERITAS0.7650.7190.7680.641
HS_fixed×mean (original)0.6230.4610.7260.545
HS_fixed × mean0.6320.4700.6940.544
HS_fixed × 5 phases0.7080.5360.7030.541
HS_all × mean0.7260.6020.7290.544
HS_all × 5 phases0.7360.5870.6990.547

Classifier settings have little effect on average, while broader layer and phase coverage raises BAcc from 0.589 to 0.642. With both axes and classifier settings matched, VERITAS still gains 8.1 percentage points on average and performs better on every dataset, supporting its benefit of structured features.

(2) Text classifier. We fine-tune ModernBERT-base on the same faithfulness labels.

Table 4. Balanced accuracy comparison with ModernBERT-base on FaithCoT-Bench.

MethodAQuALogiQATruthfulQAHLE-BioAvg.
VERITAS0.7650.7190.7680.6410.723
ModernBERT-base0.6770.5620.7130.5430.624

VERITAS outperforms ModernBERT-base on four datasets, with an average gain of 9.9 percentage points, supporting the benefit of its internal features over this supervised text baseline.


Q3. Feature configuration, anchor count, and cross-model alignment

(1) Features, κ\kappa. As stated in the main text (§3 and §4.1), VERITAS_p uses Dynamics + Map, VERITAS_c additionally uses Profile, and the default is κ\kappa=8 anchors per (layer, phase) cell. In experiments, questions are divided into a discovery set and an evaluation set, so no question appears in both.

(2) In cross-model transfer, anchor words discovered on the source model remain fixed and are read using the target model’s own lens. Each model’s layers are binned into the same number of depth stages. As a result, Map, Profile, and Dynamics are indexed by the same words on the same grid and align one-to-one across models.

Revisions:

  • App. B.5: add the full classifier setting and the matched hidden-state ablation.
  • Appendix B.8: add the cross-model alignment procedure

References:

[1] Detecting Strategic Deception Using Linear Probes. ICML, 2025.

We hope these clarifications address your concerns. Please let us know if you have any further questions.

Thanks for the detailed response. I appreciate the authors' efforts to address my main concerns, and I agree that the claims should be tightened in a further revision. I'll keep my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear reviewers and PCs/SACs/ACs:

We appreciate your time and effort in reviewing our submission.


We are encouraged by their recognition of the key contributions of our work. Across the reviews,

  • Trajectory-Based Faithfulness Formulation: Reviewers 5oA1 and 7M6t acknowledged the novelty of our trajectory-based formulation of CoT faithfulness (verbalising intermediate representations via the J-lens along model depth and CoT progression).
  • J-lens Verbalisation: Reviewer Z4A2 found combining the J-lens with CoT faithfulness interesting.
  • Comprehensive Experiments: All reviewers noted that the experiments are comprehensive.

We carefully addressed each comment raised by the reviewers. Here, we summarize the reviewers’ comments, our response, and reviewers’ feedback:

Reviewer 5oA1 (rating 3)

  • We added validation against the generating model, anchor-steering experiments, text-based and HS baselines with matched layer and phase coverage to isolate the source of the gains, and an anchor case study.
  • Follow-up: the reviewer appreciated our efforts to "address the main concerns."

Reviewer 7M6t (rating 3)

  • We added anchor-steering experiments and stronger LLM-as-Judge variants, and clarified the anchor settings.
  • Follow-up: the reviewer thought “the response is excellent” and stated that “all questions had been addressed” and that "the paper should be accepted."

Reviewer Z4A2 (rating 1)

  • Question about the faithfulness definition: Existing benchmarks adopt different practical approximations that still provide faithfulness-related signals. Moreover, prior work also uses independent models to evaluate faithfulness.
  • Source of the gains: We added hidden-state baselines with matched layer and phase coverage, a J-lens ablation, and a comparison showing that the gains mainly come from our structured features. [No comment]
  • Interpretability of the J-lens: We added a comparison showing that J-lens anchors remain more interpretable than logit-lens readouts in the middle layers. [No comment]

We would like to bring the AC's attention to Reviewer Z4A2's main reason for the strong rejection. The reviewer disagrees with our definition of CoT faithfulness, arguing that our benchmarks do not measure faithfulness.

However, we would like to highlight that:

  • Both BonaFide and FaithCoT-Bench explicitly aim to evaluate CoT faithfulness, as stated in their original papers.
  • The other two reviewers also recognized our faithfulness-related contribution. Reviewer 7M6t highlighted how VERITAS captures “faithfulness-relevant evidence,” while Reviewer 5oA1 acknowledged its value in studying “representations associated with faithfulness.”

We agree that, in principle, assessing faithfulness requires understanding the generating model's internal computation. However, as it's difficult to fully observe the internal computation, existing research adopts practical approximations, as clarified in our Appendix I. If we follow the reviewer's argument that “no ground-truth labels exist” and that these benchmarks measure only "plausibility or self-consistency", it would make empirical evaluation of CoT faithfulness infeasible.

Thus, we do not agree that this definition issue alone constitutes a sufficient reason to reject our paper, and hope the AC will consider our contributions and additional experimental evidence.


We hope these clarifications are helpful in your assessment.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.