General point of view:
The paper starts with an interesting and well-motivated problem: evaluating the faithfulness of chain-of-thought (CoT) reasoning using only generated text. The methodology is relatively clear, and the introduction explains the motivation well. However, my main concern is with the experimental evidence and the interpretation of the results. The authors propose that sentence-embedding geometry can reveal properties of reasoning faithfulness, but the experiments do not convincingly establish whether these measurements capture reasoning behavior or simply statistical properties of the generated text. Previous work has already shown the difficulty of establishing faithfulness from generated explanations (Turpin et al., 2023).
The experimental section is particularly weak regarding interpretability. Most analyses focus on AUROC and correlations between geometric features and benchmark labels, without sufficiently explaining how these features relate to faithful or unfaithful reasoning. I believe the paper needs stronger controlled experiments and a deeper analysis of its proposed geometric representation.
Abstract: Reading the abstract, it is relatively easy to understand that the authors are studying trajectories in the geometry of sentence embeddings to identify whether a model conceals its use of a misleading hint. However, this is not an entirely new research direction. Previous work, such as Truth as a Trajectory (Damirchi et al., 2026), has already investigated reasoning through geometric trajectories. Although the representations and objectives differ, the authors should better contextualize their contribution. BonaFide is mentioned without being adequately introduced. I recommend briefly explaining the benchmark before discussing its labels, making the abstract understandable without consulting the introduction.
The main contribution is also difficult to assess from the results. SemFaith achieves 0.885 AUROC, while a simpler TF-IDF baseline reaches 0.991 under the same evaluation protocol. This raises questions about the advantages of the proposed geometry. If the contribution is interpretability rather than predictive performance, the authors should emphasize this and explain what the geometric measurements reveal beyond lexical features. This distinction is important considering previous work on shortcut learning.
Introduction: The paper motivates the problem well, explains CoT, and introduces BonaFide. The method summary is clean and relatively easy to understand. However, I believe the authors should introduce the specific faithfulness problem earlier and explain more clearly that they are analyzing only generated textual traces, rather than internal reasoning representations.
Expressions such as "can the trace itself provide a useful signal about such omissions?" are relatively broad. Considering previous trajectory-based work (Damirchi et al., 2026), the authors should explicitly distinguish textual CoT trajectories from trajectories of internal model representations.
Regarding the results, I believe the paper loses relevance through the way the contribution is presented. The difference between SemFaith and TF-IDF could be interpreted negatively. An interesting point appears at the end of the related work section, where SemFaith is described as an interpretable structural predictor. I think this should be emphasized and better defined in the introduction. Also, the authors could refer to Figure 1 when explaining the individual stages of the method.
Figure 1: The figure is generally well organized and easy to understand. However, the examples are too small, especially for a printed version. Some textual examples also contain grammatical inconsistencies. The selection of the number of centroids is unclear. Why are K = 5, 10, and 20 appropriate? Established clustering validation methods could provide additional justification. Additionally, the figure shows the concatenation of features into a 24-dimensional representation, while Section 3.5 introduces an eight-dimensional representation. Although these correspond to different configurations, this distinction is not sufficiently clear in the figure.
Method:
3.1 Problem Formulation: The authors define their objective using BonaFide labels. However, I believe they should more clearly distinguish acknowledgment of a misleading hint from general reasoning faithfulness. Predicting whether a model acknowledges a hint does not necessarily establish whether its explanation reflects the actual reasoning process (Jacovi & Goldberg, 2020; Matton et al., 2025).
3.2 Step Encoding: The segmented multi-stage heuristic is not sufficiently explained. Although the authors mention splitting and merging operations, the exact criteria and their influence on the resulting representations remain unclear. Considering that all subsequent geometric measurements depend on segmentation, I would recommend a sensitivity analysis. Additionally, why is a general-purpose sentence encoder appropriate for representing reasoning steps? Semantic similarity does not necessarily imply similarity in reasoning behavior (Reimers & Gurevych, 2019).
3.3 Clustering Space Construction: The authors state that K-Means captures prototypical reasoning patterns. However, from my understanding, clustering sentence embeddings primarily identifies similarities in their semantic representation space. How do the authors establish that these clusters correspond to reasoning patterns? Previous work has shown that sentence-embedding geometry requires careful interpretation (Tehenan, 2025; Nielsen & Hansen, 2024). Why does K vary only over {5, 10, 20}? Furthermore, the authors state, "We do not assign fixed cognitive roles to their centroids". This is not necessarily contradictory, but it makes the claim about prototypical reasoning patterns less clear. I believe the authors should explain what relationship these clusters have with reasoning.
3.4 Geometric Feature Extraction: The authors mention eight geometric features, but their presentation could be clearer. Equations 2–10 contain nine numbered equations, including an intermediate similarity definition. The authors should explicitly identify the eight final features and their relationship with the proposed representation more easily to the reader. More importantly, six of these eight features are independent of the order of reasoning steps. Therefore, most of the representation describes the distribution of sentence embeddings rather than their trajectory. I believe this is a significant weakness considering the motivation of the paper. A step-permutation experiment would help establish whether ordering actually contributes to the results.
3.4.1 Outlier Features: I miss a stronger motivation for using distances to centroids as indicators of faithfulness. Furthermore, why do the authors use Euclidean distance? Sentence-embedding spaces can exhibit anisotropy and hubness (Li et al., 2020; Nielsen & Hansen, 2024). The authors should justify this choice and examine whether alternative distances or embedding normalization affect the results.
3.4.2 Coherence Features: More motivation is needed regarding why adjacent-step cosine similarity is related to faithfulness. Textual coherence does not necessarily imply faithful reasoning (Barzilay & Lapata, 2008; Jacovi & Goldberg, 2020). Also, Table 2 shows that removing coherence barely changes performance, which raises questions about the contribution of the trajectory-dependent features.
3.5 Scoring: The distinction between eight-dimensional and 24-dimensional inputs should be clearer. Also, the hyperparameters are described using implementation-specific notation (library) rather than a more general mathematical formulation (class balance and C). Another concern is that coherence features do not depend on K, so concatenating the three resolutions appears to repeat these measurements.
Experiments:
4.2 Geometric View of the Traces: This section presents trajectories through UMAP projections, but does not sufficiently explain their relevance to reasoning faithfulness. Figure 2 shows selected examples, and the authors acknowledge that these were chosen to illustrate clear signatures. However, the visualization does not explain what the observed movements mean or why particular geometric patterns should indicate concealed reasoning. The results remain descriptive rather than interpretable. I would recommend connecting the geometric changes with specific reasoning-step examples.
4.3 Held-Out-Model Prediction: The experimental results raise important concerns. SemFaith achieves 0.885 AUROC, compared with 0.869 for the length baseline and 0.991 for TF-IDF. This suggests that much of the predictive information may already be captured by simpler properties of the text (Geirhos et al., 2020; McCoy et al., 2019). I know the authors recognise this in many paragraphs, but why this information is lost?
Why do the authors not compare against pooled sentence embeddings using the same classifier? This would establish whether clustering and geometric statistics provide meaningful additional information. Also, evaluating TF-IDF combined with SemFaith would help demonstrate whether the geometric features are complementary. Regarding lines 345–349, the authors discuss existing methods but do not evaluate them using the same traces and protocol. I understand the limitation, but this makes the practical contribution more difficult to establish.
4.4 Length Restriction and Full-Sample Diagnostics: The length-matched experiment is interesting, and the improvement over the length-only baseline provides evidence of additional predictive information. However, matching length does not control differences in semantic content, writing style, or lexical cues. TF-IDF still reaches 0.981 AUROC, which suggests that these remaining differences are important.
Regarding Figure 3, the authors refer to a per-model panel in line 369, but Figure 3(c) does not identify the individual models. Furthermore, the panel is titled "Matched protocol", although its displayed AUROC range does not appear consistent with the reported matched length-control AUROC of 0.533. This makes the figure difficult to interpret, and I would recommend clarifying the evaluation protocol and labeling the models.
4.5 Feature Families and Hint Constructions: The results of Table 2 are particularly important but insufficiently discussed. Removing occupancy features reduces AUROC from 0.859 to 0.757, while removing coherence features changes it only to 0.856. This suggests that the model mainly relies on cluster distributions rather than relationships between consecutive reasoning steps. How do the authors justify the trajectory-based motivation considering these results?
Additionally, hint categories such as "error message" are mentioned without sufficient explanation in the main text. The category-level analysis also includes very few positive examples in some cases, making the conclusions difficult to generalize. A leave-one-hint-category-out evaluation would provide stronger evidence of transfer.
4.6 Where the Geometric Signal Comes From: This is one of my main concerns with the paper. The section mostly reports differences in occupancy entropy, cluster concentration, trace length, and AUROC, but does not deeply analyze how these measurements relate to reasoning faithfulness.
The authors acknowledge that some geometric differences are associated with trace length. However, the analysis does not explain why particular geometric configurations should correspond to honest acknowledgment or concealed hint use. There is no sufficiently developed interpretation connecting individual reasoning steps, their semantic relationships, and the resulting geometric measurements.
Previous work has evaluated reasoning faithfulness through causal interventions and controlled comparisons (Lanham et al., 2023; Matton et al., 2025; Zaman & Srivastava, 2025). I am not suggesting that SemFaith needs these interventions at inference time, but they could provide stronger experimental evidence for validating the geometric features.
Currently, the analysis remains superficial and mainly follows predictive correlations. I believe the experiments establish that sentence-embedding statistics correlate with BonaFide labels, but not that they meaningfully characterize faithful reasoning. This distinction is fundamental to the contribution of the paper.
4.7 Multiple Resolutions: The improvement from the strongest single resolution (0.859) to the combined representation (0.885) is modest. The authors should provide stronger evidence regarding why these particular resolutions are necessary and whether the improvement is stable across models, clustering configurations, and random seeds.
References:
- Barzilay, R., & Lapata, M. (2008). Modeling Local Coherence: An Entity-Based Approach. Computational Linguistics, 34(1), 1–34.
- Damirchi, H., Meza De la Jara, I., Abbasnejad, E., Shamsi, A., Zhang, Z., & Shi, J. (2026). Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning. ACL 2026.
- Geirhos, R., et al. (2020). Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence, 2, 665–673.
- Jacovi, A., & Goldberg, Y. (2020). Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? ACL 2020.
- Lanham, T., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.
- Li, B., et al. (2020). On the Sentence Embeddings from Pre-trained Language Models. EMNLP 2020.
- Matton, K., Ness, R., Guttag, J., & Kiciman, E. (2025). Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations. ICLR 2025.
- McCoy, R. T., Pavlick, E., & Linzen, T. (2019). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. ACL 2019.
- Nielsen, B. M. G., & Hansen, L. K. (2024). Hubness Reduction Improves Sentence-BERT Semantic Spaces. Proceedings of Machine Learning Research, 233, 181–204.
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. EMNLP-IJCNLP 2019.
- Tehenan, M. (2025). Semantic Geometry of Sentence Embeddings. Findings of EMNLP 2025.
- Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.
- Zaman, K., & Srivastava, S. (2025). A Causal Lens for Evaluating Faithfulness Metrics. EMNLP 2025.
I've read the response but I think it still does not address most of my concerns, especially the most fundamental one about framing this as faithfulness measurement, so I keep my score.