acceptodds
Under review as a conference paper at ICLR 2027

SemFaith: Semantic Geometry for Chain-of-Thought Faithfulness Evaluation

Abstract

Can a reasoning trace reveal when a model conceals its use of a misleading hint? We investigate this question through the geometry of sentence embeddings, using BonaFide's hint-acknowledgment labels as an observable test case. Our hypothesis is that how a trace's steps spread across semantic regions, how abruptly they shift topic, and how far they stray from reference centroids carry information about whether the hint is acknowledged. SemFaith segments a chain of thought (CoT) into steps, embeds them with a fixed encoder, and summarizes these measurements at three clustering resolutions; a regularized linear head maps the resulting 24 features to a score without accessing the audited model's hidden states or querying an auxiliary judge. Across five held-out Instruct models (662 CoTs), SemFaith attains 0.885 mean fold AUROC. On 67 within-model pairs closely matched for step and word counts, its AUROC is 0.788, compared with 0.533 for a length-only control, showing signal beyond gross trace length. A supervised TF-IDF baseline reaches 0.991 on the full held-out evaluation, exposing strong lexical cues in this benchmark. These results identify a compact geometric predictor of hint concealment whose signal is not explained by gross trace length, while delimiting what BonaFide can establish about general reasoning faithfulness.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought faithfulness, reasoning evaluation, semantic geometry, sentence embeddings, model auditing

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_WDSGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores the strength of geometric features to predict whether a hint is acknowledged in the given reasoning trace. They use a previous benchmark's published reasoning traces as data source and train a probe based on total of 24 outlier, coherence and clustering distribution features chosen from different levels of granularities. They compare their probe against length-only and TF-IDF baselines and their proposed method lags behind TF-IDF baseline.

Critical strengths
  1. The paper conducts an ablation study on geometric features they use to train the probe.

  2. The authors compare their method against reasonable baselines and they are honest about the results (e.g. TF-IDF outperforming their method).

Critical weaknesses
  1. The propose method does not constitute a good way to measure faithfulness. The authors do not use BonaFide benchmark to evaluate their metric, they reduced faithfulness into one way of measurement defined in that benchmark (hint acknowledgement) and chasing that particular metric. The way probes are trained (only on reasoning steps where hints are embedded and answer change is guaranteed), establishing this method as a faithfulness predictor is a serious overclaim. One possible reframing may be presenting this method as a cheap replacement for expensive monitors to decide if the model is influenced by a hint or costly training approaches (e.g. Verbalization Finetuning [1]). However, it is not possible with current experiment design and results as it does not even have held-out hint generalization tests.

  2. No statistical signifance of results are reported across paper. For instance, in L318-320 the authors claim adding multiple granulaties help by saying "so using three resolutions adds 0.026 under this protocol. This supports retaining multiple granularities in the representation." but they do not report whether this improvement is statistically significant. There are similar instances across paper and confidence intervals, standard deviation or statistical signifance are not reported in any of tables or figures.

  3. While they conduct their experiments on a wide range of 5 model selection across different sizes, their model selection lacks reasoning models, which undermines the practicality of their method even more.

  4. There are some extreme cases for hint acknowledgement: (1) model may reason about the hint but decide not to follow that or it may just casually mention the hint but not base its decision on the hint. This is the biggest reason hint-based evaluations rely on LLM judges instead of mere lexical checks. (2) hint may not be verbalized inside CoT at all but it may substantially change the CoT (for instance skipping reasoning about the baseline answer). The first one is an important case to compare the proposed geometric probe against TF-IDF to see the effect of lexical cues. However, the paper lacks such detailed analysis on failure cases. The second one could be an imporant application of the core idea of using geometric features to describe CoT progression. Connecting it to causal effects of CoT [2, 3], the CoT geometry might explain causal hint effects even when hints are not explicitly mentioned.

References:

  1. Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning, Turpin et al., 2025

  2. Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization, Zaman & Srivastava, 2026

  3. Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thought, Zhao et al., 2026

Questions for authors
  1. How are hint mentions distirbuted across CoT examples used in training and test? What if model reasons about the hint multiple steps vs only single step? How would result differ?

  2. Are the sampled 4000 reasoning steps balanced across examples?

  3. Even splitting long CoT steps with words like "therefore" and "however", can be a confounder, did you conduct any ablation on this?

  4. In some places throughout the paper, \citet command used where \citep would have been more appropriate.

1 comment

I've read the response but I think it still does not address most of my concerns, especially the most fundamental one about framing this as faithfulness measurement, so I keep my score.

Reviewer_J5auOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

General point of view: The paper starts with an interesting and well-motivated problem: evaluating the faithfulness of chain-of-thought (CoT) reasoning using only generated text. The methodology is relatively clear, and the introduction explains the motivation well. However, my main concern is with the experimental evidence and the interpretation of the results. The authors propose that sentence-embedding geometry can reveal properties of reasoning faithfulness, but the experiments do not convincingly establish whether these measurements capture reasoning behavior or simply statistical properties of the generated text. Previous work has already shown the difficulty of establishing faithfulness from generated explanations (Turpin et al., 2023).

The experimental section is particularly weak regarding interpretability. Most analyses focus on AUROC and correlations between geometric features and benchmark labels, without sufficiently explaining how these features relate to faithful or unfaithful reasoning. I believe the paper needs stronger controlled experiments and a deeper analysis of its proposed geometric representation.

Abstract: Reading the abstract, it is relatively easy to understand that the authors are studying trajectories in the geometry of sentence embeddings to identify whether a model conceals its use of a misleading hint. However, this is not an entirely new research direction. Previous work, such as Truth as a Trajectory (Damirchi et al., 2026), has already investigated reasoning through geometric trajectories. Although the representations and objectives differ, the authors should better contextualize their contribution. BonaFide is mentioned without being adequately introduced. I recommend briefly explaining the benchmark before discussing its labels, making the abstract understandable without consulting the introduction.

The main contribution is also difficult to assess from the results. SemFaith achieves 0.885 AUROC, while a simpler TF-IDF baseline reaches 0.991 under the same evaluation protocol. This raises questions about the advantages of the proposed geometry. If the contribution is interpretability rather than predictive performance, the authors should emphasize this and explain what the geometric measurements reveal beyond lexical features. This distinction is important considering previous work on shortcut learning.

Introduction: The paper motivates the problem well, explains CoT, and introduces BonaFide. The method summary is clean and relatively easy to understand. However, I believe the authors should introduce the specific faithfulness problem earlier and explain more clearly that they are analyzing only generated textual traces, rather than internal reasoning representations.

Expressions such as "can the trace itself provide a useful signal about such omissions?" are relatively broad. Considering previous trajectory-based work (Damirchi et al., 2026), the authors should explicitly distinguish textual CoT trajectories from trajectories of internal model representations.

Regarding the results, I believe the paper loses relevance through the way the contribution is presented. The difference between SemFaith and TF-IDF could be interpreted negatively. An interesting point appears at the end of the related work section, where SemFaith is described as an interpretable structural predictor. I think this should be emphasized and better defined in the introduction. Also, the authors could refer to Figure 1 when explaining the individual stages of the method.

Figure 1: The figure is generally well organized and easy to understand. However, the examples are too small, especially for a printed version. Some textual examples also contain grammatical inconsistencies. The selection of the number of centroids is unclear. Why are K = 5, 10, and 20 appropriate? Established clustering validation methods could provide additional justification. Additionally, the figure shows the concatenation of features into a 24-dimensional representation, while Section 3.5 introduces an eight-dimensional representation. Although these correspond to different configurations, this distinction is not sufficiently clear in the figure.

Method: 3.1 Problem Formulation: The authors define their objective using BonaFide labels. However, I believe they should more clearly distinguish acknowledgment of a misleading hint from general reasoning faithfulness. Predicting whether a model acknowledges a hint does not necessarily establish whether its explanation reflects the actual reasoning process (Jacovi & Goldberg, 2020; Matton et al., 2025).

3.2 Step Encoding: The segmented multi-stage heuristic is not sufficiently explained. Although the authors mention splitting and merging operations, the exact criteria and their influence on the resulting representations remain unclear. Considering that all subsequent geometric measurements depend on segmentation, I would recommend a sensitivity analysis. Additionally, why is a general-purpose sentence encoder appropriate for representing reasoning steps? Semantic similarity does not necessarily imply similarity in reasoning behavior (Reimers & Gurevych, 2019).

3.3 Clustering Space Construction: The authors state that K-Means captures prototypical reasoning patterns. However, from my understanding, clustering sentence embeddings primarily identifies similarities in their semantic representation space. How do the authors establish that these clusters correspond to reasoning patterns? Previous work has shown that sentence-embedding geometry requires careful interpretation (Tehenan, 2025; Nielsen & Hansen, 2024). Why does K vary only over {5, 10, 20}? Furthermore, the authors state, "We do not assign fixed cognitive roles to their centroids". This is not necessarily contradictory, but it makes the claim about prototypical reasoning patterns less clear. I believe the authors should explain what relationship these clusters have with reasoning.

3.4 Geometric Feature Extraction: The authors mention eight geometric features, but their presentation could be clearer. Equations 2–10 contain nine numbered equations, including an intermediate similarity definition. The authors should explicitly identify the eight final features and their relationship with the proposed representation more easily to the reader. More importantly, six of these eight features are independent of the order of reasoning steps. Therefore, most of the representation describes the distribution of sentence embeddings rather than their trajectory. I believe this is a significant weakness considering the motivation of the paper. A step-permutation experiment would help establish whether ordering actually contributes to the results.

3.4.1 Outlier Features: I miss a stronger motivation for using distances to centroids as indicators of faithfulness. Furthermore, why do the authors use Euclidean distance? Sentence-embedding spaces can exhibit anisotropy and hubness (Li et al., 2020; Nielsen & Hansen, 2024). The authors should justify this choice and examine whether alternative distances or embedding normalization affect the results.

3.4.2 Coherence Features: More motivation is needed regarding why adjacent-step cosine similarity is related to faithfulness. Textual coherence does not necessarily imply faithful reasoning (Barzilay & Lapata, 2008; Jacovi & Goldberg, 2020). Also, Table 2 shows that removing coherence barely changes performance, which raises questions about the contribution of the trajectory-dependent features.

3.5 Scoring: The distinction between eight-dimensional and 24-dimensional inputs should be clearer. Also, the hyperparameters are described using implementation-specific notation (library) rather than a more general mathematical formulation (class balance and C). Another concern is that coherence features do not depend on K, so concatenating the three resolutions appears to repeat these measurements.

Experiments: 4.2 Geometric View of the Traces: This section presents trajectories through UMAP projections, but does not sufficiently explain their relevance to reasoning faithfulness. Figure 2 shows selected examples, and the authors acknowledge that these were chosen to illustrate clear signatures. However, the visualization does not explain what the observed movements mean or why particular geometric patterns should indicate concealed reasoning. The results remain descriptive rather than interpretable. I would recommend connecting the geometric changes with specific reasoning-step examples.

4.3 Held-Out-Model Prediction: The experimental results raise important concerns. SemFaith achieves 0.885 AUROC, compared with 0.869 for the length baseline and 0.991 for TF-IDF. This suggests that much of the predictive information may already be captured by simpler properties of the text (Geirhos et al., 2020; McCoy et al., 2019). I know the authors recognise this in many paragraphs, but why this information is lost?

Why do the authors not compare against pooled sentence embeddings using the same classifier? This would establish whether clustering and geometric statistics provide meaningful additional information. Also, evaluating TF-IDF combined with SemFaith would help demonstrate whether the geometric features are complementary. Regarding lines 345–349, the authors discuss existing methods but do not evaluate them using the same traces and protocol. I understand the limitation, but this makes the practical contribution more difficult to establish.

4.4 Length Restriction and Full-Sample Diagnostics: The length-matched experiment is interesting, and the improvement over the length-only baseline provides evidence of additional predictive information. However, matching length does not control differences in semantic content, writing style, or lexical cues. TF-IDF still reaches 0.981 AUROC, which suggests that these remaining differences are important.

Regarding Figure 3, the authors refer to a per-model panel in line 369, but Figure 3(c) does not identify the individual models. Furthermore, the panel is titled "Matched protocol", although its displayed AUROC range does not appear consistent with the reported matched length-control AUROC of 0.533. This makes the figure difficult to interpret, and I would recommend clarifying the evaluation protocol and labeling the models.

4.5 Feature Families and Hint Constructions: The results of Table 2 are particularly important but insufficiently discussed. Removing occupancy features reduces AUROC from 0.859 to 0.757, while removing coherence features changes it only to 0.856. This suggests that the model mainly relies on cluster distributions rather than relationships between consecutive reasoning steps. How do the authors justify the trajectory-based motivation considering these results?

Additionally, hint categories such as "error message" are mentioned without sufficient explanation in the main text. The category-level analysis also includes very few positive examples in some cases, making the conclusions difficult to generalize. A leave-one-hint-category-out evaluation would provide stronger evidence of transfer.

4.6 Where the Geometric Signal Comes From: This is one of my main concerns with the paper. The section mostly reports differences in occupancy entropy, cluster concentration, trace length, and AUROC, but does not deeply analyze how these measurements relate to reasoning faithfulness.

The authors acknowledge that some geometric differences are associated with trace length. However, the analysis does not explain why particular geometric configurations should correspond to honest acknowledgment or concealed hint use. There is no sufficiently developed interpretation connecting individual reasoning steps, their semantic relationships, and the resulting geometric measurements.

Previous work has evaluated reasoning faithfulness through causal interventions and controlled comparisons (Lanham et al., 2023; Matton et al., 2025; Zaman & Srivastava, 2025). I am not suggesting that SemFaith needs these interventions at inference time, but they could provide stronger experimental evidence for validating the geometric features.

Currently, the analysis remains superficial and mainly follows predictive correlations. I believe the experiments establish that sentence-embedding statistics correlate with BonaFide labels, but not that they meaningfully characterize faithful reasoning. This distinction is fundamental to the contribution of the paper.

4.7 Multiple Resolutions: The improvement from the strongest single resolution (0.859) to the combined representation (0.885) is modest. The authors should provide stronger evidence regarding why these particular resolutions are necessary and whether the improvement is stable across models, clustering configurations, and random seeds.

References:

  1. Barzilay, R., & Lapata, M. (2008). Modeling Local Coherence: An Entity-Based Approach. Computational Linguistics, 34(1), 1–34.
  2. Damirchi, H., Meza De la Jara, I., Abbasnejad, E., Shamsi, A., Zhang, Z., & Shi, J. (2026). Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoning. ACL 2026.
  3. Geirhos, R., et al. (2020). Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence, 2, 665–673.
  4. Jacovi, A., & Goldberg, Y. (2020). Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? ACL 2020.
  5. Lanham, T., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.
  6. Li, B., et al. (2020). On the Sentence Embeddings from Pre-trained Language Models. EMNLP 2020.
  7. Matton, K., Ness, R., Guttag, J., & Kiciman, E. (2025). Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations. ICLR 2025.
  8. McCoy, R. T., Pavlick, E., & Linzen, T. (2019). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. ACL 2019.
  9. Nielsen, B. M. G., & Hansen, L. K. (2024). Hubness Reduction Improves Sentence-BERT Semantic Spaces. Proceedings of Machine Learning Research, 233, 181–204.
  10. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. EMNLP-IJCNLP 2019.
  11. Tehenan, M. (2025). Semantic Geometry of Sentence Embeddings. Findings of EMNLP 2025.
  12. Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.
  13. Zaman, K., & Srivastava, S. (2025). A Causal Lens for Evaluating Faithfulness Metrics. EMNLP 2025.
Critical strengths
  • The paper addresses an interesting and relevant problem: evaluating CoT faithfulness without accessing the internal representations of the model. The motivation is clear, and the proposed methodology is relatively easy to understand.
  • The authors provide a structured evaluation using held-out models, length-matched comparisons, and lexical baselines. I appreciate that they acknowledge the limitations of their results, particularly the strong performance of TF-IDF and the influence of trace length.
Critical weaknesses
  • As was described in the summary of the paper, the main weakness (from my point of view) is that the experimental evidence does not convincingly establish a relationship between the proposed geometric features and reasoning faithfulness. Most of the analysis focuses on predictive correlations rather than explaining what the geometric patterns reveal about faithful or unfaithful reasoning.

  • Six of the eight geometric features are independent of reasoning-step order, and removing coherence features barely affects performance (0.859 to 0.856). This raises serious questions about whether the method captures reasoning trajectories or simply distributions of sentence embeddings.

  • The proposed method achieves 0.885 AUROC, compared with 0.991 for TF-IDF and 0.869 for the length baseline. The authors do not establish whether the geometric representation contributes information beyond simpler semantic representations. Important comparisons, such as pooled sentence embeddings or TF-IDF combined with SemFaith, are missing.

  • The geometric interpretation in Section 4.6 is superficial, and several methodological decisions, including the selection of K and the use of Euclidean distance, lack sufficient justification. Additionally, some figures are difficult to interpret, particularly Figure 3, where model labels are missing and the panel title is inconsistent with the discussion.

Questions for authors
  1. How do the authors establish that the kk-means clusters capture distinct reasoning patterns rather than mere semantic similarities among sentence embeddings? Furthermore, what is the rationale behind selecting k∈{5,10,20}k \in \{5, 10, 20\} ? Why were standard selection heuristics, such as the elbow method or silhouette analysis, not utilized?

  2. Considering that six of the eight features do not depend on step ordering, how do the authors justify describing the representation as a reasoning trajectory? What happens if the CoT steps are randomly permuted?

  3. Why do the authors not compare against pooled sentence embeddings or evaluate TF-IDF combined with SemFaith? How can the contribution of geometric information beyond simpler representations be established?

  4. Why is Euclidean distance selected for the embedding space? Are the embeddings normalized, and how sensitive are the results to the distance metric and segmentation heuristic?

  5. In Section 4.6, what evidence establishes a meaningful relationship between geometric measurements and reasoning faithfulness rather than correlations with BonaFide labels? Could the authors provide controlled examples with similar semantic content and length but different faithfulness labels?

  6. Regarding Figure 3(c), where are the individual model labels? Does the panel represent the matched evaluation or the full held-out evaluation? The title and reported AUROC values appear inconsistent.

1 comment

I went through the general responses, and I still don’t see answers to most of the points I raised. In particular, I’m still unclear how the method measures faithfulness. This was my main concern, so I’m keeping my score.

Reviewer_Q3cUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SemFaith summarizes fixed sentence embeddings of CoT steps using distance, adjacent-step coherence, and cluster-occupancy statistics at three K-means resolutions, yielding 24 features for regularized logistic regression. On 662 Instruct-model CoTs from BonaFide, it achieves 0.885 mean held-out-model AUROC; on 67 within-model length-matched pairs, the pooled AUROC is 0.788. The intended target is primarily hint acknowledgment/concealment, rather than direct recovery of hidden reasoning.

Critical strengths
  1. Appropriate scope and informative controls. The paper carefully distinguishes prediction of an operational faithfulness label from measurement of hidden causal computation. Reporting the near-ceiling TF-IDF baseline, strong length effects, and limitations of the selected visualizations makes the conclusions well calibrated and provides useful information about the benchmark.

  2. Reasonable cross-generator evaluation with restricted access. The corrected protocol holds out entire generators and fits centroids, feature scaling, and classifiers using training models only. Length matching and feature-family ablations strengthen the analysis, while the predictor requires neither audited-model internals nor an auxiliary LLM judge.

Critical weaknesses
  1. Incremental novelty and insufficiently isolated geometric benefit. The novelty is primarily a trace-level, multi-resolution combination of established embeddings, K-means, summary statistics, and logistic regression; this differs from semantic entropy across sampled answers (Kuhn et al., 2023). Compactness is a legitimate benefit, but the 0.885 AUROC, compared with 0.869 for length and 0.991 for TF-IDF (Table 1), does not establish a predictive advantage over other compact semantic representations. A pooled-embedding baseline using the same encoder, ideally also at a matched feature budget, is missing, leaving the contribution of the particular geometric summaries insufficiently isolated.

  2. Limited empirical coverage and unquantified uncertainty. Multi-resolution geometry improves over K = 5 in all five held-out models (Supplementary Table 5), and geometry exceeds length in every model's matched subset (Supplementary Table 6), so there is consistent directional evidence. However, the 0.026 mean multi-resolution gain and the comparison on 67 matched pairs lack confidence intervals or paired uncertainty estimates. With five generators from three families on one benchmark, this limits confidence in the precision and broader generality of the effect estimates, rather than invalidating the observed improvements.

Questions for authors
  1. Clarification of multi-resolution construction. The coherence statistics in Equations (5)-(7) are independent of K, yet are repeated in all three feature blocks. Does the gain persist when these two features are included only once? Under L2 regularization, duplication changes their effective penalty, so this control would help attribute the gain to additional clustering resolutions.

  2. Evidence needed to assess novelty. Can a mean-pooled step-embedding classifier using the same encoder be compared under the identical full and length-matched protocols? A dimension-matched variant, with dimensionality reduction fitted on training folds only, would help isolate what the clustering summaries add beyond a generic semantic representation.

1 comment

Thanks for the authors' response. Removing repeated coherence features is helpful. However, my main concern is still not solved, because there is no mean-pooled step embedding baseline from the same encoder (including a same-dimension version) to show that K-means geometric features are really better than simple pooling. Besides, the experiments are still only on one dataset, so I will keep my original score.

Reviewer_bTtcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SemFaith, a method for evaluating chain-of-thought (CoT) faithfulness using semantic geometry. The authors extract geometric features from sentence embeddings of reasoning steps, including outlier distances, semantic coherence, and cluster occupancy. These features are calculated at three clustering resolutions and fed into a logistic regression classifier to predict whether a model acknowledges or conceals the influence of a misleading hint.

Critical strengths

The idea of examining the semantic structure of reasoning traces to evaluate faithfulness is interesting and worth exploring. Unlike approaches that require access to model internals or additional LLM-based judges, the proposed method works directly with the generated reasoning text.

I appreciate that the authors include simple baselines such as trace length and TF-IDF, rather than comparing only against more complex methods. The length-matched evaluation is also useful, as it attempts to examine whether the proposed features capture anything beyond differences in trace length.

Critical weaknesses
  1. Extremely poor readability and presentation.

The paper is very difficult to read and follow. It lacks a coherent flow and narrative, making it unnecessarily challenging to understand the main idea and the contributions. Even though LLMs can be used to polish scientific writing, the responsibility for ensuring that a paper is clear and understandable ultimately lies with the authors. In its current form, I found the paper hardly reviewable.

Another particularly confusing issue is the reference to an earlier version of SemFaith in Section 4.1. The authors refer to "the original SemFaith evaluation" and "the prior result" and discuss correcting an earlier evaluation that incorrectly included 100 DeepSeek Distill traces. However, it is not sufficiently clear what this previous evaluation refers to or where its details can be found. If this refers to an internal version of the work, the authors should explain that explicitly and provide the necessary context.

  1. Weak empirical justification for the proposed method.

The experimental results do not provide sufficiently convincing evidence of the usefulness of the proposed approach. A simple length-only baseline achieves 0.869 AUROC, compared to SemFaith's 0.885, an improvement of only 0.016. Moreover, a straightforward TF-IDF baseline substantially outperforms the proposed method, achieving 0.991 AUROC.

Although the length-matched experiment provides some evidence of additional signal, it is based on only 67 matched pairs. The paper also does not report confidence intervals or statistical significance tests to establish the reliability of the improvements.

Additionally, the paper relies entirely on the BonaFide dataset, but the labeling methodology is not sufficiently explained. For example, the authors report 98.9% precision against human annotations without adequately explaining the human annotation process or clearly attributing this figure to the original dataset. This makes it difficult to evaluate the reliability of the labels and the experimental findings.

Overall, I do not find the empirical evidence strong enough to justify the proposed contribution.

Questions for authors
  1. Contribution beyond simple baselines: Given that the length-only baseline achieves 0.869 AUROC compared with SemFaith's 0.885, what evidence demonstrates that the proposed semantic geometry captures meaningful information beyond trace length? Is the difference statistically significant?

  2. Performance relative to TF-IDF: TF-IDF substantially outperforms SemFaith across the reported settings. What empirical evidence supports the utility of the proposed geometric approach over this simpler representation?

  3. Dataset and labeling: Could the authors clarify how the BonaFide labels were generated and validated? Specifically, what does the reported 98.9% precision against human annotations measure, and was this validation conducted by the authors or reported in the original dataset paper?

  4. Previous evaluation: What exactly do the authors mean by the "original SemFaith evaluation" and "prior result" in Section 4.1? Were these internal experiments, or does an earlier publicly available version exist? How did the correction involving the 100 DeepSeek-R1-Distill traces affect the reported results?

1 comment

We thank the author for the response. But my major concerns are still unaddressed, for example about empirical justification of the method and results. So I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.