acceptodds
Under review as a conference paper at ICLR 2027

SemNorm: Semantic Normalization for Reasoning Traces Evaluation

Abstract

As large reasoning models generate increasingly lengthy and seemingly coherent natural language outputs, the difficulty of evaluating them at the trace-level has increased. In tandem, the verification of lengthy traces no longer provide sufficiently granular information about specific step-by-step errors or premise validity. To address these limitations in natural language reasoning traces, Semantic Normalization (SemNorm) introduces two components: 1. Atomic extraction, to process reasoning traces into atomic claims-level statements; and 2. Directed graph construction, which builds a knowledge graph from the pairwise entailment between atomic statements. This framework is founded on two observations. First, lexical-based methods are inadequate for establishing information content in diverse natural language generations. Second, embedding-based retrieval methods establish undirected semantic similarity, which leaves causal and implicative statements indistinguishable. To verify the information content in the extracted graph, SemNorm is applied as a scoring metric for reasoning quality and validity, and benchmarked against embedding-based methods as a semantic equivalence measure in perturbation-based experiments. Thus, this work discusses methods for estimating trace-level metrics, and introduces a directed reasoning graph representation for computing graph-derived scores which are comparably monotonic and on par with existing NLI-based scorers and embedding-based scorers for reasoning trace lexical, quality and correctness indicators and as a retriever model for RAG.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Chain-of-Thought (CoT), Reasoning, Information Extraction, Semantic Representation, Knowledge Representation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xdR7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SemNorm for evaluating LLM reasoning traces against reference traces. An LLM decomposes traces into atomic statements. DeBERTa-large-mnli scores all pairs in both directions: mutually entailing statements are merged into nodes, and one-way entailments become directed edges. From this the authors derive a pairwise score, a matching-based trace score (SemNormBP) and a linearized graph used as context (SemNormKG). Validation is on medical data only: sentence-level perturbations and note-level monotonicity (ACI-Bench, CliniKnote), trace-level AUROC against LLM-judge labels on MedExQA, and multi-turn answer updates compared to RAG. SemNorm is reported as on par with or better than lexical, embedding and NLI baselines.

Critical strengths
  1. The motivation is sound. Undirected similarity cannot tell apart a premise from its consequence, so a directional, entailment-based representation of reasoning traces is reasonable, especially for medical reasoning, where trace evaluation is far less developed than in math or code.
Critical weaknesses
  1. Contradiction between motivations and implementation:
    • The claim that they “opt to use smaller NLI models [...] for broader usability“ (line 052) is confounded by the methodology using 2 LLM calls for the split of the CoT into atomic segments, n2n^2 (where nn is the number of atomic segments) NLI runs for comparing all the steps. For short answers cot is viable, however for long ones is not. Additionally, they motivate their work by presenting the long CoT bottleneck, but they only use a dataset with short CoT, not showing how the method scales with length
    • They also present the problem of information granularity obtained with simple LLM evaluation. However, they only provide aggregated, quantitative evaluation, without showing some qualitative analysis (e.g., what kind of contradictions are found by the NLI model, how much the graphs are connected, the existence of false negatives given by alternative correct CoTs, etc.)
  2. Lack of deep literature understanding:
    • The rigid text-matching paradigm used in the paper assumes a single valid “golden path”, which penalise models for utilising perfectly valid alternative logical routes if their steps fail to explicitly entail the dataset's reference explanation. Modern reasoning evaluation has shifted away from this kind of reference-conformity, focusing instead on assessing the intrinsic logical soundness of the model's independently chosen trajectory. As the authors also stated in the related work, sota PRMs evaluate the validity of each intermediate step without requiring strict alignment to a single human-written reference path
    • Some relevant references are missing: a medical-domain PRM (Med-PRM, aclanthology.org/2025.emnlp-main.837.pdf) while claiming “the development of a generally applicable step-level scorer for the medical reasoning remains outstanding” (line 468); also, other methods applying NLI for similar scope are not cited like ROSCOE (ICLR 2023), ReCEval (EMNLP 2023)
Questions for authors

Clarification requests:

  1. Could you provide a qualitative case study and a visual representation of a generated SemNorm graph (in the appendix?) to demonstrate how it successfully isolates a specific hallucination compared to a standard LLM-as-a-judge baseline?
  2. How does the SemNorm metric differentiate between a model hallucinating an incorrect fact versus a model utilizing a semantically valid alternative reasoning path that simply bypasses the specific conceptual steps outlined in the ground truth?
  3. Which thresholds were used: τ/t\tau/t for the graph, edge acceptance in 3-chain recovery (cosine and NLI), and the cutoffs for accuracy and F1 in Tables 6-10? How were they chosen, and were they calibrated per method? The embeddings' recall near 1.0 with precision around 0.5 suggests an uncalibrated cutoff.
  4. Are Table 1's AUROC columns swapped? Is the Table 11 mednli value correct? What do the parentheses denote?
  5. Which LLM performs atomic extraction and which filters the contradictions, and with which prompts? Was either validated by humans?
  6. How are sentences segmented? What are the 8 perturbation types and the 4 canonical edges?
  7. Why is GIST-small (33M) described as comparable in size to DeBERTa-large? Do the Exp. 1 conclusions hold against a size-matched embedder?

Suggestions for future work:

  1. Restructure the paper and maybe reframe the paper by making the medical application more central.
  2. Include all prompts and settings in the appendix rather than deferring them to the code release.
  3. Fix typos and inconsistencies, e.g. “SemNorm (SemNorm)” (abstract), "there exists a evaluation" (Section 1), "construct an directed graph" (Section 2.2.3), "are they are" (line 162), "LLm" (line 185).
Reviewer_xVd9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose an evaluation framework for longer reasoning traces (of multiple sentences) and document length outputs. They view this evaluation as determining semantic equivalence for which they use smaller NLI models. In particular they use these NLI models to construct a digraph which captures how generated statements are entailed. I am very sorry to say that, beyond this, I have not been able to understand exactly how the authors proceed. They claim that the notion of SemNorm that they introduce conducts an (1) "atomic extraction" to decompose reasoning traces (mind you, a reasoning trace is a set of sentences) into atomic claims-level statements and (2) digraph construction that captures pairwise entailment between these atomic statements. But, unfortunately, in spite of the significant time and effort I have put into reviewing this paper, I have not been able determine exactly how the process works.

Critical strengths

It is almost impossible for me to identify the strengths of this paper without understanding exactly what the authors are doing. It seems that evaluating the complete reasoning trace, instead of just the response that the LLM produces, could be very useful in critical applications. Accounting for semantic equivalence is of course superior to just looking at lexical differences, but then there are many research works that does exactly this. Decomposing traces (again, a trace is a set of sentences, according to the authors) into atomic units could make it easier to identify exactly what claims are supported, refuted, missing, or incorrect. A digraph-based entailment could also bring more information to the evaluation task if it in fact does capture more than what pairwise NLI scores could offer.

Critical weaknesses

The glaring weakness for me is that what the authors have presented does not provide a clear picture of exactly what is being done. -- Exactly how is the reference reasoning trace R_GT obtained or validated? At even a more fundamental level, critical application typically admit multiple valid reasoning paths (e.g., the medical application that the authors mention is a case in point). Consequently, semantic equivalence to one particular reference trace does not necessarily establish correctness. -- It seems to me that the reasoning trace is being represented as an unordered set of sentences. (I admit that I may not be grasping what the authors are saying.) By doing this, they are ignoring or discarding temporal and logical ordering. Wouldn't this be a serious drawback for a framework intended to evaluate reasoning rather than factual content? -- The authors specify conditions under which an "equivalence score" equals 0 or 1. Does that mean that no intermediate values are computed? It is also not clear how repeated propositions are tackled and whether there is any thresholding of the NLI outputs. -- What is the added value of a digraph beyond what one would get from pairwise NLI scores? In particular, do graph-theoretic properties reveal reasoning inconsistencies, unsupported inferences, or other failures that simpler methods cannot identify?

Questions for authors
  1. What precisely constitutes the reference reasoning trace R_GT? Where are these traces obtained, who verifies them, and what criteria are used for verification? Does verification establish the correctness of the final answer or the validity of every reasoning step?

  2. It seems to me that the authors first define a reasoning trace as a sequence of sentences and subsequently represent it as an unordered set. What justifies discarding sentence order, given that ordering may be essential to the logical structure of reasoning?

  3. What is an indivisible atomic statement? I mean what is its definition? How is the decomposition function f implemented, and how is its correctness validated? Is the decomposition unique?

It is quite possible that I am not grasping anything that the authors are saying, but this is a rare instance that I could not understand anything that the authors are stating. I was quite excited when I read the abstract because I am quite familiar with the graph theoretic concepts especially as they relate to digraphs. But I ended up disappointed.

Reviewer_oajwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes SemNorm, a framework for evaluating natural-language reasoning traces against reference explanations. Traces are decomposed into atomic statements with an LLM, and pairwise bidirectional NLI is computed between all statements. Mutually entailing statements are merged into nodes, and on-directional entailment becomes directed edge, producing a graph with provenance attributes. Several derived scores are defined: a signed entailment-minus-contradiction score, a bipartite-matching score, and a graph-merging variant that is converted back to text for retrieval.

Critical strengths
  • the motivation of this manuscript is valid
  • Small, but accessible models are proposed. Relying on a sub-1B NLI model rather than an LLM judge for pairwise scoring is attractive for cost and reproducibility.
  • Candid reporting of negatives are appripriate
Critical weaknesses
  • The eivdence may not support the central claims, and the manuscript contradicts itself. The conclusion states that SemNorm closes the gap in reasoning trace evaluations and outperforms embedding-based models. But the result show otherwise. In table 2, SemNormBP-Trace is the worst method (adjacent monotonic rate 39.6/37.0 vs. 82.6/85.0 for KaLM). At the sentence level, all methods are near ceiling, so thisexperiment does not discriminate.
  • Step-level error localization is never evaluated.
  • All experiments are in one domain with multiple choice explanations, while the title and abstract claim general reasoning-trace evaluation
  • No runtime or cost analysis is provided against FactScore or an LLM judge.
Questions for authors

Refer weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.