- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes SemNorm for evaluating LLM reasoning traces against reference traces. An LLM decomposes traces into atomic statements. DeBERTa-large-mnli scores all pairs in both directions: mutually entailing statements are merged into nodes, and one-way entailments become directed edges. From this the authors derive a pairwise score, a matching-based trace score (SemNormBP) and a linearized graph used as context (SemNormKG). Validation is on medical data only: sentence-level perturbations and note-level monotonicity (ACI-Bench, CliniKnote), trace-level AUROC against LLM-judge labels on MedExQA, and multi-turn answer updates compared to RAG. SemNorm is reported as on par with or better than lexical, embedding and NLI baselines.
Critical strengths
- The motivation is sound. Undirected similarity cannot tell apart a premise from its consequence, so a directional, entailment-based representation of reasoning traces is reasonable, especially for medical reasoning, where trace evaluation is far less developed than in math or code.
Critical weaknesses
- Contradiction between motivations and implementation:
- The claim that they “opt to use smaller NLI models [...] for broader usability“ (line 052) is confounded by the methodology using 2 LLM calls for the split of the CoT into atomic segments, (where is the number of atomic segments) NLI runs for comparing all the steps. For short answers cot is viable, however for long ones is not. Additionally, they motivate their work by presenting the long CoT bottleneck, but they only use a dataset with short CoT, not showing how the method scales with length
- They also present the problem of information granularity obtained with simple LLM evaluation. However, they only provide aggregated, quantitative evaluation, without showing some qualitative analysis (e.g., what kind of contradictions are found by the NLI model, how much the graphs are connected, the existence of false negatives given by alternative correct CoTs, etc.)
- Lack of deep literature understanding:
- The rigid text-matching paradigm used in the paper assumes a single valid “golden path”, which penalise models for utilising perfectly valid alternative logical routes if their steps fail to explicitly entail the dataset's reference explanation. Modern reasoning evaluation has shifted away from this kind of reference-conformity, focusing instead on assessing the intrinsic logical soundness of the model's independently chosen trajectory. As the authors also stated in the related work, sota PRMs evaluate the validity of each intermediate step without requiring strict alignment to a single human-written reference path
- Some relevant references are missing: a medical-domain PRM (Med-PRM, aclanthology.org/2025.emnlp-main.837.pdf) while claiming “the development of a generally applicable step-level scorer for the medical reasoning remains outstanding” (line 468); also, other methods applying NLI for similar scope are not cited like ROSCOE (ICLR 2023), ReCEval (EMNLP 2023)
Questions for authors
Clarification requests:
- Could you provide a qualitative case study and a visual representation of a generated SemNorm graph (in the appendix?) to demonstrate how it successfully isolates a specific hallucination compared to a standard LLM-as-a-judge baseline?
- How does the SemNorm metric differentiate between a model hallucinating an incorrect fact versus a model utilizing a semantically valid alternative reasoning path that simply bypasses the specific conceptual steps outlined in the ground truth?
- Which thresholds were used: for the graph, edge acceptance in 3-chain recovery (cosine and NLI), and the cutoffs for accuracy and F1 in Tables 6-10? How were they chosen, and were they calibrated per method? The embeddings' recall near 1.0 with precision around 0.5 suggests an uncalibrated cutoff.
- Are Table 1's AUROC columns swapped? Is the Table 11 mednli value correct? What do the parentheses denote?
- Which LLM performs atomic extraction and which filters the contradictions, and with which prompts? Was either validated by humans?
- How are sentences segmented? What are the 8 perturbation types and the 4 canonical edges?
- Why is GIST-small (33M) described as comparable in size to DeBERTa-large? Do the Exp. 1 conclusions hold against a size-matched embedder?
Suggestions for future work:
- Restructure the paper and maybe reframe the paper by making the medical application more central.
- Include all prompts and settings in the appendix rather than deferring them to the code release.
- Fix typos and inconsistencies, e.g. “SemNorm (SemNorm)” (abstract), "there exists a evaluation" (Section 1), "construct an directed graph" (Section 2.2.3), "are they are" (line 162), "LLm" (line 185).