- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ReasonOps, an unsupervised method for segmenting LLM reasoning traces into discourse operators. The method extracts sentence-initial three-token phrases and clusters them into seven categories, including INFERRING, HYPOTHESIZING, and BACKTRACKING. The authors analyze 44,662 traces from 12 models across eight benchmarks, comparing operator usage across models, tasks, and answer outcomes. They also use these operators for model identification and correctness prediction, achieving a macro-AUC of 0.987 with additional text features and a global WP-AUC of 0.701, respectively. Early correctness prediction and inference-time candidate selection are also examined.
Critical strengths
S1. Large-scale analysis across models and reasoning tasks. The study covers 44,662 reasoning traces from 12 models across six families and eight benchmarks. The authors examine operator usage across different tasks, model families, problem difficulty levels, and answer outcomes. The OLMo checkpoint comparison is a useful addition. It shows clear differences in operator distributions under different post-training recipes, providing evidence that the extracted discourse patterns reflect differences in model behavior beyond the immediate task.
S2. Predictive results supported by extensive experiments. The combined operator and anchor-text features achieve a macro-AUC of 0.987 for model identification. For correctness prediction, the operator-only OST reaches a global WP-AUC of 0.701, outperforming the reported scalar baselines. It also achieves 0.664 WP-AUC when only 50% of a reasoning trace is available. The paper includes human annotation results, alternative clustering methods and sentence encoders, and a detailed hyperparameter sensitivity study. These experiments help characterize both the performance and limitations of the proposed representation.
Critical weaknesses
W1. Limited semantic consistency and coverage of the induced discourse operators. ReasonOps relies on sentence-initial phrases to assign operator labels. Such phrases can indicate discourse functions, but their interpretation may depend on the surrounding context. The human validation results suggest that some of the proposed categories are difficult to distinguish consistently. The two annotators agree with the induced labels on 62.4% and 52.0% of the evaluated spans, respectively, and agree with each other on 58.0%. Appendix N shows particularly low agreement for HYPOTHESIZING and CONSTRAINING. Another concern comes from Appendix Q. Changing the sentence encoder produces substantially different cluster assignments (ARI = 0.350–0.418), while downstream prediction scores remain relatively close. The predictive information appears more stable than the particular semantic categories used to represent it. Coverage is also not fully established. Appendix O reports that 28.0% of spans are unlabeled structural boundaries in a sample of 6,000 Claude traces. The authors acknowledge that this sample is not representative of the full corpus, but the corresponding figures for other models and benchmarks are not provided. Taken together, these results leave the consistency and coverage of the seven-category vocabulary uncertain.
W2. The contribution of grouping the original phrases into seven operators is not clearly isolated. The paper shows that the operator representation is predictive. The model-identification result reaches 0.987 macro-AUC with additional anchor-text features, while Appendix Q reports approximately 0.947 AUC using operator features alone. The operator-only OST also achieves 0.701 WP-AUC for correctness prediction. What remains unclear is whether the clustering step itself contributes to these results. The authors vary the number of operators, clustering algorithms, and sentence encoders, but do not compare the seven-category representation directly with the original three-token phrase features under matched conditions. Without this comparison, it is difficult to separate the benefit of the proposed abstraction from the predictive information already present in the sentence openings. There is also relevant prior work on reasoning-trace structure, including The CoT Encyclopedia and ReasoningFlow. Their representations differ from ReasonOps, but a clearer discussion of what the seven-operator vocabulary reveals that these approaches do not would strengthen the paper's contribution. Finally, Appendix S reports improvements over random candidate selection, but no statistically significant improvement over selection by trace length. The practical benefit of the operator representation in this application therefore remains limited.
Questions for authors
Q1. Semantic consistency of the operators. The two human annotators recover 62.4% and 52.0% of the induced labels, with considerable variation across categories. What accounts for the disagreements? Are certain operator definitions overlapping, or do the errors mainly arise from assigning labels using sentence-initial phrases?
Q2. Contribution of the clustering step. Could the authors compare the seven-operator representation against unclustered three-token phrase features using the same prediction models and data splits? This would help determine whether the clustering step adds value beyond the original lexical information.
Q3. Coverage of unlabeled spans. Appendix O reports that 28.0% of spans are unlabeled in the examined Claude sample. How does this proportion vary across models and benchmarks? Does excluding these spans affect the coverage or predictive performance of the resulting operator sequences?
Thank you for your review
Thank you sincerely for your thoughtful review. We have not decided to update the paper at this time for this conference based on our feedback from reviewers, but we are genuinely thankful for your time. We will take your comments into account in our future revisions to the paper.