- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces LOGICEVAL-X, a benchmark of 944 logical reasoning problems, and identifies difficulties in maintaining constraints during extended reasoning. Motivated by these findings, it explores symbolic solvers as tools for logical inference and proposes T-MCTS to train models to formulate problems, invoke solvers, and revise their solutions using tool feedback. The resulting LOGIC-X models achieve higher accuracy than the evaluated baselines on LOGICEVAL-X and outperform their base models on several established reasoning benchmarks.
Critical strengths
-
The paper addresses the important challenge of improving the reliability of language model reasoning and contributes a benchmark with human difficulty screening and expert verification.
-
The experiments cover several reasoning tasks and report improvements over multiple baselines, demonstrating the effectiveness of the trained models on the evaluated tasks.
Critical weaknesses
-
The connection between LOGICEVAL-X and the symbolic tool-integrated logical reasoning emphasized in the title is not sufficiently established. The benchmark evaluates final answer accuracy on complex logical problems, while the tool use study examines one approach to solving them. It remains unclear what the benchmark reveals specifically about symbolic tool use beyond general logical reasoning performance.
-
The paper does not compare T-MCTS with related methods for generating training trajectories, such as ReST-MCTS* (Zhang et al., 2024), which is discussed in Related Work. The SFT and outcome reward ablations do not establish whether T-MCTS offers advantages over existing search methods under comparable training data and computational budgets.
Questions for authors
-
What does LOGICEVAL-X reveal specifically about symbolic tool use beyond general logical reasoning performance?
-
How does T-MCTS compare with ReST-MCTS* under comparable training data and computational budgets?
5 comments
Response to Critical Weakness 2 & Question 2
Critical Weakness 2: The paper does not compare T-MCTS with related methods for generating training trajectories, such as ReST-MCTS* (Zhang et al., 2024), which is discussed in Related Work. The SFT and outcome reward ablations do not establish whether T-MCTS offers advantages over existing search methods under comparable training data and computational budgets.
Question 2: How does T-MCTS compare with ReST-MCTS* under comparable training data and computational budgets?
We thank the reviewer for raising this important comparison. We completely understand the value of comparing with ReST-MCTS*, which is a prominent work in this area. Due to computational constraints, we did not run ReST-MCTS* directly, as it requires training a separate neural Process Reward Model (PRM) from scratch for our specific domain, which is highly computationally expensive. However, we can clearly distinguish the architectural advantages of T-MCTS over ReST-MCTS* in the context of logical reasoning.
The fundamental difference lies in the source of the process reward. ReST-MCTS* relies on a trained neural PRM to evaluate intermediate reasoning steps. While effective in math domains, neural PRMs are highly susceptible to reward hacking in dense logical reasoning. They often assign high scores to fluent but logically flawed deductions. In contrast, T-MCTS relies on deterministic symbolic solver feedback (e.g., Z3, PyKE). A program's execution either succeeds or fails, providing an absolute, hallucination-free ground truth.
Furthermore, we believe our existing ablation studies in Table 4 effectively serve as a proxy to demonstrate this advantage under comparable budgets. We compared T-MCTS with Outcome RL (which uses terminal verification but lacks intermediate feedback) and sampling methods (Rejection Sampling and Best-of-N). The results show that T-MCTS significantly outperforms these baselines (e.g., 62.71% vs. 55.72% for Outcome RL). This indicates that the massive gain does not come from mere brute-force search, but precisely from our fine-grained, solver-grounded process rewards.
Finally, from an efficiency perspective, ReST-MCTS* requires training an additional PRM, which introduces extra computational overhead and potential error cascades. By bypassing the PRM entirely and using deterministic solvers, T-MCTS is more computationally efficient and reliable for logic tasks. We have added this comparative discussion to the revised Related Work section.
Words cannot fully express our gratitude for your favorable evaluation and your valuable time. Your recognition and encouragement mean a great deal to our team. We have earnestly addressed all your questions and hope our revisions meet your standards. If you have any further questions regarding our responses, please feel free to reach out, and we will gladly provide more details. If you are satisfied with our efforts, we would be sincerely grateful if you might consider maintaining or even slightly increasing the score. We will continue to work hard to ensure this paper is as rigorous and impactful as possible. Thank you again for your time, your generosity, and your uplifting support!
Apologies for the delay. Below are our detailed responses to your thoughtful feedback.
We would like to express our most sincere and heartfelt thanks for your time and for providing such a positive, encouraging, and constructive review. We deeply appreciate your recognition of our work and your thoughtful suggestions, which are incredibly valuable for further strengthening our manuscript. Following your advice, we have carefully revised the relevant sections and supplemented our analyses. Below are our detailed responses to your questions. Thank you so much for your support and for helping us make this paper better!
Response to Critical Weakness 1 & Question 1
We sincerely thank the reviewer for this profound insight. We completely agree that demonstrating how our benchmark connects to symbolic tool use is crucial for closing the loop between our analysis and the proposed method. We realize that we did not make this narrative explicit enough in the original manuscript.
The core philosophy of our work is Complexity Decoupling. In standard Chain-of-Thought (CoT), LLMs must simultaneously handle logical deduction and state-tracking. As problem complexity increases, the reasoning trace becomes extremely long. The model often forgets the initial constraints or starts to hallucinate non-existent facts during this long textual deduction. We term this failure mode "Lost in Complexity." T-MCTS addresses this by fundamentally changing the cognitive load: it trains LLMs to translate (symbolically formulate) the problem rather than solve it natively.
Within this context, LOGICEVAL-X serves not just as a general evaluation testbed, but as a diagnostic tool that reveals two critical insights specifically about symbolic tool use:
First, it reveals the necessity of decoupling. By demonstrating the severe performance degradation of LLMs under dense constraints, the benchmark proves that text-based deduction alone is insufficient, making symbolic tools a structural requirement rather than just an optional enhancement.
Second, and more importantly, it exposes the Tool Orchestration Bottleneck. As shown in our preliminary study (Table 3 and Figure 6), simply exposing LLMs to symbolic tools yields mixed results. For instance, Qwen3-8B's accuracy drops from 26.4% to 10.3% because it frequently fails at Tool Execution (e.g., syntax errors) and Feedback Repair.
Therefore, LOGICEVAL-X specifically reveals that the true challenge in symbolic tool use is not merely providing the tools, but teaching the model how to formulate, select, and repair them. This diagnostic finding directly motivates our T-MCTS framework, which uses solver feedback as a process reward to teach these orchestration skills. We have added this narrative clarification to the revised Section 5 to better bridge our benchmark and method.