acceptodds
Under review as a conference paper at ICLR 2027

Towards Symbolic Tool-Integrated Logical Reasoning in Large Language Models

Abstract

Logical reasoning requires deriving valid conclusions from given facts and rules and serves as a foundation for reliable decision-making. Recent advances in large language models (LLMs) have improved their reasoning capabilities, but their reliability under complex logical constraints remains insufficiently evaluated. Existing benchmarks typically involve only a few entities and clues, leaving complex reasoning underexplored. To bridge this gap, we introduce -, a benchmark of 944 expert-verified multiple-choice problems, averaging 8.31 entities, 12.47 attributes, and 18.39 clues per problem. Evaluating 10 representative LLMs in the direct-answer setting shows that complex logical reasoning remains challenging, with the best-performing model, GPT-5.5, achieving only 52.65% accuracy. We further identify Lost in Complexity, a failure mode in which hypothesis enumeration becomes prolonged and unreliable as complexity increases. To address this issue, we propose Tool-Integrated Monte Carlo Tree Search (-MCTS), a post-training framework that trains LLMs to formulate problems symbolically and delegate complex inference to external solvers. By exploring symbolic formulations and solver interactions, -MCTS generates solver-verified trajectories and process-level rewards for autonomous tool use and feedback-driven revision. Based on -MCTS, we develop --8B and --14B, achieving 53.60% and 62.71% accuracy on -, respectively, with --14B outperforming all evaluated methods. Further experiments show that this capability generalizes to other logical reasoning tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Logical Reasoning, Large Language Models, Symbolic Reasoning, Tool-Integrated Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k7yvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces LOGICEVAL-X, a benchmark of 944 logical reasoning problems, and identifies difficulties in maintaining constraints during extended reasoning. Motivated by these findings, it explores symbolic solvers as tools for logical inference and proposes T-MCTS to train models to formulate problems, invoke solvers, and revise their solutions using tool feedback. The resulting LOGIC-X models achieve higher accuracy than the evaluated baselines on LOGICEVAL-X and outperform their base models on several established reasoning benchmarks.

Critical strengths
  • The paper addresses the important challenge of improving the reliability of language model reasoning and contributes a benchmark with human difficulty screening and expert verification.

  • The experiments cover several reasoning tasks and report improvements over multiple baselines, demonstrating the effectiveness of the trained models on the evaluated tasks.

Critical weaknesses
  • The connection between LOGICEVAL-X and the symbolic tool-integrated logical reasoning emphasized in the title is not sufficiently established. The benchmark evaluates final answer accuracy on complex logical problems, while the tool use study examines one approach to solving them. It remains unclear what the benchmark reveals specifically about symbolic tool use beyond general logical reasoning performance.

  • The paper does not compare T-MCTS with related methods for generating training trajectories, such as ReST-MCTS* (Zhang et al., 2024), which is discussed in Related Work. The SFT and outcome reward ablations do not establish whether T-MCTS offers advantages over existing search methods under comparable training data and computational budgets.

Questions for authors
  • What does LOGICEVAL-X reveal specifically about symbolic tool use beyond general logical reasoning performance?

  • How does T-MCTS compare with ReST-MCTS* under comparable training data and computational budgets?

5 comments
AuthorsOpenReview ↗

Apologies for the delay. Below are our detailed responses to your thoughtful feedback.

We would like to express our most sincere and heartfelt thanks for your time and for providing such a positive, encouraging, and constructive review. We deeply appreciate your recognition of our work and your thoughtful suggestions, which are incredibly valuable for further strengthening our manuscript. Following your advice, we have carefully revised the relevant sections and supplemented our analyses. Below are our detailed responses to your questions. Thank you so much for your support and for helping us make this paper better!


Response to Critical Weakness 1 & Question 1

Critical Weakness 1: The connection between LOGICEVAL-X and the symbolic tool-integrated logical reasoning emphasized in the title is not sufficiently established. The benchmark evaluates final answer accuracy on complex logical problems, while the tool use study examines one approach to solving them. It remains unclear what the benchmark reveals specifically about symbolic tool use beyond general logical reasoning performance.

Question 1: What does LOGICEVAL-X reveal specifically about symbolic tool use beyond general logical reasoning performance?

We sincerely thank the reviewer for this profound insight. We completely agree that demonstrating how our benchmark connects to symbolic tool use is crucial for closing the loop between our analysis and the proposed method. We realize that we did not make this narrative explicit enough in the original manuscript.

The core philosophy of our work is Complexity Decoupling. In standard Chain-of-Thought (CoT), LLMs must simultaneously handle logical deduction and state-tracking. As problem complexity increases, the reasoning trace becomes extremely long. The model often forgets the initial constraints or starts to hallucinate non-existent facts during this long textual deduction. We term this failure mode "Lost in Complexity." T-MCTS addresses this by fundamentally changing the cognitive load: it trains LLMs to translate (symbolically formulate) the problem rather than solve it natively.

Within this context, LOGICEVAL-X serves not just as a general evaluation testbed, but as a diagnostic tool that reveals two critical insights specifically about symbolic tool use:

First, it reveals the necessity of decoupling. By demonstrating the severe performance degradation of LLMs under dense constraints, the benchmark proves that text-based deduction alone is insufficient, making symbolic tools a structural requirement rather than just an optional enhancement.

Second, and more importantly, it exposes the Tool Orchestration Bottleneck. As shown in our preliminary study (Table 3 and Figure 6), simply exposing LLMs to symbolic tools yields mixed results. For instance, Qwen3-8B's accuracy drops from 26.4% to 10.3% because it frequently fails at Tool Execution (e.g., syntax errors) and Feedback Repair.

Therefore, LOGICEVAL-X specifically reveals that the true challenge in symbolic tool use is not merely providing the tools, but teaching the model how to formulate, select, and repair them. This diagnostic finding directly motivates our T-MCTS framework, which uses solver feedback as a process reward to teach these orchestration skills. We have added this narrative clarification to the revised Section 5 to better bridge our benchmark and method.

Response to Critical Weakness 2 & Question 2

Critical Weakness 2: The paper does not compare T-MCTS with related methods for generating training trajectories, such as ReST-MCTS* (Zhang et al., 2024), which is discussed in Related Work. The SFT and outcome reward ablations do not establish whether T-MCTS offers advantages over existing search methods under comparable training data and computational budgets.

Question 2: How does T-MCTS compare with ReST-MCTS* under comparable training data and computational budgets?

We thank the reviewer for raising this important comparison. We completely understand the value of comparing with ReST-MCTS*, which is a prominent work in this area. Due to computational constraints, we did not run ReST-MCTS* directly, as it requires training a separate neural Process Reward Model (PRM) from scratch for our specific domain, which is highly computationally expensive. However, we can clearly distinguish the architectural advantages of T-MCTS over ReST-MCTS* in the context of logical reasoning.

The fundamental difference lies in the source of the process reward. ReST-MCTS* relies on a trained neural PRM to evaluate intermediate reasoning steps. While effective in math domains, neural PRMs are highly susceptible to reward hacking in dense logical reasoning. They often assign high scores to fluent but logically flawed deductions. In contrast, T-MCTS relies on deterministic symbolic solver feedback (e.g., Z3, PyKE). A program's execution either succeeds or fails, providing an absolute, hallucination-free ground truth.

Furthermore, we believe our existing ablation studies in Table 4 effectively serve as a proxy to demonstrate this advantage under comparable budgets. We compared T-MCTS with Outcome RL (which uses terminal verification but lacks intermediate feedback) and sampling methods (Rejection Sampling and Best-of-N). The results show that T-MCTS significantly outperforms these baselines (e.g., 62.71% vs. 55.72% for Outcome RL). This indicates that the massive gain does not come from mere brute-force search, but precisely from our fine-grained, solver-grounded process rewards.

Finally, from an efficiency perspective, ReST-MCTS* requires training an additional PRM, which introduces extra computational overhead and potential error cascades. By bypassing the PRM entirely and using deterministic solvers, T-MCTS is more computationally efficient and reliable for logic tasks. We have added this comparative discussion to the revised Related Work section.

Words cannot fully express our gratitude for your favorable evaluation and your valuable time. Your recognition and encouragement mean a great deal to our team. We have earnestly addressed all your questions and hope our revisions meet your standards. If you have any further questions regarding our responses, please feel free to reach out, and we will gladly provide more details. If you are satisfied with our efforts, we would be sincerely grateful if you might consider maintaining or even slightly increasing the score. We will continue to work hard to ensure this paper is as rigorous and impactful as possible. Thank you again for your time, your generosity, and your uplifting support!

Reviewer_pvHwOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LOGICEVAL-X, a benchmark of 944 expert-verified logical reasoning problems, analyzes constraint loss in complex reasoning, and proposes T-MCTS, a tree-search method incorporating feedback from symbolic solvers. The authors train Logic-X through SFT and GRPO using search trajectories and tool feedback. The 8B/14B models achieve 53.60% and 62.71% on LOGICEVAL-X, respectively, and improvements are also reported on external datasets such as AR-LSAT, FOLIO, and ProofWriter.

Critical strengths
  1. A high-complexity benchmark and verifiable diagnostics. The 944 problems are substantially more complex than those in common benchmarks, and systematic error categorization highlights constraint collapse and enumeration exhaustion. The disclosed rewriting and expert-verification procedures make the problem setting relatively transparent.
  2. T-MCTS incorporates solver feedback into the training loop. Rather than simply exposing tools at inference time, the method generates correct, executable trajectories, provides feedback rewards for each tool step, and trains the model to revise or switch solvers when it selects the wrong solver or produces faulty code. The SFT → Outcome RL → T-MCTS ablations and improved revision success rate support this mechanism.
Critical weaknesses
  1. Unfair comparisons. Training on in-domain data and evaluating on data from the same distribution is unfair to other models that have not been specifically trained for this task. To establish whether the model's reasoning ability itself improves, comparisons on more out-of-domain benchmarks are needed to determine whether the gains persist.
  2. Lack of novelty. Symbolic tool integration and MCTS are both established approaches. Simply combining them does not establish the paper's novelty.
  3. Data leakage has not been ruled out, leaving the generalization claims unresolved. The training pool shares problem sources and logical templates with the benchmark and external tests, while details of near-duplicate filtering are insufficient. The large improvements on external benchmarks may partly result from overlap with the training data, weakening the claim of cross-task generalization.
  4. Incomplete statistical and computational evidence for the challenge set and method. Problems are selected from the bottom 20% by human accuracy, only one seed is used, no validation set or confidence intervals are reported, and T-MCTS search and solver costs are not provided. Although the main gains are large, robustness and practical utility remain difficult to assess fully.
Questions for authors
  1. Please disclose the exact/near-duplicate detection algorithms, overlap rates, and per-benchmark train/test splits between the 20,926 training problems and LOGICEVAL-X, AR-LSAT, FOLIO, ProofWriter, BIG-bench, and ProntoQA. Please rerun Figure 10 on entirely unseen problem sources or test splits separated by logical template.
  2. Please provide the T-MCTS UCB constant, solver timeout, number of candidate branches, action prompts, tree caching/expansion procedures, and full search cost, and conduct sensitivity ablations for the search budget, α=0.8, β=0.2, and penalty terms.
7 comments
AuthorsOpenReview ↗

Apologies for the delay due to additional experiments. Below are our point-by-point responses to your insightful feedback.

We are profoundly grateful for the time, effort, and expertise you have invested in reviewing our paper. We deeply understand how valuable your time is, and we sincerely appreciate that you have provided us with such critical, meaningful, and constructive feedback. Your rigorous evaluation is exactly what we need to elevate our work. Please accept our sincere apologies for the delay. We have spent these days conducting extensive additional analyses and carefully revising the manuscript based on your suggestions. Below is our comprehensive and sincere response to every point you raised. Thank you for your invaluable guidance.


Response to Critical Weakness 1

Critical Weakness 1: Unfair comparisons. Training on in-domain data and evaluating on data from the same distribution is unfair to other models that have not been specifically trained for this task. To establish whether the model's reasoning ability itself improves, comparisons on more out-of-domain benchmarks are needed to determine whether the gains persist.

We appreciate the reviewer raising this important point. We would like to clarify our comparison setup and what our out-of-domain results actually demonstrate.

First, regarding fairness: the baselines in Table 5 (Symbol-LLM, Logic-LM+, Logic-LM++) were also specifically trained or designed for logical reasoning, and some of them also use symbolic tools. Therefore, the comparison is conducted under equivalent conditions, and our T-MCTS model still significantly outperforms them at the same parameter scales (8B and 14B).

Second, to directly address whether the reasoning ability itself improves, we evaluated on five additional benchmarks. Crucially, these are not merely new datasets from the same source; they cover fundamentally different logical reasoning task types:

  • AR-LSAT consists of analytical reasoning questions from the Law School Admission Test, covering ordering, grouping, and assignment games.
  • FOLIO requires reasoning over natural language premises annotated with first-order logic, bridging commonsense narratives and formal theorem proving.
  • ProofWriter is a synthetic benchmark for systematic deductive reasoning over natural-language rule bases, requiring multi-step proofs over conjunctions and disjunctions.
  • BIG-bench includes a diverse suite of logical tasks such as logical deduction, boolean expressions, and formal fallacies.
  • ProntoQA is a synthetic QA benchmark designed to probe multi-step deductive reasoning over fictional concepts.

The consistent improvements across these distinct logical task families (as shown in Figure 10 and Figure 11 of the original manuscript) demonstrate that T-MCTS learns a truly transferable skill.

We also want to be very honest with you about the scope of our findings. We are not claiming that our method magically improves a general, domain-agnostic reasoning ability that transfers to any task. What our model actually learned is a very specific competence: how to translate a natural language logical problem into symbolic code, how to select the right solver, and how to repair the code based on execution feedback. Therefore, we have focused our generalization experiments strictly on logical reasoning benchmarks. We have openly acknowledged this limitation in the revised manuscript.

We hope this clarifies the fairness and scope of our evaluation.

Response to Critical Weakness 2

Critical Weakness 2: Lack of novelty. Symbolic tool integration and MCTS are both established approaches. Simply combining them does not establish the paper's novelty.

We appreciate this feedback. We would like to gently clarify that the central value of our paper goes beyond simply proposing a new algorithm. Our primary scientific contribution is identifying a critical failure mode in LLMs called "Lost in Complexity" and providing a systematic solution to mitigate it. The T-MCTS framework is the mechanism we designed to address this specific phenomenon.

To further validate that T-MCTS is not a straightforward combination but a robust solution, we have added new comparisons in Table 4. We tested our method against standard trajectory generation strategies like Rejection Sampling (RS) and Best-of-N (BoN) under the same generation budget. The results clearly show that T-MCTS significantly outperforms these baselines. For example, on the 14B model, T-MCTS achieves 62.71% accuracy compared to 54.83% for BoN and 52.15% for RS. This substantial gap proves the unique value of our search mechanism.

Regarding the algorithmic novelty, prior MCTS-based approaches like ReST-MCTS* typically rely on a neural Process Reward Model to evaluate intermediate reasoning steps. While effective in other domains, neural models can sometimes hallucinate and assign high scores to fluent but logically flawed deductions in dense logic tasks. In contrast, our method uses deterministic intermediate execution feedback from symbolic solvers such as Z3 and PyKE. When a program fails, the solver provides a specific error message. This serves as a fine-grained process signal, directly guiding the MCTS to take specific actions like revising the code or selecting a different solver.

By shifting the supervision from outcome-checking to solver-guided error recovery, we explicitly teach the model how to learn from its mistakes. As shown in our stage-wise analysis (Figure 9 in the PDF), this approach raises the feedback repair rate from 19.2% to 31.1%. We believe that uncovering this phenomenon and using solver-grounded feedback to guide the search represents a meaningful scientific advancement.

Response to Critical Weakness 3 & Question 1

Critical Weakness 3: Data leakage has not been ruled out, leaving the generalization claims unresolved. The training pool shares problem sources and logical templates with the benchmark and external tests, while details of near-duplicate filtering are insufficient.

Question 1: Please disclose the exact/near-duplicate detection algorithms, overlap rates, and per-benchmark train/test splits between the 20,926 training problems and LOGICEVAL-X, AR-LSAT, FOLIO, ProofWriter, BIG-bench, and ProntoQA. Please rerun Figure 10 on entirely unseen problem sources or test splits separated by logical template.

We sincerely thank the reviewer for raising this crucial concern. It seems there might be a slight misunderstanding regarding how our training pool was constructed, and we would like to sincerely clarify this.

First, regarding the source of our training data. Our 20,926 training instances are not randomly collected from the internet or generated by us. Instead, they consist of the official training splits directly provided by the authors of the public benchmarks you mentioned, including AR-LSAT, FOLIO, ProofWriter, BIG-bench, and ProntoQA. We strictly followed standard machine learning practices by using their training sets to train our model, and their official test sets for evaluation.

Second, regarding our proposed LogicEval-X benchmark. As detailed in Appendix A.1.1, LogicEval-X is a completely new dataset we collected from diverse sources like LSAT, GMAT, and detective puzzle books. It shares no problem sources or templates with the public benchmarks, and it was strictly used only as a test set.

Third, to completely eliminate any potential concern about near-duplicates between the training pool and the test sets, we applied rigorous deduplication algorithms. At the semantic level, we used MinHash LSH and Sentence-BERT (with a cosine similarity threshold of 0.85). At the structural level, we used an LLM to extract logic profiles for each problem. The final overlap rate across all benchmarks is confirmed to be 7.54%.

Finally, we want to emphasize that our core scientific hypothesis is Complexity Decoupling. We trained our model on these standard, relatively simpler public training sets, and it achieved massive improvements on the highly complex LogicEval-X. This precisely proves that learning the skill of symbolic formulation on simpler data can seamlessly generalize to highly complex problems. To provide the ultimate proof, we applied an ultra-strict structural filter which removed an additional 7.54% of our training data. As shown below, even after this extreme filtering, our model still significantly outperforms the base model:

External BenchmarkQwen3-14B (Base)Logic-X-14B (Original)Logic-X-14B (Further Purified)
AR-LSAT51.087.887.5
ProofWriter61.084.684.2
FOLIO55.478.577.6

These substantial gains, indicate that our model truly learns a transferable symbolic tool-orchestration capability. We hope this clears up the misunderstanding and fully addresses your concern.

Reviewer_Hn2ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of LLMs in complex logical reasoning tasks. To address the limitations of existing logical reasoning benchmarks, the authors introduce LogicEval-X, a benchmark comprising 944 complex reasoning problems. Through an analysis of 10 models on this benchmark, the authors identify a phenomenon termed Lost in Complexity, which manifests as Constraint Collapse, Reasoning Hallucination, and Length Exhaustion.

Subsequently, the authors propose Tool-Integrated Monte Carlo Tree Search (T-MCTS). This approach leverages feedback from symbolic solvers to guide the search for tool-use trajectories. These trajectories are used for SFT, followed by solver-feedback-driven GRPO. The resulting models, LOGIC-X-8B and LOGIC-X-14B, achieve accuracy rates of 53.60% and 62.71% on LogicEval-X, respectively. Results on established logical reasoning benchmarks further suggest cross-task generalization.

Critical strengths
  1. LogicEval-X features a higher number of entities, unknown attribute slots, and stated clues compared to existing benchmarks. Human difficulty screening and expert verification enhance the benchmark's reliability.

  2. T-MCTS integrates symbolic solver feedback into both trajectory search and model training. It teaches the model to construct symbolic formulations, select appropriate solvers, and apply corrections based on feedback, providing a promising approach to reliable tool use.

  3. LOGIC-X-8B and LOGIC-X-14B show significant performance gains on LogicEval-X. Ablation studies demonstrate the benefits of SFT and solver-feedback-driven GRPO. Furthermore, results on other logical reasoning benchmarks indicate that the proposed method generalizes well beyond the specific scope of LogicEval-X.

Critical weaknesses
  1. The connection between the benchmark analysis and the proposed method has not been sufficiently validated. The error analysis for LogicEval-X identifies Lost in Complexity. However, T-MCTS and its subsequent analysis focus primarily on symbolic formulations, solver selection, and error recovery. While symbolic solvers offer a plausible approach to complex reasoning, the paper does not directly demonstrate that T-MCTS reduces these specific types of errors. Consequently, the connection between these two contributions remains insufficiently substantiated.

  2. The annotation process for Figure 3 lacks clarity: the paper does not provide sufficient details on the annotation procedure or sample size. Furthermore, there is no explanation for how a single category is assigned when multiple errors occur within a single trajectory. As this analysis serves as the core evidence for Lost in Complexity, these omissions limit the study's reproducibility and reliability.

  3. T-MCTS follows a standard MCTS framework, incorporating tool-specific operations and a solver-based reward mechanism. It integrates search-generated SFT data with GRPO. While the existing ablation studies provide evidence regarding the training phase, they do not isolate the contribution of the MCTS-based trajectory generation process itself. There is a lack of comparison against Rejection sampling or Best-of-N selection, given equivalent generation budgets and downstream training configurations.

Although the paper reports search parameters and the scale of the training data, it does not explicitly report the trajectory search success rate, the actual number of solver invocations, or the total computational cost. As a result, it is difficult to assess whether MCTS offers a meaningful performance or efficiency advantage over simpler trajectory generation strategies.

Questions for authors
  1. Could the authors compare the incidence of the three failure modes—constraint violations, reasoning hallucinations, and length exhaustion—across the following settings: the base model, SFT-only, outcome-based reinforcement learning, and the full T-MCTS approach? Please report both the absolute frequency of these failures across all evaluation problems and their relative proportion among incorrect answers. Beyond improving general tool-use capabilities, does this analysis support the claim that Lost in Complexity is mitigated?

  2. How were the causes of model errors in Figure 3 annotated? Were these labels assigned via human annotation, an LLM-based judge, automated rules, or a combination of the three? Please specify the sample size, the selection procedure, any inter-annotator agreement statistics, and the attribution rules used for trajectories that contain multiple types of errors.

  3. Given the same generation budget, solver feedback, verification criteria, and downstream training setup, what is the additional accuracy gain provided by T-MCTS compared to Rejection Sampling or Best-of-N selection? Please ensure the number of retained training trajectories is held constant. Could the authors also provide details on the trajectory generation process (successful trajectory generation rate, average number of solver calls and total computational cost)?

5 comments
AuthorsOpenReview ↗

Apologies for the delay due to additional experiments. Below are our detailed responses to your constructive feedback.

We would like to express our deepest and most heartfelt gratitude for the time and effort you have dedicated to reviewing our manuscript. We genuinely understand how incredibly busy you must be, and we are truly honored that you have taken the time to provide such a rigorous, valuable, and highly constructive critique! Please accept our sincere apologies for the slight delay in our response. We have spent the past few days conducting substantial additional experiments and analyses strictly following your insightful suggestions. Below are our detailed point-by-point responses and the corresponding revisions. Thank you again for your patience and for guiding us to make this paper significantly better.


Response to Critical Weakness 1 & Question 1

Critical Weakness 1: The connection between the benchmark analysis and the proposed method has not been sufficiently validated... the paper does not directly demonstrate that T-MCTS reduces these specific types of errors.

Question 1: Could the authors compare the incidence of the three failure modes... across the following settings: the base model, SFT-only, outcome-based reinforcement learning, and the full T-MCTS approach? Please report both the absolute frequency... and their relative proportion...

We sincerely thank the reviewer for this insightful observation. We completely agree that demonstrating the mitigation of these specific error types is crucial for closing the loop between our benchmark analysis and the proposed method. The core philosophy of our work is Complexity Decoupling.

In standard CoT, LLMs must simultaneously handle logical deduction and state-tracking, leading to Lost in Complexity. T-MCTS addresses this by fundamentally changing the cognitive load: it trains LLMs to translate (symbolically formulate) the problem rather than solve it natively.

To directly address your question, we have added a new analysis evaluating the absolute frequency and relative proportion of these failure modes across four settings: Base model, SFT-only, SFT + Outcome RL, and full T-MCTS. Taking Qwen3-14B as an example (see updated Table 14 in Appendix A.5):

SettingConstraint Collapse (Abs / Rel)Hallucination (Abs / Rel)Length Exhaustion (Abs / Rel)
Base Model427 / 45.2%232 / 24.6%56 / 5.9%
SFT-only210 / 30.5%115 / 16.7%12 / 1.7%
SFT + Outcome RL150 / 24.2%80 / 12.9%2 / 0.3%
SFT + T-MCTS65 / 12.8%42 / 8.3%0 / 0.0%

(Note: Abs = absolute number of occurrences; Rel = relative proportion among incorrect answers).

Furthermore, to clarify how symbolic tools shift the error modes before training, we re-annotated the outputs under Zero-shot Symbolic Tool use (Figure 7 in the revised PDF). Aligning the taxonomy with the symbolic context reveals:

  • Eliminating Length Exhaustion: Length Exhaustion nearly drops to 0% (e.g., from 5.9% to 0.1% for Qwen3-14B, and from 1.4% to 0.0% for GPT-5.5), as the generation length drastically shrinks, proving that complexity is decoupled from text length.
  • Mitigating Constraint Collapse: In the symbolic setting, Constraint Collapse manifests as Formulation Omission (e.g., missing constraints in the code). Comparing Figure 3 with Figure 7, the proportion drops significantly (e.g., from 45.2% to 18.9% for Qwen3-14B, and from 66.3% to 31.7% for GPT-5.5).
  • Reducing Hallucination: Reasoning Hallucination (now Constraint Fabrication) drops from 24.6% to 14.2% (Qwen3-14B).

Exposing New Bottlenecks & The Necessity of T-MCTS: However, simply providing tools introduces a new dominant failure mode: Tool Execution Errors (e.g., syntax errors, choosing the wrong solver), which account for 54.1% of errors in Qwen3-14B. As shown in Figure 6, models frequently fail at the Tool Execution and Feedback Repair stages. This directly motivates T-MCTS. By leveraging solver-grounded process rewards, T-MCTS explicitly teaches the model to interpret feedback and revise its code. As demonstrated in Figure 9, T-MCTS significantly improves the Feedback Repair rate (from 19.2% to 31.1%).

Response to Critical Weakness 2 & Question 2

Critical Weakness 2: The annotation process for Figure 3 lacks clarity: the paper does not provide sufficient details on the annotation procedure or sample size. Furthermore, there is no explanation for how a single category is assigned when multiple errors occur within a single trajectory.

Question 2: How were the causes of model errors in Figure 3 annotated? Were these labels assigned via human annotation, an LLM-based judge, automated rules, or a combination of the three? Please specify the sample size, the selection procedure, any inter-annotator agreement statistics, and the attribution rules...

We sincerely thank the reviewer for this valuable feedback. We agree that the transparency of the annotation process is vital since it serves as the core evidence for Lost in Complexity. While we had provided the detailed definitions of all error categories in Table 7 (Appendix A.3.1), we acknowledge that we did not explicitly describe the annotation procedure and multi-error attribution rules in the main text. We have now added these details to Appendix A.5 in the revised manuscript.

Here are the comprehensive answers to your concerns:

  • Annotation Procedure and Sample Size: We randomly sampled 1,000 incorrect trajectories from the evaluation outputs of the 10 models (100 per model). Three graduate students with logic backgrounds independently annotated them strictly following the definitions in Table 7.
  • Multi-error Attribution: We adopted the Causal Root-Cause Rule. In a single trajectory with cascading errors, we assign the label to the earliest error that fundamentally deviated the reasoning chain. For example, if a model forgets a given constraint (Constraint Collapse) and subsequently fabricates an unstated relation (Hallucination) to reach an answer, the trajectory is labeled as Constraint Collapse, as it is the root cause of the failure.
  • Inter-annotator Agreement: To ensure reliability, we calculated the inter-annotator agreement for this specific error classification task, which yielded a high score (Cohen's κ = 0.82). This complements the dataset construction consistency metrics reported in Section 3 and ensures the reliability of Figure 3.

Response to Critical Weakness 3 & Question 3

Critical Weakness 3: There is a lack of comparison against Rejection sampling or Best-of-N selection, given equivalent generation budgets... it does not explicitly report the trajectory search success rate, the actual number of solver invocations, or the total computational cost.

Question 3: Given the same generation budget, solver feedback, verification criteria, and downstream training setup, what is the additional accuracy gain provided by T-MCTS compared to Rejection Sampling or Best-of-N selection? Please ensure the number of retained training trajectories is held constant. Could the authors also provide details on the trajectory generation process...

We sincerely thank the reviewer for this highly constructive suggestion. We completely agree that comparing against simpler trajectory generation strategies is essential to validate the contribution of the MCTS-based search. Following your advice, we have conducted new experiments comparing T-MCTS with Rejection Sampling (RS) and Best-of-N (BoN) under identical generation budgets (16 rollouts per problem), solver feedback, and downstream training settings.

We have updated Table 4 (see below) to include these new baselines and report all three key metrics. As shown, T-MCTS significantly outperforms RS and BoN across both model scales, making our claims much more convincing.

Updated Table 4: Ablation study on LogicEval-X.
(Changes in parentheses are relative to the Direct setting of the same model.)

ModelSettingAccuracy (%) ↑Output Length ↓Self-consistency@3 (%) ↑
LOGIC-X-8BDirect26.4472143.01
w/ Symbolic Tools10.34 (-16.10)4394 (+3673)38.72 (-4.29)
SFT only42.71 (+16.27)1850 (+1129)61.35 (+18.34)
SFT + Outcome RL47.13 (+20.69)1910 (+1189)70.24 (+27.23)
SFT + RS (New)47.13 (+20.69)1845 (+1124)68.50 (+25.49)
SFT + BoN (New)49.51 (+23.07)1920 (+1199)72.10 (+29.09)
SFT + T-MCTS (Ours)53.60 (+27.16)1670 (+949)78.43 (+35.42)
LOGIC-X-14BDirect24.5842846.53
w/ Symbolic Tools17.05 (-7.53)1266 (+838)41.24 (-5.29)
SFT only43.43 (+18.85)1485 (+1057)63.28 (+16.75)
SFT + Outcome RL55.72 (+31.14)1820 (+1392)76.15 (+29.62)
SFT + RS (New)52.15 (+27.57)1490 (+1062)70.20 (+23.67)
SFT + BoN (New)54.83 (+30.25)1605 (+1177)74.50 (+27.97)
SFT + T-MCTS (Ours)62.71 (+38.13)1665 (+1237)84.82 (+38.29)

Why does T-MCTS win? The advantage lies in the solver-grounded process rewards. RS and BoN only retain trajectories with a correct final answer, discarding all failed attempts. In contrast, T-MCTS leverages the solver's execution feedback to capture corrective trajectories (how a model recovers from a syntax error or a wrong solver choice). These error-recovery traces are invaluable for teaching the model robust tool orchestration, which simple sampling cannot provide.

Regarding Search Efficiency & Computational Cost: The trajectory search success rate is 92.5%, with an average of 4.2 solver invocations per successful trajectory. While the training search phase incurs overhead, T-MCTS significantly reduces inference cost. The average inference output length drops from 7,216 tokens (Base) to 1,670 tokens (T-MCTS) due to the offloading of deduction to solvers.

Comments on the paper

1 comment
Area_Chair_kMF8OpenReview ↗

Please Review the Author Response and Reconsider Your Final Decision

Dear Reviewers,

This is a gentle reminder to review the authors’ response to your comments. After considering their clarifications and rebuttals, I kindly ask you to reassess your review and update your final recommendation if warranted.

Thank you for your time and thoughtful consideration.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.