- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReSS (Reasoning via Symbolic Scaffolds), which aims to combine the structured decision-making capabilities of decision trees with the domain knowledge of large language models (LLMs) to improve prediction accuracy and reasoning interpretability in tabular classification tasks. Specifically, the authors first train a decision tree and extract decision paths from correctly classified samples as Symbolic Scaffolds. They then use GPT-4.1 to transform these structured rules into natural language reasoning chains enriched with domain knowledge. A data augmentation strategy that preserves the original decision paths further expands the training data. Finally, the authors use these data to perform supervised fine-tuning (SFT) on Qwen-2.5-3B-Instruct, enabling the model to generate explanations and predictions without relying on the decision tree during inference. Experiments across five datasets in healthcare and finance demonstrate that ReSS outperforms various traditional machine learning and LLM baselines in classification accuracy. The authors also evaluate the faithfulness of generated reasoning through feature hallucination rates, explanation sufficiency, and necessity.
Critical strengths
-
Practical integration of symbolic decision structures and LLM domain knowledge. Compared with directly generating CoT using LLMs, ReSS employs decision tree paths to constrain the reasoning data generation process, reducing the inclusion of irrelevant features and improving the structural consistency and verifiability of reasoning supervision to some extent.
-
Comprehensive experimental comparisons and promising classification performance. The authors compare ReSS with Decision Tree, XGBoost, Direct SFT, DRC+SFT, and Direct RL across five healthcare and financial datasets, and further include TabNet and TabPFN-3 in the appendix. ReSS achieves the highest classification accuracy on all five datasets, reaching 85.71% on Diabetes, which represents an improvement of approximately 9.1 percentage points over the decision tree baseline. Furthermore, the fine-tuned 3B model outperforms GPT-4.1 without task-specific fine-tuning on all five datasets, demonstrating the effectiveness of structured reasoning supervision.
-
Multi-dimensional evaluation of reasoning quality. Beyond classification accuracy, the paper evaluates feature hallucination, explanation sufficiency, and necessity. The authors also investigate the contributions of different components through domain knowledge removal, sensitivity analysis of decision tree quality, and data augmentation ablations. These experiments provide valuable evidence for understanding the relationship between structured supervision and language model reasoning.
Critical weaknesses
-
Unclear novelty relative to existing research on tabular reasoning and knowledge distillation. Existing studies already explore the use of LLMs for tabular reasoning and knowledge distillation. For example, InterpreTabNet [1] leverages LLMs to transform important features identified by tabular models into natural language explanations. Tables2Traces [2] further constructs reasoning traces from structured tabular data to supervise LLMs, while TabReason [3] employs reinforcement learning to jointly optimize tabular prediction and explanation capabilities. In comparison, this paper mainly introduces decision tree paths as constraints for reasoning data generation. Although the specific supervision structure differs from existing approaches, the paper lacks targeted comparisons with these studies. In particular, it does not sufficiently demonstrate the unique advantages of decision tree paths over alternative feature selection or reasoning data construction strategies.
[1] InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation. ICML 2024.
[2] Tables2Traces: Distilling Tabular Data to Improve LLM Reasoning in Healthcare. EurIPS Workshop 2025.
[3] TabReason: A Reinforcement Learning-Enhanced Reasoning LLM for Explainable Tabular Data Prediction. 2025.
-
Insufficient validation of the effectiveness and necessity of decision tree supervision. The core assumption of this paper is that decision trees can provide effective structured reasoning supervision. However, the experimental results indicate that decision trees achieve lower classification accuracy than GPT-4.1 on some datasets. Although the authors demonstrate through sensitivity analysis that ReSS exhibits some robustness to decision tree prediction quality, the paper does not adequately explain why a relatively weaker decision tree can still provide more effective reasoning guidance than the LLM's own judgments.
Furthermore, the experiments lack comparisons with alternative scaffolding strategies, such as allowing GPT-4.1 to independently select important features or using randomly selected features. Therefore, the current results do not sufficiently establish whether the performance improvements specifically stem from the structured supervision provided by decision tree paths, rather than from general feature selection or filtering for correctly classified samples.
Questions for authors
Please refer to the concerns and questions raised in the Weaknesses section.
We thank the reviewer for the detailed comments. We address the raised concerns below.
Q1: Novelty relative to existing work.
A: We have cited these three works in the paper and discussed the differences. We did not compare with the existing works you mentioned for the following reasons:
[1] Si, Jacob, et al. "InterpreTabNet: distilling predictive signals from tabular data by salient feature interpretation." arXiv preprint arXiv:2406.00426 (2024)
[2] Werling, Mikkel, et al. "Tables2Traces: Distilling Tabular Data to Improve LLM Reasoning in Healthcare." (2025).
[3] Xu, Tommy, et al. "Tabreason: a reinforcement learning-enhanced reasoning LLM for explainable tabular data prediction." arXiv preprint arXiv:2505.21807 (2025).
[4] Li, Gang, et al. "Disco: Reinforcing large reasoning models with discriminative constrained optimization." Advances in Neural Information Processing Systems 38 (2026): 57304-57331.
Q2: The paper does not adequately explain why a relatively weaker decision tree can still provide more effective reasoning guidance than the LLM’s own judgments.
A: Thank you for this interesting question. The decision tree is indeed relatively weak compared with LLMs. However, this does not prevent us from curating useful reasoning data from it. We discuss this from several perspectives:
An analogy with human supervision. LLMs are post-trained on human-curated data, yet they can be much more capable than humans. Using human-curated data does not prevent us from training superintelligent models that exceed human capabilities. This is analogous to the weak-to-strong generalization argument made in [5].
Scaffolds serve as feature selection. If we leave the model to learn which features are useful on its own, it could require a huge amount of data. This has been shown in prior theoretical work (Wies et al., 2023). Please refer to the discussion in Lines 195–202 of the paper.
Decision-tree-selected features are arguably better than LLM-selected features. The DRC+SFT baseline implements the latter approach: its prompt instructs the LLM to “select only the features you judge to be the most informative and behaviorally relevant for predicting” and states that it is not required to use all features (Appendix B.2.1). Even so, its rationales still reference many more features than the tree paths do—for example, an average of 14 of the 25 features on AD, compared with approximately 4 for ReSS.
Tree scaffolds are not the only source of information used to train ReSS. The curated reasoning data also incorporate GPT-4.1’s domain knowledge, which is grounded in the scaffolds and integrated into the training rationales. Together, these two sources enable ReSS to learn a stronger model than the decision tree. When we use delexicalized features to curate the data, so that no domain knowledge is injected, the model’s performance is almost the same as that of the decision tree, as reported in Figure 4 (left).
We do not use the tree as a predictor and do not claim that it is more accurate than the LLM. Our paper demonstrates that a weaker model such as decision tree can help curate more effective structured reasoning supervision. Decision tree is just one choice. We can consider other models, including XGBoost, which is left as future work for exploration.
[5] Burns, Collin, et al. "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023." URL https://arxiv. org/abs/2312.09390.