acceptodds
Under review as a conference paper at ICLR 2027

From Decision Trees to Strong Reasoning Models for Tabular Data Classification

Abstract

Tabular data classification in high-stakes domains such as healthcare and finance requires both high accuracy and faithful, human-understandable reasoning. Symbolic decision trees provide explicit decision rules but offer limited semantic explanations, while adapting large language models (LLMs) to domain-specific tabular reasoning requires scalable and reliable reasoning supervision. We propose **ReSS**, a framework that combines symbolic decision structure with LLM semantic knowledge to curate reasoning data and train specialized tabular reasoning models. ReSS extracts decision paths from training instances whose tree predictions agree with their ground-truth labels. These paths, together with input features and labels, guide an LLM to construct natural-language rationales that follow the selected symbolic constraints and incorporate domain-specific interpretations. A scaffold-invariant augmentation strategy is proposed to expand the training dataset while preserving the symbolic constraints. The resulting data are used to fine-tune an open LLM that generates reasoning and predictions directly from input features, without requiring the decision tree at inference. We assess reasoning faithfulness through hallucination rate, explanation necessity, and explanation sufficiency. Experiments on medical and financial benchmarks demonstrate accuracy gains of up to \(10%\) over decision trees and standard fine-tuning baselines, alongside grounded and consistent explanations. Sensitivity analyses show that predictive accuracy remains relatively stable across the evaluated tree configurations, demonstrating that useful reasoning supervision can be obtained from trees of varying predictive quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Prediction, Symbolic Scaffolds, Reasoning Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GDXROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReSS (Reasoning via Symbolic Scaffolds), which aims to combine the structured decision-making capabilities of decision trees with the domain knowledge of large language models (LLMs) to improve prediction accuracy and reasoning interpretability in tabular classification tasks. Specifically, the authors first train a decision tree and extract decision paths from correctly classified samples as Symbolic Scaffolds. They then use GPT-4.1 to transform these structured rules into natural language reasoning chains enriched with domain knowledge. A data augmentation strategy that preserves the original decision paths further expands the training data. Finally, the authors use these data to perform supervised fine-tuning (SFT) on Qwen-2.5-3B-Instruct, enabling the model to generate explanations and predictions without relying on the decision tree during inference. Experiments across five datasets in healthcare and finance demonstrate that ReSS outperforms various traditional machine learning and LLM baselines in classification accuracy. The authors also evaluate the faithfulness of generated reasoning through feature hallucination rates, explanation sufficiency, and necessity.

Critical strengths
  1. Practical integration of symbolic decision structures and LLM domain knowledge. Compared with directly generating CoT using LLMs, ReSS employs decision tree paths to constrain the reasoning data generation process, reducing the inclusion of irrelevant features and improving the structural consistency and verifiability of reasoning supervision to some extent.

  2. Comprehensive experimental comparisons and promising classification performance. The authors compare ReSS with Decision Tree, XGBoost, Direct SFT, DRC+SFT, and Direct RL across five healthcare and financial datasets, and further include TabNet and TabPFN-3 in the appendix. ReSS achieves the highest classification accuracy on all five datasets, reaching 85.71% on Diabetes, which represents an improvement of approximately 9.1 percentage points over the decision tree baseline. Furthermore, the fine-tuned 3B model outperforms GPT-4.1 without task-specific fine-tuning on all five datasets, demonstrating the effectiveness of structured reasoning supervision.

  3. Multi-dimensional evaluation of reasoning quality. Beyond classification accuracy, the paper evaluates feature hallucination, explanation sufficiency, and necessity. The authors also investigate the contributions of different components through domain knowledge removal, sensitivity analysis of decision tree quality, and data augmentation ablations. These experiments provide valuable evidence for understanding the relationship between structured supervision and language model reasoning.

Critical weaknesses
  1. Unclear novelty relative to existing research on tabular reasoning and knowledge distillation. Existing studies already explore the use of LLMs for tabular reasoning and knowledge distillation. For example, InterpreTabNet [1] leverages LLMs to transform important features identified by tabular models into natural language explanations. Tables2Traces [2] further constructs reasoning traces from structured tabular data to supervise LLMs, while TabReason [3] employs reinforcement learning to jointly optimize tabular prediction and explanation capabilities. In comparison, this paper mainly introduces decision tree paths as constraints for reasoning data generation. Although the specific supervision structure differs from existing approaches, the paper lacks targeted comparisons with these studies. In particular, it does not sufficiently demonstrate the unique advantages of decision tree paths over alternative feature selection or reasoning data construction strategies.

    [1] InterpreTabNet: Distilling Predictive Signals from Tabular Data by Salient Feature Interpretation. ICML 2024.

    [2] Tables2Traces: Distilling Tabular Data to Improve LLM Reasoning in Healthcare. EurIPS Workshop 2025.

    [3] TabReason: A Reinforcement Learning-Enhanced Reasoning LLM for Explainable Tabular Data Prediction. 2025.

  2. Insufficient validation of the effectiveness and necessity of decision tree supervision. The core assumption of this paper is that decision trees can provide effective structured reasoning supervision. However, the experimental results indicate that decision trees achieve lower classification accuracy than GPT-4.1 on some datasets. Although the authors demonstrate through sensitivity analysis that ReSS exhibits some robustness to decision tree prediction quality, the paper does not adequately explain why a relatively weaker decision tree can still provide more effective reasoning guidance than the LLM's own judgments.

    Furthermore, the experiments lack comparisons with alternative scaffolding strategies, such as allowing GPT-4.1 to independently select important features or using randomly selected features. Therefore, the current results do not sufficiently establish whether the performance improvements specifically stem from the structured supervision provided by decision tree paths, rather than from general feature selection or filtering for correctly classified samples.

Questions for authors

Please refer to the concerns and questions raised in the Weaknesses section.

1 comment

We thank the reviewer for the detailed comments. We address the raised concerns below.

Q1: Novelty relative to existing work.

A: We have cited these three works in the paper and discussed the differences. We did not compare with the existing works you mentioned for the following reasons:

  • InterpreTabNet [1] employs an LLM to map the learned feature masks onto natural language text describing the learned signal, as a post-hoc explanation. This text describes which features the model relies on and how they may relate, but does not state how the value of a feature contributes to a class (e.g., that a glucose value above a certain level supports a diabetic prediction). It is therefore not a reasoning chain that leads to a prediction for an instance. We have compared with TabNet and TabPFN in Table 4 in the paper.
  • Tables2Traces [2] does curate reasoning traces from tabular data for LLM fine-tuning, but with a different goal: it aims to transform the latent knowledge embedded in tabular data into reasoning traces that strengthen an LLM's clinical reasoning, which is then evaluated on text-based medical QA benchmarks. The tabular data thus serves as a source of knowledge for the LLM, and the traces are the means of transferring it; they are not reasoning toward a prediction for an instance. ReSS has the opposite goal: it uses an LLM's domain knowledge, guided by decision paths, to reason about and classify the tabular instances.
  • TabReason [3] trains a reasoning model for tabular prediction with reinforcement learning (RL). Our Direct RL baseline follows this approach, using DisCO [4], which has been reported to outperform GRPO.

[1] Si, Jacob, et al. "InterpreTabNet: distilling predictive signals from tabular data by salient feature interpretation." arXiv preprint arXiv:2406.00426 (2024)

[2] Werling, Mikkel, et al. "Tables2Traces: Distilling Tabular Data to Improve LLM Reasoning in Healthcare." (2025).

[3] Xu, Tommy, et al. "Tabreason: a reinforcement learning-enhanced reasoning LLM for explainable tabular data prediction." arXiv preprint arXiv:2505.21807 (2025).

[4] Li, Gang, et al. "Disco: Reinforcing large reasoning models with discriminative constrained optimization." Advances in Neural Information Processing Systems 38 (2026): 57304-57331.

Q2: The paper does not adequately explain why a relatively weaker decision tree can still provide more effective reasoning guidance than the LLM’s own judgments.

A: Thank you for this interesting question. The decision tree is indeed relatively weak compared with LLMs. However, this does not prevent us from curating useful reasoning data from it. We discuss this from several perspectives:

  • An analogy with human supervision. LLMs are post-trained on human-curated data, yet they can be much more capable than humans. Using human-curated data does not prevent us from training superintelligent models that exceed human capabilities. This is analogous to the weak-to-strong generalization argument made in [5].

  • Scaffolds serve as feature selection. If we leave the model to learn which features are useful on its own, it could require a huge amount of data. This has been shown in prior theoretical work (Wies et al., 2023). Please refer to the discussion in Lines 195–202 of the paper.

  • Decision-tree-selected features are arguably better than LLM-selected features. The DRC+SFT baseline implements the latter approach: its prompt instructs the LLM to “select only the features you judge to be the most informative and behaviorally relevant for predicting” and states that it is not required to use all features (Appendix B.2.1). Even so, its rationales still reference many more features than the tree paths do—for example, an average of 14 of the 25 features on AD, compared with approximately 4 for ReSS.

  • Tree scaffolds are not the only source of information used to train ReSS. The curated reasoning data also incorporate GPT-4.1’s domain knowledge, which is grounded in the scaffolds and integrated into the training rationales. Together, these two sources enable ReSS to learn a stronger model than the decision tree. When we use delexicalized features to curate the data, so that no domain knowledge is injected, the model’s performance is almost the same as that of the decision tree, as reported in Figure 4 (left).

We do not use the tree as a predictor and do not claim that it is more accurate than the LLM. Our paper demonstrates that a weaker model such as decision tree can help curate more effective structured reasoning supervision. Decision tree is just one choice. We can consider other models, including XGBoost, which is left as future work for exploration.

[5] Burns, Collin, et al. "Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023." URL https://arxiv. org/abs/2312.09390.

Reviewer_oLPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReSS, a framework to train LMs for tabular classification that also output natural language rationales. They first fit a decision tree, keep training samples where tree prediction = gold label, then give the decision path + features + label to GPT-4.1 to write rationales that follow the path but also add domain interpretation.They also augment data by perturbing instances while keeping the same decision path. Qwen-2.5-3B-Instruct is then finetuned on this, so no tree needed at inference.

Experiments on 5 medical/financial datasets, compared w/ tree methods, direct SFT, unconstrained rationale SFT, direct RL (plus more tabular baselines in appendix). Explanations are evaluated by feature/value/comparison hallucination rates and feature masking, and there are ablations on augmentation, feature semantics, tree quality and RL after SFT.

Critical strengths
  1. Practical way to get structured rationale supervision from plain tabular data. Using decision paths to pick which features to talk about and how to organize teh teacher explanations is a nice alternative to free-form rationale generation, and it dont need experts to write each explanation. Distilling into a model that doesnt need the tree at test time is also a good design choice. Sec 3.1-3.3 + appendix explain the pipeline clearly.

  2. Consistent gains and some relevant ablations. Full ReSS has the best mean acc on all 5 datasets, incl. vs TabNet / TabPFN-3 (App E.4). Also ReSS w/o augmentation still beats DRC+SFT on all 5 in Table 2, so the gain is not just from more training data. The tree quality analysis, semantic ablation and the no-aug explanation comparison in App E.1 are useful too, overall the approach seems worth further look.

Critical weaknesses
  1. The explanation eval doesnt really establish the causal faithfulness that is claimed. In Sec 4 you intervene on features mentioned in the rationale and regenerate rationale+prediction. This shows whether those features matter for the prediction, but not whether the reasoning text itself is actually used to get the answer. A model that predicts directly from those features and just verbalizes them seperately could pass all three criteria i think.

    Even at feature level there are some gaps: stable aggregate acc ≠ stable per-instance predictions, and masking up to 4 unmentioned features is weaker than keeping only the referenced ones. The no-aug comparison in E.1 helps with the missingness-augmentation concern (thanks for that), but doesnt resolve feature relevance vs causal contribution of the reasoning. Since the paper explicitly says it validates the causal impact of rationales / gives causally grounded explanations,this affects my score quite a bit.

  2. Reliability and added value of the synthetic supervision need more validation.

    • Keeping the tree path guarantees the same tree prediction, not the same true label. Original instance being correctly classified doesnt mean the whole leaf region shares that label. Some evidence on validity / label noise of augmented samples would help.
    • Small inconsistency: Fig 5 (p.37) gives savings as 100–<500 DM but Step 6 says 500–<1000 DM. Its a training example so it doesnt directly contradict the FVH results in Table 1, but it makes how the supervision is validated and how hallucinations are audited more important.
    • Semantic ablation in 5.4 removes feature/category semantics and domain interpretation at the same time, and App B.3 also changes the student instructions. So it cant seperate the contribution of teacher semantic rationales from the student just seeing meaningful inputs. A comparison with same named inputs, same selected+augmented data and same training budget would make this much stronger.
Questions for authors
  1. Scope of faithfulness claim: do you mean feature-level faithfulness, or the stronger claim that the semantic reasoning causally contributes to the answer? For the stronger one, any results with controlled interventions on the reasoning while keeping input fixed? For feature level, could you report per-instance prediction agreement when keeping only explanation-referenced features (not just aggregate acc after masking up to 4 others)?

  2. Validating supervision / hallucination numbers: is the savings mismatch in Fig 5 just a presentation typo, or an error in the actual curated rationale? How are such errors caught/filtered? How were the Table 1 hallucination rates annotated and checked (how many samples/steps, wich annotation procedure)? And for augmented samples, anything beyond tree-path preservation that supports keeping the original label?

  3. Attribution of gains: with same feature names+labels, same selected subset, same augmented data and similar budget, how does full ReSS compare to direct-label SFT and to SFT on factual path checks w/o domain interpretation? This would show whether the semantic rationales add value beyond selection, augmentation and feature semantics. If you already have results on this it would affect my assessment a lot.

2 comments

We thank the reviewer for the detailed comments. We address the raised concerns below.

Q1 (part 1): Stronger Reasoning-level causality claim.

A: Our original feature-masking analysis supports only feature-level faithfulness. We now provide a stronger reasoning-level causality claim.

We follow the idea of [1], which tests whether a model's answer depends on its stated reasoning by modifying the reasoning and measuring how the answer changes. In particular, it identifies two ways in which reasoning can be unfaithful: (i) post-hoc reasoning, where the answer is already determined before the reasoning is written and the reasoning does not influence it, and (ii) unfaithful reasoning due to test-time computation, where the benefit comes from the extra tokens rather than from what they say.

Following this, we designed an experiment to test the causal faithfulness of ReSS generated reasoning. In detail, for each test instance we modify the last j steps of the reasoning generated by ReSS, keep the input features fixed, and force the model to answer immediately (</think><answer>). We compare three interventions:

  • Delete: remove the last j steps. This tests (i) and corresponds to early answering in [1].
  • Different class: replace each of the last j steps with a step about the same feature, taken from an instance that the model predicts as a different class. This also tests (i), and is our own adaptation to tabular input.
  • Filler: replace each of the last j steps with an equal-length filler string ("..."). This tests (ii) and corresponds to the filler-token test in [1].

Our reasoning has the format step 1, …, step n, an overall summary, answer. We treat the overall summary as the last step and index the interventions by j, counted from the end: at j=0, only the final summary is modified, and for j≥1 we additionally modify the last j reasoning steps. The interventions are cumulative: at j, the steps that were already modified at smaller j remain modified (e.g., j=2 modifies the last two steps, j=3 the last three, and so on).

In the Different class condition, the overall summary is deleted rather than replaced. The reason is that the summary explicitly states the conclusion. When we replaced only the summary with that of a different-class instance and kept every step intact, the answer flipped to the injected class almost in all test cases. So perturbing the earlier steps would reveal nothing.

We report accuracy (%) in the following table on the two datasets (Diabetes and AD) as mean ± std over 3 seeds. Reasoning chains have at most 7 steps on Diabetes and 5 on AD; instances with fewer than j steps have all of their steps modified. The unperturbed column records the ReSS's results.

Diabetes

Last j stepsUnperturbedj=0j=1j=2j=3j=4j=5j=6j=7 (all)
Delete85.71±1.0682.68±3.2782.25±4.9276.62±2.2576.62±3.4475.32±5.6673.59±5.8670.13±10.6371.00±12.07
Filler85.71±1.0683.98±1.9880.09±3.2775.76±1.9877.49±3.0080.09±2.7074.46±9.5767.97±6.4164.50±0.75
Different class85.71±1.0682.68±3.2773.59±5.8666.67±4.5661.04±6.8754.55±3.9035.50±11.3522.94±4.1722.08±3.90

AD

Last j stepsUnperturbedj=0j=1j=2j=3j=4j=5 (all)
Delete76.07±0.0673.21±0.9773.34±0.8471.63±3.3470.97±2.8862.01±4.1760.34±1.34
Filler76.07±0.0673.56±1.5072.73±1.5570.14±3.8270.18±3.7665.70±1.8360.47±2.64
Different class76.07±0.0673.21±0.9760.39±4.0439.00±3.0428.24±1.5218.36±1.9613.44±1.21

The results indicate that the answer depends on the content of the reasoning, and that this is not explained by extra test-time computation.

(i) Post-hoc reasoning. If the reasoning were post-hoc, modifying it would not change the answer. Deleting steps already lowers accuracy, so the answer depends on the reasoning being present. Replacing steps with different-class steps lowers it much further, down to 22.08 (Diabetes) and 13.44 (AD) when all steps are replaced. So the answer follows what the steps say, not only whether they are there.

(ii) Test-time computation. If the accuracy gain came from the extra tokens, replacing the reasoning with equal-length filler would preserve it. It does not: with all steps and the summary replaced by filler, accuracy falls from 85.71 to 64.50 on Diabetes and from 76.07 to 60.47 on AD.

[1] Lanham, Tamera, et al. "Measuring faithfulness in chain-of-thought reasoning." arXiv preprint arXiv:2307.13702 (2023).

Q1 (part 2): Feature-level faithfulness: keep only the features referenced in the explanation.

A: We run this test on Credit-g and AD, the two datasets with many input features (20 and 25). On HomeLoan, Diabetes and Credit-D, the inputs have only 11, 8 and 10 features, and ReSS references about four to five of them per instance. So masking up to 4 non-referenced features as done in the paper therefore already leaves only a few of them visible, and keep-only masking would add little. On Credit-g and AD, by contrast, about 15 and 21 features are not referenced, so masking 4 leaves most of them visible, and keep-only masking is a much stricter test.

For each test instance, we keep only the features referenced in ReSS's own explanation, set all other features to "unknown", and regenerate the reasoning and answer. We report the per-instance agreement (the fraction of instances whose prediction matches the original prediction on the full input) and the accuracy of the keep-only predictions. The results are averaged over three seeds.

DatasetReferenced featuresPer-instance agreementAccuracy, originalAccuracy, keep-only
Credit-g4.90 of 2089.33±3.06%77.67±1.88%75.33±3.51%
AD3.56 of 2589.85±1.68%76.07±0.06%75.76±1.03%

On average, ReSS references only 4.90 of the 20 features on Credit-g and 3.56 of the 25 on AD. Even with all remaining features masked, the prediction is unchanged for 89.33%(Credit-g) and 89.85% (AD) of instances. Accuracy changes little (76.07% → 75.76% on AD and 77.67% → 75.33% on Credit-g). This stricter test supports the feature-level sufficiency of the explanations.

Q2: Validating supervision and hallucination numbers.

A: Savings mismatch: Thank you for catching the mismatch in Figure 6 regarding savings. We checked the original instance. The curated rationale in our training data is correct; the mismatch is a transcription error that was introduced when we copied the example into the figure. We have corrected it in the revision.

Methods to compute hallucination numbers: We prompted GPT-4.1 to judge each reasoning step of the fine-tuned model’s outputs, using the three categories defined in Section 4 (FEH, FVH, CH). The rate is the number of hallucinated steps divided by the total number of steps, computed over the full test set of each dataset. The generated rationales contain on average between 4.7 and 5.6 steps per instance.

We did not separately audit the curated rationales for training, and we do not claim that they are error-free. Our evaluation instead targets the outputs of the fine-tuned model, which is what our claims concern: Table 1 reports the low hallucination rate of the generated rationales on the full test set.

Label of augmented data: Only tree-path preservation supports the original label. Augmentation inevitably introduces some label noise. However, our evidence is end-to-end: adding the augmented samples improves accuracy on all five datasets. This indicates that the net effect of the augmentation is positive despite the noise.

Q3: Attribution of gains.

A: To isolate the contribution of the semantic rationales, we compare ReSS with two models trained on exactly the same scaffold-augmented training set, with the same inputs, training instances and budget. They differ only in the supervision target:

  • Direct SFT (x → y): the model is trained to output the label directly.
  • Path-check SFT (x → z_path → y): the rationale is replaced by a templated statement of the decision-tree path (e.g., "Step 1: The Plasma glucose is 148, which is greater than 127.5."), i.e., the same features and thresholds as ReSS but without any domain interpretation. It uses the same prompt as ReSS.
  • ReSS (x → z → y): full scaffold-grounded rationale.
MethodHomeLoanDiabetesCredit-gADCredit-D
Direct SFT (x → y)80.65±1.6175.76±1.9874.00±2.0074.13±0.4278.20±0.13
Path-check SFT (x → z_path → y)81.18±0.9377.06±3.2774.33±4.5173.96±0.4677.53±0.04
ReSS (x → z → y)82.80±2.0185.71±1.0677.67±1.8876.07±0.0680.12±0.10

The results are averaged over three seeds. Our ReSS performs best on all five datasets. Since they share the same path, features and threshold checks, this gap can be attributed to the semantic content of the rationale rather than to which features are named or which checks are made.

Reviewer_H2H5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReSS, a technique for tabular classification which fine-tunes LLMs via a novel data-augmentation scheme. It evaluates two classical models, vanilla SFT, RL and LLM prompting on 5 datasets. Results show that ReSS outperforms baselines across datasets on accuracy.

Critical strengths
  • Informative ablation. The paper shows that when LLMs are fed "delexicalized features" i.e. feature names are anonymized, then performance falls back to the decision tree's level. This suggests the gains come from the LLM's domain knowledge.
  • The paper compares against three model families (classical models, SFT / RL, and zero-shot LLMs).
Critical weaknesses

Please refer to questions.

Questions for authors

1. Incomplete Metrics. The paper reports only accuracy, which is sensitive to class imbalance. Please also report macro-F1, precision and recall, and the class balance of each dataset.

2. Missing Baselines. The paper does not evaluate LightGBM, one of the strongest classical models for tabular classification. It also does not compare its augmentation against standard tabular augmentation such as SMOTE. Please add XGBoost and Direct SFT trained with (i) SMOTE and (ii) your own scaffolded data, at the same training set size. Further, please use a more modern LLM than GPT-4.1 which has been released for 3+ years now.

3. Outperforming TabPFN. App. E.4 reports that ReSS beats TabPFN-3 on all five datasets. Could you give an intuition for why? TabPFN is a leading foundation model on TabArena and I am quite surprised that a data-augmentation technique can beat it.

4. Weak Motivation for Interpretable Classification. The paper states that post-hoc explanations "often fail to constitute truly human-understandable reasoning chains" (L31), but gives no evidence for this claim. On the contrary, a rich body of work has explored the interpretability of post-hoc explainers, in particular counterfactual explanations [1]. These methods are in use in high-stakes settings such as credit decisions, criminal justice and law. Conversely, recent work shows that LLM reasoning chains can be unfaithful to the model's actual decision [2], so they are arguably less faithful than a traditional classifier with a post-hoc explainer. The paper itself cites [2] in its Related Work. For these reasons, the paper should not dismiss tabular FMs as "not interpretable" (L70).

[1] https://arxiv.org/abs/1905.07697 [2] https://arxiv.org/abs/2305.04388

6 comments

We thank the reviewer for the detailed comments. We address the raised concerns below.

Q1: Incomplete Metrics.

A: We have added macro-precision, macro-recall, and macro-F1 and the class distribution of the dataset (Appendix E.5, Tables 5–6 in the revised paper). Across all five datasets, ReSS achieves the best macro-F1, consistent with the accuracy results, indicating that our gains are not an artifact of class imbalance. Note that HomeLoan, Diabetes and Creditg are moderately imbalanced, with a minority class of 30-35%, while Credit-D is balanced, and the AD validation and test sets are nearly balanced (its official training split contains 21.3% AD)

Q2: Missing Baselines.

A: We have added the requested baselines. SMOTE and our scaffold-invariant augmentation are matched in training-set size. ReSS still achieves the highest accuracy on all five datasets.

ModelTraining dataHomeLoanDiabetesCredit-gADCredit-D
LightGBMOriginal77.96±3.3679.22±5.6672.33±2.0873.6578.78
XGBoostOriginal77.42±3.2378.78±3.7577.00±3.0074.1877.80
XGBoost+ SMOTE78.49±2.4677.92±4.6875.67±1.5375.45±0.5578.64±0.46
XGBoost+ ReSS augmentation77.96±3.3678.35±1.9874.00±2.0073.6577.87
Direct SFTOriginal80.11±2.4673.59±2.2171.33±1.1573.56±1.3677.63±0.26
Direct SFT+ SMOTE80.11±2.4676.19±3.2774.67±3.5174.22±0.8178.28±0.58
Direct SFT+ ReSS augmentation80.65±1.6175.76±1.9874.00±2.0074.13±0.4278.20±0.13
ReSS+ ReSS augmentation82.80±2.0185.71±1.0677.67±1.8876.07±0.0680.12±0.10

We also note that GPT-4.1 was released in April 2025. In our paper, it plays two roles: it is the teacher that writes the training rationales, and a zero-shot baseline (Table 3). We now add GPT-5.5 as an additional zero-shot baseline, with all other settings unchanged, and compare both with our unified ReSS model (a single model trained on all five datasets).

ModelHomeLoanDiabetesCredit-gADCredit-D
GPT-4.1 (x → y)81.18±0.9377.49±0.7535.00±3.4670.7569.97
GPT-4.1 (x → z → y)80.64±1.6177.49±1.9857.33±1.5362.0671.82
GPT-4.1 (x, S(x) → z → y)80.11±0.9374.89±0.7560.33±1.5370.2272.04
GPT-5.5 (x → y)81.18±0.9383.98±0.7576.00±4.0074.4471.19
GPT-5.5 (x → z → y)81.18±0.9380.95±1.5069.00±2.0066.1473.74
GPT-5.5 (x, S(x) → z → y)81.18±2.4672.29±5.2569.33±2.0870.7575.56
ReSS-Qwen-2.5-3B-Instruct (x → z → y)81.18±0.7681.82±2.8176.67±2.6274.35±0.3579.98±0.20

GPT-5.5 is clearly stronger than GPT-4.1 in almost all settings. Its best setting is direct prediction (x → y), which produces no rationale; there, our 3B ReSS model is comparable. When GPT-5.5 is asked to reason before answering (x → z → y), ReSS is clearly better. The most direct explanation is that chain-of-thought obtained by prompting, without fine-tuning, does not help prediction. Supplying the decision path at inference (x, S(x) → z → y) does not consistently help GPT-5.5 either, consistent with our finding for GPT-4.1 that the scaffolds are only effective when internalized through fine-tuning rather than given as prompts.

Q3: Why ReSS Outperforms TabPFN.

A: We agree that TabPFN is a strong foundation model, but it relies on a generic prior and cannot use domain knowledge. ReSS is trained with curated reasoning data, which is constructed by combining the GPT4.1's rich domain knowledge with the scaffold from a tree-based model. The scaffold specifies which features and thresholds the reasoning should use, and the LLM's domain knowledge corrects parts of the scaffold that conflict with domain knowledge.

We support this with three comparisons:

  • The scaffold improves reasoning quality. The baseline DRC+SFT solely uses the LLM generated reasoning without the scaffold for training. ReSS (w/o aug.) outperforms it on all five datasets (Table 2), showing that scaffold constraints help the LLM produce higher-quality reasoning data for fine-tuning.
  • The gain comes from the LLM's domain knowledge. In the delexicalization experiment (Fig. 4, left), we anonymize all feature names and values. The resulting model performs nearly the same as the decision tree and consistently below ReSS. So the improvement of ReSS over the tree model comes from the knowledge injected into the reasoning data, not from the scaffold structure alone. App. E.2 gives an example: on a Diabetes test sample, every condition on the tree path is a known diabetes risk factor, yet the tree predicts non-diabetic, which is wrong. The ReSS-trained model follows the same path, gives a clinical interpretation of each condition, and predicts diabetic, which is correct.
  • The gain is not only from data augmentation, and accuracy is not the only contribution. We compare Ress with TabPFN-3 and ReSS (w/o aug.) in the table below. Without augmentation, it already outperforms TabPFN-3 on HomeLoan, Diabetes and Creditg, so the improvement mainly comes from the scaffold and the LLM's domain knowledge. Moreover, TabPFN is pure prediction models that output only a label. ReSS additionally produces step-by-step, human-understandable rationales, and we verify that they are faithful: feature existence and feature value hallucination are 0% on all datasets, comparison hallucination is below 2% (Table 1, right), and the explanations are both sufficient and necessary (Fig. 2). Even where the accuracy improvement over TabPFN is small on AD, Credit-D, ReSS offers explainability that TabPFN cannot.
MethodHomeLoanDiabetesCreditgADCredit-D
TabPFN-381.18±0.7676.19±3.2475.33±4.0375.85±0.5579.36±0.12
ReSS (w/o aug.)82.26±1.3281.39±1.6276.67±2.6275.80±0.6578.68±0.25
ReSS82.80±2.0185.71±1.0677.67±1.8876.07±0.0680.12±0.10

Q4: Motivation for interpretable classification

A: We agree that the statement "Post-hoc explanations often fail to constitute truly human-understandable reasoning chains" was too strong. And we agree many tabular foundation models can be interpreted by post-hoc explanations.

What we meant is narrower: existing post-hoc explanations do not provide a reasoning chain that leads to the prediction. Counterfactual explanations such as [1] generate nearby “what-if” instances for which the classifier gives a different decision. They show alternative ways of changing the outcome, but not how the classifier arrives at its decision for the instance itself. Similarly, InterpreTabNet [2] uses an LLM to describe the learned feature masks. This text identifies which features the model relies on and how they may relate, but does not state how the value of a feature contributes to a class (e.g., that a glucose value above a certain level supports a diabetic prediction). Neither kind of explanation can be used to derive an instance’s prediction from its feature values. In ReSS, by contrast, the reasoning is generated for the purpose of prediction: the model reasons step by step over the instance’s values, each step stating a feature’s value, comparing it with a cut-off, and interpreting its implication with domain knowledge, then outputs the prediction.

We do not assume that our rationales are faithful, but evaluate them. The hallucination rate is very low (Table 1), and the referenced features are necessary and largely sufficient (Fig. 2). For the stronger claim that the reasoning itself is causal to the answer, we add interventions on the reasoning (see Reviewer oLPQ, Q1). Deleting the reasoning steps or replacing them with filler already lowers accuracy, and replacing them with steps from a different-class instance lowers it much further (Fig. 3).

[1] Mothilal, Ramaravind K., Amit Sharma, and Chenhao Tan. "Explaining machine learning classifiers through diverse counterfactual explanations." Proceedings of the 2020 conference on fairness, accountability, and transparency. 2020.

[2] Si, Jacob, et al. "InterpreTabNet: distilling predictive signals from tabular data by salient feature interpretation." arXiv preprint arXiv:2406.00426 (2024)

Thank you for the clarification, in particular for adding macro-F1 as a metric, for using GPT-5.5 as a baseline (and for pointing out that gpt-4.1 came out only last year!).

As a final follow-up, could you report AUROC for credit-g, following TabArena's protocol [1]? All of the metrics reported so far (including macro-F1) are thresholded. I noticed that the credit-g, prior work reports 79.5% accuracy for TabPFN v2 [2], above both the paper's TabPFN-3 (75.3%). I'd appreciate clarification for reproducibility here, since most of the gains in performance of ReSS are 2-3% and within 1 standard deviation.

[1] https://huggingface.co/spaces/TabArena/leaderboard?__theme=system&view=datasets [2] https://arxiv.org/abs/2502.17361

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.