acceptodds
Under review as a conference paper at ICLR 2027

MotherTree: Meta-learning on synthetic data improves decision tree training

Abstract

Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially in small-sample regimes. However, this approach does not produce a standalone model that can be inspected in isolation. We introduce MotherTree, a tabular transformer that meta-learns decision tree induction: given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. MotherTree is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. On established benchmarks with controlled sample size, the approach is competitive with size-matched trees from common algorithms: recursive partitioning, gradient-based tree learning, globally optimal trees, and distillation from tabular foundation models. Notably, MotherTree consistently improves over from-scratch gradient-based learning and acts as a strong initializer: task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. These results show that meta-learning can provide effective inductive biases for learning stand-alone, small decision tree classifiers.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Meta-learning, Decision tree learning, Synthetic data pre-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yXHLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MotherTree is a MotherNet-style hypernetwork with an ancestral-masked transformer decoder. It outputs a depth-5, axis-aligned decision tree (under a GradTree parameterization) in a single forward pass. Authors perform pre-training on TabICL/TabICLv2 synthetic priors with straight-through gradients and no reference trees. A variant, MotherTree-Tune, uses the resulting generated MotherTree model to initialize task-specific optimization similar to that done in GradTree. Evaluation covers 64 TabArena and OpenML-CC18 tasks at training budgets $B \in $ {100, …, 1000}, plus a synthetic parity-with-distractors study, and a variety of reasonably well-chosen ablations.

Critical strengths
  • The problem is well motivated within the considered research area, and a reasonably extensive body of related work is presented. Meta-learning on a synthetic prior as a wider area produces a standalone, auditable tree. Training uses instance labels rather than reference trees, which in particular, alleviates MetaTree’s main bottleneck [Zhuang et al., 2024].

  • There are clear concretely demonstrated gains over the from-scratch counterpart.

    • MotherTree-Tune beats GradTree on both benchmarks, at every budget and for both binary and multiclass tasks (Tables 5–7; Fig. 4).
    • Rank gap exceeds the critical difference at every budget (Fig. 7a–e).
    • Parity study isolates robustness to shortcut features (Fig. 5b, Fig. 8b), using a closed-form distractor-only ceiling (App. A.8.2). In the appendix, you can forego of labeling it as a TeX section which would make referencing subsections more easily digestible (e.g. in the format "Letter.Number").
Critical weaknesses
  • The paper shows no regime where the method beats the incumbent.

    • Against tuned CART, MotherTree-Tune roughly ties at B = 200 (Table 1).
    • It falls behind on TabArena from B = 400 (Table 5; Fig. 2b) and loses on multiclass tasks at larger budgets (Table 7).
    • It is never reasonably significantly separated from CART in the critical-difference diagrams (Fig. 3; Fig. 7a–e).
    • Base MotherTree also trails GradTree on TabArena multiclass at every B ≥ 200 (Table 7). This particularly contradicts the abstract’s claim that it “consistently improves” over from-scratch gradient-based learning.
  • Initialization claim lacks its key control, and several baselines seem to not be tuned properly.

    • MotherTree-Tune is compared only with randomly initialized GradTree, although a CART warm start is already used for RADDT (Table 4; [Mao et al., 2025]).
    • CART is tuned without cost-complexity pruning (which would plausibly induce overfitting most of the time), and DL85 uses 5-quantile binning (Table 4; [Aglin et al., 2020]). Especially in the case of the latter, this could be quite restrictive in rule expressivity regardless of the used algorithm. Would be interesting to see how using cost-complexity pruning for CART + increasing the number of bins for DL85 impacts performance here.
    • Distillation baselines label only the existing training inputs (App. A.4.2).
Questions for authors
  1. Is $ \mathcal{L}_\text{meta}$ (Sec. 3.4) computed on the same points used to generate the tree? If so, why not use held-out query points, as PFN and MotherNet do?
  2. How does GradTree warm-started from CART, or with multiple restarts, compare with MotherTree-Tune on the benchmarks and the parity tasks?
  3. Please specify the split protocol: the test-set size, the “balanced” split used for small data sets (L259), where the hyperparameter-tuning and early-stopping data come from within the budget B
  4. What is N at line (L131)? Is it the number of synthetic pre-training data sets? The same symbol is reused for node tokens (L181) and for the sample count (Alg. 2).
  5. How are tasks to be handled when they have more features or classes than the encoder supports? For example, TabArena’s Bioresponse has 1,777 features.
  6. In which regime would you recommend MotherTree over CART? A TabPFN reference row would show how much accuracy the interpretability constraint costs.
5 comments

We thank the reviewer for their constructive feedback on our work. We address the perceived weaknesses and questions point-by-point below.


The paper shows no regime where the method beats the incumbent

MotherTree-Tune beats the incumbent in the small-sample regime and on binary tasks:

  • In small-sample regimes, B=100B=100 and B=200B=200, MotherTree-Tune performs better than CART in aggregate on both benchmarks (Figure 2, Appendix Table 7). On OpenML CC-18, in aggregate over the whole benchmark, MotherTree-Tune performs better than CART on all sample sizes.

  • On binary tasks, MotherTree-Tune performs better than CART on TabArena until B=400B=400 and for all sample sizes on OpenML.

  • It is common that methods overlap on critical diagrams: This is true also for TabICL, TabPFN and other tabular foundation models---this does not take away from their overall contributions.

  • Finally, we emphasize the importance of the comparison with GradTree since this baseline uses the same architecture (unlike CART), and is the strongest among gradient-based/neural tree learners overall. The improvement of MotherTree-Tune over GradTree is large across sample sizes and task types (with a few exceptions on multi-class tasks). This demonstrates the value and potential of meta-learning (initialization with MotherTree) for gradient-based tree training!


Base MotherTree trails GradTree on TabArena for B≥200B\geq 200, "contradicts the abstract"

  • Thanks for spotting this! The abstract claim was referring to MotherTree-Tune. We have updated the abstract to clarify that the sentence containing ``consistently improves'' refers to using MotherTree as an initializer (i.e., MotherTree-Tune).

CART is tuned without cost-complexity pruning

We have run experiments with cost-complexity tuning on CART, but this consistently fails to improve its results---the results don't change at all. The trees do not overfit without pruning, as the depth is limited to 5 for all methods. Setting the maximum depth to 5 is critical in order to create a fair comparison with other methods producing depth-5 trees.


DL85 increased number of bins

We have tested different numbers of bins for DL85 (3, 5, and 10). The results are shown below:

For TabArena:

Method1002004008001000
DL85(bins=3)0.605±0.0090.632±0.0080.672±0.0030.704±0.0050.712±0.006
DL85(bins=5)0.611±0.0050.629±0.0100.659±0.0080.708±0.0070.714±0.010
DL85(bins=10)0.606±0.0070.627±0.0070.653±0.0060.682±0.0050.693±0.006

For OpenML-CC18:

Method1002004008001000
DL85(bins=3)0.691±0.0040.736±0.0060.768±0.0060.785±0.0050.785±0.005
DL85(bins=5)0.670±0.0050.713±0.0070.747±0.0020.761±0.0050.762±0.006
DL85(bins=10)0.649±0.0100.691±0.0070.718±0.0030.739±0.0050.737±0.005

Distillation only on existing inputs

Interesting question! For a fair comparison, we keep the task-specific training set fixed for all methods. In the revised version, we have added results for distillation on inputs sampled from the TabICLv1 (pre-training) prior. We have also added distillation from soft labels. However, the experiments show that different distillation methods from TabPFN2.6 as a teacher don't close the gap for MotherTree-Tune (Appendix 4.2 Table 4).


Is [the pre-training loss] computed on the same points used to generate the tree?

No, we utilized the same training strategy as TabPFN and MotherNet. We split the sampled prior data into context and query points. The context points are used to generate weights for our downstream tree. The query points are used to compute the loss. We have clarified this in Section 3.4 of the revised paper.

AuthorsOpenReview ↗

Rebuttal pt 2

Warm-starting GradTree

We have added warm-starting of GradTree with CART to the revised paper (see Appendix Table 5). Consistently, for both benchmarks and all sample sizes, GradTree warm-started from CART is better than distillation from TabPFN but worse than CART itself. It is also worse than MotherTree-Tune (except for a few results on multi-class tasks).

For TabArena:

Method1002004008001000
GradTree$_{\leftarrow \mathrm{CART}}$0.662±0.0060.673±0.0090.701±0.0060.729±0.0070.730±0.006
MotherTree-Tune0.676±0.0090.696±0.0060.717±0.0030.735±0.0040.739±0.006

For OpenML-CC18:

Method1002004008001000
GradTree$_{\leftarrow \mathrm{CART}}$0.731±0.0060.764±0.0090.795±0.0030.807±0.0060.808±0.006
MotherTree-Tune0.766±0.0030.786±0.0050.811±0.0050.815±0.0010.816±0.001

From the experiments above, GradTree warm-started from CART closes the gap with other distillation baselines but MotherTree-Tune leads on both benchmarks. More distillation baselines can be seen in Table 4, Appendix 4.2.


Specify the split protocol: the test-set size, the “balanced” split used for small data sets (L259)

A: The protocol is already given in Appendix A.4.4 (Algorithm 1) of the original submission.


What is N?

A: Yes, the notation NN is overloaded. We have updated the notation for the number of tree nodes and in Algorithm 2 in the revision.


How are tasks to be handled when they have more features or classes than the encoder supports?

We respond to this in revision Appendix A.4.5. For tasks with more than 100 features, we compute an ANOVA F-statistic for each feature with respect to the class label and retain the top-100 features with the largest F-value, using the scikit-learn API SelectKBest(score_func=f_classif, k=100).


In which regime would you recommend MotherTree over CART?

We recommend using MotherTree or MotherTree-Tune in small-sample tabular classification problems (number of samples less than 400). This is, in general, the regime where we should expect meta-learning to have the largest effect. Both MotherTree and MotherTree-Tune are currently strongest in binary classification. MotherTree-Tune is also strong on larger-sample binary tasks and small-sample multi-class tasks. From the results in OpenML-CC18, the binary result remains strongest, and the results in TabArena show that MotherTree-Tune is still competitive with larger sample sizes (Table 1).


We hope that our response addresses your concerns and look forward to the discussion!

We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback! If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.

Reviewer_zvycOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper introduces an end-to-end architecture for training decision trees that composes hyper network/dataset encoding, straight-through learning from GradTree, synthetic priors, and an ancestrally-masked transformer decoder.
  • This work enables a tree generator that is trained without example trees by pre-training on synthetic tables based on TabICL/TabICLv2 datasets (and removes limitations from prior work e.g. MetaTree).
  • The authors show that meta-learned priors provide a strong initialization and improve performance on 64 benchmarks tasks, outperforming from-scratch baselines (e.g. GradTree) and matching CART.
  • Ablations show that the tree-aware decoder outperforms an MLP and that generated trees use more of their leaves during inference.
Critical strengths
  • The paper removes the need for a reference tree by enabling synthetic pre-training at scale. The output of the architecture is a tree that is interpretable and auditable.
  • The paper has a clear experimental setup. All methods are evaluated under consistent training budgets and configurations, and results are reported using relevant metrics.
Critical weaknesses
  • While the paper demonstrates clear wins over parameterized baselines (e.g. GradTree), it does not show a large uplift relative to other strong baselines (e.g. CART).
  • Baselines could also be strengthened. MetaTree and DNDT are capped at 10 features, and MetaTree at 256 samples. These results may be impacted by their input limits.
  • The tree shape is fixed. Every tree is depth-5, and the method doesn't admit different configurations which may vary based on use case. The model learns from synthetic tables that are not described in the paper.
Questions for authors
  • Suggestion - To substantiate the claim about initialization, could you show the impact of GradTree when warm-started with CART?
  • Suggestion - How is the generator trained to generalize if the same rows are used for training and validation?
  • Suggestion - Does the gain comes from the ancestor rule or from using a transformer? Is it possible to update ablations to show that the decoder does indeed impact the lift?
7 comments

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each comment point by point.


No large uplift relative to other strong baselines (CART)

Figure 2 and Table 1 show large uplift for MotherTree-Tune vs CART in the low-sample regime on both benchmarks, especially in the binary tasks. These results are expanded on in Tables 7,8,9. We have added paired significance tests vs CART in Tables 11, 12.

We want to emphasize the importance of the comparison with GradTree since this baseline uses the same architecture (unlike CART), and is the strongest among gradient-based/neural tree learners overall. The improvement of MotherTree-Tune over GradTree is large across sample sizes and task types (with a few exceptions on multi-class tasks). This demonstrates the value and potential of meta-learning (initialization with MotherTree) for gradient-based tree training!


Baselines could also be strengthened

We have updated the results for DNDT and DL85 when selecting from a larger set of hyperparameters (more features, quantiles) with grid search. The original claims regarding these methods are intact.

Running MetaTree with more samples or features requires re-training. Our comparison in the initial submission is against the method as published. We present a dedicated comparison with MetaTree on tasks that satisfy MetaTree's input constraints in revision Appendix Table 6. Here, we limit the training set size of the evaluation tasks to a maximum of 256 and a subset of at most ten features, for all learners. MotherTree consistently outperforms MetaTree, also in MetaTree's domain. On a general note: When comparing with other tabular foundation models (e.g., TabPFN), we would not be expected to retrain the model from scratch.


The tree shape is fixed. Every tree is depth-5

Yes, this is a limitation of our study (acknowledged in Section 5 in the original submission). To get a task-specific topology, we can pre-train MotherTree to generate large trees that are pruned post-hoc for each task, resulting in different topologies. We have added a comment to this effect in the Limitations section of the revision. The empirical comparison in the original submission is fair since all methods are subject to this constraint.


The model learns from synthetic tables that are not described in the paper

This is not accurate. The prior used for pre-training is described in Appendix A.3 of the original submission.


Warm-starting GradTree with CART

We have added warm-starting of GradTree with CART to the revised paper (see Appendix Table 5). Consistently, for both benchmarks and all sample sizes, GradTree warm-started from CART is better than distillation from TabPFN but worse than CART itself. It is also worse than MotherTree-Tune overall, but better on large-sample multi-class tasks (Appendix Tables 7,8,9).

For TabArena (overall):

Method1002004008001000
GradTree$_{\leftarrow \mathrm{CART}}$0.662±0.0060.673±0.0090.701±0.0060.729±0.0070.730±0.006
MotherTree-Tune0.676±0.0090.696±0.0060.717±0.0030.735±0.0040.739±0.006

For OpenML-CC18 (overall):

Method1002004008001000
GradTree$_{\leftarrow \mathrm{CART}}$0.731±0.0060.764±0.0090.795±0.0030.807±0.0060.808±0.006
MotherTree-Tune0.766±0.0030.786±0.0050.811±0.0050.815±0.0010.816±0.001

From the experiments above, GradTree warm-started from CART closes the gap for other distillation baselines but MotherTree-Tune still leads evidently on both benchmarks. More distillations baselines can be seen in Table 4, Appendix 4.2.


How is the generator trained to generalize if the same rows are used for training and validation?

In MotherTree, we don't use the same rows for training and validation. We split the pre-training tasks into disjoint training and validation portions. The training portion generates the tree, and the validation portion is used to compute the loss. We have clarified the notation in Section 3.4 in the revision to make this clear.


Does the gain come from the ancestor rule or from using a transformer?

Our current experiments show that the transformer decoder with ancestral decoding improves over the MLP decoder. Our claim is that "explicitly modeling dependencies among tree nodes provides a more effective inductive bias for generating and initializing decision trees." (l.397 in the original submission). The MLP decoder lacks dependence between the outputs of different nodes. This is what we claim to improve on using the transformer, which adds such dependence. A transformer without the ancestral rule also models such dependencies explicitly, just not in a hierarchy.

We hope that our response addresses your concerns and look forward to the discussion!

Thank you for clarifying the training objective and adding details on MetaTree. I also appreciate the correction in Appendix A.3, as well as the new CART warm-start experiments.

A few follow-up questions:

  1. On CART - the advantage w/ B=100 looks concentrated in imbalanced datasets. Could you split results by class balance, add CART with class weighting, and report a metric besides AUC?
  2. On ancestral decoding - Is it possible to clarify the ablation by running this with the mask removed? Rewording Section 3.3 could also help.
  3. On warm start - GradTree/CART scores below CART on every budget (Table 7). Does the mapped tree match CART before training? After the 20% validation split, do all methods train on all B rows?

Thank you for engaging with our submission!


Performance vs. Class Imbalance

Thank you for this suggestion! We have added a breakdown of the CART and MotherTree (-Tune) results over three levels of class imbalance to Appendix 8 in our newest revision. We measure imbalance with the imbalance ratio IR=max⁡c∈Cncmin⁡c∈Cnc,\mathrm{IR} = \frac{\max_{c \in \mathcal{C}} n_c}{\min_{c \in \mathcal{C}} n_c}, where ncn_c is the number of samples for class cc. We group tasks by low, medium, and high class imbalance and have added class weighting to CART (see Appendix 8, Table 14 for a comparison with standard CART).

Indeed, we find that the MotherTree-Tune advantage is strongest, in terms of AUC, on tasks with medium and high imbalance, and CART stronger on low-imbalance tasks, see below and in Appendix 8, Table 16.

AUC (pooled across benchmarks)

LevelMethod1002004008001000
Low (n=34)CART (weight balance)0.743±0.0050.773±0.0040.799±0.0030.812±0.0030.814±0.003
MotherTree0.723±0.0040.746±0.0050.763±0.0050.769±0.0040.769±0.006
MotherTree-Tune0.738±0.0070.766±0.0040.786±0.0040.797±0.0040.794±0.008
Medium (n=18)CART (weight balance)0.689±0.0190.715±0.0160.747±0.0080.768±0.0060.772±0.007
MotherTree0.714±0.0160.736±0.0080.755±0.0060.767±0.0030.766±0.002
MotherTree-Tune0.718±0.0110.739±0.0090.755±0.0070.772±0.0070.775±0.008
High (n=16)CART (weight balance)0.615±0.0060.649±0.0140.676±0.0070.701±0.0060.709±0.008
MotherTree0.642±0.0150.667±0.0120.692±0.0080.693±0.0110.693±0.016
MotherTree-Tune0.659±0.0200.682±0.0130.710±0.0050.717±0.0150.724±0.014

We also report the balanced accuracy in revision Table 16. Here, we find that CART generally has stronger balanced accuracy, despite worse AUC for many of the tasks. This is not a metric we have produced during development and therefore haven't optimized for.


Results for MotherTree without ancestral decoding

We have added an ablation comparing ancestral attention with unrestricted attention, in the MotherTree transformer decoder, which shows that the predictive benefits of ancestral attention are modest and dataset-dependent—but sparse attention has other benefits (see below)!

On TabArena, ancestral attention yields small improvements at larger context sizes (up to 0.006 AUC), whereas the differences on OpenML-CC18 are smaller (up to 0.003 AUC). We therefore do not claim that ancestral attention consistently improves predictive performance across all settings.

For TabArena:

Method1002004008001000
MotherTree0.672 ± 0.0100.695 ± 0.0050.714 ± 0.0050.725 ± 0.0050.726 ± 0.006
MotherTree(w/o ancestral attention)0.676 ± 0.0100.696 ± 0.0060.708 ± 0.0060.720 ± 0.0030.720 ± 0.004

For OpenML-CC18:

Method1002004008001000
MotherTree0.739 ± 0.0060.763 ± 0.0050.780 ± 0.0060.785 ± 0.0060.785 ± 0.006
MotherTree(w/o ancestral attention)0.739 ± 0.0060.762 ± 0.0040.777 ± 0.0050.782 ± 0.0040.783 ± 0.004

However, ancestral attention provides a tree-structured inductive bias: each node conditions only on its ancestors and itself, reflecting the hierarchical dependencies inherent in decision trees. This design also reduces the number of allowed attention connections from O(N2)O(N^2) to O(Nd)O(Nd) for a complete binary tree with N=O(2d)N=O(2^d) nodes. We emphasize that this characterizes the sparsity of the attention pattern; the corresponding computational savings require an implementation that exploits sparse attention. In conclusion: We achieve the same predictive performance with a leaner architecture.

We have revised the discussion to distinguish the contribution of the transformer decoder from the more specific contribution of ancestral attention. Our central architectural comparison is between the tree-structured Transformer decoder and the generic MLP decoder, while the ancestral attention ablation provides further insight into the role of explicitly encoding tree hierarchy. We have reworded Section 3.3 to reflect this, on your suggestion.


Clarification for the result of GradTree warm-started from CART

  • This is correct. The GradTree/CART baseline matches the structure of the CART tree used for initialization exactly. The leaf values are initialized randomly.
  • Every method is given access to the same BB rows to use for training and validation. If they rely on a validation set, that is taken from those BB rows, as a 20% split. If the method does not rely on a validation set (MetaTree, LLM-Tree, Tsallis), they train on the full set of BB rows.
Reviewer_RV9mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MotherTree, a transformer hypernetwork that maps a labelled training set to the parameters of a hard, axis-aligned, complete binary decision tree. It combines the MotherNet data set encoder with class-wise mean pooling and a decoder whose attention mask lets each node attend only to its ancestors. It adopts GradTree’s parameterisation and straight-through gradients and is pre-trained on synthetic priors, without reference trees. Leaf values are recalibrated to the observed class proportions. A variant, MotherTree-Tune, then fine-tunes all tree parameters with GradTree’s procedure.

The evaluation covers 64 classification data sets from OpenML-CC18-Small and TabArena, at training budgets of 100 to 1,000 samples, against ten single-tree learners. MotherTree-Tune exceeds GradTree at every budget on both benchmarks. A decoder ablation and a parity-with-distractors experiment complete the study.

Critical strengths

The formulation is clean. It trains end-to-end through a hard tree without reference trees and produces a standalone model. Because MotherTree shares GradTree’s parameterisation, depth and tuning procedure, comparing the two isolates the effect of a meta-learned initialisation. On that comparison the result is robust: MotherTree-Tune outperforms both GradTree and GradTree-TabPFN at every budget on both benchmarks.

The experiments go beyond benchmarks. The decoder ablation (Figure 4) supports the ancestral mask. The parity task’s closed-form, distractor-only AUC ceiling (Appendix A.8.2) and its depth-slack parameter λ give a mechanistic test of feature selection. The paper also reports where the method falls short (multiclass TabArena, larger budgets), which appropriately limits its claims.

Critical weaknesses

The headline claims are measured against a weak reference and overstate what the tables show. GradTree, “the corresponding from-scratch learner”, has the lowest TabArena AUC of all methods. Against CART, the default learner the paper itself names, base MotherTree leads at B = 100. At B = 200 it trails on CC18, and at B = 1,000 it trails. MotherTree-Tune leads CART on CC18 at B = 100, but is otherwise within ±0.012 of it and trails it on TabArena from B = 400. The reported standard deviations are taken across data sets and splits, so they say nothing about whether these margins are significant. The critical-difference diagrams do not state which pairs are separated.

The abstract’s claim that MotherTree “consistently improves over from-scratch gradient-based learning” is also contradicted by Table 7. Base MotherTree is below GradTree on TabArena multiclass at every budget.

The numbers are hard to reconcile. The contributions list says MotherTree exceeds GradTree “by 0.069 on TabArena and 0.077 on OpenML CC-18 for B = 100”. But 0.077 is MotherTree-Tune’s gain; base MotherTree’s gain on CC18 is 0.052. The central claim is that meta-learning provides effective inductive biases for small trees. The main text should therefore state the gain over CART with paired tests and limit “consistently” to MotherTree-Tune vs. GradTree.

The comparison with other ways of transferring prior knowledge is not controlled well enough to support the claim that “parameter meta-learning is an effective mechanism for transferring prior knowledge to trees”

Distillation baseline: it samples one hard TabPFN label for each existing training point and appends it. The student therefore sees no new inputs, and the teacher’s knowledge arrives only as relabelling noise on points the student already has. Soft labels are also ruled out on the grounds that CART is non-differentiable, yet the same hard-label scheme is applied to GradTree, which is differentiable(?).

MetaTree: it is run as released, with at most 256 samples and 10 features (Table 4). That comparison cannot separate label supervision from reference-tree supervision, yet section 2 states that learning from instance labels “proved preferable in our experiments”.

Structure claims: they are supported only against GradTree (Figure 6). CART, which by construction has no contradictory paths, is absent, and Figure 6 puts leaf utilisation for both methods below about 0.7. That matters for a paper whose motivation is inspectable, standalone trees.

Questions for authors

Does MotherTree outperform CART at small budgets? Can you report paired per-data set tests (e.g. Wilcoxon signed-rank) for MotherTree and MotherTree-Tune against CART at B = 100 and B = 200, and state which pairs in Figure 3 are separated by the critical difference? Does CART with cost-complexity pruning change the picture?

What does the distillation baseline transfer? Would distillation with teacher labels on additional inputs (perturbed training points, or samples from the prior), or with soft labels for GradTree, close the gap at B = 100? Evidence that a stronger distillation baseline still trails MotherTree would materially strengthen the transfer-mechanism claim.

Please reconcile the reported gains, or explain if I misinterpreted them. Which figures in the contributions list refer to MotherTree and which to MotherTree-Tune? Why do Figure 4 and Table 5 disagree for base MotherTree on TabArena at B = 100?

3 comments

We sincerely thank the reviewer for their thoughtful and constructive feedback. Below, we address each comment point by point.


The headline claims

  • The reviewer is right that the original submission was not clear on which claims were about MotherTree (Base) and which were about MotherTree-Tune---they were stated about the family of MotherTree models as a whole. In the revision, each claim refers only to either "MotherTree" (without tuning) or "MotherTree-Tune".

  • Our contributions demonstrate that not only can meta-learning in a neural architecture generate a strong decision tree in the small-sample regime (MotherTree), but can also provide a fantastic initializer (MotherTree-Tune), transforming GradTree to be competitive with the strongest tree learners in our comparison! We have updated our contributions section to clarify this and adjusted the title slightly to place further emphasis on gradient-based learning.


The claims are measured against a weak reference

  • We respectfully disagree with this assessment. GradTree is the strongest reference we could find for gradient-based axis-aligned decision trees based on published results. Moreover, in our experiments, GradTree is actually one of the strongest methods on TabArena overall. (see e.g., Table 7) For higher sample sizes, it outperforms MetaTree, DNDT, RADDT, Tsallis, and LLM-Tree. This is precisely why we meta-learn this architecture---improving its sample efficiency has great potential! Our results show that meta-learning GradTree through MotherTree-Tune substantially improves its performance, taking it from one of the worst performers in small samples to the strongest results in the field!

Claim that MotherTree consistently improves over from-scratch gradient-based learning in the abstract

  • The abstract's claim was referring to MotherTree-Tune. We have updated the abstract to make clear that the claim ``consistently improves'' refers to using MotherTree as an initializer.

Standard deviation as margin between methods

  • Thank you for this suggestion! In the revision, we have updated the standard deviations for all the tables in the paper. They are now the standard deviation over the 5 random splits for the benchmark performance averaged across tasks.

The critical-difference diagrams do not state which pairs are separated

  • We have added an explanation for how to interpret the critical-difference diagrams in their caption

"The numbers are hard to reconcile [re: contributions in Section 1]"

  • Thank you for pointing this out! This is now resolved. In the revision, we have updated the claims in the contributions (Section 1) with the correct gains of MotherTree (without tuning) vs. GradTree.

Question: Reconciling Figure 4 and Table 5

  • We double-checked the reported results in Table 5 (original submission) and found that they were for MotherTree without tuning of leaf values. We have corrected this in the revision. Tuning leaf values in the extreme low-sample regime is worse, since the leaf values can overfit to the training data. As the number of samples increases, tuning leaf values achieves better results.

Question: Does MotherTree outperform CART at small budgets?

  • We have performed per-dataset Wilcoxon signed-rank tests at budgets B=100B=100 and B=200B=200. The results are in revision Tables 11, 12. We also report the test results for MotherTree without leaf tuning in this case.
  • The results show that MotherTree without leaf tuning outperforms CART on both benchmarks with a significant margin at B=100B=100, and on TabArena at B=200B=200. This indicates that leaf tuning could introduce overfitting in extremely scarce data settings. We argue this point in the revised paper.
  • On OpenML-CC18, MotherTree-Tune wins significantly more tasks than CART on both budgets, while base MotherTree loses out slightly. On TabArena, tuning is less effective and both models tie or perform slightly worse than CART.

Does CART with cost-complexity pruning change the picture?

  • In short, no. We have run experiments with cost-complexity tuning on CART, but this consistently fails to improve its results---the results don't change at all as the optimal pruning turns out to be keeping the learned tree. The trees do not overfit without pruning, as the depth is limited to 5 for all methods. Setting the maximum depth to 5 is critical to ensure a fair comparison with other methods producing depth-5 trees.

AuthorsOpenReview ↗

Rebuttal pt 2

Comparison with other ways of transferring prior knowledge

  • We have made substantial efforts to strengthen the comparison with other methods for transferring prior knowledge (see below).

Question: What does the distillation baseline transfer?

  • We have added new distillation baselines, with results in Table 4, Appendix 4.2 in the revision. We added both soft-label distillation to GradTree, distillation from additional inputs drawn from the pretraining prior, and warm-starting GradTree from CART. The MotherTree family remains the strongest method, which strengthens the claim that MotherTree is a good initializer.

  • There is a fixed amount of real-world samples available to any method for any tasks, including distillation methods and MotherTree. We respectfully disagree that hard labels merely add "relabelling noise." Although the distillation baseline does not learn from new input locations, this does not imply that it transfers no additional knowledge. The predictions generated by TabPFN, for example, reflect the predictive patterns and inductive biases acquired through its pretraining, which can provide informative supervision beyond the original training labels. In this way, hard-label distillation often has lower noise than the original labels!


Comparison with MetaTree

We present a dedicated comparison with MetaTree on tasks that satisfy MetaTree's input constraints in revision Appendix Table 6. Here, we limit the training set size of the evaluation tasks to a maximum of 256 and a subset of at most ten features, for all learners. MotherTree consistently outperforms MetaTree, also in MetaTree's domain.

Running MetaTree with more samples or features requires re-training. Our comparison in the initial submission is against the method as published. Generally, when comparing results to (tabular) foundation models (e.g., TabPFN), we would not be expected to retrain the foundation model from scratch.


Structure claims vs CART

We have added the structural comparison with CART to the caption of Figure 6: CART is without any uninformative paths by construction and each leaf nodes can be utilized.


We hope that our response addresses your concerns and look forward to the discussion!

We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback! If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.

Comments on the paper

2 comments

Dear reviewers and chairs,

Based on the reviewers' thoughtful and generous suggestions, we have submitted a revision that includes the following changes and additions.

  • Clarified claims regarding the performance of MotherTree and MotherTree-Tune vs. baselines, including GradTree and CART:
    • MotherTree and MotherTree-Tune provide the most value, relative to baselines, in small-sample binary tasks.
    • MotherTree is the strongest initializer of GradTree on binary tasks of all sample sizes, and competitive with CART warm-start on multi-class tasks.
  • Updated standard deviation in Table 1 (now the standard deviation over random splits for benchmark average performance )
  • Paired per-dataset significance tests of MotherTree and MotherTree-Tune vs CART (rev. Table 11 & 12).
  • Results for improved baselines (GradTree warm-started from CART, distillation with soft labels and additional inputs sampled from a prior, more hyperparameter settings for DNDT, DL85). The original claims remain unchanged (rev. Tables 1,7,8,9).
  • A dedicated comparison with MetaTree on tasks that satisfy MetaTree's input constraints (rev. Table 6).
  • Fixed ambiguous notation regarding the dataset used to generate the tree and compute the loss during pre-training (rev. section 3.1).
  • Proposed adding "gradient-based" to the title to focus the attention of the reader to the comparison with other gradient-based decision tree learners

We appreciate the time and effort the reviewers have devoted to evaluating our work. We are confident that our submission is now even stronger as a result.

AuthorsOpenReview ↗

Last days of discussion

Dear Reviewers and Area Chair,

Thank you for your thoughtful reviews and follow-up questions. We are eager to hear from you before the discussion window closes!

Since the original submission, we have made several revisions based on your feedback, including:

  • Clarified the main claims. The abstract and contributions section now clearly distinguish MotherTree as a standalone tree generator from MotherTree-Tune, which uses the generated tree to initialize GradTree. In the contributions, we clarify that the clearest gains over CART are on small-sample binary tasks.
  • Strengthened the baselines. We added CART warm-starting of GradTree and found that initializing with MotherTree still yielded higher mean AUC on both benchmarks at every sample size tested (Table 5). We also added soft-label distillation and distillation with additional synthetic inputs (Table 4), expanded the tuning of DNDT and DL85 (all tables with these methods), and compared with MetaTree under its own input constraints (Table 6). The claimed preferences for MotherTree remain intact.
  • Examined where the gains hold. We added paired per-dataset tests against CART (Tables 11–12), updated the reporting of standard deviation across splits, and compared with class-weighted CART across three levels of class label imbalance. At 100 training examples, MotherTree-Tune reached 0.659 mean AUC versus 0.615 for weighted CART on highly imbalanced tasks. Weighted CART led on low-imbalance tasks and generally had higher balanced accuracy but lower AUC (Table 16).
  • Tested the role of ancestral attention. At a reviewer’s suggestion, we removed the decoder’s attention mask in another ablation. The AUC differences were small and dataset-dependent, but most often in favor of ancestral attention. Given the potential cost savings of an exponentially sparse attention mask, ancestral attention is therefore preferable, and would be even if the AUC difference were negligible.
  • Clarifications. We also clarified which data generate the tree and which data determine the pretraining loss.

We’d welcome a brief indication of which concerns these revisions resolve and which remain. If the new evidence changes your assessment, we’d appreciate an updated score so the AC can take your current view into account.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.