- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MotherTree is a MotherNet-style hypernetwork with an ancestral-masked transformer decoder. It outputs a depth-5, axis-aligned decision tree (under a GradTree parameterization) in a single forward pass. Authors perform pre-training on TabICL/TabICLv2 synthetic priors with straight-through gradients and no reference trees. A variant, MotherTree-Tune, uses the resulting generated MotherTree model to initialize task-specific optimization similar to that done in GradTree. Evaluation covers 64 TabArena and OpenML-CC18 tasks at training budgets $B \in $ {100, …, 1000}, plus a synthetic parity-with-distractors study, and a variety of reasonably well-chosen ablations.
Critical strengths
-
The problem is well motivated within the considered research area, and a reasonably extensive body of related work is presented. Meta-learning on a synthetic prior as a wider area produces a standalone, auditable tree. Training uses instance labels rather than reference trees, which in particular, alleviates MetaTree’s main bottleneck [Zhuang et al., 2024].
-
There are clear concretely demonstrated gains over the from-scratch counterpart.
- MotherTree-Tune beats GradTree on both benchmarks, at every budget and for both binary and multiclass tasks (Tables 5–7; Fig. 4).
- Rank gap exceeds the critical difference at every budget (Fig. 7a–e).
- Parity study isolates robustness to shortcut features (Fig. 5b, Fig. 8b), using a closed-form distractor-only ceiling (App. A.8.2). In the appendix, you can forego of labeling it as a TeX section which would make referencing subsections more easily digestible (e.g. in the format "Letter.Number").
Critical weaknesses
-
The paper shows no regime where the method beats the incumbent.
- Against tuned CART, MotherTree-Tune roughly ties at B = 200 (Table 1).
- It falls behind on TabArena from B = 400 (Table 5; Fig. 2b) and loses on multiclass tasks at larger budgets (Table 7).
- It is never reasonably significantly separated from CART in the critical-difference diagrams (Fig. 3; Fig. 7a–e).
- Base MotherTree also trails GradTree on TabArena multiclass at every B ≥ 200 (Table 7). This particularly contradicts the abstract’s claim that it “consistently improves” over from-scratch gradient-based learning.
-
Initialization claim lacks its key control, and several baselines seem to not be tuned properly.
- MotherTree-Tune is compared only with randomly initialized GradTree, although a CART warm start is already used for RADDT (Table 4; [Mao et al., 2025]).
- CART is tuned without cost-complexity pruning (which would plausibly induce overfitting most of the time), and DL85 uses 5-quantile binning (Table 4; [Aglin et al., 2020]). Especially in the case of the latter, this could be quite restrictive in rule expressivity regardless of the used algorithm. Would be interesting to see how using cost-complexity pruning for CART + increasing the number of bins for DL85 impacts performance here.
- Distillation baselines label only the existing training inputs (App. A.4.2).
Questions for authors
- Is $ \mathcal{L}_\text{meta}$ (Sec. 3.4) computed on the same points used to generate the tree? If so, why not use held-out query points, as PFN and MotherNet do?
- How does GradTree warm-started from CART, or with multiple restarts, compare with MotherTree-Tune on the benchmarks and the parity tasks?
- Please specify the split protocol: the test-set size, the “balanced” split used for small data sets (L259), where the hyperparameter-tuning and early-stopping data come from within the budget B
- What is N at line (L131)? Is it the number of synthetic pre-training data sets? The same symbol is reused for node tokens (L181) and for the sample count (Alg. 2).
- How are tasks to be handled when they have more features or classes than the encoder supports? For example, TabArena’s Bioresponse has 1,777 features.
- In which regime would you recommend MotherTree over CART? A TabPFN reference row would show how much accuracy the interpretability constraint costs.
5 comments
Rebuttal pt 2
Warm-starting GradTree
We have added warm-starting of GradTree with CART to the revised paper (see Appendix Table 5). Consistently, for both benchmarks and all sample sizes, GradTree warm-started from CART is better than distillation from TabPFN but worse than CART itself. It is also worse than MotherTree-Tune (except for a few results on multi-class tasks).
For TabArena:
| Method | 100 | 200 | 400 | 800 | 1000 |
|---|---|---|---|---|---|
| GradTree$_{\leftarrow \mathrm{CART}}$ | 0.662±0.006 | 0.673±0.009 | 0.701±0.006 | 0.729±0.007 | 0.730±0.006 |
| MotherTree-Tune | 0.676±0.009 | 0.696±0.006 | 0.717±0.003 | 0.735±0.004 | 0.739±0.006 |
For OpenML-CC18:
| Method | 100 | 200 | 400 | 800 | 1000 |
|---|---|---|---|---|---|
| GradTree$_{\leftarrow \mathrm{CART}}$ | 0.731±0.006 | 0.764±0.009 | 0.795±0.003 | 0.807±0.006 | 0.808±0.006 |
| MotherTree-Tune | 0.766±0.003 | 0.786±0.005 | 0.811±0.005 | 0.815±0.001 | 0.816±0.001 |
From the experiments above, GradTree warm-started from CART closes the gap with other distillation baselines but MotherTree-Tune leads on both benchmarks. More distillation baselines can be seen in Table 4, Appendix 4.2.
Specify the split protocol: the test-set size, the “balanced” split used for small data sets (L259)
A: The protocol is already given in Appendix A.4.4 (Algorithm 1) of the original submission.
What is N?
A: Yes, the notation is overloaded. We have updated the notation for the number of tree nodes and in Algorithm 2 in the revision.
How are tasks to be handled when they have more features or classes than the encoder supports?
We respond to this in revision Appendix A.4.5. For tasks with more than 100 features, we compute an ANOVA F-statistic for each feature with respect to the class label and retain the top-100 features with the largest F-value, using the scikit-learn API SelectKBest(score_func=f_classif, k=100).
In which regime would you recommend MotherTree over CART?
We recommend using MotherTree or MotherTree-Tune in small-sample tabular classification problems (number of samples less than 400). This is, in general, the regime where we should expect meta-learning to have the largest effect. Both MotherTree and MotherTree-Tune are currently strongest in binary classification. MotherTree-Tune is also strong on larger-sample binary tasks and small-sample multi-class tasks. From the results in OpenML-CC18, the binary result remains strongest, and the results in TabArena show that MotherTree-Tune is still competitive with larger sample sizes (Table 1).
We hope that our response addresses your concerns and look forward to the discussion!
Re:
We hope that our responses and additional experimental results have helped clarify your concerns. As the discussion period draws to a close, we would welcome any further feedback! If you find our responses satisfactory, we kindly invite you to reconsider your evaluation. Thank you for your time and consideration.
Rebuttal
We thank the reviewer for their constructive feedback on our work. We address the perceived weaknesses and questions point-by-point below.
The paper shows no regime where the method beats the incumbent
MotherTree-Tune beats the incumbent in the small-sample regime and on binary tasks:
In small-sample regimes, B=100 and B=200, MotherTree-Tune performs better than CART in aggregate on both benchmarks (Figure 2, Appendix Table 7). On OpenML CC-18, in aggregate over the whole benchmark, MotherTree-Tune performs better than CART on all sample sizes.
On binary tasks, MotherTree-Tune performs better than CART on TabArena until B=400 and for all sample sizes on OpenML.
It is common that methods overlap on critical diagrams: This is true also for TabICL, TabPFN and other tabular foundation models---this does not take away from their overall contributions.
Finally, we emphasize the importance of the comparison with GradTree since this baseline uses the same architecture (unlike CART), and is the strongest among gradient-based/neural tree learners overall. The improvement of MotherTree-Tune over GradTree is large across sample sizes and task types (with a few exceptions on multi-class tasks). This demonstrates the value and potential of meta-learning (initialization with MotherTree) for gradient-based tree training!
Base MotherTree trails GradTree on TabArena for B≥200, "contradicts the abstract"
CART is tuned without cost-complexity pruning
We have run experiments with cost-complexity tuning on CART, but this consistently fails to improve its results---the results don't change at all. The trees do not overfit without pruning, as the depth is limited to 5 for all methods. Setting the maximum depth to 5 is critical in order to create a fair comparison with other methods producing depth-5 trees.
DL85 increased number of bins
We have tested different numbers of bins for DL85 (3, 5, and 10). The results are shown below:
For TabArena:
For OpenML-CC18:
Distillation only on existing inputs
Interesting question! For a fair comparison, we keep the task-specific training set fixed for all methods. In the revised version, we have added results for distillation on inputs sampled from the TabICLv1 (pre-training) prior. We have also added distillation from soft labels. However, the experiments show that different distillation methods from TabPFN2.6 as a teacher don't close the gap for MotherTree-Tune (Appendix 4.2 Table 4).
Is [the pre-training loss] computed on the same points used to generate the tree?
No, we utilized the same training strategy as TabPFN and MotherNet. We split the sampled prior data into context and query points. The context points are used to generate weights for our downstream tree. The query points are used to compute the loss. We have clarified this in Section 3.4 of the revised paper.