- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a method that utilizes LLM to build decision tree on unstructured (text) data. On the technical side, at each tree node they iteratively refine the rule via TextGrad by computing Gini + Semantic Analysis features. On the evaluation side, they use 5 small datasets, 4 LLM backbones (gemma3 4b, gpt-4.1 nano and mini, qwen3 4b) and compare against a few baselines such as CART+TF-IDF, CoT, TextGrad: they outperform LLM-based methods and tie CART+TF-IDF.
Critical strengths
- The paper is easy to follow. The authors supply examples and concrete implementation steps, so it seems easy to reproduce.
- The method is intuitive. It mimics CART in that it also minimizes the impurity index and greedily splits the space. To integrate the text, the authors use iterative refinement via TextGrad and also add semantic features.
- Compared to other naive LLM approaches, the proposed method is better on the given set of 5 datasets.
Critical weaknesses
- While the method is intuitive, it is not principled per se. For example, what if we skip the semantic analysis, , entirely? Do you do sample bagging? TextGrad has non-trivial variance, how do you battle this, e.g., it would help to see stability analysis (re-run with different seeds / initial prompts)?
- Cost analysis. The proposed method requires LLM at train and inference time and still performs only on par with CART+TF-IDF.
- Many naive baselines are skipped. For example, I would try initializing from CART+TF-DF, then ask a SOTA coding agent such as GPT-6 or Opus 5.5 to replace the TF-IDF with some semantically-informed program, e.g., suppose at node CART splits based on the terms
coughandblood, the agent could rewrite into an inspectable human-readable and transparent code snippet that would use a regexp to check the symptoms. Another approach one could try is to simply prompt the SOTA agent to produce semantically meaningful decision tree with questions at each node. This is cheap, does not require any specific method, and would be the first thing I would try in practice. - Potential data leak. Datasets such as IMDB and Spam have most likely already appeared in the pre-training corpus of the modern LLMs. Hence, I find the following claim to be doubtful. Indeed during the iterative refinement, TextGrad, may steer the prompt towards known solutions. Already in iteration 0 in the Appendix, the model has proposed to split on
severe systemic symptoms such as coughing blood, weight loss, which overlaps with WHO symptom list wording.
derived from the training data rather than from prior knowledge
Some future directions, if the authors choose to pursue this project further is to try combine their method with Jev-like model: it's more budget-friendly, faster to run, and also provides calibrated probabilities.
Questions for authors
Please, see weaknesses above.
4 comments
Answer (2/2) to reviewer z2pj
W3: Naive baselines (CART+TF-IDF rewritten by a coding agent; SOTA agent prompted to produce a tree)
We could not run these two baselines within the discussion period. The closest baseline in the paper is the concept-bottleneck baseline (Con. tag.): the same LLM as ACT proposes 50 salient concepts from the training data and labels each text with them, and a CART of depth 4 is fit on these binary features. ACT outperforms it on all five datasets (e.g., 72.8 vs. 64.7 on DIAGNO and 65.2 vs. 55.8 on BANKCHURN, averaged over LLMs). The closest published variant is LeaPR (Poesia & Sampaio, 2025), now discussed in Section 2: an LLM writes Python features for the leaf being split from its training examples, and the split with the lowest impurity is kept. These features are proposed once per leaf, whereas ACT refines each question over k steps. A program derived from TF-IDF splits also stays blind to negation and composition: a regular expression on cough and blood does not separate "coughing up blood" from "no blood when coughing", whereas an ACT node does.
Changes in the paper: Section 2 now discusses LeaPR and concept-bottleneck approaches, and Section 4 and Appendix E.1 describe the Con. tag. baseline.
W4: Potential data leakage
We agree that pretraining exposure (e.g., IMDB, SPAM) cannot be ruled out and may steer the refinement. However, all LLM-based baselines (CoT, DSPy, TextGrad, Con. tag.) use the same LLMs, so the comparison remains fair. Moreover, these baselines receive the task description and class names, whereas ACT starts from a task-agnostic seed question. The baselines are therefore, if anything, better positioned to exploit prior knowledge.
ACT also yields its largest gains on BANKCHURN, which relies on dataset-specific attribute relationships rather than pretraining-aligned semantics (Appendix B.3): 65.2% vs. 54.1% for DSPy and 53.4% for TextGrad. Regarding the WHO symptoms, DIAGNO derives from DDXPlus, whose synthetic patients are generated from a medical knowledge base compiled from the literature: the WHO symptoms are the human-defined concepts behind the data, which ACT recovers from labeled examples without being told the task. The differences are more telling (Table 4, Appendix D.1). ACT does not use loss of appetite, a TB symptom listed in the medical sources, because it also occurs in the allergic-sinusitis cases of DIAGNO and therefore does not separate the two classes in this data. Conversely, it uses shortness of breath, which is not among the symptoms we collected from these sources but was selected because it splits the training data well. Generic medical knowledge would favour the canonical TB symptoms; ACT's questions instead follow what discriminates the classes in the training data. We revised the discussion of Table 4 accordingly.
We rephrased the sentence in Section 3. It is meant to say that no task information is provided to the LLM, not that prior knowledge plays no role.
Changes in the paper: the sentence of Section 3.2 is rephrased. Section 4.1 and Appendix D.1 revise the discussion of Table 4 accordingly, and Appendix B.3 explains how DDXPlus is generated.
Future direction (Jev-like model)
We thank the reviewer for this suggestion. Since each ACT node is a yes/no question, a decision model such as TypeSafe's Jev could answer node questions at a lower inference cost, with calibrated probabilities that flag uncertain answers. This is a promising direction, with two challenges: question generation would still require a generative LLM, and using probabilities for soft routing would turn a single decision path into a weighted mixture of paths, which is harder to read and contest.
Thank you for the extensive answers! However, I choose to maintain my score. I believe the revisions needed to push this paper are substantial and would change its nature. Hence, I am not comfortable recommending acceptance.
We thank the reviewer for their feedback. In the rebuttal, we added an ablation of the semantic analysis, a significance and stability analysis (5 seeds × 4 LLMs, repeated inference runs) and per-dataset results. These show that ACT outperforms TF-IDF+CART on 4 of 5 datasets and on a new SNLI experiment. We also revised our wording on prior knowledge and data leakage.
These revisions are clarifications and additional analyses that leave the method unchanged. We would therefore be grateful if the reviewer could say which remaining concerns justify the current score and which revisions they believe would change the nature of the paper. We would be happy to provide further clarifications in the remaining time.
Answer (1/2) to reviewer z2pj
We thank the reviewer for their time and helpful comments, which we address below.
W1: Principledness, ablation of semantic analysis, bagging and TextGrad variance
Changes in the paper: Section 3.2 and Algorithm 1 now include the seed-exclusion and pruning rules. The "Ablation studies" paragraph of Section 4.1 discusses the m-ablation. Appendix H (Table 17) reports the significance analysis and Appendix J (Tables 19–20) the consistency of node answers.
W2: Cost vs. TF-IDF+CART
At inference, ACT only queries the nodes along one path. This averages about 1.1–3.1 LLM calls per sample (Table 11), and inference cost stays within the same order of magnitude as TextGrad and below DSPy in tokens (Appendix A). ACT is of course more expensive than TF-IDF+CART.
ACT is also not merely on par with TF-IDF+CART:
TF-IDF splits also ignore negation and word order ("this movie is not good"), and they often rely on uninformative terms such as our or also (Fig. 7–8).
Changes in the paper: the paragraph of Section 4.1 on TF-IDF + CART now gives per-dataset numbers, DIAGNO included. Appendix I (Table 18) reports binary SNLI, and Appendix A corrects the training-cost comparison.