acceptodds
Under review as a conference paper at ICLR 2027

How should Language Models See Tree Models? Adaptive Representations for Reasoning

Abstract

Language models increasingly reason over compact artifacts produced by machine learning pipelines rather than raw data. For tabular problems, a fitted decision tree or boosted tree ensemble is a natural such artifact, yet the language model sees only a textual representation of it. Prior work typically fixes this representation, making its effect difficult to separate from the rest of the pipeline. We introduce a representation controlled framework that varies only the tree to language representation, holding the fitted model, the language model, and the evaluation pipeline fixed. Across two tasks (rule generation and budgeted counterfactual search), five public datasets, and three language models, representation choice has a large and highly heterogeneous effect. No representation is consistently best across datasets or models, and on some datasets the spread exceeds sixty recall points. This variability motivates studying the representation as part of an iterative inference process. We first study fixed-view iteration, which keeps the tree representation unchanged across rounds, and then introduce DRALoop, which re-points the exposed information toward the current unresolved state while keeping both models frozen. Iteration alone provides substantial gains over one-shot reasoning, while DRALoop adds beyond this matched control in regimes with residual headroom. On rule generation, it exceeds the best one-shot fixed representation on all five datasets for all three language models. On counterfactual search, its strongest configuration reaches 92.0% of the attainable validity bound, compared with 81.6% for matched fixed-view iteration. These results suggest that when one learned system reasons over another, the representation connecting them can be treated as a controllable and adaptive component of inference.

Primary area:
interpretability and explainable AI
Keywords:
tree representations, language model reasoning, tabular learning, representation adaptation, test-time adaptation, rule generation, counterfactual search

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CrXtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to represent pretrained tree models as text to help LLMs extract rules and identify input changes that alter model predictions. It proposes DRALoop, which iteratively updates these representations using failure feedback, and reports adaptive gains on some tasks.

Critical strengths

None

Critical weaknesses

Missing baselines and unclear LLM necessity. The paper lacks a non-LLM baseline combining residual-driven greedy rule selection with local rule search over the same rule pool. Such a comparison is necessary to determine whether the gains primarily come from external residual computation and rule ranking rather than LLM reasoning. Counterfactual search baselines with equivalent access to tree structures are also missing.

Candidate counts do not establish equal computational budgets. Evaluating 12 candidates per method does not ensure comparable costs. The amount of exposed paths and rules, precomputed statistics, LLM input/output tokens, and invocation latency may differ substantially. Efficiency claims therefore require actual runtime and cost measurements.

Insufficient evidence for adaptive representation updates. Different datasets benefiting from different representations does not establish that updating representations within the same task improves performance. Stronger controls include a validation-selected fixed representation and a rotation of fixed representations under matched budgets. The current experiments do not adequately isolate the contributions of information exposure, residual-based filtering, iterative feedback, and LLM reasoning, leaving the central claims insufficiently supported.

Questions for authors

Could the authors report performance curves across candidate budgets alongside actual runtime, token usage, and monetary cost for DRALoop and the baselines? Equal candidate counts alone do not establish a fair efficiency comparison.

1 comment
AuthorsOpenReview ↗

Author Response to CrXt: Clarification on Experimental Controls

We thank reviewer CrXt for the detailed feedback. We clarify our contributions and address the concerns below.

  1. Non-LLM baselines and the role of the LLM: Our primary goal is not to establish that LLMs outperform specialized rule-selection or counterfactual-search algorithms. We study how different representations of the same frozen tree model affect reasoning by a fixed LLM. Our controlled experiments show substantial differences across representations, sometimes exceeding 60 recall points, without changing either model. A non-LLM residual-driven optimizer would be useful for comparing end-to-end task performance, but would not isolate the effect of representation on LLM reasoning. Such comparisons are complementary to our study rather than necessary to establish its central finding.

  2. Computational budgets and efficiency: Our experiments control the number of candidate evaluations against the frozen predictor (B = 12), rather than total computational cost. We agree that performance curves across candidate-evaluation budgets, together with runtime and token measurements, would provide a useful complementary analysis of computational cost. We do not claim computational efficiency in terms of runtime, token usage, or monetary cost.

Appendix C.2 (Table 12) additionally examines three edit sparsity limits (K = 1, 2, 3), with the candidate-evaluation budget held fixed at B = 12. Adaptive representation updates yield paired gains of +10.0, +9.2, and +5.0 percentage points, respectively, over fixed-view iteration.

  1. Evidence for adaptive representation: We clarify that DRALoop does not dynamically select among the fixed representation types. Instead, it keeps the representation mechanism fixed while updating the information exposed from the frozen tree model based on unresolved cases. Our matched fixed-view iteration baseline controls for iteration alone and shows that residual-driven updates can provide additional gains. Validation-selected fixed views and rotation among representation types are useful additional controls for future investigation, while our existing fixed-view comparison already demonstrates benefits beyond iteration alone.
Reviewer_5x5jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an adaptive tree representation learning framework, DRALoop, to guide LLMs in rule generation and counterfactual reasoning.

Critical strengths
  1. The paper investigates the effect of intermediate tree representation, which is a meaningful direction. Experiments show that the proposed DRALoop can improve LLM performance in rule generation and counterfactual reasoning.
Critical weaknesses
  1. The rule generation task lacks comparisons with some relevant and potentially competitive baselines. In particular, OCTree, which is discussed in the related work, should be included in the evaluation at least.
  2. In the counterfactual experiments, the “schema_only” representation achieves 100% validity on both the OS and Adult datasets, but DRALoop achieves only 98.5% on the Adult dataset. These results undermine the contribution and necessity of the proposed method for this task.
  3. The proposed learning process fixes the tree representation, while only updating the residual examples to condition the representation. Thus, the term “adaptive representations” in the title may be somewhat ambiguous, as it could be understood to mean adaptive tree representations.
  4. The proposed iterative tree-based feedback method is closely related to previous work such as OCTree. The paper should more clearly justify its technical contribution in the paper.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Author Response to 5x5j: Distinction from Prior Work and Interpretation of Results

We thank reviewer 5x5j for recognizing the importance of intermediate tree representations and the improvements demonstrated by DRALoop. We address the concerns below.

1 & 4. Comparison with OCTree and technical contribution: OCTree uses decision-tree reasoning as feedback to iteratively generate new features and improve predictive performance. It trains models using the generated features during this process. Our setting is different: the trained tree model remains fixed, and we study how its representation affects LLM reasoning. Our contributions are (i) a controlled study isolating the effect of representation choice and (ii) DRALoop, which adapts the information exposed from the same frozen model based on unresolved cases. We do not claim iterative feedback itself as novel. While OCTree is relevant related work, its feature-generation objective and model-training loop make it a less direct baseline for our tasks.

  1. Schema-only performance: We agree that schema-only achieves strong results on some datasets. This is consistent with our finding that no representation is universally best, and more detailed tree information is not always beneficial. On datasets where schema-only reaches 100% validity, there is no headroom for improvement on that metric. We therefore do not claim that DRALoop outperforms every fixed representation on every dataset. Our contribution is to establish the effect of representation choice and show that adapting the exposed information can improve performance beyond fixed-view iteration. On counterfactual search, the strongest DRALoop configuration achieves 92.0% of attainable validity, compared with 81.6% for matched fixed-view iteration.

  2. Meaning of adaptive representations: We agree that the terminology could be clearer. We do not adapt the fitted tree model or its parameters. Instead, DRALoop adapts the representation presented to the LLM by selecting information relevant to the remaining unresolved cases. We have clarified this distinction in Section 5 and Appendix A of the revised manuscript.

Reviewer_7SLzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how different views of a frozen tree ensemble affect LLM rule generation and counterfactual search. It proposes DRALoop, which updates the information shown to the LLM based on previous proposals and feedback, and evaluates it on five tabular datasets with three language models.

Critical strengths

The question of combining LLMs and tabular data is timely and important. The use of tree models and questioning the representation aspects of them in this context is a good thing to focus on in general IMO.

I do believe that trees are useful for representing various aspects of tabular data for LLMs.

Critical weaknesses

Outdated setup and missing baselines

I find the setup substantially out of date relative to established agentic approaches. Works such as CodeAct, Data Interpreter, and AIDE already demonstrate agents that execute code and revise their actions using feedback. Here, the LLM receives predefined information views inside a restricted proposal loop. A current strong reasoning model with Python/tools to inspect the frozen ensemble and permitted data is an important missing baseline. Without it, the need for the proposed hand-designed adaptive interface remains unclear. A budgeted tree-aware non-LLM search baseline is also missing for counterfactuals.

Setup & Magnitude of improvement

The tasks considered seem easy enough and the diff from a more "agentic" iterative loop is small enough. I think we as a community should collectively focus on more tougher up to date challenges.

Writing

The writing is hard to comprehend despite careful reading. Simple tasks are described to densely, actual representations and the way they are constructed is not written up clearly, the results tables are packed with various metrics without clear emphasis on core results.

Questions for authors
  • How does DRALoop compare with a strong code-using agent given read-only access to the same ensemble and data?
  • Do the gains persist on harder settings where tree information is necessary?
  • What concrete principle about choosing tree representations should readers take away?
1 comment
AuthorsOpenReview ↗

Author Response to 7SLz: Clarifying Contribution and Scope

We thank reviewer 7SLz for recognizing the importance of studying how tree models are represented to LLMs. We would like to clarify our contribution and address the questions raised.

Central contribution: Our paper studies two questions: (1) How does the representation of a fixed tree model affect LLM reasoning? (2) Can adapting this representation during inference improve performance? We study the first through controlled experiments that hold the tree model, LLM, and evaluation pipeline fixed. We find large differences across representations, sometimes exceeding 60 recall points. For the second, we introduce DRALoop and compare it with matched fixed-view iteration to separate the benefits of representation adaptation from those of iteration alone.

Q1. Comparison with code-using agents: CodeAct, AIDE, and Data Interpreter provide general frameworks for code execution, search, and iterative refinement. These frameworks could also be used to construct and improve representations of a fixed tree model without changing its parameters. We agree that they offer interesting alternative ways to perform representation adaptation. However, these works do not directly study the effect of tree representations on LLM reasoning. Our goal is not to establish DRALoop as the best possible adaptation method, but to show that representation matters and that adapting it can improve reasoning beyond fixed-view iteration. Comparing different approaches to representation adaptation, including code-using agents, is a valuable direction for future work.

Q2. Harder settings: Our current evaluation covers two tasks, five datasets, and three LLMs, including budgeted counterfactual search. These experiments establish substantial representation effects and show that adaptive exposure can provide additional gains over fixed-view iteration. We do not claim that DRALoop is the strongest possible solver for these tasks. We agree that evaluating more challenging tasks and stronger adaptation alternatives is useful future work.

Q3. Principle for choosing representations: Our main finding is that no single representation works best across datasets and LLMs. This suggests that representation should depend on the task and the information needed at the current step, rather than being fixed in advance. DRALoop follows this principle by updating the information exposed from the tree based on unresolved cases. On counterfactual search, its strongest configuration reaches 92.0% of attainable validity, compared with 81.6% for matched fixed-view iteration.

Presentation: We appreciate the feedback on presentation and will work to improve the clarity of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.