- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to represent pretrained tree models as text to help LLMs extract rules and identify input changes that alter model predictions. It proposes DRALoop, which iteratively updates these representations using failure feedback, and reports adaptive gains on some tasks.
Critical strengths
None
Critical weaknesses
Missing baselines and unclear LLM necessity. The paper lacks a non-LLM baseline combining residual-driven greedy rule selection with local rule search over the same rule pool. Such a comparison is necessary to determine whether the gains primarily come from external residual computation and rule ranking rather than LLM reasoning. Counterfactual search baselines with equivalent access to tree structures are also missing.
Candidate counts do not establish equal computational budgets. Evaluating 12 candidates per method does not ensure comparable costs. The amount of exposed paths and rules, precomputed statistics, LLM input/output tokens, and invocation latency may differ substantially. Efficiency claims therefore require actual runtime and cost measurements.
Insufficient evidence for adaptive representation updates. Different datasets benefiting from different representations does not establish that updating representations within the same task improves performance. Stronger controls include a validation-selected fixed representation and a rotation of fixed representations under matched budgets. The current experiments do not adequately isolate the contributions of information exposure, residual-based filtering, iterative feedback, and LLM reasoning, leaving the central claims insufficiently supported.
Questions for authors
Could the authors report performance curves across candidate budgets alongside actual runtime, token usage, and monetary cost for DRALoop and the baselines? Equal candidate counts alone do not establish a fair efficiency comparison.
Author Response to CrXt: Clarification on Experimental Controls
We thank reviewer CrXt for the detailed feedback. We clarify our contributions and address the concerns below.
Non-LLM baselines and the role of the LLM: Our primary goal is not to establish that LLMs outperform specialized rule-selection or counterfactual-search algorithms. We study how different representations of the same frozen tree model affect reasoning by a fixed LLM. Our controlled experiments show substantial differences across representations, sometimes exceeding 60 recall points, without changing either model. A non-LLM residual-driven optimizer would be useful for comparing end-to-end task performance, but would not isolate the effect of representation on LLM reasoning. Such comparisons are complementary to our study rather than necessary to establish its central finding.
Computational budgets and efficiency: Our experiments control the number of candidate evaluations against the frozen predictor (B = 12), rather than total computational cost. We agree that performance curves across candidate-evaluation budgets, together with runtime and token measurements, would provide a useful complementary analysis of computational cost. We do not claim computational efficiency in terms of runtime, token usage, or monetary cost.
Appendix C.2 (Table 12) additionally examines three edit sparsity limits (K = 1, 2, 3), with the candidate-evaluation budget held fixed at B = 12. Adaptive representation updates yield paired gains of +10.0, +9.2, and +5.0 percentage points, respectively, over fixed-view iteration.