- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to combine conflicting outputs from several classical discrete conditional-independence tests. It trains an LLM router using verifiable synthetic CI labels to predict a final independence/dependence decision from four test outputs and four hand-designed instance features, and then analyzes the router’s disagreement patterns to extract empirical applicability envelopes for the individual tests.
Critical strengths
(1) The paper studies a practically relevant problem. Classical CI tests can behave very differently across finite-sample regimes, and learning when particular tests become unreliable is potentially useful for causal discovery and other downstream applications.
(2) The applicability-envelope analysis leads to at least one concrete and useful diagnostic result. The learned MI boundary motivated an investigation of collapsed conditioning strata and revealed a degrees-of-freedom issue whose correction substantially improved CI accuracy.
Critical weaknesses
(1) The role of the 7B LLM in the proposed routing framework is not sufficiently justified. Although the router is described as reasoning over conflicting CI-test evidence, the RLVR objective supervises only the final independent/dependent decision; it does not explicitly train the model to select which CI test should be trusted, and the generated rationale is not part of the reward. The decision-relevant inputs are also largely structured numerical quantities, and it is unclear what additional information, pretrained knowledge, or reasoning capability of a 7B language model is expected to contribute beyond a standard tabular classifier. The empirical results reinforce this concern: RLVR achieves only a small average F1 gain over GBDT and random forest (0.725 vs. 0.717/0.716), and is slightly worse than GBDT on Sachs. Moreover, the LLM receives additional semantic prompt information beyond the twelve numerical inputs used by the tabular baselines, making the source of this small gain unclear despite substantially higher computational cost.
(2) The proposed applicability envelopes are inferred post hoc from the router’s binary decisions rather than learned directly. The model is trained only to predict independent/dependent, while “trust” in an individual CI test is subsequently inferred from whether the router overrules that test. The same envelope-extraction procedure is also applicable to non-LLM routers, with simpler models showing competitive or better boundary recovery on some metrics. This makes it unclear whether the main contribution is specifically an LLM-based applicability learner, or more generally a post-hoc analysis of ensemble decisions.
Questions for authors
(1) How were the four CI tests selected, and how sensitive is the method to the test library? Would the routing policy and learned applicability envelopes remain stable if tests were added, removed, or replaced?
(2) Why were conditioning order, evenness, sparsity, and degeneracy chosen as the four applicability features? Are they intended to sufficiently characterize CI-test difficulty, and how sensitive are the results to adding or removing features?
(3) What information does the LLM exploit beyond the twelve numerical inputs? Have the authors tested Qwen using only these numerical values, without method names, query identifiers, or natural-language formatting?
(4) Why not learn each CI test’s reliability directly from the instance features and ground-truth labels? Since the synthetic data provide verifiable CI labels, what does the intermediate routing model add over directly estimating test error probabilities or applicability boundaries?
(5) How stable are the learned applicability boundaries across data-generating regimes? In particular, how do they change with graph density, variable cardinality, CPT/Dirichlet concentration, and sample size? And can the authors provide broader real-world validation beyond Sachs? The current real-data and downstream causal-discovery evaluation is limited to only one benchmark.
6 comments
Part 2 of our response:
The envelopes are derived after the fact from binary decisions rather than learned, and the same extraction applied to a non-LLM router gives comparable results.
As shown above, LLM's rationale can be taken as a more straight forward "learned envelopes" stated in natural language. Meanwhile, Reading learned structure from a model’s behaviour is the usual practice such as in algorithm selection [1] and interpretability [2].
How were the four tests chosen, and are the conclusions stable when tests are added, removed or replaced?
The four tests are the defaults of pcalg, bnlearn and causal-learn. We also evaluate removing the three asymptotic tests. RLVR loses 0.006 and GBDT loses 0.073 (difference between the two +0.070 [0.047, 0.094]).
Why these four features, and how sensitive are the results to them?
Each feature corresponds to a known reason for a discrete CI test to fail. They are the coordinates in which the applicability boundaries are expressed. And as we shown in the paper and rebuttal, these features are sufficient to clearly delineate the differences between the various methods.
Why not estimate each test's reliability directly from features and ground-truth labels?
Direct estimation is how we obtain the reference boundaries of Table 2. It gives population-level boundaries and needs ground-truth labels. The router addresses the complementary task. On a new instance without a label it returns a decision, and its rationale names the tests it follows and the tests it sets aside. Table 2 and the rationale analysis above measure how far these instance-level decisions agree with the directly estimated boundaries.
How do the boundaries move with graph density, cardinality, Dirichlet concentration and sample size? Is there real data beyond Sachs?
Appendix F of the submission reports the varying of sample size and cardinality. Graph density and Dirichlet concentration act on a CI test only through the contingency table, which the four features describe. The synthetic graphs are sampled with varying edge density and the three expert networks have different conditional probability tables, so both factors vary in our data. On real data, we would like to argue that the evidence beyond synthetic DAGs also rests on the three expert networks (child, alarm, insurance). Their structures, cardinalities and conditional probability tables come from the application domain. The data are sampled from the networks, so every independence label is an exact d-separation statement. This is the property that the critique finds missing in Sachs, where the reference graph is itself uncertain. For evaluating a CI test, an exact label matters more than the provenance of the samples, because an error in the reference graph is counted as an error of the test. The three networks contribute 856 evaluation instances with real variable names.
[1] Smith-Miles, Kate, et al. "Towards objective measures of algorithm performance across instance space." Computers & Operations Research 45 (2014): 12-24.
[2] Craven, Mark, and Jude Shavlik. "Extracting tree-structured representations of trained networks." Advances in neural information processing systems 8 (1995).
Thank you for the response and additional experiments. After reading the rebuttal, some of my questions have been clarified, but I am still a bit confused about two points.
(1) For W1, I understand that the authors emphasize the usefulness of LLM-generated rationales. However, I am still not sure why we need to fine-tune a 7B LLM for this task. Since pretrained LLMs can already generate rationales, as also demonstrated by the Claude-generated rationales used for SFT, I wonder whether we could directly extract structured information from such rationales and use it to train a lightweight router. I think this would be a useful comparison to better understand the benefit of fine-tuning the entire 7B model. Also, for the changed test-library setting, wouldn't retraining a lightweight router be a relatively simple and inexpensive alternative?
(2) For W2, I agree that extracting learned structure from model behavior is common in interpretability research. However, at least from my understanding, the main task in this paper is to train a router that combines multiple CI tests to make a final decision, rather than using the LLM to systematically analyze the applicability of individual tests. The applicability envelopes seem to be a subsequent analysis of the router's behavior. Moreover, even for this analysis, the boundaries are characterized using only four pre-selected features, with limited consideration of other properties of the underlying distributions or dependence structures. I am therefore still unsure how much we can learn from these boundaries about the actual applicability of different CI tests, or whether they can provide more general insights beyond the selected features.
These are the two main points I remain uncertain about after reading the rebuttal. Thanks again for the additional clarifications.
We really appreciate reviewer's comments, and here we address each concern. Part 1 of our response:
We agree that the motivation of using LLM should be supported by experiments stronger, and we emphasize that its output rationale can increase the explainability.
We first add modern baselines UCI and wUCI [1] and take them into the input method library, then retrain the LLM:
As we can see from the tables, after including more method into library, the advantages of RLVR become larger compare to the most competitive baseline GBDT on both boundary direction and location accuracy.
We also tried to extract boundaries from the rationales. For every rationale, we record which of the tests it follows and which it sets aside. We then fit the change-point model of the paper to the set-aside rate of each test along each structural feature. We compare this with the extraction from verdicts used in Table 2. On the 2,368 disagreement instances of the unfiltered pool, the two extractions are complementary. The rationales show the boundary of χ² far more clearly than the verdicts do. The rate at which χ² is set aside rises from 0.62 to 0.91 once the order exceeds 5 (ΔBIC = −216), while its overrule rate rises from 0.08 to 0.16 (ΔBIC = −10). The verdicts show the boundary of MI more clearly, its overrule rate rises from 0.46 to 0.71 once sparsity exceeds 0.3 (ΔBIC = −157), against 0.40 to 0.53 in the rationales (ΔBIC = −29). We observe: