- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CausalDS, a synthetic benchmark for evaluating causal reasoning in LLM-based data science agents. It combines generated causal graphs, natural-language scenarios, and noisy tabular data to test causal identification, effect estimation, uncertainty quantification, abstention, and tool use. Experiments with 12 models across realistic and challenging exams show that models often recover causal structure correctly but still struggle with identifiability, calibrated uncertainty, and knowing when to abstain.
Critical strengths
Comprehensive benchmark design: CausalDS brings together causal reasoning, numerical estimation, uncertainty quantification, and abstention in a single framework, addressing an important gap in existing benchmarks. Its ability to generate fresh problems with known causal ground truth is particularly valuable for scalable evaluation.
Detailed experimental analysis: The evaluation covers 12 models, multiple difficulty settings, and repeated runs, with additional analyses of failure modes and robustness. The finding that models can identify causal structure correctly while still making errors in identifiability and uncertainty estimation is an important result.
Critical weaknesses
Limited real-world validation: Although the benchmark uses empirical distributions to guide generation, all evaluated scenarios remain synthetic, and the paper does not include independent human validation of their realism. This makes it difficult to determine how well benchmark performance reflects causal reasoning in actual data science applications.
Evaluation sensitivity: Some model rankings depend on the chosen scoring aggregation, and several task categories contain relatively few examples. Although the authors investigate these issues, they still limit the strength of conclusions about differences between closely ranked models.
Identifiability under noisy observations: The benchmark determines identifiability using the underlying conceptual causal graph, even when agents only receive noisy measurements and limited calibration data. This is a reasonable design choice, but its connection to what agents can reliably estimate from the available data deserves further justification.
Questions for authors
Could the authors provide additional evidence that the generated scenarios are realistic, for example through independent expert evaluation of the stories and causal relationships?
How reliably can agents recover identifiable causal quantities from the released noisy measurements and limited calibration data, particularly in the most difficult observation settings?
4 comments
Response to Reviewer 33xz (2/3)
- "Evaluation sensitivity":
-
"Some model rankings depend on the chosen scoring aggregation": The claim restates a disclosure from the paper, as it notes itself ("Although the authors investigate these issues") - App. A.17 (original submission) states: "the leaderboard extremes are stable under the cap and the pool weights but that the middle reshuffles if the interval term is summarized by a median instead of the capped mean". The review lists this analysis among the strengths ("additional analyses of failure modes and robustness"). We recompute the ranking under four alternative scoring rules (App. A.12, Tab. 36), and both main leaderboards print the component metrics and a second aggregate, the Composite Rank, next to the score (Tabs. 1 and 2). Group membership is stable: when the interval cap or the pool weights are changed, no model leaves its group of closely ranked models on either exam (App. A.12, Tab. 36). The one variant that moves models across groups is the interval score median on the realistic exam only, and the median is not a desirable rule. Sec. 3.8 states: "Averaging preserves propriety while a median would abandon it", hence letting the models get away with overconfidence. The large interval failures are a finding of the paper and separate the models (Sec. 4, App. A.11.4), and the review counts "errors in identifiability and uncertainty estimation" as part of "an important result", so a score should count them.
-
"several task categories contain relatively few examples": This applies to the realistic exam, which we release to the public and whose size we wanted to limit. The hard exam was built to enlarge precisely these pools: interval tasks go from 14 to 59, required abstentions from 16 to 56, and the binary pool from 34 to 97. App. A.12 states: "The hard exam is a different instrument with larger pools, and its intervals are correspondingly narrower". If the reviewer means detailed breakdowns in, e.g., App. A.10 and A.13, then this is a caveat from the paper restated: "Some realistic exam motif-by-family cells contain only one or two tasks, limiting comparisons at that level" (App. A.10). The caveat was applied to specific cells in the detailed breakdowns, which limited conclusions drawn from those cells. In other words, it was a limit on the granularity of the detailed additional analysis we can provide.
-
"they still limit the strength of conclusions about differences between closely ranked models": This perceived issue seem to stem from our own stability analysis (App. A.12). It is an additional analysis on similar exams that we ran for thoroughness, beyond the usual repetition of the same exam, so it helps to state what it varies. Each resampled exam is a similar but different exam at a high level of granularity: a task is replaced by another task of the same grading category, which can differ in everything else, namely its setting, causal graph and motif, mechanisms, data, task family and output variant, rung, and observation tier. We chose this stress test although it is the less favorable one for a ranking. The usual repetition of the same exam is reported as the run-to-run SDs in Tabs. 1 and 2. How consistently a model itself answers under repetition is a different matter, to a large extent a capability of the model (cf. Sec. 4, Fig. 4).
If the reviewer means App. A.12, we redraw the exams there in two ways. In the first, tasks are redrawn within the five grading categories, so that an exam keeps its mix of grading types. In the second, they are redrawn within the larger groups each exam was assembled from: seven groups of scenes by identifiability and graph structure on the hard exam, while the realistic exam was drawn without such groups, so its 100 tasks are redrawn as one group. In both cases the mix of grading types varies as well. We report "which orderings survive both" (Sec. 4), that is, the less favorable case, and the ranking is stable across the two: "no rank interval moves by more than one place at either end" (App. A.12).
In this analysis, we make no claims beyond what is stated. Models of the same generation are expected to rank close together on similar, but not identical exams, so overlapping ranks inside a group there are what a valid score should show. Where a difference is expected, between two generations of the same family, the newer model typically scores as well as or better (Sec. 6, Fig. 3).
-
(continued below)
Response to Reviewer 33xz (3/3)
In general, the size of an exam is a choice: "CausalDS is a generator rather than a fixed dataset" (App. A.12). The realistic exam exams is sized so that others can repeat it, and a larger or targeted exam can be generated at will with the released code. Published benchmarks in this area use comparable or smaller exams: DiscoveryBench (ICLR 2025) evaluates 239 real and 200 synthetic tasks, DSBench (ICLR 2025) 466 questions over 38 challenges with one run each, and MLE-bench (ICLR 2025) 75 tasks. Furthermore, each of our 300 tasks is a new scene with its own setting, graph, SCM, data, and story, and each exam is run (at least) three times, which is again not the norm.
Finally, these limitations are known because we measured and disclosed them: none of the benchmarks compared in Tab. 3 reports finite-exam rank uncertainty or its ranking under alternative scoring rules, including those published at top venues. We report both as additional analyses. They show how far the rankings carry over to similar exams and to other scoring rules, and the paper's conclusions do not depend on the order of closely ranked models.
- "Identifiability under noisy observations" (and the second question):
-
"its connection to what agents can reliably estimate from the available data deserves further justification": This justification is given in App. A.6, which Sec. 3.4 refers to. The agents are told the rule as well: every data-backed task states that "The observation layer affects solely the statistical estimation difficulty of the population quantity from provided data" (App. A.19.2, original submission). The connection holds by construction, in three steps that the paper states: (1) each measurement bundle has a single conceptual parent, so the observation layer "never flips an identifiability label" (Sec. 3.4); (2) the calibration rows identify each measurement model (the internal validation design of measurement-error analysis, Carroll et al., 2006, Sec. 2.3), since the conditional law of a measurement given its conceptual variable is the same in the calibration rows as in the population (App. A.6); (3) a Fisher-information screen rejects bundles with dead zones or low information over the realized support, so every released bundle is informative across its variable's range (App. A.6). In addition, the paper measures what the released data allows: for every scene, each measured variable is scored by how well it can be recovered from its measurements, including by a fit on the calibration rows alone (App. A.6). A target identified on the conceptual graph thus remains determined by the provided data, and the observation tier changes only how hard it is to estimate (App. A.6). Harder observation variants are there precisely to separate models' data-science skills while holding the rest fixed.
-
"How reliably can agents recover identifiable causal quantities from the released noisy measurements and limited calibration data, particularly in the most difficult observation settings?": As stated above, the hardest tiers are designed to separate the models according to their data science capability. Likewise, identifiability is guaranteed, as explained above.
To answer the question: the tiers do their job: they separate the models, and they do so reliably. Separation: on the hard exam, "median error grows five- to fifteen-fold from
proxytoproxy_severe, and under the severe tier the weakest models have medians many times those of the leaders" (App. A.10, Tab. 18). In each of the three families with two generations, the newer model has the lower median error at every tier (Tab. 18; at the severe tier: Astra 0.33 against GPT-5.5 1.61, Fable 0.21 against Opus 0.71, K3 0.33 against K2.6 0.48). Reliability: point estimates reproduce across restarts, with a median per-task SD of the normalized error below 0.03 for every model on the hard exam (App. A.11), and the ordering of the tiers holds within identical scenes (App. A.16, original submission).Finally, numeric tasks are scored by how close the estimate comes, so a harder tier still measures data-science skill.
-
References:
- Carroll, R. J., Ruppert, D., Stefanski, L. A., and Crainiceanu, C. M. (2006). Measurement Error in Nonlinear Models: A Modern Perspective, 2nd ed. Chapman and Hall/CRC. https://doi.org/10.1201/9781420010138.
Please let us know if you have further questions. If you feel we have addressed them, please consider raising your score.
We wish to thank the reviewer again for their time. As the discussion period closes on November 18 (AoE): if our response and the revised manuscript have addressed your concerns, please confirm and consider raising your score. We remain available for any questions.
Response to Reviewer 33xz (1/3)
We wish to thank the reviewer for their attention and the questions.
On the weaknesses and the questions:
"all evaluated scenarios remain synthetic": While we believe this isn't the crux of the stated weakness, we'd like to restate that the synthetic design is very much an either/or design choice: a benchmark is either synthetic or based on examples from existing data. This choice is stated as a strength by the review itself: "Its ability to generate fresh problems with known causal ground truth is particularly valuable for scalable evaluation." It provides a certified answer for every target, including whether it is identifiable; instances that cannot be in any training set (reducing memorization risk); and difficulty that is varied with the target held fixed. Besides the memorization issue, benchmarks built on existing data often have no certified answer and score against a reference analysis (App. A.2). As stated in the paper - App. A.17 (original submission): the benchmark measures competence "on problems with a certified answer, which is the price of having one".
"the paper does not include independent human validation of their realism": This is already freely disclosed in the paper - App. A.17 (original submission) states: "no human study has rated the stories or data for domain plausibility". First, we believe that among the synthetic benchmarks, CausalDS is likely the most realism-grounded one. Its composition is anchored in external sources: graph structures, variable types, question types, answer formats, and data observability draw on distributions estimated from real-study corpora (App. A.7), which the review grants ("uses empirical distributions to guide generation"); the mechanisms that generate the data include families derived from real corpora (Sec. 3.3, App. A.5); and the variable names and the causal relations they stand for are grounded in CauseNet, which is extracted from web and Wikipedia text (Sec. 3.2; App. A.4). What remains is the story text: first, it is constrained by the above variable mapping seeds, and is thoroughly audited for every scene: every edge and non-edge of the named graph is checked deterministically and by an LLM for real-world plausibility. A separate audit checks the story against the graph (App. A.3). The audit prompts and deterministic checker were written during intense human review of generated scenes against all failure modes we had encountered. In addition, we find no detectable dependence of the results on the particular story text: a model from a different family rewrote the variable mappings and stories of all realistic-exam scenes, and no model's results change beyond the finite-exam uncertainty (App. A.16, see the question below). Finally, human audits are not the norm among comparable benchmarks published at top venues - to name a few: Corr2Cause (ICLR 2024), CausalGame (ICML 2026), CausalProfiler (ICML 2026), and CauSim (NeurIPS 2026) all rest on generated scenarios or data, and none reports a human realism audit.
"This makes it difficult to determine how well benchmark performance reflects causal reasoning in actual data science applications": The realistic exam is composed like real causal analyses in its graph structures, variables, questions, and data access (App. A.7, previous point). No synthetic benchmark can certify transfer to a particular real study, and the claim restates the paper's disclosure of this - App. A.17 (original submission) states: "A model that performs well here may still fail on a real study". The other two reviews list this aspect as a strength: "I find that the setting of the paper applicable to real life causal questions" (qGDS) and "The benchmark closely reflects a complete causal analysis workflow" (9x5T).
"Could the authors provide additional evidence that the generated scenarios are realistic, for example through independent expert evaluation of the stories and causal relationships?": The revision adds two results (App. A.16). First, as noted above, we reran the LLM-based part of the realistic exam generation for all 100 realistic-exam scenes using a model from an unrelated family, and no solver's score, pass rate, or diagnostic loss changes beyond the finite-exam uncertainty. Second, the audits reject: in that run at least one candidate mapping or story was rejected in 84 of the 100 scenes before a version was accepted. They flagged implied links between non-adjacent variables, real-world implausible conditional independences, unrealistic types in the mapping, and missing variables and missing or contradictory causal links in the story. The paper further reports that twelve models from independent providers recover the intended graph from the story alone in 606 of 612 task-runs.
(continued below)