acceptodds
Under review as a conference paper at ICLR 2027

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Abstract

Large language models (LLMs) increasingly act as *data-science agents*, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis *or* data analysis benchmarks without causal data-generating structure or a range of causal data science tasks. Existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce **CausalDS**, a benchmark for evaluating *causal reasoning in agentic data-science workflows*. Each benchmark instance is a *scene* consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a real domain. We ground the *composition* of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the memorization risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to imperfect observations. Abstention is tested directly: for targets the graph does not identify, the graded answer is to abstain. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding. Because the generator is fully synthetic, the exam is not a fixed dataset: composition, difficulty, and size are parameters, and fresh exams can be drawn at will.

Primary area:
datasets and benchmarks
Keywords:
causal reasoning, causal inference, data science, large language models, agents, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_33xzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CausalDS, a synthetic benchmark for evaluating causal reasoning in LLM-based data science agents. It combines generated causal graphs, natural-language scenarios, and noisy tabular data to test causal identification, effect estimation, uncertainty quantification, abstention, and tool use. Experiments with 12 models across realistic and challenging exams show that models often recover causal structure correctly but still struggle with identifiability, calibrated uncertainty, and knowing when to abstain.

Critical strengths

Comprehensive benchmark design: CausalDS brings together causal reasoning, numerical estimation, uncertainty quantification, and abstention in a single framework, addressing an important gap in existing benchmarks. Its ability to generate fresh problems with known causal ground truth is particularly valuable for scalable evaluation.

Detailed experimental analysis: The evaluation covers 12 models, multiple difficulty settings, and repeated runs, with additional analyses of failure modes and robustness. The finding that models can identify causal structure correctly while still making errors in identifiability and uncertainty estimation is an important result.

Critical weaknesses

Limited real-world validation: Although the benchmark uses empirical distributions to guide generation, all evaluated scenarios remain synthetic, and the paper does not include independent human validation of their realism. This makes it difficult to determine how well benchmark performance reflects causal reasoning in actual data science applications.

Evaluation sensitivity: Some model rankings depend on the chosen scoring aggregation, and several task categories contain relatively few examples. Although the authors investigate these issues, they still limit the strength of conclusions about differences between closely ranked models.

Identifiability under noisy observations: The benchmark determines identifiability using the underlying conceptual causal graph, even when agents only receive noisy measurements and limited calibration data. This is a reasonable design choice, but its connection to what agents can reliably estimate from the available data deserves further justification.

Questions for authors

Could the authors provide additional evidence that the generated scenarios are realistic, for example through independent expert evaluation of the stories and causal relationships?

How reliably can agents recover identifiable causal quantities from the released noisy measurements and limited calibration data, particularly in the most difficult observation settings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 33xz (1/3)

We wish to thank the reviewer for their attention and the questions.

On the weaknesses and the questions:

  1. "Limited real-world validation" (and the first question):
    • "all evaluated scenarios remain synthetic": While we believe this isn't the crux of the stated weakness, we'd like to restate that the synthetic design is very much an either/or design choice: a benchmark is either synthetic or based on examples from existing data. This choice is stated as a strength by the review itself: "Its ability to generate fresh problems with known causal ground truth is particularly valuable for scalable evaluation." It provides a certified answer for every target, including whether it is identifiable; instances that cannot be in any training set (reducing memorization risk); and difficulty that is varied with the target held fixed. Besides the memorization issue, benchmarks built on existing data often have no certified answer and score against a reference analysis (App. A.2). As stated in the paper - App. A.17 (original submission): the benchmark measures competence "on problems with a certified answer, which is the price of having one".

    • "the paper does not include independent human validation of their realism": This is already freely disclosed in the paper - App. A.17 (original submission) states: "no human study has rated the stories or data for domain plausibility". First, we believe that among the synthetic benchmarks, CausalDS is likely the most realism-grounded one. Its composition is anchored in external sources: graph structures, variable types, question types, answer formats, and data observability draw on distributions estimated from real-study corpora (App. A.7), which the review grants ("uses empirical distributions to guide generation"); the mechanisms that generate the data include families derived from real corpora (Sec. 3.3, App. A.5); and the variable names and the causal relations they stand for are grounded in CauseNet, which is extracted from web and Wikipedia text (Sec. 3.2; App. A.4). What remains is the story text: first, it is constrained by the above variable mapping seeds, and is thoroughly audited for every scene: every edge and non-edge of the named graph is checked deterministically and by an LLM for real-world plausibility. A separate audit checks the story against the graph (App. A.3). The audit prompts and deterministic checker were written during intense human review of generated scenes against all failure modes we had encountered. In addition, we find no detectable dependence of the results on the particular story text: a model from a different family rewrote the variable mappings and stories of all realistic-exam scenes, and no model's results change beyond the finite-exam uncertainty (App. A.16, see the question below). Finally, human audits are not the norm among comparable benchmarks published at top venues - to name a few: Corr2Cause (ICLR 2024), CausalGame (ICML 2026), CausalProfiler (ICML 2026), and CauSim (NeurIPS 2026) all rest on generated scenarios or data, and none reports a human realism audit.

    • "This makes it difficult to determine how well benchmark performance reflects causal reasoning in actual data science applications": The realistic exam is composed like real causal analyses in its graph structures, variables, questions, and data access (App. A.7, previous point). No synthetic benchmark can certify transfer to a particular real study, and the claim restates the paper's disclosure of this - App. A.17 (original submission) states: "A model that performs well here may still fail on a real study". The other two reviews list this aspect as a strength: "I find that the setting of the paper applicable to real life causal questions" (qGDS) and "The benchmark closely reflects a complete causal analysis workflow" (9x5T).

    • "Could the authors provide additional evidence that the generated scenarios are realistic, for example through independent expert evaluation of the stories and causal relationships?": The revision adds two results (App. A.16). First, as noted above, we reran the LLM-based part of the realistic exam generation for all 100 realistic-exam scenes using a model from an unrelated family, and no solver's score, pass rate, or diagnostic loss changes beyond the finite-exam uncertainty. Second, the audits reject: in that run at least one candidate mapping or story was rejected in 84 of the 100 scenes before a version was accepted. They flagged implied links between non-adjacent variables, real-world implausible conditional independences, unrealistic types in the mapping, and missing variables and missing or contradictory causal links in the story. The paper further reports that twelve models from independent providers recover the intended graph from the story alone in 606 of 612 task-runs.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 33xz (2/3)

  1. "Evaluation sensitivity":
    • "Some model rankings depend on the chosen scoring aggregation": The claim restates a disclosure from the paper, as it notes itself ("Although the authors investigate these issues") - App. A.17 (original submission) states: "the leaderboard extremes are stable under the cap and the pool weights but that the middle reshuffles if the interval term is summarized by a median instead of the capped mean". The review lists this analysis among the strengths ("additional analyses of failure modes and robustness"). We recompute the ranking under four alternative scoring rules (App. A.12, Tab. 36), and both main leaderboards print the component metrics and a second aggregate, the Composite Rank, next to the score (Tabs. 1 and 2). Group membership is stable: when the interval cap or the pool weights are changed, no model leaves its group of closely ranked models on either exam (App. A.12, Tab. 36). The one variant that moves models across groups is the interval score median on the realistic exam only, and the median is not a desirable rule. Sec. 3.8 states: "Averaging preserves propriety while a median would abandon it", hence letting the models get away with overconfidence. The large interval failures are a finding of the paper and separate the models (Sec. 4, App. A.11.4), and the review counts "errors in identifiability and uncertainty estimation" as part of "an important result", so a score should count them.

    • "several task categories contain relatively few examples": This applies to the realistic exam, which we release to the public and whose size we wanted to limit. The hard exam was built to enlarge precisely these pools: interval tasks go from 14 to 59, required abstentions from 16 to 56, and the binary pool from 34 to 97. App. A.12 states: "The hard exam is a different instrument with larger pools, and its intervals are correspondingly narrower". If the reviewer means detailed breakdowns in, e.g., App. A.10 and A.13, then this is a caveat from the paper restated: "Some realistic exam motif-by-family cells contain only one or two tasks, limiting comparisons at that level" (App. A.10). The caveat was applied to specific cells in the detailed breakdowns, which limited conclusions drawn from those cells. In other words, it was a limit on the granularity of the detailed additional analysis we can provide.

    • "they still limit the strength of conclusions about differences between closely ranked models": This perceived issue seem to stem from our own stability analysis (App. A.12). It is an additional analysis on similar exams that we ran for thoroughness, beyond the usual repetition of the same exam, so it helps to state what it varies. Each resampled exam is a similar but different exam at a high level of granularity: a task is replaced by another task of the same grading category, which can differ in everything else, namely its setting, causal graph and motif, mechanisms, data, task family and output variant, rung, and observation tier. We chose this stress test although it is the less favorable one for a ranking. The usual repetition of the same exam is reported as the run-to-run SDs in Tabs. 1 and 2. How consistently a model itself answers under repetition is a different matter, to a large extent a capability of the model (cf. Sec. 4, Fig. 4).

      If the reviewer means App. A.12, we redraw the exams there in two ways. In the first, tasks are redrawn within the five grading categories, so that an exam keeps its mix of grading types. In the second, they are redrawn within the larger groups each exam was assembled from: seven groups of scenes by identifiability and graph structure on the hard exam, while the realistic exam was drawn without such groups, so its 100 tasks are redrawn as one group. In both cases the mix of grading types varies as well. We report "which orderings survive both" (Sec. 4), that is, the less favorable case, and the ranking is stable across the two: "no rank interval moves by more than one place at either end" (App. A.12).

      In this analysis, we make no claims beyond what is stated. Models of the same generation are expected to rank close together on similar, but not identical exams, so overlapping ranks inside a group there are what a valid score should show. Where a difference is expected, between two generations of the same family, the newer model typically scores as well as or better (Sec. 6, Fig. 3).

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 33xz (3/3)

In general, the size of an exam is a choice: "CausalDS is a generator rather than a fixed dataset" (App. A.12). The realistic exam exams is sized so that others can repeat it, and a larger or targeted exam can be generated at will with the released code. Published benchmarks in this area use comparable or smaller exams: DiscoveryBench (ICLR 2025) evaluates 239 real and 200 synthetic tasks, DSBench (ICLR 2025) 466 questions over 38 challenges with one run each, and MLE-bench (ICLR 2025) 75 tasks. Furthermore, each of our 300 tasks is a new scene with its own setting, graph, SCM, data, and story, and each exam is run (at least) three times, which is again not the norm.

Finally, these limitations are known because we measured and disclosed them: none of the benchmarks compared in Tab. 3 reports finite-exam rank uncertainty or its ranking under alternative scoring rules, including those published at top venues. We report both as additional analyses. They show how far the rankings carry over to similar exams and to other scoring rules, and the paper's conclusions do not depend on the order of closely ranked models.

  1. "Identifiability under noisy observations" (and the second question):
    • "its connection to what agents can reliably estimate from the available data deserves further justification": This justification is given in App. A.6, which Sec. 3.4 refers to. The agents are told the rule as well: every data-backed task states that "The observation layer affects solely the statistical estimation difficulty of the population quantity from provided data" (App. A.19.2, original submission). The connection holds by construction, in three steps that the paper states: (1) each measurement bundle has a single conceptual parent, so the observation layer "never flips an identifiability label" (Sec. 3.4); (2) the calibration rows identify each measurement model (the internal validation design of measurement-error analysis, Carroll et al., 2006, Sec. 2.3), since the conditional law of a measurement given its conceptual variable is the same in the calibration rows as in the population (App. A.6); (3) a Fisher-information screen rejects bundles with dead zones or low information over the realized support, so every released bundle is informative across its variable's range (App. A.6). In addition, the paper measures what the released data allows: for every scene, each measured variable is scored by how well it can be recovered from its measurements, including by a fit on the calibration rows alone (App. A.6). A target identified on the conceptual graph thus remains determined by the provided data, and the observation tier changes only how hard it is to estimate (App. A.6). Harder observation variants are there precisely to separate models' data-science skills while holding the rest fixed.

    • "How reliably can agents recover identifiable causal quantities from the released noisy measurements and limited calibration data, particularly in the most difficult observation settings?": As stated above, the hardest tiers are designed to separate the models according to their data science capability. Likewise, identifiability is guaranteed, as explained above.

      To answer the question: the tiers do their job: they separate the models, and they do so reliably. Separation: on the hard exam, "median error grows five- to fifteen-fold from proxy to proxy_severe, and under the severe tier the weakest models have medians many times those of the leaders" (App. A.10, Tab. 18). In each of the three families with two generations, the newer model has the lower median error at every tier (Tab. 18; at the severe tier: Astra 0.33 against GPT-5.5 1.61, Fable 0.21 against Opus 0.71, K3 0.33 against K2.6 0.48). Reliability: point estimates reproduce across restarts, with a median per-task SD of the normalized error below 0.03 for every model on the hard exam (App. A.11), and the ordering of the tiers holds within identical scenes (App. A.16, original submission).

      Finally, numeric tasks are scored by how close the estimate comes, so a harder tier still measures data-science skill.

References:

  • Carroll, R. J., Ruppert, D., Stefanski, L. A., and Crainiceanu, C. M. (2006). Measurement Error in Nonlinear Models: A Modern Perspective, 2nd ed. Chapman and Hall/CRC. https://doi.org/10.1201/9781420010138.

Please let us know if you have further questions. If you feel we have addressed them, please consider raising your score.

We wish to thank the reviewer again for their time. As the discussion period closes on November 18 (AoE): if our response and the revised manuscript have addressed your concerns, please confirm and consider raising your score. We remain available for any questions.

Reviewer_9x5TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CausalDS, a synthetic benchmark for evaluating causal reasoning in data-science agents. Each scene combines a hidden structural causal model, an audited natural-language description, tabular data, and a list of tasks. A separate observation layer introduces noisy measurements to vary estimation difficulty. The benchmark evaluates causal identification, numerical estimation, uncertainty quantification, abstention, and tool use on twelve agents.

Critical strengths
  1. The benchmark closely reflects a complete causal analysis workflow, combining problem interpretation, identification, data analysis, and estimation or abstention.

  2. The generation pipeline is carefully designed, providing controllable task configurations and explicit ground truth through SCMs and configurable measurement noise.

Critical weaknesses
  1. The benchmark primarily integrates existing benchmarks, without clearly establishing a new capability that this combination measures.

  2. The designed metrics are too complex, making overall scores and model comparisons difficult to interpret.

  3. The results largely confirm familiar limitations, such as overconfidence and sensitivity to noisy measurements, and offer limited new insight for future LLM causal reasoning research.

  4. Some writing reads very AI (e.g., the discussion section) and does not have a clear focus.

Questions for authors

N/A

8 comments
AuthorsOpenReview ↗

Response to Reviewer 9x5T (1/3)

We wish to thank the reviewer for their attention. We have uploaded the revised manuscript - please refer to the global comment for the list of changes.

On the weaknesses:

  1. "The benchmark primarily integrates existing benchmarks, without clearly establishing a new capability that this combination measures.":
    • "primarily integrates existing benchmarks": First, we believe that this is a misreading of the related work table. We performed a very wide literature search and were generous about giving full marks to any work that had comparable aspects, even if in a significantly different setting. We consider first-class non-identifiability, scored abstention, free-form verbalization, testing statistical uncertainty estimation, and a separate observation layer central contributions. On these, there is at most one related work with comparable features, and none has more than one:

      • none has fully integrated non-identifiable cases
      • on scored abstention, the only work that we could find is CausalT5k, which asks whether a verbal causal claim in a short vignette is warranted. Refusal there means declining to endorse the claim. It involves no data, no estimand, and no estimation.
      • on statistical uncertainty, CausalReasoningBenchmark uses real datasets from published studies and is limited to standard errors and their implied Wald intervals, scored against a reference analysis and not the ground truth.
      • on the observation layer: CausalGame is a drone design game which adds observation noise in only one of its 14 game scenarios. No causal quantity is estimated.

      Second, as App. A.2 states: "no other benchmark integrates...". It is the integration (or better, joint evaluation) that is crucial, as explained in the next point, and a non-trivial problem, as the components need to work together seamlessly.

    • "without clearly establishing a new capability that this combination measures": The capability is complete causal analysis, which the review itself lists as a strength: "The benchmark closely reflects a complete causal analysis workflow..." This is also stated in the abstract: "jointly evaluates symbolic causal reasoning, data science,...", and the Introduction: "five axes that are usually tested separately". Joint evaluation is needed: besides common sense - nowadays, agents run whole analyses unsupervised - this is also well-backed by the paper's results. For example: on the hard exam GPT-5.5 has the second-highest Pass Rate on the symbolic tasks (87.2%) and the eighth-highest on the data-backed ones (64.2%). The two exams have different leaders for the same reason: GPT-6 Astra leads the realistic exam through its identification and abstention calls and is fourth on the hard exam "despite the best Pass Rate" (Sec. 4), behind the models that estimate best under severe measurement. Benchmarks that test parts of the pipeline cannot estimate actual performance since parts depend on each other.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 9x5T (2/3)

  1. "The designed metrics are too complex, making overall scores and model comparisons difficult to interpret.":

    • "The designed metrics are too complex": As noted, joint evaluation is necessary. Thus five axes cannot be collapsed into one number without choices, and we caveat the aggregate metric explicitly - Sec. 3.8 states: "While a single-number metric can be attractive as a rough orientation, it is not the final word."

      That said, the aggregate metric is built from first principles: the basic metrics are standard (exact match, F1F_1, a normalized error for point estimates, and the interval score, a strictly proper scoring rule (Gneiting & Raftery, 2007), for intervals), and each part is aggregated as its metric requires (a median for the long-tailed point errors, and a capped mean for the interval scores, since averaging preserves propriety where a median would abandon it; Sec. 3.8, App. A.9).

      Alternatives are extensively tested: App. A.12 recomputes the ranking under four alternative scoring rules (Tab. 36), and "Under the interval-cap and equal-weight variants no rank moves more than two places and the extremes are fixed". None of the benchmarks compared in Tab. 3 reports its ranking under alternative scoring rules, including those published at top venues. The aggregate score is then composed of these parts, which are weighted by task count so that targeted exam composition is reflected by the metric.
      Given the extra page, we have revised the scoring section (Sec. 3.8) for greater clarity.

    • "making overall scores and model comparisons difficult to interpret": The main tables already also report simple metrics, as is noted: "For this reason, in Sec. 4 we report both the intermediate [...] and atomic metrics, whenever appropriate." (Sec. 3.8)

      The score is nonetheless informative. It is argued from first principles stated above, and nothing in it was tuned to a particular model or model generation. Since labs work to make each generation outperform the previous one across the board, one would expect the score to reflect this: in the plane of score against log tokens (Fig. 3), within each of the three families with two generations (GPT, Claude, Kimi) the newer model typically scores as well as or better than its predecessor, with always fewer calls and almost always fewer tokens. Furthermore, "a closed–open separation persists for older generations, too (e.g., Opus 4.8 and Gemini 3.1 vs Kimi K2.6 and Qwen 3.6)" (Sec. 4).

    As stated in the paper, the aggregate serves as rough orientation. The per-axis components reveal the mechanisms (next point).

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 9x5T (3/3)

  1. "The results largely confirm familiar limitations, such as overconfidence and sensitivity to noisy measurements, and offer limited new insight for future LLM causal reasoning research.":

    • "overconfidence": Where a finding coincides with prior work, the paper says so itself - App. A.11.4 states: "The pattern is consistent with recent uncertainty-elicitation benchmarks, in which nominal intervals from frontier models cover far below their nominal level". The claim thus restates what the paper states itself, and agreement with independent work supports the benchmark's validity. Beyond that, the paper reports specific mechanisms of failure:

      • intervals are too narrow around a correct estimate (Tab. 32).
      • over-claimed identification, tied to instruments: 502 of the 796 false-identifiability grades are on an IV motif, and none on a front-door one (App. A.11.1).
      • inconsistent abstention: on the hard exam's 22 non-identifiable effect targets, any-of-3 exceeds all-of-3 by 9 to 55 points, so a model recognizes the same non-identifiable target inconsistently (App. A.13).

      The models differ widely on these mechanisms, and the benchmark goes significantly beyond a "general limitation of LLMs"; to the best of our knowledge, no other benchmark has investigated this in such depth, which is afforded by the joint-evaluation construction.

    • "sensitivity to noisy measurements": A general sensitivity would mean a degradation across the board. Measurement difficulty is the benchmark's designed data-science-difficulty manipulation (Sec. 3.4), and the paper reports that the models respond to it very differently:

      • the tiers separate the models by generation: in each family with two generations, the newer model has the lower error at every tier of the hard exam, and the gap grows with the difficulty (Tab. 18).
      • errors at the severe tier split the field into groups (App. A.10, Tab. 18).
      • the typical failure is the interval: coverage falls with the tier while most misses stay centered on an acceptable estimate, and the models that still cover are the ones that widened their intervals (App. A.11.4).
      • the estimator does not explain the failures: failing attempts read the calibration data and use the same estimators and interval recipes as passing ones (App. A.11.3).
      • interaction style matters: the near-one-shot closed models sit with the weakest open models at the bottom of the severe-tier error (App. A.10).

      The "sensitivity" is thus specific to a model and a tier. GPT-5.5 is the clearest instance: its error is close to that of its peers at the easiest proxy tier and the largest of all models at the next one (Tab. 18). This is precisely the separator - data science competency - the observation model is targeting, and places it near the bottom of both leaderboards (Sec. 4).

    • "limited new insight for future LLM causal reasoning research": The paper reports concrete, model-specific findings on each of its five axes (App. A.11, Tab. 26), which give researchers specific targets; some examples on our axes:

      • symbolic reasoning: models read the graph and misjudge what it licenses. The most common error is a claim of identification that the graph does not support, tied to instruments and in most cases already present in the model's first plan (Sec. 4, App. A.11.1).
      • data-science execution: estimation under hard measurement separates the models (see above, Tab. 18).
      • uncertainty: intervals are too narrow around a correct estimate (see above, App. A.11.4).
      • abstention: the same non-identifiable target is recognized on some runs only (see above, App. A.13).

    In general, it seems that this weakness takes a per-axis view. These findings combine differently across models, which only the joint evaluation under different settings shows - Sec. 6 states: "The realistic exam is led by the model that makes the identification and abstention calls best ... while the hard exam by the models that estimate best under severe measurement ...". The revised Discussion (see below) states where the failure sits on each axis, and which axes remain open for every model.

  2. "Some writing reads very AI (e.g., the discussion section) and does not have a clear focus.": With the extra space afforded to us, we have revised the Discussion section so that it has a clearer structure.

Please let us know if there are further concerns. If you feel we have addressed them, please consider raising your score.

Reviewer_qGDSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors proposed a Causal Benchmark paper titled CAUSALDS. The paper contribution includes (1) an end-to-end pipeline to generate synthetic causal background + data + question + ground truth answer. (2) an unified score for aggregating multiple tasks from different directions. and (3) comprehensive evaluation of recent advance LLMs across multiple model generation / families.

Critical strengths
  1. Settings are real-life applicable I find that the setting of the paper applicable to real life causal questions so the benchmark contributes toward joint effort for enhancing LLMs' causal capability. Rich motif catalog for DAG in Appendix A.1 and SCM profile registry makes the benchmark covers a wide categories of distribution, under Pearl's Causal framework. A model with good performance on this benchmark would have certain causal graph rebuilding + tool calling + result explanation capabilities for the real life data.

  2. Experiment and Presentation The experiment section and the writing overall is clear and easy to follow. The differences for this paper and reference works are clear and meaningful. The analyses/leaderboard on realistic exam and hard exam also provide directional evidence for LLMs' Causal capability in general.

Critical weaknesses
  1. Score is Uneven by Design The proposed evaluation score follow pool size (pool with the most tasks dominates), wrong answer from different category got different penalties (missing hard numeric tasks is not penalized equally compared with binary and F1), also medians and capped mean are chosen for robustness (so large interval miss affect more than large point errors)

  2. Fully Synthetic Benchmark can has LLM confounding effects. The setup of the paper: LLM generating scene, variable, and ground truth sounds familiar to me and this works sounds like a good implementation. The narration, variable mapping, and audit is only picked as K2.5 (with K2.6 for regen). This could leads to the text and the generated variable polls bias towards "KIMI style", Rankings show no obvious Kimi advantage (K2.6 is 9th realistic / 5th hard, K3 is 6th / 7th), but no cross-generator control is reported.

Questions for authors

I do not have questions for authors. Instead I will read questions and comments from other reviewers and in this section I am providing some additional references that might be helpful and some writing suggestions. Some of the following texts are generated by LLM (and proofread by me)

Notation and terminology suggestions

  1. Better organize the Score section (Page 5) where I spent most time understanding and need pen/paper even LLM's help.
  2. Define NRelIS in the main text (§3.8), not only in the appendix. Eq. 3 uses it, along with the cap c = 10, with no justification given in the main text.
  3. Fix small notation inconsistencies. F1 appears as both F_{1,i} and "F1-Loss", and "quality categories" in the text corresponds to "pools" in the equations.

Reference suggestions (same category: causal / agent benchmarks

  1. Corr2Cause (Jin et al., 2024, "Can large language models infer causation from correlation?"). Standard static causal-inference benchmark alongside CLadder; supports CausalDS's "causal parrot" argument.
  2. ScienceAgentBench (Chen et al., 2025). Executable, data-driven scientific-discovery agent benchmark, relevant to the tool-use axis. CausalDS cites DiscoveryBench and BLADE but not this one.
  3. BoxingGym (Gandhi et al., 2025). Interactive experimental design; CausalDS's related work is thin on this side. DiscoveryWorld (Jansen et al., 2024) is an alternative for the same gap.
4 comments
AuthorsOpenReview ↗

Response to the review, 1/2

We wish to thank the reviewer for their attention and the suggestions. We have implemented the paper suggestions and included the suggested references in the revised manuscript, which we have uploaded - please refer to the global comment for the list of changes.

On the weaknesses:

  1. "Score is Uneven by Design":

    • "pool with the most tasks dominates": The task-weighted version was chosen as the default by design, as we design the exams either for realism (the "realistic"), or to target specific aspects (the "hard") along the five axes. Giving every pool the same weight would remove the purpose behind targeted exam compositions. We had already provided the equal-weight version in App. A.12, it changes no rank by more than 1-2. The claim thus restates a disclosure from the paper - Sec. 3.8 states: the score "is dominated, by construction, by whichever pool is largest unless re-weighted". For increased clarity, the scoring section now states the score as a count-weighted sum of four components.

    • "wrong answer from different category got different penalties": There are only two groups for wrong answers: bounded and unbounded losses. Everything that can be, is reduced to the first category. A wrong abstention on a numeric task is graded as a failure in the Pass Rate for this precise reason. The second category is scored by the size of the error on purpose as the magnitude of the error is very relevant information in assessing the competencies along the data-science axis, as argued in the fourth point below.

    • "missing hard numeric tasks is not penalized equally compared with binary and F1": For missing numeric answers (different case than wrong): in the unbounded case, one is left with two choices: either report them, or assign an arbitrary loss. We believe the former is more scientifically responsible, and disclose this extensively: the Valid cont. column of both leaderboards reports them in color, and Sec. 3.8 states this caveat (lines 263–266 of the original submission), so this is again already argued for and disclosed in the paper.

    • "medians and capped mean are chosen for robustness (so large interval miss affect more than large point errors)": How much "large interval miss affect more than large point errors" is set by the cap. In Tab. 36, we show that when varying the cap from c=10c=10 to c=5c=5 or c=20c=20, no model moves more than one rank on the realistic exam or two on the hard exam. Furthermore, the claim from the suggestion that "Eq. 3 uses it, along with the cap c = 10, with no justification given in the main text." is not correct: the submitted main text gives the justification directly below Eq. 3 (Sec. 3.8): "with c=10c = 10, using a median for robustness to extreme point errors and a capped mean to retain interval-miss penalties while limiting extreme contributions (App. A.9)". The revised section expands it and defines NRelIS alongside.
      CausalDSScore is designed bottom-up component-by-component, as stated above. Each pool is aggregated as its base metric requires: the interval score is proper and averaging preserves propriety while a median would abandon it, so interval tasks are the one exception to the medians and are averaged. The cap is chosen to keep a catastrophic miss from dominating (App. A.9).
      The large interval misses are a crucial finding of the paper: 72.5% of the uncovered intervals are centered on an accurate estimate and are too narrow (Tab. 32). In short, interval misses count by their size by design, and we argue that the chosen metric is well-founded on theoretical and empirical grounds.

    Furthermore, we explicitly note that CausalDSScore is just one metric, include others in the main tables (and more in the appendix), and provide extensive sensitivity analyses. Under all five scoring rules, Astra is first and Gemma last on the realistic exam, and on the hard exam GLM, Fable, DeepSeek, and Astra are the top four and Gemma, GPT-5.5, and Qwen the bottom three. None of the benchmarks compared in Tab. 3 reports its ranking under alternative scoring rules, including those published at top venues.

AuthorsOpenReview ↗

Response to the review, 2/2

  1. "Fully Synthetic Benchmark can has LLM confounding effects."

    • "LLM generating scene, variable, and ground truth": We would like to correct a possible misreading. The LLM generates only the variable mapping and the scene story. The ground truth is computed from the sampled SCM with no LLM involved. Furthermore, the variable mapping is grounded from CauseNet, so the variable names are not left to the LLM alone, which in turn restricts it in the story.

    • "no cross-generator control is reported": Cross-generator stability is now reported below, also included in the revised paper, App. A.16. A model from a different family (DeepSeek V4.1 Flash) was used to generate the verbal part of the 100 realistic-exam scenes. The exam was then run on GPT-6 Astra, GLM 5.3, and Qwen 3.6 35B:

SolverCausalDSScore change (↓\downarrow)Pass Rate change, % (↑\uparrow)Diagnostic-loss change (↓\downarrow)
GPT-6 Astra+0.083 [-0.019, 0.236]+3.9 [0.0, 9.8]-0.003 [-0.035, 0.028]
GLM 5.3+0.004 [-0.047, 0.055]+1.5 [-10.7, 13.4]-0.007 [-0.052, 0.039]
Qwen 3.6 35B+0.087 [-0.051, 0.258]+7.8 [-9.0, 24.7]-0.019 [-0.086, 0.045]

Change on the regenerated stories relative to the original ones, with 95% paired bootstrap intervals; arrows give the better direction. The diagnostic loss is the [0,1][0,1] per-task loss of our two ablations (App. A.15, App. A.17), in which a missing or failed answer counts as 1.

No solver's score, pass rate, or diagnostic loss changes beyond the finite-exam uncertainty (every interval above brackets zero), and the three solvers' changes do not differ from one another. Pass rates are equal or higher on the regenerated stories, and the diagnostic loss moves by under 0.02.

Please let us know if any concerns remain. If our responses and revisions have addressed them, please consider raising your score.

Thank you author for the explanation and the additional experiment. The revised manuscript has improved quality and I revised my scores accordingly. However my point on CausalDSScore design has some limitations still exist and my position is borderline lean accept (4-scale does not allow me to do so).

We wish to thank the reviewer for being willing to re-examine the paper and update the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and summary of changes

We thank the reviewers for their time and attention. We have uploaded a revised manuscript and replied to each review separately. For reference, we would like to clarify four aspects of the paper that our replies rely on:

1. What CausalDS measures. CausalDS evaluates a complete causal analysis, and this joint evaluation is the main focus. The parts of an analysis depend on each other, so benchmarks that test them separately cannot tell how a model performs on the whole.

2. Difficulty is by design. The observation tiers exist to separate the models by data-science skill, while the target and its identifiability stay fixed. That the models respond to them very differently is the result the benchmark is built to show.

3. What the LLM generates. An LLM writes only the variable names and the story of a scene. The answers are computed from the SCM. The verbal part is constrained by real-world causal relations from CauseNet and a deterministic checker, and goes through several stages of audits.

4. The score. Five axes cannot be collapsed into one number without choices. Given that, CausalDSScore is built from the ground up, using standard metrics as components and with an argued reason for each choice. The leaderboards also report the simpler metrics next to it, and most of the findings do not rely on CausalDSScore at all.

New results in the revision. A model from a different family regenerated the variable names and stories of all 100 realistic-exam scenes, and three models took the regenerated exam. No model's score, pass rate, or diagnostic loss changes beyond the finite-exam uncertainty.

Changes in the revised PDF

  1. Scoring section (Sec. 3.8), rewritten. The score is stated as a count-weighted sum of four components, with the reason for each choice next to the equations.
  2. New App. A.16, robustness to the story generator. See "New results" above
  3. Discussion (Sec. 6). Reorganized and expanded.
  4. Stability analysis (App. A.12), clarified. The bootstrap resamples within rough categories, so the unit of comparison are similar exams. For completeness, the appendix also reports both exams under the bounded score of App. A.15.
  5. Smaller changes. The joint evaluation is named in the contribution list (Sec. 1); Corr2Cause, ScienceAgentBench, and BoxingGym are cited.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.