- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SLEUTHBENCH, a benchmark for evaluating the statistical discovery abilities of LLM agents equipped with Python-based tools. The benchmark injects controlled data-quality problems and feature effects into public tabular datasets, allowing reference answers to be computed automatically while preserving the background structure of the original tables. It includes 17 question templates across two families, namely data-quality questions and feature-contribution questions. The authors evaluate six state-of-the-art LLMs on 70 validated dataset-template combinations, producing 1,680 graded responses in total. The models achieve relatively strong performance on data-quality detection, with an accuracy of 83.8%, but perform substantially worse on feature-contribution discovery, with an accuracy of 41.9%. The paper further proposes an Empirical Layer containing precomputed statistical artifacts, including summaries, fitted feature and interaction effects, and dataset descriptions. Providing these artifacts improves feature-contribution accuracy to 68.0%. The main contribution is a benchmark with controllable hidden signals and automatically generated ground truth, together with an empirical analysis showing a substantial gap between data-quality detection and feature-contribution discovery.
Critical strengths
-
The paper presents a clear and practically relevant evaluation framework. By injecting controlled statistical phenomena into public tabular datasets, the authors can generate reference answers automatically and reduce the influence of memorization. This directly addresses a central challenge in evaluating statistical reasoning by LLM agents.
-
The analysis goes beyond reporting a single aggregate score. It separates data-quality detection from feature-contribution discovery and evaluates the effect of the Empirical Layer. The observed performance gap between the two task families, together with the improvement from exposing statistical artifacts, provides useful insight into the specific bottlenecks of LLM agents in statistical reasoning.
Critical weaknesses
-
The ecological validity and coverage of the benchmark remain insufficiently established. Since the target phenomena are injected by the authors, models may be adapting to a limited set of injection mechanisms rather than solving genuinely open-ended statistical discovery problems. The paper should more carefully connect the injected patterns to phenomena encountered in real data analysis and discuss which real-world tasks are not covered by the current benchmark.
-
The robustness of the experimental conclusions is not sufficiently analyzed. The current evaluation depends on a fixed set of datasets, templates, models, and Python workflows, so it is unclear whether the results are sensitive to prompt wording, random seeds, dataset splits, statistical procedures, or scorer design. In particular, the improvement from the Empirical Layer may partly result from providing information that is already close to the answer. More extensive ablations and carefully matched controls are needed to establish the source of this improvement.
Questions for authors
-
What types of data-quality problems, main effects, and interaction effects are covered by the injection mechanisms? How were these patterns mapped to phenomena observed in real statistical analyses, and were domain experts involved in validating them?
-
How are reference answers and grading labels generated and verified? How does the evaluation handle multiple valid analytical approaches, models using different but statistically appropriate procedures, or cases where the grader fails to recognize a valid reasoning path?
-
How close are the statistical artifacts in the Empirical Layer to the target answers? Have the authors tested settings in which only partial summaries are provided, the artifacts are shuffled or reduced, or the model receives the original table without access to the Empirical Layer?
-
Can the authors report confidence intervals, variance estimates, or significance tests across datasets, task templates, and models?
We thank the reviewer for these questions. We clarify the details below.
Re 1: Coverage and ecological validity. Table1 in Section 3 and Appendix A describe the covered phenomena: corrupted and conditionally anomalous observations, missing-like labels and target-dependent missingness, heterogeneous outcome variability, insufficient subgroup evidence, nonlinear peaks and plateaus, and synergistic or reversing pairwise interactions. Section 3.3 connects these tasks to published observational analyses, including pneumonia and German Credit case studies. These connections motivate statistical phenomena.
Re 2: Reference answers and grading. We agree that both the gold answer and the judge must be reliable and require human examination. When we design each intervention, we verify the ground truth in two ways: i) an automatic verification script, which is discussed in the paper, and ii) we also manually compare the original dataset and the injected dataset. For example, when we add an interaction effect to the dataset, we would examine whether the feature effect plots show such a pattern. We admit that this manual check is not done in every instance and may not be comprehensive across different random seeds, but it should provide sufficient evidence that the ground truth from the intervention is correct. We will include the discussion of ii) in the paper. Once we have the unique ground truth, we will provide it to the LLM-judge and ask it to grade. The LLM-judge will only grade “correct” when the value is correct. So in our benchmark pipeline design, the LLM-judge grades have already been validated against human annotations.
Re 3: Empirical Layer and controls. The Empirical Layer provides intermediate statistical evidence, and its artifacts are computed from the dataset without access to intervention records or ground-truth answers. Their proximity to the answer varies by task: an interaction surface may reveal a relevant joint pattern, whereas descriptive summaries may provide only supporting context. Agents must still select the relevant evidence and interpret it according to the question. Making this evidence easier to access is the intended function of the Empirical Layer. To examine how agents use the Empirical Layer, we summarize component-level tool calls from their reasoning trajectories over the included injections in the table below. EBM graphs include both main effects and pairwise interactions, which are grouped together in this summary.
ebm_graphsfeature_metadatasemantic_metadatacorrelation_association_matrixoutlier_reportcolumn_statsdistribution_profilemissing_value_patternsRe 4: Robustness and uncertainty. We thank the reviewer for this suggestion. We computed 95% confidence intervals for the accuracy improvement using 1,000,000 bootstrap resamples. These results support an overall accuracy improvement from the Empirical Layer that is relatively consistent across the evaluated models, while its significance varies across data quality and feature contribution tasks, which is consistent with the result in the paper.