acceptodds
Under review as a conference paper at ICLR 2027

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

Abstract

Evaluating statistical discovery by large language model (LLM) agents requires verifiable analytical ground truth. Establishing such ground truth for real-world datasets is costly, and prior knowledge of public datasets can influence agent responses. We introduce SLEUTHBENCH, a benchmark that addresses both problems by injecting controlled data-quality problems and feature effects into public tabular datasets: the injected pattern determines the answer, so reference answers are computed automatically and memorized knowledge of the original table is insufficient, while the table keeps its background structure. The injected patterns are modeled on phenomena reported in real data analyses. The benchmark defines 17 question templates in two families: data-quality questions and feature-contribution questions. We evaluate six state-of-the-art LLMs that analyze the data using a Python coding tool, on data-science and business phrasings of 70 validated dataset–template combinations, yielding 1680 graded responses in total. The models detect data-quality problems reliably (83.8% accuracy) but recover feature contributions poorly (41.9%). Finding how features shape the target requires searching over both candidate variables and analytical procedures. To address this issue, we propose the Empirical Layer, a set of precomputed statistical artifacts comprising summaries, fitted feature and interaction effects, and dataset descriptions, which exposes candidate patterns for direct inspection. Access to these artifacts raises feature-contribution accuracy from 41.9% to 68.0%.

Primary area:
datasets and benchmarks
Keywords:
Data Science, Agent, Benchmark, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Pc6QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SLEUTHBENCH, a benchmark for evaluating the statistical discovery abilities of LLM agents equipped with Python-based tools. The benchmark injects controlled data-quality problems and feature effects into public tabular datasets, allowing reference answers to be computed automatically while preserving the background structure of the original tables. It includes 17 question templates across two families, namely data-quality questions and feature-contribution questions. The authors evaluate six state-of-the-art LLMs on 70 validated dataset-template combinations, producing 1,680 graded responses in total. The models achieve relatively strong performance on data-quality detection, with an accuracy of 83.8%, but perform substantially worse on feature-contribution discovery, with an accuracy of 41.9%. The paper further proposes an Empirical Layer containing precomputed statistical artifacts, including summaries, fitted feature and interaction effects, and dataset descriptions. Providing these artifacts improves feature-contribution accuracy to 68.0%. The main contribution is a benchmark with controllable hidden signals and automatically generated ground truth, together with an empirical analysis showing a substantial gap between data-quality detection and feature-contribution discovery.

Critical strengths
  1. The paper presents a clear and practically relevant evaluation framework. By injecting controlled statistical phenomena into public tabular datasets, the authors can generate reference answers automatically and reduce the influence of memorization. This directly addresses a central challenge in evaluating statistical reasoning by LLM agents.

  2. The analysis goes beyond reporting a single aggregate score. It separates data-quality detection from feature-contribution discovery and evaluates the effect of the Empirical Layer. The observed performance gap between the two task families, together with the improvement from exposing statistical artifacts, provides useful insight into the specific bottlenecks of LLM agents in statistical reasoning.

Critical weaknesses
  1. The ecological validity and coverage of the benchmark remain insufficiently established. Since the target phenomena are injected by the authors, models may be adapting to a limited set of injection mechanisms rather than solving genuinely open-ended statistical discovery problems. The paper should more carefully connect the injected patterns to phenomena encountered in real data analysis and discuss which real-world tasks are not covered by the current benchmark.

  2. The robustness of the experimental conclusions is not sufficiently analyzed. The current evaluation depends on a fixed set of datasets, templates, models, and Python workflows, so it is unclear whether the results are sensitive to prompt wording, random seeds, dataset splits, statistical procedures, or scorer design. In particular, the improvement from the Empirical Layer may partly result from providing information that is already close to the answer. More extensive ablations and carefully matched controls are needed to establish the source of this improvement.

Questions for authors
  1. What types of data-quality problems, main effects, and interaction effects are covered by the injection mechanisms? How were these patterns mapped to phenomena observed in real statistical analyses, and were domain experts involved in validating them?

  2. How are reference answers and grading labels generated and verified? How does the evaluation handle multiple valid analytical approaches, models using different but statistically appropriate procedures, or cases where the grader fails to recognize a valid reasoning path?

  3. How close are the statistical artifacts in the Empirical Layer to the target answers? Have the authors tested settings in which only partial summaries are provided, the artifacts are shuffled or reduced, or the model receives the original table without access to the Empirical Layer?

  4. Can the authors report confidence intervals, variance estimates, or significance tests across datasets, task templates, and models?

1 comment

We thank the reviewer for these questions. We clarify the details below.

Re 1: Coverage and ecological validity. Table1 in Section 3 and Appendix A describe the covered phenomena: corrupted and conditionally anomalous observations, missing-like labels and target-dependent missingness, heterogeneous outcome variability, insufficient subgroup evidence, nonlinear peaks and plateaus, and synergistic or reversing pairwise interactions. Section 3.3 connects these tasks to published observational analyses, including pneumonia and German Credit case studies. These connections motivate statistical phenomena.

Re 2: Reference answers and grading. We agree that both the gold answer and the judge must be reliable and require human examination. When we design each intervention, we verify the ground truth in two ways: i) an automatic verification script, which is discussed in the paper, and ii) we also manually compare the original dataset and the injected dataset. For example, when we add an interaction effect to the dataset, we would examine whether the feature effect plots show such a pattern. We admit that this manual check is not done in every instance and may not be comprehensive across different random seeds, but it should provide sufficient evidence that the ground truth from the intervention is correct. We will include the discussion of ii) in the paper. Once we have the unique ground truth, we will provide it to the LLM-judge and ask it to grade. The LLM-judge will only grade “correct” when the value is correct. So in our benchmark pipeline design, the LLM-judge grades have already been validated against human annotations.

Re 3: Empirical Layer and controls. The Empirical Layer provides intermediate statistical evidence, and its artifacts are computed from the dataset without access to intervention records or ground-truth answers. Their proximity to the answer varies by task: an interaction surface may reveal a relevant joint pattern, whereas descriptive summaries may provide only supporting context. Agents must still select the relevant evidence and interpret it according to the question. Making this evidence easier to access is the intended function of the Empirical Layer. To examine how agents use the Empirical Layer, we summarize component-level tool calls from their reasoning trajectories over the included injections in the table below. EBM graphs include both main effects and pairwise interactions, which are grouped together in this summary.

Tool / ComponentCallsPercentage
ebm_graphs3,05171.57%
feature_metadata3949.24%
semantic_metadata2916.83%
correlation_association_matrix2145.02%
outlier_report1603.75%
column_stats922.16%
distribution_profile561.31%
missing_value_patterns50.12%
Total4,263100%

Re 4: Robustness and uncertainty. We thank the reviewer for this suggestion. We computed 95% confidence intervals for the accuracy improvement using 1,000,000 bootstrap resamples. These results support an overall accuracy improvement from the Empirical Layer that is relatively consistent across the evaluated models, while its significance varies across data quality and feature contribution tasks, which is consistent with the result in the paper.

Resampling unitNumber of groupsAccuracy improvement (pp)Bootstrap 95% CI (pp)
Model6+9.72[5.67, 12.62]
Task17+9.72[−0.27, 21.00]
Reviewer_qrUaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces SleuthBench, a benchmark that injects verifiable statistical signals into public tabular datasets to evaluate LLMs’ability to identify data-quality issues and feature contributions, and provides some interesting insights.

Critical strengths
  1. The benchmark construction flow is clearly defined.
  2. The benchmark provides relatively broad coverage.
Critical weaknesses
  1. The reliability of the ground truth for automatically generated instances requires further validation. The authors should provide additional independent verification beyond the existing validations.
  2. The reliability of the scoring procedure is insufficiently validated. Case studies comparing automated scores with human judgments would help establish the credibility of the scoring results.
Questions for authors
  1. How do the authors validate the ground truth of automatically generated instances?
  2. How reliable is the scoring procedure, especially for answers requiring LLMs-based judgment?
2 comments

We sincerely thank the reviewer for their constructive suggestions and feedback.

Re1 and 2: The reliability of the ground-truth answer and the LLM judge. We agree that both the gold answer and the judge must be reliable and require human examination. When we design each intervention, we verify the ground truth in two ways: i) an automatic verification script, which is discussed in the paper, and ii) we also manually compare the original dataset and the injected dataset. For example, when we add an interaction effect to the dataset, we would examine whether the feature effect plots show such a pattern. We admit that this manual check is not done in every instance and may not be comprehensive across different random seeds, but it should provide sufficient evidence that the ground truth from the intervention is correct. We will include the discussion of ii) in the paper. Once we have the ground truth, we will provide it to the LLM-judge and ask it to grade. The LLM-judge will only grade “correct” when the value is correct. So in our benchmark pipeline design, the LLM-judge grades have already been validated against human annotations.

Thank you for your response.

Reviewer_X4TrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SLEUTHBENCH, a benchmark for evaluating whether LLM agents can discover statistical patterns in tabular data. It injects controlled data-quality issues and feature effects into public datasets, validates that the injected signal is detectable, and automatically computes reference answers. The benchmark includes 17 templates covering data-quality diagnosis and feature-contribution discovery. Across six LLMs, Python-only agents perform well on data-quality tasks (83.8%) but poorly on feature-contribution tasks (41.9%). Adding an Empirical Layer with statistical summaries and EBM-based effects improves feature-contribution accuracy to 68.0%.

Critical strengths
  • Clear benchmark design: Controlled injections provide verifiable ground truth while retaining realistic background structure from public datasets.
  • Insightful empirical finding: The paper shows that LLMs struggle more with selecting appropriate statistical analyses than with basic data-quality checks.
Critical weaknesses
  • Rule based evaluation constrcution has been studied by many previous papers. Missing discussion with them, such as [1]
  • Missing ablations: The paper should separate the effects of descriptive statistics, correlations, EBM main effects, EBM interactions, and metadata.
  • LLM-based grading needs validation: The use of an LLM judge should be supported with human-agreement or robustness analysis.

[1] Unbiased Evaluation of Large Language Models from a Causal Perspective, ICML 2025

Questions for authors
  • Please discuss the relavance with [1]
  • Have you compared against automated statistical-search or rule-based baselines?
  • Did you validate LLM-judge grades against human annotations?
1 comment

We sincerely thank the reviewer for their constructive suggestions and feedback.

Re1: Relevance to rule-based evaluation construction. We agree that controlled benchmark construction has been explored in previous work, including [1]. The key distinction between [1] and our work is the intervention target and the analysis required to answer the resulting questions. [1] primarily introduces interventions on the questions and their components, such as modifying answer options, labels, or numerical values. For interventions that preserve the underlying problem, the reasoning procedure and answer remain unchanged. In contrast, SleuthBench intervenes directly on the underlying tabular dataset, modifying the statistical relationships that the agent must discover. Even when the question and analysis procedure remain the same, the correct answer may change depending on the injected statistical phenomenon. Therefore, agents must analyze the modified dataset, and our distinction concerns the intervention on the underlying empirical data.

Re2: Comparison with automated statistical-search and rule-based methods. Our Python-only (P) setting already provides an LLM-driven automated statistical analysis baseline. Given a question and the underlying dataset, the LLM can autonomously write and execute Python code to explore the data, perform statistical analyses, and identify relevant statistical patterns, without access to the additional empirical information provided by our P+E setting.

Unlike a fixed rule-based baseline, this setting allows the agent to dynamically select its analysis strategies based on the question and observed data. Comparing P with P+E (Empirical Layer) allows us to evaluate whether providing structured empirical evidence improves performance beyond the agent's own Python-based statistical analysis.

Re3: LLM-judge grading validation. When we design each intervention, we verify the ground truth in two ways: i) an automatic verification script, which is discussed in the paper, and ii) we also manually compare the original dataset and the injected dataset. For example, when we add an interaction effect to the dataset, we would examine whether the feature effect plots show such a pattern. We admit that this manual check is not done in every instance and may not be comprehensive across different random seeds, but it should provide sufficient evidence that the ground truth from the intervention is correct. We will include the discussion of ii) in the paper. Once we have the ground truth, we will provide it to the LLM-judge and ask it to grade. The LLM-judge will only grade “correct” when the value is correct. So in our benchmark pipeline design, the LLM-judge grades have already been validated against human annotations.

Re4: Ablation on different Empirical Layer components. We thank the reviewer for highlighting the importance of understanding the contributions of different components. To examine how agents use the Empirical Layer, we summarize component-level tool calls from their reasoning trajectories over the included injections in the table below. EBM graphs include both main effects and pairwise interactions, which are grouped together in this summary.

Tool / ComponentCallsPercentage
ebm_graphs3,05171.57%
feature_metadata3949.24%
semantic_metadata2916.83%
correlation_association_matrix2145.02%
outlier_report1603.75%
column_stats922.16%
distribution_profile561.31%
missing_value_patterns50.12%
Total4,263100%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.