acceptodds
Under review as a conference paper at ICLR 2027

Looking Beyond Accuracy: Surfacing Silent Failures in Agentic Data Science

Abstract

Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of agentic data science frameworks that promise automating the design and execution of end-to-end machine learning pipelines, from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess the downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental requirements of data science: methodological data science knowledge and domain understanding. As such, only assessing downstream performance could mask methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of frameworks. Our qualitative analysis shows that DS agent pipelines are generally lacking problem solving creativity and domain knowledge, compared to competition-winning data scientists. Moreover, we observe a positive correlation between the number of data leakage failures and predictive performance, illustrating the necessity to evaluate agents beyond accuracy. We believe that principled failure mode analysis, as presented here, should be an integral part of agentic DS evaluations, and that the insights we surface are key in developing trustworthy agentic DS pipelines.

Primary area:
datasets and benchmarks
Keywords:
Data Science Agents, Large Language Models, Machine Learning Engineering Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J8rFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates methodological failures in agentic data science systems that conventional evaluations, focused on predictive performance, may overlook. The authors conduct three complementary studies: (1) a large-scale analysis of 15 failure modes across 220 data science tasks and nine agent–model configurations, (2) a controlled test suite targeting six methodological pitfalls, and (3) a comparison of agent-generated pipelines with expert solutions from Kaggle competitions.

Critical strengths
  • The paper highlights a critical limitation of existing data science agent benchmarks: high predictive performance does not necessarily imply methodological correctness—an increasingly important issue as capable agents automate complex ML workflows.

  • The evaluation spans 220 tasks, multiple agent frameworks and backbone models, controlled failure-mode tests, and comparisons with expert Kaggle solutions, offering complementary insights into the limitations of current data science agents.

  • Particularly noteworthy is the finding that data leakage is associated with better predictive rankings. The distinction between failures caused by missing pipeline stages and those arising within more sophisticated pipelines also provides valuable guidance for future agent development.

Critical weaknesses
  • The large-scale analysis relies heavily on a two-stage LLM judge to identify methodological failures, yet the paper provides no systematic human-annotated validation of the precision and recall of the 15 failure categories. Since the main conclusions depend on these judgments, stronger validation of the evaluation framework is necessary.
  • Several failure modes, such as using only one evaluation metric, naive ensembling, and the absence of feature engineering , do not necessarily indicate methodological errors in all scenarios. Their validity depends on task requirements and model characteristics. The paper should better distinguish objectively invalid practices from potentially suboptimal design choices, as this distinction directly affects the reported failure rates.
  • The paper reports that leakage-related failures are associated with improved predictive rankings, but the regression analysis controls only for agent and model identities, without fully accounting for task characteristics or pipeline complexity. A controlled comparison of identical pipelines before and after leakage correction would provide stronger evidence.
Questions for authors
  1. How were the 15 failure modes validated against expert data scientists' judgments? Can the authors report category-level precision, recall, or inter-annotator agreement?

  2. How does the failure prevalence change when subjective categories such as FM07, FM12, and FM15 are excluded?

  3. Can the authors conduct controlled leakage-removal experiments to determine whether correcting leakage consistently reduces the reported predictive performance?

  4. How representative are the controlled test results of real-world data science tasks, particularly for failures requiring domain-specific knowledge?

4 comments

Part 1/2

Thank you for the constructive review, and helpful suggestions. We address the raised questions and remarks below.

W1 & Q1: "The large-scale analysis relies heavily on a two-stage LLM judge to identify methodological failures, yet the paper provides no systematic human-annotated validation of the precision and recall of the 15 failure categories... How were the 15 failure modes validated against expert data scientists' judgments? Can the authors report category-level precision, recall, or inter-annotator agreement?"

R1: Our two-step judging procedure, prompt and model selection were initially developed by aligning against human assessments on pilot pipelines. We now add a formal human evaluation (Sec. 3.1 & 4.1).

Two data science experts independently annotated 19 randomly sampled pipelines across all failure modes. The two experts agreed on 0.947 of labels (Cohen's κ=0.80\kappa = 0.80). Our LLM judge reached 0.895 and 0.926 agreement with the two annotators (κ=0.62\kappa = 0.62 and 0.710.71, substantial agreement).

Sampled pipelines do not cover all FMs, so we performed targeted annotations on FM01 wrong task formulation, FM02 peeking at test data, FM09 feature selection before splitting, FM10 wrong model type and FM13 reusing the test set. The resulting precision is 83% overall and 100% on the two leakage modes (FM02 and FM09).

Per-category P, R, and F1 (to be included in the appendix) show no significant inter-annotator disagreement; rare categories are covered by the targeted review above.

Failure modeA1PRF1A2 posPRF1
FM01 Wrong task formulation00.00——00.00——
FM02 Peeking at test data in EDA00.00——00.00——
FM03 Not cleaning data40.801.000.8971.000.710.83
FM04 Imputation before splitting40.500.250.3321.001.001.00
FM05 Ignoring temporal order10.501.000.6710.501.000.67
FM06 No train/val split40.671.000.8050.831.000.91
FM07 No feature engineering51.001.001.0030.751.000.86
FM08 Scaling before splitting31.000.330.5011.001.001.00
FM09 Feature selection before splitting10.000.00—00.00——
FM10 Wrong model type0———0———
FM11 No regularisation in DL0———0———
FM12 Naive ensembling60.500.670.5760.500.670.57
FM13 Reusing test set1—0.00—0———
FM14 Single eval of stochastic model41.000.500.6741.000.500.67
FM15 Using only one eval metric161.000.620.77141.000.710.83
Over all modes490.710.650.68430.750.770.76

W2 & Q2: "Several failure modes, such as using only one evaluation metric, naive ensembling, and the absence of feature engineering, do not necessarily indicate methodological errors in all scenarios... How does the failure prevalence change when subjective categories such as FM07, FM12, and FM15 are excluded?"

R2: Each of these failure modes can be harmful within specific contexts. Therefore, we have used a two-stage judging tool that integrates the applicability check on a stage-level in a structured way. We agree that there could be another level of context needed, such as the severity level on the instance-level. Instances with lower severity often reflect suboptimal practices rather than strictly invalid methodology.

The judge assigns a severity score (1-3) to every finding. Across all judged pipelines, occurrences in FM07, FM12, and FM15 are indeed concentrated at severity 1-2, whereas leakage and validation failures are mainly at severity 2-3. FM14 also occurs most frequently at severity level 1-2.

Failure modefindingssev 1sev 2sev 3
FM151,08491%9%0%
FM1242650%50%0%
FM1458170%30%0%
FM078593%94%3%
FM052861%51%48%
FM097412%27%61%
FM08176%41%53%
FM063810%14%86%
FM01270%11%89%

We capture both confidence and severity; human-judge severity agreement reaches 82% (all but one within one level; full results in the extra page).

Removing those less critical failure modes drops the percentage of failure findings and the mean number of findings, with 67.1% (and 63%) of pipelines still containing at least one finding of severity $\ge$2. In the revision, we revise the distinction between failures and bad practices and report prevalence on both groups based on the severity level.

All 15 FMsWithout FM07/12/15Without FM07/12/14/15
Pipelines with $\ge$1 FM occurence94.6%80.5%67.2%
Mean FMs per pipeline2.581.240.91
Pipelines with $\ge$1 FMs of severity $\ge$281.6%67.1%63.0%
Mean # FMs of severity $\ge$2 per pipeline1.600.950.85

Part 2/2

W3 & Q3: "The paper reports that leakage-related failures are associated with improved predictive rankings, but the regression analysis controls only for agent and model identities... Can the authors conduct controlled leakage-removal experiments to determine whether correcting leakage consistently reduces the reported predictive performance?"

R3: We ran the controlled experiment as suggested: for pipelines where the judge flagged leakage, we used an LLM to fix the leakage, and another LLM judge to verify if there is any remaining leakage. We repeat this fix-verify loop until the LLM check confirms there is no more leakage, and we re-execute the pipeline on the same test set to compare scores.

We sampled 152 pipelines across 109 tasks from all five data sources, stratified by agent (ReAct, AIDE, MLEvolve), model (Gemini 3.7, Qwen 27B, Qwen 4B), leakage mode, maximum severity, number of leakage findings (1-3), and evaluation metric. 145 pipelines reached leakage-free status (5 timed out; 2 retained leakage upon manual review and were excluded).

Across the 145 repaired pipelines, 37% scored worse once leakage was removed, 10% scored better, and 52% changed by less than 0.5% as noise. Among inflated pipelines, the median inflation was 4.4%, 18 pipelines lost more than 10% of their score, and the most extreme case (where an engineered feature was equivalent to the target) essentially lost all predictive performance.

Seeing these scores change confirms the paper’s finding that leakage positively correlates with downstream performance. This complementary experiment shows the inflation is directly caused by the leakage rather than other confounding factors. We integrate this in our main text as complementary measures.

Q4: "How representative are the controlled test results of real-world data science tasks, particularly for failures requiring domain-specific knowledge?"

R4: While the empirical representativeness across domains is established by our 220-task real-world study from multiple sources (domain coverage in Table 8, Appendix A.3), the controlled suite is designed as a diagnostic unit-test tool. Each of the tasks is built on real-world datasets with a manually introduced canonical failure and an unambiguous ground-truth practice check, which is a limitation in the large-scale analysis where ground-truth is not available. The suite tests reasoning patterns where domain knowledge is required for valid modeling, such as understanding physical interaction laws (torque = weight x distance in FM04) or domain-specific data encodings (FM02). These tests measure whether agents inspect and reason based on data semantics or blindly apply common data science templates. The diagnostic findings reflect real-world behavior observed from our expert solution comparisons, where human experts spend substantial effort validating domains while agents often ignore them and look for shortcuts.

We would be grateful if you would consider updating your score after these clarifications and additions. Please let us know if you have any remaining questions.

I thank the authors for their detailed rebuttal and additional experiments, which have meaningfully strengthened the empirical analysis and addressed several of my concerns.

However, I remain concerned about the reliability of the LLM-based failure identification, the limited causal evidence linking data leakage to performance inflation, and the sensitivity of the reported failure rates to subjective failure definitions. These issues continue to affect the confidence and generalizability of the paper's central findings.

Therefore, while I appreciate the authors' substantial efforts, I will maintain my original rating of 2.

Thank you for the response.

We would like to emphasize the setup we have for each of the evaluation components ensuring reliability. Experts spent 30+ person-hours annotating 19 end-to-end data science pipelines across all 15 failure modes (two experts, independently) and verifying 42 further judge findings with their severity. The experts' own labels, without any LLM, reproduce the paper's prevalence: 95% of pipelines contain at least one failure (paper: 94.6%). And, our controlled suite does not rely on open-ended judging. Each test has a carefully designed, iterated pitfall with ground-truth checks.

For leakage presence, the observational and interventional estimates agree. Our rank regression shows leaky pipelines rank 0.41 positions higher on their task leaderboard, adjusted for agent and model. Subsequently, removing the leakage from 143 of these pipelines moves them down by 0.52 positions.

To our knowledge, this is the first study of silent failures in data science agents at scale (220 tasks covering different domains, 3 agents with different framework complexity and 3 models with different sizes). We provide evidence of how detrimental generated code can be when leakage is present. Our insights and findings consistently show that a leaderboard score alone cannot tell if the method is sound, and we need to audit the pipelines. Besides showing score is not enough, the analysis surface persistent critical failures such as leakages with different forms and flavors , that informs future research and better deployment practices.

Reviewer_8PiQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper criticizes current evaluation paradigms for agentic data science (Agentic‑DS), which mostly rely on predictive accuracy and code executability. The authors build a taxonomy of 15 methodological failure modes and adopt three evaluation protocols: large‑scale LLM‑as‑Judge analysis over 220 tasks, a controlled test suite of six curated tasks, and qualitative comparison against human Kaggle expert solutions. The work empirically shows most agent‑generated pipelines contain silent methodological failures, and data‑leakage failures positively correlate with predictive performance. The authors advocate integrating failure‑mode analysis into standard Agentic‑DS evaluation. This is a diagnostic empirical study with no new model or mitigation solution proposed.

Critical strengths
  • Originality: Propose a multi‑dimensional evaluation centered on methodological silent failures; report the counter‑intuitive positive correlation between data leakage and predictive performance. Its evaluation pipeline can inform future benchmarks.
  • Quality: Diverse multi‑source datasets; confounding factor of LLM dataset memorization is examined; popular open‑source DS agents are adopted; rich supplementary statistics and case studies are provided in appendix.
  • Significance: Warns the community against over‑reliance on predictive metrics and offers insights for building more trustworthy Agentic‑DS benchmarks.
Critical weaknesses
  • No human gold‑standard calibration for core evaluation tool: Large‑scale quantitative results fully depend on LLM‑as‑Judge without human‑annotated validation. Reliability for detecting nested leakage remains unclear.
  • Sample‑selection bias: Non‑executable runs are directly dropped in controlled experiments, which likely overestimates agent performance; the proportion and properties of discarded samples are not quantified.
  • Diagnosis‑only without intervention ablations: The paper identifies failures but does not test simple interventions such as prompt engineering or sanity‑check modules to investigate failure drivers.
  • Key findings are marginalized in appendix: Severe validation‑to‑train leakage is insufficiently analyzed in main text; recall for open‑text‑detected failures is unknown.
  • Poor reproducibility: Open‑source status for prompts, datasets and code is unclear; the qualitative Kaggle analysis relies on very limited case examples.
Questions for authors

Same as weakness

1 comment

Thank you for your thoughtful and helpful review. We address the raised remarks below.

W1: “No human calibration. <..>. Reliability for detecting nested leakage remains unclear.”

R1: Our two-step judging procedure, prompt and model selection were initially developed by aligning against human assessments on pilot pipelines. We now add a formal human evaluation (Sec. 3.1 & 4.1).

Two data science experts independently annotated 19 randomly sampled pipelines across all failure modes. The two experts agreed on 0.947 of labels (Cohen's kappa=0.80). Our LLM judge reached 0.895 and 0.926 agreement with the two annotators (kappa=0.62 and 0.71, substantial agreement).

Sampled pipelines do not cover all FMs, so we performed targeted annotations on FM01 wrong task formulation, FM02 peeking at test data, FM09 feature selection before splitting, FM10 wrong model type and FM13 reusing the test set. The resulting precision is 83% overall and 100% on the two leakage modes (FM02 and FM09).

On nested leakage reliability: across the six leakage modes, the judge's agreement with the annotators matches its overall reliability (0.89 vs 0.895 for Annotator 1; 0.96 vs 0.926 for Annotator 2), with no signs of over- or under-flagging.

W2: "Sample-selection bias: <...> the proportion and properties of discarded samples are not quantified."

R2: We assume this refers to our controlled suite experiments (Fig. 3), where non-executable runs were excluded from the failure-rate calculation. Across six controlled tests reported, only 6 out of 180 runs failed to execute and were excluded. We will explicitly report this in Section 4.3.

Non-executable runs show when agents produce invalid scripts that crash at runtime. Therefore, we do not count them as agents producing plausible solutions. If these pipelines were executable, they could either increase or decrease the presence of failure modes. The "Fail rate" in Figure 3 should then be read only as the presence of failure modes. Furthermore, our study focuses on silent failure in valid pipelines, so we identify pipelines with explicit execution failures and cannot be evaluated for downstream performance as out of scope.

W3: "The paper identifies failures but does not test simple interventions such as prompt engineering or sanity-check modules to investigate failure drivers."

R3: The primary goal of this study is systematic diagnosis of silent failures in agentic DS, to guide future work in this emerging area. Our contributions are (1) a large-scale empirical evaluation on 220 data science tasks across agent frameworks and models; (2) a unit-testing-style reusable controlled test suite that does not depend on ground truth, by injecting known, contextualized pitfalls; (3) a deep dive comparing agent solutions with expert Kaggle solutions, showing fundamental design-choice differences. The insights we surface have not been surfaced before.

We also show how agents handle explicit hints with correct practices (Fig. 4). We find that it can help but hints may also be ignored, especially in MLEvolve's iterative search-and-select loop. Therefore, even providing contextualized hints, does not solve the issues. Informed by our analysis, we inform promising directions (Sec. 5): (1) an FM-based iteration agent loop; and (2) integrating tools such as Skrub and tabular foundation models to further reduce the impact of missing preprocessing and target leakage.

W4: "Severe validation-to-train leakage is insufficiently analyzed in main text; recall for open-text-detected failures is unknown."

R4: We agree that validation-to-train leakage is an important finding. In the revision, we move the validation-to-train leakage analysis from the appendix to Section 4.2 of the main text. Open-ended methodological failures do not have ground-truth enumerations, so exact per-category recall cannot be computed. From our human validation study, we use recall from two independent annotators as a proxy, which is 0.65 for annotator 1 and 0.77 for annotator 2.

W5: "Open-source status for prompts, datasets and code is unclear; the qualitative Kaggle analysis relies on very limited case examples."

R5: We publish all prompts, code, agentic frameworks, and data scripts at https://anonymous.4open.science/r/silent-failures-agentic-ds-7E32/. We also include the generated pipeline and judge results in the repository.

Our comparison against human solutions (on Kaggle) is designed as a qualitative study to illustrate differences between agent-generated solutions and expert solutions more excplitly. These operation-level comparisons show how experts validate assumptions vs. agents making shortcuts. We will expand on this comparison to clarify the purpose of this in Sec 3.3.

We would appreciate if you would update your review if we have addressed your remarks. We also welcome any remaining questions.

Reviewer_jzQ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the results from 3 data science agents on 220 selected Kaggle competition problems and data science benchmarks to find failure modes when agents generate data science piplelines. The method employed was LLM as Judge for assessing the failure modes. Given 3 agents and 3 models for each, the experiment claims that there exists at least one failure modes on all cases. Table 1 and Table 2 shows the categorization of the failure modes. However, this paper doesn't show how each agent generates data science pipelines for each problem.

Critical strengths

Large empirical study on the results from 3 agents on data science benchmarks and Kaggle competition problems. Categorization of the failure modes with human in the loop evaluations.

Critical weaknesses

This paper is an observational study on a selected set of agents and LLMs. The main assessment is by LLMs so it could be also biased in the same way each agent generates failures. Chaning LLMs for judgement might also change the analysis. This paper also doesn't show how to fix those failure modes.

Questions for authors

Given that there are several common failure modes in each stage of the data science pipeline, what if we provide the instructions and hints for agents to avoid those failure modes up front? How each agent generate the answer?

2 comments

Thank you for the helpful feedback, which we address below.

W1: "The main assessment is by LLMs so it could be also biased in the same way each agent generates failures. Changing LLMs for judgement might also change the analysis."

R1: Our two-step judging procedure, prompt and model selection were initially developed by aligning against human assessments on pilot pipelines. We now add a formal human evaluation (Sec. 3.1 & 4.1).

Two data science experts independently annotated 19 randomly sampled pipelines across all FMs. The two experts agreed on 0.947 of labels (Cohen's kappa=0.80). Our LLM judge reached 0.895 and 0.926 agreement with the two annotators (k=0.62 and 0.71, substantial agreement).

Sampled pipelines do not cover all FMs, so we performed targeted annotations on FM01 wrong task formulation, FM02 peeking at test data, FM09 feature selection before splitting, FM10 wrong model type and FM13 reusing the test set. The resulting precision is 83% overall and 100% on the two leakage modes (FM02 and FM09).

We also re-evaluated the same pipelines (n=285 labels) with a different model family (Claude Opus 5.5):

PairAgreement [95% CI]Cohen's [95% CI]
Annotator 1 vs Annotator 20.947 [0.912, 0.975]0.80 [0.69, 0.91]
Annotator 1 vs Gemini 3.10.895 [0.860, 0.926]0.62 [0.50, 0.73]
Annotator 2 vs Gemini 3.10.926 [0.902, 0.947]0.71 [0.62, 0.80]
Annotator 1 vs Opus 5.50.860 [0.814, 0.905]0.53 [0.38, 0.67]
Annotator 2 vs Opus 5.50.912 [0.869, 0.951]0.69 [0.54, 0.83]
Gemini 3.1 vs Opus 5.50.888 [0.853, 0.919]0.61 [0.51, 0.71]

Both annotators agree more with Gemini 3.1 Pro than with Opus 5.5, justifying our LLM selection. Judges' disagreements are not random: they correlate strongly with (self-reported) judge confidence. When confidence increases from medium to high, the chance of the other judge disagreeing drops from 86% to 11%. Judges disagree with each other (11%) about as much as humans (10%). Changing the judge does not fundamentally change the results, since Gemini and Opus agree 89% of the time and disagreement concentrates in the small fraction of non-high-confidence judgments. On high-confidence judgments only, results change subtly: pipelines with at least one failure shift from 94.6% to 94.0%, and average failures per pipeline from 2.58 to 2.48. Together, these results support the validity of our LLM judge.

W2: "This paper is an observational study on a selected set of agents and LLMs….This paper also doesn't show how to fix those failure modes."

R2: The primary goal of this study is systematic diagnosis and evaluation of silent failures in agentic DS, to guide future work in this emerging area. Our contributions are (1) a large-scale empirical evaluation on 220 data science tasks across agentic frameworks and models; (2) a unit-testing-style reusable controlled test suite beyond observational study that does not depend on ground truth, by injecting known, contextualized pitfalls; (3) a deep dive comparing agent solutions with expert Kaggle solutions, showing fundamental design-choice differences.

We also show how agents react to explicit hints with correct data science practices (Figure 4). Hints help in some cases but can be ignored, especially in MLEvolve's iterative search-and-select loop. Providing instructions, even specific contextualized ones, does not completely solve issues. Informed by our analysis, we inform promising directions (Sec. 5): (1) an FM-based iteration agent loop; and (2) integrating tools such as Skrub and tabular foundation models to further reduce the impact of missing preprocessing and target leakage.

W3: "Given that there are several common failure modes in each stage of the data science pipeline, what if we provide the instructions and hints for agents to avoid those failure modes up front?"

R3: We evaluate this in our controlled suite with explicit hints (Fig. 4), from a mild hint ("inspect the target label distribution and consider whether it affects model evaluation or training") to an explicit one (class_weight='balanced', stratified split). These hints are task- and pipeline-specific, hence not available in fully automated solutions, so the hints reflect a human-in-the-loop setup. While some DS agents follow these instructions, MLEvolve ignores strong hints in three of the six cases, hence instructions are not guaranteed to survive the agent's iterative loop.

W4: "However, this paper doesn't show how each agent generates data science pipelines for each problem... How each agent generate the answer?"

R4: In our revision, we use the extra page and appendix to expand the descriptions of evaluated methods further (Sec. 4.1). Code, prompts, configurations, agent and LLM-judge traces are at: https://anonymous.4open.science/r/silent-failures-agentic-ds-7E32/

We would appreciate if you would reconsider your score if we addressed your remarks. We also welcome any open questions.

Thanks very much for your reply.

Given that the main goal of this research is "systematic diagnosis and evaluation of silent failures in agentic DS", there could be some diagnosis to explain why the failures happened. Table 1 shows the identified list of failure modes, and a task-agnostic failure mode checklist can be provided to the system prompt. It is not clear this failure modes are native limitaiton of the agent or a limitation of the prompts or skills of the agent. ReAct and AIDE agents perform well when explicit hints are given to the agent, suggesting that the failures are due to missing instructions or skills to check and correct the failure modes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.