acceptodds
Under review as a conference paper at ICLR 2027

Fail Loudly: An Auditable Runtime for Agentic Data Analysis

Abstract

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, Data Science Agents, Agentic Data Analysis, Reliable AI Systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_wigHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback, for data analysis tasks.

Critical strengths
  1. The authors open-sourced the codebase of RADRA, and I highly appreciate it.

  2. The authors done comprehensive evaluations through multiple benchmarks, i.e., DABStep benchmark, KramaBench, and DA-Code. Specifically, it outperforms the state-of-the-art baseline (DS-STAR), and also shows high generalizability across various evaluation setups.

  3. I think the idea is novel, and also important, since auditing the execution traces for any kinds of agents are an important topic to discuss. In addition, the authors provided comprehensive ablation studies to support their idea.

Critical weaknesses
  1. The key weakness I think is the absolute performance of the RADAR. Most importantly, DS-STAR is not the state-of-the-art data science agent now. As one can see in DABStep Leaderboard, now there are four more agents, i.e., NVIDIA KGMON Data Explorer, DataPilot, gg-agent-gpt-1104-1, CambioML energent.ai DS Agent. Specifically, NIVIDA KGMON Data Explorer achieved almost 90% of accuracy, which is much more higher than RADAR. I would like to recommend an acceptance if the authors can resolve this concern.
Questions for authors
  1. What is the key difference between DS-STAR's data analyzer which generates descriptions for each data, and RADAR's source map construction? Can authors provide comparative ablation study to verify that RADAR's source map construction is better design choice for data science tasks?

  2. DS-STAR also proposes simple query-aware exploration, which compares the cosine similarity between the embeddings of the data description and the user query. On the other hand, RADAR uses BM25 to retrieve the useful data files when the number of files are too large (e.g., in KramaBench). What is a better choice? Can authors provide ablation studies on this? For example, maybe we can check some precisions, recalls in KramaBench subsets

  3. In Wildfire domain in KramaBench, why RADAR fails to achieve a better score when using oracle inputs? This is weird, and I think there should be a sufficient explanation in dedicated section.

  4. Can you provide qualitative examples for visualization tasks in DA-Code benchmark?

  5. Is the proposed idea can be applied to any kind of data science agents? If so, it would be good to add on SOTA methods, and see even we can get better scores.

2 comments

Thank you for the detailed response and for conducting the additional experiments during the rebuttal period. The new empirical results and clarifications effectively address my primary concerns. Accordingly, I am happy to raise my score from 2 to 3. Please make sure to incorporate these results and discussions into the final revision.

Thank you for your thoughtful feedback and for reconsidering your assessment. We are glad that the additional experiments addressed your concerns. We will incorporate these results and discussions into the final revision.

Reviewer_MQMROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents RADAR, an auditable runtime for LLM-based data-analysis agents. Its central idea is to preserve source-level evidence and observation coverage, record typed operator invocations and execution outcomes, and validate selected analytical choices against the available runtime evidence. When a conflict is detected, RADAR can reject an operation, provide diagnostic feedback, or invoke bounded repair. The authors evaluate the system on KramaBench, DA-Code, and DABStep, reporting improvements in task performance and reductions in silent errors and non-deliveries.

The paper addresses an important reliability problem in agentic data analysis: successful execution does not necessarily imply that the result is analytically correct. Its main contribution is an integrated runtime design that connects source exploration, execution records, and rule-based validation, rather than relying exclusively on the agent to detect its own mistakes.

Critical strengths

Important and practically relevant problem. The paper clearly motivates the distinction between execution failures and silent analytical errors. Incorrect joins, missing records, and unsupported assumptions can yield plausible outputs without triggering exceptions. This is a meaningful problem for data-analysis agents.

A coherent runtime design. RADAR combines evidence-preserving exploration, typed operators, execution records, and validation checks. In particular, recording the scope and completeness of observations is a useful design choice: the absence of a value in a partial preview should not be treated as evidence that the value does not exist.

Broad empirical evaluation. The experiments cover three benchmarks with different data-analysis workflows. The reported results are promising: RADAR obtains an Overall score of 0.723 on KramaBench with full source retrieval and 0.747 with oracle sources, compared with 0.484 and 0.580 for DS-STAR, respectively. On DA-Code, RADAR scores 52.9 versus 46.4 for DS-STAR; on DABStep, it scores 52.4 versus 32.9.

Attention to reliability beyond task accuracy. The authors separately analyze silent errors and non-deliveries rather than reporting only benchmark scores. Across 1,054 tasks, they report a reduction in silent errors from 405 to 289 and in non-deliveries from 84 to 24. The examples involving unkeyed joins and missing years also help illustrate how runtime observations can expose analytical problems that ordinary execution may conceal.

Useful implementation detail. The paper describes the supported validation rules, their required observations, and cases in which checks are skipped. The accompanying code availability and appendices potentially make the system easier to inspect and reproduce.

Critical weaknesses
  1. The novelty relative to existing agent verification systems could be established more clearly. The core ingredients—source profiling, typed tool interfaces, execution tracing, rule-based validation, diagnostic feedback, and LLM-assisted repair—are individually intuitive. The paper's contribution appears to lie primarily in their integration and in the explicit linkage between observations and runtime checks. However, the paper could more clearly distinguish this contribution from prior data-analysis agents with iterative verification, execution feedback, and repair. A more systematic comparison of capabilities would help establish what is technically new beyond combining these mechanisms in one runtime.

  2. The validation coverage is necessarily limited, and the relationship between local checks and semantic correctness remains unclear. Several checks apply only to restricted SQL structures or require complete observations; unsupported cases are skipped. These conservative conditions are sensible, but they also mean that many semantically incorrect analyses may pass validation. For example, a query can use valid columns, explicit joins, and well-typed outputs while still selecting the wrong population, using the wrong statistical definition, or answering a subtly different question. The paper acknowledges that semantic interpretation remains the agent's responsibility, but it would benefit from a clearer characterization of which error classes RADAR can detect, which it cannot, and how often the implemented checks actually intervene.

  3. The silent-error evaluation needs stronger methodological clarification. The paper reports substantial reductions in silent errors, but these figures depend on classifying failed runs into silent errors and other failures. More detail on the annotation procedure, decision criteria, annotator independence, and inter-annotator agreement would make the results more convincing. It would also help to report the error rate with an explicit denominator for each benchmark and system, and to clarify how ambiguous cases are handled. Without this information, it is difficult to assess the reliability and reproducibility of the central silent-error claim.

  4. The main comparisons do not fully isolate the source of the improvements. RADAR changes several things simultaneously: source exploration, evidence supplied to the agent, typed execution, validation, diagnostics, and internal repair. The ablations are a useful start, but removing runtime validation also removes diagnostic feedback and repair. Consequently, the ablation does not isolate the effect of validation rules themselves from the additional opportunities to recover from failures. A more controlled comparison would retain the same exploration pipeline, model, tool interface, interaction budget, and repair opportunities while selectively enabling or disabling individual validation checks.

  5. The resource and interaction comparisons could be more informative. The paper reports decision-round limits, API cost, and latency, but a common limit on main-agent decision rounds does not necessarily imply equivalent total computation or feedback opportunities across systems. RADAR performs additional profiling and validation work and includes internal repair. The amortization of preprocessing is also dependent on how many queries reuse the same data lake. Reporting total model calls or tokens, validation and repair overhead, and sensitivity to preprocessing amortization would provide a clearer picture of the practical trade-off.

  6. The generality of the approach is not yet fully established. The evaluation is encouraging across three benchmarks, but the runtime rules are tailored to specific supported operation patterns, and the reported results use a particular model configuration. It remains unclear how well the checks transfer to other SQL constructs, more complex Python transformations, different data-analysis agents, or tasks whose correctness depends on domain-specific assumptions rather than observable execution state. A more explicit analysis of limitations and failure cases would help define the scope of the claims.

Questions for authors
  1. Can the authors provide a more detailed comparison between RADAR's validation mechanism and the verification, execution-feedback, and repair capabilities of existing data-analysis agents? Which component constitutes the principal methodological novelty?

  2. How were silent errors manually identified and categorized? Were outputs independently assessed by multiple annotators, and what was the agreement rate? Can the authors provide per-benchmark confusion counts or representative ambiguous cases?

  3. Can the authors isolate the effect of validation rules from the effects of diagnostic feedback and internal repair? In particular, what happens when the same exploration and repair mechanisms are retained but the pre-execution rejection checks are disabled?

  4. How often does RADAR reject a valid operation or fail to detect an incorrect one? Please report false-positive and false-negative examples, where possible, and discuss the limitations caused by unsupported SQL structures and incomplete observations.

  5. How sensitive are the reported gains to the interaction budget and total model-call/token budget? Would the improvements persist under a matched total-computation budget rather than only a shared main-agent decision-round limit?

  6. What fraction of the remaining silent errors arise from semantic mistakes that are outside the runtime's supported checks? An analysis of residual failures would help clarify the practical ceiling of the proposed approach.

3 comments

Thank you for your comments. Detailed experimental results and configurations are provided in the supplementary material.

W1/Q1. RADAR's contribution lies in an integrated auditable runtime whose components are designed to work together, rather than in any individual mechanism. Exploration preserves evidence with its provenance and observation scope. Typed operators declare inputs, arguments, and outputs, while execution records associate each invocation with its inputs, outcomes, and runtime observations. These connections allow validation rules to check analytical choices against the corresponding evidence. For example, the literal-coverage check can reject an absent filter value when Probe provides a complete enumeration of the referenced column; absence from a partial preview does not trigger rejection. Observation coverage and explicit input bindings therefore determine when the runtime can intervene.

The following table compares capabilities described in the papers and framework documentation.

SystemEvidence-preserving explorationObservation coverageTyped operatorsAuditable execution recordsEvidence-based runtime validation
AOP (CIDR 2025)△—✓✓—
DS-STAR (TMLR 2026)△——△—
Data Interpreter (ACL 2025 Findings)△—△✓—
DA-Agent (EMNLP 2024)△—△✓—
AutoGen (COLM 2024)△—✓✓—
smolagents DR△—△✓—
DS-Guru (ICLR 2026)△——△—
Blackboard (NeurIPS 2026)△—△△—
RADAR✓✓✓✓✓

✓: supported; △: partially supported; —: not described.

DS-STAR verifies plan sufficiency using an LLM supplied with the plan, code, and execution results, while Data Interpreter progressively verifies and refines execution. DA-Agent, DS-Guru, AutoGen, and smolagents DR use execution feedback to guide correction. Blackboard organizes distributed data discovery. AOP provides semantic operators, validation through external sources, and interactive pipeline adjustment. RADAR makes preserved evidence and its coverage directly usable by operation-level checks, with execution records and diagnostic feedback supporting traceable revision and bounded repair.

W2/Q4. RADAR's checks cover analytical operations and answer submission. Alongside SQL-specific checks, they validate declared result types, supporting execution evidence for empty/default answers, and output-file requirements. These submission checks also apply to results produced through Python or Bash. Section 3.2 describes these mechanisms, while Algorithm 1 illustrates the Query-specific validation path. SQL-structure restrictions concern the corresponding query checks; the applicability of other checks depends on their required execution evidence or artifact observations.

In the DA-Code, DABStep, and KramaBench runs, operator-level rules and Python input-binding checks rejected 71 invocations across 66/500 tasks (13.2%), 80 across 75/450 tasks (16.7%), and 39 across 16/104 tasks (15.4%), respectively. Across all 190 rejected invocations, 11 (5.8%) were identified as false positives, all involving Query. In the audited cases, Query validation detected 83 of 88 violations of its implemented rules (94.3%), leaving five false negatives. Python input-binding checks rejected all 23 invalid table-input declarations without rejecting any of the 933 valid declarations.

Figure 4 further shows that transformation errors decrease from 12.4% to 3.7% of delivered tasks, while scope/selection errors still account for 15.6%. These results are consistent with RADAR's ability to detect inconsistencies in analytical operations and the remaining difficulty of interpreting task-specific populations and statistical definitions. For example, in DABStep 17, the agent calculated fraud rates by transaction count instead of monetary volume and returned an incorrect answer without triggering a check. This illustrates how a computation can satisfy the implemented checks while using the wrong statistical definition.

W3/Q2. We used Codex to assist the initial annotation following the protocol in Appendix C.4, which also reports per-benchmark counts and rate denominators. Two experts independently reviewed low-confidence and inconsistent cases. Among the 63 low-confidence cases, they agreed on the category in 55 cases (87.3%); the eight category disagreements were resolved through discussion. We also sampled 20% of the high-confidence cases; both experts agreed with the LLM's labels in every sampled case.

For ambiguous cases, we use the task wording and source evidence to determine whether an analytical error can be established. For example, a KramaBench task asks: “What is the proportion (round to 3 decimal places) of fraud reporters who lost between 1$-500$ in 2024?”

  • The system uses all fraud reports as the denominator: 516,308 / 2,600,678 ≈ 0.199.
  • The reference answer uses only reports with monetary loss: 516,308 / 987,520 ≈ 0.523.

Since the question does not explicitly restrict the denominator to reports with monetary loss, we classify this case as Other failures and exclude it from the silent-error count.

W4/Q3. We added four separate ablation configurations, each disabling one runtime component: validation checks, execution diagnostics, advisory prompt insertions, or bounded internal LLM repair. All other components remain unchanged. Given the limited rebuttal period, we evaluated these finer-grained ablations with DeepSeek-V4-Flash (Preview) on all 104 KramaBench tasks and stratified random samples of 200 DA-Code tasks and 150 DABStep tasks (24 Easy and 126 Hard). Full RADAR and all ablation configurations were evaluated on the same task subsets. Scores are reported as percentages.

ConfigurationKramaBench (104)DA-Code (200)DABStep (150)
Full RADAR72.3054.3556.00
w/o validation checks67.6250.2446.00
w/o execution diagnostics69.9449.2748.67
w/o advisory prompt insertions70.0452.1251.33
w/o bounded internal LLM repair70.9151.7852.67

Removing validation checks reduces performance by 4.68, 4.11, and 10.00 percentage points on KramaBench, DA-Code, and DABStep, respectively. The separate ablations of diagnostics and repair further show that each mechanism contributes to the overall improvements.

W5/Q5. On KramaBench Full Input with the same backbone, RADAR scores 0.723 versus DS-STAR's 0.484 while using 20.99M versus 205.52M recorded query-time tokens. Its reported API cost ($1.16 vs. $3.47) and latency (89 vs. 307 s/task) are also lower. We further provide the following resource breakdown, including exploration, main-agent calls, SQL repair, and loading repair.

BenchmarkRecorded LLM callsTokens (M)Repair calls(SQL + loading)Repair token shareRepair LLM time (s/task)
KramaBench (104)1,42220.99360.42%2.90
DA-Code (500)6,13770.521220.36%1.08
DABStep (450)8,067208.492460.28%1.43

Validation checks make no LLM calls, and their execution time is included in end-to-end latency. Source Map construction also requires no LLM calls and takes 348.81 s across the six KramaBench domains, or 3.35 s/task over 104 tasks. Its preprocessing contribution is T/N for construction time T and N queries. For example, Astronomy, the most expensive domain to profile, takes 293.79 s to build, contributing 29.38 or 2.94 s/query when reused across 10 or 100 queries, respectively.

W6/Q6. Section 4.5 evaluates RADAR with three backbone models. On KramaBench Full Input, RADAR achieves scores of 0.738 with GPT-5.6 Luna and 0.460 with GPT-4.1-mini, outperforming the strongest baseline under each backbone (0.610 and 0.359, respectively). RADAR supports SQL, Python, and Bash, with validation coverage determined by the structure and runtime evidence exposed by each operator, as discussed in W2/Q4.

Scope/selection (161/289) and calculation (66/289) together account for 78.5% of RADAR’s remaining silent errors. These include errors involving population selection and statistical definitions. In DA-Code ml-cluster-007, the output uses the requested column names but fills Feature_i with row indices instead of feature values, further illustrating the need for task-specific semantic constraints. Broader coverage of SQL and Python transformations and integration with other agent architectures remain directions for future work.

Thank you again for your constructive feedback. Our responses now include a more detailed comparison with prior systems, separate runtime-component ablations, measurements of validation coverage and errors, clarification of the silent-error annotation procedure, and a breakdown of model calls, token use, and repair overhead.

As the discussion period approaches its end, we would appreciate any feedback on our responses, particularly on points that may still need clarification. Our General Response summarizes the main additions and manuscript updates.

Thank you for your time and consideration!

Reviewer_4Rh7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the critical problem of silent errors in LLM-based data-science agents. They are the situations that code executes without throwing exceptions and produces plausible-looking outputs, yet fails to answer the intended question due to incorrect choices in data scope or statistical definitions. An empirical audit by the authors of 580 failed runs from a strong baseline (DS-STAR) reveals that 69.8% of failures are silent errors rather than explicit execution crashes.

Critical strengths

I think the most consequential conceptual contribution of this paper is recognizing why naive runtime assertions fail in agentic workflows: an agent or runtime cannot safely reject a filter value or join key based on a partial file preview. By explicitly tracking observation coverage (e.g., formalizing Complete(T,c)  ⟺  0<dT,c≤τ∧∣LT,c∣=∣set(LT,c)∣=dT,c\text{Complete}(T, c) \iff 0 < d_{T,c} \le \tau \land |L_{T,c}| = |\text{set}(L_{T,c})| = d_{T,c} via Probe) , RADAR establishes a sound epistemic condition for when the runtime has sufficient evidence to reject an operation before errors propagate. Another point is that, beyond strong headline metrics across three distinct benchmarks and three model families, the experiment validation stands out.

Critical weaknesses

While RADAR exposes 8 typed operators, its pre-execution validation checks and post-execution diagnostics (empty join/filter checks) are heavily tailored to the Query operator. However, complex data wrangling, statistical modeling, and custom transformations frequently occur inside the Python or Bash operators: for instance, the paper's own motivating example in Figure 1C / Table 7 (wildfire-hard-7) performs the table merge in Python rather than Query. The agent can bypass pre-execution runtime validation simply by choosing Python over Query. This blurs the boundary between how much of RADAR's gain comes from enforced runtime checks versus heavy prompt engineering and operator structuring.

Questions for authors

First about ablation. In Table 5 and Table 9, w/o runtime validation disables pre-execution validation checks, execution diagnostics, advisory prompt insertions, and bounded internal LLM repair all at once. Can you provide a finer-grained breakdown isolating these aspects? The second is that, in Table 4, while RADAR improves dramatically on DABStep Hard tasks (47.1% vs. 22.8% for DS-STAR), its accuracy on Easy tasks drops by 5.5 percentage points (80.6% vs. 86.1%). Did you inspect the failed Easy tasks? Does this degradation stem from false-positive runtime rejections/warnings causing the agent to over-think or alter a valid simple query, or from source truncation during exploration?

5 comments

Thank you for your comments. Detailed experimental results and configurations are provided in the supplementary material.

W1. RADAR's primary goal is to achieve strong task performance across diverse data-analysis workloads. Its operator design preserves the flexibility needed for complex data wrangling, statistical modeling, and custom transformations while making analytical decisions as auditable and revisable as possible. Restricting computations to highly structured operators solely to enable stronger runtime validation would limit this flexibility and potentially compromise task performance. RADAR allows the agent to choose more structured and auditable operators when suitable, while retaining less structured operators when greater computational flexibility is needed. Query exposes column references, predicates, and join conditions, enabling stronger validation. Python supports general-purpose computations but exposes less internal structure, limiting the checks that can be applied. Bash primarily supports filesystem exploration and file inspection.

Arbitrary computations in Python and commands executed through Bash are not subject to the same Query-specific validation checks. However, both operators still participate in RADAR's auditable execution. The Python operator declares its input tables, which the runtime resolves to loaded tables in the shared execution state and provides as DataFrames. The runtime also registers returned DataFrames or Series in this state as reusable tables and records their schemas and row counts. These tables can be inspected through Probe and reused by subsequent Query or Python operations. For subsequent Query operations, the runtime can apply validation checks, including field binding, using the registered schemas. Bash executions likewise retain their outputs and exit status. Additionally, the runtime checks Python’s declared table inputs, and answer-submission checks also apply to results produced through Python or Bash, independently of Query-specific validation. Thus, while not every internal computation can be validated directly, its observable results can support subsequent inspection, validation, and revision.

Q1. We added four separate ablation configurations, each disabling one runtime component: validation checks, execution diagnostics, advisory prompt insertions, or bounded internal LLM repair. All other components remain unchanged. Given the limited rebuttal period, we evaluated these finer-grained ablations with DeepSeek-V4-Flash (Preview) on all 104 KramaBench tasks and stratified random samples of 200 DA-Code tasks and 150 DABStep tasks (24 Easy and 126 Hard). Full RADAR and all ablation configurations were evaluated on the same task subsets. Scores are reported as percentages.

ConfigurationKramaBench (104 tasks)DA-Code (200 tasks)DABStep (150 tasks)
Full RADAR72.3054.3556.00
w/o validation checks67.6250.2446.00
w/o execution diagnostics69.9449.2748.67
w/o advisory prompt insertions70.0452.1251.33
w/o bounded internal LLM repair70.9151.7852.67

Full RADAR achieves the highest score on all three benchmarks. The largest gaps relative to Full occur without validation checks on KramaBench (4.68 points) and DABStep (10.00 points), and without execution diagnostics on DA-Code (5.08 points). The configuration without advisory prompt insertions scores higher than those without validation checks or execution diagnostics on all three benchmarks.

Q2. DABStep Easy contains 72 tasks. RADAR answered 58 correctly and DS-STAR 62, so the reported 5.5-point gap corresponds to a net difference of four tasks. We inspected all 14 RADAR failures and the cases where the two systems differed in correctness.

The observed errors involved metric definitions, output units, and question interpretation. For example, on tasks #17 and #61, RADAR calculated the fraud rate using transaction counts, while the manual defines it using transaction amounts. The definition was already present in the main agent's context in both cases; the error arose in interpreting and applying it. On task #44, RADAR interpreted the requested percentage as a proportion and returned 0.618551. None of the 14 failed Easy runs triggered validation rejections or empty-result execution diagnostics. We also found no evidence that misleading runtime warnings or source truncation contributed to these failures.

The ablation results by difficulty provide complementary evidence. On the full 72-task Easy split, Full RADAR achieved 80.6% accuracy, compared with 76.4% when runtime validation was disabled, a decline of 4.2 percentage points. Removing runtime interventions therefore did not improve Easy performance in this experiment.

Thank you again for your thoughtful review. Our responses now include separate ablations of validation checks, execution diagnostics, advisory prompts, and internal repair; an analysis of all 14 DABStep Easy failures; and clarification of the scope and limitations of validation for Python and Bash.

As the discussion period approaches its end, we would appreciate any feedback on our responses, particularly on points that may still need clarification. Our General Response summarizes the main additions and manuscript updates.

Thank you for your time and consideration!

Thank you for your response. It offers reasonable clarification, more detailed ablation studies, and analysis of failure cases. It addresses most of my concerns, and I am considering raising my score to 3.

Reviewer_yqHfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors presents RADAR, a runtime for LLM-based data analysis that combines source exploration, typed execution operators, execution records, validation checks, and diagnostic feedback with bounded LLM repair. Its motivation is to reduce silent analytical errors that remain undetected despite successful execution. RADAR preserves source locations and observation coverage, checks selected operations against recorded evidence, and returns feedback for revision. Experiments on KramaBench, DA-Code, and DABStep report improvements in task performance and reductions in observed silent errors. The paper also provides component ablations and audits of runtime interventions.

Critical strengths
  1. The end-to-end gains seem substantial and extend across several settings. RADAR improves on three benchmarks.

  2. The implementation exposes concrete, inspectable intermediate information. Source locations, coverage distinctions, operator records, and detailed intervention counts make the system easier to examine. And I particularly appreciate that the authors document unsupported cases and reports actual rule-trigger frequencies.

Critical weaknesses
  1. I am not convinced that the paper establishes a distinct, reliable validation mechanism beyond a collection of useful engineering heuristics. The central distinction between an observed fact and an invalid analytical choice is unclear in Section 3.2. A complete enumeration establishes that a value is absent; it does not establish that filtering for that value is an error. For example, if a table contains only USA and Canada, asking how many records belong to Mexico has a legitimate answer of zero. As written, the literal-coverage rule in Table 6 would reject the corresponding supported query. Am I missing an additional task-dependent condition, or does the runtime reject a correct analysis in this case? This matters because the rule is presented as evidence-based validation, rather than merely a warning about a suspicious choice.

Other checks similarly enforce limited structural conventions rather than task correctness: an explicit CROSS JOIN is permitted even when the same Cartesian product would be rejected in implicit form. Python and Bash also support computations beyond the documented SQL checks. These boundaries are disclosed, but the paper does not measure false rejections or the proportion of relevant analytical operations actually covered. I therefore cannot tell whether the runtime reliably distinguishes errors from valid but unusual analyses.

and the contribution of evidence preservation is unresolved. Its ablation removes content profiling, inspection, and initial evidence together. That comparison demonstrates the value of providing more useful information, but does not isolate the value of retaining provenance and coverage. A control with the same retrieved content but without these annotations is missing.

  1. **The experiments do not distinguish “making fewer mistakes” from “detecting mistakes,” ** and the main ablation confounds validation with additional LLM assistance.

I was puzzled by the interpretation of the silent-error results. Appendix C.3 audits silent errors only among zero-scored runs. A stronger task solver will naturally produce fewer such runs, even if it has no better mechanism for recognizing its mistakes. Table 15 reports 405 silent errors among 580 zero-scored DS-STAR runs, versus 289 among 438 for RADAR. Thus, the headline 28.6% reduction in silent-error counts accompanies a large reduction in total zero-scored runs; the corresponding fractions within those runs are approximately 69.8% and 66.0%. These are different failure populations, so this is not a controlled detector comparison. Where is the evidence that the runtime identifies and corrects a particular erroneous operation, beyond the fact that the complete agent answers more tasks correctly?

The ablation does not resolve this question. Tables 5 and 9 jointly remove validation checks, diagnostic/advisory feedback, and internal LLM repair. Appendix C.1 permits up to three additional repair turns per invocation outside the main-agent round budget. Consequently, this comparison cannot attribute the improvement specifically to validation. Table 13 makes the issue particularly concrete: neither literal-coverage nor implicit-join rejection occurs on KramaBench, while advisory messages occur in 103 of 104 tasks. The showcased checks therefore cannot themselves explain the KramaBench improvement.

The reported scores remain valuable, but the experiments leave the paper’s central explanation underdetermined. Separate checks-only, feedback-only, and repair-only controls, together with matched-operation evaluation, are needed to establish that explanation.

Questions for authors
  • What makes an operation rejectable rather than merely suspicious? In the absent-value example above, how does RADAR distinguish a valid zero-result query from an incorrect filter? Please provide the relevant condition and measurements of false rejections on valid operations.

  • Which part of “runtime validation” produces the gains? Please separate deterministic checks, diagnostic/advisory feedback, and internal LLM repair while keeping exploration and the main agent fixed. How do the results change when total LLM calls or token budgets are matched?

  • Does preserving evidence metadata help beyond supplying the evidence itself? What happens when the agent receives exactly the same retrieved rows and passages, but without provenance and coverage annotations?

  • How effective is the runtime at detecting and correcting errors on matched inputs? For a fixed set of valid and erroneous operations, what are the detection rate, false-positive rate, correction success rate, and fraction skipped as unsupported? Please include computations performed through Python and Bash when reporting coverage.

  • How was the silent-error audit independently validated? Appendix C.3 describes LLM annotation with human review of low-confidence or inconsistent cases. Was a random sample of high-confidence annotations independently reviewed, with system identity concealed? Please report agreement and separately examine partially scored runs, which can also contain silent analytical errors.

5 comments

Thank you for your comments. Detailed experimental results and configurations are provided in the supplementary material.

W1/Q1. RADAR’s behavior on the valid zero-result Mexico query depends on the evidence already available for the referenced column. If Probe has not supplied a complete enumeration, the literal-coverage guard does not intervene, and the query can execute to obtain the count directly. This includes both cases where Probe has not been invoked and cases where its enumeration is incomplete.

When complete Probe evidence is available, the requested count can already be determined from that evidence. In this example, the runtime verifies that the enumeration is complete, establishing that the country column contains only USA and Canada. Consequently, no record satisfies country = 'Mexico', and the requested count is necessarily zero. The guard can reject the absent-value filter without an additional task-dependent condition, but its diagnostic provides the referenced column and the complete value set supporting this conclusion. RADAR therefore supplies the agent with sufficient evidence to answer directly, without requiring another SQL query. We verified this interaction with DeepSeek-V4-Flash (Preview): all three agent continuations from the rejected-query state returned the correct zero answer.

The join check intervenes when recorded row counts or query structure establish that both inputs can contain multiple rows, an implicit join has no ON/USING, and the enclosing SELECT has no WHERE clause. The diagnostic identifies the risk of multiplying rows and provides two executable paths: specify the pairing through ON/USING, or declare an intentional all-pairs computation through CROSS JOIN. The latter preserves the Cartesian-product semantics and supports analyses that require all combinations. The check also permits singleton broadcasts and skips a join when it cannot determine whether an input can contain multiple rows. We have clarified these conditions in Table 6.

W1/Q4. Alongside Query-specific checks, RADAR validates Python's declared table inputs and registers returned tables with their schemas and row counts. These tables can be inspected through Probe, and their schemas support validation of subsequent Query operations. Bash provides execution diagnostics through command output and exit status. Results produced through Python or Bash undergo Answer checks for result types, supporting evidence for empty/default answers, and applicable output-file requirements.

In the DA-Code, DABStep, and KramaBench runs, operator-level rules and Python input-binding checks rejected 71 invocations across 66/500 tasks (13.2%), 80 across 75/450 tasks (16.7%), and 39 across 16/104 tasks (15.4%), respectively. These comprised 94 Query invocations, 23 Python invocations, 72 Answer submissions, and one Discover invocation. Across all 190 rejected invocations, 11 (5.8%) were identified as false positives, all involving Query. Query validation detected 83 of 88 cases that violated its rules (94.3%). Python input checks detected all 23 input errors (100%) and accepted all 933 correct inputs (0% false positives). Answer checks detected all 69 answers with an incorrect type or format (100%) across 1,112 submissions. Among the 175 detected Query rule violations, Python input errors, and Answer type/format errors, 128 (73.1%) were successfully corrected following RADAR’s diagnostic feedback.

We additionally measured check applicability across 500 DA-Code, 450 DABStep, and 104 KramaBench tasks:

Operator and checkDA-CodeDABStepKramaBench
Query: at least one schema, literal, join, or window check1,295/1,471 (88.0%)5,372/5,851 (91.8%)336/640 (52.5%)
Python: declared-table input checks460/801 (57.4%)464/608 (76.3%)32/104 (30.8%)
Answer: result-type checks522/522 (100%)457/457 (100%)133/133 (100%)

Scope/selection (161/289) and calculation (66/289) account for 78.5% of RADAR's remaining silent errors. For example, in DABStep 17, the agent calculated fraud rates by transaction count instead of monetary volume and returned an incorrect answer without triggering a check. This illustrates a task-specific statistical choice that the implemented checks do not resolve.

W1/Q3. We held the retrieved rows and passages fixed and compared paired continuations with initial source locations and explicit coverage labels retained or removed. All other conditions were kept unchanged. Across 200 DA-Code, 150 DABStep, and 104 KramaBench tasks, we observed the following execution behavior:

BenchmarkFile-listing and text-reading shell calls: with / without annotationsTotal recorded operations: with / without annotations
DA-Code770 / 7853,222 / 3,272
DABStep307 / 4142,840 / 2,866
KramaBench265 / 3081,486 / 1,592

File-listing and text-reading calls are identified by commands beginning with find, ls, cat, head, tail, sed, grep, rg, wc, file, or stat, excluding commands that write supplied text or edit files in place. Removing annotations increased these calls by 1.9%, 34.9%, and 16.2%, respectively. In KramaBench legal-hard-7, both continuations answered correctly: the annotated continuation used four operations, while the other used 22, including 16 file-listing or reading commands to examine alternative reports and their year coverage. These observations support the role of evidence annotations in guiding source inspection and reducing additional file searches.

W2. Across the same 1,054 tasks, observed silent-error cases decreased from 405 for DS-STAR (38.4% of tasks) to 289 for RADAR (27.4%), while other failures decreased from 175 to 149. Silent errors showed a larger relative reduction than other failures, at 28.6% versus 14.9%. The reduction in total zero-scored runs already includes this improvement: the net decrease of 142 runs comprises 116 fewer silent-error cases and 26 fewer other failures. The conditional proportions of 69.8% and 66.0% describe the composition of each system's remaining failures. The concurrent reduction in other failures also shrinks the denominator, partially offsetting the decline in the conditional silent-error proportion despite a 28.6% reduction in silent-error cases.

To examine error recognition directly, Q4 reports operator-specific detection rates on fixed recorded operations and the correction success rate following diagnostic feedback. For example, in DA-Code di-csv-006, the agent had already proposed a query referencing a nonexistent period column. RADAR identified the missing column and reported the available fields. Following this feedback, the continuation defined period from the date using the task's intervention date and successfully executed the intended before-and-after aggregation.

W2/Q2. We added four separate ablation configurations, each disabling one runtime component: validation checks, execution diagnostics, advisory prompt insertions, or bounded internal LLM repair. All other components remain unchanged. Given the limited rebuttal period, we evaluated these finer-grained ablations with DeepSeek-V4-Flash (Preview) on all 104 KramaBench tasks and stratified random samples of 200 DA-Code tasks and 150 DABStep tasks. Full RADAR and all ablation configurations were evaluated on the same task subsets. Scores are reported as percentages.

ConfigurationKramaBench (104)DA-Code (200)DABStep (150)
Full RADAR72.3054.3556.00
w/o validation checks67.6250.2446.00
w/o execution diagnostics69.9449.2748.67
w/o advisory prompt insertions70.0452.1251.33
w/o bounded internal LLM repair70.9151.7852.67

On KramaBench, Table 13 records eight field-binding rejections and one answer-type rejection. With internal repair retained, removing validation checks reduces scores by 4.68, 4.11, and 10.00 percentage points on KramaBench, DA-Code, and DABStep, respectively.

The frequent advisory prompt insertions in Table 13 largely come from standard SQL usage instructions. We keep these instructions outside the fixed system prompt to save prompt tokens during initial source discovery and loading, when SQL-specific guidance is not yet needed. The rule becomes active after a Query or Python operation, or when SQL-related failures occur, and the same fixed guidance can recur in subsequent turns. These instructions cover quoting conventions, supported statements, and numeric conversion, accounting for 75.9% of advisory occurrences on KramaBench. A further 11.3% provide fixed failure-handling guidance: inspect the error feedback, revise the failed operation, and avoid identical retries. Together, these two shared instruction blocks account for 87.3% of advisory occurrences.

All advisory rules remain enabled in the validation-check ablation. Their contribution is measured separately by the advisory ablation, which reduces the KramaBench score from 72.30 to 70.04.

On KramaBench Full Input with the same backbone, RADAR scores 0.723 versus DS-STAR's 0.484 while using 20.99M versus 205.52M recorded query-time tokens. Its reported API cost ($1.16 vs. $3.47) and latency (89 vs. 307 s/task) are also lower. We further provide the following resource breakdown, including exploration, main-agent calls, SQL repair, and loading repair.

BenchmarkRecorded LLM callsTokens (M)Repair calls (SQL + loading)Repair token shareRepair LLM time (s/task)
KramaBench (104)1,42220.99360.42%2.90
DA-Code (500)6,13770.521220.36%1.08
DABStep (450)8,067208.492460.28%1.43

Internal repair accounts for 0.28%–0.42% of the recorded tokens. Validation checks make no LLM calls, and their execution time is included in end-to-end latency.

Q5. We used Codex to assist the initial annotation following the protocol in Appendix C.4, which also reports per-benchmark counts and rate denominators. Two experts independently reviewed low-confidence and inconsistent cases. The review was unblinded, as the tool interfaces and execution patterns in the traces reveal the originating system. Among the 63 low-confidence cases, they agreed on the category in 55 cases (87.3%); the eight category disagreements were resolved through discussion. We also sampled 20% of the high-confidence cases; both experts agreed with the LLM's labels in every sampled case.

We extended the audit to all 106 non-ML partially scored runs from the two systems, using the same silent-error criteria. We excluded DA-Code ML tasks for both systems because their scores measure predictive performance, and a valid modeling workflow can still receive a partial score. We identified silent errors in 20/53 RADAR runs (37.7%) and 33/53 DS-STAR runs (62.3%). Including these cases alongside the original zero-scored runs gives 309 silent-error cases for RADAR versus 438 for DS-STAR, a 29.5% reduction. The expanded audit shows that RADAR produces fewer silent analytical errors in both zero-scored and partially scored runs.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and Area Chair

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by your recognition of RADAR's practical design, auditable execution, and effectiveness across diverse data-analysis tasks. We have carefully considered the comments and conducted additional experiments to address the main concerns. Below, we summarize our core contributions and the new evidence and clarifications presented in this rebuttal.

For clarity and simplicity, we refer to Reviewers wigH, MQMR, 4Rh7, and yqHf as R1, R2, R3, and R4, respectively.

We also thank Reviewer wigH (R1) and 4Rh7(R3) for their positive reassessment of our work and for raising their score.

Core contributions

  1. Evidence-preserving exploration. RADAR retrieves relevant content from heterogeneous files while retaining source locations, observation scope, and coverage, preserving evidence for subsequent analysis and validation.

  2. Auditable execution and validation. Typed operators link analytical choices to runtime observations. Evidence-based checks, diagnostic feedback, and bounded repair make these choices inspectable and revisable before errors propagate.

  3. Effectiveness and practical efficiency. Across three benchmarks, RADAR achieves the highest Overall scores among evaluated agents, with 28.6% fewer observed silent errors than DS-STAR and non-deliveries reduced from 84 to 24. On KramaBench Full Input, it also has lower API cost and latency than the strongest baseline.

Additional experimental evidence

  1. Expanded baseline comparison (R1). With NVIDIA Data Explorer added, RADAR remains the strongest evaluated agent across all three benchmarks under the same model and without benchmark-specific adaptation. Data Explorer's DABStep score falls from 64.44% to 25.33% without dev-derived code and examples. These adaptation gains do not transfer with its frozen helper to KramaBench or DA-Code, where it provides no improvement.

  2. Separate runtime ablations (R2, R3, R4). We individually disable validation checks, execution diagnostics, advisory prompts, and internal LLM repair on matched sets of 104 KramaBench, 200 DA-Code, and 150 DABStep tasks. Disabling only checks reduces scores by 4.68, 4.11, and 10.00 percentage points, respectively. Each other removal also lowers all three scores.

  3. Operation-level detection and correction (R2, R4). Our detailed responses report that Query validation detects 83/88 violations of its implemented rules (94.3%). Python input checks detect all 23 invalid declarations and accept all 933 valid declarations; Answer checks detect all 69 type/format errors. Following diagnostic feedback, 128/175 detected errors (73.1%) are corrected.

  4. Source Map and retrieval comparisons (R1). Replacing Source Map with DS-STAR descriptions lowers KramaBench performance from 72.30% to 68.26%. Source Map construction takes 2.43 minutes without LLM calls, versus 90.60 minutes for descriptions. RADAR's initial exploration F1 is 76.35% versus DS-STAR's 28.89%, with cumulative recall reaching 88.54% during execution.

  5. Controlled provenance and coverage ablation (R4). Holding retrieved rows and passages fixed, we compare paired continuations with initial source locations and coverage labels retained or removed. Removing these annotations increases file-listing and text-reading calls by 16.2%, 1.9%, and 34.9% on KramaBench, DA-Code, and DABStep, respectively, while answer scores remain similar.

  6. Resource use and repair overhead (R2, R4). On KramaBench Full Input, RADAR uses 20.99M query-time tokens, compared with 205.52M for DS-STAR. Internal SQL and loading repair account for 0.42%, 0.36%, and 0.28% of RADAR's query-time tokens on KramaBench, DA-Code, and DABStep.

  7. Silent errors in partially scored runs (R4). We extend the audit to all 106 non-ML partially scored runs, finding silent errors in 20/53 RADAR runs (37.7%) versus 33/53 DS-STAR runs (62.3%). Including the original zero-scored cases gives 309 versus 438 silent errors, a 29.5% reduction.

Updates to the manuscript during rebuttal

  • Abstract; Sections 1, 4.1–4.3; Appendix C.8 (R1, R4): Added Data Explorer and revised performance comparisons, with adaptation/transfer controls and partial-score analysis.
  • Sections 4.4–4.5; Appendices C.1, C.9 (R2–R4): Added separate runtime component and provenance/coverage ablations and repair resource breakdowns.
  • Section 4.3; Appendix C.4 (R1, R2, R4): Updated silent-error and non-delivery comparisons to include Data Explorer across all 1,054 tasks in the three benchmarks (Figures 3 and 10; Table 17).
  • Appendices C.3, C.5–C.7 (R1): Added Source Map and retrieval comparisons, visualization examples, and Wildfire Oracle analysis.

We have incorporated the updates listed above into the revised manuscript.

We thank the reviewers again for their valuable feedback and respectfully ask that these additions be considered in the final assessment.

Area_Chair_RRNDOpenReview ↗

Reviewer Feedback and Final Discussion

Dear Reviewers,

The authors have submitted their rebuttals addressing your review comments, and the reviewer–author discussion period is coming to an end.

Could you please review the authors' responses, provide your feedback (if you have not), and actively engage in the final discussion before the deadline?

Thank you for your time and contributions.

Best regards,

The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.