acceptodds
Under review as a conference paper at ICLR 2027

ITERACT-BENCH: Benchmarking Agents for Interactive Data Analysis Workflows

Abstract

LLM-based data agents increasingly support analytical workflows, yet existing benchmarks typically provide requirements upfront or evaluate interaction without testing whether the resulting decisions remain consistent across dependent analyses. In practice, agents must acquire analytical requirements through clarification and feedback, then apply them as persistent constraints on subsequent computations. We introduce IterAct-Bench, a benchmark for evaluating these capabilities in interactive data analysis. Built from real-world Kaggle workflows, IterAct-Bench contains 86 tasks comprising 554 analytical stages over 727 data files. A user simulator provides specification-guided clarification and correction, while independent executable verifiers assess stage-level correctness without exposing gold outputs to the simulator. We evaluate data agents using seven frontier LLMs and four agent harnesses. The strongest configuration achieves an Average Pass Rate (APR) of 70.9%, leaving substantial room for improvement in analytical reliability. Higher APR is strongly associated with proactive clarification. Agents that seek missing requirements before submitting results tend to perform better, and this behavior varies substantially across models and harnesses. Performance also generally declines as workflows contain more stages and deeper dependencies, highlighting the difficulty of maintaining analytical correctness across complex workflows. IterAct-Bench enables systematic evaluation of how agents acquire requirements through interaction and preserve them as constraints on downstream analysis. The data and code are available at [https://huggingface.co/datasets/blue-orbit-472/IteractBench](https://huggingface.co/datasets/blue-orbit-472/IteractBench).

Primary area:
datasets and benchmarks
Keywords:
Data Agents, LLM Agents, Interactive Data Analysis, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CErQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ITERACT-BENCH evaluates whether data-analysis agents can acquire missing requirements through interaction and apply them consistently across dependent stages. The benchmark contains 86 tasks, 554 analytical stages, and 727 data files derived from Kaggle workflows. A user simulator answers clarification questions and provides corrections using a fixed specification. Independent executable verifiers evaluate finalized answers against reference outputs, which remain inaccessible to the simulator.

The experiments cover seven models and four harnesses, yielding 19 configurations with three valid trials per task. The strongest configuration achieves 70.9% task-level Average Pass Rate, where success requires every stage to pass. Specification and interaction-channel ablations demonstrate the difficulty of acquiring requirements. Additional analyses examine proactive clarification, workflow complexity, error propagation, and simulator sensitivity.

Critical strengths
  1. The benchmark connects interactive requirement acquisition with dependent analytical execution.

    Tasks require agents to retain both computational artifacts and previously established analytical rules. The persistent environment preserves actual agent state, including errors. Separating simulator-issued stage completion from executable correctness verification provides a clear evaluation boundary. This creates a useful setting for studying analytical failures that isolated, fully specified tasks may miss. (§2; Appendices A.2–A.4)

  2. The experiments provide complementary evidence about interaction behavior and evaluation reliability.

    Full-specification comparisons and clarification/correction ablations directly test the contribution of information availability and interaction channels. Matched repeated-trial analyses further show that proactive clarification remains associated with correctness within the same stage and configuration. The authors also compare simulators and manually repair identified simulator errors. These analyses substantially strengthen the empirical contribution beyond a leaderboard comparison. (§3.3; Appendices D.1, D.3, and D.5)

Critical weaknesses
  1. Evidence for preserving established requirements is less direct than evidence for acquiring them.

    The paper reports declining accuracy with dependency depth and identifies downstream propagation of earlier errors. It also acknowledges that depth is confounded with workflow length and stage difficulty. Although the fault taxonomy distinguishes Requirement Acquisition from Requirement Adherence, the latter does not specifically isolate downstream violations of requirements that were correctly applied earlier. These analyses do not clearly separate failure to retain a previously acquired and correctly applied rule from reuse of incorrect upstream artifacts or more difficult downstream computation. A targeted analysis of later violations of previously satisfied requirements would more directly support the benchmark’s distinctive claim about persistent analytical commitments. (§3.3.2; Table 7; Appendices D.4.1 and D.4.3)

  2. Simulator validation does not yet establish robustness across agent configurations.

    Appendix D.5 provides valuable simulator comparisons and human repair for Qwen3.8-Max with Pi-Agent. With the default GLM-5.2 simulator, repair recovers 7 of the 20 simulator-attributed failed trials, yielding seven additional full-task successes in the 258-trial evaluation. However, the analysis covers one evaluated agent configuration and intervenes only on failures already attributed to the simulator. The manuscript does not report a representative audit of unflagged interactions or whether simulator effects alter comparisons across agent configurations. Thus, simulator-related measurement error is partly characterized, but its effect on broader model and harness comparisons remains uncertain. (Appendix D.5; Table 14)

Questions for authors
  1. Preservation of established requirements: Can you report downstream violations of requirements that were demonstrably acquired and correctly applied earlier, distinguishing these from incorrect upstream artifacts and independent computation errors? Such an analysis would clarify how directly the benchmark measures requirement preservation.
  2. Simulator robustness: Have simulator comparisons or human audits been conducted for other agent configurations, including interactions not flagged as simulator errors? If so, please report whether they change the main model or harness comparisons.
  3. Statistical consistency: Please reconcile the proactive-interaction correlation in Figure 6(b)/§3.3.3 (ρ = 0.911) with Table 11 (ρ = 0.965). Recomputing the 16 configurations displayed in Table 10 gives approximately ρ = 0.964706 and p = 1.67 × 10⁻⁹, matching Table 11 despite its caption specifying 19 configurations. Figure 6(b) also includes three models absent from Table 10. Which configurations are included in each analysis? Please also reconcile the first-submission accuracy difference of 20.1 percentage points in §3.3.3 with 20.3 points in Appendix D.3 and Figure 12.
5 comments
AuthorsOpenReview ↗

Response to Reviewer CErQ (Part I)

We thank the reviewer for the thoughtful review and constructive feedback. We have addressed the concerns as follows.

W1: Evidence for preserving established requirements is less direct than evidence for acquiring them. The paper reports declining accuracy with dependency depth and identifies downstream propagation of earlier errors. It also acknowledges that depth is confounded with workflow length and stage difficulty. Although the fault taxonomy distinguishes Requirement Acquisition from Requirement Adherence, the latter does not specifically isolate downstream violations of requirements that were correctly applied earlier. These analyses do not clearly separate failure to retain a previously acquired and correctly applied rule from reuse of incorrect upstream artifacts or more difficult downstream computation. A targeted analysis of later violations of previously satisfied requirements would more directly support the benchmark’s distinctive claim about persistent analytical commitments. (§3.3.2; Table 7; Appendices D.4.1 and D.4.3)

Q1. Preservation of established requirements: Can you report downstream violations of requirements that were demonstrably acquired and correctly applied earlier, distinguishing these from incorrect upstream artifacts and independent computation errors? Such an analysis would clarify how directly the benchmark measures requirement preservation.

Response: Thank you for highlighting this important distinction. To distinguish violations of previously satisfied commitments from broader cross-stage dependency errors and the propagation of incorrect upstream results, we conducted a targeted trace-level analysis of 200 randomly sampled baseline trials from Qwen3.7-Max and Qwen3.8-Max using Pi-Agent. We examined their interaction and execution traces, focusing on 154 downstream stages with failing final scores.

We report three complementary metrics:

(1) Strict Preservation Violation Rate (PVR): Measures how often agents violate requirements that were previously acquired and correctly applied. The denominator consists of downstream opportunities to reapply such requirements, excluding incorrect upstream artifacts and independent computation errors. We identified 1 violation among 105 eligible opportunities (0.95%).

(2) Non-Propagation Cross-Stage Error Involvement Rate (CEIR): Measures the proportion of failed downstream stages involving newly introduced dependency-handling errors, such as misapplying established requirements, incorrectly using upstream artifacts, or omitting required dependencies. It excludes errors caused solely by propagating incorrect upstream results. Among 69 failed downstream stages with applicable upstream dependencies and assessable error mechanisms, we identified 13 such cases (18.84%).

(3) Overall CEIR: Measures the proportion of the same 69 failed downstream stages involving any cross-stage error, including both newly introduced dependency-handling errors and the propagation of incorrect upstream results. We identified 41 cases (59.42%), comprising 13 with newly introduced errors and 28 involving upstream error propagation alone.

MetricIdentified CasesRate
Strict PVR1/1050.95%
Non-Propagation CEIR13/6918.84%
Overall CEIR41/6959.42%

While direct violations of previously satisfied requirements are relatively rare, our analysis reveals broader cross-stage challenges, including incorrect reuse of upstream results, omission of required dependencies, and propagation of upstream errors. These findings provide additional evidence that ITERACT-BENCH evaluates the challenges of maintaining analytical correctness across dependent stages, beyond isolated requirement acquisition or single-stage computation.

We have added these analyses to the revised Appendix D.4.3.

AuthorsOpenReview ↗

Response to Reviewer CErQ (Part II)

W2: Simulator validation does not yet establish robustness across agent configurations. Appendix D.5 provides valuable simulator comparisons and human repair for Qwen3.8-Max with Pi-Agent. With the default GLM-5.2 simulator, repair recovers 7 of the 20 simulator-attributed failed trials, yielding seven additional full-task successes in the 258-trial evaluation. However, the analysis covers one evaluated agent configuration and intervenes only on failures already attributed to the simulator. The manuscript does not report a representative audit of unflagged interactions or whether simulator effects alter comparisons across agent configurations. Thus, simulator-related measurement error is partly characterized, but its effect on broader model and harness comparisons remains uncertain. (Appendix D.5; Table 14)

Q2. Simulator robustness: Have simulator comparisons or human audits been conducted for other agent configurations, including interactions not flagged as simulator errors? If so, please report whether they change the main model or harness comparisons.

Response: Thank you for highlighting these concerns. We conducted two analyses of simulator robustness across configurations and protocol violations in flagged and unflagged interactions.

(1) Cross-configuration simulator robustness. We extended the comparison to multiple agent and harness configurations on the same 86 tasks with three trials per task, reporting APR (%) below.

Agent/HarnessGLM-5.2UserQwen3.8-MaxUser
Qwen3.8-Max/Pi-Agent70.5069.80
GLM-5.2/Pi-Agent53.9054.26
DeepSeek-V4-Flash/Pi-Agent46.9039.15
GLM-5.2/OpenClaw26.4028.29

The Pi-Agent model ranking remains unchanged across simulators (Qwen3.8-Max > GLM-5.2 > DeepSeek-V4-Flash). For GLM-5.2, Pi-Agent consistently outperforms OpenClaw, with gaps of 27.50 and 25.97 percentage points. Although absolute APR varies, particularly for DeepSeek-V4-Flash, the tested comparisons remain stable.

(2) Human audit of simulator reliability. To investigate errors potentially missed by fault attribution (FA), we conducted a random human audit of 120 simulator decisions (40 per simulator). We examined (A) FA-flagged simulator failures, (B) FA-unflagged failures, and (C) successful trials without FA. Entries show audited decisions (human-identified protocol violations); the last column reports over-disclosure in successful trials.

User SimulatorA:FA-flaggedB:FA-unflaggedC:SuccessfulC:Over-disclosure
GLM-5.23(3)15(1)22(1)1
Qwen3.7-Max7(6)9(4)24(6)6
Qwen3.8-Max3(3)13(1)24(1)1
Total13(12)37(6)70(8)8

Importantly, 6 of 37 decisions from FA-unflagged failed trials contained simulator protocol violations, showing that FA missed some issues. We also found 8 over-disclosure violations among 70 decisions from successful trials, which failure-only audits cannot detect.

Together, these analyses show stable tested model and harness comparisons despite previously unflagged simulator violations, including over-disclosure in successful trials. Although simulator reliability is imperfect, the results support robustness within the evaluated configurations.

We have incorporated these analyses into the revised Appendix D.5.

Q3. Statistical consistency: Please reconcile the proactive-interaction correlation in Figure 6(b)/§3.3.3 (ρ = 0.911) with Table 11 (ρ = 0.965). Recomputing the 16 configurations displayed in Table 10 gives approximately ρ = 0.964706 and p = 1.67 × 10⁻⁹, matching Table 11 despite its caption specifying 19 configurations. Figure 6(b) also includes three models absent from Table 10. Which configurations are included in each analysis? Please also reconcile the first-submission accuracy difference of 20.1 percentage points in §3.3.3 with 20.3 points in Appendix D.3 and Figure 12.

Response: Thank you for identifying these inconsistencies.

(1) Proactive-interaction correlation. Figure 6(b) uses all 19 model–harness configurations (ρ=0.911\rho=0.911), whereas Table 15 uses the balanced subset of 16 (ρ=0.965\rho=0.965). The three additional Pi-Agent-only configurations are Claude Opus 5, GPT-5.6 Sol, and Kimi-K3. We corrected the Table 11 caption and clarified the scopes in Appendix D.3.

(2) First-submission accuracy. The correct difference is +20.3 percentage points, as reported in Appendix D.3 and Figure 12. The 20.1 pp in §3.3.3 was a reporting error, corrected to 20.3 pp in the revision.

AuthorsOpenReview ↗

Follow-up Before the Discussion Period Ends

Dear Reviewer CErQ,

As the discussion period is coming to an end, we would like to kindly follow up on our responses to your comments. We hope that our clarifications and additional experiments have adequately addressed your concerns.

If you have any remaining questions or concerns, we would greatly appreciate your feedback and would be happy to provide further clarification.

Thank you again for your time and thoughtful review.

Best regards, The Authors

Reviewer_ZjR5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ITERACT-BENCH evaluates LLM data agents in interactive analytical workflows where requirements must be acquired through interaction and then preserved across dependent stages. Each task is a DAG of analytical stages in a shared data environment. A user simulator holds a fixed Full Task Specification and either answers agent questions (CLARIFY) or flags observable violations (CORRECT); the agent acts via ASK/SUBMIT. The simulator never sees gold outputs or verifier results, and its COMPLETE signal means only "no observable violation" — correctness is judged afterward by independent executable verifiers.

Built from Kaggle notebooks (Jan–Jul 2026, for contamination control), the benchmark reduces 310 candidates to 86 tasks / 554 stages, averaging 6.4 stages and 4.4 hidden requirements per stage. Across 7 LLMs and 4 harnesses (19 configurations), best APR is 70.9%, and one model varies 30.6 pp across harnesses. Supplying full specifications upfront raises APR to 80.2–85.7% and shrinks the best–worst model gap from 37.2 to 5.4 pp. Proactive clarification correlates with APR (ρ = 0.911), and performance declines with dependency depth (stage pass rate 88% → 48%).

Critical strengths

S1. The simulator is prevented from becoming the grader. Table 4 draws an explicit information boundary — the simulator sees the specification and submitted values, never gold outputs, reference executions, verifier results, or agent traces — and COMPLETE means only "no observable violation," not correctness, which is judged separately by executableverifiers. This is the step LLM-as-user benchmarks most often skip, and getting it right is what makes any number from this benchmark interpretable at all.

S2. The clarification–accuracy link is tested within condition, not just across configurations. Beyond the cross-configuration correlation (ρ = 0.911), which alone would be confounded by general model capability, the authors compare repeated trials of the same stage under the same model and harness differing only in whether the agent asked before submitting (+20.1 pp first-submission accuracy). With task-level cluster bootstrap CIs and an explicit note that intervals are pointwise rather than simultaneous, Finding 4 is the one headline result I consider unaffected by the task-filter problem in W1.

Critical weaknesses

W1. The benchmark does not measure the capability it claims. The stated novelty is cross-stage commitment preservation (Table 1), yet in the paper's own taxonomy (Table 3) output specification + metric definition are 54.1% of hidden requirements while cross-stage dependency is the rarest at 5.7% — and Fig. 9's two clarification rounds are about field names, 0–1 vs. 0–100 scale, and decimal places, not analysis. Under exact JSON matching with all-stages-pass success (App. A.4), a rounding mismatch is indistinguishable from an analytical error, and AGENT_STATE_MANAGEMENT is explicitly excludedas a primary fault category (App. B.4). So nothing reported separates "lost a commitment" from "guessed a convention wrong," and 70.9% cannot be read as analytical reliability.

W2. Instrument variance exceeds the claimed effects, and tasks are selected against the evaluated models. With agent and trials fixed, swapping the simulator moves APR 70.5% → 57.8% (Table 14) — against a 0.4 pp top-model gap and a 15.6 pp harness spread. After repairing every simulator-attributed trial, 10.9 pp remains, which cannot be agent capability, contradicting App. D.5 and showing the attribution pipeline misses most of the effect; the check covers 1 of 19 configurations. Separately, 163/310 tasks (52.6%) were dropped because all four evaluated models solved them, and QA was iterated until no task-attributed failures remained among those same configurations, with no held-out split.

W3. Finding 4 is built into the design. Proactive ratio is clarify/(clarify+correct) and gold is fixed by requirements disclosed only on asking, so ρ = 0.911–0.965 is mechanical; the matched analysis does not help, since asking is a competence signal rather than a randomized treatment.

W4. The "independent" validation gates are not independent. Builder and Reviewer are both DeepSeek-V4-Pro differing only by session and prompt (App. B.3), and the screening–construction–attribution chain is entirely LLM-internal, with no agreement rates reported and no audit of unflagged trials — against a 38.7% task-defect flag rate.

W5. The harness comparison measures default prompt fit. App. D.4.2 traces OpenClaw's deficit to its Execution Bias block and SOUL.md persona, so Finding 1's harness ranking is erasable by one prompt line, and no prompt-controlled condition is run.

Questions for authors
  1. (W1) Can you decompose the 29.1% APR gap into (a) failures where the agent's analytical procedure was correct but an output convention (rounding, scale, field naming, list order) mismatched, versus (b) genuine analytical errors? Relatedly, what exactly determines which numerical fields receive "small explicit tolerances" (App. A.4), and on how many of the 554 stages is a tolerance set?
  2. (W1) Since AGENT_STATE_MANAGEMENT is excluded as a primary category, can you report any metric isolating cross-stage commitment failures — e.g., stage-level accuracy conditioned on whether an upstream-clarified commitment was correctly reapplied? Without it, what evidence supports the Table 1 claim that this is the benchmark's distinguishing property?
  3. (W2) In Table 14 the agent, harness, tasks, and 258 trials are identical, yet a 10.9 pp gap persists after repairing every simulator-attributed trial. How can that residual be "agent errors" (App. D.5)? If it cannot, what is the implied false-negative rate of the fault-attribution pipeline for simulator error?
  4. (W2) Can you repeat the simulator-swap on at least one more configuration — ideally a low-asking one such as GLM-5.2 × OpenClaw — and report whether the model and harness rankings in Fig. 4 / Table 2 are preserved?
  5. (W2) Are the reported trials the same runs used for the difficulty filter, or independent re-runs? And is there any task subset not subjected to the filter on which the four filtered models and Claude Opus 5 can be compared without selection bias?
  6. (W4) What is the agreement rate between human judgment and your LLM gates (semantic screening, blind contract review, fault attribution) on a random sample, including unflagged / passing trials? Specifically, what is the measured rate of simulator over-disclosure, which no reported statistic covers?
  7. (W3/W5) Do you have any randomized condition — forced-ask vs. forced-no-ask on the same stage, or a harness comparison under a unified system prompt? These would separate Findings 1 and 4 from design artifacts.
7 comments
AuthorsOpenReview ↗

Response to Reviewer ZjR5 (Part I)

We thank the reviewer for the detailed review and constructive feedback. We have carefully examined the concerns and prepared additional experiments and analyses addressing the benchmark's construct validity, evaluation reliability, and experimental design.

W1: The benchmark does not measure the capability it claims. [...] Fig. 9's two clarification rounds are about field names, 0–1 vs. 0–100 scale, and decimal places, not analysis. Under exact JSON matching with all-stages-pass success (App. A.4), a rounding mismatch is indistinguishable from an analytical error, [...]. So nothing reported separates "lost a commitment" from "guessed a convention wrong," and 70.9% cannot be read as analytical reliability.

Q1.1: Can you decompose the 29.1% APR gap into (a) failures where the agent's analytical procedure was correct but an output convention (rounding, scale, field naming, list order) mismatched, versus (b) genuine analytical errors?

Response: Thank you for raising this concern. We conducted a trial-level failure decomposition of the strongest configuration (Claude Opus 5 × Pi-Agent), covering all 75 failed trials out of 258 (APR = 70.9%).

Failure CategoryShare of Failures %
Output-only37.3
Involving Analytical Errors57.3
Other5.3

Output-only refers to failures where the analytical procedure was correct but output conventions, such as field naming, ordering, scale, or rounding, did not match the required specification. Involving Analytical Errors includes substantive errors in population selection, filtering, metric definitions, aggregation, or computation, including cases with additional output mismatches. Other covers failures that cannot be reliably classified into either category, such as truncated outputs caused by excessive response length.

These results show that substantive analytical errors account for the largest share of failures (57.3%), while output-only mismatches also contribute substantially (37.3%). We therefore clarify that APR measures end-to-end analytical workflow reliability under the specified output contract, rather than analytical reasoning accuracy in isolation.

We have added this failure decomposition to Appendix D.4.1.

Q1.2 Relatedly, what exactly determines which numerical fields receive "small explicit tolerances" (App. A.4), and on how many of the 554 stages is a tolerance set?

Response: We audited all 554 stages and found that explicit numerical tolerances are configured in only 10 stages (1.81%), covering 18 fields (45 numerical values). These tolerances are selectively introduced to avoid false mismatches caused by floating-point precision differences.

Specifically, we use a decimal-grid tolerance of (10−1210^{-12}) for an ADF p-value affected by last-bit numerical differences, and a relative tolerance of (10−1210^{-12}) for full-precision p-values spanning different magnitudes, where absolute tolerance could incorrectly equate very small nonzero values with zero. A task-specific relative tolerance of (10−410^{-4}) is also configured for selected regression-related metrics.

All other numerical fields use exact comparison. Tolerances are explicitly specified per field rather than applied globally. We have clarified these policies and their coverage in Appendix A.4 and Table 5.

AuthorsOpenReview ↗

Response to Reviewer ZjR5 (Part II)

W1.2: The stated novelty is cross-stage commitment preservation (Table 1), yet cross-stage dependency is the rarest requirement category at 5.7%, and AGENT_STATE_MANAGEMENT is explicitly excluded as a primary fault category. Nothing reported separates lost commitments from incorrect upstream artifacts or guessed conventions.

Q2: Since AGENT_STATE_MANAGEMENT is excluded as a primary category, can you report any metric isolating cross-stage commitment failures — e.g., stage-level accuracy conditioned on whether an upstream-clarified commitment was correctly reapplied? Without it, what evidence supports the Table 1 claim that this is the benchmark's distinguishing property?

Response: Thank you for raising this concern. We first clarify that the 5.7% in Table 3 refers specifically to explicit cross-stage dependency requirements within the hidden-requirement taxonomy, rather than the prevalence of cross-stage dependencies. In fact, 80.9% of analytical stages depend on upstream analytical states. Furthermore, AGENT_STATE_MANAGEMENT is excluded as a primary fault category to avoid double-counting downstream consequences of earlier errors under our first-divergence attribution protocol. Cross-stage failures are instead analyzed separately.

To quantify these failures, we analyzed 200 randomly selected baseline trials from Qwen3.7-Max and Qwen3.8-Max using Pi-Agent. We examined their interaction and execution traces, focusing on 154 downstream stages with failing final scores. Among these, 69 had applicable upstream dependencies and sufficient evidence to classify the error mechanisms.

We distinguish three categories:

Error CategoryFailed StagesShare
Non-Propagation Cross-Stage Errors13/6918.84%
Propagated Upstream ErrorsOnly28/6940.58%
Overall Cross-Stage Errors41/6959.42%

Non-Propagation Cross-Stage Errors refer to newly introduced errors when handling previously established requirements or computational artifacts, excluding the mere propagation of incorrect upstream results. We identified 1 requirement misapplication, 10 artifact misuse cases, and 2 dependency omissions.

Propagated Upstream Errors Only refers to downstream failures involving incorrect upstream results without additional dependency-handling errors. Overall Cross-Stage Errors combines these two categories.

These findings show that cross-stage errors involve both newly introduced dependency-handling mistakes and the propagation of incorrect upstream results. Importantly, the former cannot be explained solely by incorrect upstream artifacts or independent stage-level computation errors.

Regarding Table 1, ITERACT-BENCH's distinguishing property is its ability to evaluate both requirement acquisition and the subsequent application of established analytical commitments across dependent stages, supported by persistent execution state and independent verification. Our analysis provides empirical evidence of broader cross-stage dependency-handling failures, while distinguishing them from strict violations of previously satisfied commitments.

We have incorporated these analyses into the revised Appendix D.4.3.

AuthorsOpenReview ↗

Response to Reviewer ZjR5 (Part III)

W2.1: With agent and trials fixed, swapping the simulator moves APR 70.5% → 57.8% (Table 14) — against a 0.4 pp top-model gap and a 15.6 pp harness spread. After repairing every simulator-attributed trial, 10.9 pp remains, which cannot be agent capability, contradicting App. D.5 and showing the attribution pipeline misses most of the effect; the check covers 1 of 19 configurations.

Q3: In Table 14 the agent, harness, tasks, and 258 trials are identical, yet a 10.9 pp gap persists after repairing every simulator-attributed trial. How can that residual be "agent errors" (App. D.5)? If it cannot, what is the implied false-negative rate of the fault-attribution pipeline for simulator error?

Q4: Can you repeat the simulator-swap on at least one more configuration — ideally a low-asking one such as GLM-5.2 × OpenClaw — and report whether the model and harness rankings in Fig. 4 / Table 2 are preserved?

Response: Thank you for raising these concerns. We clarify the interpretation of the residual simulator gap and extend the cross-simulator evaluation to additional agent configurations. We address the separate task-selection concern in our response to W2.2/Q5.

(1) Residual simulator gap (Q3). We agree that the 10.9 pp gap remaining after human repair cannot be attributed entirely to intrinsic agent capability. Although the evaluated agent, tasks, and trial counts are fixed, swapping simulators changes the interaction trajectories, potentially inducing different agent decisions, computations, and subsequent failures. Our repair corrects FA-identified simulator violations but does not equalize the remaining interactions across simulators.

Thus, the residual gap reflects simulator sensitivity rather than an implied 10.9 pp FA false-negative rate. The original agent-error attribution refers to failures observed in the repaired continuations, not an explanation of the cross-simulator APR gap. A separate random human audit identified protocol violations in 6 of 37 decisions from FA-unflagged failed trials, confirming that FA diagnostics are not exhaustive, although this does not establish a trial-level false-negative rate.

(2) Cross-configuration simulator robustness (Q4). We extended the simulator-swap experiment to additional configurations, including the requested low-asking GLM-5.2/OpenClaw setting, using the same 86 tasks with three trials per configuration.

Agent / HarnessGLM-5.2 UserQwen3.8-Max User
Qwen3.8-Max / Pi-Agent70.5069.80
GLM-5.2 / Pi-Agent53.9054.26
DeepSeek-V4-Flash / Pi-Agent46.9039.15
GLM-5.2 / OpenClaw26.4028.29

The model ranking under Pi-Agent remains unchanged across the two simulators (Qwen3.8-Max > GLM-5.2 > DeepSeek-V4-Flash). GLM-5.2/Pi-Agent also consistently outperforms GLM-5.2/OpenClaw, with gaps of 27.50 and 25.97 pp, respectively. Although absolute APR varies with simulator choice, particularly for DeepSeek-V4-Flash, the tested model and harness comparisons remain stable.

These results support the robustness of the tested model and harness comparisons while confirming that simulator choice can materially affect absolute APR. We do not claim invariance across all 19 configurations or establish that the 0.4 pp difference between the two leading models is robust to simulator choice. We have clarified these limitations and added the cross-configuration results in Appendix D.5.

Reviewer_N92XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ITERACT-BENCH evaluates interactive data analysis in which agents must acquire missing requirements through clarification and preserve those requirements across dependent analytical stages. The benchmark is grounded in Kaggle workflows and combines persistent execution state, specification-guided user simulation, and executable verification. It contains 86 tasks with 554 analytical stages, and the strongest evaluated configuration achieves an Average Pass Rate of 70.9%.

Critical strengths

The strongest aspect of the paper is its treatment of clarified requirements as persistent analytical commitments. Population definitions, filtering rules, and metric conventions can affect multiple downstream computations, making interaction consequential rather than merely conversational. The benchmark explicitly preserves the agent’s actual state, including errors, rather than replacing intermediate results with reference outputs. This is a useful design choice for studying failures that propagate through a workflow.

Critical weaknesses
  1. My first concern is that some interaction difficulty may be artificial. Output specifications constitute 25.7% of hidden requirements, and generic requests for remaining requirements cannot elicit a complete answer (sec A.1, secA.3). This raises the possibility that performance partly reflects discovering benchmark-specific formatting conventions rather than resolving substantive analytical uncertainty. Providing output schemas upfront and reporting results by requirement category would clarify this concern.
  2. More evidence on simulator reliability remains consequential. With the main simulator, 7.8% of trials in the sensitivity experiment are attributed to simulator errors, and human repair increases APR from 70.5% to 73.3% (§D.5). This analysis is valuable, but covers only one agent configuration and does not establish that model rankings are stable.
Questions for authors

Could the authors clarify whether every retained task receives full human review, reconcile Section 2.2 with Appendix B.4, and explain how reference-solution correctness is independently validated beyond successful execution and same-model Builder–Reviewer checks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer N92X (Part I)

We thank the reviewer for the thoughtful review and constructive feedback. We have addressed the concerns as follows.

W1: My first concern is that some interaction difficulty may be artificial. Output specifications constitute 25.7% of hidden requirements, and generic requests for remaining requirements cannot elicit a complete answer (sec A.1, secA.3). This raises the possibility that performance partly reflects discovering benchmark-specific formatting conventions rather than resolving substantive analytical uncertainty. Providing output schemas upfront and reporting results by requirement category would clarify this concern.

Response: Thank you for this important observation. We agree that hidden output specifications may introduce avoidable formatting difficulty. To quantify their impact, we conducted an Output-Upfront ablation, revealing output-format requirements while keeping substantive analytical requirements undisclosed. Both models were evaluated on all 86 tasks with three trials per task.

AgentOriginalOutput-UpfrontImprovement
Qwen3.7-Max33.334.88+1.6pp
Qwen3.8-Max70.576.74+6.2pp

We further analyzed the remaining failed task trials under Output-Upfront, categorizing requirement-related errors using the taxonomy in Table 3. Note that categories are non-mutually exclusive; percentages are calculated over failed task trials for each model. Note that output-specification errors may still occur under Output-Upfront because agents can incorrectly implement disclosed output conventions, even when these conventions are no longer hidden.

Requirement TypeQwen3.7-Max(N=168)Qwen3.8-Max(N=60)
Metric definition88.1%85.0%
Output specification8.9%13.3%
Population definition20.2%15.0%
Filtering criteria32.7%15.0%
Grouping and comparison45.2%28.3%
Cross-stage dependency16.7%21.7%

Revealing output specifications improves APR by 1.6–6.2 percentage points, confirming that formatting requirements contribute to benchmark difficulty. However, substantial failures remain even after this intervention. In particular, metric-definition-related errors appear in 88.1% and 85.0% of the remaining failed trials, compared with 8.9% and 13.3% involving output specifications. These findings indicate that the benchmark's difficulty cannot be explained primarily by hidden formatting conventions; substantive analytical requirements remain a major source of failure.

We have added the Output-Upfront ablation and requirement-category breakdown to the revised Appendix D.1 (Tables 11–12).

W2: More evidence on simulator reliability remains consequential. With the main simulator, 7.8% of trials in the sensitivity experiment are attributed to simulator errors, and human repair increases APR from 70.5% to 73.3% (§D.5). This analysis is valuable, but covers only one agent configuration and does not establish that model rankings are stable.

Response: Thank you for highlighting this limitation. We extended the cross-simulator evaluation to additional agent and harness configurations, using the same 86 tasks with three trials per task.

Agent/HarnessGLM-5.2 UserQwen3.8-Max User
Qwen3.8-Max/Pi-Agent70.569.8
GLM-5.2/Pi-Agent53.954.26
DeepSeek-V4-Flash/Pi-Agent46.939.15
GLM-5.2/OpenClaw26.428.29

The model ranking under Pi-Agent remains unchanged across simulators (Qwen3.8-Max > GLM-5.2 > DeepSeek-V4-Flash). For GLM-5.2, Pi-Agent also consistently outperforms OpenClaw, with gaps of 27.50 and 25.97 percentage points. Although absolute APR varies with simulator choice (most notably for DeepSeek-V4-Flash), the tested model and harness rankings remain stable.

We have added this cross-configuration robustness analysis in the revised Appendix D.5 (Table 12).

AuthorsOpenReview ↗

Response to Reviewer N92X (Part II)

Q1: Could the authors clarify whether every retained task receives full human review, reconcile Section 2.2 with Appendix B.4, and explain how reference-solution correctness is independently validated beyond successful execution and same-model Builder–Reviewer checks?

Response: Thank you for requesting clarification. We address the three points below.

(1) Human review coverage. All 86 retained tasks underwent final human review for specification clarity, unintended answer leakage, and analytical difficulty. This task-level review is distinct from exhaustive manual reimplementation of every stage's reference solution.

(2) Reconciliation of §2.2 and Appendix B.4. These sections describe two different levels of review. Appendix B.4 describes targeted, in-depth human investigation of suspected task defects during post-construction QA; not every generated candidate receives this investigation. Section 2.2 additionally describes the final human review applied to every retained task. We have clarified this distinction in the revised Appendix B.4.

(3) Independent reference validation. To complement the original executable checks and Builder–Reviewer validation, we independently reproduced 18 workflows comprising 115 analytical stages using a different model family (Claude Opus 5.5), without access to the original reference implementations or gold outputs. All 115 independently computed structured outputs matched the reference outputs. This provides additional cross-model evidence of reference reproducibility.

We have added the independent validation results and clarified the scope of human review in the revised Appendix B.4.

AuthorsOpenReview ↗

Follow-up Before the Discussion Period Ends

Dear Reviewer N92X,

As the discussion period is coming to an end, we would like to kindly follow up on our responses to your comments. We hope that our clarifications and additional experiments have adequately addressed your concerns.

If you have any remaining questions or concerns, we would greatly appreciate your feedback and would be happy to provide further clarification.

Thank you again for your time and thoughtful review.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary to AC

Dear Area Chair,

We sincerely thank the reviewers and AC for their constructive feedback. We have provided detailed responses, conducted substantial additional experiments, and incorporated the results into the revised manuscript.

Our major rebuttal improvements include:

  • Benchmark validity and analytical difficulty (N92X, ZjR5). We conducted an Output-Upfront ablation (+1.6–6.2 pp APR), decomposed failures of the strongest model (57.3% involving analytical errors versus 37.3% output-only), and clarified numerical tolerance policies. These results confirm that the benchmark's difficulty extends beyond output-format conventions to substantive analytical challenges.

  • Cross-stage dependency analysis (CErQ, ZjR5). We introduced Strict Preservation Violation Rate (1/105) and a trace-level failure decomposition distinguishing newly introduced dependency errors (13/69) from upstream error propagation (28/69). We explicitly acknowledge that strict preservation violations are uncommon, while broader cross-stage dependency challenges remain.

  • Simulator reliability and benchmark quality (N92X, CErQ, ZjR5). Cross-simulator evaluations preserve the tested model and harness rankings. A human audit of 120 simulator decisions identified previously unflagged protocol violations. Independent reimplementation reproduced 115/115 reference outputs, and human–LLM fault-attribution agreement was quantified. Together, these results strengthen confidence in the benchmark's reference correctness and the robustness of its principal comparisons across the tested simulators.

  • Interaction and harness effects (ZjR5). A controlled Force-Ask intervention improves APR from 70.50% to 80.23% (+9.73 pp) without additional interaction budget. An OpenClaw prompt ablation improves APR from 26.40% to 31.01%, clarifying the contribution and limitations of harness prompting. We also clarified task-selection bias and evaluation independence.

Reviewer feedback: Reviewer N92X has confirmed that their questions are addressed and maintained a positive rating. Reviewer CErQ acknowledged the value of our additional experiments, noting that they were reasonable and helpful in clarifying the benchmark's evaluation setting and strengthening the paper. We would also appreciate consideration of our detailed responses to Reviewer ZjR5, whose follow-up remains pending.

We hope these additional experiments and revisions help clarify the benchmark's contributions, empirical findings, and limitations. We respectfully ask the AC to consider the full rebuttal and revised manuscript in the final assessment.

Thank you for your time and consideration.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.