- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ITERACT-BENCH evaluates whether data-analysis agents can acquire missing requirements through interaction and apply them consistently across dependent stages. The benchmark contains 86 tasks, 554 analytical stages, and 727 data files derived from Kaggle workflows. A user simulator answers clarification questions and provides corrections using a fixed specification. Independent executable verifiers evaluate finalized answers against reference outputs, which remain inaccessible to the simulator.
The experiments cover seven models and four harnesses, yielding 19 configurations with three valid trials per task. The strongest configuration achieves 70.9% task-level Average Pass Rate, where success requires every stage to pass. Specification and interaction-channel ablations demonstrate the difficulty of acquiring requirements. Additional analyses examine proactive clarification, workflow complexity, error propagation, and simulator sensitivity.
Critical strengths
-
The benchmark connects interactive requirement acquisition with dependent analytical execution.
Tasks require agents to retain both computational artifacts and previously established analytical rules. The persistent environment preserves actual agent state, including errors. Separating simulator-issued stage completion from executable correctness verification provides a clear evaluation boundary. This creates a useful setting for studying analytical failures that isolated, fully specified tasks may miss. (§2; Appendices A.2–A.4)
-
The experiments provide complementary evidence about interaction behavior and evaluation reliability.
Full-specification comparisons and clarification/correction ablations directly test the contribution of information availability and interaction channels. Matched repeated-trial analyses further show that proactive clarification remains associated with correctness within the same stage and configuration. The authors also compare simulators and manually repair identified simulator errors. These analyses substantially strengthen the empirical contribution beyond a leaderboard comparison. (§3.3; Appendices D.1, D.3, and D.5)
Critical weaknesses
-
Evidence for preserving established requirements is less direct than evidence for acquiring them.
The paper reports declining accuracy with dependency depth and identifies downstream propagation of earlier errors. It also acknowledges that depth is confounded with workflow length and stage difficulty. Although the fault taxonomy distinguishes Requirement Acquisition from Requirement Adherence, the latter does not specifically isolate downstream violations of requirements that were correctly applied earlier. These analyses do not clearly separate failure to retain a previously acquired and correctly applied rule from reuse of incorrect upstream artifacts or more difficult downstream computation. A targeted analysis of later violations of previously satisfied requirements would more directly support the benchmark’s distinctive claim about persistent analytical commitments. (§3.3.2; Table 7; Appendices D.4.1 and D.4.3)
-
Simulator validation does not yet establish robustness across agent configurations.
Appendix D.5 provides valuable simulator comparisons and human repair for Qwen3.8-Max with Pi-Agent. With the default GLM-5.2 simulator, repair recovers 7 of the 20 simulator-attributed failed trials, yielding seven additional full-task successes in the 258-trial evaluation. However, the analysis covers one evaluated agent configuration and intervenes only on failures already attributed to the simulator. The manuscript does not report a representative audit of unflagged interactions or whether simulator effects alter comparisons across agent configurations. Thus, simulator-related measurement error is partly characterized, but its effect on broader model and harness comparisons remains uncertain. (Appendix D.5; Table 14)
Questions for authors
- Preservation of established requirements: Can you report downstream violations of requirements that were demonstrably acquired and correctly applied earlier, distinguishing these from incorrect upstream artifacts and independent computation errors? Such an analysis would clarify how directly the benchmark measures requirement preservation.
- Simulator robustness: Have simulator comparisons or human audits been conducted for other agent configurations, including interactions not flagged as simulator errors? If so, please report whether they change the main model or harness comparisons.
- Statistical consistency: Please reconcile the proactive-interaction correlation in Figure 6(b)/§3.3.3 (ρ = 0.911) with Table 11 (ρ = 0.965). Recomputing the 16 configurations displayed in Table 10 gives approximately ρ = 0.964706 and p = 1.67 × 10⁻⁹, matching Table 11 despite its caption specifying 19 configurations. Figure 6(b) also includes three models absent from Table 10. Which configurations are included in each analysis? Please also reconcile the first-submission accuracy difference of 20.1 percentage points in §3.3.3 with 20.3 points in Appendix D.3 and Figure 12.
5 comments
Response to Reviewer CErQ (Part II)
W2: Simulator validation does not yet establish robustness across agent configurations. Appendix D.5 provides valuable simulator comparisons and human repair for Qwen3.8-Max with Pi-Agent. With the default GLM-5.2 simulator, repair recovers 7 of the 20 simulator-attributed failed trials, yielding seven additional full-task successes in the 258-trial evaluation. However, the analysis covers one evaluated agent configuration and intervenes only on failures already attributed to the simulator. The manuscript does not report a representative audit of unflagged interactions or whether simulator effects alter comparisons across agent configurations. Thus, simulator-related measurement error is partly characterized, but its effect on broader model and harness comparisons remains uncertain. (Appendix D.5; Table 14)
Q2. Simulator robustness: Have simulator comparisons or human audits been conducted for other agent configurations, including interactions not flagged as simulator errors? If so, please report whether they change the main model or harness comparisons.
Response: Thank you for highlighting these concerns. We conducted two analyses of simulator robustness across configurations and protocol violations in flagged and unflagged interactions.
(1) Cross-configuration simulator robustness. We extended the comparison to multiple agent and harness configurations on the same 86 tasks with three trials per task, reporting APR (%) below.
| Agent/Harness | GLM-5.2User | Qwen3.8-MaxUser |
|---|---|---|
| Qwen3.8-Max/Pi-Agent | 70.50 | 69.80 |
| GLM-5.2/Pi-Agent | 53.90 | 54.26 |
| DeepSeek-V4-Flash/Pi-Agent | 46.90 | 39.15 |
| GLM-5.2/OpenClaw | 26.40 | 28.29 |
The Pi-Agent model ranking remains unchanged across simulators (Qwen3.8-Max > GLM-5.2 > DeepSeek-V4-Flash). For GLM-5.2, Pi-Agent consistently outperforms OpenClaw, with gaps of 27.50 and 25.97 percentage points. Although absolute APR varies, particularly for DeepSeek-V4-Flash, the tested comparisons remain stable.
(2) Human audit of simulator reliability. To investigate errors potentially missed by fault attribution (FA), we conducted a random human audit of 120 simulator decisions (40 per simulator). We examined (A) FA-flagged simulator failures, (B) FA-unflagged failures, and (C) successful trials without FA. Entries show audited decisions (human-identified protocol violations); the last column reports over-disclosure in successful trials.
| User Simulator | A:FA-flagged | B:FA-unflagged | C:Successful | C:Over-disclosure |
|---|---|---|---|---|
| GLM-5.2 | 3(3) | 15(1) | 22(1) | 1 |
| Qwen3.7-Max | 7(6) | 9(4) | 24(6) | 6 |
| Qwen3.8-Max | 3(3) | 13(1) | 24(1) | 1 |
| Total | 13(12) | 37(6) | 70(8) | 8 |
Importantly, 6 of 37 decisions from FA-unflagged failed trials contained simulator protocol violations, showing that FA missed some issues. We also found 8 over-disclosure violations among 70 decisions from successful trials, which failure-only audits cannot detect.
Together, these analyses show stable tested model and harness comparisons despite previously unflagged simulator violations, including over-disclosure in successful trials. Although simulator reliability is imperfect, the results support robustness within the evaluated configurations.
We have incorporated these analyses into the revised Appendix D.5.
Q3. Statistical consistency: Please reconcile the proactive-interaction correlation in Figure 6(b)/§3.3.3 (ρ = 0.911) with Table 11 (ρ = 0.965). Recomputing the 16 configurations displayed in Table 10 gives approximately ρ = 0.964706 and p = 1.67 × 10⁻⁹, matching Table 11 despite its caption specifying 19 configurations. Figure 6(b) also includes three models absent from Table 10. Which configurations are included in each analysis? Please also reconcile the first-submission accuracy difference of 20.1 percentage points in §3.3.3 with 20.3 points in Appendix D.3 and Figure 12.
Response: Thank you for identifying these inconsistencies.
(1) Proactive-interaction correlation. Figure 6(b) uses all 19 model–harness configurations (), whereas Table 15 uses the balanced subset of 16 (). The three additional Pi-Agent-only configurations are Claude Opus 5, GPT-5.6 Sol, and Kimi-K3. We corrected the Table 11 caption and clarified the scopes in Appendix D.3.
(2) First-submission accuracy. The correct difference is +20.3 percentage points, as reported in Appendix D.3 and Figure 12. The 20.1 pp in §3.3.3 was a reporting error, corrected to 20.3 pp in the revision.
Follow-up Before the Discussion Period Ends
Dear Reviewer CErQ,
As the discussion period is coming to an end, we would like to kindly follow up on our responses to your comments. We hope that our clarifications and additional experiments have adequately addressed your concerns.
If you have any remaining questions or concerns, we would greatly appreciate your feedback and would be happy to provide further clarification.
Thank you again for your time and thoughtful review.
Best regards, The Authors
Response to Reviewer CErQ (Part I)
We thank the reviewer for the thoughtful review and constructive feedback. We have addressed the concerns as follows.
Response: Thank you for highlighting this important distinction. To distinguish violations of previously satisfied commitments from broader cross-stage dependency errors and the propagation of incorrect upstream results, we conducted a targeted trace-level analysis of 200 randomly sampled baseline trials from Qwen3.7-Max and Qwen3.8-Max using Pi-Agent. We examined their interaction and execution traces, focusing on 154 downstream stages with failing final scores.
We report three complementary metrics:
(1) Strict Preservation Violation Rate (PVR): Measures how often agents violate requirements that were previously acquired and correctly applied. The denominator consists of downstream opportunities to reapply such requirements, excluding incorrect upstream artifacts and independent computation errors. We identified 1 violation among 105 eligible opportunities (0.95%).
(2) Non-Propagation Cross-Stage Error Involvement Rate (CEIR): Measures the proportion of failed downstream stages involving newly introduced dependency-handling errors, such as misapplying established requirements, incorrectly using upstream artifacts, or omitting required dependencies. It excludes errors caused solely by propagating incorrect upstream results. Among 69 failed downstream stages with applicable upstream dependencies and assessable error mechanisms, we identified 13 such cases (18.84%).
(3) Overall CEIR: Measures the proportion of the same 69 failed downstream stages involving any cross-stage error, including both newly introduced dependency-handling errors and the propagation of incorrect upstream results. We identified 41 cases (59.42%), comprising 13 with newly introduced errors and 28 involving upstream error propagation alone.
While direct violations of previously satisfied requirements are relatively rare, our analysis reveals broader cross-stage challenges, including incorrect reuse of upstream results, omission of required dependencies, and propagation of upstream errors. These findings provide additional evidence that ITERACT-BENCH evaluates the challenges of maintaining analytical correctness across dependent stages, beyond isolated requirement acquisition or single-stage computation.
We have added these analyses to the revised Appendix D.4.3.