acceptodds
Under review as a conference paper at ICLR 2027

DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks

Abstract

Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: https://anonymous.4open.science/r/artifact-review-E3D3.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, benchmark, LLM agents, self-restraint, evidence-dependent decisions, multi-stage analytical tasks, Lean Six Sigma, statistical process control

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_w5kROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DMAIC-Bench, a benchmark for evaluating whether LLM agents use intermediate statistical results to decide when to continue, divert, or stop an analytical workflow. It contains 97 tasks across 15 industrial cases, including paired variants requiring opposite decisions at the same workflow stage. The authors evaluate 13 agents and find 92% accuracy on GO decisions but only 19% on NO-GO decisions. They further analyze execution traces and conduct expert assessments of final reports.

Critical strengths
  1. The benchmark goes beyond checking whether agents can perform statistical calculations. The paired GO/NO-GO design is particularly useful because it tests whether agents can change their analytical route when the evidence changes.

  2. The evaluation covers 13 agents and includes independent expert assessments of final reports. I particularly appreciate the distinction between executing an invalid analysis and actually using its result to support an unjustified conclusion.

Critical weaknesses
  1. The central observation that agents recognize problematic evidence but continue acting has already been studied. AgentAbstain (Liu et al., 2026) evaluates paired act/abstain decisions and distinguishes verbal acknowledgment from actual tool execution. Agentic Abstention (Luo et al., 2026) studies sequential decisions to stop after discovering prerequisites, while Huang et al. (2026) examine awareness-action mismatch in GUI agents. Although DMAIC-Bench introduces statistical-prerequisite workflows, I am not convinced that the reported findings establish a substantially new behavioral phenomenon. The main contribution seems to be a new application setting rather than a new insight into agent behavior.
  2. The benchmark counts downstream computation after a failed prerequisite as incorrect, even when an agent explicitly acknowledges the limitation and does not use the result to justify an invalid conclusion. The difference is substantial, with Table 33 showing that NO-GO accuracy increases from 17% under strict execution scoring to 50% under expert semantic assessment. The authors acknowledge this issue and provide alternative assessments, which I appreciate, but the headline result still relies on the stricter definition. Furthermore, Appendix K acknowledges that trace ordering does not establish whether an agent actually inspected a diagnostic before choosing its next computation. These issues weaken the interpretation of the observed failures as a lack of self-restraint.
  3. Insufficient motivation for using LLMs and missing non-LLM baselines. Perhaps most importantly, I am not convinced that the central GO/NO-GO decisions in this benchmark require an LLM. Many decisions are based on explicit statistical prerequisites and fixed thresholds, which seem better suited to deterministic rule-based checks. While LLMs may be useful for interpreting business instructions and generating reports, their role in enforcing these prerequisites is unclear. The paper also lacks a rule-based or hybrid baseline, making it difficult to assess the practical value of using LLM agents for these workflows rather than conventional statistical pipelines with explicit decision rules.
Questions for authors
  1. Given the large difference between strict routing accuracy and expert semantic accuracy, why should performing a qualified exploratory calculation count as a failure of self-restraint?
  2. Which aspects of the evaluated decision-making require language understanding or reasoning beyond what a rule-based system could provide?
5 comments
AuthorsOpenReview ↗

Response to Reviewer w5kR (1/3)

We thank the reviewer for the insightful suggestion, and we address the following concerns below.

W1. Behavioral differences between Self-restraint compared with prior Abstain related works

A diagnostic the agent computes during a workflow can change which analysis is valid next. When a prerequisite fails, the agent may need to switch methods and keep working. DMAIC-Bench shows that agents can correctly compute and acknowledge a failing result, then run the analysis it rules out in a later turn. The guarded decision graph specifies which branch the evidence warrants. Paired variants hold the workflow checkpoint fixed and vary the data so we can test whether the agent changes its route when the diagnostic changes (Sections 3.1–3.3).

Consider the measurement system analysis(MSA study) in Case 33 (Appendix G). The agent compares inspectors' pass/fail judgments on circuit boards with a reference key and computes Cohen's kappa, an agreement measure that adjusts for chance. If the first study falls below the case's threshold, the next step is to analyze which defect categories account for the disagreements and assess a second study after the inspection criteria are revised. If the second study passes, the agent should chart the lot reject rates. If it fails, the agent should stop the process assessment and escalate to an instrumented inspection. The lot data are available in either case. In one recorded run, the agent followed the first diversion correctly, computed the second study's minimum kappa as 0.5455 against a threshold of 0.60, and explicitly acknowledged that inspection reliability had not been established. It ran the lot chart in the next turn anyway.

WorkStop / withholdSelf-computed diagnosticAlternative execution check
Huang et al. (ConflictGUI)✓——
AgentAbstain✓——
Agentic Abstention✓——
DMAIC-Bench✓✓✓

Legend — marks a behavior outside the paper’s explicit evaluation targets.

Case 33, same project goal 

the case setup is: Initial failure → investigate and retest [Pass] → chart lots [Failed] → retest [Failed] → escalate (Section 2 and Table 1).

After the initial failure, both halting and investigating avoid the lot chart. The DMAIC-Bench verifier checks that the agent actually analyzes the inspector disagreements and follows the repair branch.

The recording audit establishes the order of the failing result and the downstream action (Section 4.3 and Appendix K.1). A separate analysis links the rule probes to workflow decisions and finds frequent routing errors even when the same model answered the corresponding probe correctly and the verifier confirmed the diagnostic it computed in the workflow (Appendix H.1).

AssessmentResultScope
Strict matched route accuracyGO 92% [86, 97], NO-GO 19% [8, 30]13 agents, 40 matched forks (Section 4.2)
In-context NO-GO rule probes94–100% correctRange across 10 models (Appendix H.1)
Wrong route with a correct answer on the corresponding probe74–82%Same 10 models (Appendix H.1)
Wrong route with a correct probe answer and a verified agent-computed diagnostic59–80%Same 10 models, restricted subset (Appendix H.1)
Reproducible diversion11/13 agents solved no diversion task correctly on all three attemptsAppendix J

The controls test whether access to the diagnostic, a prompt to reconsider prerequisites, or more reasoning effort changes routing. Supplying the diagnostic left NO-GO accuracy unchanged on the matched subset eligible by the rule probes. Reflection yielded small estimated gains over a neutral summary, with intervals spanning zero. At maximum reasoning effort, the GO–NO-GO gap remained in both models tested.

Control and outcomeGPT-6-astraDeepSeek-v4-proWhere reported
NO-GO accuracy on the rule-correct subset, Plain → Supplied20.0% → 20.0%7.1% → 7.1%Appendix H.1, Table 22
Change in NO-GO accuracy, reflection minus neutral summary (pp, 95% CI)+5.3 [−2.8, 15.7]+1.8 [−4.8, 9.5]Appendix O.3, Table 44
Route accuracy at maximum reasoning effort (GO / NO-GO)84.5% / 24.6%79.2% / 15.8%Appendix Q, Table 46
AuthorsOpenReview ↗

Response to Reviewer w5kR (2/3)

W2 & Q1. Qualified exploration, report validity, and action timing

A qualified exploratory calculation can be scientifically defensible. Strict routing asks whether the agent honors the prerequisite when choosing which analysis to run. An excluded calculation counts as a route error even if the agent qualifies the result in its report. The scorer credits valid steps completed before a justified stop. When a failed prerequisite calls for diversion, the trace must show that the agent carried out the warranted alternative. Expert rescoring credits acceptable withholding and valid alternatives. Direct semantic assessment asks whether the report's conclusions and recommendations are warranted.

One audited report repeated a process performance estimate while retaining the measurement limitations and withholding a conclusion that the current process was acceptable. In another, the report used an estimate computed with unresolved measurement error as a target for ongoing process control. The expert audit captures both patterns. Giving credit for acceptable conduct narrows the GO–NO-GO gap, and the gap persists when experts judge the reports directly (Section 4.4 and Appendix M, Table 39).

Study 2 outcomeNO-GOGOGO − NO-GO
Strict route accuracy17% [4, 33]90% [73, 100]73 pp [42, 96]
Route accuracy with expert credit for withholding or valid alternatives55% [41, 71]100% (20/20)45 pp [29, 59]
Expert semantic correctness50% [37, 66]75% [59, 89]25 pp [6, 42]
Reports acting on invalid evidence47% [31, 61]30% [11, 48]—
Study 2 scopeDetails
Audit units40 NO-GO units and 20 matched GO counterparts from 11 agents
WeightsSampling weights for NO-GO units and equal weights for GO counterparts
UncertaintyBrackets give case-clustered 95% confidence intervals
ManuscriptSection 4.4, Table 4 and Appendix M, Table 33

The terminal recordings show when the diagnostic was returned and when the agent issued the downstream command. A single prewritten script that runs both analyses counts as a same-turn case. In the blinded review, every continuation classified as a later-turn action included an explicit acknowledgment of the failed prerequisite before the agent continued (Section 4.3 and Appendix K.1, Table 28).

Terminal-recording auditResult
NO-GO errors classified from recordings174
Excluded analysis issued in a later turn, after the failing value was returned106
Diagnostic and excluded analysis ran in the same prewritten script63
Downstream analysis ran before the failing value was available5
Blinded reviewer agreed with the automatic classification44/50 (κ = 0.785)
Agent acknowledged the failed prerequisite before continuing, among human-classified later-turn cases27/27
AuthorsOpenReview ↗

Response to Reviewer w5kR (3/3)

W3 & Q2. Workflow planning, deterministic rules, and the advisory baseline

The LLM turns a natural-language business brief into an analysis plan. It chooses which data to request and which diagnostic to run, then revises the plan as results arrive. A deterministic tool can evaluate a specified diagnostic and enforce the permitted branches once they are encoded.

In Case 33, failure of the initial inspector-agreement study calls for examining the disagreements, revising the inspection criteria, and testing them again. Failure of the repeat study calls for escalating the measurement problem and stopping analyses that rely on those inspections. On the variant where both studies fail, every tested agent under plain instructions computed both agreement statistics correctly and took the first branch correctly. Each then charted the production lots after the repeat study failed (Appendix G).

The non-LLM baseline applies textbook thresholds and selects methods using data shapes and batch labels within a hand-coded DMAIC sequence. It uses the public task information and data steward responses. The rule pipeline handled most industrial decisions. Each NO-GO error occurred where the workflow required a specific alternative analysis after a failed check. The reference solver executes the routes specified for each task (Appendix O.2).

System on the industrial corpus (97 tasks)Mean rewardNO-GO correctGO correctMatched forks correct on both sides
Deterministic rule pipeline0.90046/52218/24832/40
Reference solver1.00052/52248/24840/40

We also ran the industrial pipeline as written on the life-science and X6 cases. The new workflows required diagnostics and branches outside its method library and rule set. For example, in the observational treatment comparison, the analyst must check whether the treatment groups are comparable, restrict the analysis to patients with sufficient overlap if needed, and stop if the available data cannot support the comparison. The GPT agents carried out analyses from the briefs and data. Their mean rewards were higher with the protocol brief in both sets of cases (Section 4.6 and Appendix O.2).

System (mean reward)X6 protocolX6 thresholds onlyLife-science protocolLife-science thresholds only
Unchanged industrial rule pipeline0.0000.0000.0000.000
GPT-6-astra0.9450.7830.9550.911
GPT-5.6-sol0.8650.7240.8970.856
GPT-5.6-luna0.6930.4980.7800.767

In the advisory experiment, a deterministic checker evaluated the prerequisite when invoked, and the agent chose what to run next. The counts measure co-occurrence within a trial. The order and checkpoint correspondence are unknown (Appendix O.2). A hybrid with mandatory gates would let the LLM interpret the brief and plan the workflow, with deterministic code blocking downstream actions when their specified prerequisites fail. When a gate blocks continuation, the agent must still decide whether the current stage calls for a repeat measurement study, an alternative analysis, or escalation (Section 4.5).

Model (32 tasks, three attempts per task)Trials using the checkerTrials with an incorrect NO-GO decision among trials with a failed check
GPT-6-astra22/967/11
DeepSeek-v4-pro84/9630/33

Thank you again for your helpful questions and suggestions.

Reviewer_aG5vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DMAIC-Bench evaluates whether agents use intermediate evidence to continue, redirect, or stop multi-stage analytical tasks. It combines paired task variants, execution scoring, and expert assessment to document failures in evidence-dependent decisions.

Critical strengths

Paired tasks distinguish justified stopping from indiscriminate refusal.

Expert assessment checks whether execution errors correspond to unjustified conclusions.

Critical weaknesses

Changing actions in response to evidence is already central to reasoning. I am not yet persuaded that the results establish what the proposed framing adds to existing accounts of evidence-dependent reasoning.

Stopping can be an outcome of reflection. Reflection revises a decision, which may include deciding to stop. A reflection comparison would help clarify the distinction suggested by the paper’s framing.

The role of self-computed evidence is not directly tested. The study does not compare identical decisions using supplied versus agent-computed evidence. That comparison would help establish whether computing the evidence affects the observed failures.

Correct calculations do not necessarily establish correct interpretation. An agent can calculate a diagnostic accurately without knowing what it implies. The results do not clearly separate missing prerequisite knowledge from failure to apply an understood rule.

The execution score can penalize otherwise defensible judgment. An agent can calculate a result, acknowledge that its assumptions fail, and correctly refuse to rely on it. The route score still counts the calculation as an error.

Questions for authors

How is self-restraint formally different from applying a decision rule to new evidence?

Does performance differ on identical supplied and self-computed evidence, after verifying rule knowledge?

Does reflection improve expert-judged decisions under an equal compute budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer aG5v (1/2)

We thank the reviewer for the suggestion, and we address the following concerns below:

W1, W4 & Q1. Applying a rule versus following it in execution

Applying a decision rule identifies the warranted action. We measure self-restraint by comparing that action with what the agent actually executes. In DMAIC-Bench, a guarded decision graph maps each computed diagnostic to admissible transitions at that checkpoint. The verifier reads the trace to see which transition the agent took and whether it ran the required alternative analysis when the evidence called for diversion. Our contribution is to measure the gap between identifying the warranted action and carrying it out when an intermediate result changes the plan. The same business goal can require a different analysis once a prerequisite fails.

Consider the inspection study in Case 33. If the initial agreement check fails, the agent should investigate why inspectors disagree and evaluate the revised criteria in a second study. A pass permits the agent to chart lot reject rates. A failed retest calls for stopping the process assessment and escalating to an instrumented inspection. The lot data are available under either outcome, so the agent must make that decision while both actions remain possible. In the recorded example, the agent took the first diversion correctly, computed the failing retest value correctly, and said the visual inspection had not been validated. It then ran the lot chart in a later turn. Paired variants hold the workflow position fixed and vary the data, testing whether the agent changes its action when the warranted branch changes (Sections 3.1–3.3 and Appendix G).

We measured interpretation separately by giving models the diagnostic and the original task context and asking what action the rule warranted. Errors remained common at the corresponding checkpoints even when the model answered the independent probe correctly and the verifier confirmed its workflow diagnostic. The recordings then show what happened within the workflow itself, including actions issued after an explicit acknowledgment of the failed prerequisite.

CheckResultScope
In-context NO-GO rule answers94–100% correct10 models, majority over 3 answers per item. 52 NO-GO and 52 GO probes (Appendix H.1)
Wrong workflow route with a correct answer on the corresponding rule probe74–82%Same models, linked decisions (Appendix H.1)
Wrong route with a correct probe answer and a verified self-computed diagnostic59–80%Same models, verified subset (Appendix H.1)
Excluded analysis issued in a later turn after the failing value returned106/174Classified recording audit (Section 4.3, Appendix K.1)
Failed prerequisite explicitly acknowledged before continuing27/27Later-turn continuations identified by the blinded reviewer (Appendix K.1)

W3 & Q2. Does supplying the diagnostic change the decision?

Supplying the diagnostic yielded the same observed NO-GO accuracy as Plain for each model on its rule-correct checkpoints. We matched records by model, case, seed and checkpoint. Both arms used the same frozen task data and reference decisions. In Plain, the agent had to obtain evidence through its own analysis. In Supplied, the brief gave diagnostic values without giving the branch verdict. We retained a checkpoint for a model only when it answered both the corresponding NO-GO probe and its matched GO probe correctly. This eligibility analysis was done after the original runs.

Unobserved routes count as errors. To compare runs with the same focal value, we selected Plain attempts whose trace matched the supplied value at its displayed precision and kept all Supplied attempts at those checkpoints. This sensitivity check also showed no consistent advantage for supplying the result.

SubsetModelCheckpoints / casesPlain correctSupplied correctSupplied − Plain, pp [95% CI]
Rule-correctGPT-6-astra15 / 116/30 (20.0%)9/45 (20.0%)0.0 [−16.7, 15.4]
Rule-correctDeepSeek-v4-pro14 / 112/28 (7.1%)3/42 (7.1%)0.0 [0, 0]
Matching focal valueGPT-6-astra12 / 103/20 (15.0%)6/36 (16.7%)+1.7 [−28.2, 24.5]
Matching focal valueDeepSeek-v4-pro12 / 102/21 (9.5%)3/36 (8.3%)−1.2 [−3.3, 0.0]

The bootstrap draws the same cases for both arms. DeepSeek's main interval has zero width because the observed rates agreed within each retained case. The focal-value check matches values recorded in completed traces and leaves open when the Plain agent saw the result. To isolate the effect of evidence source at identical values, both arms would need the same diagnostic available at the same stage before choosing the next action (Appendix H.1, Table 22).

AuthorsOpenReview ↗

Response to Reviewer aG5v (2/2)

W2 & Q3. Does prerequisite reflection improve expert-judged decisions?

In the blinded review, prerequisite reflection showed no consistent gain over a neutral summary in semantic correctness or report violations. In the experiment with a shared cumulative completion-token ceiling, the two arms agreed on both outcomes in every reviewed pair. Reflection is an intervention that may change a plan. We measure its effect on the route the agent executes and the claims it makes. Both arms were asked to log checkpoints before new data batches and before reporting. The reflection arm stated the prerequisites of its intended analysis and assessed whether its evidence satisfied them. The neutral-summary arm described progress so far. The same steward checkpoint governed data release in both arms.

The original arms shared task data, model settings and configured resource limits within each model. The DeepSeek follow-up used new cases and capped cumulative completion tokens per run, including provider-reported reasoning tokens. The cap matches the generation allowance. The logged totals below show how much prompt and completion usage still differed.

ComparisonRun scopeCumulative completion ceilingReflection's logged prompt + completion tokens
Original GPT-6-astra32 tasks, 3 attempts per arm per taskNone+0.46%
Original DeepSeek-v4-pro32 tasks, 3 attempts per arm per taskNone+3.03%
DeepSeek follow-up16 reviewed runs from 4 additional cases65,536 per run in both arms+34.4%
Model, original runsNO-GO summary → reflectionNO-GO Δ [95% CI]GO summary → reflectionGO Δ [95% CI]
GPT-6-astra21.1% → 26.3%+5.3 [−2.8, 15.7]80.7% → 81.1%+0.4 [−0.8, 1.7]
DeepSeek-v4-pro14.0% → 15.8%+1.8 [−4.8, 9.5]78.4% → 79.5%+1.1 [−0.8, 3.8]

A blinded expert judged the reports with model and condition hidden. The pairs match model, case, seed and checkpoint. For semantic correctness, the expert must judge NO-GO reports as STOP and find no unsupported conclusions or actions. GO reports need a GO judgment and must retain the warranted conclusion. A violation is an unsupported conclusion or recommended action. All reflection contrasts are reflection minus summary in percentage points. For report outcomes, we resampled by case with pairs intact.

Blinded sampleReports / pairsOutcomeReflectionSummaryΔ, pp [95% CI]
Original, both models (11 cases)72 / 36Semantically correct ↑27/3626/36+2.8 [−12.8, 16.2]
Report violations ↓5/365/360.0 [−14.7, 15.8]
DeepSeek, shared ceiling (4 new cases)16 / 8Semantically correct ↑5/85/80.0 [0, 0]
Report violations ↓1/81/80.0 [0, 0]

The original intervals span zero. The follow-up's zero-width intervals reflect identical judgments within every sampled pair, with one rating per report. The effect at equal total compute remains open (Section 4.5 and Appendix O.3, Tables 44–45).

W5. When an exploratory calculation is defensible

A report can make a sound judgment after an agent runs a calculation whose prerequisites failed. The headline NO-GO rate uses strict scoring, which asks whether the agent respected the prescribed workflow in execution. Study 2 asks separately whether the agent withheld an unsupported conclusion, used a valid alternative, or made a claim the evidence did not support. For example, once equal variances are unsupported, a declared Welch test can be a valid alternative. Likewise, an expert can accept a report that withholds an unsupported conclusion even though the trace shows the excluded calculation.

Study 2 uses population weights for sampled NO-GO decisions and equal weights for the GO counterparts. Brackets give case-clustered 95% intervals. The GO–NO-GO gaps are in percentage points.

Study 2 measure (40 NO-GO, 20 GO, 11 agents)NO-GOGOGO − NO-GO
Strict route accuracy ↑17% [4, 33]90% [73, 100]73 [42, 96]
Route accuracy with expert credit for withholding or valid alternatives ↑55% [41, 71]100% (20/20)45 [29, 59]
Expert semantic correctness ↑50% [37, 66]75% [59, 89]25 [6, 42]
Reports acting on invalid evidence ↓47% [31, 61]30% [11, 48]—

Crediting defensible conduct substantially narrows the gap. The gap persists when experts judge semantic correctness directly and when both sides of each audited pair receive equal weight. Experts also found unsupported conclusions or recommendations at reached NO-GO decisions (Section 4.4, Table 4 and Appendix M).

Thank you for the detailed response. The rebuttal strengthens the evidence for the gap between knowing a rule and applying it, but prior work already demonstrates this gap empirically [1] and establishes concrete theoretical limitations on transformer attention and sequential computation [2, 3]. The paper adds limited fundamental understanding beyond these existing results, so I will maintain my score.

[1] Hong et al. (2026). Knowing but Not Doing: Discourse-level Constraints Adherence of Large Language Models. https://aclanthology.org/2026.findings-emnlp.1083/

[2] Veličković et al. (2025). Softmax is not Enough (for Sharp Size Generalisation). https://proceedings.mlr.press/v267/velickovic25a.html

[3] Merrill & Sabharwal (2023). The Parallelism Tradeoff: Limitations of Log-Precision Transformers. https://aclanthology.org/2023.tacl-1.31/

Thank you for your prompt response and for pointing us to these relevant works. We appreciate your assessment and agree that the gap between knowing a rule and applying it has been studied in prior research. We also acknowledge your concern that our work provides limited new fundamental insight into this gap.

Our primary aim is to contribute a benchmark for evaluating evidence-dependent decisions in realistic, multi-stage analytical workflows. To construct the dataset, our team visited 15 large-scale manufacturing facilities and collected cases based on the analytical workflows followed by human engineers. Developing each case typically took two to three months. By providing paired task variants, execution-level evaluation, and expert assessment, we hope DMAIC-Bench can serve as a shared testbed for studying when agents follow, redirect, or stop a workflow in response to intermediate evidence.

We appreciate the connection you have drawn to prior empirical and theoretical work. We hope this clarifies the intended contribution of our work and that our benchmark will help researchers evaluate their methods more efficiently and accelerate future research.

Thank you again for your time and constructive feedback.

The Authors

Reviewer_T7pYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DMAIC-Bench to evaluate whether LLM agents can use intermediate evidence to decide when to continue, redirect, or stop during multi-stage analytical tasks. The benchmark contains 97 tasks across 15 industrial cases and uses paired counterfactual scenarios to distinguish justified continuation from stopping. It evaluates agents through execution traces and independent expert review. The paper identifies a substantial gap between agents' ability to perform analyses and their ability to act appropriately on diagnostic results.

Critical strengths

1. Clear Evaluation of Evidence-Dependent Agent Behavior

The paper distinguishes between correctly interpreting intermediate results and making appropriate decisions based on them. This distinction is important for evaluating the reliability of multi-stage analytical agents. The use of paired counterfactual tasks helps assess whether agents can appropriately continue, redirect, or stop based on different diagnostic outcomes.

2. Extensive Experiments and Expert Validation

The paper evaluates 13 models across 97 tasks with repeated attempts. It also includes instruction-based interventions and blinded expert assessments. By distinguishing execution errors from unsupported conclusions in final reports, the experiments provide useful insights into the limitations of current agents and strengthen the credibility of the findings.

Critical weaknesses

1. Potentially Overly Strict Evaluation Criteria

The benchmark treats forbidden downstream computations as routing errors even when agents explicitly acknowledge their limitations and avoid unsupported conclusions. I am concerned that this may overestimate the severity of self-restraint failures. The expert-based evaluation shows a substantially smaller gap between GO and NO-GO decisions than the strict execution metric. In my view, this distinction is important for interpreting the paper's main findings.

2. Limited Generalization Beyond the Evaluated Workflows

Although DMAIC-Bench contains 97 tasks, they are derived from only 15 underlying cases with repeated workflow structures. The reserved held-out cases are not used to establish generalization to unseen workflows. The additional life-sciences evaluation is also relatively small. I believe stronger evidence is needed to determine whether the observed failures generalize beyond these specific analytical settings.

Questions for authors
  1. How would the main findings change if expert-accepted exploratory computations were treated as valid rather than routing errors?

  2. Can the authors clarify how much of the observed NO-GO failure rate may be influenced by task instructions that appear to require analyses disallowed by failed prerequisites?

  3. Would the observed self-restraint gap remain consistent on genuinely unseen workflow structures or a larger set of non-DMAIC analytical tasks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer T7pY

We thank the reviewer for the clear questions about scoring, instructions, and generalization.

W1 & Q1: Crediting Expert-Accepted Computations

Crediting expert-accepted withholding and alternatives raises audited NO-GO accuracy from 17% to 55%. The Study-2 audit of 40 NO-GO and 20 paired GO units reports three criteria and report violations (Appendix M) and we add reports acting on invalid evidence:

MeasureNO-GOGOGO − NO-GO
Strict route accuracy ↑17% [4, 33]90% [73, 100]73 pp [42, 96]
Crediting expert-accepted withholding or valid alternatives ↑55% [41, 71]100% (20/20)45 pp [29, 59]
Expert semantic correctness ↑50% [37, 66]75% [59, 89]25 pp [6, 42]
Reports acting on invalid evidence ↓47% [31, 61]30% [11, 48]—

The semantic gap is smaller, at 25 pp, and contextualizes the strict execution result. Separately, the 47% estimate shows reports acting on invalid evidence at reached NO-GO decisions in the population of 11 audited agents. We have incorporated these results into Section 4.4.

Q2: Instruction Conflicts

Of the 40 matched NO-GO forks, the first coder identified 27 explicit instruction conflicts, while a blinded second coder flagged 38 as at least borderline (κ=0.752\kappa = 0.752 for explicit vs. other). Because these conflict categories are confounded with workflow stage—making subgroup accuracy comparisons inconclusive—we directly tested your suggested conditional brief across 16 matched pairs using identical wording for both GO and NO-GO variants (proceed when prerequisites hold; otherwise identify the failed prerequisites and unsupported conclusions):

ModelNovember Plain (NO-GO / GO)Conditional Brief (NO-GO / GO)
GPT-6-astra26.3% / 81.2%38.6% / 79.5%
DeepSeek-v4-pro15.8% / 76.7%25.0% / 82.0%

Although explicit conditional framing yields a descriptive increase in NO-GO accuracy, substantial gate-compliance errors persist. We have incorporated both the conflict coding and this conditional-brief control into Section 4.5 to clarify how explicit conditional instructions affect routing behavior.

W2 & Q3: Unseen Workflows and Non-DMAIC Tasks

We added two unseen non-DMAIC cases (16 tasks) involving observational overlap and diagnostic accuracy with partial verification. With threshold-only briefs and three attempts per task:

ModelNO-GOGODIVERT
GPT-6-astra81%97%33%
GPT-5.6-sol48%96%24%
GPT-5.6-luna22%86%19%
DeepSeek-v4-pro22%94%10%

We also added eight real-public-data cases (59 tasks) spanning medicine, evidence synthesis, climate, hydrology, finance, econometrics, and psychometrics in Appendix N.2, with a cross-reference in Appendix M. GPT-6-astra obtained:

Brief / reasoning settingAttemptsNO-GOGODIVERT
Threshold-only / default337/57 (65%)258/321 (80%)27/60 (45%)
Threshold-only / maximum340/57 (70%)257/321 (80%)30/60 (50%)
Explicit prerequisites / default116/19 (84%)88/107 (82%)10/20 (50%)

These additions demonstrate recurrence across diverse analytical domains, showing how failure rates vary across workflows, instructions, and models. The original five-case/32-task partition was used during development and is reported descriptively.

Thank you again for your helpful questions and suggestions.

Thank you for the detailed response and the additional experiments on evaluation criteria, instruction conflicts, and unseen workflows. The new results have addressed my main concerns and helped clarify the findings.

I appreciate the effort and have no further questions. I will maintain my positive rating.

Good luck!

AuthorsOpenReview ↗

Thank you for your positive feedback

Dear Reviewer T7pY,

Thank you very much for your kind words and positive feedback. We sincerely appreciate your time in reviewing our responses and your continued support of our work. Your constructive suggestions have been valuable in helping us improve the paper.

Best,

Authors

Reviewer_QArQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DMAIC-Bench evaluates whether agents use intermediate statistical evidence to continue, divert, or stop an analytical workflow. It contains 97 seeded tasks across 15 industrial cases in four sectors. Forty of 66 graded forks have GO and NO-GO variants at the same workflow position. A data steward controls access to batches, while a private verifier evaluates submitted traces against reference routes. Under the strict execution criterion, performing a forbidden downstream computation counts as a route error even if the report withholds its conclusion.

Across 13 models, matched-fork accuracy averages 92% on GO decisions and 19% on NO-GO decisions. Separately, a blinded expert audit estimates that reports act on invalid evidence at 47% of reached NO-GO decisions in the population of 11 audited agents, with an interval of 31% to 61%. The paper also evaluates repeated-attempt reliability, stopping-permission and rule-based interventions, and a small extension to RNA-seq and GWAS workflows.

The contribution is a useful benchmark and a substantial empirical study of analytical routing. The main interpretive challenge is distinguishing compliance with the benchmark's prescribed workflow from scientifically warranted analysis and from action taken after an agent has actually observed diagnostic evidence.

Critical strengths
  1. The paired design is useful and directly addresses indiscriminate stopping. GO and NO-GO variants at the same workflow position require different behavior, so an always-stop or always-continue policy cannot succeed on both sides. Diversion paths also test whether an agent can redirect its analysis instead of merely terminating. The verified-value subset distinguishes numerical diagnostic errors from cases where correct reported values coexist with an incorrect reference route.

  2. The expert validation provides meaningful evidence beyond agreement between the generator and its reference solver. Three certified Black Belts reviewed blinded units in two studies, with the second conducted under a frozen protocol. The paper reports agreement, preserves disagreements, provides sampling weights and uncertainty, and examines sensitivity to the sampled agent mix. The separate assessment of final reports is especially valuable because execution violations and unsupported conclusions need not coincide.

  3. The evaluation is substantial and unusually transparent about its limitations. It includes 13 models, three attempts per task, case-clustered intervals, reward-aggregation sensitivity, and a 960-trial instruction-control study. The authors report the GO-side costs of interventions, document scorer corrections, and explicitly limit the interpretation of the cross-domain pilot. The disagreement between LLM judges and human experts is also a useful finding for future automated evaluation.

Critical weaknesses
  1. The strict execution score and professionally warranted analytical behavior are materially different evaluation targets.

The scorer marks a computation on an excluded branch as a route error even when the report withholds the corresponding conclusion or uses an alternative accepted by experts. This distinction matters empirically: experts judged 15 of 32 sampled NO-GO route failures acceptable. In Table 33, crediting expert-confirmed withholding and valid alternatives raises NO-GO accuracy from 17% to 55%. On the same audited sample, the GO–NO-GO difference is 73 percentage points under strict route scoring and 25 points under expert semantic correctness.

These are different assessment criteria, not competing estimates of the same quantity. The authors acknowledge this clearly. Nevertheless, the headline execution gap can easily be read as a broader deficit in scientifically defensible decision making. The paper should make its intended target more explicit: preventing specified computations is a narrower policy than preventing unsupported conclusions.

The separate 47% expert-assessed report-violation estimate remains important evidence of substantive analytical failures. I would give that result greater prominence when discussing the practical consequences of the observed behavior.

  1. The aggregate temporal analysis does not establish that agents observed failed diagnostics before selecting their next actions.

The paper highlights 174 errors in which a correct failing diagnostic precedes the forbidden analysis. Appendix K explains that this order comes from the agent-written trace.json. The steward independently records data access, but not local computation, and one previously issued script can compute both the diagnostic and the downstream analysis before the agent receives either result.

The count therefore establishes ordering in the submitted trace, rather than 174 independently verified opportunities to revise an action after receiving diagnostic evidence. The worked transcripts do contain examples of agents acknowledging a failed prerequisite before continuing, so such behavior is not merely hypothetical. However, the aggregate interpretation requires checking the retained terminal recordings.

This limitation is acknowledged in the paper, but it is particularly important because the proposed capability concerns whether evidence governs the next action. The expert report audit and the 174-case trace analysis should also remain separate: the paper does not establish that they measure the same subset of decisions.

  1. Task interpretation and benchmark-specific conventions remain relevant confounds.

Case 33 provides a documented instruction conflict. Its Plain brief requests assessments of lot stability and the current reject level, while the NO-GO reference route prohibits the corresponding computations after the measurement study fails. A valid response could explain why those assessments cannot be supported, but that interpretation is less explicit than in the Permit-stop condition. The prevalence of similar conflicts across the corpus is not reported.

Some guard rules also encode fixed conventions rather than the full range of defensible statistical judgment. Appendix A uses only Nelson test 1 for stability decisions and does not let run or trend signals change the route. Appendix M records cases where experts favored stopping or investigating despite the benchmark's GO label. Additional control-chart rules involve trade-offs, so selecting one convention is not inherently erroneous. However, agents should either receive the relevant convention explicitly or have a clear way to receive credit for defensible alternatives.

The permission, normative, and reminder conditions are valuable controls. Their results establish heterogeneous responses to the tested instructions. They do not fully isolate failure to apply an understood prerequisite from uncertainty about the task contract or the benchmark's exact policy.

  1. The aggregate leaderboard should be distinguished more clearly from restraint-specific performance.

Mean reward combines method recall, method precision, numerical fidelity, and routing accuracy. It is a reasonable summary of overall workflow performance, but should not be interpreted as a pure restraint ranking. Table 21 already reports the more direct measure of matched forks solved correctly on both sides; presenting it alongside the main leaderboard would better connect the reported ranking to the central construct.

Individual NO-GO estimates have substantial uncertainty. Fine-grained comparisons should use paired between-model differences under case resampling. Overlap between individual confidence intervals would not, by itself, establish that models are indistinguishable. The unused held-out split and small cross-domain pilot further limit conclusions about transfer to new workflow graphs, as the authors acknowledge.

I recommend weak acceptance because the paired benchmark, substantial evaluation, and independent expert audit provide a useful contribution. The limitations above narrow the interpretation of the findings, especially claims about observed evidence-dependent action. They appear addressable through clearer framing, targeted validation of retained recordings, and better alignment between task instructions and scoring conventions.

Questions for authors
  1. Can you validate a sample of the 174 temporally ordered errors against the retained terminal recordings? Please distinguish computations performed within one precommitted script from a new action issued after the failed diagnostic was returned to the agent. The same audit could assess agreement between actual computations and the submitted trace.

  2. Please classify the existing tasks by whether their requests explicitly solicit analyses excluded by the NO-GO reference route, and report the corresponding results. For a focused instruction control, use the same conditional brief for both members of each pair: perform downstream assessments when their prerequisites hold; otherwise identify the failed prerequisites and explain which conclusions cannot be supported. This would clarify the task contract while preserving the GO objectives.

  3. Which guards are mandatory prerequisites, which are fixed benchmark conventions, and which admit context-dependent alternatives? Please explain how the scoring policy handles the expert-accepted analyses documented in Table 33, and quantify sensitivity to the additional stability signals identified in the expert audit.

  4. Can the matched-fork results be presented with a common weighting scheme across GO and NO-GO variants? The main comparison pools 146 GO and 52 NO-GO instances across 40 forks, while the expert comparison uses population-weighted NO-GO units and unweighted GO twins. A fork-balanced contrast would clarify the matched interpretation. Please also foreground the paired-fork success measure already reported in Table 21 alongside aggregate reward.

  5. Please sharpen the comparison with AgentAbstain and Agentic Abstention in Table 1. Both already examine evidence-dependent stopping after interaction, and AgentAbstain includes paired variants and trace-based checks. What precise definition of “dynamic route” motivates their entries? The distinctive contribution here appears to be computed statistical prerequisites within multistage analytical workflows. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (Kirichenko et al., NeurIPS 2025) would also provide useful background for the broader abstention discussion.

Minor reporting points: Section 4.5 appears to omit the minus signs in the interval corresponding to the 42-point GO decrease; Table 20 gives −65 to −14. Table 31's GO conduct percentages are 70/20/5, summing to 95%; please identify the remaining category or correct the entry.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QArQ (1/4)

Thank you for the detailed review. We report the requested recording audit, conditional brief, and equal-weight comparisons below.

W1. Execution compliance and scientifically warranted analysis

The headline 19% is the pooled NO-GO route accuracy at matched forks in the main evaluation across 13 models (Section 4.2). The 17% in Table 4 is the design-weighted strict route accuracy for Study 2's audited NO-GO sample. In both analyses, computing on an excluded branch counts as a route error even if the report withholds the unsupported conclusion. We now foreground the blinded audit of unsupported conclusions and recommended actions in the abstract, introduction, and Section 4.4.

Study 2 measureNO-GOGOGO − NO-GO, pp
Strict route accuracy ↑17% [4, 33]90% [73, 100]73 [42, 96]
Route accuracy with expert credit for withholding or valid alternatives ↑55% [41, 71]100% (20/20)45 [29, 59]
Expert semantic correctness ↑50% [37, 66]75% [59, 89]25 [6, 42]
Reports acting on invalid evidence ↓47% [31, 61]30% [11, 48]—
Study 2 scopeUnits or estimate
Blinded audit40 NO-GO units and 20 GO counterparts from 12 cases and 11 agents
Primary NO-GO weightingPopulation weights specified in the Study 2 sampling design (Appendix M, Table 32)
GO weightingEqual weights on the GO counterparts
Strict NO-GO route failures accepted under expert rescoring15/32 sampled failures
NO-GO report violations, sensitivity to agent mix46% [27, 55], after reweighting agents by their shares of reached NO-GO decisions
UncertaintyCase-clustered 95% confidence intervals in brackets

The primary report-violation estimate is 47% [31, 61]. Each NO-GO unit's weight is its stratum’s population size divided by its sample size. The strata were fixed before expert review by failure mechanism and benchmark scorer verdict (Table 32). The sensitivity analysis uses the same ratings and additionally reweights each agent to its share of reached NO-GO decisions in the audit population, yielding 46% [27, 55] (Appendix M).

Appendix M illustrates the practical distinction. One report warned against treating an unstable baseline as a single process, then reused the pooled Ppk, a process performance index, as a long-term performance conclusion. Another preserved the measurement caveat and withheld a claim that the current process was acceptable.

The GO–NO-GO gap narrows when we credit defensible conduct and persists under direct expert judgment (Section 4.4, Table 4).

W2 & Q1. What the terminal recordings establish

The verifier confirmed a reported failing diagnostic in 201 NO-GO route errors. Of those, 174 had the diagnostic listed before the excluded analysis in the submitted trace. These 174 cases are the denominator for the recording audit. We classified all of them using the retained terminal recordings. Within one model turn, commands are issued before any output returns. We therefore separated a single prewritten script that runs both analyses from a new command issued after the failing result was returned, when the agent had an opportunity to revise its plan.

Sequence in terminal recordingsClassifier countEstimate from blinded sample, weighted to all 174
Excluded analysis issued after the failing value returned, in a later turn106104.8 [95.3, 111.0]
Diagnostic and excluded analysis in one prewritten script6365.1 [55.5, 76.7]
Excluded analysis ran before the failing value existed54.2 [0.0, 11.4]
Later-turn cases that restated the failing value first86 of 106—

A DMAIC engineer independently coded a stratified sample for timing and trace agreement, blinded to the model and classifier label. Every continuation the engineer placed in a later turn contained an acknowledgment of the failed prerequisite before the agent ran the excluded analysis.

Validation checkResult
Blinded reviewer matched the timing classifier44/50 (88%), Cohen's κ = 0.785
Later-turn cases with explicit acknowledgment of the failed prerequisite27/27
Later-turn cases that reran an unchanged script0/27
Submitted trace matched the recorded computation in the reviewed sample50/50
Diagnostic and downstream trace entries with all decimal outputs visible in recordings434/550

Section 4.3 bases its timing claim on the later-turn cases in Appendix K.1, Table 28. Study 2 evaluates report claims in a separate audit sample (Appendix M).

AuthorsOpenReview ↗

Response to Reviewer QArQ (2/4)

W3 & Q2. Instruction conflicts and the conditional brief

In Case 33, Plain asks for lot stability and reject levels. A failed inspector-agreement retest instead warrants an explanation and escalation to instrumented inspection. At each matched NO-GO fork, we coded whether the original brief explicitly asks for an analysis excluded by the reference route.

Coding of 40 matched NO-GO forksExplicit conflictBorderlineNone
First coder27211
Blinded second coder3172
Agreement or outcomeEstimate
Agreement, explicit conflict versus otherκ = 0.752
Agreement, all three categoriesκ = 0.395
NO-GO accuracy, explicit conflict under blinded coding14.8% (77/520)
NO-GO accuracy, all other forks under blinded coding32.7% (51/156)
Other minus explicit, pp [95% CI]+17.9 [−15.5, 54.7]

The descriptive interval spans zero, and conflict labels vary with workflow stage. We tested the requested wording by adding the same instruction to both variants of each pair. Agents should perform downstream assessments when prerequisites hold, and otherwise identify the failed prerequisites and explain which conclusions the evidence cannot support. The goals and data stayed fixed, and the wording did not reveal the outcome of any prerequisite.

November Plain used two attempts per task. The conditional arm targeted three. Six DeepSeek conditional runs failed at the model endpoint and produced no scorecard. All 32 tasks remained represented in both arms despite the six failed DeepSeek conditional runs.

Model on 32 tasks (16 pairs)Scored trials, Plain → conditionalNO-GO, Plain → conditionalNO-GO Δ [95% CI]GO, Plain → conditionalGO Δ [95% CI]
GPT-6-astra64 → 9626.3% → 38.6%+12.3 [−7.7, 37.8]81.2% → 79.5%−1.7 [−8.0, 2.5]
DeepSeek-v4-pro64 → 9015.8% → 25.0%+9.2 [1.9, 17.0]76.7% → 82.0%+5.3 [0.3, 11.8]

The differences are in percentage points relative to November Plain. The 95% intervals use the paper’s case bootstrap, with the same case draws in both arms. The arms differ in scored attempts, so we report the comparison as a descriptive instruction control. The DeepSeek NO-GO interval excludes zero against November Plain and includes zero against September Plain, the baseline in Table 46. The separate rule probes show routing errors even at checkpoints with correct probe answers (Section 4.5 and Appendices H.1 and M.1).

AuthorsOpenReview ↗

Response to Reviewer QArQ (3/4)

W3 & Q3. Prerequisites, benchmark conventions, and valid alternatives

The manuscript now separates the analytical requirement from the convention that implements it. A conventional capability analysis requires a stable process. A threshold or a set of control-chart signals determines when the benchmark treats that requirement as satisfied. We use a fixed convention in the main score so paired routes can be checked consistently, and we report how the results change under a broader set of signals.

What the guard representsExampleHow the evaluation handles it
Analytical prerequisiteReliable measurement before dependent claims. Stability before interpreting standard capability indices.A failed prerequisite closes the dependent route. The graph specifies whether the agent should stop or run a valid alternative.
Fixed benchmark conventionNelson Test 1 for stability. A distinct-categories cutoff of ndc ≥ 5 in the compound gauge gate.The reference route uses the documented rule. The stability sensitivity analysis reruns the scoring under alternate rules.
Valid analytical alternativeA declared Welch test when equal variances are unsupported.The execution scorer credits that declared alternative. Table 33 separately reports credit based on expert review of the report.

Rule disclosure also varies across the controls. In Plain, the steward can reveal the predicate on denial of a locked batch. The normative brief states the qualitative prerequisite. The reminder condition publishes the numerical predicate through the steward. The conditional brief makes the task contract explicit, and the supplied-evidence condition gives the diagnostic values. Distinguishing these interventions lets readers see which information and instruction each comparison changes (Appendices D, H.1, M.1 and O.1).

Table 33 gives a NO-GO route error credit when the expert majority finds that the report drew neither an unsupported conclusion nor an unsupported action recommendation from the invalid evidence. That credit can apply when the trace still contains the computation. On GO, a route error receives credit if the expert majority judges the conduct acceptable. Expert semantic correctness is a separate judgment of the focal decision and the report's treatment of the evidence. Table 4 shows all three criteria alongside the report-violation result.

Stability ruleGO instances relabeled NO-GOContinued on relabeled decisionsPooled NO-GOPooled GO
Nelson Test 1——18.9%79.4%
Nelson Tests 1–820/68246/26014.2%78.1%

Under the broader rule, pooled NO-GO accuracy falls and remains below GO accuracy. The sensitivity analysis changes the scoring of existing runs under the expanded convention (Appendix O.1, Table 43).

Reviewer_YG8yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors test whether AI agents know when to stop during multi-step data analysis, a skill they call "self-restraint". They build DMAIC-Bench, with 97 industrial tasks, many in matched pairs where the right move is to continue in one version and to stop in the other. Across 13 models, agents continued correctly about 92% of the time but stopped correctly only about 19% of the time by a strict count, often running a forbidden analysis right after computing the warning sign. Expert review found that 47% of reports in the sampled stop cases still relied on invalid evidence, and telling models they may stop helped some while hurting their ability to continue correctly.

Critical strengths
  1. This paper looks into an interesting problem in an agentic evaluation set up. Most agent tests only check whether the task was finished or the final answer looked good. This paper checks whether the agent’s next step follows from what its own calculations showed.
  2. The evaluation is thorough.
  3. The paper supports reproducibility -- case generators, guards, scorer, audit records, and regeneration scripts are released.
  4. Useful benchmark on an interesting problem
Critical weaknesses
  1. The headline 19% depends on a very strict rule. Any forbidden calculation counts as a mistake, even if the final report does not use it. How does it reflect a real scenario?
  2. The sample size is somewhat small and the rules are one reading of the standards. There are only 15 cases, 52 stop decisions, and 40 matched pairs, so the ranges are wide (stop accuracy between 8% and 30%).
  3. Model comparisons are not fully controlled. Temperature and reasoning effort were left at each provider’s defaults, and each task was run only three times.
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Reviewer YG8y

Thank you for recognizing the importance of the problem and for raising these concerns:

W1: Strict Scoring and Practical Relevance

Strict route scoring tests whether agents respect analytical prerequisites during execution. In manufacturing, a failed stability check should trigger an investigation before the analyst proceeds with conventional capability analysis. Running the calculation anyway can waste analysis time and delay the investigation, even if the final report omits the result. If a later decision relies on that number, the team could make unnecessary machine adjustments or prematurely accept a process that still needs investigation. In one audited report, the agent calculated Ppk, a process performance index that compares the distance from the process mean to the nearest specification limit with overall process variation. The report used that value as a control target while measurement error remained unresolved (Appendix M, Table 39).

The headline 19% NO-GO accuracy measures whether agents took the warranted route during execution. Study 2 separately assessed the reports, auditing 40 NO-GO units and 20 matched GO counterparts from 11 agents. NO-GO estimates use population weights, and GO units receive equal weight. Brackets show case-clustered 95% confidence intervals.

MeasureNO-GOGOGO − NO-GO
Strict route accuracy ↑17% [4, 33]90% [73, 100]73 pp [42, 96]
Crediting expert-accepted withholding or valid alternatives ↑55% [41, 71]100% (20/20)45 pp [29, 59]
Expert semantic correctness ↑50% [37, 66]75% [59, 89]25 pp [6, 42]
Reports acting on invalid evidence ↓47% [31, 61]30% [11, 48]—

The gap narrowed under direct expert assessment, and experts still found unsupported conclusions or recommended actions based on invalid evidence at reached NO-GO decisions (Section 4.4, Table 4).

W2: Sample Size and Statistical Conventions

Building a case requires source material that documents both the workflow and the intermediate evidence that determines which analysis should follow. Records with that level of detail were extreme scarce. The 15 cases span four industrial sectors. Eight retain decision routes from completed Black Belt projects, and seven use standards-derived routes in project-informed scenarios (Appendix B). We checked source quantities and formulas and verified the decision graphs, data generators, and reference solutions. Certified Black Belt experts independently reviewed 30 blinded checkpoints covering all 15 cases. Their feedback led to scoring revisions, including credit for a declared Welch test when equal variances were unsupported (Appendices C and M).

Each case has five to eight seeded variants, yielding 97 multi-stage tasks. At a matched fork, the variants change the diagnostic evidence while keeping the workflow position fixed, so the warranted action can reverse. Each of the 13 agents ran every task three times. The 52 NO-GO decisions and 40 matched forks share the same 15 underlying workflows, so we resample by case. The 95% interval for matched NO-GO accuracy is wide at 8–30%, and a cluster-robust check gives a similar interval of 7–31% (Appendix I). The GO–NO-GO gap persists when each matched fork receives equal weight.

The GO–NO-GO gap also appeared in the cross-domain evaluation. Under thresholds-only briefs, GO accuracy exceeded NO-GO accuracy for every model tested on X6 and for GPT-6-astra at both reasoning settings on the public-data X7 cases (Section 4.6 and Appendix N.2, Tables 41–42). We rescored the same runs using Nelson Tests 1–8 in place of Test 1. Pooled NO-GO accuracy fell, and GO accuracy remained higher under both conventions (Appendix O.1, Table 43).

W3: Temperature, Reasoning Effort, and Repeated Runs

The main evaluation used provider defaults. To test how sensitive the GO–NO-GO gap was to those settings, we ran additional controls on the same 32 tasks (16 matched pairs). We varied temperature and reasoning effort for GPT-6-astra and tested maximum reasoning effort for DeepSeek-v4-pro:

ModelSettingAttemptsNO-GOGO
GPT-6-astraNovember default rerun226.3%81.2%
GPT-6-astraTemperature 0523.2%81.1%
GPT-6-astraTemperature 0.5322.8%81.4%
GPT-6-astraHigh reasoning effort522.1%82.7%
GPT-6-astraMaximum reasoning effort324.6%84.5%
DeepSeek-v4-proNovember default rerun215.8%76.7%
DeepSeek-v4-proMaximum reasoning effort315.8%79.2%

GO accuracy remained higher than NO-GO accuracy in every tested configuration, including both five-attempt conditions. Section 4.5 points to the full results in Appendix Q, Table 46.

Thank you again for your feedback, which has helped us sharpen our evaluation framing and experimental controls.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewers,

We sincerely appreciate your time and thoughtful feedback on our work. Following your constructive suggestions, we have incorporated additional analyses and extended experiments, provided detailed responses to each point, and revised the manuscript to thoroughly address all concerns.

With the discussion period approaching its end, we kindly invite you to review our rebuttal and updated version when convenient. We hope our revisions have satisfactorily resolved your questions, and we remain happy to address any remaining points.

Thank you once again for helping us improve our paper.

Best regards,

The Authors

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their careful evaluation and constructive discussion. During the rebuttal, we added experiments, recording validation, and expert assessments to strengthen our central contribution: evaluating whether intermediate statistical evidence governs an agent’s decision to continue, divert, or stop an analysis. The main updates are:

  1. Execution compliance and practical errors. We separately report strict routing, expert-accepted alternatives, and semantic correctness. Crediting defensible alternatives raises audited NO-GO accuracy from 17% to 55%. Separately, experts estimate that reports act on invalid evidence at 47% of reached NO-GO decisions in the audited-agent population.
  2. Evidence and subsequent actions. We examined all 174 relevant terminal recordings. All 27 human-coded later-turn continuations explicitly acknowledged the failed prerequisite before proceeding. Separate rule probes further show that correct rule answers do not guarantee correct workflow execution.
  3. Additional controls and baselines. We added conditional briefs, supplied-diagnostic comparisons, reflection and reasoning-effort controls, deterministic baselines, and common-weight analyses. Blinded review of 44 matched report pairs found no consistent improvement from prerequisite reflection over neutral summarization.
  4. Broader evaluation. We added two unseen non-DMAIC cases and eight real-public-data cases, showing that stopping and diversion errors recur beyond industrial workflows. We also report successful conditions and variation across models and instructions. Together, these additions strengthen DMAIC-Bench as a reproducible evaluation that connects intermediate evidence, executed routes, and the validity of final conclusions, supporting improvements to analytical agents and prerequisite safeguards.

We appreciate Reviewer T7pY’s confirmation that the additional evidence addressed their main concerns, Reviewer QArQ’s detailed methodological guidance, and Reviewers YG8y, aG5v, and w5kR’s discussion of evaluation, positioning, and practical value. Their feedback has substantially improved this work.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.