acceptodds
Desk rejected at ICLR 2027

ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Outcome-based evaluation of LLM coding agents provides limited information about defects that arise during execution. We present ProcCtrlBench, a benchmark for execution-process evaluation in LLM coding agents. ProcCtrlBench organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories, and evaluates agent trajectories through standardized process evidence rather than final outcomes alone. To support cross-platform comparison, ProcCtrlBench standardizes raw logs into a shared trajectory representation and reports calibrated scorecards over process-level findings. In addition, ProcCtrlBench uses control preservation to quantify execution-process quality, capturing whether execution remains interpretable, interruptible, correctable, reversible, and able to hand back authority when needed. We evaluate ProcCtrlBench on 200 execution trajectories sampled from three benchmarks, corresponding to 2,200 defect-level evaluation points under the full 11-defect annotation protocol. Results show that ProcCtrlBench can be instantiated with useful reliability, provides more stable semantics than direct thresholding, and reveals distinct differences in execution quality often overlooked by conventional outcome-based metrics. Our code are available at https://anonymous.4open.science/r/ProcCtrlBench-655C.

Primary area:
datasets and benchmarks
Keywords:
benchmark, process-level evaluation, control preservation, reliability assessment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P8CLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ProcCtrlBench, a proposed process-oriented benchmark for LLM coding agents. It standardizes heterogeneous execution logs into trajectories, labels eleven execution-defect types across four categories, calibrates detector outputs into defect risks, and combines defect burden with a five-part “control preservation” scorecard.

The paper evaluates the framework on 200 trajectories from AndroidBench, Terminal-Bench, and SWE-bench-Verified, reporting defect-detection metrics, calibration metrics, and system-level process scores for eleven agent-model configurations.

Critical strengths
  • The paper identifies a real limitation of endpoint-only coding-agent evaluation. The distinction between a task being solved and a task being solved through a repetitive, brittle, opaque, or difficult-to-supervise trajectory is important. The defect taxonomy in Section 3.3 provides a useful vocabulary for discussing recurrent process issues, including Duplicate Step, Dead Step, Long Chain, Context Window Thrashing, and Context Coupling.
  • Figure 1 gives a clear conceptual overview of the intended pipeline, from raw platform-specific logs through a unified trajectory representation, evidence extraction, calibration, and a multi-dimensional scorecard. In particular, separating evidence extraction from risk calibration is preferable to treating every heuristic threshold as an inherently meaningful binary defect decision.
  • The calibration motivation is sensible. Table 1 reports lower ECE for the Bayesian approach than for hard thresholding in each reported category, for example 0.214→0.118 for Context Management and 0.271→0.196 for Workflow Architecture. Figure 3 also usefully checks that the benchmark is not merely a raw duplicate count, since several cross-dimension posterior-risk correlations appear weaker than within-efficiency correlations.
  • The authors include limitations rather than presenting the scorecard as a complete assessment of coding-agent capability or safety or .
Critical weaknesses
  • The benchmark is not sufficiently specified to reproduce or interpret its scores. Section 3 leaves the feature extractors, detectors, calibration model, control-preservation function, defect-quality aggregation, and mixing weight abstract. Detector rules, normalization, weights, missing-data treatment, and exemption handling are not given in the main paper. Consequently, the fine-grained system differences in Table 2 cannot be attributed confidently to agent behavior rather than scoring choices.
  • The validity of control preservation and cross-system comparability is unclear. Passive logs alone do not establish interruptibility, correctability, reversibility, or authority handoff without operational proxies or intervention tests. Moreover, labels such as Oversized Rules, Inconsistent Tool Interface, and Weak Tool partly describe prompts and platform capabilities. Combining them with trajectory defects confounds agent-policy behavior, orchestration design, and tool quality across systems.
  • The evaluation protocol and statistical support are insufficiently clear. Section 4.1 specifies an 80/40/80 development/calibration/test split, but Tables 1 and 2 refer to the annotated set and a 200-trajectory mixed evaluation set, leaving possible in-sample evaluation unresolved. With 200 trajectories across 11 configurations, missing per-system/source counts, task overlap, and outcome distributions make rankings difficult to assess. Task-grouped splits, matched tasks, and paired uncertainty estimates are needed.
  • Calibration and detector reliability do not yet support precise scalar rankings. The hierarchical Bayesian likelihood, priors, and sharing structure are unspecified, while ECE for a hard-threshold baseline lacks a clear probability interpretation, binning protocol, and uncertainty estimates. Wrapper Workflow, Context Coupling, and Weak Tool have F1 scores of only 0.59, 0.61, and 0.53. Detector and annotation uncertainty should be propagated into system scores, with proper scoring rules and sensitivity analyses for low-reliability dimensions.
  • The claimed value beyond outcome metrics and simpler trajectory statistics is not established. The main paper lacks task-matched outcome comparisons, convincing trace examples, and baselines such as repeated-call rate, action count, or length-normalized cost. Correlations among risks do not establish nonredundant diagnostic value, and correlations with failure do not show that defects cause breakdown. Held-out ablations of the taxonomy, calibration, exemption logic, and control-preservation component are needed.
Questions for authors
  • Can you provide an executable specification of the features, detectors, thresholds, exemption rules, calibration model (including Bayesian priors and likelihoods), and all score aggregation functions and weights? Please distinguish learned components from hand-coded rules and identify what was fixed before evaluation.
  • Are Tables 1 and 2 and their uncertainty estimates computed exclusively on the 80 held-out trajectories? If not, can you report held-out results separately and clarify whether runs from the same task, repository issue, or benchmark instance can cross splits?
  • Can you provide per-system/per-source counts of trajectories, tasks, successes, and defect labels, and clarify whether systems used matched tasks, budgets, and tool access? How are agent-policy defects separated from prompt, orchestration, and tool-platform properties, and would a common tool environment change the comparison?
  • What observable evidence supports each control-preservation dimension, especially interruptibility, reversibility, and authority handoff? Can you validate these proxies through intervention tests such as interruption, checkpoint restoration, and handoff success, rather than relying only on passive traces?
  • How are probabilities, binning, rare defects, and exempt cases handled in the ECE comparison, and can you report prevalence, effective sample sizes, confidence intervals, and Brier score or log loss? Can you also show task-matched examples with similar endpoint success but different process quality, supported by raw traces and human rationale, to establish the diagnostic gain?
19 comments
AuthorsOpenReview ↗

Response to W1 [Part one]: Executable specification of features, detectors, calibration model, and score aggregation

Dear Reviewer P8CL,

We sincerely appreciate your thorough review and constructive feedback. Below, we address each of your weaknesses and questions with full mathematical specifications, strictly held-out evaluations, active intervention tests, deconfounding ablations, and task-matched trace examples incorporated into the revised manuscript

 Weakness 1 [Part one]: Executable specification of features, detectors, calibration model, and score aggregation

We thank the reviewer for pointing out the need for full mathematical transparency. We have added the executable specification in Section 3 (summarized in Data R6 below), distinguishing fixed hand-coded rules from learned calibration parameters: 

  1. Detectors, Exemptions & Missing-Data Treatment (Fixed on 80 Dev Split): Data R6 specifies the exact formula sj = fj(ϕj(T))s_j = f_j(\phi_j(T)), threshold τj\tau_j, and deterministic exemption rule exempt⁡j(T)\operatorname{exempt}_j(T) for all 11 defects. Hand-coded vs. Learned Separation: All feature extractors ϕj\phi_j, scoring rules fjf_j, thresholds τj\tau_j, exemption rules, and aggregation weights (η=0.70, wk=0.20\eta=0.70, w_k=0.20) are deterministic rules fixed on the 80 dev split before evaluation; only the hierarchical Bayesian calibration parameters (αj, βj, γc,d)(\alpha_j, \beta_j, \gamma_{c,d}) are learned (on the 40 cal split). Missing-Data Treatment: If a trajectory lacks the prerequisite structure for a defect (e.g., single-agent runs without subagent delegation for Context Coupling, or runs without wrapper sub-workflows W=∅W=\emptyset for Wrapper Workflow), the detector deterministically sets sj = 0s_j = 0 and exempt⁡j(T) = 1\operatorname{exempt}_j(T) = 1 (clamping s~i,j = −5.0\tilde{s}_{i,j} = -5.0 so posterior risk pi,j < 0.01p_{i,j} < 0.01, preventing false penalties while keeping dimension denominators ∣ Zd ∣\lvert Z_d \rvert invariant).

  2. Hierarchical Bayesian Calibration (Fit on 40 Cal Split): For defect jj in dimension d(j) ∈ {1..4}d(j) \in \{1..4\} under source c ∈ {1..3}c \in \{1..3\}, adjudicated presence yi,j ∼ Bernoulli⁡(pi,j)y_{i,j} \sim \operatorname{Bernoulli}(p_{i,j}) is fit via:

    logit⁡(pi,j) = αj + βj s~i,j + γc, d(j),  αj ∼ N(μα, d(j), σα2), \ βj ∼ N+(μβ, d(j), σβ2),\operatorname{logit}(p_{i,j}) = \alpha_j + \beta_j \tilde{s}_{i,j} + \gamma_{c, d(j)}, \quad \alpha_j \sim \mathcal{N}(\mu_{\alpha, d(j)}, \sigma_\alpha^2), \ \beta_j \sim \mathcal{N}^+(\mu_{\beta, d(j)}, \sigma_\beta^2),where s~i,j = (si,j − μdev, j)/σdev, j\tilde{s}_{i,j} = (s_{i,j} - \mu_{\text{dev}, j})/\sigma_{\text{dev}, j} is z-normalized using only the 80 dev trajectories, and deterministically clamped to s~i,j = −5.0\tilde{s}_{i,j} = -5.0 whenever exempt⁡j(Ti) = 1\operatorname{exempt}_j(T_i) = 1.

AuthorsOpenReview ↗

Response to W1 [Part two]: Executable specification of features, detectors, calibration model, and score aggregation

  1. Aggregation Weights (Fixed Before Evaluation): Dimension quality is Qd = 1 − 1∣ Zd ∣∑j ∈ Zd pjQ_d = 1 - \frac{1}{\lvert Z_d \rvert}\sum_{j \in Z_d} p_j; overall defect quality is Qdef = 0.25(Qctx + Qtool + Qwf + Qeco)Q_{\text{def}} = 0.25(Q_{\text{ctx}} + Q_{\text{tool}} + Q_{\text{wf}} + Q_{\text{eco}}); and PB⁡ = η Qdef + (1−η)CP⁡\operatorname{PB} = \eta Q_{\text{def}} + (1-\eta)\operatorname{CP} with η = 0.70\eta = 0.70.
Defect ClassContinuous Score Formula sj = fj(ϕj(T))s_j = f_j(\phi_j(T))τj\tau_{j}Deterministic Exemption Rule exemptj(T)exempt_{j}(T)
Ghost Context (GC)max⁡u ∈ T ρ(u) ⋅ (1 − κ(u)) ⋅ min⁡(1, τ(u)/10)\max_{u \in T} \rho(u) \cdot (1 - \kappa(u)) \cdot \min(1, \tau(u)/10)0.45Segment uu is a system prompt, pinned memory block, or active summary.
Oversized Rules (OR)Tokens(usys) / MaxContextBudget\text{Tokens}(u_{\text{sys}}) / \text{MaxContextBudget}0.25Task explicitly requires full API or schema specification injection in prefix.
CW Thrashing (CWT)1n ∑i I(Δ Ctxi > 0.35 ∧ Evicti+1 > 0.30) ⋅ RediscovRate\frac{1}{n} \sum_i \mathbb{I}(\Delta \text{Ctx}_i > 0.35 \wedge \text{Evict}_{i+1} > 0.30) \cdot \text{RediscovRate}0.30Multi-file batch scanning without subsequent re-read of evicted files.
Duplicate Step (DupS)max⁡i < k Sim(ui, uk) ⋅ I(Δ State(i, k) = ∅)\max_{i < k} \text{Sim}(u_i, u_k) \cdot \mathbb{I}(\Delta \text{State}(i, k) = \emptyset)0.80Intervening file edit om (i < m < k)o_m (i < m < k), test rerun, or polling command.
Tool Call Chain (TCC)max⁡w ⊂ T CycleScore(w) ⋅ (1 − InfoGain(w))\max_{w \subset T} \text{CycleScore}(w) \cdot (1 - \text{InfoGain}(w))0.65Intentional iterative pagination or directory tree traversal.
Dead Step (DeS)1n ∑i I(Ref(vi, T>i) = 0 ∧ Δ Branch(vi) = 0)\frac{1}{n} \sum_i \mathbb{I}(\text{Ref}(v_i, T_{>i}) = 0 \wedge \Delta \text{Branch}(v_i) = 0)0.35Initial exploratory status check (e.g., git status, ls) returning a clean state.
Long Chain (LC)max⁡(0, n − nexp(c)nexp(c)) ⋅ (1 − CkptRate(T))\max\left(0, \frac{n - n_{\text{exp}}(c)}{n_{\text{exp}}(c)}\right) \cdot (1 - \text{CkptRate}(T))0.50Multi-module refactoring task with verified intermediate test checkpoints.
Wrapper Workflow (WW)1∣ W ∣∑w ∈ W I(Steps⁡(w)=1 ∧ Val⁡(w)=0 ∧ Route⁡(w)=0)\frac{1}{\lvert W \rvert}\sum_{w \in W} \mathbb{I}(\operatorname{Steps}(w)=1 \land \operatorname{Val}(w)=0 \land \operatorname{Route}(w)=0)0.50Security or sandbox policy enforcement wrapper mandated by the platform.
Context Coupling (CC)\text{SCC_Ratio}(G_{\text{dep}}) \cdot \text{BidirMsgRate}(T)0.40Explicit pair-programming or critic-verifier dual-agent verification loop.
Incons. Tool Interf.SchemaDiv(Usim) ⋅ MalformedCallRate(T)\text{SchemaDiv}(U_{\text{sim}}) \cdot \text{MalformedCallRate}(T)0.35Native OS command flags differing across standard Unix utilities.
Weak Tool (WT)(1 − UseRate(u∗ ∣ App)) ⋅ FallbackSuboptRate(1 - \text{UseRate}(u^* \mid \text{App})) \cdot \text{FallbackSuboptRate}0.45Tool u∗u^* is restricted by environment permissions or missing dependencies.

Data R6: Executable specification of the 11 process-defect detectors and deterministic exemption rules. Complete Notation Legend: for context segment uu, ρ(u) ∈ [0,1]\rho(u) \in [0,1] is its context-window token share, κ(u) ∈ [0,1]\kappa(u) \in [0,1] is its subsequent AST/symbol reference rate, and τ(u)\tau(u) is persistence in steps; ΔCtxi\Delta\text{Ctx}_i and Evicti+1\text{Evict}_{i+1} are the fraction of context window added at step ii and evicted at step i+1i+1, while RediscoveryRate⁡(T)\operatorname{RediscoveryRate}(T) is the fraction of evicted files re-read later; Sim⁡(ui, uk)\operatorname{Sim}(u_i, u_k) is normalized Levenshtein similarity of tool arguments and ΔState⁡(i,k)\Delta\operatorname{State}(i,k) is the set of file/environment mutations between steps ii and kk; CycleScore⁡(w)\operatorname{CycleScore}(w) is the repetition frequency of tool-call n-grams in window ww and InfoGain⁡(w)\operatorname{InfoGain}(w) is the ratio of new AST symbols discovered; Ref⁡(vi, T>i)\operatorname{Ref}(v_i, T_{>i}) and ΔBranch⁡(vi)\Delta\operatorname{Branch}(v_i) indicate whether output viv_i is referenced in later steps or alters control flow; nexpected(c)n_{\text{expected}}(c) is the median step count of passing dev runs in source cc; SCCRatio⁡(Gdep)\operatorname{SCC_Ratio}(G_{\text{dep}}) is the fraction of nodes in strongly connected components of the delegation graph GdepG_{\text{dep}}; ParamSchemaDivergence⁡(Usim)\operatorname{ParamSchemaDivergence}(U_{\text{sim}}) is the Jensen-Shannon divergence of parameter keys across functionally equivalent tools UsimU_{\text{sim}}; and UseRate⁡(u∗ ∣ Applicable)\operatorname{UseRate}(u^* \mid \text{Applicable}) is the invocation rate of specialized AST/search tool u∗u^* when applicable.

AuthorsOpenReview ↗

Response to W2: Control Preservation Validation and Agent-Policy Deconfounding

Weakness 2: Validity of Control Preservation from passive logs and deconfounding agent policy from orchestration/tool properties

We thank the reviewer for raising this critical point and address both aspects directly:

1. Validating Control Preservation (CP⁡\operatorname{CP}) Beyond Passive Logs: Rather than assuming passive trace features imply governability, we specified closed-form observable equations for all five CP⁡\operatorname{CP} subdimensions (ψIntp, ψIntr, ψCorr, ψRev, ψHand\psi_{\text{Intp}}, \psi_{\text{Intr}}, \psi_{\text{Corr}}, \psi_{\text{Rev}}, \psi_{\text{Hand}}) and validated them through 44 active mid-trajectory intervention experiments (injected at step t = ⌊ 0.5n ⌋t = \lfloor 0.5n \rfloor: SIGINT pause/resume, synthetic tool-error injection, working-tree file corruption rollback, and forced authority handoff). Across all 44 runs, our trace-based CP⁡\operatorname{CP} scores strongly predict actual operational recovery and rollback success (Pearson r ∈ [0.74, 0.81], p < 0.001r \in [0.74, 0.81], p < 0.001; AUROC 0.81–0.880.81\text{--}0.88; see Question 4 below).

2. Deconfounding Agent Policy (QpolicyQ_{\text{policy}}) from Orchestration/Tool Properties (QinfraQ_{\text{infra}}): Because execution stability in long-horizon coding agents is co-determined by model policy and surrounding harness properties (OR, ITI, WW, WT), benchmarking both prevents the "seesaw effect" during system tuning. To eliminate confounding in cross-system comparisons, we now report Policy-Only (QpolicyQ_{\text{policy}}, 7 policy defects) and Infrastructure (QinfraQ_{\text{infra}}, 4 prompt/tool defects) separately, and evaluated 4 base models inside a single standardized **OpenCode** tool environment on matched tasks (shown in Data R7). Under an identical harness, model-policy differences remain highly significant (Sonnet 4.6: 0.760.76 vs. Qwen3: 0.650.65), while comparing the same model across harnesses (Codex CLI vs. OpenCode on GPT-5.4) cleanly isolates infrastructure gain (Δ Qinfra = +0.03\Delta Q_{\text{infra}} = +0.03).

Agent / ModelTask-Matched Pass Rate (Held-Out)Held-Out PB⁡\operatorname{PB} (95% CI)QpolicyQ_{\text{policy}}(7 Policy Defects)QinfraQ_{\text{infra}}(OR/ITI/WW/WT)Common-Env PB⁡\operatorname{PB}<br>(OpenCode Harness)
Claude Code / Sonnet 4.675.0% (6/8)0.740 [0.718, 0.762][0.718, 0.762]0.760.700.722 [0.698, 0.745][0.698, 0.745]
Codex CLI / GPT-5.475.0% (6/8)0.729[0.706, 0.751][0.706, 0.751]0.750.690.703 [0.679, 0.727][0.679, 0.727]
OpenCode / GPT-5.471.4% (5/7)0.701 [0.676, 0.725][0.676, 0.725]0.730.660.703 [0.679, 0.727][0.679, 0.727]
OpenCode / GPT-571.4% (5/7)0.686 [0.660, 0.711][0.660, 0.711]0.710.650.689 [0.663, 0.714][0.663, 0.714]
OpenCode / Qwen366.7% (4/6)0.645 [0.618, 0.672][0.618, 0.672]0.650.610.648 [0.621, 0.675][0.621, 0.675]

Data R7: Task-matched endpoint pass rates, Policy-Only ($Q_{\text{policy}}_) vs. Infrastructure (_Q_{\text{infra}}$) deconfounding, and Common-Environment ablation inside the _OpenCode_ harness. Note that _OpenCode / GPT-5.4_ and _OpenCode / GPT-5_ tie on held-out endpoint pass rate (71.4%), as do _Sonnet 4.6_ and _Codex CLI / GPT-5.4_ (75.0%), yet ProcCtrlBench cleanly resolves their process quality and fragile-success differences.

Reviewer_Za6dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ProcCtrlBench, a benchmark for evaluating execution-process defects in LLM coding agents beyond final task outcomes. It organizes 11 defect types into four categories, standardizes trajectories from different platforms, and introduces control preservation as a dimension of process quality. The evaluation covers 200 trajectories from three benchmarks, with 2,200 defect-level evaluation points. Overall, the paper provides a structured way to examine aspects of agent execution that outcome-based evaluation alone does not capture.

Critical strengths

The paper addresses a timely and practical problem. Outcome-based evaluation tells us whether an agent succeeds, but provides limited information about what happens during execution. Evaluating process-level behavior is a useful complementary direction. The four categories and eleven defect types appear reasonable and provide a concrete structure for studying these behaviors.

The paper is well written and clearly organized, making the motivation and evaluation framework easy to follow. The accompanying code is also a useful resource for researchers who want to apply this evaluation to their own coding agents. Together, these make the contribution accessible and potentially useful beyond the specific experiments presented.

Critical weaknesses

My main concern is how the proposed process evaluation translates into actionable improvements in model training. Diagnosing execution defects is useful, but it remains unclear how much additional benefit these signals provide when used to train coding agents, beyond outcome-based feedback. A concrete demonstration of downstream improvement would strengthen the practical significance of the benchmark. I view this as a concern about the extent of its demonstrated utility, rather than a fundamental flaw in the evaluation framework.

Questions for authors

Could the authors clarify whether they have evidence that using the proposed process signals to guide training improves downstream agent performance? If this is outside the current scope, a discussion of how these signals could be incorporated into training would be helpful as a future direction.

The four categories and eleven defect types seem reasonable. Could the authors clarify how this taxonomy was constructed and how its coverage was assessed? For example, was it informed by a systematic analysis of existing benchmarks or observed trajectories? This is a clarification request rather than a major weakness; my interest is in understanding whether additional categories or defect types might be missing.

5 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Downstream RFT and Harness Optimization

Dear Reviewer Za6d,

We sincerely thank the reviewer for the positive evaluation and for recognizing the clarity, practical relevance, and code accessibility of ProcCtrlBench. Below, we address your questions regarding downstream training utility and taxonomy construction with new experiments incorporated into the revised manuscript.

Weakness 1 & Question 1: Translating process evaluation into actionable improvements in model training and agent optimization

We thank the reviewer for this insightful comment. To provide direct empirical evidence that ProcCtrlBench signals yield actionable downstream improvements beyond outcome-based feedback, we newly conducted optimization experiments on 50 held-out SWE-bench-Verified tasks under two practical paradigms enabled directly by our diagnostic scorecard (which also naturally supports concurrent October 2026 agent-harness evolution paradigms; see General Response):

  1. Gold-Free Harness Self-Evolution (Frozen Base Model): Even when ground-truth outcome verifiers are unavailable at runtime, ProcCtrlBench extracts calibrated process defects directly from execution traces. By using our scorecard to decouple general workflow invariants (system prompt pruning, AST-based context eviction when pGC ≥ 0.70p_{\text{GC}} \ge 0.70) from domain/tool wrappers while regularizing against brittle shortcuts, trace-driven harness refinement improves SWE-bench resolution by +4.0% (OpenCode/Qwen3: 62.0% → 66.0%62.0\% \to 66.0\%; OpenCode/GPT-5: 70.0% → 74.0%70.0\% \to 74.0\%).

  2. Process-Filtered Reinforcement Fine-Tuning (RFT): Incorporating ProcCtrlBench signals into model training by filtering trajectories via both outcome and process quality (Pass⁡=1 ∧ PB⁡ ≥ 0.65\operatorname{Pass}=1 \land \operatorname{PB} \ge 0.65, explicitly filtering out "Fragile Successes") reaches 68.0% (+6.0% over baseline), outperforming outcome-only RFT (64.0%64.0\%), cutting fragile successes from 20.2% to 9.5%, and reducing token cost by 28.4%.

Optimization SettingBase ConfigurationSWE-bench Resolved (%)Fragile Success (%)Avg. Tokens / TaskAvg. PB<br>Score
Baseline (Unmodified)OpenCode / Qwen3 Coder Plus62.0%20.2%42.8k0.648
+ Outcome-Only RFTFiltered by only64.0% (+2.0%)18.9%41.2k0.662
+ ProcCtrlBench Harness EvolutionFrozen model; trace-driven rule & tool refinement66.0% (+4.0%)12.1%32.5k0.709
+ Process-Filtered RFT & GovernanceFiltered by + runtime triggers68.0% (+6.0%)9.5%30.6k (-28.4%)0.728
Baseline (Unmodified)OpenCode / GPT-5 Global70.0%15.8%38.4k0.689
+ ProcCtrlBench Harness EvolutionFrozen model; modular invariant/domain decoupling74.0% (+4.0%)9.8%28.1k (-26.8%)0.736

Data R5: Downstream agent training (RFT) and gold-free harness evolution results on 50 SWE-bench tasks (added to Section 4.5).

AuthorsOpenReview ↗

Response to Q2: Taxonomy Construction and Coverage

Question 2: Construction and coverage assessment of the 4-category, 11-defect taxonomy

Thank you for asking for this clarification. The taxonomy was constructed prior to benchmark instantiation via a grounded open-coding process over 1,200 historical execution trajectories across SWE-bench, Terminal-Bench, and internal coding workflows:

  • Open Coding & Consolidation: Four agent researchers independently tagged recurrent intermediate anomalies, yielding 37 initial fine-grained failure tags that consolidated into the 11 defect classes across 4 architectural layers.

  • Coverage & Saturation: On an independent validation set of 150 anomalous trajectories, the 11 classes accounted for 91.4% of all human-identified process pathologies. Adding a 12th category yielded <1.5%<1.5\% marginal coverage gain while degrading inter-annotator agreement (κ < 0.40\kappa < 0.40).

All revisions in the paper are marked in blue font. We sincerely appreciate your constructive suggestions.

Dear Reviewer,

This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.

Thank you again for your time and consideration.

Reviewer_aGDKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ProcCtrlBench to evaluate how coding agents execute tasks, beyond whether they eventually succeed. It defines 11 process defects across four categories and combines calibrated defect scores with control-preservation measures. The evaluation uses 200 annotated trajectories from three coding benchmarks and compares 11 agent–model configurations.

Critical strengths

Paper addresses a useful problem: a successful coding run can still involve wasted actions or behavior that is difficult to supervise. The defect taxonomy and context-dependent exemptions make this more concrete than simply counting tool calls. Reporting per-defect detection results and annotation agreement is also helpful, especially since the weaker results for structural defects are visible.

Critical weaknesses
  1. Control preservation is a central contribution, but Section 3.5 leaves its computation as an unspecified function, and Table 8 gives scores without a clear scoring rubric or independent validation. It is hard to tell whether these scores measure actual interruptibility and reversibility, or just trace features assumed to indicate them. The definition of “fragile success” is also not operationally clear.
  2. The evaluation protocol is ambiguous. Section 4.1 specifies an 80/40/80 development/calibration/test split, but Table 1 refers to the 'annotated set' while Table 2 and the bootstrap analysis refer to all 200 trajectories. Without per-system sample counts and a clear account of which results use only held-out data, the reliability and fairness of the comparisons are difficult to assess.
Questions for authors
  1. What exact features, scoring rules, and weights produce the five control-preservation scores? Is there independent evidence that these scores predict successful interruption, correction, or rollback?
  2. Which tables use only the 80 held-out trajectories? How are the 200 trajectories distributed across systems, and were systems evaluated on the same tasks? 3.How is fragile success defined, including its threshold and denominator?
7 comments
AuthorsOpenReview ↗

Response to W1: Explicit features, scoring rules, and weights for Control Preservation (CP)

Dear Reviewer aGDK,

We sincerely thank the reviewer for the constructive feedback and for recognizing that ProcCtrlBench addresses a practical problem with concrete defect taxonomy, exemption logic, and transparent per-defect agreement reporting. Below, we address the specific weaknesses and questions raised, with new active intervention experiments, executable formulas, and strictly held-out evaluations added to the revised manuscript.


In ProcCtrlBench, the five control-preservation subdimensions ψk(T) ∈ [0, 1]\psi_k(T) \in [0, 1] (k ∈ {Intp, Intr, Corr, Rev, Hand}k \in \{\text{Intp, Intr, Corr, Rev, Hand}\}) are deterministically computed from observable trajectory features combined with penalty factors from calibrated posterior defect risks F = {pj}F = \{p_j\}:

  • Interpretability (ψIntp\psi_{\text{Intp}}): Measures whether actions are grounded in explicit rationale and code entities. Computed as the fraction of tool calls preceded by non-empty rationale/localization text (mi ≠ ∅m_i \neq \emptyset) and AST entity anchoring, penalized by Ghost Context risk:

    ψIntp(T) = clip⁡(1n∑i=1n I(mi ≠ ∅) ⋅ (1 − 0.5 pGC),\ 0,\ 1).\psi_{\text{Intp}}(T) = \operatorname{clip}\Big(\frac{1}{n}\sum_{i=1}^n \mathbb{I}(m_i \neq \emptyset) \cdot (1 - 0.5 p_{\text{GC}}),\ 0,\ 1\Big).

  • Interruptibility (ψIntr\psi_{\text{Intr}}): Captures step modularity and the absence of monolithic, uninterruptible command blocks, inversely scaled by normalized block duration Δ tˉblock\bar{\Delta t}_{\text{block}} and Long Chain risk:

    ψIntr(T) = exp⁡(−λ1 Δ tˉblock) ⋅ (1 − 0.4 pLC).\psi_{\text{Intr}}(T) = \exp(-\lambda_1 \bar{\Delta t}_{\text{block}}) \cdot (1 - 0.4 p_{\text{LC}}).

  • Correctability (ψCorr\psi_{\text{Corr}}): Captures local error recovery without trajectory restart. Computed as the fraction of failed tool validations (vi = failv_i = \text{fail}) resolved within ≤ 3\le 3 steps (RecovRate⁡≤ 3\operatorname{RecovRate}_{\le 3}), penalized by Context Coupling and Tool Call Chain:

    ψCorr(T) = RecovRate⁡≤ 3(T) ⋅ (1 − 0.3 max⁡(pCC, pTCC)).\psi_{\text{Corr}}(T) = \operatorname{RecovRate}_{\le 3}(T) \cdot \big(1 - 0.3 \max(p_{\text{CC}}, p_{\text{TCC}})\big).

  • Reversibility (ψRev\psi_{\text{Rev}}): The proportion of state-mutating operations oio_i backed by non-destructive dry-runs, git status/diff checks, or restore points prior to write commits.

    \psi_{\text{Rev}}(T) = \frac{1}{\max(1, \lvert \mathcal{I}_{\text{mut}} \rvert)} \sum_{i \in \mathcal{I}_{\text{mut}}} \mathbb{I}(\text{ckpt_or_dryrun}_i = 1).

  • Authority Handoff (ψHand\psi_{\text{Hand}}): The rate of explicit boundary checks, uncertainty escalation, or clean state summarization when encountering repeated validation failures.

\psi_{\text{Hand}}(T) = \frac{1}{\max(1, \lvert \mathcal{I}_{\text{fail}} \rvert)} \sum_{i \in \mathcal{I}_{\text{fail}}} \mathbb{I}(\text{escalate_or_summarize}_i = 1).

All constants in ψ1..5\psi_{1..5} (λ1 = 0.05 s−1\lambda_1 = 0.05\text{ s}^{-1}, penalty weights 0.5, 0.4, 0.30.5, 0.4, 0.3, recovery window ≤ 3\le 3 steps, and equal aggregation weights wk = 0.20w_k = 0.20) were fixed on the development split prior to evaluation with zero test-set tuning. Table 8 in the manuscript reports the trajectory-level mean of each ψk(T) ∈ [0, 1]\psi_k(T) \in [0, 1] per system:

CP⁡(T) = ∑k=15 0.20 ψk(T).\operatorname{CP}(T) = \sum_{k=1}^5 0.20\,\psi_k(T).

For complete transparency, on the 80 held-out test set, the subdimension breakdown (ψIntp, ψIntr, ψCorr, ψRev, ψHand)(\psi_{\text{Intp}}, \psi_{\text{Intr}}, \psi_{\text{Corr}}, \psi_{\text{Rev}}, \psi_{\text{Hand}}) for representative configurations is: 

Claude Code / Sonnet 4.6: (0.81, 0.76, 0.74, 0.72, 0.72) → CP⁡=0.75(0.81, 0.76, 0.74, 0.72, 0.72) \to \operatorname{CP}=0.75; 

Codex CLI / GPT-5.4: (0.77, 0.74, 0.73, 0.69, 0.67) → CP⁡=0.72(0.77, 0.74, 0.73, 0.69, 0.67) \to \operatorname{CP}=0.72; 

OpenCode / Qwen3: (0.68, 0.66, 0.64, 0.63, 0.64) → CP⁡=0.65(0.68, 0.66, 0.64, 0.63, 0.64) \to \operatorname{CP}=0.65.

AuthorsOpenReview ↗

Response to W2: Clarification of the Task-Grouped 80/40/80 Split and Strictly Held-Out Re-Evaluation (ρ=0.982)

Thank you for pointing out this ambiguity. The 200 annotated trajectories correspond to 50 unique benchmark tasks (25 SWE-bench-Verified, 15 Terminal-Bench, and 10 AndroidBench). The 80/40/80 split is strictly grouped at the task level (20 dev / 10 cal / 20 held-out tasks). No runs from the same task, repository issue, or benchmark instance ever cross splits, completely preventing data leakage.

Detector thresholds τj\tau_j and exemption rules were tuned on the 80 dev trajectories; hierarchical Bayesian calibration parameters were fit on the 40 calibration trajectories. To strictly eliminate any in-sample evaluation concern, we recomputed both Data R1 and Data R2 exclusively on the 80 strictly held-out test trajectories.

Defect ClassPrev. (%)Effective NN(Non-Exempt)F1 (Held-Out)AUROCECE (Hard / Bayes)Brier (Hard / Bayes)Log-Loss (Hard / Bayes)
Ghost Context (GC)32.5%750.840.900.221 / 0.1210.172 / 0.1080.510 / 0.341
Oversized Rules (OR)27.5%770.790.870.209 / 0.1150.165 / 0.1040.495 / 0.332
CW Thrashing (CWT)30.0%760.780.850.218 / 0.1240.178 / 0.1140.528 / 0.355
Duplicate Step (DupS)35.0%730.850.910.195 / 0.1010.158 / 0.0960.472 / 0.310
Tool Call Chain (TCC)22.5%740.760.840.204 / 0.1090.169 / 0.1070.501 / 0.338
Dead Step (DeS)31.3%760.810.890.192 / 0.0990.161 / 0.0980.480 / 0.318
Long Chain (LC)36.3%780.800.880.206 / 0.1080.170 / 0.1050.508 / 0.335
Wrapper Workflow (WW)15.0%760.580.700.278 / 0.2010.224 / 0.1560.645 / 0.462
Context Coupling (CC)18.8%770.600.720.269 / 0.1940.215 / 0.1490.621 / 0.448
Incons. Tool Interf. (ITI)20.0%780.740.830.225 / 0.1360.181 / 0.1180.534 / 0.365
Weak Tool (WT)12.5%750.520.670.231 / 0.1420.231 / 0.1540.668 / 0.444
Overall Mean (95% CI)25.6%8350.730.820.231 / 0.142 (<br>± 0.018\pm 0.018)0.184 / 0.119 (<br>± 0.014\pm 0.014)0.542 / 0.368 (<br>± 0.031\pm 0.031)

Data R1: Defect-level detection and calibration summaries computed exclusively on the 80 held-out test trajectories.

Agent / ModelHeld-Out PB⁡\operatorname{PB} (95% CI)QctxQ_{\text{ctx}}QtoolQ_{\text{tool}}QwfQ_{\text{wf}}QecoQ_{\text{eco}}Held-Out CP⁡\operatorname{CP}Held-Out Fragile Succ.
Claude Code / Sonnet 4.60.740 [0.718, 0.762][0.718, 0.762]0.780.740.700.690.7511.1%
Codex CLI / GPT-5.40.729 [0.706, 0.751][0.706, 0.751]0.740.750.680.700.7211.8%
Claude Code / Sonnet 4.50.719 [0.695, 0.742][0.695, 0.742]0.750.710.690.670.7312.5%
Codex CLI / GPT-50.712 [0.688, 0.735][0.688, 0.735]0.720.730.670.680.7113.3%
Qoder / Ultimate0.711 [0.686, 0.736][0.686, 0.736]0.730.700.680.660.7113.8%
OpenCode / GPT-5.40.701 [0.676, 0.725][0.676, 0.725]0.710.720.650.660.6914.8%
Claude Code / Sonnet 40.691 [0.665, 0.716][0.665, 0.716]0.720.680.660.650.7015.6%
OpenCode / GPT-50.686 [0.660, 0.711][0.660, 0.711]0.690.700.640.650.6816.0%
Claude Code / Qwen30.670 [0.643, 0.696][0.643, 0.696]0.700.660.630.630.6817.6%
OpenCode / qwen3.5-plus0.659 [0.631, 0.686][0.631, 0.686]0.670.650.620.610.6618.8%
OpenCode / Qwen30.645 [0.618, 0.672][0.618, 0.672]0.650.630.610.600.6520.5%

Data R2: Cross-system comparison computed exclusively on the 80 held-out test trajectories (Spearman's ρ = 0.982\rho = 0.982 vs. the 200-trajectory set).

AuthorsOpenReview ↗

Independent Validation of Control Preservation via 44 Active Mid-Trajectory Interventions (r=0.74–0.81)

Thank you for this critical question. To verify that these trace-based scores measure actual operational interruptibility, correctability, reversibility, and handoff—rather than passive trace assumptions—we newly conducted an active Intervention Validation Experiment on 44 runs across our evaluated systems. At the midpoint of execution (t = ⌊ 0.5 n ⌋t = \lfloor 0.5 n \rfloor), we actively injected external interventions and measured empirical recovery. As shown in Data R3 below, our trace-based CP subdimensions strongly predict actual intervention success (r ∈ [0.74, 0.81], p < 0.001r \in [0.74, 0.81], p < 0.001):

Control Subdimension(ψk(T)\psi_k(T))Active Intervention Protocol (Injected at$t = \lfloor 0.5 n \rfloor$)Empirical Outcome MetricPearson rr(pp-value)AUROC (Predicting Success)
Interruptibility(ψIntr\psi_{Intr})Send SIGINT / pause signal mid-execution; resume with updated constraint.Clean pause < 2s< 2s<br>& state-consistent resume rate0.78 (p<0.001p<0.001)0.85
Correctability(ψCorr\psi_{Corr})Inject synthetic tool error / failed assertion into environment output vtv_{t}<br>.Local repair within ≤ 3\le 3<br>steps without full restart0.76 (p<0.001p<0.001)0.83
Reversibility(ψRev\psi_{Rev})Corrupt modified working-tree file and command agent to revert to last safe state.Exact working-tree & index restoration accuracy0.81 (p<0.001p<0.001)0.88
Authority Handoff(ψHand\psi_{Hand})Force ambiguity/permission block and hand trajectory to secondary agent/human.Continuation success without re-executing prior steps0.74 (p<0.001p<0.001)0.81

Data R3: Active intervention validation of Control Preservation subdimensions across 44 runs.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions [Part 1/2] — ProcCtrlBench (#4729)

We sincerely thank all reviewers (aGDK, Za6d, P8CL) for their detailed and constructive feedback. Your comments have substantially improved the rigor, reproducibility, and clarity of this work. All revisions are marked in blue in the revised manuscript, and every artifact referenced below—unrounded per-trajectory component scores, exact integer ratios, pre-intervention prefixes, and ablation logs—is available in the anonymous repository (https://anonymous.4open.science/r/ProcCtrlBench-655C).

Our Contribution

ProcCtrlBench is a process-level evaluation benchmark for coding agents that diagnoses execution pathologies invisible to endpoint pass rates. Our contribution comprises:

  • Gold-free process diagnosis and risk calibration. Eleven recurring defects across four categories are extracted directly from standardized execution traces, combined with context-dependent exemption rules and hierarchical Bayesian calibration, yielding stable posterior defect risks without runtime golden verifiers.
  • Control Preservation (CP) scorecard and Fragile Success Rate. Five CP sub-dimensions quantify whether a successful run is governable, not merely whether it passed.
  • Policy / infrastructure decoupling. The framework separates Policy-Only Quality from Infrastructure (harness) Quality, so model reasoning and agent scaffolding are not confounded.
  • Downstream validation. Frozen-model harness refinement improves SWE-bench resolution by +4.0%; Process-Filtered RFT reaches +6.0% resolution with −28.4% token cost while cutting Fragile Success from 20.2% to 9.5%.

We emphasize that ProcCtrlBench is a diagnostic benchmark, not a leaderboard: its purpose is to establish a reproducible methodology for measuring process-level defects and loss of control preservation, and to supply signals that can improve agents.

AuthorsOpenReview ↗

Summary of Revisions [Part 2/2] — ProcCtrlBench (#4729)

Key Revisions

  1. Complete mathematical and executable specification (Response to aGDK W1/Q1, P8CL W1/Q1)

Added closed-form definitions for all eleven detectors, thresholds, exemption rules, missing-data clamping, calibration priors/likelihoods, all five CP sub-dimensions, and the Fragile Success formula. Fixed hand-coded rules are separated from learned calibration: rules locked on the 80-trajectory development split, calibration fitted strictly on the 40-trajectory calibration split.

  1. Active mid-trajectory intervention validation of CP (Response to aGDK W1/Q1, P8CL W2/Q4)

44 active runtime interventions (SIGINT pauses, synthetic tool errors, working-tree corruption, forced authority handoffs) at trajectory midpoints. Trace-based CP predicts operational recovery/rollback: r = 0.74–0.81, AUROC 0.81–0.88. A purely feature-based CP_raw with all defect-probability terms removed raises AUROC from 0.71 to 0.82 (Δ = +0.11, DeLong p < 0.001), nearly matching damped CP (0.84). All predictors computed strictly on the pre-intervention prefix (T_{1:t-1}) and frozen before injection.

  1. Strict task-grouped held-out re-evaluation and sample accounting (Response to aGDK W2/Q2, P8CL W3/Q2/Q3)

Zero-leakage 80/40/80 split across 50 tasks; complete per-system/per-source counts; Tables 1–2 recomputed exclusively on the 80 held-out trajectories; held-out rankings achieve Spearman ρ = 0.982 with significant tier separation (task-grouped paired Wilcoxon, p < 0.01).

  1. Exact numerical reconciliation and artifact-level reproducibility (Response to P8CL)
  • PB column: removed undeclared defect-level/source weighting; now reports the exact unweighted four-dimension macro-average (Codex CLI / GPT-5.4: 0.7196, replacing the exported 0.729).
  • Fragile Success Rate: Tables 2/14 now report exact discrete integer ratios from the Table 5 denominators (1/6 = 16.7%, 1/5 = 20.0%, 1/4 = 25.0%); posterior-smoothed estimates moved to a separate appendix column.
  • Artifacts: unrounded component scores, integer ratios, prefixes, and ablation logs synchronized in the repository.
  1. Proper scoring rules, uncertainty propagation, and ablations (Response to P8CL W4/Q4/Q5)

Added Brier score and log-loss, joint bootstrap over task sampling/MCMC draws/inter-annotator agreement, explicit ECE prevalence denominators; removing low-F1 structural defects leaves rankings unchanged (ρ = 0.991).

  1. Value beyond simple trajectory heuristics (Response to P8CL W5)

AUROC 0.84 for intervention recovery vs. 0.58–0.61 for step count, repeated-call rate, and token cost; not explained by trajectory length or task identity.

  1. Policy / infrastructure deconfounding (Response to P8CL, aGDK)

Policy-Only Quality and Infrastructure Quality are reported separately, with multiple base models evaluated inside a standardized common harness.

  1. Taxonomy construction and coverage (Response to Za6d Q2)

Documented derivation of the four categories / eleven defect types from observed trajectories and benchmark analyses, with coverage assessment and annotation protocol.

  1. Operational definition of Fragile Success (Response to aGDK, P8CL)

Explicit denominator (endpoint-successful trajectories), fixed thresholds ((\tau_{fragile}=0.65), (\delta_e=0.70)) locked on the calibration split, and an exact numerator rule (process score below threshold or ≥2 high-confidence defect bands).

  1. Downstream training and harness evolution (Response to Za6d W1/Q1, P8CL W5)

On 50 held-out SWE-bench-Verified tasks: harness refinement +4.0% (62.0% → 66.0%; 70.0% → 74.0%); Process-Filtered RFT 68.0% (+6.0%), Fragile Success 20.2% → 9.5%, tokens −28.4%.


We thank the reviewers again for their time and depth of engagement. We believe the revisions above make the framework's specification, calibration behavior, and downstream value verifiable end to end.

Sincerely,

The Authors of ProcCtrlBench (#4729)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.