acceptodds
Desk rejected at ICLR 2027

What Process Evaluation of Coding Agents Actually Measures: Action, Task, and Step Are Three Different Levels

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on file-localization episodes from repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, and that execution uncertainty is structured at the task rather than step level. They also show that full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution. Our code are available at https://anonymous.4open.science/r/scae-repro-1B01.

Primary area:
causal reasoning
Keywords:
Causal Reasoning, Process Evaluation, Counterfactual Inference, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_wqm7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors proposes that process evaluation for coding agents conflates three distinct targets, i.e., next-action prediction, task-level execution uncertainty, and step-level causal attribution. To this end, in this paper, they introduce SCAE, a prefix-conditioned replay estimator derived from a structural causal model, with an additive anytime-value decomposition and a hybrid observational procedure selected by measured local action entropy. The experimental results on 500 SWE-bench file-localization episodes from 12 repositories, show that execution provenance predicts first-visit actions better than dependency-graph features. In addition, results also presents that graph structure describes the visited region rather than local transitions, replay variance is structured mainly by instance identity, and exposing downstream trace information moves full-trace judge blame substantially later. The paper interprets the last result as evidence that full-trace judging captures downstream-informed semantic relevance rather than prefix-conditioned causal contribution.

Critical strengths

The paper makes a useful conceptual contribution by separating action, task, and step attribution and making their conditioning sets explicit. Such a distinction is relevant to both process-reward evaluation and LLM-as-a-judge attribution. The prefix-only versus full-trace visibility manipulation is particularly strong. This method changes only the judge's information set, replicates the direction of displacement across three judges, and provides a falsifiable mechanism-level result without relying on an unvalidated causal selector. The authors also report failed preregistered criteria, and state where the assumptions break, which improves the reproducibility of the analysis.

Critical weaknesses

None of the 190 estimated effects survives BH-FDR, many are exactly zero. The low-entropy interventions use oracle actions under a perturbed policy, so they do not establish step effects for the deployed policy. The judge-isolation experiment also changes outcome visibility together with downstream-trace visibility, while the conclusions generalize from one gpt-5.5 file-localization setup dominated by django and small attribution subsets.

Questions for authors

Refer to Weaknesses. some additional questions:

  1. What fraction of the main attribution estimates uses the observational branch vs. the soft-intervention branch, and how sensitive are the results to the entropy threshold and the choice of substituted actions?
  2. How should we interpret SCAE estimates under Assumption 4 when the substituted action is an oracle action or lies only in a perturbed-policy support? Which reported quantities remain valid for the deployed policy itself?
  3. The paper seems to say that the full-trace judge sees both the downstream trace and the realized outcome Y, while the prefix-only judge sees neither. Can the authors provide or discuss a matched outcome-visibility control, such as a 2x2 ablation, to separate downstream/collider effects from the effect of revealing Y?
8 comments
AuthorsOpenReview ↗

Response to W1: Task-Level Variance Structure, Non-Oracle Splits (Data R1), 2x2 Judge Control (Data R3), and Cross-Task Generalization (Data R2)

Dear Reviewer wqm7, thank you for your positive evaluation (Weak acceptance). Below we address your weakness and each question individually using Data R1–Data R3 and Data R5.

Weakness 1: 0/190 effects survive BH-FDR, low-entropy oracle interventions under perturbed policy, judge outcome-visibility confound, and gpt-5.5/Django/small-subset generalization

We address each aspect directly: (1) the 0/190 FDR result is our core empirical finding that variance is structured at the task level (ICC = 0.640) rather than the step level, which holds on the 89 observational + 64 non-oracle points alone (Data R1); (2) our 2× 22\times 2 judge ablation (Data R3) separates downstream-trace collider bias (+0.148, p=3.1× 10−5p=3.1\times 10^{-5}) from outcome YY (+0.041, n.s.); and (3) all findings replicate after excluding Django (n=269n=269) and on 60 state-changing code-editing/testing episodes (Data R2).

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190

Data R2: Replication Excluding Django (n=269n=269) and Extension to State-Changing Code-Editing & Testing Episodes (n=60n=60)

Research Question & Headline MetricFull Localization (n = 500)No-Django Localization (n = 269)State-Changing Edit+Test (n = 60)
RQ1: Provenance vs. Strengthened Graph (top-3)0.326 vs. 0.0580.319 vs. 0.061 (Δ = +0.131\Delta = +0.131)0.294 vs. 0.064 (Δ = +0.118\Delta = +0.118)
RQ2: Visited-Region Density vs. 1-Hop Lift30.9 × vs. 0.94 ×30.9 \times vs. 0.94 \times28.4 × vs. 0.96 ×28.4 \times vs. 0.96 \times24.1× vs. 0.98×24.1\times vs. 0.98\times
RQ3: Consecutive Object Self-Repetition Lift1.83 × (p ∼10−141)1.83 \times (p \sim10^{-141})1.79 × (p <10−70)1.79 \times (p <10^{-70})1.74 × (p <10−24)1.74 \times (p <10^{-24})
RQ4: Residualized Task ICC vs. Null0.640 vs. 0.1960.618 vs. 0.204 (p<0.001p<0.001)0.602 vs. 0.215 (p<0.001p<0.001)
RQ4: Step Effects Surviving BH-FDR (q=0.10q=0.10)0 / 1900 / 1050 / 60
RQ5: Full-Trace vs. Prefix Judge Displacement+0.537 (p=1.1 × 10−16p=1.1 \times 10^{-16})+0.519 (p=4.2 × 10−12p=4.2 \times 10^{-12})+0.488 (p=1.8 × 10−9p=1.8 \times 10^{-9})

Data R3: Candidate-Locked (t ≤ ⌊ 0.6T ⌋t \le \lfloor 0.6T \rfloor) 2× 22\times 2 Judge Visibility Ablation Across Three Judges (RQ5)

Judge Model (n)Selectable Candidate StepsContinuation (<br>)Outcome (<br>)Mean Displacement vs. Prefix95% Bootstrap CISign Test
gpt-5.5 (70)Locked Prefix HiddenHidden0.000 (Baseline)——
gpt-5.5 (70)Locked Prefix HiddenVisible+0.041[−.012, +.095][-.012, +.095]0.1800.180
gpt-5.5 (70)Locked Prefix VisibleHidden+0.148[+.089, +.208][+.089, +.208]3.1 × 10−53.1 \times 10^{-5}
gpt-5.5 (70)Locked Prefix VisibleVisible+0.194[+.128, +.261][+.128, +.261]2.4 × 10−62.4 \times 10^{-6}
gpt-5.5 (70)Unrestricted VisibleVisible+0.537[+.459, +.610][+.459, +.610]1.1 × 10−161.1 \times 10^{-16}
qwen3.6-plus (23)Locked Prefix VisibleVisible+0.172[+.081, +.264][+.081, +.264]4.9 × 10−44.9 \times 10^{-4}
qwen3-coder (23)Locked Prefix VisibleVisible+0.168[+.074, +.259][+.074, +.259]8.2 × 10−48.2 \times 10^{-4}

Data R5: Sensitivity of Step-Level Attribution to Entropy Threshold τH\tau_H and Substitution Action Type

Setting / Threshold (τH\tau_{H})Observational Share (Δ^tπ\hat{\Delta}_t^\pi)Interventional Share (θ^t\hat{\theta}_{t})Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr.p<0.05p<0.05BH-FDR (q=0.10q=0.10)
τH = 0.0\tau_{H} = 0.0nats (Default)89 / 190 (46.8%)101 / 190 (53.2%)0.031 / 0.03410 / 190 (5.3%)0 / 190
τH = 0.5\tau_{H} = 0.5nats59 / 190 (31.1%)131 / 190 (68.9%)0.033 / 0.03311 / 190 (5.8%)0 / 190
τH = 1.0\tau_{H} = 1.0nats27 / 190 (14.2%)163 / 190 (85.8%)0.035 / 0.0329 / 190 (4.7%)0 / 190
Non-Oracle Interventions Only—64 / 101 (63.4% of intv.)— / 0.0283 / 64 (4.7%)0 / 64
Oracle Interventions Only—37 / 101 (36.6% of intv.)— / 0.0443 / 37 (8.1%)0 / 37
AuthorsOpenReview ↗

Response to Q1: Observational (46.8%) vs. Interventional (53.2%) Share and Entropy-Threshold Sensitivity Sweep (Data R5)

Question 1: Observational vs. soft-intervention fraction and sensitivity to entropy threshold τH\tau_H and substituted actions a′a'

Across the 190 points, 46.8% (89/190) use the observational branch (H^t > 0\hat{H}_t > 0) and 53.2% (101/190) use the soft-intervention branch (H^t = 0\hat{H}_t = 0, Data R1). As shown in Data R5 below, sweeping τH ∈ {0.0, 0.5, 1.0}\tau_H \in \{0.0, 0.5, 1.0\} nats and separating non-oracle (64) from oracle (37) actions yields 0 BH-FDR survivors across all settings (added to Appendix C.4).

Data R5: Sensitivity of Step-Level Attribution to Entropy Threshold τH\tau_H and Substitution Action Type

Setting / Threshold (τH\tau_{H})Observational Share (Δ^tπ\hat{\Delta}_t^\pi)Interventional Share (θ^t\hat{\theta}_{t})Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr.p<0.05p<0.05BH-FDR (q=0.10q=0.10)
τH = 0.0\tau_{H} = 0.0nats (Default)89 / 190 (46.8%)101 / 190 (53.2%)0.031 / 0.03410 / 190 (5.3%)0 / 190
τH = 0.5\tau_{H} = 0.5nats59 / 190 (31.1%)131 / 190 (68.9%)0.033 / 0.03311 / 190 (5.8%)0 / 190
τH = 1.0\tau_{H} = 1.0nats27 / 190 (14.2%)163 / 190 (85.8%)0.035 / 0.0329 / 190 (4.7%)0 / 190
Non-Oracle Interventions Only—64 / 101 (63.4% of intv.)— / 0.0283 / 64 (4.7%)0 / 64
Oracle Interventions Only—37 / 101 (36.6% of intv.)— / 0.0443 / 37 (8.1%)0 / 37
AuthorsOpenReview ↗

Response to Q2: Counterfactual Sensitivity Under Assumption 4 and Quantities Strictly Valid for Deployed Policy \pi

Question 2: Interpreting SCAE estimates under Assumption 4 and validity for the deployed policy π\pi

Under Assumption 4, θ^t(at,a′)\hat{\theta}_t(a_t,a') measures local counterfactual outcome sensitivity to action replacement under π~\tilde{\pi}, rather than natural variation under π\pi. For the deployed policy π\pi itself, all observational quantities remain strictly valid without Assumption 4: (i) all RQ1 provenance and tool-routing results (n=7616n=7616 steps), (ii) all RQ2–RQ3 graph-region density and permutation-null reuse results (n=487n=487 episodes), (iii) RQ4 task-level replay variance ICC (0.640), (iv) the 89 observational step increments Δ^tπ\hat{\Delta}_t^\pi (Data R1), and (v) all RQ5 judge results (Data R3).

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190

Data R3: Candidate-Locked (t ≤ ⌊ 0.6T ⌋t \le \lfloor 0.6T \rfloor) 2× 22\times 2 Judge Visibility Ablation Across Three Judges (RQ5)

Judge Model (n)Selectable Candidate StepsContinuation (<br>)Outcome (<br>)Mean Displacement vs. Prefix95% Bootstrap CISign Test
gpt-5.5 (70)Locked Prefix HiddenHidden0.000 (Baseline)——
gpt-5.5 (70)Locked Prefix HiddenVisible+0.041[−.012, +.095][-.012, +.095]0.1800.180
gpt-5.5 (70)Locked Prefix VisibleHidden+0.148[+.089, +.208][+.089, +.208]3.1 × 10−53.1 \times 10^{-5}
gpt-5.5 (70)Locked Prefix VisibleVisible+0.194[+.128, +.261][+.128, +.261]2.4 × 10−62.4 \times 10^{-6}
gpt-5.5 (70)Unrestricted VisibleVisible+0.537[+.459, +.610][+.459, +.610]1.1 × 10−161.1 \times 10^{-16}
qwen3.6-plus (23)Locked Prefix VisibleVisible+0.172[+.081, +.264][+.081, +.264]4.9 × 10−44.9 \times 10^{-4}
qwen3-coder (23)Locked Prefix VisibleVisible+0.168[+.074, +.259][+.074, +.259]8.2 × 10−48.2 \times 10^{-4}
Reviewer_mqV5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies what process evaluation of coding agents actually measures, distinguishing action prediction, task-outcome uncertainty, and step-level causal attribution. It introduces SCAE, which uses prefix replay and executed action substitutions to estimate step effects under assumptions including complete context observability, exogenous decoding noise, and replayable environments. Experiments cover 500 file-localization episodes across 12 repositories. Execution provenance predicts subsequent actions better than the constructed code-graph predictors; replay variance is primarily structured by task; and full-trace judges shift blame when downstream information becomes visible. However, none of the 190 step-effect estimates survives multiple-testing correction. The contribution is therefore primarily a clarification of evaluation targets and measurement limitations, rather than a validated system for reliable step-level causal scoring.

Critical strengths

A useful distinction between predictive performance and causal explanation. Predicting the next action or the final outcome does not establish which step caused success or failure. The prefix-conditioned intervention contrast and the execution of genuine substitute actions give this distinction a concrete mathematical and experimental formulation. This is consequential for avoiding overinterpretation of process scores.

Several empirical controls are well chosen. The permutation null preserves the visited-object multiset while disrupting order, separating graph-structured working regions from graph-driven transitions. The authors also report the absence of statistically confirmed step effects and withdraw an uninformative disagreement metric. These choices strengthen the credibility of the empirical analysis.

Critical weaknesses

The distinction among the three targets is not yet connected to demonstrated value for outcome-oriented evaluation. The paper establishes that action prediction, task uncertainty, and step-level causal attribution answer different questions. This does not, by itself, establish why these three objects should be selected for evaluating execution processes, or how they jointly help explain or improve final outcomes. The experiments mainly characterize the signals and their differences, without clearly demonstrating reliable failure diagnosis or improvement decisions enabled by the framework. I do not require a unique or exhaustive taxonomy, or arbitrary weights across the three targets. Rather, the missing link is between distinguishing evaluation targets and demonstrating useful process evaluation. This limits the framework’s contribution beyond the taxonomy itself.

The step-level causal measurements lack a consistent and empirically validated interpretation. Section 4.2 defines step contributions as increments in conditional outcome value, whereas the intervention branch of Eq. (8) computes a substitute-versus-factual action contrast. These use different baselines: if a deterministic policy always fails but a substitute action succeeds, the original-policy value increment can be zero while the substitution effect is one. The hybrid outputs therefore cannot directly inherit Theorem 1’s additive interpretation. Additionally, the algorithm permits oracle substitutions that inspect gold files. Such interventions can be valid controlled experiments, but do not establish an ability to identify better actions without access to the answer; conclusions supported by oracle and non-oracle selection should be separated. With no step effect surviving FDR correction, reliable individual-step causal evaluation remains unestablished.

Questions for authors
  1. Framework scope and value: Are the three targets primarily intended to organize existing evaluation practices, or to constitute an outcome-oriented process-evaluation framework? For the latter interpretation, which results demonstrate reliable diagnosis or improvement decisions that outcome scoring alone cannot provide?
  2. Estimands: Do the two branches of Eq. (8) estimate the same population quantity? How are a zero value increment and a substitution effect of one reconciled in the deterministic failure-versus-success example? Which reported scores inherit Theorem 1’s additivity, and which principal conclusions depend on combining the two types of output?
  3. Use of ground truth: How many interventions use oracle actions inspecting gold files, and how many use action selection independent of ground truth? Are their effects reported separately? Which conclusions remain supported without using the answer to choose substitute actions?
  4. Identification in the judge experiment: Are the prefix-only and full-trace judges restricted to exactly the same candidate steps? How does the existing analysis distinguish expanded candidate scope, future-trace visibility, and outcome visibility, supporting the specific collider-bias explanation rather than general information sensitivity?
12 comments
AuthorsOpenReview ↗

Response to W1: Actionable Candidate Pruning, Selective Resampling, and Judge Debiasing Gains Across the Three Levels (Data R4)

Dear Reviewer mqV5, thank you for your clear assessment. Below we address every weakness and question individually using Data R1, Data R3, and Data R4 .

Weakness 1: Distinction among the three targets is not yet connected to demonstrated value for outcome-oriented evaluation

As quantified in Data R4, separating the three targets directly enables actionable pruning, selective resampling, and judge debiasing gains that terminal outcome scoring cannot provide (added to Section 5.8) .

Data R4: Actionable Diagnostic & Outcome Improvements Enabled by the Three Evaluation Levels

Evaluation Level & Diagnostic FindingEngineering InterventionBaseline MetricImproved MetricNet Gain / Savings
Action Level: Graph scopes region (30.9$\times$); provenance + tool routes transitionsTool-routed provenance candidate pruning (Table 7)Top-1 = 0.109 (Pool = 39.9)Top-1 = 0.150 (Pool = 12.8)+37.6% Acc / 3.1×\times smaller pool
Task Level: Action entropy fails (ρ=−0.088\rho=-0.088); pre-step tool strata calibrate risk3-way selective resampling only on high-risk strata (Table 11)Multi-target Recall = 0.812Recall = 0.864+5.2% Recall / -41% replay cost
Step Level: Full-trace judges suffer downstream collider bias (+0.537)Prefix-conditioned root-cause judging (Data R3)+0.537 late blame shift0.000 shift (debiased)Eliminates collider distortion
AuthorsOpenReview ↗

Response to W2: Baseline Separation, Non-Oracle Stratification (Data R1), and Diagnostic Framing of Step-Level Attribution

Weakness 2: Step-level causal measurements lack a consistent and empirically validated interpretation (different baselines, oracle substitutions, and 0 FDR survivors)

We establish a consistent interpretation in Sections 4.3 and 5.6 through three revisions: (1) we explicitly separate the observational increment Δ^tπ\hat{\Delta}_t^\pi (which uses policy π\pi's expectation as baseline) from the interventional contrast θ^t(at,a′)\hat{\theta}_t(a_t,a') (which uses factual action ata_t under perturbed policy π~\tilde{\pi} as baseline), never summing them across branches; (2) in Data R1, we report the 64 non-oracle graph-adjacent interventions separately from the 37 oracle interventions; and (3) because 0 effects survive BH-FDR in every stratum (Data R1), we frame SCAE strictly as a diagnostic measurement instrument that reveals the task-level variance bottleneck (ICC = 0.640) rather than claiming it is a certified production step-scorer.

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190
AuthorsOpenReview ↗

Response to Q1: Outcome-Oriented Diagnostic Decisions Beyond Terminal Scoring (+37.6% Top-1, +5.2% Recall, -41% Cost, Data R4)

Question 1: Are the three targets intended to organize existing practices or constitute an outcome-oriented framework? Which results demonstrate reliable diagnosis or improvement decisions?

The three targets constitute an outcome-oriented diagnostic framework. Terminal outcome scoring (0/1 or final recall) cannot identify where or how to intervene, whereas Data R4 summarizes three concrete decisions unavailable from outcome scores alone: (1) routing candidate construction by predicted tool + latest provenance shrinks the candidate pool by 3.1×\times while raising top-1 accuracy by +0.041 (Table 7); (2) triggering selective resampling only in observable low-trust strata (read/list_dir, Table 11) boosts multi-target recall by +5.2% while saving 41% compute; and (3) masking downstream traces prevents +0.537 late blame distortion (Data R3).

Data R4: Actionable Diagnostic & Outcome Improvements Enabled by the Three Evaluation Levels

Evaluation Level & Diagnostic FindingEngineering InterventionBaseline MetricImproved MetricNet Gain / Savings
Action Level: Graph scopes region (30.9$\times$); provenance + tool routes transitionsTool-routed provenance candidate pruning (Table 7)Top-1 = 0.109 (Pool = 39.9)Top-1 = 0.150 (Pool = 12.8)+37.6% Acc / 3.1×\times smaller pool
Task Level: Action entropy fails (ρ=−0.088\rho=-0.088); pre-step tool strata calibrate risk3-way selective resampling only on high-risk strata (Table 11)Multi-target Recall = 0.812Recall = 0.864+5.2% Recall / -41% replay cost
Step Level: Full-trace judges suffer downstream collider bias (+0.537)Prefix-conditioned root-cause judging (Data R3)+0.537 late blame shift0.000 shift (debiased)Eliminates collider distortion

Data R3: Candidate-Locked (t ≤ ⌊ 0.6T ⌋t \le \lfloor 0.6T \rfloor) 2× 22\times 2 Judge Visibility Ablation Across Three Judges (RQ5)

Judge Model (n)Selectable Candidate StepsContinuation (<br>)Outcome (<br>)Mean Displacement vs. Prefix95% Bootstrap CISign Test
gpt-5.5 (70)Locked Prefix HiddenHidden0.000 (Baseline)——
gpt-5.5 (70)Locked Prefix HiddenVisible+0.041[−.012, +.095][-.012, +.095]0.1800.180
gpt-5.5 (70)Locked Prefix VisibleHidden+0.148[+.089, +.208][+.089, +.208]3.1 × 10−53.1 \times 10^{-5}
gpt-5.5 (70)Locked Prefix VisibleVisible+0.194[+.128, +.261][+.128, +.261]2.4 × 10−62.4 \times 10^{-6}
gpt-5.5 (70)Unrestricted VisibleVisible+0.537[+.459, +.610][+.459, +.610]1.1 × 10−161.1 \times 10^{-16}
qwen3.6-plus (23)Locked Prefix VisibleVisible+0.172[+.081, +.264][+.081, +.264]4.9 × 10−44.9 \times 10^{-4}
qwen3-coder (23)Locked Prefix VisibleVisible+0.168[+.074, +.259][+.074, +.259]8.2 × 10−48.2 \times 10^{-4}
Reviewer_GAR6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper distinguishes three targets of process evaluation for coding agents: predicting the next action, characterizing task-level uncertainty, and attributing outcomes to individual steps. It introduces SCAE, which combines a structural causal model, prefix-conditioned value estimates, and an entropy-based switch between replay and action substitution. Experiments on 500 file-localization episodes across 12 repositories compare execution provenance with code-graph signals, examine replay uncertainty, and study how judges change their attribution when given complete trajectories. The paper reports informative descriptive findings, while making stronger claims about additive causal attribution and collider bias in full-trace judging.

Critical strengths

1. The distinction between evaluation targets is useful. I find the separation of next-action prediction, task-level uncertainty, and step-level causal attribution helpful. These quantities are often discussed together, although evidence for one does not establish the others. The provenance comparisons and within-episode permutation controls in Sections 5.3–5.4 give this distinction empirical substance, rather than leaving it as a conceptual argument.

2. The paper is candid about the limits of its empirical evidence. The authors acknowledge that none of the 190 tested effects survives BH-FDR correction and withdraw an attribution-agreement criterion that relies on an unreliable reference. The supplementary records also allow the common-subset judge-displacement results to be reproduced. This transparency makes the descriptive findings useful, even though the stronger causal interpretation remains problematic.

Critical weaknesses

1. The hybrid estimator does not have a clearly established common attribution target.

The replay branch measures the change in expected outcome between successive prefixes, while the intervention branch compares an original action with a substituted one. These quantities need not agree. Theorem 1 establishes additivity for the prefix-value increments, but does not show that this property carries over to the hybrid estimator in Equation (8). Section 4.3 acknowledges the policy difference, yet the relationship between the two targets remains unresolved. This leaves a central part of the attribution claim unsupported.

2. The empirical evidence does not isolate the claimed mechanisms.

The judge comparison changes both the visible information and the selectable steps: the prefix condition sees only the first 60% of the trajectory, while the full-trace condition sees every step and the outcome. Later selections may therefore reflect newly available candidates. In the supplementary records, 15 of 18 later selections by Qwen3.6-plus and 14 of 15 by Qwen3-coder-plus fall beyond the prefix cutoff. A comparison using identical candidate steps is needed before attributing this displacement to collider bias.

I am also unconvinced by the conclusion that increasing replay is unlikely to help. The absence of significant step effects at the tested budget does not establish a limit on estimation precision, and Appendix D.7 acknowledges that the relevant sampling prediction remains untested.

I would raise my score if the authors resolve the attribution-target mismatch, address the candidate-set confound, and support or appropriately narrow the conclusion about additional replay.

Questions for authors

1. Clarification of the attribution target. What common population quantity do the two branches of Equation (8) estimate, and what connects the intervention branch to the additive decomposition in Equation (6)? If the targets are intentionally different, please specify which attribution and additivity claims apply to each.

2. Validation of the judge interpretation. Does the displacement persist when both conditions select from exactly the same candidate steps? Separately varying access to the continuation and the final outcome would help distinguish the proposed mechanism from the effect of making additional steps available. This control would materially affect the assessment of the collider-bias claim.

3. Verification of the reported hybrid experiment. Could the authors provide or point to the branch labels, original and substituted actions, continuation outcomes, and analysis code for the main 190-point experiment? The uploaded summaries are helpful, but they do not show how each reported estimate was produced by the two branches of Equation (8).

5 comments
AuthorsOpenReview ↗

Response to W1: Restricting Theorem 1 Additivity to Policy Increments and Separating Estimand Symbols in Eq. (8) (Data R1)

Dear Reviewer GAR6, thank you for stating the exact criteria to raise your score. Below we address every weakness and question individually using Data R1 and Data R3 .

Weakness 1: The hybrid estimator lacks a common attribution target, and Theorem 1 additivity does not carry over to Eq. (8)

We agree and have revised Section 4.3 to restrict Theorem 1's telescoping additivity (Eq. 6) strictly to the observational prefix-value increments Δtπ = vt − vt−1\Delta_t^\pi = v_t - v_{t-1} under π\pi, using distinct symbols (Δ^tπ\hat{\Delta}_t^\pi vs. θ^t(at,a′)\hat{\theta}_t(a_t,a')) in Eq. (8) and reporting both branches separately in Data R1 (Section 5.6) .

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190
AuthorsOpenReview ↗

Response to W2: Resolving the Candidate-Set Confound via Locked-Prefix Control (Data R3) and Narrowing Replay Scaling Claims

Weakness 2: Judge candidate-set confound (60% cutoff vs. full trace) and conclusion on scaling replay budget

Both issues are resolved: (1) locking full-trace judges to select exclusively from the exact same prefix candidate steps (t ≤ ⌊ 0.6T ⌋t \le \lfloor 0.6T \rfloor) in Data R3 proves that significant late displacement persists (+0.194, p=2.4× 10−6p=2.4\times 10^{-6}); and (2) we narrowed our replay claim in the Abstract, Section 1, Section 5.6 (line 408), and Section 6 to state that "at feasible budgets ($m=8$, 12.7h), task-level variance (ICC=0.640) dominates step effects, making naive replay scaling cost-prohibitive without variance-reduction techniques" .

Data R3: Candidate-Locked (t ≤ ⌊ 0.6T ⌋t \le \lfloor 0.6T \rfloor) 2× 22\times 2 Judge Visibility Ablation Across Three Judges (RQ5)

Judge Model (n)Selectable Candidate StepsContinuation (<br>)Outcome (<br>)Mean Displacement vs. Prefix95% Bootstrap CISign Test
gpt-5.5 (70)Locked Prefix HiddenHidden0.000 (Baseline)——
gpt-5.5 (70)Locked Prefix HiddenVisible+0.041[−.012, +.095][-.012, +.095]0.1800.180
gpt-5.5 (70)Locked Prefix VisibleHidden+0.148[+.089, +.208][+.089, +.208]3.1 × 10−53.1 \times 10^{-5}
gpt-5.5 (70)Locked Prefix VisibleVisible+0.194[+.128, +.261][+.128, +.261]2.4 × 10−62.4 \times 10^{-6}
gpt-5.5 (70)Unrestricted VisibleVisible+0.537[+.459, +.610][+.459, +.610]1.1 × 10−161.1 \times 10^{-16}
qwen3.6-plus (23)Locked Prefix VisibleVisible+0.172[+.081, +.264][+.081, +.264]4.9 × 10−44.9 \times 10^{-4}
qwen3-coder (23)Locked Prefix VisibleVisible+0.168[+.074, +.259][+.074, +.259]8.2 × 10−48.2 \times 10^{-4}
AuthorsOpenReview ↗

Response to Q1: Population Quantities of Eq. (8) Branches, Connection to Eq. (6) via Eq. (12), and Applicable Claims (Data R1)

Question 1: What common population quantity do the two branches of Eq. (8) estimate, what connects the intervention branch to Eq. (6), and which claims apply to each?

The two branches estimate distinct population quantities: the observational branch (H^ > τH\hat{H} > \tau_H) estimates $\Delta_t^\pi = \mathbb{E}_\pi

Y ∣ τ1:tY \mid \tau_{1:t}

 - \mathbb{E}_\pi

Y ∣ τ1:t−1Y \mid \tau_{1:t-1}

, whereas the interventional branch (, whereas the interventional branch (\hat{H} \le \tau_H$) estimates the substitution contrast θt(at, a′)\theta_t(a_t, a') under a perturbed policy π~ = (1−ϵ)π + ϵδa′\tilde{\pi} = (1-\epsilon)\pi + \epsilon\delta_{a'} (Assumption 4), connected to Eq. (6) via Eq. (12): $\Delta_t^{\tilde{\pi}} = -\epsilon\theta_t(a_t,a') + (\mathbb{E}

Y∣τ1:tY|\tau_{1:t}

 - \mathbb{E}

Y∣τ1:t−1,atY|\tau_{1:t-1},a_t

)$. Theorem 1's additivity applies only to Δtπ\Delta_t^\pi; RQ4's null test of zero step effect applies to each branch separately, yielding 0/89 and 0/101 BH-FDR survivors (Data R1, Section 4.3) .

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190
Reviewer_msSTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework that distinguishes three targets of coding-agent process evaluation, e.g., how predictable the agent’s next action is, how much its final outcome varies across repeated runs from the same task or execution prefix, and how much an individual step causally changes that outcome. It introduces SCAE, a method that estimates step-level contributions by replaying agent execution from fixed prefixes and comparing outcomes under naturally sampled or explicitly substituted actions. Validation uses 500 file-localization episodes across 12 SWE-bench-verified repositories. The results suggest that execution provenance predicts next actions better than code-graph transitions, execution uncertainty is primarily task-specific, and the replay experiments could not reliably determine how much individual steps affected the final outcome.

Critical strengths
  • The paper holds the judge model and rubric fixed while varying downstream visibility, which demonstrates that attribution depends substantially on the information presented. Replication across three judges strengthens this observation and raises an important concern about the reliability of LLM-based process evaluation.

  • The within-episode permutation baseline preserves the visited objects while removing their temporal order, helping distinguish properties of the working region from genuine sequential patterns. This supports a more precise interpretation of code-graph proximity, helping distinguish sequential behavior from patterns that arise simply from which files the agent visits.

Critical weaknesses
  • The framework combines different estimands without sufficiently establishing their correspondence. The paper moves from action-replacement effects to prefix-conditioned value increments, then combines value increments in the observational branch with alternative-action contrasts in the intervention branch. These quantities need not represent the same contribution: value increments include information revealed by tool outputs, while intervention effects depend on the chosen alternative action. Without a clearer justification of their relationship, the hybrid estimates are difficult to interpret consistently, limiting confidence in SCAE as a unified instrument for step-level causal attribution.

  • The evaluation is restricted to file localization across 12 repositories, with Django accounting for 46% of episodes; moreover, the replay-based attribution analysis covers only 38 instances. File localization excludes state-changing activities such as code editing and testing, which may exhibit substantially different action dependencies and step-level effects. Thus, the empirical evidence is insufficient to support broad conclusions about coding-agent process evaluation beyond this setting.

Questions for authors
  • Do the main findings hold when Django is excluded, and how were the 38 replay instances selected from the 500 episodes?
4 comments
AuthorsOpenReview ↗

Response to W1: Mathematical Correspondence and Stratified Breakdown of Observational vs. Interventional Estimands (Data R1)

Dear Reviewer msST, thank you for recognizing our controlled judge-visibility manipulation and permutation baseline. Below we address every weakness and question individually using explicitly indexed tables (Data R1–Data R2) .

Weakness 1: Combining different estimands (value increments with tool outputs vs. action-replacement contrasts) without establishing correspondence

We now separate the two branches of Eq. (8) into distinct symbols (Δ^tπ\hat{\Delta}_t^\pi vs. θ^t(at,a′)\hat{\theta}_t(a_t,a')) and establish their exact relationship via Eq. (12) (Appendix B.6.2) :

$\Delta_t^\pi = \underbrace{\mathbb{E}

Y ∣ τ1:t−1, do⁡(at)Y \mid \tau_{1:t-1}, \operatorname{do}(a_t)

 - \mathbb{E}_{a \sim \pi}

E[Y ∣ τ1:t−1, do⁡(a)\mathbb{E}[Y \mid \tau_{1:t-1}, \operatorname{do}(a)

]}_{\text{(A) Action effect vs. policy }\pi\text{ baseline}} + \underbrace{\mathbb{E}

Y ∣ τ1:tY \mid \tau_{1:t}

 - \mathbb{E}

Y ∣ τ1:t−1, atY \mid \tau_{1:t-1}, a_t

}_{\text{(B) Observation contribution from tool output }o_t}.$

When local overlap holds (H^(π^t) > τH)(\hat{H}(\hat{\pi}t) > \tau_H), the observational branch estimates Δtπ\Delta_t^\pi. When the policy is deterministic (H^(π^t) ≤ τH)(\hat{H}(\hat{\pi}t) \le \tau_H), Term (A) is 0 under π\pi; Eq. (8) switches to the substitution contrast $\theta_t(a_t, a') = \mathbb{E}

Y ∣ τ1:t−1, do(a′)Y \mid \tau{1:t-1}, \text{do}(a')

 - \mathbb{E}

Y ∣ τ1:t−1, do(at)Y \mid \tau{1:t-1}, \text{do}(a_t)

$ under a perturbed policy π~ = (1 − ϵ)π + ϵδa′\tilde{\pi} = (1 - \epsilon)\pi + \epsilon\delta{a'} (Assumption 4, linked via Δtπ~ = −ϵθt(at, a′) + Term B)\Delta_t^{\tilde{\pi}} = -\epsilon\theta_t(a_t, a') + \text{Term B}). We now write Eq. (8) with distinct symbols (Δ^tπ) vs. (θ^t(at, a′))(\hat{\Delta}_t^\pi) vs. (\hat{\theta}_t(a_t, a')) and report RQ4 stratified by branch in Data R1, showing 0 BH–FDR survivors in each branch independently.

Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)

Branch / Substitution TypeEstimandPoints (n)(n)Share (%)Mean ∥∥Δ^tπ∥ ∥ or ∥∥θ^t∥∥\|\|\hat{\Delta}_t^\pi\| \| or \|\|\hat{\theta}_t\|\|Uncorr. p < 0.05p < 0.05BH-FDR (q = 0.10)(q = 0.10)Survived
Observational (H^ > 0), policy (π)(\hat{H} > 0), policy (\pi)Δ^tπ\hat{\Delta}_t^\pi8946.8%0.0314 / 89 (4.5%)0 / 89
Intervention: Graph-Adjacent (non-oracle)θ^t(at, a′)\hat{\theta}_t(a_t, a')6433.7%0.0283 / 64 (4.7%)0 / 64
Intervention: Oracle Gold-File (a′ ∈ F⋆)(a' \in F^\star)θ^t(at, a⋆)\hat{\theta}_t(a_t, a^\star)3719.5%0.0443 / 37 (8.1%)0 / 37
Total Probed Prefixes—190100%0.03310 / 190 (5.3%)0 / 190
AuthorsOpenReview ↗

Response to W2: Generalization to State-Changing Code-Editing and Test-Execution Workflows on 60 Episodes (Data R2)

Weakness 2: Evaluation restricted to file localization (46% Django, 38 replay instances), excluding state-changing activities (editing and testing)

Our core findings generalize to state-changing code-editing and test-execution workflows (Data R2, right column) . While file localization was chosen to verify 1.00 byte-identical replayability without state-mutation confounds (Section 5.2), we newly evaluated 60 containerized SWE-bench patch-editing (**edit**) and testing (**pytest**) episodes: provenance still outperforms the code graph on first-visit steps (0.294 vs. 0.064), task-level variance dominates step variance (ICC = 0.602), and full-trace judges shift blame late (+0.488, p=1.8× 10−9p=1.8\times 10^{-9}). We also bounded our primary claims in the Abstract, Section 1, and Section 6 .

Data R2: Replication Excluding Django (n=269n=269) and Extension to State-Changing Code-Editing & Testing Episodes (n=60n=60)

Research Question & Headline MetricFull Localization (n = 500)No-Django Localization (n = 269)State-Changing Edit+Test (n = 60)
RQ1: Provenance vs. Strengthened Graph (top-3)0.326 vs. 0.0580.319 vs. 0.061 (Δ = +0.131\Delta = +0.131)0.294 vs. 0.064 (Δ = +0.118\Delta = +0.118)
RQ2: Visited-Region Density vs. 1-Hop Lift30.9 × vs. 0.94 ×30.9 \times vs. 0.94 \times28.4 × vs. 0.96 ×28.4 \times vs. 0.96 \times24.1× vs. 0.98×24.1\times vs. 0.98\times
RQ3: Consecutive Object Self-Repetition Lift1.83 × (p ∼10−141)1.83 \times (p \sim10^{-141})1.79 × (p <10−70)1.79 \times (p <10^{-70})1.74 × (p <10−24)1.74 \times (p <10^{-24})
RQ4: Residualized Task ICC vs. Null0.640 vs. 0.1960.618 vs. 0.204 (p<0.001p<0.001)0.602 vs. 0.215 (p<0.001p<0.001)
RQ4: Step Effects Surviving BH-FDR (q=0.10q=0.10)0 / 1900 / 1050 / 60
RQ5: Full-Trace vs. Prefix Judge Displacement+0.537 (p=1.1 × 10−16p=1.1 \times 10^{-16})+0.519 (p=4.2 × 10−12p=4.2 \times 10^{-12})+0.488 (p=1.8 × 10−9p=1.8 \times 10^{-9})
AuthorsOpenReview ↗

Response to Q1: Replication Excluding Django (n=269, Data R2) and Deterministic Headroom Selection of the 38 Replay Instances

Question 1: Do the main findings hold when Django is excluded, and how were the 38 replay instances selected from the 500 episodes?

Yes, all findings hold when excluding Django (n=269n=269 across 11 repositories, Data R2 middle column), and the 38 replay instances were deterministically selected via pre-execution headroom filters : (i) because single-target binary tasks are solved at 93.99% with zero headroom (Δt ≡ 0\Delta_t \equiv 0, Appendix C.6), we filtered for multi-target instances (∣F⋆∣ ≥ 2|F^\star| \ge 2) with non-zero recall (YrecY_{\mathrm{rec}}) headroom and length T ≥ 5T \ge 5, yielding the 38 instances (190 prefixes, added to Section 5.2 and Appendix D.1) .

Data R2: Replication Excluding Django (n=269n=269) and Extension to State-Changing Code-Editing & Testing Episodes (n=60n=60)

Research Question & Headline MetricFull Localization (n = 500)No-Django Localization (n = 269)State-Changing Edit+Test (n = 60)
RQ1: Provenance vs. Strengthened Graph (top-3)0.326 vs. 0.0580.319 vs. 0.061 (Δ = +0.131\Delta = +0.131)0.294 vs. 0.064 (Δ = +0.118\Delta = +0.118)
RQ2: Visited-Region Density vs. 1-Hop Lift30.9 × vs. 0.94 ×30.9 \times vs. 0.94 \times28.4 × vs. 0.96 ×28.4 \times vs. 0.96 \times24.1× vs. 0.98×24.1\times vs. 0.98\times
RQ3: Consecutive Object Self-Repetition Lift1.83 × (p ∼10−141)1.83 \times (p \sim10^{-141})1.79 × (p <10−70)1.79 \times (p <10^{-70})1.74 × (p <10−24)1.74 \times (p <10^{-24})
RQ4: Residualized Task ICC vs. Null0.640 vs. 0.1960.618 vs. 0.204 (p<0.001p<0.001)0.602 vs. 0.215 (p<0.001p<0.001)
RQ4: Step Effects Surviving BH-FDR (q=0.10q=0.10)0 / 1900 / 1050 / 60
RQ5: Full-Trace vs. Prefix Judge Displacement+0.537 (p=1.1 × 10−16p=1.1 \times 10^{-16})+0.519 (p=4.2 × 10−12p=4.2 \times 10^{-12})+0.488 (p=1.8 × 10−9p=1.8 \times 10^{-9})

All revisions in the paper are marked in blue font. We sincerely appreciate the reviewer's constructive suggestions and remain committed to further improving our work.

Dear Reviewer,

This is a brief note to kindly invite you to consult our rebuttal and revised manuscript, where we address the points raised in your review in detail. Please let us know if any clarification would be helpful.

Thank you again for your time and consideration.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to F1: Comprehensive View: How We Shrink the Candidate Measurement Space to the Three Representatives (Data R8)

Dear Reviewer msST,

Thank you very much for this insightful follow-up question. Providing a top-down view of the candidate measurement space—and explaining both how we narrow it down to our representative metrics and why the Δtπ\Delta_t^\pi decomposition has exactly two summands—greatly sharpens the conceptual foundation of our paper. Below we address both points directly and summarize the measurement taxonomy in Data R8 (which we will incorporate into Section 3 and Appendix B).


1. Comprehensive View: How We Shrink the Candidate Measurement Space to the Three Representatives (Data R8)

Along an agent trajectory τ1:T = (A1, O1, …, AT, OT)\tau_{1:T} = (A_1, O_1, \dots, A_T, O_T) with outcome YY, process evaluation can be parameterized along two axes: (i) the conditioning information set (prefix τ1:t−1\tau_{1:t-1} vs. full trace τ1:T\tau_{1:T}) and (ii) the target object (next transition At+1A_{t+1}, conditional outcome distribution P(Y ∣ τ1:t)P(Y \mid \tau_{1:t}), or single-step intervention do(At)do(A_t)).

To shrink the broad space of existing heuristics and metrics to a minimal, non-redundant representative at each level, we apply three filtering criteria:

  1. Prefix-Observability (No Future Leakage): The metric must condition only on information available at step tt (τ1:t\tau_{1:t}), ruling out hindsight-contaminated scores when diagnosing in-flight behavior.

  2. Resolution / Non-Degeneracy under Nulls: The metric must remain sensitive on realistic coding trajectories (e.g., avoiding metrics that collapse to 00 due to 93.99%93.99\% binary ceiling effects or metrics that are permutation invariants of the visited multiset).

  3. Actionable System Mapping: Each level's representative must map to a distinct engineering lever (scaffolding/retrieval vs. compute allocation vs. step credit assignment).

Data R8: Shrinking the Process-Evaluation Measurement Space across the Three Levels

Evaluation Level & Target QuestionBroad Candidate Measurements in LiteratureWhy Alternative Candidates Are Filtered Out (Empirical / Theoretical Evidence)Selected Representative Measurement & Why It Is Optimal
1. Action Level<br>(What governs the policy's next move At+1A_{t+1}?)• Raw tool/object reuse rates<br>• Code-graph$k$-hop proximity<br>• Unnormalized Top-kkaccuracy<br>• Tool-routed provenance distribution π^t(At+1∣τ1:t)\widehat{\pi}_t(A_{t+1}\mid\tau_{1:t})• Reuse rates have zero variance under within-episode permutation (SD=0\text{SD}=0, Section 5.5).<br>• Code-graph proximity equals permutation chance (1-hop lift=0.94×1\text{-hop lift}=0.94\times, Section 5.4).<br>• Unnormalized Top-kkignores candidate pool inflation.Closed-Form Top-kk, ECE, & Effective Pool exp⁡(H)\exp(H)on π^t\widehat{\pi}_t(Section 5.3):<br>Captures both predictive accuracy and candidate compression (39.9 → 12.839.9 \to 12.8), directly guiding candidate pruning.
2. Task Level<br>(Where does outcome uncertainty P(Y∣τ1:t)P(Y\mid\tau_{1:t})reside?)• Pre-execution forecaster v0v_0<br>• Action-level token/bucket entropy H^(π^t)\widehat{H}(\widehat{\pi}_t)<br>• Raw unresidualized replay variance<br>• Mean-residualized replay variance UtexecU_t^{\text{exec}}& ICC• Pre-execution v0v_0<br>is uninformative (AUROC=0.481\text{AUROC}=0.481, Appendix D.8).<br>• Action entropy H^(π^t)\widehat{H}(\widehat{\pi}_t)fails to proxy outcome risk (ρ=−0.088\rho=-0.088, Section 5.6).<br>• Raw variance is mechanically coupled to the mean vt(1−vt)v_t(1-v_t)<br>.Residualized Replay Variance Utexec = Var(Yrec∣τ1:t)U_t^{\text{exec}} = \text{Var}(Y_{\text{rec}}\mid\tau_{1:t})& Instance ICC (Section 5.6):<br>Isolates true outcome stochasticity from mean-variance coupling, revealing that 64.0%64.0\%of variance is task-level.
3. Step Level<br>(Did step ttcausally change YY, and what do judges measure?)• Data-flow reachability<br>• Restart-from-scratch ablation<br>• Top-1 disagreement D1D_1<br>• Binary outcome Δt(Y0/1)\Delta_t(Y_{0/1})<br>• Recall-based Δtπ / θt\Delta_t^\pi / \theta_t& Judge Displacement D3D_3• Data-flow misses empty-search redirections (Appendix B.7).<br>• Restart ablation discards realized prefix τ1:t−1\tau_{1:t-1}(Level II vs. II.5).<br>• D1D_1is$\approx 97%$even under the null (Appendix D.10).<br>• Binary Y0/1Y_{0/1}has zero headroom on 93.99%93.99\%of tasks (Appendix C.6).Prefix Increment Δtπ\Delta_t^\pi<br>/ Substitution θt(at,a′)\theta_t(a_t,a')on YrecY_{\text{rec}}, plus Positional Displacement D3D_3(Sections 4 & 5.7):<br>Maximizes intervention sensitivity (2.5052.505vs. 1.6431.643, 98.6%98.6\%headroom) and isolates judge conditioning shift without a noisy reference.
AuthorsOpenReview ↗

Response to F2: Why the  Equation Has Exactly Two Summands—and Whether Any Other Summands Exist

In our response to W1 (and Appendix B.6.2, Eq. 11–12), we wrote the step increment $\Delta_t^\pi := v_t - v_{t-1} = \mathbb{E}_\pi

Y ∣ τ1:tY \mid \tau_{1:t}

 - \mathbb{E}_\pi

Y ∣ τ1:t−1Y \mid \tau_{1:t-1}

$ as:

$\Delta_t^\pi = \underbrace{\Big(\mathbb{E}

Y ∣ τ1:t−1, At=atY \mid \tau_{1:t-1}, A_t{=}a_t

 - \mathbb{E}

Y ∣ τ1:t−1Y \mid \tau_{1:t-1}

\Big)}_{\text{Summand (A): Action-selection effect}} + \underbrace{\Big(\mathbb{E}

Y ∣ τ1:t−1, At=at, Ot=otY \mid \tau_{1:t-1}, A_t{=}a_t, O_t{=}o_t

 - \mathbb{E}

Y ∣ τ1:t−1, At=atY \mid \tau_{1:t-1}, A_t{=}a_t

\Big)}_{\text{Summand (B): Observation-revelation effect}}$

(1) Are there any other summands we need to consider?

Mathematically, under our Structural Causal Model (Section 4.1), these two summands are exhaustive (zero residual), unless context compaction occurs (which introduces a third, confounding summand):

  • Exactness via Filtration / Doob Martingale Decomposition: In one agent–environment step, the transition from prefix τ1:t−1\tau_{1:t-1} to τ1:t = (τ1:t−1, At, Ot)\tau_{1:t} = (\tau_{1:t-1}, A_t, O_t) consists of only two sequential events before the next step begins:

    1. The agent samples action At ∼ π(⋅ ∣ St)A_t \sim \pi(\cdot \mid S_t) (generating the sub-step filtration Ft−1/2 = σ(τ1:t−1, At)\mathcal{F}_{t-1/2} = \sigma(\tau_{1:t-1}, A_t)).

    2. The environment executes AtA_t and returns observation Ot = E(At, G; Uenv(t))O_t = \mathcal{E}(A_t, G; U_{\text{env}}^{(t)}) (generating Ft = σ(τ1:t−1, At, Ot)\mathcal{F}_t = \sigma(\tau_{1:t-1}, A_t, O_t)).

    3. The agent updates its internal state St+1 = T(St, At, Ot)S_{t+1} = \mathcal{T}(S_t, A_t, O_t).

  • What about a potential 3rd Summand (State-Update / Compaction Effect)? One could insert a third telescoping term for the state update St+1S_{t+1}:

    $\text{Summand (C)} := \mathbb{E}

Y ∣ τ1:t, St+1Y \mid \tau_{1:t}, S_{t+1}

 - \mathbb{E}

Y ∣ τ1:tY \mid \tau_{1:t}

.$Under Assumption 1 (Context Observability, no compaction), 

  • When Summand (C) becomes non-zero: As formalized in Appendix B.5, if a harness performs stochastic context compaction φ(ht)\varphi(h_t) or calls an external stateful memory, Summand (C) ≠ 0\text{Summand (C)} \neq 0 represents the state-compaction effect. Thus, testing Assumptions 1 and 3 is precisely what guarantees that Summands (A) and (B) exhaust 100%100\% of Δtπ\Delta_t^\pi.

(2) Why are Summand (A) and Summand (B) the most important to separate?

Separating Δtπ\Delta_t^\pi into Summand (A) and Summand (B) is critical for two reasons:

  1. They Diagnose Two Opposite Engineering Remedies (Policy vs. Environment/Tools):

    • Summand (A) (Action Effect) measures how much the agent's choice of command ata_t changed expected success relative to what policy π\pi would otherwise have sampled at τ1:t−1\tau_{1:t-1} (under Proposition 1, $\mathbb{E}
Y∣τ1:t−1, do(At=at)Y\mid\tau_{1:t-1}, do(A_t{=}a_t)

 - \mathbb{E}_{a\sim\pi}

E[Y∣τ1:t−1, do(At=a)\mathbb{E}[Y\mid\tau_{1:t-1}, do(A_t{=}a)

]$). A negative Summand (A) diagnoses a planning / policy failure (the agent chose a poor command when better commands were in support).

*   **Summand (B) (Observation Effect)** holds the chosen action $A_t = a_t$ fixed and measures how much the _returned tool output_ $o_t$ shifted expected success. Even when the agent takes the deterministic optimal action ($\text{Summand A} = 0$), Summand (B) can be large if the tool output reveals unexpected repository structure or an uninformative error—diagnosing a **tool/indexing or environment-information bottleneck** rather than a policy error.
    

2. They Explain Why Observational Value Increments (Δtπ\Delta_t^\pi) and Interventional Contrasts (θt\theta_t) Differ:

*   When the policy is deterministic at step $t$ ($\widehat{H}(\widehat{\pi}_t) = 0$), **Summand (A) vanishes to** $0$, so the observational increment $\Delta_t^\pi$ measures _only_ Summand (B) (information revealed by $o_t$). To measure the causal effect of changing the action itself at a deterministic step, one _must_ perturb the action to $a'$ to evaluate $\theta_t(a_t, a')$, which directly recovers the counterfactual action contrast (Summand A under $\tilde{\pi}$).
    

We will add Data R8 and this explicit filtration explanation of Summands (A), (B), and (C) to Section 3 and Appendix B.6.2. Thank you again for pushing us to make this foundational logic explicit!

AuthorsOpenReview ↗

Follow-up: Two New Responses Posted — F1 (Data R8: Shrinking the Candidate Measurement Space) and F2 (Why the Δ Eq. Has Exactly Two Summands)

Dear Reviewer msST,

Following up on your remaining questions, we have posted two new responses in this thread:

  1. Response to F1 — Comprehensive View: How We Shrink the Candidate Measurement Space to the Three Representatives (Data R8). It parameterizes process evaluation along two axes (conditioning information set × target object) and applies three filtering criteria — prefix-observability (no future leakage), resolution / non-degeneracy under nulls, and actionable system mapping — to justify why exactly these three representatives are selected at the action, task, and step levels.

  2. Response to F2 — Why the (Δtπ)\Delta_t^\pi) Equation Has Exactly Two Summands—and Whether Any Other Summands Exist. It gives a filtration / Doob-martingale argument showing that, under the SCM in Section 4.1, the action-selection and observation-revelation terms are exhaustive for a single step (zero residual), and identifies context compaction as the one case that would introduce an additional term.

Please let us know if any further details would be helpful for your final assessment. Thank you again for your time and feedback during the discussion period.

Sincerely,

The Authors of #4723

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.