- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors proposes that process evaluation for coding agents conflates three distinct targets, i.e., next-action prediction, task-level execution uncertainty, and step-level causal attribution. To this end, in this paper, they introduce SCAE, a prefix-conditioned replay estimator derived from a structural causal model, with an additive anytime-value decomposition and a hybrid observational procedure selected by measured local action entropy. The experimental results on 500 SWE-bench file-localization episodes from 12 repositories, show that execution provenance predicts first-visit actions better than dependency-graph features. In addition, results also presents that graph structure describes the visited region rather than local transitions, replay variance is structured mainly by instance identity, and exposing downstream trace information moves full-trace judge blame substantially later. The paper interprets the last result as evidence that full-trace judging captures downstream-informed semantic relevance rather than prefix-conditioned causal contribution.
Critical strengths
The paper makes a useful conceptual contribution by separating action, task, and step attribution and making their conditioning sets explicit. Such a distinction is relevant to both process-reward evaluation and LLM-as-a-judge attribution. The prefix-only versus full-trace visibility manipulation is particularly strong. This method changes only the judge's information set, replicates the direction of displacement across three judges, and provides a falsifiable mechanism-level result without relying on an unvalidated causal selector. The authors also report failed preregistered criteria, and state where the assumptions break, which improves the reproducibility of the analysis.
Critical weaknesses
None of the 190 estimated effects survives BH-FDR, many are exactly zero. The low-entropy interventions use oracle actions under a perturbed policy, so they do not establish step effects for the deployed policy. The judge-isolation experiment also changes outcome visibility together with downstream-trace visibility, while the conclusions generalize from one gpt-5.5 file-localization setup dominated by django and small attribution subsets.
Questions for authors
Refer to Weaknesses. some additional questions:
- What fraction of the main attribution estimates uses the observational branch vs. the soft-intervention branch, and how sensitive are the results to the entropy threshold and the choice of substituted actions?
- How should we interpret SCAE estimates under Assumption 4 when the substituted action is an oracle action or lies only in a perturbed-policy support? Which reported quantities remain valid for the deployed policy itself?
- The paper seems to say that the full-trace judge sees both the downstream trace and the realized outcome Y, while the prefix-only judge sees neither. Can the authors provide or discuss a matched outcome-visibility control, such as a 2x2 ablation, to separate downstream/collider effects from the effect of revealing Y?
8 comments
Response to Q1: Observational (46.8%) vs. Interventional (53.2%) Share and Entropy-Threshold Sensitivity Sweep (Data R5)
Question 1: Observational vs. soft-intervention fraction and sensitivity to entropy threshold and substituted actions
Across the 190 points, 46.8% (89/190) use the observational branch () and 53.2% (101/190) use the soft-intervention branch (, Data R1). As shown in Data R5 below, sweeping nats and separating non-oracle (64) from oracle (37) actions yields 0 BH-FDR survivors across all settings (added to Appendix C.4).
Data R5: Sensitivity of Step-Level Attribution to Entropy Threshold and Substitution Action Type
| Setting / Threshold () | Observational Share () | Interventional Share () | Mean | Uncorr. | BH-FDR () |
|---|---|---|---|---|---|
| nats (Default) | 89 / 190 (46.8%) | 101 / 190 (53.2%) | 0.031 / 0.034 | 10 / 190 (5.3%) | 0 / 190 |
| nats | 59 / 190 (31.1%) | 131 / 190 (68.9%) | 0.033 / 0.033 | 11 / 190 (5.8%) | 0 / 190 |
| nats | 27 / 190 (14.2%) | 163 / 190 (85.8%) | 0.035 / 0.032 | 9 / 190 (4.7%) | 0 / 190 |
| Non-Oracle Interventions Only | — | 64 / 101 (63.4% of intv.) | — / 0.028 | 3 / 64 (4.7%) | 0 / 64 |
| Oracle Interventions Only | — | 37 / 101 (36.6% of intv.) | — / 0.044 | 3 / 37 (8.1%) | 0 / 37 |
Response to Q2: Counterfactual Sensitivity Under Assumption 4 and Quantities Strictly Valid for Deployed Policy \pi
Question 2: Interpreting SCAE estimates under Assumption 4 and validity for the deployed policy
Under Assumption 4, measures local counterfactual outcome sensitivity to action replacement under , rather than natural variation under . For the deployed policy itself, all observational quantities remain strictly valid without Assumption 4: (i) all RQ1 provenance and tool-routing results ( steps), (ii) all RQ2–RQ3 graph-region density and permutation-null reuse results ( episodes), (iii) RQ4 task-level replay variance ICC (0.640), (iv) the 89 observational step increments (Data R1), and (v) all RQ5 judge results (Data R3).
Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)
| Branch / Substitution Type | Estimand | Points | Share (%) | Mean | Uncorr. | BH-FDR Survived |
|---|---|---|---|---|---|---|
| Observational | 89 | 46.8% | 0.031 | 4 / 89 (4.5%) | 0 / 89 | |
| Intervention: Graph-Adjacent (non-oracle) | 64 | 33.7% | 0.028 | 3 / 64 (4.7%) | 0 / 64 | |
| Intervention: Oracle Gold-File | 37 | 19.5% | 0.044 | 3 / 37 (8.1%) | 0 / 37 | |
| Total Probed Prefixes | — | 190 | 100% | 0.033 | 10 / 190 (5.3%) | 0 / 190 |
Data R3: Candidate-Locked () Judge Visibility Ablation Across Three Judges (RQ5)
Judge Model (n) | Selectable Candidate Steps | Continuation (<br>) | Outcome (<br>) | Mean Displacement vs. Prefix | 95% Bootstrap CI | Sign Test |
|---|---|---|---|---|---|---|
gpt-5.5 (70) | Locked Prefix | Hidden | Hidden | 0.000 (Baseline) | — | — |
gpt-5.5 (70) | Locked Prefix | Hidden | Visible | +0.041 | ||
gpt-5.5 (70) | Locked Prefix | Visible | Hidden | +0.148 | ||
gpt-5.5 (70) | Locked Prefix | Visible | Visible | +0.194 | ||
gpt-5.5 (70) | Unrestricted | Visible | Visible | +0.537 | ||
qwen3.6-plus (23) | Locked Prefix | Visible | Visible | +0.172 | ||
qwen3-coder (23) | Locked Prefix | Visible | Visible | +0.168 |
Response to W1: Task-Level Variance Structure, Non-Oracle Splits (Data R1), 2x2 Judge Control (Data R3), and Cross-Task Generalization (Data R2)
Dear Reviewer wqm7, thank you for your positive evaluation (
Weak acceptance). Below we address your weakness and each question individually using Data R1–Data R3 and Data R5.Weakness 1: 0/190 effects survive BH-FDR, low-entropy oracle interventions under perturbed policy, judge outcome-visibility confound, and gpt-5.5/Django/small-subset generalization
We address each aspect directly: (1) the 0/190 FDR result is our core empirical finding that variance is structured at the task level (ICC = 0.640) rather than the step level, which holds on the 89 observational + 64 non-oracle points alone (Data R1); (2) our 2× 2 judge ablation (Data R3) separates downstream-trace collider bias (+0.148, p=3.1× 10−5) from outcome Y (+0.041, n.s.); and (3) all findings replicate after excluding Django (n=269) and on 60 state-changing code-editing/testing episodes (Data R2).
Data R1: Stratified Breakdown of the 190 Step-Effect Estimates by Branch and Oracle Status (RQ4)
Data R2: Replication Excluding Django (n=269) and Extension to State-Changing Code-Editing & Testing Episodes (n=60)
Data R3: Candidate-Locked (t ≤ ⌊ 0.6T ⌋) 2× 2 Judge Visibility Ablation Across Three Judges (RQ5)
n)gpt-5.5(70)gpt-5.5(70)gpt-5.5(70)gpt-5.5(70)gpt-5.5(70)qwen3.6-plus(23)qwen3-coder(23)Data R5: Sensitivity of Step-Level Attribution to Entropy Threshold τH and Substitution Action Type