- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether a coding agent’s natural-language prose within a work-block can predict whether the next user message will report that something is broken, failing, or not working. Using 34,046 real coding-agent work-blocks, the authors define the next-user failure report as an observable feedback target that is distinct from execution errors, and compare structured features (context, developer history, and work statistics) with agent prose.
The results show that adding prose improves average precision from 0.211 to 0.258, with the gain persisting after excluding blocks containing explicit error/apology cues. Prose-only prediction also outperforms the feature baseline under user and temporal holdouts. At a 10% review budget, the combined model captures 35.5% of subsequent reports versus 30.2% for the feature baseline. Overall, the paper shows that agent prose contains incremental predictive information that can improve offline review allocation, but it does not establish code correctness or demonstrate that acting on the predictions improves downstream outcomes.
Critical strengths
-
The problem is practically relevant and carefully scoped.
The paper clearly distinguishes user-reported problems from execution errors and code correctness, making the target more representative of real coding-agent feedback. -
The empirical evaluation is reasonably comprehensive.
The authors test lexical-cue robustness, user holdouts, temporal splits, a zero-shot critic baseline, and fixed-budget review allocation, providing several complementary pieces of evidence that prose adds predictive value.
Critical weaknesses
-
The prediction target is noisy and does not directly correspond to agent-generated defects.
A user report may reflect a persistent issue, configuration problem, or unrelated failure rather than a problem caused by the preceding agent block. The labels are also automated, with only limited human validation, making it difficult to interpret the AP gains as genuine defect-risk detection. -
The paper mainly demonstrates offline ranking gains, not downstream utility.
Capturing more future reports within a fixed review budget is promising, but the paper does not show that using the ranking to trigger actual review or intervention reduces user-reported failures, improves code quality, or lowers review cost.
Questions for authors
-
What information in the prose actually drives the predictive gain?
Can the authors disentangle signals such as task difficulty, unresolved issues, missing verification, uncertainty, or explicit problem descriptions? Controlled paraphrase or matched-agent analyses could help distinguish genuine semantic information from superficial linguistic patterns. -
Can the ranking be evaluated prospectively through intervention?
For example, could reviewers inspect the top-ranked 10% of blocks and compare final failure rate, review cost, and code quality against random or feature-baseline selection? This would establish whether better offline prediction translates into more effective review.