acceptodds
Under review as a conference paper at ICLR 2027

Predicting User-Reported Problems from Coding-Agent Text

Abstract

Can a coding agent's prose help predict whether the user's next message will report a problem? We study 34,046 work-blocks from real coding-agent sessions and predict next-user reports that something is broken, erroring, or not working. These reports are only weakly associated with detected execution errors, making them a distinct feedback target. Adding within-block prose to structured context, developer history, and work statistics raises average precision from 0.211 to 0.258. The additive gain persists after excluding blocks with specified lexical cues. In separate user and temporal holdouts, prose alone also outperforms the feature baseline. At a 10% review budget, the combined model captures 35.5% of subsequent reports versus 30.2% for the baseline. The results establish prose as an incremental signal for offline review allocation, without claiming code correctness or intervention benefit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Coding agents, Human–AI interaction, User feedback prediction, Agent evaluation, Human Review prioritization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fDAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a coding agent’s natural-language prose within a work-block can predict whether the next user message will report that something is broken, failing, or not working. Using 34,046 real coding-agent work-blocks, the authors define the next-user failure report as an observable feedback target that is distinct from execution errors, and compare structured features (context, developer history, and work statistics) with agent prose.

The results show that adding prose improves average precision from 0.211 to 0.258, with the gain persisting after excluding blocks containing explicit error/apology cues. Prose-only prediction also outperforms the feature baseline under user and temporal holdouts. At a 10% review budget, the combined model captures 35.5% of subsequent reports versus 30.2% for the feature baseline. Overall, the paper shows that agent prose contains incremental predictive information that can improve offline review allocation, but it does not establish code correctness or demonstrate that acting on the predictions improves downstream outcomes.

Critical strengths
  1. The problem is practically relevant and carefully scoped.
    The paper clearly distinguishes user-reported problems from execution errors and code correctness, making the target more representative of real coding-agent feedback.

  2. The empirical evaluation is reasonably comprehensive.
    The authors test lexical-cue robustness, user holdouts, temporal splits, a zero-shot critic baseline, and fixed-budget review allocation, providing several complementary pieces of evidence that prose adds predictive value.

Critical weaknesses
  1. The prediction target is noisy and does not directly correspond to agent-generated defects.
    A user report may reflect a persistent issue, configuration problem, or unrelated failure rather than a problem caused by the preceding agent block. The labels are also automated, with only limited human validation, making it difficult to interpret the AP gains as genuine defect-risk detection.

  2. The paper mainly demonstrates offline ranking gains, not downstream utility.
    Capturing more future reports within a fixed review budget is promising, but the paper does not show that using the ranking to trigger actual review or intervention reduces user-reported failures, improves code quality, or lowers review cost.

Questions for authors
  1. What information in the prose actually drives the predictive gain?
    Can the authors disentangle signals such as task difficulty, unresolved issues, missing verification, uncertainty, or explicit problem descriptions? Controlled paraphrase or matched-agent analyses could help distinguish genuine semantic information from superficial linguistic patterns.

  2. Can the ranking be evaluated prospectively through intervention?
    For example, could reviewers inspect the top-ranked 10% of blocks and compare final failure rate, review cost, and code quality against random or feature-baseline selection? This would establish whether better offline prediction translates into more effective review.

Reviewer_V87NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to predict the risk that the user's next message will report a problem based on existing interaction records. Through empirical tests, it shows that using structured features and agent prose together can improve predictive performance. Building on this, the paper shows through offline evaluation that the method can help users determine which work-blocks should be prioritized for inspection.

Critical strengths

The paper investigates a valuable research question: whether the user's next message will report a problem. In practical coding collaboration, such predictions could help developers identify in advance which work-blocks need closer attention and determine inspection priorities when review resources are limited.

Critical weaknesses

There is still a clear gap between the current experiments and the intended practical use. The results mainly show that the model can better predict and rank subsequent user failure reports offline, but they do not establish whether these predictions can be used effectively during real interactions or whether they improve inspection efficiency or final task outcomes.

The methodological novelty is relatively limited, as the main approach is based on existing text representation and classification techniques. In this setting, the completeness of the baselines is important, but the paper does not include a small LLM baseline under the same data splits and supervision setting, making it difficult to judge how competitive the proposed approach is against modern text models.

The main results rely on automatically generated failure-report labels, but the validation of the ground truth is still limited. The human validation is small-scale and uses only a single human annotator, while the reported agreement is only around kappa=0.65, indicating non-negligible annotation disagreement. Therefore, the current validation may not be sufficient to fully support the reliability of the experimental conclusions.

Questions for authors

Could the authors add an experiment closer to a realistic human-agent interaction setting to test whether prediction-guided inspection actually improves inspection efficiency or final task outcomes? For example, it would be useful to quantify how much inspection effort or time a user could save while allowing only a certain fraction of failure reports to be missed.

Could the authors include a small LLM baseline under the same data splits and supervision setting? This would help clarify whether the proposed approach remains competitive with modern text models.

Could the authors provide more extensive human validation and examine whether the main performance gains still hold on human-validated labels? If possible, using multiple independent annotators would also provide a stronger assessment of ground-truth reliability.

Reviewer_A3xgOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to determine whether you can predict if a user message will report some error beyond just execution errors from the agent message preceding it. They find that including the prose rather than just taking a bunch of features (prior feedback, developer history, agent activity) improves the performance from .211 to .258 in average precision. The target is user-reported problems.

Critical strengths

Uses real interactions and isolates the improvement from the prose. The TF-IDF/SVD and gradient-boosting is a cheap way to avoid an additional LLM call at prediction time.

Critical weaknesses

It's mainly not clear to me what the contribution is. Is it just that text contains information that's missing from summary statistics? That feels like a pretty intuitive result that doesn't quite warrant a paper. The results are also not compelling enough to be of any surprise. If thge contribution of the paper is JUST that adding agent-message text helps predict user-level errors compared to NOT having the agent-level text, that feels very trivial.

Questions for authors

What is the takeaway/scientific contribution of the paper? Since the text is directly relevant to the response, an improvement over summary statistics seems expected. What substantive insight does the paper establish beyond just quantifying the improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.