- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CodeInFlight, a benchmark for evaluating coding agents under changing user intent. The benchmark constructs interaction instances from execution-validated source tasks and evaluates the same final required behavior under three delivery conditions: the final request is given upfront, the change is given after an initial declared completion, or the change arrives while the agent is still working. The benchmark covers several interaction families, including constraint updates, goal revisions, cancellation/replacement, related additions, and unrelated additions. The evaluation measures whether agents can incorporate request changes while preserving still-valid progress and satisfying the final set of required behaviors.
Critical strengths
-
The paper tackles a timely problem: coding agents handling changing user requests.
-
The benchmark covers several change types, including updates, revisions, cancellations, and additions.
-
The test categories help measure whether agents satisfy new requirements without preserving obsolete ones.
Critical weaknesses
I agree that changing user intent is an important setting. However, the key question is whether the benchmark construction faithfully captures this setting, and whether the benchmark reveals a meaningful capability gap in LLM-based coding agents.
-
First, the construction method raises validity concerns. CodeInFlight creates changing-intent interactions by splitting and recombining acceptance targets. This makes the benchmark controllable, but it is unclear whether the resulting initial requests and later updates are natural. In goal revision and cancellation/replacement cases, for example, the initial request may be a counterfactual subset of the task rather than a plausible prior user goal. A withdrawn target may reflect a convenient target split rather than a realistic requirement that a user would later retract. Similar concerns apply to additions or revisions, which may partly depend on target grouping choices. The paper should either validate the naturalness of the synthesized interactions or narrow its claim to controlled target changes.
-
Second, even if this scenario is common, the paper should better justify why this benchmark is needed. CodeInFlight should reveal something that standard repository-level coding benchmarks cannot. Note that frontier LLMs show only modest performance drops under changing-intent conditions, which is not yet suprising. The paper should clarify whether the benchmark exposes a distinct capability gap, meaningful sensitivity to change timing, or a new empirical phenomenon; otherwise, its value beyond a realistic task setting remains under-specified.
Another concern is the unclear added value of the newly mined real-repository tasks. The paper does not show what these tasks add beyond external-derived public issue-resolution tasks once both are converted into constructed changing-intent interactions. This matters because the Core evaluation is mostly external-derived: 177 / 287 instances overall and 83 / 98 cancellation instances, while real-repository tasks contribute only 48 instances. The real-repository tasks also do not appear to provide richer multi-target structure, with a median of only 1 acceptance target compared with 3 for external-derived tasks and 5 for controlled tasks. The paper should clarify the independent value of real-repository mining, ideally through source-type ablations or targeted analysis.
Questions for authors
-
How do the authors validate that the constructed initial requests and later updates are natural user-intent changes, rather than artifacts of splitting and recombining acceptance targets? In particular, for goal revision and cancellation/replacement, how often does the withdrawn target correspond to a plausible prior user requirement?
-
What new empirical insight does CodeInFlight provide beyond existing repository-level coding benchmarks? Given that frontier LLMs show only modest performance drops under changing-intent conditions, what distinct capability gap is the benchmark intended to expose?
-
What specific value do newly mined real-repository tasks add beyond external-derived public issue-resolution tasks, especially given their smaller contribution to the Core set and their lower median number of acceptance targets?
5 comments
Weakness 3 and Q3: "What specific value do newly mined real-repository tasks add beyond external-derived public issue-resolution tasks, especially given their smaller contribution to the Core set and their lower median number of acceptance targets?"
Real-repository tasks show that the finding is not an artifact of public issue-resolution data, and they add repositories that public corpora do not cover. To check the first point directly, we followed your suggestion and split the result by source for the eight strong configurations of Table 3.
| Source | Instances | Sequential Resolved | In-Flight minus Sequential |
|---|---|---|---|
| Real-repository | 48 | 88% | −4.0 |
| External-derived | 176 | 82% | −3.7 |
| Controlled | 62 | 60% | −3.0 |
Points of Resolved on the same instances and tests, In-Flight at half-way. The real-repository and external-derived losses are significant under the paper's paired bootstrap.
The loss on newly mined tasks is at least as large as on tasks derived from public corpora, and their failure rate rises by 31% under In-Flight, so the phenomenon does not depend on where the tasks come from. The second point shows up in coverage, since the 48 real-repository instances come from 48 different repositories, with Go as their most common language. As for the lower median number of acceptance targets, a single target in these tasks is typically a whole feature checked by many tests, so a median of one target does not mean a small task. Core contains fewer of them because of the cap of two tasks per repository and the structural requirements of each change family, while the release contains 634 of the 1,551 tasks. The revision adds the source split to Section 4.2 (Figure 6), with the failure rate and the ratio of lost to rescued instances for each source.
Thank you again for the precise questions. The new results answer all three of them: domain experts judge about nine in ten constructed changes plausible, the loss exposes a capability gap that a fixed-specification benchmark cannot observe, and it is largest on the newly mined real-repository tasks. If you agree that these concerns are resolved, we would be most grateful if you would take this into account in your final rating. We remain happy to discuss any remaining point.
Thank the authors for the additional experiments. However, I remain concerned about the benchmark's novelty and the value of added real-repository tasks.
We greatly appreciate the careful reading and the precise questions, and we are glad that you found the problem timely and the coverage of change types valuable. Your description of Core's composition is also correct. Your three questions pointed exactly at what the original manuscript left unproven, so we designed the new experiments around them, and the results answer them directly. We respond in the order of your review.
Weakness 1 and Q1: "it is unclear whether the resulting initial requests and later updates are natural ... How do the authors validate that the constructed initial requests and later updates are natural user-intent changes ... how often does the withdrawn target correspond to a plausible prior user requirement?"
The constructed changes are natural in about nine of ten cases, according to a blind human annotation, and the main finding holds on the natural cases. To establish this, three domain experts in software engineering rated every evaluated goal-revision and cancellation instance in Core, the two families you single out, together with 36 constraint updates. Each annotator saw the initial request and the later message and rated how plausible it is that a developer on this codebase would send that message (1 to 5).
Agreement among annotators is high (Fleiss κ = 0.86). To check that the annotators did not rate leniently, we mixed in 43 control pairs whose later message was taken from a different task, and none of them was rated plausible. Two LLM judges, applied to all 242 pairs in the four families built by splitting acceptance targets, agree: both rate 86% of them plausible and none of the controls.
The answer to your specific question follows from the table. For goal revision and cancellation, the withdrawn target corresponds to a plausible prior requirement in about nine of ten cases. In addition, the main result does not depend on the remaining tenth, since on the pairs rated plausible the eight strong configurations lose 4.1 points, against 3.6 on all of Core under the same pooling. The revision describes the setting as controlled, task-grounded request changes and adds the annotation results to Section 3.2 (Table 2), with details in Appendix E.
Weakness 2 and Q2: "What new empirical insight does CodeInFlight provide beyond existing repository-level coding benchmarks? ... what distinct capability gap is the benchmark intended to expose?"
CodeInFlight exposes all three things you list. It reveals a new phenomenon, in that every strong configuration does worse when the same change arrives during execution than after a first attempt, and In-Flight is the lowest of the three conditions for each of them. Beyond the phenomenon itself, the effect is sensitive to timing, with a loss of 4.8 points when the change arrives at 1/4 of the work that nearly vanishes at 3/4. Most relevant to your question, it isolates a distinct capability gap. The loss concentrates on cancellations that replace work in progress (9.7 points at 1/4), and it is not a matter of effort or instructions, since agents that reason by default still lose, and an explicit reminder to keep valid work and finish what is owed does not close the gap either. The reminder experiment makes this concrete.
285 and 97 instances. The cancellation entries are significant.
Even with the reminder, the agent remains well below Sequential on cancellations. A repository-level benchmark with a fixed specification observes only the Upfront condition, so none of these effects is visible there, and this is the gap that CodeInFlight is designed to fill.