acceptodds
Under review as a conference paper at ICLR 2027

CodeInFlight: Can LLMs Code Well under Changing User Intentions?

Abstract

LLM-based coding agents increasingly work alongside users who revise their requests before implementation is complete. Such revisions can invalidate an agent’s current plan or edits, requiring it to redirect unfinished work while preserving progress that remains useful. However, existing repository-level benchmarks largely assume fixed specifications or introduce new requirements only between completed turns, and therefore do not directly evaluate adaptation to changes that arrive during execution. To address this gap, we introduce CodeInFlight, a dataset and repository-level benchmark for evaluating how coding agents adapt to changing user requests while they work. We curate 1,551 execution-validated source tasks spanning 534 repositories and ten programming languages, from which we derive a stratified evaluation set of 287 interaction instances covering requirement changes, cancellation with replacement, and additional tasks. For each instance, UPFRONT, SEQUENTIAL, and IN-FLIGHT reveal the final request before coding, after the initial attempt, or during execution, respectively. Comparing each of 19 configurations with itself, we find that all six configurations resolving at least half of the instances when a change follows their first attempt resolve fewer when it arrives during execution. They generate 11% fewer output tokens but resolve 3.9 points fewer instances, for both withdrawals and additions. In paired runs of eight open-weight models, thinking raises resolution under both deliveries by similar amounts. CodeInFlight thus indicates that these higher-performing configurations adapt less reliably to changes during execution than to changes after completion.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Large Language Models, Benchmarks and Datasets, Human-Agent Collaboration, AI Agents, Human-AI Interaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4f1YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CodeInFlight, a benchmark for evaluating coding agents under changing user intent. The benchmark constructs interaction instances from execution-validated source tasks and evaluates the same final required behavior under three delivery conditions: the final request is given upfront, the change is given after an initial declared completion, or the change arrives while the agent is still working. The benchmark covers several interaction families, including constraint updates, goal revisions, cancellation/replacement, related additions, and unrelated additions. The evaluation measures whether agents can incorporate request changes while preserving still-valid progress and satisfying the final set of required behaviors.

Critical strengths
  • The paper tackles a timely problem: coding agents handling changing user requests.

  • The benchmark covers several change types, including updates, revisions, cancellations, and additions.

  • The test categories help measure whether agents satisfy new requirements without preserving obsolete ones.

Critical weaknesses

I agree that changing user intent is an important setting. However, the key question is whether the benchmark construction faithfully captures this setting, and whether the benchmark reveals a meaningful capability gap in LLM-based coding agents.

  • First, the construction method raises validity concerns. CodeInFlight creates changing-intent interactions by splitting and recombining acceptance targets. This makes the benchmark controllable, but it is unclear whether the resulting initial requests and later updates are natural. In goal revision and cancellation/replacement cases, for example, the initial request may be a counterfactual subset of the task rather than a plausible prior user goal. A withdrawn target may reflect a convenient target split rather than a realistic requirement that a user would later retract. Similar concerns apply to additions or revisions, which may partly depend on target grouping choices. The paper should either validate the naturalness of the synthesized interactions or narrow its claim to controlled target changes.

  • Second, even if this scenario is common, the paper should better justify why this benchmark is needed. CodeInFlight should reveal something that standard repository-level coding benchmarks cannot. Note that frontier LLMs show only modest performance drops under changing-intent conditions, which is not yet suprising. The paper should clarify whether the benchmark exposes a distinct capability gap, meaningful sensitivity to change timing, or a new empirical phenomenon; otherwise, its value beyond a realistic task setting remains under-specified.

Another concern is the unclear added value of the newly mined real-repository tasks. The paper does not show what these tasks add beyond external-derived public issue-resolution tasks once both are converted into constructed changing-intent interactions. This matters because the Core evaluation is mostly external-derived: 177 / 287 instances overall and 83 / 98 cancellation instances, while real-repository tasks contribute only 48 instances. The real-repository tasks also do not appear to provide richer multi-target structure, with a median of only 1 acceptance target compared with 3 for external-derived tasks and 5 for controlled tasks. The paper should clarify the independent value of real-repository mining, ideally through source-type ablations or targeted analysis.

Questions for authors
  1. How do the authors validate that the constructed initial requests and later updates are natural user-intent changes, rather than artifacts of splitting and recombining acceptance targets? In particular, for goal revision and cancellation/replacement, how often does the withdrawn target correspond to a plausible prior user requirement?

  2. What new empirical insight does CodeInFlight provide beyond existing repository-level coding benchmarks? Given that frontier LLMs show only modest performance drops under changing-intent conditions, what distinct capability gap is the benchmark intended to expose?

  3. What specific value do newly mined real-repository tasks add beyond external-derived public issue-resolution tasks, especially given their smaller contribution to the Core set and their lower median number of acceptance targets?

5 comments

We greatly appreciate the careful reading and the precise questions, and we are glad that you found the problem timely and the coverage of change types valuable. Your description of Core's composition is also correct. Your three questions pointed exactly at what the original manuscript left unproven, so we designed the new experiments around them, and the results answer them directly. We respond in the order of your review.

Weakness 1 and Q1: "it is unclear whether the resulting initial requests and later updates are natural ... How do the authors validate that the constructed initial requests and later updates are natural user-intent changes ... how often does the withdrawn target correspond to a plausible prior user requirement?"

The constructed changes are natural in about nine of ten cases, according to a blind human annotation, and the main finding holds on the natural cases. To establish this, three domain experts in software engineering rated every evaluated goal-revision and cancellation instance in Core, the two families you single out, together with 36 constraint updates. Each annotator saw the initial request and the later message and rated how plausible it is that a developer on this codebase would send that message (1 to 5).

Change familyPairsRated plausible by a majority
Goal revision (all in Core)2492%
Cancellation and replacement (all evaluated in Core)9790%
Constraint update3697%

Agreement among annotators is high (Fleiss κ = 0.86). To check that the annotators did not rate leniently, we mixed in 43 control pairs whose later message was taken from a different task, and none of them was rated plausible. Two LLM judges, applied to all 242 pairs in the four families built by splitting acceptance targets, agree: both rate 86% of them plausible and none of the controls.

The answer to your specific question follows from the table. For goal revision and cancellation, the withdrawn target corresponds to a plausible prior requirement in about nine of ten cases. In addition, the main result does not depend on the remaining tenth, since on the pairs rated plausible the eight strong configurations lose 4.1 points, against 3.6 on all of Core under the same pooling. The revision describes the setting as controlled, task-grounded request changes and adds the annotation results to Section 3.2 (Table 2), with details in Appendix E.

Weakness 2 and Q2: "What new empirical insight does CodeInFlight provide beyond existing repository-level coding benchmarks? ... what distinct capability gap is the benchmark intended to expose?"

CodeInFlight exposes all three things you list. It reveals a new phenomenon, in that every strong configuration does worse when the same change arrives during execution than after a first attempt, and In-Flight is the lowest of the three conditions for each of them. Beyond the phenomenon itself, the effect is sensitive to timing, with a loss of 4.8 points when the change arrives at 1/4 of the work that nearly vanishes at 3/4. Most relevant to your question, it isolates a distinct capability gap. The loss concentrates on cancellations that replace work in progress (9.7 points at 1/4), and it is not a matter of effort or instructions, since agents that reason by default still lose, and an explicit reminder to keep valid work and finish what is owed does not close the gap either. The reminder experiment makes this concrete.

GPT-6-Astra, change at 1/4All typesCancel and replace
In-Flight minus Sequential−4.9−13.4
In-Flight with reminder minus Sequential−2.8−8.2

285 and 97 instances. The cancellation entries are significant.

Even with the reminder, the agent remains well below Sequential on cancellations. A repository-level benchmark with a fixed specification observes only the Upfront condition, so none of these effects is visible there, and this is the gap that CodeInFlight is designed to fill.

Weakness 3 and Q3: "What specific value do newly mined real-repository tasks add beyond external-derived public issue-resolution tasks, especially given their smaller contribution to the Core set and their lower median number of acceptance targets?"

Real-repository tasks show that the finding is not an artifact of public issue-resolution data, and they add repositories that public corpora do not cover. To check the first point directly, we followed your suggestion and split the result by source for the eight strong configurations of Table 3.

SourceInstancesSequential ResolvedIn-Flight minus Sequential
Real-repository4888%−4.0
External-derived17682%−3.7
Controlled6260%−3.0

Points of Resolved on the same instances and tests, In-Flight at half-way. The real-repository and external-derived losses are significant under the paper's paired bootstrap.

The loss on newly mined tasks is at least as large as on tasks derived from public corpora, and their failure rate rises by 31% under In-Flight, so the phenomenon does not depend on where the tasks come from. The second point shows up in coverage, since the 48 real-repository instances come from 48 different repositories, with Go as their most common language. As for the lower median number of acceptance targets, a single target in these tasks is typically a whole feature checked by many tests, so a median of one target does not mean a small task. Core contains fewer of them because of the cap of two tasks per repository and the structural requirements of each change family, while the release contains 634 of the 1,551 tasks. The revision adds the source split to Section 4.2 (Figure 6), with the failure rate and the ratio of lost to rescued instances for each source.

Thank you again for the precise questions. The new results answer all three of them: domain experts judge about nine in ten constructed changes plausible, the loss exposes a capability gap that a fixed-specification benchmark cannot observe, and it is largest on the newly mined real-repository tasks. If you agree that these concerns are resolved, we would be most grateful if you would take this into account in your final rating. We remain happy to discuss any remaining point.

Thank the authors for the additional experiments. However, I remain concerned about the benchmark's novelty and the value of added real-repository tasks.

Reviewer_xQayOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

LLM-based coding agents increasingly collaborate with users who revise their requirements as development progresses. Most existing repository-level benchmarks assume fixed requirements or introduce new ones only after a work attempt is complete, making it difficult to directly assess whether agents can adapt when users change their requests during execution. The authors introduce CodeInFlight, which covers three types of changes: requirement changes, cancellations with replacements, and additional tasks.

Critical strengths
  1. The benchmark fills an important gap in evaluation and is timely given the growing prominence of coding tasks.
  2. The paper’s tables and experimental design are insightful.
Critical weaknesses
  1. The dataset construction pipeline is not described clearly enough. A more detailed explanation of the end-to-end process would strengthen the credibility of the benchmark.
  2. Figures 2 and 3 use an overly rich mix of colors and visual elements, which makes the underlying workflows harder to follow rather than clearer. Simplifying the visual design and emphasizing the key steps and transitions would improve readability.
Questions for authors
  1. I have no major concerns about the experiments or the benchmark construction. My main reservation is that the benchmark’s scope seems somewhat narrow.
2 comments

We sincerely appreciate your positive assessment and the concrete suggestions, and we are glad that you found that the benchmark fills an important and timely gap and that the tables and experimental design are insightful. We adopted all of your suggestions in the revision, and we hope the revisions further strengthen your positive view of the work.

W1: "The dataset construction pipeline is not described clearly enough. A more detailed explanation of the end-to-end process would strengthen the credibility of the benchmark."

Thank you for this suggestion. We agree, and the revision adds an end-to-end pipeline figure to Section 3.1, together with a step-by-step box in Appendix B (Figure B.1). In the original manuscript these steps were described across Section 3 and two appendices, which made the overall flow hard to see, so we now present them as five steps in order, matching the new figure.

  1. Collect source tasks from authored projects, commits mined from public repositories, and public issue-resolution corpora.
  2. Run the tests of each task in its own environment, with a failing initial state (observed directly for tier-1 tasks), a passing reference change, a negative control without the reference, verifier repeatability, and per-test scoring.
  3. Remove duplicates by repository and commit, upstream identifier, patch digest, and normalized commit subject. This yields 1,551 validated tasks.
  4. Split the goal. For each task that can support a change family (for example, at least two acceptance targets on one behavioral unit for a constraint update), split the acceptance targets into an initial and an arriving set along file or test boundaries, and fix the final required, withdrawn and regression tests before any run. This leaves 1,390 eligible tasks.
  5. Draw Core by change type and source, with language coverage and at most two tasks per repository, and freeze it before any model is evaluated. This yields 287 instances from 329 source tasks.

Read in this order, the pipeline narrows the pool step by step, from 1,551 validated tasks to 1,390 eligible ones and finally to the 287 instances in Core, and the new figure shows where each reduction happens.

W2: "Figures 2 and 3 use an overly rich mix of colors and visual elements, which makes the underlying workflows harder to follow rather than clearer."

Thank you for this helpful feedback. We agree that both figures tried to show too much at once, and we have redrawn them with a reduced palette and fewer elements. Figure 2 now shows only the three nested sets of the dataset and the three change types with their counts and sources. Figure 3 (Figure 4 in the revision, since the new pipeline figure is now Figure 3) has become a compact table of the three conditions, which shows where the change arrives and what has been built at that moment, with the same final tests for all three. The redrawn figure places the moment the change arrives at the center of each row, which is the transition the figure needs to show.

Q1: "My main reservation is that the benchmark's scope seems somewhat narrow."

The scope is broader than Core suggests, and the revision makes this clearer. Core is a balanced evaluation subset of a release with 1,551 tasks from 534 repositories in 10 languages. Beyond the release itself, the arrival point and the change family are both inputs to construction, so the same pipeline can produce further interactions and timing settings from the released tasks. The new runs at three arrival points, added during this rebuttal, are one example of that flexibility. More broadly, the findings raise a question that reaches beyond our tasks, namely whether an agent can carry a change through work that is already under way. Because the paired design only requires a task, a change and fixed final tests, it transfers directly to other agentic settings, and we hope CodeInFlight encourages researchers to study In-Flight changes in more tasks. The revision states this scope in Section 3.

We are grateful for your support, and we hope the revised paper is now easier to follow.

I thank the authors for their response and for redesigning the figures, which addressed my concerns. However, due to the constraints of ICLR's 4-point rating system, I will maintain my score of 3.

Reviewer_JsyLOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an analysis of how coding agents adapt to instruction revisions during the execution vs. after. They create CodeInFlight, a benchmark to evaluaate this, and conduct various analyses to see how agents adapt, how efficiently they do, etc.

Critical strengths

The benchmark is well-constructed and the analyses (across change types, with and without thinking, etc.) make sense. The three conditions (upfront, sequential, in-flight) make sense. The problem of mid-execution instruction changes makes sense.

Critical weaknesses

The main concern for this work is that it does not contain many insights on shortcomings of coding agents, because the differences between sequential and in-flight and between with or without thinking don't seem to be that big. Further, the key finding that in-flight reduces tokens feels so obvious that it's not clear why analyses were needed.

More detailed comments that touch on this below:

  • Since the paper's main motivation is that real users do indeed modify their requirements in flight, I wish the authors had (a) provided more background on how/when real users actually do this, and since presumably when users do this varies a lot, (b) conducted experiments that span the full gamut of possible timings, to further understand how inflight req communication affects coding agent behavior (not just one setting of "halfway"; L258).
  • L73: the authors motivate their paper by implying that it's important that we measure coding performance on the same requests with the change delivered at different times. But it's not 100% clear why the same task has to be done in multiple conditions, and by extension, why this benchmark is needed compared to existing interactive benchmarks that authors list (e.g., King & Flanigan, 2026; Raghavendra et al., 2026; Wu et al., 2026b). Part of it is that the benchmark framing is perhaps unnecessary; this could be an analysis paper? Another part is that the results in Figure 1 don't actually show that this matters that much?
  • Relatedly, in RQ1, the best models only loose 3.9% performance from an in-flight correction vs. sequential correction. Given how small that is and the 95% confidence interval being 1.5-5.7%, I have a hard time believing that this is a meaningful difference that users might actually perceive.
  • The RQs and experimental results are just not that insightful for coding agent development. In addition to varying when the in-flight request comes and seeing how that affects things, authors could conduct more qualitative analyses of the tasks that agents fail at in-flight but not sequentially.
  • It seems like the answer to RQ4 is actually that the performance difference between inflight vs. sequential is not related to whether the model is thinking vs. not. Since this is a somewhat negative result, it doesn't really tell us much about how coding agents differ when the goal change happens at different time points...
  • Perhaps I'm misunderstanding, but it seems quite obvious to me that in-flight mode would require fewer tokens than sequential, since by definition in-flight "interrupts" the agent coding. For example, it's super obvious that an in-flight cancellation would lead to saving a lot of token vs. waiting till the agent is done. This finding doesn't tell us much about agentic capabilities.

After rebuttals: changed my score from 1 -> 3.

Questions for authors

N/A

5 comments

We sincerely appreciate the detailed and candid review, and we are glad that you found the benchmark well constructed and the three conditions and analyses sensible. Your comments helped us sharpen the paper's emphasis. The original manuscript opened with token savings, while the more important finding concerns accuracy and was diluted in an average over all configurations. The revision is organized around that finding, as we explain below. We hope the responses and the revised manuscript address your concerns, and we answer each point in the order of your review.

Main concern: "the differences between sequential and in-flight and between with or without thinking don't seem to be that big. Further, the key finding that in-flight reduces tokens feels so obvious that it's not clear why analyses were needed."

The differences appear small in the original manuscript because of how the results were aggregated, not because the effect is small. We reported the mean over all 19 configurations, and as a benchmark CodeInFlight deliberately includes many weaker models that fail these tasks under every condition. For these models a change has little left to take away, so they carry little signal about adaptation. The informative comparison is therefore among strong agents, and there the result is consistent: every configuration that solves at least half of the instances does worse when the same change arrives during execution than when it follows a completed attempt, and the loss grows when the change comes earlier. The table below separates the strong configurations from the rest.

Strong configurations (Sequential Resolved ≥ 50%)UpfrontSequentialIn-Flight
Six strong configurations, mean77.677.173.3
GPT-6.1-Sol (new)89.588.885.3
Qwen3.5-397B (new)73.572.771.2

Resolved (%). In-Flight is the lowest of the three conditions for each of the eight configurations. Averaged over the eight, In-Flight is 3.5 points below Sequential and the failure rate rises from 22.0% to 25.6%, a relative increase of 16% (significant).

For a user who changes course while a strong agent is working, this means 16% more failed tasks. As for tokens, we agree that the savings are not the finding, and the revision no longer leads with them. We return to what the token data do show under Comment 6.

Comment 1: "(a) provided more background on how/when real users actually do this, and ... (b) conducted experiments that span the full gamut of possible timings ... (not just one setting of 'halfway'; L258)."

(a) Thank you for this helpful suggestion. We agree that this background was missing, and the revision now provides it. Mid-execution changes are common in deployed coding agents. In Anthropic's analysis of 500k human interruptions of Claude Code, users interrupt 5% to 9% of turns, and changing requirements mid-task is one of the stated reasons (McCain et al., 2026). A public corpus points the same way: in SWE-chat, a collection of real coding-agent sessions, users interrupt 4.5% of turns, and 8% of non-vibe-coding sessions belong to users who redirect goals mid-session (Baumann et al., 2026). Products have followed this behaviour, and GitHub Copilot's coding agent and CLI, Claude Code, and the OpenAI API all let users steer an agent while it works. The revision adds this evidence to Section 1 and Appendix A. Since such changes can arrive at any point of the work, the timing of the change is a natural axis to study, which brings us to your second point.

(b) We agree, and the full range yields a new finding: the earlier the change arrives, the larger the loss. Because the arrival point is a parameter of our harness, going beyond the halfway setting only required running arrivals at 1/4, 1/2 and 3/4 of each agent's own first-finish length, on the same instances and with the same tests. The table shows how much each agent loses at each point.

Agent1/41/23/4
GPT-6.1-Sol−6.0−3.5−0.7
GPT-6-Astra−4.9−2.5−3.2
Qwen3.5-397B−2.7−1.5+2.3
Qwen3.5-9B−5.3−0.7+0.7
Mean−4.8−2.0−0.3

In-Flight minus Sequential, points of Resolved. Upfront and Sequential are the two ends of this range.

Every agent loses most when the change arrives early. On average the cost falls from 4.8 points at 1/4 to 2.0 at 1/2 and nearly disappears close to the end, although GPT-6-Astra still loses 3.2 points at 3/4. A single halfway setting could not have revealed this sensitivity, so the revision adds this curve to Section 4.3 (Figure 7 and Table 4).

Comment 2 (L73): "it's not 100% clear why the same task has to be done in multiple conditions, and by extension, why this benchmark is needed compared to existing interactive benchmarks ... this could be an analysis paper? Another part is that the results in Figure 1 don't actually show that this matters that much?"

The same task must be run in all conditions because only then is the difference caused by timing alone. If the conditions used different tasks, a harder task and a worse adaptation would look the same. This is also where CodeInFlight departs from the interactive benchmarks we cite. They add information between completed turns or in reply to the agent's questions, which corresponds to the Sequential end of our design, so they cannot show that the same agent does worse when the same change arrives earlier. The timing curve in Part 1 is a direct example of a result that only matched instances and fixed tests make measurable. For the same reason we kept the benchmark framing, since the design itself is what others can reuse through the released tasks, the frozen Core, and a harness in which arrival time and change type are inputs. The revision explains this in Section 1 and adds a column "same tests, three arrival times" to Table 1.

Thank you also for the remark on Figure 1. You are right, and we have redrawn it. The original plotted all 19 configurations on a 0 to 100 axis, which hid the effect behind the weaker models. The revised figure focuses on the eight strong configurations and their losses, and its right panel places all 21 configurations by Sequential accuracy, so that every strong agent visibly falls below zero. The timing results now have their own place in Section 4.3, and Section 4.4 shows that cancellations arriving at 1/4 of the work raise the failure rate from 21% to 31%. All 21 configurations remain in Table 3.

Comment 3: "the best models only loose 3.9% performance ... I have a hard time believing that this is a meaningful difference that users might actually perceive."

This is an easily overlooked but meaningful difference, because the loss is concentrated rather than spread thinly across tasks. First, a user who changes a request while a strong agent is working sees 16% more failed tasks. Second, these failures are concentrated by change type. Broken down by type, nearly all of the loss sits on cancellations that replace work in progress, where the change cannot be absorbed by adding code.

Change type, pooled over six agents1/41/23/4
Cancel and replace−9.7−5.3−5.2
The other four types−0.4+0.3+1.5

In-Flight minus Sequential, points of Resolved. All three cancellation entries are significant.

Cancellations cost the most when they arrive early, and even a late one still costs about five points, while the other four types stay close to zero at every arrival point. An average over all change types therefore mixes a large loss on one kind of change with almost none on the others. The pattern also holds agent by agent: for five of the six agents at every arrival point, cancellations cost more than the other change types (Figure 8). A single strong agent shows the same picture. GPT-6-Astra loses at every arrival point, resolving 4.9, 2.5 and 3.2 points fewer at 1/4, 1/2 and 3/4, and 13.4 points fewer on cancellations at 1/4 (Appendix D).

Comment 4: "authors could conduct more qualitative analyses of the tasks that agents fail at in-flight but not sequentially."

Thank you for this suggestion, which led to a new result: strong agents mostly fail by leaving the change unfinished or by breaking work that already passed, and they declare the task complete in 84% of these failures. We examined every instance that one of the eight strong configurations solves under Sequential but fails under In-Flight, 191 cases against 111 in the opposite direction. Their test outcomes fall into three patterns.

What goes wrong (from the test outcomes)All 191Cancellations (73)
The change is not carried through75%56%
Work that already passed is broken20%33%
The agent keeps building the cancelled feature4%11%

In a typical case, after a cancellation arrived, GPT-6.1-Sol reverted the file it was editing and implemented the replacement, whose tests pass, but in doing so broke the existing checks in that file. Because these failures do not occur when the same change follows a completed attempt, they point to a concrete lever for agent design, namely re-checking what must survive a change before finishing. The revision adds this analysis to Section 4 and three case studies to Appendix C.

Comment 5: "It seems like the answer to RQ4 is actually that the performance difference between inflight vs. sequential is not related to whether the model is thinking vs. not. Since this is a somewhat negative result, it doesn't really tell us much ..."

Thank you for this observation. We agree that RQ4 read like a null result, so the revision moves it to Appendix F and states its message more directly: more reasoning makes agents stronger coders, but it does not by itself make them robust to a change in the middle of their work. In eight paired open-weight models, thinking raises In-Flight Resolved by 10 to 33 points and Sequential Resolved by 7 to 27 points. Yet all eight strong configurations in Table 3 still resolve fewer instances when the change arrives during execution, including Claude-Opus-5.5, GPT-6-Astra and GLM-5.3-Flash, which reason by default. A benchmark that measures accuracy under a fixed request would record the gain from reasoning and miss this remaining loss. Seeing it requires delivering the same change at different times, which is what CodeInFlight does.

Comment 6: "it seems quite obvious to me that in-flight mode would require fewer tokens than sequential ... This finding doesn't tell us much about agentic capabilities."

Thank you for this point, which led us to a more informative result. Saving tokens does not buy accuracy, because the work moves after the change rather than shrinking. Across the 21 configurations, the change in tokens and the change in Resolved are uncorrelated (Spearman 0.05), and all strong configurations use fewer tokens while solving fewer instances. Splitting the output at the moment of the change shows why. Under Sequential an agent spends 13.9k output tokens before the change and 2.7k after it, whereas under In-Flight it spends 7.3k before and 8.4k after, three times as much after the change, and still resolves fewer instances. Cancellations save the most tokens (20%) and lose the most accuracy, which ties this observation back to the concentration discussed under Comment 3. The revision therefore no longer presents token savings as a finding and reports this result instead (Section 4.2, Figures 5b and 5c).

Thank you again for pushing us to make the contribution clearer. The revision now addresses your main concerns with new evidence: the loss is consistent across all strong agents, it grows the earlier a change arrives, usage data show that such changes are common in practice, and the failure analysis shows where the agents break. If you find that these results address your concerns, we would sincerely appreciate it if you would consider reflecting them in your score. We would also be glad to answer any further questions during the discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We sincerely appreciate the careful reading and constructive feedback from all three reviewers. The reviewers found the benchmark well constructed, with analyses and three delivery conditions that make sense (JsyL), noted that it fills an important and timely gap with an insightful experimental design (xQay), and found that it tackles a timely problem while covering several types of change (4f1Y).

The reviews also share one central question: whether a change that arrives during execution has a meaningful effect, and whether CodeInFlight reveals behavior that existing benchmarks cannot. We thank the reviewers for raising it, and we agree that the original manuscript did not make the answer clear. It reported the timing gap as an average over all 19 configurations, many of which fail these tasks under every condition, and it placed token savings ahead of accuracy. Such an average dilutes the effect. CodeInFlight deliberately covers a wide range of models, as a benchmark should, yet adaptation can only be observed for agents that are able to complete the tasks. Once the analysis focuses on these strong agents, a consistent result emerges: agents that solve a task when the change follows their first attempt fail more often when the same change arrives while they are still working. We therefore reorganized the manuscript around this finding and added the experiments the reviewers suggested.

Core contributions

  1. A controlled benchmark for request changes during execution. Each instance keeps the same task and final tests and varies only when the change reaches the agent: before coding (Upfront), after a first attempt (Sequential), or during the work (In-Flight). This paired design isolates the effect of timing, which existing benchmarks do not measure.
  2. A reusable dataset and harness: 1,551 execution-validated tasks from 534 repositories in 10 languages, a frozen 287-instance Core over three change types, and a harness in which the arrival point is a parameter.
  3. Several new insights. For example, all eight strong agents we evaluate, from GPT-6.1-Sol and Claude Opus 5.5 to Qwen3.5-397B, solve fewer tasks when the same change arrives mid-execution, with 16% more failed tasks. The loss also grows the earlier the change arrives and concentrates on cancellations of work in progress.

Updates of experimental results during the rebuttal

  • Timing sweep (Section 4.3, Figure 7, Table 4). On the same instances and tests, the loss is 4.8 points when the change arrives at 1/4 of the work, 2.0 at 1/2, and 0.3 at 3/4. [JsyL, 4f1Y]
  • Two more strong agents (Table 3). GPT-6.1-Sol and Qwen3.5-397B also lose under In-Flight, so the loss holds for all eight strong agents. [JsyL, 4f1Y]
  • Change types (Section 4.4, Figure 8). Cancellations that replace work in progress cost 9.7 points at 1/4 and still 5.2 at 3/4, while the other four types stay near zero, and this holds for five of the six agents at every arrival point. [JsyL, 4f1Y]
  • Failure analysis (Section 4.5, Table 6, Appendix C). Of 191 lost instances, 75% leave the change unfinished and 20% break work that already passed, while the agent declares success in 84%. [JsyL]
  • Reminder baseline (Appendix D). Telling the agent to keep valid work and finish what is owed narrows the gap without closing it. [JsyL, 4f1Y]
  • Human annotation (Section 3.2, Table 2, Appendix E). Domain experts rate about nine in ten annotated changes as plausible, and the loss holds on these pairs. [4f1Y]
  • Task sources (Section 4.2, Figure 6). The loss is largest on real-repository tasks, where failures rise by 31%. [4f1Y]

Updates of discussions and presentation

  • Evidence on how often users change requests while an agent is working (Section 1, Appendix A). [JsyL]
  • Figure 1 and the abstract rewritten around accuracy, with token savings as a secondary result. [JsyL]
  • An end-to-end construction pipeline figure (Section 3.1, Appendix B). [xQay]
  • Figures 2 and 4 (formerly Figure 3) redrawn with fewer colors and elements. [xQay]
  • A column "same tests, three arrival times" in the benchmark comparison (Table 1). [JsyL]
  • The thinking analysis moved to Appendix F, framed around how reasoning affects both delivery conditions. [JsyL]

Taken together, these results show that the cost of a change depends on when it arrives and on what it withdraws, which a benchmark with a fixed specification cannot observe. The revision addresses each of these concerns with new evidence, and all changes are highlighted in blue in the updated PDF. We thank the reviewers and the AC for their time and careful assessment.

We sincerely thank all reviewers and the AC for an engaged and constructive discussion. We are grateful that Reviewer JsyL raised the score after the first round, noting that the rebuttal largely addressed the comments, and that Reviewer xQay found the experimental design insightful. The second round focused on two points, the practical size of the effect for strong agents and what CodeInFlight reveals beyond existing benchmarks, and the revision now adds three analyses on them, highlighted in blue in the updated PDF.

Final updates to the revision

  • 16% more failures for strong agents, and up to 31% for individual agents (Appendix J, Figure J.1). Because strong agents rarely fail, a loss of 3–4 points is a large share of their failures. Claude-Opus-5.5 fails 26% more often under In-Flight and GPT-6.1-Sol 31% more often, and when a cancellation arrives at 1/4 of the work, the six agents of the timing study fail 46% more often. [JsyL]
  • A loss that fixed-specification accuracy does not predict (Section 4.2, Figure J.2). Among the eight strong agents, the Spearman correlation between Upfront Resolved and the In-Flight loss is 0.00. Claude-Sonnet-5 and Qwen3.5-397B are 2.2 points apart under Upfront, but they lose 6.7 and 1.5 points under In-Flight, so a fixed-specification ranking cannot reveal this capability. [4f1Y, JsyL]
  • A source-type ablation (Appendix J, Table J.1). The finding holds on every source and on Core without the real-repository tasks, where failures still rise by 15%. Real-repository tasks give the cleanest measurement, since agents solve 88% of them under Sequential, their failures rise by 31%, and 44% of their In-Flight failures are instances that the same agent solves under Sequential. [4f1Y]

Summary of the discussion

The table maps the main questions raised in the reviews to the evidence in the revised manuscript.

QuestionEvidenceLocationReviewers
Is the effect meaningful?Every strong agent loses, with 16% more failures, and the loss is unrelated to Upfront accuracySection 4.2, Table 3, Figures J.1 and J.2JsyL, 4f1Y
Does timing matter?The loss is 4.8 points at 1/4 of the work, 2.0 at 1/2 and 0.3 at 3/4Section 4.3, Figure 7, Table 4JsyL, 4f1Y
Which changes cost most?Cancellations of work in progress cost 9.7 points at 1/4, for five of six agents at every arrival pointSection 4.4, Figure 8JsyL, 4f1Y
How do agents fail?75% leave the change unfinished, 20% break completed work, and 84% are declared completeSection 4.5, Table 6JsyL
Are the changes natural?Experts rate about nine in ten as plausible, and the loss holds on these pairsSection 3.2, Table 2, Appendix E4f1Y
What do real-repository tasks add?The largest rise in failures and the cleanest measurement of the timing effectFigure 6, Table J.14f1Y
Do users change requests mid-task?Users interrupt 5% to 9% of turns in a deployed coding agentSection 1, Appendix AJsyL
What role does thinking play?Thinking raises both conditions, and strong agents that reason by default still lose under In-FlightAppendix FJsyL
Is the presentation clear?A construction pipeline figure, and Figures 2 and 4 redrawn with fewer elementsSection 3.1, Figure 3, Appendix BxQay

Taken together, the discussion has sharpened the central result of the paper. Strong coding agents that solve a task when a change follows their first attempt fail more often when the same change arrives while they are still working, the cost depends on when the change arrives and on what it withdraws, and fixed-specification evaluation cannot reveal it. We thank the reviewers and the AC again for their time and careful assessment, which have made the paper substantially stronger.

AuthorsOpenReview ↗

Closing Summary and Thanks

Dear AC, SAC and reviewers,

As the discussion comes to a close, we would like to thank you for the time and attention you have given to CodeInFlight. During the discussion we addressed each point raised in the reviews with new experiments and analyses, all of which are included in the revised paper. Reviewer JsyL raised the rating from 1 to 3 after the first round, noting that the rebuttal largely addressed the comments, and Reviewer xQay recommends acceptance. Reviewer 4f1Y kindly acknowledged the additional experiments, and in the final round we added the analyses that the review asked for, which show that the loss reflects a capability fixed-specification benchmarks cannot measure and that it is largest on the real-repository tasks. Below we briefly recap what the paper shows and how it has changed, in case this is helpful for the final discussion.

CodeInFlight starts from a simple observation. Users often change their minds while a coding agent is still working, and in a deployed coding agent they interrupt 5% to 9% of turns, but existing benchmarks either fix the request or add information only after the agent has finished a turn. None of the twelve benchmarks in Table 1 keeps the task and the final tests fixed while varying when the change reaches the agent, which is exactly what CodeInFlight does. This comparison gives a consistent result. Every strong agent we tested, from GPT-6.1-Sol and Claude-Opus-5.5 to Qwen3.5-397B, fails more often when the same change arrives while it is still working, with 16% more failures on average. The earlier the change comes, the larger the loss, and it is largest when the change cancels work in progress. In 84% of the lost instances the agent still reports success, so the user is unlikely to notice. How much an agent loses is also unrelated to how well it does when given the full request upfront, which means this ability does not show up on existing leaderboards. With the released tasks and a harness in which the arrival point and the type of change are simple parameters, others can study it further, including ways to help agents adapt.

Much of this evidence was added during the discussion in response to the reviewers' questions. The table below compares the submitted and the revised versions.

Submitted versionRevised version
Size of the effectAn average over all 19 configurationsEvery strong agent loses, with 16% more failures and up to 31% for individual agents (Section 4.2, Figure J.1)
Timing of the changeOne arrival point, at half of the workAny arrival point, with losses of 4.8, 2.0 and 0.3 points at 1/4, 1/2 and 3/4, and of 9.7 points for cancellations at 1/4 (Sections 4.3 and 4.4)
How agents fail—75% leave the change unfinished, 20% break completed work, and 84% are declared complete (Section 4.5)
Naturalness of the changes—Experts rate about nine in ten changes as plausible, and the loss holds on them (Section 3.2)
Relation to standard accuracy—The loss is unrelated to Upfront accuracy, with Spearman ρ = 0.00 (Figure J.2)
Real-repository tasks—A source-type ablation, with the largest rise in failures (+31%) on real-repository tasks (Table J.1)
How often users change requests—Interruption rates of deployed coding agents (Section 1, Appendix A)
Configurations1921, adding GPT-6.1-Sol and Qwen3.5-397B

We are sincerely grateful to all of you. The reviewers read the paper closely and asked the questions that mattered most, from when a change arrives to whether the constructed changes are natural, and answering them has made CodeInFlight a clearer and more convincing paper than the one we submitted. We also thank the AC and SAC for overseeing the discussion and for the considerable effort this process takes. We hope the revised version shows why we find this problem worth studying, and we thank you again for the care you have put into reviewing it.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.