- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The main goal of this work is to demonstrate that frontier coding agents and LLMs struggle to adapt when the repository and issue they are working on have had async changes introduced by external factors like a co-author making changes, some tool not behaving as it would at test time, the addition of more files and new evidence, etc.
They define 12 such intervening conditions that might affect agent performance, which are realistic and well motivated.
The key distinction that they make is to measure this performance starting from a common checkpoint. They use an agent to complete the task up to a certain point, save the state at that point, and then rerun Clean and Shifted (intervened) versions of the task starting from this point.
Their results show that the best frontier coding agents are substantially affected by the shifts.
Critical strengths
- I think the results show big drops in performance when shifts are introduced, which shows a big gap in current LLM coding agents.
- The 12 categories of possible shifts introduced in the benchmark design are well motivated and realistic.
Critical weaknesses
- It is quite unclear how the tasks were actually created and how quality was ensured. There is a mention of using GPT-5.6 Sol for creating the "first" 24 tasks, but it is unclear how they were actually created. I am guessing this was an agentic process but that process is not described in the main paper.
- The authors mention recoverability checks, but how were these done? Are they also based on tests written by the same gpt-5.6-sol that was used to create these interventions?
- The results section is difficult to follow for readers.
- For a dataset/benchmark contribution, the authors should include an independent controlled study with humans in the loop to ensure that the tasks are recoverable and high quality.
Questions for authors
- For a quality check, if the agent is made fully aware of what exactly changed in the intervention can it succesfully solve the task usually? This could be a good way to ensure that the intervention does not break the task.
- What prompts were used for the data construction? What is the exact agentic process for creation?
3 comments
Closing Check-in Before Discussion Deadline (New 36-Scenario External Human Study)
Dear Reviewer NLYM,
With the author-reviewer discussion period closing tomorrow (Nov 18 AoE), we wanted to briefly check in regarding your soundness concerns—especially your recommendation for an independent human-in-the-loop verification study (W4) and full pipeline/prompt transparency (W1, W2, Q2).
Following your suggestion, we conducted a blind external software-engineer study across 36 stratified scenarios (F01–F12) using only public task interfaces, achieving a 100.0% (36/36) human recovery pass rate (, Appendix A Table 4), alongside full 6-stage role-isolated prompts (§3.1) and the full-awareness diagnostic doubling PSR (6.25% -> 13.40%, Q1).
If time permits in these final hours, we would deeply appreciate knowing whether these additions resolve your soundness concerns or if anything further can be clarified. Thank you very much for your time!
Thank you for the detailed responses and additional experiments.
Thank you for the detailed response and the new experiments. Here are my comments on each of the points:
W1. The six-stage pipeline answers my question about how the scenarios were created. Please include as much as possible in the main paper.
W2. The blind reference solution shows the tasks can still be solved after the change, so this is partly addressed. But my main question is still open: were the checker scripts, and the solutions used to test them, written by the same model that created the changes? Please say this clearly, and say how many checkers your team reviewed by hand and how many you had to fix.
W3 is satisfactorily answered.
W4: Thank you for running this study. It is what I asked for, and if the details hold up it addresses my main concern. Please report how many engineers took part, how they were recruited, and whether they solved each task themselves, in one attempt, with the same information the agents got. Please also clarify what the agreement scores were computed on: if not the four criteria (which are all 100% for Goal Interaction, making kappa undefined), please describe the items rated and how often engineers’ judgments matched the automated checkers. Finally, what happened to the scenarios rated unclear or not discoverable?
Q1,2: Table R2 is helpful. Agents keep about 80% of their Clean successes when fully informed, which together with the human study suggests the changes generally do not break the tasks.
Overall, the human study and the fully-informed experiment address much of my soundness concern, and I am raising my overall score and soundness score. My remaining questions are mainly about reporting, and I would like to see them answered in the final version.
Follow-up on Our New Human-in-the-Loop Study & Construction Pipeline Details (Parts 1–5)
Dear Reviewer NLYM,
Thank you very much for your candid and constructive feedback—your suggestions directly guided several of the most important improvements in our revised manuscript. In case helpful while navigating Parts 1–5 of our response, we wanted to briefly highlight how we addressed each of your core concerns:
operator-proposer, blindreference-recovery, andevidence-reviewer), and shows that recoverability is verified by deterministic container scripts (accepting 2 distinct valid recoveries and rejecting 4 negative controls—never LLM judges), with GPT-5.6 Sol ranking 4th overall (11.30% PSR).As we progress through the discussion phase, we would greatly appreciate hearing your thoughts on whether the independent human study, full-awareness solvability results, and pipeline documentation resolve your concerns, or if there is any further evidence we can provide. Thank you again for your time and consideration!
Best regards, The Authors