- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Switch-SWE, a benchmark for evaluating whether coding agents can resume unfinished tasks after working on other tasks within a shared conversation. The benchmark contains 200 focus tasks from 19 Python repositories, each evaluated under matched uninterrupted and switched conditions. The switched sessions interleave the focus task with four other tasks, while preserving separate task-specific workspaces. Across seven model–agent configurations, the authors observe an average decrease of 11.5 percentage points in focus-task resolution. They also introduce a Resumption Context Manager (RCM), which combines task-relevant history routing with explicit task-state memory to mitigate this degradation.
Critical strengths
-
The paper addresses an important and practically relevant problem. Coding agents increasingly operate in ongoing conversations where users move between unfinished tasks. Evaluating whether agents can recover earlier requirements and implementation progress is a useful extension beyond isolated issue resolution.
-
The paired evaluation protocol has several thoughtful controls. The uninterrupted and switched conditions preserve the focus-task requirements, base revision, and verifier. Separate workspaces prevent interruption tasks from directly overwriting the focus task’s code, helping distinguish conversational interference from filesystem conflicts.
-
The empirical finding is consistent across multiple systems. All seven evaluated model–agent configurations exhibit lower focus-task resolution under switching. The inclusion of several native coding harnesses makes the observation more informative than a single-model demonstration.
Critical weaknesses
-
The primary metric evaluates only the final focus-task outcome. Although Table 2 reports aggregate execution costs and Table 6 describes the scheduled interruptions, neither establishes how consistently agents make substantive progress on the interruption tasks. Different levels of engagement could change the amount and nature of intervening history, complicating comparisons of focus-task recovery across systems. Likewise, RCM’s improved focus-task performance does not establish whether progress on the other tasks is preserved. This does not demonstrate that any evaluated agent neglects interruptions, but it leaves an important alternative explanation insufficiently examined. Per-interruption execution statistics, meaningful progress measures, and rates of skipped or prematurely abandoned requests would strengthen the evaluation. Since these tasks are intentionally left unfinished, stage-level progress would be more informative than requiring full completion.
-
The experiments do not sufficiently distinguish unfinished-task resumption from general long-context interference. The switched condition simultaneously introduces additional tasks, longer interaction histories, more tool outputs, and potentially more frequent context compaction. The results support the conclusion that interleaving tasks under the proposed protocol reduces focus-task resolution, but do not yet isolate the contribution of resuming unfinished tasks. Comparisons against histories of completed tasks or unrelated interactions with comparable context lengths would help distinguish pending-task state management from general context interference. Further analysis across interruption counts, return intervals, and compaction events would clarify the sources of degradation. These controls would also strengthen the interpretation of RCM’s gains by showing whether they primarily arise from recovering unfinished task state or reducing irrelevant context.
Questions for authors
Please see the weakness.