acceptodds
Under review as a conference paper at ICLR 2027

Switch-SWE: Do Coding Agents Hold Focus When Switching Between Unfinished Tasks?

Abstract

Software development sessions often involve several unfinished tasks rather than one issue carried through to completion. As coding agents take on repository-level development, they need to recover an earlier task's requirements and partial progress after intervening work. Existing repository-level benchmarks typically score one issue-to-patch episode and do not directly measure whether an agent can resume a task after advancing other unfinished work in the same repository. We introduce , a benchmark for focus-task resumption in sessions that interleave unfinished tasks drawn from the same source repository, with a persistent checkout for each task. Each focus task is evaluated in matched uninterrupted and switched sessions with identical focus requirements and executable verifiers. The switched session interleaves ordered stages from other unfinished tasks in the same repository, while only the final focus patch is scored. Switch-SWE contains 200 focus tasks from 19 Python repositories, reconstructed from issue, pull-request, and review histories and rendered as 400 matched session specifications. Across 7 coding agents, interleaved unfinished work produces an average focus-task resolution gap of 11.5 percentage points, ranging from 6.0 to 15.0 percentage points. Trajectory analysis identifies cross-task requirement confusion and loss of task state as recurrent resumption behaviors. We introduce a plug-in context-management module that selects task-relevant history and maintains concise task-state summaries. On OpenCode, the module improves focus-task completion under same-repository task switching by 5.5 to 15.5 percentage points across the evaluated models.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, Harness Evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_wQmAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Switch-SWE, a benchmark for evaluating whether coding agents can resume unfinished tasks after working on other tasks within a shared conversation. The benchmark contains 200 focus tasks from 19 Python repositories, each evaluated under matched uninterrupted and switched conditions. The switched sessions interleave the focus task with four other tasks, while preserving separate task-specific workspaces. Across seven model–agent configurations, the authors observe an average decrease of 11.5 percentage points in focus-task resolution. They also introduce a Resumption Context Manager (RCM), which combines task-relevant history routing with explicit task-state memory to mitigate this degradation.

Critical strengths
  • The paper addresses an important and practically relevant problem. Coding agents increasingly operate in ongoing conversations where users move between unfinished tasks. Evaluating whether agents can recover earlier requirements and implementation progress is a useful extension beyond isolated issue resolution.

  • The paired evaluation protocol has several thoughtful controls. The uninterrupted and switched conditions preserve the focus-task requirements, base revision, and verifier. Separate workspaces prevent interruption tasks from directly overwriting the focus task’s code, helping distinguish conversational interference from filesystem conflicts.

  • The empirical finding is consistent across multiple systems. All seven evaluated model–agent configurations exhibit lower focus-task resolution under switching. The inclusion of several native coding harnesses makes the observation more informative than a single-model demonstration.

Critical weaknesses
  • The primary metric evaluates only the final focus-task outcome. Although Table 2 reports aggregate execution costs and Table 6 describes the scheduled interruptions, neither establishes how consistently agents make substantive progress on the interruption tasks. Different levels of engagement could change the amount and nature of intervening history, complicating comparisons of focus-task recovery across systems. Likewise, RCM’s improved focus-task performance does not establish whether progress on the other tasks is preserved. This does not demonstrate that any evaluated agent neglects interruptions, but it leaves an important alternative explanation insufficiently examined. Per-interruption execution statistics, meaningful progress measures, and rates of skipped or prematurely abandoned requests would strengthen the evaluation. Since these tasks are intentionally left unfinished, stage-level progress would be more informative than requiring full completion.

  • The experiments do not sufficiently distinguish unfinished-task resumption from general long-context interference. The switched condition simultaneously introduces additional tasks, longer interaction histories, more tool outputs, and potentially more frequent context compaction. The results support the conclusion that interleaving tasks under the proposed protocol reduces focus-task resolution, but do not yet isolate the contribution of resuming unfinished tasks. Comparisons against histories of completed tasks or unrelated interactions with comparable context lengths would help distinguish pending-task state management from general context interference. Further analysis across interruption counts, return intervals, and compaction events would clarify the sources of degradation. These controls would also strengthen the interpretation of RCM’s gains by showing whether they primarily arise from recovering unfinished task state or reducing irrelevant context.

Questions for authors

Please see the weakness.

Reviewer_e3euOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Switch-SWE, a benchmark for evaluating task resumption in coding agents. It constructs matched uninterrupted and switched sessions: the former completes the focus task continuously, while the latter interleaves four unfinished tasks from the same repository. Only the final focus-task patch is scored. The benchmark contains 200 focus tasks from 19 Python repositories and 400 session specifications.

Across seven model-agent configurations, switching produces an average resolution gap of 11.5 percentage points and increases computational cost. The paper further proposes RCM, which routes task-relevant history and maintains task-state memory. It substantially improves switched resolution for two models under OpenCode.

Critical strengths
  1. Clear and practically relevant problem definition. Task resumption is distinct from ordinary issue resolution, and the matched-session design isolates it explicitly.
  2. Careful evaluation protocol. The two conditions share the focus requirements, base revision, verifier, and task-specific checkout, while only the focus patch is scored.
  3. Interpretable intervention. RCM separates task-history routing from task-state memory, making its benefits easier to analyze.
Critical weaknesses

The switching effect is partially confounded. Switched sessions contain more tasks and longer histories, so the results do not fully separate genuine resumption difficulty from additional context and computation burden. Limited evaluation of RCM. RCM is tested only with OpenCode and two models, while adding auxiliary calls. Cost- or token-matched comparisons are not sufficiently reported. Potential subjectivity in task reconstruction. Multi-stage requirements are reconstructed from issue, PR, and review histories. The paper describes auditing, but does not clearly report the scale, error rate, or inter-annotator agreement of this validation. The locality analysis is not causal. Near and far conditions use different focus-task groups, so observed differences may reflect task or repository composition rather than source locality itself.

Questions for authors

How do the authors separate resumption difficulty from the effects of longer context and higher token consumption? How does RCM compare with native OpenCode under matched token, model-call, or API-cost budgets? How many task stages and requirement-verifier mappings were manually reviewed, and what was the construction error rate?

Reviewer_XNzKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Switch-SWE evaluates unfinished-task resumption through paired uninterrupted and switched coding sessions. It also proposes a Resumption Context Manager (RCM) combining task-relevant history selection and task-state memory.

Critical strengths
  1. Covers 200 focus tasks from 19 Python repositories with executable verification.
  2. Evaluates seven model–agent configurations across five harnesses, revealing performance and ranking changes under switching.
Critical weaknesses
  1. Unclear Novelty: The benchmark’s novelty over existing work on task switching and resumption is not sufficiently clear. Its distinctive contributions and insights need stronger justification.
  2. Limited method validation: RCM is evaluated only on two models within OpenCode, without comparisons against existing memory-management approaches or simple persistent task records.
Questions for authors
  1. Beyond Appendix E’s qualitative cases, how common are the failure modes? How do context growth, compaction events, and harness-specific behavior relate to outcomes?
  2. Can RCM work effectively beyond OpenCode, and how does it compare with simpler task-memory approaches under comparable budgets?
  3. Could cross-directory semantic dependencies affect the interpretation of Near/Far results? This is a secondary concern.
Reviewer_qDmhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Switch-SWE evaluates coding agents’ ability to resume unfinished tasks within one conversation, comparing paired uninterrupted and switched runs on 200 tasks from 19 Python repositories. Switching lowers resolution rates across seven configurations. The proposed RCM combines task-relevant history routing with state summaries, improving switched resolution by 5.5–15.5 points for models using OpenCode.

Critical strengths
  1. The paired sessions hold focus requests, request order, base revision, workspace layout, and verifier fixed. Separate persistent checkouts isolate each task’s code changes. Validated reference fixes and held-out fail-to-pass and pass-to-pass tests in clean environments support reliable scoring.

  2. All seven configurations show lower resolution rates under switching. RCM raises switched resolution for DeepSeek-V4.1-Flash and GLM-5.3, with little change in uninterrupted performance.

Critical weaknesses
  1. The task selection procedure is unclear. Section 3.3 reports using “earlier execution outcomes” but does not specify the models, conditions, or criteria. A diagnostic set is reasonable, but whether prior success without switching or failure under switching influenced selection matters for interpreting the reported gap.

  2. The paper states that focus-stage budgets are matched (Section 3.1), but Section 4 allows at most three timeout continuations across the entire session. It is unclear whether interruption requests could consume continuations otherwise available to focus requests. If so, part of the reported gap could reflect budget differences.

Questions for authors
  1. Which models, conditions, and criteria informed task selection? Did prior switching failures or larger uninterrupted–switched gaps affect inclusion, and were the reported runs independent of those earlier evaluations?

  2. How were timeout continuations allocated in practice? Could interruption requests use continuations otherwise available to focus requests? Do the logs confirm that focus stages had the same available budget in both conditions?

  3. Please report paired success/failure counts and uncertainty estimates for Tables 1 and 3, especially for the memory gains over +Router. Which cases does Figure 1(b) count? For DeepSeek, the categories sum to 30, matching the net decrease in successful tasks, while 137 switched runs remain unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.