acceptodds
Under review as a conference paper at ICLR 2027

SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

11 views

Abstract

Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Robustness, Agents

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_NLYMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The main goal of this work is to demonstrate that frontier coding agents and LLMs struggle to adapt when the repository and issue they are working on have had async changes introduced by external factors like a co-author making changes, some tool not behaving as it would at test time, the addition of more files and new evidence, etc.

They define 12 such intervening conditions that might affect agent performance, which are realistic and well motivated.

The key distinction that they make is to measure this performance starting from a common checkpoint. They use an agent to complete the task up to a certain point, save the state at that point, and then rerun Clean and Shifted (intervened) versions of the task starting from this point.

Their results show that the best frontier coding agents are substantially affected by the shifts.

Critical strengths
  1. I think the results show big drops in performance when shifts are introduced, which shows a big gap in current LLM coding agents.
  2. The 12 categories of possible shifts introduced in the benchmark design are well motivated and realistic.
Critical weaknesses
  1. It is quite unclear how the tasks were actually created and how quality was ensured. There is a mention of using GPT-5.6 Sol for creating the "first" 24 tasks, but it is unclear how they were actually created. I am guessing this was an agentic process but that process is not described in the main paper.
  2. The authors mention recoverability checks, but how were these done? Are they also based on tests written by the same gpt-5.6-sol that was used to create these interventions?
  3. The results section is difficult to follow for readers.
  4. For a dataset/benchmark contribution, the authors should include an independent controlled study with humans in the loop to ensure that the tasks are recoverable and high quality.
Questions for authors
  1. For a quality check, if the agent is made fully aware of what exactly changed in the intervention can it succesfully solve the task usually? This could be a good way to ensure that the intervention does not break the task.
  2. What prompts were used for the data construction? What is the exact agentic process for creation?
3 comments
AuthorsOpenReview ↗

Follow-up on Our New Human-in-the-Loop Study & Construction Pipeline Details (Parts 1–5)

Dear Reviewer NLYM,

Thank you very much for your candid and constructive feedback—your suggestions directly guided several of the most important improvements in our revised manuscript. In case helpful while navigating Parts 1–5 of our response, we wanted to briefly highlight how we addressed each of your core concerns:

  • New Independent Human-in-the-Loop Study (W4; Part 3, Appendix A, Table 4): Following your recommendation, external software engineers blind to model trajectories and checker scripts audited a stratified 36-scenario sample (3 per family across F01–F12) using only public interfaces, achieving a 100.0% (36/36) human recovery pass rate, 97.2% obligation clarity, 94.4% interface discoverability, and high inter-annotator agreement (κ = 0.89, AC1 = 0.94).
  • 6-Stage Construction Pipeline, Prompts & Deterministic Checkers (W1, W2, Q2; Parts 1–2 & 5, §3.1, Tables 3 & 4): Clarifies that all 120 scenarios follow the same 6-stage role-isolated pipeline, provides the three isolated prompt roles (operator-proposer, blind reference-recovery, and evidence-reviewer), and shows that recoverability is verified by deterministic container scripts (accepting 2 distinct valid recoveries and rejecting 4 negative controls—never LLM judges), with GPT-5.6 Sol ranking 4th overall (11.30% PSR).
  • Solvability Under Full Change Awareness & Streamlined Results (Q1, W3; Parts 2 & 4, §§3.3, 4.3–4.6): Demonstrates that explicit awareness and context reset more than double mean PSR (6.25% → 13.40%) alongside our Clean-conditioned Fully-Informed Solvability Diagnostic, and reorganizes Section 4 for much clearer readability.

As we progress through the discussion phase, we would greatly appreciate hearing your thoughts on whether the independent human study, full-awareness solvability results, and pipeline documentation resolve your concerns, or if there is any further evidence we can provide. Thank you again for your time and consideration!

Best regards, The Authors

AuthorsOpenReview ↗

Closing Check-in Before Discussion Deadline (New 36-Scenario External Human Study)

Dear Reviewer NLYM,

With the author-reviewer discussion period closing tomorrow (Nov 18 AoE), we wanted to briefly check in regarding your soundness concerns—especially your recommendation for an independent human-in-the-loop verification study (W4) and full pipeline/prompt transparency (W1, W2, Q2).

Following your suggestion, we conducted a blind external software-engineer study across 36 stratified scenarios (F01–F12) using only public task interfaces, achieving a 100.0% (36/36) human recovery pass rate (κ=0.89\kappa = 0.89, Appendix A Table 4), alongside full 6-stage role-isolated prompts (§3.1) and the full-awareness diagnostic doubling PSR (6.25% -> 13.40%, Q1).

If time permits in these final hours, we would deeply appreciate knowing whether these additions resolve your soundness concerns or if anything further can be clarified. Thank you very much for your time!

Reviewer_NLYMOpenReview ↗

Thank you for the detailed responses and additional experiments.

Thank you for the detailed response and the new experiments. Here are my comments on each of the points:

W1. The six-stage pipeline answers my question about how the scenarios were created. Please include as much as possible in the main paper.

W2. The blind reference solution shows the tasks can still be solved after the change, so this is partly addressed. But my main question is still open: were the checker scripts, and the solutions used to test them, written by the same model that created the changes? Please say this clearly, and say how many checkers your team reviewed by hand and how many you had to fix.

W3 is satisfactorily answered.

W4: Thank you for running this study. It is what I asked for, and if the details hold up it addresses my main concern. Please report how many engineers took part, how they were recruited, and whether they solved each task themselves, in one attempt, with the same information the agents got. Please also clarify what the agreement scores were computed on: if not the four criteria (which are all 100% for Goal Interaction, making kappa undefined), please describe the items rated and how often engineers’ judgments matched the automated checkers. Finally, what happened to the scenarios rated unclear or not discoverable?

Q1,2: Table R2 is helpful. Agents keep about 80% of their Clean successes when fully informed, which together with the human study suggests the changes generally do not break the tasks.

Overall, the human study and the fully-informed experiment address much of my soundness concern, and I am raising my overall score and soundness score. My remaining questions are mainly about reporting, and I would like to see them answered in the final version.

Reviewer_oRMnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

SWE-Shift introduces a benchmark for evaluating coding agents under mid-execution changes. It constructs matched Clean and Shifted continuations from the same saved state. Across 120 scenarios and 12 change families, it evaluates task completion, preservation, recovery, and evidence validity, and characterizes the resulting performance gaps across 17 coding-agent systems. The matched-continuation design provides a useful control for distinguishing baseline task difficulty from performance degradation following an intervention. The benchmark is a good initial effort torwards understanding agents' behaviours under various kinds of shifts in development.

Critical strengths

S1. The matched-continuation design separates task difficulty from agents' behaviours under shifts. Forking Clean and Shifted continuations from the same recorded history and workspace provides a useful control for assessing the effect of an intervention.

S2. Comprehensive evaluation beyond task tests. Preservation and evidence checks capture failures that functional tests alone may miss. The change-family breakdowns, qualitative cases, and diagnostic controls provide useful starting points for understanding these failures.

Critical weaknesses

W1. Post-shift requirements and their alignment with evaluation are insufficiently documented. It is unclear whether agents can reliably identify the active requirements and instructions. Thus, it's difficult to distinguish adaptation failures from ambiguities or conflicts in the benchmark.

W2. The causes of Clean failures are not sufficiently analyzed. Since Clean success includes both functional and non-functional checks, the reported rates do not distinguish agent errors from benchmark or infrastructure problems.

W3. Important details are missing about intervention and reference-solution construction, open-weight agent harnesses, and change-notice delivery. The observability controls also lack exact intermediate scores and per-family results, limiting reproducibility and interpretation.

Questions for authors

Q1. How does the agent learn what changed and which previous requirements no longer apply? How do you check for conflicts between the current requirements and the evaluator? Please clarify the remaining obligations after cancellation (pp. 34–35) and where the restriction on test-file edits is communicated (p. 38). (W1)

Q2. How many Clean failures are caused by agent errors, benchmark issues, or infrastructure failures? (W2)

Q3. How were the shifts, Clean reference repairs, and reference trajectories created and validated? Which were derived from real development events or developer patches, and which were manually authored or model-generated? (W3)

Q4. Which agent harness and version did each open-weight model use? (W3)

Q5. What is the form of the change notice, and when and how is it delivered? (W3)

3 comments
AuthorsOpenReview ↗

Follow-up & Quick Index to Parts 1–7

Dear Reviewer oRMn,

Thank you again for your positive evaluation (Rating: 3, Informativeness: 3) and insightful questions on requirement alignment, Clean failure attribution, and reproducibility details. To make our 7-part response easier to browse during this busy discussion window, we summarize where each item is addressed:

  • W1 & Q1 — Active Requirements, Cancellation & Edit Scope (Parts 1–2; Table 6 & Appendix G): Documents the conflict-free G0 → G1 bidirectional admission checks (Table 3), clarifies the post-cancellation baseline invariant on F07 cards, and explicitly prints allowed_edit_scope on the Appendix G case cards.
  • W2 & Q2 — Clean Failure Attribution (Part 3; Table 10): Shows across all 17 systems that 94.5% of Clean failures stem from functional task failures (task_success = 0), only 3.95% of cells fail non-functional scope/preservation/budget checks, and 0 cells suffer from unresolved benchmark or infrastructure faults.
  • W3 — Complete Observability Control Scores (Part 4; Table 11): Provides exact stage-by-stage PSR across all four conditions (Ordinary 6.25% → +Notice 7.28% → +State Refresh 9.33% → -Old History 13.40%) and per-family-group breakdowns.
  • Q3, Q4 & Q5 — Provenance, Open-Weight Harnesses & Notice Delivery (Parts 5–7; Tables 4 & 13): Adds component-level provenance with our 36-scenario external human audit, exact harness versions and checkpoint SHAs for all 13 open-weight models, and the 3-channel change delivery table across F01–F12.

Please feel free to let us know whenever convenient if our clarifications and new tables address all of your questions, or if there are any further details you would like us to provide. Thank you very much for your time and support!

Warm regards, The Authors

AuthorsOpenReview ↗

Thank You & Final Check Before Discussion Window Closes

Dear Reviewer oRMn,

With the author-reviewer discussion period ending tomorrow (Nov 18 AoE), we simply wanted to thank you again for your thorough review and check whether any questions remain regarding our Clean-failure attribution (94.5% functional task failures, 0 infra faults), conflict-free G0→G1G_0 \to G_1 checks, or open-weight harness/delivery tables. Please let us know if we can clarify anything further in these final hours!

Thank you for the clarifications and additional analyses, which largely address my concerns about requirement alignment, Clean failure attribution, and reproducibility. I will increase Soundness from 2 to 3 and maintain my overall rating of 3 (Weak acceptance).

Reviewer_Q6fKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This article proposes SWE Shift for evaluating the ability of code agents to respond to task related changes during execution. The intelligent agent first executes a shared trajectory, and then branches out two continuations, Clean and Shifted, from the same history and workspace; The former maintains the original environment, while the latter injects code, dependencies, runtime, requirements, or evidence changes. The benchmark includes 120 scenarios, 12 types of changes, and nine intervention locations. Success not only requires completing the original task, but may also require retaining effective modifications, adhering to updated goals, avoiding duplicate side effects, and refreshing evidence. The author evaluated 17 systems, all of which showed a significant decrease under the Shifted condition; Tools/conversation changes and outdated evidence are particularly difficult.

Critical strengths
  • Matched continuation fixes the history and workspace before intervention, effectively distinguishing between the failure of the original task and the failure caused by changes.
  • The types of changes cover code, dependencies, runtime, requirements, external side effects, and evidence status, with a comprehensive scope.
  • Evaluation not only relies on testing the original task, but also checks the saving behavior authority、 Restoration order and freshness of evidence; The qualitative case illustrates the necessity of these checks.
  • Covering 17 systems and providing analysis based on change types, intervention locations, and state processing methods.
Critical weaknesses
  • T10-T90 is constructed based on the action sequence of the reference solution, but in actual evaluation, it is only the third action of the tested agent. The granularity and actual progress of actions in different systems vary greatly, so these positions do not correspond to comparable semantic progress. Early completion credit may also earn points for later positions due to tasks being completed ahead of schedule, making timing conclusions more difficult to interpret.

  • Each checkpoint seems to run Clean and Shifted Continuation only once. For systems with randomness, two branches may produce different results even without intervention. The paper lacks Clean Clean forks or repeated runs, so it is impossible to estimate the random noise in the Clean Shifted gap, and there is no reported confidence interval.

  • Some failures are more like testing specific protocols rather than adapting to code changes. For example, the intelligent agent must perform observations, restore evidence routes, and even add test files in a specific order, which may be judged as failed due to write scope. If these requirements are not naturally and explicitly communicated to the agent, then the all zero results of F06 and F11 may primarily reflect interface discoverability.

  • The 120 scenes only come from 23 task roots and 17 repositories, and the nine locations in the same scene are highly correlated, so they cannot be considered as 1080 independent samples. In addition, GPT-5.6 Sol participated in the construction of some benchmarks, and as the tested system, the paper needs to provide a clearer explanation of the data construction and independent manual verification process.

Questions for authors
  • Can you report the proportion of early completion credits obtained by each system on T10-T90? After removing this part, does the trend in Figure 4 (b) still hold?
  • Can the failure rate be decomposed by task, preservation, freshness, authority, and recovery order? The current composite score is difficult to explain the main source of failure.
  • What specific scenarios, checkers, and reference recovery did GPT-5.6 Sol participate in? Has the final instance undergone independent manual review?
  • What information does the agent use to learn that the tool, session, or evidence route has changed in F06 and F11? If the change is clearly notified, how much will the result improve?
3 comments
AuthorsOpenReview ↗

Quick Guide to Our Response (Parts 1–9) & Check-in

Dear Reviewer Q6fK,

We hope this message finds you well, and we thank you again for your high ratings on Presentation (3) and Informativeness (3) as well as your precise methodological questions. Because our rebuttal spans Parts 1–9 due to OpenReview's length limit, here is a brief roadmap to help navigate our responses to each of your points:

  • W1 & Q1 — T10–T90 Scheduling & Early-Completion Credit (Parts 1, 5–6; Tables 7 & 8): Clarifies the T0 default (step 3) vs. the 9-position schedule k_{i,j} = ⌈j·N_i/10⌉, reports exact early-completion credit across all 17 systems (mean 0.306 pp, 100% invariant ranking), and confirms that Figure 4(b)'s frontier late-position decline becomes even steeper once early-completion credit is excluded.
  • W2 — Continuation Variance & Root-Clustered CIs (Part 2; Table 14): Adds a 3-replicate Clean-A / Clean-B / Shifted fork control alongside 23-task-root block-bootstrap 95% CIs across all headline results.
  • W3, Q2 & Q4 — Check-Level Decomposition & F06/F11 Notice Gains (Parts 3–4, 8; Tables 10 & 11): Separates Task, Preservation, Authority, Freshness, Recovery Order, and Write Scope failures; prints allowed_edit_scope directly on the Appendix G cards; and reports F06/F11 Clean baselines together with Shifted gains under explicit notification (+Notice / +State Refresh / -Old History).
  • W4 & Q3 — GPT-5.6 Sol Role Isolation & Independent Human Audit (Part 7; Table 4): Details strict context isolation across all 120 scenarios (GPT-5.6 Sol ranks 4th at 11.30% PSR with 0.00% on 5 families), 100% author-team audit, and our new 36-scenario external human verification study (100% human recovery pass rate, κ = 0.89).

As the discussion phase moves forward, we would be very grateful to hear whether these new experiments and clarifications resolve your soundness concerns, or if any questions remain that we can further address.

Sincerely, The Authors

We thank the authors for the detailed rebuttal and new analyses. However, the temporal position TkT_k still does not represent comparable semantic progress across systems, as the rebuttal itself confirms, which affects the paper's cross-system comparisons. We therefore maintain our original score.

AuthorsOpenReview ↗

Clarifying Why Cross-System Conclusions Do Not Rely on Pointwise Semantic Equivalence

Dear Reviewer Q6fK,

Thank you very much for reading our rebuttal closely and pinpointing this exact methodological question. We completely agree that at an individual index TkT_k, agents with different action granularities need not sit at the exact same fraction of semantic task completion. However, we would like to clarify (1) why agent-external scheduling is methodologically necessary for fair cross-system benchmarking, and (2) why SWE-Shift's cross-system conclusions do not rely on pointwise semantic equivalence at a single TkT_k:

  1. Why scheduling TkT_k from an agent-agnostic yardstick (NiN_i) avoids severe cross-system bias: Defining intervention timing endogenously as "a fixed fraction of each tested agent's own semantic progress or trajectory" is ill-defined and selection-biased across systems: agents fail on 33%–85% of Clean runs (often looping until budget exhaustion without reaching intermediate semantic milestones), and conditioning on eventual trajectory length introduces oracle lookahead while penalizing concise agents relative to verbose/stalling ones. Triggering shifts on a shared external horizon (ki,j=⌈j⋅Ni/10⌉k_{i,j} = \lceil j \cdot N_i / 10 \rceil)—while branching Clean and Shifted from that same agent's identical saved checkpoint—ensures every system's adaptation drop is measured causally against its own exact state at the moment of shift.

  2. Why our concrete cross-system claims hold without pointwise TkT_k alignment (using existing controls):

    • Full-horizon T10–T90 aggregation & invariant zero-modes (§4.2–4.3, Tables 7–8): Our main 17-system comparison aggregates uniformly across all 9 positions (T10–T90), where early completion averages just 0.306 pp (100% ranking-invariant when excluded, Tables 7–8). Granularity differences mainly shift which TkT_k bucket captures a system's mid-stage execution along the horizontal axis; they cannot produce the 40+ pp vertical collapse across the full grid, nor affect universal 0.00% PSR on F06 and F11 across all 17 systems at every position.
    • Within-system monotonicity for §4.4 timing trends (Figure 4b, Table 8): Crucially, §4.4 does not require System A and System B to share identical semantic progress at a single T50T_{50}. For each individual system, T10<⋯<T90T_{10} < \dots < T_{90} progresses monotonically along its own execution; Figure 4(b) compares within-system Early → Late slopes (showing frontier models degrade at later checkpoints due to history anchoring while weaker models slightly improve—a pattern that sharpens after removing early-completion credit, Table 8).
    • Pre-divergence & history-reset anchors (§3.2.1, §4.6.1, Table 11): Both the early T0 (step-3) setting—prior to multi-step editing divergence—and our -Old History control (6.25% -> 13.40% 5-system mean PSR, stripping pre-fork trajectory length differences) independently confirm that pre-fork history anchoring, rather than action-step counting, drives the adaptation gap.
  3. Explicitly scoping interpretation in §§3.2.1 & 4.4: To ensure complete precision for readers, we will explicitly state in §§3.2.1 and 4.4 that individual TkT_k indices represent matched external action-horizon checkpoints rather than cross-system-aligned semantic milestones, and scope cross-system comparisons strictly to full-grid T10–T90 metrics, paired within-system Clean/Shifted drops, and within-system Early → Late trajectory slopes.

Given that all other concerns from your initial review have been resolved and we explicitly guard against over-interpreting pointwise cross-system TkT_k slices, we respectfully ask whether you might reconsider your assessment before the discussion window closes. Thank you again for your sharp and constructive engagement!

Reviewer_nZyVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SWE-Shift introduces a benchmark for evaluating how coding agents respond to changes during execution. It branches from the same saved execution state into Clean and Shifted continuations, evaluating task completion, preservation of valid work, state recovery, and evidence validity under the original and updated conditions, respectively. The benchmark contains 120 scenarios built on 23 task roots from 17 repositories, covering 12 change families and nine intervention positions. Evaluations of 17 coding-agent systems show consistently lower success under Shifted conditions. State-handling controls, budget experiments, and trajectory analyses further investigate these failures. The main contribution is a unified framework combining matched-prefix evaluation with diverse mid-execution changes, preservation requirements, and trajectory-level checks.

Critical strengths
  • Strength1: The problem is practically important, and the matched-prefix design provides a valuable experimental control. Collaborator edits, requirement updates, tool/session changes, and outdated test results can all occur while a coding agent is working. Static task success does not adequately capture reliability in these settings. SWE-Shift generates Clean and Shifted continuations from the same saved workspace, history, and controller state, reducing differences that independent runs could introduce before the intervention (Section 3.2). This is a central methodological contribution supporting acceptance, although it does not eliminate randomness during continuation.

  • Strength2: The benchmark offers substantial coverage and concrete evidence of failures that original task tests can miss. The evaluation covers 12 change families and 17 systems, with 24 detailed cases presenting recorded actions and independent evaluator findings. These cases demonstrate that an agent can pass the original task tests while removing valid collaborator code, using incorrectly bound evidence, or violating workspace constraints (Section 4.5 and Appendix G). The appendix also reports reference recoveries, alternative valid recoveries, negative controls, and replay checks, supporting scenario relevance and solvability. State-handling and budget controls further extend the contribution beyond a performance leaderboard.

Critical weaknesses
  • W1: Strict success combines behavioral correctness with procedural compliance, limiting the interpretation of failures. The success criterion includes not only task and preservation checks but also requirements such as observing the updated state before the first mutation and making another observation after the final mutation (Section 3.3 and Appendix D, Table 6). These requirements have reasonable safety motivations, but they are not equivalent to functional correctness or adaptation capability. Appendix cases show that task behavior can pass while evidence-acquisition or write-scope requirements fail. The paper does not provide an aggregate breakdown of behavioral failures, procedure-only failures, and their overlap. Consequently, it remains unclear how much of the low success rate reflects functional failure versus protocol noncompliance, affecting the external validity and practical interpretation of the scores.

  • W2: Zero family-level PSR does not independently establish a specific adaptation deficit. The joint-success component of PSR requires both Clean and Shifted success. A zero score can therefore result from failures already present in Clean, rather than failures introduced by the update. The paper emphasizes that F06 and F11 have zero PSR across all systems, but does not report corresponding family-level Clean, Shifted, CRR, or check-level failure distributions (Section 4.3 and Figure 4). The illustrated F06/F11 cases also fail in both conditions. The zero-score observation is valid, but distinguishing baseline task difficulty from the additional difficulty introduced by change is necessary to support more specific explanations.

  • W3: The timing analysis combines different intervention-exposure states, and statistical uncertainty is insufficiently characterized. T10–T90 are scheduled positions derived from reference action sequences, rather than equivalent semantic progress across systems. PSR also credits verified early completion before the agent encounters an update (Sections 3.2.1 and 3.3; Appendix C.4). Higher late-position PSR therefore need not indicate stronger adaptation to late changes. Although the paper acknowledges these limitations, it does not separate early-completion rates, actual intervention exposure, and performance after exposure at each position. Moreover, the 120 scenarios share 23 task roots and 39 clusters, and the nine intervention positions are correlated, while the main results are reported as point estimates. Smaller system differences and diagnostic gains require uncertainty estimates that account for these dependencies.

Questions for authors
  1. Score decomposition: Among unsuccessful runs, how many pass task and behavioral-preservation checks but fail only observation-order, final-observation, or other procedural requirements? Would reporting behavioral correctness separately from procedural compliance change the main system rankings or the conclusions about F06/F11?

  2. Family-level attribution: Could you provide Clean, Shifted, CRR, and the four paired outcome categories for F06 and F11? How many cases succeed in Clean but fail in Shifted, and which checks account for these failures?

  3. Intervention exposure and timing: What are the early-completion, actual-exposure, and update-inapplicability rates at each intervention position? Do the timing trends in Section 4.4 persist when considering only trajectories that actually receive an intervention?

  4. Statistical stability: Are repeated-run results or confidence intervals accounting for task-root or cluster dependence available? Do they support the smaller system differences and gains observed in the state-handling controls?

3 comments
AuthorsOpenReview ↗

Brief Navigation Summary & Follow-up on Parts 1–7

Dear Reviewer nZyV,

Thank you once again for your constructive review and for recognizing SWE-Shift's matched-prefix control and trajectory-level evaluation as supporting acceptance. Since our detailed response had to be divided into Parts 1–7 due to character limits, we wanted to share a quick index for your convenience and gently check in:

  • W1 & Q1 (Parts 1–2, Table 10): 4-way cross-tabulation separating Behavioral Correctness vs. Procedural Compliance across all 17 systems, confirming that system rankings remain preserved under Behavioral-Only PSR.
  • W2 & Q2 (Part 3, Table 11): Complete Clean, Shifted, Joint Success, CRR, paired outcome cells (C1S1, C1S0, C0S1, C0S0), and failing-check breakdown for F06 and F11.
  • W3 & Q3 (Parts 4–5, Tables 7 & 8): Position-by-position accounting of Early Completion (P ∈ [0.00, 0.75] pp, 100% ranking-invariant), Actual Exposure, and Inapplicability across T10–T90, verifying that conditional timing trends persist (and late drops sharpen for frontier models) after removing P.
  • W3 & Q4 (Parts 6–7, Table 14): Exact 4-stage State-Access Control values (6.25% → 7.28% → 9.33% → 13.40%), 23-root block-bootstrap 95% CIs, and the 3-replicate Clean-A / Clean-B / Shifted continuation study.

Whenever your schedule permits, please let us know if these additions fully address your questions on metric transparency and statistical stability, or if there is anything else we can clarify during the discussion period. Thank you again for your time!

Best regards, The Authors

AuthorsOpenReview ↗

Brief Update on Semantic-Progress Alignment (Complementing W3) Before Discussion Closes

Dear Reviewer nZyV, As the discussion window wraps up tomorrow (Nov 18 AoE), we wanted to share one compact additional analysis prompted by our discussion with Reviewer Q6fK that also directly complements your W3 on comparable semantic progress across systems:

Using our paired Clean continuations to align all 17 systems both by each system's own empirical progress fraction (p^=k/(k+Rclean)\hat{p} = k / (k + R^{\text{clean}})) and by workspace semantic stage (Pre-Edit / In-Edit / Verification), cross-system rankings remain highly consistent (Appendix C.4 Table [X]) and the conditional timing trends hold.

Thank you again for your insightful review—please let us know if any questions remain on our Behavioral vs. Procedural decomposition (W1), F06/F11 attribution (W2), or block-bootstrap CIs (W3/Q4) before the window closes!

Area_Chair_QTc7OpenReview ↗

Reminder from AC

Dear Reviewers,

As the ICLR discussion period is closing, please take a moment to read the authors’ responses carefully and let them know which of your concerns have been addressed and which questions remain open. If anything needs clarification or would benefit from further discussion, please raise it and engage with the authors and fellow reviewers while there’s still time.

Thank you for your time, your thoughtful reviews, and your service to the ICLR community.

Best Regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.