acceptodds
Under review as a conference paper at ICLR 2027

SWE-Continuum: Multi-Stage Profiling of Behavioral Continuity in Long-Horizon Agents

21 views

Abstract

Coding benchmarks usually score the final repository state, omitting when functionality is gained, lost, or recovered. We introduce SWE-Continuum, a benchmark for multi-stage profiling of behavioral continuity, with 16 tasks and 111 requirements from 16 repositories in five languages. Tasks release requirements in a persistent workspace under fixed DAG-based schedules. A verifier checks all released requirements and baseline behavior after each stage. A construction pipeline aligns instructions, tests, and reference patches, with no-op, oracle, dry-run, and human checks. The Area Under the Progress–Expenditure Curve (AUPEC) summarizes net progress over a shared token, time, or cost budget. Across eight models, 17 of 123 valid trials contain 64 pass-to-fail transitions, yet net progress rises at 27 of the 29 checkpoints containing one. Models with nearly equal final net progress differ 4.6-fold in token expenditure; terminal net progress and token AUPEC reverse 14 of 28 model-pair rankings. These results distinguish terminal capability, behavioral continuity, and budgeted net progress.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon Agents, Coding Agents, Multi-Stage Evaluation, Behavioral Continuity, Resource Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9F9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-Continuum, a benchmark for evaluating coding agents under sequentially released requirements in a persistent repository. It tracks functional progress and regressions through cumulative testing at each stage, and proposes AUPEC to measure progress under resource budgets. Experiments on eight models show that intermediate regressions can be hidden by final scores, and model rankings can change considerably when considering token expenditure and progress trajectories.

Critical strengths
  1. The fixed requirement schedule is a reasonable design. Different agents face the same requests, while their workspace evolves independently. This makes comparison more controlled than simulator-driven benchmarks. The task construction and validation pipeline also seems carefully designed.

  2. The test-level trajectory analysis is useful. Figure 8 gives concrete cases where newly passed tests hide previous failures, and some failures are recovered later. Such information is indeed missed by terminal evaluation.

Critical weaknesses
  1. I have some doubt about how much new information AUPEC actually provides. It mixes progress timing with resource expenditure, so a model spending more tokens naturally gets lower AUPEC even when final functionality is similar. Appendix C shows that the AUPEC gap becomes much smaller when matching both final verdicts and token expenditure. The model-level AUPEC can also be almost perfectly fitted using budgeted endpoint progress and expenditure. So the large ranking changes seem mainly reflecting efficiency differences. Not very clear how much comes from the actual progress trajectory, especially regressions. The baseline preservation scores are also nearly perfect for all models, suggesting AUPEC is largely measuring how quickly new functionality is acquired.
  2. The regression results are quite concentrated. From Table 4, most regression trials and P2P transitions come from python-02. Several other tasks have no observed regression at all. This makes me wonder how representative the reported continuity problems are. Also, many tasks are constructed by removing or stubbing existing functionality and then restoring it stage by stage. Such setting is controlled, but the regression behavior may differ from real software evolution, where requirements are genuinely new and sometimes conflicting.
Questions for authors
  1. How much would the AUPEC model rankings change if using only F2P completion, without P2P preservation? This could help understand whether the ranking differences really involve behavioral continuity.

  2. In Table 4, python-02 accounts for 13 of 17 regressed trials and 42 of 46 P2P regressions. What happens to the main regression conclusions if this task is excluded?

  3. The requirement DAG can admit multiple valid release orders. Have you examined whether changing the order affects regression frequency or model rankings? This seems relevant for interpreting continuity as an agent capability.

8 comments
AuthorsOpenReview ↗

Response to Reviewer 9F9s (1/6): General Comments

We thank the reviewer for recognizing the controlled comparison enabled by the fixed requirement schedule, the careful task construction and verification process, and the value of test-level trajectory analysis. Looking only at the terminal state cannot tell how an agent reached it; newly passing tests mask damage to existing behavior, and later recoveries are equally invisible.

SWE-Continuum is a complete evaluation system, covering the Task Build Pipeline, the execution environment, and analysis metrics and visualization tools. The metrics each have their own role:

  • S measures the net functional progress at each checkpoint.

  • Per-test pass, regression, and recovery events measure behavioral continuity.

  • AUPEC measures how quickly progress accumulates within a shared resource budget.

The released analysis suite also provides finer diagnostics, such as immediate completion of newly released requirements; requirement-level first completion, regression, and recovery; features once achieved but later lost; and the cost to first reach and stably hold a target score. The main points are:

  • AUPEC provides information beyond the terminal result: it reflects when and how fast progress occurs within the budget and complements the per-test regression and recovery events; it is not meant to measure continuity on its own.

  • The uneven distribution of regressions across tasks is itself information revealed by trajectory analysis. Our conclusion does not rely on all tasks regressing equally often, but is that terminal or aggregate scores alone cannot tell whether existing behavior is preserved.

  • The delete-and-restore task construction and the fixed release order both make tasks controllable, verifiable, and fairly comparable. We explain the rationale and scope of each below.

AuthorsOpenReview ↗

Response to Reviewer 9F9s (2/6): Weakness 1

Weakness 1: Do AUPEC differences reflect efficiency rather than continuity?

AUPEC measures functional progress within a shared resource budget; it does not measure regression frequency on its own. Efficiency differences are precisely one of the things it is meant to capture. Behavioral continuity is characterized directly by per-test pass, regression, and recovery events; the two are complementary.

First, AUPEC does not merely penalize total expenditure. It integrates the progress curve within the shared budget: of two trajectories with the same terminal functionality and the same total expenditure, the one that obtains functionality earlier and keeps it gets a higher AUPEC, and continuing to spend after reaching the same functionality does not reduce the area accumulated before. It therefore measures how much usable functionality is obtained while resources are spent, not simply the terminal score divided by cost.

Second, the fit in Appendix C uses the progress at the budget boundary, S(B), not the terminal S at arbitrary expenditure. S(B) already contains whether a model reaches a given functional level within the shared budget. That in-budget endpoint progress and expenditure explain most of the model-level AUPEC differences shows that these models differ mainly in the level and speed of functionality obtained within the budget, consistent with the design goal of AUPEC. Table 16 lists the explanatory power of each endpoint variable for model-level AUPEC, including terminal S in place of S(B).

Model-level AUPEC is well fit by endpoint quantities: S(B) + log token explains 99.65%, and terminal S + linear tokens 99.38%. But this is a descriptive fit over 8 model means and does not show that the endpoints determine the area. Appendix C.1 tests the remaining trajectory differences at the trial level: 63 trial pairs with identical terminal per-test verdicts differ in AUPEC by 17.65 points on average; after also matching expenditure (token ratio at most 1.10), 8 pairs remain and still differ by 3.55, including a python-03 pair whose tokens differ by only 2.01% (45.81M vs 44.91M) and whose AUPEC differs by 9.44. The gap shrinks after matching expenditure because expenditure is itself part of AUPEC; the remaining difference comes from when progress occurs.

Finally, near-full P2P preservation does not mean that no regression occurred during development. P2P checks the existing behavior of the initial workspace; F2P functionality obtained in earlier stages can also be broken and later restored. Whichever kind of test regresses, it can be offset by the credit for new functionality at the same step. SWE-Continuum therefore records regressions and recoveries per test alongside S and AUPEC, rather than inferring them from changes in the AUPEC ranking.

The revision clearly separates these two conclusions: AUPEC shows how the resource budget changes the assessment of functional progress; per-test trajectory analysis shows the behavioral-continuity problems missed by aggregate scores.

Table 16 (revision). Variance decomposition of model-level AUPEC (R2R^2 of linear regressions over the 8 model means; tokens are total terminal expenditure; the common task set is the 6 tasks on which all 8 models have valid trials).

RegressorsAll 9 tasksCommon 6 tasks
Terminal S1.27%0.10%
S(B)73.80%74.85%
log tokens86.49%84.81%
Terminal S + log tokens94.25%92.23%
S(B) + log tokens99.65%98.03%
Terminal S + linear tokens99.38%98.29%
AuthorsOpenReview ↗

Response to Reviewer 9F9s (3/6): Weakness 2

Weakness 2: Concentration on python-02 and the delete-and-restore construction

Our conclusion does not rely on "every task regresses frequently", but on "looking only at terminal or aggregate scores cannot tell whether existing behavior is preserved", and it still holds after excluding python-02. "Delete-and-restore" is a design choice made to keep tasks controllable and verifiable.

On the concentration of regressions, the uneven distribution is itself information revealed by trajectory analysis: python-02 has 282 P2P tests, and its 6 stages repeatedly modify the same nosec parsing logic, so later changes easily break behavior that was already correct. A large P2P suite alone does not produce regressions: python-06 has 229 P2P tests and no P2P regression in any of its 11 trials; each of the other 7 primary tasks has only 12–15 P2P tests, so one broken test costs 6.67–8.33 points. Excluding python-02, of the 5 checkpoints with regressions, S still rises at 3 and falls at 2. Of the 739 checkpoints without regressions, S rises at 578 (78.2%) and never falls. Looking only at S therefore cannot distinguish "pure progress" from "progress accompanied by regressions", which is why SWE-Continuum records regressions and recoveries per test.

On task construction, only 5 of the 16 tasks are delete-and-restore reconstruction tasks (only two of the 9 primary tasks, python-03 and python-05); the other tasks start from the starting commit of a real project and implement new requirements step by step according to their dependencies. python-02, where regressions are most concentrated, is not a reconstruction task: 13 of its 14 trials regress, whereas among the two reconstruction primary tasks only 1 trial of python-05 regresses. Delete-and-restore gives every requirement the original project's implementation as an oracle, so we can verify that the task is solvable and the tests are correct, and different agents face exactly the same requirement sequence. This lets us attribute failures to the agent rather than to defects in the task. Scenarios where requirements conflict and existing designs must be overturned are equally important; our Task Spec and Task Build Pipeline do not restrict the task source and can build such tasks with the same verification process, and we will add them in later versions.

The revision discusses the scope of applicability of the delete-and-restore construction in Limitations and adds regression statistics with python-02 excluded (Section 6, Appendix I.4, and Table 10 of the revision).

Reviewer_AjcNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SWE-Continuum introduces a multi-stage benchmark for evaluating behavioral continuity in coding agents. It contains 16 tasks and 111 requirements, released sequentially in persistent workspaces with cumulative verification. The benchmark tracks functionality acquisition, regression, and recovery, and introduces AUPEC to measure functional progress relative to resource expenditure.

Experiments across eight models show that aggregate scores can conceal intermediate regressions/failures and that resource-aware evaluation can substantially change model rankings.

Critical strengths

A good evaluation protocol that is also distinct from prior work. While benchmarks such as SWE-Milestone, EvoCode-Bench, and LoopsBench examine continuous software development, SWE-Continuum combines fixed, dependency-valid requirement schedules, continued execution after failures, and cumulative verification. This enables comparisons of continuity, including regressions and subsequent recovery. The task construction and validation procedure is carefully specified too.

Informative trajectory-level and resource-aware evaluation in the paper. It demonstrates that terminal correctness does not capture development trajectory. Regressions can remain hidden despite increasing progress, and AUPEC further provides useful complementary progress measure wrt to cost. The ranking differences across eight models highlight practical relevance of the above evaluation dimensions. The paper is clearly written and well organized.

Critical weaknesses

Limited benchmark scale. The benchmark has only 16 tasks, with nine used for the main model comparison. Also, one task (python-02) accounts for 46 of the 64 observed regression transitions, suggesting that the regression findings may be concentrated in a small subset of tasks. This somewhat limits the generality of conclusions.

Benchmark reproducibility, availability. While construction protocol is described, the complete executable benchmark with environments and tests is not yet available. Given that benchmark itself is core contribution, releasing these artifacts is important for evaluation and subsequent adoption.

Questions for authors

Do the reported model-ranking reversals persist when comparing models on a common set of tasks? The current comparison includes slightly different task coverage across models.

Will authors release the executable tasks, tests, and environments alongside profiling and visualization tools?

6 comments
AuthorsOpenReview ↗

Response to Reviewer AjcN (1/4): General Comments

We thank the reviewer for recognizing how our evaluation protocol differs from prior work: a fixed, dependency-valid requirement schedule, continued execution after failures, cumulative verification, and trajectory-level, resource-aware evaluation. Looking only at the terminal state cannot tell how an agent reached it, and intermediate regressions and recoveries are masked by the aggregate score. SWE-Continuum is a complete evaluation system, from the Task Build Pipeline and the execution environment to analysis metrics and visualization tools.

The main points are:

  • Ranking flips persist on the common task set: on the 6 tasks on which all 8 models have valid trials, 14/28 pairs still flip between terminal S and AUPEC, as on the full set.

  • On scale: each task contains multiple requirements released one after another according to their dependencies (111 requirements across 16 tasks), and after each requirement all released requirements are cumulatively verified. The Task Build Pipeline can keep producing new tasks, and the public version will keep expanding the task set.

  • On open-sourcing: after anonymous review, we will release the Task Build Pipeline, the task container images with tests, the analysis suite, and the profiling tools.

AuthorsOpenReview ↗

Response to Reviewer AjcN (2/4): Weakness

Weakness: Task scale, concentration on python-02, and release of the benchmark

The core contribution of SWE-Continuum is a reproducible, extensible framework for evaluating continuous development, not merely statistics on how often regressions occur. Our experiments show how this framework reveals the development process that terminal scores cannot reflect, and how it measures functional progress within a shared budget.

On scale, the evaluation unit of SWE-Continuum is much larger than in typical benchmarks. Each task contains multiple requirements released one after another according to their dependencies, which the agent must implement one by one in the same evolving workspace while passing cumulative verification at every stage. The 16 tasks contain 111 requirements, and the 9 tasks and 123 valid trials of the main comparison contain 828 checkpoint-level evaluations. Over these 828 checkpoints, completing one requirement takes on average 4.91M tokens, 66 LLM calls, and 16.2 minutes of agent time. For reference, on the public SWE-bench Pro trajectories a single task takes 44–78 steps on average, with 3.0–5.3M input tokens in our sample; the scaffolds and models differ, so this serves as an order-of-magnitude reference. One requirement stage is thus roughly equivalent to one SWE-bench Pro task, and SWE-Continuum additionally provides continuous observation across these stages.

Using only the 6 tasks on which all 8 models have valid trials (see Q1 and Table 14), 14/28 pairs still flip between terminal S and AUPEC, consistent with the full set (11 of them are the same model pairs). The main conclusions therefore come from differences between the models themselves, not from task coverage or scale.

On python-02, the uneven distribution of regressions across tasks is itself information provided by trajectory analysis. The concentration is related to its task structure: it has 282 P2P tests, and its 6 stages repeatedly modify the same nosec parsing logic. A large P2P suite alone does not produce regressions: python-06 has 229 P2P tests and no P2P regression in any of its 11 trials. Each of the other 7 primary tasks has only 12–15 P2P tests, so one broken test costs 6.67–8.33 points. Our conclusion is not that "all tasks regress at the same rate", but that "looking only at terminal or aggregate scores cannot tell whether existing behavior is preserved". Excluding python-02, of the 5 checkpoints with regressions, S still rises at 3 and falls at 2. Of the 739 checkpoints without regressions, S rises at 578 (78.2%) and never falls. S therefore characterizes net progress, while continuity has to be characterized by the per-test regression and recovery events that SWE-Continuum also reports.

On reproduction, after anonymous review we will release the Task Build Pipeline (with the Task Spec), the task container images with tests, the analysis suite, and the profiling tools, so that researchers can both reproduce the experiments and extend the task set with the same build and verification process. The revision also states the selection criterion for the primary task set and adds the number of valid tasks, the number of trials, and confidence intervals to the result tables (Appendix D and Table 13 in the revision).

AuthorsOpenReview ↗

Response to Reviewer AjcN (3/4): Question 1

Question 1: Do ranking flips persist on a common task set?

Yes. On the 6 tasks on which all 8 models have valid trials, 14/28 pairs still flip between terminal S and AUPEC, consistent with the full set, so different task coverage is not the source of this phenomenon.

The most visible example is Qwen: 2nd on terminal S and 8th on AUPEC. Astra and DS Flash V4.1 have close terminal S, but their AUPEC is 83.28 and 48.57. Terminal S measures the functional level finally reached, while AUPEC integrates the progress curve within the shared budget, distinguishing when and with how many resources similar final results were reached. The flips are stable under the bootstrap: the 95% interval of the number of flipped pairs is [7, 18] on the common task set ([7, 16] on the full set), and 11 of the 14 pairs on each task set are the same. On the common task set, 12 pairs stay flipped in more than half of the resamples and 1 pair in more than 95%.

The revision adds the results on the common task set, the confidence intervals, and each model's number of valid tasks and trials (Tables 13 and 14 in the revision).

Table 14 (revision). Results on the 6 tasks common to all 8 models (javascript-01, javascript-02, python-02, python-03, rust-02, typescript-02). AUPEC uses the paper's shared budget B = 63.15M tokens (each token counted once: input including cache + output). 95% CIs come from a task-then-trial hierarchical bootstrap with 2,000 resamples.

ModelTerminal S [95% CI]S rankAUPEC [95% CI]AUPEC rankTokens (M)
GPT-6 Astra90.94 [76.02, 98.97]383.28 [70.15, 91.25]110.41
GPT-5.6 Luna87.82 [79.80, 95.31]572.21 [65.16, 79.76]217.15
Hy4 Preview94.52 [87.62, 98.76]168.46 [61.39, 75.57]326.20
GLM 5.3 Flash75.76 [46.51, 96.56]862.40 [37.00, 80.50]414.70
DS Flash V4-073187.78 [75.90, 95.99]659.10 [48.86, 73.06]527.99
MiMo V2.579.84 [65.92, 90.92]757.37 [49.36, 63.87]627.60
DS Flash V4.190.10 [81.89, 97.12]448.57 [27.12, 63.76]749.10
Qwen 3.8 Flash91.62 [82.22, 97.55]228.90 [15.47, 40.61]868.67
Reviewer_pp5aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper introduces SWE-Continuum, a coding benchmark of 16 tasks and 111 requirements, in which requirements arrive one at a time in the same codebase and the tests are rerun after every step
  • It aims to show when functionality is gained, broken or fixed again, which a single final score, the usual practice today, cannot capture
  • A new score is also proposed, which rewards models for making progress early within a fixed budget of tokens, time or cost
  • Eight models are compared on 9 of the 16 tasks, and the obtained results suggest that about 17 of 123 runs break previously working tests, and that ranking models by final score versus the new score flips at least half of the model pairs
  • A small extra experiment on four tasks suggests that giving the agent wrong advice causes more breakage than a neutral "check your work" prompt used as a baseline
Critical strengths
  • The paper builds its tasks with careful checks. Every new test must fail before the work starts, every existing test must pass, and the reference solution must pass at every step in two fresh environments
  • Its test-by-test tracking catches breakage that the total score would miss. In one example, a model makes 19 new tests pass while breaking 10 that passed before, so the score still goes up noticeably even though this is likely not the behavior one would want
  • The model comparison yields a useful efficiency finding. Two models reach almost the same final score, but one uses about 10 million tokens and the other almost 50 million, more than four times as many, which matters a great deal in practice
  • Reported numbers appear internally consistent, and the authors are open about gaps such as the unequal task coverage across models and the missing public release of the execution setup
Critical weaknesses
  1. The main finding about hidden breakage seems to come almost entirely from a single task. In Table 4, python-02 accounts for 13 of the 17 runs with breakage and 46 of the 64 broken tests, and without it only 4 of 109 runs show any breakage. This looks more like a property of one task than a general pattern

  2. The claim that the score still rises at 27 of 29 checkpoints with breakage largely follows from how the score is built. Every step releases new tests that can earn credit, while one broken old test costs only 1/282 of the baseline part of the score, so the result is close to expected. The paper's own extra experiment confirms this: when steps add no new requirements, "None of I's 27 regression checkpoints has rising S"

  3. AUPEC seems to add very little beyond the final score and the total cost. Appendix C.2 reports that these two quantities explain 99.7% of the variation in model-level AUPEC. The result that 14 of 28 rankings flip therefore mostly reflects how the budget trades score against cost, which relates directly to other cost-aware measures such as SWE-Effi or Cost-of-Pass that are not discussed

  4. The model comparison is too small and uneven to support the reported rankings. It uses 9 of the 16 tasks with one or two runs per task, two models have fewer valid tasks than the rest, and Table 5 reports no confidence intervals

  5. The 16 tasks were selected using the very same calibration run that was later scored (Appendix D), and the criteria for the 9-task subset do not seem to be given. This makes it difficult to judge to what extent the results would hold up under closer scrutiny

  6. The published task materials seem to contain errors that contradict the paper's own construction rules. The dependency graphs of rust-01 and rust-02 show a requirement depending on one released later, which Eq. 4 directly forbids. The python-02 instruction asks for a fallback in one sentence and forbids it in the next, so it is unclear which behavior is expected. The go-01 instruction describes Go's Sprintln function incorrectly. Two tasks tell the agent to ask the user, although there is no user to answer, which makes it very difficult for the agent to act realistically there

  7. Several closely related benchmarks with the same sequential, persistent setup are not cited, such as SWE-STEPS (ICML 2026), SlopCodeBench, SWE-CI and SWE-Bench-CL. Commit0 uses the same remove-and-restore code design and discusses the risk that models simply remember the original code, which this paper does not address at all. The related work section could be extended considerably

Questions for authors
  1. How do the headline breakage numbers look with python-02 excluded, and how often does the score rise at steps without any breakage? This would help show how much of the masking comes from agent behavior and how much from the score design

  2. Would the authors consider reporting Table 5 and the ranking flips on the tasks that all models share, with task-level confidence intervals, next to a simple score-versus-cost plot? This would clarify what AUPEC adds beyond such a plot

  3. Were the executed schedules for rust-01 and rust-02 consistent with Eq. 4, or are only the plotted graphs wrong? If the schedules were affected, fixing the faulty instructions and rerunning the affected tasks would help clarify the situation

  4. What git history do agents see after the code is removed? A simple check of how often the restored code matches the original project word for word would help separate whether the agents actually build the functionality or whether it can be attributed to remembering what the code looked like

  5. Would it be possible to run the wrong-advice experiment on the same task versions and harness as the main benchmark, ideally with both conditions starting from the same saved state? This would help isolate the effect of the advice itself from everything else that changed in this experiment

12 comments
AuthorsOpenReview ↗

Response to Reviewer pp5a (1/13): General Comments

Thank you for recognizing our new evaluation framework and for the detailed review.

Our goal is a new evaluation paradigm and framework that exposes the behavior patterns of agents on coding tasks.

SWE-Continuum addresses two limitations of current evaluation:

  • It looks only at the terminal state and cannot describe how an agent reaches it.

  • Task construction is not reliable enough, so one often cannot tell whether a failure comes from the agent's limited capability or from a poorly designed or faulty task.

SWE-Continuum is a complete evaluation system, including the Task Build Pipeline, the execution environment, analysis metrics, and visualization tools. It helps developers and agent users to:

  • understand the capability ceilings and behavior patterns of different agents, models, and harnesses with our released tasks;

  • build more tasks with our Task Spec and Task Build Pipeline, creating evaluations for their own scenarios and finding the capability–cost trade-off that suits them;

  • run their own agents in the task execution environments and use the profiling tools to obtain stage-by-stage verified records of progress and regression.

AuthorsOpenReview ↗

Response to Reviewer pp5a (2/13): Weakness 1

Weakness 1: Regressions are concentrated on python-02

Regressions are concentrated on python-02, but the finding that aggregate scores mask regressions does not depend on it. Excluding python-02, 4 of the 109 valid trials still regress, on four tasks (go-02, javascript-01, python-05, python-06), with 18 pass-to-fail transitions; S still rises at 3 of the 5 regression checkpoints.

python-02 regresses often because of its task structure: it has 282 P2P tests, the broadest baseline coverage among the primary tasks, and its 6 stages repeatedly modify the same nosec parsing logic (first per physical line, then across multi-line logical statements), so later changes easily break earlier correct behavior. A large P2P suite alone does not produce regressions: python-06 has 229 P2P tests and no P2P regression in its 11 trials.

Excluding python-02, 14 of the 18 regressions are F2P regressions: functionality acquired earlier is broken later. BPB_{P} covers only the initial workspace's P2P tests, so these are visible only in the per-test trajectory. The clearest case is MiMo on python-05 (Figure 8a): at checkpoint 6, 19 F2P tests newly pass while 10 previously passing ones fail, and S still rises by 7.63 points.

Our conclusion is that terminal or aggregate scores alone cannot tell whether existing behavior is preserved, not that every task regresses at the same rate; how regression frequency varies with task structure is itself information from per-test trajectories. The revision adds Table 9 and the comparison without python-02 (Table 10) to Appendix I, referenced from Section 4.2.

Table 9 (revision). Pass-to-fail regressions by task in the main run (123 valid trials). Breaking one P2P test costs 100/#P2P points of S (S on a 0–100 scale).

Task#P2PValid trialsTrials with regressionP2P pass-to-failF2P pass-to-failRegression checkpointsS risesS falls
go-021513130101
javascript-011215103110
javascript-021215000000
python-02282141342424240
python-031414000000
python-0514111110211
python-0622911101110
rust-021215000000
typescript-021215000000
All–12317461829272
Excluding python-02–1094414532
AuthorsOpenReview ↗

Response to Reviewer pp5a (3/13): Weakness 2

Weakness 2: Rising S at regression checkpoints follows from the scoring

The reviewer's derivation is correct, and it is exactly the paper's point: new credit at the same step offsets regressions in the aggregate score, so continuity cannot be judged from S alone, and passes, regressions, and recoveries must be recorded per test. Decomposing ΔS\Delta S at all 29 regression checkpoints of the main run also shows that "one broken old test costs only 1/282" holds only for python-02.

On python-02 (282 P2P tests), one broken test costs only 0.35 points and is outweighed by 1.45–55.07 points of same-step F2P credit; S rises at all 24 of its regression checkpoints. Apart from python-06, the other tasks have 12–15 P2P tests, so one broken test costs 6.67–8.33 points; of the 5 regression checkpoints left without python-02, S falls at 2 (go-02 checkpoint 7, python-05 checkpoint 4). Table 12 shows that without regressions S cannot fall by definition, so a rising S alone does not show that no regression occurred.

The python-05 example the reviewer cites (Figure 8a, see W1) is offsetting among F2P tests and has nothing to do with 1/282.

Under the semantic-interference condition (I), S rises at none of the 27 regression checkpoints, 26 of which are message steps releasing no new requirement.

S thus masks regressions only when the same step earns new credit, which is the normal case in multi-stage development, where every step releases a new requirement.

S is aggregate and non-blocking to keep the metric simple; the analysis tools add regression-specific diagnostics (Appendix I.2).

They are computed from the same checkpoint records: the same-step pass rate of newly released F2P tests; ever-passed F2P coverage EFE_{F} and terminal retention KFK_{F}; requirement-level first-completion, regression, and recovery events; and the time, tokens, and cost to first and stably reach S≥50/80/100S \geq 50/80/100.

The revision adds Tables 11 and 12 to Appendix I.2, and states the division of labor between S and per-test events in Section 6.

Table 11 (revision). ΔS\Delta S decomposition of all 29 regression checkpoints in the main run (S on a 0–100 scale). Each cell is the contribution to ΔS\Delta S (number of tests in parentheses); F2P new = first pass of a released F2P test; recovery = passing again after a failure. The five terms sum to ΔS\Delta S; all 29 checkpoints release a new requirement.

TaskModelTrialCheckpointF2P newF2P regressionF2P recoveryP2P regressionP2P recoveryΔS\Delta S
go-02GLMF5nMj3k7+17.58 (16)00−20.00 (3)0−2.42
javascript-01Hy4sGjm4uE4+37.97 (30)−3.80 (3)000+34.18
python-05MiMoUpDvA4a4+5.93 (7)00−7.14 (1)0−1.21
python-05MiMoUpDvA4a6+16.10 (19)−8.47 (10)000+7.63
python-06LunakWgcCh87+14.00 (7)−2.00 (1)000+12.00
python-02DS Flash V4-0731QjFUgxG6+55.07 (38)00−0.35 (1)0+54.72
python-02DS Flash V4-0731xDk9MyX5+33.33 (23)00−0.35 (1)0+32.98
python-02DS Flash V4.1s7JE6a66+5.80 (4)00−0.35 (1)0+5.44
python-02GLMHEc3F2X2+14.49 (10)00−0.35 (1)0+14.14
python-02GLMHEc3F2X3+15.94 (11)00−1.42 (4)+0.35 (1)+14.88
python-02GLMHEc3F2X6+5.80 (4)00−0.35 (1)+0.35 (1)+5.80
python-02GLMUajPB2G3+7.25 (5)−2.90 (2)0−1.42 (4)0+2.93
python-02GLMUajPB2G6+27.54 (19)−2.90 (2)+2.90 (2)−0.35 (1)+1.42 (4)+28.60
python-02LunapGv9Ty64+42.03 (29)00−0.35 (1)0+41.67
python-02LunapGv9Ty66+5.80 (4)00−0.35 (1)0+5.44
python-02LunapW7dwUy4+42.03 (29)00−0.35 (1)0+41.67
python-02LunapW7dwUy6+5.80 (4)00−2.48 (7)0+3.31
python-02Astra8qtWtUR1+1.45 (1)00−1.06 (3)0+0.39
python-02Astra8qtWtUR6+5.80 (4)00−0.35 (1)0+5.44
python-02Hy4855Zzfm6+7.25 (5)00−0.71 (2)0+6.54
python-02Hy4mgwupao6+5.80 (4)00−0.35 (1)0+5.44
python-02MiMoV5N9stR2+14.49 (10)00−0.35 (1)0+14.14
python-02MiMoV5N9stR4+37.68 (26)00−0.35 (1)0+37.33
python-02MiMoV5N9stR6+7.25 (5)00−0.35 (1)+0.35 (1)+7.25
python-02QwenCJ3at6A1+1.45 (1)00−0.35 (1)0+1.09
python-02QwenCJ3at6A2+14.49 (10)00−1.42 (4)+0.35 (1)+13.43
python-02QwenCJ3at6A6+10.14 (7)00−0.35 (1)+1.42 (4)+11.21
python-02QwenbwFWhfE1+1.45 (1)00−0.35 (1)0+1.09
python-02QwenbwFWhfE6+5.80 (4)00−0.35 (1)0+5.44

Table 12 (revision). Direction of ΔS\Delta S at checkpoints without any pass-to-fail transition. Step 1 is computed relative to the initial workspace W0W_{0} (CFC_{F} = 0, BPB_{P} = 100, S = 0).

SetRegression-free checkpointsS risesS unchangedS fallsRises (excl. step 1)
All 9 tasks799638 (79.8%)1610548 / 679
Excluding python-02739578 (78.2%)1610499 / 630
python-02 only6060 (100%)0049 / 49

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their engagement. Reviewers agree that SWE-Continuum's fixed, dependency-valid schedules, cumulative verification, and careful task construction enable controlled comparison (pp5a, AjcN, 9F9s), and that per-test trajectories reveal regressions and recoveries that terminal scores miss (Figure 8).

Our position

  • Continuity requires per-test trajectories. New credit at the same step masks regressions in aggregate scores: S rises at 27 of the 29 regression checkpoints. Under semantic interference, the same verification detects regressions in 6/10 and 5/8 trials, versus 4/12 and 1/9 under a matched review prompt (Section 4.2).
  • AUPEC measures budgeted net progress. On the six tasks common to all models, 14 of 28 rankings still flip between terminal S and AUPEC, and the flips are stable under the bootstrap (Table 14).
  • SWE-Continuum is an extensible framework. We will release the Task Build Pipeline, task container images with tests, the analysis suite, and the profiling tools.

Summary of revisions (shown in blue; new material in Appendix I)

  • Per-task regression counts and results on the other eight primary tasks (Tables 9–10, Section 4.2).
  • ΔS decomposition of all regression checkpoints (Tables 11–12).
  • Bootstrap confidence intervals, valid coverage counts, and common-task results (Tables 13–14).
  • F2P-only AUPEC, a score–token plot, and an endpoint variance analysis (Tables 15–16, Figure 13).
  • Construction pipelines, repository history, and a verbatim memorization check (Appendix I.1).
  • Task selection criterion (Appendix D), refined task wording and dependency graphs (Appendix H), and additional related work (Appendix I.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.