- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Continuum, a benchmark for evaluating coding agents under sequentially released requirements in a persistent repository. It tracks functional progress and regressions through cumulative testing at each stage, and proposes AUPEC to measure progress under resource budgets. Experiments on eight models show that intermediate regressions can be hidden by final scores, and model rankings can change considerably when considering token expenditure and progress trajectories.
Critical strengths
-
The fixed requirement schedule is a reasonable design. Different agents face the same requests, while their workspace evolves independently. This makes comparison more controlled than simulator-driven benchmarks. The task construction and validation pipeline also seems carefully designed.
-
The test-level trajectory analysis is useful. Figure 8 gives concrete cases where newly passed tests hide previous failures, and some failures are recovered later. Such information is indeed missed by terminal evaluation.
Critical weaknesses
- I have some doubt about how much new information AUPEC actually provides. It mixes progress timing with resource expenditure, so a model spending more tokens naturally gets lower AUPEC even when final functionality is similar. Appendix C shows that the AUPEC gap becomes much smaller when matching both final verdicts and token expenditure. The model-level AUPEC can also be almost perfectly fitted using budgeted endpoint progress and expenditure. So the large ranking changes seem mainly reflecting efficiency differences. Not very clear how much comes from the actual progress trajectory, especially regressions. The baseline preservation scores are also nearly perfect for all models, suggesting AUPEC is largely measuring how quickly new functionality is acquired.
- The regression results are quite concentrated. From Table 4, most regression trials and P2P transitions come from python-02. Several other tasks have no observed regression at all. This makes me wonder how representative the reported continuity problems are. Also, many tasks are constructed by removing or stubbing existing functionality and then restoring it stage by stage. Such setting is controlled, but the regression behavior may differ from real software evolution, where requirements are genuinely new and sometimes conflicting.
Questions for authors
-
How much would the AUPEC model rankings change if using only F2P completion, without P2P preservation? This could help understand whether the ranking differences really involve behavioral continuity.
-
In Table 4, python-02 accounts for 13 of 17 regressed trials and 42 of 46 P2P regressions. What happens to the main regression conclusions if this task is excluded?
-
The requirement DAG can admit multiple valid release orders. Have you examined whether changing the order affects regression frequency or model rankings? This seems relevant for interpreting continuity as an agent capability.
8 comments
Response to Reviewer 9F9s (2/6): Weakness 1
Weakness 1: Do AUPEC differences reflect efficiency rather than continuity?
AUPEC measures functional progress within a shared resource budget; it does not measure regression frequency on its own. Efficiency differences are precisely one of the things it is meant to capture. Behavioral continuity is characterized directly by per-test pass, regression, and recovery events; the two are complementary.
First, AUPEC does not merely penalize total expenditure. It integrates the progress curve within the shared budget: of two trajectories with the same terminal functionality and the same total expenditure, the one that obtains functionality earlier and keeps it gets a higher AUPEC, and continuing to spend after reaching the same functionality does not reduce the area accumulated before. It therefore measures how much usable functionality is obtained while resources are spent, not simply the terminal score divided by cost.
Second, the fit in Appendix C uses the progress at the budget boundary, S(B), not the terminal S at arbitrary expenditure. S(B) already contains whether a model reaches a given functional level within the shared budget. That in-budget endpoint progress and expenditure explain most of the model-level AUPEC differences shows that these models differ mainly in the level and speed of functionality obtained within the budget, consistent with the design goal of AUPEC. Table 16 lists the explanatory power of each endpoint variable for model-level AUPEC, including terminal S in place of S(B).
Model-level AUPEC is well fit by endpoint quantities: S(B) + log token explains 99.65%, and terminal S + linear tokens 99.38%. But this is a descriptive fit over 8 model means and does not show that the endpoints determine the area. Appendix C.1 tests the remaining trajectory differences at the trial level: 63 trial pairs with identical terminal per-test verdicts differ in AUPEC by 17.65 points on average; after also matching expenditure (token ratio at most 1.10), 8 pairs remain and still differ by 3.55, including a python-03 pair whose tokens differ by only 2.01% (45.81M vs 44.91M) and whose AUPEC differs by 9.44. The gap shrinks after matching expenditure because expenditure is itself part of AUPEC; the remaining difference comes from when progress occurs.
Finally, near-full P2P preservation does not mean that no regression occurred during development. P2P checks the existing behavior of the initial workspace; F2P functionality obtained in earlier stages can also be broken and later restored. Whichever kind of test regresses, it can be offset by the credit for new functionality at the same step. SWE-Continuum therefore records regressions and recoveries per test alongside S and AUPEC, rather than inferring them from changes in the AUPEC ranking.
The revision clearly separates these two conclusions: AUPEC shows how the resource budget changes the assessment of functional progress; per-test trajectory analysis shows the behavioral-continuity problems missed by aggregate scores.
Table 16 (revision). Variance decomposition of model-level AUPEC ( of linear regressions over the 8 model means; tokens are total terminal expenditure; the common task set is the 6 tasks on which all 8 models have valid trials).
| Regressors | All 9 tasks | Common 6 tasks |
|---|---|---|
| Terminal S | 1.27% | 0.10% |
| S(B) | 73.80% | 74.85% |
| log tokens | 86.49% | 84.81% |
| Terminal S + log tokens | 94.25% | 92.23% |
| S(B) + log tokens | 99.65% | 98.03% |
| Terminal S + linear tokens | 99.38% | 98.29% |
Response to Reviewer 9F9s (3/6): Weakness 2
Weakness 2: Concentration on python-02 and the delete-and-restore construction
Our conclusion does not rely on "every task regresses frequently", but on "looking only at terminal or aggregate scores cannot tell whether existing behavior is preserved", and it still holds after excluding python-02. "Delete-and-restore" is a design choice made to keep tasks controllable and verifiable.
On the concentration of regressions, the uneven distribution is itself information revealed by trajectory analysis: python-02 has 282 P2P tests, and its 6 stages repeatedly modify the same nosec parsing logic, so later changes easily break behavior that was already correct. A large P2P suite alone does not produce regressions: python-06 has 229 P2P tests and no P2P regression in any of its 11 trials; each of the other 7 primary tasks has only 12–15 P2P tests, so one broken test costs 6.67–8.33 points. Excluding python-02, of the 5 checkpoints with regressions, S still rises at 3 and falls at 2. Of the 739 checkpoints without regressions, S rises at 578 (78.2%) and never falls. Looking only at S therefore cannot distinguish "pure progress" from "progress accompanied by regressions", which is why SWE-Continuum records regressions and recoveries per test.
On task construction, only 5 of the 16 tasks are delete-and-restore reconstruction tasks (only two of the 9 primary tasks, python-03 and python-05); the other tasks start from the starting commit of a real project and implement new requirements step by step according to their dependencies. python-02, where regressions are most concentrated, is not a reconstruction task: 13 of its 14 trials regress, whereas among the two reconstruction primary tasks only 1 trial of python-05 regresses. Delete-and-restore gives every requirement the original project's implementation as an oracle, so we can verify that the task is solvable and the tests are correct, and different agents face exactly the same requirement sequence. This lets us attribute failures to the agent rather than to defects in the task. Scenarios where requirements conflict and existing designs must be overturned are equally important; our Task Spec and Task Build Pipeline do not restrict the task source and can build such tasks with the same verification process, and we will add them in later versions.
The revision discusses the scope of applicability of the delete-and-restore construction in Limitations and adds regression statistics with python-02 excluded (Section 6, Appendix I.4, and Table 10 of the revision).
Response to Reviewer 9F9s (1/6): General Comments
We thank the reviewer for recognizing the controlled comparison enabled by the fixed requirement schedule, the careful task construction and verification process, and the value of test-level trajectory analysis. Looking only at the terminal state cannot tell how an agent reached it; newly passing tests mask damage to existing behavior, and later recoveries are equally invisible.
SWE-Continuum is a complete evaluation system, covering the Task Build Pipeline, the execution environment, and analysis metrics and visualization tools. The metrics each have their own role:
S measures the net functional progress at each checkpoint.
Per-test pass, regression, and recovery events measure behavioral continuity.
AUPEC measures how quickly progress accumulates within a shared resource budget.
The released analysis suite also provides finer diagnostics, such as immediate completion of newly released requirements; requirement-level first completion, regression, and recovery; features once achieved but later lost; and the cost to first reach and stably hold a target score. The main points are:
AUPEC provides information beyond the terminal result: it reflects when and how fast progress occurs within the budget and complements the per-test regression and recovery events; it is not meant to measure continuity on its own.
The uneven distribution of regressions across tasks is itself information revealed by trajectory analysis. Our conclusion does not rely on all tasks regressing equally often, but is that terminal or aggregate scores alone cannot tell whether existing behavior is preserved.
The delete-and-restore task construction and the fixed release order both make tasks controllable, verifiable, and fairly comparable. We explain the rationale and scope of each below.