acceptodds
Under review as a conference paper at ICLR 2027

COPE: Recovering Long-Horizon Coding Tasks through Compression of Pull-Request Evolution

Abstract

Repository-level coding tasks are commonly derived from historical development artifacts such as issues and pull-requests (PRs). However, such artifact boundaries reflect development workflow rather than the semantic boundaries of functionality, such that the resulting tasks often capture only fragments of a feature's complete behavioral contract. We introduce **COPE** (**C**ompression **O**f **P**ull-request **E**volution), a framework that treats PR evolution history as evidence for task construction rather than as a sequence to reproduce. COPE first identifies semantically related, possibly non-contiguous PRs that shape the same functionality. It then recovers the behavioral specification revealed across their evolution and poses it as a single executable task at an earlier repository state. Each task thus compresses requirements that emerged over multiple development stages into a longer semantic horizon, aligning with semantic boundaries of true software functionality. Using COPE, we construct 24,000 long-horizon coding tasks spanning 14,868 repositories and obtain 4,478 execution-verified trajectories as a training pool. Under the same training-token budget, models trained with COPE supervision outperform those trained with single-PR and other long-horizon supervision on representative software engineering benchmarks. In particular, with only 384 trajectories, COPE PR-Tree yields relative improvements of 21.3% on SWE-Bench Verified and 33.3% on SWE-Bench Multilingual over the base model. The COPE tasks and code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents, Long-Horizon Tasks, Software Evolution, Agent Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UySVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a new data synthesis method for training SWE agents. The paper proposes that, instead of taking single PRs in order and asking an agent to reproduce them, we should instead group PRs semantically by the task that they are supposed to reflect, then create a new SWE task for this task. The paper claims significant empirical improvements, but does not demonstrate that this performs significantly better than other data generation approaches for synthesizing training data from real SWE repositories.

Critical strengths
  1. The idea of linking together PRs in this manner is a reasonable suggestion, and the figures and writing make the process clear
  2. The training dataset collected is surely a useful tool for future SWE agent training regimes
  3. Though empirical improvements are modest, it is possible that SWE-Bench-Verified does not reflect this since this benchmark is of the "rollback one PR" variety and doesn't involve more complex tasks
Critical weaknesses
  1. Empirical Validity:
  • Unclear whether this data synthesis method truly performs better than the comparisons, since the difference is < 2 %. Would you be able to provide statistical significance calculations?
  • Table 1 is nice for showing the complexity of the tasks. Would you also be to demonstrate relative complexity through the resolution rate of frontier models on these tasks?
  1. Writing & Presentation: The conclusion is much too short; I feel that I would rather have more detailed analysis of results and more comparisons to different data generation techniques. I appreciate the level of detail put into describing the method; however, I feel that there is too much space devoted to this that could be dedicated to a longer conclusion and more analysis of results.
Questions for authors
  1. Did you compute statistical significance for the dataset comparisons?
  2. What is the resolution rate on this new dataset?
3 comments

Q1: Statistical significance calculations.

A1:

We thank the reviewer for suggesting a statistical analysis. We first clarify that the improvements are not uniformly below two percentage points. PR-Tree supervision improves over Single-PR supervision by 3.20 percentage points on SWE-Bench Verified and 3.33 percentage points on SWE-Bench Multilingual. The corresponding improvements over daVinci-Agency supervision are 2.20 and 3.67 percentage points, respectively. For context, daVinci-Agency reports a 2.4-percentage-point improvement over its base model on SWE-Bench Verified.

We have supplemented the original results with paired statistical analyses of the per-instance evaluation outcomes for Qwen3-30B-A3B-Thinking-2507. The tables below report differences in task resolution rates, 95% paired-bootstrap confidence intervals, and two-sided exact McNemar pp-values for PR-Chain and PR-Tree supervision against Single-PR and daVinci-Agency supervision.

Paired statistical comparisons on SWE-Bench Verified (500 tasks) using Qwen3-30B-A3B-Thinking-2507:

Supervision comparisonGain (pp)95% CIpp-value
PR-Chain vs. Single-PR+1.80[0.20, 3.60]0.0636
PR-Chain vs. daVinci-Agency+0.80[-0.60, 2.20]0.3877
PR-Tree vs. Single-PR+3.20[0.60, 6.00]0.0293
PR-Tree vs. daVinci-Agency+2.20[0.20, 4.20]0.0433

Paired statistical comparisons on SWE-Bench Multilingual (300 tasks) using Qwen3-30B-A3B-Thinking-2507:

Supervision comparisonGain (pp)95% CIpp-value
PR-Chain vs. Single-PR+0.67[-1.00, 2.67]0.7266
PR-Chain vs. daVinci-Agency+1.00[-1.00, 3.00]0.5078
PR-Tree vs. Single-PR+3.33[0.67, 6.33]0.0414
PR-Tree vs. daVinci-Agency+3.67[0.67, 6.67]0.0266

The analysis shows that the PR-Tree gains are supported by p<0.05p < 0.05 in all four pairwise comparisons across SWE-Bench Verified and SWE-Bench Multilingual. This provides evidence that COPE can produce more effective training supervision than the compared data sources under the same loss-token budget. We have added the statistical methodology and complete results to Appendix F.2 of the revised manuscript.

Q2: Resolution rate of frontier models on COPE tasks.

A2:

We randomly sampled 500 PR-Chain tasks and 500 PR-Tree tasks and evaluated DeepSeek V4 Flash, DeepSeek V4.1 Flash, and GPT-6.1 Sol on the same task sets, using mini-SWE-agent throughout. The resolution rates are reported below.

ModelPR-ChainPR-Tree
DeepSeek V4 Flash28.20%17.60%
DeepSeek V4.1 Flash36.40%25.20%
GPT-6.1 Sol39.20%29.60%

Across all three models, PR-Tree tasks have lower resolution rates than PR-Chain tasks, with gaps of 9.60–11.20 percentage points. This consistent pattern complements the structural statistics in Table 1, providing empirical evidence that PR-Tree tasks are more challenging for coding agents. Even the strongest model in this evaluation, GPT-6.1 Sol, resolves 39.20% of PR-Chain tasks and 29.60% of PR-Tree tasks, indicating substantial remaining challenges in both task types.

We have added the evaluation setup, results, and analysis to Appendix F.3 of the revised manuscript.

Q3: Writing & Presentation

A3:

We thank the reviewer for suggesting a more thorough analysis and discussion of our results. We have expanded the analysis in Sections 4.3–4.4, substantially revised the conclusion in Section 5, and added complementary experiments in Appendix F.

In Sections 4.3–4.4, we now discuss the differing performance of PR-Chain and PR-Tree across student models and benchmarks, and explain what the Temporal ablation reveals about semantic selection under matched PR counts and training budgets. Appendix F provides additional evidence through the historical-test comparison, paired statistical analyses, and model resolution rates on COPE tasks.

The revised conclusion connects these findings to the differences between data construction approaches. We contrast COPE's unified tasks with individual-PR tasks, chronological aggregation, and daVinci-Agency's staged workflows, and discuss how jointly addressing recovered requirements may provide supervision for both overall planning and detailed implementation. We also relate the lower resolution rates of PR-Tree tasks to their varying downstream benefits, discussing why greater task difficulty does not necessarily translate into stronger supervision for every student model.

Finally, we discuss how executable verification affects task evaluation and trajectory selection. Building on this analysis, we outline future directions for improving alignment between task descriptions and synthesized verifiers, including identifying and correcting false rejections and false acceptances, as well as selecting and combining tasks to better match student capabilities and target tasks.

Reviewer_UySVOpenReview ↗

Thank you for your response

Hi, thank you for addressing my concerns, I have updated my score in accordance with this.

Thank you for your positive assessment and thoughtful engagement with our responses. Your constructive feedback has helped us strengthen the paper, and we sincerely appreciate your time and consideration.

Reviewer_fHJdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

COPE constructs coding tasks by selecting semantically related, potentially non-contiguous pull requests, recovering their accumulated behavioral requirements, and synthesizing a unified task at an earlier repository state. It produces 4,478 execution-verified trajectories from 24,000 sampled tasks. Under matched training loss-token budgets, PR-chain and PR-tree supervision improve Qwen coding models on SWE-Bench Verified and Multilingual.

Critical strengths
  1. Recovering a feature contract across non-contiguous PRs is a well-motivated way to construct richer coding supervision. F2P/P2P validation and a fresh golden precheck provide concrete executable checks on the synthesized artifacts (Sections 3.2–3.5).
  2. The downstream evidence includes token-matched supervision comparisons and a temporal-selection ablation. At 30B-A3B, PR-tree supervision reaches 29.60%/17.33% on Verified/Multilingual versus 26.40%/14.00% for single-PR supervision, supporting the utility of the data construction (Tables 2 and 4).
Critical weaknesses
  1. Student evaluation is restricted to the Qwen3 generation, although three model sizes are tested. Results on an additional generation, such as Qwen3.5 or Qwen3.6, would strengthen the evidence that COPE's benefits are robust to the choice of student model (Section 4.1; Table 3).
  2. COPE is motivated by long-horizon software engineering, but downstream evaluation covers only SWE-Bench Verified and Multilingual. Without results on SWE-Bench Pro, its benefit on a representative benchmark targeting more challenging long-horizon coding tasks remains uncertain (Section 4.1).
  3. Table 2 matches training loss-token budgets but does not clearly specify the trajectory-generation setup for each baseline. COPE reports DeepSeek V4 Flash rollouts, whereas daVinci-Agency's original paper reports GLM-4.6 (Section 4.3, arXiv:2602.02619). It is unclear whether those published trajectories were reused or regenerated, and whether Single-PR uses the same teacher. Differences in teacher, scaffold or filtering would prevent attributing the full cross-dataset gain to task construction alone. The matched-pipeline temporal ablation is useful but does not resolve this Table 2 comparison.
Questions for authors

1.why only conduct experiments on Qwen3 models? Why not qwen3.5 or qwen3.6 series. Maybe more experiment on qwen3.5 or qwen3.6 to show the generalization of your methods. 2. whether COPE improves performance on more challenging long-horizon software-engineering tasks, such as swe-bench pro. 3. Please show the rollout model, harness for daVinci-Agency, If the rollout model is different form deepseek v4 flash, I think the comparation is unfair.

1 comment

Q1 & Q2: Generalization across student generations and evaluation on SWE-Bench Pro

A1 & A2:

We thank the reviewer for suggesting evaluations on an additional student-model generation and a more challenging benchmark. Our original experiments evaluated three Qwen3 model scales to examine the effectiveness of COPE supervision within the same model family. To further assess robustness to the choice of student model and performance on more challenging software-engineering tasks, we added experiments with Qwen3.5-4B and evaluated both Qwen3.5-4B and Qwen3-30B-A3B-Thinking-2507 on SWE-Bench Pro. Training loss-token budgets were matched across supervision sources. For daVinci-Agency, we regenerated trajectories using the same teacher and rollout scaffold as COPE; details are provided in our response to Q3 below.

Student modelBaseSingle-PRdaVinci-AgencyPR-ChainPR-Tree
Qwen3-30B-A3B-Thinking-25078.62%10.53%10.81%12.03%12.58%
Qwen3.5-4B20.93%20.79%19.83%21.61%22.43%

On the newly evaluated Qwen3.5-4B, PR-Chain and PR-Tree improve the resolution rate from 20.93% to 21.61% and 22.43%, respectively, outperforming both Single-PR and daVinci-Agency supervision. These results extend the evidence for COPE’s effectiveness to an additional student-model generation.

On SWE-Bench Pro, the two COPE supervision sources also achieve the two highest scores for both student models. For Qwen3-30B-A3B, PR-Tree improves over Base, Single-PR, and regenerated daVinci-Agency supervision by 3.96, 2.05, and 1.77 percentage points, respectively. For Qwen3.5-4B, the corresponding gains are 1.50, 1.64, and 2.60 percentage points. These findings complement the original evaluations on SWE-Bench Verified and SWE-Bench Multilingual and support COPE’s training utility on more challenging long-horizon software-engineering tasks. We have added these details to Sec. 4.3 of the revised manuscript.

Q3: Trajectory-generation setup and comparison fairness

A3:

We thank the reviewer for pointing out that the trajectory-generation setup was insufficiently specified. In the original experiments, Single-PR and COPE used the same teacher model, mini-SWE-agent scaffold, and trajectory-filtering procedure. For daVinci-Agency, we used its publicly released training trajectories. Thus, Table 2 matched training loss-token budgets but did not standardize trajectory generation across all data sources.

To further control for the teacher and scaffold, we reconstructed tasks using daVinci-Agency’s open-source code, used the same teacher and LLM judge as ours, and generated trajectories with the same mini-SWE-agent scaffold used for COPE. Matching the training loss-token budget yielded 256 daVinci-Agency training trajectories. This comparison preserves daVinci-Agency’s task-construction approach and LLM-based filtering while standardizing the teacher and rollout scaffold.

We conducted additional training experiments using these regenerated trajectories and compared them with Single-PR, PR-Chain, and PR-Tree supervision on the more challenging SWE-Bench Pro benchmark, including evaluation with a newer-generation student model, Qwen3.5-4B. As reported in our joint response to Q1–Q2 above, both PR-Chain and PR-Tree outperform the regenerated daVinci-Agency supervision on both student models. These results support the training utility of COPE under a shared teacher and scaffold.

Reviewer_w7wVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

COPE address the problem where a single isolated PR doesn’t provide agent the training signal of a feature’s complete behavioral contract. It proposes to identify semantically related PRs and recover the complete behavioral specification to construct one task.

Critical strengths
  1. The scale is huge (>24K tasks from ~15k repositories).
  2. It includes meaningful semantic vs temporal selection comparisons.
Critical weaknesses
  1. Passed tests doesn't necessarily show that the recovered specification captures all relevant requirements. Some PRs might have incomplete tests. The core claim is not very well grounded.
  2. Model improvements could also be attributed to increased task complexity, instead of full specification recovery?
  3. Missing comparison (control of task size and complexity) to solidly show that PR-tree structure is more valuable than PR-chain structure.
Questions for authors
  1. how to verify / show specification completeness is achieved in the proposed method.
  2. how to isolate the benefit of semantic recovery from increased task complexity.
6 comments

Q1: Passed tests doesn't necessarily show that the recovered specification captures all relevant requirements. Some PRs might have incomplete tests.

A1:

We thank the reviewer for pointing out this common weakness of constructing tasks from historical PRs, an established practice in prior work such as SWE-Gym (https://arxiv.org/abs/2412.21139) and SWE-rebench (https://arxiv.org/abs/2505.20411). We fully agree that passing historical PR tests alone cannot establish specification completeness, as these tests may be missing or insufficient. Rather than relying solely on historical tests as in prior work, COPE further adapts and supplements them to address their coverage gaps, as detailed below.

How COPE supplements historical tests

COPE does not rely exclusively on the original PR tests. As described in Sections 3.3–3.5, the synthesis agent examines PR descriptions, code changes, and existing tests to assess coverage of the recovered behavioral requirements. For PRs without original tests, it first constructs minimal tests for the contributed functionality and then integrates them with tests from other PRs. It also adds targeted checks for retained requirements, such as boundary conditions and compatibility constraints, that existing tests do not adequately cover. Historical tests are adapted to the unified task's initial repository state and intended final behavior, rather than simply concatenated across development stages. We have clarified this procedure in Sec. 3.5 of the revised manuscript.

Comparison with original PR tests

We sampled 100 PR chains comprising 410 PRs, of which 245 included test changes. We first validated the original tests in their respective historical repository states and confirmed that 218 test suites ran successfully. The remaining 27 were excluded due to environment limitations or issues with the original implementations or tests. The 218 validated test suites span 78 chains.

On these 78 chains, we held the tasks and candidate implementations fixed and compared the original PR tests with COPE's synthesized verifier. For each task, we evaluated the unmodified base repository (Base), the reference implementation, and a fixed DeepSeek-generated patch. The available original test suites were executed separately, and an implementation was accepted only if all suites passed. Application/porting failures were recorded separately from test failures after execution. The table below summarizes the verification results.

CandidatePR tests: PassedPR tests: Porting failurePR tests: Test failureCOPE verifier: Passed
Base0 (0.00%)73 (93.59%)5 (6.41%)0 (0.00%)
Reference7 (8.97%)66 (84.62%)5 (6.41%)78 (100.00%)
DeepSeek patch5 (6.41%)71 (91.03%)2 (2.56%)24 (30.77%)

These results demonstrate the effectiveness of COPE's test adaptation: the synthesized verifier successfully validates all reference implementations and rejects all Base states, while directly combining the original PR tests encounters substantial application and porting failures.

Checks added for PRs without accompanying tests

We also inspected the 70 chains in this subset containing PRs without accompanying tests. In every inspected chain, we identified newly added checks for behaviors contributed by these PRs. This provides evidence that COPE supplements historical tests rather than relying solely on their original coverage.

In summary, the limitation raised by the reviewer is common to datasets constructed from historical PRs. Our results indicate that COPE adapts historical tests to the unified tasks and supplements them with checks for behaviors not covered by the original PR tests. We have added this comparison and the inspection results to Appendix F.1 (Table 10), and clarified the test adaptation and supplementation procedure in Sec. 3.5 of the revised manuscript.

Q2: Semantic recovery versus task complexity.

Could the model improvements be attributed to increased task complexity rather than semantic specification recovery? How can the contributions of these factors be distinguished?

A2:

Our results suggest that COPE's gains arise not merely from larger or more complex tasks, but also from recovering and organizing semantically related behavioral requirements. Increased complexity can be part of this process, as the recovered requirements must be satisfied jointly.

To examine this, the Temporal ablation in Sec. 4.4 starts from the same initial PR as each COPE chain and selects the same number of chronologically consecutive PRs. Both settings use the same task synthesis and rollout pipeline and a matched training loss-token budget. Under these controls, semantically selected PR-Chain supervision achieves 13.00% on SWE-Bench Verified, compared with 10.60% for Temporal supervision, a gain of 2.40 percentage points. This supports the contribution of semantic selection beyond simply aggregating more PRs.

Semantic recovery and increased task complexity are also not mutually exclusive. Recovering boundary conditions, compatibility requirements, and other constraints revealed by later evolution can increase the number of behavioral requirements that must be jointly satisfied. Although the Temporal ablation matches PR counts, PR-Chain tasks involve more modified files and changed lines on average, as shown in the table below, indicating a broader implementation scope. Consistent with this observation, Appendix A, Table 6 reports a lower rollout resolution rate for PR-Chain than for Temporal-PR tasks (29.13% vs. 36.33%). The resulting challenge of satisfying these requirements together may therefore be one mechanism through which semantic recovery provides useful training supervision.

MetricTemporal-PRPR-Chain
Modified files8.079.44
Changed lines613.90897.71

The verifier comparison in A1 provides complementary evidence about task construction. With tasks and candidate implementations held fixed, COPE's synthesized verifier overcomes application and porting failures encountered by the original PR tests. COPE thus contributes executable verification tailored to the unified task, in addition to recovering its behavioral requirements.

Taken together, these results support the value of semantically organizing PR evolution into unified, verifiable tasks. We have expanded the analysis in Sec. 4.4 to explain the controls in the Temporal ablation and discuss how semantic recovery relates to task complexity. We have also added the verifier comparison to Appendix F.1 of the revised manuscript.

Q3: PR-tree versus PR-chain supervision.

Without controlling for task size and complexity, how can the comparison establish that PR-tree structures provide more valuable supervision than PR-chain structures?

A3:

We appreciate this insightful question. We would like to clarify that our paper does not aim to establish that PR-tree structures are superior to PR-chain structures. Rather, COPE introduces PR-chains and PR-trees in a complementary manner to capture different semantic evolution structures. A chain captures successive changes along a single line of functional evolution, whereas a tree captures multiple related branches built on a shared functional foundation. Our comparison evaluates the training utility of these two task types under a matched loss-token budget; it is not intended to establish that tree structures are intrinsically superior to chains.

Table 3 illustrates how their relative performance varies across settings. PR-Tree achieves higher scores on SWE-Bench Verified at all three model scales. On SWE-Bench Multilingual, however, PR-Chain performs better at 4B and 8B, while PR-Tree performs better at 30B-A3B. These results show that the relative benefits of the two supervision sources depend on the student model and evaluation benchmark, rather than following a universal ordering.

The constructed PR-Tree tasks integrate requirements from multiple branches sharing a common foundation and exhibit greater overall complexity. As reported in Table 1, they contain more PRs on average, involve changes to more files, have larger code patches, and entail longer agent interactions. The reported comparison therefore evaluates the two task types with these characteristics retained, rather than isolating branching structure at a fixed task size.

We have clarified the purpose of this comparison and expanded the discussion of the benchmark- and model-dependent results in Sec. 4.3 of the revised manuscript.

Thanks for addressing these concerns. I acknowledge and accept the explanation on Q2 an Q3.

Thank you for the additional comparison with original PR tests. Two questions regarding the comparison: 1. Is there any checks to ensure reference is sound? Any false-negative analysis on rejected model patches? 2. Since the synthesis agent inspects the PR code changes, synthesized tests maybe over specific to reference. Then using reference pass rate to ground the tests validity seems to be a loop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.