- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes a new data synthesis method for training SWE agents. The paper proposes that, instead of taking single PRs in order and asking an agent to reproduce them, we should instead group PRs semantically by the task that they are supposed to reflect, then create a new SWE task for this task. The paper claims significant empirical improvements, but does not demonstrate that this performs significantly better than other data generation approaches for synthesizing training data from real SWE repositories.
Critical strengths
- The idea of linking together PRs in this manner is a reasonable suggestion, and the figures and writing make the process clear
- The training dataset collected is surely a useful tool for future SWE agent training regimes
- Though empirical improvements are modest, it is possible that SWE-Bench-Verified does not reflect this since this benchmark is of the "rollback one PR" variety and doesn't involve more complex tasks
Critical weaknesses
- Empirical Validity:
- Unclear whether this data synthesis method truly performs better than the comparisons, since the difference is < 2 %. Would you be able to provide statistical significance calculations?
- Table 1 is nice for showing the complexity of the tasks. Would you also be to demonstrate relative complexity through the resolution rate of frontier models on these tasks?
- Writing & Presentation: The conclusion is much too short; I feel that I would rather have more detailed analysis of results and more comparisons to different data generation techniques. I appreciate the level of detail put into describing the method; however, I feel that there is too much space devoted to this that could be dedicated to a longer conclusion and more analysis of results.
Questions for authors
- Did you compute statistical significance for the dataset comparisons?
- What is the resolution rate on this new dataset?
3 comments
Thank you for your response
Hi, thank you for addressing my concerns, I have updated my score in accordance with this.
Thank you for your positive assessment and thoughtful engagement with our responses. Your constructive feedback has helped us strengthen the paper, and we sincerely appreciate your time and consideration.
Q1: Statistical significance calculations.
A1:
We thank the reviewer for suggesting a statistical analysis. We first clarify that the improvements are not uniformly below two percentage points. PR-Tree supervision improves over Single-PR supervision by 3.20 percentage points on SWE-Bench Verified and 3.33 percentage points on SWE-Bench Multilingual. The corresponding improvements over daVinci-Agency supervision are 2.20 and 3.67 percentage points, respectively. For context, daVinci-Agency reports a 2.4-percentage-point improvement over its base model on SWE-Bench Verified.
We have supplemented the original results with paired statistical analyses of the per-instance evaluation outcomes for Qwen3-30B-A3B-Thinking-2507. The tables below report differences in task resolution rates, 95% paired-bootstrap confidence intervals, and two-sided exact McNemar p-values for PR-Chain and PR-Tree supervision against Single-PR and daVinci-Agency supervision.
Paired statistical comparisons on SWE-Bench Verified (500 tasks) using Qwen3-30B-A3B-Thinking-2507:
Paired statistical comparisons on SWE-Bench Multilingual (300 tasks) using Qwen3-30B-A3B-Thinking-2507:
The analysis shows that the PR-Tree gains are supported by p<0.05 in all four pairwise comparisons across SWE-Bench Verified and SWE-Bench Multilingual. This provides evidence that COPE can produce more effective training supervision than the compared data sources under the same loss-token budget. We have added the statistical methodology and complete results to Appendix F.2 of the revised manuscript.
Q2: Resolution rate of frontier models on COPE tasks.
A2:
We randomly sampled 500 PR-Chain tasks and 500 PR-Tree tasks and evaluated DeepSeek V4 Flash, DeepSeek V4.1 Flash, and GPT-6.1 Sol on the same task sets, using mini-SWE-agent throughout. The resolution rates are reported below.
Across all three models, PR-Tree tasks have lower resolution rates than PR-Chain tasks, with gaps of 9.60–11.20 percentage points. This consistent pattern complements the structural statistics in Table 1, providing empirical evidence that PR-Tree tasks are more challenging for coding agents. Even the strongest model in this evaluation, GPT-6.1 Sol, resolves 39.20% of PR-Chain tasks and 29.60% of PR-Tree tasks, indicating substantial remaining challenges in both task types.
We have added the evaluation setup, results, and analysis to Appendix F.3 of the revised manuscript.
Q3: Writing & Presentation
A3:
We thank the reviewer for suggesting a more thorough analysis and discussion of our results. We have expanded the analysis in Sections 4.3–4.4, substantially revised the conclusion in Section 5, and added complementary experiments in Appendix F.
In Sections 4.3–4.4, we now discuss the differing performance of PR-Chain and PR-Tree across student models and benchmarks, and explain what the Temporal ablation reveals about semantic selection under matched PR counts and training budgets. Appendix F provides additional evidence through the historical-test comparison, paired statistical analyses, and model resolution rates on COPE tasks.
The revised conclusion connects these findings to the differences between data construction approaches. We contrast COPE's unified tasks with individual-PR tasks, chronological aggregation, and daVinci-Agency's staged workflows, and discuss how jointly addressing recovered requirements may provide supervision for both overall planning and detailed implementation. We also relate the lower resolution rates of PR-Tree tasks to their varying downstream benefits, discussing why greater task difficulty does not necessarily translate into stronger supervision for every student model.
Finally, we discuss how executable verification affects task evaluation and trajectory selection. Building on this analysis, we outline future directions for improving alignment between task descriptions and synthesized verifiers, including identifying and correcting false rejections and false acceptances, as well as selecting and combining tasks to better match student capabilities and target tasks.