- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new method to construct long-horizon SWE traces as SFT data by creating tasks from multiple GitHub PRs that are related to one another. To collect training data, the agent is prompted to sequentially solve each PR task one by one. After the agent completes one PR task, rejection sampling is performed using an LLM judge to compare the model-generated patch against the golden patch. Rollout proceeds only on successful trials. This results in a final training set of 239 trajectories, each passing every PR task in a chain. Experiments show that this training dataset, though small, leads to larger gains than baseline datasets in various agentic tasks.
Critical strengths
- The idea of chaining relevant PRs to construct long-horizon coding tasks intuitively makes a lot of sense, and is potentially very scalable given the number of source PRs on GitHub.
Critical weaknesses
- My main concern is about the noise in how the evaluation is presented. In table 1, the non-CoPR datasets only make the model worse, and the SinglePR variant has no rejection sampling, leaving SingePRSuccess as the only strong short-horizon baseline. Some of the benchmarks, such as SWE-Bench and DS-1000, are typically not considered to be long-horizon, and that's probably why CoPR does not show a significant gain over the SinglePRSuccess baseline there. I would recommend trying some more recent benchmarks such as DeepSWE, but I'm not sure if the baseline models are strong enough for those tasks.
- How to mine PR chains from a repo is a key component for reproduction of this paper's idea, but is only described very briefly in Section 4.2. Given the number of repos used in this work is pretty small, it would be good to convince readers that the method generalizes to other repos.
Questions for authors
- Wondering if there can be such an example. Suppose we have chain (PR1, PR2). PR2 asks the agent to fix a bug. However, the agent is able to pass PR1 without even introducing the bug, and there's no need to do PR2 at all.