- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Marathoner, a method designed to enhance the long-duration execution capabilities of open-source code agents through post-training. The authors construct verifiable tasks from GitHub pull requests and extend task duration by chaining multiple tasks together. They then employ a combination of rejection sampling fine-tuning, reinforcement learning, and "late-stage reward boosting" to encourage the model to continue making meaningful progress during extended execution. Experimental results demonstrate that Marathoner-9B significantly outperforms its base model across five benchmarks involving code and terminal operations, capable of running continuously for hours and performing hundreds of tool calls.
Critical strengths
The scope of work and the research direction are commendable. This paper presents large-scale synthesis tasks, corresponding executable and verifiable training environments, and a complete post-training pipeline integrating RFT and RL. Marathoner-9B demonstrates significant improvements over the base model across 5 benchmarks.
Critical weaknesses
-
The two key methodological innovations lack compelling evidence. "Multi-Task Chaining" essentially concatenates independent tasks rather than addressing tasks with genuine long-term dependencies; the "Later Stage Bonus Reward" relies on the subjective judgment of an LLM without verification of human alignment or reward reliability.
-
The primary evaluation metric remains the final task success rate; execution time, step count, and tool call frequency are presented merely as descriptive statistics for Marathoner, without corresponding metrics for other baselines. A longer execution process could signify effective, sustained progress, or it could imply lower efficiency—such as numerous repetitive or futile attempts. Consequently, the current experiments cannot clearly distinguish whether performance gains stem from enhanced ultra-long-horizon capabilities, general improvements in programming proficiency, increased test-time computation, or even a model bias toward continuing execution rather than terminating early.
Questions for authors
-
How did the authors ensure that the 100k training PRs did not overlap with the repositories, issues, or ground-truth solutions found in the five evaluation benchmarks? Was any deduplication performed?
-
How many trajectories actually exceeded 10 hours or 1,000 tool calls? What proportion of these trajectories ultimately resulted in task completion? Can the authors report the distributions of execution time, execution steps, and tool call counts separately for the base model, the RFT model, and the final model, while distinguishing between successful and failed trajectories? The currently reported average execution times—3.56 hours on FrontierSWE and 0.58 hours on Terminal Bench 2.0—still fall short of the core claim regarding "sustained execution for over 10 hours."
-
The "Multi-Task Chaining" strategy involves randomly chaining multiple unrelated tasks; while this certainly increases the total workload, it does not necessarily require sustained reasoning or long-range planning from the model. The model might simply be completing a series of independent sub-tasks one after another. Consequently, performance gains resulting from multi-task chaining cannot directly demonstrate superior long-horizon credit assignment, memory, or planning capabilities.
-
In "Environment Scaling" experiments, were the number of training steps and the total rollout computational cost kept constant? If not, how can it be proven that the performance improvements stem primarily from environmental diversity rather than increased training computation?
-
Can the authors report the total token count, number of context overflows, and number of compression events for the case study involving "11.8 h, 872 steps, and 1,273 tool calls."
5 comments
Response to Reviewer cHgu (Part 2/5)
Comment(2): the "Later Stage Bonus Reward" relies on the subjective judgment of an LLM without verification of human alignment or reward reliability.
Response(2): Thanks for your insightful question. Yes, the Later Stage Bonus Reward is currently assigned based on LLM-based subjective judgments. Following your suggestion, we conducted an additional human evaluation to assess the reliability of these judgments and their agreement with human assessments. Specifically, we recruited five human experts with STEM backgrounds to evaluate the reward assignments. We randomly sampled 100 trajectories from Marathoner's RL training process and assigned 20 trajectories to each expert. For each trajectory, the expert was instructed to carefully examine the execution process, summarize it into several major stages with concise descriptions, and assign a binary flag (0 or 1) to each stage to indicate whether it contained highly valuable operations. If at least one stage in the second half of the execution received a flag of 1, the trajectory was assigned a Later Stage Bonus Reward of 0.5; otherwise, the bonus reward was 0. We then compared the reward assignments made by human experts with those generated by the LLM. The results are presented below. Specifically, we report the percentage of trajectories for which human experts and the LLM agree on the reward assignment, i.e., both assign a bonus reward of either 0.5 or 0.
| Human Expert #1 | Human Expert #2 | Human Expert #3 | Human Expert #4 | Human Expert #5 | Average | |
|---|---|---|---|---|---|---|
| Calibration Result | 90% | 95% | 85% | 100% | 95% | 93% |
We observe that LLM-based judgments of the Later Stage Bonus Reward exhibit high agreement with human expert assessments. This demonstrates the reliability of using LLMs to assign the Later Stage Bonus Reward and provide effective supervision for encouraging the agentic model to perform meaningful and valuable operations during later stages of execution. Notably, we adopt Qwen3.7-Max as the LLM judge for the Later Stage Bonus Reward during RL training. Qwen3.7-Max is itself a strong agentic coding model with substantial ultra-long-horizon execution capabilities, making it well-suited to identifying valuable operations and meaningful progress within long-horizon execution trajectories. Its strong coding and long-horizon reasoning capabilities may account for the high reliability of its judgments and the strong agreement with human expert assessments. We will incorporate these additional analyses into Appendix C.
Comment(3): The primary evaluation metric remains the final task success rate; execution time, step count, and tool call frequency are presented merely as descriptive statistics for Marathoner, without corresponding metrics for other baselines.
Response(3): Thanks for your insightful comment. Following your suggestion, we provide additional execution-horizon metrics for the baselines to facilitate a more comprehensive comparison, including average execution time, average number of steps, and average number of tool calls. The results are presented below:
| Models | Harness | FrontierSWE | Terminal Bench 2.0 | ||||
|---|---|---|---|---|---|---|---|
| Metric | Avg. Execution Time | Avg. Steps | Avg. Tool Calls | Avg. Execution Time | Avg. Steps | Avg. Tool Calls | |
| GPT-6-Astra | Codex | 12.84h | 895.3 | 1637.3 | 1.18h | 149.3 | 493.7 |
| Claude Fable 5.1 | Claude Code | 10.27h | 723.6 | 1438.2 | 1.27h | 163.8 | 583.3 |
| Kimi K3 | Claude Code | 9.07h | 638.4 | 1294.6 | 0.76h | 137.2 | 403.8 |
| Marathoner-9B | Claude Code | 3.56h | 426.2 | 648.3 | 0.58h | 104.8 | 239.4 |
We observe that Marathoner still falls behind flagship proprietary models, such as GPT-6-Astra and Claude Fable 5.1, in terms of average execution time, number of execution steps, and tool-call count. These frontier models demonstrate stronger capabilities in sustaining ultra-long-horizon execution and persistent task solving, which contribute to their higher task completion rates. These results suggest that Marathoner does not simply rely on excessive tool calls or unnecessarily prolonged execution to solve tasks. Instead, there remains substantial room to further strengthen its ultra-long-horizon execution capabilities, enabling it to sustain more sophisticated and persistent problem-solving processes and achieve better performance on challenging tasks.
Response to Reviewer cHgu (Part 3/5)
Comment(4): How did the authors ensure that the 100k training PRs did not overlap with the repositories, issues, or ground-truth solutions found in the five evaluation benchmarks? Was any deduplication performed?
Response(4): Thanks for your valuable question. We address this concern from two perspectives. (1) Following your suggestion, we conducted a thorough contamination analysis of our synthesized training tasks against all evaluation benchmarks. For FrontierSWE, NL2Repo, SWE-Marathon, and Terminal-Bench 2.0, which are manually constructed and curated by human experts, we primarily examine repository-level overlap between the repositories involved in benchmark tasks and those used for our training task synthesis. For SWE-Bench Verified, which is constructed based on GitHub issues, we additionally examine issue-level overlap.
Specifically, for FrontierSWE, NL2Repo, SWE-Marathon, and Terminal-Bench 2.0, we first identify all GitHub repositories associated with the benchmark tasks, including repositories directly used or mentioned as references. We then compare these repositories against the 10,000 GitHub repositories used in our training task synthesis. We find that only 17 out of the 10,000 training repositories overlap with repositories associated with these benchmarks. Furthermore, through manual inspection of all overlapping cases, we find that the task instructions and usage scenarios involving these repositories are substantially different between the benchmark tasks and our synthesized training tasks, indicating that the actual task-level duplication is very limited.
For SWE-Bench Verified, we first extract all PRs used in our training data that are associated with GitHub issues and identify their corresponding issues and repositories. We then compare both repository-level and issue-level overlap with those associated with SWE-Bench Verified. We find that only two tasks in our training data have PR-associated issues that overlap with issues in SWE-Bench Verified. Notably, the overlapping SWE-Bench Verified issues generally involve relatively simple problems and require only a small number of code changes, whereas our synthesized training tasks require PRs involving at least 100 lines of newly added code. Therefore, despite the issue-level overlap, the actual task requirements and implementation complexity are substantially different.
(2) To further eliminate potential contamination concerns, we perform complete deduplication of our training task pool, removing all tasks synthesized from repositories overlapping with the evaluation benchmarks, as well as tasks associated with overlapping GitHub issues. We then conduct both RFT and RL training using this deduplicated training dataset while keeping all other training settings unchanged. The experimental results are presented below:
| Models | Harness | FrontierSWE | NL2Repo | SWE-Marathon | Terminal Bench 2.0 | SWE-Bench Verified |
|---|---|---|---|---|---|---|
| Marathoner-9B | Claude Code | 25.4 | 34.7 | 8.2 | 57.2 | 77.5 |
| Marathoner-9B w. Data Deduplication | Claude Code | 25.6 | 34.6 | 8.5 | 57.1 | 77.4 |
We observe no substantial performance degradation in Marathoner after deduplicating the training data. This suggests that the effectiveness of our post-training pipeline does not rely on potential training–evaluation data overlap. Instead, our pipeline develops genuine and robust ultra-long-horizon execution capabilities that generalize to challenging benchmark tasks, including those associated with repositories or issues that potentially overlap with the original training data.
Response to Reviewer cHgu (Part 1/5)
Comment(1): "Multi-Task Chaining" essentially concatenates independent tasks rather than addressing tasks with genuine long-term dependencies
Response(1): Thanks for your valuable suggestion. We address this question from two perspectives.
(1) Advanced Multi-Task Chaining. Following your suggestion, we further develop an advanced Multi-Task Chaining strategy to synthesize tasks whose constituent subtasks exhibit explicit long-term logical dependencies. Specifically, given a selected GitHub repository, we first identify its major-release PRs and construct a PR list containing the titles and descriptions of these PRs. We then leverage an LLM to select groups of five major-release PRs that exhibit logical dependencies with one another. Such dependencies may involve building more complex functionalities upon capabilities introduced by previous PRs, refactoring the implementation of functionalities introduced by earlier PRs, or combining capabilities implemented in multiple previous PRs to introduce new functionalities. This selection process is based on the extracted PR titles and descriptions. For each GitHub repository, multiple groups of five interdependent major-release PRs can potentially be identified. We then synthesize challenging tasks based on these selected PR groups. Specifically, we first construct an individual task corresponding to each PR and subsequently chain these tasks together using our original Multi-Task Chaining strategy. This advanced strategy produces Frontier tasks whose constituent subtasks exhibit explicit logical dependencies, making them particularly challenging because the agentic model must maintain long-term plans, preserve execution context and memory, and coordinate multiple dependent implementation stages to successfully complete the overall task.
In practice, we construct 6,000 such tasks from 2,000 GitHub repositories. We incorporate 3,000 tasks into the RFT training stage and another 3,000 into the RL training stage, while retaining the entire original training task pool. All other training settings and hyperparameters remain unchanged. We refer to this new strategy as Advanced Multi-Task Chaining. The experimental results are presented below.
We observe that this advanced Multi-Task Chaining strategy leads to further improvements in ultra-long-horizon execution capabilities. Tasks synthesized through Advanced Multi-Task Chaining contain subtasks with explicit logical dependencies, requiring the agentic model to perform more sophisticated long-term planning and maintain execution context and memory to successfully complete the overall task. During RFT, these tasks encourage the teacher model to demonstrate sophisticated long-term planning and execution behaviors, which can then be transferred to the student model through high-quality supervision. During RL, the agentic model further practices and refines these capabilities by solving complex tasks with long-term dependencies in real-world sandbox environments, leading to more effective acquisition of ultra-long-horizon execution capabilities. We will incorporate these additional discussions into Appendix C.
(2) During the design of our method, we also considered chaining tasks across mutually related GitHub repositories. Our initial idea was to manually design rules to identify repositories with complementary functionalities, such as front-end, back-end, and database repositories. We could then construct complex tasks by chaining tasks from these different repositories, resulting in highly challenging tasks with long-term dependencies across multiple codebases. For example, an agent might need to modify the database schema, update the corresponding back-end protocols, and finally adapt the front-end interface accordingly. However, this approach requires extensive hand-crafted rules to identify relationships between repositories and construct meaningful cross-repository dependencies, making it relatively inflexible and difficult to scale. Therefore, we ultimately adopted our current Multi-Task Chaining design, which is more scalable and suitable for large-scale task synthesis. Moreover, it generates highly diverse tasks across different repositories and domains, facilitating the development of more generalizable and robust ultra-long-horizon execution capabilities.