- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper has agents solve sequences of tasks in a single persistent session by introducing sequential variants of popular software engineering benchmarks. Experiments show that persistence changes model behavior, as models tend to use less output tokens and perform less exploratory work compared to the independent baseline. In response, the authors introduce a sequential RL method to improve model capability in persistent multi-task roll-outs, and demonstrate that it strengthens capabilities.
Critical strengths
The paper’s empirical finding that sequential runs use fewer LLM turns and output tokens than independent runs, is indicative that the model is leveraging prior work and less effort when solving long-running texts. It’s an important finding because: 1., in most benchmarks, tasks are run independently; 2., in real-world workflows, tasks are often run sequentially with accumulated context; 3., it opens the door to further research, like the proposed Seq-RL, to aim to improve reliability in the persistent case.
Critical weaknesses
The novelty of creating sequential variants of existing benchmarks is diminished by the existence of sequential evaluations like ChainWorld, EvaLearn, and CL-Bench (all cited). The authors should further explain why introducing sequential variants of GAIA2 and SWE-Bench Pro is needed, rather than using these sequential benchmarks.
Questions for authors
- What is it about DeepSeek v4 Pro that makes it comparatively less capable of managing accumulated context?
- In Fig. 3, how come LLM turns isn't plotted for the GAI2 dataset or Python use plotted for SWE-Bench Pro?
- Relatedly, the authors state “Under sequential evaluation, all models follow shorter trajectories on both benchmarks,” but the trajectory reduction is only demonstrated across both benchmarks in terms of the output token metric.
2 comments
The clarification on positioning relative to ChainWorld, EvaLearn, and CL-Bench addresses my main concern. Please make the distinction (that you keep existing tasks and graders fixed over sequences of many tasks, rather than designing tasks for cross-task learning) more explicit in the revised related work. The DeepSeek hypotheses are reasonable (perhaps you will mention in them in the revised paper), and moving the GAIA2 turn counts into the main text, with a pointer to Table 5 for SWE-Bench Pro, resolves my question about Fig 3. I will maintain my score.
Response to Reviewer KBCf
We thank Reviewer KBCf for their valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.
On positioning relative to related work
Reviewer KBCf asks us to explain the need for sequential variants of GAIA2 and SWE-Bench Pro.
We will clarify the motivation in the revised related work. Our question is how retaining state across tasks affects agent reliability and effort, and whether training can improve these outcomes.
GAIA2's consistent environments and verifiers also enable RL on human-annotated scenarios, with disjoint training and evaluation universes and training histories exceeding 100K tokens. SWE-Bench Pro then tests transfer to coding without software-engineering RL data.
On DeepSeek's ability to handle long context
Reviewer KBCf asks what could explain DeepSeek v4 Pro being comparatively less capable of managing accumulated context.
On SWE-Bench Pro, DeepSeek often commits early to broad, poorly localized patches and modifies existing tests approximately tenfold more often under persistence (§4.2). We have not isolated the underlying cause, but offer two hypotheses based on these observations and public technical reports:
These remain hypotheses. Identifying which factors explain DeepSeek's larger sequential gap would require further research beyond the scope of this paper.
On reported metrics for sequential evaluation
Reviewer KBCf asks why these specific metrics were reported in Figure 3.
We selected the most striking metric changes within the available figure space. On GAIA2, we emphasized output-token reductions because fewer turns and increased Python use for batching work can both reduce generated output. LLM turn reductions are reported in Appendix Figure 10.
On SWE-Bench Pro, Mini-SWE-Agent prompts agents to use Bash commands. Most repositories primarily use TypeScript, Go, or JavaScript, and agents often batch operations through scripts and piped commands. Counting the individual commands within these batches would describe this behavior more directly than Python frequency.
Reviewer KBCf asks for evidence of shorter trajectories on both benchmarks.
We report output tokens as a measure of output effort in the main text. Their reduction can reflect fewer turns and more action batching through scripts, parallel tool calls, and piped Bash commands. Turn reductions are also reported for both benchmarks: Appendix Figure 10 for GAIA2 and Table 5 for SWE-Bench Pro. Figures 12–13 report the window-size ablations, while Tables 10–11 report turns before and after RL training.
We will clarify this metric reporting choice in the revised manuscript. Thank you for raising these questions.
Overall
Thank you again for your feedback. We will clarify the distinctions from existing sequential benchmarks and include the GAIA2 turn-count results in the main text, with explicit references to the full metric reports. Please let us know if any of your concerns need further discussion.