acceptodds
Under review as a conference paper at ICLR 2027

No Reset: Evaluating and Training Persistent Agents on Task Sequences

Abstract

AI agents are typically evaluated one task at a time, yet deployed agents operate in persistent sessions where conversation history, memory, files, and prior experience carry across tasks. We study this mismatch by introducing sequential variants of GAIA2 and SWE-Bench Pro, in which agents solve sequences of up to 40 tasks within a single persistent session. Our experiments show that persistence has a model-dependent effect on task success. Opus 4.6 and GPT-5.4 remain near their independent baselines, whereas DeepSeek v4 Pro loses 5.9 percentage points on average. More consistently, persistence changes how every model behaves: agents use up to 60% fewer output tokens, repeat less exploratory work, and, on GAIA2, increase Python's share of tool calls by up to . Trace analyses reveal both sides of this adaptation: agents reuse knowledge and customized tooling across tasks, but they can also under-explore new tasks and carry unchecked assumptions forward. To improve model capability in this setting, we introduce Sequential RL (Seq-RL), which trains on persistent multi-task rollouts with fine-grained credit assignment. Trained only on GAIA2-style data disjoint from the evaluation set, Seq-RL substantially narrows the sequential gaps on both GAIA2 splits. It also transfers zero-shot to SWE-Bench Pro, where it outperforms a single-task RL baseline trained on the same data, despite receiving no software-engineering data during RL. In short, sequential evaluation exposes new dimensions of agent reliability and in-context learning, while Seq-RL provides a promising training method for more robust long-lived agents.

Primary area:
datasets and benchmarks
Keywords:
AI agents, sequential evaluation, persistent sessions, long-context, memory, in-context learning, reinforcement learning, evaluation, episodic, continual learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KBCfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper has agents solve sequences of tasks in a single persistent session by introducing sequential variants of popular software engineering benchmarks. Experiments show that persistence changes model behavior, as models tend to use less output tokens and perform less exploratory work compared to the independent baseline. In response, the authors introduce a sequential RL method to improve model capability in persistent multi-task roll-outs, and demonstrate that it strengthens capabilities.

Critical strengths

The paper’s empirical finding that sequential runs use fewer LLM turns and output tokens than independent runs, is indicative that the model is leveraging prior work and less effort when solving long-running texts. It’s an important finding because: 1., in most benchmarks, tasks are run independently; 2., in real-world workflows, tasks are often run sequentially with accumulated context; 3., it opens the door to further research, like the proposed Seq-RL, to aim to improve reliability in the persistent case.

Critical weaknesses

The novelty of creating sequential variants of existing benchmarks is diminished by the existence of sequential evaluations like ChainWorld, EvaLearn, and CL-Bench (all cited). The authors should further explain why introducing sequential variants of GAIA2 and SWE-Bench Pro is needed, rather than using these sequential benchmarks.

Questions for authors
  • What is it about DeepSeek v4 Pro that makes it comparatively less capable of managing accumulated context?
  • In Fig. 3, how come LLM turns isn't plotted for the GAI2 dataset or Python use plotted for SWE-Bench Pro?
  • Relatedly, the authors state “Under sequential evaluation, all models follow shorter trajectories on both benchmarks,” but the trajectory reduction is only demonstrated across both benchmarks in terms of the output token metric.
2 comments
AuthorsOpenReview ↗

Response to Reviewer KBCf

We thank Reviewer KBCf for their valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

On positioning relative to related work

Reviewer KBCf asks us to explain the need for sequential variants of GAIA2 and SWE-Bench Pro.

We will clarify the motivation in the revised related work. Our question is how retaining state across tasks affects agent reliability and effort, and whether training can improve these outcomes.

  • EvaLearn studies learning from text-based problems. Deployed tool-using agents also retain files and interact with mutable environments. Evaluating the consequences of this external state is essential to understanding their behavior under persistent deployment.
  • ChainWorld compares batched and sequential presentations of 2 to 4 OSWorld tasks. These comparisons alone do not establish how retaining state affects reliability over sustained workloads. We keep tasks and graders fixed while comparing fresh and persistent sessions over up to 40 tasks. This supports evaluation over long horizons, analysis of behavior changes, and training of persistent agents, including experience reuse and context compaction.
  • CL-Bench designs tasks around learnable shared structure. We preserve existing tasks within GAIA2 universes or SWE-Bench Pro repositories without designing them to require cross-task learning. This lets us test whether ordinary session persistence helps or harms performance beyond tasks designed to favor in-context adaptation.

GAIA2's consistent environments and verifiers also enable RL on human-annotated scenarios, with disjoint training and evaluation universes and training histories exceeding 100K tokens. SWE-Bench Pro then tests transfer to coding without software-engineering RL data.

On DeepSeek's ability to handle long context

Reviewer KBCf asks what could explain DeepSeek v4 Pro being comparatively less capable of managing accumulated context.

On SWE-Bench Pro, DeepSeek often commits early to broad, poorly localized patches and modifies existing tests approximately tenfold more often under persistence (§4.2). We have not isolated the underlying cause, but offer two hypotheses based on these observations and public technical reports:

  • Training for long interactions. Models may differ in training exposure to persistent histories and compaction. OpenAI documents compaction-aware training for GPT-5.1-Codex-Max. Opus 4.6's system card also discusses long agent evaluations using compaction.
  • Retrieval and attention compression. DeepSeek's compressed and sparse attention could affect context retrieval. Its technical report reports lower 1M-token MRCR performance than Opus: 83.5 versus 92.9, both at maximum effort.

These remain hypotheses. Identifying which factors explain DeepSeek's larger sequential gap would require further research beyond the scope of this paper.

On reported metrics for sequential evaluation

Reviewer KBCf asks why these specific metrics were reported in Figure 3.

We selected the most striking metric changes within the available figure space. On GAIA2, we emphasized output-token reductions because fewer turns and increased Python use for batching work can both reduce generated output. LLM turn reductions are reported in Appendix Figure 10.

On SWE-Bench Pro, Mini-SWE-Agent prompts agents to use Bash commands. Most repositories primarily use TypeScript, Go, or JavaScript, and agents often batch operations through scripts and piped commands. Counting the individual commands within these batches would describe this behavior more directly than Python frequency.

Reviewer KBCf asks for evidence of shorter trajectories on both benchmarks.

We report output tokens as a measure of output effort in the main text. Their reduction can reflect fewer turns and more action batching through scripts, parallel tool calls, and piped Bash commands. Turn reductions are also reported for both benchmarks: Appendix Figure 10 for GAIA2 and Table 5 for SWE-Bench Pro. Figures 12–13 report the window-size ablations, while Tables 10–11 report turns before and after RL training.

We will clarify this metric reporting choice in the revised manuscript. Thank you for raising these questions.

Overall

Thank you again for your feedback. We will clarify the distinctions from existing sequential benchmarks and include the GAIA2 turn-count results in the main text, with explicit references to the full metric reports. Please let us know if any of your concerns need further discussion.

The clarification on positioning relative to ChainWorld, EvaLearn, and CL-Bench addresses my main concern. Please make the distinction (that you keep existing tasks and graders fixed over sequences of many tasks, rather than designing tasks for cross-task learning) more explicit in the revised related work. The DeepSeek hypotheses are reasonable (perhaps you will mention in them in the revised paper), and moving the GAIA2 turn counts into the main text, with a pointer to Table 5 for SWE-Bench Pro, resolves my question about Fig 3. I will maintain my score.

Reviewer_ocVcOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the effects of session persistence on AI agents through sequential evaluations adapted from GAIA2 and SWE-Bench Pro. The authors show that persistence affects task performance differently across models while consistently changing agent behavior and tool usage. They further propose Sequential Reinforcement Learning (Seq-RL), which trains agents on persistent multi-task trajectories using local and downstream rewards. Experiments demonstrate improved sequential performance and promising cross-domain generalization, particularly on SWE-Bench Pro.

Critical strengths
  1. A useful evaluation setting for persistent agents. Most agent benchmarks evaluate tasks independently, which does not reflect how agents are used in persistent sessions. I think adapting GAIA2 and SWE-Bench Pro to sequential settings is a useful contribution. The comparison also reveals changes in agent behavior that would be difficult to observe under standard single-task evaluation.

  2. Convincing empirical results, especially on cross-domain transfer. The paper evaluates multiple frontier models and provides a detailed analysis of their behavior under persistence. The Seq-RL results are particularly interesting. On SWE-Bench Pro, sequential avg@3 reaches 42.0%, compared with 24.4% for single-task RL and 25.6% for the base model, despite no software-engineering data being used for RL training. I find this transfer result one of the strongest parts of the paper.

Critical weaknesses
  1. The sources of Seq-RL's improvements are not fully isolated. Seq-RL and single-task RL use the same training scenarios but differ in compute (about 25K vs. 15K H200 GPU-hours) and training context. Also, removing downstream credit assignment leads to only modest performance drops on GAIA2 (Table 8). It is therefore difficult to tell how much of the gain comes from persistent-context training itself and how much comes from downstream credit assignment. This makes the contribution of the individual design choices less clear.

  2. Limited evidence of training stability. The training experiments use only one random seed per configuration. While the reported bootstrap confidence intervals help assess evaluation uncertainty, they do not capture variation across training runs. This is a concern for long-context RL, particularly when assessing the reliability of the cross-domain transfer results. The main gains are substantial, but their consistency across independently trained checkpoints remains unclear.

Questions for authors
  1. Seq-RL uses more training compute than the single-task RL baseline, despite using the same training scenarios. Would its advantage remain under a matched compute budget? A control that separates long-context exposure from sequential training would also help explain where the gains come from.

  2. Table 8 shows relatively small gains from downstream credit assignment on GAIA2. How does the variant without downstream credit perform on SWE-Bench Pro? I would also be interested in whether the results are sensitive to β or the prefix length.

  3. The SWE-Bench Pro results are aggregated over 420 tasks from 11 repositories. Are Seq-RL's improvements consistent across these repositories, or are they mainly driven by a few? A per-repository breakdown for the base model, single-task RL, and Seq-RL would help assess how broadly the gains generalize.

2 comments

We thank Reviewer ocVc for recognizing the practical value of persistent-agent evaluation and the strength of our cross-domain transfer results. Following your suggestions, we have added a prefilled single-task control, evaluated the local-reward-only checkpoint on SWE-Bench Pro, and provided a per-repository breakdown. We hope these additional results and clarifications address your questions and provide grounds for revising your rating upward.

On separating long-context exposure from sequential training

Reviewer ocVc asks for a control that separates long-context exposure from sequential training to clarify the sources of the gains.

Thank you for this suggestion. We report this new ablation in a shared comment because its results are relevant to all reviewers. It clarifies the contributions of long-context exposure and sequential training to performance and behavior.

On Seq-RL versus single-task RL training compute

Reviewer ocVc asks whether Seq-RL's improvements over single-task RL come from a difference in compute (GPU-hours).

To assess learning from the same training scenarios, we match the trainable scenarios, number of rollouts, and number of epochs. Persistent histories increase Seq-RL's cost per rollout, so matching GPU-hours would generally require changing task exposure and would answer a complementary question about budget efficiency. Under matched task exposure, Seq-RL continues to improve while single-task RL largely plateaus (L326; Fig. 15, L1134). This suggests limited benefit from additional epochs of the tested single-task recipe. We have not evaluated a matched-GPU-hour comparison.

On RL stability and multiple seeds for large-scale RL training

Reviewer ocVc notes that each configuration is trained once, so run-to-run variance is not captured.

We agree that multiple seeds would be ideal. We prioritized controlled ablations within the available compute budget and report one training run per configuration. Our β = 1 ablation is an independently trained Seq-RL model that also achieves substantial sequential gains, including cross-domain transfer. On sequential SWE-Bench Pro it reaches 39.1 avg@3, compared with 42.0 for full Seq-RL and 24.4 for single-task RL. It also improves over single-task RL on sequential GAIA2 (Table 8).

We have added the SWE-Bench Pro results for this ablation as requested. These results corroborate gains across training configurations, but do not measure training-seed variance. The reported confidence intervals quantify evaluation uncertainty.

ObjectiveGAIA2 Search: avg@5GAIA2 Execution: avg@5SWE-Bench Pro: avg@3
Full Seq-RL (β = 0.85)92.5 ± 3.086.2 ± 6.142.0 ± 3.4
No downstream reward (β = 1)90.1 ± 3.084.7 ± 4.239.1 ± 2.7

On SWE-Bench Pro gains across repositories

Reviewer ocVc asks whether Seq-RL's improvements are consistent across repositories or driven by a few.

We will add this breakdown to the appendix. Seq-RL has the highest sequential avg@3 of the three checkpoints in all 11 repositories, with gains of +2.5 to +27.5 percentage points over Muse Glimmer-30B. Excluding the three repositories with the largest gains, the task-weighted gain on the remaining tasks is still +13.1 points.

Sequential scores per repository (avg@3, %). Gains are in percentage points.

RepositoryMuse Glimmer-30B (baseline)Single-task RLSeq-RLGain vs baselineGain vs single-task RL
ansible28.327.555.8+27.5+28.3
vuls21.733.345.8+24.2+12.5
teleport15.015.036.7+21.7+21.7
qutebrowser29.225.850.0+20.8+24.2
navidrome13.315.033.3+20.0+18.3
flipt20.027.538.3+18.3+10.8
element-web21.719.235.8+14.2+16.7
NodeBB29.223.340.8+11.7+17.5
tutanota21.720.031.7+10.0+11.7
openlibrary50.837.556.7+5.8+19.2
webclients29.221.731.7+2.5+10.0

Overall

Thank you for requesting a more detailed analysis of long-context exposure and sequential training. We will reframe Section 5 around these additional results in the revised manuscript to strengthen the analysis of Seq-RL. We will release the training data upon publication to support further research on RL across task sequences. Please let us know if any of your concerns need further discussion.

Thank you to the authors for the detailed responses and additional experiments. These clarifications have addressed my main concerns, and I will raise my rating accordingly.

Reviewer_dykkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduce SeqRL (sequential evaluation and sequential rl) to improve the agent's performance across multiple task sessions. The experimental results on GAIA2, SWE-bench reveal that the accumulated context can improve the efficiency through knowledge reuse. However, the side effect is that it can also cause premature decisions and insufficient verification. The proposed rl method substantially improves the sequential reliability and demonstrates promising cross domain generalization.

Critical strengths
  1. The motivation of the paper is from a practical problem, and it is important to measure the performance for a real world agent that has persistent sessions across tasks.
  2. The eval perofroamcne is strong, particularly on SWE-bench (25.6 -> 42%)
  3. The rl method shows cross-domain generalization, which further demonstrates the effectiveness of the proposed method.
Critical weaknesses
  1. The paper could disentangle the effects of different components like context accumulation, compaction etc.
  2. missing references: Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2026, April). Llms get lost in multi-turn conversation. In International Conference on Learning Representations (Vol. 2026, pp. 54738-54778). Li, M., Chen, P., Zhang, Z., Yang, T., Zhang, X., Li, H., ... & Yin, B. (2026, July). Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 33364-33381).
Questions for authors

have the auhtors attempted to validate the seq-rl method on other backbone models, instead of Glimmer 30B?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dykk

We thank Reviewer dykk for their valuable feedback and questions. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

On disentangling component effects

Reviewer dykk asks for more information on the effects of context accumulation and compaction on model behavior.

We agree that the distinction between accumulated context, compaction, and cross-task adaptation should be clearer. Our existing analyses already examine these mechanisms under contrasting conditions:

  1. Long context can hurt performance. On SWE-Bench Pro, a 1M-token window keeps almost the whole history. Sequential runs then lose 4.3 to 15.1 percentage points against independent runs, compared with 0.2 to 6.1 points at 200K (Fig. 13). This observation is consistent with findings in prior work (Liu et al., 2023; Du et al., 2025; Laban et al., 2025).

  2. Compaction reduces context pressure, but it is lossy. With a 32K window on GAIA2, independent runs must also summarize their history, and they lose 2 to 12 percentage points compared with 200K (Tables 2 and 7). Sequential runs summarize more often (Table 6), and two of the three models produce more output tokens than in independent runs. This suggests that they regenerate information lost to summarization (Fig. 12).

  3. In-context adaptation helps reduce repeated work. Agents reuse knowledge, scripts, and tools from earlier tasks. They use 20 to 60% fewer output tokens, and this reduction already appears at the second task, before the context approaches its limit (Figs. 3 and 10).

The sequential gap reflects the combined effects of these three mechanisms, which can explain why it differs across models. These mechanisms depend on one another: compaction occurs because the context grows, and in-context adaptation relies on the same history that compaction shortens. Changing one mechanism can therefore affect the others. Without compaction, the full history must fit in the context window, and even at 1M, most SWE-Bench Pro sequences fill the window (Fig. 14). Our window-size ablations (32K, 200K, and 1M) shift the balance between these mechanisms without removing any of them. We have synthesized these findings in Appendix B.3 and §4.1 of the revised manuscript.

On missing citations

Reviewer dykk asks for two additional citations.

We already cite Laban et al. under its 2025 preprint date in the introduction (L059), related work (L099), and limitations (L449). We will update the reference to the published ICLR 2026 version. We will also discuss “Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewards” in the related work to better position our proposed Seq-RL method.

On validation with other backbone models

Reviewer dykk asks whether Seq-RL has been applied to models other than Muse Glimmer 30B.

Yes. We first applied Seq-RL to closed-source MoE models with 24B–42B active parameters. It was successful across these models, with the same qualitative cross-domain transfer to SWE-Bench Pro. For publication, we replicated our setup on Muse Glimmer 30B and found the same trends. Upon publication, we will release training data so the research community can experiment with new training methods for sequential task solving.

Overall

We see that you have flagged our submission for ethics review. Was this intended, or could you please provide additional context? Thank you again for your valuable feedback. We will add the clarifications described above to the revised manuscript shortly. Please let us know if any of your concerns need further discussion.

AuthorsOpenReview ↗

Thank you for the review

Thanks again for your questions and remarks, we tried to cover everything in our answers and updated the paper accordingly. Please let us know If you have any additional questions for the remainder of the rebuttal discussion, we will be happy to answer them!

Sincerely,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their feedback and for recognizing the practical importance of studying agents in persistent sessions. Our responses address three main themes:

  • Positioning: We clarify why adapting GAIA2 and SWE-Bench Pro addresses a useful question alongside existing sequential benchmarks, and discuss the additional related work suggested by the reviewers.
  • Context and behavior: We bring together the existing window-size, compaction, and trace analyses to explain what each comparison shows about accumulated context and cross-task adaptation.
  • Training contribution: Our additional control clarifies the roles of long-context exposure and sequential training. Prefilled single-task RL achieves strong sequential performance, while full Seq-RL achieves similar accuracy with fewer output tokens and turns. We will incorporate these findings into Section 5.

On separating long-context exposure from sequential training

Reviewers ocVc and dykk ask for a clearer analysis of the contributions of individual components.

Thank you for this suggestion. We believe the results are worth sharing to all reviewers. We ran an additional ablation: single-task RL with prefill, using exactly the same prefills as Seq-RL. This control exposes the model to long contexts and trains its compaction capability, while optimizing individual tasks rather than multi-task continuations.

Sequential performance on GAIA2 and SWE-Bench Pro

Scores are avg@5 for GAIA2 and avg@3 for SWE-Bench Pro, expressed as percentages.

ExperimentPrefillTrains on task sequencesGAIA2 SearchGAIA2 ExecutionSWE-Bench Pro
Single-task RLNoNo87.9 ± 3.282.1 ± 4.624.4 ± 2.9
Prefilled single-task RLYesNo92.1 ± 3.785.1 ± 4.241.3 ± 4.2
Full Seq-RLYesYes92.5 ± 3.086.2 ± 6.142.0 ± 3.4

Output effort under sequential evaluation

Values are mean output tokens and model turns per task. GAIA2-S and GAIA2-E denote the Search and Execution splits; SBP denotes SWE-Bench Pro.

ExperimentGAIA2-S tokensGAIA2-S turnsGAIA2-E tokensGAIA2-E turnsSBP tokensSBP turns
Baseline3.96K16.34.39K23.83.8K22
Single-task RL2.56K10.43.38K20.83.6K20
Prefilled single-task RL6.09K20.15.29K33.214.8K61.1
Full Seq-RL4.11K16.54.20K21.48.7K44.9

These training settings differ as follows:

  • Single-task RL trains on isolated tasks from fresh contexts.
  • Prefilled single-task RL exposes the model to long contexts and trains compaction, but receives no training signal from solving successive tasks within the same continuation.
  • Seq-RL exposes the model to long contexts and compaction while training on successive tasks within persistent continuations.

Prefilled single-task RL achieves accuracy close to Seq-RL on all sequential evaluations. This suggests that long-context exposure and compaction training account for much of the sequential accuracy improvement.

However, full Seq-RL uses fewer output tokens and turns than prefilled single-task RL across all three evaluations: approximately 21–41% fewer output tokens and 18–36% fewer turns. These results support the contribution of sequential training to more efficient task execution at similar observed accuracy. This behavior is consistent with more effective reuse of prior experience.

This ablation clarifies the contributions of long-context exposure and sequential training to performance and behavior.

Overall

The individual responses address each reviewer's questions and provide the supporting results. Please let us know if any concern needs further discussion. We will incorporate these changes into the revised manuscript. We hope these additional results and clarifications address your concerns and provide grounds for revising your rating upward.

AuthorsOpenReview ↗

Rebuttal Summary: Contributions and Revisions

We thank all reviewers for their helpful reviews and the rebuttal discussion. We beleive their feedback made the paper stronger, and we are glad that reviewers ocVc and KBCf found all their concerns addressed.

Research direction: frontier agents are deployed in persistent sessions but still trained and evaluated one task at a time. We make persistence measurable on standard benchmarks and trainable with RL.


Core contributions

  1. Sequential evaluation. Sequential variants of GAIA2 and SWE-Bench Pro: up to 40 tasks in one persistent session, with tasks and graders unchanged.
  2. Behavioral characterization. Persistence affects success in a model-dependent way and changes behavior consistently: agents use up to 60% fewer output tokens and explore less, reusing knowledge and tools across tasks.
  3. Seq-RL. RL on persistent multi-task rollouts with local and downstream rewards. Trained only on GAIA2-style data, it narrows the sequential gap on GAIA2 and transfers zero-shot to SWE-Bench Pro:
SWE-Bench Pro (zero-shot)Muse Glimmer-30B+ Single-task RL+ Seq-RL
Sequential avg@325.624.442.0\color{green}{\mathbf{42.0}}
Gap vs. independent (pp)−20.2−21.9−6.0\color{green}{\mathbf{-6.0}}

Main updates in the revised manuscript (highlighted in blue\color{blue}{\text{blue}})

UpdateWhereReviewers
Disentangling long context, compaction, and in-context adaptation. We identify three mechanisms behind the sequential gap and use 32K and 1M window ablations to show how they interact.§4.1, App. B.3dykk
Seq-RL ablations: what drives performance. Two new controlled RL training suggest that long-context exposure via prefills drives much of the accuracy gain, while sequential training adds efficiency (21–41% fewer output tokens).§5ocVc, dykk
Clarified metrics and positioning. We justify output tokens as the effort measure and now report turns in the main text. Related work positions the paper explicitly against ChainWorld, CL-Bench, EvaLearn, and RLAAR.§2, 3, 4.2KBCf, dykk

Thank you all again, the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.