acceptodds
Under review as a conference paper at ICLR 2027

Not All Experience Transfers: Toward Boundary-Aware Self-Evolution in Task Streams

Abstract

Large language model agents can improve themselves by converting accumulated experience into persistent parametric capabilities. However, existing self-evolving methods are predominantly developed and evaluated in isolated or homogeneous settings, whereas realistic deployment exposes agents to heterogeneous task streams in which tasks from different domains arrive continually without explicit boundaries or reliable feedback. This creates new challenges for recursive self-improvement, including how to accumulate experience, determine when past experience remains transferable, and suppress interference during continual parameter updates. In this work, we study parameter-level recursive self-improvement under heterogeneous task streams and propose BORDER, a framework that combines reinforcement learning with boundary-aware hierarchical experience distillation. BORDER estimates transfer boundaries directly from accumulated experience and organizes experience hierarchically according to its applicability to incoming tasks. Guided by these boundaries, transferable experience is selectively internalized into the policy through on-policy self-distillation, while reinforcement learning continues to optimize the agent using outcome-based rewards. An interference suppression mechanism further stabilizes continual parameter evolution and mitigates harmful interactions across experiences. Experiments across diverse benchmarks, task-stream settings, and model families demonstrate that BORDER consistently improves agent performance over strong self-evolving baselines, highlighting the importance of selective experience transfer for recursive self-improvement in heterogeneous task streams.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Learning, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rHcjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies self-evolving LLM agents in a streaming setting where tasks come in one at a time rather than all at once on a static dataset. The authors propose a method which combines GRPO with OPSD from skills generated over successful trajectories. The policy converts skills into abstract global skills meant to transfer, and are retrieved to help rollouts where the main model failed. BORDER achieves highest or second highest cumulative accuracy across number of domains compared to harness-only or parameter-only optimization.

Critical strengths
  • Well formulated problem setting and mostly clear method. The choices for the algorithm seem reasonable and the KL constraint mirrors prior work from the continual learning literature on mitigating catastrophic forgetting, so showing this can be executed in this agentic setting is nice.
  • The evaluation of the main results is broad. Fair baselines were chosen, and the areas where BORDER is successful and unsuccessful were both described, which increases the trustworthiness of the findings and makes the tradeoffs of the method more clear.
Critical weaknesses
  • I would have liked to have seen a better analysis of what the models are actually learning over the course of training. Do they try to build skills that are more general and transferrable rather than task specific? Is the sample efficiency improving over time (i.e. maybe the time to learn a new task presented in the stream decreases the more the agents do this approach within a stream). The demonstrated improvement is nice, but in many of the cases where agents did the second best or were first but by a narrow margin, having a better sense of what the different methods are learning would lead to a better understanding of what this approach is giving you
  • There have been several methods over the past few months which update both the harness and parameters of the model during training (primarily focused on the isolated setting). Would the authors be able to compare to at least one to see whether this method’s focus on the streaming setting leads to capabilities that those existing works are not able to attain as efficiently/effectively/etc.
Questions for authors

See weaknesses

6 comments

We are grateful to the reviewer for the thoughtful and constructive feedback! We appreciate that the reviews highlight our work for tackling the heterogeneity gap between the settings in which agent self-evolution is typically studied and those in which it is actually deployed, and for demonstrating strong and consistent gains across three benchmarks, two backbones, and two streaming scenarios. Below, we respond to each question in detail, and we will reflect the suggested revisions in the updated manuscript.

1. RE: "Skill generalization" and "Learning efficiency in the task stream"

To better understand what BORDER learns over the stream, we provide two analyses below, one on the skills it builds and one on how its performance changes within the stream. First, BORDER builds both task-specific and general skills by design. Local skills capture what worked in a particular interaction, while the global experience pool stores their abstracted, transferable form. In the representative excerpts below from BORDER's logs on BrowseComp+, the global skill discards instance-specific details (the particular game, paper size, and keyboard) while preserving the underlying multi-hop search procedure, allowing it to transfer to other questions that share the same structure.

Local skill (task-specific): Then, find the animator of Game B who used 8x11 paper and a wireless keyboard. Finally, retrieve the gaming PC's storage size from the animator's profile.

Global skill (abstracted): Then, locate the relevant intermediate entity (e.g., animator, developer, system) who satisfies a specific attribute (e.g., tool usage, role). Finally, retrieve the target attribute (e.g., storage size, version number) from the identified entity's profile.

Second, as a proxy for learning efficiency within the stream, we examine first-encounter accuracy. Since each task is evaluated before its feedback is used for updating, this accuracy reflects how well the agent handles newly arriving tasks. Comparing the first and second halves of the interleaved stream with Qwen2.5-7B-Instruct (table below), accuracy improves in all three domains, which is consistent with previously accumulated experience helping the agent handle later tasks. We also note that in the streaming setting, cumulative accuracy is the key metric, as it covers the first-encounter performance on every task in the stream. While rankings on holdout set may vary, BORDER achieves the best average cumulative accuracy in all four backbone–scenario settings.

DomainFirst halfSecond halfΔ (pp)
BFCL7.97%12.28%+4.31
BrowseComp+7.89%11.11%+3.22
Tau27.41%9.80%+2.40

2. RE: "Harness-Policy Co-Evolving Baseline"

We additionally compare BORDER with two recent methods that jointly update the harness and the model parameters, namely SkillForge [1] and HarnessForge [2]. Both baselines are run with their official implementations on Qwen3-4B-Instruct, under the same streaming protocol and task streams as our main experiments (tables below). Among parameter-updating methods, BORDER achieves the best average cumulative accuracy in both settings with competitive holdout accuracy, and extends its lead in the Interleaved setting, where both co-evolving baselines degrade.

This difference may be related to the adaptation setting. As the reviewer noted, co-evolving methods are primarily developed for non-streaming settings, where the harness and the policy are evolved over multiple rounds on a fixed task set, whereas our protocol evaluates each task before its feedback can be used for updating. This mismatch may affect co-evolving methods in two ways. First, the benefits of joint evolution typically take several rounds to emerge, whereas a single pass over a stream provides few such opportunities and leaves fewer subsequent tasks to benefit from the updates. Second, a harness evolved from a small observed prefix introduces new behavioral requirements that the policy has limited experience to accommodate, and may transfer less reliably to later tasks, particularly when domains are interleaved. This may explain why HarnessForge falls below the vanilla model on BFCL. By contrast, BORDER internalizes experience directly into the parameters without modifying the harness, which may make it better suited to limited streaming feedback.

Isolated

MethodCategoryBFCL Cum.BFCL Hold.BrowseComp+ Cum.BrowseComp+ Hold.Tau2 Cum.Tau2 Hold.Avg Cum.Avg Hold.Δ Cum.Δ Hold.
VanillaNo adaptation13.2814.583.133.1311.209.389.209.030.000.00
ACEHarness only31.2534.3825.0021.8850.0050.0035.4235.42+26.22+26.39
ReasoningBankHarness only40.6328.1314.0625.0054.6959.3836.4637.50+27.26+28.47
GRPOParameter only23.9621.868.5921.8617.5425.0016.7022.91+7.50+13.88
RLSDParameter only18.229.389.3825.0019.8031.2515.8021.88+6.60+12.85
SEEDParameter only14.0621.8812.500.0028.1328.1318.2316.67+9.03+7.64
OPSDParameter only2.860.000.263.139.146.304.093.14-5.11-5.89
SkillForge [1]Harness + Parameter21.8828.1313.8025.0025.7837.5020.4930.21+11.29+21.18
HarnessForge [2]Harness + Parameter1.303.139.3821.8817.1931.259.2918.75+0.09+9.72
BORDER (Ours)Parameter only17.9725.0021.189.3828.1337.5022.4323.96+13.23+14.93

Interleaved

MethodCategoryBFCL Cum.BFCL Hold.BrowseComp+ Cum.BrowseComp+ Hold.Tau2 Cum.Tau2 Hold.Avg Cum.Avg Hold.Δ Cum.Δ Hold.
VanillaNo adaptation13.2814.583.133.1311.209.389.209.030.000.00
ACEHarness only23.4431.2514.0618.7557.8143.7531.7731.25+22.57+22.22
ReasoningBankHarness only17.196.2515.6318.7553.1350.0028.6525.00+19.45+15.97
GRPOParameter only24.2225.0012.7615.6321.4731.2519.4823.96+10.28+14.93
RLSDParameter only19.5318.7513.2831.2517.8028.1316.8726.04+7.67+17.01
SEEDParameter only8.8512.5023.440.0013.0212.5015.108.33+5.90-0.70
OPSDParameter only1.560.000.000.006.306.302.612.08-6.59-6.95
SkillForge [1]Harness + Parameter22.1421.889.110.0025.5231.2518.9217.71+9.72+8.68
HarnessForge [2]Harness + Parameter1.563.137.5515.6315.3621.888.1613.54-1.04+4.51
BORDER (Ours)Parameter only21.7725.0019.5318.7528.7240.6323.3528.13+14.15+19.10

[1] SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support

[2] HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems

Dear Reviewer rHcj,

Thank you again for your review. We understand that the discussion schedule may not align perfectly with your availability, but we would greatly value your feedback before it closes. Could you please let us know whether our rebuttal and additional experiments have sufficiently addressed your concerns?

Thank you again for your constructive feedback. If you find that your concerns have been resolved, we would be grateful if you could reconsider your review score.

Reviewer_eJgXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article introduces BORDER, a self-evolving framework for LLM agents that closes the gap between self-evolution studied in isolated, homogeneous settings and self-evolution deployed in heterogeneous task streams where interleaved-domain tasks arrive without domain labels. It organizes evolution around three requirements: experience accumulation via outcome RL plus on-policy self-distillation of local hindsight skills (with guidance removed during optimization so behavior is internalized, not prompted); experience transfer via a boundary-aware global pool where skills are scored for transferability and de-instantiated before admission, then used only to generate extra rollouts whose environmental outcomes — not the advice — supply the learning signal; and interference suppression via success conditioning and a trajectory-level evidence gate. A central design insight is that experience is conditionally, not uniformly, transferable: reusable decision rules must be separated from instance-specific details, and transferred guidance must be re-validated by new environment outcomes before shaping policy parameters. Across three benchmarks, two backbones and both isolated and interleaved streams, BORDER achieves the highest average cumulative accuracy in all four settings, outperforming the strongest parameter-based baseline by 2.68 points on average.

Critical strengths
  • Clean problem decomposition, with a one-to-one mapping between the three identified requirements and the three modules. The paper isolates three complementary properties that self-evolution must satisfy in heterogeneous streams — experience accumulation, experience transfer, and interference suppression — and assigns one dedicated mechanism to each. This decomposition is not merely rhetorical: it directly motivates the method's structure (local vs. global experience, and the different evidential standards applied to each) and gives the reader a checklist against which to read the evaluation.

  • Reasonable experimental scope and baseline coverage. Experiments span two backbones, two streaming scenarios, and three benchmarks covering multi-step tool use, deep retrieval, and dual-control dialogue; the baselines cover both parameter-based evolution (GRPO, OPSD, RLSD, SEED) and harness-based memory/context methods (ACE, ReasoningBank), so the comparison spans both evolution paradigms rather than only the most convenient neighbors.

  • Clear presentation that is easy to follow. Figure 1 makes the isolated vs. interleaved distinction immediately graspable, the method section follows the same three-part structure as the motivation, and hyperparameters and prompt templates are fully tabulated in the appendix.

Critical weaknesses
  • Insufficient statistical power: no error bars, no significance testing, and headline margins that fall below the resolution of the metric. Results are reported as averages over three seeds, yet the paper contains no standard deviations, confidence intervals, or significance tests anywhere. This matters because the evaluation sets are small: with 32 holdout tasks per benchmark, one task is worth 3.13 points and the averaged holdout column resolves to only ~1.04 points, while the cumulative column resolves to ~0.52 points. The headline gain on Qwen2.5-7B-Instruct in the Interleaved setting is +0.80 points — about one and a half tasks — and on the very same setting BORDER is behind the strongest baselines on holdout (−1.05 points). With all methods operating at absolute accuracies between 2% and 28%, differences of this size cannot be separated from noise, so the "highest accuracy in all four settings" claim is not currently supported.

  • The ablation is run at reduced scale, and it shows that the full method is worse on holdout. The component study uses only Qwen3-4B-Instruct, only the Interleaved scenario, and only 32 training tasks per benchmark — half the main configuration — so no component is validated at the scale or in the settings actually reported in Table 1. Its outcome is also not uniformly supportive: adding experience transfer, the paper's headline conceptual contribution, lowers average holdout accuracy from 22.92 to 17.71 and drops Tau2 holdout from 40.63% to 18.75% (−21.88). By the ablation's own numbers the strongest configuration is Accum.+Supp., not BORDER (Full); the full method leads only on cumulative accuracy, which happens to be the single metric the abstract headlines.

  • Harness-based baselines run on a single backbone, reported only as a scatter plot, and BORDER loses to ACE on both metrics. ACE and ReasoningBank appear solely in Figure 3 as gains relative to Vanilla — there is no table of raw scores, so the comparison cannot be cross-checked — and only for Qwen3-4B-Instruct. Moreover, the paper's own text states that ACE retains the highest scores on both metrics in the Interleaved setting: BORDER's deficit relative to ACE narrows considerably versus Isolated (cumulative 15.63 → 1.04; holdout 11.46 → 3.12) but never closes, and BORDER additionally trails ReasoningBank by 5.30 points on cumulative accuracy. Framing this as evidence that the gap "narrows" rather than as a loss on both metrics overstates what the data show.

Questions for authors
  • The main results are averaged over three seeds but no standard deviations, confidence intervals, or significance tests are reported anywhere. Since one holdout task is worth 3.13 points (≈1.04 points on the averaged holdout column), can you provide per-cell std/CI and a paired significance test — in particular, is the +0.80-point gain on Qwen2.5-7B-Instruct (Interleaved) and the −1.05-point holdout deficit in the same setting distinguishable from noise?

  • The component study uses only Qwen3-4B-Instruct, only Interleaved, and half the training data (32 tasks per benchmark). Can you repeat it at the main configuration (64 tasks, both backbones, both scenarios)? More importantly, since adding experience transfer lowers average holdout accuracy from 22.92 to 17.71 and drops Tau2 holdout from 40.63% to 18.75%, which configuration should a practitioner actually deploy — and under what conditions should cross-task transfer be disabled? (Also, please check Table 2's Avg. Cum. column, which does not match the mean of the three per-benchmark values.)

  • ACE and ReasoningBank are run only on Qwen3-4B-Instruct and appear only as relative-gain scatter points in Figure 3. Please provide the raw cumulative and holdout scores. Since ACE retains the highest scores on both metrics in Interleaved and BORDER additionally trails ReasoningBank by 5.30 points on cumulative accuracy, how should this comparison be read — and what is the compute/rollout cost of BORDER relative to both, given the extra analysis, admission, and resampling calls?

4 comments

We sincerely thank the reviewer for the insightful and valuable comments! We are encouraged that the reviews recognize our paper for identifying and addressing the heterogeneity gap between where agent self-evolution is studied and where it is deployed, and for presenting strong and consistent results across three benchmarks, two backbones, and two streaming scenarios. We address the raised questions below and will incorporate the recommended revisions into the updated manuscript.

1. RE: "Insufficient statistical power" and "Gain on Qwen2.5-7B-Instruct":

To report per-seed statistics under a unified pipeline, we re-ran BORDER and representative baselines under the same setting as the main experiments, with three random seeds each. The tables below report each cell as mean±std, covering both backbones, both task-stream settings, and all three benchmarks. Overall, BORDER shows comparable or lower variance on the averaged metrics, and the re-run results remain in line with the conclusions of the original manuscript.

Regarding Qwen2.5-7B-Instruct (Interleaved), we would like to note that the improvement of BORDER remains consistent in this setting for two reasons. First, BORDER outperforms GRPO on both Avg C (8.49 vs. 7.49) and Avg H (11.80 vs. 10.76), with lower variance on both metrics. Second, the relatively small margins on this backbone may largely reflect its weaker base capability [1-3], as its vanilla Avg C is only 3.39 compared with 9.20 for Qwen3-4B-Instruct. Relative to this lower starting point, BORDER still yields a relative gain of about 13% over GRPO and improves the vanilla accuracy by about 2.5× (3.39 → 8.49). In addition, BORDER achieves the best Avg C and Avg H among the compared methods in all four backbone × stream settings.

Qwen3-4B-Instruct · Isolated

MethodBFCL CBFCL HBC+ CBC+ HTau2 CTau2 HAvg CAvg H
Vanilla13.2814.583.133.1311.209.389.209.03
OPSD3.64±1.830.00±0.000.78±0.690.00±0.007.30±1.607.38±1.783.91±0.802.46±0.59
GRPO22.22±1.5121.87±3.138.68±3.0021.87±6.2518.00±0.6627.09±1.8116.30±1.2223.61±1.20
BORDER20.14±2.2225.00±5.4118.17±3.0415.63±8.2727.87±2.4934.38±6.2522.06±0.3325.00±2.08

Qwen3-4B-Instruct · Interleaved

MethodBFCL CBFCL HBC+ CBC+ HTau2 CTau2 HAvg CAvg H
Vanilla13.2814.583.133.1311.209.389.209.03
OPSD1.04±0.690.00±0.000.17±0.300.00±0.006.28±0.036.33±0.102.50±0.292.11±0.04
GRPO24.31±1.4325.00±3.1211.72±6.8315.63±12.5021.22±0.3426.05±3.6119.08±2.8422.22±3.01
BORDER18.80±4.1521.88±6.2520.31±3.0725.00±3.1331.45±5.4837.50±0.0023.52±3.7328.13±1.05

Qwen2.5-7B-Instruct · Isolated

MethodBFCL CBFCL HBC+ CBC+ HTau2 CTau2 HAvg CAvg H
Vanilla0.000.004.173.135.9912.503.395.21
OPSD0.17±0.300.00±0.008.16±0.7913.54±1.816.09±0.556.39±0.244.80±0.306.64±0.68
GRPO5.21±2.078.34±1.817.97±1.0715.62±0.027.57±0.6710.42±1.806.92±0.6511.46±0.01
BORDER6.42±0.409.38±3.1311.03±0.7525.00±3.139.03±0.4012.50±3.138.83±0.2515.63±2.08

Qwen2.5-7B-Instruct · Interleaved

MethodBFCL CBFCL HBC+ CBC+ HTau2 CTau2 HAvg CAvg H
Vanilla0.000.004.173.135.9912.503.395.21
OPSD0.61±1.050.00±0.002.86±1.450.00±0.005.74±0.526.39±0.243.07±0.882.13±0.08
GRPO6.30±1.116.25±0.008.57±1.3916.67±3.617.60±1.099.38±5.417.49±1.0510.76±2.62
BORDER6.68±1.696.25±0.0010.85±0.5419.80±3.617.91±0.639.38±0.008.49±0.9011.80±1.21

[1] Qwen3 Technical Report

[2] CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

[3] CLEANER: SELF-PURIFIED TRAJECTORIES BOOST AGENTIC REINFORCEMENT LEARNING

2. RE: "Full component study"

To diagnose this more thoroughly, we conducted a full-scale ablation under the same configuration as the main experiments (64 tasks per benchmark), covering both backbones and both streaming scenarios (tables below). With the full-length streams, the full BORDER model achieves the best average cumulative accuracy in all settings. In particular, adding global transfer (BORDER Full vs. Accum. + Supp.) improves average cumulative accuracy by 1.14–2.14 points across all scenarios and backbones, and improves holdout accuracy in most settings, by up to 7.30 points.

Together with the reduced-data ablation in the original manuscript, these results help characterize when global transfer is most beneficial. We conjecture that the holdout drop in the reduced-data setting mainly reflects a cold-start effect in the early stage of the stream. Since the global experience pool is built up incrementally, this effect manifests at two levels. First, global skills are admitted only after being abstracted from successful interactions, so the pool grows conservatively and may remain limited in diversity, biased toward the task patterns seen so far. Second, the skills internalized into the policy parameters are likewise derived from fewer tasks and may not yet generalize well to unseen holdout tasks. As the stream proceeds and more diverse experience accumulates at both levels, global transfer becomes beneficial, as reflected in the full-scale results. We therefore recommend the full BORDER configuration for deployment, while noting that the benefit of cross-task transfer may be less pronounced in very short streams.

In addition, the reported Avg. Cum. pooled all valid rollouts generated when each task was first encountered, using multiple rollouts per task to reduce sampling variability and excluding those affected by environment failures. This weights benchmarks by their valid rollout counts and can therefore yield small differences from the unweighted mean of their individual accuracies. We will revise this column to report the unweighted mean and clarify its definition in the caption.

Isolated

ModelMethodBFCL C/Hτ² C/HBrowseComp+ C/HAvg C/H
4BAccum. Only14.06 / 21.8828.13 / 28.1312.50 / 0.0018.23 / 16.67
4BAccum. + Supp.9.98 / 25.0031.68 / 34.3820.92 / 6.2520.86 / 21.88
4BBORDER Full17.97 / 25.0028.13 / 37.5021.18 / 9.3822.43 / 23.96
7BAccum. Only4.42 / 6.257.83 / 9.380.26 / 0.004.17 / 5.21
7BAccum. + Supp.4.51 / 6.258.68 / 9.3810.76 / 25.007.98 / 13.54
7BBORDER Full6.51 / 9.389.38 / 12.5011.46 / 28.139.12 / 16.67

Interleaved

ModelMethodBFCL C/Hτ² C/HBrowseComp+ C/HAvg C/H
4BAccum. Only8.85 / 12.5013.02 / 12.5023.44 / 0.0015.10 / 8.33
4BAccum. + Supp.11.20 / 18.7532.90 / 28.1319.53 / 15.6321.21 / 20.83
4BBORDER Full21.77 / 25.0028.72 / 40.6319.53 / 18.7523.35 / 28.13
7BAccum. Only6.51 / 9.383.65 / 0.009.64 / 25.006.60 / 11.46
7BAccum. + Supp.6.29 / 9.388.64 / 15.639.16 / 15.638.03 / 13.54
7BBORDER Full8.33 / 6.258.64 / 9.3811.46 / 15.639.48 / 10.42

3. RE: "Raw scores for harness-based methods":

We report the raw scores underlying Figure 3 (Qwen3-4B-Instruct) in the table below, where Δ denotes the gain over the vanilla model and corresponds to the axes of Figure 3. We will also include these raw results in the appendix of the revised manuscript.

Beyond absolute scores, the two paradigms (Harness-based/Parameter-based) respond differently to heterogeneity. Moving from isolated to interleaved streams, ReasoningBank drops by 7.81 (cumulative) and 12.50 (holdout) points, and ACE by 3.65 and 4.17 points, whereas BORDER improves by 0.92 and 4.17 points. This indicate that context-based experience is more prone to cross-domain interference in heterogeneous streams, a challenge that BORDER is designed to mitigate. Moreover, within the parameter-based family, BORDER achieves the best average cumulative and holdout accuracy in both modes. The two paradigms are also orthogonal and could in principle be combined, which we leave for future work.

MethodCategoryModeBFCL Cum.BFCL Hold.BrowseComp+ Cum.BrowseComp+ Hold.Tau2 Cum.Tau2 Hold.Avg Cum.Avg Hold.Δ Cum. (x-axis)Δ Hold. (y-axis)
Vanilla––13.2814.583.133.1311.209.389.209.0300
ACEHarness-basedIsolated31.2534.3825.0021.8850.0050.0035.4235.42+26.22+26.39
ReasoningBankHarness-basedIsolated40.6328.1314.0625.0054.6959.3836.4637.50+27.26+28.47
GRPOParameter-basedIsolated23.9621.868.5921.8617.5425.0016.7022.91+7.50+13.88
RLSDParameter-basedIsolated18.229.389.3825.0019.8031.2515.8021.88+6.60+12.85
SEEDParameter-basedIsolated14.0621.8812.500.0028.1328.1318.2316.67+9.03+7.64
OPSDParameter-basedIsolated2.860.000.263.139.146.304.093.14−5.11−5.89
BORDER (Ours)Parameter-basedIsolated17.9725.0021.189.3828.1337.5022.4323.96+13.23+14.93
ACEHarness-basedInterleaved23.4431.2514.0618.7557.8143.7531.7731.25+22.57+22.22
ReasoningBankHarness-basedInterleaved17.196.2515.6318.7553.1350.0028.6525.00+19.45+15.97
GRPOParameter-basedInterleaved24.2225.0012.7615.6321.4731.2519.4823.96+10.28+14.93
RLSDParameter-basedInterleaved19.5318.7513.2831.2517.8028.1316.8726.04+7.67+17.01
SEEDParameter-basedInterleaved8.8512.5023.440.0013.0212.5015.108.33+5.90−0.70
OPSDParameter-basedInterleaved1.560.000.000.006.306.302.612.08−6.59−6.95
BORDER (Ours)Parameter-basedInterleaved21.7725.0019.5318.7528.7240.6323.3528.13+14.15+19.10

4. RE: "Cost Comparison":

We measure the end-to-end cost of each method under its native pipeline on the Qwen3-4B interleaved stream.

Time (min)ACEReasoningBankBORDER
Experience processing3.471.170.33
Rollouts and parameter learning15.3913.4789.88
Other overhead1.781.474.29
Total20.6416.1194.50

Experience processing comprises reflection and curation for ACE, trajectory evaluation and memory induction for ReasoningBank, and analysis and admission/rewrite for BORDER. Rollouts and parameter learning covers task execution and environment interaction for all methods, plus resampling and parameter updates for BORDER. Experience-processing time that occurs within this stage is counted only in the first row. Other overhead is the remainder, including startup and cleanup.

BORDER's longer time comes mainly from rollouts and parameter learning (89.88 min), which is inherent to parameter-based evolution. BORDER performs additional rollouts, resampling, and gradient updates, whereas ACE and ReasoningBank evolve in harness space with frozen parameters and therefore incur no such cost. The overhead of BORDER's experience processing itself remains small, taking 0.33 min, compared with 1.17 min for ReasoningBank and 3.47 min for ACE.

In exchange, BORDER introduces no additional overhead at inference. Since experience is internalized into the parameters, the evolved model requires no experience retrieval or context injection, whereas ACE and ReasoningBank continue to supply external experience as additional context for every query, which lengthens each prompt. We therefore consider BORDER's additional training cost a reasonable trade-off for a policy that is lighter to deploy.

Dear Reviewer eJgX,

Thank you again for your review. We understand that the discussion schedule may not align perfectly with your availability, but we would greatly value your feedback before it closes. Could you please let us know whether our rebuttal and additional experiments have sufficiently addressed your concerns?

Thank you again for your constructive feedback. If you find that your concerns have been resolved, we would be grateful if you could reconsider your review score.

Reviewer_wqvAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual self-evolution of LLM agents under heterogeneous task streams. It proposes BORDER, combining local skill distillation, global experience curation, and selective filtering to accumulate experience, support transfer, and mitigate interference. Across two backbones, three benchmarks, and two streaming settings, BORDER consistently outperforms the evaluated parameter-update baselines in average cumulative accuracy, although held-out performance and transfer benefits vary.

Critical strengths
  1. The experience-use mechanism is well motivated. BORDER combines selective distillation of local experience with global-experience-guided exploration, assigning different learning pathways to different uses of experience. This provides a concrete approach to managing experience reuse and interference.
  2. The cumulative performance gains are consistent. Across two backbones, three benchmarks, and two streaming settings, BORDER achieves the highest average cumulative accuracy among the evaluated parameter-update methods, supporting its effectiveness in the tested settings.
Critical weaknesses
  1. The independent contribution of selective distillation remains unclear. The ablation introduces success conditioning and likelihood gating together, without separate evaluations or controls matching the amount of distillation supervision. It therefore remains unclear whether likelihood gating adds value beyond success-only filtering or reduced distillation pressure.
  2. The trade-off between transfer and generalization lacks detailed diagnosis. In the reduced-data ablation, adding global transfer lowers average holdout accuracy from 22.92% to 17.71%. Although the authors explicitly acknowledge this result, its causes remain unexplored, limiting our understanding of when experience transfer helps or causes interference.
Questions for authors
  1. Section 3.1 states that domain labels are provided during training, whereas the abstract and Section 4.1 state that explicit domain labels are unavailable. Which setting was used in the reported experiments? If labels were provided, how were they used?
  2. Table 2 introduces success conditioning and likelihood gating together. Do separate ablations or controls matching the amount of distillation supervision show that likelihood gating adds value beyond success-only filtering, rather than primarily benefiting from reduced distillation pressure?
4 comments

We thank the reviewer for the careful reading and helpful suggestions! It is encouraging that the reviews find value in our study of the heterogeneity gap between how agent self-evolution is evaluated and how it is deployed in practice, and in the strong and consistent results of BORDER across three benchmarks, two backbones, and two streaming scenarios. We address each question below and will incorporate the corresponding revisions into the revised manuscript.

1. RE: "Usage of domain labels"

We thank the reviewer for pointing out this unclear wording, and we apologize for the confusion caused by the description in Section 3.1. To clarify, no explicit domain labels are provided in any of the reported experiments. Throughout training and evaluation, neither the model nor any component of BORDER receives information about which domain or benchmark a task in the heterogeneous stream comes from, consistent with the setting described in the abstract and Section 4.1. The only supervision available during training is a task-level outcome signal (i.e., whether the model's attempt on a given task is correct), which serves as the reward for reinforcement learning and carries no information about the task's domain. The "labels" mentioned in Section 3.1 refer to this task-level correctness feedback rather than to domain labels. We will revise Section 3.1 to state this explicitly and remove the ambiguous wording.

2. RE: "Ablation of Distillation Signal"

The experience interference suppression module is designed to mitigate interference among experiences from different tasks in the heterogeneous stream. Within our framework, it mainly targets the dense token-level signals from the distillation side. To this end, success conditioning (S) and evidence gating (G) serve as complementary parts of a single module, with S ensuring that distillation targets come from verified successful trajectories and G ensuring that local skills provide genuine likelihood support for those targets, and we therefore view their effect as a whole. Nevertheless, we conducted an additional ablation to examine the contribution of each, under the same setting as the component analysis for streaming self-evolution in our manuscript (Sec. 4.3). Notably, S alone reduces the amount of distillation supervision but does not improve over Accum. Only, indicating that the gains do not stem merely from reduced distillation pressure. Instead, they come from selecting reliable signals: adding G on top of S substantially improves performance, and the combination achieves the best overall performance (the mean of Avg C and Avg H), reaching 20.92. This suggests that success-only filtering and evidence gating work best in combination, helping to identify more reliable distillation signals in heterogeneous streams.

MethodBFCL C/Hτ² C/HBrowseComp+ C/HAvg C/HOverall
Accum. Only16.15 / 9.3822.22 / 31.259.05 / 0.0015.81 / 13.5414.68
Accum. Only + S4.55 / 18.7520.27 / 12.5022.22 / 3.1315.68 / 11.4613.57
Accum. Only + G5.56 / 15.6331.08 / 34.3817.22 / 18.7517.95 / 22.9220.44
Accum. Only + S + G8.59 / 18.7536.65 / 25.0017.78 / 18.7521.01 / 20.8320.92

3. RE: "Trade-off between transfer and generalization"

To diagnose this more thoroughly, we conducted a full-scale ablation under the same configuration as the main experiments (64 tasks per benchmark), covering both backbones and both streaming scenarios (tables below). With the full-length streams, the full BORDER model achieves the best average cumulative accuracy in all settings. In particular, adding global transfer (BORDER Full vs. Accum. + Supp.) improves average cumulative accuracy by 1.14–2.14 points across all scenarios and backbones, and improves holdout accuracy in most settings, by up to 7.30 points.

Together with the reduced-data ablation in the original manuscript, these results help characterize when global transfer is most beneficial. We conjecture that the holdout drop in the reduced-data setting mainly reflects a cold-start effect in the early stage of the stream. Since the global experience pool is built up incrementally, this effect manifests at two levels. First, global skills are admitted only after being abstracted from successful interactions, so the pool grows conservatively and may remain limited in diversity, biased toward the task patterns seen so far. Second, the skills internalized into the policy parameters are likewise derived from fewer tasks and may not yet generalize well to unseen holdout tasks. As the stream proceeds and more diverse experience accumulates at both levels, global transfer becomes beneficial, as reflected in the full-scale results. We therefore recommend the full BORDER configuration for deployment, while noting that the benefit of cross-task transfer may be less pronounced in very short streams. Isolated

ModelMethodBFCL C/Hτ² C/HBrowseComp+ C/HAvg C/H
4BAccum. Only14.06 / 21.8828.13 / 28.1312.50 / 0.0018.23 / 16.67
4BAccum. + Supp.9.98 / 25.0031.68 / 34.3820.92 / 6.2520.86 / 21.88
4BBORDER Full17.97 / 25.0028.13 / 37.5021.18 / 9.3822.43 / 23.96
7BAccum. Only4.42 / 6.257.83 / 9.380.26 / 0.004.17 / 5.21
7BAccum. + Supp.4.51 / 6.258.68 / 9.3810.76 / 25.007.98 / 13.54
7BBORDER Full6.51 / 9.389.38 / 12.5011.46 / 28.139.12 / 16.67

Interleaved

ModelMethodBFCL C/Hτ² C/HBrowseComp+ C/HAvg C/H
4BAccum. Only8.85 / 12.5013.02 / 12.5023.44 / 0.0015.10 / 8.33
4BAccum. + Supp.11.20 / 18.7532.90 / 28.1319.53 / 15.6321.21 / 20.83
4BBORDER Full21.77 / 25.0028.72 / 40.6319.53 / 18.7523.35 / 28.13
7BAccum. Only6.51 / 9.383.65 / 0.009.64 / 25.006.60 / 11.46
7BAccum. + Supp.6.29 / 9.388.64 / 15.639.16 / 15.638.03 / 13.54
7BBORDER Full8.33 / 6.258.64 / 9.3811.46 / 15.639.48 / 10.42

Dear Reviewer wqvA,

Thank you again for your review. We understand that the discussion schedule may not align perfectly with your availability, but we would greatly value your feedback before it closes. Could you please let us know whether our rebuttal and additional experiments have sufficiently addressed your concerns?

Thank you again for your constructive feedback. If you find that your concerns have been resolved, we would be grateful if you could reconsider your review score.

Thank you for the clarification and additional experiments, which help address my concerns. I appreciate the authors’ efforts and maintain my original rating of 3 (Weak acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.