- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies self-evolving LLM agents in a streaming setting where tasks come in one at a time rather than all at once on a static dataset. The authors propose a method which combines GRPO with OPSD from skills generated over successful trajectories. The policy converts skills into abstract global skills meant to transfer, and are retrieved to help rollouts where the main model failed. BORDER achieves highest or second highest cumulative accuracy across number of domains compared to harness-only or parameter-only optimization.
Critical strengths
- Well formulated problem setting and mostly clear method. The choices for the algorithm seem reasonable and the KL constraint mirrors prior work from the continual learning literature on mitigating catastrophic forgetting, so showing this can be executed in this agentic setting is nice.
- The evaluation of the main results is broad. Fair baselines were chosen, and the areas where BORDER is successful and unsuccessful were both described, which increases the trustworthiness of the findings and makes the tradeoffs of the method more clear.
Critical weaknesses
- I would have liked to have seen a better analysis of what the models are actually learning over the course of training. Do they try to build skills that are more general and transferrable rather than task specific? Is the sample efficiency improving over time (i.e. maybe the time to learn a new task presented in the stream decreases the more the agents do this approach within a stream). The demonstrated improvement is nice, but in many of the cases where agents did the second best or were first but by a narrow margin, having a better sense of what the different methods are learning would lead to a better understanding of what this approach is giving you
- There have been several methods over the past few months which update both the harness and parameters of the model during training (primarily focused on the isolated setting). Would the authors be able to compare to at least one to see whether this method’s focus on the streaming setting leads to capabilities that those existing works are not able to attain as efficiently/effectively/etc.
Questions for authors
See weaknesses
6 comments
2. RE: "Harness-Policy Co-Evolving Baseline"
We additionally compare BORDER with two recent methods that jointly update the harness and the model parameters, namely SkillForge [1] and HarnessForge [2]. Both baselines are run with their official implementations on Qwen3-4B-Instruct, under the same streaming protocol and task streams as our main experiments (tables below). Among parameter-updating methods, BORDER achieves the best average cumulative accuracy in both settings with competitive holdout accuracy, and extends its lead in the Interleaved setting, where both co-evolving baselines degrade.
This difference may be related to the adaptation setting. As the reviewer noted, co-evolving methods are primarily developed for non-streaming settings, where the harness and the policy are evolved over multiple rounds on a fixed task set, whereas our protocol evaluates each task before its feedback can be used for updating. This mismatch may affect co-evolving methods in two ways. First, the benefits of joint evolution typically take several rounds to emerge, whereas a single pass over a stream provides few such opportunities and leaves fewer subsequent tasks to benefit from the updates. Second, a harness evolved from a small observed prefix introduces new behavioral requirements that the policy has limited experience to accommodate, and may transfer less reliably to later tasks, particularly when domains are interleaved. This may explain why HarnessForge falls below the vanilla model on BFCL. By contrast, BORDER internalizes experience directly into the parameters without modifying the harness, which may make it better suited to limited streaming feedback.
Isolated
| Method | Category | BFCL Cum. | BFCL Hold. | BrowseComp+ Cum. | BrowseComp+ Hold. | Tau2 Cum. | Tau2 Hold. | Avg Cum. | Avg Hold. | Δ Cum. | Δ Hold. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Vanilla | No adaptation | 13.28 | 14.58 | 3.13 | 3.13 | 11.20 | 9.38 | 9.20 | 9.03 | 0.00 | 0.00 |
| ACE | Harness only | 31.25 | 34.38 | 25.00 | 21.88 | 50.00 | 50.00 | 35.42 | 35.42 | +26.22 | +26.39 |
| ReasoningBank | Harness only | 40.63 | 28.13 | 14.06 | 25.00 | 54.69 | 59.38 | 36.46 | 37.50 | +27.26 | +28.47 |
| GRPO | Parameter only | 23.96 | 21.86 | 8.59 | 21.86 | 17.54 | 25.00 | 16.70 | 22.91 | +7.50 | +13.88 |
| RLSD | Parameter only | 18.22 | 9.38 | 9.38 | 25.00 | 19.80 | 31.25 | 15.80 | 21.88 | +6.60 | +12.85 |
| SEED | Parameter only | 14.06 | 21.88 | 12.50 | 0.00 | 28.13 | 28.13 | 18.23 | 16.67 | +9.03 | +7.64 |
| OPSD | Parameter only | 2.86 | 0.00 | 0.26 | 3.13 | 9.14 | 6.30 | 4.09 | 3.14 | -5.11 | -5.89 |
| SkillForge [1] | Harness + Parameter | 21.88 | 28.13 | 13.80 | 25.00 | 25.78 | 37.50 | 20.49 | 30.21 | +11.29 | +21.18 |
| HarnessForge [2] | Harness + Parameter | 1.30 | 3.13 | 9.38 | 21.88 | 17.19 | 31.25 | 9.29 | 18.75 | +0.09 | +9.72 |
| BORDER (Ours) | Parameter only | 17.97 | 25.00 | 21.18 | 9.38 | 28.13 | 37.50 | 22.43 | 23.96 | +13.23 | +14.93 |
Interleaved
| Method | Category | BFCL Cum. | BFCL Hold. | BrowseComp+ Cum. | BrowseComp+ Hold. | Tau2 Cum. | Tau2 Hold. | Avg Cum. | Avg Hold. | Δ Cum. | Δ Hold. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Vanilla | No adaptation | 13.28 | 14.58 | 3.13 | 3.13 | 11.20 | 9.38 | 9.20 | 9.03 | 0.00 | 0.00 |
| ACE | Harness only | 23.44 | 31.25 | 14.06 | 18.75 | 57.81 | 43.75 | 31.77 | 31.25 | +22.57 | +22.22 |
| ReasoningBank | Harness only | 17.19 | 6.25 | 15.63 | 18.75 | 53.13 | 50.00 | 28.65 | 25.00 | +19.45 | +15.97 |
| GRPO | Parameter only | 24.22 | 25.00 | 12.76 | 15.63 | 21.47 | 31.25 | 19.48 | 23.96 | +10.28 | +14.93 |
| RLSD | Parameter only | 19.53 | 18.75 | 13.28 | 31.25 | 17.80 | 28.13 | 16.87 | 26.04 | +7.67 | +17.01 |
| SEED | Parameter only | 8.85 | 12.50 | 23.44 | 0.00 | 13.02 | 12.50 | 15.10 | 8.33 | +5.90 | -0.70 |
| OPSD | Parameter only | 1.56 | 0.00 | 0.00 | 0.00 | 6.30 | 6.30 | 2.61 | 2.08 | -6.59 | -6.95 |
| SkillForge [1] | Harness + Parameter | 22.14 | 21.88 | 9.11 | 0.00 | 25.52 | 31.25 | 18.92 | 17.71 | +9.72 | +8.68 |
| HarnessForge [2] | Harness + Parameter | 1.56 | 3.13 | 7.55 | 15.63 | 15.36 | 21.88 | 8.16 | 13.54 | -1.04 | +4.51 |
| BORDER (Ours) | Parameter only | 21.77 | 25.00 | 19.53 | 18.75 | 28.72 | 40.63 | 23.35 | 28.13 | +14.15 | +19.10 |
[1] SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support
[2] HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems
Dear Reviewer rHcj,
Thank you again for your review. We understand that the discussion schedule may not align perfectly with your availability, but we would greatly value your feedback before it closes. Could you please let us know whether our rebuttal and additional experiments have sufficiently addressed your concerns?
Thank you again for your constructive feedback. If you find that your concerns have been resolved, we would be grateful if you could reconsider your review score.
We are grateful to the reviewer for the thoughtful and constructive feedback! We appreciate that the reviews highlight our work for tackling the heterogeneity gap between the settings in which agent self-evolution is typically studied and those in which it is actually deployed, and for demonstrating strong and consistent gains across three benchmarks, two backbones, and two streaming scenarios. Below, we respond to each question in detail, and we will reflect the suggested revisions in the updated manuscript.
1. RE: "Skill generalization" and "Learning efficiency in the task stream"
To better understand what BORDER learns over the stream, we provide two analyses below, one on the skills it builds and one on how its performance changes within the stream. First, BORDER builds both task-specific and general skills by design. Local skills capture what worked in a particular interaction, while the global experience pool stores their abstracted, transferable form. In the representative excerpts below from BORDER's logs on BrowseComp+, the global skill discards instance-specific details (the particular game, paper size, and keyboard) while preserving the underlying multi-hop search procedure, allowing it to transfer to other questions that share the same structure.
Local skill (task-specific): Then, find the animator of Game B who used 8x11 paper and a wireless keyboard. Finally, retrieve the gaming PC's storage size from the animator's profile.
Global skill (abstracted): Then, locate the relevant intermediate entity (e.g., animator, developer, system) who satisfies a specific attribute (e.g., tool usage, role). Finally, retrieve the target attribute (e.g., storage size, version number) from the identified entity's profile.
Second, as a proxy for learning efficiency within the stream, we examine first-encounter accuracy. Since each task is evaluated before its feedback is used for updating, this accuracy reflects how well the agent handles newly arriving tasks. Comparing the first and second halves of the interleaved stream with Qwen2.5-7B-Instruct (table below), accuracy improves in all three domains, which is consistent with previously accumulated experience helping the agent handle later tasks. We also note that in the streaming setting, cumulative accuracy is the key metric, as it covers the first-encounter performance on every task in the stream. While rankings on holdout set may vary, BORDER achieves the best average cumulative accuracy in all four backbone–scenario settings.