acceptodds
Under review as a conference paper at ICLR 2027

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

Abstract

Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct behavior in these environments is often implicit, undisclosed, and subject to change over time. Recent continual-learning harnesses seek to address this challenge by enabling agents to improve from serving experience. Yet the effectiveness and limitations of these methods are not yet well characterized. Existing benchmarks provide only partial coverage: some explicitly provide the target knowledge, others assume a static environment, and those that support continual adaptation remain limited in scale and knowledge diversity. To enable systematic evaluation, we formalize an evolving-environment streaming dataset (EESD), in which agents must infer, apply, and revise latent environment knowledge from interaction and outcome feedback as hidden policies evolve, and introduce ServeLearnBench, spanning retail support, banking, and sales-pitch generation with 53 environment windows and 7,718 tasks. We evaluate five learning harnesses (RAG, Mem0, SkillOpt, Continual Harness, and Prime) across six models (GPT-5.6 Terra, Opus 5, Kimi K3, GLM-5.3, DeepSeek V4.1 Flash, and GLM-5.3 Flash), covering 28 model–harness pairs and 252 learning runs. Our evaluation reveals three main findings: a substantial gap remains between task capability and learning from experience; continual adaptation is costly and can degrade already-correct behavior; and insufficient exploration emerges as a key bottleneck to effective adaptation. Overall, ServeLearnBench provides a controlled testbed for diagnosing these limitations and tracking progress toward agents that continually and reliably improve through serving experience.

Primary area:
datasets and benchmarks
Keywords:
Continual Learning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dwr8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds a benchmark for agents that have to learn hidden and changing policies from their own serving history. The stream is split into environment windows. In each window the agent first does serving tasks and gets only an outcome score, then is tested on held out tasks with frozen state and no feedback. Tasks are split into Hidden-Dependent (visible info not enough) and Fully Specified (visible info enough), so the benchmark can measure both learning and damage to behavior that was already correct. There are three domains (Retail, Banking, Pitch) and three tiers, 53 windows and 7,718 tasks. Retail and Banking are checked by a hash of the final database plus the finish answer, Pitch by an LLM judge rubric. They test five harnesses (RAG, Mem0, SkillOpt, Continual Harness, Prime) on six models, 28 pairs. Main results: Oracle gets 95.4 on Hidden cases on average vs 14.1 for Blind, and learners are far below Oracle (best Retail L3 pair 59.6, median 22.0). Learning is costly and can hurt Fully Specified cases. Answer diversity during failure correlates with final Hidden reward.

Critical strengths
  1. The setup measures something that existing benchmarks mostly do not. The policy is hidden, feedback is only the outcome, and the policy changes, gets removed and comes back. Together with Blind and Oracle as lower and upper references, and the Hidden vs Fully Specified split, a result can be read as how much of the learnable gap a harness closes and what it breaks on the way. I think this is the main reason the paper is useful. The gap result (Oracle 95.4 vs the best learner per model at 55.8 to 73.4, Table 2) is large enough that it does not depend on the noise issues below, and the Fully Specified result (Figure 6, Retail Blind 98.9 vs learners 89.5 to 95.5) shows a cost that plain accuracy would hide.

  2. The protocol is careful and the paper is honest about its limits. Test probes run from frozen state and are excluded from every update, validation and selection, and do not even create a visible timestamp gap (App B.3). Retail and Banking use a deterministic verifier with no LLM. The appendix also says openly that harnesses are not budget matched (B.2), each cell is one run (B.4), and the Blind Fully Specified baseline is not a pure damage measure (D.6). I also liked the extra check in D.3 that counts distinct answers only before the first correct answer, because it removes the obvious mechanical link between "trying more answers" and "finding the right one". This makes me trust the benchmark itself more than the average agent benchmark.

Critical weaknesses

The harness level conclusions are not established yet, because every cell is one run and the harnesses are not compared under matched conditions. App B.4 says "Each cell summarizes one run; no standard error or confidence interval is implied." But Table 2 has very large swings that look like noise or failure modes. For example Prime on Kimi K3 Pitch L2 is 16.9, below Blind (25.9), while CH on GLM-5.3 Flash Banking FS L1 is 63.8 vs Blind 95.7. Some patterns also look systematic and unexplained: SkillOpt is near zero on Retail L2 for four of six models (0.0, 4.8, 1.1, 0.2) but higher on Retail L3 for five of six. The ranking also changes by model (CH 69.8 vs Prime 63.7 on GLM-5.3). On top of that, B.2 says Prime runs as a code agent with a stronger runtime than the ReAct loop, SkillOpt gets window segmentation through its scheduler and about 0.85 extra replay executions per ticket, and Mem0 gets an extra post-outcome memory turn. Prime is also missing for Opus 5 and GPT-5.6 Terra (omitted for cost, l.304). B.2 itself concludes that comparisons characterize complete systems and do not isolate harness effects (l.808 to 809), but the main text (l.319, l.453 to 454) states harness rankings and attributes exploration to the harness without this caveat, and with one run I can not tell which differences are real. This matters because a benchmark paper is mostly read for these rankings.

Questions for authors
  1. Can you run 3 seeds on a subset, for example two open models x all five harnesses x Retail L2/L3 and Banking L2, and report the spread? Is SkillOpt's near zero on Retail L2 (0.0, 4.8, 1.1, 0.2 on four of six models, while L3 is higher) stable, or is it the acceptance gate failing?
  2. Since B.2 says harnesses differ in runtime and extra executions (Prime's code runtime, SkillOpt's replay, Mem0's extra memory turn), can you either report a budget-matched comparison on a subset, or scope the harness rankings in the main text (l.319, l.453 to 454) to complete systems?
  3. "Effective exploration is critical" (l.130) rests on correlations across harnesses, so a harness-level factor could drive both answer diversity and reward. Can you add an intervention, for example forcing a weak harness to try untried answers after a failure? Otherwise I think "associated with" is the supported claim.
  4. The Banking prompt warns that "actual practice may differ from the documents" (D.6), and Opus 5 Blind gets only 52.7 to 69.1 on Banking Fully Specified. Does removing this warning change Blind and learner Fully Specified accuracy? This decides whether the "adaptation hurts" finding holds outside Retail.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dwr8 (1/2)

We thank the reviewer for the detailed reading of the protocol and appendix, and for recognizing the benchmark’s diagnostic value. We also appreciate the constructive suggestions and address each point below:

Q1: Can you run three seeds on two open models × all five harnesses × Retail L2/L3 and Banking L2? Is SkillOpt’s near-zero Retail L2 performance stable or caused by its acceptance gate?

A1: Thanks for the questions!

(1) Repeatability. We ran two repeats with GLM-5.3 Flash for all five harnesses on Retail L2, Retail L3, and Banking L2. The table reports held-out Hidden accuracy as mean (sample SD) across the original run and two reruns (Table 22\textcolor{blue}{\text{Table 22}} in Appendix D.13\textcolor{blue}{\text{Appendix D.13}}).

HarnessRetail L2Retail L3Banking L2
RAG13.7 (2.0)14.0 (0.5)55.0 (5.4)
Mem04.9 (0.4)14.1 (0.7)50.2 (9.1)
SkillOpt18.9 (11.1)22.7 (0.7)34.4 (1.9)
CH32.3 (3.7)28.4 (2.3)52.9 (5.2)
Prime26.0 (9.7)23.6 (3.2)76.2 (7.2)

Most configurations were stable across runs, with larger variation concentrated in a few cases. SkillOpt was more variable on Retail L2 than in the other settings. CH consistently led Retail L3 and Prime consistently led Banking L2. The lead changed on Retail L2, but the reruns support the same overall findings. We plan to extend the repeat runs to more models and settings to better characterize run-to-run variability.

(2) SkillOpt on Retail L2. We audited the GLM-5.3 and Kimi K3 runs to see whether the optimizer proposed correctly scoped rules and whether the validation gate accepted them (Table 17\textcolor{blue}{\text{Table 17}} in Appendix D.10\textcolor{blue}{\text{Appendix D.10}}).

ModelRetail tierHidden (%)Accepted / stepsCorrectly scoped / steps
GLM-5.3L20.04/711/71
GLM-5.3L34.95/566/56
Kimi K3L24.88/712/71
Kimi K3L337.78/5613/56

Correctly scoped candidates were rarer on L2 for both models. Gate decisions matched the logs. In Case 8\textcolor{blue}{\text{Case 8}} of Appendix D.10\textcolor{blue}{\text{Appendix D.10}}, the candidate update contained a correct gift-card rule and a separate incorrect rule. The validation batches contained no tasks that tested the gift-card rule. The candidate tied the current skill, and the gate rejected it because it required a higher validation score. Task mix also explained part of the gap. Applying the Retail L2 task mix to L3 scores gives the following results (Table 18\textcolor{blue}{\text{Table 18}} in Appendix D.10\textcolor{blue}{\text{Appendix D.10}}).

ModelL2 observed (%)L3 observed (%)L3 reweighted to L2 (%)
GLM-5.30.04.91.1
Kimi K34.837.718.2
Opus 51.117.07.1
GPT-5.6 Terra0.226.96.7

Overall, most configurations were stable across reruns. The SkillOpt audit highlights two issues on Retail L2: correctly scoped candidates were rare, and gaps in validation coverage could leave a useful rule unrewarded. For more details, see Appendix D.13\textcolor{blue}{\text{Appendix D.13}}, Table 22\textcolor{blue}{\text{Table 22}} (p. 36), for the repeats and Appendix D.10\textcolor{blue}{\text{Appendix D.10}}, Tables 17–18\textcolor{blue}{\text{Tables 17–18}} and Case 8\textcolor{blue}{\text{Case 8}} (p. 33), for the SkillOpt analysis.


Q2: Can you add a budget-matched subset or limit the main-text rankings to complete systems, given runtime and extra-execution differences?

A2: Thanks for raising this point. We agree that matching compute budgets is important for isolating the effects of the learning mechanisms. We matched the task streams, held-out sets, scoring, and per-task tool-call and turn limits. Matching the remaining learning-side budgets by removing method-specific executions would change the original update procedures, so the comparison would evaluate modified variants. For example, SkillOpt uses roughly 0.85 validation replays per served ticket to select updates, and Mem0 uses an extra post-outcome turn to write memories.

We therefore compare complete systems on performance and actual cost, including learning overhead. The following numbers from Figure 4\textcolor{blue}{\text{Figure 4}} aggregate serving and test results across the four open-weight models.

HarnessHidden rewardAPI cost per task (USD)
SkillOpt35.50.030
Mem036.90.049
RAG42.50.073
CH55.10.083
Prime59.00.195

Prime has higher Hidden reward and higher cost than CH, illustrating the end-to-end trade-off. For more details, see Sections 4.2 and 5.3\textcolor{blue}{\text{Sections 4.2 and 5.3}} for the scope of the rankings and exploration findings, Section 4.3\textcolor{blue}{\text{Section 4.3}} and Figure 4\textcolor{blue}{\text{Figure 4}} for the comparison, and Appendices B.2 and D.4\textcolor{blue}{\text{Appendices B.2 and D.4}} for system configurations and the cost breakdown.

AuthorsOpenReview ↗

Response to Reviewer dwr8 (2/2)

Q3: Can you force a weak harness to try untried answers after failure? Otherwise “associated with” is the supported claim.

A3: In order to test whether exploring an alternative and learning from its outcome improves held-out performance, we gave CH one chance to try a different decision after each failed serving task on Banking L2, with GLM-5.3 Flash as both actor and Refiner. During serving, only the first attempt counted toward that task’s score. The Refiner received both attempts and their outcomes, with the retry and its feedback used only for learning. At held-out test time, the actor had exactly one attempt per task from the frozen learned state.

The actor changed its decision on 98% of retries and answered correctly on 72%. The table summarizes held-out results from Table 21\textcolor{blue}{\text{Table 21}} in Appendix D.12\textcolor{blue}{\text{Appendix D.12}}.

CH configurationHeld-out Hidden (%)Held-out FS (%)
Original run47.376.6
Exploratory retry during serving91.999.0
Gain (percentage points)+44.6+22.4

The results show that an exploratory retry with feedback can improve later performance in this CH setting. We describe the cross-system relationship as an association. For more details, see Section 5.3\textcolor{blue}{\text{Section 5.3}} and Appendix D.12\textcolor{blue}{\text{Appendix D.12}}, Table 21\textcolor{blue}{\text{Table 21}}.


Q4: Does removing the warning that practice may differ from documents change Blind and learner Fully Specified accuracy?

A4: Thanks for flagging this. In our early experiment, we explicitly asked Opus 5 to follow the provided instructions. Its Blind accuracy on Fully Specified tasks was then comparable to that of the other models. For the reported benchmark, we restored the original prompt, including the warning, so that all models and harnesses received the same task instructions within each setting.

To understand why Opus 5’s Blind accuracy was lower on Fully Specified tasks, we examined its decision traces under the original prompt. We found that it sometimes inferred undocumented restrictions without task-specific evidence, even when the written rules were sufficient. It then acted on those assumptions and got tasks wrong that the written rules would have resolved. The early experiment therefore shows that prompt framing can affect Blind performance. The effect of deleting just the warning on Blind and learner accuracy remains unmeasured.


Thanks again for your careful review and constructive suggestions. We have incorporated the additional results and clarifications into the revised paper. We hope these changes address your concerns and would be happy to answer any further questions.

Reviewer_dwr8OpenReview ↗

Thank you for your detailed rebuttal

First, to comply with the PC's policy reminder: I want to clarify that I used claude to help polish my review based on my personal notes but the complete decisions were made only by myself.

I read your rebuttal and satisfied with your response. Your responses addressed all my concerns. I increased my score from 3 to 4. Wish you good luck :)

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for reading our rebuttal and increasing your rating! We’re glad our responses addressed your concerns, and we appreciate your constructive feedback.

Reviewer_sPewOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ServeLearnBench, a benchmark designed to evaluate whether large language model (LLM) agents can learn from serving experience and adapt to evolving environments. The paper formalizes the EESD, which requires agents to infer, apply, and update hidden environment knowledge from historical interactions and outcome feedback without direct access to the underlying rules. Based on this formulation, the authors construct a benchmark spanning three domains with 53 environment windows and 7718 tasks. The experiments evaluate 5 continual-learning harnesses across 6 LLMs.

Critical strengths

S1. The paper is well-structured and easy to follow.

S2. The paper is well-motivated, addressing the limitations of existing benchmarks by evaluating agents' ability to learn from experience in hidden and evolving environments.

S3. The experimental evaluation is comprehensive, covering 5 continual-learning harnesses across 6 LLMs.

Critical weaknesses

C1. Although this paper compares different learning harnesses using the same tasks and base models, the harnesses differ like execution environments, update frequencies. So the observed performance differences may be influenced by system-level configurations rather than being solely attributable to the continual-learning mechanisms themselves.

C2. The paper identifies a substantial gap between task execution and learning from experience. However, the evaluated methods update external state while keeping model parameters fixed. It is unclear whether learning failures stem from base-model limitations or the harnesses’ inability to effectively use experience.

C3. Merely reporting this phenomenon of the degradation of correct behavior offers limited novelty. Similar effects have been observed in SFT and RL. The paper should provide deeper insight into the underlying failure mechanisms. In particular, the analysis provides limited insight into why continual adaptation can degrade previously correct behavior and why some harnesses struggle to learn from and effectively reuse experience.

Questions for authors

Please mainly respond to C1, C2, and C3.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sPew (1/3)

We thank the reviewer for recognizing the motivation, clarity, and breadth of the evaluation. We also appreciate the constructive suggestions and address each point below:

Q1: Performance differences may reflect execution environments and update frequencies rather than continual-learning mechanisms alone.

A1: Thanks for raising this point. We agree that matching execution environments and update schedules is important for isolating the effects of the learning mechanisms. We matched as many conditions as the implementations allowed. All systems use the same task streams, held-out sets, and scoring, with the same base model within each model block. RAG, Mem0, SkillOpt, and CH also share a ReAct acting loop. However, fully matching the remaining conditions would change core parts of the original methods, so the comparison would evaluate modified variants. For example, moving Prime to ReAct would replace its coding runtime, removing SkillOpt’s validation replay would change how it selects updates, and imposing a common schedule would change when batch and streaming learners can use feedback.

We therefore compare each complete system end to end and report performance alongside the actual cost, including learning overhead. In Figure 4\textcolor{blue}{\text{Figure 4}}, Prime’s average Hidden reward across serving and test is 59.0 at USD 0.195 per task, while CH’s is 55.1 at USD 0.083. Sections 4.2 and 5.3\textcolor{blue}{\text{Sections 4.2 and 5.3}} now make clear that the observed differences reflect the full configurations, including their runtimes and update procedures. For more details, see Appendix B.2\textcolor{blue}{\text{Appendix B.2}} for those configurations and Appendix D.4\textcolor{blue}{\text{Appendix D.4}} for the cost breakdown.

AuthorsOpenReview ↗

Response to Reviewer sPew (2/3)

Q2: Because model parameters are fixed, is the learning failure caused by the base model or the harness?

A2: We conducted two additional analyses to examine how both the harness and the learning model contribute to adaptation.

(1) Quantitative analysis with the model or harness held fixed. We analyzed all 28 evaluated model–harness combinations. The table shows mean held-out Hidden scores across nine settings (Table 13\textcolor{blue}{\text{Table 13}} in Appendix D.7\textcolor{blue}{\text{Appendix D.7}}). Each row holds the model fixed, and each column holds the harness fixed. The dashes mark Prime runs omitted due to runtime and API cost.

ModelRAGMem0SkillOptCHPrime
Kimi K345.037.751.157.861.7
GLM-5.345.943.739.569.863.7
DeepSeek V4.1 Flash44.333.533.253.360.3
GLM-5.3 Flash38.238.435.449.761.1
Opus 540.947.850.473.4-
GPT-5.6 Terra34.931.729.655.8-

Among the four models with complete coverage, changing the harness within a model spans 24.0–30.3 points. Changing the model within CH spans 20.1 points, compared with 3.4 points within Prime. Across these four models, the spread across harness averages is 23.4 points, compared with about 8 across model averages. These comparisons show that performance depends on both the model and the harness configuration. For more details, see Figure 3\textcolor{blue}{\text{Figure 3}} and Table 13\textcolor{blue}{\text{Table 13}} in Appendix D.7\textcolor{blue}{\text{Appendix D.7}} (p. 29).

Experience reuse with the model held fixed. In Banking, we matched eight policy states across all five harnesses using GLM-5.3 Flash. For each harness, we measured accuracy on the next five encounters after its own first success (Table 14\textcolor{blue}{\text{Table 14}} in Appendix D.7\textcolor{blue}{\text{Appendix D.7}}).

ModelMatched policy statesRAGMem0SkillOptCHPrime
GLM-5.3 Flash850.0%50.0%50.0%65.0%82.5%

With the same model and policy states, RAG correctly handled 50.0% of subsequent encounters and Prime handled 82.5%. The gap shows that the evaluated systems differ in how reliably they reuse a success. For more details, see Table 14\textcolor{blue}{\text{Table 14}} in Appendix D.7\textcolor{blue}{\text{Appendix D.7}} (p. 29).

(2) Learner-swap ablation with the actor held fixed. On Banking L2, we held the GLM-5.3 Flash actor and the harness fixed and changed only the learner from GLM-5.3 Flash to GLM-5.3. We swapped the Refiner in CH and the optimizer in SkillOpt (Table 19\textcolor{blue}{\text{Table 19}} in Appendix D.11\textcolor{blue}{\text{Appendix D.11}}).

HarnessHidden: Flash → GLM-5.3 (%)FS: Flash → GLM-5.3 (%)
CH47.3 → 80.676.6 → 93.0
SkillOpt34.9 → 40.970.1 → 87.6

With the same actor, CH gained 33.3 points on Hidden tasks and improved in all five windows. SkillOpt gained 6.0 points on Hidden tasks and 17.5 points on Fully Specified (FS) tasks. A stronger learning model can improve both adaptation to hidden rules and accuracy on fully specified tasks even with the actor fixed. In Banking L2, the Hidden gain from the same learner swap was larger in CH than in SkillOpt. Together, the results show that both the harness and the learning model matter for adaptation. For more details, see Tables 13–14\textcolor{blue}{\text{Tables 13–14}} in Appendix D.7\textcolor{blue}{\text{Appendix D.7}} and Tables 19, 20\textcolor{blue}{\text{Tables 19, 20}} in Appendix D.11\textcolor{blue}{\text{Appendix D.11}} (p. 34).

AuthorsOpenReview ↗

Response to Reviewer sPew (3/3)

Q3: Behavioral degradation has precedents in SFT/RL; provide deeper insight into degradation and failure to reuse experience.

A3: Thanks for raising this point. We agree that degradation of previously correct behavior has been observed in SFT and RL. Our contribution on this point is to trace how updates made by the evolution algorithms in continual-learning harnesses can degrade previously correct behavior.

To examine behavioral degradation and failures of experience reuse in more depth, we audited archived GLM-5.3 Flash trajectories on Banking L2. For FS degradation, we paired three CH runs with Blind by task ID and used verifier scores to select all 138 pairs where Blind succeeded and CH failed. We used Claude Code with Opus 5.5 to compare the actor’s decision and saved learned state with the applicable policy and cite the excerpts supporting each label. We checked those excerpts against the saved files. For experience reuse, we matched serving tasks by policy state and decision. We traced the first success directly in RAG’s index and retrieval logs. For failed CH follow-ups, we used Claude Code with Opus 5.5 in read-only mode to inspect the actor traces and Refiner updates.

FS degradation. Of the 138 task–run pairs where Blind succeeded and CH failed, the audit classified 126 (91.3%) as overgeneralization (Appendix D.8\textcolor{blue}{\text{Appendix D.8}}). In Case 1\textcolor{blue}{\text{Case 1}} of Appendix D.8\textcolor{blue}{\text{Appendix D.8}}, a CH rule learned from overseas electronics failures dropped the electronics condition and led the actor to escalate an eligible USD 492.99 travel charge. Blind correctly approved it.

Experience reuse. RAG retrieved the first successful exemplar in 26 of 162 eligible follow-ups (Table 16\textcolor{blue}{\text{Table 16}} in Appendix D.9\textcolor{blue}{\text{Appendix D.9}}). Accuracy was 26/26 when the exemplar was retrieved and 84/136 when it was omitted. In Case 7\textcolor{blue}{\text{Case 7}} of Appendix D.9\textcolor{blue}{\text{Appendix D.9}}, dropping a CH rule’s jewelry condition made a later travel outcome appear to contradict the rule, and the Refiner deleted the rule along with its supporting success record. Together, these traces show how overgeneralized rules can degrade previously correct behavior and how retrieval or consolidation can prevent reuse of successful experience. For more details, see Appendix D.8\textcolor{blue}{\text{Appendix D.8}}, Table 15\textcolor{blue}{\text{Table 15}} and Cases 1–5\textcolor{blue}{\text{Cases 1–5}} (pp. 30–31), Appendix D.9\textcolor{blue}{\text{Appendix D.9}}, Table 16\textcolor{blue}{\text{Table 16}} and Cases 6–7\textcolor{blue}{\text{Cases 6–7}} (p. 32), and Appendix D.7\textcolor{blue}{\text{Appendix D.7}}, Table 14\textcolor{blue}{\text{Table 14}} (p. 29).


Thanks again for your careful review and constructive suggestions. We have incorporated the additional results and clarifications into the revised paper. We hope these changes address your concerns and would be happy to answer any further questions.

Reviewer_sPewOpenReview ↗

ACK of Rebuttal

The authors have revised the paper and addressed several key concerns well. One more suggestion is to move some key results from appendix to main text. I will increase my score to the positive.

Reviewer_dgXROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ServeLearnBench evaluates agents that must infer hidden environment knowledge from serving interactions and outcome feedback as policies change over time. It contains 53 environment windows and 7,718 tasks across retail support, banking, and sales-pitch generation. Evaluations of five learning harnesses across six models examine adaptation performance, computational cost, degradation on fully specified tasks, and associations between exploration and learning.

Critical strengths
  1. The benchmark combines hidden knowledge, changing policies, and held-out evaluation in a clearly specified streaming protocol. Blind and Oracle references, together with Hidden-Dependent and Fully Specified slices, help distinguish knowledge-acquisition failures from underlying task limitations and collateral degradation.

  2. The evaluation provides useful evidence beyond aggregate accuracy, including learning speed, policy withdrawal, post-success knowledge reuse, and serving-versus-learning costs. These analyses expose practical limitations of existing systems and make the benchmark informative for future continual-learning research.

Critical weaknesses
  1. Harness comparisons include substantial differences in acting runtime, learning computation, replay access, and available information. Appendix B appropriately discloses these differences, including Prime’s stronger execution runtime and SkillOpt’s access to window segmentation through scheduling. However, the resulting rankings characterize complete systems rather than isolate continual-learning mechanisms. Some interpretations therefore need to remain explicitly limited to the evaluated configurations.

  2. Each evaluation cell represents one run, and some task instances share a small number of effective attribute combinations (Appendix B.4). The apparent scale should therefore not be interpreted as independent replication. Similarly, the exploration analysis is observational: diversity of answers can covary with runtime, compute, and overall system competence. The correlation is informative but does not establish exploration as the causal bottleneck.

Questions for authors
  1. How stable are the main rankings and exploration–performance associations across repeated runs or alternative task orders?

  2. Can you provide a controlled comparison holding the acting runtime and feedback access fixed, or further distinguish conclusions about complete systems from conclusions about learning mechanisms?

4 comments
AuthorsOpenReview ↗

Response to Reviewer dgXR (1/2)

We thank the reviewer for recognizing the streaming protocol, the diagnostic controls, and the analyses beyond aggregate accuracy. We also appreciate the constructive suggestions and address each point below:

Q1: How stable are the main rankings and exploration–performance associations across repeated runs, and how should task counts be interpreted when instances share attribute combinations?

A1: Thanks for the questions! We are happy to provide more details on the repeated experiements.

(1) Repeatability across runs. To assess ranking stability, we ran two additional runs with GLM-5.3 Flash for each of the five harnesses on Retail L2, Retail L3, and Banking L2. Each fresh run counts as one replication.

Held-out Hidden accuracy (%), mean (sample SD) across the original run and two reruns (Table 22\textcolor{blue}{\text{Table 22}} in Appendix D.13\textcolor{blue}{\text{Appendix D.13}}).

HarnessRetail L2Retail L3Banking L2
RAG13.7 (2.0)14.0 (0.5)55.0 (5.4)
Mem04.9 (0.4)14.1 (0.7)50.2 (9.1)
SkillOpt18.9 (11.1)22.7 (0.7)34.4 (1.9)
CH32.3 (3.7)28.4 (2.3)52.9 (5.2)
Prime26.0 (9.7)23.6 (3.2)76.2 (7.2)

Most configurations were stable across runs, with larger variation concentrated in a few cases. CH consistently led Retail L3 and Prime consistently led Banking L2, while the lead changed on Retail L2. The reruns support the same overall findings. We plan to extend the repeat runs to more models and settings to better characterize run-to-run variability. For more details, see Table 22\textcolor{blue}{\text{Table 22}} in Appendix D.13\textcolor{blue}{\text{Appendix D.13}} (p. 36).

(2) Exploration–performance correlation. Using the aggregation from Figure 7\textcolor{blue}{\text{Figure 7}}, with one point per harness, Spearman ρ on these same three settings was 0.60 for the original runs, 0.70 for the pooled reruns, and 0.60 for all runs pooled. The original six settings gave ρ = 0.90 (Table 23\textcolor{blue}{\text{Table 23}} in Appendix D.13\textcolor{blue}{\text{Appendix D.13}}). Overall, the reruns show that most configurations were stable and that the exploration–performance association remained positive. For more details, see Tables 22–23\textcolor{blue}{\text{Tables 22–23}} in Appendix D.13\textcolor{blue}{\text{Appendix D.13}} (p. 36).


Q2: Can you hold the acting runtime and feedback access fixed, or further distinguish complete-system conclusions from learning-mechanism conclusions?

A2: Thanks for raising this point. We agree that matching the acting runtime, feedback access, and compute budget is important for isolating the effects of the learning mechanisms. We matched the task streams, held-out sets, scoring, and base model within each model block. RAG, Mem0, SkillOpt, and CH also share a ReAct acting loop. However, fully matching how systems receive and use feedback would change core parts of the original methods, so the comparison would evaluate modified variants. For example, CH sends the serving score to a separate Refiner, while Prime uses it in the same session. Unifying those feedback paths would change how the systems update.

We therefore compare each complete system end to end and report performance alongside the actual cost, including learning overhead. In Figure 4\textcolor{blue}{\text{Figure 4}}, Prime’s average Hidden reward across serving and test is 59.0 at USD 0.195 per task, while CH’s is 55.1 at USD 0.083. Prime achieves a higher Hidden reward at a higher per-task cost, illustrating the cost–performance trade-off between the two complete systems. Sections 4.2 and 5.3\textcolor{blue}{\text{Sections 4.2 and 5.3}} now scope the rankings and exploration findings to the evaluated systems. Appendix B.2\textcolor{blue}{\text{Appendix B.2}} describes the operating conditions, and Appendix D.4\textcolor{blue}{\text{Appendix D.4}} breaks down the costs. For more details, see Section 4.3\textcolor{blue}{\text{Section 4.3}} for the cost–performance comparison.

AuthorsOpenReview ↗

Response to Reviewer dgXR (2/2)

Q3: Answer diversity can covary with runtime, compute, and overall system competence. Does the observed correlation establish exploration as a causal bottleneck?

A3: Across harnesses, runtime and compute can affect both answer diversity and reward. Prime builds on CH’s learning design and uses a coding/RLM runtime, while RAG, Mem0, SkillOpt, and CH share the ReAct acting loop. Matching runtime, replay, and update schedules would change core parts of the methods, as discussed in A2. We therefore report the performance and overhead of each complete system and interpret the cross-harness relationship as an association (Figure 4\textcolor{blue}{\text{Figure 4}} and Appendix B.2\textcolor{blue}{\text{Appendix B.2}}).

Additionally, in order to test how much an extra opportunity to explore can help within the same harness, we gave CH one retry after a failed serving task on Banking L2. Both the actor and Refiner used GLM-5.3 Flash. During serving, only the first attempt counted toward that task’s score. The Refiner received both attempts and their outcomes, with the retry and its feedback used only for learning. At held-out test time, the actor had exactly one attempt per task from the frozen learned state. Held-out Hidden accuracy rose from 47.3% to 91.9% (+44.6 points) and Fully Specified accuracy from 76.6% to 99.0% (+22.4 points) (Table 21\textcolor{blue}{\text{Table 21}} in Appendix D.12\textcolor{blue}{\text{Appendix D.12}}). This shows that an exploratory retry with feedback can substantially improve subsequent performance in this CH setting. For more details, see Appendix D.12\textcolor{blue}{\text{Appendix D.12}}, Table 21\textcolor{blue}{\text{Table 21}} (p. 35).


Thanks again for your careful review and constructive suggestions. We have incorporated the additional results and clarifications into the revised paper. We hope these changes address your concerns and would be happy to answer any further questions.

Thank you for the response. Most of my concerns have been resolved, and i will keep my positive score

AuthorsOpenReview ↗

Thank you for your follow-up

Thank you for reading our rebuttal and for your continued support. We’re glad our responses addressed most of your concerns, and we appreciate your constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

Dear reviewers,

We thank all the reviewers for their thoughtful reviews and constructive feedback, and for recognizing the value of evaluating agents in hidden and evolving environments. We have prepared point-by-point responses for each reviewer and revised the manuscript in response to your suggestions. Below we summarize the main changes that address the reviewers’ common concerns.

Repeatability. Most configurations were stable across the three GLM-5.3 Flash runs, with larger variation concentrated in a few cases. CH consistently led Retail L3 and Prime consistently led Banking L2, while the lead changed on Retail L2. The reruns support the same overall findings (Appendix D.13\textcolor{blue}{\text{Appendix D.13}}, Table 22\textcolor{blue}{\text{Table 22}} (p. 36)). Across the same three settings, the association between exploration and held-out performance remained positive in the reruns (Appendix D.13\textcolor{blue}{\text{Appendix D.13}}, Table 23\textcolor{blue}{\text{Table 23}}, p. 36).

System comparisons and learning models. We compare end-to-end systems and include learning overhead in the cost–performance analysis (Figures 1 and 4\textcolor{blue}{\text{Figures 1 and 4}} and Appendix D.4\textcolor{blue}{\text{Appendix D.4}}). Aligning update schedules and replay budgets would change how the systems learn (Appendix B.2\textcolor{blue}{\text{Appendix B.2}}). We clarified the scope of the rankings in the main text. With the Flash actor fixed, replacing CH’s Refiner raised Banking L2 Hidden accuracy from 47.3% to 80.6% and FS accuracy from 76.6% to 93.0% (Appendix D.11\textcolor{blue}{\text{Appendix D.11}}, Tables 19–20\textcolor{blue}{\text{Tables 19–20}}, p. 34). With GLM-5.3 Flash and the same eight Banking policy states, accuracy over the five encounters after each harness’s first success varied across harnesses (Appendix D.7\textcolor{blue}{\text{Appendix D.7}}, Table 14\textcolor{blue}{\text{Table 14}}, p. 29).

Failure mechanisms. In the GLM-5.3 Flash Banking L2 audit, overgeneralization was the most frequent category among FS cases where Blind succeeded and CH failed. The traces also document how successful experience was missed or discarded (Appendices D.8–D.9\textcolor{blue}{\text{Appendices D.8–D.9}}, pp. 30–32). The SkillOpt audit finds few correctly scoped candidates and shows how validation coverage can block an update that contains a useful rule (Appendix D.10\textcolor{blue}{\text{Appendix D.10}}, Tables 17–18\textcolor{blue}{\text{Tables 17–18}} and Case 8\textcolor{blue}{\text{Case 8}}, p. 33).

Exploratory retries. Giving CH one retry after a serving failure raised held-out Hidden accuracy from 47.3% to 91.9% and FS accuracy from 76.6% to 99.0% on Banking L2. Only first attempts counted toward serving scores. The retry and its outcome went only to the Refiner as learning evidence. Each held-out task allowed exactly one attempt from frozen state (Appendix D.12\textcolor{blue}{\text{Appendix D.12}}, Table 21\textcolor{blue}{\text{Table 21}}, p. 35).


Paper Revision Summary

All changes in the revised paper are highlighted in blue.

1. Additional evaluation and analysis

  1. Added Appendix D.7\textcolor{blue}{\text{Appendix D.7}}. Compared performance with the model or harness held fixed and measured reuse after first success on matched policy states. (sPew Q2 and Q3)

  2. Added Appendices D.8–D.9\textcolor{blue}{\text{Appendices D.8–D.9}}. Audited why Fully Specified performance degrades and why successful experiences go unused, with quantitative results and trace excerpts. (sPew Q3)

  3. Added Appendix D.10\textcolor{blue}{\text{Appendix D.10}}. Examined SkillOpt’s low Retail L2 scores through candidate quality, validation coverage, and task composition. (dwr8 Q1)

  4. Added Appendix D.11\textcolor{blue}{\text{Appendix D.11}}. Compared two learning models with the actor fixed in CH and SkillOpt. (sPew Q2)

  5. Added Appendix D.12\textcolor{blue}{\text{Appendix D.12}}. Tested an exploratory retry during serving and evaluated the learned state on single-attempt held-out tasks. (dgXR Q3, dwr8 Q3)

  6. Added Appendix D.13\textcolor{blue}{\text{Appendix D.13}}. Reported repeated runs for all five harnesses and recomputed the exploration–performance correlation on matched settings. (dgXR Q1, dwr8 Q1)

2. Writing and clarifications

  1. Updated the Introduction\textcolor{blue}{\text{Introduction}}, Sections 4.2, 5.1, and 5.3\textcolor{blue}{\text{Sections 4.2, 5.1, and 5.3}}, and Appendices B.2 and D.3\textcolor{blue}{\text{Appendices B.2 and D.3}}. Clarified the scope of the performance and learning-speed rankings, explained differences in runtime and feedback access, and described the exploration results (across systems and within a model) as associations. (dgXR Q2 and Q3, sPew Q1, dwr8 Q2 and Q3)

  2. Updated Appendix D.6\textcolor{blue}{\text{Appendix D.6}}. Added the Opus 5 instruction-following prompt observation and clarified how to interpret the Banking Fully Specified results. (dwr8 Q4)


We thank the reviewers again for their constructive feedback. We hope the responses and revised manuscript address your concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.