acceptodds
Under review as a conference paper at ICLR 2027

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

Abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering () harness and train one model to act, reflect, and curate task-local playbooks. Our method, , evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across different agentic benchmarks, shows faster convergence and higher converged performance than existing baselines, including outcome-only . The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness, Self improvement, Reinforcement Learning, Agentic RL, LLMs, RLVR

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_FGHFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Experience-Handoff Reinforcement Learning, which jointly trains a shared model in the Actor, Reflector, and Curator roles within a fixed ACE harness. ACE-RL evaluates candidate playbooks through their downstream task rewards and uses a sparse sampling tree to construct role-specific comparison groups. The method is evaluated on Mastermind, MBPP, AppWorld, and MiniSWEBench against configurations including actor-only RL and joint PPO. The paper also analyzes inference scaling, computational costs, and the contents of the resulting playbooks.

Critical strengths
  1. The paper addresses an important problem in agentic RL by extending optimization beyond immediate task execution to the generation and reuse of experience.
  2. ACE-RL provides a framework for jointly training acting, reflection, and memory curation using verifiable downstream rewards.
  3. The method achieves competitive performance across diverse agentic benchmarks, supported by additional analyses of inference scaling, training behavior, and computational costs.
Critical weaknesses
  1. ACE-RL jointly trains the Actor, Reflector, and Curator, but does not isolate the benefit of optimizing each role. The actor-only baselines are trained without experience handoff, making them insufficient controls for determining whether directly optimizing Reflector and Curator is necessary, raising the concern of much of the improvement may come from Actor adaptation rather than effective experience transfer.
  2. ACE-RL introduces substantial computational overhead from candidate-evaluation rollouts. It generates approximately 2.6–3.3× as many tokens as GRPO per training step, while only 55–60% of these tokens contribute to policy updates. The paper lacks compute-matched comparisons or a systematic analysis of performance gains relative to total training and inference costs, leaving the practical efficiency of the proposed method unclear.
  3. ACE-RL evaluates each candidate playbook using only one downstream Actor rollout, whose reward reflect more of sampling randomness rather than the actual quality of the generated experience, making credit assignment largely noisy. Selecting the highest-reward candidate among noisy evaluations introduce more noise and the risk of a decrease in exploration diversity.
Questions for authors
  1. Appendix G.1 reports that all 300 Mastermind tips are identical template strings. What causes this behavior, and does it emerge gradually during training? Could it be related to the template examples included in the Curator prompts? Does it implies entropy collapse during training?
  2. How does ACE-RL's RSI@K compare with Pass@K using the same trained Actor without experience handoff? In particular, given the strong first-attempt performance on MBPP and the weaker inference scaling on AppWorld, how much of the improvement can be attributed to actually utilizing experience from previous attempts?
  3. During training, ACE-RL selects the best candidate playbook based on additional downstream rollouts. However, Section 4.2 states that no such evaluation rollouts are performed at inference. How are playbook updates accepted or rejected during inference, and how does this difference affect the learned experience-handoff behavior?
  4. ACE-RL evaluates each candidate playbook using only one downstream Actor rollout, which may introduce substantial noise into the Reflector and Curator advantages. Have the authors investigated how the number of evaluation rollouts per candidate affects training stability, final performance, and computational cost? Additionally, does the unequal number of candidate evaluations across different reflections affect the reliability of credit assignment?
2 comments

Thank you for the detailed rebuttal. I appreciate the improved compute matching, which largely addresses my concerns about training efficiency. Although the gains from experience handoff remain inconsistent across benchmarks, I find the new evidence sufficient to strengthen the paper's empirical support. I will therefore raise my rating to 3 (Weak Accept).

Thanks for reading our rebuttal, glad to hear that our revision was helpful!

Reviewer_ktx6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Experience-Handoff RL, which builds upon Agentic Context Engineering (ACE), and use GRPO to jointly optimize the rollouts across the three stages (Act, Reflect, Curate) in the ACE harness. The authors did experiments with a few agentic tasks, e.g. AppWorld, 100 instances of SWE-Bench, and show that compared to only training on the task rollout alone underperform joint training on the three stages' rollouts, under the assumption that 3 reflection attempts are allowed in test time.

Critical strengths
  • Good research problem: The idea of co-evolving memory harness and policy makes sense and has recently been an area that the research community show a lot of interests in.
  • The proposed method makes sense and is well-motivated: Jointly optimizing model all parts of the harness and test model's performance when reflection allowed is by itself a setting that has research value, and the proposed method show empirical gain under this setting.
Critical weaknesses

Overall, the key weakness I see in the paper is inaccuracy of the conclusion + claims made in the paper. To me this comes from multiple factors:

  • Unfair evaluation / baseline comparison: It's important to note that RSI @ k for k > 1 assumes the model would have access to the reward from previous attempts. The paper directly compares RSI @ k from ACE-RL with Pass @ k from GRPO in many of the results (e.g. Table 2 and Figure 5). But IMO the only conclusion we might be able to draw from this comparison is that model learns to reflect better. But it looks like the paper is using this as evidence to claim ACE-RL is overall better than GRPO/PPO, to quote -- Test rewards favor it over actor-only RL on all four benchmarks in conclusion. Compared to pass @ k eval, RSI @ k (1) assumes access to reward, (2) uses a lot more test-time compute, so it wouldn't be a fair comparison. In fact, Figure 5(a) shows side by side comparison of RSI @ 3 between GRPO and ACE-RL, and it is not clear ACE-RL has a clear edge under similar compute budget (i.e. effective batch size) per training step.

  • Missing pass @ k evaluation for ACE-RL: The authors need to show pass @ k evaluation for ACE-RL in order for us to understand whether gain comes from retries alone.

  • Inaccurate details / terminologies esp. relates to RL & existing work: e.g. the paper contains hand-wavy description of PPO vs. GRPO. Note that the biggest and frankly only difference between modern PPO vs GRPO implementation is how advantage estimation or importance sampling is done (through a separate critic model for learning to estimate token-level critic value / advantage). Otherwise both depend on a reward function providing "verifiable reward". But the paper contain very noticeable inaccurate accounts for how PPO works, which is not helping the presentation of the work in this paper. e.g. In table 1, the "learned critic" vs. "verifiable reward" distinction doesn't make sense at all, these are things that can co-exist. Also the word "harness" is used in accurately in the paper -- you are not optimizing the "harness" itself, you are optimizing on the rollouts coming from the harness, where you are not changing the harness, you are steer the model to execute harness so that it leads to "better" expected reward.

Questions for authors

See above

Reviewer_RpdgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training a model to write experience that subsequent attempts on the same task inherit. Within a fixed ACE harness, one shared model acts, reflects on its trajectory and edits a playbook. ACE-RL scores candidate playbooks with fresh actor executions and uses a sparse sampling tree to construct comparison groups sharing an input for each role, jointly optimizing their behavior. The authors report higher rewards than plain GRPO deployed in the same harness on Mastermind, MBPP, AppWorld and MiniSWEBench, alongside a chain alternative, cost measurements and qualitative playbook analysis. The results support the overall feasibility of training this harness, but do not yet isolate informative handoff or establish the practical advantage of the specific grouping mechanism under matched resources.

Critical strengths
  1. The method turns reflection and editable playbooks into training objects that can be executed and rewarded through task outcomes, jointly training all three roles in one shared model. In Table 2, ACE-RL has higher mean reward on all four tasks than plain GRPO deployed in the same RSI@3 harness. This provides concrete evidence that training for a fixed harness is feasible.

  2. The appendices are transparent about important limitations: they directly compare a chain alternative, explain the local gradient approximations and generation costs, and report the absence of informative handoff on the two small model benchmarks. This allows system performance and mechanism explanations to be assessed separately, and the negative findings are useful in their own right.

Critical weaknesses

Informative experience transfer is not isolated from actor and prompt adaptation. Plain GRPO is trained with plain prompts and only later deployed in ACE. Page 9 reports that the first Mastermind harness visit drops from vanilla 0.64 to 0.35 before any handoff, while ACE-RL already reaches 0.95 on its first MBPP visit, above GRPO's 0.86 across eight independent attempts. Appendix G.1 further reports that all 300 Mastermind tips are the same template and the 15 MBPP tips mostly restate reflection case labels, explicitly attributing these gains mainly to actor adaptation. AppWorld and MiniSWEBench playbooks contain task information, but textual diversity and two success sequences do not alone establish a causal benefit. Without original, empty and length matched generic playbook comparisons at the same checkpoint, actor prompt and budget, together with an actor-only control trained under the harness prompts, the central contribution remains difficult to attribute.

The practical increment of the specific grouping design and the efficiency claims are insufficiently established. Appendix B does include a relevant ablation, but Table 5 reports main versus chain rewards of 0.736 versus 0.746, 0.973 versus 0.973, 0.649 versus 0.580 and 0.227 versus 0.242 on Mastermind, MBPP, AppWorld and MiniSWEBench, respectively, without a consistent ordering. Each method is trained once and the intervals cover task variation only. Table 4 reports that ACE-RL generates 3.29 and 3.03 times GRPO's tokens per step on MBPP and AppWorld, so convergence by optimizer steps does not establish a matched resource advantage. Delayed-joint PPO also changes advantage estimation, filtering and actor training source, with different candidate scoring counts described on page 7 and in D.3; its failure cannot be attributed solely to critic based credit assignment. These gaps limit the mechanism contribution beyond existing downstream reward and chain training approaches.

Questions for authors
  1. Controls that could change the attribution: On at least one longer task, can the same trained checkpoint, actor prompt and budget be used to compare original, empty and length matched generic playbooks, reporting paired task differences? How does an actor-only control trained under the same harness prompts perform? This would separate the contribution of experience content from actor adaptation.

  2. Evidence that could change the mechanism and efficiency assessment: Can ACE-RL, chain ACE-RL and GRPO trained with the harness prompts be compared by total generated tokens or tool time? Do additional training runs or measured credit variance support a stable gain from the specific grouping design? If such evidence is unavailable, please scope the conclusions to the reported single runs and step budgets, rather than presenting a theoretical variance motivation as an established performance or resource advantage.

  3. Protocol and metric clarification: How does the test time acceptance rule on page 3 correspond to evaluation without additional candidate scoring rollouts on pages 7 and 39? Does delayed-joint PPO use one or eight evaluations per candidate, and what are the realized trajectory and token counts for each method? Please separate full MiniSWEBench issue resolution from mean shaped reward, which gives partial credit to unresolved diffs. MBPP correctness claims should likewise be scoped to the three exposed scored tests or supported by additional unseen tests. These targeted controls address central claims rather than requiring every possible ablation.

Reviewer_xWmtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper proposes Experience-Handoff RL (EHRL): training one LLM both to act and to write the experience its future attempts will inherit, using only task outcomes.
  • As a testbed, the authors fix the Agentic Context Engineering (ACE) harness and train one model in three roles: actor (solves the task), reflector (extracts an insight from a trajectory) and curator (edits a task-specific playbook).
  • The proposed method, ACE-RL:
    • scores each candidate playbook by running one fresh actor rollout with it and reading the task verifier, with no learned reward model;
    • credits each reflection with the mean score of the playbooks curated from it;
    • forms a separate GRPO comparison group for each role through a sparse sampling tree (four reflections of one seed trajectory, four edits of one reflection).
  • Validation: Qwen3-0.6B on Mastermind and MBPP, and Qwen3-4B-Instruct on AppWorld and MiniSWEBench, compared with PPO, GRPO and a critic-based "delayed-joint PPO" baseline.
  • ACE-RL reports the highest final test reward on all four benchmarks (Table 2). The paper also compares RSI@K (best of K sequential attempts in the harness) with pass@K (best of K independent samples).
Critical strengths
  • Well-motivated problem with a simple, verifiable signal. Training the model for the harness it is deployed in, rather than only its actor, is a timely and relevant question. Scoring a playbook by executing it gives a reward for intermediate text without a learned reward model. The results show that a harness-trained checkpoint beats GRPO deployed in the same harness at most checkpoints (29/29 on Mastermind, 36/40 on AppWorld), confirming that actor-only training does not transfer well to harnessed use.
  • Clear formulation and transparent reporting. The objective (Eq. 4–5), the sampling tree (Fig. 3) and the derivation in Appendix D are clearly presented, and all prompts are provided. The authors openly report diagnostics that qualify their own claims (Appendix B, G.1, F.5). This candor makes the paper's evidence easy to assess.
Critical weaknesses
  • The central claim, that the model learns to write useful experience, is not supported.
    • Appendix G.1 shows the learned handoff is degenerate on two of the four benchmarks:
      • On Mastermind, every tip written by the trained curator is the template string "Tip content from FAILED insights." (300 tips over 100 tasks).
      • On MBPP, the tips merely restate the reflector's case label.
    • The authors themselves conclude that gains on these benchmarks come from "an actor trained to work inside the harness rather than from the content of the handoff."
    • At matched numbers of actor episodes (Fig. 5b), ACE-RL's RSI@K is within 0.03 of GRPO's pass@K on Mastermind, below it on AppWorld (0.77 vs. 0.81), and within the task CI on MiniSWEBench.
    • The MBPP advantage is already present at K=1, before any handoff occurs. GRPO's harnessed score also drops at the first visit because of the prompt change (0.35 vs. 0.64 on Mastermind, Sec. 5.2). This points to the training-versus-deployment prompt mismatch, not learned experience, as the main driver.
    • The headline evidence therefore does not separate "learning to hand off experience" from "learning to act under the harness prompt."
  • The experimental comparisons are confounded.
    • The proposed technical contribution, per-role comparison groups, shows no measurable benefit. A chain-wise variant without per-role groups matches ACE-RL within the CI on all four benchmarks and is ahead on Mastermind and MiniSWEBench (Appendix B, Table 5).
    • What remains, rewarding intermediate text by the success of the rollout it guides, closely follows Reflect–Retry–Reward (Bensal et al., 2025) and Memory-R1.
    • The comparisons favor ACE-RL:
      • ACE-RL generates about 3× the completion tokens of GRPO per step (Table 4) but is compared at equal step counts.
      • Table 2 compares ACE-RL's RSI@3 (best of three attempts) with the baselines' pass@1.
      • GRPO is never trained on the harness prompt.
      • The critic-based baseline collapses to 0.00 on three benchmarks even though its critic's explained variance is near 1, suggesting an implementation problem rather than a meaningful comparison.
    • All results come from a single training run per method, with CIs over at most 57–100 tasks. The AppWorld and MiniSWEBench gains over GRPO in the harness fall within overlapping CIs (0.649 ± 0.125 vs. 0.561 ± 0.130; 0.227 ± 0.083 vs. 0.180 ± 0.076).
    • On MBPP, the feedback shows the same asserts that are scored (Appendix E). The paper's own example (Fig. 2b) succeeds by "return[ing] the digit for each tested position from the feedback", which suggests the reported 0.973 may partly reflect solutions fitted to the visible tests.
Questions for authors

Clarification

  1. How does ACE-RL compare with a GRPO baseline trained on the harness actor prompt (with an empty or inherited playbook) at matched generated-token or rollout budgets? This would separate harness-prompt adaptation from learned handoff.
  2. How does performance change when the ACE-RL actor is given an empty, shuffled or base-model-written playbook at evaluation? This would directly measure the value of the playbook content.
  3. Why does the Mastermind curator converge to a fixed template string? Does this indicate that the curator reward is uninformative or exploitable?
  4. On MBPP, what fraction of solved held-out tasks hard-code the expected outputs of the displayed asserts? Do the gains hold on hidden tests (e.g., MBPP+)?
  5. What causes delayed-joint PPO to collapse despite a near-perfect explained variance? Was the baseline tuned (e.g., learning rate, critic warm-up)?
  6. Can you report results over multiple training seeds, at least for AppWorld and MiniSWEBench, where the margins fall within the CI?

Suggestions

  1. Evaluating playbooks that transfer across tasks, rather than task-local ones, and larger models would test whether learned handoff matters beyond adapting to the prompt.
  2. Directly measuring the variance reduction from per-role groups would clarify when the sparse tree is worth its complexity.

Comments on the paper

1 comment

Hello reviewers, this is a quick reminder that the rebuttal period is ending soon. We highly value your reviews and it helped us improving our paper, including core methods, analysis, and structure. Please check the updated version and (or) our rebuttal to discuss if there are any concerns left. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.