- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Experience-Handoff Reinforcement Learning, which jointly trains a shared model in the Actor, Reflector, and Curator roles within a fixed ACE harness. ACE-RL evaluates candidate playbooks through their downstream task rewards and uses a sparse sampling tree to construct role-specific comparison groups. The method is evaluated on Mastermind, MBPP, AppWorld, and MiniSWEBench against configurations including actor-only RL and joint PPO. The paper also analyzes inference scaling, computational costs, and the contents of the resulting playbooks.
Critical strengths
- The paper addresses an important problem in agentic RL by extending optimization beyond immediate task execution to the generation and reuse of experience.
- ACE-RL provides a framework for jointly training acting, reflection, and memory curation using verifiable downstream rewards.
- The method achieves competitive performance across diverse agentic benchmarks, supported by additional analyses of inference scaling, training behavior, and computational costs.
Critical weaknesses
- ACE-RL jointly trains the Actor, Reflector, and Curator, but does not isolate the benefit of optimizing each role. The actor-only baselines are trained without experience handoff, making them insufficient controls for determining whether directly optimizing Reflector and Curator is necessary, raising the concern of much of the improvement may come from Actor adaptation rather than effective experience transfer.
- ACE-RL introduces substantial computational overhead from candidate-evaluation rollouts. It generates approximately 2.6–3.3× as many tokens as GRPO per training step, while only 55–60% of these tokens contribute to policy updates. The paper lacks compute-matched comparisons or a systematic analysis of performance gains relative to total training and inference costs, leaving the practical efficiency of the proposed method unclear.
- ACE-RL evaluates each candidate playbook using only one downstream Actor rollout, whose reward reflect more of sampling randomness rather than the actual quality of the generated experience, making credit assignment largely noisy. Selecting the highest-reward candidate among noisy evaluations introduce more noise and the risk of a decrease in exploration diversity.
Questions for authors
- Appendix G.1 reports that all 300 Mastermind tips are identical template strings. What causes this behavior, and does it emerge gradually during training? Could it be related to the template examples included in the Curator prompts? Does it implies entropy collapse during training?
- How does ACE-RL's RSI@K compare with Pass@K using the same trained Actor without experience handoff? In particular, given the strong first-attempt performance on MBPP and the weaker inference scaling on AppWorld, how much of the improvement can be attributed to actually utilizing experience from previous attempts?
- During training, ACE-RL selects the best candidate playbook based on additional downstream rollouts. However, Section 4.2 states that no such evaluation rollouts are performed at inference. How are playbook updates accepted or rejected during inference, and how does this difference affect the learned experience-handoff behavior?
- ACE-RL evaluates each candidate playbook using only one downstream Actor rollout, which may introduce substantial noise into the Reflector and Curator advantages. Have the authors investigated how the number of evaluation rollouts per candidate affects training stability, final performance, and computational cost? Additionally, does the unequal number of candidate evaluations across different reflections affect the reliability of credit assignment?
2 comments
Thanks for reading our rebuttal, glad to hear that our revision was helpful!
Thank you for the detailed rebuttal. I appreciate the improved compute matching, which largely addresses my concerns about training efficiency. Although the gains from experience handoff remain inconsistent across benchmarks, I find the new evidence sufficient to strengthen the paper's empirical support. I will therefore raise my rating to 3 (Weak Accept).