acceptodds
Under review as a conference paper at ICLR 2027

HELICES: Adaptive Expansion of Capability Frontiers for Agent Self-Evolution

Abstract

Self-evolution of large language model (LLM)-based agents aims to continually improve their capabilities by discovering and internalizing effective behaviors from past interactions. Yet existing approaches often separate these processes: experience extraction does not consistently adapt to what the agent still needs to learn. We argue that experience extraction and internalization should be coupled around a dynamic capability frontier. Accordingly, we introduce HELICES, a method for adaptive capability-frontier expansion that couples experience extraction and internalization as the policy evolves. HELICES probes this frontier with self-generated natural-language hints, using gains over unguided execution to identify effective behaviors not yet mastered. HELICES jointly optimizes hint generation and utilization through shared parameters, and then internalizes independently verified gains by fully supervising actions and selectively weighting reasoning tokens based on hint attribution. Together, adaptive probing and verified internalization continually expand the capability frontier for independent task solving. Across three agent benchmarks, HELICES surpasses strong baselines by up to 7.2 percentage points and generalizes to unseen tasks without hints.

Primary area:
reinforcement learning
Keywords:
Self-Evolving Agents;Co-Evolution;Dynamic Capability Expansion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_orCsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HELICES is a two-stage self-evolution method built around a "capability frontier": tasks the agent can solve only with a hint it writes for itself. In the first stage, one shared model is trained with GRPO as both hint generator and solver for 120 updates, so the frontier keeps moving. In the second stage, verified hinted improvements are distilled into the hint-free policy over 30 updates. Two filters decide what gets distilled: a confirmation rollout and a gate on reasoning tokens. All evaluation is hint-free. On ALFWorld, Search-QA and Mind2Web at 3B and 7B, HELICES reports 94.3 / 97.1 on ALFWorld. A long appendix adds multi-seed runs, budget-matched comparisons and mechanism studies.

Critical strengths

S1. The ALFWorld result is replicated over three full-pipeline seeds (Tables 13–14). It is also compared against a near-episode-matched Evolving-RL with GPU-hours reported (Tables 17–18). Evaluation is hint-free throughout, which is the right deployment setting.

S2. The internalization step gets real mechanism controls. The gate is tested against a shuffled gate with matched sparsity over five paired seeds (Tables 24–25), and a generator×solver crossing separates better hints from better hint use (Appendix F.1). These only explain about 2 pp of the gain, though.

Critical weaknesses

W1. The gain mostly comes from the joint stage, and I'm not convinced frontier probing is the reason. At 7B the joint checkpoint already scores 94.86 and internalization adds 2.3 pp (Table 3), with intervals spanning zero (Table 52). Hints themselves help little when used: +2.16 pp in the fixed-probe study, and |B| never exceeds 26/274. Yet Frozen Generator, Static probes and Separate G/S all land at 93–95, against 80 for GRPO. A 13–15 pp gap that doesn't care whether the generator learns points to something these runs share, such as solver-only-on-hinted-rollouts, batch composition, or codebase and hyperparameter differences from the reproduced GRPO. I'd like a same-code joint stage with an empty hint, and GRPO run with Table 9's settings.

W2. Result provenance is inconsistent, and the SOTA claim doesn't hold. Appendix D.3 says Table 1 rows are single runs, but they equal the three-seed means of Tables 13–16 (e.g. 3B: 1188/1260 = 94.29, which matches no single seed). The HELICES rows in Table 21 (continuation seeds) and Table 19 (independent seeds) are identical in all six statistics. The 7B base model on the same Seen tasks reads 16.4 (Table 1), 41.2 (Table 39) and about 48.9 (Fig. 3). Using the paper's own weighting, SEED-3B and SDAR-7B in Appendix D.4 beat HELICES on Search-QA. SEED is the closest method and belongs in the main table, or "state-of-the-art" should go.

Questions for authors
  1. [Clarification] Are the Table 1 cells three-seed means or single runs? If means, were the baselines also evaluated three times?
  2. [Clarification] Why are the HELICES rows of Tables 19 and 21 identical, and likewise the 7B means in Tables 4 and 13? Which experiment produced them?
  3. [Clarification] Was GRPO run in your codebase with your hyperparameters? Can you report a joint stage with an empty or random hint? A clear win here would change my view of W1.
  4. [Clarification] What explains the three different base-model ALFWorld numbers (16.4 / 41.2 / about 48.9)?
  5. [Clarification] At 7B, unguided success rises from 43.6 to 78.4 within 20 joint updates, while guided success is no higher than unguided. What do you think transfers?
11 comments

Q1 & Q2: How were results aggregated across training seeds, and why did some values repeat across tables?

Answer 1 & 2: Thank you for pointing out these issues. Both arose during the same update of the multi-seed results.

  • Training seeds and result sources. Table 1 reports mean scores for GRPO, Evolving-RL, and HELICES over three independent training seeds (42, 84, and 126). Each run starts from the corresponding base model and follows the full training procedure. Tables 13–16 in Appendix D.5 report the per-seed results and standard deviations. Due to limited computational resources, we report single-seed results for the other baselines that we reproduced or evaluated ourselves. Results quoted from prior papers follow their original evaluation settings. We updated the main table with the multi-seed results but overlooked the corresponding update to Appendix D.3. Its description of the result sources therefore still reflected the earlier experimental setup and no longer matched the main table.

  • Distinguishing full training from continuation training. Tables 13 and 19 report three full training runs, each starting from the base model with a different seed. Tables 4 and 21 each report three continuation runs from the corresponding shared checkpoint at step 120 of joint training. During the same update of the multi-seed results, we mixed up the statistics for full training and continuation training. As a result, the HELICES endpoint statistics in Table 21 repeated those in Table 19, and the 7B HELICES means in Table 4 repeated those in Table 13.

  • Revisions and verification. We thank the reviewer for bringing these issues to our attention. After reviewing the original records, we have recomputed the statistics for each experimental setting and corrected Tables 4 and 21. The experimental setup for the main table, its notes, and Appendix D.3 have been updated to specify the training seeds, evaluation passes, and aggregation procedure for each method. For transparency, checkpoint details and per-seed results for both types of experiments have been added to the appendix. The corresponding values cited throughout the main text and appendix have also been checked.

Q3: GRPO implementation and joint-training results with empty or random hints

Answer: We appreciate the suggestion. We report two sets of hint ablations.

We have trained the GRPO baseline using the same general training hyperparameters and data splits as HELICES, and evaluated it under the same unguided protocol. To examine the role of hints in joint training, we compare different hint conditions in two settings: training from the same base model and continuing training from a shared early joint-training checkpoint.

  • Training from the same base model

    All conditions start from Qwen2.5-7B-Instruct with training seed 42 and complete 120 updates with matched Solver interaction budgets. GRPO uses unguided trajectories; joint-training variants use empty hints, random hints of similar length from other tasks in the batch, or hints generated from the current task’s trajectory and feedback. We report unguided performance at the final joint-training checkpoint, before internalization.

Training conditionALFWorld Seen SRALFWorld Unseen SRMind2Web AAMind2Web Task SR
GRPO78.57%79.85%23.33%1.57%
Joint training with empty hints80.71%80.60%22.99%1.27%
Joint training with random hints2.14%2.24%22.44%1.57%
Full HELICES94.29%91.79%28.05%2.09%
  • Continuing from the shared J1 checkpoint

    To examine the role of hints after the same initial capability gains, all conditions continue from the shared J1 checkpoint (step 20) for 100 updates with matched Solver interaction budgets. We compare GRPO, empty hints, random hints, Frozen Generator, Static probes, and Separate G/S, with unguided evaluation at step 120.

Training conditionALFWorld Seen SRALFWorld Unseen SRMind2Web AAMind2Web Task SR
Shared J1 checkpoint78.57%77.61%16.74%1.19%
J1 → GRPO86.43%84.33%24.30%1.64%
J1 → Joint training with empty hints87.14%83.58%23.50%1.79%
J1 → Joint training with random hints82.86%81.34%24.80%1.42%
J1 → Frozen Generator91.43%88.06%25.60%2.01%
J1 → Static probes89.29%85.07%26.30%1.86%
J1 → Separate G/S92.86%91.04%27.10%1.94%
J1 → Full HELICES94.29%91.79%28.05%2.09%
Frozen Generator fixes the Generator parameters at J1. Static probes reuses task–trajectory–hint combinations cached at J1. Separate G/S initializes both models from J1 and trains them without parameter sharing. Both Full HELICES rows report the same endpoint. ALFWorld results use one evaluation pass, with J1 taken from the first recorded pass. Mind2Web AA is weighted by the number of tasks in each test split.

Conclusions.

  • Hints can still help when the Generator stops updating. Frozen Generator continues to generate hints, Static probes reuses existing hints, and Separate G/S continues to learn how to generate and use hints. The strong performance of these variants is consistent with hints playing an important role in training. Comparisons with Full HELICES measure the additional effects of continued adaptation and parameter sharing. The empty-hint control tests whether joint training retains the full gains after removing task-relevant hints.

  • Task-relevant hints help the model improve through self-evolution. Full HELICES outperforms the empty-hint and random-hint variants, even though all three use the same joint-training procedure. Learning with task-relevant hints produces additional capability gains that carry over to the final unguided evaluation.

  • Training outcomes depend on hint quality and on how well the model already generates and uses hints. On ALFWorld, the random-hint condition performs substantially better when training starts from J1. Frozen Generator and Static probes also support further gains. These results suggest that the value of hint-guided training depends on what the model has already learned. Full HELICES improves further by adapting to the current task and model state.

  • Hints continue to help after early training. From the same J1 checkpoint, Full HELICES still outperforms continued GRPO and joint training with empty or random hints. Hints help the model build early capabilities and support later learning, further improving its ability to solve tasks without hints.

  • Parameter sharing adds to the gains from learning with hints. Separate G/S performs close to Full HELICES, reflecting the value of cooperation between the Generator and Solver through hints. Sharing parameters links learning in the two roles and further improves the final performance of Full HELICES.

Q4: Differences among the three reported ALFWorld base-model scores

Answer: Thank you for noticing the discrepancy.

  • Table 1 reports 16.4% for Vanilla under the direct action output setup used in baselines such as SkillRL and Evolving-RL. In this setup, the model selects an action directly from the current observation. Skill-Prompt and Few-Shot (text) use the same setup. The remaining methods use a ReAct-style interaction framework, in which the model reasons explicitly from the current observation before choosing each action. The 48.9% initial base-model score in Figure 3 also comes from the ReAct setup.

  • The 48.9% in Figure 3 and the 41.2% in the multi-round alternating experiments come from the same base model before training. As noted in Appendix G, the two experiments use different evaluation setups. For Figure 3, we obtained the initial score with a standalone evaluation script that calls SGLang. The alternating experiments use the native Slime evaluator, which uses the same prompt format and action parsing as training. We used this evaluator at each checkpoint to track capability changes across rounds of joint training and internalization.

  • We inspected trajectories from both evaluators and found a possible issue when the native Slime evaluator runs three evaluation passes in parallel. If the same task appears more than once in a batch, the evaluator may fail to track a separate step budget for each trajectory. Some trajectories can then run out of steps early. The base model typically takes more steps to complete a task, so this issue is more likely to affect its score. We rarely saw the issue at later checkpoints as the model improved. We now use the standalone SGLang evaluation script throughout. The discrepancy prompted us to check the evaluation pipeline more closely.

Reviewer_qmhHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HELICES, an agent self-evolution method that couples experience extraction and internalization around a dynamic capability frontier, which comprises three stages: 1) Adaptive Probing: a shared policy jointly learns to generate hints from its own trajectories and feedback and to solve tasks with these hints via GRPO; 2) Verification: hint-induced gains are accepted only if they are reproduced on an independent seed; and 3) Hint-Attribution-Guided Internalization: verified trajectories are learned without hints, with full supervision on actions and selective weighting on reasoning tokens based on hint attribution. The paper also provides a task-level formulation of capability regions and coverage change. Experiments on ALFWorld, Search-QA, and Mind2Web with Qwen2.5-3B/7B show the effectiveness of HELICES under hint-free evaluation.

Critical strengths
  1. The motivation of coupling experience extraction and internalization is reasonable, and the task-level formulation of capability regions is a useful lens to distinguish frontier expansion from improvements in average success rate.
  2. The independent confirmation step is well-motivated, and the fresh-seed audit (Table 5) shows that confirmed hints yield more reproducible gains and fewer harmful cases than rejected ones.
  3. All evaluations are conducted without test-time hints across three benchmarks of different types and two model scales, and the appendix provides extensive implementation details, multi-seed results, and ablations.
  4. The figures are clear and help readers understand the overall pipeline of the proposed method.
Critical weaknesses
  1. Limited novelty in a crowded direction. The claimed "missing feedback loop" between experience extraction and internalization has already been studied extensively, and most components of HELICES have direct precedents, many of which are not cited: 1) extracting hints from agent failures, generating guided trajectories, and fine-tuning on them with the hints removed has already been applied to ALFWorld and WebShop [1]; 2) using hints during RL while optimizing a hint-free policy has been explored by [2, 3, 4, 5]; 3) jointly training a guidance generator and a solver, with the generator rewarded by the solver's guided success, is the two-player setting of [6]; 4) self-mining guidance, validating it with paired guided/unguided rollouts, and selectively distilling guided action tokens into the plain policy is almost exactly [7], and using the log-probability shift between guided and unguided contexts as a token-level signal is proposed by [8, 9]. Given these works, the contribution of HELICES appears to be a combination of existing components with a theoretical framing, and the paper does not clearly identify what is fundamentally new.

  2. The theoretical analysis is weakly connected to the actual method. Section 3 relies on an idealized model, and the authors acknowledge in several places that it does not describe what HELICES actually optimizes: 1) the sufficient condition in Eq. (8) requires a trajectory-level KL of at most −log⁡α-\log\alpha (about 0.22 nats for α=0.8\alpha=0.8) over trajectories of hundreds of tokens, which is unlikely to hold in practice; 2) the objective actually used (Eqs. 15–16, with gated reasoning tokens, sample weights, and GRPO terms) differs from the full-trajectory KL projection, so the bound cannot be applied to it; 3) Eq. (7) is an accounting identity rather than a result, and the theory defines regions with a threshold α\alpha while the empirical analysis in Section 5.3 uses "3/3 successes", which corresponds to α→1\alpha\to1 estimated from only three evaluations. As a result, the theory reads more as a post-hoc framing than as support for the design choices.

  3. Unclear or inconsistent formulation and method details. Several parts of Sections 3–4 are hard to follow or appear inconsistent: 1) Δ\Delta is overloaded, denoting the paired success gain Δk(h;x,ω)\Delta_k(h;x,\omega) in Section 3.2, the token-level log-likelihood contrast Δit\Delta_{it} in Eq. (13), and further quantities in Tables 3 and 5; similarly, ϕk\phi_k and θk\theta_k are first defined separately and then identified; 2) more importantly, because all GG candidates in a group share the same source outcome J(τi0)J(\tau_i^0), group normalization cancels it, so the success component of the generator advantage AqA^q in Eq. (10) is identical to the solver advantage AhA^h (the authors partly admit this at the end of Section 4.2). The "gain over unguided execution", which the abstract presents as the core probing signal, therefore contributes nothing beyond the optional efficiency and truncation terms; 3) key quantities are deferred to the appendix or left undefined in the main text, including the exact form of rprober^{\text{probe}}, the base weights bitb_{it}, the sample weights wiw_i, how the confirmation seed is chosen, and the meaning of the "independently normalized objectives" in Eq. (16); 4) the hint representation is specified only for ALFWorld (JSON evidence/advice fields), and it is unclear what hints look like for Search-QA and Mind2Web, or how "protocol-valid" is defined there.

  4. Incomplete and potentially unfair experimental comparison. 1) The most closely related methods [2, 6, 7, 8, 9] are neither discussed nor compared, and SEED is reported only in the appendix under a different protocol; 2) the evaluation protocol is not matched: in Table 1, HELICES on ALFWorld is evaluated with three passes, whereas the baselines use a single pass on the 140 Seen tasks only; Skill0 and AgentOCR use Qwen2.5-VL backbones; and the baseline numbers mix reproduction, checkpoint re-evaluation, and values copied from the original papers; 3) the discovery and confirmation stages require substantial additional environment interactions that are excluded from the "matched budget" claim, so a compute- or interaction-matched comparison is needed; 4) apart from ALFWorld, the improvements are marginal (e.g., +0.7 / +0.4 pp over SAPO on single-hop / multi-hop QA at 7B, Mind2Web SR below 3%), and Bamboogle contains only 11 questions, so a single question changes the score by 9.1 pp. No significance tests are reported for these comparisons.

  5. Outdated backbones and saturated benchmarks. 1) All experiments use Qwen2.5-3B/7B-Instruct, released about two years ago. It is unclear whether the gains hold for current models with much stronger in-context learning and reasoning abilities (e.g., the Qwen3 series), where hints may be less necessary or behave differently. Prior work [10] also suggests that self-generated in-context teachers are unreliable at the 3B scale; 2) ALFWorld is largely saturated: plain GRPO already reaches 80%, several baselines exceed 90% at 7B, and HELICES reaches 97.1% at 7B, leaving little room to demonstrate frontier expansion. Search-QA and Mind2Web are also standard benchmarks from earlier work. To support the claim of continual capability expansion, the method should be evaluated on more challenging and realistic agentic benchmarks such as Terminal-Bench, GDPval, SWE-bench, or τ²-bench, where current models are still far from solving the tasks.

  6. Missing key ablations and analyses. Several analyses are needed to understand where the gains come from: 1) the effect of the hint form, e.g., conditional rules vs. free-form reflections, episode-level vs. step-level hints, abstract principles vs. partial solutions, and hint length or specificity; 2) whether joint training of the generator matters, compared with a frozen generator, an external (stronger) hint generator, or separate parameters for the two roles, and whether rprober^{\text{probe}} can be replaced by guided success alone; 3) a comparison of the hard attribution gate with on-policy (self-)distillation methods (e.g., OPD/OPSD and [9, 10]) that use the dense token-level KL to the hint-conditioned teacher; 4) a sensitivity analysis of λint\lambda_{\text{int}}, which is very small (0.005), together with the gradient contribution of the internalization term relative to GRPO, since Table 3 shows that internalization adds only about 2 pp; 5) a breakdown of whether the acquired tasks reflect transferable decision rules or task-specific memorization, e.g., performance on held-out task families or environments not used for hint generation.

[1] Fine-tuning with RAG for Improving LLM Learning of New Skills. arXiv:2510.01375, 2025.

[2] Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models. arXiv:2506.13923, 2025.

[3] QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. ICLR 2026.

[4] Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning. ICLR 2026.

[5] Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback. ICML 2026.

[6] RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems. arXiv:2510.02263, 2025.

[7] SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training. arXiv:2606.02355, 2026.

[8] OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning. arXiv:2606.26790, 2026.

[9] Reinforcement Learning via Self-Distillation. ICML 2026.

[10] Self-Distillation Enables Continual Learning. ICML 2026.

Questions for authors
  1. What exactly does a "fresh realization" mean in independent confirmation? In ALFWorld, does the partner seed change the environment layout (object placement, initial room), or only the sampling randomness of the solver? If the task instance is the same, the confirmation may not be truly independent of discovery. In addition, for Search-QA, how do you prevent a hint derived from the source trajectory and its feedback from leaking the gold answer or key entities?

  2. Could you provide concrete hint examples for Search-QA and Mind2Web, together with statistics on hint length, protocol-invalid rate, and the fraction of hints that are task-specific rather than transferable decision rules? Do hints generated for one task ever help other tasks?

  3. In Figure 4, most net acquisition happens in the first one or two cycles (e.g., +82 and +66 tasks for 3B), while later cycles contribute only a few tasks. Does this indicate that the frontier saturates quickly on ALFWorld? How would the method behave on a benchmark where the hint-exposed region does not shrink so fast, and does the "continual" expansion claim still hold in that case?

11 comments

W1: What distinguishes HELICES from existing approaches to guidance and internalization?

Answer W1: Thank you for identifying these related studies.

1. The autonomous feedback loop between experience extraction and capability internalization remains underexplored.

The core focus of our research is to establish a continual self-evolution loop in which experience extraction and capability internalization reinforce each other, relying solely on the agent’s own interactions with the environment to continually expand the capability frontier for independent task solving. Although prior work has explored related mechanisms, this autonomous feedback loop remains underexplored.

  • Externally supplied experience does not improve its extraction mechanism as the policy learns. RAG fine-tuning [1] uses external GPT-4o hints to support LoRA fine-tuning, without feeding student improvements back into training the hint generator. Guide [2] generates guidance from reference solutions using an external model, while QuestA [3] uses fragments of existing correct solutions. Scaf-GRPO [4] adjusts the use of externally generated hints but does not train their generator. OPID [8] extracts skills from new trajectories, but policy updates do not optimize the external analyzer. These methods support internalization without improving the mechanism that supplies experience.

  • Learning from self-generated experience does not necessarily optimize experience extraction. The self-critique version of Critique-GRPO [5] improves answers using its own critiques, but does not directly optimize critique generation based on the resulting solution quality. SIRI [7] mines new skills as the policy changes, but explicitly notes the absence of a dedicated training signal for skill mining. SDPO [9] internalizes predictions conditioned on retrospective feedback, without explicitly extracting experience and optimizing its extraction through execution outcomes. These methods improve the policy using its own feedback, but do not directly train experience extraction based on which experience improves behavior.

  • Some methods learn experience extraction but require external teacher initialization. RLAD [6] uses an external model to generate and filter abstractions, while SEED initializes trajectory analysis through supervised fine-tuning on externally annotated skills. Their subsequent self-improvement continues to reinforce additional knowledge supplied by stronger models, placing them outside the standard self-evolution setting. HELICES establishes and improves the feedback loop without such teacher-supervised initialization.

Prior work has not fully explored this autonomous feedback loop, which remains a distinct and valuable research problem in self-evolution.

2. The three modules make distinct methodological contributions.

  • Experience extraction adapts to current difficulties and improves through execution feedback. The methods discussed above supply guidance, adjust its use, or optimize responses to it without directly training experience extraction through its execution outcomes. RLAD [6] generates abstractions from the problem, whereas HELICES extracts experience from current interactions and feedback to target the actual difficulties encountered by the policy. HELICES therefore learns both what experience to extract as tasks and capabilities change and how to extract effective experience through continued execution feedback.

  • Gain verification separates candidate discovery from independent confirmation. SIRI [7] evaluates skill utility through differences in returns between groups of rollouts with and without skills. After discovering an improvement, HELICES fixes the hint and repeats the paired guided and unguided comparison using fresh executions. Confirmation trajectories enter internalization only when both comparisons show the same type of improvement. This tests whether the observed benefit is reproducible, reducing the risk of learning from chance successes.

  • Internalization fully supervises actions and selectively reinforces hint-supported reasoning. RAG fine-tuning [1] fine-tunes on trajectories after removing hints. SIRI [7] selectively distills actions based on utility and action advantages while excluding reasoning text. OPID [8], SDPO [9], and SEED construct learning signals by reassessing output probabilities under additional context. HELICES fully supervises actions in independently confirmed trajectories and applies weighted supervision only to reasoning whose hint support exceeds a threshold. It distinguishes actions that should be learned in full from reasoning worth reinforcing, preserving complete effective behaviors while reducing indiscriminate learning from incidental reasoning.

Distinct motivations and mechanisms give each module methodological value; similar technical forms do not make HELICES a simple combination of existing components.

W2: How does the theoretical framework support the practical method?

Answer W2: Thank you for raising these questions about the connection between theory and implementation.

As explicitly stated in the paper, our theoretical framework does not provide a direct guarantee for the full HELICES training objective. It formalizes capability expansion through an idealized model of internalization. The appendix analyzes its connection to the practical objective, while controlled experiments evaluate the actual design choices.

The framework connects experience extraction, gain verification, and capability internalization. HELICES uses hints to discover behaviors the policy cannot yet reliably perform independently, then internalizes them into unguided capabilities. As capabilities improve, experience extraction adapts to changing learning needs through new interactions and feedback. Independent confirmation checks whether observed gains recur before trajectories enter internalization. For a fixed successful target distribution, Equation (8) links lower KL divergence from the unguided trajectory distribution to a higher lower bound on independent success probability.

  • Equation (8) gives a sufficient, not necessary, condition for capability acquisition. The condition of approximately 0.223 nats at α=0.8\alpha=0.8 is indeed strict. However, KL depends on the match between distributions; trajectory length alone does not establish that the condition cannot hold. The policy may also succeed through trajectories outside the selected target distribution. Equation (8) establishes the mathematical connection between successful-target fitting and independent success, while fresh unguided evaluations assess actual capability gains.

  • The difference between the practical objective and full-trajectory projection is explicitly analyzed. Gating, sample weighting, and GRPO change the update direction. The appendix accounts for these deviations from the full-projection gradient under a fixed target and the stated optimization assumptions. The guarantee is therefore conditional, and controlled ablations test the effectiveness of the specific gating and weighting choices.

  • Equation (7) decomposes capability changes, and the 3/3 criterion applies a fixed threshold. Equation (7) separates coverage changes into acquisition and forgetting, allowing us to examine the sources and retention of new capabilities. With three evaluations, empirical success rates are limited to 00, 1/31/3, 2/32/3, and 11. At α=0.8\alpha=0.8, only 3/3 successes satisfy p^≥α\hat p\geq\alpha; this does not imply α→1\alpha\rightarrow1. We acknowledge the limited estimation precision and report sensitivity analyses under alternative success-count criteria. Across both model scales and both training stages, previously hint-dependent tasks show higher subsequent unguided acquisition rates under all three criteria.

The framework remains open to further development. This study connects experience extraction at current capability limits with independent capability acquisition through internalization. Future work will improve experience generation and selection, alongside training methods for learning and retaining effective behaviors.

W3: How are the notation, rewards, normalization, and hint protocols defined?

Answer W3: We appreciate these suggestions for making the formulation and implementation clearer.

1. Notation and parameter sharing.

We will use distinct symbols for success-rate gains, token-level log-likelihood differences, and metric changes in tables to avoid overloading Δ\Delta. In the theory, ϕk\phi_k and θk\theta_k denote the Generator and Solver parameters, respectively. HELICES shares parameters during joint training, when ϕk=θk\phi_k=\theta_k. During internalization, a copy of the Generator is frozen while the Solver continues to update. We will clarify these meanings and stages when the symbols first appear.

2. Baseline cancellation and the role of unguided experience.

The shared unguided success baseline does cancel under within-group normalization. For valid candidates in the same group, the Generator and Solver receive identical advantages when only the success reward is considered. The original text did not clearly distinguish the role of unguided experience in generating guidance from the learning signal obtained after normalizing gain differences. Unguided trajectories and environmental feedback already condition hint generation: the Generator proposes guidance based on current policy behavior and learns from guided execution which guidance the Solver can use effectively. As the policy changes, new experience and execution feedback continually improve both guidance generation and use. We will revise the relevant statements in the abstract, introduction, and method section.

3. Reward, weight, and normalization definitions.

We will supplement the appendix with the definitions needed to understand the training procedure.

In the ALFWorld configuration, the Generator reward for a valid hint is

rprobe=(Jh−J0)+0.2J0Jhclip⁡(L0−LhTmax⁡,−1,1)−0.1I[z0=0,zh=1].r_{\mathrm{probe}}=(J^h-J^0)+0.2J^0J^h\operatorname{clip}\left(\frac{L^0-L^h}{T_{\max}},-1,1\right)-0.1\mathbb{I}[z^0=0,z^h=1].

Here, JJ, LL, and zz denote task success, execution length, and truncation, respectively. Superscripts 00 and hh indicate unguided and guided executions, and Tmax⁡T_{\max} is the step limit. The efficiency term applies only when both executions succeed and is omitted for tasks with fixed or poorly measurable execution costs.

During internalization, the base weight bitb_{it} is 1 for action tokens and 0.25 for reasoning tokens, with attribution gating subsequently applied to reasoning. The sample weights wiw_i are 4 for failure-to-success improvements and 1 for efficiency improvements. Confirmation uses the sampling seed and unguided outcome of another source trajectory in the group that did not contribute to generating the hint. Specifically, candidate jj is paired with (j+1) mod G(j+1)\bmod G.

In Equation (16), separate normalization means that GRPO is averaged over source trajectories, while the internalization term is normalized by the total weight of accepted samples; the two are then combined using λint\lambda_{\mathrm{int}}. The token average within each internalization trajectory includes model-generated positions with zero gating weight. We will add these definitions and task-specific configurations to the appendix so that each loss calculation can be reproduced.

4. Search-QA and Mind2Web hint protocols.

Our response to Q2 details the hint representations and the checks that define protocol validity for these benchmarks.

Reviewer_dQiHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HELICES, a method that improves LLM agents by extracting experience and then internalizing it into the policy. From interaction trajectories and environment feedback, the model writes natural-language hints. These hints target behaviors the agent can carry out when guided but cannot yet do reliably on its own. Hint generation and hint-guided solving are trained jointly with shared-parameter GRPO. HELICES then re-runs the task to check whether a hint's gain holds up, and internalizes the accepted trajectories without hints, using full action supervision, attribution-weighted reasoning supervision, and unguided GRPO. On ALFWorld, Search-R1, and Mind2Web with Qwen2.5-3B/7B, it improves overall hint-free performance over several RL and experience-based baselines. The paper also reports multi-seed runs, mechanism ablations, interaction-cost comparisons, and repeated-cycle experiments.

Critical strengths
  1. The proposed method combines adaptive hint probing, independent gain confirmation, and hint-free internalization, which sounds interesting.
  2. This paper conducts the full pipeline over multiple independent seeds, compares methods under matched interaction and compute budgets, ablates components, and studies repeated cycles.
Critical weaknesses
  1. The paper should position itself more clearly against existing self-distillation methods. HELICES shares a basic idea with on-policy self-distillation: information available only during training is used to push the model toward behavior it must later produce unaided. OPSD does this by matching teacher and student distributions, where the teacher sees privileged context. SEED is closer still: for agents, it combines self-generated hindsight skills, guidance-conditioned probability shifts, and RL. HELICES does have its own mechanisms (a trained hint prober, independent gain confirmation, and attribution-gated internalization), but the paper should state which parts are new and which extend these prior methods. This point concerns novelty rather than the reported gains.
  2. The added value of the internalization objective is not fully established. The ablations show that independent confirmation and attribution-based token weighting both help, and the attribution-specificity controls show the effect is more than just sparser supervision. What is missing is a controlled comparison between the proposed internalization objective and closely related self-distillation objectives. The attribution signal is a contrast between guidance-conditioned and unguided log-probabilities, so a direct comparison with OPSD- or SEED-style distillation would show what the new objective adds.
Questions for authors
  1. How does HELICES differ methodologically from prior on-policy self-distillation work, SEED in particular? Which parts of coupling guidance generation with policy improvement are new, and which extend existing self-evolving hindsight-supervision methods?
  2. Could the HELICES internalization loss be compared with an OPSD- or SEED-style distillation objective, holding fixed the accepted hints, training trajectories, backbone, and interaction budget? This would show whether the gains come mainly from better hint generation and verification or from the attribution-gated supervision rule. If a full experiment is not feasible, a conceptual argument or partial evidence would still help.
  3. The repeated acquisition/retention results are reported on ALFWorld only. Do the authors expect alternating probing and internalization to keep its advantage over the single-switch schedule on Search-R1 and Mind2Web as well? Evidence on this would help in judging the broader self-evolution claim.
3 comments

Q1: How does HELICES differ from SEED and other on-policy self-distillation methods in coupling guidance generation with policy improvement?

Answer 1: Thanks for the inspiring question. HELICES differs from prior methods in both principle and methodology.

  • Conceptually, HELICES is a self-evolving algorithm that improves by discovering and internalizing effective behaviors from environment interaction. SEED requires skill annotations generated by an external model for SFT initialization. Such methods therefore belong to a fundamentally different class of algorithms from HELICES. HELICES provides a path toward autonomous improvement on new tasks for which ready-made solutions are not yet available.

  • Methodologically, the core innovation of HELICES lies in actively constructing reliable learning targets to address current gaps in policy capabilities. This directly links the optimization of experience generation to gains in unguided task-solving ability.

    • Directly optimizing guidance generation. Existing methods, such as OPSD and SEED, convert reference answers, feedback, or experience into supervision to improve the policy. HELICES uses the model being trained to generate effective guidance for the current policy based on the hints' actual effects on execution.
    • Separating gain discovery from independent confirmation. SIRI selects effective skills by repeatedly comparing executions with and without skills. HELICES separates gain discovery and confirmation for the same hint. It uses confirmation trajectories for internalization only if the improvement is reproduced in a separate set of comparison executions that was not used during discovery. This reduces unreliable supervision caused by chance successes.
    • Distinct internalization rules for actions and reasoning. OPSD applies the same distillation rule to actions and reasoning, while SIRI distills only actions. HELICES fully supervises the actions in confirmed trajectories and selectively supervises reasoning supported by the hint. This preserves successful behavior and reduces imitation of incidental reasoning.

    HELICES makes three methodological contributions: it directly optimizes guidance generation, independently confirms gains, and applies distinct supervision rules to actions and reasoning. The latter two improve existing mechanisms for gain verification and experience internalization.

Q2: Internalization loss comparisons under fixed supervision and matched budgets

Answer 2: We appreciate the suggestion. We compare internalization objectives and assess independent confirmation and attribution gating.

We compare four internalization conditions on ALFWorld and Mind2Web. For each benchmark, all four conditions begin internalization training from the same HELICES model checkpoint obtained after 120 joint-training steps. The SEED and OPSD loss comparisons use identical hints and auxiliary supervision trajectories under the same confirmation setting, with matched numbers of training updates and total interaction budgets. SEED and OPSD use their respective original internalization rules. SEED-style and OPSD-style retain the corresponding rules and add independent confirmation when selecting internalization targets. For hints that show gains during discovery, we resample trajectories with and without the hint on the same task. We use the guided trajectories from the confirmation stage for internalization only if the hint's gains are reproduced in this reassessment.

Ablation results for independent confirmation under the SEED and OPSD internalization rules

MethodALFWorld Seen SRALFWorld Unseen SRMind2Web AAMind2Web Task SR
SEED94.52%94.28%29.02%1.94%
SEED-style95.71%96.02%29.50%2.01%
OPSD95.48%93.78%29.16%2.09%
OPSD-style96.67%95.52%29.75%2.16%
HELICES97.14%96.27%30.62%2.16%
  • Independent confirmation improves performance under both the SEED and OPSD rules. Both methods show improvements in ALFWorld Seen and Unseen success rates, as well as Mind2Web action accuracy and task success rate. These results provide preliminary evidence that independent confirmation is applicable across different internalization rules.

  • Independent confirmation can filter out trajectories that succeed by chance. The original SEED pipeline uses skill annotations generated by an external model for initialization, while OPSD provides its teacher branch with reference solutions. Both methods depend on guidance quality. Directly internalizing trajectories that succeed by chance may reinforce irrelevant reasoning or actions that are difficult to reproduce. Independent confirmation can reduce the number of such trajectories included in training.

  • Attribution gating still contributes to internalization after independent confirmation. The paper's ALFWorld ablation shows that removing attribution gating for reasoning tokens reduces both Seen and Unseen success rates even when independent confirmation is retained. We hold the supervision data fixed and match the budgets in the loss comparisons above. The HELICES row reports reference results from the paper.

Q3: Multi-round alternating training on Search-R1 and Mind2Web versus a single phase switch

Answer 3: Thank you for asking about performance beyond ALFWorld. We compare schedules and report six-round results.

Alternating training and a schedule with a single phase switch achieve comparable performance under the same training budget. Both are viable options. Our additional six-round experiments show that alternating training can also yield cumulative capability gains across different tasks. We conducted additional experiments on Mind2Web and single-hop Search-R1 using six rounds of alternating training. Each round includes 20 joint-training updates (J) and 5 internalization updates (I), giving 150 updates across all six rounds. The results are presented below.

Table R1. Results of six rounds of alternating training on Mind2Web

StageUpdate intervalUnguided AAGuided AAFrontier transferUnexposed leapCoverage lossNet gain
Base08.80%7.08%————
J10→2016.74%16.74%16160−42+134
I120→2517.02%17.86%831−34+5
J225→4521.38%25.19%24141−79+86
I245→5021.63%24.97%2220−36+6
J350→7022.30%30.01%4278−109+11
I370→7522.59%30.33%1522−31+6
J475→9524.95%32.22%6180−88+53
I495→10025.33%31.83%2025−36+9
J5100→12027.68%32.30%4254−52+44
I5120→12527.94%32.14%1417−24+7
J6125→14529.74%32.86%3343−37+39
I6145→15030.10%32.97%1012−17+5

Table R2. Results of six rounds of alternating training on single-hop Search-R1

StageComparison intervalUnguided EMGuided EMFrontier transferUnexposed leapCoverage lossNet gain
Base042.65%41.78%————
J10→2048.01%46.54%307543+62
I120→2548.70%47.66%192132+8
J225→4549.91%49.22%253344+14
I245→5050.09%48.44%171227+2
J350→7050.78%51.12%163442+8
I370→7552.16%51.30%161717+16
J475→9553.03%52.60%263248+10
I495→10052.68%53.11%9821−4
J5100→12051.90%52.51%212454−9
I5120→12552.25%53.46%111320+4
J6125→14552.85%50.87%183748+7
I6145→15053.46%52.68%71616+7
Total215322412+125
  • Alternating training over multiple rounds also produces net growth in capability coverage beyond ALFWorld. Both Mind2Web and single-hop Search-R1 show expanded capability coverage after multiple rounds, with joint training and internalization each contributing to the net growth. These results provide empirical evidence that repeated probing and internalization are applicable to web navigation and retrieval-based question answering.

  • Unguided action accuracy on Mind2Web improves after every internalization stage. These gains are still observed in the later stages of training. Alternating training achieves a final unguided action accuracy close to the result reported in the paper for the schedule with a single phase switch.

  • The model solves more single-hop Search-R1 questions correctly without hints after multiple rounds of alternating training. Unguided exact-match accuracy still shows net gains in the later stages of training, and its final value exceeds that of the base model.

We have added the corresponding figures to the appendix section Alternating Training on Mind2Web and Search-R1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.