- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HELICES is a two-stage self-evolution method built around a "capability frontier": tasks the agent can solve only with a hint it writes for itself. In the first stage, one shared model is trained with GRPO as both hint generator and solver for 120 updates, so the frontier keeps moving. In the second stage, verified hinted improvements are distilled into the hint-free policy over 30 updates. Two filters decide what gets distilled: a confirmation rollout and a gate on reasoning tokens. All evaluation is hint-free. On ALFWorld, Search-QA and Mind2Web at 3B and 7B, HELICES reports 94.3 / 97.1 on ALFWorld. A long appendix adds multi-seed runs, budget-matched comparisons and mechanism studies.
Critical strengths
S1. The ALFWorld result is replicated over three full-pipeline seeds (Tables 13–14). It is also compared against a near-episode-matched Evolving-RL with GPU-hours reported (Tables 17–18). Evaluation is hint-free throughout, which is the right deployment setting.
S2. The internalization step gets real mechanism controls. The gate is tested against a shuffled gate with matched sparsity over five paired seeds (Tables 24–25), and a generator×solver crossing separates better hints from better hint use (Appendix F.1). These only explain about 2 pp of the gain, though.
Critical weaknesses
W1. The gain mostly comes from the joint stage, and I'm not convinced frontier probing is the reason. At 7B the joint checkpoint already scores 94.86 and internalization adds 2.3 pp (Table 3), with intervals spanning zero (Table 52). Hints themselves help little when used: +2.16 pp in the fixed-probe study, and |B| never exceeds 26/274. Yet Frozen Generator, Static probes and Separate G/S all land at 93–95, against 80 for GRPO. A 13–15 pp gap that doesn't care whether the generator learns points to something these runs share, such as solver-only-on-hinted-rollouts, batch composition, or codebase and hyperparameter differences from the reproduced GRPO. I'd like a same-code joint stage with an empty hint, and GRPO run with Table 9's settings.
W2. Result provenance is inconsistent, and the SOTA claim doesn't hold. Appendix D.3 says Table 1 rows are single runs, but they equal the three-seed means of Tables 13–16 (e.g. 3B: 1188/1260 = 94.29, which matches no single seed). The HELICES rows in Table 21 (continuation seeds) and Table 19 (independent seeds) are identical in all six statistics. The 7B base model on the same Seen tasks reads 16.4 (Table 1), 41.2 (Table 39) and about 48.9 (Fig. 3). Using the paper's own weighting, SEED-3B and SDAR-7B in Appendix D.4 beat HELICES on Search-QA. SEED is the closest method and belongs in the main table, or "state-of-the-art" should go.
Questions for authors
- [Clarification] Are the Table 1 cells three-seed means or single runs? If means, were the baselines also evaluated three times?
- [Clarification] Why are the HELICES rows of Tables 19 and 21 identical, and likewise the 7B means in Tables 4 and 13? Which experiment produced them?
- [Clarification] Was GRPO run in your codebase with your hyperparameters? Can you report a joint stage with an empty or random hint? A clear win here would change my view of W1.
- [Clarification] What explains the three different base-model ALFWorld numbers (16.4 / 41.2 / about 48.9)?
- [Clarification] At 7B, unguided success rises from 43.6 to 78.4 within 20 joint updates, while guided success is no higher than unguided. What do you think transfers?
11 comments
Q3: GRPO implementation and joint-training results with empty or random hints
Answer: We appreciate the suggestion. We report two sets of hint ablations.
We have trained the GRPO baseline using the same general training hyperparameters and data splits as HELICES, and evaluated it under the same unguided protocol. To examine the role of hints in joint training, we compare different hint conditions in two settings: training from the same base model and continuing training from a shared early joint-training checkpoint.
-
Training from the same base model
All conditions start from Qwen2.5-7B-Instruct with training seed 42 and complete 120 updates with matched Solver interaction budgets. GRPO uses unguided trajectories; joint-training variants use empty hints, random hints of similar length from other tasks in the batch, or hints generated from the current task’s trajectory and feedback. We report unguided performance at the final joint-training checkpoint, before internalization.
Training condition ALFWorld Seen SR ALFWorld Unseen SR Mind2Web AA Mind2Web Task SR GRPO 78.57% 79.85% 23.33% 1.57% Joint training with empty hints 80.71% 80.60% 22.99% 1.27% Joint training with random hints 2.14% 2.24% 22.44% 1.57% Full HELICES 94.29% 91.79% 28.05% 2.09%
-
Continuing from the shared J1 checkpoint
To examine the role of hints after the same initial capability gains, all conditions continue from the shared J1 checkpoint (step 20) for 100 updates with matched Solver interaction budgets. We compare GRPO, empty hints, random hints, Frozen Generator, Static probes, and Separate G/S, with unguided evaluation at step 120.
Training condition ALFWorld Seen SR ALFWorld Unseen SR Mind2Web AA Mind2Web Task SR Shared J1 checkpoint 78.57% 77.61% 16.74% 1.19% J1 → GRPO 86.43% 84.33% 24.30% 1.64% J1 → Joint training with empty hints 87.14% 83.58% 23.50% 1.79% J1 → Joint training with random hints 82.86% 81.34% 24.80% 1.42% J1 → Frozen Generator 91.43% 88.06% 25.60% 2.01% J1 → Static probes 89.29% 85.07% 26.30% 1.86% J1 → Separate G/S 92.86% 91.04% 27.10% 1.94% J1 → Full HELICES 94.29% 91.79% 28.05% 2.09%
Frozen Generator fixes the Generator parameters at J1. Static probes reuses task–trajectory–hint combinations cached at J1. Separate G/S initializes both models from J1 and trains them without parameter sharing. Both Full HELICES rows report the same endpoint. ALFWorld results use one evaluation pass, with J1 taken from the first recorded pass. Mind2Web AA is weighted by the number of tasks in each test split.
Conclusions.
-
Hints can still help when the Generator stops updating. Frozen Generator continues to generate hints, Static probes reuses existing hints, and Separate G/S continues to learn how to generate and use hints. The strong performance of these variants is consistent with hints playing an important role in training. Comparisons with Full HELICES measure the additional effects of continued adaptation and parameter sharing. The empty-hint control tests whether joint training retains the full gains after removing task-relevant hints.
-
Task-relevant hints help the model improve through self-evolution. Full HELICES outperforms the empty-hint and random-hint variants, even though all three use the same joint-training procedure. Learning with task-relevant hints produces additional capability gains that carry over to the final unguided evaluation.
-
Training outcomes depend on hint quality and on how well the model already generates and uses hints. On ALFWorld, the random-hint condition performs substantially better when training starts from J1. Frozen Generator and Static probes also support further gains. These results suggest that the value of hint-guided training depends on what the model has already learned. Full HELICES improves further by adapting to the current task and model state.
-
Hints continue to help after early training. From the same J1 checkpoint, Full HELICES still outperforms continued GRPO and joint training with empty or random hints. Hints help the model build early capabilities and support later learning, further improving its ability to solve tasks without hints.
-
Parameter sharing adds to the gains from learning with hints. Separate G/S performs close to Full HELICES, reflecting the value of cooperation between the Generator and Solver through hints. Sharing parameters links learning in the two roles and further improves the final performance of Full HELICES.
Q4: Differences among the three reported ALFWorld base-model scores
Answer: Thank you for noticing the discrepancy.
-
Table 1 reports 16.4% for Vanilla under the direct action output setup used in baselines such as SkillRL and Evolving-RL. In this setup, the model selects an action directly from the current observation. Skill-Prompt and Few-Shot (text) use the same setup. The remaining methods use a ReAct-style interaction framework, in which the model reasons explicitly from the current observation before choosing each action. The 48.9% initial base-model score in Figure 3 also comes from the ReAct setup.
-
The 48.9% in Figure 3 and the 41.2% in the multi-round alternating experiments come from the same base model before training. As noted in Appendix G, the two experiments use different evaluation setups. For Figure 3, we obtained the initial score with a standalone evaluation script that calls SGLang. The alternating experiments use the native Slime evaluator, which uses the same prompt format and action parsing as training. We used this evaluator at each checkpoint to track capability changes across rounds of joint training and internalization.
-
We inspected trajectories from both evaluators and found a possible issue when the native Slime evaluator runs three evaluation passes in parallel. If the same task appears more than once in a batch, the evaluator may fail to track a separate step budget for each trajectory. Some trajectories can then run out of steps early. The base model typically takes more steps to complete a task, so this issue is more likely to affect its score. We rarely saw the issue at later checkpoints as the model improved. We now use the standalone SGLang evaluation script throughout. The discrepancy prompted us to check the evaluation pipeline more closely.
Q1 & Q2: How were results aggregated across training seeds, and why did some values repeat across tables?
Answer 1 & 2: Thank you for pointing out these issues. Both arose during the same update of the multi-seed results.
Training seeds and result sources. Table 1 reports mean scores for GRPO, Evolving-RL, and HELICES over three independent training seeds (42, 84, and 126). Each run starts from the corresponding base model and follows the full training procedure. Tables 13–16 in Appendix D.5 report the per-seed results and standard deviations. Due to limited computational resources, we report single-seed results for the other baselines that we reproduced or evaluated ourselves. Results quoted from prior papers follow their original evaluation settings. We updated the main table with the multi-seed results but overlooked the corresponding update to Appendix D.3. Its description of the result sources therefore still reflected the earlier experimental setup and no longer matched the main table.
Distinguishing full training from continuation training. Tables 13 and 19 report three full training runs, each starting from the base model with a different seed. Tables 4 and 21 each report three continuation runs from the corresponding shared checkpoint at step 120 of joint training. During the same update of the multi-seed results, we mixed up the statistics for full training and continuation training. As a result, the HELICES endpoint statistics in Table 21 repeated those in Table 19, and the 7B HELICES means in Table 4 repeated those in Table 13.
Revisions and verification. We thank the reviewer for bringing these issues to our attention. After reviewing the original records, we have recomputed the statistics for each experimental setting and corrected Tables 4 and 21. The experimental setup for the main table, its notes, and Appendix D.3 have been updated to specify the training seeds, evaluation passes, and aggregation procedure for each method. For transparency, checkpoint details and per-seed results for both types of experiments have been added to the appendix. The corresponding values cited throughout the main text and appendix have also been checked.