- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Explorer-zero alternates training a task proposer and solver in resettable stateful environments. The proposer explores, carries out a task, and records intermediate and final states as verification targets. The solver receives the generated instruction and obtains partial reward for matching intermediate checkpoints, in addition to final completion reward. Proposer incentives favour intermediate difficulty, subtasks and action diversity. Across ALFWorld, ScienceWorld and AppWorld, the method outperforms the evaluated self-play baselines without human-authored training tasks. A 30B AppWorld model reaches 59.67% task goal completion versus 51.93% for Self-Challenging and 62.65% for human-data RL. The theory characterises reduced reward collapse conditional on all rollouts failing and having varied subtask counts.
Critical strengths
-
Reusing demonstrated intermediate states as reward checkpoints provides a coherent way to couple feasible task generation with process supervision. It avoids requiring a separate task-specific judge for each proposal, and the paper makes the resettable, inspectable-environment assumption explicit. This is a useful training-system contribution rather than merely another prompting scaffold.
-
Evaluation uses the benchmarks' independent success criteria, not the proposer-generated checks. Consistent gains across three environments and two AppWorld model sizes therefore support downstream utility despite possible imperfections in the self-generated curriculum. The interactive-proposer ablation and complexity-stratified analysis further connect the gains to the intended long-horizon setting.
Critical weaknesses
-
Semantic reliability of the generated verification targets is not sufficiently established. A reached state proves feasibility, but does not ensure that its retained changes exactly express the final instruction or that intermediate checkpoints represent substantive progress. Figure 8 provides a concrete example: a login checkpoint survives the validity filter, and another subtask mentions adding songs although the saved state contains only an empty playlist. Equation 7 also permits a failed final task to receive reward 1 after all checkpoints have been visited. These issues do not invalidate the independent benchmark gains, but they limit the claim that the generated supervision reliably encodes task completion.
-
The reported evidence does not isolate the proposed reward-collapse mechanism. Table 2 changes both the generated task distribution and solver rewards, as the authors acknowledge. Figure 3 reports fewer zero-reward rollouts, whereas the proposition concerns equal rewards within a rollout group; a group can have nonzero identical rewards and still collapse. The gains establish the value of the combined approach more strongly than the particular optimisation explanation. Reported deviations also come from evaluation sampling, not independent training runs.
Questions for authors
-
Clarification and targeted validation: how do the matching rules handle incidental state changes, logically equivalent solutions and unrelated destructive changes? Could you report a small audit of existing generated instructions/checkpoints, including the frequency of setup-only or mismatched checkpoints and failed-final rollouts receiving reward 1?
-
Existing-log analysis: what fraction of GRPO groups has zero reward variance with and without subtask supervision, especially conditional on all final tasks failing? This directly tests the stated mechanism. If available, a fixed generated-task pool evaluated under binary versus checkpoint rewards would separate reward granularity from curriculum changes, but the group-variance statistic is the primary request.
3 comments
Addressing concerns
Dear Reviewer QCJF,
Thank you again for your comments. As the discussion period is closing, we are looking forward to your comments on our rebuttal. We appreciate your valuable time and efforts in reviewing our work.
Some key takeaways from reviews and rebuttal:
- Directly measuring reward collapse: Rescoring identical saved trajectories shows that checkpoint rewards restore nonzero reward variance in 31–52% of groups where every rollout fails the final task, directly testing the proposed mechanism without changing tasks or trajectories.
- Auditing supervision reliability: Our manual audit finds instruction–target agreement in 93 of 100 accepted tasks, with all seven mismatches occurring in AppWorld; two of 41 AppWorld intermediate checkpoints were setup-only. More details in Appendix B.
- Alternative solutions and verification limitations: The diverse-solution study achieves all-pass@5 of 88.0% on generated tasks versus 87.1% on ALFWorld validation tasks, while we acknowledge that reference-state matching can restrict valid alternative goal states and does not explicitly penalize unwanted side effects.
- Reward for failed-final rollouts: The missing weighting factor in Equation 7 has been corrected: final success receives 1.0, while reward for a failed final task is bounded by 0.15 (even if all subtasks succeed).
Regards,
Authors
Thank you. Rescoring identical rollout groups addresses my reward-collapse concern, and the fixed-task-pool experiment further supports the contribution of checkpoint rewards. The clarification that failed-task rewards were capped at 0.15 in the experiments resolves my concern about Equation 7. The audit is also informative, although the AppWorld mismatches and lack of penalties for unwanted side effects remain important limitations. With these limitations stated clearly, my main concerns are substantially addressed. I remain at weak acceptance.
We thank reviewer QCJF for the careful assessment and for recognizing the value of grounded process supervision and independent benchmark evaluation. We address the reviewer's concerns below.
(1a) How does verification handle alternative solutions, incidental changes, and unwanted side effects?
Our verifier accepts different action sequences that reach the same checked goal state. Matching the proposer's reference can nevertheless be restrictive when a task admits valid solutions corresponding to different checkpoint states. In our targeted ALFWorld oracle study with GPT-5.5, over 70% of tasks had at least two distinct sampled action paths, while all-pass@5—the fraction of tasks for which all five attempts succeed—was 88.0% on generated tasks, compared with 87.1% on ALFWorld validation tasks using its own verifier (see our response to fXMZ for details).
(1b) Could the authors audit generated instructions and checkpoints, including setup-only or mismatched checkpoints and failed-final rollouts receiving reward 1?
We manually audited 100 accepted tasks and their checkpoints, split equally between early and late training. Instructions and targets agreed in 93 cases, with all seven mismatches in AppWorld:
AppWorld mismatches fell from 5/20 early to 2/20 late: early errors included drafts instead of sent emails and clauses retained after failed API calls; the two late cases overstated an operation's effect or retained an unwanted deletion. We add the audit and clarify the checkpoint examples in Appendix E and Figure 8. Paper revisions are highlighted in blue.
When looking at intermediate checkpoints in these tasks, all 89 checkpoints audited in ALFWorld and ScienceWorld were supported by their subtask descriptions. Among the 41 AppWorld checkpoints, two were setup-only (4.9%). Separately, seven AppWorld tasks had instruction–target mismatches; only one of those tasks also contained a mislabelled intermediate checkpoint.
Finally, the reviewer correctly identified a missing weight in Eq. 7: the corrected equation assigns success reward 1 and bounds failed-task checkpoint credit by 0.15. We correct Eq. 7 and Appendix B, as discussed in our response to reviewer 8Dtg.
(2) What fraction of GRPO groups has zero reward variance with and without subtask supervision, especially when every rollout fails?
On the same saved rollouts, checkpoint rewards restore nonzero variance in 31–52% of all-final-failure groups. We rescore all logged updates under binary final-success and fine-grained subtask rewards, using groups of 8 rollouts for the same task and determining final success with the generated verifier. A group counts as collapsed whenever all 8 rewards are equal, including equal positive rewards:
Because the tasks and trajectories are identical in both reward columns, this comparison isolates how checkpoint rewards distinguish different amounts of progress. It directly measures group collapse, whereas Figure 3 measures individual zero-reward rollouts. We add this analysis to Appendix E.