- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HERO alternates the optimization of a task-conditioned harness generator and an executor initialized from the same pretrained model. Execution outcomes provide preferences for generator DPO, while successful trajectories provide executor supervision after their inputs are reconstructed under a fixed deployment harness. The updated executor then supplies feedback for subsequent generator adaptation. With Qwen3.5-9B, the final executor achieves 97.01% success on ALFWorld's official unseen split and 45.60% on WebShop's test split without generator calls at deployment. Additional analyses examine generator adaptation, trajectory sources, and student performance under matched supervision budgets.
Critical strengths
-
The separation between experience collection and deployment is useful. Reconstructing trajectories collected under generated harnesses into supervision under a common deployment harness provides a practical mechanism for transferring training-time guidance into executor parameters. This permits deployment without additional generator calls and makes the deployed system's interface clear.
-
The reported executor performance is promising, and the supplementary analysis provides meaningful controls. Appendix D.1 matches the teacher-source comparison on 307 tasks, 3,696 action examples, 121,496 supervised tokens, and 462 updates. The authors also report uncertainty for paired comparisons and acknowledge inconclusive downstream differences. These details make the evidence more informative than the final success rates alone.
Critical weaknesses
-
The incremental downstream benefit of reciprocal optimization remains insufficiently established. In Table 3 and supplementary Table S10, students trained from the naive and optimized generators both achieve 93.28% unseen success; the optimized-generator student's advantage over the fixed-harness student is inconclusive. This matched comparison conditions on commonly solved tasks, so it does not capture all possible benefits from collection yield, but it also does not establish a DPO-specific advantage in student success. Generator improvements with a frozen executor provide useful evidence about collection-time behavior, while the central claim concerns the alternating training process. A matched multi-round self-training control is therefore important for attributing the final gains. The stage description also needs clarification: Section 3.5 describes E1 supervision from G0/E0, whereas Appendix B.2 and Table S5 identify G1/E0 as the WebShop E1 teacher. This prevents a reliable attribution of the first-round improvement from the main-text description alone.
-
The historical evaluation sequence and external comparisons do not fully establish comparability. The supplement identifies the 69.40% ALFWorld starting score as belonging to a separate early study, while the main sequence reports 68.66%. It also states that the main unseen E0 evaluation used the original configuration and E1–E3 used a later one. Thus, the difference between the starting scores has an explanation, but the full E0-to-E3 curve still combines evaluation configurations. External comparisons additionally involve different backbones or training budgets. The reported results demonstrate a promising pipeline, but they leave uncertainty about how much improvement is attributable specifically to reciprocal optimization. This unresolved attribution is the main reason for my weak-rejection recommendation.
Questions for authors
-
Please reconcile Section 3.5 with Appendix B.2 and Table S5: which generator/executor pair supplied the first executor-training data in each benchmark? For ALFWorld, what changed between the original and later evaluation configurations, and is an E0 result under the later configuration already available? A consistent configuration would make the training gains easier to interpret.
-
Is there an existing multi-round comparison against fixed-harness self-training or a frozen-generator variant with comparable collection and optimization budgets? If not, one limited comparison on a representative setting would be useful; I am not requesting a full matrix of new baselines and seeds. The matched supervision details in Appendix D.1 are already clear. For any such comparison, please distinguish the marginal collection costs reported in Table S13 from upstream generator optimization costs. Evidence that repeated generator adaptation improves downstream executor performance would affect much of my judgement.
2 comments
Thank you for the substantial follow-up. The common-configuration ALFWorld reevaluation resolves my concern about the E0–E3 comparison, and the benchmark-specific explanation clarifies which generator supplied the first-stage training data. I also appreciate the three-round self-training controls.
Those controls leave my central attribution concern unresolved. On the official WebShop test set, HERO reaches 42.6%, compared with 42.4% for fixed-H0 self-training and 41.4% for frozen-G0 self-training. The differences are statistically unresolved in this single-seed comparison. Meanwhile, HERO uses 9.22 core training GPU-hours versus 4.62 for fixed H0, and performance at equal total compute was not tested. The results demonstrate substantial improvement from iterative self-training, but do not yet establish an additional final-executor benefit from repeated generator adaptation.
Improvements in collection behavior and trajectory length remain useful findings. I encourage the paper to present those as the supported contribution and to qualify claims about downstream capability or cost efficiency from reciprocal optimization. For now, I am keeping my rating at 2.
Response to Reviewer gZ5S
Thank you for your careful review and for recognizing the fixed-deployment design and the matched-budget analysis. We appreciate your specific suggestions for a multi-round comparison and clearer training-stage descriptions; we address both, together with the cost and evaluation questions, below.
Multi-round controls
We added the requested three-round WebShop comparison with Qwen3.5-9B. HERO, fixed-H0 self-training, and frozen-G0 self-training share the initial checkpoint, 1,000 training tasks, four allocated training attempts per task per round, and 256 SFT updates per round. This follow-up uses a smaller training set than the original 2,000-task experiment. Each method learns from its own successful trajectories and previous-round executor. The final E3 models use H0 on the same 500 official test tasks.
HERO exceeds fixed H0 on development success by 3.1, 2.9, and 2.1 points across the three rounds. Final test success is 42.6% for HERO, 42.4% for fixed H0, and 41.4% for frozen G0, compared with 31.4% for the initial executor. The observed gain over frozen G0 is 1.2 points. These single-seed final-test differences remain statistically unresolved.
Collection and generator-training costs
Core training costs are 4.62, 6.07, and 9.22 GPU-hours for fixed H0, frozen G0, and HERO, including candidate, control, and teacher executions plus DPO and SFT. Including recorded serving, export, and evaluation intervals gives 8.16, 9.26, and 14.42 GPU-hours; shared setup, archival, and failed operations without complete timing records are excluded. Attempts and executor updates are matched, while teacher-data volume and compute differ. Equal-total-compute performance was not tested.
For the historical G-star generator, the two documented DPO stages cost approximately 2.71 and 2.17 GPU-hours. These upstream costs are reported separately from collection with the trained generator. They exclude trajectory collection and incompletely recorded exploration, so they are partial training costs.
First-stage teacher data
On ALFWorld, E1 learns from G0/E0 successes in the initial collection bank. On WebShop, E1 learns from fresh G1/E0 executions after DPO. Subsequent teachers are G2/E1 and G3/E2. We have prepared a correction to Section 3.5 that distinguishes these initializations. The new controlled HERO runs use a post-DPO teacher in every round.
ALFWorld reevaluation
The 69.40% score belongs to a separate preliminary study; the historical main sequence used 68.66% for E0 under earlier serving settings. We have now reevaluated unchanged E0–E3 under one configuration on all 134 official unseen tasks. Success is 67.91%, 89.55%, 95.52%, and 97.01%, a 29.10-point E0-to-E3 increase. Earlier evaluation results had already informed subsequent experimental choices.