acceptodds
Under review as a conference paper at ICLR 2027

HERO: Agent Self-Improvement through Harness–Executor Reciprocal Optimization

Abstract

The ability to learn from experience and continually improve is fundamental to intelligence. We introduce HERO, an agent self-improvement framework that adapts experience collection alongside executor learning. A task-conditioned harness generator and an executor, initialized from the same pretrained model, learn through their own environment interactions. Direct preference optimization trains the generator using preferences derived from verified execution outcomes. Successful action trajectories provide supervision for executor fine-tuning, with student inputs reconstructed under a fixed deployment harness. The updated executor then supplies feedback for subsequent generator adaptation. This alternating process uses self-generated experience without additional teacher-provided solutions or critiques. We evaluate HERO with Qwen3.5-9B on ALFWorld and WebShop. On ALFWorld, the final executor reaches 97.0% success on the official unseen split. On WebShop, three consecutive DPO–SFT stages improve success on the official 500-task test split from 31.2% to 45.6%. The final executors achieve these results under fixed deployment harnesses, requiring no extra generator calls. Analysis shows that the trained generator yields more clean successful trajectories and shorter average rollouts than its untrained counterpart. These findings highlight HERO as a way to shape the experience from which an executor learns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Self-Improvement, Harness Optimization, Direct Preference Optimization, Self-Training, Co-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gZ5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HERO alternates the optimization of a task-conditioned harness generator and an executor initialized from the same pretrained model. Execution outcomes provide preferences for generator DPO, while successful trajectories provide executor supervision after their inputs are reconstructed under a fixed deployment harness. The updated executor then supplies feedback for subsequent generator adaptation. With Qwen3.5-9B, the final executor achieves 97.01% success on ALFWorld's official unseen split and 45.60% on WebShop's test split without generator calls at deployment. Additional analyses examine generator adaptation, trajectory sources, and student performance under matched supervision budgets.

Critical strengths
  1. The separation between experience collection and deployment is useful. Reconstructing trajectories collected under generated harnesses into supervision under a common deployment harness provides a practical mechanism for transferring training-time guidance into executor parameters. This permits deployment without additional generator calls and makes the deployed system's interface clear.

  2. The reported executor performance is promising, and the supplementary analysis provides meaningful controls. Appendix D.1 matches the teacher-source comparison on 307 tasks, 3,696 action examples, 121,496 supervised tokens, and 462 updates. The authors also report uncertainty for paired comparisons and acknowledge inconclusive downstream differences. These details make the evidence more informative than the final success rates alone.

Critical weaknesses
  1. The incremental downstream benefit of reciprocal optimization remains insufficiently established. In Table 3 and supplementary Table S10, students trained from the naive and optimized generators both achieve 93.28% unseen success; the optimized-generator student's advantage over the fixed-harness student is inconclusive. This matched comparison conditions on commonly solved tasks, so it does not capture all possible benefits from collection yield, but it also does not establish a DPO-specific advantage in student success. Generator improvements with a frozen executor provide useful evidence about collection-time behavior, while the central claim concerns the alternating training process. A matched multi-round self-training control is therefore important for attributing the final gains. The stage description also needs clarification: Section 3.5 describes E1 supervision from G0/E0, whereas Appendix B.2 and Table S5 identify G1/E0 as the WebShop E1 teacher. This prevents a reliable attribution of the first-round improvement from the main-text description alone.

  2. The historical evaluation sequence and external comparisons do not fully establish comparability. The supplement identifies the 69.40% ALFWorld starting score as belonging to a separate early study, while the main sequence reports 68.66%. It also states that the main unseen E0 evaluation used the original configuration and E1–E3 used a later one. Thus, the difference between the starting scores has an explanation, but the full E0-to-E3 curve still combines evaluation configurations. External comparisons additionally involve different backbones or training budgets. The reported results demonstrate a promising pipeline, but they leave uncertainty about how much improvement is attributable specifically to reciprocal optimization. This unresolved attribution is the main reason for my weak-rejection recommendation.

Questions for authors
  1. Please reconcile Section 3.5 with Appendix B.2 and Table S5: which generator/executor pair supplied the first executor-training data in each benchmark? For ALFWorld, what changed between the original and later evaluation configurations, and is an E0 result under the later configuration already available? A consistent configuration would make the training gains easier to interpret.

  2. Is there an existing multi-round comparison against fixed-harness self-training or a frozen-generator variant with comparable collection and optimization budgets? If not, one limited comparison on a representative setting would be useful; I am not requesting a full matrix of new baselines and seeds. The matched supervision details in Appendix D.1 are already clear. For any such comparison, please distinguish the marginal collection costs reported in Table S13 from upstream generator optimization costs. Evidence that repeated generator adaptation improves downstream executor performance would affect much of my judgement.

2 comments
AuthorsOpenReview ↗

Response to Reviewer gZ5S

Thank you for your careful review and for recognizing the fixed-deployment design and the matched-budget analysis. We appreciate your specific suggestions for a multi-round comparison and clearer training-stage descriptions; we address both, together with the cost and evaluation questions, below.

Multi-round controls

We added the requested three-round WebShop comparison with Qwen3.5-9B. HERO, fixed-H0 self-training, and frozen-G0 self-training share the initial checkpoint, 1,000 training tasks, four allocated training attempts per task per round, and 256 SFT updates per round. This follow-up uses a smaller training set than the original 2,000-task experiment. Each method learns from its own successful trajectories and previous-round executor. The final E3 models use H0 on the same 500 official test tasks.

HERO exceeds fixed H0 on development success by 3.1, 2.9, and 2.1 points across the three rounds. Final test success is 42.6% for HERO, 42.4% for fixed H0, and 41.4% for frozen G0, compared with 31.4% for the initial executor. The observed gain over frozen G0 is 1.2 points. These single-seed final-test differences remain statistically unresolved.

Collection and generator-training costs

Core training costs are 4.62, 6.07, and 9.22 GPU-hours for fixed H0, frozen G0, and HERO, including candidate, control, and teacher executions plus DPO and SFT. Including recorded serving, export, and evaluation intervals gives 8.16, 9.26, and 14.42 GPU-hours; shared setup, archival, and failed operations without complete timing records are excluded. Attempts and executor updates are matched, while teacher-data volume and compute differ. Equal-total-compute performance was not tested.

For the historical G-star generator, the two documented DPO stages cost approximately 2.71 and 2.17 GPU-hours. These upstream costs are reported separately from collection with the trained generator. They exclude trajectory collection and incompletely recorded exploration, so they are partial training costs.

First-stage teacher data

On ALFWorld, E1 learns from G0/E0 successes in the initial collection bank. On WebShop, E1 learns from fresh G1/E0 executions after DPO. Subsequent teachers are G2/E1 and G3/E2. We have prepared a correction to Section 3.5 that distinguishes these initializations. The new controlled HERO runs use a post-DPO teacher in every round.

ALFWorld reevaluation

The 69.40% score belongs to a separate preliminary study; the historical main sequence used 68.66% for E0 under earlier serving settings. We have now reevaluated unchanged E0–E3 under one configuration on all 134 official unseen tasks. Success is 67.91%, 89.55%, 95.52%, and 97.01%, a 29.10-point E0-to-E3 increase. Earlier evaluation results had already informed subsequent experimental choices.

Thank you for the substantial follow-up. The common-configuration ALFWorld reevaluation resolves my concern about the E0–E3 comparison, and the benchmark-specific explanation clarifies which generator supplied the first-stage training data. I also appreciate the three-round self-training controls.

Those controls leave my central attribution concern unresolved. On the official WebShop test set, HERO reaches 42.6%, compared with 42.4% for fixed-H0 self-training and 41.4% for frozen-G0 self-training. The differences are statistically unresolved in this single-seed comparison. Meanwhile, HERO uses 9.22 core training GPU-hours versus 4.62 for fixed H0, and performance at equal total compute was not tested. The results demonstrate substantial improvement from iterative self-training, but do not yet establish an additional final-executor benefit from repeated generator adaptation.

Improvements in collection behavior and trajectory length remain useful findings. I encourage the paper to present those as the supported contribution and to qualify claims about downstream capability or cost efficiency from reciprocal optimization. For now, I am keeping my rating at 2.

Reviewer_6fiuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HERO alternates DPO training of a harness generator with SFT of an executor on successful trajectories reconstructed under a fixed deployment interface. Improving the learning value of self-generated experience is a worthwhile goal, and the results support executor improvements and some reduction in trajectory length. However, the added benefit of generator training over ordinary multi-round self-training remains to be isolated. Experiments use ALFWorld and WebShop, with additional comparisons of trajectory sources and student outcomes.

Critical strengths
  1. Section 3.4 reconstructs action inputs under the deployment interface and removes the generated prompts. This allows the executor to use its learned capabilities through a fixed interface without additional generator calls.

  2. Table 1 reports success rates of 97.01% on ALFWorld unseen tasks and 45.60% on WebShop test tasks, demonstrating practically meaningful executor improvements from self-generated interaction data.

  3. With student-training budgets matched in Table 3, the optimized generator reduces the student's mean execution length from 17.06 to 15.67 steps. The accompanying analysis on jointly solved tasks provides useful evidence of efficiency gains beyond success rate alone.

Critical weaknesses
  1. The contribution of reciprocal optimization is not isolated from ordinary multi-round self-training. In Table 3, students trained from optimized and untrained generators both achieve 93.28% success, despite shorter trajectories with the optimized generator. There is no multi-round fixed-harness control matched for total environment interactions and training, and the costs of candidate-harness executions, control executions, and DPO/SFT stages are not reported. The current evidence therefore does not establish the additional capability or cost-efficiency benefit of training the generator.

  2. Figure 3 uses the original ALFWorld evaluation configuration for E0 and a later configuration for E1-E3. The improvement from the initial to the final executor therefore cannot be attributed entirely to learning.

  3. Figure 3 reports one evaluation per task with seed 42. Given the 3% point WebShop advantage over Agent SFT and the small student differences in Table 3, the lack of independent training runs and paired uncertainty estimates limits the strength of these comparisons.

Questions for authors
  1. Starting from the same model and matching total environment interactions and SFT updates, what final success rates do multi-round fixed-H0 self-training, self-training with G0 kept frozen throughout, and full HERO achieve?

  2. What are the total interaction and compute costs, including candidate-harness executions, control executions, and generator training? Does the advantage persist at a fixed total cost?

  3. Can you reevaluate E0-E3 under the same ALFWorld configuration and explain the differences between the original and later evaluation settings?

  4. Can you report multiple independent training runs and paired task-level results to assess the stability of the WebShop advantage and the student differences in Table 3?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6fiu

Thank you for your thoughtful review and for highlighting the value of context reconstruction and the observed executor improvements. We appreciate your suggestions on multi-round controls, cost accounting, and uncertainty, which guided the additional experiments and analyses below.

Multi-round controls

We added three-round WebShop controls for fixed-H0 and frozen-G0 self-training. Together with HERO, they start from the same Qwen3.5-9B checkpoint and use the same 1,000 training tasks, four allocated training attempts per task per round, and 256 SFT updates per round with global batch size 16. This follow-up uses a smaller training set than the original 2,000-task experiment. Each method learns from its own successful trajectories and previous-round executor. Final evaluation uses E3 under H0 on the same 500 official test tasks.

HERO leads fixed H0 on development success by 3.1, 2.9, and 2.1 points across the three rounds. Final development success is 40.6% for HERO, 38.5% for fixed H0, and 41.6% for frozen G0. Final test success is 42.6%, 42.4%, and 41.4%, respectively, compared with 31.4% for the initial executor. The observed test gain over frozen G0 is 1.2 points. These controls directly compare the complete training procedures under a shared attempt and update budget.

Trajectory efficiency

The earlier matched-data analysis provides complementary evidence about execution efficiency. Naive- and DPO-source students both achieve 93.28% unseen success. On the 119 tasks solved by both the DPO-source and fixed-H0-source students, the former averages 12.54 environment steps versus 13.75, an 8.80% reduction. This retrospective comparison uses H0 for both students. We report this efficiency change separately from success-rate differences.

Interaction and compute costs

Core training costs, including candidate, control, and teacher executions, DPO, and SFT, are 4.62, 6.07, and 9.22 GPU-hours for fixed H0, frozen G0, and HERO. Including recorded serving, export, and evaluation intervals gives 8.16, 9.26, and 14.42 GPU-hours, respectively; these exclude shared setup, archival, and failed operations without complete timing records. Attempts and SFT updates are matched; teacher-data yield and GPU time differ. Performance at equal total GPU-hours was not tested.

Consistent ALFWorld evaluation

We reevaluated the unchanged E0–E3 checkpoints on all 134 official unseen tasks under the same serving configuration. Success is 67.91%, 89.55%, 95.52%, and 97.01%, a 29.10-point E0-to-E3 increase. The earlier E0 score used different serving settings; this reevaluation removes that mismatch. Earlier evaluation results had already informed subsequent experimental choices.

Paired evaluation

We assessed the new comparisons using paired bootstrap over instruction groups. The final-test differences remain statistically unresolved. Each condition uses one training seed, so this analysis quantifies task-group uncertainty for the evaluated checkpoints, not variation across independent training runs.

The core concern remains unresolved, so I maintain my rating of 2

Reviewer_YecDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HERO trains two LoRA adapters on Qwen3.5-9B. The first is a task-conditioned harness generator trained with DPO on success/failure preferences collected from a frozen executor. The second is an executor trained with SFT on successful trajectories, rebuilt under the fixed deployment harness H0. The two are updated in turn for 3 rounds, with no teacher model. The executor improves from 68.66 to 97.01 on ALFWorld (unseen split) and from 31.20 to 45.60 on WebShop, with no generator calls at deployment (Table 1).

Critical strengths

The deployment design is practical. Rebuilding the training contexts under H0 (Eq. 7) means all gains carry over to a fixed harness at no extra inference cost, and no external teacher is used. This makes the absolute numbers notable for a 9B model.

Critical weaknesses

The paper does not show that the learned generator drives the improvement. There is no matched baseline of iterative rejection-sampling SFT under H0 (in the style of ReST or AgentEvol).

The framework is only tested on Qwen3.5-9B, and it is not clear how the framework performs for smaller model, or larger model.

Questions for authors

On ALFWorld, what data guided the configuration, number of rounds and hyperparameters? Why was E0 evaluated under a different configuration?

How much does the ALFWorld-specific lexical filter contribute, and would it work in other environments?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YecD

Thank you for your careful review and for recognizing the practical value of fixed-harness deployment without an external teacher. Your questions motivated the additional self-training controls and the clarifications on evaluation settings, model size, and the lexical filter below.

Multi-round self-training controls

We added the requested three-round comparison on WebShop. HERO, fixed-H0 self-training, and frozen-G0 self-training start from the same Qwen3.5-9B checkpoint and use the same 1,000 training tasks, four allocated training attempts per task per round, and 256 executor SFT updates per round. This follow-up uses a smaller training set than the original 2,000-task experiment. Each method learns from its own successful trajectories and continues from its own previous-round executor. All final evaluations use H0 on the same 500 official test tasks.

HERO exceeds fixed-H0 self-training on development success in all three rounds, by 3.1, 2.9, and 2.1 percentage points. Final test success is 42.6% for HERO, 42.4% for fixed H0, and 41.4% for frozen G0, compared with 31.4% for the initial executor. The observed gain over frozen G0 is 1.2 points. The final-test differences remain statistically unresolved in this single-seed comparison.

ALFWorld evaluation and configuration selection

The common-configuration reevaluation confirms the overall executor improvement. Keeping E0–E3 unchanged, we evaluated all four checkpoints on the 134 official unseen tasks with the same serving settings: success is 67.91%, 89.55%, 95.52%, and 97.01%, a 29.10-point increase from E0 to E3. The original E0 score came from an earlier serving configuration; the new comparison removes that mismatch.

The historical sequence was exploratory: earlier evaluations informed subsequent settings and the decision to add a third round. Generator selection used Development90 success, then input-token cost and the earlier checkpoint to break ties. The third round's budget and final checkpoint were fixed before its collection. We will document this selection history explicitly; the reevaluation does not change the prior exposure to evaluation results.

Model size

We also completed the three-round comparison with Qwen3.5-4B. In that setting, HERO did not exceed the self-training controls, so the observed 9B ranking does not generalize across both tested model sizes. Larger models have not been evaluated.

Lexical filter

The ALFWorld filter screens generated guidance for task-inconsistent transformation and appliance terms. It flagged 73 of 3,491 and 432 of 3,310 successful generated candidates in two collections. These counts measure screening frequency; an ablation would be needed to quantify its effect on performance. WebShop does not use this filter, and we have not tested its transfer to other environments.

Reviewer_YecDOpenReview ↗

Further comments on the reply

Thanks for the authors' replies and response.

Regarding the model size, it was mentioned that "We also completed the three-round comparison with Qwen3.5-4B. In that setting, HERO did not exceed the self-training controls, so the observed 9B ranking does not generalize across both tested model sizes. Larger models have not been evaluated." Where are the results, and why the 9B ranking does not generalize across both tested model size? Why are larger models not evaluated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.