acceptodds
Under review as a conference paper at ICLR 2027

ZeroHero: Training Small Language Agents from Scratch

Abstract

Large pretrained language models provide capable policies for interactive tasks, but how much model capacity is necessary for a specialized language agent? We introduce ZeroHero, a framework for training small language agents from scratch through expert-trajectory pretraining and adaptive post-training. A central challenge is deciding how to allocate training between expert supervision and the agent's own experience as its competence changes. We introduce Bandit-based Adaptive Training Selection (BATS), which uses a contextual bandit to select blocks of DAgger-style imitation learning or PPO updates based on measured learning progress. We study this approach under Reinforcement Learning with Oracle Dialogue (RLOD), where an agent combines environment actions with natural-language questions to an oracle that provides otherwise hidden task information. We construct dialogue-augmented ALFWorld household tasks and Craftax navigation tasks, and train 5–10M-parameter executors from random initialization, first on successful expert trajectories and then through online interaction. The expert supplies training supervision, while an LLM oracle remains available during execution. On ALFWorld, BATS achieves 89.6% success, compared with 67.9% for the expert, and outperforms fixed PPO, DAgger, and mixed-loss training from the same starting checkpoint. BATS also supports recovery from weak starting policies where pure PPO collapses. Across environments and starting policies, it selects different training strategies, ranging from predominantly reinforcement learning to sustained expert-guided imitation. These results show that compact language agents can surpass their demonstration expert and highlight the importance of adapting training regimes to the task and the learner's competence.

Primary area:
reinforcement learning
Keywords:
Small language models, oracle dialogue, imitation learning, reinforcement learning, contextual bandits, adaptive training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XN2aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how much model capacity is needed for a specialized language agent. It introduces ZeroHero, a framework that trains small LLM agents from scratch using successful expert demonstrations, followed by online imitation learning and reinforcement learning. To balance learning from expert supervision against learning from the agent’s own experience, the framework includes BATS, a controller that selects between blocks of DAgger and PPO updates. The paper also introduces dialogue-augmented versions of ALFWorld and Craftax, in which the agent can query a privileged LLM oracle for information. Experiments demonstrate improvements in both augmented environments.

Critical strengths

The results suggest that a model with roughly 5M parameters can achieve a high success rate in an interactive language environment. This is an interesting finding about the capacity required for specialized agents. The experiments also cover several useful dimensions: two environments, three initialization strengths, and three random seeds for the main small-model results. This breadth provides some confidence that the findings extend beyond a single experimental setting.

Critical weaknesses
  1. The practical motivation and scope are unclear. The paper focuses on training specialized agents from scratch, but it does not establish when this approach would be preferable to adapting a pretrained model. Concrete use cases involving both specialized pretraining and subsequent task-specific imitation learning or RL would strengthen the motivation. Likewise, the importance of optimizing training-budget allocation at this small scale needs clearer justification. If the intended value is to inform training at larger scales, an experiment that derives an allocation strategy at a small scale and validates it on a larger model would provide important evidence.

  2. The experiments do not clearly isolate BATS’s benefit as a compute-allocation method. A BATS training cycle is not a uniform unit of computation or interaction, so comparisons based on update counts do not establish compute efficiency. Although the paper acknowledges the optimization mismatch in mixed-loss training, comparisons under matched GPU time or estimated computation are needed to support the allocation claim. Figure 4 also appears consistent with simpler schedules: ALFWorld favors imitation early and PPO later, while Craftax relies primarily on imitation. Without comparisons against fixed schedules, such as allocating the first fifth of training to DAgger and the remainder to PPO, it is difficult to determine whether adaptive control is necessary.

Questions for authors
  1. Why are dialogue-augmented versions of ALFWorld and Craftax necessary for this study? Do the same qualitative trends hold in the original environments? What evidence separates the contribution of the training algorithm from the assistance provided by the privileged oracle?

  2. How are Gemma, the zero-shot Qwen models, and Qwen3-4B trained, where applicable, and evaluated? Do they use the same environment configurations, oracle access, interaction budgets, and evaluation protocols as BATS and the controlled baselines?

  3. Have the author compared BATS with fixed DAgger-to-PPO schedules and imitation-only training under matched compute and interaction budgets? Such results would materially affect my assessment of whether the adaptive controller provides a meaningful advantage.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

Thank you for recognizing the breadth of our evaluation and for asking how we account for training costs.

W1. Why train a compact agent? For a narrow task executed repeatedly, a specialized model can reduce the cost of rolling out a policy. On ALFWorld-Dialogue, our 5M-parameter executor achieved 89.6% test success from the strong checkpoint with a 50/50 DAgger-to-PPO schedule, compared with 88.3% for RL-adapted Qwen3-4B. Expert corrections after pretraining teach the agent what to do in unfamiliar states, and PPO uses task rewards to refine the policy. We have not tested whether a schedule learned at this scale transfers to a larger model.

W2 and Q3. Simple schedules and resource use. We compared imitation-only DAgger, a fixed halfway switch to PPO, a switch chosen retrospectively at a slowdown in DAgger validation performance, and BATS. At equal training-cycle counts on ALFWorld-Dialogue, the slowdown switch led from the weak initialization (80.6% test success), the halfway switch led from medium (88.8%), and BATS led from strong (91.0%). From strong, the halfway switch achieved 89.6% and DAgger alone achieved 73.1%. BATS used about 19k expert labels and 20 active hours, versus 140k labels and 24 hours for the halfway switch. It also used more agent decisions (1.8M versus 1.3M). We can quantify the saving in expert labels, but these runs do not establish an advantage at matched compute and interaction budgets.

Q1. Oracle access. Dialogue lets agents ask for help when an observation is ambiguous. Within each environment, all methods could query the same oracle under the same rules. We also evaluated agents without dialogue. In Crafter, BATS and PPO reached similar mean achievement counts (7.44 and 7.39). In WebShop, PPO had the highest purchase success and DAgger the highest mean reward. These experiments establish that the pipeline works without dialogue. They do not isolate the oracle’s contribution within a given task.

Q2. Gemma and Qwen evaluation. Each agent received the same type of public observation and interaction history, could act or ask a question, and had the same per-episode budget. We ran one episode per held-out task with the same success criterion. Only the oracle saw private task information, including when Gemma acted as the expert. Gemma 4 26B-A4B and zero-shot Qwen3-4B and Qwen3-8B received no task-specific training. The separate Qwen3-4B (GiGPO) row reports the RL-fine-tuned agent. Appendix C gives the model-specific decoding settings.

Reviewer_XN2aOpenReview ↗

Response

Thank you for clarifying the baseline evaluation protocols and adding the requested fixed-schedule comparisons. These address my questions about baseline comparability and the missing schedule baselines.

However, my concerns about BATS’s added value remain. The fixed halfway schedule outperforms BATS from weak and medium initializations, while BATS’s advantage from the strong initialization is small and based on a single seed (Table 2). The expert-label savings are useful, but the comparisons do not establish an efficiency advantage under matched compute and interaction budgets.

The experiments without dialogue broaden the evaluation but do not isolate oracle assistance in the original tasks or show a consistent advantage for BATS. I also retain reservations about the practical benefits of training these agents from scratch. Given these remaining concerns about the contribution, efficiency, and generality, I will therefore maintain my score.

Reviewer_kexTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work focuses on the problem of training small language agents from scratch. The author proposed ZeroHero, which is a framework for training small language agents from scratch through expert trajectory pre training and adaptive post training. This work was validated on ALFWorld with a success rate of 89.6%, compared to 67.9% for experts. The main contribution of this work is to propose a framework that combines expert trajectory pre training with adaptive post training of small language agents, while using BATS and RLOD to achieve SLM performance that surpasses expert LLM.

Critical strengths

This work focuses on how to train SLM from scratch and propose a complete training framework. Compared to some tricks in the post training stage, this problem is more fundamental and has certain research value and uniqueness.

Critical weaknesses

At present, the experiment is being conducted in a relatively structured environment, and it is uncertain whether this method will still work in a more complex and open environment.

Questions for authors

Have you attempted to evaluate the effectiveness of this method in more complex open environments?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Comments

Thank you for recognizing the value of training compact agents from scratch and for asking how they perform in more open environments.

We evaluated our 5M-parameter agent in two settings without a dialogue oracle.

Crafter. The agent explores a procedurally generated world, gathers resources, and crafts tools. The pretrained policy averaged 6.13 distinct achievements per episode on 100 held-out worlds. After 150 post-training iterations, mean achievement counts on the development evaluation were 6.89 for DAgger, 7.39 for PPO, and 7.44 for BATS.

WebShop. The agent searches a catalog and purchases a product that meets a customer’s request. We evaluated all four policies on the same held-out tasks. After 100 iterations, BATS improved full-purchase success from 32.6% to 34.9% and mean reward from 0.747 to 0.782. PPO achieved the highest full-purchase success (43.3%), while DAgger achieved the highest mean reward (0.811), which also credits partially matching purchases.

The results show that the training pipeline extends to both tasks. BATS and PPO performed similarly in Crafter, and the choice of method in WebShop depended on which outcome we measured. The general response reports the results for both environments.

Reviewer_GTQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ZeroHero, a framework for training small, task-specific language agents from scratch using expert demonstrations and adaptive post-training. The central methodological contribution is BATS, which employs a contextual bandit to dynamically select between DAgger-style imitation learning and PPO-based reinforcement learning according to the agent's current behavioral statistics and measured learning progress. The authors also introduce RLOD, a setting in which an agent can interact with the environment and query a privileged LLM oracle for hidden task information. Experiments are conducted on dialogue-augmented versions of ALFWorld and Craftax, using 5M–10M parameter language models initialized from scratch. The reported results suggest that BATS can outperform fixed IL, RL, and mixed-loss baselines under certain initializations. In particular, it achieves 89.6% success on ALFWorld-Dialogue, outperforming the demonstration expert.

Critical strengths
  1. An interesting and practically relevant problem. Training compact, task-specific agents from scratch is a meaningful research direction.
  2. Promising empirical results in certain settings. The results on ALFWorld-Dialogue are encouraging. BATS achieves 89.6% success from a strong initialization, compared with 74.4% for PPO and 73.9% for DAgger.
Critical weaknesses
  1. My primary concern is that the central motivation behind BATS is not new. For example, ADVISOR in NeurIPS 2021 explicitly studies how to dynamically balance imitation and reinforcement learning according to the learner's observations. It adaptively weights the IL and RL objectives during training, allowing the policy to repeatedly shift between imitation and exploration rather than following a fixed training schedule. Although BATS differs in its implementation, i.e., selecting complete optimization regimes at the training-block level instead of continuously adjusting loss weights at the observation level, the underlying principle of adaptive IL/RL allocation is closely related.

  2. The contextual-bandit formulation is insufficiently justified, particularly regarding optimization stability and non-stationarity. The IL and RL's effects on the shared policy are highly non-stationary: the benefit of either operator depends on the entire preceding optimization trajectory. However, the LinUCB controller estimates the utility of each operator using short-term changes in success rate. Such measurements can be noisy and may not reliably reflect the long-term benefit of a particular optimization regime.

  3. Missing baselines and confounded comparisons make it difficult to attribute improvements specifically to BATS. An important missing comparison is against simple but carefully tuned training schedules, such as DAgger followed by PPO, periodic alternation between IL and RL, or switching based on a threshold in validation performance or learning progress.

Reference

[1] Bridging the Imitation Gap by Adaptive Insubordination. NeurIPS 2021.

Questions for authors

Please see the weakness part above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

Thank you for recognizing the value of training compact agents from scratch and for pointing us to ADVISOR.

W1. Relation to ADVISOR

ADVISOR [1] and BATS both adapt how much the learner relies on imitation and reinforcement learning. ADVISOR uses an auxiliary policy to estimate how well the learner can predict the expert’s action from a given observation. That estimate sets the relative weights of the imitation and RL losses for that observation. BATS makes one decision for a whole training block: it selects DAgger or PPO based on recent policy progress. A PPO block can therefore proceed without requesting new expert action labels. We are running a direct comparison with ADVISOR and will report the results when the experiments are complete.

W2. What the controller can infer from one block

We use LinUCB because it gives us a simple way to choose between IL and RL as training progresses, without fixing a switch point in advance. It uses the policy’s current behavior to estimate the immediate gain from each operator and gives an exploration bonus to the operator whose gain is more uncertain in that context. We agree that this signal can miss delayed benefits, such as an IL update that makes later RL updates more effective. Our experiments show that the resulting schedules can improve final held-out success over either operator alone in some settings. How reliably a single block’s gain predicts the better long-term choice remains an open question.

W3. Comparison with simpler schedules

We compared BATS with imitation-only DAgger, a fixed halfway switch from DAgger to PPO, and a switch selected retrospectively from a slowdown in DAgger validation performance. With the same number of training cycles on ALFWorld-Dialogue, the slowdown switch gave the highest test success from the weak initialization (80.6%), the halfway switch led from the medium initialization (88.8%), and BATS led from the strong initialization (91.0%, versus 89.6% for the halfway switch).

[1] Bridging the Imitation Gap by Adaptive Insubordination. NeurIPS 2021.

AuthorsOpenReview ↗

Additional ADVISOR experiments

W1 follow-up. ADVISOR on ALFWorld-Dialogue

Following up on the comparison promised in our earlier reply, we evaluated ADVISOR from the same weak, medium, and strong initializations as our baselines. The table reports validation success at training cycle 300 on the same games. Each entry is a single run. The comparison matches training cycles, while rollout and optimization budgets vary by method.

MethodWeakMediumStrong
ADVISOR21.7%53.3%76.7%
BATS73.3%78.3%91.7%
DAgger70.0%80.0%80.0%
PPO0.0%63.3%88.3%
Mixed loss0.0%66.7%90.0%

We adapted ADVISOR by treating each complete text command as one action and using the same expert-query budget and state selection as DAgger. With a single expert target and alpha=1alpha=1, ADVISOR’s KL-based imitation weight ww equals the auxiliary policy’s probability of the full expert command, including EOS. The main policy weights imitation by ww and PPO by 1−w1-w.

The gap between BATS and ADVISOR was largest for the weak initialization. ADVISOR was motivated by settings where the expert has privileged information. Here, expert model sees the same public observation as the learner. A weak learner may therefore assign low probability to a useful command even when the observation provides enough information to infer it. This would downweight a correction the learner could benefit from. Consistent with this interpretation, the mean imitation weight on labeled decisions through cycle 300 was 8.0% for weak, 10.2% for medium, and 13.8% for strong initialization.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewers: Addressing General Concerns

We sincerely thank all reviewers for their time and constructive feedback. We particularly appreciate their recognition of the importance and research value of training compact, task-specific language agents from scratch (@GTQJ, @kexT), the promising empirical results demonstrating the capabilities of small language models (@GTQJ, @XN2a), and the breadth of our experimental evaluation (@XN2a).

Two questions came up across the reviews. First, how well does ZeroHero work in more complex environments without a dialogue oracle? Second, how does BATS compare with simpler schedules for switching between imitation learning and reinforcement learning? We conducted additional experiments to address both questions.

1. Generalization to non-dialogue environments (@kexT, @XN2a)

We evaluated ZeroHero in Crafter and WebShop. In Crafter, the agent explores a procedurally generated world, gathers resources, and crafts tools. In WebShop, it searches a catalog and purchases a product that satisfies a customer’s request. In both environments, the agent acts without access to a dialogue oracle.

Crafter. We pretrained a randomly initialized 5M-parameter agent on reasoning and action traces generated by Qwen3.8-27B. The training set contained 20,949 examples from 96 worlds. The epoch-12 checkpoint averaged 6.13 distinct achievements per episode on 100 held-out worlds. We initialized DAgger, PPO, and BATS from this checkpoint and report their development scores after 150 training iterations.

WebShop. We pretrained a randomly initialized 5M-parameter agent on reasoning and action traces generated by GiGPO-trained Qwen2.5-7B expert. The training set contained 22,005 steps across 4,068 instructions. We report pretrained checkpoint and its performance after 100 iterations of DAgger, PPO, or BATS. All four policies were evaluated on the same held-out tasks. Success requires a purchase with reward 1; mean reward also credits purchases that satisfy some of the requested attributes.

PolicyCrafter mean achievementsWebShop success rateWebShop mean reward
Zero-shot Qwen3.8-27B11.8036.0%0.466
ZeroHero after pretraining6.1332.6%0.747
+ DAgger6.8942.9%0.811
+ PPO7.3943.3%0.780
+ BATS7.4434.9%0.782

All three post-training methods improved on their starting policies in both environments. In Crafter, PPO and BATS reached similar achievement counts. In WebShop, PPO had the highest full-purchase success rate, while DAgger had the highest mean reward. These results extend the ZeroHero pipeline to tasks where the agent learns entirely through demonstrations and interaction with the environment.

2. Comparison with simpler training schedules (@GTQJ, @XN2a)

We compared BATS with imitation-only DAgger and two schedules that switch once from DAgger to PPO. The fixed schedule switches halfway through training. The slowdown schedule uses the DAgger validation curve to select a switch point retrospectively. The table reports ALFWorld-Dialogue test success after the same number of training cycles.

Post-training methodWeak initializationMedium initializationStrong initialization
DAgger only71.6%76.9%73.1%
Fixed DAgger -> PPO, 50/5079.1%88.8%89.6%
DAgger -> PPO at validation slowdown80.6%82.8%85.1%
BATS71.6%73.1%91.0%

A fixed schedule can work well, but the preferred switch point depends on the environment and the starting policy. On ALFWorld-Dialogue, switching at the slowdown point worked better for Weak, while the halfway switch worked better for Medium and Strong. The trade-off is practical: switching a weak policy to PPO before it can earn reward leaves little signal for learning. Indeed, PPO alone from the weak ALFWorld initialization failed to solve any test tasks. Conversely, delaying PPO when a stronger policy already earns reward leaves fewer updates to learn from successful trajectories.

BATS addresses this choice one block at a time. It uses measured learning progress to select the next update and can return to imitation later. It makes no claim to identify an optimal switching point. Our broader contribution is ZeroHero, a pipeline for training small language agents from scratch through expert-trajectory pretraining followed by interaction with the environment. BATS gives that pipeline an adaptive way to allocate post-training when a suitable schedule is unknown in advance.

We will incorporate these experiments, their evaluation protocols, and the comparisons with simpler schedules into the revised paper.

AuthorsOpenReview ↗

Additional BATS arguments

  1. When does adaptive scheduling help over fixed alternation? (@GTQJ, @XN2a)

Both reviewers raised the possibility that a simple schedule could explain the gains we observe. Our ALFWorld comparison above examines how well it works to start with DAgger and switch once to PPO. Periodic alternation raises a further question. If expert supervision becomes useful several times during training, is it enough to return to the expert at fixed intervals, or does it help to choose those intervals from the learner’s behavior?

We designed SkillLadder, a controlled tabular task, to study this question. The agent must complete eight skills in sequence, each requiring four correct actions. In the default setting, the expert knows three steps of each skill and guesses between two actions on the remaining step. DAgger can teach the learner enough to attempt a new skill, and PPO can use reward to resolve the step on which the expert is unreliable. Once the learner passes that skill consistently, it reaches the next one more often. The expert can now correct mistakes in states the learner rarely visited during its initial imitation phase.

We tested two settings. In the default setting, the expert chooses the correct action on its uncertain step 50% of the time for every skill. In the uneven setting, that probability falls to 3% for every second skill. We carried both the tuned 2:8 schedule and the BATS controller settings directly from the first setting to the second.

Expert settingTraining scheduleFinal mean successSeeds sustaining ≥90% successMean expert labels
DefaultBest of five single switches (160 DAgger cycles, then PPO)64.1%0/205,999
DefaultRandom choice of operator every two cycles98.7%20/201,729
DefaultFixed alternation (2 DAgger cycles, then 8 PPO cycles, repeated)99.4%20/20592
DefaultBATS99.2%20/20966
UnevenSame 2:8 alternation81.3%3/107,196
UnevenBATS98.8%10/105,033

Mean success averages the probabilities of completing the first one, two, …, eight skills. We measure time to sustained 90% success from the first cycle at which mean success reaches 90% and stays at or above it through the end of the run. In each training cycle, the learner collects trajectories and updates its policy. We matched cycle counts within each setting (400 for default, 800 for uneven) and counted expert labels over the full run. Compute and interaction costs can differ across methods.

A tuned periodic schedule matched BATS on the default ladder. The 2:8 schedule reached sustained 90% success in a median of 104 cycles, compared with 106 for BATS and 163 for random selection. Before reaching that threshold, BATS selected a median of eight distinct DAgger phases per run. The longest uninterrupted DAgger phase lasted a median of 14 cycles, or seven consecutive controller decisions. BATS therefore returned to imitation repeatedly and sometimes stayed with it for several blocks. A fixed ratio tuned for this setting achieved comparable success with fewer labels.

The 2:8 schedule transferred poorly when expert reliability varied across skills. It sustained 90% success on 3/10 seeds, compared with 10/10 for BATS using the same controller settings. A fixed mixed loss with β = 0.3 also worked well here. It reached sustained 90% success in a median of 512 cycles, close to 506 for BATS. Over the full 800-cycle runs, it used an average of 34,145 expert labels, compared with 5,033 for BATS. These results support choosing when to return to imitation from observed learning progress when a suitable fixed schedule is unknown in advance.

AuthorsOpenReview ↗

Revised paper

Thank you for your feedback. We have uploaded a revised paper incorporating the experiments discussed in our responses. Table 2 compares BATS with two schedules that switch once from DAgger to PPO. Appendix G.1 reports the comparison with ADVISOR (Table 6), and Appendix G.2 presents the Crafter and WebShop experiments without a dialogue oracle (Tables 7–8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.