- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how much model capacity is needed for a specialized language agent. It introduces ZeroHero, a framework that trains small LLM agents from scratch using successful expert demonstrations, followed by online imitation learning and reinforcement learning. To balance learning from expert supervision against learning from the agent’s own experience, the framework includes BATS, a controller that selects between blocks of DAgger and PPO updates. The paper also introduces dialogue-augmented versions of ALFWorld and Craftax, in which the agent can query a privileged LLM oracle for information. Experiments demonstrate improvements in both augmented environments.
Critical strengths
The results suggest that a model with roughly 5M parameters can achieve a high success rate in an interactive language environment. This is an interesting finding about the capacity required for specialized agents. The experiments also cover several useful dimensions: two environments, three initialization strengths, and three random seeds for the main small-model results. This breadth provides some confidence that the findings extend beyond a single experimental setting.
Critical weaknesses
-
The practical motivation and scope are unclear. The paper focuses on training specialized agents from scratch, but it does not establish when this approach would be preferable to adapting a pretrained model. Concrete use cases involving both specialized pretraining and subsequent task-specific imitation learning or RL would strengthen the motivation. Likewise, the importance of optimizing training-budget allocation at this small scale needs clearer justification. If the intended value is to inform training at larger scales, an experiment that derives an allocation strategy at a small scale and validates it on a larger model would provide important evidence.
-
The experiments do not clearly isolate BATS’s benefit as a compute-allocation method. A BATS training cycle is not a uniform unit of computation or interaction, so comparisons based on update counts do not establish compute efficiency. Although the paper acknowledges the optimization mismatch in mixed-loss training, comparisons under matched GPU time or estimated computation are needed to support the allocation claim. Figure 4 also appears consistent with simpler schedules: ALFWorld favors imitation early and PPO later, while Craftax relies primarily on imitation. Without comparisons against fixed schedules, such as allocating the first fifth of training to DAgger and the remainder to PPO, it is difficult to determine whether adaptive control is necessary.
Questions for authors
-
Why are dialogue-augmented versions of ALFWorld and Craftax necessary for this study? Do the same qualitative trends hold in the original environments? What evidence separates the contribution of the training algorithm from the assistance provided by the privileged oracle?
-
How are Gemma, the zero-shot Qwen models, and Qwen3-4B trained, where applicable, and evaluated? Do they use the same environment configurations, oracle access, interaction budgets, and evaluation protocols as BATS and the controlled baselines?
-
Have the author compared BATS with fixed DAgger-to-PPO schedules and imitation-only training under matched compute and interaction budgets? Such results would materially affect my assessment of whether the adaptive controller provides a meaningful advantage.
2 comments
Response
Thank you for clarifying the baseline evaluation protocols and adding the requested fixed-schedule comparisons. These address my questions about baseline comparability and the missing schedule baselines.
However, my concerns about BATS’s added value remain. The fixed halfway schedule outperforms BATS from weak and medium initializations, while BATS’s advantage from the strong initialization is small and based on a single seed (Table 2). The expert-label savings are useful, but the comparisons do not establish an efficiency advantage under matched compute and interaction budgets.
The experiments without dialogue broaden the evaluation but do not isolate oracle assistance in the original tasks or show a consistent advantage for BATS. I also retain reservations about the practical benefits of training these agents from scratch. Given these remaining concerns about the contribution, efficiency, and generality, I will therefore maintain my score.
Response to Reviewer Comments
Thank you for recognizing the breadth of our evaluation and for asking how we account for training costs.
W1. Why train a compact agent? For a narrow task executed repeatedly, a specialized model can reduce the cost of rolling out a policy. On ALFWorld-Dialogue, our 5M-parameter executor achieved 89.6% test success from the strong checkpoint with a 50/50 DAgger-to-PPO schedule, compared with 88.3% for RL-adapted Qwen3-4B. Expert corrections after pretraining teach the agent what to do in unfamiliar states, and PPO uses task rewards to refine the policy. We have not tested whether a schedule learned at this scale transfers to a larger model.
W2 and Q3. Simple schedules and resource use. We compared imitation-only DAgger, a fixed halfway switch to PPO, a switch chosen retrospectively at a slowdown in DAgger validation performance, and BATS. At equal training-cycle counts on ALFWorld-Dialogue, the slowdown switch led from the weak initialization (80.6% test success), the halfway switch led from medium (88.8%), and BATS led from strong (91.0%). From strong, the halfway switch achieved 89.6% and DAgger alone achieved 73.1%. BATS used about 19k expert labels and 20 active hours, versus 140k labels and 24 hours for the halfway switch. It also used more agent decisions (1.8M versus 1.3M). We can quantify the saving in expert labels, but these runs do not establish an advantage at matched compute and interaction budgets.
Q1. Oracle access. Dialogue lets agents ask for help when an observation is ambiguous. Within each environment, all methods could query the same oracle under the same rules. We also evaluated agents without dialogue. In Crafter, BATS and PPO reached similar mean achievement counts (7.44 and 7.39). In WebShop, PPO had the highest purchase success and DAgger the highest mean reward. These experiments establish that the pipeline works without dialogue. They do not isolate the oracle’s contribution within a given task.
Q2. Gemma and Qwen evaluation. Each agent received the same type of public observation and interaction history, could act or ask a question, and had the same per-episode budget. We ran one episode per held-out task with the same success criterion. Only the oracle saw private task information, including when Gemma acted as the expert. Gemma 4 26B-A4B and zero-shot Qwen3-4B and Qwen3-8B received no task-specific training. The separate Qwen3-4B (GiGPO) row reports the RL-fine-tuned agent. Appendix C gives the model-specific decoding settings.