acceptodds
Withdrawn from ICLR 2027

Go for the Reward: Monte Carlo Tree Search for Training Sub-Billion Language Model Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training sub-billion language model agents without external teachers is difficult when their initial policies rarely complete tasks and receive little guidance from task-level failure feedback. We introduce Go for the Reward (GFR), a framework that learns directly from environment interaction without teacher demonstrations or task-specific supervised initialization. Inspired by the search–learning loop of Go-playing AI, GFR alternates training-time Monte Carlo tree search with policy learning. Search explores alternative continuations from shared interaction prefixes to discover successful trajectories. Execution constraints complement these trajectories with feedback on malformed or inadmissible actions and redundant interactions, including in unsuccessful attempts. The policy learns from this experience through successful-trajectory supervision, outcome-supported sibling-branch comparisons, and execution penalties, and then guides subsequent search. Starting from pretrained Qwen3-0.6B, GFR achieves 39.55% unseen success on ALFWorld and a WebShop Task Score of 59.61 without evaluation-time search. Both results are the highest among the compared teacher-free methods. On ALFWorld, this teacher-free training reaches performance comparable to teacher-assisted supervised fine-tuning followed by reinforcement learning. GFR also learns more efficient execution paths, reducing environment interactions by about 31% relative to its no-tree variant on a shared set of successfully completed WebShop tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Language Model, Monte Carlo Tree Search, Reinforcement Learning, Agentic Tool-use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_nCqKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GFR, a method that guides sub-billion parameter models to discover successful trajectories through self-interaction with the environment using MCTS during training. The Qwen3-0.6B model is then trained using three types of signals: CE loss on successful trajectories, negative loss from sibling branches, and penalties for execution constraints (such as formatting errors, illegal actions, and repetitive actions). Experiments conducted on ALFWorld and WebShop with comprehensive baselines demonstrate that GFR significantly outperforms most teacher-free baselines on Qwen3-0.6B, while also reducing the number of actions required on WebShop.

Critical strengths
  1. The paper studies a meaningful setting: bootstrapping sub-billion language model agents without external teachers or task-specific SFT warm-start. The comparison includes several relevant teacher-free baselines on ALFWorld and WebShop, and the results show clear gains over the pretrained model and most compared methods.

  2. GFR combines search-generated successful trajectories with execution feedback, rather than relying only on terminal rewards. The ablations, especially the No UL comparison, provide useful evidence that execution-level feedback can reduce redundant behavior and lead to shorter executions.

Critical weaknesses
  1. The effective scope of GFR remains unclear.

The main experiments use only Qwen3-0.6B with one fixed seed. Since GFR depends on search discovering successful trajectories, its gains may vary substantially with the initial policy quality: weaker models may still fail to find useful successes, while stronger models may already obtain enough successful trajectories through rejection sampling or standard RL. Results across model scales or initial success rates would make the claim about sub-billion agents much stronger.

  1. Concerns about limitations of environments.

GFR appears to rely on environment properties that may not hold in more open-ended agent tasks. Training-time search assumes reset and replay of action prefixes, public admissibility checks, reliable execution feedback, and hand-designed redundancy detection. These assumptions are reasonable for ALFWorld and WebShop, but it is unclear how the method would transfer to tasks such as coding or real web interaction, where actions are less easily filtered and repeated states are harder to define.

Questions for authors
  1. Could the authors clarify the role and empirical contribution of the auxiliary Q-head in GFR? The Q-head is trained on detached context-action representations and contributes one value pseudocount during search, but there is no ablation that removes it. How much does this learned value prior affect search quality and final performance, compared with the other learning signals such as successful-path CE, branch penalties, and execution constraints?

  2. Can the authors clarify how sensitive GFR is to the initial model capability and model scale? The current experiments only use Qwen3-0.6B. Would GFR still provide similar gains for a weaker model that rarely finds successful trajectories, or for a stronger model where rejection sampling or standard RL may already find enough successful trajectories?

  3. Can the authors report variance across multiple training seeds, especially for ALFWorld where the gap to teacher-assisted SFT+GRPO is small? More experimental results should be provided to make the paper's conclusions more convincing.

Reviewer_aNT6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Go for the Reward (GFR), a framework for training agents without external teacher demonstrations or task-specific initialization. GFR alternates policy-guided Monte Carlo tree search with policy learning. The policy learns from successful-trajectory supervision, outcome-supported sibling-branch comparisons, and execution penalties.The trained policy operates without search at evaluation time. Experiments with Qwen3-0.6B on ALFWorld and WebShop show promising results, including 39.55% success on ALFWorld tasks and a WebShop Task Score of 59.61.

Critical strengths
  1. The paper addresses an important challenge in training small language-model agents: sparse task-level rewards provide little useful learning signal when the initial policy rarely succeeds. Using environment-grounded search to discover successful continuations is a reasonable and potentially effective way to overcome this cold-start problem.
  2. The evaluation includes both ALFWorld and WebShop and compares against multiple teacher-free approaches. The comparison with teacher-assisted SFT followed by GRPO is also useful for understanding the potential of teacher-free training.
Critical weaknesses
  1. The contributions of search and the individual learning objectives are not fully disentangled. GFR combines tree search, successful-path supervision, sibling-branch penalties, and execution constraints. The ablations provide some evidence for the role of these components, but the results do not fully establish which components drive task-completion improvements versus more efficient execution. A clearer separation of success-rate gains and execution-efficiency gains would strengthen the paper.
  2. The evaluation is limited in model and environment diversity. The main experiments use Qwen3-0.6B on ALFWorld and WebShop. It remains unclear how the method behaves across different model sizes, initial policy competence levels, and more complex long-horizon environments. These experiments would help establish the scope and generality of the proposed approach.
Questions for authors
  1. Can you quantify the separate effects of tree search, successful-trajectory imitation, sibling-branch penalties, and execution constraints on task success and execution efficiency?
  2. How sensitive is GFR to the initial policy's success rate? Are there regimes in which tree search rarely discovers successful paths, or where the search cost outweighs its benefit?
Reviewer_FYGtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a sub-billion language-model agent (Qwen3-0.6B) can learn multi-step tasks without a teacher model or a supervised warm-start. This is challenging because the base model achieves small success rate (1.43% and 0.75% on ALFWorld seen and unseen), so group-based RL methods such as GRPO get almost no learning signal. The paper proposes GFR based on AlphaGo Zero's idea of alternating search and learning. It builds a search tree for each task during training. After initializing the tree, the algorithm resets the environment and replays actions with a shared prefix. It applies PUCT with progressive widening and a auxiliary value head to explore new actions. The model uses three kinds of training signals to update its weights. (i) It imitates successful paths and keeps replaying recent successes from earlier rounds. (ii) It is mildly penalized for actions that did worse than a successful sibling action. (iii) It is penalized for malformed, inadmissible and redundant actions, and these penalties give a learning signal even when nothing succeeds.

GFR achieves 38.57% / 39.55% on ALFWorld seen / unseen respectively. The eight teacher-free baselines reach at most 6.43% / 3.73%, and a teacher-assisted pipeline (SFT on Qwen3-8B successes, then GRPO) reaches 37.86% / 37.31%. On WebShop, GFR has the best Task Score (59.61) and uses the fewest actions, while Tree-GRPO has a slightly higher success rate (29.80% vs. 28.40%). WebShop ablations suggest that search and the extra losses mainly shorten successful episodes rather than raise the score. Removing the unlikelihood loss on redundant actions hurts both benchmarks.

Critical strengths
  1. Teacher-free learning from a weak initial policy. GFR enabled learning from an initial policy that rarely succeeds. On ALFWorld, unseen success increased from below 1% to 39.55%, which is from 1/1341/134 to 53/13453/134 tasks. Every compared teacher-free method stays at or below 5/1345/134 (Table 2). The result is strong in two ways. (i) The gains were retained without evaluation-time search. (ii) GFR achieved comparable ALFWorld performance to the SFT-success + GRPO reference without using teacher demonstrations. Only one seed is reported, but the gap is much larger than the evaluation noise, since a rough binomial standard error is about 4 points at n=134n = 134. It shows that a sub-billion agent can learn from its own environment interactions without an external teacher.

  2. Training signals even when no trajectory succeeds. Execution constraints address format, admissibility and redundancy, including in unsuccessful attempts. This directly targets the problem that motivates the paper. The No UL ablation removes only the unlikelihood (UL) loss and keeps search-time action blocking. Both the scores and the behavior get worse without UL. (i) ALFWorld success drops from 38.57% / 39.55% to 25.71% / 25.37%, and the WebShop Task Score drops from 59.61 to 40.74 (Table 6). (ii) The standalone behavior becomes much more repetitive. Repeated observation–action pairs occur on 70.80% of WebShop tasks without UL, compared with 2.60% for GFR.

Critical weaknesses
  1. The contribution of tree search is not isolated. GFR combines tree search, learning from successful experience and execution constraints together, but it is unclear to me what search itself contributes. (i) The search and task-learning ablations are only reported on WebShop (Table 5). There, No tree replaces search with independent trajectories under the same nominal path budget, and the resulting policies differ in execution length despite similar Task Scores (No tree 59.95 vs. GFR 59.61, L.427–428). (ii) On ALFWorld, where the main gain appears, only UL is ablated (Table 6). Even No UL (25.37% unseen) stays far above every teacher-free baseline (at most 3.73%). So most of the ALFWorld gain is not linked to any component. It could come from search, but also from learning from execution feedback in unsuccessful attempts, which is how the paper explains its advantage over Tree-GRPO (L.382–383). This decides whether the main lesson is about tree search or about the training objective.

  2. The baselines are hard to interpret. I have three concerns here. (i) For GRPO and GiGPO, the paper only states the budget and full-parameter updates (L.994, L.1014). Settings such as the reward shaping and evaluation decoding are not given. This matters because GFR itself uses format and admissibility signals, and the base model takes invalid actions on 98.8% of WebShop tasks (Fig. 6c). (ii) Several baselines look weaker than expected. SDPO and OPSD score 0.00 on every metric (Table 2). GRPO also gains much less than in the GiGPO paper (Feng et al., 2025, Table 1), which uses the same nominal budget and reports GRPO raising Qwen2.5-1.5B-Instruct from 4.1% to 72.8% on ALFWorld. Here GRPO only reaches 5.00% / 3.73% from 1.43% / 0.75%. Model size and thinking mode differ, so some gap is expected, but it should be explained. (iii) The experiments use one fixed seed, and differences are reported descriptively (L.1012–1013). This makes the small WebShop margins over Tree-GRPO hard to judge. The claim that both headline results are the highest among the compared teacher-free methods (L.025–026) depends on these baselines.

Overall: The ALFWorld result is promising, and the paper is candid about the limits of its comparisons (B.1–B.2). However, the evidence does not yet show the role of tree search, and the baselines are under-specified. Both issues look fixable. For now I lean toward weak rejection, and I would raise my score if Questions 1 and 2 are answered convincingly.

Questions for authors
  1. Could you run "No tree" on ALFWorld under the same 75-round, 19,200-path budget?
  2. For GRPO, GiGPO and T$^2$PO, what were the group size, the reward (including any format or invalid-action penalty; the GiGPO paper uses −0.1-0.1 for invalid actions), the KL coefficient and the evaluation decoding? What explains the much smaller gains than in the GiGPO paper, and why do SDPO and OPSD score 0.00 everywhere?
  3. Could you add at least two more training seeds for GFR on both benchmarks? Please also report WebShop success rates for the ablations.
Reviewer_A7LsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Small language-model agents trained with RL on multi-step tasks often get no learning signal at the start: if a weak policy almost never completes a task, every attempt fails, and group-relative methods such as GRPO have nothing to compare. The paper asks whether a sub-billion model can bootstrap from its own interaction, with no teacher and no supervised warm start.

GFR alternates search and learning, in the spirit of AlphaGo Zero. During training only, it builds a search tree per task by resetting the environment and replaying earlier actions, so it can try different next actions from the same point. The policy then imitates successful paths, is pushed away from sibling actions that succeeded less often, and is penalized for malformed, invalid and repeated actions, using hand-written repetition detectors for each environment. At test time it acts alone, without search.

Starting from Qwen3-0.6B, GFR lifts ALFWorld success from almost zero to far above every RL and self-distillation baseline, roughly matches a reference trained on a larger teacher's demonstrations, and gets the best WebShop Task Score. Ablations show the repetition penalty matters a lot, and that on WebShop search mainly shortens paths rather than raising the score.

Critical strengths

The untrained model solves 1 of 134 unseen ALFWorld tasks; GFR solves 53, with no search at test time, while no RL baseline solves more than 5 (Table 2). GFR also matches a reference that first fine-tunes on the successful runs of a larger Qwen3-8B model and then applies GRPO: 54 vs 53 seen tasks, 53 vs 50 unseen (Table 3). The paper reports results that cut against it. Table 2 bolds Tree-GRPO's higher WebShop success (29.80 vs 28.40), Table 5 bolds the no-search variant's higher Task Score, and Appendix B.2 says budgets match trajectory counts, not compute, and that every result is one seed. Every rate I checked works out to a whole number of tasks.

S2. Evidence on how the agent behaves, not just whether it succeeds.

Removing the repetition penalty (unlikelihood training, UL) has a large and consistent cost: ALFWorld falls from 38.57/39.55 to 25.71/25.37 seen/unseen, and the WebShop Task Score from 59.61 to 40.74 (Table 6). Figure 6 shows why. Without UL, the agent repeats the same action on an unchanged page in 70.8% of WebShop tasks (2.6% with it) and runs out of actions without buying in 34.4% (0.2%). On tasks that every variant solves, GFR's paths are 31% shorter than the no-search variant's (3.74 vs 5.43 actions) and about half as long as the no-UL variant's (3.61 vs 7.55).

Critical weaknesses

W1. The paper credits tree search for the gains but never tests that claim where the gains are.

On WebShop, removing search does not lower the score: the no-search variant gets 59.95 and a variant that keeps search but drops the penalties gets 59.70, against GFR's 59.61 (Table 5). There, search only shortens paths, and only together with the penalties. On ALFWorld, where the big gain appears, there is no run without search at all, and the one ablation shows that the hand-written repetition penalty alone is worth about 13 to 14 points. Besides search, GFR differs from GRPO in at least six ways: imitation of successes with a replay memory, format training, penalties for invalid actions, the repetition penalty, gated updates with rollbacks, and a value head. Several of these could be added to GRPO directly, so the gap to GRPO cannot yet be credited to search (Q1).

W2. The comparisons are too thin to support "highest" or "comparable".

Every result is one training seed and one test attempt per task. On 134 unseen tasks the standard error is about 4.2 points, so the match with the teacher-assisted reference rests on 3 tasks (53 vs 50). Several baselines end at or below the untrained model: SDPO and OPSD score zero on every metric, EvoTool falls below the base model on WebShop, and GKD scores 0.00 on unseen ALFWorld tasks after 27.86 on seen ones. That pattern suggests broken setups rather than weak methods. The paper reports no tuning for GRPO and GiGPO and does not say how they were sampled at test time; GFR used temperature 1.0, the base and 8B models 0.6. Finally, the teacher behind the reference, Qwen3-8B, scores only 27.61 on unseen tasks itself, below GFR (Q2-Q4).

Questions for authors

Q1. Does search matter on ALFWorld? Please run the no-search variant and the search-without-penalties variant on ALFWorld with the same seed and budget, plus GRPO with GFR's format, invalid-action and repetition losses added. How much of the gain from 0.75 to 39.55 survives without search? This is the experiment that separates the effect of search from the effect of the extra losses.

Q2. Baseline strength. Your GRPO and GiGPO reach only 3-5% on ALFWorld, so it matters whether they were set up well. What learning rate, KL coefficient and invalid-action handling did GRPO and GiGPO use, and were they tuned? Do they learn with thinking mode turned off? What sampling settings evaluated each baseline? How do your GRPO and GiGPO results compare with those the GiGPO authors report for the verl-agent setup you follow?

Q3. Seeds. Can you report mean and standard deviation over at least 3 training seeds, with several test attempts per task, for GFR, the no-search variant and GRPO on ALFWorld? With one run, a 3-task margin cannot be told apart from noise.

Q4. Baselines that collapsed. Why do SDPO and OPSD score exactly zero and always use the full action budget, and why is SDPO reported at round 70? Why does GKD score 27.86 on seen tasks but 0.00 on unseen ones? Which teacher do GKD and GEPA use?

Q5. Generality. The repetition penalty is worth more than any other measured component, and it depends on hand-written rules. How much work per environment do the repetition detectors and the action blocking take? Does GFR still beat GRPO with both removed? How would it work in an environment that cannot be reset and replayed?

Minor suggestions (do not affect the score).

(a) Add a Limitations section; the paper has none. It should cover the need for a resettable simulator, the hand-written detectors, the single model (Qwen3-0.6B) and the single seed.

(b) Position GFR against prior agent self-training methods that learn from exploration or tree search on the same environments, and say which of them need a teacher or a supervised warm start.

(c) The abstract's "the highest among the compared teacher-free methods" should name the metric, since Tree-GRPO has the higher WebShop success rate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.