- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GFR, a method that guides sub-billion parameter models to discover successful trajectories through self-interaction with the environment using MCTS during training. The Qwen3-0.6B model is then trained using three types of signals: CE loss on successful trajectories, negative loss from sibling branches, and penalties for execution constraints (such as formatting errors, illegal actions, and repetitive actions). Experiments conducted on ALFWorld and WebShop with comprehensive baselines demonstrate that GFR significantly outperforms most teacher-free baselines on Qwen3-0.6B, while also reducing the number of actions required on WebShop.
Critical strengths
-
The paper studies a meaningful setting: bootstrapping sub-billion language model agents without external teachers or task-specific SFT warm-start. The comparison includes several relevant teacher-free baselines on ALFWorld and WebShop, and the results show clear gains over the pretrained model and most compared methods.
-
GFR combines search-generated successful trajectories with execution feedback, rather than relying only on terminal rewards. The ablations, especially the No UL comparison, provide useful evidence that execution-level feedback can reduce redundant behavior and lead to shorter executions.
Critical weaknesses
- The effective scope of GFR remains unclear.
The main experiments use only Qwen3-0.6B with one fixed seed. Since GFR depends on search discovering successful trajectories, its gains may vary substantially with the initial policy quality: weaker models may still fail to find useful successes, while stronger models may already obtain enough successful trajectories through rejection sampling or standard RL. Results across model scales or initial success rates would make the claim about sub-billion agents much stronger.
- Concerns about limitations of environments.
GFR appears to rely on environment properties that may not hold in more open-ended agent tasks. Training-time search assumes reset and replay of action prefixes, public admissibility checks, reliable execution feedback, and hand-designed redundancy detection. These assumptions are reasonable for ALFWorld and WebShop, but it is unclear how the method would transfer to tasks such as coding or real web interaction, where actions are less easily filtered and repeated states are harder to define.
Questions for authors
-
Could the authors clarify the role and empirical contribution of the auxiliary Q-head in GFR? The Q-head is trained on detached context-action representations and contributes one value pseudocount during search, but there is no ablation that removes it. How much does this learned value prior affect search quality and final performance, compared with the other learning signals such as successful-path CE, branch penalties, and execution constraints?
-
Can the authors clarify how sensitive GFR is to the initial model capability and model scale? The current experiments only use Qwen3-0.6B. Would GFR still provide similar gains for a weaker model that rarely finds successful trajectories, or for a stronger model where rejection sampling or standard RL may already find enough successful trajectories?
-
Can the authors report variance across multiple training seeds, especially for ALFWorld where the gap to teacher-assisted SFT+GRPO is small? More experimental results should be provided to make the paper's conclusions more convincing.