- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GET, an on-policy self-distillation method for training multi-turn language agents alongside GRPO. For each decision, GET builds a privileged teacher context by combining the immediate execution feedback of the student's action with an "Evidence Card" that summarizes peer actions and episode outcomes from observation-matched transitions in other rollouts of the same task. The teacher re-scores the student's sampled tokens under this context, and the resulting token-level log-probability gaps are mapped via Raw or Gate weighting rules to form an auxiliary distillation loss. Experiments on ALFWorld and Search-QA with Qwen2.5-3B and Qwen3-1.7B show improvements over vanilla GRPO and skill-conditioned distillation.
Critical strengths
-
Constructing privileged teacher context directly from on-policy rollout groups via observation matching and deterministic rendering is practical, avoiding extra LLM calls for skill generation.
-
The empirical analysis goes beyond terminal success rates by examining environment-verified intermediate milestones on ALFWorld and search call efficiency on Search-QA, offering helpful insights into how local supervision changes agent behavior.
Critical weaknesses
-
The core components of GET closely overlap with existing methods discussed in Section 5, yet none of them are empirically compared. Such as GiGPO, SDPO, and MOPD and Combining observation matching with peer/feedback distillation feels incremental. Furthermore, the experiments only compare against GRPO and SDAR, omitting direct multi-turn RL and peer-distillation SOTA baselines.
-
The performance gap between the Raw and Gate weighting rules fluctuates drastically across models without a clear explanation. On Table 1, Qwen2.5-3B with Gate achieves 84.31 while Raw drops sharply to 70.07. On Qwen3-1.7B, Raw substantially outperforms Gate, yet suffers from severe collapse on specific subtasks. In addition, Table 2 reveals high variance across random seeds for GET, raising concerns that reporting only seed 0 in the main tables overstates the method's actual stability and gains. Finally, all experiments are restricted to very small (1.7B and 3B) and somewhat dated (Qwen2.5 from 2024) backbones, leaving it unclear whether GET still provides meaningful gains on stronger, larger, and more recent models
Questions for authors
-
Given that GiGPO also leverages observation matching across rollout groups and methods like MOPD/SDPO use peer trajectories or execution feedback for self-distillation, why are these closely related methods not included as baselines? How does GET perform when compared directly against GiGPO or MOPD under the same training budget?
-
Why do Raw and Gate exhibit completely opposite behaviors on Qwen2.5-3B and Qwen3-1.7B on ALFWorld, and what causes the severe degradation on tasks like Pick2 under Raw? Moreover, since the evaluation only uses small 1.7B/3B models (including the older Qwen2.5-3B), can the authors provide results on larger or more recent backbones to show that explicit peer evidence cards remain beneficial when the base policy is stronger?