- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes SteerDistill, a quantization-aware distillation method for multi-turn LLM agents. It identifies behavioral divergences between failed quantized-student rollouts and successful full-precision reference trajectories. At these turns, the teacher provides structured guidance, the student regenerates its response, and an environment-specific verifier checks the correction before the rollout continues. The corrected behavior is then internalized without steering prompts, using forward KL from the steered student at intervention turns and reverse KL toward the full-precision teacher elsewhere. Experiments across five benchmarks and three models show improved quantized performance close to full precision, with better training efficiency and shorter successful trajectories on τ2-Bench.
Critical strengths
-
The method is well motivated and easy to understand. Quantization can introduce small errors that cause the rest of an agent's interaction to fail. The method identifies the step where the rollout goes wrong, uses a teacher-generated prompt to help the student revise its response, and checks whether the revised action is correct. It then trains the student to produce this behavior without the prompt, so teacher guidance is no longer needed at inference time.
-
The experiments show strong performance recovery and promising efficiency gains. Across different models and benchmarks, the method recovers much of the performance lost through quantization and outperforms other quantization-aware distillation methods. The reported results also show better training efficiency and fewer output tokens per successful task, suggesting practical benefits beyond accuracy.
Critical weaknesses
-
Some training choices need more ablation. At the corrected steps, the model learns from the student with teacher guidance. At other steps, it learns from the full-precision teacher. Why not use the full-precision teacher as the target at all steps? The teacher-takeover ablation changes how the corrected response is generated, so it does not directly answer this question. It would also be useful to train only on the verified corrections and skip the other steps. This could show whether the remaining supervision is needed and whether training can be made cheaper.
-
The initial GRPO training is not well justified. The paper applies task-specific GRPO training to the backbone models before quantization, but it does not clearly explain why this step is needed. Since the goal is to recover performance lost through quantization, I would like to see results starting directly from the original backbone models. This would help show whether the gains depend on the initial GRPO training.
Questions for authors
-
How do you handle valid alternatives to the teacher's actions? The student may take a different but valid path from the teacher. How do you distinguish this from an actual mistake, and have you checked how often it happens?
-
How well does SteerDistill work without the initial GRPO training? Do you have results starting directly from the original backbone models?
2 comments
Thank you for the detailed response. The additional experiments addressed my concerns about the training choice and my questions, so I will keep my score and raise my soundness to 3.
Thank you for your careful review and constructive feedback.
The new ablations support using steered-student targets on corrected turns and retaining supervision on other turns. We also evaluate SteerDistill without initial GRPO and examine valid alternatives to teacher actions.
For the main Qwen3-8B W4A16 comparison, we now report four-seed BFCL V4 means for SteerDistill, On-Policy QAD, and QeRL (Appendix C.2).
Q1. Handling valid alternatives to teacher actions
We retain successful student rollouts unchanged and treat different actions as equivalent when they produce matching environment transitions. Only failed rollouts are revised, starting at the first behavioral divergence—a transition that differs from the teacher’s. We have clarified these rules in Section 3.1 and Appendix A.1 of the revised paper.
This check missed valid alternatives in 18 of 302 examined failed rollouts (6.0%) from Qwen3-8B (W4A16) training in AgentWorldModel. At the first divergence in each rollout, we kept the student’s action and let the teacher continue the task. We counted the action as a valid alternative only if the teacher completed the task without correcting that action. We have added this analysis to Appendix C.4 of the revised paper.
Q2 / W2. Performance without initial GRPO
Without initial GRPO, SteerDistill outperforms both distillation baselines and matched-budget QeRL.
With initial GRPO, it achieves better final W4A16 performance with fewer total GPU hours than QeRL with or without initial GRPO.
Results starting directly from the original Qwen3-8B checkpoint, without initial GRPO (quantized methods use W4A16):
With longer training, QeRL scores higher and exceeds the original BF16 model on BFCL (42.74 vs. 40.40), reflecting task learning beyond quantization recovery. Distillation relies on the frozen original backbone as its teacher, while QeRL continues learning from task rewards.
Initial GRPO strengthens both the frozen teacher and the student's initialization. The comparison below starts from the same original backbone and includes all training stages:
We have added these comparisons to Appendix C.1 (Table 5) and clarified the role of initial GRPO in the experimental setup.
W1. Distillation targets and supervision across turns
Using full-precision teacher targets at all turns or training only on verified corrections gives lower scores than SteerDistill. Results on Qwen3-8B under W4A16:
We have added the BFCL V4 results to Table 2 (Q5) of the revised paper.