acceptodds
Under review as a conference paper at ICLR 2027

SteerDistill: Quantizing LLM Agents via Online Teacher-Steered Trajectory Distillation

Abstract

Quantization makes multi-turn LLM agents brittle: a single erroneous decision alters every subsequent interaction and can derail the task. We observe that quantization rarely breaks an agent's overall plan; instead, it corrupts a few executable decisions, such as a tool call with the wrong interface or a misgrounded argument. Existing recovery methods either learn from fixed full-precision data that misses the states a quantized agent reaches, or supervise the student's own rollouts without locating the error and rely on later rollouts to avoid it; neither corrects the turn at which a rollout goes wrong. We introduce SteerDistill, a quantization-aware distillation method built on *online teacher steering*. When a quantized student's rollout diverges from a successful full-precision trajectory, the teacher issues a steering prompt at the divergent turn; the student regenerates that turn itself, the environment verifies the correction, and the rollout continues from the corrected state. The steered behavior is then distilled into the student without the prompt, so no teacher is needed at inference. Across five agentic benchmarks and three models with 4-bit weights, \nickname outperforms the strongest quantized baseline by 1.67–2.18 average points and stays within one point of BF16. On -Bench, it also exceeds quantization-aware reinforcement learning with 67% fewer training GPU hours.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, LLM Quantization, Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_34AMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes SteerDistill, a quantization-aware distillation method for multi-turn LLM agents. It identifies behavioral divergences between failed quantized-student rollouts and successful full-precision reference trajectories. At these turns, the teacher provides structured guidance, the student regenerates its response, and an environment-specific verifier checks the correction before the rollout continues. The corrected behavior is then internalized without steering prompts, using forward KL from the steered student at intervention turns and reverse KL toward the full-precision teacher elsewhere. Experiments across five benchmarks and three models show improved quantized performance close to full precision, with better training efficiency and shorter successful trajectories on τ2-Bench.

Critical strengths
  1. The method is well motivated and easy to understand. Quantization can introduce small errors that cause the rest of an agent's interaction to fail. The method identifies the step where the rollout goes wrong, uses a teacher-generated prompt to help the student revise its response, and checks whether the revised action is correct. It then trains the student to produce this behavior without the prompt, so teacher guidance is no longer needed at inference time.

  2. The experiments show strong performance recovery and promising efficiency gains. Across different models and benchmarks, the method recovers much of the performance lost through quantization and outperforms other quantization-aware distillation methods. The reported results also show better training efficiency and fewer output tokens per successful task, suggesting practical benefits beyond accuracy.

Critical weaknesses
  1. Some training choices need more ablation. At the corrected steps, the model learns from the student with teacher guidance. At other steps, it learns from the full-precision teacher. Why not use the full-precision teacher as the target at all steps? The teacher-takeover ablation changes how the corrected response is generated, so it does not directly answer this question. It would also be useful to train only on the verified corrections and skip the other steps. This could show whether the remaining supervision is needed and whether training can be made cheaper.

  2. The initial GRPO training is not well justified. The paper applies task-specific GRPO training to the backbone models before quantization, but it does not clearly explain why this step is needed. Since the goal is to recover performance lost through quantization, I would like to see results starting directly from the original backbone models. This would help show whether the gains depend on the initial GRPO training.

Questions for authors
  1. How do you handle valid alternatives to the teacher's actions? The student may take a different but valid path from the teacher. How do you distinguish this from an actual mistake, and have you checked how often it happens?

  2. How well does SteerDistill work without the initial GRPO training? Do you have results starting directly from the original backbone models?

2 comments

Thank you for your careful review and constructive feedback.

The new ablations support using steered-student targets on corrected turns and retaining supervision on other turns. We also evaluate SteerDistill without initial GRPO and examine valid alternatives to teacher actions.

For the main Qwen3-8B W4A16 comparison, we now report four-seed BFCL V4 means for SteerDistill, On-Policy QAD, and QeRL (Appendix C.2).

Q1. Handling valid alternatives to teacher actions

We retain successful student rollouts unchanged and treat different actions as equivalent when they produce matching environment transitions. Only failed rollouts are revised, starting at the first behavioral divergence—a transition that differs from the teacher’s. We have clarified these rules in Section 3.1 and Appendix A.1 of the revised paper.

This check missed valid alternatives in 18 of 302 examined failed rollouts (6.0%) from Qwen3-8B (W4A16) training in AgentWorldModel. At the first divergence in each rollout, we kept the student’s action and let the teacher continue the task. We counted the action as a valid alternative only if the teacher completed the task without correcting that action. We have added this analysis to Appendix C.4 of the revised paper.

Q2 / W2. Performance without initial GRPO

Without initial GRPO, SteerDistill outperforms both distillation baselines and matched-budget QeRL.

With initial GRPO, it achieves better final W4A16 performance with fewer total GPU hours than QeRL with or without initial GRPO.

Results starting directly from the original Qwen3-8B checkpoint, without initial GRPO (quantized methods use W4A16):

MethodGPU hoursBFCL V4 Overall (%)τ²-Bench Pass@1 (%)
BF16—40.4030.00
BitDistiller—35.3424.29
On-Policy QAD—36.1624.58
QeRL3236.0724.95
SteerDistill3239.6727.67
QeRL (converged)13242.7428.53

With longer training, QeRL scores higher and exceeds the original BF16 model on BFCL (42.74 vs. 40.40), reflecting task learning beyond quantization recovery. Distillation relies on the frozen original backbone as its teacher, while QeRL continues learning from task rewards.

Initial GRPO strengthens both the frozen teacher and the student's initialization. The comparison below starts from the same original backbone and includes all training stages:

MethodTraining procedureTotal GPU hoursBFCL V4 Overall (%)τ²-Bench Pass@1 (%)
QeRLDirect RL to convergence13242.7428.53
QeRLBF16 GRPO → RL≈169 (73 + 96)44.9529.84
SteerDistillBF16 GRPO → QAD≈105 (73 + 32)46.8731.79

We have added these comparisons to Appendix C.1 (Table 5) and clarified the role of initial GRPO in the experimental setup.

W1. Distillation targets and supervision across turns

Using full-precision teacher targets at all turns or training only on verified corrections gives lower scores than SteerDistill. Results on Qwen3-8B under W4A16:

ConfigurationBFCL V4τ²-Bench
SteerDistill46.8731.79
Teacher at corrected turns44.9029.30
No loss at other turns43.8527.67
  • Full-precision teacher targets at all turns: We kept the corrected responses fixed and replaced the steered-student targets with full-precision teacher targets. This suggests that matching the distribution that produced the verified correction is more effective than applying teacher supervision to the same response.
  • Training only on verified corrections: Removing the loss at all other turns also lowered performance, indicating that supervision on those turns contributes to performance.

We have added the BFCL V4 results to Table 2 (Q5) of the revised paper.

Thank you for the detailed response. The additional experiments addressed my concerns about the training choice and my questions, so I will keep my score and raise my soundness to 3.

Reviewer_4z7vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SteerDistill, a quantization-aware distillation framework for recovering the agentic capabilities of low-bit LLMs. The authors observe that quantization often preserves an agent's high-level plan but introduces errors in individual executable decisions, which can compound across multi-turn interactions. To address this, SteerDistill aligns failed quantized-student rollouts with successful full-precision teacher trajectories and identifies the first behaviorally divergent turn. The teacher then provides targeted guidance, allowing the student to regenerate a corrected action, which is verified by the environment before continuing the rollout. The corrected behavior is subsequently distilled into the unassisted student using forward KL at intervention turns and reverse KL elsewhere. Across five agentic benchmarks and three model families, SteerDistill outperforms existing quantization-aware training and distillation baselines, recovering performance close to BF16 while reporting improved training efficiency.

Critical strengths
  1. Well-motivated problem and targeted approach: The paper identifies an important challenge in quantized LLM agents, where individual execution errors can derail otherwise valid multi-turn trajectories. Correcting these errors at the point of divergence is an intuitive and well-motivated approach.

  2. Effective trajectory correction and distillation: The combination of teacher-guided student regeneration, environment-based verification, and turn-specific KL distillation is interesting. In particular, allowing the student to generate its own corrections rather than directly substituting teacher responses is a sensible design choice.

  3. Strong experimental results and ablations: The method consistently improves over multiple quantization baselines across different models, precisions, and agentic benchmarks. The ablation studies provide useful evidence supporting the individual design choices.

Critical weaknesses
  1. Limited comparisons with closely related methods: While the paper provides extensive comparisons with quantization-aware baselines, it lacks experimental comparisons with related guided on-policy distillation methods such as Guided-OPD, FutureBridge-OPD, and OVCSD. These would help better establish the relative benefits of the proposed trajectory correction mechanism.

  2. Dependence on successful teacher references and environment verification: The method requires successful teacher trajectories, environment-state restoration, and reliable local verification. These assumptions may limit its applicability to more complex, open-ended agentic environments.

  3. Limited quantitative analysis of quantization-induced failures: A central motivation is that quantization errors are concentrated in a few executable decisions rather than the overall reasoning process. However, this observation is primarily supported by qualitative examples. A more systematic analysis of the frequency and types of such failures would strengthen the motivation.

Questions for authors
  1. How does SteerDistill compare with related methods such as Guided-OPD and OVCSD under the same quantization setting?

  2. What fraction of failed student trajectories have successful teacher references, and how often can the identified divergences be corrected?

  3. Could the authors provide a quantitative analysis supporting the observation that quantization failures are primarily caused by a few incorrect executable decisions rather than broader reasoning errors?

2 comments

Thank you for your careful review and constructive feedback.

Our new comparisons show that SteerDistill outperforms all three related methods adapted to QAD. We also analyze student failure types and report teacher reference coverage and correction success.

For the main Qwen3-8B W4A16 comparison, we now report four-seed BFCL V4 means for SteerDistill, On-Policy QAD, and QeRL (Appendix C.2).

Q1 / W1. Comparison with related distillation methods

SteerDistill outperforms all three related methods adapted to the same QAD setting. We use Qwen3-8B under W4A16; “→ QAD” denotes adapting the original method to QAD. Results:

MethodBFCL V4 Overall (%)τ²-Bench Pass@1 (%)
On-Policy QAD44.5328.77
Guided-OPD → QAD45.0429.42
OVCSD → QAD45.2330.30
FutureBridge-OPD → QAD45.8130.71
SteerDistill46.8731.79

All three adaptations improve on On-Policy QAD, yet SteerDistill exceeds the strongest, FutureBridge-OPD, by 1.06 points on BFCL V4 and 1.08 points on τ²-Bench. These gains support our approach of guiding the student to produce locally verified corrections and then distilling the steered-student distributions, rather than supplying teacher-generated turns or continuations.

We have added the adaptation details and results to Appendix C.3 (Table 7), with a summary in Section 4.2.

Q2 / W2. Teacher reference coverage and correction success

Out of 1,079 failed student rollouts, 480 (44.5%) had successful teacher references. Some rollouts encountered further divergences after an earlier correction, giving 580 divergent turns in total. Of these, 545 (94.0%) were successfully corrected within four attempts per turn. We collected these statistics throughout SteerDistill training with Qwen3-8B under W4A16, using the AgentWorldModel environment.

From the first to the last 10% of training trajectories, student failures decreased from 136 to 72 out of 388 rollouts (35.1% → 18.6%), while teacher reference coverage among failed rollouts decreased from 58.8% to 37.5%. This is consistent with the student approaching teacher performance: it fails less often, and its remaining failures increasingly overlap with the teacher’s. We have added these rates and trends to Appendix C.4 of the revised paper.

Q3 / W3. Executable-decision errors versus broader reasoning errors

Among 1,133 failed rollouts, 948 (83.7%) primarily involved local executable-decision errors. We evaluated the PTQ-initialized Qwen3-8B (W4A16) on BFCL V4 before QAD, and used GPT-6.1 Sol to classify the failed rollouts.

Failure typeCountShare
Local: incorrect tool calls53046.8%
Local: missing or misordered operations35831.6%
Local: errors in interpreting results or judging completion605.3%
Broader reasoning failures18516.3%

Most broader reasoning failures (169 of 185) involved getting stuck in repetitive reasoning, without taking action or producing an answer. We have added this analysis to Appendix C.5 (Table 9) of the revised paper.

W2. Successful teacher references, state restoration, and local verification

A BF16 teacher is already available in our QAD setting: it is the model whose agent capabilities we aim to recover. We use its successful rollouts to guide corrections; student rollouts without a successful reference are retained unchanged for distillation (Section 3.1).

Beyond our evaluation suite, benchmarks such as AppWorld and OSWorld support state checkpointing and execution-based checks, providing a basis for local verification against teacher references. If state restoration or reliable local verification is unavailable, we can skip trajectory revision and continue with ordinary on-policy QAD. We have acknowledged this limitation of trajectory revision in the paper’s Limitations section.

Dear reviewer, as the discussion period is coming to a close, we would like to ask whether our response and additional experiments have addressed your concerns. Please let us know if you have other questions or comments. Thank you for your time and valuable feedback.

Reviewer_HKh8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a quantization-aware distillation method for multi-turn LLM agents, called SteerDistill. The authors argue that quantizations mostly corrupts a few decisions while the overall plan remains fine. SteerDistill relies on a teacher to write a steering prompt at the turns that teacher and student diverge. Then the student would regenerate the action given that prompt and verifier makes sure that action has had the intended consequence. The steered behavior is then distilled into the student without the prompt with a combination of forward and reverse KL objective.

Critical strengths

The evaluations are broad. It covers five benchmarks, three models and three precisions. The proposed methodology is sound and the ablation experiments are isolates the main design choices.

Critical weaknesses

1- The comparison to the baselines is not completely fair: SteerDistill uses a teacher to generate steering prompts with up to 4 steering attempts per turn and a verifier which the baselines do not get. It is useful to compare SteerDistill with the baseline in terms of GPU hours as in Figure 3(a), but it does not say whether teacher and verifier time is counted. Moreover, expect figure 3, no CI or standard deviation is reported to understand the significance of the results since most of them are 1-2 points improvements compared to the baselines.

2- Without verification the method scores 42.18 on BFCL V4, below On-Policy QAD (44.43) and QeRL (44.91). The gains may come largely from the verifier filtering. Also, the acceptance rate is not reported to see how effective is really the steering prompt.

Questions for authors
  • What fraction of failed quantized rollouts contain a single divergent turn, and how many divergences does a typical failed rollout have?

  • What is the acceptance rate of the local verifier?

2 comments

Thank you for your careful review and constructive feedback.

Our four-seed BFCL V4 evaluations on Qwen3-8B under W4A16 confirm consistent gains over On-Policy QAD and QeRL. We have updated the corresponding BFCL V4 scores and five-benchmark averages in the revised paper. We also report statistics on divergent turns and verifier acceptance, and clarify that GPU hours include teacher steering and verification.

Q1. Number of divergent turns in failed rollouts

Out of 480 failed rollouts with successful teacher references, 394 (82.1%) had one divergent turn, 72 (15.0%) had two, and 14 (2.9%) had three or more. We collected these trajectories throughout SteerDistill training with Qwen3-8B under W4A16, using the AgentWorldModel environment.

Q2 / W2. Local verifier acceptance rate

The local verifier accepted 545 of 715 correction attempts (76.2%), counting both initial attempts and retries. Across 580 divergent turns, 545 (94.0%) were successfully corrected within four attempts. We collected these statistics throughout SteerDistill training with Qwen3-8B under W4A16, using the AgentWorldModel environment. We have added these rates to Appendix C.4 of the revised paper.

W1. Comparison fairness and training cost

SteerDistill achieves higher success rates with fewer GPU hours, including the costs of teacher steering, retries, and verification.

  • Teacher supervision. During training, On-Policy QAD, BitDistiller, and SteerDistill use the same full-precision teacher checkpoint. SteerDistill also uses this teacher to construct locally verified corrections.

  • Training cost. Fig. 3(a) compares end-to-end GPU hours on τ²-Bench using Qwen3-8B under W4A16:

    MethodGPU hoursτ²-Bench Pass@1 (%)
    On-Policy QAD4028.75
    QeRL9629.84
    SteerDistill3231.68

    We have clarified in the revised caption that these costs include teacher steering and verification.

  • Contribution of steering. With the same verifier and up to four attempts per turn, adding teacher steering raised the BFCL V4 score from 44.46 to 46.87 (Table 2, Q2).

At inference, all methods use the same evaluation settings without teacher assistance.

W1. Variation across evaluation runs

Across four independent evaluation runs, SteerDistill consistently outperforms On-Policy QAD and QeRL. Results for Qwen3-8B under W4A16 (BFCL V4 Overall and τ²-Bench Pass@1, %):

MethodBFCL V4: submittedBFCL V4: mean ± SDτ²-Bench: mean ± SD
On-Policy QAD44.4344.53 ± 0.1928.77 ± 0.24
QeRL44.9144.95 ± 0.2329.84 ± 0.27
SteerDistill46.8446.87 ± 0.1431.79 ± 0.20

The mean gains are substantially larger than the standard deviations of the individual methods. The τ²-Bench means were already based on four evaluation runs; we now also report their standard deviations. We have added these results to Appendix C.2 (Table 6) of the revised paper.

W2. Role of verifier filtering in the gains

Retries and verification alone gave nearly the same BFCL V4 score as On-Policy QAD (44.46 vs. 44.53). In contrast, adding teacher steering raised the score to 46.87, with the same verifier and up to four attempts per turn (Qwen3-8B, W4A16; Table 2, Q2).

Dear reviewer, as the discussion period is coming to a close, we would like to ask whether our response and additional experiments have addressed your concerns. Please let us know if you have other questions or comments. Thank you for your time and valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for the valuable feedback and discussion. We have revised the manuscript and added experiments (Qwen3-8B under W4A16, unless otherwise noted) and analyses to address the reviewers’ main concerns. Before the discussion period ends, we summarize our responses and the corresponding revisions below.

  • Comparison fairness, evaluation variability, and verification (HKh8). We clarified that the training costs in Fig. 3 include steering, retries, and verification, and added four-run evaluation results showing gains larger than the observed standard deviations (Appendix C.2). With the same verifier and retry budget, adding teacher steering raises BFCL from 44.46 to 46.87, showing that filtering alone does not explain the gains (Table 2, Q2).

  • Comparison with related agent distillation methods (4z7v). We implemented and evaluated QAD adaptations of Guided-OPD, OVCSD, and FutureBridge-OPD under the same quantization setting. SteerDistill outperforms all three on both benchmarks, exceeding the strongest by 1.06 points on BFCL and 1.08 on τ²-Bench (Appendix C.3).

  • Distillation targets and supervision at other turns (34AM). With corrected responses fixed, replacing steered-student targets with teacher targets lowers BFCL from 46.87 to 44.90. Removing supervision at all other turns lowers it to 43.85, supporting the use of both steered-student targets and supervision beyond the corrected turns (Table 2, Q5).

  • Role of initial GRPO (34AM). Without initial GRPO, SteerDistill still outperforms the distillation baselines and matched-budget QeRL, although QeRL scores higher when trained longer on task rewards. Including initial GRPO, our full pipeline achieves higher τ²-Bench Pass@1 than direct QeRL trained to convergence (31.79% vs. 28.53%) at lower total cost (approximately 105 vs. 132 GPU hours; Appendix C.1).

  • Quantitative support for local correction (HKh8, 4z7v). Our new analysis of the quantized student before distillation classifies 83.7% of 1,133 failed BFCL rollouts as primarily involving local execution errors. The added training statistics report teacher-reference coverage and verifier acceptance, and show that 94.0% of targeted divergent turns are corrected within four attempts (Appendices C.4–C.5).

  • Applicability and valid alternative actions (4z7v, 34AM). We clarified that successful student rollouts are retained unchanged, actions with equivalent environment effects are accepted, and rollouts without successful teacher references are still used for distillation. The revision also states the requirements for state restoration and reliable verification, and reports valid alternatives missed by transition matching in 18 of 302 examined failed rollouts (§3.1; Appendices A.1, C.4; Limitations).

Thank you again for considering our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.