- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FarOPD, an on-policy distillation method for multi-turn LLM agents. It combines local token-level supervision with averaged rewards from subsequent actions, using future teacher–student mismatch to guide earlier action updates.
Critical strengths
The method has a clear motivation and is simple to implement. Experiments cover different agent benchmarks, student sizes, and both single- and multi-teacher settings, showing consistent improvements over the evaluated OPD baselines. The appendix also provides detailed experimental settings and derivations.
Critical weaknesses
(1) The differences from existing future-reward-based distillation methods need further clarification. MiniLLM also uses teacher-student log-ratio future rewards. Similarly, FarOPD extends this framework to multi-turn interaction via cross-action feedback and action averaging. However, the paper does not fully demonstrate what extra benefits these improvements bring, beyond adapting existing reward formulas to interaction trajectories.
(2) The theoretical analysis is based on cumulative trajectory-level KL, whereas the practical method uses averaged future rewards. Table 3 shows that the unaveraged future-reward variant fails to train under the reported settings. Although averaging improves task performance, the authors could further justify replacing cumulative future rewards with averaged rewards and provide a more detailed analysis of why the unaveraged variant collapses during training.
Questions for authors
(1) Could the authors clarify the key differences between FarOPD and MiniLLM, and compare FarOPD with a direct adaptation of MiniLLM to multi-turn interactions?
(2) How does averaging future rewards relate to the cumulative trajectory-level KL objective in Theorem 1? Could the authors analyze why the unaveraged variant collapses in Table 3 and report the trajectory-level KL during training to show whether the averaged update still reduces the original objective?
2 comments
W2 & Q2
The authors could further justify replacing cumulative future rewards with averaged rewards and provide a more detailed analysis of why the unaveraged variant collapses during training.
Thank you for raising this concern. We added a rescaled cumulative future reward ablation with Qwen2.5-0.5B. Rescaling restores successful training, supporting reward magnitude as a contributor to the collapse of the unaveraged variant.
Without averaging, the future reward accumulates log-ratios over all subsequent action tokens, so its magnitude can grow substantially with response length and remaining horizon. This large cumulative term is added to every token's local reward and can dominate the training signal, disrupting the balance between local and future supervision and destabilizing training.
To test this explanation, we rescaled cumulative future rewards by a fixed coefficient to match the root-mean-square magnitude of FarOPD's averaged future rewards ( 0.00046 on ALFWorld and 0.00057 on WebShop). These small coefficients reflect the substantial scale difference between cumulative and averaged rewards. Successful training after rescaling supports this explanation, while FarOPD's stronger performance shows the benefit of action averaging beyond controlling the overall reward magnitude.
| Method | WebShop Succ. | WebShop Score | ALFWorld Succ. |
|---|---|---|---|
| Vanilla OPD | 68.49 | 83.86 | 83.57 |
| Rescaled future reward | 65.63 | 73.97 | 88.57 |
| FarOPD | 74.22 | 84.85 | 90.00 |
We have added this comparison in Section 4.4 and Appendix C.4 of the revised version.
How does averaging future rewards relate to the cumulative trajectory-level KL objective in Theorem 1? Could the authors report the trajectory-level KL during training to show whether the averaged update still reduces the original objective?
Thank you for raising this question. Action averaging provides a practical approximation to the original trajectory-level KL gradient, and our additional experiments show that the averaged update can substantially reduce this KL during training.
As mentioned in lines 236–238 and 301–304, action averaging reweights future rewards, so the expected-update equivalence holds for the averaged reward's corresponding advantage. This reweighting may introduce bias relative to the original trajectory-level KL gradient.
We have additionally reported the trajectory-level reverse KL during FarOPD training in Appendix C.5 of the revision, showing a substantial decrease on both ALFWorld and WebShop. These results provide empirical evidence that FarOPD's averaged update can reduce the original trajectory-level KL objective.
Thank you for this comment. FarOPD develops hindsight credit assignment for multi-turn agents to account for how actions affect future interactions, whereas MiniLLM targets single-turn response generation.
When directly adapted to agents, MiniLLM (response-level) shows no clear improvement over Vanilla OPD and falls short of FarOPD. The key distinction is that an agent’s action changes subsequent interaction contexts through environment transitions and thereby affects future teacher–student mismatch (Fig. 1). Theorem 1 captures this dependence by separating the hindsight component from the trajectory-level reverse-KL gradient:
−∇θJ=gL+gW+gF,
where gL+gW contains the response-level matching studied by MiniLLM, while gF captures an action’s effects on future mismatch through environment transitions. This reveals the distinct problem of credit assignment in multi-turn agent distillation.
Furthermore, simply extending MiniLLM’s token-level future rewards across turns is insufficient, as the trajectory-level adaptation still falls short of FarOPD (see below). These results support explicitly structuring credit around actions for effective future-action supervision. FarOPD uses action averaging to give future actions equal weight and prevent longer responses from dominating. We also drop the within-action term, focusing the future reward on the action’s effects on future interactions rather than autoregressive credit within a response. Table 3 supports both designs.
We ran two MiniLLM adaptations with Qwen2.5-0.5B:
where Tturn and Ttraj denote the final action-token positions of the current turn and full trajectory, respectively, and γ=0.95 is the discount factor. Environment observations condition generation but do not contribute rewards. The results are shown below.
We have added this discussion and both MiniLLM comparisons in Appendix C.1 of the revised version.