acceptodds
Desk rejected at ICLR 2027

FarOPD: Learning from Future Interactions in On-Policy Distillation for Multi-Turn Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) provides dense teacher supervision for training large language model (LLM) agents on student-generated trajectories. However, each action shapes future observations and contexts, making next-token optimization alone myopic in multi-turn settings: actions with similar local supervision can lead to substantially different future interactions. We show that the trajectory-level reverse-KL objective intrinsically connects an action's supervision to its influence on future interactions. Teacher-student mismatch thus provides not only a local matching signal, but also hindsight credit for earlier actions. Based on this insight, we propose FarOPD (On-Policy Distillation with Future Action Rewards), which combines local-token supervision with a future-action reward derived from subsequent interactions. This makes action updates sensitive to future teacher-student mismatch, extending local imitation toward trajectory-level matching. Extensive experiments on three multi-turn agent benchmarks demonstrate consistent improvements over OPD baselines, with success rate gains of up to 8.4% relative to Vanilla OPD. These gains extend to multi-teacher distillation and are especially pronounced on tasks requiring longer interactions.

Primary area:
reinforcement learning
Keywords:
LLM-based Agents, Multi-turn Agents, On-policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VsqvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FarOPD, an on-policy distillation method for multi-turn LLM agents. It combines local token-level supervision with averaged rewards from subsequent actions, using future teacher–student mismatch to guide earlier action updates.

Critical strengths

The method has a clear motivation and is simple to implement. Experiments cover different agent benchmarks, student sizes, and both single- and multi-teacher settings, showing consistent improvements over the evaluated OPD baselines. The appendix also provides detailed experimental settings and derivations.

Critical weaknesses

(1) The differences from existing future-reward-based distillation methods need further clarification. MiniLLM also uses teacher-student log-ratio future rewards. Similarly, FarOPD extends this framework to multi-turn interaction via cross-action feedback and action averaging. However, the paper does not fully demonstrate what extra benefits these improvements bring, beyond adapting existing reward formulas to interaction trajectories.

(2) The theoretical analysis is based on cumulative trajectory-level KL, whereas the practical method uses averaged future rewards. Table 3 shows that the unaveraged future-reward variant fails to train under the reported settings. Although averaging improves task performance, the authors could further justify replacing cumulative future rewards with averaged rewards and provide a more detailed analysis of why the unaveraged variant collapses during training.

Questions for authors

(1) Could the authors clarify the key differences between FarOPD and MiniLLM, and compare FarOPD with a direct adaptation of MiniLLM to multi-turn interactions?

(2) How does averaging future rewards relate to the cumulative trajectory-level KL objective in Theorem 1? Could the authors analyze why the unaveraged variant collapses in Table 3 and report the trajectory-level KL during training to show whether the averaged update still reduces the original objective?

2 comments

W1 & Q1. Could the authors clarify the key differences between FarOPD and MiniLLM, and compare FarOPD with a direct adaptation of MiniLLM to multi-turn interactions?

Thank you for this comment. FarOPD develops hindsight credit assignment for multi-turn agents to account for how actions affect future interactions, whereas MiniLLM targets single-turn response generation.

When directly adapted to agents, MiniLLM (response-level) shows no clear improvement over Vanilla OPD and falls short of FarOPD. The key distinction is that an agent’s action changes subsequent interaction contexts through environment transitions and thereby affects future teacher–student mismatch (Fig. 1). Theorem 1 captures this dependence by separating the hindsight component from the trajectory-level reverse-KL gradient:

−∇θJ=gL+gW+gF,-\nabla_\theta\mathcal J=g_{\mathrm L}+g_{\mathrm W}+g_{\mathrm F},

where gL+gWg_{\mathrm L}+g_{\mathrm W} contains the response-level matching studied by MiniLLM, while gFg_{\mathrm F} captures an action’s effects on future mismatch through environment transitions. This reveals the distinct problem of credit assignment in multi-turn agent distillation.

Furthermore, simply extending MiniLLM’s token-level future rewards across turns is insufficient, as the trajectory-level adaptation still falls short of FarOPD (see below). These results support explicitly structuring credit around actions for effective future-action supervision. FarOPD uses action averaging to give future actions equal weight and prevent longer responses from dominating. We also drop the within-action term, focusing the future reward on the action’s effects on future interactions rather than autoregressive credit within a response. Table 3 supports both designs.

We ran two MiniLLM adaptations with Qwen2.5-0.5B:

  • Response-level retains the future return within the current response: Utresp=1Tturn−t+1∑j=tTturnγj−trjU_t^{\mathrm{resp}}=\frac{1}{T_{\mathrm{turn}}-t+1}\sum_{j=t}^{T_{\mathrm{turn}}}\gamma^{j-t}r_j
  • Trajectory-level concatenates action tokens across turns and extends the future return to the entire trajectory: Uttraj=1Ttraj−t+1∑j=tTtrajγj−trjU_t^{\mathrm{traj}}=\frac{1}{T_{\mathrm{traj}}-t+1}\sum_{j=t}^{T_{\mathrm{traj}}}\gamma^{j-t}r_j

where TturnT_{\mathrm{turn}} and TtrajT_{\mathrm{traj}} denote the final action-token positions of the current turn and full trajectory, respectively, and γ=0.95\gamma=0.95 is the discount factor. Environment observations condition generation but do not contribute rewards. The results are shown below.

MethodWebShop Succ.WebShop ScoreALFWorld Succ.
Vanilla OPD68.4983.8683.57
MiniLLM (Response-level)65.6382.2584.29
MiniLLM (Trajectory-level)68.7576.8185.71
FarOPD74.2284.8590.00

We have added this discussion and both MiniLLM comparisons in Appendix C.1 of the revised version.

W2 & Q2

The authors could further justify replacing cumulative future rewards with averaged rewards and provide a more detailed analysis of why the unaveraged variant collapses during training.

Thank you for raising this concern. We added a rescaled cumulative future reward ablation with Qwen2.5-0.5B. Rescaling restores successful training, supporting reward magnitude as a contributor to the collapse of the unaveraged variant.

Without averaging, the future reward accumulates log-ratios over all subsequent action tokens, so its magnitude can grow substantially with response length and remaining horizon. This large cumulative term is added to every token's local reward and can dominate the training signal, disrupting the balance between local and future supervision and destabilizing training.

To test this explanation, we rescaled cumulative future rewards by a fixed coefficient to match the root-mean-square magnitude of FarOPD's averaged future rewards (α=\alpha = 0.00046 on ALFWorld and 0.00057 on WebShop). These small coefficients reflect the substantial scale difference between cumulative and averaged rewards. Successful training after rescaling supports this explanation, while FarOPD's stronger performance shows the benefit of action averaging beyond controlling the overall reward magnitude.

MethodWebShop Succ.WebShop ScoreALFWorld Succ.
Vanilla OPD68.4983.8683.57
Rescaled future reward65.6373.9788.57
FarOPD74.2284.8590.00

We have added this comparison in Section 4.4 and Appendix C.4 of the revised version.

How does averaging future rewards relate to the cumulative trajectory-level KL objective in Theorem 1? Could the authors report the trajectory-level KL during training to show whether the averaged update still reduces the original objective?

Thank you for raising this question. Action averaging provides a practical approximation to the original trajectory-level KL gradient, and our additional experiments show that the averaged update can substantially reduce this KL during training.

As mentioned in lines 236–238 and 301–304, action averaging reweights future rewards, so the expected-update equivalence holds for the averaged reward's corresponding advantage. This reweighting may introduce bias relative to the original trajectory-level KL gradient.

We have additionally reported the trajectory-level reverse KL during FarOPD training in Appendix C.5 of the revision, showing a substantial decrease on both ALFWorld and WebShop. These results provide empirical evidence that FarOPD's averaged update can reduce the original trajectory-level KL objective.

Reviewer_L9iEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FarOPD improves distillation for multi-turn LLM agents by using teacher-student feedback from later interactions. This eventually can guide earlier actions. FarOPD cleverly combines standard token-level optimization with feedback from future actions. The authors provide results for three benchmarks and show FarOPD achieves consistent improvements (focus was on long-horizon tasks).

Critical strengths

Thank you so much for submitting your paper. Overall, I have understood the concept very well. Below are the main strengths from my point of view in no particular order.

  • Clever Theory and Approach: I particularly liked the approach you have gone here. Multi-turn LLM optimization is notoriously difficult and many works exist that try to tackle this from several different points of view. A clear theoretical approach is however often missing. You tackle it from a principled point of view by showing that the trajectory-level reverse-KL analysis indicates a future teacher–student mismatch that can be re-used to guide earlier actions. Your multi-turn distillation approach is thus fairly new or at least very well motivated and thus a compelling theory-backed argument on why this could improve training.
  • Promising Initial Results: Your experiments go on to show that these improvements seem to hold in practice. You show consistent trends on three standard tasks that are mainly targeted to long-horizon agentic tasks. This is also the core problem/task you want to solve and where FarOPD can make a difference. The results are also substantial and as far as I can judge.

There are also other very-welcomed points about the paper that I would like to mention:

  • Ablations: are very principled and answer Q2 clearly.
  • Overall presentation: figures are easy to read and theoretical explanations are clear.
  • Appendix: I also appreciated the expanded method details in the appendix, although, I could not find the time to read it all.
Critical weaknesses

Despite the aforementioned strengths, there are some weaknesses that follow from your theoretical background that need to be addressed.

  • Averaging Comaprison could be better: As far as I see (from Table 3), summing future rewards causes a complete training collapse. But I believe it also dramatically changes the reward scale and thus its dependence on trajectory length. Thus, without appropriately rescaling the summed rewards or retuning optimization, I believe that this comparison cannot really say that action averaging is intrinsically much better. The comparison against token averaging is more informative. I would suggest that you resolve this by showing, for example, a cumulative future reward baseline or a similar experiment that would support your method better. This would help distinguish the benefits of action averaging from those of simply controlling reward scale.
  • Model Generalizability: I appreciate the three model sizes, but results only on one model family (Qwen2.5) are not sufficient to bolster your approach's generalizability. I would like to see at least one other model family to be able to justify generalizability of your method.

There are some other (mostly weaker) points that I would like to rephrase as questions (down below) and are not critical.

Questions for authors

As mentioned above, there are some non-critical but nevertheless important points that I would like to phrase as questions to the authors.

1) Seeds: You mention three seeds but do not report statistical variance. I do not mind this at all and think it is cleaner for the paper presentation in the main body. However, you could add a few sentences or sub-results in the appendix that show that this does not change the impact/magnitude/usability of your method. Reviewers that are not familiar with Webshop/ALFWorld might criticize this, however, I do not. So this is not so much of a question, rather a support on your end where I would like to help in case a reviewer attacks this. But is there any reason you did not add it other than presentation?

2) Theory might not be complete: Formulating as a question since I am not quite sure. You mention that the exact gradient includes within-action credit and cumulative future rewards. I agree at first glance. FarOPD thus omits the former and averages the latter. The paper acknowledges these approximations, so this is not an undisclosed correctness error. However, the bias–variance proposition is conditional and does not establish that the implemented estimator actually has lower variance or mean-squared error in these experiments. Can you comment on that?

3) Reproducibility: Currently, I do not see a repository (which is fine). After checking the reproducibility statement, I think exact dataset splits, evaluation-set sizes, and teacher-training configurations could be more detailed. Can you commit to providing those? (does not need to be in the rebuttal). Alternatively, just add a repository as a zip file.

4) Table 2: I am not sure if I follow the presentation of the GiGPO results in this table. Why do we have results for Webshop for GiGPO (ALFWorld)?

5) Related Work: I think your related work review is good but should add some important papers that are close in spirit, in particular TSR [1], gammaOPD [2], and SERL [3]. These works primarily address search-guided rollout construction, temporal credit assignment in on-policy distillation, and another version of hindsight-based supervision for multi-turn agents. In particular, discussing gammaOPD’s treatment of future teacher–student mismatch and SERL’s use of environment-conditioned feedback would help clarify what is distinctive about FarOPD’s cross-action credit assignment and action-averaging scheme.

I would appreciate if you address these points!

[1] TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents [2] Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation [3] What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents

1 comment

W1. A cumulative future reward baseline or a similar experiment that would support your method better.

Thank you for this helpful suggestion. As suggested, we added a rescaled cumulative future reward baseline with Qwen2.5-0.5B. This variant trains successfully after rescaling, but matching reward scale alone does not recover FarOPD’s performance.

Specifically, we scale cumulative future rewards by a fixed coefficient calibrated separately for each environment to match the root-mean-square magnitude of FarOPD’s averaged future rewards (α=\alpha = 0.00046 on ALFWorld and 0.00057 on WebShop). This controls the overall reward magnitude while preserving cumulative token weighting.

MethodWebShop Succ.WebShop ScoreALFWorld Succ.
Vanilla OPD68.4983.8683.57
Rescaled future reward65.6373.9788.57
FarOPD74.2284.8590.00

These results support the benefit of action averaging beyond matching overall reward magnitude. This experiment has been added in Section 4.4 of the revised version.

W2. Model Generalizability.

We appreciate your suggestion to evaluate another model family. We have therefore extended our evaluation to Llama-3.2, distilling task-specific Llama-3.2-3B RL teachers into Llama-3.2-1B-Instruct students. FarOPD consistently outperforms both Vanilla OPD and TCOD-F2B, improving success rates over the best baseline by 7.81 percentage points on WebShop and 5.48 percentage points on ALFWorld. These results support FarOPD’s generalization beyond the Qwen2.5 family. Section 4.2 now summarizes this extension, and Appendix C.2 reports the full results.

MethodWebShop Succ.WebShop ScoreALFWorld Succ.
RL Teacher56.2567.0173.33
Zero-shot0.261.380.71
Vanilla OPD47.4058.7068.10
TCOD-F2B46.3560.8170.71
FarOPD55.2171.1076.19

Q1. Report statistical variance.

Thank you for this helpful suggestion. We omitted standard deviations from the main text to keep the presentation clear. We have now reported results as mean ± standard deviation in Appendix C.3 and Appendix C.2 of the revision, covering the Qwen2.5 and Llama-3.2 experiments, respectively.

Q2. The bias–variance proposition is conditional.

Thank you for this careful observation. Proposition 2 analyzes the bias–variance trade-off of action averaging from a theoretical perspective. It shows that averaging can make gradient estimates more accurate when the benefit of reducing random noise outweighs the systematic error it introduces.

Table 3 complements this theoretical analysis by evaluating the practical benefit of action averaging. The results support the effectiveness of this design, while Proposition 2 characterizes the conditions for improving estimator MSE. We have clarified these respective roles in Section 3.3 of the revised version.

Q3. Code Repository.

Our code is available in the anonymous repository, which is linked in the abstract. More detailed configurations for data preparation, evaluation, and teacher training can be found in the corresponding scripts and configuration files.

Q4. Why do we have results for Webshop for GiGPO (ALFWorld)?

Thank you for raising this question. GiGPO (ALFWorld) denotes the RL teacher trained on ALFWorld. Its WebShop results are included to illustrate the domain specialization of RL teachers: each teacher performs well in its own domain but transfers poorly to the other. This evaluation is explained in Section 4.3 of the revised version.

Q5. Related Work.

Thank you for these suggestions. We have added these discussion to Related Work of the revised version.

Reviewer_y8QEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on-policy distillation (OPD) for multi-turn LLM agents. Theorem 1 splits the trajectory-level reverse-KL gradient into a local part and a “hindsight” part that depends on future mismatch. Based on this, FarOPD adds one extra reward per action to the token-level OPD reward. It takes the mean log-ratio of each later action, then averages over the remaining turns (Eq. 8). The students are Qwen2.5-0.5B/1.5B, and the teachers are GiGPO-trained Qwen2.5-7B models. In six single-teacher settings, FarOPD beats vanilla OPD by 3.4 to 5.7 points in success rate (Table 1).

Critical strengths
  1. The change costs almost nothing, and it helps every time. FarOPD beats vanilla OPD in all 10 settings (Tables 1-2). It reuses the log-ratios OPD already computes, with no extra teacher calls, rollouts or critic. No single margin clears the noise by much, but 10 wins out of 10 is unlikely by chance.

  2. Others can reuse the averaging ablation (Table 3, 0.5B only). Summing the future rewards makes training collapse (0.00 success). Per-action averaging beats per-token averaging by 8.3 points on WebShop (74.22 vs 65.89). On ALFWorld the gap is 3.6 points (90.00 vs 86.43). Per-token averaging resembles MiniLLM’s length normalization, and on WebShop it even falls below vanilla OPD.

Critical weaknesses
  1. The paper does not test whether hindsight credit is what helps. The method uses one rollout per task and no baseline to center the reward. In Fig. 4, the future and local rewards have about the same sign and size. So part of the gain may simply come from a larger negative reward. A shuffled or batch-mean future reward would check this. The gain over vanilla OPD looks real. But at 1.5B, the margin over the best baseline is only about 2.6 points, from 3-seed means with no std.

  2. MiniLLM (Gu et al., ICLR 2024) already has the main theory. Its Sec. 2.2 splits the reverse-KL gradient into a single-step term and a future term. Its Eq. 6 normalizes the future term by the remaining length. Theorem 1 carries this across turns, but the paper cites MiniLLM only as the origin of OPD. Apart from the theorem, the new parts are per-action averaging and dropping the within-action term. Given margins near noise, this is a small step for ICLR.

Questions for authors
  1. Four ALFWorld averages in Table 1 do not match the per-category numbers. For example, 0.5B vanilla OPD gives 351/420 = 83.57, not 84.52. Which numbers are correct?

  2. App. A.4 says the advantage of the averaged reward differs from A^H in Eq. (6). Does FarOPD then optimize a different, length-normalized objective?

3 comments

W1-1. The paper does not test whether hindsight credit is what helps. ... So part of the gain may simply come from a larger negative reward. A shuffled or batch-mean future reward would check this.

Thank you for this helpful suggestion. As suggested, we added shuffled and rescaled future reward ablations with Qwen2.5-0.5B. Shuffling future rewards reduces performance and matching reward scale alone does not recover FarOPD’s performance. These results highlight the importance of FarOPD’s future-action reward and action averaging designs. This experiment has been added in the Section 4.4 in the revision.

  • Shuffled future reward: Randomly reassign future rewards across actions within a batch while preserving their values.
  • Rescaled future reward: Scale cumulative future rewards by a fixed coefficient to match the root-mean-square magnitude of FarOPD’s future rewards.
MethodWebShop Succ.WebShop ScoreALFWorld Succ.
Vanilla OPD68.4983.8683.57
Shuffled future reward64.8471.6883.57
Rescaled future reward65.6373.9788.57
FarOPD74.2284.8590.00

W1-2. The method uses one rollout per task and no baseline to center the reward.

Thank you for raising this question. As mentioned in lines 222–224, a context-dependent baseline contributes zero to the expected update and therefore need not be explicitly estimated. The same identity holds for FarOPD’s action-averaged reward and its corresponding centered advantage (lines 237–238). One on-policy rollout per task provides a Monte Carlo sample of the update, and training combines these samples across tasks to estimate the expected update.

We have clarified these points in Section 3.2 and Appendix A.3 of the revised version.

W1-3. At 1.5B, the margin over the best baseline is only about 2.6 points, from 3-seed means with no std.

Thank you for raising this concern. We performed a bootstrap analysis of the success-rate gains at 1.5B, finding a statistically significant improvement on WebShop and positive estimated gains on ALFWorld and ScienceWorld.

Appendix C.3 reports the mean ± std results and the 95% bootstrap confidence intervals. The results are summarized below.

BenchmarkFarOPDBest BaselineImprovement95% CI
WebShop76.82 ± 2.3474.22 ± 2.26+2.60[0.52, 4.72]
ALFWorld91.43 ± 0.7188.57 ± 0.71+2.86[−0.48, 6.43]
ScienceWorld60.16 ± 0.7857.55 ± 1.63+2.61[−0.54, 5.98]

W2. MiniLLM already has the main theory.

Thank you for this comment. FarOPD develops hindsight credit assignment for multi-turn agents to account for how actions affect future interactions, whereas MiniLLM targets single-turn response generation.

When directly adapted to agents, MiniLLM (response-level) shows no clear improvement over Vanilla OPD and falls short of FarOPD. The key distinction is that an agent’s action changes subsequent interaction contexts through environment transitions and thereby affects future teacher–student mismatch (Fig. 1). Theorem 1 captures this dependence by separating the hindsight component from the trajectory-level reverse-KL gradient:

−∇θJ=gL+gW+gF,-\nabla_\theta\mathcal J=g_{\mathrm L}+g_{\mathrm W}+g_{\mathrm F},

where gL+gWg_{\mathrm L}+g_{\mathrm W} contains the response-level matching studied by MiniLLM, while gFg_{\mathrm F} captures an action’s effects on future mismatch through environment transitions. This reveals the distinct problem of credit assignment in multi-turn agent distillation.

Furthermore, simply extending MiniLLM’s token-level future rewards across turns is insufficient, as the trajectory-level adaptation still falls short of FarOPD (see below). These results support explicitly structuring credit around actions for effective future-action supervision. FarOPD uses action averaging to give future actions equal weight and prevent longer responses from dominating. We also drop the within-action term, focusing the future reward on the action’s effects on future interactions rather than autoregressive credit within a response. Table 3 supports both designs.

We ran two MiniLLM adaptations with Qwen2.5-0.5B:

  • Response-level retains the future return within the current response: Utresp=1Tturn−t+1∑j=tTturnγj−trjU_t^{\mathrm{resp}}=\frac{1}{T_{\mathrm{turn}}-t+1}\sum_{j=t}^{T_{\mathrm{turn}}}\gamma^{j-t}r_j
  • Trajectory-level concatenates action tokens across turns and extends the future return to the entire trajectory: Uttraj=1Ttraj−t+1∑j=tTtrajγj−trjU_t^{\mathrm{traj}}=\frac{1}{T_{\mathrm{traj}}-t+1}\sum_{j=t}^{T_{\mathrm{traj}}}\gamma^{j-t}r_j

where TturnT_{\mathrm{turn}} and TtrajT_{\mathrm{traj}} denote the final action-token positions of the current turn and full trajectory, respectively, and γ=0.95\gamma=0.95 is the discount factor. Environment observations condition generation but do not contribute rewards. The results are shown below.

MethodWebShop Succ.WebShop ScoreALFWorld Succ.
Vanilla OPD68.4983.8683.57
MiniLLM (Response-level)65.6382.2584.29
MiniLLM (Trajectory-level)68.7576.8185.71
FarOPD74.2284.8590.00

We have added this discussion and both MiniLLM comparisons in Appendix C.1 of the revised version.

Q1. Four ALFWorld averages in Table 1 do not match the per-category numbers.

Thank you for carefully checking these results. We apologize for this reporting oversight. The per-category results are correct, and we have corrected the four inconsistent averages in the revision. This occurred because, when consolidating the multi-seed results, we updated the per-category values but inadvertently left these four averages unchanged. The corrections are shown below:

MethodALFWorld Avg.
0.5B Vanilla OPD84.52 → 83.57
0.5B SOD84.76 → 84.52
1.5B Vanilla OPD86.43 → 86.19
1.5B TCOD-B2F88.81 → 88.57

These corrections are minor and do not change any pairwise performance ordering or affect the paper's conclusions. FarOPD consistently outperforms all baselines. Thank you again for bringing this issue to our attention.

Q2. App. A.4 says the advantage of the averaged reward differs from A^H in Eq. (6). Does FarOPD then optimize a different, length-normalized objective?

Thank you for raising this question. FarOPD does not define a new length-normalized KL objective. Instead, action averaging provides a practical approximation to the original trajectory-level KL gradient.

As mentioned in lines 236–238, action averaging reweights future rewards, so the expected-update equivalence holds for the averaged reward's corresponding advantage A~tH\widetilde A_t^{\mathrm H}, rather than the original AtHA_t^{\mathrm H} in Eq. (6). We explicitly acknowledge in lines 301–304 that this reweighting may introduce bias relative to the original trajectory-level KL gradient. The practical optimization therefore uses the surrogate defined in Eq. (10), rather than the exact KL gradient.

Thank you for highlighting this ambiguity. We have clarified this distinction in Section 3.3 and Appendix A.4 of the revised version.

Comments on the paper

2 comments

Thank you for the detailed clarifications and additional experiments. They address my main concerns. My score is already positive, and I’m happy to keep it as is.

Dear Reviewer Vsqv,

Thank you very much for your positive feedback. We are glad that our response and additional experiments have addressed your main concerns. We sincerely appreciate your constructive feedback and suggestions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.