acceptodds
Under review as a conference paper at ICLR 2027

SAPD: Learning from Agent Experience without Environment Interaction

Abstract

Scaling the training of Large Language Model (LLM) agents typically requires a large collection of executable environments, whose construction, maintenance, and operation constrain the expansion of training experience. Existing offline trajectories offer an opportunity to reuse interaction experience, yet effectively refining the current policy without revisiting these environments remains challenging: supervised fine-tuning (SFT) can lead to catastrophic forgetting, while online reinforcement learning requires real-time interaction with the environment. To address this challenge, we propose Step-Advantage Policy Distillation (SAPD), a method for learning from offline trajectories using relative step advantages. SAPD decomposes multi-turn trajectories into individual decision contexts for on-policy distillation, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training. The entire process requires neither new environment observations nor additional multi-turn interactions. Experiments on ALFWorld, WebShop, and Search-based QA show that SAPD improves performance across different teacher configurations, mitigates catastrophic forgetting compared with SFT, and runs at least faster per training step than online OPD. We will release our code upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm agent, on-policy distillation, environment-free agentic learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jC63OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAPD learns from recorded agent histories without executing new actions during policy refinement. It generates response groups under offline prefixes, combines token-level distillation with centered action-level teacher–student scores, and adds local rule rewards. Experiments cover ALFWorld, WebShop, search QA, sequential retention, and training-time comparisons with online distillation.

Critical strengths
  1. The environment-free refinement formulation is useful and clearly bounded: response generation is on-policy conditional on recorded histories, without claiming on-policy state coverage. The step signal and local rewards provide an implementable use of existing trajectories.
  2. The sequential-training comparison provides strong evidence of retention relative to SFT in the tested order. The online-OPD comparison reports cumulative time as well as per-step time, making the efficiency claim more informative.
Critical weaknesses
  1. The robustness analysis removes a shared score offset, but does not establish that shared offsets dominate actual teacher errors under prefix mismatch. Relative preference errors and variable action lengths remain, as the paper acknowledges. The experiments do not directly validate that this error model explains the observed gains.
  2. Several incremental gains, particularly on search QA, are small, and the main tables do not provide repeated-training uncertainty. This makes it difficult to distinguish consistent step-advantage benefits from training variation across the teacher configurations.
Questions for authors
  1. Do controlled shared-offset, response-dependent-error, and action-length experiments support the proposed explanation for robustness?
  2. What is the variation across independent training runs for SAPD versus OPD+Rule, especially on search QA?
3 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you very much for reviewing our paper and for acknowledging the value of our work in the review. Regarding the concerns raised in the review, we provide the following responses. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to CW1 and Q1: Shared-Offset Assumption

Many thanks for your constructive comments. To connect the shared-offset analysis to the observed learning gains, we provide an online-prefix comparison and ablations of centering and standard-deviation normalization, alongside the main experimental results. We also clarify how response-dependent preference errors and variable action lengths are represented in our error model.

Shared offsets, residual errors, and evidence from the main experiments.

For responses sharing a prefix hh, our model writes ristep=ri⋆+b(h)+eir_i^{\mathrm{step}}=r_i^\star+b(h)+e_i. Group centering gives Aistep=(ri⋆−rˉ⋆)+(ei−eˉ)A_i^{\mathrm{step}}=(r_i^\star-\bar r^\star)+(e_i-\bar e), removing the shared component while explicitly retaining response-dependent residual errors. Thus, the model accommodates imperfect relative preferences. The empirical question is whether removing the shared component yields a useful relative signal for learning in the presence of these residual errors.

Variable action lengths. As detailed in Appendix A, if the token-level error contains a shared component btok(h)b_{\mathrm{tok}}(h), summing over NiN_i action tokens gives Δi=Nibtok(h)+ui\Delta_i=N_i b_{\mathrm{tok}}(h)+u_i, where uiu_i is the accumulated token-level residual. For a reference length N0(h)N_0(h), this fits the aggregate model with b(h)=N0(h)btok(h)b(h)=N_0(h)b_{\mathrm{tok}}(h) and ei=(Ni−N0(h))btok(h)+uie_i=(N_i-N_0(h))b_{\mathrm{tok}}(h)+u_i. Length-dependent variation is therefore included in the residual term, and the analysis applies without assuming equal action lengths. The observed learning gains below show that the centered signal is useful in our evaluated settings, where responses can differ in both length and scoring error.

The experiments support this expectation. In Table 1 in the main paper, SAPD outperforms OPD+Rule by 6.5 and 4.4 percentage points on ALFWorld and WebShop, respectively, with a GRPO teacher, and by 2.6 and 6.3 percentage points with an SFT teacher. These results demonstrate that AistepA_i^{\mathrm{step}} contributes to improved learning. We next present controlled experiments to examine the role of mismatch-induced shared offsets in these gains, followed by ablations of centering and standard-deviation normalization.

Additional controlled experiments with smaller shared offset.

To assess the role of the shared offset b(h)b(h), we compare SAPD and OPD+Rule in a setting where the mismatch that motivates this offset is removed. Specifically, we replace fixed offline prefixes with histories generated online by the current student through environment interactions. In this controlled comparison, we focus on the mismatch between fixed offline histories and histories generated by the current student policy. Under our error model, using these on-policy histories is therefore expected to substantially reduce the shared offset associated with this source of mismatch.

If removing this offset is an important source of SAPD’s gains under offline prefixes, we would expect its advantage over OPD+Rule to diminish in this online setting. The results in Table 1 support this prediction: SAPD’s gains decrease from 6.5 to 0.8 percentage points on ALFWorld and from 4.4 to 1.6 percentage points on WebShop.

This contrast provides empirical support for the role of mismatch-induced shared offsets in explaining SAPD’s gains, complementing the algebraic cancellation established in Proposition 1. While changing the prefix source does not isolate b(h)b(h) from all other distributional changes, the smaller gains under on-policy histories are consistent with our proposed mechanism. We have added these results to Appendix C.5.

Table 1: Comparison across prefix sources

MethodPrefix SourceALFWorldWebShop
GRPO Teacher 3B—89.164.8
OPD+Ruleoffline62.849.0
SAPDoffline69.353.4
OPD+RuleOnline student-generated62.552.3
SAPDOnline student-generated63.353.9
AuthorsOpenReview ↗

Author Response (2/3)

Response to CW1 and Q1: Shared-Offset Assumption

Additional experiments with uncentered or standardized signals

To further examine how the shared-offset explanation relates to the observed gains, we compare three signal constructions: the uncentered score Airaw=ristepA_i^{\mathrm{raw}}=r_i^{\mathrm{step}}, the group-centered advantage Aictr=ristep−rˉstepA_i^{\mathrm{ctr}}=r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}, and the standardized advantage Aistd=ristep−rˉstepstd⁡(rstep)+ηA_i^{\mathrm{std}}=\frac{r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}}{\operatorname{std}(r^{\mathrm{step}})+\eta}, where η\eta is a numerical stabilizer. We keep the other training hyperparameters fixed, although we do not explicitly match the realized update magnitudes across variants.

Effect of group centering. Under our error model, removing centering reintroduces the shared offset b(h)b(h), together with rˉ⋆\bar r^\star and eˉ\bar e, since Airaw−Aictr=rˉstep=rˉ⋆+b(h)+eˉA_i^{\mathrm{raw}}-A_i^{\mathrm{ctr}}=\bar r^{\mathrm{step}}=\bar r^\star+b(h)+\bar e. As shown in Table 2, centering improves performance by 2.8 percentage points on ALFWorld and 8.4 percentage points on WebShop. These results support the benefit of removing the group-level score component, which includes the shared offset. This complements the online-prefix comparison above: SAPD’s gains diminish when the mismatch is reduced, while performance also declines when centering is removed. Both observations are consistent with our explanation that removing mismatch-induced shared offsets contributes to SAPD’s gains, although neither comparison isolates b(h)b(h) alone.

Effect of standard-deviation normalization. Centering alone also outperforms standardization, achieving 96.6% versus 72.1% on ALFWorld and 59.4% versus 39.6% on WebShop. Dividing by a small group standard deviation can amplify weak within-group score differences, potentially increasing the influence of scoring noise. The observed degradation is consistent with our motivation to preserve the original scale of these differences. However, without explicitly matching update magnitudes, we cannot attribute the degradation solely to noise amplification, since standardization also changes the scale and relative weight of the step objective during optimization.

Together, the smaller gains with on-policy prefixes and the performance drop when centering is removed provide complementary empirical support for the role of mismatch-induced shared offsets in SAPD’s gains. The main results and ablations further show that the centered relative signal provides useful supervision in the presence of response-dependent errors and variable action lengths. The normalization comparison favors group centering without standard-deviation normalization under the tested configuration. We have added the prefix comparison to Appendix C.5 and the signal ablations to Appendix D.2.

Table 2: Ablation of centering and standard-deviation normalization

MethodALFWorldWebShop
SFT Teacher 3B94.559.7
AictrA_i^{\mathrm{ctr}}96.659.4
AirawA_i^{\mathrm{raw}}93.851.0
AistdA_i^{\mathrm{std}}72.139.6

AuthorsOpenReview ↗

Author Response (3/3)

Response to CW2 and Q2: Results Across Random Seeds

Thank you for your comments. We conducted three independent training runs each for SAPD and OPD+Rule with an SFT teacher. The table reports the mean and sample standard deviation of success rates. These results are also reported in Appendix C.7.

Table 3: Results across independent random seeds

MethodALFWorldWebShopSearch QA
OPD+Rule95.3 ± 0.853.1 ± 3.438.6 ± 0.3
SAPD98.4 ± 0.859.6 ± 0.938.8 ± 0.1

SAPD achieves higher mean performance on all three tasks, with particularly consistent gains on ALFWorld and WebShop of 3.1 and 6.5 percentage points, respectively.

We agree that SAPD’s improvement on Search-based QA is small, with a mean improvement of only 0.2 percentage points. This is consistent with the observation in Section 5.2 that long-horizon tasks benefit more from SAPD. ALFWorld allows up to 50 interaction steps and WebShop up to 15, compared with only four for Search-based QA. Over longer interactions, early decisions shape subsequent observations and actions, allowing errors and mismatches between teacher and student behavior to accumulate. SAPD compares candidate actions under the same prefix, reducing the impact of context mismatch and providing more stable guidance for policy optimization.


We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.


Reviewer_V8VQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAPD, a method for training LLM agents using offline trajectories without further environment interaction. It uses recorded histories as prefixes and generates new responses from the student model. It then uses teacher scores to compute relative step advantages for policy optimization.

Critical strengths
  1. The paper addresses an important problem of reducing environment interaction costs in agent training.
  2. The proposed step advantage method is simple and improves policy distillation using offline trajectories.
  3. The experiments cover three agent benchmarks and show improvements in task performance and training efficiency.
Critical weaknesses
  1. The paper assumes that teacher scoring errors share a common offset under prefix mismatch, but this assumption is not verified. The paper states in Appendix A.5 that experiments are needed to test “sensitivity to response-dependent perturbations and teacher ranking errors” and “the effect of standard-deviation normalization under weak within-group score differences.” However, these experiments are not reported.
  2. The paper mainly compares SAPD with SFT, GRPO, and OPD variants. Comparisons with more recent offline and prefix-based distillation methods under the same settings are missing.
  3. The first two rows of Table 1 (SFT and GRPO) are not clearly explained, especially their model and training settings. Table 3 also does not specify which teacher model is used.
Questions for authors

Please see the weaknesses above.

8 comments
AuthorsOpenReview ↗

Author Response (1/4)

Thank you very much for your careful review of our paper and for providing so many valuable suggestions. We respond to your concerns below. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to CW1: Shared-Offset Assumption

Many thanks for highlighting the empirical questions raised in Appendix A.5. We agree that Proposition 1 establishes the cancellation of a shared offset under the stated error model, but does not by itself verify the structure of teacher errors in practice. We address this concern through three sets of experiments: those reported in the main paper that support our assumption, additional controlled experiments with a smaller shared offset, and experiments using uncentered or standardized signals.

Evidence from the main experiments and the scope of Proposition 1.

Under our error model, responses sharing the same prefix hh have step scores ristep=ri⋆+b(h)+eir_i^{\mathrm{step}}=r_i^\star+b(h)+e_i, where b(h)b(h) is a shared offset and eie_i is a response-dependent residual error. Group centering gives Aistep=(ri⋆−rˉ⋆)+(ei−eˉ)A_i^{\mathrm{step}}=(r_i^\star-\bar r^\star)+(e_i-\bar e). It therefore removes the shared offset while retaining the centered residual error. Centering does not change within-group response rankings or correct ranking errors caused by these residuals; its usefulness depends on the remaining relative signal retaining information that helps policy improvement.

The experiments support this expectation. In Table 1 in the main paper, SAPD outperforms OPD+Rule by 6.5 and 4.4 percentage points on ALFWorld and WebShop, respectively, with a GRPO teacher, and by 2.6 and 6.3 percentage points with an SFT teacher. These results demonstrate that AistepA_i^{\mathrm{step}} contributes to improved learning. We next present controlled experiments to examine the role of mismatch-induced shared offsets in these gains, followed by ablations of centering and standard-deviation normalization.

Additional controlled experiments with smaller shared offset.

To assess the role of the shared offset b(h)b(h), we compare SAPD and OPD+Rule in a setting where the mismatch that motivates this offset is removed. Specifically, we replace fixed offline prefixes with histories generated online by the current student through environment interactions. In this controlled comparison, we focus on the mismatch between fixed offline histories and histories generated by the current student policy. Under our error model, using these on-policy histories is therefore expected to substantially reduce the shared offset associated with this source of mismatch.

If removing this offset is an important source of SAPD’s gains under offline prefixes, we would expect its advantage over OPD+Rule to diminish in this online setting. The results in Table 1 support this prediction: SAPD’s gains decrease from 6.5 to 0.8 percentage points on ALFWorld and from 4.4 to 1.6 percentage points on WebShop.

This contrast provides empirical support for the role of mismatch-induced shared offsets in explaining SAPD’s gains, complementing the algebraic cancellation established in Proposition 1. While changing the prefix source does not isolate b(h)b(h) from all other distributional changes, the smaller gains under on-policy histories are consistent with our proposed mechanism. We have added these results to Appendix C.5.

Table 1: Comparison across prefix sources

MethodPrefix SourceALFWorldWebShop
GRPO Teacher 3B—89.164.8
OPD+Ruleoffline62.849.0
SAPDoffline69.353.4
OPD+RuleOnline student-generated62.552.3
SAPDOnline student-generated63.353.9
AuthorsOpenReview ↗

Author Response (2/4)

Response to CW1: Shared-Offset Assumption

Additional experiments with uncentered or standardized signals

To examine the shared-offset explanation and the normalization question raised in Appendix A.5, we compare three signal constructions: the uncentered score Airaw=ristepA_i^{\mathrm{raw}}=r_i^{\mathrm{step}}, the group-centered advantage Aictr=ristep−rˉstepA_i^{\mathrm{ctr}}=r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}, and the standardized advantage Aistd=ristep−rˉstepstd⁡(rstep)+ηA_i^{\mathrm{std}}=\frac{r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}}{\operatorname{std}(r^{\mathrm{step}})+\eta}, where η\eta is a numerical stabilizer. We keep the other training hyperparameters fixed, although we do not explicitly match the realized update magnitudes across variants.

Effect of group centering. Under our error model, removing centering reintroduces the shared offset b(h)b(h), together with rˉ⋆\bar r^\star and eˉ\bar e, since Airaw−Aictr=rˉstep=rˉ⋆+b(h)+eˉA_i^{\mathrm{raw}}-A_i^{\mathrm{ctr}}=\bar r^{\mathrm{step}}=\bar r^\star+b(h)+\bar e. As shown in Table 2, centering improves performance by 2.8 percentage points on ALFWorld and 8.4 percentage points on WebShop. These results support the benefit of removing the group-level score component, which includes the shared offset. This complements the online-prefix comparison above: SAPD’s gains diminish when the mismatch is reduced, while performance also declines when centering is removed. Both observations are consistent with our explanation that removing mismatch-induced shared offsets contributes to SAPD’s gains, although neither comparison isolates b(h)b(h) alone.

Effect of standard-deviation normalization. Centering alone also outperforms standardization, achieving 96.6% versus 72.1% on ALFWorld and 59.4% versus 39.6% on WebShop. Dividing by a small group standard deviation can amplify weak within-group score differences, potentially increasing the influence of scoring noise. The observed degradation is consistent with our motivation to preserve the original scale of these differences. However, without explicitly matching update magnitudes, we cannot attribute the degradation solely to noise amplification, since standardization also changes the scale and relative weight of the step objective during optimization.

Overall, these results favor group centering without standard-deviation normalization under the tested configuration and provide complementary evidence for the shared-offset explanation. These results suggest that the centered relative signal retains useful information for policy improvement despite response-dependent scoring errors. We have added these results to Appendix D.2.

Summary of the additional evidence. Together, the online-prefix comparison and centering ablation provide complementary empirical support for the role of mismatch-induced shared offsets in SAPD’s gains. The learning improvements suggest that the centered relative signal retains useful information for policy improvement under the teacher errors encountered in our experiments. The normalization ablation further supports group centering without standard-deviation normalization under the tested configuration.

Table 2: Ablation of centering and standard-deviation normalization

MethodALFWorldWebShop
SFT Teacher 3B94.559.7
AictrA_i^{\mathrm{ctr}}96.659.4
AirawA_i^{\mathrm{raw}}93.851.0
AistdA_i^{\mathrm{std}}72.139.6

AuthorsOpenReview ↗

Author Response (3/4)

Response to CW2: Additional Baselines

Thank you for the comment. Our work focuses on multi-turn OPD that directly reuses existing interaction trajectories without additional environment interactions. To our knowledge, SAPD is the first study of this setting.

The closest recent offline OPD method is ReOPD, which uses trajectories generated by a teacher through environment interactions. We report the ReOPD comparison in Appendix C.2. Under the same teacher-generated prefix setting, with a 1.5B student and a GRPO-trained 3B teacher, the following results demonstrate SAPD’s effectiveness.

Table 3: Comparison with ReOPD

MethodALFWorldWebShop
GRPO Teacher 3B89.164.8
ReOPD75.066.7
SAPD76.670.8

Existing published prefix-based methods have not explored our fully environment-free setting. To further compare with recent methods addressing prefix mismatch, we adapt RLSD [1], which has been accepted as a NeurIPS 2026 Spotlight and has 172 citations on Google Scholar, to offline prefixes. To adapt RLSD to our environment-free setting, we construct step advantages from teacher distillation scores while retaining its other mechanisms. With a 1.5B student and a 3B SFT teacher, the results are in Table 4, showing that SAPD achieves better performance than adapted RLSD, supporting the effectiveness of SAPD in addressing context mismatch under environment-free settings. This comparison is included in Appendix C.3.

Table 4: Comparison with adapted RLSD

MethodALFWorldWebShop
SFT Teacher 3B94.559.7
Adapted RLSD93.054.2
SAPD96.659.4

Reviewer_Wu65OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAPD refines an agent using fixed decision histories extracted from offline trajectories. The student generates several single-turn responses for each history, and a frozen teacher scores the generated tokens without executing the proposed actions. SAPD sums teacher–student log-probability differences over action tokens, centers these scores within each prefix-conditioned group, and broadcasts the resulting step advantages over the response tokens. Training combines this objective with token-level distillation and local format/repetition rewards. A shared-offset analysis motivates relative scoring while explicitly retaining limitations from response-dependent errors and variable action length. Experiments cover ALFWorld, WebShop, and Search-based QA, including two teacher configurations, component and scoring-mask ablations, sequential training, and a runtime comparison with online OPD.

Critical strengths
  1. SAPD requires no new environment observations during refinement and learns from student-generated alternatives rather than simply imitating the recorded next action. Table 1 compares against OPD+Rule on the same offline prefixes, providing a relevant control for the added step objective.

  2. Table 1 compares against OPD+Rule on the same offline prefixes, and the improvements are most meaningful on ALFWorld and WebShop: with the GRPO teacher, SAPD improves from 62.8% to 69.3% and from 49.0% to 53.4%, respectively; with the SFT teacher, WebShop improves from 53.1% to 59.4%.

Critical weaknesses
  1. Proposition 1 establishes that centering removes a shared score offset, but it does not show that teacher errors on mismatched histories are predominantly shared offsets or that the remaining rankings are reliable.

  2. Table 3 compares only SAPD and SFT in one task order. It therefore shows that the complete SAPD procedure forgets less than the particular SFT procedure, but not whether step advantages improve retention over offline OPD+Rule or ReOPD.

Questions for authors
  1. With identical prefixes, candidate responses, rule rewards, and matched update magnitudes, how do centered step scores compare with uncentered sequence scores and with standardized step advantages?
5 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you very much for reviewing our paper and for providing so many valuable suggestions. In response to your concerns, we provide the following replies. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to CW1 and Q1: Shared-Offset Assumption

Many thanks for your constructive comments. We address this concern through three sets of experiments: those reported in the main paper that support our assumption, additional controlled experiments with a smaller shared offset, and experiments using uncentered or standardized signals.

Evidence from the main experiments about the shared offset and response rankings.

Under our assumption, for responses sharing the same prefix hh, the step score is ristep=ri⋆+b(h)+eir_i^{\mathrm{step}}=r_i^\star+b(h)+e_i, where ri⋆r_i^\star is the ideal action score, b(h) is a shared offset, and eie_i is a response-dependent residual error. Group centering gives Aistep=ristep−rˉstep=(ri⋆−rˉ⋆)+(ei−eˉ)A_i^{\mathrm{step}}=r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}=(r_i^\star-\bar r^\star)+(e_i-\bar e). Thus, the ideal centered advantage Ai⋆=ri⋆−rˉ⋆A_i^\star=r_i^\star-\bar r^\star is independent of the shared offset b(h)b(h). Although the centered advantage AistepA_i^{\mathrm{step}} sacrifices token-level signals, we believe that it provides more accurate relative sequence-level supervision by removing the shared offset. We therefore expect adding this signal to the original token-level OPD objective to improve learning.

The experiments support this expectation. In Table 1 in the main paper, SAPD outperforms OPD+Rule by 6.5 and 4.4 percentage points on ALFWorld and WebShop, respectively, with a GRPO teacher, and by 2.6 and 6.3 percentage points with an SFT teacher. These results demonstrate that AistepA_i^{\mathrm{step}} contributes to improved learning. We next present controlled experiments to examine the role of mismatch-induced shared offsets in these gains, followed by ablations of centering and standard-deviation normalization.

Additional controlled experiments with smaller shared offset.

To assess the role of the shared offset b(h)b(h), we compare SAPD and OPD+Rule in a setting where the mismatch that motivates this offset is removed. Specifically, we replace fixed offline prefixes with histories generated online by the current student through environment interactions. In this controlled comparison, we focus on the mismatch between fixed offline histories and histories generated by the current student policy. Under our error model, using these on-policy histories is therefore expected to substantially reduce the shared offset associated with this source of mismatch.

If removing this offset is an important source of SAPD’s gains under offline prefixes, we would expect its advantage over OPD+Rule to diminish in this online setting. The results in Table 1 support this prediction: SAPD’s gains decrease from 6.5 to 0.8 percentage points on ALFWorld and from 4.4 to 1.6 percentage points on WebShop.

This contrast provides empirical support for the role of mismatch-induced shared offsets in explaining SAPD’s gains, complementing the algebraic cancellation established in Proposition 1. While changing the prefix source does not isolate b(h)b(h) from all other distributional changes, the smaller gains under on-policy histories are consistent with our proposed mechanism. We have added these results to Appendix C.5.

Table 1: Comparison across prefix sources

MethodPrefix SourceALFWorldWebShop
GRPO Teacher 3B—89.164.8
OPD+Ruleoffline62.849.0
SAPDoffline69.353.4
OPD+RuleOnline student-generated62.552.3
SAPDOnline student-generated63.353.9
AuthorsOpenReview ↗

Author Response (2/3)

Response to CW1 and Q1: Shared-Offset Assumption

Additional experiments with uncentered or standardized signals

To directly address Q1 and further examine the shared-offset explanation raised in CW1, we compare three signal constructions: the uncentered score Airaw=ristepA_i^{\mathrm{raw}}=r_i^{\mathrm{step}}, the group-centered advantage Aictr=ristep−rˉstepA_i^{\mathrm{ctr}}=r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}, and the standardized advantage Aistd=ristep−rˉstepstd⁡(rstep)+ηA_i^{\mathrm{std}}=\frac{r_i^{\mathrm{step}}-\bar r^{\mathrm{step}}}{\operatorname{std}(r^{\mathrm{step}})+\eta}, where η\eta is a numerical stabilizer. We keep the other training hyperparameters fixed, although we do not explicitly match the realized update magnitudes across variants.

Effect of group centering. Under our error model, removing centering reintroduces the shared offset b(h)b(h), together with rˉ⋆\bar r^\star and eˉ\bar e, since Airaw−Aictr=rˉstep=rˉ⋆+b(h)+eˉA_i^{\mathrm{raw}}-A_i^{\mathrm{ctr}}=\bar r^{\mathrm{step}}=\bar r^\star+b(h)+\bar e. As shown in Table 2, centering improves performance by 2.8 percentage points on ALFWorld and 8.4 percentage points on WebShop. These results support the benefit of removing the group-level score component, which includes the shared offset. This complements the online-prefix comparison above: SAPD’s gains diminish when the mismatch is reduced, while performance also declines when centering is removed. Both observations are consistent with our explanation that removing mismatch-induced shared offsets contributes to SAPD’s gains, although neither comparison isolates b(h)b(h) alone.

Effect of standard-deviation normalization. Centering alone also outperforms standardization, achieving 96.6% versus 72.1% on ALFWorld and 59.4% versus 39.6% on WebShop. Dividing by a small group standard deviation can amplify weak within-group score differences, potentially increasing the influence of scoring noise. The observed degradation is consistent with our motivation to preserve the original scale of these differences. However, without explicitly matching update magnitudes, we cannot attribute the degradation solely to noise amplification, since standardization also changes the scale and relative weight of the step objective during optimization.

Overall, these results favor group centering without standard-deviation normalization under the tested configuration and provide complementary evidence for the shared-offset explanation. These results suggest that the centered relative signal retains useful information for policy improvement despite response-dependent scoring errors. We have added these results to Appendix D.2.

Table 2: Ablation of centering and standard-deviation normalization

MethodALFWorldWebShop
SFT Teacher 3B94.559.7
AictrA_i^{\mathrm{ctr}}96.659.4
AirawA_i^{\mathrm{raw}}93.851.0
AistdA_i^{\mathrm{std}}72.139.6

AuthorsOpenReview ↗

Author Response (3/3)

Response to CW2: Retention in Sequential Training Compared with Distillation Baselines

Thank you for the suggestion. It should be clarified that we attribute the retention benefits to on-policy student-generated responses in our introduction, whereas the step-advantage signal is designed to improve supervision under context mismatch. We provide additional experimental evidence to support this interpretation.

We add sequential-training experiments using OPD+Rule and ReOPD with on-policy student-generated responses, following the sequential-training protocol in Section 5.4 of the paper. For ReOPD, we use a GRPO-trained teacher following the setting in the original ReOPD paper. The additional results are provided in Appendix C.6. As shown in Table 3 below, after all three training stages, OPD+Rule shows decreases of 5.8 and 7.2 percentage points on WebShop and ALFWorld, respectively, while ReOPD shows decreases of 10.4 and 6.3 percentage points. These decreases are substantially smaller than those of sequential SFT (49.3 and 84.4 percentage points) and closer to those of SAPD (8.6 and 5.6 percentage points).

These results suggest that retention during sequential training may primarily benefit from on-policy student-generated responses. Despite this, OPD+Rule remains limited in performance, and ReOPD still requires environment interactions to generate teacher contexts. In contrast, SAPD optimizes online student-generated responses while reusing offline prefixes, preserving the retention benefits of OPD-style methods without relying on environment interactions.

Table 3: Performance under sequential training

Training StageWebShopALFWorldSearch
SAPD: SFT teacher 3B
Stage1: WebShop59.416.416.2
Stage 2: ALFWorld57.096.113.9
Stage 3: Search50.890.536.6
OPD+Rule: SFT Teacher 3B
Stage1: WebShop52.814.211.6
Stage 2: ALFWorld46.989.89.4
Stage 3: Search47.082.736.9
ReOPD: GRPO Teacher 3B
Stage1: WebShop64.116.310.3
Stage 2: ALFWorld56.367.211.3
Stage 3: Search53.760.941.6

We sincerely thank you for your thoughtful feedback, which has helped us strengthen both the analysis and the empirical evaluation. Our setting enables policy refinement from existing trajectories even when live interactions are costly or constrained by service availability and operating hours. We hope the additional experiments and clarifications address your concerns, and we would appreciate your reconsideration of our work in light of this additional evidence.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.