- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SAPD learns from recorded agent histories without executing new actions during policy refinement. It generates response groups under offline prefixes, combines token-level distillation with centered action-level teacher–student scores, and adds local rule rewards. Experiments cover ALFWorld, WebShop, search QA, sequential retention, and training-time comparisons with online distillation.
Critical strengths
- The environment-free refinement formulation is useful and clearly bounded: response generation is on-policy conditional on recorded histories, without claiming on-policy state coverage. The step signal and local rewards provide an implementable use of existing trajectories.
- The sequential-training comparison provides strong evidence of retention relative to SFT in the tested order. The online-OPD comparison reports cumulative time as well as per-step time, making the efficiency claim more informative.
Critical weaknesses
- The robustness analysis removes a shared score offset, but does not establish that shared offsets dominate actual teacher errors under prefix mismatch. Relative preference errors and variable action lengths remain, as the paper acknowledges. The experiments do not directly validate that this error model explains the observed gains.
- Several incremental gains, particularly on search QA, are small, and the main tables do not provide repeated-training uncertainty. This makes it difficult to distinguish consistent step-advantage benefits from training variation across the teacher configurations.
Questions for authors
- Do controlled shared-offset, response-dependent-error, and action-length experiments support the proposed explanation for robustness?
- What is the variation across independent training runs for SAPD versus OPD+Rule, especially on search QA?
3 comments
Author Response (2/3)
Response to CW1 and Q1: Shared-Offset Assumption
Additional experiments with uncentered or standardized signals
To further examine how the shared-offset explanation relates to the observed gains, we compare three signal constructions: the uncentered score , the group-centered advantage , and the standardized advantage , where is a numerical stabilizer. We keep the other training hyperparameters fixed, although we do not explicitly match the realized update magnitudes across variants.
Effect of group centering. Under our error model, removing centering reintroduces the shared offset , together with and , since . As shown in Table 2, centering improves performance by 2.8 percentage points on ALFWorld and 8.4 percentage points on WebShop. These results support the benefit of removing the group-level score component, which includes the shared offset. This complements the online-prefix comparison above: SAPD’s gains diminish when the mismatch is reduced, while performance also declines when centering is removed. Both observations are consistent with our explanation that removing mismatch-induced shared offsets contributes to SAPD’s gains, although neither comparison isolates alone.
Effect of standard-deviation normalization. Centering alone also outperforms standardization, achieving 96.6% versus 72.1% on ALFWorld and 59.4% versus 39.6% on WebShop. Dividing by a small group standard deviation can amplify weak within-group score differences, potentially increasing the influence of scoring noise. The observed degradation is consistent with our motivation to preserve the original scale of these differences. However, without explicitly matching update magnitudes, we cannot attribute the degradation solely to noise amplification, since standardization also changes the scale and relative weight of the step objective during optimization.
Together, the smaller gains with on-policy prefixes and the performance drop when centering is removed provide complementary empirical support for the role of mismatch-induced shared offsets in SAPD’s gains. The main results and ablations further show that the centered relative signal provides useful supervision in the presence of response-dependent errors and variable action lengths. The normalization comparison favors group centering without standard-deviation normalization under the tested configuration. We have added the prefix comparison to Appendix C.5 and the signal ablations to Appendix D.2.
Table 2: Ablation of centering and standard-deviation normalization
| Method | ALFWorld | WebShop |
|---|---|---|
| SFT Teacher 3B | 94.5 | 59.7 |
| 96.6 | 59.4 | |
| 93.8 | 51.0 | |
| 72.1 | 39.6 |
Author Response (3/3)
Response to CW2 and Q2: Results Across Random Seeds
Thank you for your comments. We conducted three independent training runs each for SAPD and OPD+Rule with an SFT teacher. The table reports the mean and sample standard deviation of success rates. These results are also reported in Appendix C.7.
Table 3: Results across independent random seeds
| Method | ALFWorld | WebShop | Search QA |
|---|---|---|---|
| OPD+Rule | 95.3 ± 0.8 | 53.1 ± 3.4 | 38.6 ± 0.3 |
| SAPD | 98.4 ± 0.8 | 59.6 ± 0.9 | 38.8 ± 0.1 |
SAPD achieves higher mean performance on all three tasks, with particularly consistent gains on ALFWorld and WebShop of 3.1 and 6.5 percentage points, respectively.
We agree that SAPD’s improvement on Search-based QA is small, with a mean improvement of only 0.2 percentage points. This is consistent with the observation in Section 5.2 that long-horizon tasks benefit more from SAPD. ALFWorld allows up to 50 interaction steps and WebShop up to 15, compared with only four for Search-based QA. Over longer interactions, early decisions shape subsequent observations and actions, allowing errors and mismatches between teacher and student behavior to accumulate. SAPD compares candidate actions under the same prefix, reducing the impact of context mismatch and providing more stable guidance for policy optimization.
We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.
Author Response (1/3)
Thank you very much for reviewing our paper and for acknowledging the value of our work in the review. Regarding the concerns raised in the review, we provide the following responses. The corresponding revisions are highlighted in blue in the revised manuscript.
Response to CW1 and Q1: Shared-Offset Assumption
Many thanks for your constructive comments. To connect the shared-offset analysis to the observed learning gains, we provide an online-prefix comparison and ablations of centering and standard-deviation normalization, alongside the main experimental results. We also clarify how response-dependent preference errors and variable action lengths are represented in our error model.
Shared offsets, residual errors, and evidence from the main experiments.
For responses sharing a prefix h, our model writes ristep=ri⋆+b(h)+ei. Group centering gives Aistep=(ri⋆−rˉ⋆)+(ei−eˉ), removing the shared component while explicitly retaining response-dependent residual errors. Thus, the model accommodates imperfect relative preferences. The empirical question is whether removing the shared component yields a useful relative signal for learning in the presence of these residual errors.
Variable action lengths. As detailed in Appendix A, if the token-level error contains a shared component btok(h), summing over Ni action tokens gives Δi=Nibtok(h)+ui, where ui is the accumulated token-level residual. For a reference length N0(h), this fits the aggregate model with b(h)=N0(h)btok(h) and ei=(Ni−N0(h))btok(h)+ui. Length-dependent variation is therefore included in the residual term, and the analysis applies without assuming equal action lengths. The observed learning gains below show that the centered signal is useful in our evaluated settings, where responses can differ in both length and scoring error.
The experiments support this expectation. In Table 1 in the main paper, SAPD outperforms OPD+Rule by 6.5 and 4.4 percentage points on ALFWorld and WebShop, respectively, with a GRPO teacher, and by 2.6 and 6.3 percentage points with an SFT teacher. These results demonstrate that Aistep contributes to improved learning. We next present controlled experiments to examine the role of mismatch-induced shared offsets in these gains, followed by ablations of centering and standard-deviation normalization.
Additional controlled experiments with smaller shared offset.
To assess the role of the shared offset b(h), we compare SAPD and OPD+Rule in a setting where the mismatch that motivates this offset is removed. Specifically, we replace fixed offline prefixes with histories generated online by the current student through environment interactions. In this controlled comparison, we focus on the mismatch between fixed offline histories and histories generated by the current student policy. Under our error model, using these on-policy histories is therefore expected to substantially reduce the shared offset associated with this source of mismatch.
If removing this offset is an important source of SAPD’s gains under offline prefixes, we would expect its advantage over OPD+Rule to diminish in this online setting. The results in Table 1 support this prediction: SAPD’s gains decrease from 6.5 to 0.8 percentage points on ALFWorld and from 4.4 to 1.6 percentage points on WebShop.
This contrast provides empirical support for the role of mismatch-induced shared offsets in explaining SAPD’s gains, complementing the algebraic cancellation established in Proposition 1. While changing the prefix source does not isolate b(h) from all other distributional changes, the smaller gains under on-policy histories are consistent with our proposed mechanism. We have added these results to Appendix C.5.
Table 1: Comparison across prefix sources