- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PESD augments GRPO with selective self-distillation on failed on-policy rollouts. Using completed-episode hindsight, retrieved skills, and available successful references, it partitions actions by context-induced log-probability shifts into positive distillation, conservative unlikelihood updates, or RL-only optimization. Additional guidance is progressively reduced across task–outcome revisits, and privileged information is removed at inference. Experiments evaluate 1.7B and 3B models on ALFWorld and Search-QA, supplemented by component ablations and a manual audit of 1,000 actions.
Critical strengths
-
The problem formulation is well motivated: actions within a failed trajectory receive different forms of supervision. Assigning distinct optimization roles to actions within the same failed rollout is more nuanced than suppressing all actions simply because the trajectory failed.
-
The experiments across model scales and the evaluation on unseen tasks are reasonably comprehensive.
Critical weaknesses
-
The core contribution is to partition actions into three categories, P, N, and U, based on log probabilities and apply a different update strategy to each. In my view, these hand-crafted decision rules may improve performance because the model's log probabilities, even with the additional context, still do not provide a reliable supervisory signal. Similarly, dividing the additional guidance into three levels appears to reflect limitations in model capability: an explicit hierarchy is needed to reduce the supervision's dependence on specific reference material. This raises the question of whether the method would remain effective for somewhat stronger models, or whether its benefits are limited to small, relatively weak models.
-
Invalid actions account for 67.2% of the 311 actions in the positive region P, exceeding the 40% invalid-action rate across all audited actions. Specifically, 209 of these 311 actions were manually labeled invalid. Why are these invalid actions reinforced? In addition, the negative region N contains only 24 actions, providing a small sample to support the conclusions. The audit covers only 20 failed trajectories from a single model at a single training checkpoint.
-
The Search-QA evaluation protocol does not provide sufficiently robust support for the claimed overall advantage. Appendix A explicitly states that the 125 Bamboogle questions are divided into two slices, one containing 124 questions and the other a single question, and the slice scores are then averaged with equal weight. Consequently, a single question accounts for 50% of the score on this dataset.
Questions for authors
Please refer to the weakness
4 comments
Additional ALFWorld Evalution on Qwen2.5-7B-Instruct
we are running PESD with Qwen2.5-7B-Instruct on ALFWorld (In Progress). -> we also evaluate PESD with Qwen2.5-7B-Instruct on ALFWorld (Results reported below).
We additionally test PESD with Qwen2.5-7B-Instruct on ALFWorld to evaluate whether its benefits extend to stronger backbones. As shown in the table below, PESD outperforms SDAR by 6.3 points on average with Qwen2.5-7B-Instruct. The results suggest that PESD maintains its advantage with a stronger backbone.
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. |
|---|---|---|---|---|---|---|---|
| SDAR | 94.7 | 75.0 | 100 | 86.7 | 68.2 | 78.9 | 85.9 |
| PESD | 100 | 91.7 | 100 | 86.7 | 84.0 | 83.3 | 92.2 |
Thank you for the additional experiments. The larger-model results, comparisons using corrected Bamboogle aggregation, and experiments with matched training budgets have alleviated my concerns on these points. However, the role of positively reinforcing invalid actions remains unclear, and the existing ablations do not isolate its effect. I am still awaiting the promised control that excludes invalid actions from positive reinforcement and will reassess once the results are available. For now, I maintain my original rating.
Additional ablation results.
Thank you for your consideration and for acknowledging the additional experiments. We have added the ablation (excluding invalid actions from positive reinforcement) results. As shown below, excluding invalid actions from positive reinforcement does not lead to an overall improvement. We further evaluated this variant with 10 random seeds. Its best run achieved an average success rate of 93.0, matching the best performance achieved by the original PESD. We have also added an analysis of the training dynamics w/ and w/o invalid actions to the revised manuscript (Appendix D.4). We hope these additional results address your remaining concerns. Thank you again for your patience and constructive feedback.
| Variant | Pick | Look | Clean | Heat | Cool | Pick2 | Avg. |
|---|---|---|---|---|---|---|---|
| PESD | 99.0 ± 1.5 | 86.1 ± 7.9 | 94.9 ± 3.6 | 97.8 ± 3.2 | 84.0 ± 0.0 | 87.0 ± 2.6 | 92.2 ± 1.3 |
| w/o invalid in | 100.0 ± 0.0 | 88.9 ± 4.8 | 94.9 ± 2.3 | 95.5 ± 3.9 | 88.0 ± 4.0 | 72.2 ± 9.6 | 91.1 ± 0.9 |
Response to Reviewer b8mm
We thank the reviewer for the thoughtful feedback. Our responses are below.
(W1) Decision rules, privilege hierarchy, and stronger models
We agree that log-probability shifts are not a perfect signal, but this is not a weakness of small models in particular. A shift tells us how the privileged context changes the model's preference for an action, not whether the action is actually good. The thresholds simply decide when the signal is strong enough to act on. This matters most on the negative side, since unlikelihood pushes an action down without offering a better one. Without these safeguards, Bare Reverse collapses to 66.4, and dropping the consistency gate (ρ=0) gives 85.9. SDAR and PCSD filter teacher signals for the same reason. The hierarchy is also not a workaround for weak models. It addresses the fact that privileged information is gone at inference. If strong guidance stays on throughout training, the teacher leans on the reference rather than on the policy's own behavior. To test whether the benefits extend to stronger models, we also evaluate PESD with Qwen2.5-7B-Instruct on ALFWorld (Results reported below).
(W2) Invalid actions in P and audit scale
P is defined by a likelihood increase under the privileged context, not by correctness, so it can include invalid actions. This is why the forward update on P is kept small: the coefficient is 0.01, tokens are weighted by persistence, and RL with the invalid-action penalty keeps pushing these actions down. Even so, adding forward supervision on P still improves over reverse-only (90.6 → 92.2). Filtering P by correctness during training would need a reliable process reward model, which would add another reward signal and make the effect of self-distillation harder to isolate. N is small on purpose, since reverse updates should only hit clearly rejected actions. The audit is a manual sanity check of this selectivity, not a precise estimate. On the same 1,000 hand-labeled actions, lowering the consistency threshold ρ (from 0.6 to 0.3 to 0) grows N from 24 to 104 to 111 and raises the share of correct actions in it from 8.3% to 14.4% to 15.3%, in line with the performance drop (92.2 → 91.4 → 85.9).
(W3) Search-QA aggregation
We agree. This aggregation comes from the evaluation pipeline we inherited from SDAR (verl-agent), and it affects all methods equally. Bamboogle drops by 32.5–36.3 points for every method (Appendix A). We rerank all methods in two ways, using the sample-level Bamboogle score (Updated Bam.) and leaving Bamboogle out (w/o Bam.):
PESD stays on top in both cases, by 1.5–2.1 points, similar to the original protocol.
We thank the reviewer again for the helpful comments.