- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The key idea of the paper is: Instead of just using environmental feedback for self-distillation as a signal for policy supervision, also use it to explicitly model environmental feedback given an action and history. Both objectives are optimized jointly through the same underlying model.
Critical strengths
- A useful approach to leverage environmental feedback.
- Strong empirical results showing SELF outperforms other methods, while preserving the general capabilities of the base model.
Critical weaknesses
- W1: As feedback modeling is coupled with action selection, I am concerned that SELF might be biased towards predictable outcomes. In execution oriented tasks, like the ones explored in this paper, it might be helpful. However, in many other environments, exploring informative but uncertain actions is vital. I am concerned that the algorithm would guide the policy towards redundant actions, like Example 4 on page 25, where the action is to read a config file, and the predictor predicts the learning_rate will be 1e-6. If this answer is so obvious, why is the task even relevant anymore and the action not redundant? Would the algorithm bias towards opening this predictable file instead of another task-relevant "config2" file with more non-standard values?
- W2: I find the ablation studies with each interesting. However, changing either \lambda_o or \lambda_a, as long as the ratio is the same, would lead to the same minimizer (scale of the loss doesn't affect the minimizer), so from a conceptual point of view Table 3 and 4 would be better reorganized if showing the effects of the ratio and the scale separately.
- W3: I would appreciate more discussion about some of the results presented. Namely, you show SDPO performs worse with full feedback than just observation. This is a counterintuitive result, could you give any insights about why this could be happening?
- Presentation:
- I believe the paper would benefit from introducing all figures in the text. For example, Figures 1 is never mentioned in the text. The appendix pointers would be more useful if they named the relevant sections explicitly. For example, I was wondering how the optimization of Equation 8 worked, and it would have been helpful if afterwards Appendix b were mentioned.
- Figures and tables are redundant showing the exact same information. Figure 2 is redundant with Table 2, I would remove one from main text and instead bring show new relevant info to provide context, like what exactly TGC and SGC are measuring. Similarly, content of tables 3, 4 and 5 is the same as Figure 3. I would suggest authors make a better use of the space.
- There is a floating "Qwen3-..." text in Fig. 2b that probably shouldn't be there
Questions for authors
- Q1: Could you address the potential limitation of bias towards predictable actions? Could you evaluate SELF on environments that rely heavily on exploration? Could you measure action coverage or redundant actions?
- Q2: To distinguish the effect of balancing the losses from the scale of training steps, could you separately sweep the ratio while holding the overall scale fixed, and sweep the overall scale while holding the ratio fixed?
- Q3: Do you expect performance to extend outside of Qwen family?
Response to Rebuttal
I thank the authors for the detailed rebuttal and the additional experiments. The new feedback NLL and greedy-decoding results address my concern about the feedback-modeling direction.
Regarding the comparison with HERO, SELF achieves 35.5% SR versus 34.7%. Given the standard deviations reported in Table 2, this 0.8% difference is within the noise, so I read this result as comparable performance rather than a clear improvement.
The new results also show that GRPO + lenv outperforms SELF. I acknowledge that SELF is computationally cheaper. However, the main claimed contribution of the paper is outperforming reward-trained GRPO, not training efficiency, and these results suggest that the advantage over GRPO comes from the auxiliary loss (which is known to improve GRPO) rather than from self-distillation itself.
As for the noisy-reward experiment, as I understand from the rebuttal, the performance drop is reported for vanilla GRPO rather than for GRPO + lenv, which is expected. To demonstrate the value of SELF when rewards are unavailable, I believe the evaluation should be extended to reward-free settings.
Finally, the rebuttal does not address my main concern (W1): the novelty of SELF relative to ECHO, PaW, Early Experience and RLTF, and what the method contributes beyond combining established components. Therefore, I maintain my score.