acceptodds
Under review as a conference paper at ICLR 2027

Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

Abstract

Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model's ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in -bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language agents, Self-distillation, Reward-free agent learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Wpy2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The key idea of the paper is: Instead of just using environmental feedback for self-distillation as a signal for policy supervision, also use it to explicitly model environmental feedback given an action and history. Both objectives are optimized jointly through the same underlying model.

Critical strengths
  • A useful approach to leverage environmental feedback.
  • Strong empirical results showing SELF outperforms other methods, while preserving the general capabilities of the base model.
Critical weaknesses
  • W1: As feedback modeling is coupled with action selection, I am concerned that SELF might be biased towards predictable outcomes. In execution oriented tasks, like the ones explored in this paper, it might be helpful. However, in many other environments, exploring informative but uncertain actions is vital. I am concerned that the algorithm would guide the policy towards redundant actions, like Example 4 on page 25, where the action is to read a config file, and the predictor predicts the learning_rate will be 1e-6. If this answer is so obvious, why is the task even relevant anymore and the action not redundant? Would the algorithm bias towards opening this predictable file instead of another task-relevant "config2" file with more non-standard values?
  • W2: I find the ablation studies with each λ=0\lambda=0 interesting. However, changing either \lambda_o or \lambda_a, as long as the ratio is the same, would lead to the same minimizer (scale of the loss doesn't affect the minimizer), so from a conceptual point of view Table 3 and 4 would be better reorganized if showing the effects of the ratio λo/λa\lambda_o / \lambda_a and the scale separately.
  • W3: I would appreciate more discussion about some of the results presented. Namely, you show SDPO performs worse with full feedback than just observation. This is a counterintuitive result, could you give any insights about why this could be happening?
  • Presentation:
    • I believe the paper would benefit from introducing all figures in the text. For example, Figures 1 is never mentioned in the text. The appendix pointers would be more useful if they named the relevant sections explicitly. For example, I was wondering how the optimization of Equation 8 worked, and it would have been helpful if afterwards Appendix b were mentioned.
    • Figures and tables are redundant showing the exact same information. Figure 2 is redundant with Table 2, I would remove one from main text and instead bring show new relevant info to provide context, like what exactly TGC and SGC are measuring. Similarly, content of tables 3, 4 and 5 is the same as Figure 3. I would suggest authors make a better use of the space.
    • There is a floating "Qwen3-..." text in Fig. 2b that probably shouldn't be there
Questions for authors
  • Q1: Could you address the potential limitation of bias towards predictable actions? Could you evaluate SELF on environments that rely heavily on exploration? Could you measure action coverage or redundant actions?
  • Q2: To distinguish the effect of balancing the losses from the scale of training steps, could you separately sweep the ratio λo/λa\lambda_o / \lambda_a while holding the overall scale fixed, and sweep the overall scale while holding the ratio fixed?
  • Q3: Do you expect performance to extend outside of Qwen family?
Reviewer_ytncOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SELF, a reward-free framework for training LLM agents from environmental feedback. A single shared model is jointly trained with hindsight self-distillation, where an EMA teacher conditioned on the realized feedback supervises the student’s actions via reverse KL, and feedback modeling, where the model learns to predict the environment’s response to its actions via NLL. The authors claim the two objectives are mutually reinforcing and that SELF outperforms self-distillation, memory-based and reward-trained GRPO baselines on τ\tau-bench Retail and AppWorld with Qwen3-4B/8B, while preserving general capabilities.

Critical strengths
  1. The paper is well written and well structured. The problem of learning from environmental feedback without verifiable rewards is clearly motivated, and the method is easy to follow.
  2. The method is simple and practical. It requires no separate environment model, no additional rollouts to construct targets, and no extra inference-time cost, since only the student policy is deployed.
  3. The paper evaluates retention of general capabilities (MMLU, MMLU-Pro, IFEval) after agent training, which is good practice and often omitted in similar work.
  4. The appendix is detailed (full algorithm, gradient derivations, prompts, evaluation protocols), which aids reproducibility.
Critical weaknesses
  1. SELF combines two established ingredients: feedback-conditioned self-distillation (SDPO; Hübotter et al., ICML 2026) and an auxiliary next-observation likelihood on the shared policy. The latter has already been paired with several policy objectives: GRPO (ECHO; Shrivastava et al., 2026); general RL (PaW; Lu et al., 2026), which also reports improved observation prediction under joint training, a version of the “mutual reinforcement” claim; and imitation learning in a reward-free setting that includes τ\tau-bench Retail (implicit world modeling in Early Experience; Zhang et al., ICML 2026). RLTF (Song et al., 2026) proposes both feedback-conditioned self-distillation and feedback modeling. Given this body of work, it is expected that such a combination can improve performance, and the paper cites these works without explaining what is gained over them. While examining the effect of adding an environment-modeling loss to self-distillation is of some interest, the contribution is seemed to be incremental for me.

  2. The headline claim, “outperforming both existing self-distillation approaches and reward-trained GRPO,” is not fair contribution. In the paper’s own results, pure self-distillation underperforms GRPO (SDPO 30.4; SELF(λo\lambda_o=0) 29.4; GRPO 33.3). The advantage over GRPO therefore stems from lenvl_{env}, a term that is orthogonal to the policy objective and is known to improve GRPO as well: ECHO reports TerminalBench-2.0 pass@1 raise from 2.70 to 5.17 for Qwen3-8B when adding it. The fair counterpart is therefore GRPO + lenvl_{env}. More broadly, the benefit of adding an environment-prediction term has already been demonstrated in multiple settings (ECHO, PaW, Early Experience), and SELF provides one more instance of it. A substantive contribution would be a systematic study of when and why this term helps across tasks and policy objectives.

  3. Experiments cover two benchmarks and a single model family (Qwen3 4B/8B). All ablations and mechanism analyses use one model (4B), and the λo\lambda_o sweep is limited to Retail. Many methods operate in this setting and should be compared against, e.g., HERO (Liu et al., 2026), which also tackles τ\tau-bench Retail with self-distillation from environment observations. Benchmarks used by closely related work (TerminalBench in ECHO; ALFWorld, WebShop and others in Early Experience and PaW) would better test generality. In addition, no validation split is described, and the default λo\lambda_o=0.2 coincides with the best test result in Table 3, which raises the question of whether hyperparameters were selected on the test set.

  4. The reverse direction (self-distillation improves feedback modeling) rests solely on an LLM-judge accuracy of single sampled predictions, which may reflect factors other than the quality of the learned environment model. Several observations make the conclusion difficult to draw. (A) At λa\lambda_a=0, the model is trained directly on feedback prediction, yet its accuracy barely changes relative to Base (Retail 32->37%; AppWorld 11->9%), suggesting that the metric or this run may not be informative. (B) Accuracy is not reported in Table 3, so the improvement may simply follow overall model quality, since a stronger model may predict feedback more easily, rather than indicate a specific benefit of self-distillation. (C) lenvl_{env} is a mass-covering (forward-KL) objective, while lsdl_{sd} is mode-seeking (reverse KL) on shared parameters, so higher sampled accuracy with larger lambdaalambda_a may reflect reduced sampling entropy rather than better environment modeling. To support the claim, It would be informative to see teacher-forced feedback NLL on held-out transitions for all settings in Tables 3 and 4 (including lambdaolambda_o=0), feedback accuracy for the lambdaolambda_o sweep, and greedy-decoding accuracy and predictive entropy.

Questions for authors
  1. How were lambdaolambda_o and lambdaalambda_a selected?
  2. How does SELF compare to GRPO + lenvl_{env}(as in ECHO)? This is necessary to attribute the gain over GRPO to self-distillation rather than to the auxiliary loss.
  3. Can you provide the measurements requested in W4 (feedback NLL for all settings in Tables 3 and 4, accuracy for the lambdaolambda_o sweep, greedy-decoding accuracy and predictive entropy)?
  4. Can you report comparisons with closely related methods on the same benchmarks (e.g., HERO on τ\tau-bench Retail), or results on an additional benchmarks common for that field?
1 comment
Reviewer_ytncOpenReview ↗

Response to Rebuttal

I thank the authors for the detailed rebuttal and the additional experiments. The new feedback NLL and greedy-decoding results address my concern about the feedback-modeling direction.

Regarding the comparison with HERO, SELF achieves 35.5% SR versus 34.7%. Given the standard deviations reported in Table 2, this 0.8% difference is within the noise, so I read this result as comparable performance rather than a clear improvement.

The new results also show that GRPO + lenvl_{env} outperforms SELF. I acknowledge that SELF is computationally cheaper. However, the main claimed contribution of the paper is outperforming reward-trained GRPO, not training efficiency, and these results suggest that the advantage over GRPO comes from the auxiliary loss (which is known to improve GRPO) rather than from self-distillation itself.

As for the noisy-reward experiment, as I understand from the rebuttal, the performance drop is reported for vanilla GRPO rather than for GRPO + lenvl_{env}, which is expected. To demonstrate the value of SELF when rewards are unavailable, I believe the evaluation should be extended to reward-free settings.

Finally, the rebuttal does not address my main concern (W1): the novelty of SELF relative to ECHO, PaW, Early Experience and RLTF, and what the method contributes beyond combining established components. Therefore, I maintain my score.

Reviewer_pqiuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how language agents can learn from environmental feedback without task rewards. Building on hindsight self-distillation methods such as SDPO, where a self-teacher conditioned on realized feedback is distilled into the student, the authors argue that conditioning alone underuses feedback. Their method, SELF, adds an auxiliary NLL objective in which the same model predicts the realized feedback from the history and action, alongside reverse-KL distillation from an EMA teacher. On τ-bench Retail and AppWorld with Qwen3-4B and Qwen3-8B, SELF outperforms SDPO, OEL, Mem0 and reward-trained GRPO (e.g., 43.2 vs. 36.8 and 39.1 pass@4 on Retail with Qwen3-8B). Sweeps over the two loss weights are presented as evidence that feedback modeling and distillation reinforce each other, and MMLU, MMLU-Pro and IFEval results indicate general capabilities are retained.

Critical strengths
  1. The method is simple and cheap, and the gains are consistent. SELF adds one NLL term on data the method already collects, and it improves over SDPO across two backbones and two quite different environments. Raising the feedback-modeling weight from 0 to 0.2 moves student success from 29.4 to 38.7.

  2. The diagnostics are informative. Evaluating the privileged teacher directly, the honest observation that the best teacher does not give the best student, and the cross-rollout control all go beyond headline numbers.

  3. Capability retention and cost are considered. SELF stays within about 0.1 points of the base model on MMLU, MMLU-Pro and IFEval, while SDPO loses up to 3.4 points, and it needs about 40% of GRPO's GPU-hours.

Critical weaknesses
  1. The learning signal is not explained. The teacher is conditioned on the feedback produced by the very action it scores, with no outcome label, so it is unclear why it would prefer better actions rather than simply re-predict that action. The mutual-reinforcement evidence is also confounded: with λa = 0 the policy collapses and the data distribution changes completely.

  2. The closest baselines are missing. RLTF, Early Experience and PaW are cited but not compared, and SDPO plus a next-observation NLL loss is essentially SELF. SDPO gets worse in the Full setting, which suggests a configuration problem, and the GRPO setup and compute budgets are not reported.

  3. The evaluation is weak. Retail uses pass@4 instead of pass@1 or pass^k, feedback-prediction accuracy relies on an unvalidated LLM judge, and training rounds and steps are not reported. There are also several figure and table errors, including swapped axis labels, a wrong table reference, and appendix examples that come from neither benchmark.

Questions for authors
  1. Since the teacher sees f_i, the outcome of a_i, but not a_i, what does it actually reward? Could you show examples, or token-level statistics, of where the teacher moves probability mass relative to the student, and say whether it mostly re-predicts a_i?

  2. Please add "SDPO + next-observation NLL" baselines in the style of Early Experience/PaW and RLTF, and an offline feedback-modeling control that trains on fixed base-policy transitions. This would separate the distillation effect from the data-distribution effect in Table 4.

  3. What are T, K, the number of training trajectories per round, and the GRPO configuration (group size, reward, steps)? Were the methods compared at matched sample or compute budgets?

  4. Please report pass@1 or pass^k on Retail, and validate the LLM judge against human or exact-match labels on a subset.

  5. Why does adding ground-truth and success labels (Full) hurt SDPO so much? Was SDPO tuned separately for that setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.