acceptodds
Under review as a conference paper at ICLR 2027

GET: Direct Peer Evidence for Agentic Self-Distillation

Abstract

Outcome-supervised reinforcement learning provides language agents with limited guidance for intermediate decisions. On-policy self-distillation (OPSD) supplies token-level supervision from privileged context, making the selection and organization of decision-relevant experience a central design question. We introduce Group Evidence Teacher (GET), which organizes observation-matched peer experience into a decision-local reference alongside immediate feedback on the student's own action. For each decision, GET selects transitions from other rollouts of the same task and deterministically compiles their actions and source episodes' outcomes into an Evidence Card, without generating skills or corrective guidance. During training, the same policy re-scores the student-sampled response using a decision-local context built from available action feedback and peer evidence; the resulting token-level log-probability differences determine the weights of an auxiliary update alongside group-relative reinforcement learning. On ALFWorld, GET achieves mean success across three seeds. Adding peer evidence to action feedback yields a -percentage-point gain in task success in a seed-0 component comparison; environment-verified replay of these policies shows a -point gain in required object-state changes across cleaning, heating, and cooling tasks. On Search-QA, GET outperforms Skill in both micro- and macro-averaged exact match across both backbones and weighting rules. With Qwen3-1.7B and Raw OPSD, GET achieves higher test accuracy with fewer search calls per question than Skill.

Primary area:
reinforcement learning
Keywords:
Agentic RL, OPSD, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_t7LeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GET, an on-policy self-distillation method for training multi-turn language agents alongside GRPO. For each decision, GET builds a privileged teacher context by combining the immediate execution feedback of the student's action with an "Evidence Card" that summarizes peer actions and episode outcomes from observation-matched transitions in other rollouts of the same task. The teacher re-scores the student's sampled tokens under this context, and the resulting token-level log-probability gaps are mapped via Raw or Gate weighting rules to form an auxiliary distillation loss. Experiments on ALFWorld and Search-QA with Qwen2.5-3B and Qwen3-1.7B show improvements over vanilla GRPO and skill-conditioned distillation.

Critical strengths
  1. Constructing privileged teacher context directly from on-policy rollout groups via observation matching and deterministic rendering is practical, avoiding extra LLM calls for skill generation.

  2. The empirical analysis goes beyond terminal success rates by examining environment-verified intermediate milestones on ALFWorld and search call efficiency on Search-QA, offering helpful insights into how local supervision changes agent behavior.

Critical weaknesses
  1. The core components of GET closely overlap with existing methods discussed in Section 5, yet none of them are empirically compared. Such as GiGPO, SDPO, and MOPD and Combining observation matching with peer/feedback distillation feels incremental. Furthermore, the experiments only compare against GRPO and SDAR, omitting direct multi-turn RL and peer-distillation SOTA baselines.

  2. The performance gap between the Raw and Gate weighting rules fluctuates drastically across models without a clear explanation. On Table 1, Qwen2.5-3B with Gate achieves 84.31 while Raw drops sharply to 70.07. On Qwen3-1.7B, Raw substantially outperforms Gate, yet suffers from severe collapse on specific subtasks. In addition, Table 2 reveals high variance across random seeds for GET, raising concerns that reporting only seed 0 in the main tables overstates the method's actual stability and gains. Finally, all experiments are restricted to very small (1.7B and 3B) and somewhat dated (Qwen2.5 from 2024) backbones, leaving it unclear whether GET still provides meaningful gains on stronger, larger, and more recent models

Questions for authors
  1. Given that GiGPO also leverages observation matching across rollout groups and methods like MOPD/SDPO use peer trajectories or execution feedback for self-distillation, why are these closely related methods not included as baselines? How does GET perform when compared directly against GiGPO or MOPD under the same training budget?

  2. Why do Raw and Gate exhibit completely opposite behaviors on Qwen2.5-3B and Qwen3-1.7B on ALFWorld, and what causes the severe degradation on tasks like Pick2 under Raw? Moreover, since the evaluation only uses small 1.7B/3B models (including the older Qwen2.5-3B), can the authors provide results on larger or more recent backbones to show that explicit peer evidence cards remain beneficial when the base policy is stronger?

Reviewer_JfEHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose GET, an on-policy self-distillation method for agentic RL that augments GRPO with an auxiliary distillation loss. The teacher is the same policy, given privileged training-time context that consists of the environment feedback on the student’s own action and an Evidence Card, a deterministically constructed summary of the actions and episode outcomes from other rollouts in the same group that reached the same observation. The teacher–student log-probability gap on the sampled tokens determines the weights of the auxiliary loss, either directly or through a sigmoid gate (Gate), following SDAR. The method is evaluated on ALFWorld and Search-QA's subgroups with Qwen2.5-3B and Qwen3-1.7B. The authors compare against GRPO and skill-conditioned self-distillation (SDAR) and report improvements over the skill-based variant across all configurations. The paper also includes ablations of the context components and of the peer selection strategy, a milestone-level analysis of task progress on ALFWorld, an LLM-judge comparison of Evidence Cards versus skills, and an analysis of search-call counts.

Critical strengths
  1. The method is evaluated on two different domains (ALFWorld, Search-QA), two backbones (Qwen2.5-3B, Qwen3-1.7B), and two weighting rules (Raw, Gate).
  2. Evidence Cards are built deterministically from rollouts that are already collected, without additional LLM generation or an external skill bank. The overhead is small.
  3. Insightful milestone analysis. The environment-verified milestone analysis shows where the gains come from: they concentrate at the required object-state change rather than at object acquisition.
  4. Clear presentation. The paper is easy to follow, and the method, evaluation protocols and analyses are documented in detail. The authors also report negative or inconclusive results (e.g., the continuation study) rather than omitting them.
Critical weaknesses
  1. The SDAR reproduction performs noticeably worse than in the original paper. Original paper report +9.4 points over GRPO on Qwen2.5-3B ALFWorld, while here SDAR underperforms GRPO. The paper does not discuss this discrepancy. In several configurations Skill is below plain GRPO, so the gains over it are not very informative. GiGPO, the closest prior work is not compared against.
  2. The components already exist in prior work: feedback and peer attempts as self-teacher context (SDPO, MOPD), state-matched grouping (GiGPO, SMRC-SD), and the sigmoid gate (SDAR). The contribution is mostly their combination. The paper does not show why this combination is preferable to the closest alternatives.
  3. Weak evidence from the LLM-judge study. Preference for state-specific cards over task-level skills is expected by construction, and a judge’s preference does not imply a better training signal. The study also shows a notable position bias. The related continuation study gives a difference whose confidence interval includes zero.
  4. Evidence Cards disappear during training. Cards require at least two distinct peer actions. As the policy converges, coverage drops from 87% to 61%, so the auxiliary signal weakens over training. This suggests the method mostly helps early on, and the paper does not analyze this.
Questions for authors
  1. Why does your SDAR reproduction underperform GRPO, while the original paper reports +9.4 points over GRPO on Qwen2.5-3B ALFWorld? Are there differences in setup or evaluation that explain this?
  2. Could you compare against GiGPO and at least one other closely related method (e.g., SDPO or MOPD) on at least one benchmark?
  3. What do you see as the key novel element beyond combining existing components, and which experiment isolates it?
  4. How does Evidence Card coverage change during GET training, and does the benefit persist with longer training?
Reviewer_RazHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a method for constructing privileged context for agentic self-distillation. GET selects observation-matched transitions from other rollouts of the same task and organizes peer actions and episode outcomes into Evidence Cards. These cards are combined with immediate action feedback to help a frozen policy snapshot re-score the student’s original responses. The resulting token-level signals provide auxiliary supervision alongside GRPO, using either signed weights or the gating mechanism adopted from SDAR [1]. The method is evaluated on ALFWorld and Search-QA, with additional analyses of task progress and interaction efficiency.

Critical strengths

This paper constructs its teacher context directly from collected rollouts and immediate feedback, without an additional language-model guidance-generation stage. Excluding the target trajectory is a sensible safeguard against directly recycling its own peer statistics. The method fits naturally into group-based agent training and adds no evidence-retrieval component at deployment.

GET achieves promising results on ALFWorld and Search-QA. The environment-based analysis also provides useful evidence that the gains reflect improvements in task execution.

Critical weaknesses

The methods are somewhat proposed in previous work: local grouping in GiGPO [2], peer-conditioned teaching in MOPD [3], grouped search hindsight in SD-Search [4], and state-compatible privileged guidance in SMRC-SD [5]. GET’s main distinction lies in combining directly rendered, observation-matched peer evidence with immediate feedback. However, comparisons primarily against GRPO and Skill do not adequately establish the advantage of this design over closely related alternatives. In addition, the selector comparison changes both the selected evidence and the amount of auxiliary supervision, making it difficult to attribute the improvement specifically to observation matching. Since evidence construction is the central contribution, stronger isolation of its benefit is important.

Matching observations does not necessarily imply matching decision states, since relevant history, inventory, or completed subgoals may differ. Likewise, an action’s association with a successful episode does not establish its local usefulness, and repeated visits from the same trajectory may amplify that trajectory’s influence. Although the reported performance is encouraging, the current analyses do not sufficiently establish when the constructed evidence is trustworthy or whether the resulting teacher scores consistently reflect better action quality. The external-judge preference study provides limited validation of this mechanism, while the continuation results remain inconclusive. These issues limit confidence in the reliability and general applicability of the proposed supervision.

Questions for authors
  1. Can the authors more directly establish GET’s advantage over local-grouping or peer-context alternatives, such as GiGPO [2] and SD-Search [4], under comparable training and evaluation conditions? A Search-QA feedback-only comparison would also help clarify the additional contribution of peer evidence.

  2. How do the authors assess whether observation-matched peer transitions are compatible with the student’s actual decision state? Please clarify how differences in relevant history, inventory, and completed subgoals affect the reliability of the constructed evidence.

  3. Does the advantage of observation matching remain when the amount of auxiliary supervision is controlled? Such a comparison would help distinguish the benefit of selecting more relevant evidence from the effect of changing supervision strength.

  4. Can the authors demonstrate that changes in teacher scores reflect useful evidence rather than repeated action text or differences in context structure? Controls that retain the card format while removing or shuffling outcome information would be informative. This concern is related to the replay-context confounding studied by OCSD [6].

[1] Self-Distilled Agentic Reinforcement Learning.

[2] Group-in-Group Policy Optimization for LLM Agent Training.

[3] Multi-Rollout On-Policy Distillation via Peer Successes and Failures.

[4] SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning.

[5] When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents.

[6] Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.