acceptodds
Under review as a conference paper at ICLR 2027

ReSCC: Calibrating Agent Self-Distillation with Retrieved Action Outcomes

Abstract

Teacher-guided self-distillation provides long-horizon agents with dense token- level supervision, complementing the sparse and coarse feedback from terminal rewards. Confidence-based gating reflects how strongly a teacher prefers a pre- diction but does not establish whether the resulting action improves downstream outcomes. We propose RESCC (Retrieval-Simulated Counterfactual Credit), which calibrates teacher guidance using action outcomes from previously collected training trajectories. RESCC retrieves similar state–action histories and compares the current action with well-supported alternatives using behavior-probability correction and a cross-fitted outcome model. It converts this evidence into a bounded signed weight on the distillation loss and falls back to the original update when historical support is insufficient. GRPO and online rollouts remain unchanged. Across ALFWorld, WebShop, and Search-QA with Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B, RESCC achieves improvements of up to 19.5 percentage points over existing baselines. Ablation studies show that outcome-aware retrieval, behavior correction, support-aware abstention, and bidirectional credit all contribute to the performance gains. Executed-alternative evaluations further show that RESCC aligns substantially better with realized action outcomes than confidence-based signals, with reliability improving as historical support increases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
on policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GMdEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ReSCC extends SDAR, which adds a token-level self-distillation loss, gated by the teacher–student probability gap, on top of GRPO. Because that gate reflects teacher's preference rather than action consequences, ReSCC stores past trajectories and, for each token/action, retrieves similar historical state/task/action by cosine similarity. It then estimates the outcome of the taken action versus the best well-supported alternative using inverse-behavior-probability weighting plus a cross-fitted outcome model. If the evidence is reliable (within a confidence interval), it scales the self-distillation loss up or down by a bounded factor; otherwise, it falls back to plain SDAR. GRPO and rollouts are unchanged. Authors claim that it consistently beats GRPO and SDAR on average across ALFWorld, WebShop, and Search-QA, with the largest gains on WebShop.

Critical strengths

Problem is genuine; the teacher cannot be blindly trusted, especially seeing that the connection between task and its success may not be quite crucial on a specific token which shows disagreement with a teacher and its subsequent gating influence. (Table 6 quantifies this: 17% confident-teacher-but-harmful, 14% helpful-exception)

Method of splitting source trajectory and scoring held-out folds does reduce overfitting and leakage in the outcome model (all decisions from one trajectory share a fold, so the model never predicts a record it was trained on). Also, confidence levels to decide when to trust historical evidence and when to discard them is good.

Critical weaknesses

Token level self-distillation guidance is not new, and several recent methods address it; the experiments should compare against the most recent state of the art (e.g., UCoB, SEED, StepOPSD) not just GRPO and SDAR. Retrieving similar historical states using cosine similarity alone is also questionable. The paper does not analyze how sensitive the method is to retrieve the wrong state versus the wrong action, nor how this affects performance. Because the memory is filled by the agent's own earlier policies, it may carry selection bias - frequently chosen, high-reward actions are over-represented, and the method gives no account of how newly explored actions would be handled or how cold start works when the action space changes. Finally, the core problem remains open: how is a terminal reward linked to the quality of a specific action in the trajectory? No empirical evidence shows that action variability per task is small enough for terminal returns to be a reliable signal.

Questions for authors
  1. Were the SDAR and other baselines re-run with your seeds and budget, or copied from Lu et al.? Can you report baseline variance and significance tests for the gains over SDAR, especially on Qwen2.5-7B and Search-QA, where ReSCC is flat or lower?
  2. Can the ablations (Table 2) be repeated on at least one other backbone? Several deltas are under one standard deviation.
  3. The memory contains trajectories from older policies, and you store the policy version but never use it. Did you test recency weighting or on-policy-only memory?
  4. How does the method treat newly explored actions or a changing action space (cold start)? Does it systematically favour historically frequent, high-reward actions?
Reviewer_poF8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an interesting yet intuitive limitation of teacher-guided self-distillation, where teacher confidence does not necessarily reflect downstream action outcomes. The proposed ReSCC retrieves similar historical experiences and estimates the outcome difference between the selected action and supported alternatives using behavior correction and a cross-fitted outcome model. This estimate is then converted into a bounded weight on the self-distillation loss, leaving GRPO unchanged. When historical evidence is insufficient, the method falls back to SDAR. Experiments cover three Qwen backbones across three long-horizon agent benchmarks. ReSCC achieves an average improvement of 4.0 percentage points over SDAR, although the gains are not consistent across all settings. Ablations generally support the effectiveness of individual components, with relatively modest incremental gains.

Critical strengths
  1. Clear motivation. The paper identifies a meaningful weakness in confidence-gated self-distillation: teacher preference does not necessarily reflect downstream action outcomes. The relatively low pairwise AUC of 0.58 between the teacher–student probability gap and executed action outcomes further supports this observation.
  2. Straightforward integration. ReSCC reuses trajectories collected during training without requiring additional online rollouts. It also functions as a plug-in for SDAR, modifying only the weighting of the distillation loss while leaving GRPO unchanged.
  3. Comprehensive empirical evaluation.
Critical weaknesses
  1. Dependence on historical trajectory quality. ReSCC relies on historical outcomes to calibrate teacher supervision, but its effectiveness may depend heavily on the quality and diversity of collected trajectories. While the authors investigate retrieval quality and support-aware abstention, these do not necessarily guarantee that sufficiently supported historical comparisons are reliable, particularly when successful trajectories or useful alternatives are scarce.
  2. Unclear distinction from step-level credit assignment. The estimated outcome difference between selected and alternative actions resembles a step-level advantage signal. Although the GIGPO experiment suggests complementarity, it remains unclear why this signal should specifically reweight self-distillation rather than directly contribute to the RL objective.
Questions for authors
  1. Historical trajectory quality: What happens when the memory contains predominantly unsuccessful trajectories or lacks genuinely beneficial alternatives? Would ReSCC largely fall back to SDAR, or could it potentially reinforce suboptimal actions based on misleading historical comparisons?
  2. Performance consistency: ReSCC underperforms SDAR in several settings, particularly on Search-QA and WebShop accuracy with Qwen2.5-7B. More broadly, the average improvements appear concentrated in Qwen3-1.7B. Could the authors discuss the possible reasons for these differences across backbones?
Reviewer_rbtgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Aiming at the problem of "when the teacher's signal is credible" in long-range agent self-distillation - the existing gating only reflects the teacher's preference rather than the consequences of the action, RESCC retrieves similar state-action records from the training history, uses a behavior modification + cross-fitting outcome model to estimate the outcome difference between "the action taken and the best evidence-based alternative action". If there is insufficient support, abstain and fall back, otherwise the distillation loss is reweighted with a bounded bidirectional weight (GRPO remains unchanged). Under the 3 benchmark × 3 Qwen backbone, the average is +12.0pp compared to GRPO and +4.0pp compared to SDAR; ablation and counterfactual execution evaluation (AUC 0.79 vs 0.58) verify that each component is effective.

Critical strengths

The problem framing is insightful and pluggable: Formalize "teacher confidence ≠ action consequences" into a counterfactual comparison that can be estimated from historical trajectories, forming a complete closed loop (retrieval → estimation → support inspection → bounded two-way weighting → abstention and fallback), no GRPO, and clear project boundaries.

Critical weaknesses

Key details are not explained: the executed-alternative experimental protocol, memory refresh strategy, outcome model architecture, hyperparameter sensitivity, code/reproduction resources are all missing.

Questions for authors

Key question (to the author): The core evidence in the full article is AUC 0.79 - how many states was this number measured on? What are the three environments of ALFWorld / WebShop / Search-QA? Only when the sample size is large and all three environments are established, can I believe that "the historical ending is more accurate than the teacher's confidence level"; if there are only dozens or hundreds of states, or it all depends on a certain environment, this conclusion needs to be re-evaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.