- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
ReSCC extends SDAR, which adds a token-level self-distillation loss, gated by the teacher–student probability gap, on top of GRPO. Because that gate reflects teacher's preference rather than action consequences, ReSCC stores past trajectories and, for each token/action, retrieves similar historical state/task/action by cosine similarity. It then estimates the outcome of the taken action versus the best well-supported alternative using inverse-behavior-probability weighting plus a cross-fitted outcome model. If the evidence is reliable (within a confidence interval), it scales the self-distillation loss up or down by a bounded factor; otherwise, it falls back to plain SDAR. GRPO and rollouts are unchanged. Authors claim that it consistently beats GRPO and SDAR on average across ALFWorld, WebShop, and Search-QA, with the largest gains on WebShop.
Critical strengths
Problem is genuine; the teacher cannot be blindly trusted, especially seeing that the connection between task and its success may not be quite crucial on a specific token which shows disagreement with a teacher and its subsequent gating influence. (Table 6 quantifies this: 17% confident-teacher-but-harmful, 14% helpful-exception)
Method of splitting source trajectory and scoring held-out folds does reduce overfitting and leakage in the outcome model (all decisions from one trajectory share a fold, so the model never predicts a record it was trained on). Also, confidence levels to decide when to trust historical evidence and when to discard them is good.
Critical weaknesses
Token level self-distillation guidance is not new, and several recent methods address it; the experiments should compare against the most recent state of the art (e.g., UCoB, SEED, StepOPSD) not just GRPO and SDAR. Retrieving similar historical states using cosine similarity alone is also questionable. The paper does not analyze how sensitive the method is to retrieve the wrong state versus the wrong action, nor how this affects performance. Because the memory is filled by the agent's own earlier policies, it may carry selection bias - frequently chosen, high-reward actions are over-represented, and the method gives no account of how newly explored actions would be handled or how cold start works when the action space changes. Finally, the core problem remains open: how is a terminal reward linked to the quality of a specific action in the trajectory? No empirical evidence shows that action variability per task is small enough for terminal returns to be a reliable signal.
Questions for authors
- Were the SDAR and other baselines re-run with your seeds and budget, or copied from Lu et al.? Can you report baseline variance and significance tests for the gains over SDAR, especially on Qwen2.5-7B and Search-QA, where ReSCC is flat or lower?
- Can the ablations (Table 2) be repeated on at least one other backbone? Several deltas are under one standard deviation.
- The memory contains trajectories from older policies, and you store the policy version but never use it. Did you test recency weighting or on-policy-only memory?
- How does the method treat newly explored actions or a changing action space (cold start)? Does it systematically favour historically frequent, high-reward actions?