- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCOUT, a state-conditional counterfactual privilege distillation framework for multi-turn LLM agents. The method addresses the potential mismatch between a fixed successful task plan and the states actually reached by a student policy. SCOUT conditions a frozen teacher on three complementary views: Gold, Skill, and Neutral. A parameter-free gate dynamically combines their token probability distributions according to Gold compatibility and skill applicability. Experiments on ALFWorld, WebShop, and search-based question answering demonstrate consistent improvements over Gold-only on-policy self-distillation, both with and without GRPO.
Critical strengths
- Meaningful problem formulation and state-adaptive supervision. The paper identifies a relevant limitation of using fixed privileged plans in multi-turn agent training. Its state-conditional mixture of Gold, Skill, and Neutral teacher distributions provides a conceptually reasonable approach to adapting distillation targets to the student's actual interaction history.
- Consistent empirical improvements across models and benchmarks. SCOUT demonstrates improvements across three model backbones and multiple agent environments. The evaluation covers both pure distillation and GRPO-augmented training, while component ablations provide supporting evidence for the contributions of different views and adaptive weighting.
Critical weaknesses
- Insufficient evidence for the claimed state-mismatch and recovery mechanism. The central motivation is that a fixed Gold plan may provide inappropriate supervision when the student deviates from the successful trajectory. However, the Gold teacher is already conditioned on the student's actual interaction history. Therefore, it is unclear why the Gold teacher cannot infer appropriate recovery actions from the combination of the successful plan and current observations. Although the Skill view provides explicit recovery guidance, the experiments do not directly establish that its benefits arise from correcting state-mismatched supervision rather than simply providing more structured task knowledge. A controlled comparison of teacher predictions at identical off-track states would strengthen this claim.
- Insufficient comparisons with simpler alternatives and limited computational analysis. The paper demonstrates the importance of individual views through ablation studies, but does not sufficiently compare SCOUT with simpler alternatives, such as adding explicit recovery instructions to the Gold teacher or providing Gold and Skill information jointly within a single teacher context. Consequently, it remains unclear whether the full multi-view probability mixture is necessary. Furthermore, the additional teacher evaluations, skill extraction, and environment-specific state signals introduce training complexity, while computational overhead and cost-matched comparisons are not adequately reported.
Questions for authors
- Can the authors compare Gold, Skill, and mixed teacher distributions at identical off-track student states, particularly the probabilities assigned to valid recovery actions? This would help establish whether the improvements specifically arise from better recovery supervision.
- How does SCOUT compare against a Gold teacher with explicit state-recovery instructions or a single teacher conditioned jointly on Gold and Skill information? What additional training cost does the three-view framework introduce?
3 comments
[Part 2/2] Thank you for responses. We look forward to hearing back from you.
Response to W2 & Q2: Simpler alternatives and training cost.
Thank you for suggesting these comparisons. We add two controlled baselines on ALFWorld with Qwen3.5-4B under pure distillation.
- Gold + recovery instruction adds a fixed instruction to the Gold context. It asks the teacher to check whether the plan applies and to recover when needed. It does not use Skill.
- Gold + Skill, single context adds the same retrieved skills used by SCOUT to the Gold context. One scoring pass gives one teacher distribution.
All methods use the same initialization, distillation loss formulation, training data, interaction budget, checkpoint selection rule, and evaluation setting. Training hyperparameters follow the paper.
Table III. ALFWorld success (%) with Qwen3.5-4B, pure distillation without GRPO.
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | Overall |
|---|---|---|---|---|---|---|---|
| Gold-only OPSD | 83.6 | 18.3 | 41.4 | 17.1 | 23.2 | 64.2 | 44.8 |
| Gold + recovery instruction | 83.1 | 19.4 | 40.8 | 19.7 | 23.9 | 61.0 | 44.6 |
| Gold + Skill, single context | 85.3 | 17.2 | 40.2 | 18.8 | 26.8 | 61.8 | 45.3 |
| Full SCOUT | 85.9 | 18.3 | 44.3 | 23.1 | 32.6 | 62.6 | 48.1 |
The recovery instruction gives 44.6% Overall success, close to Gold-only OPSD (44.8%). Putting Gold and Skill in a single context gives 45.3%. SCOUT reaches 48.1%, which is 3.5 and 2.8 points higher than these baselines. So asking the teacher to recover does not help by itself. Giving both types of information in one context also does not reach SCOUT in this setting. These results support using a state-conditional probability mixture to control the contribution of each view.
For computation, SCOUT uses three teacher forward passes to score the existing student sequence. It does not generate three teacher rollouts. Skill extraction uses autoregressive teacher generation. The teacher is often idle during student rollouts or parameter updates, and we run skill extraction during this time. Extracted skills are stored in a buffer for later retrieval and reuse.
Table IV. Average wall-clock time per step on WebShop over about 150 shared valid training steps.
| Method | Time per step (s) |
|---|---|
| GRPO | 141.00 |
| OPSD+GRPO | 156.79 |
| SCOUT+GRPO | 164.71 |
SCOUT+GRPO takes 16.82% more time than GRPO and 5.05% more than OPSD+GRPO. On our current infrastructure, scoring three views does not triple the training time. We will add these baselines and timing results to the revised paper.
Thank you for the detailed responses and the additional experiments. I appreciate the effort to address my concerns, particularly the controlled comparisons at identical off-track states and the additional baselines. These experiments provide useful evidence and partially alleviate my concerns. However, my main remaining concern is the distinction between demonstrating the effectiveness of additional structured recovery knowledge and establishing the necessity of SCOUT's specific multi-view probability-mixture mechanism.
The current results provide evidence that Skill-based recovery guidance is beneficial, but the comparisons are still somewhat limited in demonstrating that the proposed distribution-level integration is the key factor behind the improvements, rather than the additional structured knowledge itself.
Given these remaining concerns, I consider the response insufficient to substantially change my original assessment. I therefore maintain my original Weak Reject rating.
[Part 1/2] Thank you for responses. We look forward to hearing back from you.
Response to W1 & Q1: State mismatch and recovery supervision
Thank you for suggesting a controlled comparison at the same states. The Gold teacher sees the actual interaction history of the student and may infer recovery actions. But a fixed successful plan may leave out the corrective steps needed in the current state. SCOUT gives this guidance explicitly through the Skill view. It adjusts the contribution of each view by Gold compatibility and skill applicability (Sections 3.2 to 3.4).
We ran an offline test on ALFWorld with Qwen3.5-4B. It compares Gold, Skill, and SCOUT at exactly the same states. Each comparison uses the same teacher checkpoint, interaction history, action output prefix, and candidate action set. We score each complete action string token by token, including its end marker. We then normalize the probabilities over the shared candidate set. For SCOUT, we first mix the token distributions as in Section 3.4 and then compute action probabilities.
We use 76 mismatched states. In each of them, the student holds a wrong object and must put it down. We measure the mean relative probability of two actions. The first is a designated recovery action validated by the environment. The second is an inapplicable Gold action. We also remove only the recovery rules from Skill. This separates recovery instructions from other structured knowledge. The preconditions, action schema, expected effects, retrieved skills, and original gating weights stay unchanged.
Table I. Mean probability normalized over the candidate action set in 76 mismatched states (%).
At the same mismatched states, SCOUT raises the recovery action probability from 5.41% to 75.46% compared with Gold. It also lowers the probability of the inapplicable Gold action from 19.04% to 0.17%. So even when Gold sees the actual history, the SCOUT target gives more support to the corrective action needed in the current state. Removing the recovery rules lowers the recovery action probability by 11.64 points. It also raises the support for the inapplicable action. So recovery instructions add value beyond the skill knowledge kept in this ablation.
The two cases below show how the explicit recovery rules help at specific states.
Table II. Relative probability of the designated recovery action in two mismatched cases (%).
Removing the recovery rules lowers the recovery action probability from 86.02% to 77.60% in the first case. In the second case, it drops from 67.19% to 53.08%. The same-state comparison and the targeted ablation support our explanation. Skill gives applicable recovery guidance, and SCOUT puts it into the supervision target.