- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ReSAIL, a method targeting performance collapse across repeated cycles of self-distillation training. It does this by prioritizing distillation on steps where the PI-conditioned behavior of the teacher differs the most from the ordinary behavior, balancing the resulting losses across trajectories, and adding a regularization loss that pushes the student’s PI-conditioned output distributions toward those of the frozen teacher. The paper presents strong results on ALFWorld and TextCraft, and ablates and analyzes the components of the method.
Critical strengths
- The writing is clear.
- The method tackles an interesting problem and shows substantial gains.
- The chosen comparison methods and set of carried out ablations are good.
Critical weaknesses
- Since ReSAIL is framed mainly around tackling collapse in deployment performance across cycles, I think it would be good to know what is the behavior of the method across more than 3 cycles. Moreover, while the experimental scope seems reasonable for the amount of used compute, the fact results use a single training run for each method and that only ALFWorld and TextCraft test the full method is a limitation in terms of the amount of evidence for the method.
Questions for authors
-
Do you have an intuition for why filtering for the steps with highest PI sensitivity helps? ReSAIL is not the first paper to use selection/filtering in the context of distillation. Did the motivation for that component of ReSAIL come from past work?
-
The motivation for Privileged retention is presented as trying to preserve the PI-conditioned behavior as the student becomes the teacher in the next cycle. Under that framing, why not keep the initial model frozen as the privileged teacher across cycles? More generally, how do the authors view the relationship between preserving PI-conditioned behavior and allowing it to improve across cycles, as is actually observed in their experiments? Moreover, most of the benefit from retention in deployment performance is already obtained by ordinary retention. Could these suggest that a more appropriate framing for the introduced regularization technique be something else rather than specifically ‘to preserve PI-conditioned behavior’?
Secondary feedback/questions:
-
The paper states ‘We provide the first evidence that performance collapse in iterative agent self-distillation can be effectively mitigated through a more robust learning mechanism.‘ This seems like a strong claim. Is this really the first work addressing this? The paper itself mentions Chen et al. (2026a) as targeting collapse over target degradations across successive cycles.
-
How was the hyperparameter selection procedure carried out for the method and the baselines? While the hyperparameter values and some sensitivity analyses are provided, I could not find an explanation of the selection procedure or whether the methods received comparable tuning.
5 comments
Official Comment by Authors (2/3)
Q3: Privileged Retention and Evolving Teacher Competence
A3: Thank you for pointing out this ambiguity. We have revised “preserving PI-conditioned behavior” to “sustaining PI-conditioned competence” in the main paper to clarify that our goal is to maintain useful PI-conditioned supervision while allowing the teacher's competence to improve across cycles.
To evaluate your proposed alternative, we kept the initial teacher fixed across three OEL cycles on ALFWorld with Qwen3-4B, while otherwise following the standard OEL training procedure.
Table 3. ALFWorld success rates for OEL with evolving or frozen teachers and full ReSAIL (%; mean ± SD over three evaluations).
| Split | Setting | C1 | C2 | C3 |
|---|---|---|---|---|
| ID | OEL (evolving teacher) | 54.9 ± 4.7 | 56.5 ± 2.0 | 47.4 ± 1.2 |
| OEL (frozen initial teacher) | 54.9 ± 4.7 | 56.0 ± 2.4 | 56.8 ± 1.2 | |
| OEL+ReSAIL | 62.2 ± 2.0 | 70.1 ± 3.2 | 74.2 ± 1.4 | |
| OOD | OEL (evolving teacher) | 47.1 ± 2.5 | 46.1 ± 2.1 | 38.5 ± 1.2 |
| OEL (frozen initial teacher) | 47.1 ± 2.5 | 51.3 ± 4.7 | 52.9 ± 1.6 | |
| OEL+ReSAIL | 57.3 ± 4.0 | 63.5 ± 4.7 | 67.2 ± 2.3 |
Freezing the initial teacher mitigates the performance decline observed in standard OEL. In contrast, OEL+ReSAIL achieves substantially higher performance across cycles while continuing to update the teacher.
As you note, ordinary-view retention (OR) already provides substantial deployment gains. PR specifically targets the PI-conditioned competence needed for subsequent supervision. At C3, PR outperforms OR by 8.1 percentage points with PI, compared with only 1.3 percentage points without PI (Figure 3).
To examine whether this advantage translates into more effective supervision, we used each of the OR and PR checkpoints from C1–C3 to train a fresh student from the same Qwen3-4B initialization, holding the training data, PI, distillation objective, and budget fixed.
Table 4. ALFWorld success rates of fresh students trained by OR and PR teachers (%; mean ± SD over three evaluations).
| Split | Teacher source | C1 teacher | C2 teacher | C3 teacher |
|---|---|---|---|---|
| ID | OEL+ReSAIL (OR) | 62.8 ± 2.4 | 71.6 ± 3.9 | 70.8 ± 3.6 |
| OEL+ReSAIL (PR) | 69.3 ± 1.2 | 74.2 ± 1.6 | 77.1 ± 1.8 | |
| OOD | OEL+ReSAIL (OR) | 48.4 ± 2.1 | 62.2 ± 3.2 | 67.4 ± 2.0 |
| OEL+ReSAIL (PR) | 53.1 ± 4.3 | 64.1 ± 3.4 | 74.0 ± 5.0 |
PR checkpoints yield higher mean student success rates than OR checkpoints at every source cycle on both splits. Later PR checkpoints also produce progressively better students, indicating that teaching effectiveness can improve across cycles.
We further extended the OR–PR comparison to six cycles to examine whether these advantages are also reflected in longer-term deployment performance.
Table 5. Six-cycle ALFWorld deployment success rates with OR and PR (%; mean ± SD across two training seeds).
| Split | Method | C1 | C2 | C3 | C4 | C5 | C6 |
|---|---|---|---|---|---|---|---|
| ID | OEL+ReSAIL (OR) | 61.6 ± 2.4 | 71.0 ± 0.6 | 75.0 ± 2.9 | 75.8 ± 1.5 | 75.5 ± 1.5 | 68.5 ± 2.6 |
| OEL+ReSAIL (PR) | 63.7 ± 2.0 | 68.8 ± 1.8 | 75.8 ± 2.2 | 79.2 ± 1.8 | 77.0 ± 1.3 | 75.1 ± 2.8 | |
| OOD | OEL+ReSAIL (OR) | 55.1 ± 4.2 | 64.7 ± 2.8 | 67.7 ± 2.6 | 66.9 ± 0.7 | 66.3 ± 2.0 | 59.4 ± 3.3 |
| OEL+ReSAIL (PR) | 57.0 ± 0.4 | 65.6 ± 2.9 | 68.8 ± 2.2 | 70.6 ± 4.1 | 68.5 ± 1.8 | 63.7 ± 1.3 |
PR maintains higher mean success rates than OR on both splits from C3 through C6, with a larger gap at C6 than at C3. These results further support the benefit of sustaining PI-conditioned competence across cycles.
Official Comment by Authors (3/3)
Q4: Clarifying the Novelty Claim and Its Scope
A4: Existing PI-based self-distillation methods, such as SDPO and OEL, predominantly use on-policy distillation, in which a PI-conditioned teacher supervises student-generated responses. Our original use of “self-distillation” referred to this prevalent paradigm.
Chen et al. (2026a) identify performance collapse across successive cycles and address it by switching to off-policy context distillation using teacher-generated trajectories collected through additional environment interaction. In contrast, ReSAIL demonstrates that collapse can be effectively mitigated within the original on-policy paradigm, without requiring these additional teacher–environment interactions.
We have therefore revised the abstract and Introduction to explicitly specify “on-policy self-distillation” and narrowed the scope of our “first evidence” claim accordingly. We have also clarified the distinction between our approach and Chen et al. (2026a) in Related Work.
Q5: Hyperparameter Selection
A5: Our hyperparameter selection procedure was as follows:
- Training settings and budgets. We followed the optimization settings of SDPO [1] and matched training budgets across methods within each benchmark. Given the varying training set sizes across benchmarks, we adjusted the number of task instances used per cycle accordingly.
- Additional baseline configuration checks. For the self-distillation baselines (SDPO and OEL), we compared reverse KL, forward KL, and Jensen–Shannon (JS) divergence objectives. Given their similar performance, we retained reverse KL.
- ReSAIL-specific hyperparameters. Since TextCraft provides fewer trajectories per update than ALFWorld, we used a higher selection ratio to retain sufficient training steps for distillation. The retention weight was chosen through a coarse comparison of candidate values; 0.5 and 1 performed similarly, and no finer search was conducted.
We have clarified these details in the revised paper.
[1] Hübotter et al. Reinforcement Learning via Self-Distillation. 2026.
Official Comment by Authors (1/3)
Thank you for your review and suggestions on the experimental evaluation. We address these suggestions and your questions about the method’s motivation and design below.
Q1: Longer Runs and Broader Experimental Evaluation
A1: We address your concerns about the evaluation with the following additional experiments.
1. Longer runs and independent training seeds. We extended the ALFWorld experiments from three to six cycles using Qwen3-4B, with two independent training seeds each for OEL and OEL+ReSAIL.
Table 1. Six-cycle ALFWorld success rates (%; mean ± SD across two training seeds).
OEL+ReSAIL maintains higher mean success rates than OEL on both splits throughout all six cycles. Although performance declines after Cycle 4, the results support sustained mitigation of cross-cycle degradation over the six evaluated cycles.
2. Full-method evaluation on ToolAlpaca.
Following the official SDPO settings for ToolAlpaca, we evaluated SDPO and OEL with and without the full ReSAIL method using Qwen3-4B over three training cycles.
Table 2. ToolAlpaca success rates (%; mean ± SD over six evaluations). Base model (C0): 56.4 ± 0.8%.
On ToolAlpaca, both SDPO+ReSAIL and OEL+ReSAIL improve across all three cycles, whereas both baselines decline sharply at Cycle 3.
Q2: Motivation for Sensitivity-Guided Selection
A2: Data selection is widely used in deep learning, including distillation. Our design is motivated by differences in the additional guidance that PI provides across steps in agent trajectories, rather than by prior distillation work. At low-sensitivity steps, the teacher's predictions are similar with and without PI, suggesting that PI provides little additional guidance. At high-sensitivity steps, PI has a stronger influence on the teacher's predictions. We therefore prioritize these steps to focus distillation on behaviors most affected by PI.