acceptodds
Under review as a conference paper at ICLR 2027

ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

Abstract

Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our experiments with existing methods reveal a collapse in deployment performance across cycles, while task performance with privileged information (PI) also declines. We address this collapse by prioritizing informative interaction steps for distillation and preserving PI-conditioned behavior as the student becomes the next teacher. We introduce **Re**tentive and **S**elective **A**ugmentation for **I**terative Self-Disti**l**lation (**ReSAIL**), a plug-in augmentation for iterative PI-based self-distillation. ReSAIL selects interaction steps where PI most strongly changes the teacher's predictions and balances the resulting distillation losses across trajectories. It also regularizes the student's PI-conditioned output distributions toward those of the frozen teacher at selected and unselected steps to preserve PI-conditioned behavior for supervision in the next cycle. On ALFWorld and TextCraft, ReSAIL sustains substantial gains across model scales over three cycles, with an average absolute gain of 22.5% in final-cycle success rates when added to self-distillation baselines. Sensitivity-guided selection of offline data also improves action prediction accuracy for multimodal GUI agents on AITZ. These findings provide the first evidence that a more robust learning mechanism can effectively mitigate performance collapse in iterative agent self-distillation over deployment trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, iterative self-distillation, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wKycOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ReSAIL, a method targeting performance collapse across repeated cycles of self-distillation training. It does this by prioritizing distillation on steps where the PI-conditioned behavior of the teacher differs the most from the ordinary behavior, balancing the resulting losses across trajectories, and adding a regularization loss that pushes the student’s PI-conditioned output distributions toward those of the frozen teacher. The paper presents strong results on ALFWorld and TextCraft, and ablates and analyzes the components of the method.

Critical strengths
  • The writing is clear.
  • The method tackles an interesting problem and shows substantial gains.
  • The chosen comparison methods and set of carried out ablations are good.
Critical weaknesses
  • Since ReSAIL is framed mainly around tackling collapse in deployment performance across cycles, I think it would be good to know what is the behavior of the method across more than 3 cycles. Moreover, while the experimental scope seems reasonable for the amount of used compute, the fact results use a single training run for each method and that only ALFWorld and TextCraft test the full method is a limitation in terms of the amount of evidence for the method.
Questions for authors
  • Do you have an intuition for why filtering for the steps with highest PI sensitivity helps? ReSAIL is not the first paper to use selection/filtering in the context of distillation. Did the motivation for that component of ReSAIL come from past work?

  • The motivation for Privileged retention is presented as trying to preserve the PI-conditioned behavior as the student becomes the teacher in the next cycle. Under that framing, why not keep the initial model frozen as the privileged teacher across cycles? More generally, how do the authors view the relationship between preserving PI-conditioned behavior and allowing it to improve across cycles, as is actually observed in their experiments? Moreover, most of the benefit from retention in deployment performance is already obtained by ordinary retention. Could these suggest that a more appropriate framing for the introduced regularization technique be something else rather than specifically ‘to preserve PI-conditioned behavior’?

Secondary feedback/questions:

  • The paper states ‘We provide the first evidence that performance collapse in iterative agent self-distillation can be effectively mitigated through a more robust learning mechanism.‘ This seems like a strong claim. Is this really the first work addressing this? The paper itself mentions Chen et al. (2026a) as targeting collapse over target degradations across successive cycles.

  • How was the hyperparameter selection procedure carried out for the method and the baselines? While the hyperparameter values and some sensitivity analyses are provided, I could not find an explanation of the selection procedure or whether the methods received comparable tuning.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for your review and suggestions on the experimental evaluation. We address these suggestions and your questions about the method’s motivation and design below.

Q1: Longer Runs and Broader Experimental Evaluation

A1: We address your concerns about the evaluation with the following additional experiments.

1. Longer runs and independent training seeds. We extended the ALFWorld experiments from three to six cycles using Qwen3-4B, with two independent training seeds each for OEL and OEL+ReSAIL.

Table 1. Six-cycle ALFWorld success rates (%; mean ± SD across two training seeds).

SplitMethodC1C2C3C4C5C6
IDOEL55.9 ± 1.357.2 ± 0.951.0 ± 5.244.9 ± 3.536.8 ± 2.430.9 ± 3.5
OEL+ReSAIL63.7 ± 2.068.8 ± 1.875.8 ± 2.279.2 ± 1.877.0 ± 1.375.1 ± 2.8
OODOEL48.0 ± 1.347.9 ± 2.641.1 ± 3.738.5 ± 3.335.2 ± 2.630.6 ± 2.4
OEL+ReSAIL57.0 ± 0.465.6 ± 2.968.8 ± 2.270.6 ± 4.168.5 ± 1.863.7 ± 1.3

OEL+ReSAIL maintains higher mean success rates than OEL on both splits throughout all six cycles. Although performance declines after Cycle 4, the results support sustained mitigation of cross-cycle degradation over the six evaluated cycles.

2. Full-method evaluation on ToolAlpaca.

Following the official SDPO settings for ToolAlpaca, we evaluated SDPO and OEL with and without the full ReSAIL method using Qwen3-4B over three training cycles.

Table 2. ToolAlpaca success rates (%; mean ± SD over six evaluations). Base model (C0): 56.4 ± 0.8%.

MethodC1C2C3
SDPO60.0 ± 1.160.3 ± 1.932.4 ± 2.1
SDPO+ReSAIL60.5 ± 0.661.3 ± 2.063.2 ± 0.9
OEL59.6 ± 0.858.6 ± 2.715.0 ± 1.1
OEL+ReSAIL59.3 ± 0.861.5 ± 2.062.7 ± 1.2

On ToolAlpaca, both SDPO+ReSAIL and OEL+ReSAIL improve across all three cycles, whereas both baselines decline sharply at Cycle 3.

Q2: Motivation for Sensitivity-Guided Selection

A2: Data selection is widely used in deep learning, including distillation. Our design is motivated by differences in the additional guidance that PI provides across steps in agent trajectories, rather than by prior distillation work. At low-sensitivity steps, the teacher's predictions are similar with and without PI, suggesting that PI provides little additional guidance. At high-sensitivity steps, PI has a stronger influence on the teacher's predictions. We therefore prioritize these steps to focus distillation on behaviors most affected by PI.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Q3: Privileged Retention and Evolving Teacher Competence

A3: Thank you for pointing out this ambiguity. We have revised “preserving PI-conditioned behavior” to “sustaining PI-conditioned competence” in the main paper to clarify that our goal is to maintain useful PI-conditioned supervision while allowing the teacher's competence to improve across cycles.

To evaluate your proposed alternative, we kept the initial teacher fixed across three OEL cycles on ALFWorld with Qwen3-4B, while otherwise following the standard OEL training procedure.

Table 3. ALFWorld success rates for OEL with evolving or frozen teachers and full ReSAIL (%; mean ± SD over three evaluations).

SplitSettingC1C2C3
IDOEL (evolving teacher)54.9 ± 4.756.5 ± 2.047.4 ± 1.2
OEL (frozen initial teacher)54.9 ± 4.756.0 ± 2.456.8 ± 1.2
OEL+ReSAIL62.2 ± 2.070.1 ± 3.274.2 ± 1.4
OODOEL (evolving teacher)47.1 ± 2.546.1 ± 2.138.5 ± 1.2
OEL (frozen initial teacher)47.1 ± 2.551.3 ± 4.752.9 ± 1.6
OEL+ReSAIL57.3 ± 4.063.5 ± 4.767.2 ± 2.3

Freezing the initial teacher mitigates the performance decline observed in standard OEL. In contrast, OEL+ReSAIL achieves substantially higher performance across cycles while continuing to update the teacher.

As you note, ordinary-view retention (OR) already provides substantial deployment gains. PR specifically targets the PI-conditioned competence needed for subsequent supervision. At C3, PR outperforms OR by 8.1 percentage points with PI, compared with only 1.3 percentage points without PI (Figure 3).

To examine whether this advantage translates into more effective supervision, we used each of the OR and PR checkpoints from C1–C3 to train a fresh student from the same Qwen3-4B initialization, holding the training data, PI, distillation objective, and budget fixed.

Table 4. ALFWorld success rates of fresh students trained by OR and PR teachers (%; mean ± SD over three evaluations).

SplitTeacher sourceC1 teacherC2 teacherC3 teacher
IDOEL+ReSAIL (OR)62.8 ± 2.471.6 ± 3.970.8 ± 3.6
OEL+ReSAIL (PR)69.3 ± 1.274.2 ± 1.677.1 ± 1.8
OODOEL+ReSAIL (OR)48.4 ± 2.162.2 ± 3.267.4 ± 2.0
OEL+ReSAIL (PR)53.1 ± 4.364.1 ± 3.474.0 ± 5.0

PR checkpoints yield higher mean student success rates than OR checkpoints at every source cycle on both splits. Later PR checkpoints also produce progressively better students, indicating that teaching effectiveness can improve across cycles.

We further extended the OR–PR comparison to six cycles to examine whether these advantages are also reflected in longer-term deployment performance.

Table 5. Six-cycle ALFWorld deployment success rates with OR and PR (%; mean ± SD across two training seeds).

SplitMethodC1C2C3C4C5C6
IDOEL+ReSAIL (OR)61.6 ± 2.471.0 ± 0.675.0 ± 2.975.8 ± 1.575.5 ± 1.568.5 ± 2.6
OEL+ReSAIL (PR)63.7 ± 2.068.8 ± 1.875.8 ± 2.279.2 ± 1.877.0 ± 1.375.1 ± 2.8
OODOEL+ReSAIL (OR)55.1 ± 4.264.7 ± 2.867.7 ± 2.666.9 ± 0.766.3 ± 2.059.4 ± 3.3
OEL+ReSAIL (PR)57.0 ± 0.465.6 ± 2.968.8 ± 2.270.6 ± 4.168.5 ± 1.863.7 ± 1.3

PR maintains higher mean success rates than OR on both splits from C3 through C6, with a larger gap at C6 than at C3. These results further support the benefit of sustaining PI-conditioned competence across cycles.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q4: Clarifying the Novelty Claim and Its Scope

A4: Existing PI-based self-distillation methods, such as SDPO and OEL, predominantly use on-policy distillation, in which a PI-conditioned teacher supervises student-generated responses. Our original use of “self-distillation” referred to this prevalent paradigm.

Chen et al. (2026a) identify performance collapse across successive cycles and address it by switching to off-policy context distillation using teacher-generated trajectories collected through additional environment interaction. In contrast, ReSAIL demonstrates that collapse can be effectively mitigated within the original on-policy paradigm, without requiring these additional teacher–environment interactions.

We have therefore revised the abstract and Introduction to explicitly specify “on-policy self-distillation” and narrowed the scope of our “first evidence” claim accordingly. We have also clarified the distinction between our approach and Chen et al. (2026a) in Related Work.

Q5: Hyperparameter Selection

A5: Our hyperparameter selection procedure was as follows:

  • Training settings and budgets. We followed the optimization settings of SDPO [1] and matched training budgets across methods within each benchmark. Given the varying training set sizes across benchmarks, we adjusted the number of task instances used per cycle accordingly.
  • Additional baseline configuration checks. For the self-distillation baselines (SDPO and OEL), we compared reverse KL, forward KL, and Jensen–Shannon (JS) divergence objectives. Given their similar performance, we retained reverse KL.
  • ReSAIL-specific hyperparameters. Since TextCraft provides fewer trajectories per update than ALFWorld, we used a higher selection ratio to retain sufficient training steps for distillation. The retention weight was chosen through a coarse comparison of candidate values; 0.5 and 1 performed similarly, and no finer search was conducted.

We have clarified these details in the revised paper.

[1] Hübotter et al. Reinforcement Learning via Self-Distillation. 2026.

Reviewer_rGctOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a method to address collapse after several rounds of self distillation.

  1. Rank interaction steps by how much privileged information changes the teacher’s predictions, using JSD and distill only the most sensitive steps.
  2. Average the student-to-teacher KL loss - take the token average over the trajectory, and then average over the trajectory to prevent long trajectories from dominating
  3. Add a retention loss - Give both the old teacher and the new student the same privileged information, and train the student to preserve the teacher’s predictions.
Critical strengths

The paper identifies a common problem in self distillation and proposes a method to address the same.

Critical weaknesses
  • The cause for training collapse is unclear. ReSAIL improves over baselines, but it seems unclear from the paper that losing privileged-information competence is the main cause of collapse.
  • The writing for this paper should be substantially improved.
    • The results are hard to parse.
    • Despite multiple passes, the ablations remain hard to understand for me.
Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your review. We address your concerns about the causes of collapse and the clarity of our experimental results below.

Q1: Causes of Collapse and the Role of PI-Conditioned Competence

A1: Training collapse may arise from multiple factors, and identifying its underlying causes is beyond the scope of our work. Our original submission does not claim that declining PI-conditioned competence is the main cause of collapse. Our primary contribution is a method for mitigating collapse in iterative agent self-distillation. With the evaluation tasks and PI held fixed, PI-conditioned success declined alongside deployment performance (Figure 1 in the main paper). Since each updated student becomes the next teacher, this observation motivates sustaining PI-conditioned competence to maintain useful supervision across cycles.

With SGS and TLB held fixed, adding PR improves PI-conditioned performance and mitigates deployment degradation (Table 2 in the main paper). To further substantiate the motivation for PR and its effectiveness, we conduct three complementary experiments: (1) a comparison of fixed and evolving teachers, (2) an evaluation of OR and PR checkpoints as teachers for fresh students, and (3) an extension of the OR–PR comparison to six training cycles.

1. Fixed versus evolving teachers. Using Qwen3-4B on ALFWorld, we compare standard OEL, which updates the teacher between cycles, with a variant that keeps the initial C0 teacher fixed across C1–C3. The student continues training across cycles in both settings.

Table 1. ALFWorld success rates with fixed and evolving teachers (%; mean ± SD over three evaluations).

SplitSettingC1C2C3
IDOEL (fixed initial teacher)54.9 ± 4.756.0 ± 2.456.8 ± 1.2
OEL (evolving teacher)54.9 ± 4.756.5 ± 2.047.4 ± 1.2
OODOEL (fixed initial teacher)47.1 ± 2.551.3 ± 4.752.9 ± 1.6
OEL (evolving teacher)47.1 ± 2.546.1 ± 2.138.5 ± 1.2

Keeping the initial teacher fixed avoids the decline observed with the evolving teacher on both splits over C1–C3. This supports the involvement of teacher updates in the observed degradation.

2. Teaching effectiveness of OR versus PR checkpoints. We compare C1–C3 checkpoints trained with ordinary-view retention (OR) or privileged-view retention (PR), with SGS and TLB held fixed. Each checkpoint teaches a fresh student initialized from the same Qwen3-4B base model. We use the same training data, PI, distillation objective, and training budget across these comparisons.

Table 2. Student success rates on ALFWorld (%; mean ± SD over three evaluations). C1–C3 indicate the cycles from which the teacher checkpoints are taken.

SplitTeacher sourceC1 teacherC2 teacherC3 teacher
IDOEL+ReSAIL (OR)62.8 ± 2.471.6 ± 3.970.8 ± 3.6
OEL+ReSAIL (PR)69.3 ± 1.274.2 ± 1.677.1 ± 1.8
OODOEL+ReSAIL (OR)48.4 ± 2.162.2 ± 3.267.4 ± 2.0
OEL+ReSAIL (PR)53.1 ± 4.364.1 ± 3.474.0 ± 5.0

PR teachers yield higher mean student success rates than OR teachers at every source cycle on both splits, and later PR checkpoints produce progressively better students.

3. Longer-term deployment performance. We extended the Qwen3-4B experiments on ALFWorld to six cycles with two independent training seeds per method to examine the longer-term benefit of PR over OR.

Table 3. Six-cycle ALFWorld success rates (%; mean ± SD across two training seeds).

SplitMethodC1C2C3C4C5C6
IDOEL55.9 ± 1.357.2 ± 0.951.0 ± 5.244.9 ± 3.536.8 ± 2.430.9 ± 3.5
OEL+ReSAIL (OR)61.6 ± 2.471.0 ± 0.675.0 ± 2.975.8 ± 1.575.5 ± 1.568.5 ± 2.6
OEL+ReSAIL (PR)63.7 ± 2.068.8 ± 1.875.8 ± 2.279.2 ± 1.877.0 ± 1.375.1 ± 2.8
OODOEL48.0 ± 1.347.9 ± 2.641.1 ± 3.738.5 ± 3.335.2 ± 2.630.6 ± 2.4
OEL+ReSAIL (OR)55.1 ± 4.264.7 ± 2.867.7 ± 2.666.9 ± 0.766.3 ± 2.059.4 ± 3.3
OEL+ReSAIL (PR)57.0 ± 0.465.6 ± 2.968.8 ± 2.270.6 ± 4.168.5 ± 1.863.7 ± 1.3

PR maintains higher mean success rates than OR on both splits from C3 through C6, with a larger gap at the final cycle. These results support the additional benefit of PR during extended training.

Together with the PI-conditioned evaluations, these findings provide further empirical support for sustaining PI-conditioned competence as an effective strategy for improving subsequent distillation and mitigating cross-cycle degradation.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q2: Clarity of the Main Results and Ablation Studies

A2: We clarify the comparisons in the main results and the purpose of each ablation below.

Main results (Table 1 in the main paper). We evaluate Qwen3-4B and Qwen3-8B on ALFWorld (ID/OOD) and TextCraft over three training cycles. We compare RFT, offline GRPO, EPD, SDPO, and OEL, and apply ReSAIL to SDPO and OEL. Each augmented method achieves higher mean success rates than its corresponding baseline at every reported cycle. Both augmented methods maintain non-decreasing mean success rates across the three cycles in all evaluated settings, supporting sustained improvement beyond the first cycle.

Ablation studies. The ablations address three separate questions:

  • What does each component contribute? (Table 2 in the main paper) Each row represents a separate configuration with different components enabled. OEL+SGS improves first-cycle performance over OEL. Adding TLB to OEL+SGS improves C3 performance, while adding PR to OEL+SGS+TLB yields continued gains through C3.
  • Does PI sensitivity identify useful steps? (Table 3(a) in the main paper) Top, Random, and Bottom each select 5% of steps, with TLB and PR fixed. Top achieves the highest mean success rates on both splits, supporting PI sensitivity as a selection criterion.
  • Does the retention view matter? (Table 3(b) and Figure 3 in the main paper) With SGS and TLB fixed, we compare no retention, ordinary-view retention, and privileged-view retention. Table 3(b) in the main paper reports deployment success without PI; Figure 3 in the main paper evaluates the same retention variants both with and without PI. Both retention variants improve deployment performance over no retention. The performance gap between PR and OR is larger when evaluation includes PI.

We would be happy to clarify any specific comparison or result that remains unclear.

Reviewer_YAFmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReSAIL, which aims to reduce performance collapse when agents repeatedly learn from their own experience. It selects steps where privileged information changes the teacher’s predictions the most, balances the selected losses across trajectories, and preserves the student’s predictions when privileged information is supplied. Added to OEL and an offline version of SDPO, it improves success rates across three training cycles on ALFWorld and TextCraft.

Critical strengths
  • The improvements are large and consistent across both model sizes and both base methods. On ALFWorld OOD with Qwen3-8B, adding ReSAIL raises final-cycle success from 41.4 to 78.4 for OEL and from 49.2 to 75.3 for SDPO. The ReSail methods also maintain their gains across all three cycles.

  • The method is easy to understand and intuitive. The ablation study shows how each component works and shows some expected findings (e.g. Table 2 shows that selection is important for early performance, but PR is necessary for maintaining performance). The introduced method is therefore more understandable.

Critical weaknesses
  • The evidence for training stability is limited to three cycles. As the method mainly argues for 'preventing collapse', it would be interesting to see whether it only mitigates the problem or completely solves it. Longer runs with several training seeds would help establish how reliably ReSAIL prevents the observed collapse.

  • The benefit of retaining the privileged view over ordinary KL regularization is less clear. In Table 3(b), ordinary-view retention already reaches a very close performance PR. Figure 3 shows a larger difference when privileged information is supplied, but does not directly show that this makes the model a better teacher in the next cycle. A controlled comparison of the resulting teachers would strengthen this explanation.

Questions for authors
  • Does PR still beat ordinary-view retention over longer runs?
  • How does SGS compare with simply picking the steps with the largest distillation loss?
  • Have you tried training beyond Cycle 3? Does performance remain stable?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your review and for recognizing the consistent improvements and informative ablations. We address your questions below with additional experiments and analysis.

Q1: Stability over More Cycles and the Benefits of Privileged Retention

A1: To assess stability beyond Cycle 3, we extended the experiments from three to six training cycles using Qwen3-4B on ALFWorld, with two independent training seeds per method. We compare OEL, OEL+ReSAIL with ordinary-view retention (OR), and our full method with privileged-view retention (PR).

Table 1. Six-cycle ALFWorld success rates (%; mean ± SD across two training seeds).

SplitMethodC1C2C3C4C5C6
IDOEL55.9 ± 1.357.2 ± 0.951.0 ± 5.244.9 ± 3.536.8 ± 2.430.9 ± 3.5
OEL+ReSAIL (OR)61.6 ± 2.471.0 ± 0.675.0 ± 2.975.8 ± 1.575.5 ± 1.568.5 ± 2.6
OEL+ReSAIL (PR)63.7 ± 2.068.8 ± 1.875.8 ± 2.279.2 ± 1.877.0 ± 1.375.1 ± 2.8
OODOEL48.0 ± 1.347.9 ± 2.641.1 ± 3.738.5 ± 3.335.2 ± 2.630.6 ± 2.4
OEL+ReSAIL (OR)55.1 ± 4.264.7 ± 2.867.7 ± 2.666.9 ± 0.766.3 ± 2.059.4 ± 3.3
OEL+ReSAIL (PR)57.0 ± 0.465.6 ± 2.968.8 ± 2.270.6 ± 4.168.5 ± 1.863.7 ± 1.3

OEL+ReSAIL (PR) continues to improve beyond Cycle 3, reaching its peak at Cycle 4 on both splits. Although performance subsequently declines, it remains above Cycle 1 levels through Cycle 6, while OEL falls below its Cycle 1 performance. These results support sustained mitigation of degradation over the six evaluated cycles, rather than its complete elimination. PR also maintains higher mean success rates than OR from Cycle 3 through Cycle 6 on both splits, with a larger gap at Cycle 6 than at Cycle 3.

Q2: A Controlled Comparison of Teaching Effectiveness

A2: Following your suggestion, we compare the teaching effectiveness of OR and PR checkpoints from C1–C3 by evaluating the resulting students. Each checkpoint serves as the teacher for a fresh student initialized from the same Qwen3-4B base model. All six groups use the same training data, privileged information, distillation objective, and training budget.

Table 2. Student success rates on ALFWorld (%; mean ± SD over three evaluations).

SplitTeacher sourceC1 teacherC2 teacherC3 teacher
IDOEL+ReSAIL (OR)62.8 ± 2.471.6 ± 3.970.8 ± 3.6
OEL+ReSAIL (PR)69.3 ± 1.274.2 ± 1.677.1 ± 1.8
OODOEL+ReSAIL (OR)48.4 ± 2.162.2 ± 3.267.4 ± 2.0
OEL+ReSAIL (PR)53.1 ± 4.364.1 ± 3.474.0 ± 5.0

PR teachers yield higher mean student success rates than OR teachers for each source cycle on both splits. These results directly support the additional teaching benefit of PR over ordinary-view retention.

Q3: Comparing Sensitivity-Guided Selection and Largest-Loss Selection

A3: To evaluate the suggested largest-loss alternative, we compare it with SGS using Qwen3-4B on ALFWorld, with two independent training seeds per method. Both methods select the top 5% of steps by their respective scores, with TLB, PR, task sampling, and the training budget held fixed. Largest-loss selection ranks steps by token-averaged student–teacher distillation KL divergence, whereas SGS ranks them by the change in the frozen teacher’s predictions induced by PI.

Table 3. ALFWorld success rates under SGS and largest-loss selection (%; mean ± SD across two training seeds).

SplitSelectionC1C2C3
IDLargest-loss61.7 ± 2.670.2 ± 1.775.3 ± 1.1
SGS63.7 ± 2.068.8 ± 1.875.8 ± 2.2
OODLargest-loss55.5 ± 1.859.1 ± 2.261.3 ± 1.3
SGS57.0 ± 0.465.6 ± 2.968.8 ± 2.2

SGS achieves higher mean OOD success rates in all three cycles, and the gap widens with each cycle. On the ID split, the two methods perform similarly: SGS has slightly higher means in C1 and C3, while largest-loss is slightly higher in C2. This similarity is consistent with the strong correlation between the two scores. In the SGS run, their Spearman correlation ranges from 0.917 to 0.932 before the first update of each cycle and from 0.779 to 0.817 before the final update of each cycle (see the newly added Figure 7 in Appendix D.5). Despite this strong correlation, SGS retains an advantage on OOD tasks.

Reviewer_YAFmOpenReview ↗

Response to authors

Thank you for the additional experiments. The six-cycle experiments address my concern about the evaluation iterations. Please incorporate these results into the revised paper. I maintain my positive view of the paper.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your positive feedback. We will incorporate the additional results into the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We sincerely thank all reviewers for their thorough reviews and constructive feedback. We are encouraged by their recognition of the value of addressing collapse in self-distillation (rGct, wKyc), the substantial performance gains (YAFm, wKyc), and the clear presentation (YAFm, wKyc).

We summarize the additional evidence and revisions below.

  1. Longer-term and broader evaluation. The six-cycle experiments support ReSAIL’s longer-term effectiveness (Appendix D.7), while the ToolAlpaca evaluation supports its applicability to a broader range of tasks (Appendix D.8).

  2. Teaching effectiveness of PR. The controlled teaching experiments support PR’s benefit for subsequent supervision (Appendix D.4), while the fixed-teacher control helps clarify the role of teacher updates in cross-cycle degradation (Appendix D.6).

  3. Comparison with largest-loss selection. The comparison supports the effectiveness of PI sensitivity as a step-selection criterion beyond the magnitude of the distillation loss (Appendix D.5).

  4. Clarified motivation and scope. We clarified that PR aims to sustain PI-conditioned competence while allowing the teacher to improve across cycles. We also clarified how ReSAIL mitigates collapse in iterative on-policy agent self-distillation and distinguished our approach from Chen et al.

Changes in the revised paper are marked in blue. Detailed comparisons and explanations are provided in our responses under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.