acceptodds
Under review as a conference paper at ICLR 2027

DoubleFree: Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

Abstract

Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves - consolidation speedups and reduces GPU-hours by -, while maintaining comparable multi-domain performance.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Multi-Task, LLM Post-Training, RL, Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6aR9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an algorithm that reduces the cost of the RL-then-MOPD pipeline for multi-task post-training. Domain specialists are trained with RL, then distilled into one student through multi teacher, on-policy distillation. Doublefree reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online, teacher-free student training. They benchmark this on three small language models.

Critical strengths
  1. The algorithm results in large efficiency gains across three models from two families, and the cost is cheap.
  2. The paper includes both quantitative and theoretical support for the algorithm.
Critical weaknesses
  1. The improvements are modest in some cases (like DoubleFree-MPD vs Lightning MOPD) and full results are only reported for Qwen3-4B.
  2. Pre-scoring already exists in Lightning MOPD and Offline MPD, so trajectory reuse is the contribution.
Questions for authors
  1. How was the Balance selector chosen for Qwen3-4B in Table 3? If it was chosen using the benchmark results in Tables 4 and 7, please report Random as the primary 4B setting or select on a held-out split.
  2. Please report Lightning MOPD and Offline MPD (time, GPU-hours, accuracy) on Qwen3-8B and SmolLM3-3B, and multi-seed results for at least Standard MOPD and Lightning MOPD.
  3. What are Standard MOPD’s GPU-hours when teachers are time-shared with, or colocated in, the student GPU pool?
  4. Can you outline what contributions there are here, other than trajectory re-use?
2 comments

We thank the reviewer for the insightful comments and hope the responses below address the concerns.

W1 & Q2: Additional models and multi-seed results

Time and GPU-hours are reported in Table 4 of our response to Reviewer MmbC. Table 5 reports additional accuracy results. Due to limited GPU availability, additional Standard MOPD seeds on these two models are still in progress.

Table 5. Accuracy comparison on SmolLM3-3B and Qwen3-8B.

ModelMethodSeedsAIME24AIME25AMC23MATH500LCB v6BCBHumanEval+IFBenchIFEvalAll AVG
SmolLM3-3BStandard MOPD4237.2926.8873.4483.2027.007.0161.5944.0081.3349.91
Offline MPD4235.3126.8874.1481.6025.215.0764.0242.0080.7849.10
Lightning MOPD42–4436.67 ± 0.4225.31 ± 0.6874.35 ± 0.3579.80 ± 2.3126.00 ± 0.435.94 ± 0.9660.37 ± 2.6645.67 ± 1.8681.95 ± 0.8749.54 ± 0.18
DoubleFree-MPD42–4436.39 ± 1.3626.39 ± 0.7775.00 ± 0.2881.47 ± 0.6426.38 ± 0.376.17 ± 0.5963.21 ± 2.1441.78 ± 0.3881.08 ± 0.2849.39 ± 0.19
Qwen3-8BStandard MOPD4259.1751.0491.3387.4036.0717.9982.9346.3387.8061.65
Offline MPD4260.4253.0290.9487.6035.4318.7582.9344.0087.8061.53
Lightning MOPD42–4460.87 ± 1.1153.92 ± 0.9791.82 ± 0.6388.73 ± 0.6435.79 ± 0.5017.91 ± 0.3082.52 ± 0.7046.00 ± 1.0087.12 ± 0.7561.93 ± 0.42
DoubleFree-MPD42–4459.93 ± 0.6952.15 ± 0.2291.12 ± 0.6688.27 ± 0.6134.90 ± 0.4816.53 ± 0.1883.54 ± 2.7944.00 ± 0.5887.49 ± 0.7561.20 ± 0.20

Consistent with the paper, All AVG equally weights the three domain averages, each computed by equally weighting its benchmarks.

Q1: Random as the primary selector

We agree that Random should be the primary setting. At submission, Balance was the only Qwen3-4B selector with completed multi-seed evaluation. We have now added Random’s multi-seed results and will report it in the main table, retaining the other selectors as ablations. Across three seeds, All AVG is 58.30 for Standard MOPD and 58.29 for DoubleFree-MPD Random (58.18 for Balance). Our main conclusions do not depend on a particular selector.

Table 6. Multi-seed accuracy comparison on Qwen3-4B.

MethodAIME24AIME25AMC23MATH500Math AVGLCB v6BCBHumanEval+Code AVGIFBenchIFEvalIF AVGAll AVG
Standard MOPD55.10 ± 0.8352.57 ± 0.4391.98 ± 0.3390.80 ± 0.5372.61 ± 0.1732.40 ± 0.299.99 ± 0.1380.28 ± 0.9340.89 ± 0.4340.00 ± 0.8882.81 ± 0.8561.40 ± 0.6158.30 ± 0.30
DoubleFree-MPD Random53.92 ± 0.4251.84 ± 1.4291.80 ± 0.6789.93 ± 1.2171.87 ± 0.5732.74 ± 0.5810.22 ± 0.2279.68 ± 1.2740.88 ± 0.5440.44 ± 1.5083.80 ± 1.3162.12 ± 1.2958.29 ± 0.74

Q3: Teacher time-sharing and GPU costs

We evaluated Qwen3-4B Standard MOPD with three teachers time-sharing the student’s four-GPU pool. As shown in Table 7, training completed in 13.90 hours (55.62 GPU-hours), including teacher-switching overhead. Compared with the dedicated 4+3-GPU deployment, time-sharing reduced GPU-hours by 39.8%, with a 5.4% increase in wall-clock time. Against this shared-pool implementation, DoubleFree-MPD still achieves a 3.16× speedup and uses 68.3% fewer GPU-hours.

Table 7. Qwen3-4B training efficiency across teacher deployments.

MethodTeacher deploymentTraining GPUsTime (h)GPU-hoursSpeedupGPU-hours reduction
Standard MOPDDedicated4+313.1992.31——
Standard MOPDTime-shared413.9055.621.00×0%
DoubleFree-MPDOffline scoring44.4017.613.16×68.3%

W2 & Q4: Contributions

DoubleFree is a simple and effective method for efficient multi-task LLM training. It exploits the connection between specialist exploration and student consolidation in the RL-then-MOPD pipeline. Pre-scoring is already used in single-domain methods such as Lightning OPD. It becomes particularly valuable in multi-task settings, where concurrently serving multiple teachers requires GPU resources that grow with domain count. Trajectory reuse decouples student training time from online rollout latency, while offline teacher supervision decouples its GPU requirements from the number of domains. These two forms of decoupling improve multi-task training scalability in both time and space. Our theoretical analysis characterizes the discrepancy from on-policy updates, while experiments evaluate the quality–cost trade-off.

While the GPU usage reduction is significant in some cases, you can see that the GPU hours savings vs Lightning MOPD in the Qwen-3 case is minimal. I think this needs to be more adequately explored to more clearly demonstrate the algorithm generalizes, given that GPU time savings is the contribution. I also don't feel there's enough novel contribution in the paper to merit publication. This is more clever engineering than an innovation. I will be maintaining my score.

Reviewer_MmbCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DoubleFree to reduce the cost of multi-teacher policy distillation. The main idea is to reuse trajectories collected during domain-specific RL training and precompute the final teachers’ token-level log-probabilities. During student training, the method loads these trajectories and teacher scores, while recomputing the student’s log-probabilities. This avoids additional rollouts and online teacher serving during distillation. The authors also provide theoretical analysis of the difference between updates using historical trajectories and fresh student rollouts. Experiments on three models show lower training costs with performance close to standard MOPD.

Critical strengths
  1. The motivation is clear. Reusing RL trajectories and caching teacher scores can reduce the cost of multi-teacher distillation.
  2. The method is simple and easy to implement in existing training frameworks.
  3. Experiments on three models show substantial cost reductions with performance close to standard MOPD.
Critical weaknesses
  1. I think the novelty needs to be clarified. Reusing historical trajectories and caching teacher scores are natural ideas given existing offline distillation and replay-based methods. My main concern is the distribution mismatch between historical trajectories and the evolving student. Although the paper provides theoretical analysis, it remains unclear whether this mismatch stays small during training and how the method handles cases where the student moves far from the historical policies.
  2. The performance is generally close to standard MOPD, with small drops in some settings. I think this trade-off is acceptable because the main goal is to reduce training cost. However, the additional efficiency gain over existing offline distillation methods needs further evaluation.
  3. The connection between the proposed method and multi-teacher distillation needs clarification. Trajectory reuse and offline teacher scoring seem broadly applicable to policy distillation, including the single-teacher setting. It is unclear which part of the method specifically addresses challenges introduced by using multiple teachers.
Questions for authors

Could the authors clarify the main differences between DoubleFree and REGEN, and what advantages DoubleFree offers over REGEN?

Could the authors clarify the main differences and advantages over the async OPD?

[1] Kang, Wonjun, et al. "AsyncOPD: How Stale Can On-Policy Distillation Be?." arXiv preprint arXiv:2606.24143 (2026). [2] Chen, Yunjie, et al. "REGEN: Replay-recycling for expert-to-generalist distillation with offline reinforcement learning." arXiv preprint arXiv:2607.19450 (2026).

1 comment

We want to thank the reviewer for the insightful comments and suggestions. We provide detailed responses point by point. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.

W3 & Q1 & Q2: Novelty and comparison with related methods

Systems efficiency and multi-task scalability: DoubleFree addresses the rollout cost and GPU demands of multi-task consolidation from a systems perspective. By reusing RL trajectories and applying pre-scoring, it eliminates additional rollouts and decouples training GPU requirements from domain count, improving training scalability in both time and space. See our response to Reviewer 6aR9 (W2 & Q4) for details.

Related work:

  • REGEN. We thank the reviewer for highlighting this concurrent work. Both methods reuse RL trajectories for efficient multi-task consolidation. REGEN uses reward-based offline RL with sequence-level advantages, whereas DoubleFree-MPD retains token-level supervision from final RL teachers, providing finer-grained guidance beyond scalar rewards.

  • AsyncOPD. This paper optimizes the staleness–throughput trade-off within the OPD stage while retaining online rollout generation and online teacher scoring. DoubleFree targets the RL-then-MOPD pipeline, reusing trajectories from the preceding RL stage instead of stale rollouts from preceding OPD update steps. This completely eliminates additional generation and online teacher serving, improving multi-task training scalability in both time and space.

W1: Distribution mismatch and policy drift

Regarding distribution mismatch, the right panel of Figure 5(c) in Appendix C.1 shows that the student–behavior token log-probability gap decreases overall from an early-stage average of 0.176 to a late-stage average of 0.035. This indicates closer agreement between the student and the generating policies on replayed tokens, providing a token-level diagnostic of off-policy mismatch during training.

To address potential drift over longer training horizons, a possible extension is an interleaved pipeline that alternates shorter RL and MOPD stages. Each round trains specialists from the current student, then consolidates their capabilities using trajectories collected during that round. The updated student initializes the next round, refreshing specialists and replay data, which may help limit distribution mismatch. We have preliminarily explored this pipeline but observed no additional accuracy gains in our current setting, where historical reuse already preserves accuracy well. Concurrent work, Pistis[1], also explores interleaved RL and OPD on multimodal models.

[1] ByteDance. Pistis Technical Report. https://arxiv.org/html/2609.28554v1, 2026.

W2: Efficiency gains over offline baselines

We have extended the comparison to SmolLM3-3B and Qwen3-8B, as reported in Table 4. DoubleFree-MPD achieves 1.94–2.57× speedups and 48.5–61.1% savings over Offline MPD, and 1.08–2.42× speedups and 7.3–58.8% GPU-hours savings over Lightning MOPD. The gain over Lightning is modest on Qwen3-8B, where Lightning trains on shorter trajectories generated by the initial student (939 vs. 1,985 tokens).

Table 4. Efficiency comparison on SmolLM3-3B and Qwen3-8B with offline distillation baselines.

ModelMethodTraining GPUsGPU-hours ↓Time (h) ↓Avg. response length (tokens)DoubleFree speedup ↑GPU-hours saved by DoubleFree ↑
SmolLM3-3BStandard MOPD4+361.588.801,441.333.63×84.2%
Lightning MOPD423.545.891,349.892.42×58.8%
Offline MPD424.956.241,418.642.57×61.1%
DoubleFree-MPD49.702.431,349.831.00×0%
Qwen3-8BStandard MOPD8+6225.6516.122,405.744.52×87.4%
Lightning MOPD830.753.84939.261.08×7.3%
Offline MPD855.336.922,407.171.94×48.5%
DoubleFree-MPD828.503.561,985.321.00×0%

The last two columns report DoubleFree-MPD’s speedup and GPU-hours savings relative to the method in the corresponding row, within the same model.

Reviewer_CRXaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DoubleFree, an efficient approach to consolidating domain-specific RL experts into a shared student. Instead of generating fresh student rollouts, DoubleFree reuses trajectories collected during expert RL and optionally precomputes final-teacher token log-probabilities. The paper introduces MPD and SFT variants, provides a conditional bound on local update discrepancies relative to on-policy distillation, and evaluates the approach across three models and three domains.

Critical strengths
  1. Practical efficiency improvements with competitive performance. Across the evaluated settings, DoubleFree reports 2.99–5.75× consolidation speedups and 80.9–90.1% reductions in allocated GPU-hours across its two variants, while retaining broadly comparable average benchmark performance. The cost breakdown includes offline preparation and teacher resource allocation, making the reported savings interpretable.

  2. Simple and useful reuse of existing training artifacts. Exploiting trajectories already produced during specialist RL is a practical design choice that requires limited changes to existing training frameworks. The evaluation covers math, code, and instruction following across two model families.

Critical weaknesses
  1. The connection between the theoretical conditions and practical training remains insufficiently established. The update-discrepancy bound depends on distribution mismatch and update-contribution moments, but the experiments do not establish whether these quantities make the bound meaningfully small during training. Figure 4 measures a prefix-level coverage proxy rather than the full-trajectory discrepancy appearing in the theorem. Moreover, similar final accuracy does not directly establish similar update directions. The analysis therefore provides a plausible explanation, but limited evidence about when historical reuse reliably approximates OPD or when that approximation fails.

  2. The incremental benefit of using the full exploration history is not sufficiently isolated. On Qwen3-4B, DoubleFree-MPD improves over the closer offline baseline, Lightning MOPD, by approximately 1.13× in time and 11.3% in GPU-hours; this baseline is not evaluated on the other two models. Comparisons against cached late-stage RL trajectories and mixtures of early- and late-stage trajectories are also missing. These controls would clarify whether intermediate exploration stages provide a distinct benefit beyond avoiding additional generation.

Questions for authors
  1. At matched student checkpoints and prompts, can you directly measure the relative error and cosine similarity between fresh-rollout and historical-reuse update directions? Does the coverage diagnostic predict these differences?
  2. How does full-history reuse compare with reusing only late-stage RL trajectories or an early/late mixture under matched budgets? Does shuffling the historical trajectories change the results, given the chronological replay described in Appendix C.1?
1 comment

We want to thank the reviewer for the insightful comments. We hope our responses can address your concerns and lead you to consider increasing your rating of our work.

W1 & Q1:Gradient comparisons and distribution mismatch

We measured full-parameter distillation gradients at early, middle, and late Qwen3-4B checkpoints using identical prompts. We compared historical-reuse and fresh-rollout gradients (reuse–fresh), using two independently sampled fresh-rollout gradients (fresh–fresh) as a reference for sampling variability. Values are averaged across all domains.

Table 1. Gradient comparisons at matched checkpoints and prompts.

CheckpointCosine: fresh–freshCosine: reuse–freshRelative error: fresh–freshRelative error: reuse–fresh
Early0.4470.2950.9391.794
Middle0.2800.2001.2112.590
Late0.2230.2272.5211.796

The difference between the mean fresh–fresh and reuse–fresh cosine similarities narrows from 0.153 to 0.080 and then 0.004. The right panel of Figure 5(c) also shows that the student–behavior token log-probability gap decreases overall from 0.176 to 0.035.

Our theory conditionally bounds update differences through distribution mismatch and the magnitude of trajectory-level gradient contributions. We agree that comparable final accuracy alone does not guarantee aligned update directions. To test the relevance of Figure 4’s coverage diagnostic, we paired it with fresh–reuse gradient measurements at intermediate student checkpoints, using matched prompts per domain. Higher mismatch was associated with lower gradient cosine similarity (Spearman=-0.60) and higher relative error (0.48). These results provide preliminary evidence that the coverage diagnostic is informative about update disagreement.

W2 & Q2: Stage coverage and replay order ablation

Thank you for these valuable comments. Efficiency results on SmolLM3-3B and Qwen3-8B are reported in Table 4 of our response to Reviewer MmbC, and the corresponding accuracy results are provided in Table 5 of our response to Reviewer 6aR9.

For stage coverage, we compared three trajectory sources on SmolLM3-3B and Qwen3-4B:

  • History-25%: sample one step from every four consecutive RL steps across steps 1–200.
  • Late-only: use steps 151–200.
  • Early+Late: sample 25 steps from each of steps 1–50 and 151–200.

Within each model, all three strategies reuse trajectories from the same RL history pool but differ in stage coverage. They use the same cache size, training-update budget, student initialization, and final teachers.

In Table 2, History-25% achieves comparable or slightly higher All AVG on both models (Bold indicates the highest score). The underlined entries highlight more pronounced accuracy drops relative to History-25%, suggesting a benefit from broader coverage across exploration stages in these runs.

Table 2. Stage coverage ablation.

ModelStrategyAIME24AIME25AMC23MATH500LCB v6BCBHumanEval+IFBenchIFEvalAll AVG
SmolLM3-3BHistory-25%34.2727.0874.6180.6026.506.9364.0245.6781.1550.01
SmolLM3-3BLate-only36.4625.7374.0683.4026.296.0865.24<u>40.33</u>81.8949.52
SmolLM3-3BEarly+Late35.3127.0874.4582.2025.575.2460.3741.6782.2649.04
Qwen3-4BHistory-25%56.3553.3392.2789.8032.5010.7377.4442.0084.4758.80
Qwen3-4BLate-only<u>54.17</u><u>51.77</u>91.6490.2031.5010.3078.0541.6785.4058.48
Qwen3-4BEarly+Late55.7355.4291.8889.8031.8611.4979.88<u>38.67</u>84.2958.59

For replay order, we compared chronological and shuffled full-history replay on Qwen3-4B across three seeds. Within each seed, we varied only the replay order, holding the historical pool and training settings fixed. Table 3 shows comparable overall performance under both schedules.

These results suggest that comparable performance does not require chronological replay. We will clarify in Appendix C.1 that the decreasing student–behavior gap is observed under chronological replay, without attributing this trend to replay order itself.

Table 3. Replay order ablation on Qwen3-4B.

Replay orderAIME24AIME25AMC23MATH500LCB v6BCBHumanEval+IFBenchIFEvalAll AVG
Chronological53.92 ± 0.4251.84 ± 1.4291.80 ± 0.6789.93 ± 1.2132.74 ± 0.5810.22 ± 0.2279.68 ± 1.2740.44 ± 1.5083.80 ± 1.3158.29 ± 0.74
Shuffled56.18 ± 0.4254.27 ± 0.5591.88 ± 0.4989.73 ± 0.5832.83 ± 0.239.88 ± 1.1080.08 ± 1.2739.56 ± 1.1783.92 ± 0.3758.56 ± 0.15

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.