- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces an algorithm that reduces the cost of the RL-then-MOPD pipeline for multi-task post-training. Domain specialists are trained with RL, then distilled into one student through multi teacher, on-policy distillation. Doublefree reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online, teacher-free student training. They benchmark this on three small language models.
Critical strengths
- The algorithm results in large efficiency gains across three models from two families, and the cost is cheap.
- The paper includes both quantitative and theoretical support for the algorithm.
Critical weaknesses
- The improvements are modest in some cases (like DoubleFree-MPD vs Lightning MOPD) and full results are only reported for Qwen3-4B.
- Pre-scoring already exists in Lightning MOPD and Offline MPD, so trajectory reuse is the contribution.
Questions for authors
- How was the Balance selector chosen for Qwen3-4B in Table 3? If it was chosen using the benchmark results in Tables 4 and 7, please report Random as the primary 4B setting or select on a held-out split.
- Please report Lightning MOPD and Offline MPD (time, GPU-hours, accuracy) on Qwen3-8B and SmolLM3-3B, and multi-seed results for at least Standard MOPD and Lightning MOPD.
- What are Standard MOPD’s GPU-hours when teachers are time-shared with, or colocated in, the student GPU pool?
- Can you outline what contributions there are here, other than trajectory re-use?
2 comments
While the GPU usage reduction is significant in some cases, you can see that the GPU hours savings vs Lightning MOPD in the Qwen-3 case is minimal. I think this needs to be more adequately explored to more clearly demonstrate the algorithm generalizes, given that GPU time savings is the contribution. I also don't feel there's enough novel contribution in the paper to merit publication. This is more clever engineering than an innovation. I will be maintaining my score.
We thank the reviewer for the insightful comments and hope the responses below address the concerns.
W1 & Q2: Additional models and multi-seed results
Time and GPU-hours are reported in Table 4 of our response to Reviewer MmbC. Table 5 reports additional accuracy results. Due to limited GPU availability, additional Standard MOPD seeds on these two models are still in progress.
Table 5. Accuracy comparison on SmolLM3-3B and Qwen3-8B.
Consistent with the paper, All AVG equally weights the three domain averages, each computed by equally weighting its benchmarks.
Q1: Random as the primary selector
We agree that Random should be the primary setting. At submission, Balance was the only Qwen3-4B selector with completed multi-seed evaluation. We have now added Random’s multi-seed results and will report it in the main table, retaining the other selectors as ablations. Across three seeds, All AVG is 58.30 for Standard MOPD and 58.29 for DoubleFree-MPD Random (58.18 for Balance). Our main conclusions do not depend on a particular selector.
Table 6. Multi-seed accuracy comparison on Qwen3-4B.
Q3: Teacher time-sharing and GPU costs
We evaluated Qwen3-4B Standard MOPD with three teachers time-sharing the student’s four-GPU pool. As shown in Table 7, training completed in 13.90 hours (55.62 GPU-hours), including teacher-switching overhead. Compared with the dedicated 4+3-GPU deployment, time-sharing reduced GPU-hours by 39.8%, with a 5.4% increase in wall-clock time. Against this shared-pool implementation, DoubleFree-MPD still achieves a 3.16× speedup and uses 68.3% fewer GPU-hours.
Table 7. Qwen3-4B training efficiency across teacher deployments.
W2 & Q4: Contributions
DoubleFree is a simple and effective method for efficient multi-task LLM training. It exploits the connection between specialist exploration and student consolidation in the RL-then-MOPD pipeline. Pre-scoring is already used in single-domain methods such as Lightning OPD. It becomes particularly valuable in multi-task settings, where concurrently serving multiple teachers requires GPU resources that grow with domain count. Trajectory reuse decouples student training time from online rollout latency, while offline teacher supervision decouples its GPU requirements from the number of domains. These two forms of decoupling improve multi-task training scalability in both time and space. Our theoretical analysis characterizes the discrepancy from on-policy updates, while experiments evaluate the quality–cost trade-off.