- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Task Specialization Fine-Tuning (TSFT), which allocates a limited fine-tuning budget across specialized policies initialized from a pretrained model. It predicts task-wise learning curves, uses integer linear programming to maximize predicted joint task coverage, and periodically updates the predictions and allocation. Experiments span routing problems, continuous control, and LLM reinforcement fine-tuning, accompanied by model comparisons, sensitivity studies, and computational analyses.
Critical strengths
-
Well-motivated allocation problem. TSFT accounts for heterogeneous fine-tuning gains and overlapping policy coverage, providing a useful formulation beyond uniform specialization.
-
Broad empirical investigation. Experiments cover several distinct domains, with meaningful gains in selected settings. Alternative prediction models, hyperparameter studies, and solver-cost comparisons help substantiate the framework’s feasibility.
Critical weaknesses
-
The allocation baselines are insufficiently competitive and controlled. A greedy allocator using the same predictions and global coverage objective would help isolate the value of exact optimization. Strong learning-curve-aware methods, such as Freeze-Thaw BO or ifBO, are also absent, although adapting them to the portfolio objective would be necessary.
-
The contribution of online replanning is not directly established. Execution-interval sensitivity does not replace a matched comparison between one-shot allocation and repeated model estimation and replanning. Consequently, the benefit of feedback remains insufficiently separated from forecasting and joint coverage optimization.
-
Performance advantages are uneven and incompletely replicated. Some gains are small, including 0.2 percentage points on CVRPTW 3/150 and 0.8 points on the five-seed Meta-World 5/25 setting. Ant coverage is highly threshold-sensitive, while the main Meta-World and LLM experiments use one training seed, limiting evidence for consistently substantial improvements.
-
The theoretical guarantee remains conditional. The bound assumes coverage-prediction and final-allocation errors without establishing that these are small for the implemented online procedure. Exact ILP solutions at individual planning steps therefore provide limited guarantees for the final executed allocation
Questions for authors
-
Are scheduling-validation examples strictly separated from final test examples? How is adaptive model-selection bias controlled for the LLM benchmarks described as OOD?
-
Could you report per-benchmark pass@1, raw Ant returns for the compared methods, and ranking stability across reasonable coverage thresholds?
-
How stable are the main Meta-World and LLM results across training seeds and scheduling randomness? How are Random and Adaptive results averaged?
-
What are the end-to-end costs, including training, repeated evaluation, model storage, and inference selection? How much portfolio coverage can a practical policy selector recover?
6 comments
Response to Reviewer 6Jpo (2/4)
W3 & Q2 & Q3: Performance advantages are uneven and incompletely replicated.
Our experiments demonstrate TSFT's improvements over strong baselines across a broad range of domains. For example, MOORE is a state-of-the-art baseline on Meta-World, yet TSFT achieves a further 14% improvement in the challenging 10/50 setting. To further substantiate TSFT's effectiveness and the robustness of our findings, we provide detailed raw results and additional experiments below.
-
Q2: Could you report per-benchmark pass@1, raw Ant returns for the compared methods, and ranking stability across reasonable coverage thresholds?
-
Please see the following table for detailed per-benchmark pass@1 results. Starting from a pretrained LLM with an average pass@1 of 26.89%, TSFT achieves 39.31% across all benchmarks, outperforming all non-oracle baselines and approaching oracle-level performance.
LLM-FT 4/100 Oracle Oracle-Warmup Uniform Adaptive LinUCB TSFT DAPO-17K 32.00% 32.00% 23.00% 21.60% 18.00% 34.00% MATH-500 76.00% 76.00% 73.00% 75.00% 77.00% 76.00% GSM8K 84.00% 84.00% 86.00% 84.80% 84.00% 86.00% CodeContests+ 36.23% 36.23% 38.80% 36.01% 36.23% 34.13% AIME 2024 10.00% 10.00% 3.33% 6.00% 6.67% 6.67% AIME 2025 13.33% 13.33% 10.00% 10.00% 10.00% 10.00% Minerva Math 20.00% 20.00% 17.00% 18.40% 17.00% 20.00% MBPP 61.00% 60.00% 61.00% 62.20% 56.00% 60.00% BigCodeBench 27.00% 27.00% 24.00% 23.80% 19.00% 27.00% Average 39.95% 39.84% 37.35% 37.53% 35.99% 39.31% -
In the original paper, we evaluated Ant using only three random seeds. We have since identified a library incompatibility issue affecting the Ant experiments. To address this issue, we are rerunning the experiments with five random seeds and reporting the average raw performance (i.e., mean episodic returns) in the following tables. Note that CRL's optimization objective is global coverage rather than returns: higher coverage does not necessarily translate into higher returns, although the two metrics are correlated. We will incorporate the updated results into the revised paper once all experiments are completed.
Ant 3/50 3/100 4/100 Oracle 1688.7 2130.8 2391.3 Oracle-Warmup 2019.6 2296.8 2386.5 Pretrained 1145.4 1145.4 1145.4 MTL 1661.2 2132.1 2132.1 Random 1897.8 1992.4 2016.0 Uniform 1897.9 2009.5 2086.8 Adaptive 1792.0 2072.8 2080.6 LinUCB 1449.5 1705.4 1657.0 TSFT 1920.3 2123.2 2158.2 -
The following table presents the (coverage) ranking stability of the compared methods. We evaluate all baselines on Ant across diverse settings, varying the number of policies, training budgets, and coverage thresholds. The results show that TSFT consistently ranks among the Top2 methods in most settings, demonstrating its effectiveness and robustness across different configurations.
Threshold N/K TSFT Random Uniform Adaptive LinUCB 1,000 3/50 1 5 2 3 4 1,000 3/100 1 4 5 2 3 1,000 4/100 1= 4 5 1= 3 1,500 3/50 1 5 3 2 4 1,500 3/100 2 5 4 3 1 1,500 4/100 2 5 4 1 3 2,000 3/50 1 4 5 3 2 2,000 3/100 1 4 5 3 2 2,000 4/100 1 4 5 2 3 2,500 3/50 1 5 3 4 2 2,500 3/100 1 5 4 2 3 2,500 4/100 1 5 4 3 2 3,000 3/50 1 3 4 2 5 3,000 3/100 1 5 4 2 3 3,000 4/100 1 5 4 3 2 3,500 3/50 2 1 3 4 5 3,500 3/100 1 5 3 4 2 3,500 4/100 2 5 4 1 3 Average 1.22 4.39 3.94 2.50 2.89
-
Response to Reviewer 6Jpo (3/4)
-
Q3: More training seeds. How are Random and Adaptive results averaged?
Thanks for your comment. Regarding training seeds: The LLM experiments require
16 NVIDIA H200 GPUs, which are unfortunately unavailable to us during the limited rebuttal period. Nevertheless, we agree that additional runs would further strengthen the empirical evidence. To address this concern, we conduct additional runs on CVRP and Meta-World, whereSeed1corresponds to the original results andSeed2represents the newly obtained results. As shown in the table below, the additional results are consistent with the empirical findings reported in the original paper, providing further support for the effectiveness and robustness of TSFT. Meanwhile, we are continuing to conduct additional experiments (e.g., with more policies and seeds) and plan to run the LLM experiments with additional seeds once sufficient computational resources become available. We will incorporate the complete results into the final version of the paper.Regarding scheduling randomness, we evaluate Random and Adaptive over five allocation seeds in the main LLM and Meta-World experiments. For each run, we compute coverage using the task-specific thresholds and then report the equally weighted arithmetic mean across runs. TSFT does not sample allocations: given the same observations and fixed solver settings, its scheduling decisions are deterministic. This eliminates allocation-sampling variability, while training-seed variability remains relevant.
| CVRP 3/100 (Seed1) | CVRP 3/100 (Seed2) | CVRP 3/150 (Seed1) | CVRP 3/150 (Seed2) | Meta-World 10/50 (Seed1) | Meta-World 10/50 (Seed2) | |
|---|---|---|---|---|---|---|
| Oracle | 26.7% | 26.0% | 29.3% | 29.0% | 86.0% | 84.0% |
| Oracle-Warmup | 21.4% | 21.5% | 24.5% | 24.7% | 84.0% | 80.0% |
| Pretrained | 6.8% | 6.7% | 6.8% | 6.7% | 58.0% | 62.0% |
| MTL | 9.5% | 9.3% | 11.0% | 10.9% | 60.0% | 72.0% |
| Random | 10.1% | 11.3% | 11.0% | 11.1% | 71.2% | 65.2% |
| Uniform | 7.9% | 6.6% | 15.4% | 12.7% | 70.0% | 68.0% |
| Adaptive | 10.4% | 12.6% | 16.1% | 18.5% | 71.2% | 70.4% |
| LinUCB | 12.6% | 14.0% | 20.5% | 20.5% | 68.0% | 74.0% |
| TSFT | 18.7% | 20.4% | 23.0% | 21.6% | 74.0% | 76.0% |
Q1: Are scheduling-validation examples strictly separated from final test examples? How is adaptive model-selection bias controlled for the LLM benchmarks described as OOD?
Yes, we seperate validation samples used for online budget allocation and test samples used for final evaluation. We will clarify this in the revision.
The CRL setting aims to maximize task coverage within a predefined, bounded context space, reflecting practical applications where a practitioner seeks strong performance across a specified collection of related tasks. Accordingly, validation feedback from all tasks is available to guide budget allocation, while their test data remain held out. Moreover, ID and OOD refer to benchmark distributions relative to the corpora used to fine-tune the specialized policies, rather than whether a benchmark is excluded from the allocation procedure. Both groups provide validation feedback for allocation, but neither provides test feedback. We acknowledge that the original terminology could suggest that the OOD benchmarks are entirely unseen during allocation, and we have revised the paper to make this distinction explicit.
Response to Reviewer 6Jpo (1/4)
We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for providing constructive feedback. Below, we respond to each of your comments in detail, where
Wdenotes Weakness (W1–W4) andQdenotes Question (Q1–Q4).W1: The allocation baselines are insufficiently competitive and controlled.
We add a
Greedybaseline that replaces exact ILP solving by sequentially allocating each budget unit to the policy with the largest predicted marginal gain in global coverage, while retaining TSFT’s surrogate model and online replanning procedure. We conduct experiments on CVRP. The results shown in the below table support the value of exact optimization.We further add portfolio adaptations of
Freeze-Thaw BOandifBO, accounting for coverage overlap when evaluating each policy’s continuation. We conduct experiments on CVRP and report results in the below table. These BO methods exploit partial learning curves and predictive uncertainty, but are originally designed to identify the best individual configuration. Our portfolio adaptations select which policy to continue while holding the others at their current checkpoints. In contrast, TSFT jointly optimizes the remaining budget across policies, accounting for how their future coverage sets overlap and complement one another. Together, these two experiments consistently demonstrate the effectiveness of TSFT compared with strong baselines.W2: The contribution of online replanning is not directly established.
Online replanning is designed to mitigate the effects of modeling errors by incorporating new observations and updating subsequent allocations. To assess its contribution, we add a one-shot allocation baseline: after the same warmup, it fits the performance models and solves the resulting ILP once, then executes the allocation without further model estimation or replanning.
Final coverage: We conduct additional experiments on the largest CVRP setting (4/150), involving four policies and a total budget of 150 units. We consider two warmup budgets, with 3 and 5 units per policy, respectively. As shown in the table below, TSFT consistently outperforms one-shot allocation, achieving improvements of 9.0% and 5.2%, respectively. Notably, the performance gap is larger under a smaller warmup budget, highlighting the importance of online replanning when initial observations are limited.
Prediction accuracy: Beyond final coverage, we observe that the average prediction MSE decreases from 5.178 (one-shot model) to 1.936 (TSFT's final-round model) as more observations become available. Fig. 2 further shows that the average recall of coverage-set predictions improves with additional observations, providing further empirical evidence of the effectiveness of online model re-estimation.
In summary, while one-shot allocation remains competitive by leveraging performance forecasting and exact optimization, its effectiveness depends heavily on the accuracy of the initial models, particularly under limited warmup budgets (e.g., W = 3 × N). In contrast, TSFT is more robust to initial modeling errors, as it continuously refines predictions using observed fine-tuning outcomes and adapts subsequent allocations accordingly. These results provide direct empirical evidence of the benefits of online re-estimation and replanning.