acceptodds
Under review as a conference paper at ICLR 2027

Task Specialization Fine-Tuning for Contextual Reinforcement Learning

Abstract

Contextual Reinforcement Learning (CRL) seeks to generalize classical RL by maximizing task coverage across a context space of related tasks. While prior works often train from scratch and rely on either multi-task learning for a single policy or strategically training multiple policies, we advocate for a unified alternative: pretraining a single policy with good initial performance, followed by fine-tuning multiple policies for task specialization. This new paradigm, however, introduces unique challenges, such as heterogeneous marginal returns and sample inefficiency. This raises a critical research question: given a pretrained policy and a constrained budget, *how much* fine-tuning should each task region receive to enable sample-efficient CRL? To this end, we propose *Task Specialization Fine-Tuning (TSFT)*, an online framework that formalizes budget allocation as a Markov decision process and uses a simple parametric function to predict fine-tuning performance. TSFT leverages dynamic programming to derive the optimal allocation, with a receding-horizon variant that trades optimality for efficiency. Extensive experiments across diverse decision domains, including combinatorial optimization, continuous control, and LLM fine-tuning, demonstrate that TSFT significantly outperforms baselines in task coverage and approaches oracle performance. Our work charts a new direction for model-based CRL, aligning with the modern pretrain-finetune era.

Primary area:
optimization
Keywords:
Fine-tuning, Discrete Optimization, Control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6JpoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Task Specialization Fine-Tuning (TSFT), which allocates a limited fine-tuning budget across specialized policies initialized from a pretrained model. It predicts task-wise learning curves, uses integer linear programming to maximize predicted joint task coverage, and periodically updates the predictions and allocation. Experiments span routing problems, continuous control, and LLM reinforcement fine-tuning, accompanied by model comparisons, sensitivity studies, and computational analyses.

Critical strengths
  1. Well-motivated allocation problem. TSFT accounts for heterogeneous fine-tuning gains and overlapping policy coverage, providing a useful formulation beyond uniform specialization.

  2. Broad empirical investigation. Experiments cover several distinct domains, with meaningful gains in selected settings. Alternative prediction models, hyperparameter studies, and solver-cost comparisons help substantiate the framework’s feasibility.

Critical weaknesses
  1. The allocation baselines are insufficiently competitive and controlled. A greedy allocator using the same predictions and global coverage objective would help isolate the value of exact optimization. Strong learning-curve-aware methods, such as Freeze-Thaw BO or ifBO, are also absent, although adapting them to the portfolio objective would be necessary.

  2. The contribution of online replanning is not directly established. Execution-interval sensitivity does not replace a matched comparison between one-shot allocation and repeated model estimation and replanning. Consequently, the benefit of feedback remains insufficiently separated from forecasting and joint coverage optimization.

  3. Performance advantages are uneven and incompletely replicated. Some gains are small, including 0.2 percentage points on CVRPTW 3/150 and 0.8 points on the five-seed Meta-World 5/25 setting. Ant coverage is highly threshold-sensitive, while the main Meta-World and LLM experiments use one training seed, limiting evidence for consistently substantial improvements.

  4. The theoretical guarantee remains conditional. The bound assumes coverage-prediction and final-allocation errors without establishing that these are small for the implemented online procedure. Exact ILP solutions at individual planning steps therefore provide limited guarantees for the final executed allocation

Questions for authors
  1. Are scheduling-validation examples strictly separated from final test examples? How is adaptive model-selection bias controlled for the LLM benchmarks described as OOD?

  2. Could you report per-benchmark pass@1, raw Ant returns for the compared methods, and ranking stability across reasonable coverage thresholds?

  3. How stable are the main Meta-World and LLM results across training seeds and scheduling randomness? How are Random and Adaptive results averaged?

  4. What are the end-to-end costs, including training, repeated evaluation, model storage, and inference selection? How much portfolio coverage can a practical policy selector recover?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6Jpo (1/4)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for providing constructive feedback. Below, we respond to each of your comments in detail, where W denotes Weakness (W1–W4) and Q denotes Question (Q1–Q4).

W1: The allocation baselines are insufficiently competitive and controlled.

We add a Greedy baseline that replaces exact ILP solving by sequentially allocating each budget unit to the policy with the largest predicted marginal gain in global coverage, while retaining TSFT’s surrogate model and online replanning procedure. We conduct experiments on CVRP. The results shown in the below table support the value of exact optimization.

CVRP3/1003/1504/150
Greedy5.1%8.6%10.2%
TSFT18.7%23.0%26.4%

We further add portfolio adaptations of Freeze-Thaw BO and ifBO, accounting for coverage overlap when evaluating each policy’s continuation. We conduct experiments on CVRP and report results in the below table. These BO methods exploit partial learning curves and predictive uncertainty, but are originally designed to identify the best individual configuration. Our portfolio adaptations select which policy to continue while holding the others at their current checkpoints. In contrast, TSFT jointly optimizes the remaining budget across policies, accounting for how their future coverage sets overlap and complement one another. Together, these two experiments consistently demonstrate the effectiveness of TSFT compared with strong baselines.

CVRP3/1003/1504/150
Freeze-Thaw BO7.4%11.6%12.7%
ifBO9.9%22.4%9.5%
TSFT18.7%23.0%26.4%

W2: The contribution of online replanning is not directly established.

Online replanning is designed to mitigate the effects of modeling errors by incorporating new observations and updating subsequent allocations. To assess its contribution, we add a one-shot allocation baseline: after the same warmup, it fits the performance models and solves the resulting ILP once, then executes the allocation without further model estimation or replanning.

  • Final coverage: We conduct additional experiments on the largest CVRP setting (4/150), involving four policies and a total budget of 150 units. We consider two warmup budgets, with 3 and 5 units per policy, respectively. As shown in the table below, TSFT consistently outperforms one-shot allocation, achieving improvements of 9.0% and 5.2%, respectively. Notably, the performance gap is larger under a smaller warmup budget, highlighting the importance of online replanning when initial observations are limited.

    CVRP 4/150W = 3 * NW = 5 * N
    One-Shot16.1%21.2%
    TSFT25.1%26.4%
  • Prediction accuracy: Beyond final coverage, we observe that the average prediction MSE decreases from 5.178 (one-shot model) to 1.936 (TSFT's final-round model) as more observations become available. Fig. 2 further shows that the average recall of coverage-set predictions improves with additional observations, providing further empirical evidence of the effectiveness of online model re-estimation.

In summary, while one-shot allocation remains competitive by leveraging performance forecasting and exact optimization, its effectiveness depends heavily on the accuracy of the initial models, particularly under limited warmup budgets (e.g., W = 3 × N). In contrast, TSFT is more robust to initial modeling errors, as it continuously refines predictions using observed fine-tuning outcomes and adapts subsequent allocations accordingly. These results provide direct empirical evidence of the benefits of online re-estimation and replanning.

AuthorsOpenReview ↗

Response to Reviewer 6Jpo (2/4)

W3 & Q2 & Q3: Performance advantages are uneven and incompletely replicated.

Our experiments demonstrate TSFT's improvements over strong baselines across a broad range of domains. For example, MOORE is a state-of-the-art baseline on Meta-World, yet TSFT achieves a further 14% improvement in the challenging 10/50 setting. To further substantiate TSFT's effectiveness and the robustness of our findings, we provide detailed raw results and additional experiments below.

  • Q2: Could you report per-benchmark pass@1, raw Ant returns for the compared methods, and ranking stability across reasonable coverage thresholds?

    • Please see the following table for detailed per-benchmark pass@1 results. Starting from a pretrained LLM with an average pass@1 of 26.89%, TSFT achieves 39.31% across all benchmarks, outperforming all non-oracle baselines and approaching oracle-level performance.

      LLM-FT 4/100OracleOracle-WarmupUniformAdaptiveLinUCBTSFT
      DAPO-17K32.00%32.00%23.00%21.60%18.00%34.00%
      MATH-50076.00%76.00%73.00%75.00%77.00%76.00%
      GSM8K84.00%84.00%86.00%84.80%84.00%86.00%
      CodeContests+36.23%36.23%38.80%36.01%36.23%34.13%
      AIME 202410.00%10.00%3.33%6.00%6.67%6.67%
      AIME 202513.33%13.33%10.00%10.00%10.00%10.00%
      Minerva Math20.00%20.00%17.00%18.40%17.00%20.00%
      MBPP61.00%60.00%61.00%62.20%56.00%60.00%
      BigCodeBench27.00%27.00%24.00%23.80%19.00%27.00%
      Average ↑\uparrow39.95%39.84%37.35%37.53%35.99%39.31%
    • In the original paper, we evaluated Ant using only three random seeds. We have since identified a library incompatibility issue affecting the Ant experiments. To address this issue, we are rerunning the experiments with five random seeds and reporting the average raw performance (i.e., mean episodic returns) in the following tables. Note that CRL's optimization objective is global coverage rather than returns: higher coverage does not necessarily translate into higher returns, although the two metrics are correlated. We will incorporate the updated results into the revised paper once all experiments are completed.

      Ant3/503/1004/100
      Oracle1688.72130.82391.3
      Oracle-Warmup2019.62296.82386.5
      Pretrained1145.41145.41145.4
      MTL1661.22132.12132.1
      Random1897.81992.42016.0
      Uniform1897.92009.52086.8
      Adaptive1792.02072.82080.6
      LinUCB1449.51705.41657.0
      TSFT1920.32123.22158.2
    • The following table presents the (coverage) ranking stability of the compared methods. We evaluate all baselines on Ant across diverse settings, varying the number of policies, training budgets, and coverage thresholds. The results show that TSFT consistently ranks among the Top2 methods in most settings, demonstrating its effectiveness and robustness across different configurations.

      ThresholdN/KTSFTRandomUniformAdaptiveLinUCB
      1,0003/5015234
      1,0003/10014523
      1,0004/1001=451=3
      1,5003/5015324
      1,5003/10025431
      1,5004/10025413
      2,0003/5014532
      2,0003/10014532
      2,0004/10014523
      2,5003/5015342
      2,5003/10015423
      2,5004/10015432
      3,0003/5013425
      3,0003/10015423
      3,0004/10015432
      3,5003/5021345
      3,5003/10015342
      3,5004/10025413
      Average ↓\downarrow1.224.393.942.502.89
AuthorsOpenReview ↗

Response to Reviewer 6Jpo (3/4)

  • Q3: More training seeds. How are Random and Adaptive results averaged?

    Thanks for your comment. Regarding training seeds: The LLM experiments require 16 NVIDIA H200 GPUs, which are unfortunately unavailable to us during the limited rebuttal period. Nevertheless, we agree that additional runs would further strengthen the empirical evidence. To address this concern, we conduct additional runs on CVRP and Meta-World, where Seed1 corresponds to the original results and Seed2 represents the newly obtained results. As shown in the table below, the additional results are consistent with the empirical findings reported in the original paper, providing further support for the effectiveness and robustness of TSFT. Meanwhile, we are continuing to conduct additional experiments (e.g., with more policies and seeds) and plan to run the LLM experiments with additional seeds once sufficient computational resources become available. We will incorporate the complete results into the final version of the paper.

    Regarding scheduling randomness, we evaluate Random and Adaptive over five allocation seeds in the main LLM and Meta-World experiments. For each run, we compute coverage using the task-specific thresholds and then report the equally weighted arithmetic mean across runs. TSFT does not sample allocations: given the same observations and fixed solver settings, its scheduling decisions are deterministic. This eliminates allocation-sampling variability, while training-seed variability remains relevant.

CVRP 3/100 (Seed1)CVRP 3/100 (Seed2)CVRP 3/150 (Seed1)CVRP 3/150 (Seed2)Meta-World 10/50 (Seed1)Meta-World 10/50 (Seed2)
Oracle26.7%26.0%29.3%29.0%86.0%84.0%
Oracle-Warmup21.4%21.5%24.5%24.7%84.0%80.0%
Pretrained6.8%6.7%6.8%6.7%58.0%62.0%
MTL9.5%9.3%11.0%10.9%60.0%72.0%
Random10.1%11.3%11.0%11.1%71.2%65.2%
Uniform7.9%6.6%15.4%12.7%70.0%68.0%
Adaptive10.4%12.6%16.1%18.5%71.2%70.4%
LinUCB12.6%14.0%20.5%20.5%68.0%74.0%
TSFT18.7%20.4%23.0%21.6%74.0%76.0%

Q1: Are scheduling-validation examples strictly separated from final test examples? How is adaptive model-selection bias controlled for the LLM benchmarks described as OOD?

Yes, we seperate validation samples used for online budget allocation and test samples used for final evaluation. We will clarify this in the revision.

The CRL setting aims to maximize task coverage within a predefined, bounded context space, reflecting practical applications where a practitioner seeks strong performance across a specified collection of related tasks. Accordingly, validation feedback from all tasks is available to guide budget allocation, while their test data remain held out. Moreover, ID and OOD refer to benchmark distributions relative to the corpora used to fine-tune the specialized policies, rather than whether a benchmark is excluded from the allocation procedure. Both groups provide validation feedback for allocation, but neither provides test feedback. We acknowledge that the original terminology could suggest that the OOD benchmarks are entirely unseen during allocation, and we have revised the paper to make this distinction explicit.

Reviewer_YejuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a novel framework TSFT to enable efficient task specialization in CRL, that formulates CRL as a model-based resource allocation problem based on two core assumptions: (1) an overall fixed budget, and (2) predefined global convergence.

Critical strengths
  • Presents an innovative and well-positioned problem formulation for CRL, shifting focus from "where" to "how much," which offers strong potential for real-world applications.
  • Mathematical formulations are cleanly presented, well-defined, and self-contained.
Critical weaknesses
  • Lacks comparisons with other multi-policy CRL approaches as competitive baselines.
  • Sensitivity analysis for the performance threshold is limited. In Figure 8, performance vary with different thresholds, making it unclear how to determine appropriate values across various baselines and evaluation datasets.
  • Lacks a computational cost comparison between TSFT and baselines.
Questions for authors

Please check the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Yeju (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for providing constructive feedback. Below, we respond to each of your comments in detail, where W denotes Weakness (W1–W3).

W1: Comparisons with other multi-policy CRL approaches.

Thanks for your comment. We compare TSFT with SD-MBTL [1], a recent multi-policy CRL method. Specifically, we conduct experiments on CartPole and Ant using five random seeds. For Ant, we identified a library incompatibility issue affecting our original experiments and have therefore rerun all Ant experiments. Detailed results are provided in our response to Reviewer 6Jpo W3 & Q2 & Q3.

The following table summarizes the comparison with SD-MBTL, reporting coverage / average episodic returns achieved by each method. Under the same total budget, we evaluate SD-MBTL with N=1N=1 and N=3N=3 policies. Note that the optimization objective of CRL is global coverage rather than average returns: higher coverage does not necessarily translate into higher returns, although the two metrics are correlated. SD-MBTL performs well on CartPole but struggles on the more challenging Ant benchmark. One possible explanation is that existing multi-policy CRL approaches are generally designed for settings with relatively large or unconstrained training budgets, whereas TSFT explicitly optimizes policy specialization under a limited budget, thereby achieving better sample efficiency. We will incorporate the updated results into the revised paper once all experiments are completed.

[1] Structure Detection for Contextual Reinforcement Learning. In AAAI 2026

CartPole 3/100CartPole 4/100Ant 3/100Ant 4/100
SD-MBTL (N=1N=1)79.4% / 434.979.4% / 434.90.5% / 9530.5% / 953
SD-MBTL (N=3N=3)83.4% / 443.883.4% / 443.80.3% / 12720.3% / 1272
TSFT84.0% / 446.287.6% / 452.121.0% / 212319.9% / 2132

W2: Sensitivity analysis for the performance threshold is limited.

In Fig. 7 (now), we conduct a sensitivity analysis on CVRP with ϵ∈[1.15%,1.40%]\epsilon \in [1.15\%, 1.40\%], since coverage is inherently threshold-dependent. We do not consider smaller thresholds because the coverage of all methods approaches zero under the current training budget. Achieving meaningful coverage in such challenging settings would require substantially larger training budgets. We also note that the performance metric for CVRP is the optimality gap, for which even a reduction of 0.1% can be significant in both research and practical applications. Moreover, TSFT is threshold-aware: its allocation decisions are guided by predicted coverage, allowing it to adapt its allocation strategy to different thresholds while maximizing global coverage.

To address this concern, we conduct an additional sensitivity analysis on CartPole 3/50 (see the table below), considering a broader range of performance thresholds from 400 to 500. We also examine the stability of coverage rankings across thresholds ranging from 1000 to 3500 on Ant (see our response to Reviewer 6Jpo W3 & Q2 & Q3). Together, these results demonstrate that TSFT consistently achieves competitive and stable coverage across a wide range of thresholds, demonstrating its ability to adapt allocation strategies to different thresholds. We will clarify the rationale for threshold selection and incorporate these additional results into the revised paper.

CartPole 3/50ϵ=400\epsilon=400ϵ=450\epsilon=450ϵ=475\epsilon=475ϵ=490\epsilon=490ϵ=500\epsilon=500
Oracle90.00%89.82%89.58%89.28%89.16%
Oracle-Warmup88.66%88.10%87.44%87.04%86.78%
Pretrained77.20%73.62%70.50%68.22%66.58%
MTL81.20%78.86%76.36%74.70%73.76%
Random80.56%78.66%77.02%75.96%75.44%
Uniform77.02%75.60%74.34%73.44%72.82%
Adaptive79.74%78.34%76.58%75.86%75.06%
LinUCB85.84%83.52%81.00%79.28%77.66%
TSFT86.04%84.72%82.46%81.68%82.20%
AuthorsOpenReview ↗

Response to Reviewer Yeju (2/2)

W3: Lacks a computational cost comparison between TSFT and baselines.

TSFT supports online evaluation without saving intermediate checkpoints. Moreover, all multi-policy methods use the same inference-selection procedure. Therefore, compared with baselines using the same number of policies, TSFT incurs no additional model storage or inference-selection cost. Its additional computational cost primarily comes from ILP optimization, performance-model fitting, and repeated policy evaluation:

  • ILP optimization: Table 10 reports the computational cost per planning step, which is generally below one second.
  • Performance-model fitting: As discussed in Section 4.1, fitting can be parallelized across thousands of tasks and completed within seconds.
  • Repeated Policy evaluation (Validation): Repeated evaluation provides the observations needed for model fitting and constitutes the main overhead. In our experiments, the total evaluation time is on the order of hours, substantially less than the days required for policy training. To reduce this overhead, we also investigate a cheaper evaluation test in Table 11. Using only one-tenth of the evaluation samples reduces evaluation costs by approximately 10$\times$ while maintaining comparable final coverage (e.g., 22.3% vs. 23.0% on CVRP 3/150).

The following table summarizes the computational costs. Note that the reported times are cumulative across all planning steps in the online process.

CVRP 3/150Policy TrainingValidationModel FittingILP SolvingFinal Evaluation
Random172h00010m
Adaptive172h5.0h0010m
TSFT172h6.3h20.2s1.19s10m
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer Yeju,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

As the discussion period will close soon, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.

We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!

Best regards,
Authors

AuthorsOpenReview ↗

Reminder: Author Rebuttal Acknowledgement

Dear Reviewer Yeju,

As the discussion period will close in less than 9 hours, if you still have some questions, let us discuss in the openreview system.

Best regards,
Authors

Reviewer_AfGaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies budget allocation for task specialization in contextual reinforcement learning (CRL). Rather than training separate policies from scratch, the authors propose pretraining a shared policy and subsequently fine-tuning multiple specialized policies for different regions of the context space. The proposed Task Specialization Fine-Tuning (TSFT) framework models the performance of each specialized policy across tasks using parametric learning curves. These predictions are used to formulate a maximum-coverage problem, solved through integer linear programming (ILP). Experiments on combinatorial optimization, continuous control, and LLM fine-tuning show that TSFT generally improves task coverage over uniform, adaptive, and bandit-based allocation strategies, often approaching oracle performance.

Critical strengths
  1. The budget-allocation formulation is relevant and provides a useful extension to existing contextual RL approaches.

  2. Strong experimental results across combinatorial optimization, continuous control, and LLM fine-tuning.

  3. The paper includes extensive analyses of surrogate models, computational efficiency, and sensitivity to hyperparameters.

Critical weaknesses
  1. The related work could better discuss active multi-task fine-tuning and task scheduling [1, 2]. These address related resource-allocation problems, although TSFT's multi-policy coverage objective is distinct.

  2. The theoretical bound relies on assumptions about surrogate-model accuracy and planning optimality, limiting its practical guarantees.

  3. The LLM evaluation protocol needs clarification. Checkpoints are repeatedly evaluated on the benchmark tasks used for allocation. It is unclear whether final evaluation uses independent samples.

  4. The combinatorial optimization and LLM results use single seeds. Additional runs would strengthen the empirical conclusions.

[1] Bagatella et al. (2025). Active Fine-Tuning of Multi-Task Policies. ICML 2025.

[2] Cho et al. (2024). Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling. ICML 2024.

Questions for authors
  1. How does TSFT relate to active fine-tuning and task scheduling methods [1, 2]?

  2. Are the evaluation samples used for online budget allocation separate from those used for final LLM evaluation?

  3. Can the authors provide additional seeds for the combinatorial optimization and LLM experiments?

[1] Bagatella et al. (2025). Active Fine-Tuning of Multi-Task Policies. ICML 2025.

[2] Cho et al. (2024). Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling. ICML 2024.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AfGa (1/2)

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper and for providing constructive feedback. Below, we respond to each of your comments in detail, where W denotes Weakness (W1–W4) and Q denotes Question (Q1–Q3).

W1 & Q1: Relevance to active fine-tuning and task scheduling methods.

Thank you for highlighting these relevant works! We have added a discussion to the related work section of the revised paper.

  • Conceptually, TSFT differs from AMF [1] and SMT [2] in following perspectives: 1) Policy structure: Both AMF and SMT optimize a single shared multi-task policy, whereas TSFT optimizes multiple specialized policies from a pretrained (multi-task) policy. 2) Allocation criterion: AMF selects tasks for expert demonstrations based on expected information gain, while SMT schedules experience collection by prioritizing difficult tasks. TSFT allocates fine-tuning budget across specialized policies based on predicted future coverage and the remaining budget. 3) Optimization objective: AMF and SMT aim to improve a shared policy’s performance across tasks. TSFT maximizes the number of tasks covered by at least one specialized policy, explicitly accounting for overlapping coverage. Together, these conceptual differences lead to our distinct problem formulation (i.e., a variant of the MCP).

  • Experimentally, AMF focuses on demonstration-based fine-tuning for robotic control, while SMT studies multi-task robotic manipulation on Meta-World. TSFT is evaluated across combinatorial optimization, continuous control, and LLM fine-tuning, demonstrating its applicability across diverse domains.

W2: Practical guarantees of theoretical bound.

Our bound is conditional and does not establish an unconditional near-optimality guarantee for the online procedure. Our original purpose is to decompose the gap to the oracle into contributions from warmup allocation, surrogate-model error, and algorithmic planning error. This decomposition helps readers better understand TSFT by clarifying the sources of suboptimality, identifying the conditions under which it can approach oracle performance, and suggesting directions for further improvement. We have revised the theoretical discussion in Appendix C to explicitly state these limitations.

Moreover, we further provide empirical evidence to quantify the errors of our assumptions under the TSFT online procedure.

  • Surrogate Coverage Error (Assumption 1): We measure how much predicted coverage differs from actual coverage across all feasible post-warmup allocations. In specific, we evaluate CVRP 3/150 and compute the worst-case coverage-prediction error, max⁡K∈ΩW∣G(K)−G^(K)∣\max_{\mathcal{K}\in\Omega_W} |G(\mathcal{K})-\widehat{G}(\mathcal{K})| (Eq. (32)), together with the mean absolute error over the same feasible post-warmup allocation set. Errors are normalized by the size of the context space and expressed in coverage percentage points. We report results every 10 budget units, refitting the surrogate models using only observations available at each step. As shown in the below table, the maximum error decreases from 16.00% to 3.90%, while the mean error decreases from 4.41% to 0.46%, despite intermediate fluctuations. These results provide empirical evidence of improved surrogate accuracy during online execution, with small absolute coverage errors in the later stages.
  • Final-Allocation Error (Assumption 2): We measure how much predicted coverage the final executed allocation loses relative to the best allocation under a fixed surrogate. In specific, on CVRP 3/150, we compute max⁡K∈ΩWG^(K)−G^(KTSFT)\max_{\mathcal {K}\in\Omega_W} \widehat{G} (\mathcal{K}) - \widehat{G}(\mathcal{K}_{\mathrm{TSFT}}) using the last surrogate model employed for planning. The best post-warmup allocation achieves 22.50% predicted coverage, while the executed allocation achieves 19.80%, yielding an error of 2.70%. This quantifies the final-allocation error of the implemented online procedure and provides empirical evidence of limited absolute error.
Consumed budgetMaximum absolute errorMean absolute error
3016.00%4.41%
409.80%1.93%
5013.20%3.19%
6010.00%1.98%
709.20%1.60%
807.40%1.19%
905.90%0.95%
1005.50%0.75%
1105.20%0.58%
1204.70%0.50%
1304.70%0.48%
1403.90%0.46%
AuthorsOpenReview ↗

Response to Reviewer AfGa (2/2)

W3 & Q2: It is unclear whether LLM final evaluation uses independent samples.

Yes, we seperate validation samples used for online budget allocation and test samples used for final LLM evaluation. We will clarify this in the revision.

W4 & Q3: Additional runs on combinatorial optimization and LLM experiments.

Thanks for your comment. The LLM experiments require 16 NVIDIA H200 GPUs, which are unfortunately unavailable to us during the limited rebuttal period. Nevertheless, we agree that additional runs would further strengthen the empirical evidence. To address this concern, we conduct additional runs on CVRP and Meta-World, where Seed1 corresponds to the original results and Seed2 represents the newly obtained results. As shown in the table below, the additional results are consistent with the empirical findings reported in the original paper, providing further support for the effectiveness and robustness of TSFT.

We are continuing to conduct additional experiments (e.g., with more policies) and plan to run the LLM experiments with additional seeds once sufficient computational resources become available. We will incorporate the complete results into the final version of the paper.

CVRP 3/100 (Seed1)CVRP 3/100 (Seed2)CVRP 3/150 (Seed1)CVRP 3/150 (Seed2)Meta-World 10/50 (Seed1)Meta-World 10/50 (Seed2)
Oracle26.7%26.0%29.3%29.0%86.0%84.0%
Oracle-Warmup21.4%21.5%24.5%24.7%84.0%80.0%
Pretrained6.8%6.7%6.8%6.7%58.0%62.0%
MTL9.5%9.3%11.0%10.9%60.0%72.0%
Random10.1%11.3%11.0%11.1%71.2%65.2%
Uniform7.9%6.6%15.4%12.7%70.0%68.0%
Adaptive10.4%12.6%16.1%18.5%71.2%70.4%
LinUCB12.6%14.0%20.5%20.5%68.0%74.0%
TSFT18.7%20.4%23.0%21.6%74.0%76.0%

[1] Bagatella et al. (2025). Active Fine-Tuning of Multi-Task Policies. ICML 2025.

[2] Cho et al. (2024). Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling. ICML 2024.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer AfGa,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

As the discussion period will close soon, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.

We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!

Best regards,
Authors

AuthorsOpenReview ↗

Reminder: Author Rebuttal Acknowledgement

Dear Reviewer AfGa,

As the discussion period will close in less than 9 hours, if you still have some questions, let us discuss in the openreview system.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.