acceptodds
Under review as a conference paper at ICLR 2027

Task Vectors in the Loop: Online Geometric Coordination for Multi-Task RL for LLM

Abstract

Reinforcement-learning (RL) post-training has advanced the reasoning and problem-solving capabilities of large language models, but extending these gains across heterogeneous tasks requires balancing cross-task transfer with task specialization. Mixed RL and post-hoc model merging approach this challenge from opposite directions: joint training allows transfer but risks interference, whereas independent training preserves specialization but leaves task-specific updates to be reconciled afterward. Rather than treating specialization and joint learning as competing choices, we use brief specialization to reveal task structure that guides joint optimization. We introduce Task-Vector-Guided Reinforcement Learning (TVG-RL), which turns short task-specific warm-ups into a geometric prior for multi-task RL. Starting from a common initialization, TVG-RL extracts task vectors and decomposes their structure into shared and task-private subspaces. It then trains a single policy from the original base, using this decomposition to project and rescale subsequent RL gradients. Shared and private components receive distinct treatment, while a complementary component permits updates outside these task-conditioned shared–private subspaces. The resulting method requires neither fully trained experts nor a final model-merging stage. Experiments across eight heterogeneous benchmarks show improvements over direct mixed-data RL on most tasks, alongside gains over the evaluated post-hoc model-merging baselines. These findings support a role for task specialization beyond producing standalone experts: the geometry it reveals can help a shared policy learn multiple capabilities together.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL for LLM, Model Merging, Task Vector

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_a1euOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper addresses cross-task transfer and interference in multi-task RL post-training. The authors propose Task-Vector-Guided Reinforcement Learning (TVG-RL). Starting from a common initialization, brief task-specific warm-ups produce task vectors. Their geometry is decomposed into shared, task-private, and complementary components, which are used to project and rescale subsequent multi-task RL gradients. The method trains a single shared policy without requiring fully trained experts or a final merging stage. In the five-task Qwen2.5-7B setting, TVG-RL achieves 79.00% KK accuracy, improving over mixRL’s 44.75% by 34.25 percentage points. Its HH reward reaches 16.21, compared with 5.35 for the strongest merging baseline. In the four-task Qwen3-1.7B setting, TVG-RL achieves the highest mean score among evaluated generalist methods on each of the five reported benchmarks, including held-out GPQA-D. The reported online overhead is approximately 3.1%–3.8%, excluding warm-up training and basis construction.

Critical strengths

The paper makes a clear and differentiated contribution by turning task vectors from post-hoc merging tools into online optimization priors for multi-task RL. Brief specialization reveals task structure that guides shared-policy learning, avoiding fully trained experts and a final merge.

Why this matters for the overall assessment: This change in training paradigm has independent research value and would be the primary basis for acceptance if the controlled comparisons and ablations hold.

Critical weaknesses
    • The key design choices in the shared/private decomposition are not sufficiently validated. The subspace definition, selected tensors, low-dimensional representation, and scaling coefficients may determine the outcome, yet their stability and tuning cost are not established.

    • Why this matters for the overall assessment: If the method works only under carefully selected decomposition settings, the contribution would look more like task-specific engineering, substantially weakening the case for acceptance.

    • The efficiency and scalability claims are incomplete. Warm-up and basis construction are excluded from the reported online overhead, and the evaluation covers only two model scales and a limit1. ed task set.

    • Why this matters for the overall assessment: The paper claims efficient online coordination relative to expert training and merging, but end-to-end cost and scaling with the number of tasks are not established; this directly affects its practical significance and acceptance strength.

Questions for authors
  1. Including task-specific warm-ups, vector decomposition, and basis construction, what are the total FLOPs, GPU time, and memory requirements relative to mixRL and to independently training and merging experts?

  2. Could the authors provide sensitivity analyses for the shared/private decomposition, including the selected layers, subspace rank, projection coefficients, and task sampling weights?

Reviewer_highOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TVG-RL for MTRL post-training of LLMs. TVG-RL first runs short independent task-specific GRPO warmups from a common base, and extracts one shared direction and one task-private direction per task to guide the joint training process. The evaluation results demonstrate that TVG-RL outperforms mixRL and post-hoc model merging methods on most benchmarks with negligible online overhead.

Critical strengths
  • I like the overall intuition of this paper. TVG-RL uses task-vector geometry, normally a post-hoc merging tool, as an online optimization prior for joint RL-based training. I think it is a natural and potentially useful idea.

  • All baselines are consistently re-evaluated in a single pipeline, and a zero-shot transfer benchmark is included.

Critical weaknesses
  • There are no ablation study, so it is unknown which part of the method causes the gains. It is missing hyperparameter sweeps, component ablations (e.g.m shared-only, private-only, w/o norm restoration), and a random-subspace control.

  • Key baselines are missing, and several claims are overstated. First, there is no comparison to multi-task optimization literature (PCGrad, GradVac, MGDA, NashMTL, etc.) although the method is closely aligned to these. The comparison to CARE-RL, which is described as closely related, is also missing.

Questions for authors
  • Can you provide ablations on α_p, α_⊥, warm-up length, shared-only vs. private-only, norm restoration, and hard projection? Most importantly, how does TVG-RL perform with random orthonormal directions in place of the task-vector bases?

  • Why is mixRL omitted in the 1.7B setting? Can you add it, along with PCGrad or GradVac-style gradient surgery?

  • Can you report results over multiple training seeds, given the single-seed protocol and the 100-problem KK test set?

Reviewer_rb5HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multi-task RL for LLMs, where learning one capability can interfere with another.

TVG-RL uses short single-task warm-ups to estimate shared and task-specific directions. It then restarts from the base model and uses these directions to reweight each task's gradients during joint training.

The strongest result is on Qwen2.5-7B: KK accuracy rises from 44.75% with mixRL to 79.00%. HH, Science, and IF also improve. The 1.7B comparison covers merging baselines but omits mixRL.

Critical strengths
  1. I like the use of short specialist runs to guide joint training. Table 1 shows a large KK improvement, and HH, Science, and IF also improve. The paper also tests a second backbone and held-out GPQA-D.

  2. Appendix C makes the distributed projection easy to follow. Each rank contributes to a few scalar coefficients, which avoids gathering the full gradients. The measured online overhead is 3.1-3.8%, excluding warm-ups and basis construction. The paper also reports raw metrics and results without HH.

Critical weaknesses
  1. My main concern is whether the proposed geometry is actually responsible for the gains. The method changes the component weights and restores gradient norms at the same time. Without random-basis controls or component ablations, I cannot tell how much the shared-private construction contributes. MGS and CARE-RL are discussed in the paper, but neither is compared experimentally.

  2. Appendix D.2 restores each transformed gradient to its original norm, but Eq. (9) and Algorithm 1 leave this out. Proposition 1 concerns the linear transformation before this rescaling. I would like to see the actual update written consistently throughout the paper, with a clear statement of which part the proposition covers.

  3. Table 2 leaves out mixRL on Qwen3-1.7B, which is an important comparison for a method intended to improve joint training. The GPQA-D leads over the best merging baseline are also small, 0.51 and 0.25 percentage points. I find the in-domain gains more convincing than the held-out results.

Questions for authors
  1. Does the learned basis beat a matched random basis when the weights and norm restoration stay the same? Please also compare with MGS or CARE-RL.

  2. Did the main runs use the norm-restoration step in Appendix D.2? Please make Algorithm 1 match the update used in training and show what happens without that step.

  3. How does mixRL perform on Qwen3-1.7B? Please also report the warm-up and basis-construction cost so the total training cost can be compared.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Rebuttal

We sincerely thank the AC and all reviewers for their time and constructive feedback. We would like to summary the rebuttal.

We are encouraged that Reviewer a1eu recognizes the research value of introducing task vectors as priors for multi-task RL post-training and the distributed projection algorithm design. We also appreciate that Reviewer rb5H and Reviewer high find the underlying idea natural and promising, and recognize the value of using short task-specific warm-ups to guide joint training.

During the rebuttal, we have made the following additions and clarifications:

  • Added controlled experiments with random task-vector bases and different component weights, together with comparisons against PCGrad and CARE-RL, to further validate the contribution of task-derived geometric structure.

  • Added mixRL result on Qwen3-1.7B, providing further evidence of the effectiveness and consistency of TVG-RL.

  • Clarified the implementation and role of norm restoration, refined the theoretical analysis to clarify the scope of our propositions, and provided a comprehensive computational cost analysis.

We sincerely appreciate the reviewers' thoughtful comments and the opportunity to strengthen our work through this process. We will incorporate these additional results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.