acceptodds
Under review as a conference paper at ICLR 2027

No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation

19 views

Abstract

Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced by on-policy distillation (OPD) remains largely unexplored. OPD trains a stu- dent using teacher feedback on student-generated trajectories, yielding parameter updates that differ from those produced by the teacher model, usually by rein- forcement learning (RL). We therefore ask whether OPD task vectors can comple- ment their RL teacher updates and compose effectively across tasks. Across five domains and two model architectures, we find evidence for both forms of compos- ability. Within a task, merging OPD and RL task vectors can outperform both constituent models, even when the OPD student is weaker than its RL teacher. Across tasks, OPD task-vector compositions achieve higher average scores than corresponding RL compositions in seven of eight backbone–merging-rule com- parisons. Parameter-space analyses reveal substantial non-collinearity between OPD and RL updates. Experiment in CODE domain on SMOLLM3-3B shows that the combined direction outperforms either constituent direction at the tested global update norm, supporting directional complementarity in this configuration. Across tasks, OPD updates also show lower overlap among the top-10% feed- forward channels ranked by update energy. Together, these results show that weaker standalone performance does not imply weaker task-vector composabil- ity. OPD task vectors can complement stronger RL teacher updates and combine effectively across tasks, highlighting composability as a distinct property for un- derstanding and evaluating post-training updates.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Task vector, On-Policy Distillation, Post training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mRGVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates the composability of task vectors derived from On-Policy Distillation (OPD). Experiments show that merging OPD task vectors with RL teacher vectors can improve single-task performance, and that cross-task merging with OPD vectors generally outperforms merging RL task vectors. The authors validate these findings across two model architectures, five task domains, and multiple merging methods, and further explore potential explanations through parameter-space analyses.

Critical strengths
  1. Novel research perspective. The paper investigates the composability of OPD task vectors, highlighting the distinction between standalone model performance and task-vector composability. This provides a new perspective on understanding OPD.

  2. Reasonably convincing experimental results. Across two model architectures and multiple task domains, cross-task merging with OPD outperforms RL merging in 7 out of 8 comparisons. Moreover, some same-task merging configurations even surpass the RL teacher models.

  3. In-depth parameter-space analysis. The authors analyze the differences between OPD and RL in terms of update norms, sparsity, directional similarity, and channel overlap. They also introduce norm-matching experiments to investigate the directional complementarity of task vectors.

Critical weaknesses
  1. Confounding factors in the conclusions about cross-task composability. The norms of OPD task vectors are substantially smaller than those of RL task vectors. For example, on SmolLM3, the norms of the Math and Code OPD vectors are only 59.7% and 68.7% of their RL counterparts, respectively. However, Table 2 lacks a norm-matched RL merging baseline. Therefore, the observed advantage of OPD may arise from smaller parameter perturbations rather than inherently better task composability.
  2. Insufficient evidence for same-task complementarity. The most significant improvement in Table 1 comes from Qwen Code, yet the norm-matching experiment in Section 4.2 is conducted only on SmolLM3 Code. Furthermore, the experiments do not compare against an optimally scaled RL-only direction. Consequently, the current evidence is insufficient to establish that the main performance gains indeed originate from the complementary directions provided by OPD.
  3. Questionable hyperparameter settings for the merging baselines. Raw Sum is essentially Task Arithmetic (TA) with λ = 1. However, Table 2 shows that TA performs substantially worse than Raw Sum across all configurations. Given the authors' claim that the best merging hyperparameters were selected, why was λ = 1, which would yield a better result, not included in the TA configurations? This raises concerns about the hyperparameter selection and fairness of the baseline comparisons.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

A Gentle Follow-up on Further Feedback

Dear Reviewer,

Thank you very much for your time and valuable feedback.

As the discussion period is approaching its end, with fewer than three days remaining, we would kindly like to ask whether you have any further questions or concerns regarding our responses. We would be more than happy to provide additional clarification if needed.

We sincerely appreciate your time, consideration, and constructive feedback.

Best regards,
The Authors

AuthorsOpenReview ↗

A Gentle Follow-up Before the Discussion Period Ends

Dear Reviewer,

Thank you again for your time and thoughtful feedback.

As the discussion period is drawing to a close, we would like to respectfully follow up on our previous message. We would be very grateful if you could let us know whether our responses have adequately addressed your concerns, or if there are any remaining questions that we could further clarify.

We sincerely appreciate your time and consideration.

Best regards,
The Authors

Reviewer_yR6qOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how well task vectors of models trained with on-policy distillation (OPD) merge with other task vectors. Each OPD student starts from a shared initial model and is trained with the RL expert for the corresponding task as its teacher, and a task vector is defined as the parameter difference between the trained model and the initial model.

The authors compare three settings: merging an OPD student with its RL teacher on the same task, merging OPD or RL experts across multiple tasks, and jointly combining both training methods across multiple tasks. Experiments cover math, code, and instruction following on SmolLM3-3B, with science and tool use additionally evaluated on Qwen3-4B, using TA, Raw Sum, TIES, and TSV-M.

On the Qwen code task, the OPD student and RL teacher score 37.18% and 37.51%, respectively, and merging the two with TIES yields 41.24%, which is 3.73%p higher than the teacher (Table 1). In cross-task merging, OPD achieves a higher average score than RL in 7 out of 8 comparisons (Table 2). Furthermore, averaging the OPD and RL updates per task and then summing them across tasks yields a higher domain-average score than single-source Raw Sum for both model families (Table 3).

In the equal-norm control experiment on SmolLM3 code, the combined direction achieves higher average performance than either single direction at the tested global update norm. The main contribution is demonstrating, through these results, the need to evaluate a model’s standalone performance and its ability to contribute to merging as separate properties.

Critical strengths
  1. The paper provides useful empirical evidence for distinguishing standalone performance from merging contribution. On the Qwen code task, merging the OPD student, which has lower standalone performance, with the RL teacher improves over the teacher by 3.73%p with TIES and by 3.50%p even with fixed-coefficient Raw Sum (Table 1). This suggests that selecting expert models based on standalone performance alone may miss candidates that are valuable for merging.

  2. The paper combines comparisons across multiple settings with a control experiment that examines directional complementarity. Results are compared across two model families and multiple tasks and merging methods. In particular, on SmolLM3 code, at the same global update norm, the combined direction achieves an average score 1.77%p higher than the RL-only direction and 2.07%p higher than the OPD-only direction (Table 5). This provides evidence that the observed benefit in this setting is difficult to explain by update magnitude alone.

Critical weaknesses
  1. The procedure for selecting merging configurations is underspecified. Appendix B.4 states that the best-performing configuration among evaluated candidates is reported, but the candidate ranges, final TA and TIES settings, selection metric, use of a separate validation set, and search budget for OPD and RL are unclear. Notably, Raw Sum corresponds to TA with λ=1, yet it outperforms TA in Table 2, which raises the question of whether λ=1 was included in the TA search. These details are important for assessing the fairness and reproducibility of the comparisons.

  2. The uncertainty of the main performance differences is difficult to assess. Results are reported as means over three experimental repetitions, without per-repetition scores or confidence intervals, and it is unclear whether these repetitions involve independently trained checkpoints or repeated evaluations of the same checkpoint. In the seven comparisons in Table 2 where OPD performs better, the gains in average score range from 0.47 to 0.98 percentage points. Reporting uncertainty estimates would help assess the stability of these gains and clarify the statistical analyses mentioned in the reproducibility statement.

Questions for authors
  1. Merging configuration selection: Could you describe in more detail the candidate ranges, final hyperparameters, and selection metric for each merging method? I would also like to know whether a separate validation set was used, whether the same search budget was applied to OPD and RL, and whether λ=1 was included among the TA candidates.
  2. Repeated runs: Do the three runs correspond to independently trained checkpoints, or to repeated evaluations of the same checkpoint? Providing per-run results and confidence intervals for the main differences in Tables 1–3 would help assess the stability of the results.
  3. Additional control for update magnitude: In cross-task merging, does the OPD advantage persist if the norm of the RL updates is matched to that of OPD, or if the scale of each method is tuned using the same validation procedure? This comparison would clarify the respective roles of update magnitude and direction in the observed benefit.
3 comments
AuthorsOpenReview ↗

A Gentle Follow-up on Further Feedback

Dear Reviewer,

Thank you very much for your time and valuable feedback.

As the discussion period is approaching its end, with fewer than three days remaining, we would kindly like to ask whether you have any further questions or concerns regarding our responses. We would be more than happy to provide additional clarification if needed.

We sincerely appreciate your time, consideration, and constructive feedback.

Best regards,
The Authors

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.

I will maintain my current score.

Thank you very much for your continued positive assessment of our work and for your thoughtful feedback throughout the discussion. We sincerely appreciate your time, consideration, and constructive suggestions.

Reviewer_hpDtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the compositional abilities of on-policy distillation (OPD), within-task, across-task, and joint-task (averaging both within and across tasks). It shows 3 findings: within a task, composing OPD + teacher-RL can lead to better results than RL even though the OPD model is weaker. Across tasks, OPD-merged models can have better results compared to RL, showing better relative improvement compared to their baseline. Then, RL + OPD joint-task composition (so in-task + across-task) can show better performance compared to RL or OPD mixtures alone. The paper also proposes a comparison in weight space between RL and OPD.

Critical strengths

Strengths:

  1. Well written, contributions/findings are clear. Nice findings for the within-task setting, which show that composing with the teacher can give better results. The finding is also well-proportioned: the authors do not claim to always outperform, but rather that they sometimes do, even with a weaker OPD checkpoint.

  2. Interesting from the geometric perspective, showing that models from OPD have lower overlap between tasks.

Critical weaknesses

Weaknesses:

  1. Statistical significance is not always properly addressed. The authors repeat several times that they do not establish statistical significance (e.g. L291–292), so they are aware of this weakness, but still didn't fix it. Namely, in Table 1, 7/12 methods show (sometimes small) improvements, and no method outperforms consistently on the other benchmarks. In Table 2 the gains of OPD merging seem clearer, but they still lack baselines that take the merging scale into account: simple ones that just vary the scale to isolate the scale difference between RL and OPD. E.g. for Table 1, λ·RL with λ = ‖o + r‖ / ‖r‖ (which should be > 1); this exists in Table 5 but only for one task.

  2. The claims in the last section about the weight structure seem loosely connected to the rest of the paper: there is no correlation between the geometric metrics and the merge gains, and no clear conclusion nor experiments to study the impact of these. As is, Section 4 seems a bit off.

Questions for authors

Questions to authors:

  1. (Material) Could you add statistical significance results, at least from the 3 generations you are talking about? Several seeds of OPD training would also help: I understand the RL checkpoints do not come from you, but it would allow error bars/intervals. Comparing multiple OPD checkpoints and reporting their base results could convince me that the results are within or above noise, and make me improve my grade.

  2. (Clarification) How do you choose the hyperparameters for model merging? "Best-performing one among the evaluated candidates" is not clear. In particular, TA should always be higher than Raw Sum if λ = 1 is in the grid: how do you choose this λ grid?

  3. (Clarification) What exactly is an "MLP channel" here? Does it count all the MLP layers gate/up/down? Only some? Is it at the neuron level?

  4. (Suggestion) For cross-task composition, it would be interesting to have Pareto fronts with varying per-task weights. If the OPD Pareto front is above the RL one even if both endpoints are lower, this could further motivate better composition with OPD.

  5. (Suggestion) Could you re-evaluate the Open-MOPD results?

Minor: Table 5 is referenced as 4.2 in the main text.

3 comments
AuthorsOpenReview ↗

A Gentle Follow-up on Further Feedback

Dear Reviewer,

Thank you very much for your time and valuable feedback.

As the discussion period is approaching its end, with fewer than three days remaining, we would kindly like to ask whether you have any further questions or concerns regarding our responses. We would be more than happy to provide additional clarification if needed.

We sincerely appreciate your time, consideration, and constructive feedback.

Best regards,
The Authors

I would like to thank the authors for their detailed rebutal, and maintained my acceptance grade.

Thank you very much for your positive assessment and valuable feedback. We truly appreciate your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.