acceptodds
Under review as a conference paper at ICLR 2027

Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

15 views

Abstract

Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Reinforcement Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YBmMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOPED for integrating multiple visual capabilities into a single VLM. it first trains capabilty-specific experts with RL, extracts textual skills from successful & failed expert trajectories. Then, it uses these skills to condition the experts during multi-teacher on-policy distillation. The student itself does not use the skills at inference time. Eight multimodal benchmarks and Qwen3-VL-4B/8B are used in experiments. SCOPED improves over the corresponding backbone and mixed-capability RL. Additionally, the ablation studies and analysis verify the effectiveness of each component of SCOPED.

Critical strengths
  1. The experiments are fairly extensive. The paper evaluates across eight benchmarks and two model sales, including useful analysis of gradient interference, expert capability retention, trajectory consistency and visual attention.
  2. The paper also provides reasonable ablations separating the effects of expert specialization, on-policy distillation and skill augmentation.
Critical weaknesses

I see some practical value in the overall engineering pipeline, but I do not think the current methodological contribution is substantial enough for a full paper.

  1. The core pipeline largely combines two existing ideas, i.e., 1) training capability-specific experts and consolidating them through multi-teacher on-policy distillation, as well as 2) using trajectory-derived skills as privileged context for teacher supervision. Both ingredients are already discussed in the paper's related work. I do not see a substantially new distillation mechanism beyond this combination.

  2. Furthermore, the method is almost modality-agnostic. I am not sure what is specifically visual about this method. Frankly speaking, the same approache could be applied to almost any modality or task. Vision seems more like the application setting here, rather than something the method is actually designed around. The ablation studies also suggest this point. The most of the gain already comes from multi-expert OPD. On the 8B model, SCOPED without skill augmentation reaches 72.39 average score, while full SCOPED reaches 73.13.

Questions for authors

See above weaknesses.

3 comments

We thank the reviewer for recognizing the extensive experiments and component analyses. We further clarify the methodological contribution and scope of SCOPED.

Response to W1 – Methodological Contribution Beyond Existing Distillation Frameworks

SCOPED addresses heterogeneous capability integration by combining independent RL, capability-specific experience abstraction, and skill-conditioned multi-teacher distillation. Unlike standard MOPD, which directly transfers expert predictive distributions, SCOPED extracts reusable guidance from successful and failed expert trajectories and uses it as privileged information to condition the corresponding teachers along student-generated on-policy trajectories.

Compared with Skill-SD and related approaches, our focus is on integrating multiple independently RL-optimized experts with heterogeneous reward objectives into a unified student, rather than single-model self-distillation. This design enables capability-specific guidance to enhance expert supervision without requiring skills or expert routing during inference.

Response to W2 – Visual Specificity and Generality

SCOPED is not inherently restricted to visual tasks, which also highlights its potential generality. We choose multimodal visual understanding because its heterogeneous capabilities provide a representative setting for studying optimization interference and knowledge integration.

Specifically, perception, OCR and chart understanding, symbolic reasoning, and multi-image reasoning involve different input structures, reward functions, and reasoning requirements. These differences motivate capability-specific expert learning and experience abstraction, followed by unified capability integration.

Our experiments across these four capability groups demonstrate that SCOPED effectively preserves complementary expertise in a unified VLM. The framework also has the potential to generalize to language-only models and other modalities, which we consider a promising direction for future investigation.

Response to W2 – Incremental Benefits of Skill Augmentation

Skill conditioning provides capability-specific guidance exclusively to the teachers during distillation, while the unified student requires neither skills nor expert routing at inference time.

Under the same expert and distillation configurations, SCOPED improves upon MOPD by +0.74 and +0.45 on the 8B and 4B models, respectively. Furthermore, amplitude retention and directional alignment analyses show that SCOPED better preserves specialized expert behaviors. These results support the effectiveness of skill conditioning in improving heterogeneous knowledge transfer beyond standard multi-expert distillation.

Dear Reviewer,

Thank you again for your valuable feedback. We hope that our responses have addressed your concerns and would greatly appreciate any further thoughts you may have.

We would be happy to provide additional clarification if needed.

Thank you for your time and consideration.

Thank you for the response, but my main concerns remain unresolved. First, I have pointed this out as a weakness in my review, and your response merely repeats the same weakness. The response does not provide any new methodological depth. Also, the "response to W2" is completely self-contradictory. If this paper intends to build a general framework, the method should be designed and evaluated across modalities accordingly. If not, this paper should provide genuine vision-specific insights and designs. Claiming generality here as the response for lacking visual specificity is rather unconvincing. Last, simply repeating the numbers does not change the fact.

Thus, I do not think this paper is ready for acceptance at its current stage. Maintaining my current score without lowering it further is my bottom line;

Reviewer_y34rOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCOPED integrates several RL-trained capability experts into one VLM. First, separate Qwen3-VL experts are trained with RL for four capabilities: perception, OCR/chart, symbolic reasoning, and multi-image reasoning. Each expert then reflects on its own successful and failed rollouts and writes a natural-language “skill” for its capability. Finally, multi-teacher on-policy distillation (reverse KL) transfers the experts into a single student; the teacher sees its skill, but the student does not. Across eight benchmarks, SCOPED-8B improves the Qwen3-VL-8B backbone by 5.83 points on average and beats Mix-RL, SFT, parameter merging, and distillation without skills. Further analyses report gradient conflicts in Mix-RL and show that skills make expert trajectories more consistent and attention more grounded.

Critical strengths

Consistent integration gains: SCOPED-8B improves on the backbone on all eight benchmarks and beats Mix-RL by 3.82 points. The same ordering of methods holds at 4B, so the expert-then-distill recipe looks robust.

Near-oracle consolidation: The unified student roughly matches the best per-benchmark expert (73.13 vs 73.21 in Table 7), which shows that on-policy distillation can merge four separately trained experts with little loss.

Useful component ablations: Comparisons against parameter merging, distillation without skills, and success-only or failure-only skill sources isolate each design choice reasonably well.

Critical weaknesses

Train–test overlap inflates headline comparisons: RL data include PixMo-Count and We-Math 2.0 Pro, and these benchmarks are also used for evaluation. The skills also contain benchmark-format rules (e.g., “bare numeric value”, “option letter A–D”). This makes comparisons with closed-source and general open models, and the “beats Qwen3-VL-32B” claim, not like-for-like.

The skill contribution is small and unverified: Skills add only +0.74 average over distillation without skills (72.39 → 73.13), with several benchmarks unchanged, from single runs without variance. This is too small to support the paper’s main framing.

The reward-hacking motivation is never tested: The paper gives no evidence that the experts reward-hack or that skills remove such behavior. Rewards like token-F1 for OCR are plausibly hackable, yet hacking is never measured.

Limited novelty and unmatched compute: The method combines multi-teacher OPD with skill-conditioned (privileged-context) distillation, as in Skill-SD and context distillation. Mix-RL is a single run, whereas SCOPED uses four expert RL runs plus distillation, and stronger joint-training baselines (e.g., gradient surgery, OptMerge) are missing.

Analyses are overclaimed: The gradient cosines are on the order of 10−310^{-3} to 10−210^{-2}, close to what is expected in high dimensions, so they do not show a “severe see-saw effect.” Higher trajectory-embedding similarity may simply reflect skill-template boilerplate rather than better behavior.

Questions for authors

How much do the RL training sets (PixMo-Count, We-Math 2.0 Pro, Orsta) overlap with the evaluation splits, and how do results change on benchmarks fully excluded from training?

Can you report mean and standard deviation over several seeds for SCOPED with and without skill augmentation, to show the +0.74 gain is reliable?

Is there direct evidence that the raw experts reward-hack (e.g., verbose OCR outputs that game token-F1) and that skill conditioning reduces it?

What data and targets did the SFT baseline use, and how does SCOPED compare with Mix-RL given matched total compute?

3 comments

We thank the reviewer for the constructive feedback. We address the concerns as follows.

Response to W1, Q1 – Data Splits and Excluded-Benchmark Results

PixMo-Count uses its official training split for RL and a separate evaluation split. We-Math training uses We-Math 2.0 Pro, while evaluation uses the original We-Math testmini (1,740 questions). Skills are extracted from training-set trajectories. We additionally recompute the averages after excluding both benchmarks:

BenchmarkMix-RLMOPDSCOPED
CountBenchQA89.8089.4190.43
CC-OCR V286.0188.8988.79
ChartQA85.0088.1688.28
VisualPuzzles40.1043.7546.58
MUIRBench67.8076.2776.58
MMSI-Bench29.1031.5032.00
Average66.3069.6670.44

SCOPED outperforms MOPD on five of six benchmarks, averaging +0.78 over MOPD and +4.14 over Mix-RL. We will clarify the dataset sources and splits.

Response to W2, Q2 – Skill Gains and Reliability

Skill conditioning improves the 8B and 4B averages by +0.74 and +0.45, respectively. The 8B model improves on six benchmarks, while the 4B model improves on all eight. Amplitude retention and directional alignment analyses further show stronger preservation of expert-induced behaviors. Multi-seed training variance has not yet been measured.

Response to W1 – Skills Beyond Answer Formatting

Beyond answer formatting, skills capture trajectory-derived strategies for visual evidence checking, numerical comparison, cross-image matching, and error avoidance. Ablating the trajectory sources yields the following gains over the corresponding original RL experts:

BenchmarkSuccessFailureCombined
PixMo-Count+1.13+1.51+2.46
ChartQA+0.48+0.56+0.96
We-Math+1.15+0.75+1.90
MMSI-Bench+0.80+0.60+1.30
8-task average+0.59+0.55+1.09

Combining successful and failed experiences achieves the largest gains, demonstrating their complementary contributions.

Response to W3, W5, Q3 – Mechanism and Analysis

Our gradient analysis examines local directional disagreements during Mix-RL, while skill conditioning operates during subsequent distillation. Reward hacking is a potential optimization risk rather than a directly measured reduction in our experiments; we will clarify the wording. Trajectory similarity measures consistency, while benchmark performance, expert retention, and visual attention provide complementary evidence.

Response to W4, Q4 – SFT and Computational Cost

SFT uses the same approximately 38K mixed-domain samples, with*ground-truth answers as supervision targets. SFT, Mix-RL, and Domain RL all use 3 epochs. Four experts each train on 4 A800 GPUs for approximately 24 hours, totaling roughly 400 GPU-hours MOPD and SCOPED each use 8 A800 GPUs for approximately 24 hours (roughly 90 GPU-hours). SCOPED's main training stages total approximately 580 GPU-hours, excluding offline skill extraction.

Mix-RL versus MOPD evaluates capability integration, while MOPD versus SCOPED isolates skill augmentation under nearly identical distillation runtimes.

Dear Reviewer,

Thank you again for your valuable feedback. We hope that our responses have addressed your concerns and would greatly appreciate any further thoughts you may have.

We would be happy to provide additional clarification if needed.

Thank you for your time and consideration.

The rebuttal addresses most of my concerns. However, after reading other reviewers' comments, I think the paper still needs revise before publication. From my perspective, I would increase my rate while lowering my confidence.

Reviewer_LpfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SCOPED, a framework designed to integrate heterogeneous visual capabilities into a unified VLM while mitigating cross-capability interference inherent in joint reinforcement learning. The authors evaluate SCOPED across eight benchmarks where SCOPED-8B outperforms its base Qwen3-VL-8B model, a mixed-capability RL baseline and a multi-expert distillation variant without skill conditioning.

Critical strengths
  • The paper evaluates SCOPED across benchmarks covering distinct capabilities on both 4 B and 8 B backbone models. Comprehensive comparisons against many baselines
  • Comprehensive diagnostic analyses of capability interference and retention
Critical weaknesses
  • The central conceptual novelty- extracting reusable skills from expert trajectories via reflection - yields only a +0.74 point gain on the 8B model and +0.45 on the 4B model over skill-less multi-expert distillation
  • SCOPED incurs noticeable computational overhead by training four independent experts, collecting and reflecting on trajectories, and running an additional distillation phase. However, the paper lacks a compute-matched baseline
  • Missing related work on multi objective RL https://aclanthology.org/2026.findings-acl.1717/
Questions for authors
  • What are the total GPU-hours, FLOPs, and token counts required for SCOPED relative to Mix-RL and skill-less distillation? How does SCOPED perform relative to a Mix-RL baseline trained with an equal computational budget?
  • Can you directly quantify whether skill conditioning reduces cross-capability gradient conflicts or mitigates reward-hacking behaviors during the distillation phase compared to distillation without skills?
2 comments

We thank the reviewer for recognizing the extensive evaluation and capability analyses in our paper. We provide the following clarifications.

Response to W1 – Contribution of Skill Augmentation

SCOPED addresses two complementary aspects of capability integration: independent RL and multi-expert distillation enable capability acquisition and consolidation, while skill conditioning further improves expert supervision. On the 8B model, Mix-RL, MOPD, and SCOPED achieve average scores of 69.31, 72.39, and 73.13, respectively. Skill augmentation also brings an additional +0.45 gain on the 4B model. Beyond benchmark performance, our analyses of amplitude retention and directional alignment show that SCOPED better preserves expert-induced behaviors than MOPD. These results demonstrate that skill augmentation contributes to both performance improvement and specialized capability preservation.

Response to W2, Q1 – Computational Cost and Training Budget

We further clarify the actual training costs. Four capability-specific experts are trained independently, each using 4 NVIDIA A800 GPUs for approximately 24 hours, totaling roughly 400 GPU-hours. Each expert is trained only on its corresponding subset of approximately 9.5K samples, with approximately 38K samples across all four domains. Mix-RL instead jointly trains on the same combined dataset. Both RL strategies use 3 epochs, maintaining identical training data coverage and epoch counts.

During multi-expert distillation, both MOPD and SCOPED use 8 A800 GPUs for approximately 24 hours, corresponding to roughly 190 GPU-hours each. The combined cost of expert training and distillation for SCOPED is approximately 580 GPU-hours, excluding offline skill extraction. Importantly, MOPD and SCOPED share the same experts and distillation configuration, with nearly identical training times. Thus, skill conditioning introduces no substantial additional training-time overhead during distillation.

Response to Q2 – Gradient Conflict and Reward-Hacking Analysis

Our gradient analysis investigates optimization interference during mixed-capability RL, whereas skill conditioning operates during subsequent expert distillation. The amplitude retention, directional alignment, and visual attention analyses provide complementary evidence that skill-conditioned supervision improves the preservation of expert capabilities.

Reward hacking is discussed as a potential risk of reward-based optimization rather than a directly measured outcome of SCOPED. We will refine the corresponding description to distinguish this motivation from the empirically observed improvements in expert supervision and capability retention.

Response to W3 – Missing Related Work on Multi-Objective RL

We thank the reviewer for pointing out RECAP (Phan et al., 2026). RECAP mitigates capability forgetting during joint RL through general-capability replay and dynamic objective weighting. In contrast, SCOPED independently acquires specialized expertise and subsequently integrates heterogeneous capabilities through skill-conditioned multi-teacher distillation. We will incorporate RECAP into Related Work and clarify the distinction between these optimization strategies.

Dear Reviewer,

Thank you again for your valuable feedback. We hope that our responses have addressed your concerns and would greatly appreciate any further thoughts you may have.

We would be happy to provide additional clarification if needed.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.