acceptodds
Under review as a conference paper at ICLR 2027

Skill-: Skill-Policy Co-Evolution for Self-Improving Multimodal Reasoning

Abstract

Post-training with explicit reasoning trajectories is important for enhancing the reasoning capabilities of Multimodal Large Language Models (MLLMs). However, obtaining high-quality reasoning trajectories typically requires costly human annotation or supervision from stronger external models. To reduce this cost, self-improvement paradigms have emerged, enabling models to generate, verify, and learn from their own reasoning experience. Despite their effectiveness, existing self-improvement methods remain largely trajectory-centric, repeatedly consuming sample-specific reasoning trajectories while leaving persistently unresolved samples with little contribution to subsequent improvement. In this work, we introduce Skill-, a skill-policy co-evolution framework that turns self-generated multimodal experience into reusable reasoning skills and continuously expands their capability coverage. In Skill-, a shared MLLM policy jointly performs skill selection, utilization, and evolution, with stage-specific credit assignment yielding learning signals according to the downstream consequences of each decision. The newly-recovered successful experience further improves the parametric policy and evolves the skill library, allowing reusable skills and the policy to improve iteratively across evolution rounds. Experiments show that Skill- achieves up to higher evolutionary efficiency and greater data efficiency than existing self-improvement methods. Across diverse benchmarks spanning mathematical, chart/document, and general multimodal reasoning, Skill- also achieves stronger performance against representative multimodal reasoning models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Self-Improvement, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DcJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Skill-π addresses limited experience reuse in multimodal self-improvement by turning successful trajectories into reusable skills and jointly updating a shared policy and skill library. Its stage-specific credit assignment supports skill selection, utilization, and evolution, with reported improvements across multimodal reasoning benchmarks.

Critical strengths

The framework offers a well-motivated approach to cross-sample experience reuse, with explicit skill-induction constraints and a principled separation of selection and utilization credit. Controlled comparisons and complementary analyses of performance, skill dynamics, negative transfer, and computational cost provide useful empirical support.

Critical weaknesses
  1. Contribution isolation. The GRPO controls retain skill guidance, leaving a matched no-skill RL baseline missing. Unclear inference-time library use further prevents separating parameter learning from prompting and additional computation. Vision-SR1, SkillRL, and Skill1 are cited but not compared, without specific explanations of comparability. This leaves the incremental contribution relative to closely related approaches unclear.

  2. Transfer and credit attribution. Recovery lacks a fresh unguided resampling control, and source–target deduplication is unspecified. Utility also conflates skill contribution with task difficulty, policy capability, and selection bias.

Questions for authors
  1. Could you add matched no-skill GRPO and clarify final library use, including evaluation with and without skills at the same checkpoint?

  2. Could you provide fresh unguided recovery controls, source–target deduplication rules, and difficulty/policy-matched evidence for skill utility?

  3. Could you compare Vision-SR1, SkillRL, and Skill1 where applicable, or explain the specific differences that preclude comparison?

Reviewer_B1LkOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Skill-π, a framework for self-imimproving multimodal reasoning through skill-policy co-evolution. Instead of learning only from individual reasoning trajectories, the method extracts reusable skills from successful attempts and uses them to guide further exploration. A shared MLLM handles skill selection, utilization, and evolution, with separate learning signals for each stage. The skill library and model are then updated over multiple rounds. Experiments on mathematical, medical, and general multimodal reasoning tasks show improvements over several existing methods, particularly in data efficiency.

Critical strengths

Clear Presentation: The paper is well organized and easy to follow. Figure 2 is particularly helpful, showing how skills are constructed, retrieved, and used, as well as how the results feed back into model training. The inputs and outputs at different stages are also clearly illustrated.

Interesting Research Direction: The paper tackles an important problem in multimodal self-improvement. Rather than repeatedly learning from individual successful trajectories, Skill-π tries to build reusable reasoning skills while improving the underlying model. Using a shared MLLM for skill selection, utilization, and evolution is an interesting design choice, especially for learning across different multimodal tasks.

Critical weaknesses
  1. Limited RL Baselines: The RL comparison is rather limited. Skill-π uses a GRPO-based objective, and most of the ablations focus on how the learning signals are assigned within this framework. I would like to see comparisons with other RL methods, such as DAPO, under the same training conditions. Without these comparisons, it is difficult to tell how much of the improvement comes from the skill-policy co-evolution framework itself rather than the choice of RL algorithm and training strategy.

  2. Incomplete Analysis in Figure 3: Figure 3(a) and Figure 3(c) do not clearly specify which models and datasets were used, making the results difficult to interpret. Another concern is that the performance of Skill-π is still increasing at the end of both curves, with no clear sign of convergence. Since continued improvement is one of the main claims of the paper, I think a longer training run is needed. The authors should report the results until the curves have largely stabilized, or explain why training was stopped at this point. Otherwise, it is hard to judge whether the reported gains would persist after convergence.

Questions for authors
  1. Training the Shared MLLM: I had some trouble understanding how the three stages are combined during training. Appendix C provides the prompts, but it is still difficult to reconstruct the actual training data. Are selection, utilization, and evolution samples mixed in the same batch? It would help to show one complete example, with the input, model output, reward, and loss mask for each stage. Another question is why a shared MLLM is preferred over three separate models. Was this choice tested experimentally?

  2. OOD Generalization: How well do the learned skills transfer to unseen datasets? For example, the authors could train on Geometry3K and evaluate on another visual math benchmark without further training. I am also curious about transfer between domains. Since skill retrieval uses visual-state compatibility, including domain information, can a skill learned from mathematical diagrams be reused for chart reasoning? The current experiments do not directly answer these questions.

  3. Long-Term Skill Evolution: The experiments stop after three evolution rounds. This seems rather short for a method focused on continuous self-improvement. What happens after ten or more rounds? Figure 3(b) already shows that the library size does not always increase. I wonder whether longer training would lead to redundant skills, conflicting strategies, or eventually a drop in performance.

  4. Skill Library vs. Policy Learning: Table 4 removes individual stages, but I think the contribution of the skill library and the policy updates should be separated more carefully. One useful experiment would be to freeze the policy while updating the library, and then reverse the setup. How much does each contribute? I would also test the final model without skill retrieval. It is important to know whether the improvement is retained in the model weights or still depends heavily on the external library.

  5. Training Cost: The data efficiency results are encouraging, but training examples are only part of the cost. Skill-π uses 16 skill-conditioned rollouts per query, plus retrieval and library updates. Appendix B.4 reports GPU-hours for Skill-π, but not for the competing methods. How much compute does each method need to reach the same accuracy? A comparison against VISTA and ReSTEM using total GPU-hours would be more informative than sample counts alone.

Reviewer_o4QaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Skill-π, a framework that improves multimodal reasoning through the co-evolution of a skill library and a model policy. The method extracts reusable skills from successful reasoning trajectories and uses them to guide subsequent exploration. The authors also introduce a stage-specific credit assignment mechanism to optimize skill selection, utilization, and evolution. Experiments demonstrate performance improvements across multiple multimodal reasoning benchmarks.

Critical strengths
  1. The paper addresses the limitations of trajectory-based self-improvement by introducing reusable skills. This provides a valuable direction for improving the self-improvement capabilities of multimodal models.
  2. The stage-specific credit assignment mechanism distinguishes skill selection, utilization, and evolution. This design enables more targeted learning signals and better integration of skill learning with policy optimization.
  3. Skill-π improves the average performance of Qwen2.5-VL-7B and Qwen3-VL-8B by 7.2 and 5.1$%$, respectively, across seven multimodal benchmarks. The ablation studies further demonstrate the effectiveness of the main components and the stage-specific credit assignment mechanism.
Critical weaknesses
  1. Existing studies, such as Skill1 and SkillRL, have also explored skill reuse and joint policy optimization. The paper could better clarify its unique contributions, particularly regarding stage-specific credit assignment and cross-instance skill abstraction.
  2. Figure 4 demonstrates the recovery of previously unsuccessful samples using skill guidance. However, the paper does not compare this with additional unguided sampling under the same rollout budget. Therefore, it is difficult to isolate the contribution of skill guidance itself.
  3. The reported efficiency improvements lack sufficiently clear calculation details. In addition, some accuracy values in Table 1 appear inconsistent with the reported test set sizes.
Questions for authors
  1. What are the key methodological differences between Skill-π and existing approaches such as Skill1 and SkillRL? Could the authors further clarify the novel aspects of the proposed credit assignment mechanism?
  2. For the unsuccessful samples in Figure 4, what is the recovery rate when additional unguided sampling is performed under the same rollout budget? Does skill guidance still provide a significant improvement?
  3. How are the reported 3× evolutionary efficiency and 2.7× data efficiency calculated? Do these comparisons account for the additional computational costs of skill construction, retrieval, and evolution?
  4. Appendix A states that the SLAKE test set contains 355 samples. However, the accuracies of 79.38% for ReSTEM (3B, Iteration 2) and 78.21% for R3V (3B, Iteration 1) in Table 1 cannot correspond to integer numbers of correct predictions over 355 samples. I could not find an explanation in the paper indicating whether these values were averaged across multiple runs or computed using another method. Could the authors clarify how these results were obtained?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.