acceptodds
Under review as a conference paper at ICLR 2027

SkillCard: Compiling and Evolving Procedural Visual Skills as Single Images

Abstract

Skills enable agents to tackle long-horizon tasks by capturing reusable procedural knowledge from human expertise and past interactions. However, most existing skills are textual and may omit visual cues needed to distinguish states and verify action outcomes. Recent approaches incorporate screenshots and keyframes as visual references alongside textual procedures. Yet preserving this evidence does not by itself make it actionable, as agents must still integrate separately presented instructions and visual evidence to determine when to act, what change to expect, and whether recovery is needed. As new experience reveals gaps in guidance, revisions must also keep the instructions and visual evidence consistent. The challenge is therefore not merely to retain visual information, but to organize it into reusable procedural knowledge that remains coherent as it evolves. To this end, we propose SkillCard, which represents each procedural visual skill as a standalone image while keeping its underlying source editable. Specifically, compiling organizes instructions and visual evidence into reusable cards, while evolving revises their sources based on agent experience and recompiles the cards. Together, these processes support skill reuse at inference and the co-evolution of skills and policies during training. Experiments on Visual ALFWorld and Visual WebShop show that SkillCard improves success rates by over 10% compared with base models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agents, Reinforcement Learning, Vision-Language Models, Skill Learning, Agentic AI, Experience Replay, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FezmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

SkillCard compiles procedural instructions and visual evidence into single-image skills, with editable sources revised from agent experience. Experiments show gains on VisualALFWorld and VisualWebShop, alongside lower skill-related inference token costs.

Critical strengths
  • Separating editable sources from rendered cards is a practical design.
  • SkillCard improves over SkillRL by 7.1/4.5 points on VisualALFWorld ID/OOD.
  • Results across three backbones suggest the approach is broadly useful.
Critical weaknesses
  • The main novelty appears to be single-image compilation. State-conditioned visual procedures and skill–policy co-evolution already exist, so the paper needs to establish what this representation adds through concrete experiments and clear position on how this work among those related prior efforts.
  • The experiments do not show that compiling a skill into one image is responsible for the gains. Differences in instruction quality, selected evidence, or retrieval remain alternative explanations. (Section 3.3 acknowledges that the variants retain different content; Table 3 does not compare identical content across presentation formats.)
  • The benefit of evolution, a central contribution, is not demonstrated separately from policy training and the initial skill bank. (Appendix A.5 explicitly notes the absence of fixed-bank training and local-patch-versus-full-rewrite controls.)
Questions for authors

Does SkillCard outperform an interleaved text-and-image version of exactly the same source,?

How much does updating the cards improve over training with the initial bank fixed?

Do local patches preserve useful knowledge better than full rewrites?

Reviewer_Gs58OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillCard, a framework for representing a reusable multimodal procedural skill as a single rendered image. The method separates an editable source package containing procedural units, keyframes, captions, and layout metadata from a policy-facing image card; the source can be revised from agent trajectories through Patchify and recompiled during policy learning. SkillCard is evaluated on visualized ALFWorld and WebShop settings with local and commercial vision-language models. It reports improvements over direct inference, textual skills, and several multimodal/procedural-memory baselines, and studies card layout, retrieval, token cost, and image resolution.

Critical strengths
  1. Interesting and well-structured framework. The paper presents an interesting high-level idea for organizing and evolving multimodal procedural skills. I appreciate the separation between editable source packages and rendered images, which allows procedural knowledge to be maintained independently of its visual representation. The idea of evolving skills by updating the underlying source packages and recompiling the corresponding cards is also intuitive and appealing. Together, these design choices provide a clear framework for building reusable visual skills that can evolve over time.

  2. Consistent improvements across models and benchmarks. The paper conducts experiments across multiple vision-language models and two visual interaction benchmarks, covering both frozen-policy evaluation and reinforcement learning. The consistent improvements over existing baselines provide encouraging evidence for the effectiveness of the overall SkillCard framework.

Critical weaknesses
  1. Heavy reliance on teacher model and insufficient details. The framework relies heavily on a teacher model for constructing and updating source packages, but key implementation details remain unclear, including how procedural units and keyframes are selected and how Patchify operation is conducted and validated. Moreover, the impact of teacher model quality is not investigated, raising concerns about the soundness and reproducibility of the framework.

  2. Insufficient ablations to disentangle the contributions of different components. Although the overall framework achieves promising results, the experiments do not sufficiently isolate the contributions of its key components. First, it remains unclear how much of the improvement comes specifically from compiling procedural knowledge into a structured single image, rather than from the underlying skill content or selected visual evidence. Second, the reinforcement learning results do not disentangle the contributions of policy optimization and skill evolution. More controlled comparisons, such as content-matched representation baselines and fixed-bank versus evolving-bank training, would help clarify which components are responsible for the observed improvements.

Questions for authors
  1. Which teacher model is used for operations like initial source construction and Patchify? How sensitive is the framework to the teacher model choice?

  2. Please provide a more detailed explanation of how the projection (PrP_r), renderer (CrC_r), and Patchify operations are implemented in practice.

  3. How were the default values of the source budget (bb) and retrieval budget (kk) determined? And please also provide a complete definition of the rendering configuration (ωr\omega_r).

  4. For frozen-policy evaluation, how skill sources are constructed from rollouts on training tasks? Is the bank generated in a single pass or evolved through multiple steps/iters before evaluation?

  5. For reinforcement-learning evaluation, which policy checkpoint and skill bank version are used at evaluation time?

  6. Which kind of representation is used by default in Table 1 and 2?

  7. Please provide two additional comparisons: (1) GRPO training with a fixed SkillCard bank, without skill evolution, and (2) frozen-policy evaluation using the raw resource package presented as text+image rather than compiled into a single image. These comparisons would help isolate the contributions of skill evolution and rendering skills into images, respectively.

  8. Please clarify how baselines like VisualSkill and SkillRL are reproduced and adapted to ensure fair comparisons.

Reviewer_qieZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillCard, a framework to represent procedural knowledge as a single image by combining instructions, visual evidence, and execution steps. It also introduces a way to update these skills from agent experience and jointly optimize the policy using the GRPO algorithm. Experiments were performed on Visual ALFWorld and Visual WebShop, which show improvements over existing skill-based methods.

Critical strengths
  • The motivation is interesting especially I found the idea of connecting procedural instructions with visual evidence for long-horizon tasks.
  • The editable source representation is a useful design choice that allows instructions, visual evidence, and layout revised at the same time.
  • The experimental evaluation is comprehensive. The authors have tested different model backbones and RL settings, along with both in-domain and out-of-domain tasks. They also compare against relevant text-based and multimodal skill approaches.
Critical weaknesses
  1. Although the idea of the paper is interesting, but I think the novelty is somewhat limited compared to MMSkills and VisualSkill. Those works have already used visual information with procedural instructions. In this work, the authors mainly organized them into a single image. I am not fully convinced how much additional benefit this representation provides compared to existing methods.

  2. I also have some concerns about the Patchify mechanism. Here, they update the skills based on agent trajectories and feedback, but there is no validation to check whether the updated skill is actually good or not. Since these updated skills later used for training, I assume an incorrect update may also affect the policy, which needs to be cleared with more discussion

  3. Another concern in the RL experiments where authors compare SkillCard with GRPO and other methods, but there is no experiment where GRPO is trained with a fixed SkillCard bank. Without this comparison, it is hard to understand whether the improvement is actually coming from skill evolution or just because the model is getting additional guidance from SkillCard.

  4. I also think the evaluation failed to provide enough understanding of why SkillCard is working better. The authors mainly report the final task success rate, but their motivation was about recognizing visual conditions, verifying action outcomes and recovering from failures. I couldn't find a detailed analysis showing improvement in these particular abilities. Also in ALFWorld, the agent already gets high-level actions and an oracle controller is used, so it is not clear how much visual understanding is actually required from the model.

  5. The authors perform three runs but I did not understand why they report the best result instead of the average. Some of the improvements over existing SOTA methods are quite small. So it is difficult to know whether these improvements are consistent across different runs. Reporting average performance with standard deviation, at least for the best methods, would be more convincing.

Questions for authors
  • Did the authors compare SkillCard with a structured text and image representation using exactly the same information?
  • Can authors provide more detailed analysis of where SkillCard helps during task execution? For example, is it reducing incorrect action selection, improving recognition of correct conditions, or helping agents verify successful outcomes?

Comments on the paper

1 comment

In the end, sincerely thank the reviewers and area chairs for taking time to review our paper and for the replies during the discussion. Your feedback helped us clarify our work and see where it could be improved. Regardless of the results, we are grateful for the time and care you devoted to our submission!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.