SkillCard: Compiling and Evolving Procedural Visual Skills as Single Images
Abstract
Skills enable agents to tackle long-horizon tasks by capturing reusable procedural knowledge from human expertise and past interactions. However, most existing skills are textual and may omit visual cues needed to distinguish states and verify action outcomes. Recent approaches incorporate screenshots and keyframes as visual references alongside textual procedures. Yet preserving this evidence does not by itself make it actionable, as agents must still integrate separately presented instructions and visual evidence to determine when to act, what change to expect, and whether recovery is needed. As new experience reveals gaps in guidance, revisions must also keep the instructions and visual evidence consistent. The challenge is therefore not merely to retain visual information, but to organize it into reusable procedural knowledge that remains coherent as it evolves. To this end, we propose SkillCard, which represents each procedural visual skill as a standalone image while keeping its underlying source editable. Specifically, compiling organizes instructions and visual evidence into reusable cards, while evolving revises their sources based on agent experience and recompiles the cards. Together, these processes support skill reuse at inference and the co-evolution of skills and policies during training. Experiments on Visual ALFWorld and Visual WebShop show that SkillCard improves success rates by over 10% compared with base models.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.