- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work introduces ViSkill, a visual-native skill learning framework for vision-language model agents. Most existing agent skill libraries are text-centric, so they lose spatial and geometric details and treat skill construction separately from policy optimization. To address this, the authors turn successful interaction experiences into composite visual skill cards. Each card combines action-annotated trajectory keyframes with high-level policy descriptions in a single image, keeping the spatial and procedural context of the task intact.
Critical strengths
-
ViSkill uses visual skill cards instead of text-only skills. It puts keyframes and short policy text together, so VLMs can directly see object positions and action order. This works better than plain text or OCR-based text on spatial planning tasks.
-
ViSkill adds a skill-guided reward. If an episode fails but follows part of a retrieved skill trajectory, the agent gets partial credit. This gives denser feedback and helps learning faster without changing the main reward ranking.
-
ViSkill connects retrieval and skill updating. Skills are matched by environmental geometry, not just meaning, and useful new trajectories are added back to the library. This lets the system improve over time.
Critical weaknesses
-
Putting keyframes and a short policy text in one image can help a VLM see spatial layout directly, and the partial credit reward is a reasonable way to ease sparse rewards. My concern is that the surrounding framework is not new enough. Memory bank, retrieval, and RL loop already appear in AtlasVA and SPyCE [1]. ViSkill mostly changes the storage format, so I would not call it a new paradigm.
-
The card itself is also too flat. It mixes the whole trajectory and text without separating plan from action. On long tasks, a small VLM may not know which part to attend to. Retrieval is another weak point because it depends on hand made geometric coordinates. That may work in grid worlds, but I doubt it transfers to open scenes. The library update also seems too static. Eq. 5 controls size, but I did not see merging, conflict resolution, or semantic deduplication, so bad old skills may crowd out better new ones.
-
Single runs on a 3B model, no variance, closed tasks, and a saturated arm result do not prove robustness. The rendering and feature pipeline also needs latency and memory costs.
[1] Zhang, R. and Qiu, W., 2026. SPyCE: Skill-Policy Co-evolution for Multimodal Agents. arXiv preprint arXiv:2607.13854.
Questions for authors
In Eq. 6, if the agent oscillates, does the farthest matched point make the reward flat and encourage local stagnation? Is matching direction sensitive? Also, during online distillation, is there any check that the small model's text matches the trajectory, or can hallucinated or truncated cards enter the library?