acceptodds
Withdrawn from ICLR 2027

ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action–state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, \method achieves an overall success rate of , rising to with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Agents, Skill Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tEWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces ViSkill, a visual-native skill learning framework for vision-language model agents. Most existing agent skill libraries are text-centric, so they lose spatial and geometric details and treat skill construction separately from policy optimization. To address this, the authors turn successful interaction experiences into composite visual skill cards. Each card combines action-annotated trajectory keyframes with high-level policy descriptions in a single image, keeping the spatial and procedural context of the task intact.

Critical strengths
  1. ViSkill uses visual skill cards instead of text-only skills. It puts keyframes and short policy text together, so VLMs can directly see object positions and action order. This works better than plain text or OCR-based text on spatial planning tasks.

  2. ViSkill adds a skill-guided reward. If an episode fails but follows part of a retrieved skill trajectory, the agent gets partial credit. This gives denser feedback and helps learning faster without changing the main reward ranking.

  3. ViSkill connects retrieval and skill updating. Skills are matched by environmental geometry, not just meaning, and useful new trajectories are added back to the library. This lets the system improve over time.

Critical weaknesses
  1. Putting keyframes and a short policy text in one image can help a VLM see spatial layout directly, and the partial credit reward is a reasonable way to ease sparse rewards. My concern is that the surrounding framework is not new enough. Memory bank, retrieval, and RL loop already appear in AtlasVA and SPyCE [1]. ViSkill mostly changes the storage format, so I would not call it a new paradigm.

  2. The card itself is also too flat. It mixes the whole trajectory and text without separating plan from action. On long tasks, a small VLM may not know which part to attend to. Retrieval is another weak point because it depends on hand made geometric coordinates. That may work in grid worlds, but I doubt it transfers to open scenes. The library update also seems too static. Eq. 5 controls size, but I did not see merging, conflict resolution, or semantic deduplication, so bad old skills may crowd out better new ones.

  3. Single runs on a 3B model, no variance, closed tasks, and a saturated arm result do not prove robustness. The rendering and feature pipeline also needs latency and memory costs.

[1] Zhang, R. and Qiu, W., 2026. SPyCE: Skill-Policy Co-evolution for Multimodal Agents. arXiv preprint arXiv:2607.13854.

Questions for authors

In Eq. 6, if the agent oscillates, does the farthest matched point make the reward flat and encourage local stagnation? Is matching direction sensitive? Also, during online distillation, is there any check that the small model's text matches the trajectory, or can hallucinated or truncated cards enter the library?

Reviewer_JuZGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ViSkill introduces a framework for storing successful VLM-agent interactions as composite visual skill cards. The method combines geometry-aware retrieval, online skill distillation, skill-guided reward shaping, and PPO optimization. An optional cold-start procedure initializes the library with solver-generated trajectories. Experiments cover Sokoban, FrozenLake, and PrimitiveSkill.

Critical strengths
  1. Representing successful trajectories as composite visual cards is intuitive and appropriate for spatially grounded tasks. The comparison with text-based skill representations suggests that retaining visual structure can improve skill reuse.

  2. The paper includes component ablations and training-dynamics analyses. The shared-backbone comparisons against VAGEN and AtlasVA are useful, and the results support the benefits of explicit experience reuse alongside policy optimization.

Critical weaknesses

W1. Marginal improvements over the strongest related baseline.

The primary performance improvements over AtlasVA are only three percentage points on Sokoban and one percentage point on FrozenLake. Both approaches achieve 100% success on PrimitiveSkill. These small differences are insufficient to establish a robust advantage without uncertainty estimates across independent training runs. The paper's claim of superior overall performance is numerically supported, but the evidence for a substantial algorithmic improvement over strong memory-augmented RL approaches remains limited.

W2. Limited benchmark difficulty and insufficient evidence of general visual skill learning.

The evaluated environments consist of two discrete grid-world tasks and a set of short-horizon manipulation primitives. FrozenLake uses deterministic transitions with slippery dynamics disabled, while PrimitiveSkill episodes involve only a few interaction turns. Moreover, the strongest related baseline already solves all five PrimitiveSkill categories perfectly. This benchmark saturation makes it difficult to establish whether ViSkill improves complex visual reasoning, long-horizon planning, or compositional skill transfer.

W3. The novelty of visual skill cards relative to multimodal demonstration retrieval is not fully isolated.

The proposed skill representation combines annotated trajectory frames with a textual strategy rendered into a composite image. Although the comparison against text-only and OCR-based representations supports the usefulness of visual information, it does not conclusively establish the benefit of the specific visual-native skill representation over simply retrieving and providing successful visual demonstrations. A stronger comparison would expose the same frames, actions, and strategy information through a standard multimodal in-context demonstration interface under matched context budgets.

W4. Retrieval relies on task-specific geometric representations and restricted transfer.

The primary retrieval mechanism uses manually specified geometric descriptors. In PrimitiveSkill, retrieval additionally uses task identifiers and corresponding object coordinates, and skills are never retrieved across different manipulation tasks. This restricts the demonstration of compositional or cross-task skill transfer. The DINOv2 ablation on the two grid-world environments partially mitigates concerns about handcrafted features, but does not establish an equally general retrieval mechanism for the 3D setting.

W5. Skill quality filtering and reward shaping rely on substantial task-specific structure.

The quality gate uses trajectory-length budgets derived from task-specific solvers or scripted oracle plans. The auxiliary reward also requires task-dependent progress matching against a reference trajectory. These mechanisms are straightforward in grid worlds and structured manipulation tasks but may be difficult to implement in open-ended environments without reliable state abstraction or reference solutions. Although the fixed-budget ablation demonstrates that solver-derived thresholds are not strictly necessary in the 2D tasks, the generality of the full training signal remains insufficiently validated.

W6. Trajectory-alignment rewards may discourage alternative successful strategies.

The skill-guided reward measures progress toward states represented by a retrieved reference trajectory. However, multiple valid strategies can solve the same task through substantially different state sequences. Favoring agreement with one successful trajectory may bias optimization toward previously observed solutions and penalize novel but equally valid behaviors. The paper acknowledges this limitation but does not quantify its consequences in tasks containing diverse solution paths.

W7. The claimed co-evolution mechanism is not fully disentangled from ordinary policy improvement.

The paper shows that a library accumulated during joint policy optimization is more effective than one accumulated without optimization. However, a better-trained policy naturally generates more successful and higher-quality trajectories. Therefore, the observed transfer improvement does not by itself demonstrate that the feedback loop between skills and policy learning provides benefits beyond stronger demonstration collection. Comparisons using libraries matched for trajectory success, quality, diversity, and size would better isolate genuine co-evolution effects.

Questions for authors
  1. Are the geometric descriptors used for retrieval computed exclusively from the agent's RGB observations at inference time? If not, can the authors provide a comparison under strictly matched observation access?

  2. Can the authors provide multi-seed uncertainty estimates against AtlasVA and isolate the contribution of visual skill representations under matched token and auxiliary-supervision budgets? How does performance change without solver-derived trajectory-length thresholds?

Reviewer_DwRkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ViSkill, a framework for reinforcing VLM agents with evolving visual-native skills. Successful trajectories are distilled into visual skill cards containing annotated observations and strategy descriptions. These skills are retrieved based on geometric similarity and used to guide both inference and reward shaping during PPO training. Experiments on Sokoban, FrozenLake, and PrimitiveSkill demonstrate improvements over existing baselines, with an overall success rate of 0.89, increasing to 0.91 with cold-start initialization.

Critical strengths
  • I find the motivation of incorporating visual evidence as part of a skill very intuitive in solving multi-turn visual tasks. Although prior work like AtlasVA already shifts to such paradigm, it's convincing to observe more work that supports the usefulness of visual evidence as part of skills for decision-making processes.
  • What I find as most interesting from prior work like AtlasVA that relies on spatial heatmaps for skill storage and retrieval, this work constructs skills based at the trajectory level, and empirically the trained models achieve better performance compared to the prior work.
Critical weaknesses

I am generally unconvinced on the generality of the claims and the empirical utilities of the framework- see below:

  • The evaluation environments are homogeneous and largely synthetic, limiting the generalizability of the scientific conclusions and the effectiveness of the framework. Also, the framework seems to have been only validated on Qwen2.5-VL-3B
  • The improvements over competitive baselines ie AtlasVA are relatively marginal given the additional complexity.
  • The framework introduces numerous additional hyperparameters and appears sensitive to their selection.

See questions how I felt these could be addressed.

Questions for authors
  1. Can authors incorporate more tasks for evaluation between this work, AtlasVA, along with the untrained baseline and the strongest proprietary models? For example the other two tasks in VAGEN that authors did not evaluate with, as well as more realistic tasks such as those in OSWorld or VisualAgentBench. Incorporating more and newer model families would be more helpful e.g., Qwen3.5, Gemma4.
  2. Can authors provide a more comprehensive comparison against AtlasVA how frameworks are different conceptually, and a cost comparison in terms of training and inference FLOPs, latency to justify the marginal gain as reported in the main table?
  3. Does a fixed set of optimal hyperparameters on one environment and model transfer across environments and models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.