acceptodds
Withdrawn from ICLR 2027

SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual Multi-Agent Systems (VMAS) extend vision-language models (VLMs) through collaborative reasoning, but adapting agent perception and communication to the visual demands of each input remains challenging. Existing approaches often treat visual evidence acquisition, communication topology, and experience refinement separately, limiting how evolving skills can shape subsequent team behavior. We introduce **SkillGraph**, a framework for *vision-driven self-adaptation* that couples these processes through visual information. Given an image and a question, SkillGraph retrieves reusable skills to instantiate agents. A Multimodal Graph Transformer (MMGT) combines skill representations with question and image features to compute agent-specific selective attention and visually grounded representations. The resulting attention maps define agent-specific image crops, which are provided to the corresponding agents alongside the original image. The visually grounded representations further predict a directed communication graph, allowing visual context to guide both evidence acquisition and collaboration. During training, failed executions are diagnosed to refine a multimodal Skill Bank, whose updated skill representations are fed back into MMGT to reshape subsequent visual attention and communication topology. Experiments across four benchmarks, five representative multi-agent structures, and four VLM backbones show that SkillGraph consistently improves performance while reducing inference-time token consumption. Controlled analyses further validate the contributions of visual conditioning, agent-specific visual inputs, and skill evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Multi-Agent Systems, Multi-Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6jcKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SkillGraph proposes a visual multi-agent framework that jointly adapts agent-specific visual evidence and communication structures through evolving skills. It uses a Multimodal Graph Transformer to generate attention-guided image crops and predict query-dependent communication graphs, while a failure-driven Skill Bank updates agent instructions based on execution feedback. Experiments on four visual question-answering benchmarks with multiple VLM backbones and communication topologies report consistent accuracy improvements and reduced inference-time token consumption.

Critical strengths
  1. The paper presents a coherent framework that connects skill retrieval, agent-specific visual attention, adaptive communication topology, and failure-driven skill refinement. The integration allows agent capabilities and collaboration structures to adapt to different visual tasks.
Critical weaknesses
  1. The authors claim that MMGT jointly learns agent-specific visual attention and communication topology. However, its training objective directly optimizes only the graph sampling policy, without explicitly optimizing the quality of visual attention.
  2. The three main components of the proposed framework have relatively direct foundations in existing techniques. The proposed co-evolution mechanism mainly involves revising skill descriptions, re-encoding skill embeddings, and alternating MMGT optimization. It does not introduce a dedicated objective or learning algorithm for jointly optimizing skills and communication topology, which limits its methodological novelty.
  3. Although the authors propose Skill–Collaboration Co-Evolution, the Skill Designer only modifies the trigger conditions and reasoning strategies of existing skills or creates new skills. It does not explicitly optimize the communication relationships among agents.
Questions for authors
  1. The proposed framework alternates between MMGT optimization using the current Skill Bank and skill refinement based on execution failures. Could the authors compare this approach with independently optimizing the Skill Bank and MMGT, followed by combining them at inference time? Such a comparison would help demonstrate whether the proposed co-evolution mechanism is necessary.
  2. SkillGraph provides agents with additional attention-guided image crops, so the performance improvements may result from access to higher-resolution local visual evidence rather than the proposed skill-conditioned selective attention. Could the authors compare against random crops or uniform grid crops under the same visual token budget to verify the effectiveness of the learned attention mechanism?
  3. The reported inference efficiency improvements may primarily result from reducing the number of communication edges rather than learning a more effective communication topology. Could the authors provide additional ablation experiments comparing learned communication graphs with random or heuristic graphs under the same communication edge budget?
Reviewer_F2fEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillGraph, a framework that jointly learns agents’ visual attention and communication topology using a multimodal graph Transformer, while iteratively updating a skill bank based on failure feedback collected during training. The approach aims to enable adaptive collaboration in multi-agent vision systems while keeping the vision-language model backbones frozen. Experiments across four visual question answering benchmarks, multiple model backbones, and different communication topologies show consistent performance improvements. The authors also report reduced inference token consumption.

Critical strengths
  1. SkillGraph integrates visual attention learning, communication topology optimization, and skill bank evolution into a unified multi-agent collaboration framework. This design enables agents to adapt visual information selection and inter-agent communication to task requirements while keeping the vision-language model backbones frozen, offering methodological novelty and practical value.

  2. The paper evaluates SkillGraph across four visual question answering benchmarks, multiple model backbones, and different communication topologies, with ablation studies examining the contributions of key components. The results demonstrate consistent performance improvements across multiple settings, alongside reduced inference token consumption.

Critical weaknesses
  1. Insufficient Validation of Learned Visual Attention. The crop contexts are treated as fixed inputs during gradient updates, and the paper does not directly optimize the utility of the selected regions as visual evidence. It also lacks comparisons against simple cropping baselines. Consequently, it remains unclear whether the proposed module genuinely learns task-relevant visual attention.

  2. Lack of Comparisons with Established Adaptive Baselines. The experiments primarily compare SkillGraph against fixed communication topologies, with only a custom VLM Meta-Agent as an additional baseline. The advantages of SkillGraph over published methods for adaptive communication topology design and multimodal skill accumulation are not sufficiently established. This limits the evidence for the necessity and substantive benefits of the proposed joint adaptation framework.

  3. Incomplete Assessment of Computational Efficiency. The paper primarily supports its efficiency claims through reductions in inference token consumption. However, token counts alone do not capture end-to-end inference costs. Additional overhead from visual processing, failure diagnosis, and skill bank updates should also be accounted for to provide a more comprehensive assessment of the method's computational efficiency.

Questions for authors
  1. Can the authors provide comparisons with simple cropping baselines or other evidence that the learned attention maps identify task-relevant visual regions, given that the crops are treated as fixed inputs during gradient updates?

  2. Can the authors compare SkillGraph with established adaptive topology and multimodal skill-evolution methods under comparable settings to better establish its advantages over existing approaches?

  3. Can the authors report end-to-end inference costs, including latency and the overhead of visual processing, failure diagnosis, and skill bank updates, rather than relying solely on inference token consumption?

Reviewer_YUxqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SkillGraph, a framework for improving visual multi-agent systems by jointly adapting agent skills, visual attention, and communication topology. It uses a Multimodal Graph Transformer to select relevant image regions for each agent and determine how agents exchange information. During training, a Skill Bank is updated using feedback from failed executions, allowing the system to improve its strategies over time. Experiments on four visual question-answering benchmarks and several vision-language models show improvements over fixed-topology multi-agent baselines and a reduction in inference-time token usage.

Critical strengths
  • Adapting both the visual information available to each agent and the way agents communicate is a reasonable approach to reducing unnecessary computation and improving collaboration.
  • The method brings together skill retrieval, attention-guided image crops, adaptive communication, and skill evolution in a single framework. The connection between these components is a useful aspect of the work.
  • The experiments cover four benchmarks and multiple vision-language backbones, including models of different sizes. The improvements are generally consistent across these settings, which provides some evidence that the method is not limited to a single model or task.
  • The authors include ablation studies, a cross-dataset skill-transfer analysis, and an evaluation of failure-diagnosis reliability. These analyses help explain the proposed approach beyond the main performance comparisons.
Critical weaknesses
  • The main results combine several changes at once: skill evolution, adaptive communication, and additional visual crops. Although the paper includes ablations, the component analysis appears to focus on only MMBench and RealWorldQA. It is therefore difficult to tell whether the contribution of each component holds across all four benchmarks.
  • The training process relies on ground-truth answers to diagnose failures and revise skills, so using the term self-improving/evolving/adaptation is not very accurate.
  • The optimization of visual attention is not fully clear. In the training algorithm, it is unclear how directly the task reward improves the quality of the visual crops, rather than affecting them indirectly through shared representations.
  • Most experiments use fixed communication topologies as baselines, while the comparison with an adaptive VLM meta-agent relies on a context-engineered approach. Comparisons with stronger learned-topology or adaptive multimodal multi-agent methods would be more beneficial.
  • The reported performance gains are sometimes small for larger backbones. Also, the token-cost comparison focuses on the random-topology baseline and does not report end-to-end inference latency or the full computational overhead of the framework.
  • Important settings, including some graph-prediction and crop-extraction parameters, are not sufficiently clear for reproducing the full method.
Questions for authors
  • How does the task reward improve the attention maps and the quality of the selected image crops, given that the crops are treated as fixed during gradient computation?
  • Could the authors provide comparisons with stronger adaptive-topology baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.