acceptodds
Withdrawn from ICLR 2027

Self-Evolving Visual Agents via Margin Reward

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool invocation usually relies on curated question-answer pairs or expert demonstrations at scale, which are expensive to acquire. In this study, we propose Marvo, a simple-yeteffective self-evolution approach that enables learning tool invocation from unlabeled images. Marvo consists of two agents: (1) a challenger that generates question-answer pairs from unlabeled images, and (2) a solver that learns tool invocation by solving questions through reinforcement learning. To encourage effective tool invocation, a good training question should be difficult enough to warrant tool use, yet remain within the solver's capabilities. To this end, we introduce a margin reward to the challenger, defined as solver's performance difference with and without access to tools. With the margin reward, we create an adaptive curriculum that encourages the solver to learn effective tool invocation. Across benchmarks covering agentic reasoning, visual search, mathematical, and general real-world reasoning, Marvo improves average accuracy by up to 7.1% - demonstrating margin reward as a preferred way to scale question complexity without human supervision.

Primary area:
reinforcement learning
Keywords:
Self-Evolution, Visual Agents, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ERTsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Marvo, a self-evolution framework that trains visual agents using question–answer pairs generated from unlabeled images. A challenger and a solver are alternately optimized, with the challenger rewarded by the solver’s accuracy difference with and without tools to encourage questions that benefit from tool use. Experiments across two model backbones and seven benchmarks show improvements over the initial checkpoints and the R-Zero baseline, while extended training reveals limitations from declining generated-data quality.

Critical strengths

This work presents a well-motivated reward for tool-useful task generation. The margin reward directly measures the solver’s benefit from tool access, providing a more targeted signal than uncertainty alone. Using this signal to train the challenger offers a simple and interpretable way to align generated tasks with the goal of improving tool use.

Critical weaknesses
  1. Limited methodological contribution and incomplete isolation of the reward’s value. The challenger–solver framework largely follows R-Zero, making the margin reward the central methodological contribution. Although comparisons with R-Zero are useful, the paper does not establish whether the full margin formulation offers substantial benefits over simpler objectives, such as rewarding tool-assisted success or combining uncertainty with tool necessity. This leaves the significance of the specific reward design insufficiently established.

  2. Generated-data reliability limits sustained self-evolution. Tables 6–7 show that additional evolution rounds lead to performance degradation alongside declining question quality: the external judge’s estimated accuracy of generated questions falls from 68.5% in Round 1 to 43.6% in Round 8. Although reporting this limitation is valuable, the method lacks an effective mechanism to prevent unreliable self-generated supervision, limiting the evidence for scalable, sustained improvement.

Questions for authors
  1. Contribution of the tool-free baseline. While Table 1 compares against uncertainty-based rewards and Table 4 changes the reference harness, can the authors isolate the contribution of subtracting tool-free performance by comparing against tool-assisted success alone, with filtering and training budgets held fixed?
  2. Scalability of self-evolution. The later-round degradation reported in Appendix F.2 raises concerns about whether Marvo can sustain improvement as evolution continues. Is this primarily limited by the models’ initial capabilities, or by errors accumulating and being reinforced through self-training? Can the authors also investigate whether stronger models mitigate this degradation, or whether stricter filtering with independent checks of question and answer validity enables more sustained improvement?
  3. Consistency of solver rewards. Section 2.4 describes the solver reward as binary answer correctness, while Table 5 also lists a tool-use bonus of +0.2 and a format penalty of −0.5. Can the authors clarify how these components are combined, which experiments use them, and whether Marvo and R-Zero use identical solver rewards? This clarification would help establish whether the comparison isolates the effect of the challenger’s margin reward.
Reviewer_nqnGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Marvo, a self-evolution framework for training visual agents to use tools without manually annotated question-answer pairs. It introduces a margin reward that measures the solver's performance difference with and without tool access, encouraging the challenger to generate questions where tool use is beneficial. The challenger and solver are trained alternately using GRPO. Experiments on Qwen2.5-VL-7B and Qwen3-VL-4B show improvements across seven benchmarks covering visual search, mathematical reasoning, and agentic tasks.

Critical strengths
  • The research motivation is clear and sound. The concept of leveraging the performance gap between "tool-assisted reasoning" and "reasoning without tools" to steer question generation is both simple and intuitive.
  • The proposed margin reward directly accounts for the utility of tools, rather than relying solely on problem difficulty or the solver’s internal uncertainty.
  • The experimental evaluation covers two model families and a variety of visual reasoning benchmarks. After three rounds of evolution, Marvo consistently outperforms R-Zero in terms of overall accuracy.
  • The additional experiments with a zoom-in tool and different initialization checkpoints provide useful insights into the applicability of the method.
Critical weaknesses
  • The challenger generates both questions and answers without external ground-truth verification. As shown in Table 7, the estimated quality of the generated answers declines as the number of evolution rounds increases, which may introduce noisy supervision signals.
  • The long-term stability of the self-evolution process is not fully established. Appendix F.2 indicates that, after five to six rounds of evolution, the model’s performance on several benchmarks begins to decline. This raises a question: can the proposed marginal reward truly support sustained self-improvement?
  • The paper lacks sufficient direct evidence to demonstrate that performance gains actually stem from "better tool-use decisions." While Figure 6 shows an increase in the number of tool calls, a higher frequency of calls does not equate to greater efficiency in tool usage. An analysis of tool-call success rates or the occurrence of unnecessary tool calls would provide a more compelling argument.
  • The effectiveness of the marginal reward is primarily demonstrated through comparisons with R-Zero. Including comparisons against other, simpler reward designs—such as those based solely on "tool-assisted accuracy" or "uncertainty" under identical conditions—would help to more clearly delineate the specific benefits provided by this objective function.
  • The paper does not report experimental results based on multiple random seeds. Given the relatively modest performance gains over R-Zero (particularly for the 4B model), conducting additional experiments would help verify the reliability of the results.
Questions for authors

Please see the Weaknesses part.

Reviewer_RHUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the challenge that visual agents often rely on costly human annotations to learn tool use, this paper introduces Marvo, a self-evolving framework that operates on unlabeled images. The framework employs a dual-agent evolutionary system consisting of a "Challenger" (which proposes tasks) and a "Solver" (which attempts to solve them). Building upon the R-Zero paradigm, Marvo incorporates a "marginal reward"—defined as the performance gap of the Solver when operating with versus without access to tools—to adaptively filter for high-value tasks that cannot be solved through visual intuition alone but require the use of external tools. In benchmarks covering agent reasoning, visual search, and multimodal mathematics, this approach improved average accuracy by up to 7.1%, significantly enhancing the stability and effectiveness of tool use.

Critical strengths
  1. Enables unsupervised self-evolution: Starting solely from unlabeled images—without the need for human-annotated question-answer pairs or costly expert interaction trajectories—this method significantly lowers the data threshold and costs associated with teaching visual agents to use tools.
  2. Marginal reward mechanism highlights "tool necessity": Unlike traditional uncertainty metrics that rely solely on model perplexity, the marginal reward directly measures the performance gap between using a tool and not using one. This filters for tasks that cannot be solved through visual perception alone but can be answered correctly when a tool is employed.
  3. High training stability and strong generalization across tools and architectures: In contrast to the baseline method R-Zero, which frequently suffers from performance regressions during training, Marvo demonstrates more stable, near-monotonic growth across iterations. Furthermore, this mechanism is versatile—it applies not only to complex Python code interpreters but also to lightweight tools like image cropping and zooming—and consistently delivers performance gains across models of varying parameter scales.
Critical weaknesses
  1. Relatively early-stage models evaluated, leaving scalability to larger models unverified: In terms of horizontal comparisons across self-evolution mechanisms, the paper relies almost exclusively on R-Zero, an early baseline adapted from text-only reasoning tasks. While several frontier models and multimodal visual agents (e.g., DeepEyes V2, PyVision-RL) are included as reference points, the study lacks side-by-side, multi-round dynamic evolution comparisons under identical experimental settings against more concurrent multimodal self-evolution or reinforcement learning frameworks, making the baselines somewhat limited in breadth and recency. Furthermore, given the adoption of Qwen2.5-VL-7B, it remains an open question whether newer model generations (e.g., Qwen3.5) or Mixture-of-Experts (MoE) architectures would exhibit different dynamics. In particular, if a base model already suffers from low policy entropy, whether the Marvo framework can still provide effective training signals remains to be seen.

  2. Severe degradation and distribution shift in synthetic data and pseudo-label quality across rounds: Evaluations using the Qwen3.8-27B judge reveal that as both the challenger and solver drift away from the true data distribution over multiple interaction rounds, the accuracy of proposed questions and pseudo-labels plummets drastically from 68.5% in Round 1 to 43.6% in Round 8, leading to severe deterioration in training signal quality during later stages. This presents a substantial challenge to the long-term reliability of the synthesized data.

Questions for authors
  1. Given that a four-option multiple-choice question carries a 25% probability of success through random guessing, does this introduce significant variance-related noise into the calculation of marginal rewards? In real-world scenarios, interactions involving visual agents are predominantly open-ended and generative—such as open-set question answering, complex coordinate prediction, or multi-step tool manipulation—rather than strictly multiple-choice. Can this framework enable models to learn how to invoke tools to arrive at the correct answer for open-ended tasks?

  2. Does the "drift" observed in the challenger and solver over later stages indicate a theoretical upper limit to the current self-evolving paradigm? To ensure the sustainability of this paradigm—and to leverage the evolution of model capabilities for generating increasingly complex problems—how do the authors plan to develop effective calibration or filtering mechanisms to prevent the models from collapsing after multiple rounds of interaction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.