acceptodds
Under review as a conference paper at ICLR 2027

VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning

24 views

Abstract

Visual tool use enables vision-language models (VLMs) to acquire task-relevant evidence, but its effectiveness can be limited by predefined visual operations and the tool-use strategies learned during training. Representing these operations and strategies as reusable external skills allows them to be refined through self-evolution without updating model parameters. However, skill refinement can introduce performance regressions. Meanwhile, inspecting accumulated task execution trajectories together with skill evolution history may lead to evolution evidence overload. To address these challenges, we propose VisualEvolve, a self-evolving framework that evolves executable visual operations and tool-use strategies for VLMs through a Select–Execute–Evolve–Evaluate cycle. Skill-Library Snapshot Selection uses performance on a fixed validation set disjoint from execution tasks to select an effective historical snapshot for subsequent evolution, limiting the propagation of performance regressions. Progressive Evidence Access (PEA) guides capability-gap diagnosis through a coarse-to-fine hierarchy comprising the Skill-Performance Report, Evolution Log, Trajectory Summaries, and relevant raw trajectory segments. Across four VLM backbones and six evaluation sets from three benchmarks, VisualEvolve improves over the Initial Skill Library in 23 of 24 settings, with a maximum gain of 15.1 percentage points on VisualProbe Hard using Qwen3.5-Plus. Ablation studies support the contributions of Snapshot Selection and PEA, while further experiments show that skills can be improved using experience from different data settings and directly reused across backbones. Moreover, VisualEvolve enables effective skill self-evolution with over an order of magnitude fewer task-solving trajectories than estimated for representative RL-based methods. To facilitate future research, our code is available at https://anonymous.4open.science/r/VisualEvolve-6E80/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Reasoning, Vision-Language Models, Agent Skills, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_veV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VisualEvolve, a framework that improves visual reasoning by updating an external skill library while keeping the VLM frozen. It uses task outcomes and execution traces to revise tool-use instructions and, when needed, executable visual operations. Each round starts from the best historical library on a fixed validation set, and the agent inspects feedback through reports, summaries, and raw traces. Across four backbones and three benchmarks, the evolved library improves over the initial library in 23 of 24 settings. Ablations support both design choices, and transfer experiments show that evolved skills remain useful across backbones.

Critical strengths
  • The improvements are consistent across backbones. The evolved library outperforms the initial library in 23 of 24 settings. This provides useful evidence that updating external skills can improve visual reasoning without changing model parameters.

  • The evolved skills transfer across models. A library developed with Kimi-K2.5 improves all three other backbones without modification. This suggests that the skills capture useful procedures beyond the model used to develop them.

Critical weaknesses
  1. The added value over existing skill-evolution methods is unclear. Dynamo [1] already evolves visual skills and tools, while Meta-Harness [2] uses historical code, scores, and traces to guide updates. The paper compares mainly against an initial skill library and training-based systems, leaving its advantage over these closer approaches unresolved.

  2. The experiments do not directly test the evidence-overload claim. The ablations show that PEA improves accuracy, but do not show whether it helps the agent find relevant information as more history accumulates. This supports PEA’s usefulness, while leaving its stated motivation only partly tested.

[1] Sun et al. Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agents. 2026. https://arxiv.org/abs/2606.30185

[2] Lee et al. Meta-Harness: End-to-End Optimization of Model Harnesses. 2026. https://arxiv.org/abs/2603.28052

Questions for authors
  • What evidence shows that VisualEvolve improves over existing feedback-driven skill-refinement approaches when using the same backbone, data, and comparable optimization budget?

  • What evidence shows that PEA helps the agent find relevant evidence more effectively as the evolution history grows?

4 comments
AuthorsOpenReview ↗

Response to Reviewer veV1 (1/3): W1

We sincerely thank you for the careful review and for recognizing the consistent gains across backbones and the cross-backbone transfer of the evolved skills. Line numbers refer to the revised PDF, where new content is marked in blue.

  • W1: The added value over existing skill-evolution methods is unclear. Dynamo [1] already evolves visual skills and tools, while Meta-Harness [2] uses historical code, scores, and traces to guide updates. The paper compares mainly against an initial skill library and training-based systems, leaving its advantage over these closer approaches unresolved.

Thank you for this comment. We would like to clarify two points. The comparison with these closer approaches is reported in our response to Q1 (next comment).

  1. Dynamo [1] and Meta-Harness [2] were published after the ICLR submission deadline. The two works that you cite as arXiv preprints were formally published in October 2026, Dynamo at EMNLP 2026 and Meta-Harness at COLM 2026. In addition, we found no open-source implementation of Dynamo, so we cannot reproduce Dynamo. However, to address your concerns, we additionally conducted two visual skill evolution experiments with Meta-Harness and SkillHone under the same backbone, data, and optimization budget. We report the results in our response to Q1 (next comment).

  2. The added value of VisualEvolve lies in two mechanisms that address two important challenges of visual skill evolution. Skill updates can introduce performance regressions, and the accumulated trajectories and evolution history can cause evolution evidence overload (Sec. 1, L053–079). To address these two challenges, we propose Skill-Library Snapshot Selection and Progressive Evidence Access (PEA) (Sec. 1, contributions, L108–111). Below, we contrast each mechanism with Meta-Harness and Dynamo.

For Skill-Library Snapshot Selection, we would like to clarify three points.

  • Snapshot Selection does not select the final skill library at the end of evolution, which every variant in our ablation studies (Table 2) already does. Snapshot Selection selects the starting skill library of every round. At the beginning of each round, VisualEvolve selects the archived snapshot with the highest validation accuracy, instead of always continuing from the latest snapshot (Sec. 3.3, L194–196). In contrast, Meta-Harness lets its proposer decide which prior candidate to start from.

  • Snapshots whose validation accuracy drops are not discarded. These snapshots stay in the archive and serve as diagnostic evidence. In contrast, Dynamo keeps a candidate update only if the candidate improves accuracy on Dynamo's training cases, and discards every other candidate.

  • Snapshot Selection is necessary in multi-round evolution and complementary to PEA. Removing Snapshot Selection lowers macro-average accuracy from 77.6% to 72.2%. Without the Skill-Performance Report and the Evolution Log of PEA, Snapshot Selection reaches only 71.8% (Table 2, row "w/o SPR + EL"), close to the 71.2% of the variant with neither Snapshot Selection nor these two components (row "w/o Evaluate").

For PEA, we would like to clarify three points.

  • Meta-Harness gives its proposer access to the source code, scores, and traces of prior harness candidates (Sec. 2, L138–139), and the proposer accesses this history directly. PEA targets the drawback of accessing the accumulated evolution history directly. As the evolution records accumulate, inspecting all the records at once can cause evolution evidence overload (Sec. 1, L077–079).

  • To address this drawback, PEA organizes the evidence into a coarse-to-fine hierarchy. The Self-Evolving Agent first reads the Skill-Performance Report, then the Evolution Log, then the Trajectory Summaries, and accesses raw trajectory segments only when finer-grained evidence is needed (Sec. 3.5, L218–221).

  • PEA is necessary and complementary to Snapshot Selection. Removing PEA lowers macro-average accuracy from 77.6% to 71.3% at nearly the same token consumption, and PEA without Snapshot Selection reaches only 72.2% (Table 2, row "w/o Selection"). Our response to W2 & Q2 shows what the Self-Evolving Agent reads as the evolution history grows, with PEA and without PEA.

We have added a discussion of Dynamo to Sec. 2 (L135–137).

AuthorsOpenReview ↗

Response to Reviewer veV1 (2/3): Q1

  • Q1: What evidence shows that VisualEvolve improves over existing feedback-driven skill-refinement approaches when using the same backbone, data, and comparable optimization budget?

Thank you for this question. As clarified in our response to W1, Dynamo and Meta-Harness were published after the ICLR submission deadline, and we found no open-source implementation of Dynamo. Nevertheless, to address your concerns, we additionally conducted experiments with Meta-Harness and SkillHone (another self-evolution method discussed in Sec. 2) under the same backbone, the same data, and the same optimization budget (new, Appendix B, Table 3), and with three backbones without a budget cap (new, Table 1). In all experiments, each method uses one backbone for all agents, and the three methods share the same Initial Skill Library, evolution data, and 200-example validation set. Each method is tested with the snapshot that has the highest validation accuracy in that method's own run.

Same optimization budget (Table 3, Qwen3.5-Plus, 200M tokens counting all input and output tokens). Accuracy (%).

MethodTokens (M)RoundsV*BenchHRBench 4KHRBench 8KVisualProbe EasyVisualProbe MediumVisualProbe HardMacro Avg.
Meta-Harness200789.588.487.070.248.546.271.6
SkillHone2003986.487.085.568.848.142.569.7
VisualEvolve192.731692.191.090.077.360.454.777.6

Three backbones, without a budget cap (Table 1, where Meta-Harness and SkillHone run 20 rounds and VisualEvolve runs 16 rounds). Accuracy (%).

BackboneMethodV*BenchHRBench 4KHRBench 8KVisualProbe EasyVisualProbe MediumVisualProbe HardMacro Avg.
Qwen3.5-PlusInitial Skill Library86.487.083.368.148.139.668.8
Qwen3.5-PlusSkillHone85.988.086.967.447.041.569.5
Qwen3.5-PlusMeta-Harness90.189.088.472.351.949.173.5
Qwen3.5-PlusVisualEvolve92.191.090.077.360.454.777.6
Qwen3.6-PlusInitial Skill Library90.688.587.466.753.436.870.6
Qwen3.6-PlusSkillHone89.589.088.567.448.538.770.3
Qwen3.6-PlusMeta-Harness90.691.590.471.655.646.274.3
Qwen3.6-PlusVisualEvolve91.192.090.973.858.250.076.0
Doubao-Seed-2.0-ProInitial Skill Library89.586.886.075.959.042.573.3
Doubao-Seed-2.0-ProSkillHone89.087.587.077.360.445.374.4
Doubao-Seed-2.0-ProMeta-Harness89.587.987.578.061.648.175.4
Doubao-Seed-2.0-ProVisualEvolve90.689.489.079.463.850.977.2
  1. Under the same backbone, data, and budget, VisualEvolve achieves the highest accuracy on all six evaluation sets (Table 3). In macro-average accuracy, VisualEvolve exceeds Meta-Harness by 6.0 points and SkillHone by 7.9 points.

  2. The gains are largest on the VisualProbe sets (Table 3). VisualEvolve exceeds Meta-Harness by 11.9 points on VisualProbe Medium, and exceeds SkillHone by 12.3 points on VisualProbe Medium and by 12.2 points on VisualProbe Hard.

  3. Meta-Harness and SkillHone still trail VisualEvolve without the 200M-token budget. Meta-Harness completes only 7 rounds within 200M tokens because Meta-Harness generates three candidate revisions per round (Sec. 4.1, L313–316). Without a budget cap, the 20-round run of Meta-Harness with Qwen3.5-Plus uses 594M tokens, about three times the 192.73M tokens of VisualEvolve, and Meta-Harness still trails VisualEvolve by 4.1 points. The 20-round run of SkillHone uses 115M tokens and reaches 69.5%, close to the 69.7% that SkillHone reaches under 200M tokens (Table 1, Appendix B).

  4. Without a budget cap, VisualEvolve achieves the highest accuracy on all 18 combinations of three backbones and six evaluation sets (Table 1).

We have added the comparison across three backbones to Sec. 4.1–4.2 and Table 1, and the comparison under a matched token budget to Appendix B (Table 3).

AuthorsOpenReview ↗

Response to Reviewer veV1 (3/3): W2 & Q2

  • W2 & Q2: The experiments do not directly test the evidence-overload claim. The ablations show that PEA improves accuracy, but do not show whether it helps the agent find relevant information as more history accumulates. What evidence shows that PEA helps the agent find relevant evidence more effectively as the evolution history grows?

Thank you for this question. To address your concern, we tested the evidence-overload claim more directly. We additionally analyzed what the Self-Evolving Agent ("the agent" below) reads, and how the updates of the agent turn out, as more history accumulates, with Progressive Evidence Access (PEA) and without PEA (new, Appendix C, Table 6). The results show that PEA helps the agent find relevant evidence more effectively as the evolution history grows.

This analysis uses Qwen3.5-Plus as the backbone of all agents, as in our ablation studies (Sec. 4.3). Over 16 rounds, the archived evolution evidence grows nearly 40×, from ≈71K to ≈2.83M tokens (Appendix C, Table 4). Table 6 compares rounds 1–8 with rounds 9–16. The columns "VisualEvolve" are the run with PEA. In the columns "w/o PEA", the agent has access to the same validation-accuracy history, skill-change history, and raw trajectories, but accesses these evolution records directly (Sec. 4.3, L414–416).

Metric (Table 6, Qwen3.5-Plus)w/o PEA, rounds 1–8w/o PEA, rounds 9–16VisualEvolve, rounds 1–8VisualEvolve, rounds 9–16
Mean context length (tokens)47,51248,29441,08442,768
Raw trajectory segments accessed per round7.37.14.43.4
Historical skill diffs inspected per round1.31.91.84.4
Trajectory summaries accessed per round––10.49.5
Updates improving on their starting library3 of 81 of 83 of 83 of 8
Best-so-far validation accuracy at the end (%)64.065.068.071.5

A historical skill diff is a recorded historical modification of a skill. An update improves on its starting library if the updated skill library has a higher validation accuracy than the snapshot from which the update starts. "–" denotes unavailable evidence.

  1. With PEA, the agent inspects more historical skill diffs as more history accumulates. From rounds 1–8 to rounds 9–16, the number of historical skill diffs inspected per round rises from 1.8 to 4.4 with PEA, but only from 1.3 to 1.9 without PEA.

  2. With PEA, the agent reads Trajectory Summaries and accesses fewer raw trajectory segments. With PEA, the agent accesses about 10 trajectory summaries per round, and accesses 4.4 raw trajectory segments per round in rounds 1–8 and 3.4 in rounds 9–16. Without PEA, the agent accesses about 7 raw trajectory segments per round in both halves. This matches the coarse-to-fine access of PEA (Sec. 3.5, L218–221).

  3. PEA lowers the context burden. The mean context length is 41,084 and 42,768 tokens with PEA, compared with 47,512 and 48,294 tokens without PEA, although the agent with PEA inspects more historical skill diffs.

  4. The evidence that the agent finds with PEA is relevant, as judged by the updates. With PEA, the number of improving updates does not drop as more history accumulates. We judge the relevance of the evidence by the outcome of the updates. With PEA, 3 of the 8 updates improve on their starting library in both halves. Without PEA, this number drops from 3 of 8 in rounds 1–8 to 1 of 8 in rounds 9–16.

  5. The advantage of PEA grows as more history accumulates. The advantage of PEA in best-so-far validation accuracy is 4.0 points after round 8 (68.0% versus 64.0%) and 6.5 points after round 16 (71.5% versus 65.0%). In rounds 9–16, best-so-far validation accuracy rises by 3.5 points with PEA and by 1.0 point without PEA, at comparable total token consumption (192.7M tokens with PEA and 190.8M tokens without PEA, Appendix C).

In summary, without PEA, both consequences of evidence overload that we state in Sec. 1 (L077–079) appear, namely a higher context burden (point 3) and capability gaps that are harder to diagnose as more history accumulates (point 4). With PEA, the agent inspects more historical skill diffs and accesses fewer raw trajectory segments as more history accumulates (points 1 and 2), and the evidence that the agent finds leads to better updates (points 4 and 5).

We have added this analysis to Appendix C (Tables 4–6). Thank you again for your suggestions. We would be glad to discuss any further questions during the discussion period.

Dear Reviewer veV1,

Thank you very much for the time and effort you have devoted to reviewing our paper. We sincerely appreciate your helpful comments and suggestions.

We have carefully responded to each of your comments. In particular, we compared VisualEvolve with Meta-Harness and SkillHone under the same backbone, data, and optimization budget (W1 & Q1; Tables 1 and 3 of the revised PDF), and we directly tested the evidence-overload claim by analyzing what the Self-Evolving Agent reads as the evolution history grows, with and without PEA (W2 & Q2; Appendix C, Tables 4–6).

As the discussion deadline approaches, we would be very grateful if you could review our responses and let us know whether any concerns remain. We would welcome the opportunity to discuss them further and provide any additional clarification that may be helpful.

Thank you again for your time and consideration.

Best regards,
The authors

Reviewer_Bbs5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces VisualEvolve, a training-free framework that improves visual reasoning by analyzing execution trajectories and continuously updating external skills, without modifying model parameters.

Critical strengths
  • The experiments cover four VLMs (including frontier models like K3), multiple benchmarks, cross-model transfer, and cost analysis. The performance improvements are also quite promising.

  • The framework can automatically improve tool-use strategies and even develop new visual operations without updating model weights, making it a relatively low-cost and practical approach.

Critical weaknesses
  • The overall framework is quite general, with few components specifically designed for vision.

    • Snapshot Selection is essentially choosing the best-performing skill repository on a validation set, which is fairly standard.
    • This makes me wonder whether VisualEvolve is really a visual reasoning method, as the same framework could easily apply to coding agents, web agents, or general tool-using agents.
    • Applying a general method to vision can still be valuable, but the authors need to better explain why existing self-evolution methods are insufficient and what unique design choices VisualEvolve brings. The current evidence isn't very convincing.
  • The Skill Performance Score seems rather coarse.

    • It estimates a skill's contribution based on accuracy changes when that skill is loaded, but multiple skills are often used together, and their usage patterns may also change across versions.
    • For example, in an observe → search → crop → verify pipeline, it's hard to tell which skill actually led to the improvement.
    • Although the authors acknowledge that SPS is only an approximation, I'd like to see leave-one-skill-out or skill combination ablations to validate its reliability.
  • The paper mainly reports overall accuracy gains and which skills are added or updated across iterations.

    • While the appendix includes some skill-loading statistics and examples, these are mostly descriptive.
    • I'd like to see a more systematic analysis of how skill updates change visual search behavior, tool-call frequency, localization quality, visual evidence gathering, and error correction.
    • In particular, it's unclear whether the gains come from better tool-use strategies or newly introduced visual operations. Right now, we can see that the skills change and accuracy improves, but the connection between the two remains unclear.
Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bbs5 (1/4): W1, first part

We sincerely thank you for the detailed and constructive review and for recognizing the promising gains across four VLM backbones and six evaluation sets, the cross-backbone transfer of the evolved skills, and the practical value of improving tool-use strategies and developing new visual operations without updating model weights. Table and line numbers refer to the revised PDF, where new content is marked in blue. Table 7 was Table 3 in the original submission.

  • W1 (first part): The overall framework is quite general, with few components specifically designed for vision. Snapshot Selection is essentially choosing the best-performing skill repository on a validation set, which is fairly standard. This makes me wonder whether VisualEvolve is really a visual reasoning method, as the same framework could easily apply to coding agents, web agents, or general tool-using agents.

We would like to clarify that we do not claim a general framework. Our paper focuses on two challenges of visual skill evolution. Skill updates can introduce performance regressions, and the accumulated evolution history can cause evolution evidence overload (Sec. 1, L053–079).

1. Is VisualEvolve really a visual reasoning method? We understand your concern that the same framework could apply to coding agents, web agents, or general tool-using agents. VisualEvolve differs in three respects.

  • Input. Every task is a visual reasoning task that consists of an image and a question (Sec. 4.1).

  • Output. Solving a task requires visual reasoning, and the resulting execution trajectory contains images, such as the images produced by the crop operation. The Self-Evolving Agent inspects these images during capability-gap diagnosis (Evolve Meta-Skill, Appendix I.2).

  • Object of evolution. VisualEvolve evolves tool-use strategies and executable visual operations. Each skill consists of a tool-use strategy and an optional executable visual operation (Sec. 3.3). The evolved skill library adds tool-use strategies such as observe, search, and verify, and for rotated or flipped images it adds two new executable visual operations, rotate and flip (Table 7).

2. Is Snapshot Selection fairly standard? Snapshot Selection does more than choose the best-performing skill repository on a validation set. There are three key features:1)Every evolution round starts from the archived snapshot with the highest validation accuracy; 2)A snapshot whose validation accuracy drops remains in the archive as evolution evidence, and 3)Snapshot Selection is complementary to our second mechanism, Progressive Evidence Access. We will explain these three characteristics in our next comment, which answers your question about our unique design choices.

AuthorsOpenReview ↗

Response to Reviewer Bbs5 (2/4): W1, second part

  • W1 (second part): Applying a general method to vision can still be valuable, but the authors need to better explain why existing self-evolution methods are insufficient and what unique design choices VisualEvolve brings. The current evidence isn't very convincing.

Thank you for this suggestion. We first show with new experiments why existing self-evolution methods are insufficient, and we then explain our unique design choices.

1. Why are existing self-evolution methods insufficient? To address your concern, we conducted new experiments with two existing self-evolution methods, Meta-Harness and SkillHone (new Table 3 and new rows in Table 1). All methods share the same backbone, Initial Skill Library, evolution data, and validation set.

Table 3 (Qwen3.5-Plus, same budget of 200M tokens, accuracy in %).

MethodTokens (M)V*BenchHRBench 4KHRBench 8KVisualProbe EasyVisualProbe MediumVisualProbe HardMacro Avg.
Meta-Harness20089.588.487.070.248.546.271.6
SkillHone20086.487.085.568.848.142.569.7
VisualEvolve192.7392.191.090.077.360.454.777.6

Table 1 (three backbones, no token budget, macro-average accuracy in %).

MethodQwen3.5-PlusQwen3.6-PlusDoubao-Seed-2.0-Pro
Initial Skill Library68.870.673.3
SkillHone69.570.374.4
Meta-Harness73.574.375.4
VisualEvolve77.676.077.2

Under the same budget, SkillHone and Meta-Harness improve macro-average accuracy over the Initial Skill Library (68.8%) by only 0.9 and 2.8 points, while VisualEvolve improves macro-average accuracy by 8.8 points. The same ordering holds with all three backbones when no token budget is set.

2. What unique design choices does VisualEvolve bring? Multi-round skill evolution for visual reasoning faces two challenges. Skill updates can introduce performance regressions, and the accumulated evolution history can cause evolution evidence overload (Sec. 1, L053–079). We propose one mechanism for each challenge (Sec. 1, L108–111), and the two mechanisms are complementary.

Skill-Library Snapshot Selection addresses the performance regressions.

  • Every evolution round starts from the archived snapshot with the highest validation accuracy, not from the latest snapshot. In each round, the updated skill library is evaluated on the fixed validation set and archived as a new snapshot. If a skill update lowers validation accuracy, the next round does not start from this snapshot. Snapshot Selection selects the archived snapshot with the highest validation accuracy as the starting skill library (Sec. 3.3). In contrast, Meta-Harness lets its proposer decide which prior candidate to start from, and SkillHone continues from the last accepted revision.

  • A snapshot whose validation accuracy drops is kept in the archive, not discarded. The Evolution Log records the validation accuracy of each snapshot and the skill modifications between consecutive snapshots, so the Self-Evolving Agent can locate regressions and identify update directions to avoid (Sec. 3.5).

  • Snapshot Selection is necessary. Removing Snapshot Selection lowers macro-average accuracy from 77.6% to 72.2% (Table 2).

Progressive Evidence Access (PEA) addresses the evolution evidence overload.

  • Existing methods retain rich evolution histories. Meta-Harness gives its proposer access to the source code, scores, and traces of prior harness candidates, and SkillHone records diagnoses, revisions, and feedback (Sec. 2). As these records accumulate, inspecting them all at once can cause evolution evidence overload, which makes capability gaps harder to diagnose (Sec. 1, L077–079).

  • PEA lets the Self-Evolving Agent access the evidence from coarse to fine. The Self-Evolving Agent first identifies the relevant skills through the Skill-Performance Report, then traces the historical modifications of these skills through the Evolution Log, then analyzes the Trajectory Summaries of the current round, and accesses raw trajectory segments only when finer-grained evidence is needed (Sec. 3.5).

  • PEA is necessary. Removing PEA lowers macro-average accuracy from 77.6% to 71.3% (Table 2).

The two mechanisms are complementary. Snapshot Selection determines the starting skill library of each round, and PEA determines how the accumulated evidence is accessed. In Table 2, macro-average accuracy is 71.2% with neither Snapshot Selection nor the two history reports of PEA (the Skill-Performance Report and the Evolution Log), 71.8% with Snapshot Selection only, 72.2% with the two reports only, and 77.6% with both.

AuthorsOpenReview ↗

Response to Reviewer Bbs5 (3/4): W2

  • W2: The Skill Performance Score seems rather coarse. ... multiple skills are often used together, and their usage patterns may also change across versions. ... I'd like to see leave-one-skill-out or skill combination ablations to validate its reliability.

Thank you for this suggestion. To address your concern, we conducted the leave-one-skill-out ablation and a skill-combination ablation on the final skill library (new Table 9). We remove one evolved skill at a time, or crop and observe together, and evaluate each reduced skill library on the fixed validation set of 200 examples, on which the final skill library reaches 71.5% validation accuracy (Qwen3.5-Plus). Load rate is the percentage of validation examples on which the skill is loaded with the final skill library. SPS is computed from the Initial Skill Library to the final skill library. In the last row, the two values are for crop and for observe.

Skill removedLoad rate (%)SPSValidation acc. (%)Δ Acc.
observe35.5+5.665.0−6.5
verify10.5+9.567.0−4.5
search5.0+10.063.5−8.0
count6.5+38.565.5−6.0
color21.5+16.367.5−4.0
read_text15.0+10.066.5−5.0
crop + observe68.0 / 35.5+19.3 / +5.662.0−9.5

The table supports three observations, which together explain how reliable SPS is and how VisualEvolve uses it.

  1. The skill-removal results support SPS as an indicator of skill utility. All six evolved skills have a positive SPS, and removing any one of them lowers validation accuracy by 4.0 to 8.0 points.

  2. SPS provides only an approximate estimate of the contribution of an individual skill, as you note and as we state in Sec. 3.5. A higher SPS does not necessarily correspond to a larger accuracy drop after skill removal. For example, count has the largest SPS (+38.5), but removing search causes the largest drop among the six single-skill removals (−8.0). One reason is that multiple skills are loaded together, and skill removal changes how the remaining skills are used. Jointly removing crop and observe reduces validation accuracy by 9.5 points and increases the load rate of search from 5.0% to 27.0% (reported in the text below Table 9). As for the usage patterns that change across versions, SPS is computed on all validation examples on which the skill is loaded under at least one of the two compared snapshots (Sec. 3.5), so SPS takes such changes into account.

  3. Because SPS is an approximate estimate, VisualEvolve uses SPS only to prioritize skills for inspection, not to attribute an improvement to a single skill. In a pipeline such as observe → search → crop → verify, the Self-Evolving Agent prioritizes the skills with large absolute SPS for further inspection (Sec. 3.5). The Self-Evolving Agent then inspects the historical modifications of these skills in the Evolution Log and the Trajectory Summaries before updating the skill library. This skill screening is useful. Removing the Skill-Performance Report, which provides SPS, lowers macro-average accuracy from 77.6% to 73.6% (Table 2, row "w/o SPR").

Reviewer_KVDEOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

VisualEvolve is a framework for improving visual reasoning by evolving an external skill library while keeping the VLM backbone frozen. It follows a Select–Execute–Evolve–Evaluate cycle: selecting a historical skill library, collecting execution trajectories and correctness feedback, revising skill instructions and optional executable visual operations, and evaluating the updated library on a fixed validation set. Its two main components are Skill-Library Snapshot Selection, which chooses the historical library with the highest validation accuracy, and Progressive Evidence Access, which organizes performance reports, revision histories, trajectory summaries, and relevant raw traces for capability-gap diagnosis. Experiments across four VLM backbones and six evaluation sets show improvements over the initial skill library, accompanied by component ablations, cross-backbone transfer experiments, and cost analyses.

Critical strengths

Consistent empirical gains across multiple visual reasoning settings. The paper provides useful evidence that evolving an external skill library can improve visual reasoning without updating model parameters. Across four VLM backbones and six evaluation sets, VisualEvolve improves over the initial skill library in 23 of 24 settings. For example, the macro-average accuracy of Qwen3.5-Plus increases from 68.8% to 77.6% (Table 1). The ablation results also support the contributions of Snapshot Selection and Progressive Evidence Access within the framework (Table 2). These results give the approach practical value and support the effectiveness of the integrated workflow across the evaluated models and tasks.

Critical weaknesses

Limited methodological novelty and insufficient new insight beyond established agent practices. My main concern is that the framework largely combines familiar elements of agent systems: external skills, execution feedback, trajectory summarization, model-driven skill revision, and validation-based version selection. Snapshot Selection chooses the best historical skill library according to validation accuracy, while Progressive Evidence Access organizes diagnosis through performance reports, revision histories, summaries, and raw traces. Although these are sensible design choices, the paper does not clearly establish a sufficiently distinct methodological contribution or generalizable insight beyond applying and organizing these practices for visual reasoning.

This concern is reinforced by the lack of direct comparisons with closely related skill/harness optimization methods, such as SkillHone and Meta-Harness, which are discussed in Section 2. The internal ablations demonstrate that the selected components are useful within this implementation, but they do not establish the additional benefit over existing self-evolving agents under comparable models, feedback, and computational budgets. Consequently, the experiments demonstrate the practical effectiveness of this combination more clearly than they establish what new knowledge the paper contributes about agent skill evolution.

Questions for authors

Clarification of the contribution: What is the specific methodological advance over existing skill/harness optimization approaches such as SkillHone and Meta-Harness? In particular, what generalizable insight do Snapshot Selection and Progressive Evidence Access establish beyond validation-based version selection and structured use of execution history?

Comparative evidence that could affect the assessment: Can you provide evidence under matched backbones, feedback, and computational budgets comparing VisualEvolve with an existing skill-evolution method, or with a straightforward baseline that iteratively revises skills from execution feedback and retains the best validated version? This comparison would help determine the additional benefit of the proposed evidence-organization mechanism beyond the gains from performing multiple rounds of skill optimization.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KVDE (1/3): W1

We sincerely thank you for the constructive review and for recognizing the consistent empirical gains, the supporting ablations, and the practical value of the approach. Line numbers refer to the revised PDF, where new content is marked in blue.

  • W1: Limited methodological novelty. The framework largely combines familiar elements of agent systems (external skills, execution feedback, trajectory summarization, model-driven skill revision, and validation-based version selection).

Thank you for this comment. We would like to clarify that we do not claim the familiar elements that you list as contributions. Our paper focuses on two important challenges of visual skill evolution. Skill updates can introduce performance regressions, and the accumulated trajectories and evolution history can cause evolution evidence overload (Sec. 1, L053–079). To address these two challenges, we propose Skill-Library Snapshot Selection and Progressive Evidence Access (PEA) (Sec. 1, contributions, L108–111).

To show that the gains do not come from the familiar elements, we refer to our ablation studies, which separate the familiar elements from the two proposed mechanisms (Table 2, Qwen3.5-Plus, macro-average accuracy in % over the six evaluation sets). In the table, "✓" means kept and "×" means removed. SPR, EL, and TS denote the three components of PEA, namely the Skill-Performance Report, the Evolution Log, and the Trajectory Summaries. The row "Single-Pass" queries the backbone in a single turn without external tools. The Evolve stage, in which the skills are revised, is kept in every other row.

VariantSelectExecuteEvaluateSPRELTSMacro Avg.
w/o Selection×✓✓✓✓✓72.2
w/o Execute✓×✓✓✓×69.9
w/o Evaluate×✓×××✓71.2
w/o SPR✓✓✓×✓✓73.6
w/o EL✓✓✓✓×✓75.0
w/o TS✓✓✓✓✓×73.0
w/o SPR + EL✓✓✓××✓71.8
w/o PEA✓✓✓×××71.3
Single-Pass××××××64.7
VisualEvolve✓✓✓✓✓✓77.6

In this table, the row "w/o Evaluate" is the system that is built from the five elements that you list.

  • External skills. The skills are kept in a skill library outside the frozen VLM.

  • Execution feedback. The Execute stage is kept, so the Worker (the agent that solves each task) solves the evolution tasks and returns trajectories with correctness labels.

  • Trajectory summarization. The Trajectory Summaries (TS) are kept.

  • Model-driven skill revision. The Self-Evolving Agent (the agent that diagnoses failures and revises the skill library) revises the skills in every round.

  • Validation-based version selection. After evolution ends, the snapshot with the highest validation accuracy is returned (Table 2 note, L395–397). The "×" under Select only means that no snapshot is selected at the start of each round.

However, the familiar elements alone are not sufficient. With all five familiar elements, the row "w/o Evaluate" reaches 71.2%, only 2.4 points above the Initial Skill Library (68.8%, Table 1), whereas VisualEvolve reaches 77.6%.

The remaining 6.4 points come from the two components that the row "w/o Evaluate" removes. The first component is Snapshot Selection, which starts every round from the archived snapshot with the highest validation accuracy instead of only returning this snapshot at the end. The second component consists of the Skill-Performance Report and the Evolution Log of PEA (SPR and EL). Both components rely on the Evaluate stage, which evaluates every new snapshot on the validation set.

In summary, our methodological contribution is the two proposed mechanisms, not the familiar elements. The familiar elements account for 2.4 of the 8.8 points that VisualEvolve gains over the Initial Skill Library, and the remaining 6.4 points come from Snapshot Selection, the Skill-Performance Report, and the Evolution Log. Our response to Q1 explains the two mechanisms.

AuthorsOpenReview ↗

Response to Reviewer KVDE (2/3): Q1

  • Q1: What is the specific methodological advance over SkillHone and Meta-Harness? What generalizable insight do Snapshot Selection and Progressive Evidence Access (PEA) establish beyond validation-based version selection and structured use of execution history?

Thank you for this question. The specific methodological advance over SkillHone and Meta-Harness is the two mechanisms, Snapshot Selection and PEA. Snapshot Selection shows that the starting skill library of each round matters. PEA shows that a growing evolution history has to be organized from coarse to fine.

Snapshot Selection, beyond validation-based version selection. We would like to clarify three points.

  • To address performance regressions, we propose Snapshot Selection. Snapshot Selection does not simply select the final skill library at the end of evolution, but selects the starting skill library of every round. At the beginning of each round, VisualEvolve selects the archived snapshot with the highest validation accuracy, instead of always continuing from the latest snapshot (Sec. 3.3, L194–196). In contrast, Meta-Harness lets its proposer decide which prior candidate to start from, and SkillHone continues from the last accepted revision.

  • Snapshots whose validation accuracy drops are not used as starting points, but are not discarded. These snapshots stay in the archive, and the Evolution Log uses these snapshots to show which historical skill modifications lowered validation accuracy (Sec. 3.5, L249–251). In contrast, SkillHone keeps a revision only when the revision passes its acceptance check.

  • Table 2 shows that Snapshot Selection is necessary in multi-round evolution and complementary to PEA. Removing Snapshot Selection lowers macro-average accuracy from 77.6% to 72.2% (row "w/o Selection"), and Snapshot Selection without the Skill-Performance Report and the Evolution Log reaches only 71.8% (row "w/o SPR + EL").

PEA, beyond structured use of execution history. Snapshot Selection decides where each round starts, and PEA decides how the accumulated evolution history is accessed.

  • PEA targets a drawback of accessing this history directly. Meta-Harness gives its proposer access to the source code, scores, and traces of prior candidates, and SkillHone records diagnoses, revisions, and feedback (Sec. 2, L137–139). However, as the evolution records accumulate, inspecting all the records at once can cause evolution evidence overload, which increases the context burden and makes capability gaps harder to diagnose (Sec. 1, L077–079).

  • To address this drawback, PEA organizes the evidence into a coarse-to-fine hierarchy. The Self-Evolving Agent reads the Skill-Performance Report first, then the Evolution Log, then the Trajectory Summaries, and raw trajectory segments only when finer-grained evidence is needed (Sec. 3.5, L218–221).

  • Table 2 shows that PEA is necessary and complementary to Snapshot Selection. Removing PEA lowers macro-average accuracy from 77.6% to 71.3% (row "w/o PEA").

To further address your concern about the insight of PEA, we additionally analyzed what the Self-Evolving Agent reads as the evolution history grows, with PEA and without PEA (new Table 6). The columns "VisualEvolve" are the run with PEA, and in the columns "w/o PEA" the Self-Evolving Agent accesses the same records directly (Sec. 4.3).

Metric (Table 6, Qwen3.5-Plus)w/o PEA, rounds 1–8w/o PEA, rounds 9–16VisualEvolve, rounds 1–8VisualEvolve, rounds 9–16
Mean context length (tokens)47,51248,29441,08442,768
Raw trajectory segments accessed per round7.37.14.43.4
Historical skill diffs inspected per round1.31.91.84.4
Trajectory summaries accessed per round––10.49.5
Updates improving on their starting library3 of 81 of 83 of 83 of 8
Best-so-far validation accuracy at the end (%)64.065.068.071.5

A historical skill diff is a recorded historical modification of a skill. An update improves on its starting library if the update raises validation accuracy above the snapshot from which the update starts.

  1. Without PEA, the Self-Evolving Agent inspects fewer than 2 historical skill diffs per round (1.3 and 1.9), and the number of updates that improve on their starting library drops from 3 of 8 to 1 of 8.

  2. With PEA, the Self-Evolving Agent inspects 1.8 historical skill diffs per round in rounds 1–8 and 4.4 in rounds 9–16, accesses fewer raw trajectory segments than without PEA, and 3 of the 8 updates improve on their starting library in both halves.

In summary, Snapshot Selection goes beyond validation-based version selection by selecting the starting skill library of every round, and PEA goes beyond structured use of execution history by organizing the growing evolution history from coarse to fine.

We have added this analysis to Appendix C (Tables 4–6).

AuthorsOpenReview ↗

Response to Reviewer KVDE (3/3): W2 & Q2

  • W2 & Q2: The paper lacks direct comparisons with closely related skill/harness optimization methods, such as SkillHone and Meta-Harness, under comparable models, feedback, and computational budgets. Can you provide evidence under matched backbones, feedback, and computational budgets comparing VisualEvolve with an existing skill-evolution method, or with a straightforward baseline that iteratively revises skills from execution feedback and retains the best validated version?

Thank you for this constructive suggestion. We would like to clarify that SkillHone and Meta-Harness were arXiv preprints when we submitted our paper. SkillHone was formally published at EMNLP 2026 (Industry Track) and Meta-Harness at COLM 2026, both in October 2026, after the ICLR submission deadline. For this reason, our original submission discusses both methods in Sec. 2 without an experimental comparison.

Nevertheless, to address your concerns, we additionally conducted experiments with SkillHone and Meta-Harness under matched backbones, feedback, and computational budgets (new Table 3 in Appendix B of the revised PDF). All three methods use Qwen3.5-Plus for all agents, the same Initial Skill Library, the same evolution data, and the same 200-example validation set. Each method receives an evolution budget of 200M tokens and is tested with the snapshot that has the highest validation accuracy in the run of that method. The table reports accuracy (%).

MethodTokens (M)V*BenchHRBench 4KHRBench 8KVisualProbe EasyVisualProbe MediumVisualProbe HardMacro Avg.
Meta-Harness20089.588.487.070.248.546.271.6
SkillHone20086.487.085.568.848.142.569.7
VisualEvolve192.7392.191.090.077.360.454.777.6
  1. VisualEvolve outperforms the two existing self-evolution methods under matched backbones, feedback, and computational budgets. VisualEvolve achieves the highest accuracy on all six evaluation sets. In macro-average accuracy, VisualEvolve exceeds Meta-Harness by 6.0 points and SkillHone by 7.9 points. SkillHone remains close to the Initial Skill Library because the Worker rarely uses the scripts and reference documents that SkillHone adds, and 99.9% of the operation calls of the Worker still invoke the original crop script (new, Appendix B).

  2. The straightforward baseline that you describe is already in our ablation studies (Table 2, row "w/o Evaluate", Qwen3.5-Plus). This baseline iteratively revises the skills from execution feedback and retains the best validated version after evolution ends (Table 2 note, L395–397). This baseline reaches 71.2% macro-average accuracy, and VisualEvolve reaches 77.6%.

  3. The same ablation studies give the additional benefit of the evidence-organization mechanism beyond the gains from performing multiple rounds of skill optimization. Multiple rounds of skill optimization alone raise accuracy by 2.4 points, from 68.8% with the Initial Skill Library to 71.2% with the straightforward baseline. To isolate the benefit of the evidence-organization mechanism, PEA, we compare VisualEvolve with the row "w/o PEA" of Table 2, which differs from VisualEvolve only in PEA. Without PEA, VisualEvolve reaches 71.3%, close to the 71.2% of the straightforward baseline. PEA raises accuracy by 6.3 points, from 71.3% to 77.6%. The run without PEA and the run with PEA both perform 16 rounds and consume nearly the same number of tokens (190.8M and 192.7M), so this gain does not come from more rounds or more tokens.

In summary, VisualEvolve outperforms the two existing self-evolution methods under matched backbones, feedback, and computational budgets, and outperforms the straightforward baseline by 6.4 points. The evidence-organization mechanism accounts for 6.3 points beyond the gains from performing multiple rounds of skill optimization.

We have added the comparison with SkillHone and Meta-Harness to Sec. 4.1–4.2, Table 1, and Appendix B (Table 3). Thank you again for your comments.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for the constructive feedback and for recognizing the consistent gains across four VLM backbones and six evaluation sets (veV1, Bbs5, KVDE), the cross-backbone transfer of the evolved skills (veV1, Bbs5), and the practical value of improving visual reasoning without updating model weights (Bbs5, KVDE). We respond to each reviewer below the corresponding review. Here we summarize the main changes in the revised PDF, where new content is marked in blue. Table and line numbers in our responses refer to the revised PDF, in which Tables 3–6, 8, and 9 are new.

1. Comparison with existing self-evolution methods (veV1, Bbs5, KVDE). Dynamo, Meta-Harness, and SkillHone first appeared as arXiv preprints and were formally published in October 2026, after the ICLR submission deadline. In addition, we found no open-source implementation of Dynamo. Nevertheless, to address this shared concern, we conducted new experiments with Meta-Harness and SkillHone (new Table 3 in Appendix B). All three methods use Qwen3.5-Plus for all agents, the same Initial Skill Library, the same evolution data, the same validation set, and an evolution budget of 200M tokens. The table reports accuracy (%).

MethodTokens (M)V*BenchHRBench 4KHRBench 8KVisualProbe EasyVisualProbe MediumVisualProbe HardMacro Avg.
Meta-Harness20089.588.487.070.248.546.271.6
SkillHone20086.487.085.568.848.142.569.7
VisualEvolve192.7392.191.090.077.360.454.777.6

VisualEvolve achieves the highest accuracy on all six evaluation sets. In macro-average accuracy, VisualEvolve exceeds Meta-Harness by 6.0 points and SkillHone by 7.9 points. We have also added both methods to Table 1 for three backbones (Qwen3.5-Plus, Qwen3.6-Plus, and Doubao-Seed-2.0-Pro), where VisualEvolve again achieves the highest accuracy on every evaluation set.

2. What our paper contributes (veV1, Bbs5, KVDE). Our paper focuses on two important challenges of visual skill evolution, and proposes one mechanism for each challenge (Sec. 1). We do not claim a general-purpose agent framework.

  • Skill-Library Snapshot Selection addresses performance regressions. Every round starts from the archived snapshot with the highest validation accuracy, not from the latest snapshot. Removing Snapshot Selection lowers macro-average accuracy from 77.6% to 72.2% (Table 2).

  • Progressive Evidence Access (PEA) addresses evolution evidence overload. PEA organizes the accumulated evolution history from coarse to fine. Removing PEA lowers macro-average accuracy from 77.6% to 71.3% (Table 2).

3. New analyses requested by the reviewers. We have added the following analyses to the revised PDF.

  • What the Self-Evolving Agent reads as the evolution history grows, with PEA and without PEA (veV1, KVDE, Appendix C, Tables 4–6).

  • Leave-one-skill-out and skill-combination ablations (Bbs5, Appendix D, Table 9).

  • Inference behavior before and after skill evolution (Bbs5, Appendix D, Table 8).

In summary, VisualEvolve addresses two challenges of skill evolution for visual reasoning, performance regressions with Skill-Library Snapshot Selection and evidence overload with Progressive Evidence Access. We hope that these clarifications and new results address your concerns, and we would appreciate it if you could reconsider your assessment in light of these results. We are happy to discuss any further questions.

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.