acceptodds
Under review as a conference paper at ICLR 2027

ViewCraft: Evolving What Multimodal Agents See and How They Read It

Abstract

Reusable skills improve multimodal agents without updating model weights, but their effectiveness depends on both the evidence presented and the procedure used to read it. We introduce ViewCraft, which jointly evolves an executable evidence interface and its reading procedure. The interface selects, transforms, and organizes observations; the reader interprets the resulting views. Failure traces guide revisions, replay and paired evaluation retain useful changes, and interface projection recovers visual edits from eligible rejected joint proposals. Across eight benchmarks, ViewCraft improves over direct execution by an average of points in task score, with gains on every benchmark. The advantage persists at a matched input-token budget. On all five multi-source tasks, the learned interface uses – fewer input tokens than Initial skill. The learned skills also transfer across executors and outperform direct execution in every evaluated transfer, supporting evidence construction and reading as a coupled, reusable skill.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agents, Skill Evolution, Visual Skills

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ko22OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ViewCraft, a training-free framework that improves frozen multimodal agents by jointly evolving an executable evidence interface and the procedure used to interpret its outputs. The interface controls evidence selection, transformation, and presentation, while how to reason over that evidence is also specified. Optimization combines failure-guided proposals, visual replay, paired development evaluation, and interface projection. Useful visual changes from rejected joint proposals are retained for further refinement. Across 8 image, document, and video benchmarks, ViewCraft improves the average task score over direct inference by 6.1 points. Ablations and cross-executor experiments further examine component interactions and skill transfer.

Critical strengths

A well-motivated approach to multimodal agent improvement. The paper distinguishes between making relevant evidence available and interpreting it correctly. Jointly optimizing these two components is practically meaningful. Substantial experimental controls support the main result. The evaluation includes text-only and interface-only optimization, alternative search schedules, and adapted MPO and Dynamo baselines within a shared search space. The token-matched Direct and self-consistency controls also help establish that the gains are not explained simply by providing more input or sampling more answers (Tables 2~4).

Critical weaknesses

The comparison with related methods has a limited scope. For example, GEPA is not compared in the experiments. MPO and Dynamo are evaluated through adaptations to ViewCraft’s interface and evaluation protocol, while it establishes an advantage over those particular adaptations rather than over the complete original systems. A clearer account of which original capabilities are preserved or restricted would strengthen the comparative conclusions.

Questions for authors
  1. For the MPO and Dynamo baselines, which components of the original algorithms are retained, and which are changed to fit the shared interface? Could these adaptations materially affect their relative performance?
  2. What limitations do you anticipate when applying ViewCraft to smaller VLMs? In particular, could weaker evidence interpretation or self-diagnosis require a different division of work between the optimizer and executor?
  3. Could you provide further practical guidance on the performance–cost tradeoff, including deployment costs and search costs relative to the adapted baselines?
Reviewer_vUiNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ViewCraft optimizes a reusable reading procedure together with an executable interface that selects, transforms, and arranges visual evidence for a frozen multimodal model. Failure traces guide proposals, replay filters visually unchanged edits, and paired development evaluation selects updates. When an eligible joint proposal fails, interface projection retests its visual edits with the incumbent reader. Across eight benchmark subsets, the reported mean task score rises from 0.631 for Direct to 0.692 for ViewCraft. The paper tests alternative search strategies, reader/interface combinations, answer-format controls, input-token matching, and transfer across three executors on four tasks.

Critical strengths
  1. The paper evaluates an executable evidence interface against several useful controls. ViewCraft improves over Direct on all eight tasks in Table 2 and retains a reported 5.2-point advantage over input-token-matched Direct in Table 23. Tables 3-4 compare restricted, alternating, random, and adapted MPO/Dynamo searches. These comparisons provide evidence that the gains depend on the learned interface and read ing procedure, beyond the amount of input alone.

  2. The interventions examine the proposed explanation for the gains. Terminal crossings, alternative coordinate partitions, fixed-trajectory transplants, and contract controls test dependence between the learned components (Fig. 5c; Tables 16-19). Tables 15 and 24 further test the retention mechanisms; removing projection reduces the reported mean score by about 0.8 points. The inclusion of regressions and deployment costs helps define the method's limits.

Critical weaknesses
  1. Parts of the score accounting and statistical analysis need to be reconciled with the stated protocol. Table 5 says that self-transfer pools three runs. For VideoHolmes, 183 binary test items imply a denominator of 549, yet the Sonnet self-transfer score of 0.507 cannot be obtained by directly rounding an integer count over 549: the adjacent values are 278/549 = 0.506375 and 279/549 = 0.508197. Intermediate rounding or a different effective denominator might explain this, and the aggregation should be specified.

ReasonMap has only four test diagrams. Under the described procedure of averaging runs per item and then exchanging whole-source labels, there are 16 paired swaps. The minimum conventional exact tail probability is 0.0625 for a one-sided test, or 0.125 for a two-sided absolute-difference test. Table 13 reports p = 0.05. Exact enumeration, together with the tail definition, would resolve the discrepancy. The bootstrap should also state how the same source is linked across seeds. These que tions mainly concern transfer accounting and uncertainty estimates.

  1. The comparison with the nearest methods and the projection procedure need more implementation detail. MPO already jointly optimizes textual and non-textual prompts, while Dynamo evolves visual tools with paired skills (Choi et al., 2026; Sun et al., 2026). ViewCraft's more specific contribution is its evidence-interface state and retention procedure. The same-space comparisons are useful, but the paper does not fully specify the proposal grammar, initial states, optimizer/verifier prompts, or baseline adaptations needed to reconstruct Table 4.

Algorithm 1 also says to reuse validated rendering after restoring the incumbent reader, although the reader controls subsequent zoom/view requests. It is unclear whether this means reusing renderer validation or replaying the rejected proposal's observation trajectory. The projected skill should be evaluated using he request policy it will use at deployment.

Questions for authors
  1. Can Table 5 be regenerated from the per-run correct counts and denominators? Please specify the repetition counts, weights, and rounding order, particularly for Sonnet self-transfer on VideoHolmes.

  2. What statistic and tail definition produced the ReasonMap p-value? Please enumerate the 16 whole-diagram swaps under the stated protocol and explain how the bootstrap preserves source identity across seeds.

  3. Which original MPO and Dynamo mechanisms are retained in the adaptations? Please provide the existing proposal grammar/interpreter, starting states, prompts, UCB settings, and per-task/seed budgets needed to reconstruct the comparison.

  4. Is the projected state executed afresh with the incumbent reader's requests? What is reused at Algorithm 1, line 13? A rejected joint trace and its successful projection would clarify this. Please also specify whether evidence support is a hard guard or only a tiebreak/projection signal.

Reviewer_jtP2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper jointly optimizes visual evidence selection, transformation, and presentation together with the instructions a frozen multimodal model uses to interpret that evidence. It uses execution feedback and paired evaluation to refine reusable skills, including a mechanism that retains useful visual edits from rejected joint updates.

Critical strengths
  1. The paper reports good empirical improvements across eight benchmarks. They remain positive against an input-token-matched baseline.
  2. There is substantial diagnostic evaluation including component ablations, alternative search strategies, and transfers across three executors which help distinguish the contribution of visual preparation from that of reading instructions.
Critical weaknesses
  1. The expression of the paper made it quite difficult to follow. Certain vague phrases are introduced without clear connection to the manuscript until reading much further in. ("interface contracts", "projection", "coordinate partition", etc.). Even the first sentence overloads a large amount of unexplained jargon ("jointly evolving controls obscure causal attribution and can entangle adaptation with held-out feedback"). The results in the paper seem strong but the expression makes it quite difficult to parse / adds unnecessary complexity in simply following the work.

  2. Equation 2 is a probability decomposition, not an explanation of how to construct better inputs. The optimizer prompts and expression interpreter are also needed to reproduce the search but are not supplied in the PDF.

  3. The novelty is somewhat limited, as Choi et al.’s Multimodal Prompt Optimization already jointly optimizes textual and non-textual prompts; Sun et al.’s Dynamo already evolves reusable reasoning skills alongside executable visual tools. Both are acknowledged here. ViewCraft’s novelty therefore rests principally on its particular representation and filtering/retry rules, rather than joint visual-textual evolution itself. Its adapted baseline implementations should be better justified, as a result.

Questions for authors
  1. Can you provide one complete evolution round, showing the original views/instructions, proposed edits, measured acceptance scores, and interface-only retry, alongside the exact optimizer prompt and rendering rules?

  2. What capability absent from MPO and Dynamo explains the remaining advantage? The baseline adaptations should be ideally documented + show which changes preserve or restrict the original methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.