acceptodds
Withdrawn from ICLR 2027

VisualCoding: Executable Skills for Reliable Agentic Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and produce the final answer. Errors and hallucinations at these steps can prevent acquired experience from translating into reliable single-run performance. We propose VisualCoding, which combines execution-guided acquisition of reusable visual programs with direct adoption of their answers on applicable inputs. Execution-Guided Program Acquisition composes visual primitives and additional code into family-level solvers, then iteratively tests and revises them on labeled training instances. Scope-Aware Direct Execution runs a retained solver and returns its valid in-scope answer. We evaluate VisualCoding on four models across four benchmarks spanning 69 task families. VisualCoding exceeds the strongest evaluated baseline in mean accuracy by 4.45 percentage points with Gemini-3.1-Pro and by 14.78 points when reusing the frozen Pro-acquired artifacts with Flash-Lite. Ablations support execution-guided acquisition and show that direct execution outperforms textual or callable reuse of the same programs. Code is available at: https://anonymous.4open.science/r/VisualCoding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Agentic Learning, Visual Reasoning, Agent System

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jBKVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VisualCoding, a framework for turning repeated visual task patterns into reusable executable programs. For each known task family, a writer model uses labeled training examples to construct and iteratively refine a Python solver, while an additional scope rule decides when the solver is reliable enough to use. The authors claim three main contributions: (1) execution-guided acquisition of family-level visual solvers rather than only textual skills, (2) scope-aware direct execution that adopts reliable program outputs without requiring the deployment model to reinterpret them, and (3) reuse of the acquired programs across different executors.

At test time, in-scope program outputs are directly adopted as answers, otherwise the system falls back to a text skill or the base agent. The method is evaluated on TIR-Bench, MIRA, GameQA, and PuzzleVQA, including both the original strong model and weaker/different deployment models. The main empirical result is that directly executing retained programs is generally more reliable than asking an agent to reinterpret textual skills or invoke the same programs through another reasoning step, with the largest gains appearing on highly structured task families such as jigsaw, maze, and geometric matching problems.

Critical strengths
  1. The paper is empirically careful. Beyond the main tables, the authors report family-level results, acquisition and deployment ablations, scope analyses, sampling-based comparisons, and detailed implementation information in the appendix. This makes it possible to inspect which gains come from particular task families and which components are actually responsible for the improvements, rather than relying only on aggregate accuracy.

  2. The paper provides useful evidence that directly reusing an acquired executable procedure can be more reliable than asking an agent to reinterpret or re-apply the same procedure at test time. The deployment ablations compare textual reuse, callable-program reuse, and direct execution of the same acquired programs, and the scope mechanism provides a practical fallback when a program is not considered reliable. I view these as useful engineering observations, even though they do not, in my opinion, amount to a sufficiently strong or novel contribution for acceptance.

Critical weaknesses
  1. The novelty is limited relative to closely related prior work, and the remaining methodological contribution appears incremental. The central idea of having a strong model synthesize reusable Python tools for a recurring task class and then reusing them with cheaper or different models has clear precedent in Large Language Models as Tool Makers (Cai et al., ICLR 2024). CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models (Qian et al., Findings of EMNLP 2023) already studies executable tool creation with execution-based rectification, while CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (Yuan et al., ICLR 2024) constructs and validates reusable executable tools from training examples and evaluates them on VQA. None of these closely related works is cited or compared. VisualCoding adds a precision-constrained scope rule and a more systematic cross-instance refinement procedure, but the measured contribution of the latter is modest: FULLC improves over independent search by only +2.4 points on TIR and +0.6 on MIRA, while the first jigsaw candidate already solves 35/40 training examples. I therefore view the methodological advance over this prior line of work as fairly limited.

  2. The empirical success is concentrated on highly regular, family-specific tasks, which limits the significance of the claimed improvement in visual reasoning. The method assumes known task families and fits a frozen solver to each family using labeled examples; its strongest cases are tasks such as jigsaw assembly, maze search, mirror matching, and geometric measurement, where nearly the same algorithmic structure can be reused across instances. The headline TIR improvement is heavily dominated by jigsaw, while several less rigid families show little gain or degradation. The cross-executor results also provide weaker evidence of general transfer than the main text suggests: by construction, DeepSeek and GPT reuse the same frozen Flash-Lite routes and stored program readings (App. C.3), so the program takeover set and program outputs are shared across executors. Moreover, Flash-Lite-specific calibration enables program routes for 40 of 69 families, compared with 27 under the Pro policy (Table 9), meaning that part of the large Flash-Lite gain comes from routing more families to already-frozen program answers rather than from improved reasoning by the target executor. Overall, the experiments demonstrate effective reuse of task-family-specific programs, but provide limited evidence for broader or open-ended visual reasoning ability.

Questions for authors

Could the authors clarify what they see as the key conceptual distinction between the proposed framework and the closely related prior work discussed above, particularly existing program-based and verification-based approaches? More importantly, the paper frames the method as improving visual reasoning ability, but it is not yet clear to me that the current evidence isolates an improvement in visual reasoning itself, rather than an improvement from externalizing the task into executable programs. Is there additional evidence or analysis showing that the proposed method actually improves the model's visual reasoning capability?

Reviewer_Wv1rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces VisualCoding, a framework for building reusable programs for known visual task families. A strong model writes and revises these programs using execution feedback from labeled training examples. At test time, the system returns the program’s answer when validity and scope checks pass; otherwise, it falls back to a base agent. Experiments on four benchmarks, using subsets in some cases, report improvements over tool-based agents and text-skill methods, although several important comparisons are missing.

Critical strengths

The paper addresses a practical problem: agents may fail to correctly implement a useful procedure or report its computed answer. Retaining executable programs and directly using their outputs is a sensible design, and the experiments show clear gains on some task families. However, executable skill/experience/knowledge reuse has been widely explored, and the related-work discussion does not fully clarify the additional contribution.

Critical weaknesses
  1. Important comparisons are missing, including existing visual programming methods, small open-weight VLMs with SFT or distillation, and a simple family-level solver built with the same data and budget.
  2. The evidence for generalizable skill learning is limited. Programs are developed using labeled examples from known task families and tested on further examples from those same families.
Questions for authors
  1. A key baseline is missing: ask the same big model to write a solver for each task family, test and revise it using the same training data and budget, and then freeze it. Without this comparison, it is unclear whether the gains come from the proposed framework or mainly from preparing dedicated code for known task types.

  2. ViperGPT, VisProg, and TVP already study visual program execution or reusable tools (cited in this manuscript). VisReP (CVPR24) and DWIM (ICCV25) also study learning from program execution or tool-use experience. The paper should explain its differences more clearly and compare with relevant methods. Calling the retained programs “skills” does not by itself establish a new contribution.

  3. The experiments do not include small open-weight VLMs, such as Qwen or InternVL models around 7B–8B. Relevant methods such as Beacon and PyVision-RL should also be considered. These models are tested in the benchmark you used (or public available SOTA). Since the proposed method uses training labels and substantial Pro computation, comparisons with SFT or distillation under comparable data and resource conditions would help establish its value.

  4. Different deployment models reuse the same programs and stored program outputs. On these questions, the weaker model can be bypassed. This shows that an external solver can support different models, but does not show that weaker models learn to use skills better. It also remains unclear whether smaller models can acquire useful programs without a stronger model.

  5. Dataset selection could be clearly explained if possible. For example, GameQA contains around 140K questions, including 15,047 test questions, while this paper uses only 514 test questions from 17 families. Appendix C.1 reports the split sizes but does not clearly explain the selection process, excluded tasks, or difficulty distribution. The full set testing would be more convinced.

  6. In Table 11, jigsaw accounts for about 90% (table is misaligned, hard to read) of the net improvement over the tool agent on TIR-Bench, while some other families show lower accuracy.

  7. Programs are acquired and deployed separately for known task families using predefined routes. The paper does not clearly test whether the system can select the right skill from mixed inputs. It also lacks tests on unseen task types, new combinations, changed visual styles, or harder instances.

  8. How does this method differ from existing self-evolving models or agents that improve their programs and tools through execution feedback? A comparison under the same training data and computation budget would help clarify the contribution. Also, what are its advantages over using a strong model to generate in-context examples during training and retrieving relevant examples at test time? Comparisons using the same training data and computation budget would help clarify the contribution.

Reviewer_f7JzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VisualCoding uses labeled examples and execution feedback to build reusable programs for known visual task families. At deployment, it runs the frozen program and returns its answer when validity and scope checks pass. Experiments on four benchmarks examine how programs are acquired, how their outputs are used, and how the frozen programs and routing rules transfer across executors.

Critical strengths
  • The deployment ablations reuse the same programs. TIR-Bench accuracy is 76.3% with direct execution, 69.4% with textual reuse, and 73.2% with callable code. Returning a computed answer to the model for continuation loses a net six correct answers on both TIR-Bench and MIRA. These controls locate errors in how an existing program is used.
  • Flash-Lite improves from 43.42% to 61.92% with the frozen Pro policy, then reaches 68.57% with its own calibration. Transfer therefore helps before target-specific route calibration, though recalibration adds further gains. The comparison keeps the programs’ internal dependencies fixed.
Critical weaknesses
  • On MIRA’s puzzle family, the program takes over all 17 test questions but scores 23.5%, compared with 52.9% for the tool agent. This is a substantial regression that passes both scope and deployment checks. To assess whether it is an isolated failure, I would like to see test-time adoption coverage, accepted-output accuracy, and counts of corrected and harmed answers for each benchmark. The no-scope acquisition ablation changes the programs, so it does not isolate the contribution of scope filtering for the frozen programs.
  • The acquisition results leave me less certain about the value of detailed execution feedback. On MIRA, full feedback reaches 46.6%, independent search 46.0%, and aggregate-only feedback 46.5%. These are three-run means without individual results or variability estimates. TIR-Bench shows a larger gain, but the paper does not establish how consistently feedback-guided revision improves on simpler search and feedback. This matters because program acquisition is a central part of the proposed method.
Questions for authors
  • What failed on MIRA’s puzzle family? Can you report coverage, accepted-output accuracy, and corrected versus harmed answer counts for each benchmark?
  • What are the three individual acquisition results, and how much does the full-feedback advantage over independent search and aggregate-only feedback vary across runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.