- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VisualCoding, a framework for turning repeated visual task patterns into reusable executable programs. For each known task family, a writer model uses labeled training examples to construct and iteratively refine a Python solver, while an additional scope rule decides when the solver is reliable enough to use. The authors claim three main contributions: (1) execution-guided acquisition of family-level visual solvers rather than only textual skills, (2) scope-aware direct execution that adopts reliable program outputs without requiring the deployment model to reinterpret them, and (3) reuse of the acquired programs across different executors.
At test time, in-scope program outputs are directly adopted as answers, otherwise the system falls back to a text skill or the base agent. The method is evaluated on TIR-Bench, MIRA, GameQA, and PuzzleVQA, including both the original strong model and weaker/different deployment models. The main empirical result is that directly executing retained programs is generally more reliable than asking an agent to reinterpret textual skills or invoke the same programs through another reasoning step, with the largest gains appearing on highly structured task families such as jigsaw, maze, and geometric matching problems.
Critical strengths
-
The paper is empirically careful. Beyond the main tables, the authors report family-level results, acquisition and deployment ablations, scope analyses, sampling-based comparisons, and detailed implementation information in the appendix. This makes it possible to inspect which gains come from particular task families and which components are actually responsible for the improvements, rather than relying only on aggregate accuracy.
-
The paper provides useful evidence that directly reusing an acquired executable procedure can be more reliable than asking an agent to reinterpret or re-apply the same procedure at test time. The deployment ablations compare textual reuse, callable-program reuse, and direct execution of the same acquired programs, and the scope mechanism provides a practical fallback when a program is not considered reliable. I view these as useful engineering observations, even though they do not, in my opinion, amount to a sufficiently strong or novel contribution for acceptance.
Critical weaknesses
-
The novelty is limited relative to closely related prior work, and the remaining methodological contribution appears incremental. The central idea of having a strong model synthesize reusable Python tools for a recurring task class and then reusing them with cheaper or different models has clear precedent in Large Language Models as Tool Makers (Cai et al., ICLR 2024). CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models (Qian et al., Findings of EMNLP 2023) already studies executable tool creation with execution-based rectification, while CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (Yuan et al., ICLR 2024) constructs and validates reusable executable tools from training examples and evaluates them on VQA. None of these closely related works is cited or compared. VisualCoding adds a precision-constrained scope rule and a more systematic cross-instance refinement procedure, but the measured contribution of the latter is modest: FULLC improves over independent search by only +2.4 points on TIR and +0.6 on MIRA, while the first jigsaw candidate already solves 35/40 training examples. I therefore view the methodological advance over this prior line of work as fairly limited.
-
The empirical success is concentrated on highly regular, family-specific tasks, which limits the significance of the claimed improvement in visual reasoning. The method assumes known task families and fits a frozen solver to each family using labeled examples; its strongest cases are tasks such as jigsaw assembly, maze search, mirror matching, and geometric measurement, where nearly the same algorithmic structure can be reused across instances. The headline TIR improvement is heavily dominated by jigsaw, while several less rigid families show little gain or degradation. The cross-executor results also provide weaker evidence of general transfer than the main text suggests: by construction, DeepSeek and GPT reuse the same frozen Flash-Lite routes and stored program readings (App. C.3), so the program takeover set and program outputs are shared across executors. Moreover, Flash-Lite-specific calibration enables program routes for 40 of 69 families, compared with 27 under the Pro policy (Table 9), meaning that part of the large Flash-Lite gain comes from routing more families to already-frozen program answers rather than from improved reasoning by the target executor. Overall, the experiments demonstrate effective reuse of task-family-specific programs, but provide limited evidence for broader or open-ended visual reasoning ability.
Questions for authors
Could the authors clarify what they see as the key conceptual distinction between the proposed framework and the closely related prior work discussed above, particularly existing program-based and verification-based approaches? More importantly, the paper frames the method as improving visual reasoning ability, but it is not yet clear to me that the current evidence isolates an improvement in visual reasoning itself, rather than an improvement from externalizing the task into executable programs. Is there additional evidence or analysis showing that the proposed method actually improves the model's visual reasoning capability?