acceptodds
Withdrawn from ICLR 2027

Trace2Prog: Compiling and Calibrating Agent Experience into Reusable Visual Programs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recurring visual question answering (QA), different questions may share the same solution procedure despite changes in their visual inputs. However, multimodal agents often reconstruct this procedure from scratch for every question. Reusing compiled procedures can reduce repeated processing, but a program that worked before may fail under new visual conditions. The key challenge is deciding when to use a compiled program and when to fall back to the agent. To this end, we introduce Trace2Prog, a compile–calibrate–deploy framework for recurring visual QA. It compiles past executions into candidate programs, validates them under the source visual condition, and uses a small labeled target set to determine whether they remain suitable for reuse. For each new question, Trace2Prog uses a validated program when suitable and otherwise falls back to the multimodal agent. On 480 controlled requests across four task types and three renderer profiles, Trace2Prog achieves 93.1% accuracy, compared with 48.3% for the baseline tool agent, 86.7% for per-query program synthesis, and 87.1% for success-only compilation. On 3,215 TIR-Bench and PuzzleVQA requests, Trace2Prog improves overall accuracy by 0.25–2.18 percentage points without introducing observed additional errors. Across 2,160 program–request pairs, target calibration reduces harmful replacements from 87 to 24.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multimodal agents, program synthesis, skill reuse, visual reasoning, selective deployment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7kq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trace2Prog, which compiles the historical execution experience of agents into reusable Python programs. The accuracy and quality of each program are verified in a small number of sandbox environments, and a small amount of labeled data is used to test whether it can outperform the agent, which determines whether it is retained for reuse. At test time, a lightweight VLM router selects, for each task, either an appropriate reusable program or fallback agent to act. The method ultimately surpasses the original agent on controlled synthetic tasks and shows preliminary gains on real-world benchmarks.

Critical strengths
  • S1. A meaningful research question with practical value. The paper addresses the problem that visual agents must repeatedly reason over and execute code for recurring tasks, and proposes a training-free experience-reuse framework that improves performance. By cutting redundant overhead, the approach has genuine practical value for real-world settings.
  • S2. An in-depth exploration of the reliability of experience reuse. The paper goes beyond how to compile historical experience into reusable programs, examining how to assess program quality, decide whether a program is worth substituting for the original agent, and determine when deployment is appropriate. In particular, measuring the practical benefit of experience reuse through Fix, Hurt, and Net offers an interesting and valuable perspective on reliable agent experience reuse.
  • S3. Clear problem definition and well-organized presentation. The paper is well structured and easy to follow. The three-stage Compile–Calibrate–Deploy framework is logically coherent and readable.
Critical weaknesses
  • W1. The method achieves gains in controlled environments but only limited gains on the full TIR-Bench and PuzzleVQA. Moreover, the experiments concentrate on visual puzzles and other highly rule-based tasks, with no validation on more open-ended tasks such as mathematical reasoning or complex tool use, which limits the method's practical value and generalization in real-world.
  • W2. The main expeirments compare mainly against baselines such as Per-query Synthesis and Success-only Compilation, lacking systematic comparison with recent agent experence-reuse methods (e.g., MMSkills, FlowEvo, CRAFT). The current experiments are insuficient to demonstrate a clear advantage of Trace2Prog over these recent approaches.
  • W3. The method requires labeled samples from the target environment to judge whether a program is worth reusing, and the types of target visual variations in the controlled experiments already appear during calibration. The effectiveness of Trace2Prog under entirely novel visual conditions is not sufficently demonstrated.

[1] MMSkills: Towards Multimodal Skills for General Visual Agents

[2] FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills

[3] CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets

Questions for authors
  • Q1. The method achieves very large gains in controlled experiments but only limited gains on the full benchmarks. Is this mainly because the reusable programs have low task coverage? How would coverage and actual gains change on mathematcal reasoning, code execution, or more complex agent tasks? Could this limit the method's real-world applicability?

  • Q2. The paper mainly relies on strong API models for program compilation and agent reasoning. Does the framework remain applicable to smaller open-source models (e.g., 4B/8B)? How would models of differing capability affect program quality and the final gains?

Reviewer_dPgbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Trace2Prog for recurring visual QA. First an LLM compiler converts the trajectory of agent into standalone programs of solving procedure. A small set of held out data are used to select these frozen programs. When deploying, a VLM router and a classifier are used to judge whether a QA can reuse these programs or need falling back to the multimodal agent. This system is evaluated on 480 controlled requests, and almost doubles the accuracy from 48.3% (direct multimodal agent) to 93.1%.

Critical strengths
  1. Good balance of performance and efficiency. Trace2Prog achieves higher performance while reduces cost by reusing calibrated programs.
  2. No hurt observed during deployment. Table 1 shows label-free deployment adds 8-70 correct answers with no observed hurt and zero wrong contract deployment.
Critical weaknesses
  1. The contribution of calibration is relatively small. On 480 controlled requests, the performance gain mostly shows the role of contract selection and weak fallback, while evidence to support the importance of calibration is weak.
  2. Overfitting concern on controlled set. On TIR-Bench and PuzzleVQA, Trace2Prog's improvement is only 0.25–2.18%, which is very small compared with 44.8% improvement on 480 controlled requests.
  3. Trace2Prog performs worse than the cached experience XSkill adaptation (59 vs 55 on TIR-Bench, 141 vs 190 on PuzzleVQA).
Questions for authors
  1. Can you report Track II results on contracts chosen without development stage inspection (e.g. randomly sampled TIR-Bench or PuzzleVQA that meets the data requirements)?
  2. What's the token budget for agent in each track, and XSkill? Can you give cost comparisons (compile + calibrate + route) versus XSkill?
Reviewer_bLHwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a framework for reusing past successful programs in new settings for visual programming. They propose a compile-calibrate-deploy method that first collects past agent traces into standalone programs, which are tested on held-out samples to ensure they perform better than a baseline fallback agent, and are then used directly to solve new problems where applicable, automatically falling back to a baseline agent if the program fails, abstains, or yields invalid outputs.

Critical strengths
  • The authors address a real and critical problem in visual programming: programs are not always reusable and can sometimes lead to a correct answer by accident, diminishing their usability and undermining their motivation.
  • The framework's decoupling of the different components (compile-calibrate-deploy) seems to have interesting applications, such as using a stronger model for compilation and a weaker routing model for deployment. However, beyond the default setup of using a stronger model for compilation and a weaker model for deployment, there was no significant further analysis of any potential trade-offs.
Critical weaknesses
  • I am concerned that the novelty of the method is only incremental compared to standardized methods in digital agents (e.g., SkillWeaver, ASI) for collecting tested code components and leveraging them as a robust library for subsequent queries.
  • This might just be me, but I am having significant trouble parsing the tables. For example, in Table 1, if the models were evaluated on 1,299 questions, why are the results reported over the full 3,215 samples? I understand the need to report final numbers on a full benchmark, but it might be better to define explicit splits and report the numbers more intuitively.

Zheng, Boyuan, et al. "Skillweaver: Web agents can self-improve by discovering and honing skills." arXiv preprint arXiv:2504.07079 (2025).

Wang, Zora Zhiruo, et al. "Inducing programmatic skills for agentic tasks." COLM 2025.

Questions for authors
  • I am unclear on what the exact contribution of this work is. Is it the decoupled framework? Is it applying skill induction to static visual reasoning tasks? If it is the former, it would have been interesting to see different ablations with various models to understand the benefits of such a decoupled framework. If it is the latter, it would be useful to explicitly highlight the contribution beyond what is already established in digital agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.